diff --git a/.agents/skills/ai-chat/SKILL.md b/.agents/skills/ai-chat/SKILL.md new file mode 100644 index 0000000000..f670b2ffb7 --- /dev/null +++ b/.agents/skills/ai-chat/SKILL.md @@ -0,0 +1,40 @@ +--- +name: ai-chat +description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window. +--- + +## Always benchmark before and after + +No context or behavior change ships without an `ai_evals` A/B on the affected mode. +Add or adjust cases for exactly what you changed — see the `ai-evals` skill for +authoring and the full run reference. + +Run the affected mode **before** your change and **after**, same model(s), same cases. + +## Measure the window first, and cumulative second + +Optimize **`finalContextTokens`** (window occupancy — what drives overflow and +compaction), then cumulative prompt tokens. + +## Context discipline + +The dominant fixed cost is per-iteration overhead: the system prompt **plus every +tool schema** is re-sent on every loop iteration. So: + +- **Every tool and every parameter is a permanent tax.** Justify each one and measure + it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead + params rather than leaving them in the schema. +- **Tool results return the minimum.** Never echo content the model already has. The + canonical mistake: a write tool that returns the whole edited artifact right after + the model authored it — return `{ success, message }` instead. When you touch a + *shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check + this invariant for **all** the write tools routing through it — the echo has + regressed before via a shared refactor. + +## Prompts and tool descriptions are part of the surface + +The system prompt and tool descriptions steer behavior as much as the tools +themselves, and are benchmarkable the same way. A description that advertises +truncation makes the model self-limit; the path-conventions block changes where +drafts land. Treat prompt/description edits as real changes and A/B them — a +pure-prompt change is a legitimate, measurable improvement. \ No newline at end of file diff --git a/.agents/skills/ai-evals/SKILL.md b/.agents/skills/ai-evals/SKILL.md new file mode 100644 index 0000000000..ad334e2c6b --- /dev/null +++ b/.agents/skills/ai-evals/SKILL.md @@ -0,0 +1,87 @@ +--- +name: ai-evals +description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes. +--- + +# AI evals — authoring and running benchmark cases + +`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes: +`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production +prompts, tools, and guidance in this checkout. Each attempt runs the real production +path, deterministic validation, then LLM judging. + +The goal is to test current production guidance with realistic user requests — **not** +to pin one exact implementation shape. + +## Running benchmarks + +```bash +cd ai_evals +bun install # first time; frontend modes also need `cd frontend && bun install` +bun run cli -- models # list model aliases +bun run cli -- cases global # list cases for a mode +bun run cli -- run global global-test1-script-create --model sonnet +``` + +Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill +backend's `/api/w//ai/proxy`, so you need **any** reachable backend: + +```bash +WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1: WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \ + bun run cli -- run global --models sonnet,gpt-5.5,gemini-3.1-pro-preview +``` + +- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace + creation 400s ("reached workspace limit"). Always set + `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to + reuse one. The only side effect of a run is upserting an `f/evals/ai/` + resource there. +- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a + separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under + test. + +## Authoring core rules + +1. Write prompts like a real user request. +2. Prefer behavior, inputs, constraints, and outcomes over internal implementation. +3. Keep deterministic validation narrow and hard. +4. Put semantic expectations in `judgeChecklist`. +5. Use `expected` fixtures only when exact structure really matters. + +### Prompt writing + +Prompts should sound like something a user would naturally ask. Do not write prompts +as if the user knows Windmill internals unless the case explicitly tests a power-user +workflow. + +Good: +- "Create a flow that routes support requests based on customer tier." +- "Add a reset button that sets the counter back to 0." +- "Create a flow that reuses the existing greeting script instead of duplicating the logic." + +Bad: +- "Use `branchone` with 3 branches and a default branch." +- "Create a `rawscript` step with this exact topology." +- "This is a benchmark harness." + +### Deterministic validation + +Use deterministic checks only for hard failures: missing required files; unexpected +extra files when the prompt says not to create them; syntax errors; unresolved flow +refs; missing required special modules or suspend config; obvious corruption. + +Do **not** encode one preferred implementation. Bad hard checks: exact step topology +for a creation flow; exact branch structure when the prompt only asked for routing; +exact input shape when multiple reasonable shapes are acceptable. + +### Judge checklist + +Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior +that must be present, important constraints, and key completion criteria — not +low-level implementation details unless truly required. + +Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing +workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a +`rawscript` node". + +See `ai_evals/README.md` for the full case format, fields, and fixture details. \ No newline at end of file diff --git a/.claude/skills/ai-chat/SKILL.md b/.claude/skills/ai-chat/SKILL.md new file mode 120000 index 0000000000..f8c527f3c9 --- /dev/null +++ b/.claude/skills/ai-chat/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-chat/SKILL.md \ No newline at end of file diff --git a/.claude/skills/ai-evals/SKILL.md b/.claude/skills/ai-evals/SKILL.md new file mode 120000 index 0000000000..8b714cb50d --- /dev/null +++ b/.claude/skills/ai-evals/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-evals/SKILL.md \ No newline at end of file diff --git a/.github/DockerfileBackendTests b/.github/DockerfileBackendTests index 88275f204b..9b29f72a64 100644 --- a/.github/DockerfileBackendTests +++ b/.github/DockerfileBackendTests @@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv ENV TZ=Etc/UTC diff --git a/.github/workflows/ai-agent-tests.yml b/.github/workflows/ai-agent-tests.yml new file mode 100644 index 0000000000..3a5c51f153 --- /dev/null +++ b/.github/workflows/ai-agent-tests.yml @@ -0,0 +1,132 @@ +name: AI Agent Integration Tests + +# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against +# real LLM providers. Runs only when AI-agent backend code or the tests change, +# because each run makes real (paid) LLM calls. To avoid spending on every commit, +# the PR side triggers only when a PR is marked ready for review (out of draft) — +# not on `synchronize` — plus push to main and manual dispatch. +on: + workflow_dispatch: + push: + branches: [main] + paths: + - "integration_tests/ai_agent_tests/**" + - "backend/windmill-ai/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-worker/src/ai_executor.rs" + - "backend/windmill-worker/src/ai/**" + - "backend/windmill-worker/src/memory_common.rs" + - "backend/windmill-common/src/flow_conversations.rs" + - ".github/workflows/ai-agent-tests.yml" + pull_request: + types: [opened, reopened, ready_for_review] + paths: + - "integration_tests/ai_agent_tests/**" + - "backend/windmill-ai/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-worker/src/ai_executor.rs" + - "backend/windmill-worker/src/ai/**" + - "backend/windmill-worker/src/memory_common.rs" + - "backend/windmill-common/src/flow_conversations.rs" + - ".github/workflows/ai-agent-tests.yml" + +concurrency: + group: ai-agent-tests-${{ github.ref }} + cancel-in-progress: true + +jobs: + ai_agent_e2e: + # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while + # still a draft. `ready_for_review` always arrives non-draft. + if: github.event_name != 'pull_request' || github.event.pull_request.draft == false + runs-on: ubicloud-standard-16 + services: + postgres: + image: postgres:16 + ports: + - 5432:5432 + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache-workspaces: backend + toolchain: 1.93.0 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.10 + + - uses: actions/setup-node@v4 + with: + node-version: "20" + + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + + # CE build (no enterprise/license needed for AI agents). `quickjs` powers + # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool + # test. Bun tool scripts run via the always-on worker (BUN_PATH). + - name: Build Windmill + working-directory: ./backend + env: + SQLX_OFFLINE: true + CARGO_BUILD_JOBS: 12 + RUSTFLAGS: "" + run: cargo build --features quickjs,mcp + + - name: Start Windmill + working-directory: ./backend + env: + DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill + BUN_PATH: bun + NODE_BIN_PATH: node + RUST_LOG: info + run: | + mkdir -p ../integration_tests/logs + ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 & + echo "Waiting for Windmill to be ready..." + for i in $(seq 1 60); do + if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then + echo "Windmill is ready" + break + fi + sleep 2 + done + curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; } + + - name: Run AI agent integration tests + timeout-minutes: 20 + working-directory: ./integration_tests/ai_agent_tests + env: + WINDMILL_URL: http://localhost:8000 + # Only the providers we have org secrets for. Other providers + # (Azure, Bedrock, OpenRouter) are skipped by conftest when their + # keys are absent — see skip_provider_without_credentials. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }} + run: | + python -m venv .venv + .venv/bin/pip install -r requirements.txt + # The S3/vision-attachment tests need MinIO large-file storage and + # image-capable provider setup; out of scope for this cost-controlled + # smoke. Add MinIO secrets + a storage service to enable them. + .venv/bin/python -m pytest -v \ + --ignore=test_user_attachments.py \ + --ignore=test_user_images.py \ + --ignore=test_image_output.py + + - name: Archive Windmill logs + uses: actions/upload-artifact@v4 + if: always() + with: + name: ai-agent-tests-windmill-logs + path: integration_tests/logs diff --git a/.github/workflows/ai-evals-test.yml b/.github/workflows/ai-evals-test.yml new file mode 100644 index 0000000000..a255a4df94 --- /dev/null +++ b/.github/workflows/ai-evals-test.yml @@ -0,0 +1,166 @@ +name: AI Evals (global mode) + +# Smoke-tests the production global AI chat proxy/frontend execution path via +# the ai_evals harness, one case across one cheap model per provider. Runs only +# when the eval harness or the global chat code change, since each run makes real +# (paid) LLM calls. The backend is built from source purely as the AI proxy the +# harness routes model calls through; the global tools/drafts run in-process in +# the Vitest bridge against production frontend code. To avoid spending on every +# commit, the PR side triggers only when a PR is marked ready for review (out of +# draft) — not on `synchronize` — plus push to main and manual dispatch. +on: + workflow_dispatch: + push: + branches: [main] + paths: + - "ai_evals/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-ai/**" + - "frontend/src/lib/components/copilot/**" + # The eval harness runs production frontend code in-process; these are the + # AI/draft-specific deps outside copilot/ that the global smoke exercises. + - "frontend/src/lib/userDraft.svelte.ts" + - "frontend/src/lib/userDraftDbSyncer.svelte.ts" + - "frontend/src/lib/infer.ts" + - ".github/workflows/ai-evals-test.yml" + pull_request: + types: [opened, reopened, ready_for_review] + paths: + - "ai_evals/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-ai/**" + - "frontend/src/lib/components/copilot/**" + # The eval harness runs production frontend code in-process; these are the + # AI/draft-specific deps outside copilot/ that the global smoke exercises. + - "frontend/src/lib/userDraft.svelte.ts" + - "frontend/src/lib/userDraftDbSyncer.svelte.ts" + - "frontend/src/lib/infer.ts" + - ".github/workflows/ai-evals-test.yml" + +concurrency: + group: ai-evals-test-${{ github.ref }} + cancel-in-progress: true + +jobs: + ai_evals_global: + # Provider secrets are unavailable to forked and Dependabot PRs. + if: >- + github.event_name != 'pull_request' || + ( + github.event.pull_request.draft == false && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.user.login != 'dependabot[bot]' + ) + runs-on: ubicloud-standard-16 + services: + postgres: + image: postgres:16 + ports: + - 5432:5432 + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache-workspaces: backend + toolchain: 1.93.0 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.10 + + - uses: actions/setup-node@v4 + with: + # Node 22.19+ is required by the frontend's undici 8.x, which the + # Vitest bridge loads; Node 20 fails with markAsUncloneable. + node-version: "22" + + # CE build used only as the AI proxy (login, workspace, provider resource, + # /ai/proxy). No worker execution or MCP needed — global tools/drafts run + # in the Vitest bridge. quickjs matches the standard CE feature set. + - name: Build Windmill (AI proxy) + working-directory: ./backend + env: + SQLX_OFFLINE: true + CARGO_BUILD_JOBS: 12 + RUSTFLAGS: "" + run: cargo build --features quickjs + + - name: Start Windmill + working-directory: ./backend + env: + DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill + RUST_LOG: info + run: | + mkdir -p ../ai_evals/logs + ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 & + echo "Waiting for Windmill to be ready..." + for i in $(seq 1 60); do + if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then + echo "Windmill is ready" + break + fi + sleep 2 + done + curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; } + + - name: Install frontend deps + generate client + working-directory: ./frontend + run: | + npm ci + npm run generate-backend-client + + - name: Run global AI evals + timeout-minutes: 20 + working-directory: ./ai_evals + env: + WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000 + WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests + # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }} + DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }} + run: | + bun install + mkdir -p results + # One cheap model per provider (anthropic/openai/googleai/deepseek). + fail=0 + for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do + echo "::group::global-test1-script-create ($m)" + if ! bun run cli -- run global global-test1-script-create \ + --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then + echo "$m: harness/proxy errored" + fail=1 + echo "::endgroup::" + continue + fi + # The CLI exits 0 when the harness records failed attempts, so gate + # on execution-only pass counts while ignoring model output quality. + if jq -e \ + '.attemptCount > 0 and .passedAttempts == .attemptCount' \ + "results/ci-$m.json" > /dev/null; then + echo "$m: OK — proxy/frontend execution completed" + else + echo "$m: FAILED proxy/frontend execution" + jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true + fail=1 + fi + echo "::endgroup::" + done + [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; } + + - name: Archive logs and results + uses: actions/upload-artifact@v4 + if: always() + with: + name: ai-evals-global-logs + path: | + ai_evals/logs + ai_evals/results diff --git a/.github/workflows/backend-test-windows.yml b/.github/workflows/backend-test-windows.yml index 1c73e5d429..7065547b28 100644 --- a/.github/workflows/backend-test-windows.yml +++ b/.github/workflows/backend-test-windows.yml @@ -58,7 +58,9 @@ jobs: - uses: denoland/setup-deno@v2 with: - deno-version: v2.x + # Pin to the Deno version shipped in the runtime image (Dockerfile) so CI + # tests what production runs, instead of floating on the latest v2.x. + deno-version: 2.2.1 - uses: actions/setup-go@v2 with: @@ -74,7 +76,7 @@ jobs: - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: diff --git a/.github/workflows/backend-test.yml b/.github/workflows/backend-test.yml index 8f1f15447c..0be727d4bd 100644 --- a/.github/workflows/backend-test.yml +++ b/.github/workflows/backend-test.yml @@ -50,7 +50,9 @@ jobs: dotnet-version: "9.0.x" - uses: denoland/setup-deno@v2 with: - deno-version: v2.x + # Pin to the Deno version shipped in the runtime image (Dockerfile) so CI + # tests what production runs, instead of floating on the latest v2.x. + deno-version: 2.2.1 - uses: actions/setup-go@v2 with: go-version: 1.21.5 @@ -62,7 +64,7 @@ jobs: node-version: "20" - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: php-version: "8.3" diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index 0d9df9f0ac..f5db652084 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -37,6 +37,44 @@ jobs: with: fetch-depth: 1 + # Make the EE source (the *_ee.rs files in the companion repo) available so the + # reviewer can see EE-only code (e.g. windmill-queue/src/jobs_ee.rs), not just the + # CE surface. The EE ref is read from the PR head's backend/ee-repo-ref.txt (via the + # API, so it reflects the PR's EE pin regardless of which ref is checked out here). + - name: Check EE access + id: ee + env: + EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }} + run: | + if [ -z "$EE_TOKEN" ] || [ -z "$PR_NUMBER" ]; then + echo "available=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + HEAD_SHA=$(gh api "repos/${{ github.repository }}/pulls/$PR_NUMBER" --jq .head.sha) + REF=$(gh api "repos/${{ github.repository }}/contents/backend/ee-repo-ref.txt?ref=$HEAD_SHA" --jq .content | base64 -d | tr -d '[:space:]') + if [ -z "$REF" ]; then + echo "available=false" >> "$GITHUB_OUTPUT" + else + echo "available=true" >> "$GITHUB_OUTPUT" + echo "ee_repo_ref=$REF" >> "$GITHUB_OUTPUT" + fi + + - name: Checkout EE repository + if: steps.ee.outputs.available == 'true' + uses: actions/checkout@v4 + with: + repository: windmill-labs/windmill-ee-private + path: ./windmill-ee-private + ref: ${{ steps.ee.outputs.ee_repo_ref }} + token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} + fetch-depth: 1 + + - name: Substitute EE code + if: steps.ee.outputs.available == 'true' + run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private + - name: Run Claude PR Action uses: anthropics/claude-code-action@v1 with: diff --git a/.github/workflows/refresh-docs-snapshot.yml b/.github/workflows/refresh-docs-snapshot.yml new file mode 100644 index 0000000000..a000f44df8 --- /dev/null +++ b/.github/workflows/refresh-docs-snapshot.yml @@ -0,0 +1,52 @@ +name: Refresh docs snapshot + +# The backend embeds a vendored docs snapshot (backend/windmill-api/docs_snapshot/*.gz) +# so in-product docs search works with no runtime egress. This job re-fetches it from +# windmill.dev on a schedule and opens a PR when it changed, keeping the embedded docs +# fresh independently of the release cadence (the binary embeds whatever is on the +# source tree at build time, so a merged refresh rides into the next release build). +on: + schedule: + - cron: "0 6 * * 1" # Mondays 06:00 UTC + workflow_dispatch: + +jobs: + refresh: + runs-on: ubicloud + permissions: + contents: write + pull-requests: write + steps: + - uses: actions/create-github-app-token@v2 + id: app + with: + app-id: ${{ vars.INTERNAL_APP_ID }} + private-key: ${{ secrets.INTERNAL_APP_KEY }} + - uses: actions/checkout@v4 + with: + token: ${{ steps.app.outputs.token }} + - name: Fetch + re-gzip docs snapshot + run: cd backend/windmill-api/docs_snapshot && ./fetch.sh + - name: Sanity-check the fetched corpus + # curl -f in fetch.sh rejects HTTP errors, but not a valid-but-garbage 200 + # (truncated file, error page). Guard against embedding a broken snapshot. + run: | + cd backend/windmill-api/docs_snapshot + test "$(wc -c < llms-full.txt.gz)" -gt 100000 + test "$(wc -c < llms.txt.gz)" -gt 1000 + pages=$(gzip -dc llms-full.txt.gz | grep -c '^Source:' || true) + echo "pages in snapshot: $pages" + test "${pages:-0}" -ge 200 + - uses: peter-evans/create-pull-request@v6 + with: + token: ${{ steps.app.outputs.token }} + branch: chore/refresh-docs-snapshot + add-paths: backend/windmill-api/docs_snapshot/*.gz + commit-message: "chore: refresh vendored docs snapshot" + title: "chore: refresh vendored docs snapshot" + body: | + Automated refresh of the embedded docs snapshot + (`backend/windmill-api/docs_snapshot/*.gz`) from windmill.dev. + + Review the diff for unexpected churn (a bad upstream docs deploy would + show up as a large drop in pages or content) before merging. diff --git a/CHANGELOG.md b/CHANGELOG.md index 73f6ba3f12..361b6fbd91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,198 @@ # Changelog +## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28) + + +### Features + +* **apps:** add labels input to app editor deploy drawer ([#9828](https://github.com/windmill-labs/windmill/issues/9828)) ([da45e69](https://github.com/windmill-labs/windmill/commit/da45e699c8aefeede172c90769ef4f4b182fec0c)) +* column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) ([#9814](https://github.com/windmill-labs/windmill/issues/9814)) ([003a262](https://github.com/windmill-labs/windmill/commit/003a262a4e9d6c2a63ada01aa8429aea1fbb6031)) + + +### Bug Fixes + +* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e)) +* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d)) +* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62)) + +## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26) + + +### Features + +* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e)) +* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69)) +* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048)) +* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5)) +* capture managed-materialize output schema as asset metadata ([#2](https://github.com/windmill-labs/windmill/issues/2)a) ([#9812](https://github.com/windmill-labs/windmill/issues/9812)) ([ade74b2](https://github.com/windmill-labs/windmill/commit/ade74b297f6a03441e700a20ffc2d7291c8a85fd)) +* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57)) + + +### Bug Fixes + +* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98)) +* **flows:** reject corrupt step paths at deploy + atomic cache writes ([#9751](https://github.com/windmill-labs/windmill/issues/9751)) ([#9813](https://github.com/windmill-labs/windmill/issues/9813)) ([3cda447](https://github.com/windmill-labs/windmill/commit/3cda44762148bcd2ee5c0ea821db884950376ead)) +* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971)) +* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32)) +* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b)) +* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a)) + + +### Performance Improvements + +* **audit:** re-anchor S3 audit export on enable + opt-in backfill ([#9818](https://github.com/windmill-labs/windmill/issues/9818)) ([577ceee](https://github.com/windmill-labs/windmill/commit/577ceeee8679f054c6898d1a7889df30ab830f8f)) + +## [1.740.0](https://github.com/windmill-labs/windmill/compare/v1.739.0...v1.740.0) (2026-06-25) + + +### Features + +* **api:** add structured endpoint for flow logs ([#9797](https://github.com/windmill-labs/windmill/issues/9797)) ([ba768fe](https://github.com/windmill-labs/windmill/commit/ba768fee888682cb50142d6e76c0422c40307f46)) +* bounded-cascade selective execution for pipelines (UI + CLI) ([#9695](https://github.com/windmill-labs/windmill/issues/9695)) ([248540a](https://github.com/windmill-labs/windmill/commit/248540ac4d6e4ee9ee7c3e6f2cc822c63cc6426e)) +* data tests for ducklake pipeline materialization ([#9708](https://github.com/windmill-labs/windmill/issues/9708)) ([f6998ec](https://github.com/windmill-labs/windmill/commit/f6998ec54cba2507703790bf33427e7567d42c4b)) +* detect and guard against deploying stale drafts ([#9768](https://github.com/windmill-labs/windmill/issues/9768)) ([d865518](https://github.com/windmill-labs/windmill/commit/d8655189347f58df9d17e83dc55798baf7964279)) +* ducklake time-travel UX (snapshot history + AT VERSION reads) ([#9709](https://github.com/windmill-labs/windmill/issues/9709)) ([d131d75](https://github.com/windmill-labs/windmill/commit/d131d754e1fc9674abf5de383d2bc93596df9bd1)) +* self-host docs search for chat, mcp, cli; drop inkeep ([#9772](https://github.com/windmill-labs/windmill/issues/9772)) ([9d61e4e](https://github.com/windmill-labs/windmill/commit/9d61e4e59e4101de84217f7c7846f1aa94e84d89)) + + +### Bug Fixes + +* allow hyphens in postgresql database name validation ([#9782](https://github.com/windmill-labs/windmill/issues/9782)) ([170cd79](https://github.com/windmill-labs/windmill/commit/170cd79aaf92152fc3c0f675f155853c7f0e5b25)) +* **debounce:** never supersede a running debounce survivor ([#9780](https://github.com/windmill-labs/windmill/issues/9780)) ([5549bdc](https://github.com/windmill-labs/windmill/commit/5549bdc67a5559a764616c44b1018543bc0568fe)) +* decrypt secret variables via external backend in common resolvers ([#9784](https://github.com/windmill-labs/windmill/issues/9784)) ([cd42c6c](https://github.com/windmill-labs/windmill/commit/cd42c6ca18261328055554788932c3fe876a4a5b)) +* enforce containment of python module dir for preview jobs ([#9704](https://github.com/windmill-labs/windmill/issues/9704)) ([88fca6a](https://github.com/windmill-labs/windmill/commit/88fca6a8c130b9e3b0f0cd410e422d4e074fc11f)) +* **frontend:** apply script editor timeout to preview/Test runs ([#9794](https://github.com/windmill-labs/windmill/issues/9794)) ([6664ce6](https://github.com/windmill-labs/windmill/commit/6664ce6dc0c5fbc283303148de06d6bb85e4acf7)) +* **frontend:** nested-loop "Test this step" resolves iter to innermost loop ([#9778](https://github.com/windmill-labs/windmill/issues/9778)) ([74ebfc6](https://github.com/windmill-labs/windmill/commit/74ebfc67f069047875db738926865bd4bd6fe9e9)) +* opt out of Deno minimum-dependency-age for private npm registries ([#9802](https://github.com/windmill-labs/windmill/issues/9802)) ([b28f974](https://github.com/windmill-labs/windmill/commit/b28f974e5069f635419d9ea56fad6a0e417894e8)) +* pass SSL cert env vars to `uv python install` ([#9790](https://github.com/windmill-labs/windmill/issues/9790)) ([962758c](https://github.com/windmill-labs/windmill/commit/962758c02de5f6d962c681fe9c39769b99429e8d)) +* **python:** re-verify wheel RECORD on local cache reuse (once per worker) ([#9775](https://github.com/windmill-labs/windmill/issues/9775)) ([6c71c33](https://github.com/windmill-labs/windmill/commit/6c71c33470e3ea547f3b994db829eb4d04882443)) +* **python:** serialize concurrent installs into shared wheel cache dir ([#9787](https://github.com/windmill-labs/windmill/issues/9787)) ([11d83ab](https://github.com/windmill-labs/windmill/commit/11d83ab1ec559be5d3263010228e6db65358e04b)) +* re-pin stale-draft fork base when restoring an app deployment ([#9792](https://github.com/windmill-labs/windmill/issues/9792)) ([b9711e5](https://github.com/windmill-labs/windmill/commit/b9711e5ace8585315a1c2b85bb25ac8dd7832d6f)) +* restore libargon2-1 for PHP runtime in server image ([#9795](https://github.com/windmill-labs/windmill/issues/9795)) ([e9cb806](https://github.com/windmill-labs/windmill/commit/e9cb80639b2dec63ede69fc3a4e3720bb1a3c319)) +* use transaction for parallel_monitor_lock DELETE in last-iteration path ([#9789](https://github.com/windmill-labs/windmill/issues/9789)) ([754cae9](https://github.com/windmill-labs/windmill/commit/754cae956ac8d431ddeb055453835e249cdd07b7)) + + +### Performance Improvements + +* drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes ([#9786](https://github.com/windmill-labs/windmill/issues/9786)) ([aa098c7](https://github.com/windmill-labs/windmill/commit/aa098c70c0271b2b1917749d1f607c0559cf04de)) +* eliminate dual-connection DB pool contention across worker, queue, and api ([#9798](https://github.com/windmill-labs/windmill/issues/9798)) ([0dbd9c1](https://github.com/windmill-labs/windmill/commit/0dbd9c1231b00d4693af68835fe1d9e7c8869b43)) + +## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24) + + +### Features + +* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6)) +* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011)) +* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209)) +* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381)) +* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d)) +* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548)) +* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38)) +* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2)) +* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c)) + + +### Bug Fixes + +* **ai-chat:** strip unclosed <summary> tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d)) +* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36)) +* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1)) +* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d)) +* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43)) +* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c)) +* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23)) +* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5)) +* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb)) +* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37)) + + +### Performance Improvements + +* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3)) +* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d)) + +## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23) + + +### Features + +* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc)) +* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b)) + + +### Bug Fixes + +* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a)) +* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e)) +* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9)) +* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57)) +* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7)) +* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3)) +* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6)) + + +### Performance Improvements + +* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10)) + +## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23) + + +### Features + +* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904)) + + +### Bug Fixes + +* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911)) +* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6)) +* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a)) +* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39)) +* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd)) + + +### Performance Improvements + +* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011)) + +## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23) + + +### Features + +* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd)) + + +### Bug Fixes + +* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72)) +* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c)) +* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76)) +* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9)) +* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6)) + +## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22) + + +### Features + +* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2)) +* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039)) +* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f)) +* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb)) + + +### Bug Fixes + +* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d)) +* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2)) +* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f)) +* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126)) +* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654)) +* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8)) + ## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20) diff --git a/Dockerfile b/Dockerfile index d327c9b394..df03ffbdc0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -163,14 +163,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH RUN apt-get update \ - && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \ + && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg libargon2-1 \ && if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* # Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \ - && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ + && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ && apt-get update \ && apt-get install -y --no-install-recommends postgresql-client \ && apt-get clean \ @@ -184,11 +184,11 @@ RUN if [ "$WITH_GIT" = "true" ]; then \ else echo 'Building the image without git'; fi; RUN if [ "$WITH_POWERSHELL" = "true" ]; then \ - if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \ + if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu72 -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \ && rm -rf /var/lib/apt/lists/* && \ dpkg --install 'pwsh.deb' && \ rm 'pwsh.deb'; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu72 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \ && rm -rf /var/lib/apt/lists/* && \ mkdir -p /opt/microsoft/powershell/7 && \ tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \ @@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # Install UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv # Preinstall python runtimes to temp build location (will copy with world-writable perms later) # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run diff --git a/ai_evals/AGENTS.md b/ai_evals/AGENTS.md index af5427abae..6e63c4037e 100644 --- a/ai_evals/AGENTS.md +++ b/ai_evals/AGENTS.md @@ -1,208 +1,14 @@ -# AI Evals Authoring Guide +# AI Evals -This folder contains black-box benchmark cases for: +Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`, +`script`, `cli`, `global`). -- `flow` -- `app` -- `script` -- `cli` -- `global` +**Authoring and running cases is documented in the `ai-evals` skill** — load it +before adding/changing a case or running a benchmark. Claude Code reads +`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read +`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in +Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`. -The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape. +For AI chat / copilot changes that these evals measure, see the `ai-chat` skill. -## Core rules - -1. Write prompts like a real user request. -2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details. -3. Keep deterministic validation narrow and hard. -4. Put semantic expectations in `judgeChecklist`. -5. Use `expected` fixtures only when exact structure really matters. - -## Prompt writing - -Prompts should sound like something a user would naturally ask. - -Good: - -- "Create a flow that routes support requests based on customer tier." -- "Add a reset button that sets the counter back to 0." -- "Create a flow that reuses the existing greeting script instead of duplicating the logic." - -Bad: - -- "Use `branchone` with 3 branches and a default branch." -- "Create a `rawscript` step with this exact topology." -- "This is a benchmark harness." - -Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow. - -## Flow-specific rules - -This is the main principle you asked for: - -- flow prompts should read like requests from a user who does not know the product internals -- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs - -That means: - -- creation cases should describe the business behavior and expected result -- modification cases may mention existing step names, because the user can see the current flow -- only mention special Windmill constructs when the case is explicitly about those constructs - -Examples: - -- acceptable creation prompt: - "Create a purchase approval flow that pauses for approval and asks the approver for a comment." -- avoid: - "Create a suspend step with one required event and a resume form." - -For flow cases, do not fail a case just because the model chose a different valid topology. - -## App-specific rules - -App prompts should focus on user-visible behavior: - -- what the UI should let the user do -- what should persist -- what backend behavior is needed - -Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app. - -## CLI-specific rules - -CLI prompts can be more explicit about paths and file names because real CLI users often do specify them. - -Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction. - -When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior. - -## Global-specific rules - -Global prompts should exercise workspace-level drafting behavior: - -- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant -- writing AI drafts rather than saving or deploying by default -- producing coherent multi-artifact changes when the request crosses artifact boundaries - -Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them. - -Datatable cases should set `skipJudge: true` and validate through tool-use -(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions -(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`, -`['update', 'insert into']`). Two reasons the judge is unreliable here: - -- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` - produce no drafts, and the global judge only sees the drafts artifact — it - scores a no-draft conversational answer as empty (same as the - `askUserQuestion` cases). -- Even a case that *does* produce a draft (a script reading the data table via - `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK - reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the - SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']` - plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from - runtime SDK use). - -`stringIncludesAnyOf` is existential over calls (at least one matching call), so a -mutation case still passes when the model mixes its UPDATE/INSERT with -verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful -within a case — writes persist, so a model that re-queries to verify its -CREATE/UPDATE sees the change and does not loop. But the engine is best-effort -(SELECT returns all rows of the referenced/first table with no WHERE/projection), -so still never assert specific returned row values. Seed data via -`workspace.datatables` in the `initial` fixture (see README). - -## Deterministic validation - -Use deterministic validation only for hard failures such as: - -- missing required files -- unexpected extra files when the prompt says not to create them -- syntax errors -- unresolved flow refs -- missing required special modules or suspend config -- obvious artifact corruption - -Do not use deterministic validation to enforce one preferred implementation for broad creation tasks. - -Examples of bad hard checks: - -- exact step topology for a creation flow -- exact branch structure when the prompt only asked for routing behavior -- exact input shape when multiple reasonable shapes are acceptable - -## Judge checklist - -Every non-trivial case should have a `judgeChecklist`. - -The checklist should capture: - -- the user-visible behavior that must be present -- important constraints -- key completion criteria - -The checklist should not duplicate low-level implementation details unless they are truly required by the task. - -Good checklist items: - -- "the flow calculates the order total with 8% tax" -- "the app persists recipes appropriately for a raw Windmill app" -- "the flow reuses the existing workspace script instead of rewriting the logic" - -Bad checklist items: - -- "uses `branchone`" -- "contains a `rawscript` node" - -## When to use `expected` - -Use `expected` fixtures when the case is structure-sensitive, for example: - -- exact file creation -- exact script content -- modification cases where a specific file must change in a specific way -- cases where preserving an existing structure is part of the requirement - -Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass. - -## When to use `initial` - -Use `initial` when the benchmark is about: - -- editing an existing artifact -- reusing existing workspace assets -- preserving existing behavior while adding a change - -If the case is greenfield, prefer no `initial`. - -## Case design ladder - -Prefer suites that get gradually harder: - -1. trivial create case -2. realistic create case -3. reuse-existing-assets case -4. modification case -5. refactor case -6. edge-case or niche product behavior - -The last cases in a suite should cover unusual or product-specific behavior. - -## Anti-patterns - -Avoid these: - -- benchmark framing in prompts -- over-specified internal topology for creation tasks -- judge checklists that just restate implementation details -- deterministic validation that encodes one preferred solution -- fixtures that are so minimal or brittle that they create false negatives - -## Before adding a case - -Ask: - -1. Would a real user plausibly write this prompt? -2. If the model solves it in a different valid way, would the case still pass? -3. Are the hard deterministic checks only catching objectively broken output? -4. Does the `judgeChecklist` describe the real success criteria? -5. If this case fails, will the reason be understandable from the saved artifacts? +The full case format, fields, and fixture details remain in `ai_evals/README.md`. diff --git a/ai_evals/README.md b/ai_evals/README.md index 88825f4ee7..33bb47ae46 100644 --- a/ai_evals/README.md +++ b/ai_evals/README.md @@ -75,6 +75,8 @@ Public CLI surface: - `--model `: choose the model under test - `--models `: run the same cases sequentially against several model aliases - `--verbose`: stream assistant output for frontend runs +- `--skip-judge`: skip LLM judge scoring for the run +- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring - `--record`: append a compact tracked summary line to `ai_evals/history/.jsonl` for full-suite runs only - `--backend-validation `: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals @@ -99,7 +101,7 @@ Notes: - the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5` - frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there -- the judge model is separate and currently defaults to `claude-sonnet-4-6` +- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs ## Case Format diff --git a/ai_evals/adapters/frontend/benchmarkRunner.ts b/ai_evals/adapters/frontend/benchmarkRunner.ts index 32107eadf1..b45f880699 100644 --- a/ai_evals/adapters/frontend/benchmarkRunner.ts +++ b/ai_evals/adapters/frontend/benchmarkRunner.ts @@ -25,6 +25,12 @@ export async function runFrontendBenchmarkFromEnv(): Promise ); const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1"; const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1"; + const executionOnly = + process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1"; + const judgeModel = + process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly + ? null + : DEFAULT_JUDGE_MODEL; const model = resolveEvalModel( mode, process.env.WMILL_FRONTEND_AI_EVAL_MODEL, @@ -48,7 +54,8 @@ export async function runFrontendBenchmarkFromEnv(): Promise cases: selectedCases, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, + executionOnly, concurrency: verbose ? 1 : undefined, verbose, onProgress: emitProgress @@ -60,7 +67,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise mode, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, caseResults, }); } diff --git a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts index a6d78da36b..e553d15f56 100644 --- a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts @@ -50,6 +50,20 @@ export interface GlobalLiveEditorDraftFixture { value?: unknown; } +// Identity the global system prompt builds paths from. Production reads +// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs +// in, so without this the prompt sees an empty username (`u//...`) and no +// path-selection case is meaningful. Seeded per-case via the initial fixture and +// passed straight to `prepareGlobalSystemMessage` (no global-store mutation). +export interface GlobalUserFixture { + username: string; + is_admin?: boolean; + /** Folders the user can write to (the writable set whoami returns). */ + folders?: string[]; + /** Folders the user can read; read-only folders = folders_read \ folders. */ + folders_read?: string[]; +} + export interface GlobalEvalResult { success: boolean; state: GlobalDraftState; @@ -65,6 +79,7 @@ export interface GlobalEvalResult { export interface GlobalEvalOptions { workspaceFixtures?: BenchmarkWorkspaceRunnables; liveEditorDrafts?: GlobalLiveEditorDraftFixture[]; + user?: GlobalUserFixture; model?: string; maxIterations?: number; provider?: AIProvider; @@ -90,9 +105,11 @@ export async function runGlobalEval( const model = options.model ?? "claude-haiku-4-5-20251001"; const injectActiveEditorContext = process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1"; + // Pass the seeded identity straight to the prompt builder rather than mutating + // the process-global `userStore`, so concurrent cases never race on it. const rawResult = await runEval({ userPrompt, - systemMessage: prepareGlobalSystemMessage(), + systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }), userMessage: prepareGlobalUserMessage( userPrompt, [], diff --git a/ai_evals/adapters/frontend/mockBackend.ts b/ai_evals/adapters/frontend/mockBackend.ts index b78351512d..379eb6d447 100644 --- a/ai_evals/adapters/frontend/mockBackend.ts +++ b/ai_evals/adapters/frontend/mockBackend.ts @@ -90,6 +90,13 @@ export function resetBenchmarkMockBackend(): void { benchmarkDrafts.clear() } +// Stand-in for FolderService.createFolder so the global create_folder tool runs in +// memory instead of mutating the real backend. Folders aren't otherwise modelled +// (no folder-listing in evals), so this just echoes the created name. +export function createBenchmarkFolder(_workspace: string, name: string): string { + return name +} + export function registerBenchmarkWorkspace(workspace: string): void { benchmarkWorkspaces.add(workspace) } diff --git a/ai_evals/adapters/frontend/runtime.ts b/ai_evals/adapters/frontend/runtime.ts index 347e15191c..9a54e86084 100644 --- a/ai_evals/adapters/frontend/runtime.ts +++ b/ai_evals/adapters/frontend/runtime.ts @@ -24,6 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: { runs: number; model?: string; verbose?: boolean; + skipJudge?: boolean; + executionOnly?: boolean; backendValidation?: string; }): Promise { const tempDir = await mkdtemp( @@ -40,6 +42,9 @@ export async function runFrontendBenchmarkAdapter(input: { WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "", WMILL_FRONTEND_AI_EVAL_PROGRESS: "1", WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0", + WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE: + input.skipJudge || input.executionOnly ? "1" : "0", + WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0", WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "", }; diff --git a/ai_evals/adapters/frontend/vitestAdapter.test.ts b/ai_evals/adapters/frontend/vitestAdapter.test.ts index 621ecaefcd..dcaa1d2ca4 100644 --- a/ai_evals/adapters/frontend/vitestAdapter.test.ts +++ b/ai_evals/adapters/frontend/vitestAdapter.test.ts @@ -49,6 +49,7 @@ vi.mock('$lib/gen', async () => { listBenchmarkFlows, listBenchmarkJobs, listBenchmarkScripts, + createBenchmarkFolder, createBenchmarkHttpTrigger, createBenchmarkSchedule, previewBenchmarkSchedule, @@ -90,6 +91,12 @@ vi.mock('$lib/gen', async () => { ? listBenchmarkDrafts(data.workspace) : actual.DraftService.listDrafts(data) }), + FolderService: wrapService(actual.FolderService, { + createFolder: async (data: { workspace: string; requestBody: { name: string } }) => + hasBenchmarkWorkspace(data.workspace) + ? createBenchmarkFolder(data.workspace, data.requestBody.name) + : actual.FolderService.createFolder(data) + }), ScriptService: wrapService(actual.ScriptService, { listScripts: async (data: { workspace: string }) => hasBenchmarkWorkspace(data.workspace) diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index 319d13b272..31b097867b 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -3,8 +3,9 @@ Create a draft Bun script at `f/evals/global/greet_user`. It should take a string `name` input and return `Hello, ${name}!`. Leave it as an AI draft only; do not deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json runtime: - maxTurns: 8 + maxTurns: 10 validate: draftCountExactly: 1 requiredDrafts: @@ -1113,3 +1114,137 @@ - renames the formatCurrency definition, imports, and all call sites to formatMoney - leaves the unrelated formatCurrencyPrecise helper unchanged - leaves the result as an AI draft only + +# --- Path selection (u/ vs f/) --- +# These cases assert how the assistant picks a workspace path when the user gives +# none: a bare name defaults to the personal scope `u//`, an existing folder +# whose purpose matches is used, a non-admin targets a writable folder and never a +# read-only one, and shared intent with no matching folder asks rather than invents. +# Each depends on the seeded `user` fixture (username / is_admin / folders / +# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed. + +- id: global-path1-bare-name-defaults-to-personal + prompt: |- + Stage a quick draft helper that takes a string and returns it trimmed of + leading and trailing whitespace. Just keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + pathStartsWith: u/admin/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - stages a single script draft for a trim helper + - defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given + - does not invent an f/ path + - leaves the result as a draft only + +- id: global-path2-match-existing-folder + prompt: |- + Draft a flow for the marketing team's weekly campaign report. + It should take a week number and return a short summary string. + Keep it as a draft only. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: flow + pathStartsWith: f/marketing/ + toolExpect: + requiredToolsUsed: + - write_flow + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - drafts a flow for the marketing campaign report + - places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder + - leaves the result as a draft only + +- id: global-path3-shared-intent-unknown-folder-asks + prompt: |- + Put together a draft onboarding checklist flow for the People Ops team to use + when a new hire joins. Keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - askUserQuestion + forbiddenToolsUsed: + - write_flow + - write_script + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops) + - asks which folder to use instead of guessing or inventing one + - does not create a draft until the folder is known + +- id: global-path4-nonadmin-avoids-readonly-folder + prompt: |- + Draft a small flow that returns today's date as an ISO string, and stage it in + one of our shared team folders. Keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: flow + pathStartsWith: f/team_a/ + forbiddenDrafts: + - type: flow + pathStartsWith: f/team_b/ + toolExpect: + requiredToolsUsed: + - write_flow + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - drafts a flow that returns the current date as an ISO string + - places it in team_a (writable by this non-admin user) and not team_b (read-only) + - leaves the result as a draft only + +- id: global-path5-create-folder-then-draft + prompt: |- + Create a new shared folder called "analytics" for our data work, then draft a + script in it that returns the current timestamp as an ISO string. Keep the + script as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + pathStartsWith: f/analytics/ + toolExpect: + requiredToolsUsed: + - create_folder + - write_script + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - creates a new shared folder named "analytics" via create_folder + - drafts a script placed in that folder (f/analytics/...) returning an ISO timestamp + - leaves the script as a draft only diff --git a/ai_evals/cli/index.ts b/ai_evals/cli/index.ts index f92d6d7027..504b8a8d13 100644 --- a/ai_evals/cli/index.ts +++ b/ai_evals/cli/index.ts @@ -25,7 +25,9 @@ import { import { runSuite } from "../core/runSuite"; import { EVAL_MODES, type EvalMode } from "../core/types"; import { DEFAULT_JUDGE_MODEL } from "../core/judge"; -import { createCliModeRunner } from "../modes/cli"; +// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes +// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps +// (e.g. @cliffy/*) just to load this entrypoint. import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime"; import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings"; import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend"; @@ -97,6 +99,11 @@ async function main() { "comma-separated model aliases to run sequentially", ) .option("--verbose", "stream assistant output during frontend runs") + .option("--skip-judge", "skip LLM judge scoring for this run") + .option( + "--execution-only", + "only require the model/proxy/frontend loop to complete", + ) .option( "--record", "append a compact summary line to ai_evals/history/.jsonl", @@ -115,6 +122,8 @@ async function main() { model?: string; models?: string; verbose?: boolean; + skipJudge?: boolean; + executionOnly?: boolean; record?: boolean; backendValidation?: string; }, @@ -127,6 +136,8 @@ async function main() { model: options.model, models: options.models, verbose: options.verbose ?? false, + skipJudge: options.skipJudge ?? false, + executionOnly: options.executionOnly ?? false, record: options.record ?? false, backendValidation: options.backendValidation, }); @@ -175,6 +186,8 @@ async function handleRun(input: { model?: string; models?: string; verbose: boolean; + skipJudge: boolean; + executionOnly: boolean; record: boolean; backendValidation?: string; }) { @@ -230,6 +243,8 @@ async function handleRun(input: { input.runs, getCliEvalModel(model), runModel, + input.skipJudge, + input.executionOnly, ) : await runFrontendBenchmarkAdapter({ mode: input.mode, @@ -237,6 +252,8 @@ async function handleRun(input: { runs: input.runs, model: model.id, verbose: input.verbose, + skipJudge: input.skipJudge, + executionOnly: input.executionOnly, backendValidation, }); @@ -278,20 +295,25 @@ async function runCliBenchmark( runs: number, model: ReturnType, runModel: string, + skipJudge: boolean, + executionOnly: boolean, ) { + const { createCliModeRunner } = await import("../modes/cli"); + const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL; const caseResults = await runSuite({ modeRunner: createCliModeRunner(model), cases, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, + executionOnly, }); return buildRunResult({ mode: "cli", runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, caseResults, }); } diff --git a/ai_evals/core/cases.test.ts b/ai_evals/core/cases.test.ts index 9955a73fa9..5d6e3245db 100644 --- a/ai_evals/core/cases.test.ts +++ b/ai_evals/core/cases.test.ts @@ -212,6 +212,9 @@ describe("loadCases", () => { }, ], }); + expect(caseEntry?.initialPath).toContain( + "ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json" + ); expect(caseEntry?.toolExpect).toMatchObject({ requiredToolsUsed: ["write_script"], forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"], diff --git a/ai_evals/core/runSuite.test.ts b/ai_evals/core/runSuite.test.ts new file mode 100644 index 0000000000..26f300a5aa --- /dev/null +++ b/ai_evals/core/runSuite.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from "bun:test"; +import { runSuite } from "./runSuite"; +import type { ModeRunner } from "./types"; + +const modeRunner: ModeRunner = { + mode: "global", + concurrency: 1, + loadInitial: async () => undefined, + loadExpected: async () => undefined, + run: async () => ({ + success: true, + actual: { ok: true }, + assistantMessageCount: 1, + toolCallCount: 0, + toolsUsed: [], + skillsInvoked: [], + tokenUsage: null, + }), + validate: () => [], +}; + +describe("runSuite", () => { + it("skips judge checks when the run disables judge scoring", async () => { + const [caseResult] = await runSuite({ + modeRunner, + cases: [ + { + id: "case-1", + prompt: "Create a draft script", + judgeChecklist: ["the output satisfies the prompt"], + }, + ], + runs: 1, + runModel: "model-under-test", + judgeModel: null, + }); + + const [attempt] = caseResult.attempts; + expect(attempt.passed).toBe(true); + expect(attempt.judgeScore).toBeNull(); + expect(attempt.judgeSummary).toBeNull(); + expect(attempt.checks.map((check) => check.name)).toEqual([ + "run succeeded", + ]); + }); + + it("only requires run success when execution-only is enabled", async () => { + let loadExpectedCalls = 0; + let validateCalls = 0; + let backendValidateCalls = 0; + + const executionOnlyRunner: ModeRunner< + undefined, + undefined, + { ok: boolean } + > = { + ...modeRunner, + loadExpected: async () => { + loadExpectedCalls++; + return undefined; + }, + validate: () => { + validateCalls++; + return [{ name: "validator failed", passed: false }]; + }, + backendValidate: async () => { + backendValidateCalls++; + return { + checks: [{ name: "backend validation failed", passed: false }], + }; + }, + }; + + const [caseResult] = await runSuite({ + modeRunner: executionOnlyRunner, + cases: [ + { + id: "case-1", + prompt: "Create a draft script", + expectedPath: "fixtures/expected.json", + toolExpect: { requiredToolsUsed: ["write_script"] }, + judgeChecklist: ["the output satisfies the prompt"], + }, + ], + runs: 1, + runModel: "model-under-test", + judgeModel: "judge-model", + executionOnly: true, + }); + + const [attempt] = caseResult.attempts; + expect(attempt.passed).toBe(true); + expect(attempt.judgeScore).toBeNull(); + expect(attempt.judgeSummary).toBeNull(); + expect(attempt.checks.map((check) => check.name)).toEqual([ + "run succeeded", + ]); + expect(loadExpectedCalls).toBe(0); + expect(validateCalls).toBe(0); + expect(backendValidateCalls).toBe(0); + }); +}); diff --git a/ai_evals/core/runSuite.ts b/ai_evals/core/runSuite.ts index 1438749a11..4a8c9dab7b 100644 --- a/ai_evals/core/runSuite.ts +++ b/ai_evals/core/runSuite.ts @@ -15,11 +15,13 @@ export async function runSuite(input: { runs: number; runModel: string | null; judgeModel?: string | null; + executionOnly?: boolean; concurrency?: number; verbose?: boolean; onProgress?: (event: FrontendBenchmarkProgressEvent) => void; }): Promise { - const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL; + const judgeModel = + input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel; const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency); const results = new Array(input.cases.length); let cursor = 0; @@ -52,6 +54,7 @@ export async function runSuite(input: { runs: input.runs, judgeModel, judgeThreshold: input.modeRunner.judgeThreshold ?? 80, + executionOnly: input.executionOnly ?? false, modeRunner: input.modeRunner, totalCases: input.cases.length, verbose: input.verbose ?? false, @@ -72,8 +75,9 @@ async function runCaseAttempts(input: { caseIndex: number; evalCase: EvalCase; runs: number; - judgeModel: string; + judgeModel: string | null; judgeThreshold: number; + executionOnly: boolean; modeRunner: ModeRunner; totalCases: number; verbose: boolean; @@ -99,7 +103,9 @@ async function runCaseAttempts(input: { try { const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath); - const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath); + const expected = input.executionOnly + ? undefined + : await input.modeRunner.loadExpected(input.evalCase.expectedPath); const run = await input.modeRunner.run(input.evalCase.prompt, initial, { evalCase: input.evalCase, caseId: input.evalCase.id, @@ -162,22 +168,30 @@ async function runCaseAttempts(input: { }); const checks: BenchmarkCheck[] = [ buildCheck("run succeeded", run.success, run.error), - ...input.modeRunner.validate({ - evalCase: input.evalCase, - prompt: input.evalCase.prompt, - initial, - expected, - actual: run.actual, - run, - }), - ...validateToolExpectations({ - run, - toolExpect: input.evalCase.toolExpect, - }), ]; + if (!input.executionOnly) { + checks.push( + ...input.modeRunner.validate({ + evalCase: input.evalCase, + prompt: input.evalCase.prompt, + initial, + expected, + actual: run.actual, + run, + }), + ...validateToolExpectations({ + run, + toolExpect: input.evalCase.toolExpect, + }) + ); + } const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? []; - if (run.success && input.modeRunner.backendValidate) { + if ( + run.success && + !input.executionOnly && + input.modeRunner.backendValidate + ) { try { const backendValidation = await input.modeRunner.backendValidate({ evalCase: input.evalCase, @@ -218,7 +232,12 @@ async function runCaseAttempts(input: { let judgeScore: number | null = null; let judgeSummary: string | null = null; - if (run.success && !input.evalCase.skipJudge) { + if ( + run.success && + !input.executionOnly && + input.judgeModel !== null && + !input.evalCase.skipJudge + ) { const judge = await judgeOutput({ mode: input.modeRunner.mode, prompt: input.evalCase.prompt, diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json b/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json new file mode 100644 index 0000000000..8c81142fb0 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json @@ -0,0 +1,6 @@ +{ + "user": { + "username": "admin", + "is_admin": true + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json new file mode 100644 index 0000000000..236f091bc2 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["evals"], + "folders_read": ["evals"] + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json b/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json new file mode 100644 index 0000000000..7f7e237f36 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["marketing", "data_engineering", "shared_utils"], + "folders_read": ["marketing", "data_engineering", "shared_utils"] + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json b/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json new file mode 100644 index 0000000000..6320cef4d3 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "bob", + "is_admin": false, + "folders": ["team_a"], + "folders_read": ["team_a", "team_b"] + } +} diff --git a/ai_evals/modes/global.ts b/ai_evals/modes/global.ts index 2cf5413f59..050a4caad9 100644 --- a/ai_evals/modes/global.ts +++ b/ai_evals/modes/global.ts @@ -4,6 +4,7 @@ import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureL import { runGlobalEval, type GlobalLiveEditorDraftFixture, + type GlobalUserFixture, } from "../adapters/frontend/core/global/globalEvalRunner"; import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend"; import type { FrontendEvalModelConfig } from "../core/models"; @@ -15,6 +16,7 @@ import { getFrontendApiKey } from "./frontendCommon"; export interface GlobalInitialFixture { workspace?: BenchmarkWorkspaceRunnables; liveEditorDrafts?: GlobalLiveEditorDraftFixture[]; + user?: GlobalUserFixture; } export function createGlobalModeRunner( @@ -38,6 +40,7 @@ export function createGlobalModeRunner( { workspaceFixtures: initial?.workspace, liveEditorDrafts: initial?.liveEditorDrafts, + user: initial?.user, maxIterations: context.evalCase?.runtime?.maxTurns, provider: modelConfig.provider, model: modelConfig.model, @@ -104,6 +107,7 @@ async function loadGlobalInitialFixture(path: string): Promise>$2 FROM v2_job WHERE id = ANY($1)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca" +} diff --git a/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json new file mode 100644 index 0000000000..cef272c219 --- /dev/null +++ b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "language!: _", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null + ] + }, + "hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8" +} diff --git a/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json b/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json new file mode 100644 index 0000000000..e576cf8fae --- /dev/null +++ b/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "kind!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "parent_job", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null, + true, + true + ] + }, + "hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4" +} diff --git a/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json new file mode 100644 index 0000000000..e3f72d9c3a --- /dev/null +++ b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT version, columns AS \"columns: Json>\"\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC\n LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "columns: Json>", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7" +} diff --git a/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json b/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json new file mode 100644 index 0000000000..3e617e3f0c --- /dev/null +++ b/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd" +} diff --git a/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json new file mode 100644 index 0000000000..36ea3d7fb2 --- /dev/null +++ b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab" +} diff --git a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json similarity index 59% rename from backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json rename to backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json index 0769d083d6..4e5f9f6ed7 100644 --- a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json +++ b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json @@ -1,12 +1,12 @@ { "db_name": "PostgreSQL", - "query": "SELECT 1", + "query": "SELECT NOT pg_is_in_recovery()", "describe": { "columns": [ { "ordinal": 0, "name": "?column?", - "type_info": "Int4" + "type_info": "Bool" } ], "parameters": { @@ -16,5 +16,5 @@ null ] }, - "hash": "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5" + "hash": "282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf" } diff --git a/backend/.sqlx/query-299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4.json b/backend/.sqlx/query-299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4.json new file mode 100644 index 0000000000..642723decf --- /dev/null +++ b/backend/.sqlx/query-299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '10 minutes'\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4" +} diff --git a/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json new file mode 100644 index 0000000000..f940f25b4a --- /dev/null +++ b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json @@ -0,0 +1,62 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT version, columns AS \"columns: Json>\",\n snapshot_id, job_id, captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "columns: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "snapshot_id", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 4, + "name": "captured_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false, + true, + true, + false + ] + }, + "hash": "2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb" +} diff --git a/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json b/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json index 7e13b3aa44..241778d359 100644 --- a/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json +++ b/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json @@ -42,7 +42,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json b/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json new file mode 100644 index 0000000000..b475d068d5 --- /dev/null +++ b/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a" +} diff --git a/backend/.sqlx/query-394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db.json b/backend/.sqlx/query-394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db.json new file mode 100644 index 0000000000..3470285737 --- /dev/null +++ b/backend/.sqlx/query-394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db" +} diff --git a/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json b/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json new file mode 100644 index 0000000000..7d46bd5a66 --- /dev/null +++ b/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4" + ] + }, + "nullable": [] + }, + "hash": "39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91" +} diff --git a/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json b/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json new file mode 100644 index 0000000000..dd243cc2bb --- /dev/null +++ b/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job_queue SET running = true WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1" +} diff --git a/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json b/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json new file mode 100644 index 0000000000..8a79de3aa4 --- /dev/null +++ b/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) as \"c!\" FROM v2_job_debounce_batch", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2" +} diff --git a/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json new file mode 100644 index 0000000000..eabe7f9bf6 --- /dev/null +++ b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84" +} diff --git a/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json new file mode 100644 index 0000000000..8d9c34ed2b --- /dev/null +++ b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79" +} diff --git a/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json b/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json new file mode 100644 index 0000000000..6fd7d38f69 --- /dev/null +++ b/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e" +} diff --git a/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json new file mode 100644 index 0000000000..a6ab79cd7c --- /dev/null +++ b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n FROM workspace f\n JOIN workspace p ON p.id = f.parent_workspace_id\n WHERE f.id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "deleted!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688" +} diff --git a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json b/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json deleted file mode 100644 index 2f62420f87..0000000000 --- a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_result_stream_v2\n WHERE job_id NOT IN (SELECT id FROM v2_job_queue)\n AND job_id NOT IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at > NOW() - INTERVAL '60 seconds'\n )\n RETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51" -} diff --git a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json b/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json deleted file mode 100644 index f63afbc986..0000000000 --- a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8", - "UuidArray" - ] - }, - "nullable": [ - false - ] - }, - "hash": "45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f" -} diff --git a/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json b/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json deleted file mode 100644 index 3995bffd64..0000000000 --- a/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618" -} diff --git a/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json b/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json new file mode 100644 index 0000000000..0c4d90b073 --- /dev/null +++ b/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM flow_conversation_message m\n USING flow_conversation c\n WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c" +} diff --git a/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json b/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json index b90ba150d9..c269f7f340 100644 --- a/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json +++ b/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json @@ -41,7 +41,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json new file mode 100644 index 0000000000..43f527b3aa --- /dev/null +++ b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf" +} diff --git a/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json new file mode 100644 index 0000000000..e99432f11f --- /dev/null +++ b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('max_connections')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617" +} diff --git a/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json b/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json new file mode 100644 index 0000000000..c2c041e131 --- /dev/null +++ b/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext($1))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f" +} diff --git a/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json b/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json new file mode 100644 index 0000000000..07d1753aa0 --- /dev/null +++ b/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301" +} diff --git a/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json new file mode 100644 index 0000000000..5bd3cf80a7 --- /dev/null +++ b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265" +} diff --git a/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json b/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json index 2e189af5e8..d81d6ebd94 100644 --- a/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json +++ b/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json @@ -35,7 +35,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json b/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json new file mode 100644 index 0000000000..8c46b8250c --- /dev/null +++ b/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT attempt FROM native_retry_attempt WHERE job_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "attempt", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false + ] + }, + "hash": "54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c" +} diff --git a/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json b/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json index 0191cf8bfd..91c941593f 100644 --- a/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json +++ b/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json @@ -42,7 +42,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json new file mode 100644 index 0000000000..711970ff32 --- /dev/null +++ b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('server_version_num')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee" +} diff --git a/backend/.sqlx/query-57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89.json b/backend/.sqlx/query-57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89.json new file mode 100644 index 0000000000..f634fc2d4b --- /dev/null +++ b/backend/.sqlx/query-57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claim_self AS (\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE id = $1 AND consumed_at IS NULL\n RETURNING debounce_batch\n ), claim_rest AS (\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM claim_self)\n AND id <> $1 AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT debounce_batch FROM claim_self) AS claimed_batch,\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claim_rest) AS \"claimed_ids!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "had_row!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "claimed_batch", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "prev_consumed_by", + "type_info": "Uuid" + }, + { + "ordinal": 3, + "name": "claimed_ids!", + "type_info": "UuidArray" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89" +} diff --git a/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json b/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json index bb326fab87..0c445c1ae5 100644 --- a/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json +++ b/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json @@ -35,7 +35,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json b/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json new file mode 100644 index 0000000000..0535031aac --- /dev/null +++ b/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dispatch_event WHERE workspace_id = $1 AND producer_job_id = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5" +} diff --git a/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json b/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json index 713ccb9dd3..36ddb8ab9f 100644 --- a/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json +++ b/backend/.sqlx/query-5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55.json @@ -15,7 +15,7 @@ ] }, "nullable": [ - null + true ] }, "hash": "5a219a2532517869578c4504ff3153c43903f929ae5d62fbba12610f89c36d55" diff --git a/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json b/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json new file mode 100644 index 0000000000..28c09ca528 --- /dev/null +++ b/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\",\n result AS \"result: Json>\",\n started_at AS \"started_at!\"FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "success!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "result: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "started_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [ + null, + true, + true + ] + }, + "hash": "5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391" +} diff --git a/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json new file mode 100644 index 0000000000..20c5c58c40 --- /dev/null +++ b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n FROM app a\n JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "low_code!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "raw!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174" +} diff --git a/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json b/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json index 4bb89660ec..1e39bfdaab 100644 --- a/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json +++ b/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json @@ -46,7 +46,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18.json b/backend/.sqlx/query-70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18.json new file mode 100644 index 0000000000..e885c72039 --- /dev/null +++ b/backend/.sqlx/query-70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '1 hour'\n RETURNING 1\n ) SELECT count(*) FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18" +} diff --git a/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json b/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json new file mode 100644 index 0000000000..d55722ed24 --- /dev/null +++ b/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8" +} diff --git a/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json b/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json new file mode 100644 index 0000000000..694ed1887f --- /dev/null +++ b/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM debounce_key WHERE key = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5" +} diff --git a/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json new file mode 100644 index 0000000000..f909a33b4c --- /dev/null +++ b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"x\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5" +} diff --git a/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json b/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json new file mode 100644 index 0000000000..82dcf60cc1 --- /dev/null +++ b/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684" +} diff --git a/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json b/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json new file mode 100644 index 0000000000..79b43245ef --- /dev/null +++ b/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "concurrency_settings", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "debouncing_settings", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "retry_settings", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [ + true, + true, + true + ] + }, + "hash": "82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496" +} diff --git a/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json b/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json new file mode 100644 index 0000000000..03fc734037 --- /dev/null +++ b/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab" +} diff --git a/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json b/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json deleted file mode 100644 index 96e3439612..0000000000 --- a/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b" -} diff --git a/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json new file mode 100644 index 0000000000..ffce4491e3 --- /dev/null +++ b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json @@ -0,0 +1,67 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\",\n language AS \"language!: windmill_common::scripts::ScriptLang\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "language!: windmill_common::scripts::ScriptLang", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d" +} diff --git a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json similarity index 68% rename from backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json rename to backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json index 56b887e8b9..f8f8511709 100644 --- a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json +++ b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json @@ -1,16 +1,17 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()", + "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()\n WHERE (background_task_state.value->>'last_xmin')::bigint <= $4", "describe": { "columns": [], "parameters": { "Left": [ "Text", "Jsonb", - "Text" + "Text", + "Int8" ] }, "nullable": [] }, - "hash": "8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d" + "hash": "881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829" } diff --git a/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json new file mode 100644 index 0000000000..eac4547361 --- /dev/null +++ b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO materialized_asset_schema\n (workspace_id, asset_kind, asset_path, version, columns,\n snapshot_id, job_id, captured_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7, now())", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Varchar", + "Int8", + "Jsonb", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0" +} diff --git a/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json b/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json index 13402c8387..90031c14d6 100644 --- a/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json +++ b/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json @@ -35,7 +35,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json new file mode 100644 index 0000000000..0f75ec3883 --- /dev/null +++ b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf" +} diff --git a/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json b/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json new file mode 100644 index 0000000000..794f62e49d --- /dev/null +++ b/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1 AND id = ANY($2))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f" +} diff --git a/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json b/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json new file mode 100644 index 0000000000..e2132d7737 --- /dev/null +++ b/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d" +} diff --git a/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json b/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json new file mode 100644 index 0000000000..58630c5356 --- /dev/null +++ b/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96" +} diff --git a/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json b/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json new file mode 100644 index 0000000000..6a39c40970 --- /dev/null +++ b/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77" +} diff --git a/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json new file mode 100644 index 0000000000..24e387a783 --- /dev/null +++ b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614" +} diff --git a/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json b/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json new file mode 100644 index 0000000000..819928ddc1 --- /dev/null +++ b/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END\n FROM workspace WHERE id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d" +} diff --git a/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json b/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json new file mode 100644 index 0000000000..a8dc380f40 --- /dev/null +++ b/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claimed AS (\n -- Claim the whole batch in ONE update so concurrent same-batch\n -- survivors lock rows in identical scan order (no lock-ordering\n -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual\n -- and skips rows the other already took. A claim therefore consumes\n -- every still-unclaimed row of the batch atomically.\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM mine)\n AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\",\n EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "had_row!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "prev_consumed_by", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "claimed_ids!", + "type_info": "UuidArray" + }, + { + "ordinal": 3, + "name": "claimed_self!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be" +} diff --git a/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json b/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json new file mode 100644 index 0000000000..c3cee83397 --- /dev/null +++ b/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b" +} diff --git a/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json new file mode 100644 index 0000000000..a604c85831 --- /dev/null +++ b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "aurora!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "rds!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "cloudsql!", + "type_info": "Bool" + }, + { + "ordinal": 3, + "name": "azure!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db" +} diff --git a/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json b/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json new file mode 100644 index 0000000000..3413ddc87f --- /dev/null +++ b/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "e!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c" +} diff --git a/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json b/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json new file mode 100644 index 0000000000..406cbdb176 --- /dev/null +++ b/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT (hash)\n DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42" +} diff --git a/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json new file mode 100644 index 0000000000..dc82db7591 --- /dev/null +++ b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"cutoff?\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "cutoff?", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e" +} diff --git a/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json b/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json index 52e1ced19a..da7e3d5787 100644 --- a/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json +++ b/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json @@ -46,7 +46,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json b/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json new file mode 100644 index 0000000000..5b0f0b319c --- /dev/null +++ b/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtextextended($1, 0::int8))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238" +} diff --git a/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json b/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json new file mode 100644 index 0000000000..fc002b7074 --- /dev/null +++ b/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json @@ -0,0 +1,25 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH target AS (\n SELECT id FROM v2_job_queue WHERE id = $1 AND NOT running FOR UPDATE\n ), completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n $3::text::jsonb,\n s.flow_status,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n WHERE q.id IN (SELECT id FROM target)\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id IN (SELECT id FROM target)\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n SELECT $4, $1, $2 WHERE EXISTS (SELECT 1 FROM target)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135" +} diff --git a/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json b/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json new file mode 100644 index 0000000000..22618ad999 --- /dev/null +++ b/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354" +} diff --git a/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json b/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json new file mode 100644 index 0000000000..f9fbc7a58b --- /dev/null +++ b/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM flow_conversation_message WHERE job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636" +} diff --git a/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json new file mode 100644 index 0000000000..b2e218e511 --- /dev/null +++ b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "UuidArray", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75" +} diff --git a/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json b/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json new file mode 100644 index 0000000000..2d1e21019e --- /dev/null +++ b/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba" +} diff --git a/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json b/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json new file mode 100644 index 0000000000..876465fb8d --- /dev/null +++ b/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT j.runnable_path AS \"runnable_path!\", j.kind::text AS \"kind!\",\n q.runnable_settings_handle\n FROM v2_job j JOIN v2_job_queue q ON q.id = j.id\n WHERE j.workspace_id = $1 AND j.trigger_kind = 'asset'\n ORDER BY j.runnable_path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "kind!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "runnable_settings_handle", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + null, + true + ] + }, + "hash": "c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59" +} diff --git a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json b/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json deleted file mode 100644 index a68387f905..0000000000 --- a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_perms\nWHERE job_id NOT IN (SELECT id FROM v2_job_queue)\nRETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b" -} diff --git a/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json b/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json new file mode 100644 index 0000000000..ca39442f59 --- /dev/null +++ b/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "description", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b" +} diff --git a/backend/.sqlx/query-c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed.json b/backend/.sqlx/query-c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed.json new file mode 100644 index 0000000000..7361b645b5 --- /dev/null +++ b/backend/.sqlx/query-c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES\n ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'),\n ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'),\n ($3, nextval('debounce_batch_seq'), NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed" +} diff --git a/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json b/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json index 4629a9bd0a..161b9d36f9 100644 --- a/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json +++ b/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json @@ -42,7 +42,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json new file mode 100644 index 0000000000..2c775af0b1 --- /dev/null +++ b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', now(),\n 'last_oldest_inflight_ts',\n COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1" +} diff --git a/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json new file mode 100644 index 0000000000..4364da5dd0 --- /dev/null +++ b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230" +} diff --git a/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json b/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json index 0eed069163..e60f5cc187 100644 --- a/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json +++ b/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json @@ -47,7 +47,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] } } diff --git a/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json b/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json new file mode 100644 index 0000000000..8c2a7716fd --- /dev/null +++ b/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\"\n FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "success!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994" +} diff --git a/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json b/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json new file mode 100644 index 0000000000..87cd891113 --- /dev/null +++ b/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf" +} diff --git a/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json new file mode 100644 index 0000000000..20ae255eb1 --- /dev/null +++ b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + false + ] + }, + "hash": "e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd" +} diff --git a/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json b/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json new file mode 100644 index 0000000000..121d7fe04a --- /dev/null +++ b/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "description", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "instructions", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d" +} diff --git a/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json b/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json new file mode 100644 index 0000000000..29ce0c6f2c --- /dev/null +++ b/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61" +} diff --git a/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json b/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json new file mode 100644 index 0000000000..f236c2aae0 --- /dev/null +++ b/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES\n ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'),\n ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'),\n ($3, nextval('debounce_batch_seq'), NULL),\n ($4, nextval('debounce_batch_seq'), now() - interval '20 minutes')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid", + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2" +} diff --git a/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json new file mode 100644 index 0000000000..dd7f4da810 --- /dev/null +++ b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.id = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null, + null, + false + ] + }, + "hash": "ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db" +} diff --git a/backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json b/backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json similarity index 55% rename from backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json rename to backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json index 4b96e7005b..84a2320a80 100644 --- a/backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json +++ b/backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json @@ -1,15 +1,23 @@ { "db_name": "PostgreSQL", - "query": "WITH to_delete AS (\n SELECT id FROM v2_job_queue\n JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = $1\n AND j.workspace_id = $2\n AND flow_step_id IS NULL\n AND running = false\n FOR UPDATE\n ), deleted AS (\n DELETE FROM v2_job_queue\n WHERE id IN (SELECT id FROM to_delete)\n RETURNING id\n ) DELETE FROM v2_job WHERE id IN (SELECT id FROM deleted)", + "query": "WITH to_delete AS (\n SELECT id FROM v2_job_queue\n JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = $1\n AND j.workspace_id = $2\n AND flow_step_id IS NULL\n AND running = false\n FOR UPDATE\n )\n DELETE FROM v2_job_queue\n WHERE id IN (SELECT id FROM to_delete)\n RETURNING id", "describe": { - "columns": [], + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + } + ], "parameters": { "Left": [ "Text", "Text" ] }, - "nullable": [] + "nullable": [ + false + ] }, - "hash": "d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a" + "hash": "f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3" } diff --git a/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json b/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json new file mode 100644 index 0000000000..1039dd3950 --- /dev/null +++ b/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN NULL ELSE debounce_key.job_id END,\n job_id = EXCLUDED.job_id,\n debounced_times = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN 0 ELSE debounce_key.debounced_times + 1 END,\n first_started_at = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN now() ELSE debounce_key.first_started_at END\n RETURNING debounced_times, first_started_at, previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT $1, COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq'))\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "debounced_times", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "first_started_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "job_id_to_debounce", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2" +} diff --git a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json similarity index 64% rename from backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json rename to backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json index af4ada67f2..84eb4caba9 100644 --- a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json +++ b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = 'anonymous'\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", + "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = $4\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", "describe": { "columns": [ { @@ -11,6 +11,7 @@ ], "parameters": { "Left": [ + "Text", "Text", "Text", "Text" @@ -20,5 +21,5 @@ null ] }, - "hash": "a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d" + "hash": "f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7" } diff --git a/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json b/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json new file mode 100644 index 0000000000..81b34464f4 --- /dev/null +++ b/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN NULL ELSE debounce_key.job_id END,\n job_id = EXCLUDED.job_id,\n debounced_times = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN 0 ELSE debounce_key.debounced_times + 1 END,\n first_started_at = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN now() ELSE debounce_key.first_started_at END\n RETURNING debounced_times, first_started_at, previous_job_id AS job_id_to_debounce\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "debounced_times", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "first_started_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "job_id_to_debounce", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5" +} diff --git a/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json b/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json new file mode 100644 index 0000000000..132d1da5a1 --- /dev/null +++ b/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH ids AS (SELECT id FROM v2_job WHERE workspace_id = $1),\n _de AS (DELETE FROM dispatch_event WHERE workspace_id = $1),\n _fc AS (DELETE FROM flow_conversation_message WHERE job_id IN (SELECT id FROM ids))\n DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM ids)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645" +} diff --git a/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json new file mode 100644 index 0000000000..484dd23e2a --- /dev/null +++ b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN usr.email IS NULL THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "status!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441" +} diff --git a/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json new file mode 100644 index 0000000000..5ac11c6a5e --- /dev/null +++ b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS \"day!\",\n id AS \"id!\",\n timestamp AS \"ts!\",\n row_to_json(r)::text AS \"line!\"\n FROM (\n SELECT workspace_id, id, timestamp, username, operation,\n action_kind::text AS action_kind, resource, parameters, email, span\n FROM audit_partitioned\n WHERE timestamp >= $1 AND timestamp < $2\n AND (timestamp, id) > ($3, $4)\n ORDER BY timestamp, id\n LIMIT $5\n ) r\n ORDER BY timestamp, id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "day!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "id!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "ts!", + "type_info": "Timestamptz" + }, + { + "ordinal": 3, + "name": "line!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Timestamptz", + "Timestamptz", + "Timestamptz", + "Int8", + "Int8" + ] + }, + "nullable": [ + null, + false, + false, + null + ] + }, + "hash": "fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c" +} diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 63534975f5..7072389c21 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -180,9 +180,9 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.102" +version = "1.0.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" +checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3" [[package]] name = "ar_archive_writer" @@ -237,9 +237,9 @@ checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" [[package]] name = "arrayvec" -version = "0.7.6" +version = "0.7.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" +checksum = "f02882884d3e1bc524fb12c79f107f6ad0e1cfd498c536ffb494301740995dfe" [[package]] name = "arrow" @@ -1826,13 +1826,13 @@ dependencies = [ [[package]] name = "bstr" -version = "1.12.1" +version = "1.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" +checksum = "5cee35f73844aa3014bb606320a6c1f010249dbdf43342fe54b5a4f6a8ed4b79" dependencies = [ "memchr", "regex-automata", - "serde", + "serde_core", ] [[package]] @@ -1855,9 +1855,9 @@ dependencies = [ [[package]] name = "byte-unit" -version = "5.2.3" +version = "5.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "37bcaa4a0975bed4a760af3efe4368825098ce5f9d37a30c5a021d635dc63d8f" +checksum = "4a813de7f2bbedb7dce265b64f1cf5908ebe4d56281ece8d847e98113788b9b0" dependencies = [ "rust_decimal", "schemars 1.2.1", @@ -2101,9 +2101,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chacha20" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" dependencies = [ "cfg-if", "cpufeatures 0.3.0", @@ -2445,9 +2445,9 @@ dependencies = [ [[package]] name = "crc-any" -version = "2.5.0" +version = "2.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a62ec9ff5f7965e4d7280bd5482acd20aadb50d632cf6c1d74493856b011fa73" +checksum = "46db9f663dfb869b80fcf59e32d7a80fc6c464a4f6328f3f06a00f5e36d05f8c" dependencies = [ "debug-helper", ] @@ -3412,9 +3412,9 @@ checksum = "c286de4e81ea2590afc24d754e0f83810c566f50a1388fa75ebd57928c0d9745" [[package]] name = "debug-helper" -version = "0.3.13" +version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f578e8e2c440e7297e008bb5486a3a8a194775224bbc23729b0dbdfaeebf162e" +checksum = "80a4af69c60438a1a82af89d362f4729fd38db7b73f305a237636fad31ceb2bf" [[package]] name = "debugid" @@ -4371,18 +4371,18 @@ dependencies = [ [[package]] name = "enum-ordinalize" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a1091a7bb1f8f2c4b28f1fe2cef4980ca2d410a3d727d67ecc3178c9b0800f0" +checksum = "07f808d588c10e464ea6f7d3eaed500049eff30aaac103460f61828c2d65b3eb" dependencies = [ "enum-ordinalize-derive", ] [[package]] name = "enum-ordinalize-derive" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" +checksum = "42e528e2d34ba8a67a1a650b86beae8ef69fc5fdb638016f386b973226590432" dependencies = [ "proc-macro2", "quote", @@ -5233,9 +5233,9 @@ dependencies = [ [[package]] name = "gosyn" -version = "0.2.10" +version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c99c1502d84229dc7ddb6af755f40ebe80e7e932fa78ecef979cedcf9999ba93" +checksum = "fed1657682b1c3f63ece1fe5b60fc6c5f5923612a20d19f6af38ce79eaf361e3" dependencies = [ "anyhow", "strum", @@ -5712,9 +5712,9 @@ dependencies = [ [[package]] name = "hyper-http-proxy" -version = "1.1.0" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ad4b0a1e37510028bc4ba81d0e38d239c39671b0f0ce9e02dfa93a8133f7c08" +checksum = "8021e0ae20c08eadc94d0bdafdeda66d4f0858541c146ae6e46b219bfe58497e" dependencies = [ "bytes", "futures-util", @@ -5726,7 +5726,6 @@ dependencies = [ "hyper-util", "native-tls", "pin-project-lite", - "rustls-native-certs 0.7.3", "tokio", "tokio-native-tls", "tokio-rustls 0.26.4", @@ -6715,9 +6714,9 @@ dependencies = [ [[package]] name = "log" -version = "0.4.32" +version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" [[package]] name = "loom" @@ -7046,9 +7045,9 @@ checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] name = "memmap2" -version = "0.9.10" +version = "0.9.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" dependencies = [ "libc", "stable_deref_trait", @@ -8963,21 +8962,21 @@ dependencies = [ [[package]] name = "quick_cache" -version = "0.6.23" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a3db184a8b66cfe87f0263a1de147a6b554c864d1767c6f7fa4eb0e5497b565" +checksum = "403c1a912fec895cafb223201e368234842acb9220aaf08ab042ae89ba5f135c" dependencies = [ - "ahash 0.8.12", "equivalent", - "hashbrown 0.16.1", + "foldhash 0.2.0", + "hashbrown 0.17.1", "parking_lot", ] [[package]] name = "quinn" -version = "0.11.9" +version = "0.11.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" +checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" dependencies = [ "bytes", "cfg_aliases", @@ -8995,9 +8994,9 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.14" +version = "0.11.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" +checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e" dependencies = [ "aws-lc-rs", "bytes", @@ -9031,9 +9030,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.45" +version = "1.0.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" dependencies = [ "proc-macro2", ] @@ -12189,9 +12188,9 @@ dependencies = [ [[package]] name = "time" -version = "0.3.49" +version = "0.3.51" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "711a53c2d47bbd818258c498c8dbfe186a2526c631495cfe7e078567f86b8469" +checksum = "85c17d80feb7334b40c484e45ed1a5273dfd8bfda537c3be2e74a06a6686f327" dependencies = [ "deranged", "num-conv", @@ -12209,9 +12208,9 @@ checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" [[package]] name = "time-macros" -version = "0.2.29" +version = "0.2.30" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "71c652a3727a9cbb9a02f707f530b618ce00d0ccd762009c8c23bd191df3c17d" +checksum = "dcef1a61bdb119096e153208ec5cbec23944ce8bca13be5c7f60c634f7403935" dependencies = [ "num-conv", "time-core", @@ -13340,9 +13339,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.23.3" +version = "1.23.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" +checksum = "bf80a72845275afea99e7f2b434723d3bc7e38470fcd1c7ed39a599c73319a53" dependencies = [ "getrandom 0.4.3", "js-sys", @@ -13735,7 +13734,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -13817,7 +13816,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.734.0" +version = "1.742.0" dependencies = [ "async-stream", "async-trait", @@ -13850,7 +13849,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13863,7 +13862,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "argon2", @@ -14001,7 +14000,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14024,20 +14023,22 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", "serde", "serde_json", "sqlx", + "tracing", "windmill-api-auth", "windmill-common", + "windmill-parser-sql-asset", ] [[package]] name = "windmill-api-auth" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14063,7 +14064,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.734.0" +version = "1.742.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14073,7 +14074,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14090,7 +14091,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14112,7 +14113,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14135,7 +14136,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14151,7 +14152,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14172,7 +14173,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14193,7 +14194,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14207,7 +14208,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -14242,7 +14243,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14267,7 +14268,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14285,7 +14286,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14307,7 +14308,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14327,7 +14328,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14364,7 +14365,7 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14392,7 +14393,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.734.0" +version = "1.742.0" dependencies = [ "lazy_static", "serde", @@ -14404,7 +14405,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.734.0" +version = "1.742.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14429,7 +14430,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14443,7 +14444,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.734.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14476,7 +14477,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.734.0" +version = "1.742.0" dependencies = [ "chrono", "lazy_static", @@ -14490,7 +14491,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14509,7 +14510,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.734.0" +version = "1.742.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14611,7 +14612,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.734.0" +version = "1.742.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14630,7 +14631,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.734.0" +version = "1.742.0" dependencies = [ "regex", "serde", @@ -14645,7 +14646,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14669,7 +14670,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "futures", @@ -14686,7 +14687,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.734.0" +version = "1.742.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14702,7 +14703,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -14723,7 +14724,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -14754,7 +14755,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "arc-swap", @@ -14779,7 +14780,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-stream", @@ -14813,7 +14814,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "futures", @@ -14831,7 +14832,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.734.0" +version = "1.742.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14840,7 +14841,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14852,7 +14853,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14864,7 +14865,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "gosyn", @@ -14876,7 +14877,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14888,7 +14889,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14900,7 +14901,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "nu-parser", @@ -14911,7 +14912,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14922,7 +14923,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14934,7 +14935,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14945,7 +14946,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -14967,7 +14968,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14979,7 +14980,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14993,7 +14994,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15010,7 +15011,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15023,7 +15024,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -15035,7 +15036,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15053,7 +15054,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15069,7 +15070,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15085,7 +15086,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -15096,7 +15097,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -15130,11 +15131,12 @@ dependencies = [ "uuid", "windmill-audit", "windmill-common", + "windmill-jseval", ] [[package]] name = "windmill-runtime-nativets" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "const_format", @@ -15173,7 +15175,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.734.0" +version = "1.742.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15184,17 +15186,19 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", "axum 0.8.9", + "base64 0.22.1", "chrono", "futures", "hex", "http 1.4.2", "hyper 1.10.1", "lazy_static", + "magic-crypt", "quick_cache", "reqwest 0.13.1", "serde", @@ -15216,7 +15220,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15240,7 +15244,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15273,7 +15277,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15306,7 +15310,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15326,7 +15330,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15360,7 +15364,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15396,7 +15400,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15419,7 +15423,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15443,7 +15447,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -15467,7 +15471,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15502,7 +15506,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15530,7 +15534,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15555,7 +15559,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15574,7 +15578,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-once-cell", @@ -15684,7 +15688,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.734.0" +version = "1.742.0" dependencies = [ "bytes", "futures", @@ -16502,9 +16506,9 @@ dependencies = [ [[package]] name = "zlib-rs" -version = "0.6.3" +version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" +checksum = "977347db8caa080403f6b6b7c1cda9479a8e869316f7e13a59b19076a40f94e3" [[package]] name = "zmij" diff --git a/backend/Cargo.toml b/backend/Cargo.toml index a4dd1ce96f..45682215cf 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.734.0" +version = "1.742.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.734.0" +version = "1.742.0" authors = ["Ruben Fiszel "] edition = "2021" @@ -121,6 +121,7 @@ embedding = ["windmill-api/embedding"] parquet = ["windmill-api/parquet", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker/parquet"] prometheus = ["windmill-common/prometheus", "windmill-api/prometheus", "windmill-worker/prometheus", "windmill-queue/prometheus", "dep:prometheus"] flow_testing = ["windmill-worker/flow_testing"] +failpoints = ["windmill-worker/failpoints", "windmill-queue/failpoints"] quickjs = ["windmill-worker/quickjs", "windmill-api/quickjs"] openidconnect = ["windmill-api/openidconnect", "windmill-common/openidconnect", "windmill-object-store/openidconnect"] cloud = ["windmill-queue/cloud", "windmill-worker/cloud", "windmill-common/cloud", "windmill-api/cloud"] diff --git a/backend/THREAT_MODEL.md b/backend/THREAT_MODEL.md index dc98a5a625..9501b5bf0d 100644 --- a/backend/THREAT_MODEL.md +++ b/backend/THREAT_MODEL.md @@ -88,7 +88,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | EP12 Stored-content rendering | App builder HTML component, markdown, S3 download response headers | stored user content → admin browser (same origin) | Admin session, account takeover | | EP13 Log/file reading & export endpoints | `service_logs`, `jobs_u/getupdate` log file read (symlinks), workspace/tarball export | authed/unauth request → arbitrary file or admin-only config | Arbitrary files, global settings | | EP14 Secret-value & resource-value caches | In-memory caches in `windmill-store` keyed (historically un-keyed) by path | cache lookup crossing identity/folder boundary | Secret variables, resource creds | -| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS=false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | +| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS` now defaults to `true`; can still be overridden to `false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | | EP16 Supply chain | Cached hub scripts, GitHub workflow actions, vendored deps, Docker base image | build/update-time input → host & build integrity | Worker host, build integrity | | EP17 Token lifecycle | Token create/rescope/refresh, script-issued JWTs | scoped caller → broader privilege | Tokens, accounts, isolation | @@ -103,7 +103,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | T5 | Worker compromise & cross-tenant access via weak-by-default isolation (nsjail off by default → user code runs with only PID-ns `unshare`); sandbox escape where nsjail/dind/podman is enabled | remote_auth | EP9, EP15 | Worker host, isolation, downstream | critical | likely | unmitigated | nsjail off by default everywhere (`DISABLE_NSJAIL=true`); shipped compose gives PID-ns `unshare` only (`FAVOR_UNSHARE_PID=true`), bare installs get no isolation. Where nsjail enabled: read-only remounts, jail-tmp refusal, podman socket gating | GHSA-6qr8-xhg4-453q, GHSA-3vpp-vf62-wqp6, f8467f38c8, df5aec0f5d, f1b6746e0e | | T6 | Disclosure of secrets, resource credentials, and workspace encryption keys across the authorization boundary (AI proxy, MCP, caches, export); database read additionally yields plaintext instance-level `global_settings` secrets | remote_auth | EP6, EP14, EP13 | Secret variables, encryption keys, resource creds, global settings | critical | likely | partially_mitigated | RLS on `$var:`, cache scoping by caller, admin checks on export; per-workspace secret *variables* encrypted at rest, but `global_settings` is plaintext under the default DB secret backend | GHSA-jwg4-v3cj-rvfm, GHSA-8m2p-2crh-9h3w, GHSA-6635-6fch-v8px, GHSA-437f-725p-7w84, GHSA-f27g-j463-q85w (CVE-2026-26964), GHSA-j679-v6vj-jfxc, GHSA-6vrr-fq33-qpfp, 0ba128afe7, 7836a4e733, ff8e39c69b | | T7 | Full instance compromise from insecure deployment defaults (dind control, default admin/`changeme`, exposed Postgres, publicly readable SUPERADMIN_SECRET) | remote_unauth | EP15 | All assets | critical | likely | partially_mitigated | first-time-setup warning on default admin; docs recommend hardening | GHSA-3vpp-vf62-wqp6, GHSA-24fr-44f8-fqwg (CVE-2026-29059), GHSA-6q36-5p3h-766j | -| T8 | Unauthenticated RCE via the Debugger WebSocket in the default `windmill_extra` configuration | remote_unauth | EP15 | Worker host, all assets | critical | possible | unmitigated | `REQUIRE_SIGNED_DEBUG_REQUESTS` exists but defaults to false | GHSA-725h-99vx-9xr4 | +| T8 | Unauthenticated RCE via the Debugger WebSocket: `/ws_debug/*` exposed by the gateway/ingress with the debugger service as the auth boundary; signature gate was bypassable via `program`-mode launches (read+exec an arbitrary server-side file path, never signed) even with signing on, and the WS handshake had no Origin check (CSWSH) | remote_unauth | EP15 | Worker host, all assets | critical | possible | partially_mitigated | `program`-mode launches now rejected when `REQUIRE_SIGNED_DEBUG_REQUESTS` is on (signing covers every launch, not just inline `code`); shipped `docker-compose` now defaults `REQUIRE_SIGNED_DEBUG_REQUESTS=true`; opt-in `DEBUG_ALLOWED_ORIGINS` allowlist rejects cross-origin handshakes. Residual: code default is secure but operators can still set `=false`; origin allowlist is opt-in | GHSA-725h-99vx-9xr4 | | T9 | Supply-chain compromise via cached hub scripts, GitHub workflow command injection, or vulnerable base-image deps | supply_chain | EP16 | Worker host, build integrity | critical | possible | partially_mitigated | hub-script re-pin to patched versions; HUB_BASE_URL override | GHSA-w2m9-q5f7-3gpq, edf340c4d4, GHSA-8rq7-w7g6-8wvr, GHSA-vch9-39v5-4wg7 (CVE-2024-37371) | | T10 | Unauthenticated disclosure of job results, args, logs, and admin config via missing-authz public endpoints | remote_unauth | EP2, EP13 | Job results/args/logs, global settings, scripts | high | likely | partially_mitigated | anonymous-job checks, log-endpoint authz hardening | GHSA-qfg7-x243-5hg4, GHSA-v448-fmm4-52fp, 108a88a180, bb90f4ce83 | | T11 | Stored XSS leading to admin/account takeover via app HTML component, markdown, or S3 download content-type | remote_auth | EP12 | Admin session, accounts | high | likely | partially_mitigated | DOMPurify markdown sanitization, `X-Content-Type-Options: nosniff` + CSP sandbox on downloads | GHSA-9c5c-hh3c-r9mc, GHSA-qxj7-hpx3-r892, GHSA-cf2x-rg8c-v63v, bb78b1c06d, 625b67dff0 | diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index b3422ed267..d432aa6060 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -1454aaa9e60e17cfb3c22002594900dbaca38455 +409324cb0835bd42a9fc817e891343b7363a0c59 diff --git a/backend/migrations/20260617144417_add_ai_skill.down.sql b/backend/migrations/20260617144417_add_ai_skill.down.sql new file mode 100644 index 0000000000..8fd6e1606d --- /dev/null +++ b/backend/migrations/20260617144417_add_ai_skill.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS ai_skill; diff --git a/backend/migrations/20260617144417_add_ai_skill.up.sql b/backend/migrations/20260617144417_add_ai_skill.up.sql new file mode 100644 index 0000000000..b9a5003572 --- /dev/null +++ b/backend/migrations/20260617144417_add_ai_skill.up.sql @@ -0,0 +1,15 @@ +-- Workspace-scoped AI chat skills (Claude/Codex-style SKILL.md instructions). +-- `name` is the skill folder slug; `description` is advertised in the AI chat +-- system prompt, `instructions` is the SKILL.md body fetched on demand. +CREATE TABLE ai_skill ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE, + name VARCHAR(255) NOT NULL, + description TEXT NOT NULL, + instructions TEXT NOT NULL, + edited_at TIMESTAMPTZ NOT NULL DEFAULT now(), + edited_by VARCHAR(255) NOT NULL DEFAULT '', + PRIMARY KEY (workspace_id, name) +); + +GRANT ALL ON ai_skill TO windmill_user; +GRANT ALL ON ai_skill TO windmill_admin; diff --git a/backend/migrations/20260619150718_native_retry_settings.down.sql b/backend/migrations/20260619150718_native_retry_settings.down.sql new file mode 100644 index 0000000000..459ebe4eaf --- /dev/null +++ b/backend/migrations/20260619150718_native_retry_settings.down.sql @@ -0,0 +1,4 @@ +ALTER TABLE runnable_settings +DROP COLUMN IF EXISTS retry_settings; + +DROP TABLE IF EXISTS retry_settings; diff --git a/backend/migrations/20260619150718_native_retry_settings.up.sql b/backend/migrations/20260619150718_native_retry_settings.up.sql new file mode 100644 index 0000000000..e03192c710 --- /dev/null +++ b/backend/migrations/20260619150718_native_retry_settings.up.sql @@ -0,0 +1,16 @@ +CREATE TABLE IF NOT EXISTS retry_settings( + hash BIGINT PRIMARY KEY, + constant_attempts INTEGER, + constant_seconds INTEGER, + exponential_attempts INTEGER, + exponential_multiplier INTEGER, + exponential_seconds INTEGER, + exponential_random_factor INTEGER, + retry_if_expr TEXT +); + +ALTER TABLE runnable_settings +ADD COLUMN IF NOT EXISTS retry_settings BIGINT DEFAULT NULL; + +GRANT ALL ON retry_settings TO windmill_admin; +GRANT ALL ON retry_settings TO windmill_user; diff --git a/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql b/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql new file mode 100644 index 0000000000..3d532de9cb --- /dev/null +++ b/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql @@ -0,0 +1,3 @@ +-- No-op: this migration only re-pins the function's search_path. Reverting would +-- mean restoring the hardcoded `SET search_path = public`, which is the very bug +-- this repairs, so there is nothing to undo. diff --git a/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql b/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql new file mode 100644 index 0000000000..f8e448891f --- /dev/null +++ b/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql @@ -0,0 +1,22 @@ +-- Repair instances that already applied the folder-labels migrations while the +-- function hardcoded `SET search_path = public`. On a non-public schema (PG_SCHEMA) +-- the function was pinned to `public`, so at runtime it read the wrong `folder` +-- table (or a stray public.folder) instead of the workspace's real one. +-- +-- `FROM CURRENT` snapshots the migration connection's search_path (the actual +-- Windmill schema) into the function, keeping the SECURITY DEFINER injection +-- hardening. On public-schema installs this re-pins to `public`, i.e. a no-op. +-- Idempotent: redefining with the same body is harmless on already-correct installs. +CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[] +LANGUAGE sql STABLE SECURITY DEFINER SET search_path FROM CURRENT AS $$ + SELECT ( + SELECT array_agg(l ORDER BY first_ord) + FROM ( + SELECT u.l, min(u.ord) AS first_ord + FROM unnest(f.labels) WITH ORDINALITY AS u(l, ord) + GROUP BY u.l + ) deduped + ) + FROM folder f + WHERE f.workspace_id = w_id AND item_path LIKE 'f/%' AND f.name = split_part(item_path, '/', 2) +$$; diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql new file mode 100644 index 0000000000..3c69d46fc8 --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data backfill: once a raw app's draft is retyped to 'raw_app' it +-- is indistinguishable from one saved as 'raw_app' by the per-kind code, so the +-- original typ='app' state cannot be reconstructed. No-op on revert. diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql new file mode 100644 index 0000000000..d4b068a80c --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql @@ -0,0 +1,35 @@ +-- The pre-per-user `DRAFT_TYPE` enum had only ('script','flow','app'): a raw +-- app's draft was therefore stored as typ='app'. The new model splits app vs +-- raw_app into distinct draft kinds chosen from the deployed app's `raw_app` +-- flag, so a raw app's pre-migration draft is invisible to the per-kind lookups +-- (editor overlay, migrate-legacy, get-for-user), which all query typ='raw_app'. +-- Realign every such draft (any owner, including the legacy NULL-email row) to +-- 'raw_app' when the deployed app at that path is a raw app. + +-- Drop, don't retype, a stale 'app' row when a 'raw_app' draft already exists +-- for the same owner (the newer 'raw_app' row, saved with the per-kind code, is +-- authoritative) — retyping would collide on the draft_pkey_with_user / +-- draft_pkey_legacy partial unique indexes over (workspace_id, path, typ, email). +DELETE FROM draft d +USING app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE + AND EXISTS ( + SELECT 1 FROM draft d2 + WHERE d2.workspace_id = d.workspace_id + AND d2.path = d.path + AND d2.typ = 'raw_app' + AND d2.email IS NOT DISTINCT FROM d.email + ); + +UPDATE draft d +SET typ = 'raw_app' +FROM app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE; diff --git a/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql b/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql new file mode 100644 index 0000000000..0a09f0f42a --- /dev/null +++ b/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS native_retry_attempt; diff --git a/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql b/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql new file mode 100644 index 0000000000..e193fb8b8f --- /dev/null +++ b/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql @@ -0,0 +1,18 @@ +-- Sparse marker: one row per native retry attempt (a re-pushed Script job that +-- gained a retry). Presence = "this job is a native retry attempt"; `attempt` is +-- the chain position (1-based). Lets consumers — asset-cascade dispatch, the +-- per-occurrence schedule-handler counting, and the run-page chain — identify +-- retries explicitly instead of inferring from incidental fields (parent_job + +-- runnable + flow_innermost), which collides with schedule handlers and WAC +-- inline children. Sparse: only failed-and-retried jobs under a retry policy +-- produce rows. Lifecycle: removed with their job in the retention sweep (no FK, +-- to keep the bulk job delete cheap). +CREATE TABLE IF NOT EXISTS native_retry_attempt ( + job_id UUID PRIMARY KEY, + -- `integer` matches the retry policy's i32 attempt count; avoids any narrowing + -- on the maybe_enqueue read/write path. + attempt INTEGER NOT NULL +); + +GRANT ALL ON native_retry_attempt TO windmill_admin; +GRANT ALL ON native_retry_attempt TO windmill_user; diff --git a/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql new file mode 100644 index 0000000000..9399c18a6b --- /dev/null +++ b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql @@ -0,0 +1,16 @@ +-- Re-establish the ON DELETE CASCADE foreign keys. Once the cascades were gone the explicit +-- delete paths may have left orphan rows (or none were created); purge any orphans first so +-- the constraints can be validated. +DELETE FROM dispatch_event WHERE producer_job_id NOT IN (SELECT id FROM v2_job); +DELETE FROM flow_conversation_message WHERE job_id IS NOT NULL AND job_id NOT IN (SELECT id FROM v2_job); +DELETE FROM zombie_job_counter WHERE job_id NOT IN (SELECT id FROM v2_job); + +ALTER TABLE dispatch_event + ADD CONSTRAINT dispatch_event_producer_job_id_fkey + FOREIGN KEY (producer_job_id) REFERENCES v2_job(id) ON DELETE CASCADE; +ALTER TABLE flow_conversation_message + ADD CONSTRAINT flow_conversation_message_job_id_fkey + FOREIGN KEY (job_id) REFERENCES v2_job(id) ON DELETE CASCADE; +ALTER TABLE zombie_job_counter + ADD CONSTRAINT zombie_job_counter_job_id_fkey + FOREIGN KEY (job_id) REFERENCES v2_job(id) ON DELETE CASCADE; diff --git a/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql new file mode 100644 index 0000000000..ebaa555b0e --- /dev/null +++ b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql @@ -0,0 +1,11 @@ +-- Drop the ON DELETE CASCADE foreign keys from v2_job's sparse side tables. +-- These cascades made bulk retention deletes (DELETE FROM v2_job WHERE id = ANY(...)) +-- fire a per-row RI trigger for each FK; for flow_conversation_message, whose job_id +-- column is unindexed, that meant a sequential scan of the whole table per deleted row +-- (benchmarked at ~14x the base delete time). Deletion of these tables is now handled +-- explicitly by windmill_common::jobs::delete_jobs and the workspace/export delete paths, +-- following the existing no-FK precedent of job_logs / job_stats / native_retry_attempt. + +ALTER TABLE dispatch_event DROP CONSTRAINT IF EXISTS dispatch_event_producer_job_id_fkey; +ALTER TABLE flow_conversation_message DROP CONSTRAINT IF EXISTS flow_conversation_message_job_id_fkey; +ALTER TABLE zombie_job_counter DROP CONSTRAINT IF EXISTS zombie_job_counter_job_id_fkey; diff --git a/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql b/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql new file mode 100644 index 0000000000..0e90a4b693 --- /dev/null +++ b/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql @@ -0,0 +1 @@ +DROP INDEX IF EXISTS ix_resume_job_flow; diff --git a/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql b/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql new file mode 100644 index 0000000000..7edded95e3 --- /dev/null +++ b/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql @@ -0,0 +1,5 @@ +-- Index resume_job's FK column to v2_job_queue. Without it, every per-job-completion +-- DELETE FROM v2_job_queue (the system's hottest delete path) cascades into a sequential +-- scan of resume_job. The table is small (only currently-suspended flows), so the scan is +-- cheap today, but the index makes the cascade an index probe and removes the footgun. +CREATE INDEX IF NOT EXISTS ix_resume_job_flow ON resume_job (flow); diff --git a/backend/migrations/20260625135355_debounce_batch_consumed.down.sql b/backend/migrations/20260625135355_debounce_batch_consumed.down.sql new file mode 100644 index 0000000000..8ce64f87ac --- /dev/null +++ b/backend/migrations/20260625135355_debounce_batch_consumed.down.sql @@ -0,0 +1,4 @@ +DROP INDEX IF EXISTS idx_v2_job_debounce_batch_consumed_at; +ALTER TABLE v2_job_debounce_batch + DROP COLUMN IF EXISTS consumed_at, + DROP COLUMN IF EXISTS consumed_by; diff --git a/backend/migrations/20260625135355_debounce_batch_consumed.up.sql b/backend/migrations/20260625135355_debounce_batch_consumed.up.sql new file mode 100644 index 0000000000..2559cbb74e --- /dev/null +++ b/backend/migrations/20260625135355_debounce_batch_consumed.up.sql @@ -0,0 +1,17 @@ +-- Claim-based, exactly-once consumption of debounce batches. +-- A batch row is "claimed" by the survivor that accumulates its args. Stamping the +-- row consumed (instead of deleting it) lets a later-pulled survivor of the same +-- batch tell "my contribution was already processed" (consumed_at set -> no-op) +-- apart from "I was never batched" (no row at all; CE / legacy -> run my own args). +-- consumed_at doubles as the GC timestamp. NULL = not yet consumed. +-- consumed_by records which job claimed the row, so a job that is re-pulled (e.g. +-- crash recovery) can tell its own prior claim (keep its accumulated args) apart from +-- a sibling survivor having swept it in (run empty). +ALTER TABLE v2_job_debounce_batch + ADD COLUMN IF NOT EXISTS consumed_at TIMESTAMP WITH TIME ZONE, + ADD COLUMN IF NOT EXISTS consumed_by UUID; + +-- Keeps the GC sweep (delete consumed rows past a grace period) cheap. +CREATE INDEX IF NOT EXISTS idx_v2_job_debounce_batch_consumed_at + ON v2_job_debounce_batch (consumed_at) + WHERE consumed_at IS NOT NULL; diff --git a/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql b/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql new file mode 100644 index 0000000000..8a653a9842 --- /dev/null +++ b/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS materialized_asset_schema; diff --git a/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql b/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql new file mode 100644 index 0000000000..9c548aded7 --- /dev/null +++ b/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql @@ -0,0 +1,37 @@ +-- Captured output schema of a managed `// materialize` asset (gap #2a). +-- After a managed materialize commits, the worker DESCRIBEs the written table +-- and records its column list here as asset-level metadata. Schema is a +-- property of the asset/table, not of a partition slice, so it lives in its own +-- table keyed by (workspace, asset_kind, asset_path) rather than as a column on +-- materialized_partition (which would duplicate the identical schema across +-- every partition row). This is the producer-side capture that #2b (save-time +-- consumer-ref contract enforcement) reads back. +-- +-- Versioning across re-materializations: a new `version` row is inserted only +-- when the captured column set differs from the latest stored version for the +-- asset; an unchanged re-materialize re-affirms the latest row in place. So the +-- table is a compact schema-evolution history and MAX(version) is the current +-- contract. +CREATE TABLE IF NOT EXISTS materialized_asset_schema ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + asset_kind ASSET_KIND NOT NULL, + asset_path VARCHAR(255) NOT NULL, + -- Monotonic per (workspace, asset_kind, asset_path), starting at 1; only + -- bumped when the schema actually changes. + version BIGINT NOT NULL, + -- The captured columns, ordered as the table presents them: + -- [{"name": "...", "type": "..."}, ...]. + columns JSONB NOT NULL, + -- DuckLake snapshot the schema was captured from (NULL for non-ducklake / + -- substrates without snapshots). + snapshot_id BIGINT, + job_id UUID, + captured_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, asset_kind, asset_path, version) +); + +-- Default privileges (migration 20250205131523) only apply to objects created +-- by the role that set them, so grant explicitly — the API reads/writes this +-- table as the invoking role (same fix as script_trigger in 20260619112847). +GRANT ALL ON materialized_asset_schema TO windmill_user; +GRANT ALL ON materialized_asset_schema TO windmill_admin; diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql new file mode 100644 index 0000000000..1ce2c852b7 --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql @@ -0,0 +1,21 @@ +-- Restore the previous anchor: epoch sentinel + preserve-cursor (DO NOTHING). +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', '1970-01-01T00:00:00+00:00' + ) + ) + ON CONFLICT (name) DO NOTHING; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP FUNCTION IF EXISTS audit_logs_s3_oldest_inflight_ts(); diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql new file mode 100644 index 0000000000..ff2ed110c6 --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql @@ -0,0 +1,86 @@ +-- Anchors the audit→object-store export cursor when the setting is enabled. +-- +-- `last_ts`/`last_oldest_inflight_ts` must be a *recent* floor, not epoch: the +-- export's `timestamp >= floor` predicate is the only partition-pruning bound (the +-- `age(xmin)` cursor is unindexable), so an epoch floor would scan the whole +-- `audit_partitioned` table on the first run and never finish under a +-- `statement_timeout`. The floor must be at or below the timestamp of any row whose +-- xid >= this snapshot xmin; the oldest in-flight `xact_start` is that bound when +-- stats are visible (no restricted role / prepared 2PC txn), else a bounded 7-day +-- window. +-- +-- `ON CONFLICT DO UPDATE ... WHERE last_xmin <` keeps the cursor monotonic: a +-- re-enable re-anchors it forward (so the export resumes from ~now rather than +-- rescanning the disabled gap — that gap is the backfill's job), but it never moves +-- backwards, so it is HA-safe and can't be regressed by a slower concurrent writer. +-- +-- The task name literal must match +-- `windmill_common::global_settings::AUDIT_LOGS_S3_EXPORT_TASK`. + +-- Oldest in-flight `xact_start` when this role can observe *all* sessions (so the +-- min is a true cluster-wide bound), else NULL — callers substitute a conservative +-- window. The stats-visibility check goes through `is_superuser` (a preset GUC, no +-- catalog read) first, then a best-effort `pg_has_role` probe guarded by EXCEPTION: +-- `pg_has_role` reads `pg_authid`, which some managed providers (e.g. Cloud SQL) +-- forbid even to read from an elevated context, raising "Modifying pg_authid or +-- pg_auth_members is not allowed in elevated context". The EXCEPTION block runs in +-- its own subtransaction, so a failure there returns NULL (→ conservative fallback) +-- without aborting the caller — the migration-time UPDATE below, the trigger, or the +-- export task, none of which must fail just because the optimization is unavailable. +CREATE OR REPLACE FUNCTION audit_logs_s3_oldest_inflight_ts() +RETURNS timestamptz AS $$ +DECLARE + v_can_read_all_stats boolean := current_setting('is_superuser') = 'on'; +BEGIN + IF NOT v_can_read_all_stats THEN + BEGIN + v_can_read_all_stats := pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'); + EXCEPTION WHEN OTHERS THEN + v_can_read_all_stats := false; + END; + END IF; + IF v_can_read_all_stats AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) THEN + RETURN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL); + END IF; + RETURN NULL; +END; +$$ LANGUAGE plpgsql; + +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', now(), + 'last_oldest_inflight_ts', + COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days') + ) + ) + ON CONFLICT (name) DO UPDATE + SET value = EXCLUDED.value + WHERE (background_task_state.value->>'last_xmin')::bigint + < (EXCLUDED.value->>'last_xmin')::bigint; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +-- Recovery for a legacy epoch-sentinel checkpoint (`last_ts = epoch`, never +-- drained). It cannot be safely resumed: its un-drained backlog can be arbitrarily +-- old, so stamping a recent floor over the old xmin would prune the older rows while +-- the cursor advanced past them (silent loss), and keeping the epoch floor would +-- reintroduce the full scan. Re-anchor it to now like a fresh enable; the pre-anchor +-- window is recoverable via the opt-in backfill, not silently dropped. +UPDATE background_task_state +SET value = jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', to_jsonb(now()), + 'last_oldest_inflight_ts', + to_jsonb(COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))) +WHERE name = 'audit_logs_s3_export' + AND (value->>'last_ts')::timestamptz <= 'epoch'::timestamptz; diff --git a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs index 812bfa785d..f7c578a274 100644 --- a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs +++ b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs @@ -9,8 +9,9 @@ use sqlparser::{ parser::Parser, }; use windmill_parser::asset_parser::{ - asset_was_used, merge_assets, parse_asset_syntax, parse_pipeline_annotations, AssetKind, - AssetUsageAccessType, ParseAssetsOutput, ParseAssetsResult, + asset_was_used, merge_assets, merge_column_lineage, parse_asset_syntax, + parse_pipeline_annotations, AssetKind, AssetUsageAccessType, ColumnLineage, ColumnRef, + ParseAssetsOutput, ParseAssetsResult, }; use AssetUsageAccessType::*; @@ -33,7 +34,55 @@ pub fn parse_assets(input: &str) -> anyhow::Result { } } - let pipeline = parse_pipeline_annotations(input); + let mut pipeline = parse_pipeline_annotations(input); + // Scope inferred lineage to a single output asset so columns from an + // auxiliary CTAS aren't attributed to the materialized one (the flat list + // has no per-entry output on the wire). The `// materialize` target, when + // declared, IS the output: keep entries tagged with it plus untagged + // top-level-SELECT entries (which describe that target). Without a declared + // target, keep inference only when every tagged entry shares one output + // asset — otherwise it's ambiguous which asset the flat list describes, so + // drop it rather than show false dependencies. + let target = pipeline + .materialize + .as_ref() + .map(|m| (m.target_kind, m.target_path.clone())); + let inferred: Vec = match &target { + Some(t) => collector + .column_lineage + .into_iter() + .filter(|(out, _)| out.as_ref().map_or(true, |o| o == t)) + .map(|(_, cl)| cl) + .collect(), + None => { + let mut first: Option<&(AssetKind, String)> = None; + let mut ambiguous = false; + for (out, _) in &collector.column_lineage { + if let Some(o) = out { + match first { + None => first = Some(o), + Some(f) if f != o => { + ambiguous = true; + break; + } + _ => {} + } + } + } + if ambiguous { + Vec::new() + } else { + collector + .column_lineage + .into_iter() + .map(|(_, cl)| cl) + .collect() + } + } + }; + // Body-inferred column lineage, with `// column` annotations taking + // precedence per output column (explicit declaration overrides inference). + pipeline.column_lineage = merge_column_lineage(inferred, pipeline.column_lineage); Ok(ParseAssetsOutput::new( merge_assets(collector.assets), Vec::new(), @@ -54,6 +103,16 @@ struct AssetCollector { cte_name_stack: Vec>, // Locally created tables (not attached to an asset) local_table_names: HashSet, + // Inferred column-level lineage: one entry per output column of an + // output-producing query, mapping it to the upstream source columns its + // expression reads. Each is tagged with the *output asset* it belongs to — + // `Some((kind, path))` for a CTAS / CREATE VIEW into a real asset, `None` + // for a top-level managed-materialize SELECT (its output is the `// + // materialize` target, known only in `parse_assets`). `parse_assets` uses + // the tag to scope the flat list to a single output asset so columns from an + // auxiliary output don't get attributed to the materialized one. Best-effort: + // dynamic/raw SQL, INSERT…SELECT, and wildcards are left to annotations. + column_lineage: Vec<(Option<(AssetKind, String)>, ColumnLineage)>, } impl AssetCollector { @@ -65,15 +124,24 @@ impl AssetCollector { currently_used_asset: None, cte_name_stack: Vec::new(), local_table_names: HashSet::new(), + column_lineage: Vec::new(), } } - /// If the name resolves to an attached asset, record it. Otherwise, register it as a local - /// table/view so that subsequent references are not mistakenly attributed to the active asset. - fn track_table_definition(&mut self, name: &ObjectName) { - if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { - self.assets.push(asset); - } else if let Some(simple_name) = get_trivial_obj_name(name) { + /// Record a `CREATE TABLE`/`VIEW` target. A *temporary* table/view is always + /// local — even a one-part name under an active `USE dl`, which would + /// otherwise resolve to an asset (`ducklake://…/tmp`) and then leak as a + /// column source for later references. A non-temp name that resolves to an + /// attached asset is recorded as that asset; anything else is registered + /// local so subsequent references aren't attributed to the active asset. + fn track_table_definition(&mut self, name: &ObjectName, is_temporary: bool) { + if !is_temporary { + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.assets.push(asset); + return; + } + } + if let Some(simple_name) = get_trivial_obj_name(name) { self.local_table_names.insert(simple_name.to_lowercase()); } } @@ -280,6 +348,22 @@ impl AssetCollector { } } + // Infer the output-column lineage of a query that produces an asset, tagging + // each entry with its `output` asset. Called only for an output-producing + // query — a top-level managed-materialize SELECT (`output: None`, resolved + // to the `// materialize` target later) or a CTAS / CREATE VIEW into a real + // asset (`output: Some`). A CTAS into a local/temp staging table is never + // an output, so it's simply not passed here. + fn infer_query_output( + &mut self, + query: &sqlparser::ast::Query, + output: Option<(AssetKind, String)>, + ) { + if let Some(select) = query.body.as_select() { + self.infer_column_lineage(&select.projection, &select.from, output); + } + } + fn handle_table_with_joins( &mut self, table_with_joins: &sqlparser::ast::TableWithJoins, @@ -302,17 +386,57 @@ impl AssetCollector { } } - // Extract columns from SELECT items and create individual asset results for each column - // Only processes columns that reference known assets to avoid false positives - fn extract_column_assets( - &mut self, - projection: &[SelectItem], + // The alias-map entry (key → asset) for one FROM/JOIN table factor, or + // `None` if it isn't an asset-backed table. The key is its alias, else the + // bare table name; S3 table-functions and string-literal tables are only + // keyed when aliased (an unaliased one is ambiguous). Returns the asset with + // a single matched relation so the caller can attribute qualified columns. + fn table_alias_entry(&self, relation: &TableFactor) -> Option<(String, ParseAssetsResult)> { + let TableFactor::Table { name, alias, args, .. } = relation else { + return None; + }; + let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); + if has_args { + let alias = alias.as_ref()?; + let asset = self.get_s3_asset_from_table_function(relation)?; + return Some((alias.name.value.clone(), asset)); + } + let asset = self + .get_associated_asset_from_obj_name(name, Some(R)) + .or_else(|| self.get_s3_asset_from_str_literal_table(relation))?; + if get_str_lit_from_obj_name(name).is_some() { + // String-literal S3 table: only unambiguous when aliased. + let alias = alias.as_ref()?; + return Some((alias.name.value.clone(), asset)); + } + let key = match alias { + Some(a) => a.name.value.clone(), + None => name + .0 + .last() + .and_then(|id| id.as_ident()) + .map(|id| id.value.clone()) + .unwrap_or_default(), + }; + Some((key, asset)) + } + + // Resolve a query's FROM clause into (single-table asset, alias→asset map). + // `single_table` is `Some` only for an unambiguous one-table FROM with no + // joins (so bare column refs can be attributed); `table_to_asset` keys by + // alias/table name for qualified refs and includes every JOINed table. + // Shared by `extract_column_assets` (read columns) and `infer_column_lineage` + // (output→input edges) so both resolve identically. + fn build_from_maps( + &self, from_tables: &[sqlparser::ast::TableWithJoins], + ) -> ( + Option, + BTreeMap, ) { - // Check if this is a single-table SELECT (to avoid ambiguity). - // For S3 table functions (read_parquet/read_csv/read_json), detect the asset even - // though args are present, since we know the file path from the string literal arg. - let single_table = if from_tables.len() == 1 { + // Single unambiguous table only when there's exactly one FROM entry AND + // it has no joins — otherwise a bare column could belong to any side. + let single_table = if from_tables.len() == 1 && from_tables[0].joins.is_empty() { let relation = &from_tables[0].relation; if let TableFactor::Table { name, args, .. } = relation { let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); @@ -329,51 +453,30 @@ impl AssetCollector { None }; - // Build a map of table aliases/names to assets for multi-table queries. - // For S3 table functions, only aliased references are unambiguous - // (e.g. SELECT t.col1 FROM read_parquet('s3://...') AS t). + // Alias → asset for qualified column refs, across every FROM entry AND + // its JOINed tables (so `c.col` in `FROM a JOIN c` resolves). let mut table_to_asset: BTreeMap = BTreeMap::new(); for table_with_joins in from_tables { - if let TableFactor::Table { name, alias, args, .. } = &table_with_joins.relation { - let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); - if has_args { - // For table functions, only add to the alias map when an alias is present - if let Some(alias) = alias { - if let Some(asset) = - self.get_s3_asset_from_table_function(&table_with_joins.relation) - { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } - } else if let Some(asset) = self - .get_associated_asset_from_obj_name(name, Some(R)) - .or_else(|| { - self.get_s3_asset_from_str_literal_table(&table_with_joins.relation) - }) - { - // For string literal S3 tables (e.g. FROM 's3:///file.parquet'), only add to - // the alias map when an alias is present (to avoid false positives). - // For regular named tables, use alias or table name as key. - let is_str_literal = get_str_lit_from_obj_name(name).is_some(); - if is_str_literal { - if let Some(alias) = alias { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } else { - let table_key = if let Some(alias) = alias { - alias.name.value.clone() - } else { - name.0 - .last() - .and_then(|id| id.as_ident()) - .map(|id| id.value.clone()) - .unwrap_or_default() - }; - table_to_asset.insert(table_key, asset); - } + if let Some((k, a)) = self.table_alias_entry(&table_with_joins.relation) { + table_to_asset.insert(k, a); + } + for join in &table_with_joins.joins { + if let Some((k, a)) = self.table_alias_entry(&join.relation) { + table_to_asset.insert(k, a); } } } + (single_table, table_to_asset) + } + + // Extract columns from SELECT items and create individual asset results for each column + // Only processes columns that reference known assets to avoid false positives + fn extract_column_assets( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); // Process each SELECT item for item in projection { @@ -446,6 +549,136 @@ impl AssetCollector { } } } + + // Infer column-level lineage for an output-producing query's projection: + // each *named* output column → the upstream source columns its expression + // reads. Covers passthroughs (`amount`) and computed columns + // (`amount + tax AS total`) alike. Skipped: wildcards and unaliased + // expressions (no stable output name), and inputs that don't resolve to a + // known asset. A column with no resolved inputs is dropped. + // + // Best-effort and intentionally flat: results from every output query in the + // script accumulate into one list (the graph hangs them off the materialize + // write-edge), with no per-output-table association. This is exact for the + // common single-output member (a managed-materialize SELECT, or one CTAS), + // but a multi-statement script that stages through a TEMP table reports the + // *intermediate* column names (the final SELECT reads the temp table, whose + // columns don't resolve to an asset, so they drop out). A `// column` + // annotation overrides any output column where inference is wrong or coarse. + fn infer_column_lineage( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + output: Option<(AssetKind, String)>, + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); + for item in projection { + let (out_col, expr) = match item { + SelectItem::ExprWithAlias { expr, alias } => (alias.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::Identifier(id)) => (id.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::CompoundIdentifier(parts)) => { + match parts.last() { + Some(last) => (last.value.clone(), expr), + None => continue, + } + } + _ => continue, + }; + let mut collector = ColumnIdentCollector { refs: Vec::new(), query_depth: 0 }; + let _ = expr.visit(&mut collector); + let mut inputs: Vec = Vec::new(); + for parts in &collector.refs { + if let Some(cr) = self.resolve_column_ref(parts, &single_table, &table_to_asset) { + if !inputs.contains(&cr) { + inputs.push(cr); + } + } + } + if !inputs.is_empty() { + self.column_lineage + .push((output.clone(), ColumnLineage { column: out_col, inputs })); + } + } + } + + // Resolve identifier `parts` (e.g. `["t","amount"]` or `["amount"]`) to the + // source asset column it reads, mirroring `extract_column_assets`' + // resolution: a bare ident needs an unambiguous single-table FROM; a + // qualified ident resolves its prefix via the alias map, or (≥3 parts) as a + // db/schema-qualified object name. + fn resolve_column_ref( + &self, + parts: &[String], + single_table: &Option, + table_to_asset: &BTreeMap, + ) -> Option { + let asset_to_ref = |asset: &ParseAssetsResult, col: &str| ColumnRef { + from_kind: asset.kind, + from_path: asset.path.clone(), + from_column: col.to_string(), + }; + match parts { + [col] => single_table.as_ref().map(|a| asset_to_ref(a, col)), + [.., col] => { + let prefix = parts.first()?; + if let Some(asset) = table_to_asset.get(prefix) { + Some(asset_to_ref(asset, col)) + } else if parts.len() >= 3 { + let obj_parts: Vec = parts[..parts.len() - 1] + .iter() + .map(|p| ObjectNamePart::Identifier(sqlparser::ast::Ident::new(p.clone()))) + .collect(); + let asset = + self.get_associated_asset_from_obj_name(&ObjectName(obj_parts), Some(R))?; + Some(asset_to_ref(&asset, col)) + } else { + None + } + } + [] => None, + } + } +} + +// Collects the identifier paths an expression reads, for column-lineage +// inference: `Expr::Identifier(a)` → `["a"]`, `Expr::CompoundIdentifier(t.a)` → +// `["t","a"]`. The derived `Visit` walk recurses through operators, functions, +// casts and CASE, so every leaf identifier of the outer expression is captured. +struct ColumnIdentCollector { + refs: Vec>, + // Depth of nested (sub)queries inside the expression. Identifiers are only + // captured at depth 0: a scalar/correlated subquery's columns belong to ITS + // own FROM, not the outer projection's, so descending would misattribute + // (e.g. `(SELECT x FROM other) AS c FROM orders` must NOT bind `c` to + // `orders.x`). Subquery-derived columns are simply left to annotations. + query_depth: usize, +} + +impl Visitor for ColumnIdentCollector { + type Break = (); + + fn pre_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth += 1; + std::ops::ControlFlow::Continue(()) + } + + fn post_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth = self.query_depth.saturating_sub(1); + std::ops::ControlFlow::Continue(()) + } + + fn pre_visit_expr(&mut self, expr: &Expr) -> std::ops::ControlFlow { + if self.query_depth == 0 { + match expr { + Expr::Identifier(id) => self.refs.push(vec![id.value.clone()]), + Expr::CompoundIdentifier(parts) => self + .refs + .push(parts.iter().map(|id| id.value.clone()).collect()), + _ => {} + } + } + std::ops::ControlFlow::Continue(()) + } } impl Visitor for AssetCollector { @@ -505,7 +738,11 @@ impl Visitor for AssetCollector { ) -> std::ops::ControlFlow { match statement { sqlparser::ast::Statement::Query(q) => { + // A top-level SELECT is the managed-materialize output, so its + // columns ARE the materialized asset's columns (output resolved + // to the `// materialize` target in `parse_assets`). self.handle_query_reads(q); + self.infer_query_output(q, None); } sqlparser::ast::Statement::Insert(insert) => { @@ -658,18 +895,32 @@ impl Visitor for AssetCollector { } sqlparser::ast::Statement::CreateTable(create_table) => { - self.track_table_definition(&create_table.name); + self.track_table_definition(&create_table.name, create_table.temporary); // `CREATE TABLE x AS SELECT … FROM y` reads y. The AS-query // isn't a `Statement::Query`, so its FROM tables are only - // caught here. + // caught here. Only infer output lineage when `x` is a real + // asset — a CTAS into a local/temp staging table is not the + // materialized output (its columns aren't the asset's). if let Some(query) = &create_table.query { self.handle_query_reads(query); + // Infer only when `x` is a real asset (its output columns + // ARE that asset's), tagged with it so `parse_assets` can + // scope lineage per output. A local/temp staging table is + // not an asset → not inferred. + if let Some(asset) = + self.get_associated_asset_from_obj_name(&create_table.name, Some(W)) + { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } } } - sqlparser::ast::Statement::CreateView { name, query, .. } => { - self.track_table_definition(name); + sqlparser::ast::Statement::CreateView { name, query, temporary, .. } => { + self.track_table_definition(name, *temporary); self.handle_query_reads(query); + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } } // DROP TABLE/VIEW is a write to the dropped object — the @@ -687,7 +938,9 @@ impl Visitor for AssetCollector { | sqlparser::ast::ObjectType::MaterializedView ) { for name in names { - self.track_table_definition(name); + // DROP is a write to the named object; resolve it as an + // asset if it is one (not a temp-creation context). + self.track_table_definition(name, false); } } } @@ -1920,8 +2173,9 @@ mod ctas_read_tests { assets ); assert!( - assets.iter().any(|a| a.path == "main/exciting_809" - && a.access_type == Some(W)), + assets + .iter() + .any(|a| a.path == "main/exciting_809" && a.access_type == Some(W)), "expected write of main/exciting_809, got {:?}", assets ); @@ -1935,9 +2189,292 @@ mod ctas_read_tests { "#; let assets = parse_assets(input).unwrap().assets; assert!( - assets.iter().any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), + assets + .iter() + .any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), "expected read of main/fx_rates, got {:?}", assets ); } + + fn lineage(input: &str) -> Vec { + parse_assets(input).unwrap().column_lineage + } + + #[test] + fn test_infer_lineage_computed_and_passthrough() { + // CTAS with a computed column (amount + tax) and a passthrough (id). + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_bare_column_single_table() { + // Managed-materialize form: a plain top-level SELECT, bare columns + // attributed to the single FROM table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + SELECT amount AS revenue FROM orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "revenue".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_annotation_overrides() { + // The `// column` annotation for `order_total` wins; `id` stays inferred. + let input = r#" + -- column order_total <- datatable://prod/manual.grand_total + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + // Annotation entry is authoritative and listed first. + assert_eq!(got[0].column, "order_total"); + assert_eq!(got[0].inputs[0].from_path, "prod/manual"); + assert_eq!(got[0].inputs[0].from_column, "grand_total"); + // Inferred `id` survives; inferred `order_total` dropped (no dup). + assert!(got.iter().any(|c| c.column == "id")); + assert_eq!(got.iter().filter(|c| c.column == "order_total").count(), 1); + } + + #[test] + fn test_infer_lineage_skips_local_staging_ctas() { + // A CTAS into a TEMP/local table is NOT the materialized output, so its + // columns must not be reported (they'd be anchored to the script's + // `// materialize` target as if they were the final asset's columns). + // The final SELECT reads the local staging table → unresolved → empty. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TEMP TABLE tmp AS SELECT dl.orders.amount AS amt FROM dl.orders; + SELECT amt AS total FROM tmp; + "#; + assert!( + lineage(input).is_empty(), + "staging columns must not be reported as final output; got {:?}", + lineage(input) + ); + } + + #[test] + fn test_infer_lineage_ctas_into_asset_still_inferred() { + // A CTAS whose target IS an asset is the output, so it's still inferred. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT dl.orders.amount AS amt FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "amt".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_temp_table_under_use_is_local() { + // A one-part TEMP table name under an active `USE dl` must NOT resolve to + // an asset (`warehouse/tmp`); it's local, so the final SELECT reading it + // can't invent `warehouse/tmp.amt` as a column source for the output. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + CREATE TEMP TABLE tmp AS SELECT amount AS amt FROM orders; + SELECT amt AS total FROM tmp; + "#; + let got = lineage(input); + assert!( + got.is_empty(), + "temp staging under USE must not leak warehouse/tmp as a source; got {:?}", + got + ); + // And no phantom `warehouse/tmp` asset is recorded. + let assets = parse_assets(input).unwrap().assets; + assert!( + !assets.iter().any(|a| a.path == "warehouse/tmp"), + "temp table must not be recorded as an asset; got {:?}", + assets + ); + } + + #[test] + fn test_infer_lineage_scopes_to_materialize_target() { + // A script with a `// materialize` target plus an AUXILIARY CTAS into a + // different asset: only the materialized target's columns are reported; + // the auxiliary output's columns must not be attributed to it. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.audit AS SELECT dl.orders.id AS aid FROM dl.orders; + SELECT dl.orders.amount AS total FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }], + "auxiliary `audit` columns must not appear on the materialized target" + ); + } + + #[test] + fn test_infer_lineage_drops_ambiguous_multi_output() { + // No `// materialize` target and two real CTAS outputs: which asset the + // flat lineage describes is ambiguous, so inference is dropped rather + // than attributed to an arbitrary one. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.a AS SELECT dl.orders.id AS x FROM dl.orders; + CREATE TABLE dl.b AS SELECT dl.orders.amount AS y FROM dl.orders; + "#; + assert!( + lineage(input).is_empty(), + "ambiguous multi-output must drop inference" + ); + } + + #[test] + fn test_infer_lineage_wildcard_yields_nothing() { + // `SELECT *` has no enumerable output columns → no inferred lineage. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT * FROM dl.orders; + "#; + assert!(lineage(input).is_empty()); + } + + #[test] + fn test_infer_lineage_resolves_joined_inputs() { + // Columns from BOTH sides of an explicit JOIN must resolve, incl. a + // computed column mixing the two. A bare column is dropped (ambiguous + // across the join) rather than misattributed to the first table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT o.id, c.region AS cust_region, o.amount + c.discount AS net + FROM dl.orders o + JOIN dl.customers c ON c.id = o.customer_id; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "cust_region".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "region".to_string(), + }], + }, + ColumnLineage { + column: "net".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "discount".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_does_not_descend_into_subqueries() { + // A scalar subquery's bare column (`amount`) belongs to the subquery's + // own FROM, NOT the outer `dl.orders` — it must not be attributed to the + // outer table. The subquery column is left to annotations; the + // passthrough `id` still resolves. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, (SELECT amount FROM dl.other LIMIT 1) AS c + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }], + "subquery column `c` must be dropped, not misattributed to orders" + ); + } } diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index b5d69db97b..096d93163b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.734.0" +version = "1.742.0" dependencies = [ "aho-corasick", "anyhow", @@ -6272,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.734.0" +version = "1.742.0" dependencies = [ "proc-macro2", "quote", @@ -6284,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.734.0" +version = "1.742.0" dependencies = [ "convert_case", "serde", @@ -6293,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6305,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6317,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "gosyn", @@ -6329,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6341,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6353,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "nu-parser", @@ -6364,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6375,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6387,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6398,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -6420,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6432,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6446,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "convert_case", @@ -6463,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6476,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -6488,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6506,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6522,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6538,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6570,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -6581,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.734.0" +version = "1.742.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 5e38fac8cf..e2afd2483b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.734.0" +version = "1.742.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/parsers/windmill-parser/src/asset_parser.rs b/backend/parsers/windmill-parser/src/asset_parser.rs index 61e30a343f..2b7f6ecdbf 100644 --- a/backend/parsers/windmill-parser/src/asset_parser.rs +++ b/backend/parsers/windmill-parser/src/asset_parser.rs @@ -112,6 +112,16 @@ pub struct ParseAssetsOutput { // Drives the worker's write-strategy + snapshot capture. #[serde(skip_serializing_if = "Option::is_none", default)] pub materialize: Option, + // `// data_test …` — data-quality assertions run against the + // materialized asset after the write commits. Accumulating (multiple + // lines allowed). Drives the worker's post-materialize verifier probes. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub data_tests: Vec, + // `// column <- .[, …]` — declared column-level lineage, + // one entry per output column. Accumulating. Pure metadata: drives the + // column-lineage graph view, executes nothing. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub column_lineage: Vec, } #[derive(Serialize, Debug, PartialEq, Clone)] @@ -235,6 +245,74 @@ pub struct MaterializeSpec { pub unique_key: Option, } +// `// data_test …` — a data-quality assertion run against the +// freshly-materialized asset (post DELETE+INSERT), failing the run on +// violation. The first extensible annotation family: the parser turns a +// `data_test` line into one of a known *vocabulary* of checks, and the +// runtime turns each check into a SQL "verifier" probe. A sibling annotation +// family (e.g. column-lineage) follows the same shape — a keyword head +// selecting a variant, the rest parsed per-variant — rather than growing a +// new closed list. See `docs/ducklake-materialization.md` §"Extensible +// annotations". Multiple `// data_test` lines accumulate (unlike the +// single-value annotations above, which are first-write-wins). +// +// Built-ins mirror dbt's generic data tests; `Custom` is the escape hatch +// (dbt's singular test): a DuckDB script path whose SELECT returns the +// violating rows. The keyword is `data_test` — NOT `test` — to stay clear +// of the unrelated `// test:` CI-test annotation (see +// `windmill_common::schema::parse_ci_test_annotation`), matching dbt 1.8's +// own `tests:` → `data_tests:` rename. +#[derive(Serialize, Debug, PartialEq, Clone)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum DataTest { + // `// data_test unique ` — no two non-NULL rows share `column`. + Unique { column: String }, + // `// data_test not_null ` — `column` is never NULL. + NotNull { column: String }, + // `// data_test accepted_values = a,b,c` — every non-NULL value of + // `column` is one of `values` (comma-separated; surrounding quotes stripped). + AcceptedValues { column: String, values: Vec }, + // `// data_test relationships -> .` — referential + // integrity: every non-NULL `column` value exists in `to_path`'s `to_column`. + Relationships { column: String, to_kind: AssetKind, to_path: String, to_column: String }, + // `// data_test ` — escape hatch: a deployed DuckDB script + // whose trailing SELECT returns the violating rows (non-empty ⇒ fail). + Custom { path: String }, +} + +// `// column <- .[, …]` — declared column-level +// lineage: one output column of this script's produced asset and the upstream +// source columns it derives from. A sibling of `DataTest` in the extensible +// annotation family (`docs/pipelines-vs-dbt.md` §3): same parse shape — a head +// token (the output column) then a per-variant tail — but accumulating, one +// line per output column. Unlike `data_test` these are pure metadata: they +// drive the column-lineage graph view, never a runtime probe. +// +// dbt derives column lineage from SQL-AST parsing; Windmill is polyglot +// (Python/TS/Bash/SQL in one DAG), so a uniform AST is not available. The +// annotation is the explicit, language-agnostic declaration — the same +// "annotations are real comments parsed strictly" stance as the rest of the +// pipeline grammar. Body-inferred per-asset column *sets* (`columns` on +// `ParseAssetsResult`) complement it but cannot express column→column edges. +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnLineage { + // The produced asset's output column this line describes. + pub column: String, + // Upstream source columns it derives from (≥1; malformed refs dropped). + pub inputs: Vec, +} + +// One `.` upstream reference inside a `// column` line. The +// asset URI accepts the default-syntax shorthands (like `// materialize` / +// `// data_test relationships`); the column is the segment after the final +// `.` (so a schema-qualified `warehouse/main.orders.amount` keeps `amount`). +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnRef { + pub from_kind: AssetKind, + pub from_path: String, + pub from_column: String, +} + // `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger // firing runs the script (current behaviour). `All` = AND: the script // runs only once every partition-bearing input has materialized at the @@ -266,6 +344,8 @@ pub struct PipelineAnnotations { pub tag: Option, pub retry: Option, pub materialize: Option, + pub data_tests: Vec, + pub column_lineage: Vec, } impl ParseAssetsOutput { @@ -290,10 +370,34 @@ impl ParseAssetsOutput { tag: pipeline.tag, retry: pipeline.retry, materialize: pipeline.materialize, + data_tests: pipeline.data_tests, + column_lineage: pipeline.column_lineage, } } } +// Combine column lineage inferred from the body (SQL AST) with lineage declared +// via `// column` annotations. The annotation is the *override*: where both +// describe the same output column, the explicit declaration wins and the +// inferred entry is dropped. Inferred entries are also deduped by output column +// among themselves (first wins). Used by the language asset-parsers so a +// `// column` line can correct a mis-inferred edge without disabling inference +// for the rest of the columns. +pub fn merge_column_lineage( + inferred: Vec, + annotated: Vec, +) -> Vec { + let mut seen: std::collections::HashSet = + annotated.iter().map(|c| c.column.clone()).collect(); + let mut out = annotated; + for c in inferred { + if seen.insert(c.column.clone()) { + out.push(c); + } + } + out +} + #[derive(Debug, Clone, Serialize)] pub struct DelegateToGitRepoDetails { pub resource: String, @@ -487,9 +591,13 @@ fn parse_kv_opts(s: &str) -> BTreeMap { out } -// Scan raw source for pipeline annotations. Language-agnostic: any line -// whose first non-whitespace tokens are a comment prefix (`//`, `#`, or -// `--`) followed by one of the recognized keywords: +// Scan the leading comment header for pipeline annotations. Only the +// contiguous block of comment lines at the top of the file is considered +// (blank lines tolerated, scan stops at the first line of actual code) so +// that ordinary comments in the body can't false-positive as annotations. +// Language-agnostic: any header line whose first non-whitespace tokens are +// a comment prefix (`//`, `#`, or `--`) followed by one of the recognized +// keywords: // - `pipeline` → opt-in marker (must be alone on the line) // - `on ` → asset / native trigger edge (including // the marker-only `on schedule` form) @@ -527,6 +635,9 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { for raw_line in code.lines() { let line = raw_line.trim_start(); + if line.is_empty() { + continue; + } let rest = if let Some(r) = line.strip_prefix("//") { r } else if let Some(r) = line.strip_prefix("--") { @@ -534,7 +645,11 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { } else if let Some(r) = line.strip_prefix('#') { r } else { - continue; + // Annotations live in the leading comment header. Stop at the first + // line of actual code so comments inside the body (e.g. a regular + // `# tag ...` prose comment) can't false-positive as annotations. + // Mirrors BashAnnotations::sandbox_image / ssh_target. + break; }; let rest = rest.trim_start(); @@ -584,7 +699,14 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { if let Some(after_kw) = consume_keyword(rest, "tag") { let name = after_kw.trim(); - if !name.is_empty() && out.tag.is_none() { + // Worker tags are single-word identifiers (e.g. `heavy`, `gpu`). + // A value with whitespace or beyond the `script.tag` column width + // is almost certainly a regular comment starting with "# tag ...". + if !name.is_empty() + && !name.contains(char::is_whitespace) + && name.len() <= 50 + && out.tag.is_none() + { out.tag = Some(name.to_string()); } continue; @@ -608,6 +730,29 @@ pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { continue; } + // `data_test` is checked before `on`/asset shorthands and is a complete + // word (so it never collides with the `// test:` CI annotation, which + // has no whitespace after `test`). Accumulates — every well-formed line + // adds a check; malformed lines are dropped (fail-safe, the missing + // check is then simply absent from the graph + run). + if let Some(after_kw) = consume_keyword(rest, "data_test") { + if let Some(spec) = parse_data_test_spec(after_kw.trim()) { + out.data_tests.push(spec); + } + continue; + } + + // `// column <- .[, …]` — accumulating column lineage. + // A complete word, so it never swallows a body comment that happens to + // start with `column` followed by non-lineage prose (that has no `<-` + // and is dropped fail-safe). Checked before `on`/asset shorthands. + if let Some(after_kw) = consume_keyword(rest, "column") { + if let Some(spec) = parse_column_lineage_spec(after_kw.trim()) { + out.column_lineage.push(spec); + } + continue; + } + if let Some(after_kw) = consume_keyword(rest, "on") { let spec_text = after_kw.trim(); if spec_text.is_empty() { @@ -684,6 +829,123 @@ fn parse_materialize_spec(s: &str) -> Option { Some(MaterializeSpec { target_kind, target_path: path.to_string(), manual, append, unique_key }) } +// Parse a `// data_test …` right-hand side into one `DataTest`. The +// leading token selects the variant; the remainder is parsed per-variant. +// Anything not matching a built-in keyword is the `Custom` escape hatch — a +// single script-path token. Returns `None` for malformed input so a typo +// fails safe (the check is dropped, never silently mis-parsed). +// +// This is the extension seam: a new built-in is one match arm + its parser; +// a sibling annotation family (column-lineage) reuses the same head-keyword +// dispatch shape rather than adding a parallel closed list. +fn parse_data_test_spec(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() { + return None; + } + let mut it = s.splitn(2, char::is_whitespace); + let head = it.next()?; + let rest = it.next().unwrap_or("").trim(); + match head { + "unique" => Some(DataTest::Unique { column: single_ident(rest)? }), + "not_null" => Some(DataTest::NotNull { column: single_ident(rest)? }), + "accepted_values" => parse_accepted_values(rest), + "relationships" => parse_relationships(rest), + // Custom escape hatch: the whole right-hand side must be one path token + // (`head` with no trailing content). Trailing content after a + // non-built-in head is a malformed built-in (e.g. `uniq order_id`) and + // is rejected rather than misread as a path. + _ if rest.is_empty() => Some(DataTest::Custom { path: head.to_string() }), + _ => None, + } +} + +// A single bare identifier token (column name). Rejects empty / multi-token +// input. The identifier is double-quoted + escaped at codegen, so any +// character is safe here; we only enforce "exactly one token". +fn single_ident(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() || s.split_whitespace().count() != 1 { + return None; + } + Some(s.to_string()) +} + +// Strip one layer of matching surrounding single or double quotes. +fn unquote(s: &str) -> &str { + let b = s.as_bytes(); + if b.len() >= 2 && (b[0] == b'"' || b[0] == b'\'') && b[b.len() - 1] == b[0] { + &s[1..s.len() - 1] + } else { + s + } +} + +// ` = a,b,c` — column, then `=`, then a comma-separated value list. +// Surrounding quotes are stripped per value; empty values are dropped; a +// value may not itself contain a comma (v1 limitation). +fn parse_accepted_values(s: &str) -> Option { + let (col, vals) = s.split_once('=')?; + let column = single_ident(col)?; + let values: Vec = vals + .split(',') + .map(|v| unquote(v.trim()).to_string()) + .filter(|v| !v.is_empty()) + .collect(); + if values.is_empty() { + return None; + } + Some(DataTest::AcceptedValues { column, values }) +} + +// ` -> .` — referential integrity. The referenced +// column is the segment after the final `.`; everything before it is the +// asset URI (default-syntax shorthands enabled, like `// materialize`). +fn parse_relationships(s: &str) -> Option { + let (col, target) = s.split_once("->")?; + let column = single_ident(col)?; + let target = target.trim(); + let (asset_uri, ref_col) = target.rsplit_once('.')?; + let to_column = single_ident(ref_col)?; + let (to_kind, to_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if to_path.is_empty() { + return None; + } + Some(DataTest::Relationships { column, to_kind, to_path: to_path.to_string(), to_column }) +} + +// Parse a `// column <- [, …]` right-hand side. The head +// (before `<-`) is the output column; the tail is a comma-separated list of +// `.` upstream references. Mirrors `parse_accepted_values`' +// "drop empties, require ≥1" stance: individually malformed refs are dropped +// and the line is kept iff at least one ref parses; a missing `<-`, a non-ident +// output column, or zero valid refs drops the whole line (fail-safe). +fn parse_column_lineage_spec(s: &str) -> Option { + let (out_col, refs) = s.split_once("<-")?; + let column = single_ident(out_col)?; + let inputs: Vec = refs + .split(',') + .filter_map(|r| parse_column_ref(r.trim())) + .collect(); + if inputs.is_empty() { + return None; + } + Some(ColumnLineage { column, inputs }) +} + +// `.` — the referenced column is the segment after the final +// `.`; everything before it is the asset URI (default-syntax shorthands +// enabled, like `// materialize`). Same shape as `parse_relationships`' target. +fn parse_column_ref(s: &str) -> Option { + let (asset_uri, ref_col) = s.rsplit_once('.')?; + let from_column = single_ident(ref_col)?; + let (from_kind, from_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if from_path.is_empty() { + return None; + } + Some(ColumnRef { from_kind, from_path: from_path.to_string(), from_column }) +} + // Parse a `// partitioned [opts]` right-hand side. Recognized kinds: // `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start), // and `dynamic key=""` (plus optional format). @@ -1074,6 +1336,56 @@ mod pipeline_annotation_tests { assert!(out.tag.is_none()); } + #[test] + fn tag_with_whitespace_is_skipped() { + // A regular English comment starting with "# tag " must not be + // mistaken for a worker-tag annotation (worker tags are single words). + let out = + parse_pipeline_annotations("# tag this function so we remember to refactor it later"); + assert!(out.tag.is_none()); + } + + #[test] + fn tag_too_long_is_skipped() { + let long = "x".repeat(51); + let out = parse_pipeline_annotations(&format!("// tag {long}")); + assert!(out.tag.is_none()); + } + + #[test] + fn annotations_in_body_are_ignored() { + // Only the leading comment header is scanned. A regular `# tag ...` + // prose comment buried in the body — the WIN-2090 false-positive that + // crashed the `script.tag` INSERT — must not be treated as an + // annotation once real code has started. + let code = concat!( + "import pandas as pd\n", + "\n", + "def main():\n", + " # tag each row with its source so downstream steps can filter\n", + " # on s3://should/not/parse\n", + " return pd.DataFrame()\n", + ); + let out = parse_pipeline_annotations(code); + assert!(out.tag.is_none()); + assert!(out.triggers.is_empty()); + } + + #[test] + fn header_allows_blank_lines_before_code() { + // Blank lines (e.g. after a shebang) don't end the header; the first + // line of real code does. + let code = concat!( + "#!/usr/bin/env python\n", + "\n", + "# tag heavy\n", + "import os\n", + "# tag light\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!(out.tag.as_deref(), Some("heavy")); + } + #[test] fn retry_count_only() { let out = parse_pipeline_annotations("// retry 3"); @@ -1240,4 +1552,230 @@ mod pipeline_annotation_tests { assert_eq!(m.get("b").unwrap(), "fine"); assert!(m.get("garbage").is_none()); } + + #[test] + fn data_test_builtins() { + let code = concat!( + "// data_test unique order_id\n", + "// data_test not_null user_id\n", + "// data_test accepted_values status = paid,pending,refunded\n", + "// data_test relationships user_id -> datatable://prod/users.id\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.data_tests, + vec![ + DataTest::Unique { column: "order_id".to_string() }, + DataTest::NotNull { column: "user_id".to_string() }, + DataTest::AcceptedValues { + column: "status".to_string(), + values: vec![ + "paid".to_string(), + "pending".to_string(), + "refunded".to_string() + ], + }, + DataTest::Relationships { + column: "user_id".to_string(), + to_kind: AssetKind::DataTable, + to_path: "prod/users".to_string(), + to_column: "id".to_string(), + }, + ] + ); + } + + #[test] + fn data_test_accepts_quotes_and_spacing() { + let out = parse_pipeline_annotations("// data_test accepted_values kind = \"a b\", 'c' ,d"); + assert_eq!( + out.data_tests, + vec![DataTest::AcceptedValues { + column: "kind".to_string(), + values: vec!["a b".to_string(), "c".to_string(), "d".to_string()], + }] + ); + } + + #[test] + fn data_test_custom_escape_hatch() { + // A non-built-in single token is a custom script path; default-syntax + // asset shorthands are NOT triggered here (a path is just a path). + let out = parse_pipeline_annotations("// data_test f/tests/orders_amount_sane"); + assert_eq!( + out.data_tests, + vec![DataTest::Custom { path: "f/tests/orders_amount_sane".to_string() }] + ); + } + + #[test] + fn data_test_relationships_ducklake_shorthand() { + let out = parse_pipeline_annotations( + "// data_test relationships sku -> ducklake://warehouse/dim_products.sku", + ); + assert_eq!( + out.data_tests, + vec![DataTest::Relationships { + column: "sku".to_string(), + to_kind: AssetKind::Ducklake, + to_path: "warehouse/dim_products".to_string(), + to_column: "sku".to_string(), + }] + ); + } + + #[test] + fn data_test_malformed_dropped_fail_safe() { + // A misspelled built-in with trailing content is not a valid path token + // → dropped, not misread as a custom test. An empty value list, a + // missing arrow target, and a bare keyword are all dropped too. + let out = parse_pipeline_annotations(concat!( + "// data_test uniq order_id\n", // typo'd built-in + arg + "// data_test accepted_values s =\n", // no values + "// data_test relationships a -> b\n", // no `.refcol` + "// data_test unique\n", // missing column + "// data_test\n", // bare keyword + )); + assert!(out.data_tests.is_empty()); + } + + #[test] + fn data_test_not_confused_with_ci_test_annotation() { + // `// test:` is the unrelated CI-test annotation — it must NOT be + // parsed as a data test (no whitespace after `test`, and the keyword + // is `data_test` anyway). + let out = parse_pipeline_annotations("// test: f/foo/bar\n// data_test unique id"); + assert_eq!( + out.data_tests, + vec![DataTest::Unique { column: "id".to_string() }] + ); + } + + #[test] + fn column_lineage_basic() { + let code = concat!( + "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n", + "// column user_name <- datatable://prod/users.name\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.column_lineage, + vec![ + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ColumnLineage { + column: "user_name".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/users".to_string(), + from_column: "name".to_string(), + }], + }, + ] + ); + } + + #[test] + fn column_lineage_schema_qualified_keeps_last_dot_as_column() { + // The column is the segment after the FINAL dot, so a schema-qualified + // ducklake table (`main.dim_products`) survives intact. + let out = parse_pipeline_annotations( + "// column sku <- ducklake://warehouse/main.dim_products.sku", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "sku".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/main.dim_products".to_string(), + from_column: "sku".to_string(), + }], + }] + ); + } + + #[test] + fn column_lineage_drops_malformed_refs_keeps_valid() { + // `bad_no_dot` has no `.col` and is dropped; the line survives on its + // one valid ref. Mirrors accepted_values' drop-empties-keep-≥1 stance. + let out = parse_pipeline_annotations( + "// column total <- bad_no_dot, datatable://prod/orders.amount", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn merge_column_lineage_annotation_overrides_inferred() { + let inferred = vec![ + ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "amount".to_string(), + }], + }, + ColumnLineage { + column: "qty".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "qty".to_string(), + }], + }, + ]; + // Annotation redefines `total` (wins) and leaves `qty` to inference. + let annotated = vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/x".to_string(), + from_column: "grand_total".to_string(), + }], + }]; + let merged = merge_column_lineage(inferred, annotated); + assert_eq!(merged.len(), 2); + // Annotation entry kept first and authoritative. + assert_eq!(merged[0].column, "total"); + assert_eq!(merged[0].inputs[0].from_column, "grand_total"); + // Inferred `qty` survives (no annotation for it); inferred `total` dropped. + assert_eq!(merged[1].column, "qty"); + } + + #[test] + fn column_lineage_malformed_lines_dropped_fail_safe() { + // No arrow, a multi-token output column, and a line whose every ref is + // malformed are all dropped entirely. + let out = parse_pipeline_annotations(concat!( + "// column no_arrow datatable://prod/x.y\n", // missing `<-` + "// column a b <- datatable://prod/x.y\n", // output not a single ident + "// column total <- bad_no_dot\n", // no valid ref + "// column\n", // bare keyword + )); + assert!(out.column_lineage.is_empty()); + } } diff --git a/backend/parsers/windmill-parser/src/sql_materialize.rs b/backend/parsers/windmill-parser/src/sql_materialize.rs index c1b0cc97aa..7a3c72bdce 100644 --- a/backend/parsers/windmill-parser/src/sql_materialize.rs +++ b/backend/parsers/windmill-parser/src/sql_materialize.rs @@ -482,7 +482,8 @@ pub fn build_wrap_blocks( partition_value_sql: &str, partitioned: bool, strategy: MaterializeStrategy, -) -> Vec { + tests: &[DataTestResolved], +) -> Result, String> { let target_qualified = format!("{TARGET_ALIAS}.{target_table}"); let cg = MaterializeCodegen { target_qualified: &target_qualified, @@ -492,6 +493,14 @@ pub fn build_wrap_blocks( partitioned, strategy, }; + let ctx = DataTestCtx { + target_qualified: &target_qualified, + asset_path, + partition_col, + partition_value_sql, + partitioned, + }; + let test_sql = build_data_test_checks(tests, &ctx)?; let mut blocks: Vec = Vec::new(); // Setup blocks come from the splitter with their `;` stripped — re-terminate // each so that when the executor re-joins and re-splits the assembled query, @@ -499,15 +508,20 @@ pub fn build_wrap_blocks( // don't merge into one malformed statement. blocks.extend(plan.setup.iter().map(|s| terminate(s))); blocks.push(target_attach.to_string()); + // Referenced-asset ATTACHes (relationships tests) — read-only, before the + // write and the summary that probes them. + blocks.extend(test_sql.attaches); blocks.extend(cg.statements()); + // The summary read carries the per-test breakdown (when any tests apply). blocks.push(materialize_result_sql( &target_qualified, asset_path, partition_col, partition_value_sql, partitioned, + &test_sql.checks, )); - blocks + Ok(blocks) } /// The trailing one-row summary the materialize run returns: the asset it @@ -520,6 +534,7 @@ pub fn materialize_result_sql( partition_col: &str, partition_value_sql: &str, partitioned: bool, + checks: &[DataTestCheck], ) -> String { let (count_expr, partition_sel) = if partitioned { // Row count is the slice this run wrote (the partition); `partition` @@ -536,10 +551,69 @@ pub fn materialize_result_sql( String::new(), ) }; - format!( - "SELECT 'ducklake://{asset_path}' AS materialized, \ + // Capture the materialized output schema (gap #2a) in the same summary row — + // no extra round-trip. `DESCRIBE SELECT * FROM ` yields one row per + // column (`column_name`, `column_type`); fold them into a list-of-struct the + // worker reads back and persists as asset metadata. The write just + // committed, so the latest snapshot (no `AT (VERSION)` needed) is exactly the + // slice recorded in `snapshot_id`. + // + // Two correctness details: + // - `_wm_ord` (a `row_number()` over the DESCRIBE) is captured so the + // list-of-struct is ordered *explicitly* (`list(... ORDER BY _wm_ord)`). + // DESCRIBE returns columns in physical order; without the explicit ORDER + // the `list()` aggregate could reorder them and spuriously bump the schema + // version on a re-materialize. + // - For a `// partitioned` asset the physical table carries the synthetic + // `_wm_partition` column; it must be filtered out so the recorded schema is + // the producer's logical output, not Windmill's storage detail (this is the + // grain #2b contract enforcement reads back). + let partition_filter = if partitioned { + format!( + " WHERE column_name <> '{}'", + partition_col.replace('\'', "''") + ) + } else { + String::new() + }; + let schema_capture = format!( + "(SELECT list({{'name': column_name, 'type': column_type}} ORDER BY _wm_ord) \ + FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ + FROM (DESCRIBE SELECT * FROM {target_qualified}){partition_filter})) AS output_schema" + ); + let base_cols = format!( + "'ducklake://{asset_path}' AS materialized, \ {partition_sel}{count_expr} AS rows, \ - (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id;" + (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id, \ + {schema_capture}" + ); + if checks.is_empty() { + return format!("SELECT {base_cols};"); + } + // Per-test breakdown. Each check's violating-count is computed once as a CTE + // column (`c0`, `c1`, …); the `data_tests` list-of-struct then references + // those columns — DuckDB rejects scalar subqueries *inside* a struct/list + // literal, hence the CTE. Names are single-quote-escaped. The result row + // carries the whole breakdown so the worker runs every test (no + // abort-on-first) and decides pass/fail itself. + let cte_cols = checks + .iter() + .enumerate() + .map(|(i, c)| format!("{} AS c{i}", c.violating)) + .collect::>() + .join(", "); + let list_items = checks + .iter() + .enumerate() + .map(|(i, c)| { + let name = c.name.replace('\'', "''"); + format!("{{'test': '{name}', 'violating': c{i}}}") + }) + .collect::>() + .join(", "); + format!( + "WITH _wm_tr AS (SELECT {cte_cols}) \ + SELECT {base_cols}, [{list_items}] AS data_tests FROM _wm_tr;" ) } @@ -553,6 +627,271 @@ fn terminate(stmt: &str) -> String { } } +// --------------------------------------------------------------------------- +// Data tests (`// data_test`) +// --------------------------------------------------------------------------- +// +// A data test is the FIRST extensible annotation: the parser yields a +// `DataTest` from a known vocabulary, and this module turns each into a +// *check* — a `(name, violating-row-count query)` pair — that runs against the +// freshly-materialized target after the write commits. The materialize summary +// query embeds every check's count in one `data_tests` column, so all tests +// run in a single pass (no abort-on-first) and the worker, not the SQL, +// decides pass/fail and reports the full per-test breakdown. +// +// The pattern is deliberately open: a verifier is just `(name, count query)`. +// Built-ins differ only in their count query; the `Custom` escape hatch +// supplies its own (a user SELECT returning the violating rows). A sibling +// annotation family (column-lineage) can emit its own checks through the same +// `push_check` shape rather than bolting on a parallel mechanism. See +// `docs/ducklake-materialization.md`. + +use crate::asset_parser::{AssetKind, DataTest}; + +/// Target context a data-test probe runs against — the materialized table and +/// the partition slice (when partitioned, tests are scoped to the slice just +/// written, so a rerun/backfill is independent of other partitions' data). +#[derive(Debug, Clone)] +pub struct DataTestCtx<'a> { + /// Fully-qualified materialized target, e.g. `_wm_target.orders`. + pub target_qualified: &'a str, + /// `/` of the target, for human-readable probe messages. + pub asset_path: &'a str, + /// Physical partition column on the managed table. + pub partition_col: &'a str, + /// SQL literal/expression for the current partition value (already escaped). + pub partition_value_sql: &'a str, + /// Whether the target is partitioned (scopes probes to the slice). + pub partitioned: bool, +} + +/// A data test resolved enough to generate SQL. Built-ins carry only their +/// parsed `DataTest`; `Custom` additionally carries the fetched script body +/// (the parser crate can't fetch it — the worker does and passes it in). +#[derive(Debug, Clone)] +pub enum DataTestResolved { + BuiltIn(DataTest), + Custom { path: String, body: String }, +} + +/// One compiled data-test check: a human-readable `name` and a scalar SQL +/// expression (`violating`) yielding the number of rows that violate it (0 = +/// pass). The materialize summary query embeds every check's count so the +/// worker gets the whole breakdown in one result — all tests run (no +/// abort-on-first) and the worker, not the SQL, decides pass/fail. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DataTestCheck { + pub name: String, + /// Scalar subquery yielding the violating-row count, e.g. + /// `(SELECT count(*) AS v FROM (…))`. + pub violating: String, +} + +/// The SQL a set of data tests compiles to: referenced-asset `ATTACH` +/// statements (resolved by the executor's ATTACH-transform pass) and the +/// per-test checks, both in declaration order. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct DataTestChecks { + pub attaches: Vec, + pub checks: Vec, +} + +/// Alias prefix for a relationships test's referenced asset, attached +/// read-only alongside the target. `_wm_ref_` so it never collides with the +/// user's aliases or the reserved `_wm_target`. +const REF_ALIAS_PREFIX: &str = "_wm_ref_"; + +// Double-quote a SQL identifier, escaping embedded quotes — so an arbitrary +// column name from an annotation can't break out of the identifier. +fn quote_ident(id: &str) -> String { + format!("\"{}\"", id.replace('"', "\"\"")) +} + +// Quote a possibly schema-qualified table reference (`schema.table`) by quoting +// each dotted segment independently: `main.dim_products` → `"main"."dim_products"`. +// Quoting the whole thing would make DuckDB read it as one table name containing +// a literal dot, querying the wrong table. +fn quote_qualified(name: &str) -> String { + name.split('.') + .map(quote_ident) + .collect::>() + .join(".") +} + +// Single-quote a SQL string literal, escaping embedded quotes. +fn quote_lit(s: &str) -> String { + format!("'{}'", s.replace('\'', "''")) +} + +// The `WHERE`/`AND` fragment scoping a probe to the current partition, or +// empty when unpartitioned. `prefix` is `WHERE ` or `AND ` per call site. +fn partition_scope(ctx: &DataTestCtx, prefix: &str, table_alias: Option<&str>) -> String { + if !ctx.partitioned { + return String::new(); + } + let col = match table_alias { + Some(a) => format!("{a}.{}", quote_ident(ctx.partition_col)), + None => quote_ident(ctx.partition_col), + }; + format!("{prefix}{col} = {}", ctx.partition_value_sql) +} + +// Record one check: its display `name` plus `count_query` (which yields a +// single-column violating-row count) wrapped as a scalar subquery. +fn push_check(out: &mut DataTestChecks, name: String, count_query: String) { + out.checks + .push(DataTestCheck { name, violating: format!("({count_query})") }); +} + +/// Compile resolved data tests into ATTACH statements + per-test checks for +/// `ctx`'s target. Pure: returns SQL text, executes nothing. Errors carry an +/// actionable message (e.g. a relationships target that isn't an attachable +/// table). +pub fn build_data_test_checks( + tests: &[DataTestResolved], + ctx: &DataTestCtx, +) -> Result { + let t = ctx.target_qualified; + let mut out = DataTestChecks::default(); + // Dedup ref attaches by (kind, name): a database can't be attached twice, + // so multiple relationships into the same db share one alias. + let mut ref_aliases: Vec<(AssetKind, String, String)> = Vec::new(); + + for resolved in tests { + match resolved { + DataTestResolved::BuiltIn(DataTest::Unique { column }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let q = format!( + "SELECT count(*) AS v FROM (SELECT {c} FROM {t} WHERE {c} IS NOT NULL{scope} \ + GROUP BY {c} HAVING count(*) > 1)" + ); + push_check(&mut out, format!("unique({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::NotNull { column }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let q = format!("SELECT count(*) AS v FROM {t} WHERE {c} IS NULL{scope}"); + push_check(&mut out, format!("not_null({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::AcceptedValues { column, values }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let list = values + .iter() + .map(|v| quote_lit(v)) + .collect::>() + .join(", "); + let q = format!( + "SELECT count(*) AS v FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}" + ); + push_check(&mut out, format!("accepted_values({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::Relationships { + column, + to_kind, + to_path, + to_column, + }) => { + let (ref_name, ref_table) = to_path.split_once('/').ok_or_else(|| { + format!("data_test relationships: target `{to_path}` must be `/
`") + })?; + if ref_table.is_empty() { + return Err(format!( + "data_test relationships: target `{to_path}` has no table" + )); + } + let scheme = match to_kind { + AssetKind::Ducklake => "ducklake", + AssetKind::DataTable => "datatable", + other => { + return Err(format!( + "data_test relationships: target kind {other:?} is not an attachable \ + table (use ducklake:// or datatable://)" + )) + } + }; + // The materialize target's ducklake is already attached as + // `_wm_target`; a reference into that same lake must reuse it + // rather than ATTACH the same database again under a fresh alias + // (DuckDB forbids attaching one database twice). `asset_path` is + // the target's `/
`, so its lake is the part before + // the first `/`. + let target_lake = ctx.asset_path.split('/').next().unwrap_or(""); + let alias = if *to_kind == AssetKind::Ducklake && ref_name == target_lake { + TARGET_ALIAS.to_string() + } else { + // Reuse an existing alias for the same (kind, name), else mint one. + match ref_aliases + .iter() + .find(|(k, n, _)| k == to_kind && n == ref_name) + { + Some((_, _, a)) => a.clone(), + None => { + let a = format!("{REF_ALIAS_PREFIX}{}", ref_aliases.len()); + // Escape the name — it is interpolated into a + // single-quoted DuckDB literal (defense-in-depth: the + // name is deploy-time annotation content, but the parser + // places no character restriction on asset paths). + let esc_name = ref_name.replace('\'', "''"); + out.attaches + .push(format!("ATTACH '{scheme}://{esc_name}' AS {a};")); + ref_aliases.push((*to_kind, ref_name.to_string(), a.clone())); + a + } + } + }; + let c = quote_ident(column); + let rc = quote_ident(to_column); + // `ref_table` may be schema-qualified (`schema.table`); quote each + // segment so the dot stays a schema separator, not a literal. + let rt = quote_qualified(ref_table); + let scope = partition_scope(ctx, " AND ", Some("_wm_src")); + let q = format!( + "SELECT count(*) AS v FROM {t} _wm_src \ + WHERE _wm_src.{c} IS NOT NULL{scope} \ + AND NOT EXISTS (SELECT 1 FROM {alias}.{rt} _wm_ref \ + WHERE _wm_ref.{rc} = _wm_src.{c})" + ); + push_check( + &mut out, + format!("relationships({column} -> {to_path}.{to_column})"), + q, + ); + } + // A parsed Custom must be resolved (body fetched) before codegen. + DataTestResolved::BuiltIn(DataTest::Custom { path }) => { + return Err(format!( + "data_test custom `{path}`: body not resolved before codegen (internal)" + )); + } + DataTestResolved::Custom { path, body } => { + // dbt singular-test convention: the body is a *single* SELECT + // (or CTE) returning the violating rows. It is embedded as a + // subquery (`FROM ()`), so a multi-statement body would + // produce invalid SQL — validate up front with an actionable + // error. It runs in the target's connection (can read + // `_wm_target` + the user's attaches); partition substitution is + // already applied by the worker. + let stmts = split_statements(body); + if stmts.is_empty() { + return Err(format!("data_test custom `{path}`: empty test body")); + } + if stmts.len() > 1 { + return Err(format!( + "data_test custom `{path}`: must be a single SELECT returning the \ + violating rows (found {} statements)", + stmts.len() + )); + } + let q = format!("SELECT count(*) AS v FROM ({})", stmts[0]); + push_check(&mut out, format!("custom({path})"), q); + } + } + } + Ok(out) +} + #[cfg(test)] mod tests { use super::*; @@ -792,7 +1131,9 @@ mod tests { "'2026-06-19'", true, MaterializeStrategy::Replace, - ); + &[], + ) + .unwrap(); // setup block first, then the target ATTACH, then codegen, then result. assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl")); // every setup block must be `;`-terminated so re-splitting can't merge it @@ -814,4 +1155,275 @@ mod tests { assert!(last.contains("WHERE _wm_partition = '2026-06-19') AS rows")); assert!(last.contains("ducklake_snapshots('_wm_target')")); } + + // -- data tests --------------------------------------------------------- + + fn ctx_partitioned() -> DataTestCtx<'static> { + DataTestCtx { + target_qualified: "_wm_target.orders", + asset_path: "analytics/orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + } + } + fn ctx_unpartitioned() -> DataTestCtx<'static> { + DataTestCtx { partitioned: false, ..ctx_partitioned() } + } + + #[test] + fn data_test_unique_and_not_null_partition_scoped() { + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Unique { column: "order_id".into() }), + DataTestResolved::BuiltIn(DataTest::NotNull { column: "user_id".into() }), + ]; + let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap(); + assert!(sql.attaches.is_empty()); + assert_eq!(sql.checks.len(), 2); + // short, asset-free names (the asset is shown once by the breakdown). + assert_eq!(sql.checks[0].name, "unique(order_id)"); + assert_eq!(sql.checks[1].name, "not_null(user_id)"); + // each `violating` is a scalar count subquery. + assert!(sql.checks[0] + .violating + .starts_with("(SELECT count(*) AS v FROM")); + // unique: groups non-null keys within the slice, having count>1 + assert!(sql.checks[0] + .violating + .contains("GROUP BY \"order_id\" HAVING count(*) > 1")); + assert!(sql.checks[0] + .violating + .contains("\"order_id\" IS NOT NULL AND \"_wm_partition\" = '2026-06-19'")); + // not_null: null rows in the slice + assert!(sql.checks[1] + .violating + .contains("WHERE \"user_id\" IS NULL AND \"_wm_partition\" = '2026-06-19'")); + } + + #[test] + fn data_test_unpartitioned_has_no_partition_scope() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::NotNull { + column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0].violating.contains("WHERE \"id\" IS NULL")); + assert!(!sql.checks[0].violating.contains("_wm_partition")); + } + + #[test] + fn data_test_accepted_values_escapes_literals() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::AcceptedValues { + column: "status".into(), + values: vec!["paid".into(), "o'brien".into()], + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0] + .violating + .contains("NOT IN ('paid', 'o''brien')")); + assert!(sql.checks[0].violating.contains("\"status\" IS NOT NULL")); + } + + #[test] + fn data_test_relationships_attaches_ref_and_dedups() { + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Relationships { + column: "user_id".into(), + to_kind: AssetKind::DataTable, + to_path: "prod/users".into(), + to_column: "id".into(), + }), + // second relationship into the SAME db reuses the alias (no 2nd attach) + DataTestResolved::BuiltIn(DataTest::Relationships { + column: "buyer_id".into(), + to_kind: AssetKind::DataTable, + to_path: "prod/buyers".into(), + to_column: "id".into(), + }), + ]; + let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap(); + assert_eq!(sql.attaches.len(), 1, "same db attached once"); + assert_eq!(sql.attaches[0], "ATTACH 'datatable://prod' AS _wm_ref_0;"); + assert!(sql.checks[0] + .violating + .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"users\"")); + assert!(sql.checks[1] + .violating + .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"buyers\"")); + assert!(sql.checks[0] + .violating + .contains("_wm_src.\"_wm_partition\" = '2026-06-19'")); + assert_eq!( + sql.checks[0].name, + "relationships(user_id -> prod/users.id)" + ); + } + + #[test] + fn data_test_relationships_escapes_ref_name_in_attach() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "k".into(), + to_kind: AssetKind::DataTable, + to_path: "ev'il/users".into(), + to_column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + // single quote in the name is doubled so it can't break out of the literal + assert_eq!(sql.attaches[0], "ATTACH 'datatable://ev''il' AS _wm_ref_0;"); + } + + #[test] + fn data_test_relationships_same_lake_reuses_target() { + // A relationship into the SAME ducklake as the materialize target + // (asset_path = "analytics/orders") must NOT re-ATTACH it — _wm_target + // already holds that catalog; reuse it. + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "user_id".into(), + to_kind: AssetKind::Ducklake, + to_path: "analytics/users".into(), + to_column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!( + sql.attaches.is_empty(), + "same-lake ref must not ATTACH again" + ); + assert!(sql.checks[0] + .violating + .contains("NOT EXISTS (SELECT 1 FROM _wm_target.\"users\"")); + } + + #[test] + fn data_test_relationships_schema_qualified_target() { + // `/.
` — the schema-qualified table must quote each + // segment so the dot stays a separator, not part of one identifier. + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "sku".into(), + to_kind: AssetKind::Ducklake, + to_path: "warehouse/main.dim_products".into(), + to_column: "sku".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert_eq!( + sql.attaches[0], + "ATTACH 'ducklake://warehouse' AS _wm_ref_0;" + ); + assert!( + sql.checks[0] + .violating + .contains("FROM _wm_ref_0.\"main\".\"dim_products\""), + "schema-qualified target should be quoted per segment: {}", + sql.checks[0].violating + ); + } + + #[test] + fn data_test_relationships_rejects_non_attachable_kind() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "k".into(), + to_kind: AssetKind::S3Object, + to_path: "bucket/file".into(), + to_column: "c".into(), + })]; + assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err()); + } + + #[test] + fn data_test_custom_wraps_body() { + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SELECT * FROM _wm_target.orders WHERE amount < 0;".into(), + }]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + // trailing ; stripped, wrapped as a count subquery + assert!(sql.checks[0].violating.contains( + "SELECT count(*) AS v FROM (SELECT * FROM _wm_target.orders WHERE amount < 0)" + )); + assert_eq!(sql.checks[0].name, "custom(f/tests/amount)"); + } + + #[test] + fn data_test_custom_rejects_multi_statement_body() { + // The body is embedded as a subquery, so a setup-then-SELECT body would + // produce invalid SQL — reject it up front with an actionable error. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SET threads = 1; SELECT * FROM _wm_target.orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("single SELECT"), "unexpected error: {err}"); + } + + #[test] + fn data_test_unresolved_custom_is_internal_error() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Custom { + path: "f/x".into(), + })]; + assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err()); + } + + #[test] + fn materialize_result_sql_embeds_data_tests_breakdown() { + let checks = vec![ + DataTestCheck { + name: "unique(order_id)".into(), + violating: "(SELECT count(*) AS v FROM q0)".into(), + }, + DataTestCheck { + name: "custom(f/t)".into(), + violating: "(SELECT count(*) AS v FROM q1)".into(), + }, + ]; + let sql = materialize_result_sql( + "_wm_target.orders", + "analytics/orders", + "_wm_partition", + "'2026-06-19'", + false, + &checks, + ); + // counts computed once in a CTE, referenced by the list-of-struct. + assert!(sql.starts_with("WITH _wm_tr AS (SELECT (SELECT count(*) AS v FROM q0) AS c0,")); + assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0}, ")); + assert!(sql.contains("{'test': 'custom(f/t)', 'violating': c1}] AS data_tests")); + assert!(sql.contains("FROM _wm_tr;")); + // no tests -> plain summary, no CTE / data_tests column. + let plain = materialize_result_sql( + "_wm_target.orders", + "analytics/orders", + "_wm_partition", + "'x'", + false, + &[], + ); + assert!(plain.starts_with("SELECT 'ducklake://analytics/orders' AS materialized")); + assert!(!plain.contains("data_tests")); + // Schema capture (gap #2a) is in every summary, tests or not. Unpartitioned + // → explicit ordering, no partition-column filter. + for s in [&sql, &plain] { + assert!(s.contains( + "(SELECT list({'name': column_name, 'type': column_type} ORDER BY _wm_ord) \ + FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ + FROM (DESCRIBE SELECT * FROM _wm_target.orders))) AS output_schema" + )); + assert!(!s.contains("WHERE column_name <>")); + } + } + + #[test] + fn materialize_result_sql_schema_excludes_partition_column() { + // Partitioned → the synthetic `_wm_partition` column is filtered out so + // the captured schema is the producer's logical output only. + let sql = materialize_result_sql( + "_wm_target.orders_daily", + "analytics/orders_daily", + "_wm_partition", + "'2026-06-19'", + true, + &[], + ); + assert!(sql.contains( + "FROM (DESCRIBE SELECT * FROM _wm_target.orders_daily) \ + WHERE column_name <> '_wm_partition')) AS output_schema" + )); + } } diff --git a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json index 083ee7a8ba..4ecbc992bc 100644 --- a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json +++ b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json @@ -301,5 +301,210 @@ "tag": null, "retry": null } + }, + { + "name": "data_test built-ins accumulate in order", + "code": "-- pipeline\n-- materialize ducklake://analytics/orders key=order_id\n-- data_test unique order_id\n-- data_test not_null user_id\n-- data_test accepted_values status = paid,pending,refunded\n-- data_test relationships user_id -> datatable://prod/users.id\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "unique_key": "order_id" + }, + "data_tests": [ + { "type": "unique", "column": "order_id" }, + { "type": "not_null", "column": "user_id" }, + { + "type": "accepted_values", + "column": "status", + "values": ["paid", "pending", "refunded"] + }, + { + "type": "relationships", + "column": "user_id", + "to_kind": "datatable", + "to_path": "prod/users", + "to_column": "id" + } + ] + } + }, + { + "name": "data_test accepted_values strips quotes and spacing", + "code": "# data_test accepted_values kind = \"a b\", 'c' ,d\nprint(1)", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { "type": "accepted_values", "column": "kind", "values": ["a b", "c", "d"] } + ] + } + }, + { + "name": "data_test custom escape hatch is a script path", + "code": "// data_test f/tests/orders_amount_sane\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [{ "type": "custom", "path": "f/tests/orders_amount_sane" }] + } + }, + { + "name": "data_test relationships with ducklake shorthand target", + "code": "// data_test relationships sku -> ducklake://warehouse/dim_products.sku\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "relationships", + "column": "sku", + "to_kind": "ducklake", + "to_path": "warehouse/dim_products", + "to_column": "sku" + } + ] + } + }, + { + "name": "malformed data_test lines are dropped fail-safe", + "code": "// data_test uniq order_id\n// data_test accepted_values s =\n// data_test relationships a -> b\n// data_test unique\n// data_test\n// data_test unique id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [{ "type": "unique", "column": "id" }] + } + }, + { + "name": "ci test annotation is not a data test", + "code": "// test: f/foo/bar\n// data_test unique id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [{ "type": "unique", "column": "id" }] + } + }, + { + "name": "column lineage maps output columns to upstream sources", + "code": "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n// column user_name <- datatable://prod/users.name\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "order_total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "tax" + } + ] + }, + { + "column": "user_name", + "inputs": [ + { "from_kind": "datatable", "from_path": "prod/users", "from_column": "name" } + ] + } + ] + } + }, + { + "name": "column lineage keeps duplicate input refs (dedup is a view concern)", + "code": "// column total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.amount\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + } + ] + } + ] + } + }, + { + "name": "column lineage keeps schema-qualified table, drops malformed refs", + "code": "// column sku <- ducklake://warehouse/main.dim_products.sku, bad_no_dot\n// column no_arrow datatable://prod/x.y\n// column total <- bad_no_dot\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "sku", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/main.dim_products", + "from_column": "sku" + } + ] + } + ] + } } ] diff --git a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs index 5e863fc06e..883ddebcbc 100644 --- a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs +++ b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs @@ -38,6 +38,15 @@ struct Expected { // deserializing; only fixtures exercising materialization set it. #[serde(default)] materialize: Option, + // Snake_case `DataTest` serde shape (e.g. {"type":"unique","column":"x"}), + // compared against `serde_json::to_value(got.data_tests)`. Absent === []. + #[serde(default)] + data_tests: Vec, + // Snake_case `ColumnLineage` serde shape (e.g. {"column":"x","inputs": + // [{"from_kind":"datatable","from_path":"p","from_column":"c"}]}), compared + // against `to_value(got.column_lineage)`. Absent === []. + #[serde(default)] + column_lineage: Vec, } #[derive(Deserialize)] @@ -189,5 +198,20 @@ fn pipeline_annotation_fixtures_match() { want.is_some() ), } + + let got_tests = serde_json::to_value(&got.data_tests).expect("data_tests serialize"); + assert_eq!( + got_tests, + serde_json::Value::Array(f.expected.data_tests.clone()), + "{ctx}: data tests" + ); + + let got_lineage = + serde_json::to_value(&got.column_lineage).expect("column_lineage serialize"); + assert_eq!( + got_lineage, + serde_json::Value::Array(f.expected.column_lineage.clone()), + "{ctx}: column lineage" + ); } } diff --git a/backend/src/main.rs b/backend/src/main.rs index 57cc02598a..94b8e2f4f6 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -1446,19 +1446,45 @@ Windmill Community Edition {GIT_VERSION} } else { None }; - monitor_db( - &conn, - &base_internal_url, - server_mode, - worker_mode, - false, - tx.clone(), - Some(MonitorIteration { - rd_shift, - iter: monitor_iteration, - }), + // Hard cap on a single monitor pass. monitor_db runs all its + // periodic tasks under one join!, so a single task stuck on a + // non-DB await (statement_timeout only bounds DB statements) + // would otherwise freeze the whole loop indefinitely — silently + // stopping critical maintenance like audit-partition creation. + // Larger than statement_timeout (5min) so a slow-but-progressing + // statement is never killed prematurely. + const MONITOR_DB_TIMEOUT: Duration = Duration::from_secs(600); + let monitor_timed_out = tokio::time::timeout( + MONITOR_DB_TIMEOUT, + monitor_db( + &conn, + &base_internal_url, + server_mode, + worker_mode, + false, + tx.clone(), + Some(MonitorIteration { + rd_shift, + iter: monitor_iteration, + }), + ), ) - .await; + .await + .is_err(); + if monitor_timed_out { + windmill_common::utils::report_critical_error( + format!( + "monitor task did not finish within {}s and was aborted; \ + a background maintenance task is likely stuck. \ + Continuing to the next iteration.", + MONITOR_DB_TIMEOUT.as_secs() + ), + db.clone(), + None, + None, + ) + .await; + } monitor_iteration += 1; if let Some(handle) = warn_handle { handle.abort(); diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index be54edf468..817887eaad 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -75,6 +75,7 @@ use windmill_common::{ WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, }, indexer::load_indexer_config, + jobs::delete_jobs, jwt::JWT_SECRET, oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH, server::load_smtp_config, @@ -1275,6 +1276,19 @@ pub async fn delete_expired_items(db: &DB) -> () { tracing::error!("Error deleting autoscaling event on CE: {:?}", e); } + // native_retry_attempt has no FK to v2_job (kept off the hot bulk delete). + // Retention sweeps markers alongside the jobs it deletes, but direct job + // deletions (workspace/job/schedule clearing) leave markers orphaned — reap + // any whose job is gone. The table is sparse, so this anti-join is cheap. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)" + ) + .execute(db) + .await + { + tracing::error!("Error reaping orphaned native retry markers: {:?}", e); + } + if let Err(e) = windmill_queue::cascade::reap_stale_join_slots(db).await { tracing::error!("Error reaping stale join_pending_inputs slots: {:?}", e); } @@ -1324,6 +1338,9 @@ pub async fn delete_expired_items(db: &DB) -> () { let cleanup_start = Instant::now(); let mut total_deleted = 0u64; let mut batch_num = 0i32; + // Watermark carried across batches so each one resumes after the rows the previous batch + // already processed instead of re-scanning the (potentially undeletable) oldest prefix. + let mut completed_at_floor: Option> = None; // Process batches until no more expired jobs or max batches reached loop { @@ -1336,14 +1353,17 @@ pub async fn delete_expired_items(db: &DB) -> () { } // Each batch runs in its own transaction to avoid long-running locks - let batch_result = delete_expired_jobs_batch(db, job_retention_secs, batch_size).await; + let batch_result = + delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor) + .await; match batch_result { - Ok(deleted_count) => { + Ok((deleted_count, max_completed_at)) => { if deleted_count == 0 { // No more expired jobs to delete break; } + completed_at_floor = max_completed_at.or(completed_at_floor); total_deleted += deleted_count as u64; batch_num += 1; } @@ -1510,12 +1530,20 @@ pub async fn check_expiring_tokens(db: &DB) { /// Delete a batch of expired jobs with LIMIT and SKIP LOCKED for high-scale environments. /// Uses a single transaction per batch to minimize lock duration. -/// Returns the number of jobs deleted in this batch. +/// +/// `completed_at_floor` is the watermark from the previous batch in the same cleanup run (the +/// max `completed_at` it deleted); pass `None` for the first batch. It is re-applied as +/// `completed_at >= floor` so the scan resumes past the rows already processed instead of +/// re-walking them (see the inline comment on the DELETE for why this matters). +/// +/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the +/// returned watermark back in as `completed_at_floor` for the next batch. async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result { + completed_at_floor: Option>, +) -> error::Result<(usize, Option>)> { let mut tx = db.begin().await?; // Fetch active ROOT job IDs that started before the retention period. We only care about @@ -1531,26 +1559,70 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas - // ORDER BY completed_at ensures we delete oldest jobs first - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` is a watermark carried across batches within a cleanup run: it is the + // max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor` + // lets each batch resume after the rows the previous batch already processed instead of + // re-scanning them. This matters when the oldest rows are undeletable (children of a + // still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that + // same protected prefix on every batch, turning a cleanup run quadratic in prefix size. + // Floor only ever skips rows the current run already deleted, was protecting, or skip-locked — + // all correctly deferred to the next run, identical to the unbounded scan's semantics. + // + // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at + // deletes oldest jobs first. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + // Common case: no old root flow is still running, so nothing is protected and the + // v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely. + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`: + // the subquery form lets the planner build a one-time hashed SubPlan and apply it as a + // filter on the ordered index scan, giving O(1) membership per candidate instead of a + // per-row linear array scan (which degrades sharply when many root jobs are active). The + // `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); @@ -1589,10 +1661,21 @@ async fn delete_expired_jobs_batch( Err(e) => tracing::error!("Error deleting job logs: {:?}", e), } - if let Err(e) = sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", &deleted_jobs) - .execute(&mut *tx) - .await + // Native retry markers have no FK (to keep this bulk delete cheap) — sweep + // them with their jobs here too (the periodic retention path), same as the + // other side tables. The table is created by a startup migration, so it + // always exists by the time cleanup runs. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + &deleted_jobs + ) + .execute(&mut *tx) + .await { + tracing::error!("Error deleting native retry markers: {:?}", e); + } + + if let Err(e) = delete_jobs(&mut *tx, &deleted_jobs).await { tracing::error!("Error deleting job: {:?}", e); } @@ -1610,7 +1693,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok(deleted_count) + Ok((deleted_count, max_completed_at)) } async fn delete_log_files_from_disk_and_store( @@ -1638,11 +1721,30 @@ async fn delete_log_files_from_disk_and_store( .collect(); let stream = futures::stream::iter(s3_paths).boxed(); let mut result = os.delete_stream(stream); + let mut deleted = 0u64; + let mut not_found = 0u64; + let mut failed = 0u64; while let Some(r) = result.next().await { - if let Err(e) = r { - tracing::error!("Failed to delete from object store: {e}"); + match r { + Ok(_) => deleted += 1, + // Deleting a non-existent object is a successful no-op. S3's + // DeleteObjects ignores missing keys, but GCS returns 404 per + // delete, surfacing as NotFound — count it separately rather + // than logging it as an error. + Err(windmill_object_store::object_store_reexports::ObjectStoreError::NotFound { .. }) => { + not_found += 1; + } + Err(e) => { + failed += 1; + tracing::error!("Failed to delete from object store: {e}"); + } } } + if deleted + not_found + failed > 0 { + tracing::info!( + "object store log cleanup: {deleted} deleted, {not_found} already absent (404), {failed} failed" + ); + } } } } @@ -2614,6 +2716,9 @@ pub async fn monitor_db( if let Err(e) = cleanup_debounce_orphaned_keys(&db).await { tracing::error!("Error cleaning up debounce keys: {:?}", e); } + if let Err(e) = cleanup_consumed_debounce_batches(&db).await { + tracing::error!("Error cleaning up consumed debounce batches: {:?}", e); + } } } }; @@ -4316,6 +4421,40 @@ RETURNING key,job_id Ok(()) } +/// GC for claim-based debounce batches: once a batch row has been consumed (its +/// args accumulated into some survivor's run), it only lingers to let a later-pulled +/// survivor of the same batch tell "already consumed" from "never batched". A generous +/// grace period (>> any debounce window) makes that decision safe; after it, the rows +/// are dead weight. A re-pulled survivor whose row was GC'd correctly falls back to its +/// own (already-accumulated, persisted) args, so the grace period is not correctness- +/// critical. +async fn cleanup_consumed_debounce_batches(db: &DB) -> error::Result<()> { + // Only reclaim a consumed row once its job has LEFT the queue. A consumed sibling + // (its contribution already accumulated by another survivor) can sit queued well + // past any time-based grace under a concurrency limit / worker backlog; removing its + // row while still queued would make its eventual pull treat it as never-batched and + // re-run its item (a duplicate). Keeping the row until the job is no longer queued + // guarantees that pull still sees "already consumed" and runs empty. The age floor + // is just a safety margin on top. + let deleted = sqlx::query_scalar!( + "WITH del AS ( + DELETE FROM v2_job_debounce_batch + WHERE consumed_at IS NOT NULL + AND consumed_at < now() - interval '10 minutes' + AND id NOT IN (SELECT id FROM v2_job_queue) + RETURNING 1 + ) SELECT count(*) FROM del" + ) + .fetch_one(db) + .await? + .unwrap_or(0); + + if deleted > 0 { + tracing::info!("Cleaned up {deleted} consumed debounce batch rows"); + } + Ok(()) +} + async fn cleanup_debounce_keys_for_completed_jobs(db: &DB) -> error::Result<()> { // If min version doesn't support runnable settings, clean up debounce keys for completed jobs if !windmill_common::min_version::MIN_VERSION_SUPPORTS_RUNNABLE_SETTINGS_V0 @@ -4349,39 +4488,90 @@ RETURNING key,job_id Ok(()) } -async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { - let result = sqlx::query_scalar!( - "DELETE FROM job_perms -WHERE job_id NOT IN (SELECT id FROM v2_job_queue) -RETURNING job_id" - ) - .fetch_all(db) - .await?; +// Per-statement cap keeps each delete short and lock-light; the per-cycle batch +// cap bounds total work per monitor iteration so monitor_db stays responsive. +// A large backlog drains across several iterations rather than one long delete. +// +// These sweeps anti-join the whole table to find orphans, so their cost tracks the heap's +// physical size. job_perms / job_result_stream_v2 are high-churn (one row per job, deleted +// here), so their bloat — not the query shape — is what makes the sweep slow. These sweeps run +// every monitor cycle, but the bulk vacuuming_tables() runs only ~hourly, so dead tuples pile +// up between bulk vacuums; each sweep VACUUMs its own table right after deleting (see below) to +// keep the heap near the live working set. The outer `ctid IN (SELECT ... LIMIT)` is +// deliberate: a `job_id IN (...)` rewrite adds a second scan/probe for the delete and +// benchmarks slower, so don't "simplify" it. +const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000; +const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10; - if !result.is_empty() { - tracing::info!("Cleaned up {} orphaned job_perms rows", result.len()); +// Reclaim the dead tuples a sweep just created so the next sweep's anti-join scans a lean heap +// instead of a bloated one. Plain VACUUM (not FULL) only takes SHARE UPDATE EXCLUSIVE, so +// concurrent reads/writes (every job create touches job_perms) keep running, and the visibility +// map lets it skip unchanged pages so repeated runs are cheap. SKIP_LOCKED means HA replicas +// don't pile up: one vacuums, the rest skip rather than queue behind it. +async fn vacuum_after_sweep(db: &DB, table: &str) { + if let Err(e) = sqlx::query(&format!("VACUUM (SKIP_LOCKED) {table}")) + .execute(db) + .await + { + tracing::warn!("Error vacuuming {table} after orphan cleanup: {e:?}"); + } +} + +async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_perms + WHERE ctid IN ( + SELECT jp.ctid FROM job_perms jp + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id) + LIMIT 100000 + )" + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } + + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_perms rows"); + vacuum_after_sweep(db, "job_perms").await; } Ok(()) } async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> { - let result = sqlx::query!( - "DELETE FROM job_result_stream_v2 - WHERE job_id NOT IN (SELECT id FROM v2_job_queue) - AND job_id NOT IN ( - SELECT id FROM v2_job_completed - WHERE completed_at > NOW() - INTERVAL '60 seconds' - ) - RETURNING job_id", - ) - .fetch_all(db) - .await?; + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_result_stream_v2 + WHERE ctid IN ( + SELECT jrs.ctid FROM job_result_stream_v2 jrs + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id) + AND NOT EXISTS ( + SELECT 1 FROM v2_job_completed c + WHERE c.id = jrs.job_id + AND c.completed_at > NOW() - INTERVAL '60 seconds' + ) + LIMIT 100000 + )", + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } - if result.len() > 0 { - tracing::info!( - "Cleaned up {} orphaned job_result_stream_v2 rows", - result.len() - ); + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows"); + vacuum_after_sweep(db, "job_result_stream_v2").await; } Ok(()) } @@ -4417,11 +4607,19 @@ async fn audit_log_retention_days() -> i64 { } } +/// Number of days ahead (including today) for which an audit partition must +/// always exist. A missing partition in this window means audit inserts fail +/// once that date is reached — and because some callers (notably login) write +/// the audit row in the same transaction as their own work, that failure +/// poisons the whole transaction, so a missing partition is a hard outage, not +/// just a dropped audit row. +const AUDIT_PARTITION_LOOKAHEAD_DAYS: i64 = 3; + async fn manage_audit_partitions(db: &DB, retention_days: i64) { let today = chrono::Utc::now().date_naive(); - // Create partitions for today and the next 3 days - for days_ahead in 0..=3i64 { + // Create partitions for today and the next few days + for days_ahead in 0..=AUDIT_PARTITION_LOOKAHEAD_DAYS { let date = today + chrono::Duration::days(days_ahead); let next_date = date + chrono::Duration::days(1); let partition_name = format!("audit_{}", date.format("%Y%m%d")); @@ -4437,9 +4635,6 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { } } - // Drop expired partitions - let cutoff_date = today - chrono::Duration::days(retention_days); - let partitions = sqlx::query_scalar::<_, String>( "SELECT c.relname::text \ FROM pg_inherits i \ @@ -4449,28 +4644,62 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { .fetch_all(db) .await; - match partitions { - Ok(partitions) => { - for partition_name in partitions { - if let Some(date_str) = partition_name.strip_prefix("audit_") { - if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { - if date < cutoff_date { - let quoted_name = - format!("\"{}\"", partition_name.replace('"', "\"\"")); - let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); - match sqlx::query(&sql).execute(db).await { - Ok(_) => tracing::info!( - "Dropped expired audit partition {partition_name}" - ), - Err(e) => tracing::error!( - "Error dropping audit partition {partition_name}: {e:?}" - ), - } + let partitions = match partitions { + Ok(partitions) => partitions, + Err(e) => { + tracing::error!("Error listing audit partitions: {e:?}"); + return; + } + }; + + // Verify the lookahead window is actually covered. If a create above failed + // (or this loop has not run for several days), alert loudly instead of + // letting it surface days later as failed audit inserts and broken logins. + let existing: std::collections::HashSet<&str> = partitions.iter().map(|s| s.as_str()).collect(); + let missing: Vec = (0..=AUDIT_PARTITION_LOOKAHEAD_DAYS) + .map(|days_ahead| { + format!( + "audit_{}", + (today + chrono::Duration::days(days_ahead)).format("%Y%m%d") + ) + }) + .filter(|name| !existing.contains(name.as_str())) + .collect(); + if !missing.is_empty() { + report_critical_error( + format!( + "Audit log partitions missing after maintenance run: {}. \ + Audit inserts will fail once these dates are reached, which also \ + breaks logins (the login audit row shares the login transaction). \ + Check for earlier 'Error creating audit partition' logs and verify \ + the audit-partition maintenance loop is still running.", + missing.join(", ") + ), + db.clone(), + None, + None, + ) + .await; + } + + // Drop expired partitions + let cutoff_date = today - chrono::Duration::days(retention_days); + for partition_name in &partitions { + if let Some(date_str) = partition_name.strip_prefix("audit_") { + if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { + if date < cutoff_date { + let quoted_name = format!("\"{}\"", partition_name.replace('"', "\"\"")); + let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); + match sqlx::query(&sql).execute(db).await { + Ok(_) => { + tracing::info!("Dropped expired audit partition {partition_name}") } + Err(e) => tracing::error!( + "Error dropping audit partition {partition_name}: {e:?}" + ), } } } } - Err(e) => tracing::error!("Error listing audit partitions: {e:?}"), } } diff --git a/backend/tests/asset_trigger_dispatch.rs b/backend/tests/asset_trigger_dispatch.rs index e79937f702..198472d404 100644 --- a/backend/tests/asset_trigger_dispatch.rs +++ b/backend/tests/asset_trigger_dispatch.rs @@ -54,6 +54,14 @@ async fn seed_script( .bind(language) .execute(db) .await?; + // These tests use #[sqlx::test] isolated DBs that share one workspace id and + // reuse script paths, while the same path is seeded with different content + // (hence different hashes) across tests. The process-global deployed-script + // caches are keyed by (workspace, path)/(workspace, hash), so a concurrent + // test resolves a path to a hash that lives in another test's DB and the + // dispatch 404s. Disable them so every resolution reads the test's own DB. + windmill_common::DEPLOYED_SCRIPT_CACHE_DISABLED + .store(true, std::sync::atomic::Ordering::Relaxed); Ok(h) } @@ -392,14 +400,44 @@ async fn end_to_end_asset_dispatch(db: Pool) -> anyhow::Result<()> { ); } - // producer with parent_job (flow step) is ineligible + // flow step (carries flow_step_id) is ineligible clear_dispatched(&db).await?; let id = seed_producer_job(&db, json!({})).await?; let mut mini = make_mini(id, PRODUCER); - mini.parent_job = Some(Uuid::new_v4()); + mini.flow_step_id = Some("a".to_string()); let r = dispatch_asset_triggers(&db, &mini).await; assert_eq!(r.dispatched.len(), 0, "flow step producer ineligible"); + // A native retry attempt has a native_retry_attempt marker — it stays eligible, + // so a subscriber that recovers on retry still cascades. + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + sqlx::query("INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, 1)") + .bind(id) + .execute(&db) + .await?; + let mut mini = make_mini(id, PRODUCER); + mini.parent_job = Some(Uuid::new_v4()); + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!( + r.dispatched.len(), + 2, + "native retry attempt (marked) still dispatches" + ); + + // A parented Script child WITHOUT the marker — a schedule handler or a WAC + // inline child (which re-runs the same runnable) — must NOT cascade. + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + let mut mini = make_mini(id, PRODUCER); + mini.parent_job = Some(Uuid::new_v4()); // no marker => not a retry + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!( + r.dispatched.len(), + 0, + "parented child without the marker (handler/WAC inline) does not cascade" + ); + // producer with kind=Flow is ineligible let id = seed_producer_job(&db, json!({})).await?; let mut mini = make_mini(id, PRODUCER); @@ -640,13 +678,13 @@ async fn debounce_setting_applied_to_dispatched_subscriber( Ok(()) } -/// `// retry []` opts the subscriber into the flow-runtime retry -/// path. Implementation-wise, the dispatcher wraps the script in a one-step -/// flow (`JobPayload::SingleStepFlow`) so the existing flow retry machinery -/// handles re-runs. Subscribers without retry continue to be pushed as -/// `JobKind::Script` (no wrapping). +/// `// retry []` opts the subscriber into native retry: the +/// dispatcher pushes a real `JobKind::Script` (not a one-step flow) carrying +/// the policy in its `runnable_settings_handle`, so a failed subscriber re-runs +/// natively and stays eligible to trigger its own downstream. Subscribers +/// without retry are pushed as a plain `Script` with no settings handle. #[sqlx::test(fixtures("base"))] -async fn retry_setting_wraps_dispatched_subscriber_as_flow( +async fn retry_setting_dispatches_subscriber_as_native_script( db: Pool, ) -> anyhow::Result<()> { initialize_tracing().await; @@ -654,8 +692,8 @@ async fn retry_setting_wraps_dispatched_subscriber_as_flow( seed_script(&db, SUB_S3, "echo retrying", "bash").await?; seed_script(&db, SUB_RES, "echo plain", "bash").await?; seed_asset_write(&db, PRODUCER, "s3object", "f/blob").await?; - // Retry policy on the s3 edge; res edge stays vanilla so we also assert - // the "no retry" path keeps the cheaper ScriptHash push. + // Retry policy on the s3 edge; res edge stays vanilla so we also assert the + // "no retry" path carries no settings handle. seed_subscription_with_retry(&db, SUB_S3, "s3://f/blob", 3, 5).await?; seed_subscription(&db, SUB_RES, "script", "s3://f/blob").await?; @@ -663,37 +701,46 @@ async fn retry_setting_wraps_dispatched_subscriber_as_flow( let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; assert_eq!(r.dispatched.len(), 2, "both subscribers dispatched"); - let kinds: Vec<(String, String)> = sqlx::query!( - r#"SELECT runnable_path AS "runnable_path!", kind::text AS "kind!" - FROM v2_job - WHERE workspace_id = $1 AND trigger_kind = 'asset' - ORDER BY runnable_path"#, + let rows: Vec<(String, String, Option)> = sqlx::query!( + r#"SELECT j.runnable_path AS "runnable_path!", j.kind::text AS "kind!", + q.runnable_settings_handle + FROM v2_job j JOIN v2_job_queue q ON q.id = j.id + WHERE j.workspace_id = $1 AND j.trigger_kind = 'asset' + ORDER BY j.runnable_path"#, WS, ) .fetch_all(&db) .await? .into_iter() - .map(|r| (r.runnable_path, r.kind)) + .map(|r| (r.runnable_path, r.kind, r.runnable_settings_handle)) .collect(); - let s3_kind = kinds + let s3 = rows .iter() - .find(|(p, _)| p == SUB_S3) - .map(|(_, k)| k.as_str()) - .unwrap_or(""); - let res_kind = kinds + .find(|(p, _, _)| p == SUB_S3) + .expect("s3 dispatched"); + let res = rows .iter() - .find(|(p, _)| p == SUB_RES) - .map(|(_, k)| k.as_str()) - .unwrap_or(""); + .find(|(p, _, _)| p == SUB_RES) + .expect("res dispatched"); + // Native retry: a real Script (not a SingleStepFlow), with the policy in the + // runnable_settings_handle. assert_eq!( - s3_kind, "singlestepflow", - "retry-bearing subscriber wraps as SingleStepFlow so flow-runtime retry kicks in" + s3.1, "script", + "retry subscriber dispatched as a native Script" + ); + assert!( + s3.2.is_some(), + "retry subscriber carries a runnable_settings_handle (the retry policy)" ); assert_eq!( - res_kind, "script", - "no-retry subscriber stays as the cheaper ScriptHash push" + res.1, "script", + "no-retry subscriber stays a plain ScriptHash push" + ); + assert!( + res.2.is_none(), + "no-retry subscriber carries no settings handle" ); Ok(()) diff --git a/backend/tests/batch_rerun.rs b/backend/tests/batch_rerun.rs index b750379834..b1708465f9 100644 --- a/backend/tests/batch_rerun.rs +++ b/backend/tests/batch_rerun.rs @@ -71,6 +71,7 @@ fn ssf_script_payload(hash: Option, retry: Option) -> JobPayl path: SCRIPT_PATH.to_string(), hash, flow_version: None, + language: None, args: Default::default(), retry, error_handler_path: None, @@ -92,6 +93,7 @@ fn ssf_flow_payload() -> JobPayload { path: FLOW_PATH.to_string(), hash: None, flow_version: Some(FLOW_VERSION), + language: None, args: Default::default(), retry: None, error_handler_path: None, diff --git a/backend/tests/debounce_e2e.rs b/backend/tests/debounce_e2e.rs new file mode 100644 index 0000000000..9a2351f135 --- /dev/null +++ b/backend/tests/debounce_e2e.rs @@ -0,0 +1,112 @@ +// End-to-end debounce test: drives the FULL real path — real `push()` (EE +// `maybe_debounce` collapsing the batch), real `pull()`, real +// `maybe_apply_debouncing` (claim + accumulate), and a real worker executing the +// surviving flow — then asserts the executed result contains every accumulated item. +// Runs on --features deno_core,enterprise,private (debounce is EE/compile-gated). +#[cfg(all(feature = "deno_core", feature = "enterprise", feature = "private"))] +mod debounce_e2e { + use serde_json::json; + use sqlx::{Pool, Postgres}; + use uuid::Uuid; + use windmill_common::flows::FlowValue; + use windmill_common::jobs::JobPayload; + use windmill_common::worker::Connection; + use windmill_test_utils::*; + + async fn initialize_tracing() { + use std::sync::Once; + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let _ = windmill_common::tracing_init::initialize_tracing( + "test", + &windmill_common::utils::Mode::Standalone, + "test", + ); + }); + } + + /// A one-step flow that debounces on `items` and returns `flow_input.items`, + /// so the flow's result is exactly the accumulated batch. + fn debounce_flow() -> FlowValue { + serde_json::from_value(json!({ + "modules": [{ + "id": "a", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main(items: any[]) { return items }", + "input_transforms": { + "items": { "type": "javascript", "expr": "flow_input.items" } + } + } + }], + "debounce_delay_s": 1, + "debounce_key": "e2e_debounce_key", + "debounce_args_to_accumulate": ["items"] + })) + .expect("valid flow value") + } + + /// Fire three same-key messages; only the survivor runs, and it must execute once + /// with ALL accumulated items (none dropped, none duplicated). + #[sqlx::test(fixtures("base"))] + async fn debounce_accumulation_runs_once_with_all_items( + db: Pool, + ) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Push 3 debounced flow jobs (same key) — each collapses the previous; the last + // is the survivor. scheduled_for is ~1s out, so all 3 land before any worker pull. + let mut survivor = Uuid::nil(); + let mut superseded = Vec::new(); + for n in [1i64, 2, 3] { + if survivor != Uuid::nil() { + superseded.push(survivor); + } + survivor = RunJob::from(JobPayload::RawFlow { + value: debounce_flow(), + path: None, + restarted_from: None, + }) + .arg("items", json!([n])) + .push(&db) + .await; + } + + // Run a real worker until the survivor completes. + let listener = listen_for_completed_jobs(&db).await; + in_test_worker(Connection::Sql(db.clone()), listener.find(&survivor), port).await; + + let cj = completed_job(survivor, &db).await; + assert!(cj.success, "survivor flow must succeed"); + let mut result: Vec = + serde_json::from_value(cj.json_result().expect("result present")) + .expect("result is an array of numbers"); + result.sort(); + assert_eq!( + result, + vec![1, 2, 3], + "survivor must execute exactly once with ALL accumulated items" + ); + + // The two superseded messages must have been debounced (skipped), not run. + // (use a lightweight status query — skipped jobs have NULL started_at, which the + // full CompletedJob row decoder rejects) + for s in superseded { + let skipped: Option = + sqlx::query_scalar("SELECT status = 'skipped' FROM v2_job_completed WHERE id = $1") + .bind(s) + .fetch_optional(&db) + .await?; + assert_eq!( + skipped, + Some(true), + "superseded message {s} must be debounced (skipped), not executed" + ); + } + + Ok(()) + } +} diff --git a/backend/tests/fixtures/jobs_read_auth.sql b/backend/tests/fixtures/jobs_read_auth.sql index e6b28fba0c..456ac8c801 100644 --- a/backend/tests/fixtures/jobs_read_auth.sql +++ b/backend/tests/fixtures/jobs_read_auth.sql @@ -19,6 +19,45 @@ INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, sc ARRAY['jobs:read', 'if_jobs:filter_tags:deno'] ); +-- App embed token for the admin viewer (test-user). Mirrors a minted sandboxed +-- low-code app token: carries the `app_embed` sentinel plus the embed scope set. +-- Used to assert the token is confined to jobs the viewer LAUNCHED, not every job +-- the (admin) viewer could otherwise read. +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES ( + encode(sha256('EMBED_APP_TOKEN'::bytea), 'hex'), 'EMBED_APP_', 'EMBED_APP_TOKEN', + 'test@windmill.dev', 'app embed token', false, + ARRAY['apps:run', 'jobs:read', 'app_embed', 'resources:run', 'users:read', 'folders:read'] +); + +-- A completed app-component job LAUNCHED BY the admin viewer (created_by = +-- test-user), running as the app owner. The embed token must keep reading its own +-- launched job (the `created_by == viewer` fast path). +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner, args +) VALUES ( + '12121212-1212-1212-1212-121212121212', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false, + '{"own": "arg"}' +); +INSERT INTO public.v2_job_completed (id, workspace_id, duration_ms, status, result) VALUES + ('12121212-1212-1212-1212-121212121212', 'test-workspace', 1000, 'success'::job_status, + '{"own": "EMBED_OWN_RESULT"}'); + +-- A QUEUED job launched by the admin embed viewer (created_by = test-user). The +-- embed token may cancel its own launched job; it must NOT cancel another user's. +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner +) VALUES ( + '13131313-1313-1313-1313-131313131313', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false +); +INSERT INTO public.v2_job_queue (id, workspace_id, scheduled_for, running, tag) VALUES + ('13131313-1313-1313-1313-131313131313', 'test-workspace', '2023-01-01 00:00:00', false, 'deno'); + -- RUNNING job: queued (no completed row) and owned by test-user-2. Used to check -- that `completed/get_result_maybe?get_started=true` authorizes before disclosing -- running-state to a non-reader. diff --git a/backend/tests/jobs_read_auth.rs b/backend/tests/jobs_read_auth.rs index f19daf0f92..1eab0bffee 100644 --- a/backend/tests/jobs_read_auth.rs +++ b/backend/tests/jobs_read_auth.rs @@ -38,6 +38,10 @@ const TOP_SECRET_FLOW: &str = "ffffffff-ffff-ffff-ffff-ffffffffffff"; const DEEP_LEAF_JOB: &str = "88888888-8888-8888-8888-888888888888"; // A queued/running job (no completed row) owned by test-user-2. const RUNNING_JOB: &str = "77777777-7777-7777-7777-777777777777"; +// An app-component job launched BY the admin embed viewer (created_by test-user). +const EMBED_OWN_JOB: &str = "12121212-1212-1212-1212-121212121212"; +// A QUEUED job launched by the embed viewer (created_by test-user) — cancelable by it. +const EMBED_OWN_QUEUED: &str = "13131313-1313-1313-1313-131313131313"; // Secrets that must never leak to an unauthorized viewer. const RESULT_SECRET: &str = "RESULT_SECRET"; @@ -59,6 +63,19 @@ async fn get(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCod (status, body) } +async fn post(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCode, String) { + let mut req = client() + .post(format!("{base}/{path}")) + .json(&serde_json::json!({})); + if let Some(token) = token { + req = req.header("Authorization", format!("Bearer {token}")); + } + let resp = req.send().await.expect("request"); + let status = resp.status(); + let body = resp.text().await.expect("body"); + (status, body) +} + #[sqlx::test(fixtures("base", "jobs_read_auth"))] async fn test_single_job_read_authorization(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -88,6 +105,10 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul format!("get_completed_logs_tail/{VICTIM}"), ), ("get_flow_all_logs", format!("get_flow_all_logs/{VICTIM}")), + ( + "get_flow_all_logs_structured", + format!("get_flow_all_logs_structured/{VICTIM}"), + ), ( "completed/get_timing", format!("completed/get_timing/{VICTIM}"), @@ -281,6 +302,75 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul "top flow in an unreadable folder must stay denied (got {status}): {body}" ); + // ---- APP EMBED TOKEN: confined to jobs the viewer LAUNCHED, not everything + // the (admin) viewer can otherwise read. The token carries the `app_embed` + // sentinel; an admin's normal token reads VICTIM (asserted above), but the + // embed token must stop at the `created_by == viewer` grant so user-authored + // app JS can't reuse it to read unrelated jobs by UUID. + // Its own launched component job (created_by == viewer) still reads. + let (status, body) = get( + &base, + &format!("completed/get_result/{EMBED_OWN_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must read a job it launched (got {status}): {body}" + ); + assert!( + body.contains("EMBED_OWN_RESULT"), + "embed token should get its own launched job result: {body}" + ); + // The VICTIM job — created by another user but readable by this admin viewer's + // normal token (asserted above) — is denied to the embed token across result / + // logs / live update. NotFound (not 403) so the untrusted app can't even probe + // existence, and no secret leaks. + for path in [ + format!("completed/get_result/{VICTIM}"), + format!("get_logs/{VICTIM}"), + format!("getupdate/{VICTIM}?only_result=true"), + ] { + let (status, body) = get(&base, &path, Some("EMBED_APP_TOKEN")).await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not read a job it did not launch ({path}, got {status}): {body}" + ); + for secret in [RESULT_SECRET, ARGS_SECRET, LOGS_SECRET] { + assert!( + !body.contains(secret), + "embed token response for {path} leaked `{secret}`: {body}" + ); + } + } + + // ---- APP EMBED TOKEN: cancellation confined to the app's own jobs. The token + // may cancel a job it launched (created_by == viewer), but `cancel_job_api` + // denies (NotFound) a job created by someone else, even though cancel + // otherwise has no per-job ownership check. + let (status, body) = post( + &base, + &format!("queue/cancel/{EMBED_OWN_QUEUED}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must cancel a job it launched (got {status}): {body}" + ); + let (status, body) = post( + &base, + &format!("queue/cancel/{RUNNING_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not cancel another user's job (got {status}): {body}" + ); + // ---- UNAUTHENTICATED, unchanged: an anonymous-created job is readable // without a token (public trigger / public app result polling). let (status, body) = get(&base, &format!("completed/get_result/{ANON_JOB}"), None).await; diff --git a/backend/tests/suspend_resume.rs b/backend/tests/suspend_resume.rs index b8468f0cbd..704fa0d256 100644 --- a/backend/tests/suspend_resume.rs +++ b/backend/tests/suspend_resume.rs @@ -628,4 +628,53 @@ mod suspend_resume { ); Ok(()) } + + /// A step that declares a `suspend` but is skipped via `skip_if` never arms + /// its approval, so it must not gate the following step. If it did, the flow + /// would park forever waiting for a resume event that is never dispatched. + #[cfg(feature = "deno_core")] + #[sqlx::test(fixtures("base"))] + async fn skipped_suspend_step_does_not_block_next_step( + db: Pool, + ) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + + let flow: FlowValue = serde_json::from_value(json!({ + "modules": [ + { + "id": "a", + "skip_if": { "type": "javascript", "expr": "true" }, + "suspend": { "required_events": 1, "timeout": 86400 }, + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main() { return 1 }", + "input_transforms": {}, + }, + }, + { + "id": "b", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main() { return 42 }", + "input_transforms": {}, + }, + }, + ], + })) + .unwrap(); + + let result = + RunJob::from(JobPayload::RawFlow { value: flow, path: None, restarted_from: None }) + .run_until_complete(&db, false, server.addr.port()) + .await + .json_result() + .unwrap(); + + assert_eq!(result, json!(42)); + Ok(()) + } } diff --git a/backend/tests/v2_job_delete_orphans.rs b/backend/tests/v2_job_delete_orphans.rs new file mode 100644 index 0000000000..95cd1673b6 --- /dev/null +++ b/backend/tests/v2_job_delete_orphans.rs @@ -0,0 +1,237 @@ +//! v2_job no longer cascades to its sparse side tables `dispatch_event`, +//! `flow_conversation_message`, and `zombie_job_counter` — their `ON DELETE CASCADE` +//! foreign keys were dropped (migration `drop_v2_job_side_table_cascades`) to keep bulk +//! retention deletes cheap. Every path that deletes a v2_job must therefore clean those +//! tables explicitly. These tests assert no orphan side rows survive the deletion paths: +//! direct job deletion (`delete_jobs`), schedule clearing, and workspace deletion — plus a +//! regression test that the `/jobs/delete` purge endpoint stays workspace-scoped (a workspace +//! admin must not be able to delete another workspace's side rows by passing foreign job ids). +//! +//! Uses runtime `sqlx::query` (not the compile-time macros) so no offline query cache is +//! needed, matching delete_after_secs.rs. + +use sqlx::{Pool, Postgres}; +use uuid::Uuid; +use windmill_test_utils::*; + +const WS: &str = "test-workspace"; + +/// Insert one row in each side table that used to cascade from `job_id`, in workspace `ws`. +async fn seed_side_rows(db: &Pool, ws: &str, job_id: Uuid) -> anyhow::Result<()> { + sqlx::query( + "INSERT INTO dispatch_event + (workspace_id, producer_job_id, subscriber_path, asset_kind, asset_path, outcome) + VALUES ($1, $2, 'f/sub', 'resource', 'f/res', 'dispatched')", + ) + .bind(ws) + .bind(job_id) + .execute(db) + .await?; + + let conv_id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by) + VALUES ($1, $2, 'f/flow', 'test-user')", + ) + .bind(conv_id) + .bind(ws) + .execute(db) + .await?; + // created_seq is assigned by a trigger; inserting a value is rejected. + sqlx::query( + "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id) + VALUES ($1, 'assistant', 'hi', $2)", + ) + .bind(conv_id) + .bind(job_id) + .execute(db) + .await?; + + sqlx::query("INSERT INTO zombie_job_counter (job_id, counter) VALUES ($1, 1)") + .bind(job_id) + .execute(db) + .await?; + Ok(()) +} + +async fn count(db: &Pool, sql: &str, job_id: Uuid) -> anyhow::Result { + Ok(sqlx::query_scalar::<_, i64>(sql) + .bind(job_id) + .fetch_one(db) + .await?) +} + +/// (dispatch_event, flow_conversation_message, zombie_job_counter, v2_job) row counts for `job_id`. +async fn counts(db: &Pool, job_id: Uuid) -> anyhow::Result<(i64, i64, i64, i64)> { + Ok(( + count( + db, + "SELECT count(*) FROM dispatch_event WHERE producer_job_id = $1", + job_id, + ) + .await?, + count( + db, + "SELECT count(*) FROM flow_conversation_message WHERE job_id = $1", + job_id, + ) + .await?, + count( + db, + "SELECT count(*) FROM zombie_job_counter WHERE job_id = $1", + job_id, + ) + .await?, + count(db, "SELECT count(*) FROM v2_job WHERE id = $1", job_id).await?, + )) +} + +async fn insert_job(db: &Pool, ws: &str, job_id: Uuid) -> anyhow::Result<()> { + sqlx::query("INSERT INTO v2_job (id, workspace_id, kind) VALUES ($1, $2, 'script')") + .bind(job_id) + .bind(ws) + .execute(db) + .await?; + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn test_delete_jobs_removes_side_rows(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let job_id = Uuid::new_v4(); + insert_job(&db, WS, job_id).await?; + seed_side_rows(&db, WS, job_id).await?; + assert_eq!( + counts(&db, job_id).await?, + (1, 1, 1, 1), + "seed should create one row per table" + ); + + let mut conn = db.acquire().await?; + windmill_common::jobs::delete_jobs(&mut conn, &[job_id]).await?; + drop(conn); + + let (de, fcm, zombie, job) = counts(&db, job_id).await?; + assert_eq!( + (de, fcm, zombie, job), + (0, 0, 0, 0), + "delete_jobs left orphans: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn test_clear_schedule_removes_side_rows(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + // clear_schedule deletes queued, non-running jobs whose v2_job is a schedule trigger. + let job_id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, kind, trigger_kind, trigger) + VALUES ($1, $2, 'script', 'schedule', 'f/sched')", + ) + .bind(job_id) + .bind(WS) + .execute(&db) + .await?; + sqlx::query( + "INSERT INTO v2_job_queue (id, workspace_id, scheduled_for, running) + VALUES ($1, $2, now(), false)", + ) + .bind(job_id) + .bind(WS) + .execute(&db) + .await?; + seed_side_rows(&db, WS, job_id).await?; + + let mut tx = db.begin().await?; + windmill_queue::schedule::clear_schedule(&mut tx, "f/sched", WS).await?; + tx.commit().await?; + + let (de, fcm, zombie, job) = counts(&db, job_id).await?; + assert_eq!( + (de, fcm, zombie, job), + (0, 0, 0, 0), + "clear_schedule left orphans: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn test_workspace_delete_removes_side_rows(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let job_id = Uuid::new_v4(); + insert_job(&db, WS, job_id).await?; + seed_side_rows(&db, WS, job_id).await?; + + // SECRET_TOKEN is the base fixture's instance-superadmin token for test@windmill.dev. + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let resp = reqwest::Client::new() + .delete(format!( + "http://localhost:{port}/api/workspaces/delete/{WS}" + )) + .header("Authorization", "Bearer SECRET_TOKEN") + .send() + .await?; + let status = resp.status(); + assert!( + status.is_success(), + "delete workspace failed: {status} {}", + resp.text().await? + ); + + let (de, fcm, zombie, job) = counts(&db, job_id).await?; + assert_eq!( + (de, fcm, zombie, job), + (0, 0, 0, 0), + "workspace deletion left orphans: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} + +/// The `/jobs/delete` purge endpoint must scope every side-table delete to the path +/// workspace. A `test-workspace` admin passing a job id from another workspace must not be +/// able to delete that workspace's job or side rows (the side tables no longer cascade, so +/// the scoping has to live in each explicit delete). +#[sqlx::test(fixtures("base"))] +async fn test_jobs_export_delete_is_workspace_scoped(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + // A job with side rows living in a *different* workspace. + sqlx::query( + "INSERT INTO workspace (id, name, owner) VALUES ('ws-other', 'ws-other', 'test-user')", + ) + .execute(&db) + .await?; + let other_job = Uuid::new_v4(); + insert_job(&db, "ws-other", other_job).await?; + seed_side_rows(&db, "ws-other", other_job).await?; + + // A test-workspace admin calls the purge endpoint with the FOREIGN job id. + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let resp = reqwest::Client::new() + .post(format!("http://localhost:{port}/api/w/{WS}/jobs/delete")) + .header("Authorization", "Bearer SECRET_TOKEN") + .json(&[other_job]) + .send() + .await?; + assert!( + resp.status().is_success(), + "purge request failed: {} {}", + resp.status(), + resp.text().await? + ); + + // The other workspace's job and all its side rows must survive untouched. + let (de, fcm, zombie, job) = counts(&db, other_job).await?; + assert_eq!( + (de, fcm, zombie, job), + (1, 1, 1, 1), + "jobs_export purge crossed workspaces: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} diff --git a/backend/tests/wm_token_superadmin_guard.rs b/backend/tests/wm_token_superadmin_guard.rs new file mode 100644 index 0000000000..650cf384d4 --- /dev/null +++ b/backend/tests/wm_token_superadmin_guard.rs @@ -0,0 +1,206 @@ +//! A WM_TOKEN (job JWT) running as a superadmin must not be able to perform +//! global user/token management — promotion, password reset, user creation, +//! token creation/impersonation, offboarding, or exporting the user table. +//! A non-admin `wm_deployers` member can mint +//! such a token implicitly via an app/flow `on_behalf_of`, so trusting it would +//! let them establish *persistent* superadmin. A real superadmin who needs this +//! from a script must use a dedicated superadmin API token (which only a real +//! superadmin can create), not `$WM_TOKEN`. +//! +//! The fixture provides `test@windmill.dev` (instance superadmin, token +//! `SECRET_TOKEN`) and `test2@windmill.dev` (non-superadmin, `SECRET_TOKEN_2`). + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_common::auth::create_jwt_token; +use windmill_common::db::Authed; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder { + builder.header("Authorization", format!("Bearer {}", token)) +} + +/// Mint a WM_TOKEN: an internally-signed job JWT (note the `job_id` claim) for +/// `email`, exactly as a running app/flow job is issued. +async fn wm_token(email: &str, is_admin: bool) -> String { + let authed = Authed { + email: email.to_string(), + username: "runner".to_string(), + is_admin, + is_operator: false, + groups: vec![], + folders: vec![], + scopes: None, + token_prefix: None, + }; + create_jwt_token( + authed, + "test-workspace", + 3600, + Some(uuid::Uuid::new_v4()), + Some("app".to_string()), + None, + None, + ) + .await + .expect("mint wm_token") +} + +#[sqlx::test(fixtures("preserve_on_behalf_of"))] +async fn test_wm_token_cannot_manage_superadmin_users(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // The server decodes WM_TOKENs with the same in-process JWT secret, so + // setting it once lets us mint a valid one below. + set_jwt_secret().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/users"); + + // A superadmin-capable WM_TOKEN — the exact thing a deployer obtains via an + // app on_behalf_of pointed at a superadmin. + let sa_wm = wm_token("test@windmill.dev", true).await; + + // 1. Cannot mint a (superadmin) token. + let resp = authed(client().post(format!("{base}/tokens/create")), &sa_wm) + .json(&json!({})) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not create tokens: {}", + resp.text().await? + ); + + // 2. Cannot impersonate (mint a token as another user). + let resp = authed(client().post(format!("{base}/tokens/impersonate")), &sa_wm) + .json(&json!({ "impersonate_email": "test2@windmill.dev" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not impersonate: {}", + resp.text().await? + ); + + // 3. Cannot promote a user to superadmin. + let resp = authed( + client().post(format!("{base}/update/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({ "is_super_admin": true })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not promote users: {}", + resp.text().await? + ); + + // 4. Cannot reset its own (the superadmin's) password. + let resp = authed(client().post(format!("{base}/setpassword")), &sa_wm) + .json(&json!({ "password": "hunter2" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not reset passwords: {}", + resp.text().await? + ); + + // 4b. Cannot delete a user. + let resp = authed( + client().delete(format!("{base}/delete/test2@windmill.dev")), + &sa_wm, + ) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not delete users: {}", + resp.text().await? + ); + + // 4c. Cannot change a user's login type. + let resp = authed( + client().post(format!("{base}/set_login_type/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({ "login_type": "password" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not change login type: {}", + resp.text().await? + ); + + // 4d. Cannot offboard a global user (deletes user, tokens, password, invites, + // instance-group membership and reassigns their assets). + let resp = authed( + client().post(format!("{base}/offboard/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({})) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not offboard users: {}", + resp.text().await? + ); + + // 4e. Cannot export the global user table (leaks every user's password_hash). + let resp = authed(client().get(format!("{base}/export")), &sa_wm) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not export global users: {}", + resp.text().await? + ); + + // 5. Escape hatch / no false positive: a real superadmin API token + // (SECRET_TOKEN, no job_id) can still create tokens. + let resp = authed( + client().post(format!("{base}/tokens/create")), + "SECRET_TOKEN", + ) + .json(&json!({ "label": "ci" })) + .send() + .await?; + assert_eq!( + resp.status(), + 201, + "a real superadmin token must still create tokens: {}", + resp.text().await? + ); + + // 6. No collateral: a non-superadmin WM_TOKEN can still create its own + // token — the guard only fires for superadmin-capable job tokens. + let user_wm = wm_token("test2@windmill.dev", false).await; + let resp = authed(client().post(format!("{base}/tokens/create")), &user_wm) + .json(&json!({ "label": "from-script" })) + .send() + .await?; + assert_eq!( + resp.status(), + 201, + "non-superadmin WM_TOKEN must still create its own token: {}", + resp.text().await? + ); + + Ok(()) +} diff --git a/backend/tests/worker.rs b/backend/tests/worker.rs index 4ed12d3a4a..2349d476b7 100644 --- a/backend/tests/worker.rs +++ b/backend/tests/worker.rs @@ -5601,6 +5601,74 @@ async fn test_stop_after_all_iters_if_bad_expr_parallel_forloop( Ok(()) } +// Regression for the savepoint added around `evaluate_stop_after_all_iters_if`. +// The failpoint makes the in-evaluation DB read fail with a transaction-aborting +// error (SELECT 1/0). The caller swallows that error and keeps using the outer +// status-update transaction (later reads + commit). Without the savepoint the +// outer transaction would be aborted and the commit would fail, so the flow job +// would never be marked completed (the worker would error/retry). With the +// savepoint the read failure is isolated, the iteration is marked failed, and +// the flow completes — which is what this test asserts. +#[cfg(all(feature = "failpoints", feature = "quickjs", feature = "python"))] +#[sqlx::test(fixtures("base"))] +async fn test_stop_after_all_iters_if_db_error_isolated_by_savepoint( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + let port = 123; + let flow: FlowValue = serde_json::from_value(serde_json::json!({ + "modules": [ + { + "id": "a", + "value": { + "type": "forloopflow", + "iterator": { "type": "javascript", "expr": "result.items" }, + "skip_failures": false, + "parallel": true, + "modules": [{ + "value": { + "input_transforms": { + "n": { "type": "javascript", "expr": "flow_input.iter.value" }, + }, + "type": "rawscript", + "language": "python3", + "content": "def main(n): return n", + }, + }], + }, + "stop_after_all_iters_if": { + "expr": "__wm_failpoint_abort_tx__", + "skip_if_stopped": false, + }, + }, + ], + })) + .unwrap(); + let job = JobPayload::RawFlow { value: flow, path: None, restarted_from: None }; + + // If the savepoint failed to isolate the aborted read, the status-update + // transaction would be poisoned and the job would never complete, so this + // call would hang until the worker times out rather than returning. + let cjob = RunJob::from(job) + .arg("items", json!([1, 2, 3])) + .run_until_complete(&db, false, port) + .await; + + assert!( + !cjob.success, + "iteration should be marked failed after the injected read error" + ); + let result = cjob.json_result().unwrap(); + let error_msg = result["error"]["message"].as_str().unwrap_or(""); + assert!( + error_msg.contains("stop_after_all_iters_if"), + "error should mention stop_after_all_iters_if, got: {error_msg}" + ); + + Ok(()) +} + #[cfg(all(feature = "quickjs", feature = "python"))] #[sqlx::test(fixtures("base"))] async fn test_results_length_in_input_transform(db: Pool) -> anyhow::Result<()> { diff --git a/backend/windmill-api-assets/Cargo.toml b/backend/windmill-api-assets/Cargo.toml index 1ea16c3f0d..c6e7676b6b 100644 --- a/backend/windmill-api-assets/Cargo.toml +++ b/backend/windmill-api-assets/Cargo.toml @@ -11,8 +11,10 @@ path = "src/lib.rs" [dependencies] windmill-api-auth.workspace = true windmill-common = { workspace = true, default-features = false } +windmill-parser-sql-asset.workspace = true axum.workspace = true chrono.workspace = true serde.workspace = true serde_json.workspace = true sqlx.workspace = true +tracing.workspace = true diff --git a/backend/windmill-api-assets/src/lib.rs b/backend/windmill-api-assets/src/lib.rs index a982b02d35..29122bd793 100644 --- a/backend/windmill-api-assets/src/lib.rs +++ b/backend/windmill-api-assets/src/lib.rs @@ -23,6 +23,7 @@ pub fn workspaced_service() -> Router { .route("/graph", get(asset_graph)) .route("/pipelines", get(list_pipeline_folders)) .route("/partitions", get(list_partitions)) + .route("/asset_schemas", get(list_asset_schemas)) .route("/record_materialization", post(record_materialization)) } @@ -53,17 +54,40 @@ async fn list_partitions( Ok(Json(rows)) } +// Per-asset captured output schema versions for a ducklake asset (gap #2a) — +// the schema-evolution history persisted after each managed `// materialize`. +// Newest version first; materialization targets are ducklake-only in v1, so the +// kind is fixed. +async fn list_asset_schemas( + authed: ApiAuthed, + Path(w_id): Path, + Extension(user_db): Extension, + Query(q): Query, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = windmill_common::materialization::list_asset_schemas( + &mut *tx, + &w_id, + AssetKind::Ducklake, + &q.path, + ) + .await?; + tx.commit().await?; + Ok(Json(rows)) +} + // Record a materialization outcome from a polyglot (Python/TS) `wmill.ducklake` // helper running as a pipeline step. The DuckDB `// materialize` engine records // this itself; the SDK helpers post here instead so SDK-materialized slices show -// up in the grid identically. RLS-scoped to the caller's workspace. +// up in the grid identically. When the helper also captured the output schema, +// that schema version is upserted too. RLS-scoped to the caller's workspace. async fn record_materialization( authed: ApiAuthed, Path(w_id): Path, Extension(user_db): Extension, Json(req): Json, ) -> JsonResult<()> { - let mut tx = user_db.begin(&authed).await?; + let mut tx = user_db.clone().begin(&authed).await?; windmill_common::materialization::record_materialization( &mut *tx, &w_id, @@ -78,6 +102,37 @@ async fn record_materialization( ) .await?; tx.commit().await?; + // Schema capture is independently best-effort (its own transaction for the + // per-asset advisory lock) and must never roll back the partition record + // above — mirroring the worker's `record_mat`. A lost schema version + // degrades the history, not the run. Only a successful (`Materialized`) write + // advances the recorded schema — a failed/running write must not (and a + // client shouldn't be able to bump the history by attaching a schema to one). + let is_materialized = matches!( + req.status, + windmill_common::materialization::MaterializationStatus::Materialized + ); + if let (true, Some(columns)) = (is_materialized, req.schema.as_ref()) { + let res: windmill_common::error::Result<()> = async { + let mut tx = user_db.clone().begin(&authed).await?; + windmill_common::materialization::record_asset_schema( + &mut tx, + &w_id, + req.asset_kind, + &req.asset_path, + columns, + req.snapshot_id, + req.job_id, + ) + .await?; + tx.commit().await?; + Ok(()) + } + .await; + if let Err(e) = res { + tracing::warn!("failed to record captured asset schema: {e:#}"); + } + } Ok(Json(())) } @@ -450,6 +505,59 @@ struct GraphRunnableNode { // pipeline-member visual state on the frontend. #[serde(skip_serializing_if = "std::ops::Not::not", default)] in_pipeline: bool, + // Annotation badges parsed from the deployed script body, so the canvas + // shows partition/freshness/tag/retry/data-test chips on *deployed* nodes + // (not only on live-edited drafts, which the frontend parses itself). Kept + // in lockstep with the TS `AssetGraphRunnableNode` fields the node renders. + #[serde(skip_serializing_if = "Option::is_none", default)] + partition_kind: Option, + #[serde(skip_serializing_if = "Option::is_none", default)] + freshness: Option, + #[serde(skip_serializing_if = "Option::is_none", default)] + tag: Option, + #[serde(skip_serializing_if = "Option::is_none", default)] + retry: Option, + #[serde(skip_serializing_if = "Vec::is_empty", default)] + data_tests: Vec, + // `// column <- .` declared column-level lineage, surfaced + // so the canvas can draw the column-lineage view on deployed nodes (not + // only live drafts). Lockstep with TS `AssetGraphRunnableNode.column_lineage`. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + column_lineage: Vec, + // `// materialize ` target — the asset this script's `column_lineage` + // describes. Lets the column-graph anchor lineage to the exact output asset + // instead of guessing a ducklake write-edge (a multi-output script writes + // several). Absent for scripts with no `// materialize` annotation. + #[serde(skip_serializing_if = "Option::is_none", default)] + materialize_target: Option, + // Managed `// materialize` write strategy (`replace` | `append` | `merge`), + // absent for non-materializing or `manual` scripts. Surfaced so the asset + // panel can tell whether the captured schema can evolve: only whole-table + // `replace` (CREATE OR REPLACE) can change columns run-to-run; `append` / + // `merge` / any partitioned write INSERTs into a fixed-schema table. + #[serde(skip_serializing_if = "Option::is_none", default)] + materialize_strategy: Option, +} + +// The output asset a producer's column lineage belongs to (the `// materialize` +// target). Kept minimal — the column graph only needs (kind, path) to anchor. +#[derive(Serialize, Debug)] +struct MaterializeTargetNode { + kind: windmill_common::assets::AssetKind, + path: String, +} + +// The partition's kind word for the node badge (the full PartitionSpec carries +// tz/format/start, which the badge doesn't need). +fn partition_kind_word(kind: &windmill_common::assets::PartitionKind) -> &'static str { + use windmill_common::assets::PartitionKind::*; + match kind { + Daily => "daily", + Hourly => "hourly", + Weekly => "weekly", + Monthly => "monthly", + Dynamic { .. } => "dynamic", + } } // Lineage edge from parsed r/w usages. One per (runnable, asset, access_type) @@ -642,15 +750,22 @@ async fn asset_graph( .await?; // Which scripts in scope are pipeline members (have `// pipeline`). + // Pipeline members + their latest deployed body, so the graph can surface + // annotation badges (partition/freshness/tag/retry/data_test) on deployed + // nodes. `DISTINCT ON (path) … ORDER BY created_at DESC` picks the newest + // non-archived version per path (a redeploy archives the prior one, but be + // defensive against transient overlaps). let pipeline_member_paths = sqlx::query!( r#" - SELECT path AS "path!" + SELECT DISTINCT ON (path) path AS "path!", content AS "content!", + language AS "language!: windmill_common::scripts::ScriptLang" FROM script WHERE workspace_id = $1 AND auto_kind = 'pipeline' AND archived = false AND deleted = false AND ($2::text IS NULL OR path LIKE $2) + ORDER BY path, created_at DESC "#, &w_id, folder_filter.as_deref(), @@ -681,6 +796,48 @@ async fn asset_graph( tx.commit().await?; + // Parse each pipeline member's body once into its badge annotations, keyed + // by path, for the runnable-node construction below. + let annotations_by_path: std::collections::HashMap< + String, + windmill_common::assets::PipelineAnnotations, + > = pipeline_member_paths + .iter() + .map(|r| { + ( + r.path.clone(), + windmill_common::assets::parse_pipeline_annotations(&r.content), + ) + }) + .collect(); + // Column-level lineage per member. The annotation-only lineage (already + // parsed above) is the baseline. For DuckDB scripts we additionally run the + // full SQL asset parser to infer output→input column edges from the AST; it + // merges them with the `// column` annotations (annotation wins). If the SQL + // can't be parsed (DuckDB accepts grammar `sqlparser` rejects), we fall back + // to the annotation-only baseline rather than dropping explicit annotations. + let column_lineage_by_path: std::collections::HashMap< + String, + Vec, + > = pipeline_member_paths + .iter() + .map(|r| { + let annotated = || { + annotations_by_path + .get(&r.path) + .map(|a| a.column_lineage.clone()) + .unwrap_or_default() + }; + let lineage = if r.language == windmill_common::scripts::ScriptLang::DuckDb { + windmill_parser_sql_asset::parse_assets(&r.content) + .map(|o| o.column_lineage) + .unwrap_or_else(|_| annotated()) + } else { + annotated() + }; + (r.path.clone(), lineage) + }) + .collect(); let pipeline_member_script_paths: std::collections::HashSet = pipeline_member_paths.into_iter().map(|r| r.path).collect(); let existing_script_paths: std::collections::HashSet = @@ -804,7 +961,50 @@ async fn asset_graph( .map(|(usage_kind, path)| { let in_pipeline = usage_kind == AssetUsageKind::Script && pipeline_member_script_paths.contains(&path); - GraphRunnableNode { path, usage_kind, in_pipeline } + // Annotation badges, only for pipeline-member scripts (the only + // bodies we parsed). Gate on the runnable kind too: a flow sharing a + // path with a pipeline script must not inherit its badges. + let ann = (usage_kind == AssetUsageKind::Script) + .then(|| annotations_by_path.get(&path)) + .flatten(); + GraphRunnableNode { + in_pipeline, + partition_kind: ann + .and_then(|a| a.partition.as_ref()) + .map(|p| partition_kind_word(&p.kind).to_string()), + freshness: ann + .and_then(|a| a.freshness.as_ref()) + .map(|f| f.duration.clone()), + tag: ann.and_then(|a| a.tag.clone()), + retry: ann.and_then(|a| a.retry.clone()), + data_tests: ann.map(|a| a.data_tests.clone()).unwrap_or_default(), + // Inferred (DuckDB AST) + annotation column lineage, gated to + // scripts like the badges above. + column_lineage: (usage_kind == AssetUsageKind::Script) + .then(|| column_lineage_by_path.get(&path)) + .flatten() + .cloned() + .unwrap_or_default(), + materialize_target: ann.and_then(|a| a.materialize.as_ref()).map(|m| { + MaterializeTargetNode { + kind: windmill_common::assets::asset_kind_from_parser(m.target_kind), + path: m.target_path.clone(), + } + }), + materialize_strategy: ann.and_then(|a| a.materialize.as_ref()).and_then(|m| { + if m.manual { + None + } else if m.append { + Some("append".to_string()) + } else if m.unique_key.is_some() { + Some("merge".to_string()) + } else { + Some("replace".to_string()) + } + }), + path, + usage_kind, + } }) .collect(); runnables.sort_by(|a, b| a.path.cmp(&b.path)); diff --git a/backend/windmill-api-auth/src/auth.rs b/backend/windmill-api-auth/src/auth.rs index 314ba99450..a6a0fc4a8e 100644 --- a/backend/windmill-api-auth/src/auth.rs +++ b/backend/windmill-api-auth/src/auth.rs @@ -191,7 +191,11 @@ impl AuthCache { is_operator: claims.is_operator, groups: claims.groups, folders: claims.folders, - scopes: None, + // Honor the scopes embedded in the JWT (mirrors the EE + // jwt_ext_ branch). The route middleware only enforces + // scopes when Some, so a None-scoped JWT (e.g. the job + // WM_TOKEN) keeps full user privileges as before. + scopes: claims.scopes, username_override, token_prefix: claims.audit_span, read_only: false, diff --git a/backend/windmill-api-auth/src/lib.rs b/backend/windmill-api-auth/src/lib.rs index b9e6a748d4..d2347aa22e 100644 --- a/backend/windmill-api-auth/src/lib.rs +++ b/backend/windmill-api-auth/src/lib.rs @@ -205,6 +205,33 @@ pub async fn require_super_admin(db: &DB, email: &str) -> error::Result<()> { } } +/// Forbid sensitive global user/token management when authenticated as a +/// superadmin *via a job token* (`WM_TOKEN`). +/// +/// A `WM_TOKEN`'s identity is derived from an app/flow `on_behalf_of`, which a +/// non-admin `wm_deployers` member can point at a superadmin. Trusting it for +/// these operations would let them establish *persistent* superadmin (promote a +/// user, reset a superadmin's password, mint a superadmin token, ...). `job_id` +/// is set only for `WM_TOKEN`s; regular session/API tokens have it `None`, so a +/// real superadmin who needs this from a script uses a dedicated superadmin API +/// token (which only a real superadmin can create) instead of `$WM_TOKEN`. +pub async fn forbid_superadmin_job_token( + db: &DB, + email: &str, + job_id: Option, +) -> error::Result<()> { + if job_id.is_some() && is_super_admin_email(db, email).await? { + return Err(Error::NotAuthorized( + "This operation cannot be performed with a job token ($WM_TOKEN) that runs as a \ + superadmin. If a script genuinely needs to do this, create a dedicated superadmin \ + token from the User settings drawer (the 'Tokens' section), store it as a secret, \ + and use that token explicitly instead of $WM_TOKEN." + .to_owned(), + )); + } + Ok(()) +} + pub fn check_scopes(authed: &ApiAuthed, required: F) -> error::Result<()> where F: FnOnce() -> String, @@ -996,6 +1023,18 @@ pub fn require_path_read_access_for_preview( return Ok(()); }; + // Reject path traversal before any privilege-based short-circuit. A Preview's + // path is request-supplied and bypasses the DB `proper_id` CHECK that deployed + // runnables get; it then flows to the worker where it builds on-disk module + // directories. A `..` segment or an absolute path could let a write escape the + // per-job dir. + if path.starts_with('/') || path.split('/').any(|seg| seg == "..") || path.contains('\0') { + return Err(Error::BadRequest(format!( + "Invalid path for preview job: {}", + path + ))); + } + if authed.is_admin { return Ok(()); } @@ -1053,6 +1092,45 @@ mod tests { } } + // Regression tests for the Preview path traversal: a Preview's path skips the + // DB `proper_id` CHECK and reaches the worker, where it builds on-disk module + // dirs. Traversal must be rejected even for admins, who otherwise bypass the + // namespace/folder access check. + #[test] + fn preview_path_rejects_traversal() { + let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() }; + for path in [ + "u/admin/../../../../../../tmp/evil/payload", + "../../tmp/evil", + "/tmp/evil", + "u/admin/ok/../../../../etc/cron.d/x", + ] { + assert!( + require_path_read_access_for_preview(&admin, &Some(path.to_string())).is_err(), + "expected traversal path to be rejected: {path}" + ); + } + } + + #[test] + fn preview_path_allows_legitimate_paths() { + let alice = ApiAuthed { username: "alice".into(), ..Default::default() }; + assert!(require_path_read_access_for_preview(&alice, &None).is_ok()); + assert!(require_path_read_access_for_preview(&alice, &Some(String::new())).is_ok()); + assert!( + require_path_read_access_for_preview(&alice, &Some("u/alice/my_script".into())).is_ok() + ); + + let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() }; + assert!( + require_path_read_access_for_preview(&admin, &Some("hub/foo/bar/baz".into())).is_ok() + ); + // `..` only as a substring of a segment is a valid name, not traversal. + assert!( + require_path_read_access_for_preview(&admin, &Some("f/team/my..script".into())).is_ok() + ); + } + #[test] fn predicate_no_scopes_allows_all() { let authed = authed_with_scopes(None); diff --git a/backend/windmill-api-auth/src/scopes.rs b/backend/windmill-api-auth/src/scopes.rs index 87ca3a8862..041885369c 100644 --- a/backend/windmill-api-auth/src/scopes.rs +++ b/backend/windmill-api-auth/src/scopes.rs @@ -274,6 +274,7 @@ pub enum ScopeDomain { Configs, OAuth, AI, + AiSkills, Indexer, Teams, // Microsoft Teams integration @@ -294,6 +295,7 @@ pub enum ScopeDomain { RawApps, // Raw application data AgentWorkers, // Agent workers management Mcp, // MCP + Docs, // Self-hosted documentation search (read-only) } impl ScopeDomain { @@ -329,6 +331,7 @@ impl ScopeDomain { Self::Configs => "configs", Self::OAuth => "oauth", Self::AI => "ai", + Self::AiSkills => "ai_skills", Self::Capture => "capture", Self::Drafts => "drafts", Self::Favorites => "favorites", @@ -344,6 +347,7 @@ impl ScopeDomain { Self::Teams => "teams", Self::GitSync => "git_sync", Self::Mcp => "mcp", + Self::Docs => "docs", } } @@ -378,6 +382,7 @@ impl ScopeDomain { "configs" => Some(Self::Configs), "oauth" => Some(Self::OAuth), "ai" => Some(Self::AI), + "ai_skills" => Some(Self::AiSkills), "indexer" | "srch" => Some(Self::Indexer), "teams" => Some(Self::Teams), "native_triggers" => Some(Self::NativeTriggers), @@ -394,6 +399,7 @@ impl ScopeDomain { "raw_apps" => Some(Self::RawApps), "agent_workers" => Some(Self::AgentWorkers), "mcp" => Some(Self::Mcp), + "docs" => Some(Self::Docs), _ => None, } } @@ -448,6 +454,30 @@ pub fn check_route_access( // Find the domain and kind for this route let (required_domain, required_kind, route_suffix) = extract_domain_from_route(route_path)?; + // App embed tokens (sentinel) carry broad read scopes (`jobs:read`, + // `users:read`, `folders:read`) that exist only for a handful of routes. The + // whole `/users`, `/folders` and `/jobs` routers are CORS-enabled for the + // opaque app iframe, so default-deny everything in those domains except the + // intended routes — otherwise the token could enumerate/export workspace data. + if has_app_embed_sentinel(Some(token_scopes)) { + if let Some(suffix) = route_suffix.as_deref() { + if app_embed_route_denied(required_domain, suffix) { + return Err(Error::PermissionDenied( + "Access denied. App embed token cannot access this route.".to_string(), + )); + } + // The by-id job cancel is a POST (write) that the token's `jobs:read` + // wouldn't satisfy, but cancelling the app's own component runs is + // intended (most components supersede an in-flight run on re-run). Permit + // it here; `cancel_job_api` confines it to jobs the app launched + // (created_by == viewer). A read_only token is still rejected by the + // separate read-only check. + if suffix.starts_with("jobs_u/queue/cancel/") { + return Ok(()); + } + } + } + // MCP scopes (mcp:all, mcp:favorites, mcp:hub:*, etc.) use a custom format // that doesn't fit the standard domain:action model. Verify the token has at // least one mcp: scope; MCP handlers do their own fine-grained checking. @@ -534,7 +564,7 @@ const FLOW_JOBS: [&'static str; 6] = [ lazy_static::lazy_static! { static ref RUN_PATH_ACTIONS: Vec<&'static str> = { - let mut v = vec!["jobs/resume/", "jobs/run/batch_rerun_jobs", "jobs/run/workflow_as_code", "jobs/run/dependencies","jobs/run/flow_dependencies", "apps_u/execute_component"]; + let mut v = vec!["jobs/resume/", "jobs/run/batch_rerun_jobs", "jobs/run/workflow_as_code", "jobs/run/dependencies","jobs/run/flow_dependencies", "apps_u/execute_component", "apps_u/upload_s3_file"]; v.extend(SCRIPT_JOBS); v.extend(FLOW_JOBS); @@ -637,6 +667,92 @@ const RUN_WHITELISTED_GET_PATHS: [&'static str; 20] = [ "jobs/completed/get_result_maybe/", ]; +/// Sentinel scope in app embed tokens. Grants nothing itself; `check_route_access` +/// uses it to deny the workspace-wide job enumeration routes `jobs:read` would +/// otherwise reach, so an embedded app reads only jobs it launched (by id). +pub const APP_EMBED_SENTINEL: &str = "app_embed"; + +/// True if a token's scopes include the app-embed sentinel (a sandboxed app iframe +/// token). Such tokens carry the viewer's identity but represent untrusted app JS, +/// so several handlers confine them to the app's own resources/runs. +pub fn has_app_embed_sentinel(scopes: Option<&[String]>) -> bool { + scopes.is_some_and(|s| s.iter().any(|x| x == APP_EMBED_SENTINEL)) +} + +/// Routes an app embed token (sentinel) is denied. Its broad scopes (`apps:run`, +/// `jobs:read`, `users:read`, `folders:read`) exist only for a fixed set of routes a +/// running app uses, but the whole `/apps`, `/jobs`, `/users`, `/folders` routers are +/// CORS-enabled for the opaque app iframe. Default-deny those domains via an explicit +/// allowlist so the token can't reach workspace inventory, counts, exports, or +/// capability-minting routes (job signatures / resume URLs). +fn app_embed_route_denied(domain: ScopeDomain, suffix: &str) -> bool { + match domain { + ScopeDomain::Apps => !app_embed_apps_route_allowed(suffix), + ScopeDomain::Jobs => !app_embed_job_route_allowed(suffix), + ScopeDomain::Users => suffix != "users/whoami", + ScopeDomain::Folders => suffix != "folders/listnames", + _ => false, + } +} + +/// App routes a running app uses: its own definition (`apps/get/p/`, further +/// path-scoped by `apps:read:`) and the public app-serving endpoints +/// (`apps_u/*`: public_app, public_resource, get_data, and the path-taking +/// `execute_component` / `download_s3_file`, which re-check `apps:run|read:` +/// in their handlers so they stay confined to this app). Everything else in the +/// domain — workspace app inventory (`exists`, `custom_path_exists`, `list`, +/// `list_paths*`, `secret_of`, history, management) — is denied. +fn app_embed_apps_route_allowed(suffix: &str) -> bool { + // The embed-token mint endpoints live under `apps_u/` but they create + // credentials. A running app never calls them — the trusted embedder session/JWT + // mints the token and hands it to the iframe — so deny them here, otherwise an + // app embed token could renew itself indefinitely past the 12h expiry. + if suffix.starts_with("apps_u/embed_token") { + return false; + } + suffix.starts_with("apps/get/p/") || suffix.starts_with("apps_u/") +} + +/// Job routes a running app uses (the by-id poll/cancel surface driven by the +/// frontend JobLoader). Everything else in the jobs domain — enumeration, counts, +/// exports, and the `job_signature`/`resume_urls` capability-minting routes — is +/// denied. By-id reads are further confined to the app's own runs by +/// `require_job_read_access` (the `app_embed` cutoff). +fn app_embed_job_route_allowed(suffix: &str) -> bool { + // `get_root_job_id` is intentionally absent: its handler has no access check at + // all (returns any job's root id by id) and the app never calls it, so denying + // it costs nothing and avoids leaking a foreign job's flow lineage. + const ALLOWED: [&str; 15] = [ + "jobs_u/get/", + "jobs_u/getupdate/", + "jobs_u/getupdate_sse/", + "jobs_u/get_logs/", + "jobs_u/get_completed_logs_tail/", + "jobs_u/get_args/", + "jobs_u/get_flow/", + "jobs_u/get_flow_all_logs/", + "jobs_u/get_flow_debug_info/", + "jobs_u/get_log_file/", + "jobs_u/completed/get/", + "jobs_u/completed/get_result/", + "jobs_u/completed/get_result_maybe/", + "jobs_u/completed/get_timing/", + "jobs_u/queue/cancel/", + ]; + ALLOWED.iter().any(|p| suffix.starts_with(p)) +} + +/// Resource routes a metadata-only `resources:run` scope (app embed tokens) may +/// GET: pickers (`/list`) and type schemas. Excludes every value-returning route +/// (`get`, `get_value`, `get_value_interpolated`, `list_search`) so resource +/// values — which can hold credentials — are never exposed. +fn resource_metadata_route_allowed(suffix: &str) -> bool { + suffix == "resources/list" + || suffix.starts_with("resources/list_names/") + || suffix.starts_with("resources/exists/") + || suffix.starts_with("resources/type/") +} + fn scope_grants_access( scope: &ScopeDefinition, required_domain: ScopeDomain, @@ -656,6 +772,14 @@ fn scope_grants_access( let scope_action = ScopeAction::from_str(&scope.action) .ok_or_else(|| Error::BadRequest(format!("Invalid scope action: {}", scope.action)))?; + // App embed tokens carry `resources:run`: metadata-only resource access via + // default-deny + allowlist (so a new value route is never exposed by accident). + // See `resource_metadata_route_allowed`. + if scope_domain == ScopeDomain::Resources && scope_action == ScopeAction::Run { + return Ok(required_action == ScopeAction::Read + && route_path.is_some_and(resource_metadata_route_allowed)); + } + if !scope_action.includes(&required_action) && !(scope_domain == ScopeDomain::Jobs && required_action == ScopeAction::Read @@ -699,6 +823,23 @@ pub fn check_read_only_for_route(route_path: &str, http_method: &str) -> Result< } } +/// The minimal scope string that grants access to exactly `{method} {path}`, as +/// `check_route_access` would require it. Used to mint a least-privilege JWT for +/// a single proxied request (the MCP endpoint proxy), so the minted token can do +/// only that one operation rather than acting as a blank check. +/// +/// `path` is the request path (e.g. `/api/w/{workspace}/variables/get/...`). +/// Returns `None` if the route's domain can't be determined — the caller should +/// then fail closed. +pub fn scope_for_route(method: &str, path: &str) -> Option { + let action = map_http_method_to_action(method, path); + let (domain, kind, _suffix) = extract_domain_from_route(path).ok()?; + Some(match (domain, action, kind) { + (ScopeDomain::Jobs, ScopeAction::Run, Some(kind)) => format!("jobs:run:{}", kind), + (domain, action, _) => format!("{}:{}", domain.as_str(), action.as_str()), + }) +} + /// Helper function to check if scopes allow access to a route pub fn check_scopes_for_route( token_scopes: Option<&[String]>, @@ -789,6 +930,12 @@ mod tests { assert_eq!(domain, ScopeDomain::FlowConversations); assert_eq!(kind, None); assert_eq!(route_suffix, Some("flow_conversations/list".to_string())); + + let (domain, kind, route_suffix) = + extract_domain_from_route("/api/w/test_workspace/ai_skills/list").unwrap(); + assert_eq!(domain, ScopeDomain::AiSkills); + assert_eq!(kind, None); + assert_eq!(route_suffix, Some("ai_skills/list".to_string())); } #[test] @@ -845,6 +992,10 @@ mod tests { ScopeDomain::from_str("flow_conversations"), Some(ScopeDomain::FlowConversations) ); + assert_eq!( + ScopeDomain::from_str("ai_skills"), + Some(ScopeDomain::AiSkills) + ); // Test canonical string conversion assert_eq!(ScopeDomain::Acls.as_str(), "acls"); @@ -854,6 +1005,41 @@ mod tests { ScopeDomain::FlowConversations.as_str(), "flow_conversations" ); + assert_eq!(ScopeDomain::AiSkills.as_str(), "ai_skills"); + } + + #[test] + fn test_ai_skills_scope_access() { + let read_scopes = vec!["ai_skills:read".to_string()]; + assert!( + check_route_access(&read_scopes, "/api/w/test_workspace/ai_skills/list", "GET").is_ok() + ); + assert!(check_route_access( + &read_scopes, + "/api/w/test_workspace/ai_skills/get/foo", + "GET" + ) + .is_ok()); + assert!(check_route_access( + &read_scopes, + "/api/w/test_workspace/ai_skills/upload", + "POST" + ) + .is_err()); + + let write_scopes = vec!["ai_skills:write".to_string()]; + assert!(check_route_access( + &write_scopes, + "/api/w/test_workspace/ai_skills/upload", + "POST" + ) + .is_ok()); + assert!(check_route_access( + &write_scopes, + "/api/w/test_workspace/ai_skills/delete/foo", + "DELETE" + ) + .is_ok()); } #[test] @@ -1083,4 +1269,38 @@ mod tests { let scopes = vec!["jobs:read".to_string(), "mcp:all".to_string()]; assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok()); } + + #[test] + fn test_scope_for_route() { + // The minted scope must be exactly what check_route_access requires for + // the same route, so a JWT carrying it passes for that one route only. + assert_eq!( + scope_for_route("GET", "/api/w/ws/variables/get/u/x/y").as_deref(), + Some("variables:read") + ); + assert_eq!( + scope_for_route("POST", "/api/w/ws/variables/create").as_deref(), + Some("variables:write") + ); + assert_eq!( + scope_for_route("DELETE", "/api/w/ws/resources/delete/u/x/y").as_deref(), + Some("resources:write") + ); + // jobs run paths carry the runnable kind. + assert_eq!( + scope_for_route("POST", "/api/w/ws/jobs/run/p/u/x/y").as_deref(), + Some("jobs:run:scripts") + ); + assert_eq!( + scope_for_route("POST", "/api/w/ws/jobs/run/f/u/x/y").as_deref(), + Some("jobs:run:flows") + ); + + // The minted scope actually satisfies the route check it targets. + let s = scope_for_route("POST", "/api/w/ws/variables/create").unwrap(); + assert!(check_route_access(&[s], "/api/w/ws/variables/create", "POST").is_ok()); + + // Unknown route -> None so the caller fails closed. + assert!(scope_for_route("GET", "/healthz").is_none()); + } } diff --git a/backend/windmill-api-configs/src/lib.rs b/backend/windmill-api-configs/src/lib.rs index e18afc35c8..f776712063 100644 --- a/backend/windmill-api-configs/src/lib.rs +++ b/backend/windmill-api-configs/src/lib.rs @@ -217,7 +217,7 @@ async fn delete_config( let mut tx = db.begin().await?; let deleted = sqlx::query!("DELETE FROM config WHERE name = $1 RETURNING name", name) - .fetch_all(&db) + .fetch_all(&mut *tx) .await?; audit_log( diff --git a/backend/windmill-api-groups/src/granular_acls.rs b/backend/windmill-api-groups/src/granular_acls.rs index 1f88c33e9f..663644e8e0 100644 --- a/backend/windmill-api-groups/src/granular_acls.rs +++ b/backend/windmill-api-groups/src/granular_acls.rs @@ -105,8 +105,6 @@ async fn add_granular_acl( return Err(Error::BadRequest("Invalid kind".to_string())); } - let mut tx = user_db.begin(&authed).await?; - let identifier = if kind == "group_" || kind == "folder" || kind == "volume" { "name" } else { @@ -140,6 +138,8 @@ async fn add_granular_acl( } } + let mut tx = user_db.begin(&authed).await?; + if kind == "folder" { if let Some(obj) = sqlx::query_scalar!( "SELECT owners FROM folder WHERE name = $1 AND workspace_id = $2", diff --git a/backend/windmill-api-integration-tests/tests/workspaces.rs b/backend/windmill-api-integration-tests/tests/workspaces.rs index fcbdeea8ba..92f179c39b 100644 --- a/backend/windmill-api-integration-tests/tests/workspaces.rs +++ b/backend/windmill-api-integration-tests/tests/workspaces.rs @@ -646,6 +646,20 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { .unwrap(); assert_eq!(resp.json::().await?, true); + // Regression: changing a fork's workspace id must preserve its parent + // linkage. Dropping it leaves a wm-fork- workspace with no parent — a + // "fork of nothing" that can no longer be compared or merged. + let parent: Option = + sqlx::query_scalar("SELECT parent_workspace_id FROM workspace WHERE id = $1") + .bind("wm-fork-renamed") + .fetch_one(&db) + .await?; + assert_eq!( + parent.as_deref(), + Some("new-test-ws"), + "renamed fork must keep its parent_workspace_id" + ); + // --- create_fork over an existing (active) workspace id: clear 400, not a raw SQL 500 --- let resp = authed(client().post(format!("{new_ws_base}/create_fork"))) .json(&json!({ diff --git a/backend/windmill-api-jobs/src/execution.rs b/backend/windmill-api-jobs/src/execution.rs index 0581b7acd9..8ec60a69ba 100644 --- a/backend/windmill-api-jobs/src/execution.rs +++ b/backend/windmill-api-jobs/src/execution.rs @@ -55,7 +55,7 @@ pub async fn check_tag_available_for_workspace( ) -> error::Result<()> { if let Some(tag) = tag.as_deref().filter(|t| !t.is_empty()) { let tags = get_scope_tags(authed); - check_tag_available_for_workspace_internal(&db, w_id, tag, &authed.email, tags).await + check_tag_available_for_workspace_internal(db, w_id, tag, &authed.email, tags).await } else { Ok(()) } diff --git a/backend/windmill-api-jobs/src/jobs_export.rs b/backend/windmill-api-jobs/src/jobs_export.rs index d8f1826948..0b188c0fa0 100644 --- a/backend/windmill-api-jobs/src/jobs_export.rs +++ b/backend/windmill-api-jobs/src/jobs_export.rs @@ -659,8 +659,34 @@ pub async fn delete_jobs( .await? .rows_affected(); + // job_ids are request-supplied, so scope every side-table delete to the workspace exactly + // like the v2_job delete below — otherwise a workspace admin could erase another + // workspace's side rows by passing foreign job ids. zombie_job_counter and + // flow_conversation_message have no workspace_id, so scope them via v2_job / their conversation. + // (Side-table list kept in sync with windmill_common::jobs::delete_jobs.) let zombie_deleted = sqlx::query!( - "DELETE FROM zombie_job_counter WHERE job_id = ANY($1)", + "DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1 AND id = ANY($2))", + &w_id, + &job_ids + ) + .execute(&mut *tx) + .await? + .rows_affected(); + + let dispatch_event_deleted = sqlx::query!( + "DELETE FROM dispatch_event WHERE workspace_id = $1 AND producer_job_id = ANY($2)", + &w_id, + &job_ids + ) + .execute(&mut *tx) + .await? + .rows_affected(); + + let conversation_message_deleted = sqlx::query!( + "DELETE FROM flow_conversation_message m + USING flow_conversation c + WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)", + &w_id, &job_ids ) .execute(&mut *tx) @@ -688,6 +714,8 @@ pub async fn delete_jobs( + queue_deleted + completed_deleted + zombie_deleted + + dispatch_event_deleted + + conversation_message_deleted + jobs_deleted; tracing::info!( diff --git a/backend/windmill-api-jobs/src/types.rs b/backend/windmill-api-jobs/src/types.rs index 6252cef9b7..c2175753e6 100644 --- a/backend/windmill-api-jobs/src/types.rs +++ b/backend/windmill-api-jobs/src/types.rs @@ -303,6 +303,7 @@ pub struct UnifiedJob { pub preprocessed: Option, pub worker: Option, pub runnable_settings_handle: Option, + pub is_retry: Option, } const CJ_FIELDS: &[&str] = &[ @@ -344,6 +345,7 @@ const CJ_FIELDS: &[&str] = &[ "v2_job.preprocessed", "v2_job_completed.worker", "null as runnable_settings_handle", + "EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", ]; const QJ_FIELDS: &[&str] = &[ @@ -385,6 +387,7 @@ const QJ_FIELDS: &[&str] = &[ "v2_job.preprocessed", "v2_job_queue.worker", "v2_job_queue.runnable_settings_handle", + "EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", ]; impl UnifiedJob { @@ -438,6 +441,7 @@ impl From for Job { priority: uj.priority, labels: uj.labels, preprocessed: uj.preprocessed, + is_retry: uj.is_retry, }, )), "QueuedJob" => Job::QueuedJob(JobExtended::new( @@ -487,6 +491,7 @@ impl From for Job { preprocessed: uj.preprocessed, runnable_settings_handle: uj.runnable_settings_handle, labels: uj.labels, + is_retry: uj.is_retry, }, )), t => panic!("job type {} not valid", t), diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index ea54b83c31..9aecd66947 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -1284,28 +1284,10 @@ async fn create_script_internal<'c>( if let Err(e) = windmill_parser::sql_materialize::classify_wrap(&ns.content) { return Err(Error::BadRequest(e.message())); } - // Managed materialize strips line comments when it wraps the SELECT, - // so a `-- $name (TYPE)` declaration is lost while its `$name` - // reference survives in the embedded SELECT — it would run unbound. - // Managed materialize takes no SQL args (the partition is supplied by - // the engine, not bound). Reject declared args with a clear error. - if let Ok(sig) = windmill_parser_sql::parse_duckdb_sig(&ns.content) { - if !sig.args.is_empty() { - let names = sig - .args - .iter() - .map(|a| format!("${}", a.name)) - .collect::>() - .join(", "); - return Err(Error::BadRequest(format!( - "managed `// materialize` cannot take SQL arguments ({names}): wrapping your \ - SELECT drops the `-- $arg` declarations, so they would run unbound. The \ - partition is supplied by the engine — reference its value with the \ - `{{partition}}` token, or use `// materialize manual` to write the DDL (and \ - bind args) yourself." - ))); - } - } + // SQL args are supported: managed materialize strips line comments + // (including `-- $name (type)` declarations) when it wraps the SELECT, + // but the executor parses the signature from the un-wrapped script, so + // `$name` references in the SELECT stay bound at run time. } // `key=` (merge) and `append` are mutually exclusive reconciliation // strategies; append (INSERT-only) wins. Surface the conflict rather @@ -1332,20 +1314,10 @@ async fn create_script_internal<'c>( if let Some(t) = pipeline_annotations.tag.clone() { ns.tag = Some(t); } - // `// retry []` is PARSED but PARKED: a retried subscriber - // is wrapped in a SingleStepFlow, whose run is a flow step and therefore - // ineligible for asset dispatch (asset_dispatch::is_eligible_kind) — so a - // retried subscriber would silently become a cascade dead-end (P1). We do - // not persist it to script_trigger; the cascade ignores retry until this - // is fixed. TODO(pipeline-retry): re-enable once cascade dispatch handles - // flow-wrapped producers. - if pipeline_annotations.retry.is_some() { - tracing::warn!( - "`// retry` on {} is not yet supported in the asset cascade and is ignored \ - (a retried subscriber cannot trigger its downstream). TODO(pipeline-retry).", - ns.path - ); - } + // `// retry []` is persisted to `script_trigger` below (asset + // edges only) and drives native subscriber retry in the cascade: a failed + // subscriber re-runs as a `Script` job (not a flow step), so it stays + // eligible for asset dispatch and can trigger its own downstream on success. // Asset presence is server-authoritative: re-parse the deployed content // (same parsers the frontend wasm wraps) and union with the client list. // The `asset` rows written below drive the asset-trigger cascade, so a @@ -1388,6 +1360,7 @@ async fn create_script_internal<'c>( RunnableSettings { debouncing_settings: ns.debouncing_settings.insert_cached(&db).await?, concurrency_settings: ns.concurrency_settings.insert_cached(&db).await?, + retry_settings: None, }, &db, ) @@ -1737,6 +1710,24 @@ async fn create_script_internal<'c>( .and_then(parse_duration_secs), _ => None, }; + // `// retry` applies only to the asset cascade: a failed subscriber is + // re-run natively (a `Script` job, not a flow step), so it can still + // trigger its own downstream on success — see asset_dispatch. + let (retry_count, retry_delay_s) = match spec { + TriggerSpec::Asset { .. } => ( + pipeline_annotations + .retry + .as_ref() + // `// retry ` count is u32; saturate the narrowing to i16. + .map(|r| r.count.min(i16::MAX as u32) as i16), + pipeline_annotations + .retry + .as_ref() + .and_then(|r| r.delay.as_deref()) + .and_then(parse_duration_secs), + ), + _ => (None, None), + }; insert_script_trigger( &mut *tx, &w_id, @@ -1746,9 +1737,8 @@ async fn create_script_internal<'c>( &trigger_ref, pipeline_join_all, debounce_s, - // retry parked — see TODO(pipeline-retry) above. - None, - None, + retry_count, + retry_delay_s, ) .await?; } diff --git a/backend/windmill-api-settings/src/audit_logs_s3.rs b/backend/windmill-api-settings/src/audit_logs_s3.rs index 9fa4574a1e..d7d749cb58 100644 --- a/backend/windmill-api-settings/src/audit_logs_s3.rs +++ b/backend/windmill-api-settings/src/audit_logs_s3.rs @@ -20,9 +20,11 @@ use windmill_common::DB; pub struct AuditLogsS3ExportStatus { /// xid cursor: rows of transactions below this have been exported. pub last_xmin: i64, - /// Partition-pruning floor (the epoch sentinel while still bootstrapping). + /// Partition-pruning floor: the latest audit-row timestamp the cursor has + /// reached (also the read side's 7-day-fallback anchor). pub last_ts: Option>, - /// True until the initial post-enable backlog has been fully drained. + /// True while the exporter is draining a backlog — the last run was capped + /// at `MAX_XID_INTERVAL` xids and has not yet caught up to the live snapshot. pub bootstrapping: bool, /// The latest audit-row timestamp actually written to object storage so /// far (monotonic) — the "how current is the mirror" figure. @@ -50,8 +52,7 @@ pub async fn get_status(db: &DB) -> error::Result::from_timestamp(0, 0).unwrap(); - let bootstrapping = last_ts.map(|t| t <= epoch).unwrap_or(true); + let bootstrapping = v.get("draining").and_then(|x| x.as_bool()).unwrap_or(false); Ok(Some(AuditLogsS3ExportStatus { last_xmin: v.get("last_xmin").and_then(|x| x.as_i64()).unwrap_or(0), last_ts, diff --git a/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs new file mode 100644 index 0000000000..0c5c55f7ed --- /dev/null +++ b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs @@ -0,0 +1,708 @@ +#![cfg(feature = "parquet")] +//! Opt-in historical backfill of audit logs to the instance object store. +//! +//! The steady-state exporter (the EE `export_audit_logs_to_object_store`) cursors +//! on transaction xmin and, by design, only exports rows committed *after* the +//! feature was enabled — it never rescans history (an `age(xmin)` predicate is +//! unindexable, so scanning the whole partitioned table can't survive a +//! `statement_timeout`). This module covers the complementary need: exporting a +//! chosen historical `[from, to)` window (e.g. the gap left while the export was +//! disabled) on demand. +//! +//! It is safe to run on a large table because it scans strictly by `timestamp` +//! (the partition key — pruned and indexed) in bounded keyset pages, so every +//! query touches at most one page worth of rows and survives a statement timeout. +//! It does not touch the xmin cursor / checkpoint at all. +//! +//! Objects are written next to the steady-state ones under `logs/audit/dt=/` +//! as `audit_backfill__.ndjson`, with the exact same row shape, so +//! a consumer reads them uniformly. The key includes the requested window so two +//! different backfill ranges never overwrite each other (a per-page `min_id` alone +//! is not unique across windows). Re-running the *same* window is deterministic +//! (audit history is append-only), so it overwrites the same objects rather than +//! duplicating. A window that overlaps already-exported steady-state rows simply +//! re-emits them under a different key; consumers dedupe by `id`. +//! +//! Scope: like the steady-state export, this reads only `audit_partitioned`. The +//! pre-partitioning `audit` table is intentionally not exported; a window that +//! overlaps any legacy `audit` row is rejected (see [`try_start`]) so a backfill +//! never silently reports success while omitting them. +//! +//! Progress is persisted in `background_task_state` (name [`TASK_NAME`]) so any +//! API replica can serve the status endpoint, mirroring `log_cleanup`. + +use std::sync::Arc; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use tokio::sync::RwLock; + +use crate::background_task; +use windmill_common::error::{self}; +use windmill_common::tracing_init::LOGS_AUDIT; +use windmill_common::{DB, INSTANCE_NAME}; + +use windmill_object_store::object_store_reexports::{ObjectStore, Path as ObjectPath}; + +pub const TASK_NAME: &str = "audit_logs_s3_backfill"; + +/// Rows fetched per keyset page. Bounds each query so it stays well under any +/// `statement_timeout` even on a busy partition, and bounds peak memory (one +/// page of ndjson is buffered before the day-grouped PUTs). +const PAGE_ROWS: i64 = 10_000; + +/// Test-only override for [`PAGE_ROWS`] (0 = use the default), so a test can force +/// multi-page / page-spanning-day keyset behaviour with only a handful of rows. +#[cfg(test)] +static PAGE_ROWS_OVERRIDE: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0); + +fn page_rows() -> i64 { + #[cfg(test)] + { + match PAGE_ROWS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed) { + 0 => PAGE_ROWS, + n => n, + } + } + #[cfg(not(test))] + { + PAGE_ROWS + } +} + +#[derive(Clone, Serialize, Deserialize)] +pub struct AuditBackfillProgress { + pub running: bool, + pub started_at: DateTime, + pub finished_at: Option>, + /// Human-readable description of the current phase. + pub phase: String, + /// Inclusive lower / exclusive upper bound of the window being exported. + pub from: DateTime, + pub to: DateTime, + /// Audit rows written to object storage so far. + pub rows_written: u64, + /// Object PUTs issued so far (one per day per page). + pub objects_written: u64, + /// Keyset cursor: the timestamp of the last row exported (how far the + /// backfill has progressed through the window). + pub last_ts: Option>, + pub errors: u64, + pub last_error: Option, +} + +impl AuditBackfillProgress { + fn new_running(from: DateTime, to: DateTime) -> Self { + Self { + running: true, + started_at: Utc::now(), + finished_at: None, + phase: "starting".to_string(), + from, + to, + rows_written: 0, + objects_written: 0, + last_ts: None, + errors: 0, + last_error: None, + } + } +} + +struct Session { + db: DB, + owner: String, + progress: RwLock, +} + +impl Session { + async fn update(&self, f: F) { + let snapshot = { + let mut p = self.progress.write().await; + f(&mut p); + p.clone() + }; + if let Err(e) = + background_task::update_state(&self.db, TASK_NAME, &self.owner, &snapshot).await + { + tracing::warn!("audit backfill: failed to persist progress: {e:#}"); + } + } + + async fn record_error(&self, msg: String) { + tracing::error!("audit backfill: {msg}"); + self.update(|p| { + p.errors = p.errors.saturating_add(1); + p.last_error = Some(msg); + }) + .await; + } + + async fn release(&self) { + let snapshot = { + let mut p = self.progress.write().await; + p.running = false; + p.finished_at = Some(Utc::now()); + p.phase = "done".to_string(); + p.clone() + }; + tracing::info!( + "audit backfill finished: {} row(s) in {} object(s) for [{}, {}), {} error(s)", + snapshot.rows_written, + snapshot.objects_written, + snapshot.from, + snapshot.to, + snapshot.errors + ); + if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await + { + tracing::warn!("audit backfill: failed to release lease: {e:#}"); + } + } +} + +#[derive(Deserialize)] +pub struct BackfillRequest { + pub from: DateTime, + pub to: DateTime, +} + +/// Atomically claim the backfill lease, or error if one is already running. +pub async fn try_start(db: &DB, from: DateTime, to: DateTime) -> error::Result<()> { + if from >= to { + return Err(error::Error::BadRequest( + "audit backfill: `from` must be strictly before `to`".to_string(), + )); + } + // The backfill keyset-pages by `(timestamp, id)` over rows visible at scan time and + // declares the window fully exported once the scan runs dry. But a row's `timestamp` + // is its inserting transaction's `xact_start`, so a transaction that started inside + // `[from, to)` yet commits after the scan has passed that timestamp — or any row + // committed when `to` is in the future — would be silently omitted. Require `to` to + // be at or before the oldest in-flight `xact_start`: everything strictly older than + // the oldest running transaction is already committed and stable. + // + // That bound is only sound when we can see every xmin-holding transaction. A role + // without pg_read_all_stats/superuser sees only its own sessions, and a prepared + // (2PC) transaction is invisible to pg_stat_activity — in either case an old + // transaction could still commit rows inside an accepted window after our scan ends. + // Since a backfill asserts completeness, we REJECT in those cases (the function + // returns NULL). (The continuous exporter, which only claims bounded lag, keeps the + // 7-day fallback instead.) The probe lives in the `audit_logs_s3_oldest_inflight_ts()` + // SQL function (migration 20260626132251) so its `pg_has_role`/`pg_authid` read is + // wrapped in a subtransaction EXCEPTION: managed providers (e.g. Cloud SQL) forbid + // reading pg_authid from an elevated context, which would otherwise surface here as + // an opaque error instead of NULL → the actionable rejection below. + let settled_cutoff: Option> = + sqlx::query_scalar!(r#"SELECT audit_logs_s3_oldest_inflight_ts() AS "cutoff?""#) + .fetch_one(db) + .await?; + let Some(settled_cutoff) = settled_cutoff else { + return Err(error::Error::BadRequest( + "audit backfill: cannot determine a trustworthy settled-time boundary, so completeness \ + can't be guaranteed. The windmill DB role needs pg_read_all_stats (or superuser) and \ + there must be no prepared (2PC) transactions in progress — otherwise an old or \ + invisible transaction could later commit audit rows inside the requested window and \ + the backfill would miss them. Grant the privilege / resolve prepared transactions and \ + retry." + .to_string(), + )); + }; + if to > settled_cutoff { + return Err(error::Error::BadRequest(format!( + "audit backfill: `to` ({to}) must be at or before {settled_cutoff}, the newest point \ + guaranteed settled (the oldest in-flight transaction's start); choose an earlier \ + upper bound." + ))); + } + // The backfill (like the steady-state export) reads only `audit_partitioned`. Audit + // history from before partitioning was introduced lives in the legacy `audit` table + // and is intentionally not exported. If the requested window overlaps any legacy row, + // reject — otherwise a "completed" backfill would silently omit them. Checking the + // legacy table directly (rather than min(audit_partitioned)) also covers an upgraded + // instance whose `audit_partitioned` is still empty, where a min() guard would no-op. + // Non-macro query: no compile-time-checked entry needed. + let overlaps_legacy: bool = sqlx::query_scalar::<_, bool>( + "SELECT EXISTS (SELECT 1 FROM audit WHERE timestamp >= $1 AND timestamp < $2)", + ) + .bind(from) + .bind(to) + .fetch_one(db) + .await?; + if overlaps_legacy { + return Err(error::Error::BadRequest( + "audit backfill: the requested window overlaps rows in the legacy (pre-partitioning) \ + `audit` table, which is not exported to object storage. Restrict the window to the \ + partitioned era (after audit-log partitioning was introduced)." + .to_string(), + )); + } + let claimed = background_task::try_claim( + db, + TASK_NAME, + &*INSTANCE_NAME, + &AuditBackfillProgress::new_running(from, to), + ) + .await?; + if !claimed { + return Err(error::Error::BadRequest( + "An audit log backfill is already running".to_string(), + )); + } + Ok(()) +} + +/// Fetch the current backfill status. Any API server can call this. +pub async fn get_status(db: &DB) -> error::Result> { + let Some(r) = background_task::get(db, TASK_NAME).await? else { + return Ok(None); + }; + match serde_json::from_value::(r.value) { + Ok(mut p) => { + // get() collapses `running` to false when the heartbeat is stale. + p.running = r.running; + Ok(Some(p)) + } + Err(e) => Err(error::Error::internal_err(format!( + "deserialize audit backfill progress: {e:#}" + ))), + } +} + +pub fn spawn_backfill(db: DB, from: DateTime, to: DateTime) { + use futures::FutureExt; + use std::panic::AssertUnwindSafe; + + tokio::spawn(async move { + let session = Arc::new(Session { + db: db.clone(), + owner: INSTANCE_NAME.clone(), + progress: RwLock::new(AuditBackfillProgress::new_running(from, to)), + }); + + let s = session.clone(); + let task = async move { + let store = match windmill_object_store::get_object_store().await { + Some(st) => st, + None => { + s.record_error("Object storage is not configured".to_string()) + .await; + return; + } + }; + if let Err(e) = run_backfill(&s, &db, &store, from, to).await { + s.record_error(format!("backfill failed: {e:#}")).await; + } + }; + + // catch_unwind so a panic can't leave the lease held forever. + if let Err(panic) = AssertUnwindSafe(task).catch_unwind().await { + let msg = panic + .downcast_ref::<&str>() + .map(|s| s.to_string()) + .or_else(|| panic.downcast_ref::().cloned()) + .unwrap_or_else(|| "unknown panic".to_string()); + session + .record_error(format!("backfill task panicked: {msg}")) + .await; + } + + session.release().await; + }); +} + +/// Export `[from, to)` in keyset pages ordered by `(timestamp, id)`. Each page is +/// a bounded, partition-pruned scan; rows are grouped by UTC day and written one +/// object per day per page. +async fn run_backfill( + session: &Session, + db: &DB, + store: &Arc, + from: DateTime, + to: DateTime, +) -> error::Result<()> { + session.update(|p| p.phase = "exporting".to_string()).await; + + // Keyset cursor over (timestamp, id). `id` starts below any real value so the + // first page includes rows at exactly `from`. + let mut cursor_ts = from; + let mut cursor_id: i64 = -1; + let page_rows = page_rows(); + // Namespace object keys by the requested window. The per-page `min_id` alone is not + // unique across runs: a narrower, overlapping backfill can start a day's page at the + // same first row (same `min_id`) but contain fewer rows, and `put` would overwrite a + // broader run's object — silently dropping the rows only that object held. Including + // the window makes different ranges write disjoint keys (same window re-runs stay + // idempotent); consumers already dedupe overlapping rows by `id`. + let window_key = format!("{}_{}", from.timestamp_millis(), to.timestamp_millis()); + + loop { + let rows = sqlx::query!( + r#"SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS "day!", + id AS "id!", + timestamp AS "ts!", + row_to_json(r)::text AS "line!" + FROM ( + SELECT workspace_id, id, timestamp, username, operation, + action_kind::text AS action_kind, resource, parameters, email, span + FROM audit_partitioned + WHERE timestamp >= $1 AND timestamp < $2 + AND (timestamp, id) > ($3, $4) + ORDER BY timestamp, id + LIMIT $5 + ) r + ORDER BY timestamp, id"#, + from, + to, + cursor_ts, + cursor_id, + page_rows + ) + .fetch_all(db) + .await?; + + if rows.is_empty() { + break; + } + + // Group this page's ndjson lines by day, preserving (timestamp, id) order, + // and track the min id per day for a deterministic, collision-free key. + let mut by_day: Vec<(String, i64, String)> = Vec::new(); // (day, min_id, ndjson) + for row in &rows { + match by_day.last_mut() { + Some((day, _min_id, acc)) if *day == row.day => { + acc.push('\n'); + acc.push_str(&row.line); + } + _ => by_day.push((row.day.clone(), row.id, row.line.clone())), + } + } + + for (day, min_id, ndjson) in &by_day { + let object_path = ObjectPath::from(format!( + "{LOGS_AUDIT}dt={day}/audit_backfill_{window_key}_{min_id}.ndjson" + )); + store + .put(&object_path, ndjson.clone().into_bytes().into()) + .await + .map_err(|e| error::Error::internal_err(format!("upload {object_path}: {e:#}")))?; + let n = ndjson.lines().count() as u64; + // Persist progress (and refresh the lease heartbeat) after every object, + // not just once the page completes: a stale heartbeat lets another replica + // re-claim the lease and run a concurrent backfill, so the gap between + // heartbeats must stay well under STALE_HEARTBEAT_SECS even if a page's + // uploads are slow. + session + .update(|p| { + p.rows_written = p.rows_written.saturating_add(n); + p.objects_written = p.objects_written.saturating_add(1); + }) + .await; + } + + // Advance the keyset cursor past the last row of this page. + let last = rows.last().expect("page is non-empty"); + cursor_ts = last.ts; + cursor_id = last.id; + + let new_last_ts = last.ts; + session.update(|p| p.last_ts = Some(new_last_ts)).await; + + // A short page means the window is exhausted. + if (rows.len() as i64) < page_rows { + break; + } + } + + Ok(()) +} + +#[cfg(all(test, feature = "parquet"))] +mod tests { + use super::*; + use futures::stream::StreamExt; + use std::sync::atomic::Ordering; + use std::sync::Arc; + use windmill_object_store::object_store_reexports::{InMemory, ObjectStore, Path as OsPath}; + + /// A private, per-test object store. `run_backfill` takes the store as a parameter, + /// so tests use a local one and never touch the process-global `OBJECT_STORE_SETTINGS` + /// (which would otherwise race across the parallel test runner). + fn local_store() -> (Arc, Arc) { + let store = Arc::new(InMemory::new()); + let dynstore: Arc = store.clone(); + (store, dynstore) + } + + /// Serializes the tests that touch the `PAGE_ROWS_OVERRIDE` process global (read + /// inside `run_backfill`) so they can't observe each other's value under the parallel + /// runner. + static PAGE_OVERRIDE_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + + /// Resets `PAGE_ROWS_OVERRIDE` on drop so a failing assertion can't leak a non-default + /// page size into another test. + struct ResetPageOverride; + impl Drop for ResetPageOverride { + fn drop(&mut self) { + PAGE_ROWS_OVERRIDE.store(0, Ordering::Relaxed); + } + } + + /// Insert an audit row `days` days in the past (creating the daily partition if + /// needed). The row's `timestamp` defaults to that point, landing it in the + /// matching partition. + async fn insert_audit_days_ago(db: &DB, operation: &str, days: i64) -> i64 { + sqlx::query(&format!( + "DO $$ DECLARE d date := current_date - {days}; BEGIN \ + EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_partitioned \ + FOR VALUES FROM (%L) TO (%L)', 'audit_'||to_char(d,'YYYYMMDD'), d, d + 1); END $$;" + )) + .execute(db) + .await + .ok(); + sqlx::query_scalar::<_, i64>(&format!( + "INSERT INTO audit_partitioned + (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{{}}'::jsonb, + now() - interval '{days} days') + RETURNING id" + )) + .bind(operation) + .fetch_one(db) + .await + .expect("insert audit row") + } + + /// Insert an audit row at an exact timestamp (creating the daily partition if + /// needed), for tests that need distinct in-day timestamps. + async fn insert_audit_at(db: &DB, operation: &str, ts: DateTime) -> i64 { + sqlx::query(&format!( + "DO $$ DECLARE d date := '{}'; BEGIN \ + EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_partitioned \ + FOR VALUES FROM (%L) TO (%L)', 'audit_'||to_char(d,'YYYYMMDD'), d, d + 1); END $$;", + ts.format("%Y-%m-%d") + )) + .execute(db) + .await + .ok(); + sqlx::query_scalar::<_, i64>( + "INSERT INTO audit_partitioned + (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{}'::jsonb,$2) + RETURNING id", + ) + .bind(operation) + .bind(ts) + .fetch_one(db) + .await + .expect("insert audit row") + } + + /// All ids across every `audit_backfill_*.ndjson` object, and the set of object + /// paths (to assert pagination/day keying). + async fn backfilled(store: &InMemory) -> (Vec, Vec) { + let prefix = OsPath::from("logs/audit"); + let metas = store + .list(Some(&prefix)) + .collect::>() + .await + .into_iter() + .map(|m| m.expect("list object")) + .collect::>(); + let mut ids = Vec::new(); + let mut paths = Vec::new(); + for meta in metas { + paths.push(meta.location.to_string()); + let bytes = store + .get(&meta.location) + .await + .expect("get object") + .bytes() + .await + .expect("read bytes"); + for line in std::str::from_utf8(&bytes).unwrap().lines() { + if line.is_empty() { + continue; + } + let v: serde_json::Value = serde_json::from_str(line).expect("valid ndjson"); + ids.push(v.get("id").and_then(|x| x.as_i64()).expect("row has id")); + } + } + ids.sort(); + paths.sort(); + (ids, paths) + } + + fn session(db: &DB, from: DateTime, to: DateTime) -> Session { + Session { + db: db.clone(), + owner: INSTANCE_NAME.clone(), + progress: RwLock::new(AuditBackfillProgress::new_running(from, to)), + } + } + + // End-to-end backfill: a settled multi-day window is exported in bounded keyset + // pages (forced to 2 rows/page) — every in-window row lands exactly once, rows + // outside [from,to) are excluded, a day that spans a page boundary produces more + // than one object, and re-running is idempotent (same keys overwritten, no dupes). + #[sqlx::test(migrations = "../migrations")] + async fn backfill_exports_window_in_pages(db: DB) -> anyhow::Result<()> { + let _serial = PAGE_OVERRIDE_LOCK.lock().await; + let _reset = ResetPageOverride; // restores the page override even on panic + let (store, dyn_store) = local_store(); + // Force multi-page / page-spanning-day keyset behaviour with a handful of rows. + PAGE_ROWS_OVERRIDE.store(2, Ordering::Relaxed); + + // In window [now-6d, now-2d): days 5, 4, 3 ago. + let mut want = Vec::new(); + for i in 0..3 { + want.push(insert_audit_days_ago(&db, &format!("bf.d5.{i}"), 5).await); + } + for i in 0..2 { + want.push(insert_audit_days_ago(&db, &format!("bf.d4.{i}"), 4).await); + } + for i in 0..2 { + want.push(insert_audit_days_ago(&db, &format!("bf.d3.{i}"), 3).await); + } + want.sort(); + // Out of window: before `from` and at/after `to`. + let before = insert_audit_days_ago(&db, "bf.before", 7).await; + let after = insert_audit_days_ago(&db, "bf.after", 1).await; + + let from = Utc::now() - chrono::Duration::days(6); + let to = Utc::now() - chrono::Duration::days(2); + + let s = session(&db, from, to); + run_backfill(&s, &db, &dyn_store, from, to).await?; + + let (ids, paths) = backfilled(&store).await; + assert_eq!(ids, want, "exactly the in-window rows, each once: {ids:?}"); + assert!( + !ids.contains(&before) && !ids.contains(&after), + "rows outside [from,to) must not be exported" + ); + // 3 rows on the day-5 partition at a 2-row page size => that day spans pages, + // so it yields >1 object — proving keyset paging across a day boundary. + let day5_objects = paths + .iter() + .filter(|p| p.contains("audit_backfill_")) + .count(); + assert!( + day5_objects >= 4, + "expected multiple paged objects (incl. a split day), got {paths:?}" + ); + { + let p = s.progress.read().await; + assert_eq!(p.rows_written, want.len() as u64, "progress row count"); + } + + // Idempotent re-run: deterministic keys are overwritten, never duplicated. + let s2 = session(&db, from, to); + run_backfill(&s2, &db, &dyn_store, from, to).await?; + let (ids2, _) = backfilled(&store).await; + assert_eq!(ids2, want, "re-run stays exactly once per row: {ids2:?}"); + + Ok(()) + } + + // A narrower backfill overlapping a broader one must not overwrite (and drop rows + // from) the broader run's object: the object key includes the window. The two + // windows share a day and the same first row (so the same `min_id`), but the + // narrower one holds fewer rows. + #[sqlx::test(migrations = "../migrations")] + async fn backfill_window_in_key_prevents_overwrite(db: DB) -> anyhow::Result<()> { + // Hold the lock so no concurrent test's PAGE_ROWS_OVERRIDE is observed; this test + // wants the default (large) page size so each day is one object per window. + let _serial = PAGE_OVERRIDE_LOCK.lock().await; + let (store, dyn_store) = local_store(); + + // Four rows on the same day at distinct times. + let base = Utc::now() - chrono::Duration::days(5); + let r0 = insert_audit_at(&db, "ov.0", base).await; + let r1 = insert_audit_at(&db, "ov.1", base + chrono::Duration::seconds(10)).await; + let r2 = insert_audit_at(&db, "ov.2", base + chrono::Duration::seconds(20)).await; + let r3 = insert_audit_at(&db, "ov.3", base + chrono::Duration::seconds(30)).await; + + // Broad run covers all four (one object for the day, keyed by r0). + let a_from = base - chrono::Duration::seconds(1); + let a_to = base + chrono::Duration::seconds(31); + run_backfill(&session(&db, a_from, a_to), &db, &dyn_store, a_from, a_to).await?; + + // Narrow run starts at the same first row (same min_id) but holds only r0, r1. + let b_from = base - chrono::Duration::seconds(1); + let b_to = base + chrono::Duration::seconds(15); + run_backfill(&session(&db, b_from, b_to), &db, &dyn_store, b_from, b_to).await?; + + let (ids, _) = backfilled(&store).await; + for id in [r0, r1, r2, r3] { + assert!( + ids.contains(&id), + "row {id} lost — a narrower overlapping window overwrote the broader run's \ + object: {ids:?}" + ); + } + Ok(()) + } + + // The endpoint rejects a window whose upper bound is not yet settled (a row's + // timestamp is its txn's xact_start, so a future/live `to` could miss late + // commits), but accepts a window safely in the past. + #[sqlx::test(migrations = "../migrations")] + async fn backfill_rejects_unstable_window(db: DB) -> anyhow::Result<()> { + let future = Utc::now() + chrono::Duration::days(1); + let past_from = Utc::now() - chrono::Duration::days(2); + let err = try_start(&db, past_from, future).await.unwrap_err(); + assert!( + matches!(err, error::Error::BadRequest(_)), + "a future `to` must be rejected as unstable, got {err:?}" + ); + + // A window fully in the settled past is accepted. + let from = Utc::now() - chrono::Duration::days(3); + let to = Utc::now() - chrono::Duration::days(2); + try_start(&db, from, to) + .await + .expect("a settled past window is accepted"); + Ok(()) + } + + /// Insert a row into the legacy (non-partitioned) `audit` table at an exact time. + async fn insert_legacy_audit_at(db: &DB, operation: &str, ts: DateTime) { + sqlx::query( + "INSERT INTO audit (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{}'::jsonb,$2)", + ) + .bind(operation) + .bind(ts) + .execute(db) + .await + .expect("insert legacy audit row"); + } + + // A window overlapping rows in the legacy (non-partitioned) `audit` table is rejected: + // those rows are not exported, so the backfill must not report success while silently + // omitting them. Covers the empty-`audit_partitioned` case (a min(partitioned) guard + // would no-op there). + #[sqlx::test(migrations = "../migrations")] + async fn backfill_rejects_window_overlapping_legacy(db: DB) -> anyhow::Result<()> { + // A legacy row ~5 days ago, and no partitioned rows at all. + insert_legacy_audit_at(&db, "legacy.row", Utc::now() - chrono::Duration::days(5)).await; + + // A window covering it is rejected. + let from = Utc::now() - chrono::Duration::days(6); + let to = Utc::now() - chrono::Duration::days(2); + let err = try_start(&db, from, to).await.unwrap_err(); + assert!( + matches!(err, error::Error::BadRequest(_)), + "a window overlapping legacy audit rows must be rejected, got {err:?}" + ); + + // A window clear of any legacy row is accepted. + let from_ok = Utc::now() - chrono::Duration::days(2); + let to_ok = Utc::now() - chrono::Duration::days(1); + try_start(&db, from_ok, to_ok) + .await + .expect("a window with no legacy overlap is accepted"); + Ok(()) + } +} diff --git a/backend/windmill-api-settings/src/lib.rs b/backend/windmill-api-settings/src/lib.rs index 7a278afc49..a7e1322ad9 100644 --- a/backend/windmill-api-settings/src/lib.rs +++ b/backend/windmill-api-settings/src/lib.rs @@ -14,6 +14,8 @@ use std::{ #[cfg(feature = "parquet")] mod audit_logs_s3; #[cfg(feature = "parquet")] +mod audit_logs_s3_backfill; +#[cfg(feature = "parquet")] mod background_task; #[cfg(feature = "private")] mod ee; @@ -204,7 +206,12 @@ pub fn global_service() -> Router { ) .route("/run_log_cleanup", post(run_log_cleanup)) .route("/log_cleanup_status", get(log_cleanup_status)) - .route("/audit_logs_s3_status", get(audit_logs_s3_status)); + .route("/audit_logs_s3_status", get(audit_logs_s3_status)) + .route("/audit_logs_s3_backfill", post(run_audit_logs_s3_backfill)) + .route( + "/audit_logs_s3_backfill_status", + get(audit_logs_s3_backfill_status), + ); } #[cfg(not(feature = "parquet"))] @@ -256,52 +263,300 @@ pub async fn test_s3_bucket( use bytes::Bytes; use futures::StreamExt; - require_super_admin(&db, &authed.email).await?; + // The probe executes on the API server itself. On multi-tenant Cloud that is a shared control + // plane, so we constrain untrusted callers to remove the SSRF / credential-exfiltration / + // local-filesystem surface (see validate_object_storage_test). On self-hosted instances the + // object store usually lives on the local/private network and all authenticated users are + // trusted, so testing there stays unrestricted. Super admins keep the unrestricted path too. + let is_super_admin = is_super_admin_email(&db, &authed.email).await?; + let restrict = !is_super_admin && *CLOUD_HOSTED; + if restrict { + validate_object_storage_test(&test_s3_bucket).await?; + } let client = build_object_store_from_settings(test_s3_bucket, Some(&db)) .await? .store; - let mut list = client.list(Some( - &windmill_object_store::object_store_reexports::Path::from("".to_string()), - )); - let first_file = list.next().await; - if first_file.is_some() { - if let Err(e) = first_file.as_ref().unwrap() { - tracing::error!("error listing bucket: {e:#}"); - error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}")); + let run = async { + let mut list = client.list(Some( + &windmill_object_store::object_store_reexports::Path::from("".to_string()), + )); + let first_file = list.next().await; + if first_file.is_some() { + if let Err(e) = first_file.as_ref().unwrap() { + tracing::error!("error listing bucket: {e:#}"); + error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}")); + } + tracing::info!("Listed files: {:?}", first_file.unwrap()); + } else { + tracing::info!("No files in blob storage"); } - tracing::info!("Listed files: {:?}", first_file.unwrap()); + + let path = windmill_object_store::object_store_reexports::Path::from(format!( + "/test-s3-bucket-{uuid}", + uuid = uuid::Uuid::new_v4() + )); + tracing::info!("Testing blob storage at path: {path}"); + client + .put( + &path, + windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"), + ) + .await + .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?; + let content = client + .get(&path) + .await + .map_err(to_anyhow)? + .bytes() + .await + .map_err(to_anyhow)?; + if content != Bytes::from_static(b"hello") { + return Err(error::Error::internal_err( + "Failed to read back from blob storage".to_string(), + )); + } + client.delete(&path).await.map_err(to_anyhow)?; + Ok::("Tested blob storage successfully".to_string()) + }; + + if restrict { + // The object-store client is built with timeouts disabled, so a malicious endpoint could + // otherwise hold the API server connection open indefinitely. + tokio::time::timeout(Duration::from_secs(15), run) + .await + .map_err(|_| { + error::Error::internal_err("Object storage connectivity test timed out".to_string()) + })? } else { - tracing::info!("No files in blob storage"); + run.await + } +} + +// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller on +// Cloud. The probe runs on the shared API server, so without these constraints an authenticated +// user could coerce the server into connecting to arbitrary internal endpoints (SSRF), signing +// requests with the instance role (credential exfiltration), or reading/writing the server's local +// disk (filesystem object store). +#[cfg(feature = "parquet")] +async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Result<()> { + fn non_empty(opt: &Option) -> bool { + opt.as_ref().is_some_and(|s| !s.is_empty()) } - let path = windmill_object_store::object_store_reexports::Path::from(format!( - "/test-s3-bucket-{uuid}", - uuid = uuid::Uuid::new_v4() - )); - tracing::info!("Testing blob storage at path: {path}"); - client - .put( - &path, - windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"), - ) + // Reject backends that rely on the server's identity or local filesystem, require explicit + // credentials for the rest (so the server never falls back to its own ambient credentials), and + // resolve the host the client will actually connect to. We derive the *effective* endpoint here + // — mirroring build_*_from_settings: the region/account-derived default and the virtual-hosted + // bucket prefix — rather than only validating a caller-supplied `endpoint`, so caller-controlled + // `region`/`account_name`/`bucket` cannot smuggle an internal host past the check (e.g. an empty + // endpoint with region = "@169.254.169.254/" otherwise resolves to the cloud metadata service). + let effective_endpoint: Option = match settings { + ObjectSettings::Filesystem(_) => { + return Err(error::Error::NotAuthorized( + "Testing a local filesystem object store requires a super admin".to_string(), + )); + } + ObjectSettings::AwsOidc(_) => { + return Err(error::Error::NotAuthorized( + "Testing OIDC-based object storage requires a super admin".to_string(), + )); + } + ObjectSettings::S3(s3) => { + if !(non_empty(&s3.access_key) && non_empty(&s3.secret_key)) { + return Err(error::Error::NotAuthorized( + "Testing S3 storage without explicit credentials requires a super admin" + .to_string(), + )); + } + let region = s3 + .region + .clone() + .filter(|r| !r.is_empty()) + .or_else(|| std::env::var("AWS_REGION").ok().filter(|r| !r.is_empty())) + .unwrap_or_else(|| "us-east-1".to_string()); + let raw_endpoint = s3 + .endpoint + .clone() + .filter(|e| !e.is_empty()) + .or_else(|| std::env::var("S3_ENDPOINT").ok().filter(|e| !e.is_empty())) + .unwrap_or_else(|| format!("s3.{region}.amazonaws.com")); + Some(windmill_object_store::render_endpoint( + raw_endpoint, + !s3.allow_http.unwrap_or(true), + s3.port, + s3.path_style, + s3.bucket.clone().unwrap_or_default(), + )) + } + ObjectSettings::Azure(azure) => { + if !non_empty(&azure.access_key) { + return Err(error::Error::NotAuthorized( + "Testing Azure storage without an explicit access key requires a super admin" + .to_string(), + )); + } + Some( + azure + .endpoint + .clone() + .filter(|e| !e.is_empty()) + .unwrap_or_else(|| format!("{}.blob.core.windows.net", azure.account_name)), + ) + } + ObjectSettings::Gcs(gcs) => { + if gcs.service_account_key.is_empty() { + return Err(error::Error::NotAuthorized( + "Testing GCS storage without a service account key requires a super admin" + .to_string(), + )); + } + // The service-account-key JSON can override the data-plane URL (`gcs_base_url`) and the + // OAuth token endpoint (`token_uri`); the GCS client connects to whatever they point at. + // Validate every http(s) URL embedded in the key. When none override it, the host stays + // the public storage.googleapis.com, so no further check is needed. + if let Ok(serde_json::Value::Object(map)) = + serde_json::from_str::(&gcs.service_account_key) + { + for value in map.values() { + if let Some(url) = value.as_str() { + // Match how the URL parser reads the value: leading whitespace/control is + // ignored and the scheme is case-insensitive. + let url = + url.trim_start_matches(|c: char| c.is_whitespace() || c.is_control()); + if strip_http_scheme(url).is_some() { + validate_public_endpoint(url).await?; + } + } + } + } + None + } + }; + + // Block non-public network targets (internal services, cloud metadata, loopback, ...). + if let Some(endpoint) = effective_endpoint { + validate_public_endpoint(&endpoint).await?; + } + Ok(()) +} + +#[cfg(feature = "parquet")] +async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> { + let host = extract_host(endpoint).ok_or_else(|| { + error::Error::BadRequest(format!("Invalid object storage endpoint: {endpoint}")) + })?; + + let addrs: Vec = tokio::net::lookup_host((host.as_str(), 443u16)) .await - .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?; - let content = client - .get(&path) - .await - .map_err(to_anyhow)? - .bytes() - .await - .map_err(to_anyhow)?; - if content != Bytes::from_static(b"hello") { - return Err(error::Error::internal_err( - "Failed to read back from blob storage".to_string(), - )); + .map_err(|e| { + error::Error::BadRequest(format!( + "Could not resolve object storage endpoint '{host}': {e}" + )) + })? + .collect(); + + if addrs.is_empty() { + return Err(error::Error::BadRequest(format!( + "Could not resolve object storage endpoint '{host}'" + ))); + } + + // Reject if any resolved address is non-public, which also defeats the simplest DNS-rebinding + // attempts (a name resolving to both a public and a private address). + for addr in addrs { + if is_forbidden_ip(addr.ip()) { + return Err(error::Error::NotAuthorized( + "Testing object storage at a private, loopback, or link-local endpoint requires a super admin" + .to_string(), + )); + } + } + Ok(()) +} + +// Strip a leading `http://`/`https://` scheme case-insensitively (URL schemes are +// case-insensitive), returning the remainder when one was present. +#[cfg(feature = "parquet")] +fn strip_http_scheme(s: &str) -> Option<&str> { + for scheme in ["https://", "http://"] { + let b = scheme.as_bytes(); + if s.len() >= b.len() && s.as_bytes()[..b.len()].eq_ignore_ascii_case(b) { + return Some(&s[b.len()..]); + } + } + None +} + +#[cfg(feature = "parquet")] +fn extract_host(endpoint: &str) -> Option { + let mut s = endpoint.trim(); + if let Some(rest) = strip_http_scheme(s) { + s = rest; + } + s = s.split(['/', '?', '#', '\\']).next().unwrap_or(s); + if let Some((_, rest)) = s.rsplit_once('@') { + s = rest; + } + let host = if let Some(rest) = s.strip_prefix('[') { + // IPv6 literal, e.g. [::1]:9000 + rest.split(']').next().unwrap_or(rest) + } else { + // host or host:port + s.split(':').next().unwrap_or(s) + } + .trim(); + if host.is_empty() { + None + } else { + Some(host.to_string()) + } +} + +#[cfg(feature = "parquet")] +fn is_forbidden_ip(ip: std::net::IpAddr) -> bool { + use std::net::{IpAddr, Ipv4Addr}; + match ip { + IpAddr::V4(v4) => { + v4.is_loopback() + || v4.is_private() + || v4.is_link_local() // 169.254.0.0/16, incl. the cloud metadata endpoint + || v4.is_unspecified() + || v4.is_broadcast() + || v4.is_documentation() + || v4.is_multicast() + || v4.octets()[0] == 0 // 0.0.0.0/8 + || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xc0) == 64) // 100.64.0.0/10 CGNAT + } + IpAddr::V6(v6) => { + // Any IPv4 embedded in an IPv6 address (IPv4-mapped ::ffff:0:0/96, IPv4-compatible + // ::/96, or NAT64 64:ff9b::/96) is re-checked against the IPv4 rules, so e.g. + // 64:ff9b::169.254.169.254 cannot route to the metadata endpoint in a NAT64 network. + let seg = v6.segments(); + let is_v4_compatible = seg[0..6] == [0, 0, 0, 0, 0, 0]; + let is_nat64 = seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0]; + if let Some(v4) = v6.to_ipv4_mapped() { + return is_forbidden_ip(IpAddr::V4(v4)); + } + if is_v4_compatible || is_nat64 { + let embedded = Ipv4Addr::new( + (seg[6] >> 8) as u8, + (seg[6] & 0xff) as u8, + (seg[7] >> 8) as u8, + (seg[7] & 0xff) as u8, + ); + if is_forbidden_ip(IpAddr::V4(embedded)) { + return true; + } + } + v6.is_loopback() + || v6.is_unspecified() + || v6.is_multicast() + || (seg[0] & 0xfe00) == 0xfc00 // fc00::/7 unique local + || (seg[0] & 0xffc0) == 0xfe80 // fe80::/10 link-local + } } - client.delete(&path).await.map_err(to_anyhow)?; - Ok("Tested blob storage successfully".to_string()) } #[cfg(feature = "parquet")] @@ -353,6 +608,32 @@ async fn audit_logs_s3_status( Ok(Json(audit_logs_s3::get_status(&db).await?)) } +#[cfg(feature = "parquet")] +async fn run_audit_logs_s3_backfill( + Extension(db): Extension, + authed: ApiAuthed, + Json(req): Json, +) -> error::Result { + require_super_admin(&db, &authed.email).await?; + if !matches!(get_license_plan().await, LicensePlan::Enterprise) { + return Err(error::Error::BadRequest( + "Audit log export to object storage is an Enterprise feature".to_string(), + )); + } + audit_logs_s3_backfill::try_start(&db, req.from, req.to).await?; + audit_logs_s3_backfill::spawn_backfill(db.clone(), req.from, req.to); + Ok(axum::http::StatusCode::ACCEPTED) +} + +#[cfg(feature = "parquet")] +async fn audit_logs_s3_backfill_status( + Extension(db): Extension, + authed: ApiAuthed, +) -> error::JsonResult> { + require_super_admin(&db, &authed.email).await?; + Ok(Json(audit_logs_s3_backfill::get_status(&db).await?)) +} + #[derive(Deserialize)] pub struct TestKey { pub license_key: String, @@ -1275,8 +1556,8 @@ async fn setup_custom_instance_pg_database_inner( // Validate name to ensure it only contains alphanumeric characters // Prevents SQL injection on the instance database lazy_static::lazy_static! { - // Must start with a letter, then alphanumeric/underscore - static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_]*$").unwrap(); + // Must start with a letter, then alphanumeric/underscore/hyphen + static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_-]*$").unwrap(); } let dbname = dbname.trim(); if dbname.is_empty() { @@ -1292,7 +1573,7 @@ async fn setup_custom_instance_pg_database_inner( } if !VALID_NAME.is_match(dbname) { return Err(error::Error::BadRequest( - "Database name must start with a letter and contain only alphanumeric characters or underscores".to_string(), + "Database name must start with a letter and contain only alphanumeric characters, underscores, or hyphens".to_string(), )); } // Additional check: block PostgreSQL reserved/special names @@ -1861,3 +2142,111 @@ mod tests { ); } } + +#[cfg(all(test, feature = "parquet"))] +mod object_storage_test_hardening { + use super::{extract_host, is_forbidden_ip, validate_object_storage_test}; + use std::net::IpAddr; + use windmill_object_store::ObjectSettings; + + // IP literals (not hostnames) keep validate_public_endpoint deterministic — `lookup_host` + // parses them without any network round-trip. + fn gcs_settings(gcs_base_url: &str) -> ObjectSettings { + serde_json::from_value(serde_json::json!({ + "type": "Gcs", + "bucket": "b", + "serviceAccountKey": { "gcs_base_url": gcs_base_url, "client_email": "x@y.z" } + })) + .unwrap() + } + + #[tokio::test] + async fn rejects_gcs_internal_base_url() { + // gcs_base_url in the service-account key must not smuggle an internal host past the check, + // including via a mixed-case scheme (URL schemes are case-insensitive). + for url in [ + "http://169.254.169.254", + "HTTP://169.254.169.254", + "Https://10.0.0.5", + ] { + assert!( + validate_object_storage_test(&gcs_settings(url)) + .await + .is_err(), + "{url} should be rejected" + ); + } + } + + #[tokio::test] + async fn allows_gcs_public_base_url() { + assert!( + validate_object_storage_test(&gcs_settings("https://8.8.8.8")) + .await + .is_ok() + ); + } + + fn ip(s: &str) -> IpAddr { + s.parse().unwrap() + } + + #[test] + fn forbids_internal_ips() { + for s in [ + "127.0.0.1", // loopback + "169.254.169.254", // cloud metadata (link-local) + "10.0.0.5", // private + "172.16.3.4", // private + "192.168.1.10", // private + "0.0.0.0", // unspecified + "100.64.0.1", // CGNAT + "::1", // IPv6 loopback + "fe80::1", // IPv6 link-local + "fc00::1", // IPv6 unique local + "::ffff:127.0.0.1", // IPv4-mapped loopback + "::ffff:169.254.169.254", // IPv4-mapped metadata + "::169.254.169.254", // IPv4-compatible metadata + "64:ff9b::169.254.169.254", // NAT64-embedded metadata + "64:ff9b::a9fe:a9fe", // NAT64-embedded metadata (hex form) + ] { + assert!(is_forbidden_ip(ip(s)), "{s} should be forbidden"); + } + } + + #[test] + fn allows_public_ips() { + for s in ["8.8.8.8", "1.1.1.1", "52.95.110.1", "2606:4700:4700::1111"] { + assert!(!is_forbidden_ip(ip(s)), "{s} should be allowed"); + } + } + + #[test] + fn extracts_host_from_endpoint() { + let cases = [ + ("s3.amazonaws.com", Some("s3.amazonaws.com")), + ("https://minio.internal:9000", Some("minio.internal")), + ("http://10.0.0.5:9000/bucket", Some("10.0.0.5")), + ("user:pass@host.example:443", Some("host.example")), + ("[::1]:9000", Some("::1")), + ("https://[fe80::1]/x", Some("fe80::1")), + ("", None), + // Injection via region/bucket interpolation into the default endpoint string: the + // userinfo `@` and the path `/` must not hide the real authority from the host check. + ( + "https://s3.@169.254.169.254/.amazonaws.com", + Some("169.254.169.254"), + ), + ( + "https://@169.254.169.254/mybucket.s3.amazonaws.com", + Some("169.254.169.254"), + ), + ("s3.#@169.254.169.254/x.amazonaws.com", Some("s3.")), + // Scheme is case-insensitive. + ("HTTP://169.254.169.254", Some("169.254.169.254")), + ]; + for (input, expected) in cases { + assert_eq!(extract_host(input).as_deref(), expected, "input: {input}"); + } + } +} diff --git a/backend/windmill-api-settings/src/log_cleanup.rs b/backend/windmill-api-settings/src/log_cleanup.rs index 890b46ba9e..f5e5eab680 100644 --- a/backend/windmill-api-settings/src/log_cleanup.rs +++ b/backend/windmill-api-settings/src/log_cleanup.rs @@ -27,11 +27,14 @@ use uuid::Uuid; use crate::background_task; use windmill_common::error::{self}; +use windmill_common::jobs::delete_jobs; use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE}; use windmill_common::worker::WINDMILL_DIR; use windmill_common::{DB, INSTANCE_NAME, JOB_RETENTION_SECS, SERVICE_LOG_RETENTION_SECS}; -use windmill_object_store::object_store_reexports::{ObjectStore, Path as ObjectPath}; +use windmill_object_store::object_store_reexports::{ + ObjectStore, ObjectStoreError, Path as ObjectPath, +}; pub const TASK_NAME: &str = "log_cleanup"; @@ -61,6 +64,10 @@ pub struct LogCleanupProgress { pub total_jobs: u64, pub processed_jobs: u64, pub s3_deleted: u64, + /// Number of delete calls that returned 404 (object already absent — a no-op + /// success). GCS returns 404 per missing key where S3's DeleteObjects stays silent. + #[serde(default)] + pub s3_not_found: u64, /// Number of S3 objects inspected during the orphan scan phase. pub orphans_scanned: u64, /// Number of orphan S3 objects deleted (no corresponding DB row). @@ -81,6 +88,7 @@ impl LogCleanupProgress { total_jobs: 0, processed_jobs: 0, s3_deleted: 0, + s3_not_found: 0, orphans_scanned: 0, orphans_deleted: 0, errors: 0, @@ -132,6 +140,13 @@ impl Session { p.phase = "done".to_string(); p.clone() }; + tracing::info!( + "log cleanup finished: {} object(s) deleted from object store, {} already absent (404), {} orphans deleted, {} error(s)", + snapshot.s3_deleted, + snapshot.s3_not_found, + snapshot.orphans_deleted, + snapshot.errors + ); if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await { tracing::warn!("log cleanup: failed to release lease: {e:#}"); @@ -179,21 +194,32 @@ pub async fn get_status(db: &DB) -> error::Result> { async fn s3_bulk_delete( store: &Arc, paths: Vec, -) -> (u64 /* deleted */, u64 /* errors */) { +) -> ( + u64, /* deleted */ + u64, /* not_found */ + u64, /* errors */ +) { let stream = futures::stream::iter(paths.into_iter().map(Ok)).boxed(); let mut deleted = 0u64; + let mut not_found = 0u64; let mut errors = 0u64; let mut res = store.delete_stream(stream); while let Some(r) = res.next().await { match r { Ok(_) => deleted += 1, + // Deleting a non-existent object is a successful no-op. S3's DeleteObjects + // ignores missing keys, but GCS returns 404 per delete, surfacing as + // NotFound — track it separately so it isn't reported as an error. + Err(ObjectStoreError::NotFound { .. }) => { + not_found += 1; + } Err(e) => { errors += 1; tracing::warn!("log cleanup: failed to delete object: {e:#}"); } } } - (deleted, errors) + (deleted, not_found, errors) } /// Delete the given relative paths from the local filesystem under `base_dir`. @@ -265,7 +291,7 @@ async fn cleanup_service_logs( .iter() .map(|p| ObjectPath::from(format!("{}{}", LOGS_SERVICE, p))) .collect(); - let (deleted, errors) = s3_bulk_delete(store, s3_paths).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, s3_paths).await; disk_bulk_delete(&*TMP_WINDMILL_LOGS_SERVICE, &rel_paths).await; session @@ -275,6 +301,7 @@ async fn cleanup_service_logs( p.total_service = p.processed_service; } p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; @@ -313,19 +340,21 @@ async fn cleanup_job_logs( return Ok(()); } + let mut completed_at_floor: Option> = None; loop { - let (deleted_count, rel_paths) = - delete_expired_jobs_batch(db, retention_secs, JOB_BATCH).await?; + let (deleted_count, rel_paths, max_completed_at) = + delete_expired_jobs_batch(db, retention_secs, JOB_BATCH, completed_at_floor).await?; if deleted_count == 0 { break; } + completed_at_floor = max_completed_at.or(completed_at_floor); let s3_paths: Vec = rel_paths .iter() .map(|p| ObjectPath::from(p.clone())) .collect(); - let (deleted, errors) = s3_bulk_delete(store, s3_paths).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, s3_paths).await; disk_bulk_delete(&*WINDMILL_DIR, &rel_paths).await; session @@ -335,6 +364,7 @@ async fn cleanup_job_logs( p.total_jobs = p.processed_jobs; } p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; @@ -355,7 +385,8 @@ async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result<(usize, Vec)> { + completed_at_floor: Option>, +) -> error::Result<(usize, Vec, Option>)> { let mut tx = db.begin().await?; let active_root_job_ids: Vec = sqlx::query_scalar!( @@ -368,29 +399,61 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` carries a watermark across batches so each one resumes after the rows + // the previous batch processed instead of re-scanning the (potentially undeletable) oldest + // prefix; the empty-active-roots branch skips the v2_job join entirely. See + // backend/src/monitor.rs::delete_expired_jobs_batch for the full rationale. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); if deleted_count == 0 { tx.commit().await?; - return Ok((0, Vec::new())); + return Ok((0, Vec::new(), max_completed_at)); } if let Err(e) = sqlx::query!( @@ -421,10 +484,21 @@ async fn delete_expired_jobs_batch( } }; - if let Err(e) = sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", &deleted_jobs) - .execute(&mut *tx) - .await + // Native retry markers have no FK (to keep this bulk delete cheap) — sweep + // them with their jobs here, same as the other side tables above. The table + // is created by a startup migration, so it always exists by the time cleanup + // runs. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + &deleted_jobs + ) + .execute(&mut *tx) + .await { + tracing::error!("log cleanup: error deleting native retry markers: {e:?}"); + } + + if let Err(e) = delete_jobs(&mut *tx, &deleted_jobs).await { tracing::error!("log cleanup: error deleting job: {e:?}"); } @@ -440,7 +514,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok((deleted_count, log_paths)) + Ok((deleted_count, log_paths, max_completed_at)) } /// Scan S3 under the `logs/` prefix for orphan log files and delete them. @@ -561,11 +635,12 @@ async fn flush_service_orphans( batch: &mut Vec, ) { let paths = std::mem::take(batch); - let (deleted, errors) = s3_bulk_delete(store, paths).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, paths).await; session .update(|p| { p.orphans_deleted = p.orphans_deleted.saturating_add(deleted); p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; @@ -616,11 +691,12 @@ async fn flush_job_orphans( return; } - let (deleted, errors) = s3_bulk_delete(store, to_delete).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, to_delete).await; session .update(|p| { p.orphans_deleted = p.orphans_deleted.saturating_add(deleted); p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; diff --git a/backend/windmill-api-users/src/users.rs b/backend/windmill-api-users/src/users.rs index db9ecb118d..2963b91a50 100644 --- a/backend/windmill-api-users/src/users.rs +++ b/backend/windmill-api-users/src/users.rs @@ -27,7 +27,7 @@ use axum::{ Json, Router, }; use hyper::{header::LOCATION, StatusCode}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin, OptJobAuthed}; use windmill_common::usernames::{ generate_instance_wide_unique_username, get_instance_username_or_create_pending, }; @@ -1415,11 +1415,13 @@ async fn convert_user_to_group( async fn update_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(email_to_update): Path, Extension(db): Extension, Json(eu): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; let mut new_super_admin: Option = None; @@ -1581,10 +1583,12 @@ async fn update_user( async fn delete_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(email_to_delete): Path, Extension(db): Extension, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!("DELETE FROM token WHERE email = $1", &email_to_delete) @@ -1840,7 +1844,7 @@ async fn delete_workspace_user( username_to_delete, &w_id, ) - .fetch_optional(&db) + .fetch_optional(&mut *tx) .await?; let email_to_delete = not_found_if_none(email_to_delete_o, "User", &username_to_delete)?; @@ -1877,9 +1881,11 @@ async fn set_login_type( Extension(db): Extension, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(et): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!( @@ -2019,8 +2025,6 @@ async fn refresh_token( authed: ApiAuthed, cookies: Cookies, ) -> Result { - let mut tx = db.begin().await?; - if let Some(thresh_s) = query.if_expiring_in_less_than_s { let t_hash = windmill_common::auth::hash_token(&token); let not_expired = sqlx::query_scalar!("SELECT true FROM token WHERE token_hash = $1 and expiration IS NOT NULL and expiration > now() + $2::int * '1 sec'::interval", &t_hash, thresh_s) @@ -2033,6 +2037,8 @@ async fn refresh_token( } } + let mut tx = db.begin().await?; + let super_admin = sqlx::query_scalar!( "SELECT super_admin FROM password WHERE email = $1 AND disabled = false", &authed.email @@ -2159,8 +2165,10 @@ pub async fn create_session_token<'c>( async fn create_token( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(token_config): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; check_token_create_rate_limit(&authed.username)?; windmill_api_auth::ensure_scopes_within_caller(&authed, token_config.scopes.as_deref())?; @@ -2176,6 +2184,7 @@ async fn create_token( async fn impersonate( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(new_token): Json, ) -> Result<(StatusCode, String)> { use windmill_common::min_version::MIN_VERSION_SUPPORTS_TOKEN_HASH; @@ -2189,6 +2198,7 @@ async fn impersonate( Some(&token) }; require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; if new_token.impersonate_email.is_none() { return Err(Error::BadRequest( @@ -2707,8 +2717,10 @@ struct ExportedGlobalUser { async fn export_global_users( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, ) -> JsonResult> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; let users = sqlx::query_as!( ExportedGlobalUser, @@ -2744,9 +2756,11 @@ async fn export_global_users() -> JsonResult { async fn overwrite_global_users( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(users): Json>, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!("DELETE FROM password") .execute(&mut *tx) diff --git a/backend/windmill-api-workspaces/src/deployment_requests.rs b/backend/windmill-api-workspaces/src/deployment_requests.rs index 7782382073..a279f82d6c 100644 --- a/backend/windmill-api-workspaces/src/deployment_requests.rs +++ b/backend/windmill-api-workspaces/src/deployment_requests.rs @@ -717,16 +717,27 @@ async fn create_deployment_request_comment( // ---- helpers ------------------------------------------------------------ async fn parent_of_fork(db: &DB, w_id: &str) -> Result { - sqlx::query_scalar!( - "SELECT parent_workspace_id FROM workspace WHERE id = $1", + // Resolve the fork's parent and require it to still exist and be active. A + // parent that is archived (soft-deleted) can no longer be accessed, so a + // diff or deployment request against it targets an unreachable workspace. + let parent = sqlx::query!( + "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\" + FROM workspace f + JOIN workspace p ON p.id = f.parent_workspace_id + WHERE f.id = $1", w_id, ) .fetch_optional(db) - .await? - .flatten() - .ok_or_else(|| { - Error::BadRequest(format!( + .await?; + + match parent { + None => Err(Error::BadRequest(format!( "workspace {w_id} is not a fork (no parent_workspace_id)" - )) - }) + ))), + Some(p) if p.deleted => Err(Error::BadRequest(format!( + "parent workspace {} of fork {w_id} is archived", + p.id + ))), + Some(p) => Ok(p.id), + } } diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 2363773446..250ce7059a 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -244,6 +244,7 @@ pub fn global_service() -> Router { .route("/list_as_superadmin", get(list_workspaces_as_super_admin)) .route("/list", get(list_workspaces)) .route("/users", get(user_workspaces)) + .route("/session_workspace_status", post(session_workspace_status)) .route("/create", post(create_workspace)) .route("/create_fork", post(deprecated_create_workspace_fork)) .route("/exists", post(exists_workspace)) @@ -2395,7 +2396,7 @@ async fn edit_ducklake_config( "#, &w_id ) - .fetch_one(&db) + .fetch_one(&mut *tx) .await? .unwrap_or(serde_json::Value::Null); let old_ducklakes: HashMap = @@ -4919,6 +4920,47 @@ async fn user_workspaces( Ok(Json(WorkspaceList { email, workspaces })) } +#[derive(Deserialize)] +struct SessionWorkspaceStatusRequest { + workspace_ids: Vec, +} + +/// Reconciliation support for client-side AI sessions, which the backend cannot touch +/// directly. The client posts the workspace ids its sessions reference and uses the +/// per-id status to keep sessions in sync with workspace lifecycle: `deleted` (no row / +/// no access → unresolvable) drops the sessions, `archived` (soft-deleted, still a +/// member) archives them, `active` restores ones previously archived-by-workspace. +/// Archived and hard-deleted workspaces are absent from `user_workspaces`, so this is the +/// only way the client learns about a change made while it was away or on another device. +async fn session_workspace_status( + Extension(db): Extension, + ApiAuthed { email, .. }: ApiAuthed, + Json(req): Json, +) -> JsonResult> { + if req.workspace_ids.len() > 1000 { + return Err(Error::BadRequest( + "Too many workspace ids (max 1000)".to_string(), + )); + } + let rows = sqlx::query!( + "SELECT req.id AS \"id!\", + (CASE + WHEN usr.email IS NULL THEN 'deleted' + WHEN workspace.deleted THEN 'archived' + ELSE 'active' + END) AS \"status!\" + FROM unnest($1::text[]) AS req(id) + LEFT JOIN workspace ON workspace.id = req.id + LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + &req.workspace_ids[..], + email, + ) + .fetch_all(&db) + .await?; + let statuses = rows.into_iter().map(|r| (r.id, r.status)).collect(); + Ok(Json(statuses)) +} + pub async fn check_w_id_conflict<'c>(tx: &mut Transaction<'c, Postgres>, w_id: &str) -> Result<()> { if w_id == "global" { return Err(windmill_common::error::Error::BadRequest( diff --git a/backend/windmill-api-workspaces/src/workspaces_extra.rs b/backend/windmill-api-workspaces/src/workspaces_extra.rs index aa64514b4c..3cb2b29f79 100644 --- a/backend/windmill-api-workspaces/src/workspaces_extra.rs +++ b/backend/windmill-api-workspaces/src/workspaces_extra.rs @@ -65,13 +65,22 @@ pub(crate) async fn change_workspace_id( old_id, rw.new_id ); - // Create new workspace with new id and name + // Create new workspace with new id and name. A fork that keeps a wm-fork- + // id must carry its parent_workspace_id over, otherwise it becomes a + // parentless "fork of nothing" with no source to compare or merge against. + // A non-fork target id means the workspace is being promoted out of a fork, + // so the parent pointer is intentionally cleared. info!("Creating new workspace row"); + let new_is_fork = rw.new_id.starts_with(WM_FORK_PREFIX); sqlx::query!( - "INSERT INTO workspace SELECT $1, $2, owner, false, premium FROM workspace WHERE id = $3", + "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id) + SELECT $1, $2, owner, false, premium, + CASE WHEN $4 THEN parent_workspace_id ELSE NULL END + FROM workspace WHERE id = $3", &rw.new_id, &rw.new_name, - &old_id + &old_id, + new_is_fork ) .execute(&mut *tx) .await?; @@ -347,6 +356,18 @@ pub(crate) async fn change_workspace_id( .execute(&mut *tx) .await?; + // Re-parent child forks: any fork whose parent_workspace_id was the old id + // must follow the renamed parent to the new id, otherwise it is left + // pointing at the soft-deleted old shell (whose data has moved here). + info!("Re-parenting child forks to the new workspace id"); + sqlx::query!( + "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2", + &rw.new_id, + &old_id + ) + .execute(&mut *tx) + .await?; + info!("Updating workspace_protection_rule table"); sqlx::query!( "UPDATE workspace_protection_rule SET workspace_id = $1 WHERE workspace_id = $2", @@ -745,6 +766,18 @@ pub(crate) async fn delete_workspace( sqlx::query!("DELETE FROM v2_job_queue WHERE workspace_id = $1", &w_id) .execute(&mut *tx) .await?; + // dispatch_event / flow_conversation_message / zombie_job_counter no longer cascade from + // v2_job (see migration drop_v2_job_side_table_cascades); delete them before v2_job so the + // workspace's jobs leave no orphan side rows. One round-trip, scanning v2_job once. + sqlx::query!( + "WITH ids AS (SELECT id FROM v2_job WHERE workspace_id = $1), + _de AS (DELETE FROM dispatch_event WHERE workspace_id = $1), + _fc AS (DELETE FROM flow_conversation_message WHERE job_id IN (SELECT id FROM ids)) + DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM ids)", + &w_id + ) + .execute(&mut *tx) + .await?; sqlx::query!("DELETE FROM v2_job WHERE workspace_id = $1", &w_id) .execute(&mut *tx) .await?; diff --git a/backend/windmill-api/Cargo.toml b/backend/windmill-api/Cargo.toml index 94e72e7c4a..6ce463a5aa 100644 --- a/backend/windmill-api/Cargo.toml +++ b/backend/windmill-api/Cargo.toml @@ -43,7 +43,7 @@ mcp = ["dep:windmill-mcp", "windmill-mcp/server", "windmill-mcp/auth", "windmill bedrock = ["windmill-ai/bedrock"] python = ["windmill-dep-map/python", "dep:windmill-parser-py", "dep:windmill-parser-py-imports", "windmill-api-scripts/python", "windmill-api-configs/python", "windmill-api-agent-workers?/python", "windmill-trigger/python", "windmill-common/python"] no_auth = ["windmill-api-auth/no_auth", "windmill-store/no_auth", "windmill-api-users/no_auth"] -quickjs = ["windmill-jseval/quickjs"] +quickjs = ["windmill-jseval/quickjs", "windmill-queue/quickjs"] [dependencies] windmill-ai = { workspace = true, default-features = false } diff --git a/backend/windmill-api/docs_snapshot/README.md b/backend/windmill-api/docs_snapshot/README.md new file mode 100644 index 0000000000..09a1ac6615 --- /dev/null +++ b/backend/windmill-api/docs_snapshot/README.md @@ -0,0 +1,19 @@ +# Vendored docs snapshot + +`llms.txt.gz` (curated page index) and `llms-full.txt.gz` (full corpus, every docs +page concatenated and delimited by `Source:` lines) are a gzipped snapshot of +`https://www.windmill.dev/llms.txt` and `/llms-full.txt`. + +They are embedded into the binary by `../src/docs/corpus.rs` (`include_bytes!`) and +decompressed/parsed once at first use. This lets in-product docs search +(`GET /api/docs/search`, `GET /api/docs/page`) — used by the AI chat, the MCP +`searchDocs`/`readDocsPage` tools, and the `wmill docs` CLI — work with **no +runtime network egress**, including on air-gapped instances. + +The tradeoff is staleness: the snapshot is pinned to whatever was published when +`fetch.sh` was last run. Refresh on each release: + +```bash +./fetch.sh # re-downloads and re-gzips both files +git add llms.txt.gz llms-full.txt.gz +``` diff --git a/backend/windmill-api/docs_snapshot/fetch.sh b/backend/windmill-api/docs_snapshot/fetch.sh new file mode 100755 index 0000000000..bf24d42f7b --- /dev/null +++ b/backend/windmill-api/docs_snapshot/fetch.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Refresh the vendored documentation snapshot embedded into windmill-api. +# +# The backend self-hosts the docs corpus (see ../src/docs/) so docs search works +# with no runtime egress. This snapshot is pinned to whatever was published on +# windmill.dev when this script was last run — re-run it on each release to keep +# the in-product docs search reasonably fresh, then commit the updated *.gz. +set -euo pipefail + +DOCS_ORIGIN="${DOCS_ORIGIN:-https://www.windmill.dev}" +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +fetch() { + local name="$1" + echo "Fetching ${DOCS_ORIGIN}/${name} ..." + curl -fSL "${DOCS_ORIGIN}/${name}" -o "${DIR}/${name}" + # -9 max compression; -n omit the original name/timestamp so the artifact is + # reproducible and diffs only when the docs actually change. + gzip -9 -n -c "${DIR}/${name}" > "${DIR}/${name}.gz" + rm -f "${DIR}/${name}" + echo " wrote ${name}.gz ($(wc -c < "${DIR}/${name}.gz") bytes)" +} + +fetch "llms.txt" +fetch "llms-full.txt" +echo "Done. Commit the updated *.gz files." diff --git a/backend/windmill-api/docs_snapshot/llms-full.txt.gz b/backend/windmill-api/docs_snapshot/llms-full.txt.gz new file mode 100644 index 0000000000..f5420b1cb5 Binary files /dev/null and b/backend/windmill-api/docs_snapshot/llms-full.txt.gz differ diff --git a/backend/windmill-api/docs_snapshot/llms.txt.gz b/backend/windmill-api/docs_snapshot/llms.txt.gz new file mode 100644 index 0000000000..abdd2b0b27 Binary files /dev/null and b/backend/windmill-api/docs_snapshot/llms.txt.gz differ diff --git a/backend/windmill-api/openapi-deref.json b/backend/windmill-api/openapi-deref.json index 135131af41..e511578821 100644 --- a/backend/windmill-api/openapi-deref.json +++ b/backend/windmill-api/openapi-deref.json @@ -1,7 +1,7 @@ { "openapi": "3.0.3", "info": { - "version": "1.728.0", + "version": "1.740.0", "title": "Windmill API", "contact": { "name": "Windmill Team", @@ -174,52 +174,124 @@ } } }, - "/inkeep": { - "post": { - "summary": "query Windmill AI documentation assistant (EE only)", - "operationId": "queryDocumentation", + "/docs/search": { + "get": { + "summary": "Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more.", + "operationId": "searchDocs", "x-mcp-tool": true, "tags": [ "documentation" ], - "requestBody": { - "description": "query to send to the AI documentation assistant", - "required": true, - "content": { - "application/json": { - "schema": { - "type": "object", - "properties": { - "query": { - "type": "string", - "description": "The documentation query to send to the AI assistant" - } - }, - "required": [ - "query" - ] - } + "parameters": [ + { + "name": "query", + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better.", + "in": "query", + "required": true, + "schema": { + "type": "string" } } - }, + ], "responses": { "200": { - "description": "AI documentation assistant response", + "description": "matching documentation pages", "content": { "application/json": { "schema": { "type": "object", - "description": "Response from Inkeep service" + "properties": { + "text": { + "type": "string", + "description": "Model-ready rendering of the results" + }, + "results": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string" + }, + "title": { + "type": "string" + }, + "score": { + "type": "integer" + }, + "snippets": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "url", + "title", + "score", + "snippets" + ] + } + } + }, + "required": [ + "text", + "results" + ] } } } + } + } + } + }, + "/docs/page": { + "get": { + "summary": "Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section.", + "operationId": "readDocsPage", + "x-mcp-tool": true, + "tags": [ + "documentation" + ], + "parameters": [ + { + "name": "url", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted.", + "in": "query", + "required": true, + "schema": { + "type": "string" + } }, - "403": { - "description": "Enterprise Edition required", + { + "name": "section", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page.", + "in": "query", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "documentation page content", "content": { - "text/plain": { + "application/json": { "schema": { - "type": "string" + "type": "object", + "properties": { + "text": { + "type": "string" + }, + "source_url": { + "type": "string" + } + }, + "required": [ + "text", + "source_url" + ] } } } @@ -1531,6 +1603,56 @@ } } }, + "/workspaces/session_workspace_status": { + "post": { + "summary": "get the lifecycle status of workspaces referenced by client-side sessions", + "operationId": "getSessionWorkspaceStatus", + "tags": [ + "workspace" + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "workspace_ids": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "workspace_ids" + ] + } + } + } + }, + "responses": { + "200": { + "description": "map of workspace id to status (active, archived, or deleted)", + "content": { + "application/json": { + "schema": { + "type": "object", + "additionalProperties": { + "type": "string", + "enum": [ + "active", + "archived", + "deleted" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/workspaces/get_as_superadmin": { "get": { "summary": "get workspace as super admin (require to be super admin)", @@ -2492,6 +2614,10 @@ "type": "integer", "format": "int64" }, + "s3_not_found": { + "type": "integer", + "format": "int64" + }, "orphans_scanned": { "type": "integer", "format": "int64" @@ -2593,6 +2719,124 @@ } } }, + "/settings/audit_logs_s3_backfill": { + "post": { + "summary": "start an opt-in historical backfill of audit logs to object storage", + "operationId": "runAuditLogsS3Backfill", + "tags": [ + "setting" + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "from": { + "type": "string", + "format": "date-time", + "description": "inclusive lower bound of the window to export" + }, + "to": { + "type": "string", + "format": "date-time", + "description": "exclusive upper bound of the window to export" + } + }, + "required": [ + "from", + "to" + ] + } + } + } + }, + "responses": { + "202": { + "description": "backfill started" + } + } + } + }, + "/settings/audit_logs_s3_backfill_status": { + "get": { + "summary": "get status of the audit-log object-store historical backfill", + "operationId": "getAuditLogsS3BackfillStatus", + "tags": [ + "setting" + ], + "responses": { + "200": { + "description": "current backfill status (null if never run)", + "content": { + "application/json": { + "schema": { + "nullable": true, + "type": "object", + "properties": { + "running": { + "type": "boolean" + }, + "started_at": { + "type": "string", + "format": "date-time" + }, + "finished_at": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "phase": { + "type": "string" + }, + "from": { + "type": "string", + "format": "date-time" + }, + "to": { + "type": "string", + "format": "date-time" + }, + "rows_written": { + "type": "integer", + "format": "int64" + }, + "objects_written": { + "type": "integer", + "format": "int64" + }, + "last_ts": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "errors": { + "type": "integer", + "format": "int64" + }, + "last_error": { + "type": "string", + "nullable": true + } + }, + "required": [ + "running", + "started_at", + "phase", + "from", + "to", + "rows_written", + "objects_written", + "errors" + ] + } + } + } + } + } + } + }, "/settings/send_stats": { "post": { "summary": "send stats", @@ -9704,6 +9948,10 @@ "type": "string", "description": "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side (client_credentials flow only). The token URL is never caller-supplied." }, + "cc_token_url": { + "type": "string", + "description": "Bring-your-own token endpoint override (client_credentials flow only). Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; ignored/rejected on the shared-instance path." + }, "mcp_server_url": { "type": "string", "description": "MCP server URL for MCP OAuth token refresh" @@ -9785,6 +10033,10 @@ "cc_instance": { "type": "string", "description": "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side. The token URL is never caller-supplied." + }, + "cc_token_url": { + "type": "string", + "description": "Bring-your-own token endpoint override. Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; rejected on the shared-instance path." } } } @@ -11614,6 +11866,32 @@ } } }, + "/apps_u/embed_token_by_custom_path/{custom_path}": { + "get": { + "summary": "get app embed token by custom path", + "operationId": "getAppEmbedTokenByCustomPath", + "tags": [ + "app" + ], + "parameters": [ + { + "$ref": "#/components/parameters/CustomPath" + } + ], + "responses": { + "200": { + "description": "embed token", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/EmbedTokenResponse" + } + } + } + } + } + } + }, "/scripts/hub/get/{path}": { "get": { "summary": "get hub script content by path", @@ -12188,6 +12466,14 @@ "parameters": [ { "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "all_users", + "in": "query", + "description": "List every draft in the workspace (all users), not just the current user's own + legacy rows. Other users' rows come back with `mine=false` (view-only).", + "schema": { + "type": "boolean" + } } ], "responses": { @@ -12225,6 +12511,27 @@ "created_at": { "type": "string", "format": "date-time" + }, + "can_write": { + "type": "boolean", + "description": "Whether the current user may deploy/discard this draft (same check the deploy/discard endpoints enforce)." + }, + "mine": { + "type": "boolean", + "description": "The row belongs to the current user (own draft or the legacy no-owner row) and is therefore actionable. Always true in the default listing; with `all_users=true`, other users' rows are false (view-only)." + }, + "draft_users": { + "description": "Draft authors at this (path, kind) — the legacy NULL-email row surfaced as a null username.\nPopulated only for the shared full-page-editor kinds (script/flow/app/raw_app); omitted for\ndrawer kinds, which keep their drafts private. Feeds the Draft badge's owner-avatar circles.\n", + "type": "array", + "items": { + "type": "object", + "properties": { + "username": { + "type": "string", + "nullable": true + } + } + } } }, "required": [ @@ -12232,7 +12539,9 @@ "path", "draft_only", "legacy_draft", - "created_at" + "created_at", + "can_write", + "mine" ] } } @@ -12302,6 +12611,55 @@ } } }, + "/w/{workspace}/drafts/get_own/{kind}/{path}": { + "get": { + "summary": "fetch the current user's own draft content at a path (any kind)", + "operationId": "getOwnDraft", + "tags": [ + "draft" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "kind", + "in": "path", + "required": true, + "schema": { + "$ref": "#/components/schemas/UserDraftItemKind" + } + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "responses": { + "200": { + "description": "the user's draft content, or null when none exists", + "content": { + "application/json": { + "schema": { + "nullable": true, + "type": "object", + "properties": { + "value": {}, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "value", + "created_at" + ] + } + } + } + } + } + } + }, "/w/{workspace}/drafts/update/{kind}/{path}": { "post": { "summary": "upsert (or clear) the current user's draft at a path", @@ -12348,6 +12706,11 @@ "legacy": { "type": "boolean", "description": "Delete-only. Target the legacy workspace-level row (email NULL) instead of the current user's row. Used to discard a legacy draft from the review page." + }, + "created_at": { + "type": "string", + "format": "date-time", + "description": "Upsert-only override for the stored creation timestamp. Normal saves omit it (stamped server-side); the localStorage→DB migration passes the draft's original write time so migrated drafts keep their age." } } } @@ -12385,6 +12748,67 @@ } } }, + "/w/{workspace}/drafts/migrate_legacy/{kind}/{path}": { + "post": { + "summary": "resolve a legacy (workspace-level) draft (admin only)", + "description": "Delete a legacy draft (email NULL) or assign it to the authed admin as a per-user draft. Workspace admins / superadmins only.", + "operationId": "migrateLegacyDraft", + "tags": [ + "draft" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "kind", + "in": "path", + "required": true, + "schema": { + "$ref": "#/components/schemas/UserDraftItemKind" + } + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "delete", + "assign_to_self" + ], + "description": "delete the legacy draft, or take ownership of it." + } + }, + "required": [ + "action" + ] + } + } + } + }, + "responses": { + "200": { + "description": "migration result", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, "/w/{workspace}/scripts/create": { "post": { "summary": "create script", @@ -16099,6 +16523,202 @@ } } }, + "/w/{workspace}/ai_skills/list": { + "get": { + "summary": "list the workspace AI chat skills (name + description only)", + "operationId": "listAiSkills", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + } + ], + "responses": { + "200": { + "description": "skill listing", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "required": [ + "name", + "description" + ], + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + } + } + } + } + } + } + } + } + } + }, + "/w/{workspace}/ai_skills/get/{name}": { + "get": { + "summary": "get a workspace AI chat skill including its instructions", + "operationId": "getAiSkill", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "skill", + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "name", + "description", + "instructions" + ], + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + }, + "instructions": { + "type": "string" + } + } + } + } + } + } + } + } + }, + "/w/{workspace}/ai_skills/upload": { + "post": { + "summary": "upsert workspace AI chat skills (admin only)", + "operationId": "uploadAiSkills", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "skills" + ], + "properties": { + "skills": { + "type": "array", + "maxItems": 50, + "items": { + "type": "object", + "required": [ + "name", + "description", + "instructions" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "pattern": "^[a-z0-9-]+$" + }, + "description": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + }, + "instructions": { + "type": "string", + "minLength": 1, + "maxLength": 65536 + } + } + } + } + } + } + } + } + }, + "responses": { + "200": { + "description": "uploaded", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, + "/w/{workspace}/ai_skills/delete/{name}": { + "delete": { + "summary": "delete a workspace AI chat skill (admin only)", + "operationId": "deleteAiSkill", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "deleted", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, "/w/{workspace}/apps/get_data/v/{secretWithExtension}": { "get": { "summary": "get raw app data by", @@ -16114,6 +16734,7 @@ "name": "secretWithExtension", "in": "path", "required": true, + "description": "App version secret suffixed with the requested file type extension. Supported extensions are `.js` (JavaScript bundle), `.css` (stylesheet), and `.html` (sandboxed wrapper document).", "schema": { "type": "string" } @@ -16127,6 +16748,16 @@ "schema": { "type": "string" } + }, + "text/css": { + "schema": { + "type": "string" + } + }, + "text/html": { + "schema": { + "type": "string" + } } } } @@ -16518,6 +17149,35 @@ } } }, + "/w/{workspace}/apps/embed_token/p/{path}": { + "get": { + "summary": "get app embed token by path", + "operationId": "getAppEmbedTokenByPath", + "tags": [ + "app" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "responses": { + "200": { + "description": "embed token", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/EmbedTokenResponse" + } + } + } + } + } + } + }, "/w/{workspace}/apps/get/lite/{path}": { "get": { "summary": "get app lite by path", @@ -16720,6 +17380,40 @@ } } }, + "/w/{workspace}/apps_u/embed_token/{secret}": { + "get": { + "summary": "get app embed token by secret", + "operationId": "getAppEmbedTokenBySecret", + "tags": [ + "app" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "secret", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "embed token", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/EmbedTokenResponse" + } + } + } + } + } + } + }, "/w/{workspace}/apps_u/public_resource/{path}": { "get": { "summary": "get public resource", @@ -17864,6 +18558,14 @@ "type": "boolean" } }, + { + "name": "timeout", + "description": "custom timeout in seconds for this preview run", + "in": "query", + "schema": { + "type": "integer" + } + }, { "$ref": "#/components/parameters/NewJobId" } @@ -19889,6 +20591,89 @@ } } }, + "/w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}": { + "get": { + "summary": "get all logs for a flow job in a structured format", + "operationId": "getFlowAllLogsStructured", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/JobId" + } + ], + "responses": { + "200": { + "description": "structured logs of all flow steps, one entry per job", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "job_id": { + "type": "string" + }, + "label": { + "type": "string", + "description": "human-readable label describing the job's position in the flow tree" + }, + "kind": { + "type": "string", + "description": "job kind (script, flow, forloopflow, ...)" + }, + "flow_step_id": { + "type": "string", + "nullable": true + }, + "step_path": { + "type": "string", + "nullable": true, + "description": "materialized step path (e.g. \"a/b\")" + }, + "depth": { + "type": "integer", + "description": "depth in the flow tree (0 for the root flow job)" + }, + "parent_module_type": { + "type": "string", + "nullable": true, + "description": "parent module type (forloopflow, branchall, ...)" + }, + "sibling_index": { + "type": "integer", + "description": "1-based index of this job among siblings sharing the same step" + }, + "sibling_count": { + "type": "integer", + "description": "total number of siblings sharing the same step" + }, + "logs": { + "type": "string" + } + }, + "required": [ + "job_id", + "label", + "kind", + "depth", + "sibling_index", + "sibling_count", + "logs" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/jobs_u/get_completed_logs_tail/{id}": { "get": { "summary": "get completed job logs tail", @@ -20401,6 +21186,192 @@ } } }, + "/w/{workspace}/jobs_u/dispatch_events/{id}": { + "get": { + "summary": "list asset-trigger dispatch events for a producer job", + "description": "Returns the chronological log of decisions the asset-trigger dispatcher made after this producer job completed. Each row is one (subscriber, asset write) decision: `dispatched` (with `child_job_id`), `join_pending` (with `received_inputs` / `required_inputs` / `partition`), or `skipped` (with `reason`). Rows are reaped automatically when the producer's `v2_job` row is deleted by the retention sweep.\n", + "operationId": "listDispatchEvents", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/JobId" + } + ], + "responses": { + "200": { + "description": "dispatch events for this producer job", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "subscriber_path": { + "type": "string" + }, + "asset_kind": { + "type": "string", + "enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + }, + "asset_path": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "dispatched", + "join_pending", + "skipped" + ] + }, + "child_job_id": { + "type": "string", + "format": "uuid" + }, + "partition": { + "type": "string" + }, + "received_inputs": { + "type": "integer" + }, + "required_inputs": { + "type": "integer" + }, + "debounce_s": { + "type": "integer" + }, + "reason": { + "type": "string" + }, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "subscriber_path", + "asset_kind", + "asset_path", + "outcome", + "created_at" + ] + } + } + } + } + } + } + } + }, + "/w/{workspace}/jobs/asset_dispatch_edges": { + "get": { + "summary": "list asset-cascade producer→child job edges for a folder", + "description": "Returns the `dispatched` asset-trigger edges (producer job → child job) whose subscriber lives under `path_start`. Lets a pipeline view reconstruct the cascade tree of a folder by job id and group connected runs. Visibility follows the producer job's RLS.\n", + "operationId": "listAssetDispatchEdges", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "path_start", + "in": "query", + "required": true, + "description": "Folder path prefix the children live under, e.g. `f/orders/`.", + "schema": { + "type": "string" + } + }, + { + "name": "created_after", + "in": "query", + "required": false, + "description": "Only edges dispatched at/after this instant.", + "schema": { + "type": "string", + "format": "date-time" + } + } + ], + "responses": { + "200": { + "description": "asset-cascade edges for the folder", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "producer_job_id": { + "type": "string", + "format": "uuid" + }, + "child_job_id": { + "type": "string", + "format": "uuid", + "description": "Set for `dispatched`; absent for `join_pending` inputs." + }, + "subscriber_path": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "dispatched", + "join_pending" + ] + }, + "asset_kind": { + "type": "string", + "enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + }, + "asset_path": { + "type": "string" + }, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "producer_job_id", + "subscriber_path", + "outcome", + "asset_kind", + "asset_path", + "created_at" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/jobs/completed/delete/{id}": { "post": { "summary": "delete completed job (erase content but keep run id)", @@ -21097,6 +22068,10 @@ } } } + }, + "view_token": { + "type": "string", + "description": "Share-read-link token for the flow. An authenticated workspace member can append it as a `view_token` query param on the run page to read a flow they don't otherwise have access to." } } } @@ -21517,6 +22492,10 @@ "approver" ] } + }, + "view_token": { + "type": "string", + "description": "Share-read-link token for the parent flow. An authenticated workspace member can append it as a `view_token` query param on the run page to read a flow they don't otherwise have access to." } }, "required": [ @@ -32737,6 +33716,241 @@ } } }, + "/w/{workspace}/assets/graph": { + "get": { + "summary": "Get the workspace-wide asset <-> runnable graph", + "operationId": "getAssetsGraph", + "tags": [ + "asset" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "asset_kinds", + "in": "query", + "description": "Filter by asset kinds (comma-separated list)", + "schema": { + "type": "string" + } + }, + { + "name": "folder", + "in": "query", + "description": "Scope the graph to runnables in a single folder", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "asset graph nodes, lineage edges and trigger edges", + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "assets", + "runnables", + "edges", + "triggers" + ], + "properties": { + "assets": { + "type": "array", + "items": { + "type": "object", + "required": [ + "kind", + "path" + ], + "properties": { + "kind": { + "$ref": "#/components/schemas/AssetKind" + }, + "path": { + "type": "string" + } + } + } + }, + "runnables": { + "type": "array", + "items": { + "type": "object", + "required": [ + "path", + "usage_kind" + ], + "properties": { + "path": { + "type": "string" + }, + "usage_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "in_pipeline": { + "type": "boolean", + "description": "True iff the script is a pipeline member (deployed with `// pipeline`). Omitted when false." + } + } + } + }, + "edges": { + "type": "array", + "items": { + "type": "object", + "required": [ + "runnable_path", + "runnable_kind", + "asset_kind", + "asset_path" + ], + "properties": { + "runnable_path": { + "type": "string" + }, + "runnable_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "asset_kind": { + "$ref": "#/components/schemas/AssetKind" + }, + "asset_path": { + "type": "string" + }, + "access_type": { + "$ref": "#/components/schemas/AssetUsageAccessType" + } + } + } + }, + "triggers": { + "type": "array", + "items": { + "oneOf": [ + { + "type": "object", + "description": "Asset trigger edge (`// on `)", + "required": [ + "trigger_kind", + "asset_kind", + "asset_path", + "runnable_kind", + "runnable_path" + ], + "properties": { + "trigger_kind": { + "type": "string", + "enum": [ + "asset" + ] + }, + "asset_kind": { + "$ref": "#/components/schemas/AssetKind" + }, + "asset_path": { + "type": "string" + }, + "runnable_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "runnable_path": { + "type": "string" + } + } + }, + { + "type": "object", + "description": "Native trigger edge (schedule, email, kafka, ...). `path` is the trigger row's path.", + "required": [ + "trigger_kind", + "path", + "runnable_kind", + "runnable_path" + ], + "properties": { + "trigger_kind": { + "type": "string", + "enum": [ + "schedule", + "email", + "kafka", + "mqtt", + "nats", + "postgres", + "sqs", + "gcp" + ] + }, + "path": { + "type": "string" + }, + "runnable_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "runnable_path": { + "type": "string" + } + } + } + ] + } + } + } + } + } + } + } + } + } + }, + "/w/{workspace}/assets/pipelines": { + "get": { + "summary": "List folders that contain at least one pipeline-member script", + "operationId": "listPipelineFolders", + "tags": [ + "asset" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + } + ], + "responses": { + "200": { + "description": "folders containing pipeline scripts, with their script counts", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "required": [ + "folder", + "script_count" + ], + "properties": { + "folder": { + "type": "string", + "description": "The folder name (without the `f/` prefix)" + }, + "script_count": { + "type": "integer", + "format": "int64", + "description": "Number of pipeline-member scripts in the folder" + } + } + } + } + } + } + } + } + } + }, "/w/{workspace}/volumes/list": { "get": { "summary": "List all volumes in the workspace", @@ -33686,7 +34900,8 @@ "trigger_cli", "trigger_nextcloud", "trigger_google", - "trigger_github" + "trigger_github", + "data_pipeline" ] }, "OpenFlow": { @@ -36308,6 +37523,10 @@ "parent_hash": { "type": "string" }, + "auto_parent": { + "type": "boolean", + "description": "When true, the backend resolves the parent to the current deployed head for this path within the transaction (ignoring parent_hash), instead of failing with a \"lineage must be linear\" error when the supplied parent_hash is stale." + }, "summary": { "type": "string" }, @@ -36709,6 +37928,9 @@ "preprocessed": { "type": "boolean" }, + "is_retry": { + "type": "boolean" + }, "worker": { "type": "string" } @@ -36866,6 +38088,9 @@ "preprocessed": { "type": "boolean" }, + "is_retry": { + "type": "boolean" + }, "worker": { "type": "string" } @@ -37373,6 +38598,12 @@ "type": "string" } }, + "folders_read": { + "type": "array", + "items": { + "type": "string" + } + }, "folders_owners": { "type": "array", "items": { @@ -37400,6 +38631,7 @@ "operator", "disabled", "folders", + "folders_read", "folders_owners" ] }, @@ -39303,7 +40535,8 @@ "gcp", "azure", "google", - "github" + "github", + "asset" ] }, "TriggerMode": { @@ -43149,6 +44382,10 @@ }, "on_behalf_of_email": { "type": "string" + }, + "sandbox": { + "type": "boolean", + "description": "Publisher opt-in to app sandbox isolation (alpha). When true the app is isolated from each viewer's Windmill session. When false/absent the app runs same-origin with the viewer's full session (the default, pre-isolation behavior).\n" } } }, @@ -43430,6 +44667,44 @@ "version" ] }, + "EmbedTokenResponse": { + "type": "object", + "properties": { + "token": { + "type": "string", + "nullable": true, + "description": "Narrowly-scoped embed token for the iframe. Absent for fully anonymous or raw apps, which load without a scoped token." + }, + "expiration": { + "type": "string", + "format": "date-time", + "nullable": true, + "description": "Expiration of the embed token." + }, + "raw_app": { + "type": "boolean", + "description": "Raw apps render single-iframe and skip the opaque-viewer indirection and the embed token entirely." + }, + "sandbox": { + "type": "boolean", + "description": "Publisher opted this app into sandbox isolation. When false the viewer runs the app same-origin with its full session." + }, + "app_path": { + "type": "string", + "nullable": true, + "description": "The resolved app path; the embedder uses it to scope the app's backing localStorage per app." + }, + "workspace_id": { + "type": "string", + "nullable": true, + "description": "The resolved workspace; pairs with app_path so apps at the same path in different workspaces don't share a localStorage store." + } + }, + "required": [ + "raw_app", + "sandbox" + ] + }, "FlowVersion": { "type": "object", "properties": { diff --git a/backend/windmill-api/openapi-deref.yaml b/backend/windmill-api/openapi-deref.yaml index e45131d81b..e17c38efaf 100644 --- a/backend/windmill-api/openapi-deref.yaml +++ b/backend/windmill-api/openapi-deref.yaml @@ -1,6 +1,6 @@ openapi: 3.0.3 info: - version: 1.728.0 + version: 1.740.0 title: Windmill API contact: name: Windmill Team @@ -146,18 +146,18 @@ paths: checks: type: object description: Detailed health checks - required: &ref_352 + required: &ref_354 - database - readiness - properties: &ref_353 + properties: &ref_355 database: type: object description: Database health status - required: &ref_354 + required: &ref_356 - healthy - latency_ms - pool - properties: &ref_355 + properties: &ref_357 healthy: type: boolean description: Whether the database is reachable @@ -168,11 +168,11 @@ paths: pool: type: object description: Database connection pool statistics - required: &ref_356 + required: &ref_358 - size - idle - max_connections - properties: &ref_357 + properties: &ref_359 size: type: integer description: Current number of connections in the pool @@ -186,13 +186,13 @@ paths: description: Workers health status nullable: true type: object - required: &ref_358 + required: &ref_360 - healthy - active_count - worker_groups - min_version - versions - properties: &ref_359 + properties: &ref_361 healthy: type: boolean description: Whether any workers are active @@ -219,10 +219,10 @@ paths: description: Job queue status nullable: true type: object - required: &ref_360 + required: &ref_362 - pending_jobs - running_jobs - properties: &ref_361 + properties: &ref_363 pending_jobs: type: integer format: int64 @@ -234,9 +234,9 @@ paths: readiness: type: object description: Server readiness status - required: &ref_362 + required: &ref_364 - healthy - properties: &ref_363 + properties: &ref_365 healthy: type: boolean description: Whether the server is ready to accept requests @@ -275,40 +275,107 @@ paths: text/plain: schema: type: string - /inkeep: - post: - summary: query Windmill AI documentation assistant (EE only) - operationId: queryDocumentation + /docs/search: + get: + summary: >- + Full-text search across the entire Windmill documentation. Provide one + or more keywords; returns the most relevant docs pages, each with its + Source URL and short matching snippets. Use this FIRST to find relevant + pages by their content (a flag, function, error message, config key or + concept). If the snippets answer the question, answer directly; + otherwise call readDocsPage with a returned Source URL to read more. + operationId: searchDocs x-mcp-tool: true tags: - documentation - requestBody: - description: query to send to the AI documentation assistant - required: true - content: - application/json: - schema: - type: object - properties: - query: - type: string - description: The documentation query to send to the AI assistant - required: - - query + parameters: + - name: query + description: >- + Keywords to search for in the documentation body, e.g. "chromium + worker tag" or "retry exponential backoff". Fewer, more distinctive + words match better. + in: query + required: true + schema: + type: string responses: '200': - description: AI documentation assistant response + description: matching documentation pages content: application/json: schema: type: object - description: Response from Inkeep service - '403': - description: Enterprise Edition required + properties: + text: + type: string + description: Model-ready rendering of the results + results: + type: array + items: + type: object + properties: + url: + type: string + title: + type: string + score: + type: integer + snippets: + type: array + items: + type: string + required: + - url + - title + - score + - snippets + required: + - text + - results + /docs/page: + get: + summary: >- + Fetch the markdown of a single Windmill documentation page. Provide the + `url` of a page found via searchDocs (its Source URL). If the page is + large, this returns its list of section headings instead of the full + content; call again with the `section` argument set to one of those + headings to read that section. + operationId: readDocsPage + x-mcp-tool: true + tags: + - documentation + parameters: + - name: url + description: >- + The docs page to read, as a Source URL returned by searchDocs (e.g. + https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. + /docs/core_concepts/jobs) is also accepted. + in: query + required: true + schema: + type: string + - name: section + description: >- + Optional. A heading title from the page outline to read just that + section instead of the full page. + in: query + schema: + type: string + responses: + '200': + description: documentation page content content: - text/plain: + application/json: schema: - type: string + type: object + properties: + text: + type: string + source_url: + type: string + required: + - text + - source_url /openapi.yaml: get: summary: get openapi yaml spec @@ -488,24 +555,24 @@ paths: - name: before description: filter on started before (inclusive) timestamp in: query - schema: &ref_304 + schema: &ref_306 type: string format: date-time - name: after description: filter on created after (exclusive) timestamp in: query - schema: &ref_305 + schema: &ref_307 type: string format: date-time - name: username description: filter on exact username of user in: query - schema: &ref_313 + schema: &ref_315 type: string - name: operation description: filter on exact or prefix name of operation in: query - schema: &ref_314 + schema: &ref_316 type: string - name: operations in: query @@ -520,12 +587,12 @@ paths: - name: resource description: filter on exact or prefix name of resource in: query - schema: &ref_315 + schema: &ref_317 type: string - name: action_kind description: filter on type of operation in: query - schema: &ref_316 + schema: &ref_318 type: string enum: - Create @@ -562,12 +629,12 @@ paths: application/json: schema: type: object - properties: &ref_403 + properties: &ref_405 email: type: string password: type: string - required: &ref_404 + required: &ref_406 - email - password responses: @@ -749,6 +816,10 @@ paths: type: array items: type: string + folders_read: + type: array + items: + type: string folders_owners: type: array items: @@ -757,7 +828,7 @@ paths: nullable: true allOf: - type: object - properties: &ref_400 + properties: &ref_402 source: type: string enum: @@ -775,7 +846,7 @@ paths: description: >- The instance group name (when source is 'instance_group') - required: &ref_401 + required: &ref_403 - source is_service_account: type: boolean @@ -788,6 +859,7 @@ paths: - operator - disabled - folders + - folders_read - folders_owners /w/{workspace}/users/update/{username}: post: @@ -813,7 +885,7 @@ paths: application/json: schema: type: object - properties: &ref_405 + properties: &ref_407 is_admin: type: boolean operator: @@ -1187,7 +1259,7 @@ paths: type: array items: type: object - properties: &ref_419 + properties: &ref_421 jwt_hash: type: integer format: int64 @@ -1210,7 +1282,7 @@ paths: last_used_at: type: string format: date-time - required: &ref_420 + required: &ref_422 - jwt_hash - email - username @@ -1342,7 +1414,7 @@ paths: type: array items: type: object - properties: &ref_406 + properties: &ref_408 label: type: string scopes: @@ -1351,7 +1423,7 @@ paths: type: string expiration: type: string - required: &ref_407 + required: &ref_409 - label - scopes description: Tokens owned by this user (will be deleted) @@ -1395,7 +1467,7 @@ paths: application/json: schema: type: object - properties: &ref_408 + properties: &ref_410 reassign_to: type: string description: 'Target for reassignment: ''u/{username}'' or ''f/{folder}''' @@ -1409,7 +1481,7 @@ paths: type: boolean default: true description: Whether to also remove the user from the workspace - required: &ref_409 + required: &ref_411 - reassign_to responses: '200': @@ -1428,7 +1500,7 @@ paths: on success. summary: type: object - properties: &ref_410 + properties: &ref_412 scripts_reassigned: type: integer flows_reassigned: @@ -1445,7 +1517,7 @@ paths: type: integer drafts_deleted: type: integer - required: &ref_411 + required: &ref_413 - scripts_reassigned - flows_reassigned - apps_reassigned @@ -1475,12 +1547,12 @@ paths: application/json: schema: type: object - properties: &ref_412 + properties: &ref_414 workspaces: type: array items: type: object - properties: &ref_414 + properties: &ref_416 workspace_id: type: string username: @@ -1489,11 +1561,11 @@ paths: type: object properties: *ref_12 required: *ref_13 - required: &ref_415 + required: &ref_417 - workspace_id - username - preview - required: &ref_413 + required: &ref_415 - workspaces /users/offboard/{email}: post: @@ -1515,12 +1587,12 @@ paths: application/json: schema: type: object - properties: &ref_416 + properties: &ref_418 reassignments: type: object additionalProperties: type: object - properties: &ref_417 + properties: &ref_419 reassign_to: type: string description: 'Target: ''u/{username}'' or ''f/{folder}''' @@ -1529,7 +1601,7 @@ paths: description: >- Required when reassign_to is a folder. Username to use as permissioned_as. - required: &ref_418 + required: &ref_420 - reassign_to description: Map of workspace_id to reassignment config delete_user: @@ -1589,7 +1661,7 @@ paths: application/json: schema: type: array - items: &ref_563 + items: &ref_565 type: object properties: workspace_id: @@ -1699,7 +1771,7 @@ paths: application/json: schema: type: object - properties: &ref_504 + properties: &ref_506 email: type: string workspaces: @@ -1774,9 +1846,43 @@ paths: - username - color - disabled - required: &ref_505 + required: &ref_507 - email - workspaces + /workspaces/session_workspace_status: + post: + summary: >- + get the lifecycle status of workspaces referenced by client-side + sessions + operationId: getSessionWorkspaceStatus + tags: + - workspace + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + workspace_ids: + type: array + items: + type: string + required: + - workspace_ids + responses: + '200': + description: map of workspace id to status (active, archived, or deleted) + content: + application/json: + schema: + type: object + additionalProperties: + type: string + enum: + - active + - archived + - deleted /w/{workspace}/workspaces/get_as_superadmin: get: summary: get workspace as super admin (require to be super admin) @@ -1836,7 +1942,7 @@ paths: application/json: schema: type: object - properties: &ref_506 + properties: &ref_508 id: type: string name: @@ -1845,7 +1951,7 @@ paths: type: string color: type: string - required: &ref_507 + required: &ref_509 - id - name responses: @@ -2021,7 +2127,7 @@ paths: properties: &ref_24 logs: type: object - properties: &ref_472 + properties: &ref_474 super_admin: type: string enum: &ref_21 @@ -2548,6 +2654,9 @@ paths: s3_deleted: type: integer format: int64 + s3_not_found: + type: integer + format: int64 orphans_scanned: type: integer format: int64 @@ -2618,6 +2727,90 @@ paths: - bootstrapping - last_run_exported - updated_at + /settings/audit_logs_s3_backfill: + post: + summary: start an opt-in historical backfill of audit logs to object storage + operationId: runAuditLogsS3Backfill + tags: + - setting + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + from: + type: string + format: date-time + description: inclusive lower bound of the window to export + to: + type: string + format: date-time + description: exclusive upper bound of the window to export + required: + - from + - to + responses: + '202': + description: backfill started + /settings/audit_logs_s3_backfill_status: + get: + summary: get status of the audit-log object-store historical backfill + operationId: getAuditLogsS3BackfillStatus + tags: + - setting + responses: + '200': + description: current backfill status (null if never run) + content: + application/json: + schema: + nullable: true + type: object + properties: + running: + type: boolean + started_at: + type: string + format: date-time + finished_at: + type: string + format: date-time + nullable: true + phase: + type: string + from: + type: string + format: date-time + to: + type: string + format: date-time + rows_written: + type: integer + format: int64 + objects_written: + type: integer + format: int64 + last_ts: + type: string + format: date-time + nullable: true + errors: + type: integer + format: int64 + last_error: + type: string + nullable: true + required: + - running + - started_at + - phase + - from + - to + - rows_written + - objects_written + - errors /settings/send_stats: post: summary: send stats @@ -2828,11 +3021,11 @@ paths: type: array items: type: object - properties: &ref_546 + properties: &ref_548 name: type: string value: {} - required: &ref_547 + required: &ref_549 - name - value /settings/instance_config: @@ -2930,9 +3123,9 @@ paths: application/json: schema: type: object - required: &ref_375 + required: &ref_377 - keys - properties: &ref_376 + properties: &ref_378 keys: type: array items: @@ -3049,11 +3242,11 @@ paths: type: array items: type: object - required: &ref_373 + required: &ref_375 - workspace_id - path - error - properties: &ref_374 + properties: &ref_376 workspace_id: type: string description: Workspace ID where the secret is located @@ -4348,13 +4541,13 @@ paths: application/json: schema: type: object - required: &ref_555 + required: &ref_557 - all_ahead_items_visible - all_behind_items_visible - skipped_comparison - diffs - summary - properties: &ref_556 + properties: &ref_558 all_ahead_items_visible: type: boolean description: >- @@ -4375,7 +4568,7 @@ paths: description: List of differences found between workspaces items: type: object - required: &ref_557 + required: &ref_559 - kind - path - ahead @@ -4383,7 +4576,7 @@ paths: - has_changes - exists_in_source - exists_in_fork - properties: &ref_558 + properties: &ref_560 kind: type: string enum: @@ -4428,7 +4621,7 @@ paths: summary: description: Summary statistics of the comparison type: object - required: &ref_559 + required: &ref_561 - total_diffs - total_ahead - total_behind @@ -4442,7 +4635,7 @@ paths: - schedules_changed - triggers_changed - conflicts - properties: &ref_560 + properties: &ref_562 total_diffs: type: integer description: Total number of items with differences @@ -4639,12 +4832,12 @@ paths: type: array items: type: object - properties: &ref_532 + properties: &ref_534 pattern: type: string allow: type: string - required: &ref_533 + required: &ref_535 - pattern - allow secondary_storage: @@ -4777,7 +4970,7 @@ paths: auto_invite: type: object description: Configuration for auto-inviting users to the workspace - properties: &ref_364 + properties: &ref_366 enabled: type: boolean default: false @@ -4818,7 +5011,7 @@ paths: type: object additionalProperties: type: object - properties: &ref_383 + properties: &ref_385 resource_path: type: string models: @@ -4827,7 +5020,7 @@ paths: type: string web_search_enabled: type: boolean - required: &ref_384 + required: &ref_386 - resource_path - models default_model: @@ -4873,7 +5066,7 @@ paths: error_handler: type: object description: Configuration for the workspace error handler - properties: &ref_365 + properties: &ref_367 path: type: string description: Path to the error handler script or flow @@ -4890,7 +5083,7 @@ paths: success_handler: type: object description: Configuration for the workspace success handler - properties: &ref_366 + properties: &ref_368 path: type: string description: Path to the success handler script or flow @@ -5194,7 +5387,7 @@ paths: type: array items: type: object - properties: &ref_509 + properties: &ref_511 importer_path: type: string importer_kind: @@ -5208,7 +5401,7 @@ paths: items: type: string nullable: true - required: &ref_510 + required: &ref_512 - importer_path - importer_kind /w/{workspace}/workspaces/get_imports/{importer_path}: @@ -5266,13 +5459,13 @@ paths: type: array items: type: object - properties: &ref_511 + properties: &ref_513 imported_path: type: string count: type: integer format: int64 - required: &ref_512 + required: &ref_514 - imported_path - count /w/{workspace}/workspaces/get_dependency_map: @@ -5295,7 +5488,7 @@ paths: type: array items: type: object - properties: &ref_508 + properties: &ref_510 workspace_id: type: string nullable: true @@ -5827,7 +6020,7 @@ paths: type: array items: type: object - properties: &ref_385 + properties: &ref_387 provider: type: string enum: *ref_51 @@ -5835,7 +6028,7 @@ paths: type: array items: type: string - required: &ref_386 + required: &ref_388 - provider - models default_model: @@ -5926,10 +6119,10 @@ paths: Request body for editing the workspace error handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: &ref_367 + oneOf: &ref_369 - type: object description: New grouped format for editing error handler - properties: &ref_368 + properties: &ref_370 path: type: string description: Path to the error handler script or flow @@ -5947,7 +6140,7 @@ paths: description: >- Legacy flat format for editing error handler (deprecated, use new format) - properties: &ref_369 + properties: &ref_371 error_handler: type: string description: Path to the error handler script or flow @@ -5986,10 +6179,10 @@ paths: Request body for editing the workspace success handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: &ref_370 + oneOf: &ref_372 - type: object description: New grouped format for editing success handler - properties: &ref_371 + properties: &ref_373 path: type: string description: Path to the success handler script or flow @@ -6001,7 +6194,7 @@ paths: description: >- Legacy flat format for editing success handler (deprecated, use new format) - properties: &ref_372 + properties: &ref_374 success_handler: type: string description: Path to the success handler script or flow @@ -6118,10 +6311,10 @@ paths: type: array items: type: object - required: &ref_524 + required: &ref_526 - datatable_name - schemas - properties: &ref_525 + properties: &ref_527 datatable_name: type: string schemas: @@ -6163,10 +6356,10 @@ paths: type: array items: type: object - required: &ref_526 + required: &ref_528 - datatable_name - schemas - properties: &ref_527 + properties: &ref_529 datatable_name: type: string schemas: @@ -6211,12 +6404,12 @@ paths: application/json: schema: type: object - required: &ref_528 + required: &ref_530 - datatable_name - schema_name - table_name - columns - properties: &ref_529 + properties: &ref_531 datatable_name: type: string schema_name: @@ -6977,7 +7170,7 @@ paths: type: array items: type: object - properties: &ref_402 + properties: &ref_404 email: type: string executions: @@ -7040,7 +7233,7 @@ paths: type: array items: type: object - properties: &ref_520 + properties: &ref_522 name: type: string description: @@ -7050,7 +7243,7 @@ paths: type: array items: type: object - properties: &ref_518 + properties: &ref_520 value: type: string label: @@ -7060,11 +7253,11 @@ paths: nullable: true requires_resource_path: type: boolean - required: &ref_519 + required: &ref_521 - value - label - requires_resource_path - required: &ref_521 + required: &ref_523 - name - scopes /users/tokens/create: @@ -7080,7 +7273,7 @@ paths: application/json: schema: type: object - properties: &ref_421 + properties: &ref_423 label: type: string expiration: @@ -7121,7 +7314,7 @@ paths: application/json: schema: type: object - properties: &ref_422 + properties: &ref_424 label: type: string expiration: @@ -7131,7 +7324,7 @@ paths: type: string workspace_id: type: string - required: &ref_423 + required: &ref_425 - impersonate_email responses: '201': @@ -7249,7 +7442,7 @@ paths: type: array items: type: object - properties: &ref_108 + properties: &ref_109 label: type: string expiration: @@ -7273,7 +7466,7 @@ paths: type: string read_only: type: boolean - required: &ref_109 + required: &ref_110 - token_prefix - created_at - last_used_at @@ -7329,7 +7522,7 @@ paths: application/json: schema: type: object - properties: &ref_426 + properties: &ref_428 path: type: string description: The path to the variable @@ -7358,7 +7551,7 @@ paths: type: string ws_specific: type: boolean - required: &ref_427 + required: &ref_429 - path - value - is_secret @@ -7480,7 +7673,7 @@ paths: application/json: schema: type: object - properties: &ref_428 + properties: &ref_430 path: type: string description: The path to the variable @@ -7853,7 +8046,7 @@ paths: type: array items: type: object - properties: &ref_424 + properties: &ref_426 name: type: string value: @@ -7862,7 +8055,7 @@ paths: type: string is_custom: type: boolean - required: &ref_425 + required: &ref_427 - name - value - description @@ -8049,12 +8242,12 @@ paths: description: >- A workspace protection rule defining restrictions and bypass permissions - required: &ref_566 + required: &ref_568 - name - rules - bypass_groups - bypass_users - properties: &ref_567 + properties: &ref_569 name: type: string description: Unique name for the protection rule @@ -8066,7 +8259,7 @@ paths: description: Configuration of protection restrictions items: &ref_64 type: string - enum: &ref_568 + enum: &ref_570 - DisableDirectDeployment - DisableWorkspaceForking - RestrictDeployToDeployers @@ -8227,11 +8420,11 @@ paths: type: array items: type: object - required: &ref_569 + required: &ref_571 - username - email - is_admin - properties: &ref_570 + properties: &ref_572 username: type: string email: @@ -8286,10 +8479,10 @@ paths: type: array items: type: object - required: &ref_571 + required: &ref_573 - username - email - properties: &ref_572 + properties: &ref_574 username: type: string email: @@ -8918,6 +9111,13 @@ paths: substituted into the fixed-host registry template server-side (client_credentials flow only). The token URL is never caller-supplied. + cc_token_url: + type: string + description: >- + Bring-your-own token endpoint override (client_credentials + flow only). Only honored together with + cc_client_id/cc_client_secret and mutually exclusive with + cc_instance; ignored/rejected on the shared-instance path. mcp_server_url: type: string description: MCP server URL for MCP OAuth token refresh @@ -8985,6 +9185,13 @@ paths: client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side. The token URL is never caller-supplied. + cc_token_url: + type: string + description: >- + Bring-your-own token endpoint override. Only honored + together with cc_client_id/cc_client_secret and mutually + exclusive with cc_instance; rejected on the shared-instance + path. responses: '200': description: OAuth token response @@ -9241,7 +9448,7 @@ paths: application/json: schema: type: object - properties: &ref_433 + properties: &ref_435 path: type: string description: The path to the resource @@ -9258,7 +9465,7 @@ paths: type: string ws_specific: type: boolean - required: &ref_434 + required: &ref_436 - path - value - resource_type @@ -9349,7 +9556,7 @@ paths: application/json: schema: type: object - properties: &ref_435 + properties: &ref_437 path: type: string description: The path to the resource @@ -9802,7 +10009,7 @@ paths: - name: name in: path required: true - schema: &ref_278 + schema: &ref_280 type: string responses: '200': @@ -9935,7 +10142,7 @@ paths: application/json: schema: type: object - properties: &ref_436 + properties: &ref_438 schema: {} description: type: string @@ -10312,7 +10519,7 @@ paths: description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: &ref_124 + properties: &ref_125 summary: type: string description: Short description of what this flow does @@ -10324,7 +10531,7 @@ paths: description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: &ref_611 + properties: &ref_613 modules: type: array description: >- @@ -10356,7 +10563,7 @@ paths: in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms - properties: &ref_328 + properties: &ref_330 input_transforms: type: object description: >- @@ -10377,7 +10584,7 @@ paths: step. Use for hardcoded values or resource references like '$res:path/to/resource' - properties: &ref_140 + properties: &ref_142 value: description: >- The static value. For resources, use @@ -10386,7 +10593,7 @@ paths: type: string enum: - static - required: &ref_141 + required: &ref_143 - type - type: object description: >- @@ -10534,7 +10741,7 @@ paths: - r - w - rw - required: &ref_329 + required: &ref_331 - type - content - language @@ -10544,7 +10751,7 @@ paths: Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code - properties: &ref_330 + properties: &ref_332 input_transforms: type: object description: >- @@ -10584,7 +10791,7 @@ paths: description: >- If true, this script is a trigger that can start the flow - required: &ref_331 + required: &ref_333 - type - path - input_transforms @@ -10593,7 +10800,7 @@ paths: Reference to an existing flow by path. Use this to call another flow as a subflow - properties: &ref_332 + properties: &ref_334 input_transforms: type: object description: >- @@ -10618,7 +10825,7 @@ paths: type: string enum: - flow - required: &ref_333 + required: &ref_335 - type - path - input_transforms @@ -10631,7 +10838,7 @@ paths: 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations - properties: &ref_334 + properties: &ref_336 modules: type: array description: >- @@ -10680,7 +10887,7 @@ paths: discriminator: *ref_86 squash: type: boolean - required: &ref_335 + required: &ref_337 - modules - iterator - skip_failures @@ -10692,7 +10899,7 @@ paths: condition after each iteration. Use stop_after_if on modules to control loop termination - properties: &ref_336 + properties: &ref_338 modules: type: array description: >- @@ -10730,7 +10937,7 @@ paths: discriminator: *ref_86 squash: type: boolean - required: &ref_337 + required: &ref_339 - modules - skip_failures - type @@ -10742,7 +10949,7 @@ paths: one with a true expression runs. If no branches match, the default branch executes - properties: &ref_338 + properties: &ref_340 branches: type: array description: >- @@ -10794,7 +11001,7 @@ paths: type: string enum: - branchone - required: &ref_339 + required: &ref_341 - branches - default - type @@ -10805,7 +11012,7 @@ paths: BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently - properties: &ref_340 + properties: &ref_342 branches: type: array description: >- @@ -10846,7 +11053,7 @@ paths: If true, all branches execute concurrently. If false, they execute sequentially - required: &ref_341 + required: &ref_343 - branches - type - type: object @@ -10854,7 +11061,7 @@ paths: Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder - properties: &ref_342 + properties: &ref_344 type: type: string enum: @@ -10864,7 +11071,7 @@ paths: description: >- If true, marks this as a flow identity (special handling) - required: &ref_343 + required: &ref_345 - type - type: object description: >- @@ -10872,7 +11079,7 @@ paths: accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task - properties: &ref_344 + properties: &ref_346 input_transforms: type: object description: >- @@ -10884,22 +11091,22 @@ paths: Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined - oneOf: &ref_346 + oneOf: &ref_348 - type: object description: >- Static provider configuration passed directly to the AI agent - properties: &ref_596 + properties: &ref_598 value: type: object description: >- Complete AI provider configuration with resource reference and model selection - properties: &ref_594 + properties: &ref_596 kind: type: string description: Supported AI provider types - enum: &ref_321 + enum: &ref_323 - openai - azure_openai - anthropic @@ -10922,7 +11129,7 @@ paths: description: >- Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro') - required: &ref_595 + required: &ref_597 - kind - resource - model @@ -10930,7 +11137,7 @@ paths: type: string enum: - static - required: &ref_597 + required: &ref_599 - type - value - type: object @@ -10950,7 +11157,7 @@ paths: satisfy the parameter. properties: *ref_91 required: *ref_92 - discriminator: &ref_347 + discriminator: &ref_349 propertyName: type mapping: static: >- @@ -11020,27 +11227,27 @@ paths: Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined - oneOf: &ref_348 + oneOf: &ref_350 - type: object description: >- Static memory configuration passed directly to the AI agent - properties: &ref_602 + properties: &ref_604 value: description: Conversation memory configuration - oneOf: &ref_600 + oneOf: &ref_602 - type: object description: No conversation memory/context - properties: &ref_322 + properties: &ref_324 kind: type: string enum: - 'off' - required: &ref_323 + required: &ref_325 - kind - type: object description: Automatic context management - properties: &ref_324 + properties: &ref_326 kind: type: string enum: @@ -11055,11 +11262,11 @@ paths: description: >- Identifier for persistent memory across agent invocations - required: &ref_325 + required: &ref_327 - kind - type: object description: Explicit message history - properties: &ref_326 + properties: &ref_328 kind: type: string enum: @@ -11069,7 +11276,7 @@ paths: items: type: object description: A single message in conversation history - properties: &ref_598 + properties: &ref_600 role: type: string enum: @@ -11078,13 +11285,13 @@ paths: - system content: type: string - required: &ref_599 + required: &ref_601 - role - content - required: &ref_327 + required: &ref_329 - kind - messages - discriminator: &ref_601 + discriminator: &ref_603 propertyName: kind mapping: 'off': '#/components/schemas/MemoryOff' @@ -11094,7 +11301,7 @@ paths: type: string enum: - static - required: &ref_603 + required: &ref_605 - type - value - type: object @@ -11114,7 +11321,7 @@ paths: satisfy the parameter. properties: *ref_91 required: *ref_92 - discriminator: &ref_349 + discriminator: &ref_351 propertyName: type mapping: static: >- @@ -11227,7 +11434,7 @@ paths: A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool - properties: &ref_350 + properties: &ref_352 id: type: string description: >- @@ -11245,12 +11452,12 @@ paths: The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference - oneOf: &ref_609 + oneOf: &ref_611 - description: >- A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module - allOf: &ref_604 + allOf: &ref_606 - type: object properties: tool_type: @@ -11283,7 +11490,7 @@ paths: Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers - properties: &ref_605 + properties: &ref_607 tool_type: type: string enum: @@ -11307,7 +11514,7 @@ paths: MCP server items: type: string - required: &ref_606 + required: &ref_608 - tool_type - resource_path - type: object @@ -11315,20 +11522,20 @@ paths: A tool implemented as a websearch tool. The AI can call this like any other websearch tool - properties: &ref_607 + properties: &ref_609 tool_type: type: string enum: - websearch - required: &ref_608 + required: &ref_610 - tool_type - discriminator: &ref_610 + discriminator: &ref_612 propertyName: tool_type mapping: flowmodule: '#/components/schemas/FlowModuleTool' mcp: '#/components/schemas/McpToolValue' websearch: '#/components/schemas/WebsearchToolValue' - required: &ref_351 + required: &ref_353 - id - value type: @@ -11354,7 +11561,7 @@ paths: description: >- If true, the agent can execute multiple tool calls in parallel - required: &ref_345 + required: &ref_347 - tools - type - input_transforms @@ -11527,7 +11734,7 @@ paths: Retry configuration for failed module executions type: object - properties: &ref_320 + properties: &ref_322 constant: type: object description: >- @@ -11568,14 +11775,14 @@ paths: description: >- Conditional retry based on error or result - properties: &ref_199 + properties: &ref_201 expr: type: string description: >- JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables - required: &ref_200 + required: &ref_202 - expr debouncing: description: >- @@ -11716,7 +11923,7 @@ paths: description: >- A sticky note attached to a flow for documentation and annotation - properties: &ref_149 + properties: &ref_151 id: type: string description: Unique identifier for the note @@ -11775,7 +11982,7 @@ paths: description: >- For group notes, the IDs of nodes contained within this group - required: &ref_150 + required: &ref_152 - id - text - color @@ -11795,7 +12002,7 @@ paths: collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id. - properties: &ref_151 + properties: &ref_153 summary: type: string description: Display name for this group @@ -11821,10 +12028,10 @@ paths: color: type: string description: Color for the group in the flow editor - required: &ref_152 + required: &ref_154 - start_id - end_id - required: &ref_612 + required: &ref_614 - modules schema: type: object @@ -11838,7 +12045,7 @@ paths: description: >- The flow will be run with the permissions of the user with this email. - required: &ref_125 + required: &ref_126 - summary - value /apps/hub/list: @@ -11951,7 +12158,7 @@ paths: - name: custom_path in: path required: true - schema: &ref_137 + schema: &ref_97 type: string responses: '200': @@ -11961,7 +12168,7 @@ paths: schema: allOf: - type: object - properties: &ref_132 + properties: &ref_133 id: type: integer workspace_id: @@ -11982,7 +12189,7 @@ paths: value: {} policy: type: object - properties: &ref_131 + properties: &ref_132 triggerables: type: object additionalProperties: @@ -12014,6 +12221,14 @@ paths: type: string on_behalf_of_email: type: string + sandbox: + type: boolean + description: > + Publisher opt-in to app sandbox isolation (alpha). + When true the app is isolated from each viewer's + Windmill session. When false/absent the app runs + same-origin with the viewer's full session (the + default, pre-isolation behavior). execution_mode: type: string enum: @@ -12035,7 +12250,7 @@ paths: items: type: string default: [] - required: &ref_133 + required: &ref_134 - id - workspace_id - path @@ -12052,6 +12267,64 @@ paths: properties: workspace_id: type: string + /apps_u/embed_token_by_custom_path/{custom_path}: + get: + summary: get app embed token by custom path + operationId: getAppEmbedTokenByCustomPath + tags: + - app + parameters: + - name: custom_path + in: path + required: true + schema: *ref_97 + responses: + '200': + description: embed token + content: + application/json: + schema: + type: object + properties: &ref_135 + token: + type: string + nullable: true + description: >- + Narrowly-scoped embed token for the iframe. Absent for + fully anonymous or raw apps, which load without a scoped + token. + expiration: + type: string + format: date-time + nullable: true + description: Expiration of the embed token. + raw_app: + type: boolean + description: >- + Raw apps render single-iframe and skip the opaque-viewer + indirection and the embed token entirely. + sandbox: + type: boolean + description: >- + Publisher opted this app into sandbox isolation. When + false the viewer runs the app same-origin with its full + session. + app_path: + type: string + nullable: true + description: >- + The resolved app path; the embedder uses it to scope the + app's backing localStorage per app. + workspace_id: + type: string + nullable: true + description: >- + The resolved workspace; pairs with app_path so apps at the + same path in different workspaces don't share a + localStorage store. + required: &ref_136 + - raw_app + - sandbox /scripts/hub/get/{path}: get: summary: get hub script content by path @@ -12062,7 +12335,7 @@ paths: - name: path in: path required: true - schema: &ref_97 + schema: &ref_98 type: string responses: '200': @@ -12081,7 +12354,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script details @@ -12112,7 +12385,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script pick recorded @@ -12175,7 +12448,7 @@ paths: type: number kind: type: string - enum: &ref_98 + enum: &ref_99 - script - failure - trigger @@ -12246,7 +12519,7 @@ paths: type: string kind: type: string - enum: *ref_98 + enum: *ref_99 score: type: number required: @@ -12308,7 +12581,7 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: &ref_122 + schema: &ref_123 type: boolean - name: created_by description: >- @@ -12316,7 +12589,7 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: &ref_123 + schema: &ref_124 type: string - name: path_start description: mask to filter matching starting path @@ -12458,7 +12731,7 @@ paths: items: allOf: - type: object - properties: &ref_105 + properties: &ref_106 workspace_id: type: string hash: @@ -12501,7 +12774,7 @@ paths: type: string language: type: string - enum: &ref_99 + enum: &ref_100 - python3 - deno - go @@ -12599,20 +12872,20 @@ paths: additionalProperties: type: object description: An additional module file associated with a script - properties: &ref_101 + properties: &ref_102 content: type: string description: The source code content of this module language: type: string - enum: *ref_99 + enum: *ref_100 lock: type: string nullable: true description: >- Lock file content for this module's dependencies - required: &ref_102 + required: &ref_103 - content - language labels: @@ -12627,7 +12900,7 @@ paths: description: > Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder. - required: &ref_106 + required: &ref_107 - hash - path - summary @@ -12708,6 +12981,14 @@ paths: in: path required: true schema: *ref_4 + - name: all_users + in: query + description: >- + List every draft in the workspace (all users), not just the current + user's own + legacy rows. Other users' rows come back with + `mine=false` (view-only). + schema: + type: boolean responses: '200': description: the user's drafts @@ -12726,7 +13007,7 @@ paths: Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. - enum: &ref_100 + enum: &ref_101 - script - flow - app @@ -12751,6 +13032,7 @@ paths: - trigger_nextcloud - trigger_google - trigger_github + - data_pipeline path: type: string summary: @@ -12779,12 +13061,43 @@ paths: created_at: type: string format: date-time + can_write: + type: boolean + description: >- + Whether the current user may deploy/discard this draft + (same check the deploy/discard endpoints enforce). + mine: + type: boolean + description: >- + The row belongs to the current user (own draft or the + legacy no-owner row) and is therefore actionable. Always + true in the default listing; with `all_users=true`, + other users' rows are false (view-only). + draft_users: + description: > + Draft authors at this (path, kind) — the legacy + NULL-email row surfaced as a null username. + + Populated only for the shared full-page-editor kinds + (script/flow/app/raw_app); omitted for + + drawer kinds, which keep their drafts private. Feeds the + Draft badge's owner-avatar circles. + type: array + items: + type: object + properties: + username: + type: string + nullable: true required: - kind - path - draft_only - legacy_draft - created_at + - can_write + - mine /w/{workspace}/drafts/get/{kind}/{path}: get: summary: >- @@ -12808,11 +13121,11 @@ paths: the Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. - enum: *ref_100 + enum: *ref_101 - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: username in: query required: false @@ -12838,6 +13151,48 @@ paths: - created_at '404': description: no draft for that owner at that path + /w/{workspace}/drafts/get_own/{kind}/{path}: + get: + summary: fetch the current user's own draft content at a path (any kind) + operationId: getOwnDraft + tags: + - draft + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: kind + in: path + required: true + schema: + type: string + description: > + Closed set of item kinds a user can autosave as a draft. Mirrors + the + + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 + - name: path + in: path + required: true + schema: *ref_98 + responses: + '200': + description: the user's draft content, or null when none exists + content: + application/json: + schema: + nullable: true + type: object + properties: + value: {} + created_at: + type: string + format: date-time + required: + - value + - created_at /w/{workspace}/drafts/update/{kind}/{path}: post: summary: upsert (or clear) the current user's draft at a path @@ -12859,11 +13214,11 @@ paths: the Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. - enum: *ref_100 + enum: *ref_101 - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: required: true content: @@ -12891,6 +13246,14 @@ paths: Delete-only. Target the legacy workspace-level row (email NULL) instead of the current user's row. Used to discard a legacy draft from the review page. + created_at: + type: string + format: date-time + description: >- + Upsert-only override for the stored creation timestamp. + Normal saves omit it (stamped server-side); the + localStorage→DB migration passes the draft's original write + time so migrated drafts keep their age. responses: '200': description: save result @@ -12910,6 +13273,57 @@ paths: required: - status - current_timestamp + /w/{workspace}/drafts/migrate_legacy/{kind}/{path}: + post: + summary: resolve a legacy (workspace-level) draft (admin only) + description: >- + Delete a legacy draft (email NULL) or assign it to the authed admin as a + per-user draft. Workspace admins / superadmins only. + operationId: migrateLegacyDraft + tags: + - draft + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: kind + in: path + required: true + schema: + type: string + description: > + Closed set of item kinds a user can autosave as a draft. Mirrors + the + + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 + - name: path + in: path + required: true + schema: *ref_98 + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + action: + type: string + enum: + - delete + - assign_to_self + description: delete the legacy draft, or take ownership of it. + required: + - action + responses: + '200': + description: migration result + content: + text/plain: + schema: + type: string /w/{workspace}/scripts/create: post: summary: create script @@ -12953,11 +13367,18 @@ paths: application/json: schema: type: object - properties: &ref_392 + properties: &ref_394 path: type: string parent_hash: type: string + auto_parent: + type: boolean + description: >- + When true, the backend resolves the parent to the current + deployed head for this path within the transaction (ignoring + parent_hash), instead of failing with a "lineage must be + linear" error when the supplied parent_hash is stale. summary: type: string description: @@ -12972,7 +13393,7 @@ paths: type: string language: type: string - enum: *ref_99 + enum: *ref_100 kind: type: string enum: @@ -13055,7 +13476,7 @@ paths: type: string kind: type: string - enum: &ref_306 + enum: &ref_308 - s3object - resource - ducklake @@ -13080,8 +13501,8 @@ paths: additionalProperties: type: object description: An additional module file associated with a script - properties: *ref_101 - required: *ref_102 + properties: *ref_102 + required: *ref_103 labels: type: array items: @@ -13091,7 +13512,7 @@ paths: description: >- When true (set by the CLI / git sync), deploying this script does not delete an existing user draft at the same path. - required: &ref_393 + required: &ref_395 - path - summary - content @@ -13117,7 +13538,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: Workspace error handler enabled required: true @@ -13224,19 +13645,19 @@ paths: application/json: schema: type: object - properties: &ref_390 + properties: &ref_392 workspace_id: type: string language: type: string - enum: *ref_99 + enum: *ref_100 name: type: string description: type: string content: type: string - required: &ref_391 + required: &ref_393 - workspace_id - language - content @@ -13263,7 +13684,7 @@ paths: required: true schema: type: string - enum: *ref_99 + enum: *ref_100 - name: name in: query required: false @@ -13291,7 +13712,7 @@ paths: required: true schema: type: string - enum: *ref_99 + enum: *ref_100 - name: name in: query required: false @@ -13323,7 +13744,7 @@ paths: type: array items: type: object - properties: &ref_103 + properties: &ref_104 id: type: integer archived: @@ -13336,13 +13757,13 @@ paths: type: string language: type: string - enum: *ref_99 + enum: *ref_100 workspace_id: type: string created_at: type: string format: date-time - required: &ref_104 + required: &ref_105 - workspace_id - language - created_at @@ -13365,7 +13786,7 @@ paths: required: true schema: type: string - enum: *ref_99 + enum: *ref_100 - name: name in: query required: false @@ -13378,8 +13799,8 @@ paths: application/json: schema: type: object - properties: *ref_103 - required: *ref_104 + properties: *ref_104 + required: *ref_105 /w/{workspace}/scripts/archive/p/{path}: post: summary: archive script by path @@ -13394,7 +13815,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script archived @@ -13416,7 +13837,7 @@ paths: - name: hash in: path required: true - schema: &ref_107 + schema: &ref_108 type: string responses: '200': @@ -13425,8 +13846,8 @@ paths: application/json: schema: type: object - properties: *ref_105 - required: *ref_106 + properties: *ref_106 + required: *ref_107 /w/{workspace}/scripts/delete/h/{hash}: post: summary: delete script by hash (erase content but keep hash, require admin) @@ -13442,7 +13863,7 @@ paths: - name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 responses: '200': description: script details @@ -13450,8 +13871,8 @@ paths: application/json: schema: type: object - properties: *ref_105 - required: *ref_106 + properties: *ref_106 + required: *ref_107 /w/{workspace}/scripts/delete/p/{path}: post: summary: delete script at a given path (require admin) @@ -13467,7 +13888,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: keep_captures description: keep captures in: query @@ -13529,7 +13950,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: with_starred_info in: query schema: @@ -13549,8 +13970,8 @@ paths: schema: allOf: - type: object - properties: *ref_105 - required: *ref_106 + properties: *ref_106 + required: *ref_107 - type: object description: > Overlay fields added to every "get by path" response that @@ -13600,7 +14021,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: triggers count @@ -13608,7 +14029,7 @@ paths: application/json: schema: type: object - properties: &ref_129 + properties: &ref_130 primary_schedule: type: object properties: @@ -13660,7 +14081,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: tokens list @@ -13670,8 +14091,8 @@ paths: type: array items: type: object - properties: *ref_108 - required: *ref_109 + properties: *ref_109 + required: *ref_110 /w/{workspace}/scripts/history/p/{path}: get: summary: get history of a script by path @@ -13686,7 +14107,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script history @@ -13696,12 +14117,12 @@ paths: type: array items: type: object - properties: &ref_110 + properties: &ref_111 script_hash: type: string deployment_msg: type: string - required: &ref_111 + required: &ref_112 - script_hash /w/{workspace}/scripts/list_paths_from_workspace_runnable/{path}: get: @@ -13717,7 +14138,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: list of script paths @@ -13739,7 +14160,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 tags: - script responses: @@ -13749,8 +14170,8 @@ paths: application/json: schema: type: object - properties: *ref_110 - required: *ref_111 + properties: *ref_111 + required: *ref_112 /w/{workspace}/scripts/history_update/h/{hash}/p/{path}: post: summary: update history of a script @@ -13765,11 +14186,11 @@ paths: - name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: Script deployment message required: true @@ -13857,7 +14278,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script content @@ -13881,12 +14302,12 @@ paths: - name: token in: path required: true - schema: &ref_310 + schema: &ref_312 type: string - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script content @@ -13908,7 +14329,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: does it exists @@ -13930,7 +14351,7 @@ paths: - name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 - name: with_starred_info in: query schema: @@ -13946,8 +14367,8 @@ paths: application/json: schema: type: object - properties: *ref_105 - required: *ref_106 + properties: *ref_106 + required: *ref_107 /w/{workspace}/scripts/raw/h/{path}: get: summary: raw script by hash @@ -13962,7 +14383,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: script content @@ -13984,7 +14405,7 @@ paths: - name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 responses: '200': description: script details @@ -14034,7 +14455,7 @@ paths: type: array items: type: object - properties: &ref_112 + properties: &ref_113 test_script_path: type: string job_id: @@ -14048,7 +14469,7 @@ paths: type: string format: date-time nullable: true - required: &ref_113 + required: &ref_114 - test_script_path /w/{workspace}/scripts/ci_test_results_batch: post: @@ -14097,8 +14518,8 @@ paths: type: array items: type: object - properties: *ref_112 - required: *ref_113 + properties: *ref_113 + required: *ref_114 /w/{workspace}/scripts/raw_temp/store: post: summary: store raw script content temporarily for CLI lock generation @@ -14165,7 +14586,7 @@ paths: type: string language: type: string - enum: *ref_99 + enum: *ref_100 name: description: named workspace dependency (null for default) type: string @@ -14262,7 +14683,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: scheduled_for description: when to schedule this job (leave empty for immediate run) in: query @@ -14284,20 +14705,20 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: &ref_114 + schema: &ref_115 type: string format: uuid - name: tag description: Override the tag to use in: query - schema: &ref_115 + schema: &ref_116 type: string - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: &ref_116 + schema: &ref_117 type: string - name: job_id description: >- @@ -14306,7 +14727,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: &ref_117 + schema: &ref_118 type: string format: uuid - name: invisible_to_owner @@ -14345,23 +14766,23 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14369,7 +14790,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14378,19 +14799,19 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: &ref_118 + schema: &ref_119 type: string - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: &ref_119 + schema: &ref_120 type: string - name: skip_preprocessor description: skip the preprocessor in: query - schema: &ref_120 + schema: &ref_121 type: boolean requestBody: description: script args @@ -14420,23 +14841,23 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14444,7 +14865,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14453,13 +14874,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14467,12 +14888,12 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: &ref_121 + schema: &ref_122 type: string - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 responses: '200': description: job result @@ -14493,7 +14914,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14502,13 +14923,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14516,11 +14937,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14568,13 +14989,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14582,11 +15003,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14633,13 +15054,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14647,7 +15068,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14655,11 +15076,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14686,7 +15107,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14695,13 +15116,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14709,11 +15130,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14755,7 +15176,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14764,13 +15185,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14778,7 +15199,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14786,11 +15207,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14836,13 +15257,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14850,11 +15271,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14908,13 +15329,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14922,7 +15343,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14930,11 +15351,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14968,23 +15389,23 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14992,7 +15413,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -15001,17 +15422,17 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -15047,23 +15468,23 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -15071,7 +15492,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -15080,13 +15501,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -15094,11 +15515,11 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -15133,17 +15554,17 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -15151,7 +15572,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -15160,17 +15581,17 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -15213,17 +15634,17 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -15231,7 +15652,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -15240,13 +15661,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -15254,11 +15675,11 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -15398,14 +15819,14 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: path_start description: mask to filter matching starting path in: query @@ -15480,15 +15901,15 @@ paths: type: array items: allOf: - - allOf: &ref_128 + - allOf: &ref_129 - type: object description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: *ref_124 - required: *ref_125 + properties: *ref_125 + required: *ref_126 - type: object - properties: &ref_514 + properties: &ref_516 workspace_id: type: string path: @@ -15502,7 +15923,7 @@ paths: type: boolean extra_perms: type: object - additionalProperties: &ref_513 + additionalProperties: &ref_515 type: boolean starred: type: boolean @@ -15535,7 +15956,7 @@ paths: Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder. - required: &ref_515 + required: &ref_517 - path - edited_by - edited_at @@ -15595,7 +16016,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 tags: - flow responses: @@ -15607,7 +16028,7 @@ paths: type: array items: type: object - properties: &ref_126 + properties: &ref_127 id: type: integer created_at: @@ -15615,7 +16036,7 @@ paths: format: date-time deployment_msg: type: string - required: &ref_127 + required: &ref_128 - id - created_at /w/{workspace}/flows/get_latest_version/{path}: @@ -15630,7 +16051,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 tags: - flow responses: @@ -15640,8 +16061,8 @@ paths: application/json: schema: type: object - properties: *ref_126 - required: *ref_127 + properties: *ref_127 + required: *ref_128 /w/{workspace}/flows/list_paths_from_workspace_runnable/{runnable_kind}/{path}: get: summary: list flow paths from workspace runnable @@ -15656,7 +16077,7 @@ paths: - name: runnable_kind in: path required: true - schema: &ref_136 + schema: &ref_139 type: string enum: - script @@ -15664,7 +16085,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: match_path_start in: query schema: @@ -15700,7 +16121,7 @@ paths: content: application/json: schema: - allOf: *ref_128 + allOf: *ref_129 /w/{workspace}/flows/history_update/v/{version}: post: summary: update flow history @@ -15751,7 +16172,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: with_starred_info in: query schema: @@ -15770,7 +16191,7 @@ paths: application/json: schema: allOf: - - allOf: *ref_128 + - allOf: *ref_129 - type: object description: > Overlay fields added to every "get by path" response that @@ -15820,7 +16241,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: flow status @@ -15848,7 +16269,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: triggers count @@ -15856,7 +16277,7 @@ paths: application/json: schema: type: object - properties: *ref_129 + properties: *ref_130 /w/{workspace}/flows/list_tokens/{path}: get: summary: get tokens with flow scope @@ -15871,7 +16292,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: tokens list @@ -15881,8 +16302,8 @@ paths: type: array items: type: object - properties: *ref_108 - required: *ref_109 + properties: *ref_109 + required: *ref_110 /w/{workspace}/flows/toggle_workspace_error_handler/{path}: post: summary: Toggle ON and OFF the workspace error handler for a given flow @@ -15897,7 +16318,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: Workspace error handler enabled required: true @@ -15929,7 +16350,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: flow details @@ -15967,13 +16388,13 @@ paths: application/json: schema: allOf: - - allOf: &ref_130 + - allOf: &ref_131 - type: object description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: *ref_124 - required: *ref_125 + properties: *ref_125 + required: *ref_126 - type: object properties: path: @@ -16047,7 +16468,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: Partially filled flow required: true @@ -16055,7 +16476,7 @@ paths: application/json: schema: allOf: - - allOf: *ref_130 + - allOf: *ref_131 - type: object properties: deployment_message: @@ -16087,7 +16508,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: archiveFlow required: true @@ -16120,7 +16541,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: keep_captures description: keep captures in: query @@ -16166,14 +16587,14 @@ paths: type: array items: type: object - required: &ref_377 + required: &ref_379 - id - workspace_id - flow_path - created_at - updated_at - created_by - properties: &ref_378 + properties: &ref_380 id: type: string format: uuid @@ -16266,14 +16687,14 @@ paths: type: array items: type: object - required: &ref_379 + required: &ref_381 - id - conversation_id - message_type - content - created_at - created_seq - properties: &ref_380 + properties: &ref_382 id: type: string format: uuid @@ -16379,14 +16800,14 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: path_start description: mask to filter matching starting path in: query @@ -16419,7 +16840,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: &ref_212 + schema: &ref_214 type: boolean responses: '200': @@ -16430,7 +16851,7 @@ paths: type: array items: type: object - properties: &ref_522 + properties: &ref_524 workspace_id: type: string path: @@ -16460,7 +16881,7 @@ paths: description: > Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder. - required: &ref_523 + required: &ref_525 - workspace_id - path - summary @@ -16600,6 +17021,141 @@ paths: text/plain: schema: type: string + /w/{workspace}/ai_skills/list: + get: + summary: list the workspace AI chat skills (name + description only) + operationId: listAiSkills + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + responses: + '200': + description: skill listing + content: + application/json: + schema: + type: array + items: + type: object + required: + - name + - description + properties: + name: + type: string + description: + type: string + /w/{workspace}/ai_skills/get/{name}: + get: + summary: get a workspace AI chat skill including its instructions + operationId: getAiSkill + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: skill + content: + application/json: + schema: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + description: + type: string + instructions: + type: string + /w/{workspace}/ai_skills/upload: + post: + summary: upsert workspace AI chat skills (admin only) + operationId: uploadAiSkills + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - skills + properties: + skills: + type: array + maxItems: 50 + items: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + minLength: 1 + maxLength: 64 + pattern: ^[a-z0-9-]+$ + description: + type: string + minLength: 1 + maxLength: 1024 + instructions: + type: string + minLength: 1 + maxLength: 65536 + responses: + '200': + description: uploaded + content: + text/plain: + schema: + type: string + /w/{workspace}/ai_skills/delete/{name}: + delete: + summary: delete a workspace AI chat skill (admin only) + operationId: deleteAiSkill + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: deleted + content: + text/plain: + schema: + type: string /w/{workspace}/apps/get_data/v/{secretWithExtension}: get: summary: get raw app data by @@ -16614,6 +17170,10 @@ paths: - name: secretWithExtension in: path required: true + description: >- + App version secret suffixed with the requested file type extension. + Supported extensions are `.js` (JavaScript bundle), `.css` + (stylesheet), and `.html` (sandboxed wrapper document). schema: type: string responses: @@ -16623,6 +17183,12 @@ paths: text/javascript: schema: type: string + text/css: + schema: + type: string + text/html: + schema: + type: string /w/{workspace}/apps/list_search: get: summary: list apps for search @@ -16672,14 +17238,14 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: path_start description: mask to filter matching starting path in: query @@ -16726,7 +17292,7 @@ paths: type: array items: type: object - properties: &ref_516 + properties: &ref_518 id: type: integer workspace_id: @@ -16813,7 +17379,7 @@ paths: description: > Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder. - required: &ref_517 + required: &ref_519 - id - workspace_id - path @@ -16858,7 +17424,7 @@ paths: type: string policy: type: object - properties: *ref_131 + properties: *ref_132 deployment_message: type: string custom_path: @@ -16919,7 +17485,7 @@ paths: type: string policy: type: object - properties: *ref_131 + properties: *ref_132 deployment_message: type: string custom_path: @@ -16993,7 +17559,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: with_starred_info in: query schema: @@ -17021,8 +17587,8 @@ paths: schema: allOf: - type: object - properties: *ref_132 - required: *ref_133 + properties: *ref_133 + required: *ref_134 - type: object description: > Overlay fields added to every "get by path" response that @@ -17058,6 +17624,30 @@ paths: "diff vs deployed" UI in that case. properties: *ref_78 required: *ref_79 + /w/{workspace}/apps/embed_token/p/{path}: + get: + summary: get app embed token by path + operationId: getAppEmbedTokenByPath + tags: + - app + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: path + in: path + required: true + schema: *ref_98 + responses: + '200': + description: embed token + content: + application/json: + schema: + type: object + properties: *ref_135 + required: *ref_136 /w/{workspace}/apps/get/lite/{path}: get: summary: get app lite by path @@ -17072,7 +17662,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: app lite details @@ -17080,8 +17670,8 @@ paths: application/json: schema: type: object - properties: *ref_132 - required: *ref_133 + properties: *ref_133 + required: *ref_134 /w/{workspace}/apps/history/p/{path}: get: summary: get app history by path @@ -17096,7 +17686,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: app history @@ -17106,12 +17696,12 @@ paths: type: array items: type: object - properties: &ref_134 + properties: &ref_137 version: type: integer deployment_msg: type: string - required: &ref_135 + required: &ref_138 - version /w/{workspace}/apps/get_latest_version/{path}: get: @@ -17125,7 +17715,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 tags: - app responses: @@ -17135,8 +17725,8 @@ paths: application/json: schema: type: object - properties: *ref_134 - required: *ref_135 + properties: *ref_137 + required: *ref_138 /w/{workspace}/apps/list_paths_from_workspace_runnable/{runnable_kind}/{path}: get: summary: list app paths from workspace runnable @@ -17151,11 +17741,11 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_136 + schema: *ref_139 - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 responses: '200': description: list of app paths @@ -17183,7 +17773,7 @@ paths: - name: version in: path required: true - schema: &ref_311 + schema: &ref_313 type: integer requestBody: description: App deployment message @@ -17224,8 +17814,33 @@ paths: application/json: schema: type: object - properties: *ref_132 - required: *ref_133 + properties: *ref_133 + required: *ref_134 + /w/{workspace}/apps_u/embed_token/{secret}: + get: + summary: get app embed token by secret + operationId: getAppEmbedTokenBySecret + tags: + - app + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: secret + in: path + required: true + schema: + type: string + responses: + '200': + description: embed token + content: + application/json: + schema: + type: object + properties: *ref_135 + required: *ref_136 /w/{workspace}/apps_u/public_resource/{path}: get: summary: get public resource @@ -17313,8 +17928,8 @@ paths: application/json: schema: type: object - properties: *ref_132 - required: *ref_133 + properties: *ref_133 + required: *ref_134 /w/{workspace}/apps/delete/{path}: delete: summary: delete app @@ -17361,7 +17976,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: update app required: true @@ -17377,7 +17992,7 @@ paths: value: {} policy: type: object - properties: *ref_131 + properties: *ref_132 deployment_message: type: string custom_path: @@ -17418,7 +18033,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: update app required: true @@ -17437,7 +18052,7 @@ paths: value: {} policy: type: object - properties: *ref_131 + properties: *ref_132 deployment_message: type: string custom_path: @@ -17484,7 +18099,7 @@ paths: - name: custom_path in: path required: true - schema: *ref_137 + schema: *ref_97 responses: '200': description: custom path exists @@ -17515,7 +18130,7 @@ paths: type: array items: type: object - properties: &ref_138 + properties: &ref_140 s3: type: string filename: @@ -17524,7 +18139,7 @@ paths: type: string presigned: type: string - required: &ref_139 + required: &ref_141 - s3 required: - s3_objects @@ -17537,8 +18152,8 @@ paths: type: array items: type: object - properties: *ref_138 - required: *ref_139 + properties: *ref_140 + required: *ref_141 /w/{workspace}/apps_u/execute_component/{path}: post: summary: executeComponent @@ -17553,7 +18168,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: update app required: true @@ -17738,7 +18353,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 - name: scheduled_for description: when to schedule this job (leave empty for immediate run) in: query @@ -17753,17 +18368,17 @@ paths: - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17771,7 +18386,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -17780,7 +18395,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the flow owner (default false) in: query @@ -17841,17 +18456,17 @@ paths: - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17859,7 +18474,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -17868,7 +18483,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the flow owner (default false) in: query @@ -17936,14 +18551,14 @@ paths: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: &ref_142 + oneOf: &ref_144 - type: object description: >- Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource' - properties: *ref_140 - required: *ref_141 + properties: *ref_142 + required: *ref_143 - type: object description: >- JavaScript expression evaluated at runtime. Can @@ -17961,7 +18576,7 @@ paths: parameter. properties: *ref_91 required: *ref_92 - discriminator: &ref_143 + discriminator: &ref_145 propertyName: type mapping: static: '#/components/schemas/schemas-StaticTransform' @@ -17981,8 +18596,8 @@ paths: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_142 - discriminator: *ref_143 + oneOf: *ref_144 + discriminator: *ref_145 use_latest_version: type: boolean responses: @@ -18008,7 +18623,7 @@ paths: - name: id in: path required: true - schema: &ref_176 + schema: &ref_178 type: string format: uuid - name: scheduled_for @@ -18027,11 +18642,11 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -18039,7 +18654,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -18048,7 +18663,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the flow owner (default false) in: query @@ -18122,7 +18737,7 @@ paths: - name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 - name: scheduled_for description: when to schedule this job (leave empty for immediate run) in: query @@ -18137,23 +18752,23 @@ paths: - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -18161,7 +18776,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -18170,7 +18785,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the script owner (default false) in: query @@ -18210,12 +18825,17 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the script owner (default false) in: query schema: type: boolean + - name: timeout + description: custom timeout in seconds for this preview run + in: query + schema: + type: integer - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -18223,7 +18843,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 requestBody: description: preview required: true @@ -18231,7 +18851,7 @@ paths: application/json: schema: type: object - properties: &ref_145 + properties: &ref_147 content: type: string description: The code to run @@ -18247,7 +18867,7 @@ paths: additionalProperties: true language: type: string - enum: *ref_99 + enum: *ref_100 tag: type: string kind: @@ -18269,8 +18889,8 @@ paths: additionalProperties: type: object description: An additional module file associated with a script - properties: *ref_101 - required: *ref_102 + properties: *ref_102 + required: *ref_103 temp_script_refs: type: object nullable: true @@ -18280,7 +18900,7 @@ paths: local content instead of the deployed script additionalProperties: type: string - required: &ref_146 + required: &ref_148 - args responses: '201': @@ -18308,7 +18928,7 @@ paths: application/json: schema: type: object - properties: &ref_429 + properties: &ref_431 content: type: string description: The code to run @@ -18318,8 +18938,8 @@ paths: additionalProperties: true language: type: string - enum: *ref_99 - required: &ref_430 + enum: *ref_100 + required: &ref_432 - content - args - language @@ -18343,7 +18963,7 @@ paths: - name: path in: path required: true - schema: *ref_97 + schema: *ref_98 requestBody: description: script args required: true @@ -18351,7 +18971,7 @@ paths: application/json: schema: type: object - properties: &ref_144 + properties: &ref_146 args: type: object description: The arguments to pass to the script or flow @@ -18376,7 +18996,7 @@ paths: - name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 requestBody: description: script args required: true @@ -18384,7 +19004,7 @@ paths: application/json: schema: type: object - properties: *ref_144 + properties: *ref_146 responses: '200': description: script result @@ -18418,8 +19038,8 @@ paths: application/json: schema: type: object - properties: *ref_145 - required: *ref_146 + properties: *ref_147 + required: *ref_148 responses: '200': description: job result @@ -18454,12 +19074,12 @@ paths: application/json: schema: type: object - properties: &ref_431 + properties: &ref_433 args: type: object description: The arguments to pass to the script or flow additionalProperties: true - required: &ref_432 + required: &ref_434 - args responses: '201': @@ -18492,15 +19112,15 @@ paths: type: array items: type: object - properties: &ref_147 + properties: &ref_149 raw_code: type: string path: type: string language: type: string - enum: *ref_99 - required: &ref_148 + enum: *ref_100 + required: &ref_150 - raw_code - path - language @@ -18544,8 +19164,8 @@ paths: type: array items: type: object - properties: *ref_147 - required: *ref_148 + properties: *ref_149 + required: *ref_150 entrypoint: type: string required: @@ -18585,7 +19205,7 @@ paths: description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: &ref_153 + properties: &ref_155 modules: type: array description: >- @@ -18694,8 +19314,8 @@ paths: description: >- A sticky note attached to a flow for documentation and annotation - properties: *ref_149 - required: *ref_150 + properties: *ref_151 + required: *ref_152 groups: type: array description: Semantic groups of modules for organizational purposes @@ -18708,9 +19328,9 @@ paths: naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id. - properties: *ref_151 - required: *ref_152 - required: &ref_154 + properties: *ref_153 + required: *ref_154 + required: &ref_156 - modules required: - path @@ -18742,7 +19362,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the script owner (default false) in: query @@ -18755,7 +19375,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -18769,14 +19389,14 @@ paths: application/json: schema: type: object - properties: &ref_156 + properties: &ref_158 value: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_153 - required: *ref_154 + properties: *ref_155 + required: *ref_156 path: type: string args: @@ -18787,7 +19407,7 @@ paths: type: string restarted_from: type: object - properties: &ref_155 + properties: &ref_157 flow_job_id: type: string format: uuid @@ -18820,7 +19440,7 @@ paths: `RestartedFlow` against `nested.flow_job_id` instead of fresh-launching it. type: object - properties: *ref_155 + properties: *ref_157 temp_script_refs: type: object nullable: true @@ -18831,7 +19451,7 @@ paths: of the deployed script additionalProperties: type: string - required: &ref_157 + required: &ref_159 - value - content - args @@ -18867,8 +19487,8 @@ paths: application/json: schema: type: object - properties: *ref_156 - required: *ref_157 + properties: *ref_158 + required: *ref_159 responses: '200': description: job result @@ -18893,7 +19513,7 @@ paths: application/json: schema: type: object - properties: &ref_530 + properties: &ref_532 entrypoint_function: type: string description: Name of the function to execute for dynamic select @@ -18914,7 +19534,7 @@ paths: description: Path to the deployed script or flow runnable_kind: type: string - enum: &ref_204 + enum: &ref_206 - script - flow required: @@ -18932,11 +19552,11 @@ paths: description: Code content for inline execution language: type: string - enum: *ref_99 + enum: *ref_100 required: - source - code - required: &ref_531 + required: &ref_533 - entrypoint_function - runnable_ref responses: @@ -18962,27 +19582,27 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: worker description: >- filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: &ref_160 + schema: &ref_162 type: string - name: script_path_exact description: >- @@ -18990,7 +19610,7 @@ paths: (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: &ref_161 + schema: &ref_163 type: string - name: script_path_start description: >- @@ -18998,12 +19618,12 @@ paths: 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: &ref_162 + schema: &ref_164 type: string - name: schedule_path description: mask to filter by schedule path in: query - schema: &ref_163 + schema: &ref_165 type: string - name: trigger_path description: >- @@ -19011,7 +19631,7 @@ paths: 'f/trigger1,f/trigger2') and negation by prefixing all values with '!' (e.g. '!f/trigger1,!f/trigger2') in: query - schema: &ref_312 + schema: &ref_314 type: string - name: trigger_kind description: >- @@ -19020,34 +19640,34 @@ paths: (e.g. '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: &ref_192 + schema: &ref_194 type: string - name: script_hash description: mask to filter exact matching path in: query - schema: &ref_164 + schema: &ref_166 type: string - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: &ref_165 + schema: &ref_167 type: string format: date-time - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: &ref_166 + schema: &ref_168 type: string format: date-time - name: success description: filter on successful jobs in: query - schema: &ref_174 + schema: &ref_176 type: boolean - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: &ref_168 + schema: &ref_170 type: boolean - name: job_kinds description: >- @@ -19055,36 +19675,36 @@ paths: ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: &ref_169 + schema: &ref_171 type: string - name: suspended description: filter on suspended jobs in: query - schema: &ref_170 + schema: &ref_172 type: boolean - name: running description: filter on running jobs in: query - schema: &ref_167 + schema: &ref_169 type: boolean - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: &ref_171 + schema: &ref_173 type: string - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: &ref_173 + schema: &ref_175 type: string - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: &ref_175 + schema: &ref_177 type: boolean - name: tag description: >- @@ -19092,7 +19712,7 @@ paths: 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: &ref_172 + schema: &ref_174 type: string - name: page description: which page to return (start at 1, default 1) @@ -19123,7 +19743,7 @@ paths: type: array items: type: object - properties: &ref_195 + properties: &ref_197 workspace_id: type: string id: @@ -19198,14 +19818,14 @@ paths: by extension its DT_TOKEN. flow_status: type: object - properties: &ref_179 + properties: &ref_181 step: type: integer modules: type: array items: type: object - properties: &ref_158 + properties: &ref_160 type: type: string enum: @@ -19359,20 +19979,20 @@ paths: type: array items: type: boolean - required: &ref_159 + required: &ref_161 - type user_states: additionalProperties: true preprocessor_module: allOf: - type: object - properties: *ref_158 - required: *ref_159 + properties: *ref_160 + required: *ref_161 failure_module: allOf: - type: object - properties: *ref_158 - required: *ref_159 + properties: *ref_160 + required: *ref_161 - type: object properties: parent_module: @@ -19387,13 +20007,13 @@ paths: items: type: string format: uuid - required: &ref_180 + required: &ref_182 - step - modules - failure_module workflow_as_code_status: type: object - properties: &ref_181 + properties: &ref_183 scheduled_for: type: string format: date-time @@ -19409,13 +20029,13 @@ paths: description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_153 - required: *ref_154 + properties: *ref_155 + required: *ref_156 is_flow_step: type: boolean language: type: string - enum: *ref_99 + enum: *ref_100 email: type: string visible_to_owner: @@ -19434,9 +20054,11 @@ paths: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string - required: &ref_196 + required: &ref_198 - id - running - canceled @@ -19552,14 +20174,14 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: &ref_178 + schema: &ref_180 type: string - name: worker description: >- @@ -19567,117 +20189,117 @@ paths: (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_160 + schema: *ref_162 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_161 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_162 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_163 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_164 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_165 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_166 + schema: *ref_168 - name: created_before description: filter on created before (inclusive) timestamp in: query - schema: &ref_186 + schema: &ref_188 type: string format: date-time - name: created_after description: filter on created after (exclusive) timestamp in: query - schema: &ref_187 + schema: &ref_189 type: string format: date-time - name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: &ref_188 + schema: &ref_190 type: string format: date-time - name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: &ref_189 + schema: &ref_191 type: string format: date-time - name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: &ref_190 + schema: &ref_192 type: string format: date-time - name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: &ref_191 + schema: &ref_193 type: string format: date-time - name: running description: filter on running jobs in: query - schema: *ref_167 + schema: *ref_169 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_168 + schema: *ref_170 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_169 + schema: *ref_171 - name: suspended description: filter on suspended jobs in: query - schema: *ref_170 + schema: *ref_172 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_172 + schema: *ref_174 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_173 + schema: *ref_175 - name: page description: which page to return (start at 1, default 1) in: query @@ -19754,96 +20376,96 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_161 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_162 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_163 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_164 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_165 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_166 + schema: *ref_168 - name: success description: filter on successful jobs in: query - schema: *ref_174 + schema: *ref_176 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_168 + schema: *ref_170 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_169 + schema: *ref_171 - name: suspended description: filter on suspended jobs in: query - schema: *ref_170 + schema: *ref_172 - name: running description: filter on running jobs in: query - schema: *ref_167 + schema: *ref_169 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_173 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_175 + schema: *ref_177 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_172 + schema: *ref_174 - name: page description: which page to return (start at 1, default 1) in: query @@ -19929,7 +20551,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: list of OTEL Span objects (compatible with OpenTelemetry Span proto) @@ -19957,7 +20579,7 @@ paths: schema: description: job trigger kind (schedule, http, websocket...) type: string - enum: &ref_177 + enum: &ref_179 - webhook - default_email - email @@ -19973,6 +20595,7 @@ paths: - azure - google - github + - asset - name: trigger_path description: The path of the trigger (can contain forward slashes) in: path @@ -20023,7 +20646,7 @@ paths: schema: description: job trigger kind (schedule, http, websocket...) type: string - enum: *ref_177 + enum: *ref_179 - name: trigger_path description: The path of the trigger (can contain forward slashes) in: path @@ -20070,68 +20693,68 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_178 + schema: *ref_180 - name: worker description: >- filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_160 + schema: *ref_162 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_161 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_162 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_163 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_164 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_165 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_166 + schema: *ref_168 - name: success description: filter on successful jobs in: query - schema: *ref_174 + schema: *ref_176 - name: status description: >- filter on the exact completed job status. Unlike `success=true` @@ -20151,30 +20774,30 @@ paths: ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_169 + schema: *ref_171 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_173 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_175 + schema: *ref_177 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_172 + schema: *ref_174 - name: page description: which page to return (start at 1, default 1) in: query @@ -20212,7 +20835,7 @@ paths: type: array items: type: object - properties: &ref_193 + properties: &ref_195 workspace_id: type: string id: @@ -20289,23 +20912,23 @@ paths: by extension its DT_TOKEN. flow_status: type: object - properties: *ref_179 - required: *ref_180 + properties: *ref_181 + required: *ref_182 workflow_as_code_status: type: object - properties: *ref_181 + properties: *ref_183 raw_flow: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_153 - required: *ref_154 + properties: *ref_155 + required: *ref_156 is_flow_step: type: boolean language: type: string - enum: *ref_99 + enum: *ref_100 is_skipped: type: boolean email: @@ -20328,9 +20951,11 @@ paths: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string - required: &ref_194 + required: &ref_196 - id - created_by - duration_ms @@ -20374,7 +20999,7 @@ paths: items: type: object description: Completed job with full data for export/import operations - properties: &ref_182 + properties: &ref_184 id: type: string format: uuid @@ -20469,7 +21094,7 @@ paths: type: boolean language: type: string - enum: *ref_99 + enum: *ref_100 is_skipped: type: boolean email: @@ -20512,7 +21137,7 @@ paths: status: type: string description: Actual job status from database - required: &ref_183 + required: &ref_185 - id - created_by - created_at @@ -20540,8 +21165,8 @@ paths: items: type: object description: Completed job with full data for export/import operations - properties: *ref_182 - required: *ref_183 + properties: *ref_184 + required: *ref_185 responses: '200': description: Successfully imported completed jobs @@ -20578,7 +21203,7 @@ paths: items: type: object description: Queued job with full data for export/import operations - properties: &ref_184 + properties: &ref_186 id: type: string format: uuid @@ -20668,7 +21293,7 @@ paths: type: boolean language: type: string - enum: *ref_99 + enum: *ref_100 email: type: string visible_to_owner: @@ -20709,7 +21334,7 @@ paths: suspend_until: type: string format: date-time - required: &ref_185 + required: &ref_187 - id - created_by - created_at @@ -20737,8 +21362,8 @@ paths: items: type: object description: Queued job with full data for export/import operations - properties: *ref_184 - required: *ref_185 + properties: *ref_186 + required: *ref_187 responses: '200': description: Successfully imported queued jobs @@ -20793,123 +21418,123 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_178 + schema: *ref_180 - name: worker description: >- filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_160 + schema: *ref_162 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_161 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_162 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_163 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_164 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_165 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_166 + schema: *ref_168 - name: created_before description: filter on created before (inclusive) timestamp in: query - schema: *ref_186 + schema: *ref_188 - name: created_after description: filter on created after (exclusive) timestamp in: query - schema: *ref_187 + schema: *ref_189 - name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_188 + schema: *ref_190 - name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_189 + schema: *ref_191 - name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: *ref_190 + schema: *ref_192 - name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: *ref_191 + schema: *ref_193 - name: running description: filter on running jobs in: query - schema: *ref_167 + schema: *ref_169 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_168 + schema: *ref_170 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_169 + schema: *ref_171 - name: suspended description: filter on suspended jobs in: query - schema: *ref_170 + schema: *ref_172 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_172 + schema: *ref_174 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_173 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_175 + schema: *ref_177 - name: per_page description: number of items to return for a given page (default 30, max 100) in: query @@ -20921,7 +21546,7 @@ paths: (e.g. '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: *ref_192 + schema: *ref_194 - name: is_skipped description: is the job skipped in: query @@ -20989,28 +21614,28 @@ paths: schema: type: array items: - oneOf: &ref_197 - - allOf: - - type: object - properties: *ref_193 - required: *ref_194 - - type: object - properties: - type: - type: string - enum: - - CompletedJob + oneOf: &ref_199 - allOf: - type: object properties: *ref_195 required: *ref_196 + - type: object + properties: + type: + type: string + enum: + - CompletedJob + - allOf: + - type: object + properties: *ref_197 + required: *ref_198 - type: object properties: type: type: string enum: - QueuedJob - discriminator: &ref_198 + discriminator: &ref_200 propertyName: type /jobs/db_clock: get: @@ -21078,7 +21703,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: no_logs in: query schema: @@ -21101,8 +21726,8 @@ paths: content: application/json: schema: - oneOf: *ref_197 - discriminator: *ref_198 + oneOf: *ref_199 + discriminator: *ref_200 /w/{workspace}/jobs_u/get_root_job_id/{id}: get: summary: get root job id @@ -21117,7 +21742,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: get root job id @@ -21140,7 +21765,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: remove_ansi_warnings in: query schema: @@ -21166,7 +21791,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: concatenated logs of all flow steps @@ -21174,6 +21799,73 @@ paths: text/plain: schema: type: string + /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}: + get: + summary: get all logs for a flow job in a structured format + operationId: getFlowAllLogsStructured + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: id + in: path + required: true + schema: *ref_178 + responses: + '200': + description: structured logs of all flow steps, one entry per job + content: + application/json: + schema: + type: array + items: + type: object + properties: + job_id: + type: string + label: + type: string + description: >- + human-readable label describing the job's position in + the flow tree + kind: + type: string + description: job kind (script, flow, forloopflow, ...) + flow_step_id: + type: string + nullable: true + step_path: + type: string + nullable: true + description: materialized step path (e.g. "a/b") + depth: + type: integer + description: depth in the flow tree (0 for the root flow job) + parent_module_type: + type: string + nullable: true + description: parent module type (forloopflow, branchall, ...) + sibling_index: + type: integer + description: >- + 1-based index of this job among siblings sharing the + same step + sibling_count: + type: integer + description: total number of siblings sharing the same step + logs: + type: string + required: + - job_id + - label + - kind + - depth + - sibling_index + - sibling_count + - logs /w/{workspace}/jobs_u/get_completed_logs_tail/{id}: get: summary: get completed job logs tail @@ -21188,7 +21880,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: completed job logs tail @@ -21210,7 +21902,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: job args @@ -21261,7 +21953,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: running in: query schema: @@ -21308,11 +22000,11 @@ paths: type: string flow_status: type: object - properties: *ref_179 - required: *ref_180 + properties: *ref_181 + required: *ref_182 workflow_as_code_status: type: object - properties: *ref_181 + properties: *ref_183 /w/{workspace}/jobs_u/getupdate_sse/{id}: get: summary: get job updates via server-sent events @@ -21327,7 +22019,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: running in: query schema: @@ -21400,7 +22092,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: flow debug info details @@ -21421,7 +22113,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: job details @@ -21429,8 +22121,8 @@ paths: application/json: schema: type: object - properties: *ref_193 - required: *ref_194 + properties: *ref_195 + required: *ref_196 /w/{workspace}/jobs_u/completed/get_result/{id}: get: summary: get completed job result @@ -21445,7 +22137,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: suspended_job in: query schema: @@ -21482,10 +22174,10 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: get_started in: query - schema: &ref_318 + schema: &ref_320 type: boolean responses: '200': @@ -21519,7 +22211,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: job timing details @@ -21538,6 +22230,154 @@ paths: type: integer required: - created_at + /w/{workspace}/jobs_u/dispatch_events/{id}: + get: + summary: list asset-trigger dispatch events for a producer job + description: > + Returns the chronological log of decisions the asset-trigger dispatcher + made after this producer job completed. Each row is one (subscriber, + asset write) decision: `dispatched` (with `child_job_id`), + `join_pending` (with `received_inputs` / `required_inputs` / + `partition`), or `skipped` (with `reason`). Rows are reaped + automatically when the producer's `v2_job` row is deleted by the + retention sweep. + operationId: listDispatchEvents + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: id + in: path + required: true + schema: *ref_178 + responses: + '200': + description: dispatch events for this producer job + content: + application/json: + schema: + type: array + items: + type: object + properties: + subscriber_path: + type: string + asset_kind: + type: string + enum: + - s3object + - resource + - variable + - ducklake + - datatable + - volume + asset_path: + type: string + outcome: + type: string + enum: + - dispatched + - join_pending + - skipped + child_job_id: + type: string + format: uuid + partition: + type: string + received_inputs: + type: integer + required_inputs: + type: integer + debounce_s: + type: integer + reason: + type: string + created_at: + type: string + format: date-time + required: + - subscriber_path + - asset_kind + - asset_path + - outcome + - created_at + /w/{workspace}/jobs/asset_dispatch_edges: + get: + summary: list asset-cascade producer→child job edges for a folder + description: > + Returns the `dispatched` asset-trigger edges (producer job → child job) + whose subscriber lives under `path_start`. Lets a pipeline view + reconstruct the cascade tree of a folder by job id and group connected + runs. Visibility follows the producer job's RLS. + operationId: listAssetDispatchEdges + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: path_start + in: query + required: true + description: Folder path prefix the children live under, e.g. `f/orders/`. + schema: + type: string + - name: created_after + in: query + required: false + description: Only edges dispatched at/after this instant. + schema: + type: string + format: date-time + responses: + '200': + description: asset-cascade edges for the folder + content: + application/json: + schema: + type: array + items: + type: object + properties: + producer_job_id: + type: string + format: uuid + child_job_id: + type: string + format: uuid + description: Set for `dispatched`; absent for `join_pending` inputs. + subscriber_path: + type: string + outcome: + type: string + enum: + - dispatched + - join_pending + asset_kind: + type: string + enum: + - s3object + - resource + - variable + - ducklake + - datatable + - volume + asset_path: + type: string + created_at: + type: string + format: date-time + required: + - producer_job_id + - subscriber_path + - outcome + - asset_kind + - asset_path + - created_at /w/{workspace}/jobs/completed/delete/{id}: post: summary: delete completed job (erase content but keep run id) @@ -21552,7 +22392,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: job details @@ -21560,8 +22400,8 @@ paths: application/json: schema: type: object - properties: *ref_193 - required: *ref_194 + properties: *ref_195 + required: *ref_196 /w/{workspace}/jobs_u/queue/cancel/{id}: post: summary: cancel queued or running job @@ -21576,7 +22416,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 requestBody: description: reason required: true @@ -21640,7 +22480,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 requestBody: description: reason required: true @@ -21702,7 +22542,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: scheduled for timestamp @@ -21724,7 +22564,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: resume_id in: path required: true @@ -21755,7 +22595,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: resume_id in: path required: true @@ -21805,7 +22645,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: approver in: query schema: @@ -21866,7 +22706,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: approver in: query schema: @@ -22040,6 +22880,13 @@ paths: type: integer approver: type: string + view_token: + type: string + description: >- + Share-read-link token for the flow. An authenticated + workspace member can append it as a `view_token` query + param on the run page to read a flow they don't otherwise + have access to. /w/{workspace}/jobs_u/resume/{id}/{resume_id}/{signature}: get: summary: resume a job for a suspended flow @@ -22054,7 +22901,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -22062,7 +22909,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 - name: resume_id in: path required: true @@ -22097,7 +22944,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: resume_id in: path required: true @@ -22139,7 +22986,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: key in: path required: true @@ -22171,7 +23018,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: key in: path required: true @@ -22197,7 +23044,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 requestBody: required: true content: @@ -22225,7 +23072,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: resume_id in: path required: true @@ -22260,7 +23107,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: resume_id in: path required: true @@ -22302,7 +23149,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 - name: resume_id in: path required: true @@ -22326,8 +23173,8 @@ paths: type: object properties: job: - oneOf: *ref_197 - discriminator: *ref_198 + oneOf: *ref_199 + discriminator: *ref_200 approvers: type: array items: @@ -22340,6 +23187,13 @@ paths: required: - resume_id - approver + view_token: + type: string + description: >- + Share-read-link token for the parent flow. An + authenticated workspace member can append it as a + `view_token` query param on the run page to read a flow + they don't otherwise have access to. required: - job - approvers @@ -22402,7 +23256,7 @@ paths: application/json: schema: type: object - properties: &ref_438 + properties: &ref_440 path: type: string description: >- @@ -22495,7 +23349,7 @@ paths: nullable: true type: object description: Retry configuration for failed module executions - properties: &ref_201 + properties: &ref_203 constant: type: object description: Retry with constant delay between attempts @@ -22530,8 +23384,8 @@ paths: retry_if: type: object description: Conditional retry based on error or result - properties: *ref_199 - required: *ref_200 + properties: *ref_201 + required: *ref_202 no_flow_overlap: type: boolean description: >- @@ -22584,7 +23438,7 @@ paths: type: array items: type: string - required: &ref_439 + required: &ref_441 - path - schedule - timezone @@ -22628,7 +23482,7 @@ paths: application/json: schema: type: object - properties: &ref_440 + properties: &ref_442 schedule: type: string description: >- @@ -22703,7 +23557,7 @@ paths: nullable: true type: object description: Retry configuration for failed module executions - properties: *ref_201 + properties: *ref_203 no_flow_overlap: type: boolean description: >- @@ -22759,7 +23613,7 @@ paths: type: array items: type: string - required: &ref_441 + required: &ref_443 - schedule - timezone - args @@ -22863,7 +23717,7 @@ paths: schema: allOf: - type: object - properties: &ref_202 + properties: &ref_204 path: type: string description: >- @@ -22984,7 +23838,7 @@ paths: nullable: true type: object description: Retry configuration for failed module executions - properties: *ref_201 + properties: *ref_203 summary: type: string nullable: true @@ -23052,7 +23906,7 @@ paths: description: > Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder. - required: &ref_203 + required: &ref_205 - path - edited_by - edited_at @@ -23146,7 +24000,7 @@ paths: filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: path description: filter by path (script path) in: query @@ -23208,8 +24062,8 @@ paths: type: array items: type: object - properties: *ref_202 - required: *ref_203 + properties: *ref_204 + required: *ref_205 /w/{workspace}/schedules/list_with_jobs: get: summary: list schedules with last 20 jobs @@ -23237,10 +24091,10 @@ paths: schema: type: array items: - allOf: &ref_437 + allOf: &ref_439 - type: object - properties: *ref_202 - required: *ref_203 + properties: *ref_204 + required: *ref_205 - type: object properties: jobs: @@ -23318,10 +24172,10 @@ paths: application/json: schema: type: object - properties: &ref_205 + properties: &ref_207 info: type: object - properties: &ref_447 + properties: &ref_449 title: type: string version: @@ -23350,28 +24204,28 @@ paths: type: string required: - name - required: &ref_448 + required: &ref_450 - title - version url: type: string openapi_spec_format: type: string - enum: &ref_442 + enum: &ref_444 - yaml - json http_route_filters: type: array items: type: object - properties: &ref_443 + properties: &ref_445 folder_regex: type: string path_regex: type: string route_path_regex: type: string - required: &ref_444 + required: &ref_446 - folder_regex - path_regex - route_path_regex @@ -23379,7 +24233,7 @@ paths: type: array items: type: object - properties: &ref_445 + properties: &ref_447 user_or_folder_regex: type: string enum: @@ -23392,8 +24246,8 @@ paths: type: string runnable_kind: type: string - enum: *ref_204 - required: &ref_446 + enum: *ref_206 + required: &ref_448 - user_or_folder_regex - user_or_folder_regex_value - path @@ -23422,7 +24276,7 @@ paths: application/json: schema: type: object - properties: *ref_205 + properties: *ref_207 responses: '200': description: Downloaded OpenAPI spec @@ -23451,7 +24305,7 @@ paths: type: array items: type: object - properties: &ref_206 + properties: &ref_208 path: type: string description: >- @@ -23505,7 +24359,7 @@ paths: HTTP method (get, post, put, delete, patch) that triggers this endpoint type: string - enum: &ref_208 + enum: &ref_210 - get - post - put @@ -23527,7 +24381,7 @@ paths: 'async' returns job ID immediately, 'sync_sse' streams results via Server-Sent Events type: string - enum: &ref_209 + enum: &ref_211 - sync - async - sync_sse @@ -23537,7 +24391,7 @@ paths: 'windmill' (Windmill token), 'api_key', 'basic_http', 'custom_script', 'signature' type: string - enum: &ref_210 + enum: &ref_212 - none - windmill - api_key @@ -23555,7 +24409,7 @@ paths: mode: description: job trigger mode type: string - enum: &ref_211 + enum: &ref_213 - enabled - disabled - suspended @@ -23576,7 +24430,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -23592,7 +24446,7 @@ paths: type: array items: type: string - required: &ref_207 + required: &ref_209 - path - script_path - route_path @@ -23625,8 +24479,8 @@ paths: application/json: schema: type: object - properties: *ref_206 - required: *ref_207 + properties: *ref_208 + required: *ref_209 responses: '201': description: http trigger created @@ -23656,7 +24510,7 @@ paths: application/json: schema: type: object - properties: &ref_449 + properties: &ref_451 path: type: string description: >- @@ -23716,7 +24570,7 @@ paths: HTTP method (get, post, put, delete, patch) that triggers this endpoint type: string - enum: *ref_208 + enum: *ref_210 is_async: type: boolean description: Deprecated, use request_type instead @@ -23726,14 +24580,14 @@ paths: 'async' returns job ID immediately, 'sync_sse' streams results via Server-Sent Events type: string - enum: *ref_209 + enum: *ref_211 authentication_method: description: >- How requests are authenticated - 'none' (public), 'windmill' (Windmill token), 'api_key', 'basic_http', 'custom_script', 'signature' type: string - enum: *ref_210 + enum: *ref_212 is_static_website: type: boolean description: >- @@ -23757,7 +24611,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -23773,7 +24627,7 @@ paths: type: array items: type: string - required: &ref_450 + required: &ref_452 - path - script_path - is_flow @@ -23839,9 +24693,9 @@ paths: application/json: schema: allOf: - - allOf: &ref_213 + - allOf: &ref_215 - type: object - properties: &ref_219 + properties: &ref_221 path: type: string description: >- @@ -23884,7 +24738,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 labels: type: array items: @@ -23913,7 +24767,7 @@ paths: Frontend appends a `*` to the displayed name. type: boolean - required: &ref_220 + required: &ref_222 - path - script_path - permissioned_as @@ -23924,7 +24778,7 @@ paths: - is_flow - mode type: object - properties: &ref_214 + properties: &ref_216 route_path: type: string description: >- @@ -23953,7 +24807,7 @@ paths: HTTP method (get, post, put, delete, patch) that triggers this endpoint type: string - enum: *ref_208 + enum: *ref_210 authentication_resource_path: type: string nullable: true @@ -23975,14 +24829,14 @@ paths: 'async' returns job ID immediately, 'sync_sse' streams results via Server-Sent Events type: string - enum: *ref_209 + enum: *ref_211 authentication_method: description: >- How requests are authenticated - 'none' (public), 'windmill' (Windmill token), 'api_key', 'basic_http', 'custom_script', 'signature' type: string - enum: *ref_210 + enum: *ref_212 is_static_website: type: boolean description: >- @@ -24013,8 +24867,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_215 + properties: *ref_203 + required: &ref_217 - route_path - request_type - authentication_method @@ -24105,7 +24959,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: http trigger list @@ -24114,10 +24968,10 @@ paths: schema: type: array items: - allOf: *ref_213 + allOf: *ref_215 type: object - properties: *ref_214 - required: *ref_215 + properties: *ref_216 + required: *ref_217 /w/{workspace}/http_triggers/exists/{path}: get: summary: does http trigger exists @@ -24163,7 +25017,7 @@ paths: type: string http_method: type: string - enum: *ref_208 + enum: *ref_210 trigger_path: type: string workspaced_route: @@ -24203,7 +25057,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -24236,7 +25090,7 @@ paths: application/json: schema: type: object - properties: &ref_451 + properties: &ref_453 path: type: string description: >- @@ -24261,7 +25115,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 filters: type: array description: >- @@ -24292,7 +25146,7 @@ paths: Messages to send immediately after connecting (can be raw strings or computed by runnables) items: - anyOf: &ref_216 + anyOf: &ref_218 - type: object properties: raw_message: @@ -24335,7 +25189,7 @@ paths: nullable: true description: Optional periodic heartbeat message configuration type: object - properties: &ref_217 + properties: &ref_219 interval_secs: type: integer minimum: 1 @@ -24352,7 +25206,7 @@ paths: Optional. Top-level JSON field to extract from incoming messages. The extracted value replaces {{state}} in the heartbeat message. - required: &ref_218 + required: &ref_220 - interval_secs - message error_handler_path: @@ -24365,7 +25219,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24381,7 +25235,7 @@ paths: type: array items: type: string - required: &ref_452 + required: &ref_454 - path - script_path - url @@ -24418,7 +25272,7 @@ paths: application/json: schema: type: object - properties: &ref_453 + properties: &ref_455 url: type: string description: >- @@ -24470,7 +25324,7 @@ paths: Messages to send immediately after connecting (can be raw strings or computed by runnables) items: - anyOf: *ref_216 + anyOf: *ref_218 url_runnable_args: description: The arguments to pass to the script or flow nullable: true @@ -24488,8 +25342,8 @@ paths: nullable: true description: Optional periodic heartbeat message configuration type: object - properties: *ref_217 - required: *ref_218 + properties: *ref_219 + required: *ref_220 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -24500,7 +25354,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24516,7 +25370,7 @@ paths: type: array items: type: string - required: &ref_454 + required: &ref_456 - path - script_path - url @@ -24582,12 +25436,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_221 + - allOf: &ref_223 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_222 + properties: &ref_224 url: type: string description: >- @@ -24636,7 +25490,7 @@ paths: Messages to send immediately after connecting (can be raw strings or computed by runnables) items: - anyOf: *ref_216 + anyOf: *ref_218 url_runnable_args: description: The arguments to pass to the script or flow nullable: true @@ -24656,8 +25510,8 @@ paths: nullable: true description: Optional periodic heartbeat message configuration type: object - properties: *ref_217 - required: *ref_218 + properties: *ref_219 + required: *ref_220 error_handler_path: type: string description: >- @@ -24670,8 +25524,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_223 + properties: *ref_203 + required: &ref_225 - url - filters - can_return_message @@ -24758,7 +25612,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: websocket trigger list @@ -24767,10 +25621,10 @@ paths: schema: type: array items: - allOf: *ref_221 + allOf: *ref_223 type: object - properties: *ref_222 - required: *ref_223 + properties: *ref_224 + required: *ref_225 /w/{workspace}/websocket_triggers/exists/{path}: get: summary: does websocket trigger exists @@ -24819,7 +25673,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -24889,7 +25743,7 @@ paths: application/json: schema: type: object - properties: &ref_486 + properties: &ref_488 path: type: string description: >- @@ -24958,7 +25812,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -24969,7 +25823,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24985,7 +25839,7 @@ paths: type: array items: type: string - required: &ref_487 + required: &ref_489 - path - script_path - is_flow @@ -25022,7 +25876,7 @@ paths: application/json: schema: type: object - properties: &ref_488 + properties: &ref_490 kafka_resource_path: type: string description: >- @@ -25098,7 +25952,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -25114,7 +25968,7 @@ paths: type: array items: type: string - required: &ref_489 + required: &ref_491 - path - script_path - kafka_resource_path @@ -25180,12 +26034,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_224 + - allOf: &ref_226 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_225 + properties: &ref_227 kafka_resource_path: type: string description: >- @@ -25262,8 +26116,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_226 + properties: *ref_203 + required: &ref_228 - kafka_resource_path - group_id - topics @@ -25350,7 +26204,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: kafka trigger list @@ -25359,10 +26213,10 @@ paths: schema: type: array items: - allOf: *ref_224 + allOf: *ref_226 type: object - properties: *ref_225 - required: *ref_226 + properties: *ref_227 + required: *ref_228 /w/{workspace}/kafka_triggers/exists/{path}: get: summary: does kafka trigger exists @@ -25411,7 +26265,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -25530,7 +26384,7 @@ paths: application/json: schema: type: object - properties: &ref_490 + properties: &ref_492 path: type: string description: >- @@ -25573,7 +26427,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -25584,7 +26438,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -25600,7 +26454,7 @@ paths: type: array items: type: string - required: &ref_491 + required: &ref_493 - path - script_path - is_flow @@ -25636,7 +26490,7 @@ paths: application/json: schema: type: object - properties: &ref_492 + properties: &ref_494 nats_resource_path: type: string description: >- @@ -25686,7 +26540,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -25702,7 +26556,7 @@ paths: type: array items: type: string - required: &ref_493 + required: &ref_495 - path - script_path - nats_resource_path @@ -25767,12 +26621,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_227 + - allOf: &ref_229 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_228 + properties: &ref_230 nats_resource_path: type: string description: >- @@ -25824,8 +26678,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_229 + properties: *ref_203 + required: &ref_231 - nats_resource_path - use_jetstream - subjects @@ -25911,7 +26765,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: nats trigger list @@ -25920,10 +26774,10 @@ paths: schema: type: array items: - allOf: *ref_227 + allOf: *ref_229 type: object - properties: *ref_228 - required: *ref_229 + properties: *ref_230 + required: *ref_231 /w/{workspace}/nats_triggers/exists/{path}: get: summary: does nats trigger exists @@ -25972,7 +26826,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -26035,7 +26889,7 @@ paths: application/json: schema: type: object - properties: &ref_473 + properties: &ref_475 queue_url: type: string description: The full URL of the AWS SQS queue to poll for messages @@ -26044,7 +26898,7 @@ paths: Authentication type - 'credentials' for access key/secret, 'oidc' for OpenID Connect type: string - enum: &ref_230 + enum: &ref_232 - oidc - credentials aws_resource_path: @@ -26079,7 +26933,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -26090,7 +26944,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -26106,7 +26960,7 @@ paths: type: array items: type: string - required: &ref_474 + required: &ref_476 - queue_url - aws_resource_path - path @@ -26142,7 +26996,7 @@ paths: application/json: schema: type: object - properties: &ref_475 + properties: &ref_477 queue_url: type: string description: The full URL of the AWS SQS queue to poll for messages @@ -26151,7 +27005,7 @@ paths: Authentication type - 'credentials' for access key/secret, 'oidc' for OpenID Connect type: string - enum: *ref_230 + enum: *ref_232 aws_resource_path: type: string description: >- @@ -26184,7 +27038,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -26195,7 +27049,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -26211,7 +27065,7 @@ paths: type: array items: type: string - required: &ref_476 + required: &ref_478 - queue_url - aws_resource_path - path @@ -26277,12 +27131,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_231 + - allOf: &ref_233 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_232 + properties: &ref_234 queue_url: type: string description: The full URL of the AWS SQS queue to poll for messages @@ -26291,7 +27145,7 @@ paths: Authentication type - 'credentials' for access key/secret, 'oidc' for OpenID Connect type: string - enum: *ref_230 + enum: *ref_232 aws_resource_path: type: string description: >- @@ -26329,8 +27183,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_233 + properties: *ref_203 + required: &ref_235 - queue_url - aws_resource_path - aws_auth_resource_type @@ -26416,7 +27270,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: sqs trigger list @@ -26425,10 +27279,10 @@ paths: schema: type: array items: - allOf: *ref_231 + allOf: *ref_233 type: object - properties: *ref_232 - required: *ref_233 + properties: *ref_234 + required: *ref_235 /w/{workspace}/sqs_triggers/exists/{path}: get: summary: does sqs trigger exists @@ -26477,7 +27331,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -26542,17 +27396,17 @@ paths: type: array items: type: object - properties: &ref_577 + properties: &ref_579 service_name: type: string - enum: &ref_234 + enum: &ref_236 - nextcloud - google - github oauth_data: nullable: true type: object - properties: &ref_235 + properties: &ref_237 client_id: type: string description: The OAuth client ID for the workspace @@ -26567,7 +27421,7 @@ paths: type: string format: uri description: The OAuth redirect URI - required: &ref_236 + required: &ref_238 - client_id - client_secret - base_url @@ -26576,7 +27430,7 @@ paths: type: string nullable: true description: Path to the resource storing the OAuth token - required: &ref_578 + required: &ref_580 - service_name /w/{workspace}/native_triggers/integrations/{service_name}/exists: get: @@ -26594,7 +27448,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 responses: '200': description: integration exists @@ -26618,7 +27472,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 requestBody: description: new native trigger service required: true @@ -26626,8 +27480,8 @@ paths: application/json: schema: type: object - properties: *ref_235 - required: *ref_236 + properties: *ref_237 + required: *ref_238 responses: '201': description: native trigger service created @@ -26651,7 +27505,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 requestBody: description: redirect_uri required: true @@ -26659,10 +27513,10 @@ paths: application/json: schema: type: object - properties: &ref_237 + properties: &ref_239 redirect_uri: type: string - required: &ref_238 + required: &ref_240 - redirect_uri responses: '200': @@ -26687,7 +27541,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 responses: '200': description: whether instance sharing is available @@ -26711,7 +27565,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 requestBody: description: redirect_uri required: true @@ -26719,8 +27573,8 @@ paths: application/json: schema: type: object - properties: *ref_237 - required: *ref_238 + properties: *ref_239 + required: *ref_240 responses: '200': description: authorization URL using instance credentials @@ -26744,7 +27598,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 responses: '200': description: native trigger service deleted @@ -26768,7 +27622,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 requestBody: description: OAuth callback data required: true @@ -26817,7 +27671,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 requestBody: description: new native trigger configuration required: true @@ -26826,7 +27680,7 @@ paths: schema: type: object description: Data for creating or updating a native trigger - properties: &ref_239 + properties: &ref_241 script_path: type: string description: The path to the script or flow that will be triggered @@ -26843,7 +27697,7 @@ paths: type: string nullable: true description: Short summary to be displayed when listed - required: &ref_240 + required: &ref_242 - script_path - is_flow - service_config @@ -26855,13 +27709,13 @@ paths: schema: type: object description: Response returned when a native trigger is created - properties: &ref_580 + properties: &ref_582 external_id: type: string description: >- The external ID of the created trigger from the external service - required: &ref_581 + required: &ref_583 - external_id /w/{workspace}/native_triggers/{service_name}/update/{external_id}: post: @@ -26884,7 +27738,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 - name: external_id in: path required: true @@ -26899,8 +27753,8 @@ paths: schema: type: object description: Data for creating or updating a native trigger - properties: *ref_239 - required: *ref_240 + properties: *ref_241 + required: *ref_242 responses: '200': description: native trigger updated @@ -26929,7 +27783,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 - name: external_id in: path required: true @@ -26946,7 +27800,7 @@ paths: description: >- Full trigger response containing both Windmill data and external service data - properties: &ref_575 + properties: &ref_577 external_id: type: string description: The unique identifier from the external service @@ -26955,7 +27809,7 @@ paths: description: The workspace this trigger belongs to service_name: type: string - enum: *ref_234 + enum: *ref_236 script_path: type: string description: The path to the script or flow that will be triggered @@ -26982,7 +27836,7 @@ paths: type: object description: Configuration data from the external service additionalProperties: true - required: &ref_576 + required: &ref_578 - external_id - workspace_id - service_name @@ -27011,7 +27865,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 - name: external_id in: path required: true @@ -27042,7 +27896,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 - name: page description: which page to return (start at 1, default 1) in: query @@ -27076,7 +27930,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: native triggers list @@ -27087,7 +27941,7 @@ paths: items: type: object description: A native trigger stored in Windmill - properties: &ref_573 + properties: &ref_575 external_id: type: string description: The unique identifier from the external service @@ -27096,7 +27950,7 @@ paths: description: The workspace this trigger belongs to service_name: type: string - enum: *ref_234 + enum: *ref_236 script_path: type: string description: The path to the script or flow that will be triggered @@ -27119,7 +27973,7 @@ paths: type: string nullable: true description: Short summary to be displayed when listed - required: &ref_574 + required: &ref_576 - external_id - workspace_id - service_name @@ -27143,7 +27997,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 - name: external_id in: path required: true @@ -27173,7 +28027,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 responses: '200': description: sync completed successfully @@ -27198,7 +28052,7 @@ paths: type: array items: type: object - properties: &ref_582 + properties: &ref_584 id: type: string name: @@ -27209,7 +28063,7 @@ paths: type: string path: type: string - required: &ref_583 + required: &ref_585 - id - name - path @@ -27234,7 +28088,7 @@ paths: type: array items: type: object - properties: &ref_584 + properties: &ref_586 id: type: string summary: @@ -27242,7 +28096,7 @@ paths: primary: type: boolean default: false - required: &ref_585 + required: &ref_587 - id - summary /w/{workspace}/native_triggers/google/drive/files: @@ -27285,12 +28139,12 @@ paths: application/json: schema: type: object - properties: &ref_588 + properties: &ref_590 files: type: array items: type: object - properties: &ref_586 + properties: &ref_588 id: type: string name: @@ -27300,13 +28154,13 @@ paths: is_folder: type: boolean default: false - required: &ref_587 + required: &ref_589 - id - name - mime_type next_page_token: type: string - required: &ref_589 + required: &ref_591 - files /w/{workspace}/native_triggers/google/drive/shared_drives: get: @@ -27329,12 +28183,12 @@ paths: type: array items: type: object - properties: &ref_590 + properties: &ref_592 id: type: string name: type: string - required: &ref_591 + required: &ref_593 - id - name /w/{workspace}/native_triggers/github/repos: @@ -27358,7 +28212,7 @@ paths: type: array items: type: object - properties: &ref_592 + properties: &ref_594 full_name: type: string name: @@ -27367,7 +28221,7 @@ paths: type: string private: type: boolean - required: &ref_593 + required: &ref_595 - full_name - name - owner @@ -27384,7 +28238,7 @@ paths: required: true schema: type: string - enum: *ref_234 + enum: *ref_236 - name: workspace_id in: path required: true @@ -27433,7 +28287,7 @@ paths: application/json: schema: type: object - properties: &ref_456 + properties: &ref_458 mqtt_resource_path: type: string description: >- @@ -27443,16 +28297,16 @@ paths: type: array items: type: object - properties: &ref_241 + properties: &ref_243 qos: type: string - enum: &ref_455 + enum: &ref_457 - qos0 - qos1 - qos2 topic: type: string - required: &ref_242 + required: &ref_244 - qos - topic description: >- @@ -27466,7 +28320,7 @@ paths: nullable: true description: MQTT v3 specific configuration (clean_session) type: object - properties: &ref_243 + properties: &ref_245 clean_session: type: boolean v5_config: @@ -27475,7 +28329,7 @@ paths: MQTT v5 specific configuration (clean_start, topic_alias_maximum, session_expiry_interval) type: object - properties: &ref_244 + properties: &ref_246 clean_start: type: boolean topic_alias_maximum: @@ -27486,7 +28340,7 @@ paths: nullable: true description: MQTT protocol version ('v3' or 'v5') type: string - enum: &ref_245 + enum: &ref_247 - v3 - v5 path: @@ -27508,7 +28362,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -27519,7 +28373,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -27535,7 +28389,7 @@ paths: type: array items: type: string - required: &ref_457 + required: &ref_459 - path - script_path - is_flow @@ -27570,7 +28424,7 @@ paths: application/json: schema: type: object - properties: &ref_458 + properties: &ref_460 mqtt_resource_path: type: string description: >- @@ -27580,8 +28434,8 @@ paths: type: array items: type: object - properties: *ref_241 - required: *ref_242 + properties: *ref_243 + required: *ref_244 description: >- Array of MQTT topics to subscribe to, each with topic name and QoS level @@ -27593,19 +28447,19 @@ paths: nullable: true description: MQTT v3 specific configuration (clean_session) type: object - properties: *ref_243 + properties: *ref_245 v5_config: nullable: true description: >- MQTT v5 specific configuration (clean_start, topic_alias_maximum, session_expiry_interval) type: object - properties: *ref_244 + properties: *ref_246 client_version: nullable: true description: MQTT protocol version ('v3' or 'v5') type: string - enum: *ref_245 + enum: *ref_247 path: type: string description: >- @@ -27625,7 +28479,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -27636,7 +28490,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -27652,7 +28506,7 @@ paths: type: array items: type: string - required: &ref_459 + required: &ref_461 - path - script_path - is_flow @@ -27717,12 +28571,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_246 + - allOf: &ref_248 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_247 + properties: &ref_249 mqtt_resource_path: type: string description: >- @@ -27732,8 +28586,8 @@ paths: type: array items: type: object - properties: *ref_241 - required: *ref_242 + properties: *ref_243 + required: *ref_244 description: >- Array of MQTT topics to subscribe to, each with topic name and QoS level @@ -27741,14 +28595,14 @@ paths: nullable: true description: MQTT v3 specific configuration (clean_session) type: object - properties: *ref_243 + properties: *ref_245 v5_config: nullable: true description: >- MQTT v5 specific configuration (clean_start, topic_alias_maximum, session_expiry_interval) type: object - properties: *ref_244 + properties: *ref_246 client_id: type: string nullable: true @@ -27757,7 +28611,7 @@ paths: nullable: true description: MQTT protocol version ('v3' or 'v5') type: string - enum: *ref_245 + enum: *ref_247 server_id: type: string description: >- @@ -27782,8 +28636,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_248 + properties: *ref_203 + required: &ref_250 - subscribe_topics - mqtt_resource_path - type: object @@ -27868,7 +28722,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: mqtt trigger list @@ -27877,10 +28731,10 @@ paths: schema: type: array items: - allOf: *ref_246 + allOf: *ref_248 type: object - properties: *ref_247 - required: *ref_248 + properties: *ref_249 + required: *ref_250 /w/{workspace}/mqtt_triggers/exists/{path}: get: summary: does mqtt trigger exists @@ -27929,7 +28783,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -27993,7 +28847,7 @@ paths: schema: type: object description: Data for creating or updating a Google Cloud Pub/Sub trigger. - properties: &ref_249 + properties: &ref_251 gcp_resource_path: type: string description: >- @@ -28001,7 +28855,7 @@ paths: credentials for authentication. subscription_mode: type: string - enum: &ref_254 + enum: &ref_256 - existing - create_update description: >- @@ -28019,7 +28873,7 @@ paths: description: Base URL for push delivery endpoint. delivery_type: type: string - enum: &ref_251 + enum: &ref_253 - push - pull description: >- @@ -28030,7 +28884,7 @@ paths: nullable: true type: object description: Configuration for push delivery mode. - properties: &ref_252 + properties: &ref_254 audience: type: string description: >- @@ -28041,7 +28895,7 @@ paths: description: >- If true, push messages will include OIDC authentication tokens. - required: &ref_253 + required: &ref_255 - authenticate - base_endpoint path: @@ -28063,7 +28917,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 auto_acknowledge_msg: type: boolean description: >- @@ -28090,7 +28944,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -28106,7 +28960,7 @@ paths: type: array items: type: string - required: &ref_250 + required: &ref_252 - path - script_path - is_flow @@ -28143,8 +28997,8 @@ paths: schema: type: object description: Data for creating or updating a Google Cloud Pub/Sub trigger. - properties: *ref_249 - required: *ref_250 + properties: *ref_251 + required: *ref_252 responses: '200': description: gcp trigger updated @@ -28203,15 +29057,15 @@ paths: application/json: schema: allOf: - - allOf: &ref_255 + - allOf: &ref_257 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object description: >- A Google Cloud Pub/Sub trigger that executes a script or flow when messages are received. - properties: &ref_256 + properties: &ref_258 gcp_resource_path: type: string description: >- @@ -28230,7 +29084,7 @@ paths: (internal use). delivery_type: type: string - enum: *ref_251 + enum: *ref_253 description: >- Delivery mode for messages. 'push' for HTTP push delivery where messages are sent to a webhook @@ -28240,11 +29094,11 @@ paths: nullable: true type: object description: Configuration for push delivery mode. - properties: *ref_252 - required: *ref_253 + properties: *ref_254 + required: *ref_255 subscription_mode: type: string - enum: *ref_254 + enum: *ref_256 description: >- The mode of subscription. 'existing' means using an existing GCP subscription, while 'create_update' @@ -28268,8 +29122,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_257 + properties: *ref_203 + required: &ref_259 - gcp_resource_path - topic_id - subscription_id @@ -28357,7 +29211,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: gcp trigger list @@ -28366,13 +29220,13 @@ paths: schema: type: array items: - allOf: *ref_255 + allOf: *ref_257 type: object description: >- A Google Cloud Pub/Sub trigger that executes a script or flow when messages are received. - properties: *ref_256 - required: *ref_257 + properties: *ref_258 + required: *ref_259 /w/{workspace}/gcp_triggers/exists/{path}: get: summary: does gcp trigger exists @@ -28421,7 +29275,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -28488,10 +29342,10 @@ paths: application/json: schema: type: object - properties: &ref_462 + properties: &ref_464 subscription_id: type: string - required: &ref_463 + required: &ref_465 - subscription_id responses: '200': @@ -28546,10 +29400,10 @@ paths: application/json: schema: type: object - properties: &ref_460 + properties: &ref_462 topic_id: type: string - required: &ref_461 + required: &ref_463 - topic_id responses: '200': @@ -28578,12 +29432,12 @@ paths: schema: type: object description: Data for creating or updating an Azure Event Grid trigger. - properties: &ref_258 + properties: &ref_260 azure_resource_path: type: string azure_mode: type: string - enum: &ref_260 + enum: &ref_262 - basic_push - namespace_push - namespace_pull @@ -28615,7 +29469,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 error_handler_path: type: string error_handler_args: @@ -28625,7 +29479,7 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string preserve_permissioned_as: @@ -28634,7 +29488,7 @@ paths: type: array items: type: string - required: &ref_259 + required: &ref_261 - path - script_path - is_flow @@ -28671,8 +29525,8 @@ paths: schema: type: object description: Data for creating or updating an Azure Event Grid trigger. - properties: *ref_258 - required: *ref_259 + properties: *ref_260 + required: *ref_261 responses: '200': description: azure trigger updated @@ -28731,20 +29585,20 @@ paths: application/json: schema: allOf: - - allOf: &ref_261 + - allOf: &ref_263 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object description: >- An Azure Event Grid trigger that executes a script or flow when events arrive. - properties: &ref_262 + properties: &ref_264 azure_resource_path: type: string azure_mode: type: string - enum: *ref_260 + enum: *ref_262 description: Azure Event Grid trigger mode. scope_resource_id: type: string @@ -28780,8 +29634,8 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_201 - required: &ref_263 + properties: *ref_203 + required: &ref_265 - azure_resource_path - azure_mode - scope_resource_id @@ -28862,7 +29716,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: azure trigger list @@ -28871,13 +29725,13 @@ paths: schema: type: array items: - allOf: *ref_261 + allOf: *ref_263 type: object description: >- An Azure Event Grid trigger that executes a script or flow when events arrive. - properties: *ref_262 - required: *ref_263 + properties: *ref_264 + required: *ref_265 /w/{workspace}/azure_triggers/exists/{path}: get: summary: check whether an azure trigger exists @@ -28925,7 +29779,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -28957,10 +29811,10 @@ paths: application/json: schema: type: object - properties: &ref_466 + properties: &ref_468 azure_resource_path: type: string - required: &ref_467 + required: &ref_469 - azure_resource_path responses: '200': @@ -28990,10 +29844,10 @@ paths: application/json: schema: type: object - properties: &ref_468 + properties: &ref_470 scope_resource_id: type: string - required: &ref_469 + required: &ref_471 - scope_resource_id responses: '200': @@ -29025,12 +29879,12 @@ paths: application/json: schema: type: object - properties: &ref_470 + properties: &ref_472 scope_resource_id: type: string topic_name: type: string - required: &ref_471 + required: &ref_473 - scope_resource_id - topic_name responses: @@ -29063,10 +29917,10 @@ paths: application/json: schema: type: object - properties: &ref_464 + properties: &ref_466 azure_mode: type: string - enum: *ref_260 + enum: *ref_262 description: Azure Event Grid trigger mode. scope_resource_id: type: string @@ -29075,7 +29929,7 @@ paths: nullable: true subscription_name: type: string - required: &ref_465 + required: &ref_467 - azure_mode - scope_resource_id - subscription_name @@ -29111,7 +29965,7 @@ paths: items: type: object description: An ARM resource the service principal can see. - properties: &ref_264 + properties: &ref_266 id: type: string name: @@ -29120,7 +29974,7 @@ paths: type: string type: type: string - required: &ref_265 + required: &ref_267 - id - name - type @@ -29151,8 +30005,8 @@ paths: items: type: object description: An ARM resource the service principal can see. - properties: *ref_264 - required: *ref_265 + properties: *ref_266 + required: *ref_267 /w/{workspace}/postgres_triggers/postgres/version/{path}: get: summary: get postgres version @@ -29215,19 +30069,19 @@ paths: application/json: schema: type: object - properties: &ref_480 + properties: &ref_482 postgres_resource_path: type: string relations: type: array items: type: object - properties: &ref_267 + properties: &ref_269 schema_name: type: string table_to_track: type: array - items: &ref_478 + items: &ref_480 type: object properties: table_name: @@ -29240,14 +30094,14 @@ paths: type: string required: - table_name - required: &ref_268 + required: &ref_270 - schema_name - table_to_track language: type: string - enum: &ref_479 + enum: &ref_481 - Typescript - required: &ref_481 + required: &ref_483 - postgres_resource_path - relations - language @@ -29272,7 +30126,7 @@ paths: - name: id in: path required: true - schema: &ref_309 + schema: &ref_311 type: string responses: '200': @@ -29305,7 +30159,7 @@ paths: type: array items: type: object - properties: &ref_477 + properties: &ref_479 slot_name: type: string active: @@ -29332,7 +30186,7 @@ paths: application/json: schema: type: object - properties: &ref_266 + properties: &ref_268 name: type: string responses: @@ -29364,7 +30218,7 @@ paths: application/json: schema: type: object - properties: *ref_266 + properties: *ref_268 responses: '200': description: postgres replication slot deleted @@ -29415,7 +30269,7 @@ paths: in: path required: true description: The name of the publication - schema: &ref_269 + schema: &ref_271 type: string responses: '200': @@ -29424,18 +30278,18 @@ paths: application/json: schema: type: object - properties: &ref_270 + properties: &ref_272 table_to_track: type: array items: type: object - properties: *ref_267 - required: *ref_268 + properties: *ref_269 + required: *ref_270 transaction_to_track: type: array items: type: string - required: &ref_271 + required: &ref_273 - transaction_to_track /w/{workspace}/postgres_triggers/publication/create/{publication}/{path}: post: @@ -29456,7 +30310,7 @@ paths: in: path required: true description: The name of the publication - schema: *ref_269 + schema: *ref_271 requestBody: description: new publication for postgres required: true @@ -29464,8 +30318,8 @@ paths: application/json: schema: type: object - properties: *ref_270 - required: *ref_271 + properties: *ref_272 + required: *ref_273 responses: '201': description: publication created @@ -29492,7 +30346,7 @@ paths: in: path required: true description: The name of the publication - schema: *ref_269 + schema: *ref_271 requestBody: description: update publication for postgres required: true @@ -29500,8 +30354,8 @@ paths: application/json: schema: type: object - properties: *ref_270 - required: *ref_271 + properties: *ref_272 + required: *ref_273 responses: '201': description: publication updated @@ -29528,7 +30382,7 @@ paths: in: path required: true description: The name of the publication - schema: *ref_269 + schema: *ref_271 responses: '200': description: postgres publication deleted @@ -29554,7 +30408,7 @@ paths: application/json: schema: type: object - properties: &ref_482 + properties: &ref_484 replication_slot_name: type: string description: Name of the PostgreSQL logical replication slot to use @@ -29582,7 +30436,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 postgres_resource_path: type: string description: >- @@ -29593,8 +30447,8 @@ paths: Configuration for creating/managing the publication (tables, operations) type: object - properties: *ref_270 - required: *ref_271 + properties: *ref_272 + required: *ref_273 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -29605,7 +30459,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -29621,7 +30475,7 @@ paths: type: array items: type: string - required: &ref_483 + required: &ref_485 - path - script_path - is_flow @@ -29656,7 +30510,7 @@ paths: application/json: schema: type: object - properties: &ref_484 + properties: &ref_486 replication_slot_name: type: string description: Name of the PostgreSQL logical replication slot to use @@ -29684,7 +30538,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 postgres_resource_path: type: string description: >- @@ -29695,8 +30549,8 @@ paths: Configuration for creating/managing the publication (tables, operations) type: object - properties: *ref_270 - required: *ref_271 + properties: *ref_272 + required: *ref_273 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -29707,7 +30561,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -29723,7 +30577,7 @@ paths: type: array items: type: string - required: &ref_485 + required: &ref_487 - path - script_path - is_flow @@ -29789,12 +30643,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_272 + - allOf: &ref_274 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_273 + properties: &ref_275 postgres_resource_path: type: string description: >- @@ -29832,8 +30686,8 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_201 - required: &ref_274 + properties: *ref_203 + required: &ref_276 - postgres_resource_path - replication_slot_name - publication_name @@ -29919,7 +30773,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: postgres trigger list @@ -29928,10 +30782,10 @@ paths: schema: type: array items: - allOf: *ref_272 + allOf: *ref_274 type: object - properties: *ref_273 - required: *ref_274 + properties: *ref_275 + required: *ref_276 /w/{workspace}/postgres_triggers/exists/{path}: get: summary: does postgres trigger exists @@ -29980,7 +30834,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -30043,7 +30897,7 @@ paths: application/json: schema: type: object - properties: &ref_494 + properties: &ref_496 path: type: string script_path: @@ -30063,11 +30917,11 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_201 + properties: *ref_203 mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 permissioned_as: type: string description: >- @@ -30083,7 +30937,7 @@ paths: type: array items: type: string - required: &ref_495 + required: &ref_497 - path - script_path - local_part @@ -30117,7 +30971,7 @@ paths: application/json: schema: type: object - properties: &ref_496 + properties: &ref_498 path: type: string script_path: @@ -30137,7 +30991,7 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_201 + properties: *ref_203 permissioned_as: type: string description: >- @@ -30153,7 +31007,7 @@ paths: type: array items: type: string - required: &ref_497 + required: &ref_499 - path - script_path - is_flow @@ -30215,12 +31069,12 @@ paths: application/json: schema: allOf: - - allOf: &ref_275 + - allOf: &ref_277 - type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 type: object - properties: &ref_276 + properties: &ref_278 local_part: type: string workspaced_local_part: @@ -30234,8 +31088,8 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_201 - required: &ref_277 + properties: *ref_203 + required: &ref_279 - local_part - type: object description: > @@ -30319,7 +31173,7 @@ paths: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 responses: '200': description: email trigger list @@ -30328,10 +31182,10 @@ paths: schema: type: array items: - allOf: *ref_275 + allOf: *ref_277 type: object - properties: *ref_276 - required: *ref_277 + properties: *ref_278 + required: *ref_279 /w/{workspace}/email_triggers/exists/{path}: get: summary: does email trigger exists @@ -30413,7 +31267,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 force: type: boolean description: > @@ -30443,9 +31297,9 @@ paths: type: array items: type: object - required: &ref_498 + required: &ref_500 - name - properties: &ref_499 + properties: &ref_501 name: type: string summary: @@ -30475,9 +31329,9 @@ paths: type: array items: type: object - required: &ref_279 + required: &ref_281 - name - properties: &ref_280 + properties: &ref_282 name: type: string summary: @@ -30496,14 +31350,14 @@ paths: type: array items: type: object - properties: &ref_500 + properties: &ref_502 workspace_id: type: string workspace_name: type: string role: type: string - required: &ref_501 + required: &ref_503 - name /groups/get/{name}: get: @@ -30515,7 +31369,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: instance group @@ -30523,8 +31377,8 @@ paths: application/json: schema: type: object - required: *ref_279 - properties: *ref_280 + required: *ref_281 + properties: *ref_282 /groups/create: post: summary: create instance group @@ -30562,7 +31416,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: update instance group required: true @@ -30598,7 +31452,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: instance group deleted @@ -30616,7 +31470,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: user to add to instance group required: true @@ -30646,7 +31500,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: user to remove from instance group required: true @@ -30681,7 +31535,7 @@ paths: type: array items: type: object - properties: &ref_281 + properties: &ref_283 name: type: string summary: @@ -30702,7 +31556,7 @@ paths: enum: - superadmin - devops - required: &ref_282 + required: &ref_284 - name /groups/overwrite: post: @@ -30719,8 +31573,8 @@ paths: type: array items: type: object - properties: *ref_281 - required: *ref_282 + properties: *ref_283 + required: *ref_284 responses: '200': description: success message @@ -30756,7 +31610,7 @@ paths: type: array items: type: object - properties: &ref_283 + properties: &ref_285 name: type: string summary: @@ -30769,7 +31623,7 @@ paths: type: object additionalProperties: type: boolean - required: &ref_284 + required: &ref_286 - name /w/{workspace}/groups/listnames: get: @@ -30842,7 +31696,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: updated group required: true @@ -30874,7 +31728,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: group deleted @@ -30896,7 +31750,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: group @@ -30904,8 +31758,8 @@ paths: application/json: schema: type: object - properties: *ref_283 - required: *ref_284 + properties: *ref_285 + required: *ref_286 /w/{workspace}/groups/adduser/{name}: post: summary: add user to group @@ -30920,7 +31774,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: added user to group required: true @@ -30952,7 +31806,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: added user to group required: true @@ -30984,7 +31838,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 - name: page description: which page to return (start at 1, default 1) in: query @@ -31043,7 +31897,7 @@ paths: type: array items: type: object - properties: &ref_286 + properties: &ref_288 name: type: string owners: @@ -31069,7 +31923,7 @@ paths: (relative to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: &ref_285 + items: &ref_287 type: object required: - path_glob @@ -31098,7 +31952,7 @@ paths: Labels set on the folder. Items inside the folder inherit them, exposed as `inherited_labels` on scripts and flows and stamped into job labels at run time. - required: &ref_287 + required: &ref_289 - name - owners - extra_perms @@ -31165,7 +32019,7 @@ paths: to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: *ref_285 + items: *ref_287 labels: type: array items: @@ -31193,7 +32047,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: update folder required: true @@ -31219,7 +32073,7 @@ paths: to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: *ref_285 + items: *ref_287 labels: type: array items: @@ -31245,7 +32099,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: folder deleted @@ -31267,7 +32121,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: folder @@ -31275,8 +32129,8 @@ paths: application/json: schema: type: object - properties: *ref_286 - required: *ref_287 + properties: *ref_288 + required: *ref_289 /w/{workspace}/folders/exists/{name}: get: summary: exists folder @@ -31291,7 +32145,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: folder exists @@ -31313,7 +32167,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: folder @@ -31355,7 +32209,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: owner user to folder required: true @@ -31389,7 +32243,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: added owner to folder required: true @@ -31425,7 +32279,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 - name: page description: which page to return (start at 1, default 1) in: query @@ -31489,7 +32343,7 @@ paths: type: array items: type: object - properties: &ref_502 + properties: &ref_504 worker: type: string worker_instance: @@ -31535,7 +32389,7 @@ paths: type: string native_mode: type: boolean - required: &ref_503 + required: &ref_505 - worker - worker_instance - ping_at @@ -31700,7 +32554,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: a config @@ -31709,12 +32563,12 @@ paths: schema: type: object nullable: true - properties: &ref_389 + properties: &ref_391 alerts: type: array items: type: object - properties: &ref_387 + properties: &ref_389 name: type: string tags_to_monitor: @@ -31727,7 +32581,7 @@ paths: type: integer alert_time_threshold_seconds: type: integer - required: &ref_388 + required: &ref_390 - name - tags_to_monitor - jobs_num_threshold @@ -31743,7 +32597,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 requestBody: description: worker group required: true @@ -31766,7 +32620,7 @@ paths: - name: name in: path required: true - schema: *ref_278 + schema: *ref_280 responses: '200': description: Delete config @@ -31789,12 +32643,12 @@ paths: type: array items: type: object - properties: &ref_548 + properties: &ref_550 name: type: string config: type: object - required: &ref_549 + required: &ref_551 - name /configs/list_autoscaling_events/{worker_group}: get: @@ -31825,7 +32679,7 @@ paths: type: array items: type: object - properties: &ref_552 + properties: &ref_554 id: type: integer format: int64 @@ -31892,7 +32746,7 @@ paths: type: string language: type: string - enum: *ref_99 + enum: *ref_100 required: - workspace_id - language @@ -31918,7 +32772,7 @@ paths: type: string language: type: string - enum: *ref_99 + enum: *ref_100 workspace_dep_names: type: array items: @@ -32258,7 +33112,7 @@ paths: properties: trigger_kind: type: string - enum: &ref_288 + enum: &ref_290 - webhook - http - websocket @@ -32304,11 +33158,11 @@ paths: required: true schema: type: string - enum: *ref_288 + enum: *ref_290 - name: runnable_kind in: path required: true - schema: *ref_136 + schema: *ref_139 - name: path in: path required: true @@ -32330,7 +33184,7 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_136 + schema: *ref_139 - name: path in: path required: true @@ -32344,17 +33198,17 @@ paths: type: array items: type: object - properties: &ref_553 + properties: &ref_555 trigger_config: {} trigger_kind: type: string - enum: *ref_288 + enum: *ref_290 error: type: string last_server_ping: type: string format: date-time - required: &ref_554 + required: &ref_556 - trigger_kind /w/{workspace}/capture/list/{runnable_kind}/{path}: get: @@ -32370,7 +33224,7 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_136 + schema: *ref_139 - name: path in: path required: true @@ -32379,7 +33233,7 @@ paths: in: query schema: type: string - enum: *ref_288 + enum: *ref_290 - name: page description: which page to return (start at 1, default 1) in: query @@ -32397,10 +33251,10 @@ paths: type: array items: type: object - properties: &ref_289 + properties: &ref_291 trigger_kind: type: string - enum: *ref_288 + enum: *ref_290 main_args: {} preprocessor_args: {} id: @@ -32408,7 +33262,7 @@ paths: created_at: type: string format: date-time - required: &ref_290 + required: &ref_292 - trigger_kind - main_args - preprocessor_args @@ -32428,7 +33282,7 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_136 + schema: *ref_139 - name: path in: path required: true @@ -32473,8 +33327,8 @@ paths: application/json: schema: type: object - properties: *ref_289 - required: *ref_290 + properties: *ref_291 + required: *ref_292 delete: summary: delete a capture operationId: deleteCapture @@ -32566,13 +33420,13 @@ paths: schema: *ref_4 - name: runnable_id in: query - schema: &ref_291 + schema: &ref_293 type: string - name: runnable_type in: query - schema: &ref_292 + schema: &ref_294 type: string - enum: &ref_398 + enum: &ref_400 - ScriptHash - ScriptPath - FlowPath @@ -32589,7 +33443,7 @@ paths: filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: include_preview in: query schema: @@ -32603,7 +33457,7 @@ paths: type: array items: type: object - properties: &ref_293 + properties: &ref_295 id: type: string name: @@ -32617,7 +33471,7 @@ paths: type: boolean success: type: boolean - required: &ref_294 + required: &ref_296 - id - name - args @@ -32667,10 +33521,10 @@ paths: schema: *ref_4 - name: runnable_id in: query - schema: *ref_291 + schema: *ref_293 - name: runnable_type in: query - schema: *ref_292 + schema: *ref_294 - name: page description: which page to return (start at 1, default 1) in: query @@ -32688,8 +33542,8 @@ paths: type: array items: type: object - properties: *ref_293 - required: *ref_294 + properties: *ref_295 + required: *ref_296 /w/{workspace}/inputs/create: post: summary: Create an Input for future use in a script or flow @@ -32703,10 +33557,10 @@ paths: schema: *ref_4 - name: runnable_id in: query - schema: *ref_291 + schema: *ref_293 - name: runnable_type in: query - schema: *ref_292 + schema: *ref_294 requestBody: description: Input required: true @@ -32714,12 +33568,12 @@ paths: application/json: schema: type: object - properties: &ref_394 + properties: &ref_396 name: type: string args: type: object - required: &ref_395 + required: &ref_397 - name - args - created_by @@ -32749,14 +33603,14 @@ paths: application/json: schema: type: object - properties: &ref_396 + properties: &ref_398 id: type: string name: type: string is_public: type: boolean - required: &ref_397 + required: &ref_399 - id - name - is_public @@ -32782,7 +33636,7 @@ paths: - name: input in: path required: true - schema: &ref_317 + schema: &ref_319 type: string responses: '200': @@ -32815,7 +33669,7 @@ paths: properties: s3_resource: type: object - properties: &ref_295 + properties: &ref_297 bucket: type: string region: @@ -32830,7 +33684,7 @@ paths: type: string pathStyle: type: boolean - required: &ref_296 + required: &ref_298 - bucket - region - endPoint @@ -32908,8 +33762,8 @@ paths: properties: s3_resource: type: object - properties: *ref_295 - required: *ref_296 + properties: *ref_297 + required: *ref_298 responses: '200': description: Connection settings @@ -32930,10 +33784,10 @@ paths: type: boolean client_kwargs: type: object - properties: &ref_297 + properties: &ref_299 region_name: type: string - required: &ref_298 + required: &ref_300 - region_name required: - endpoint_url @@ -32988,8 +33842,8 @@ paths: type: boolean client_kwargs: type: object - properties: *ref_297 - required: *ref_298 + properties: *ref_299 + required: *ref_300 required: - endpoint_url - use_ssl @@ -33047,8 +33901,8 @@ paths: application/json: schema: type: object - properties: *ref_295 - required: *ref_296 + properties: *ref_297 + required: *ref_298 /w/{workspace}/job_helpers/test_connection: get: summary: Test connection to the workspace object storage @@ -33112,10 +33966,10 @@ paths: type: array items: type: object - properties: &ref_299 + properties: &ref_301 s3: type: string - required: &ref_300 + required: &ref_302 - s3 restricted_access: type: boolean @@ -33148,7 +34002,7 @@ paths: application/json: schema: type: object - properties: &ref_303 + properties: &ref_305 mime_type: type: string size_in_bytes: @@ -33212,7 +34066,7 @@ paths: application/json: schema: type: object - properties: &ref_301 + properties: &ref_303 msg: type: string content: @@ -33224,7 +34078,7 @@ paths: - Csv - Parquet - Unknown - required: &ref_302 + required: &ref_304 - content_type /w/{workspace}/job_helpers/list_git_repo_files: get: @@ -33272,8 +34126,8 @@ paths: type: array items: type: object - properties: *ref_299 - required: *ref_300 + properties: *ref_301 + required: *ref_302 restricted_access: type: boolean required: @@ -33332,8 +34186,8 @@ paths: application/json: schema: type: object - properties: *ref_301 - required: *ref_302 + properties: *ref_303 + required: *ref_304 /w/{workspace}/job_helpers/load_git_repo_file_metadata: get: summary: >- @@ -33364,7 +34218,7 @@ paths: application/json: schema: type: object - properties: *ref_303 + properties: *ref_305 /w/{workspace}/job_helpers/check_s3_folder_exists: get: summary: Check if S3 path exists and is a folder @@ -33815,7 +34669,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 requestBody: description: parameters for statistics retrieval required: true @@ -33844,46 +34698,46 @@ paths: type: array items: type: object - properties: &ref_534 + properties: &ref_536 id: type: string name: type: string - required: &ref_535 - - id - scalar_metrics: - type: array - items: - type: object - properties: &ref_536 - metric_id: - type: string - value: - type: number required: &ref_537 - id - - value - timeseries_metrics: + scalar_metrics: type: array items: type: object properties: &ref_538 + metric_id: + type: string + value: + type: number + required: &ref_539 + - id + - value + timeseries_metrics: + type: array + items: + type: object + properties: &ref_540 metric_id: type: string values: type: array items: type: object - properties: &ref_540 + properties: &ref_542 timestamp: type: string format: date-time value: type: number - required: &ref_541 + required: &ref_543 - timestamp - value - required: &ref_539 + required: &ref_541 - id - values /w/{workspace}/job_metrics/set_progress/{id}: @@ -33900,7 +34754,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 requestBody: description: parameters for statistics retrieval required: true @@ -33934,7 +34788,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: job progress between 0 and 99 @@ -33952,11 +34806,11 @@ paths: - name: before description: filter on started before (inclusive) timestamp in: query - schema: *ref_304 + schema: *ref_306 - name: after description: filter on created after (exclusive) timestamp in: query - schema: *ref_305 + schema: *ref_307 - name: with_error in: query required: false @@ -34028,12 +34882,12 @@ paths: type: array items: type: object - properties: &ref_542 + properties: &ref_544 concurrency_key: type: string total_running: type: number - required: &ref_543 + required: &ref_545 - concurrency_key - total_running /concurrency_groups/prune/{concurrency_id}: @@ -34046,7 +34900,7 @@ paths: - name: concurrency_id in: path required: true - schema: &ref_319 + schema: &ref_321 type: string responses: '200': @@ -34066,7 +34920,7 @@ paths: - name: id in: path required: true - schema: *ref_176 + schema: *ref_178 responses: '200': description: concurrency key for given job @@ -34102,104 +34956,104 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_178 + schema: *ref_180 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_161 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_162 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_163 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_164 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_165 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_166 + schema: *ref_168 - name: running description: filter on running jobs in: query - schema: *ref_167 + schema: *ref_169 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_168 + schema: *ref_170 - name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_188 + schema: *ref_190 - name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_189 + schema: *ref_191 - name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: *ref_190 + schema: *ref_192 - name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: *ref_191 + schema: *ref_193 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_169 + schema: *ref_171 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_172 + schema: *ref_174 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_173 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_175 + schema: *ref_177 - name: page description: which page to return (start at 1, default 1) in: query @@ -34215,7 +35069,7 @@ paths: (e.g. '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: *ref_192 + schema: *ref_194 - name: is_skipped description: is the job skipped in: query @@ -34268,17 +35122,17 @@ paths: application/json: schema: type: object - properties: &ref_544 + properties: &ref_546 jobs: type: array items: - oneOf: *ref_197 - discriminator: *ref_198 + oneOf: *ref_199 + discriminator: *ref_200 obscured_jobs: type: array items: type: object - properties: &ref_399 + properties: &ref_401 typ: type: string started_at: @@ -34291,7 +35145,7 @@ paths: Obscured jobs omitted for security because of too specific filtering type: boolean - required: &ref_545 + required: &ref_547 - jobs - obscured_jobs /srch/w/{workspace}/index/search/job: @@ -34335,7 +35189,7 @@ paths: type: array items: type: object - properties: &ref_550 + properties: &ref_552 dancer: type: string hit_count: @@ -34414,7 +35268,7 @@ paths: type: array items: type: object - properties: &ref_551 + properties: &ref_553 dancer: type: string /srch/index/search/count_service_logs: @@ -34689,7 +35543,7 @@ paths: type: string kind: type: string - enum: *ref_306 + enum: *ref_308 usages: type: array items: @@ -34702,13 +35556,13 @@ paths: type: string kind: type: string - enum: &ref_308 + enum: &ref_310 - script - flow - job access_type: type: string - enum: &ref_307 + enum: &ref_309 - r - w - rw @@ -34718,7 +35572,7 @@ paths: description: The columns used (for tables) additionalProperties: type: string - enum: *ref_307 + enum: *ref_309 nullable: true created_at: type: string @@ -34791,7 +35645,7 @@ paths: type: string kind: type: string - enum: *ref_308 + enum: *ref_310 responses: '200': description: all assets used by the given usage paths, in the same order @@ -34811,10 +35665,10 @@ paths: type: string kind: type: string - enum: *ref_306 + enum: *ref_308 access_type: type: string - enum: *ref_307 + enum: *ref_309 nullable: true /w/{workspace}/assets/list_favorites: get: @@ -34842,6 +35696,181 @@ paths: path: type: string description: The asset path + /w/{workspace}/assets/graph: + get: + summary: Get the workspace-wide asset <-> runnable graph + operationId: getAssetsGraph + tags: + - asset + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: asset_kinds + in: query + description: Filter by asset kinds (comma-separated list) + schema: + type: string + - name: folder + in: query + description: Scope the graph to runnables in a single folder + schema: + type: string + responses: + '200': + description: asset graph nodes, lineage edges and trigger edges + content: + application/json: + schema: + type: object + required: + - assets + - runnables + - edges + - triggers + properties: + assets: + type: array + items: + type: object + required: + - kind + - path + properties: + kind: + type: string + enum: *ref_308 + path: + type: string + runnables: + type: array + items: + type: object + required: + - path + - usage_kind + properties: + path: + type: string + usage_kind: + type: string + enum: *ref_310 + in_pipeline: + type: boolean + description: >- + True iff the script is a pipeline member (deployed + with `// pipeline`). Omitted when false. + edges: + type: array + items: + type: object + required: + - runnable_path + - runnable_kind + - asset_kind + - asset_path + properties: + runnable_path: + type: string + runnable_kind: + type: string + enum: *ref_310 + asset_kind: + type: string + enum: *ref_308 + asset_path: + type: string + access_type: + type: string + enum: *ref_309 + nullable: true + triggers: + type: array + items: + oneOf: + - type: object + description: Asset trigger edge (`// on `) + required: + - trigger_kind + - asset_kind + - asset_path + - runnable_kind + - runnable_path + properties: + trigger_kind: + type: string + enum: + - asset + asset_kind: + type: string + enum: *ref_308 + asset_path: + type: string + runnable_kind: + type: string + enum: *ref_310 + runnable_path: + type: string + - type: object + description: >- + Native trigger edge (schedule, email, kafka, ...). + `path` is the trigger row's path. + required: + - trigger_kind + - path + - runnable_kind + - runnable_path + properties: + trigger_kind: + type: string + enum: + - schedule + - email + - kafka + - mqtt + - nats + - postgres + - sqs + - gcp + path: + type: string + runnable_kind: + type: string + enum: *ref_310 + runnable_path: + type: string + /w/{workspace}/assets/pipelines: + get: + summary: List folders that contain at least one pipeline-member script + operationId: listPipelineFolders + tags: + - asset + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + responses: + '200': + description: folders containing pipeline scripts, with their script counts + content: + application/json: + schema: + type: array + items: + type: object + required: + - folder + - script_count + properties: + folder: + type: string + description: The folder name (without the `f/` prefix) + script_count: + type: integer + format: int64 + description: Number of pipeline-member scripts in the folder /w/{workspace}/volumes/list: get: summary: List all volumes in the workspace @@ -34862,13 +35891,13 @@ paths: type: array items: type: object - required: &ref_564 + required: &ref_566 - name - size_bytes - file_count - created_at - created_by - properties: &ref_565 + properties: &ref_567 name: type: string size_bytes: @@ -34983,13 +36012,13 @@ paths: type: array items: type: object - required: &ref_381 + required: &ref_383 - name - description - instructions - path - method - properties: &ref_382 + properties: &ref_384 name: type: string description: The tool name/operation ID @@ -35140,12 +36169,12 @@ components: so the home page can render a "Draft" badge. Gated to non-operators + page 0 + no narrowing filters on the backend so picker callers stay deployed-only and pagination stays clean. - schema: *ref_212 + schema: *ref_214 Id: name: id in: path required: true - schema: *ref_309 + schema: *ref_311 Key: name: key in: path @@ -35161,7 +36190,7 @@ components: in: path required: true description: The name of the publication - schema: *ref_269 + schema: *ref_271 VersionId: name: version in: path @@ -35172,7 +36201,7 @@ components: name: token in: path required: true - schema: *ref_310 + schema: *ref_312 AccountId: name: id in: path @@ -35187,17 +36216,17 @@ components: name: path in: path required: true - schema: *ref_97 + schema: *ref_98 ScriptHash: name: hash in: path required: true - schema: *ref_107 + schema: *ref_108 JobId: name: id in: path required: true - schema: *ref_176 + schema: *ref_178 Path: name: path in: path @@ -35207,7 +36236,7 @@ components: name: custom_path in: path required: true - schema: *ref_137 + schema: *ref_97 PathId: name: id in: path @@ -35217,12 +36246,12 @@ components: name: version in: path required: true - schema: *ref_311 + schema: *ref_313 Name: name: name in: path required: true - schema: *ref_278 + schema: *ref_280 Page: name: page description: which page to return (start at 1, default 1) @@ -35241,12 +36270,12 @@ components: '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: *ref_192 + schema: *ref_194 OrderDesc: name: order_desc description: order by desc order (default true) in: query - schema: *ref_122 + schema: *ref_123 CreatedBy: name: created_by description: >- @@ -35254,7 +36283,7 @@ components: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_123 + schema: *ref_124 Label: name: label description: >- @@ -35262,7 +36291,7 @@ components: 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_178 + schema: *ref_180 Worker: name: worker description: >- @@ -35270,26 +36299,26 @@ components: 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_160 + schema: *ref_162 ParentJob: name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_114 + schema: *ref_115 WorkerTag: name: tag description: Override the tag to use in: query - schema: *ref_115 + schema: *ref_116 CacheTtl: name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_116 + schema: *ref_117 NewJobId: name: job_id description: >- @@ -35297,7 +36326,7 @@ components: randomly using the ULID scheme. If a job id already exists in the queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_117 + schema: *ref_118 IncludeHeader: name: include_header description: > @@ -35307,19 +36336,19 @@ components: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_118 + schema: *ref_119 QueueLimit: name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_119 + schema: *ref_120 SkipPreprocessor: name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_120 + schema: *ref_121 Payload: name: payload description: > @@ -35328,7 +36357,7 @@ components: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_121 + schema: *ref_122 ScriptStartPath: name: script_path_start description: >- @@ -35336,12 +36365,12 @@ components: 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_162 + schema: *ref_164 SchedulePath: name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_163 + schema: *ref_165 TriggerPath: name: trigger_path description: >- @@ -35349,7 +36378,7 @@ components: 'f/trigger1,f/trigger2') and negation by prefixing all values with '!' (e.g. '!f/trigger1,!f/trigger2') in: query - schema: *ref_312 + schema: *ref_314 ScriptExactPath: name: script_path_exact description: >- @@ -35357,87 +36386,87 @@ components: (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_161 + schema: *ref_163 ScriptExactHash: name: script_hash description: mask to filter exact matching path in: query - schema: *ref_164 + schema: *ref_166 CreatedBefore: name: created_before description: filter on created before (inclusive) timestamp in: query - schema: *ref_186 + schema: *ref_188 CreatedAfter: name: created_after description: filter on created after (exclusive) timestamp in: query - schema: *ref_187 + schema: *ref_189 StartedBefore: name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_165 + schema: *ref_167 StartedAfter: name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_166 + schema: *ref_168 Before: name: before description: filter on started before (inclusive) timestamp in: query - schema: *ref_304 + schema: *ref_306 CompletedBefore: name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_188 + schema: *ref_190 CompletedAfter: name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_189 + schema: *ref_191 CreatedAfterQueue: name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: *ref_191 + schema: *ref_193 CreatedBeforeQueue: name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: *ref_190 + schema: *ref_192 Success: name: success description: filter on successful jobs in: query - schema: *ref_174 + schema: *ref_176 ScheduledForBeforeNow: name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_168 + schema: *ref_170 Suspended: name: suspended description: filter on suspended jobs in: query - schema: *ref_170 + schema: *ref_172 Running: name: running description: filter on running jobs in: query - schema: *ref_167 + schema: *ref_169 AllowWildcards: name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_175 + schema: *ref_177 ArgsFilter: name: args description: filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_171 + schema: *ref_173 Tag: name: tag description: >- @@ -35445,37 +36474,37 @@ components: 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_172 + schema: *ref_174 ResultFilter: name: result description: filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_173 + schema: *ref_175 After: name: after description: filter on created after (exclusive) timestamp in: query - schema: *ref_305 + schema: *ref_307 Username: name: username description: filter on exact username of user in: query - schema: *ref_313 + schema: *ref_315 Operation: name: operation description: filter on exact or prefix name of operation in: query - schema: *ref_314 + schema: *ref_316 ResourceName: name: resource description: filter on exact or prefix name of resource in: query - schema: *ref_315 + schema: *ref_317 ActionKind: name: action_kind description: filter on type of operation in: query - schema: *ref_316 + schema: *ref_318 JobKinds: name: job_kinds description: >- @@ -35483,34 +36512,34 @@ components: 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_169 + schema: *ref_171 RunnableId: name: runnable_id in: query - schema: *ref_291 + schema: *ref_293 RunnableTypeQuery: name: runnable_type in: query - schema: *ref_292 + schema: *ref_294 InputId: name: input in: path required: true - schema: *ref_317 + schema: *ref_319 GetStarted: name: get_started in: query - schema: *ref_318 + schema: *ref_320 ConcurrencyId: name: concurrency_id in: path required: true - schema: *ref_319 + schema: *ref_321 RunnableKind: name: runnable_kind in: path required: true - schema: *ref_136 + schema: *ref_139 schemas: UserDraftOverlay: type: object @@ -35534,25 +36563,25 @@ components: description: | Closed set of item kinds a user can autosave as a draft. Mirrors the Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. - enum: *ref_100 + enum: *ref_101 OpenFlow: type: object description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: *ref_124 - required: *ref_125 + properties: *ref_125 + required: *ref_126 FlowValue: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_153 - required: *ref_154 + properties: *ref_155 + required: *ref_156 Retry: type: object description: Retry configuration for failed module executions - properties: *ref_201 + properties: *ref_203 StopAfterIf: type: object description: Early termination condition for a module @@ -35702,7 +36731,7 @@ components: retry: description: Retry configuration for failed module executions type: object - properties: *ref_320 + properties: *ref_322 debouncing: description: Debounce configuration for this step (EE only) type: object @@ -35735,8 +36764,8 @@ components: description: >- Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_142 - discriminator: *ref_143 + oneOf: *ref_144 + discriminator: *ref_145 StaticTransform: type: object description: >- @@ -35793,7 +36822,7 @@ components: kind: type: string description: Supported AI provider types - enum: *ref_321 + enum: *ref_323 resource: type: string description: >- @@ -35813,16 +36842,16 @@ components: oneOf: - type: object description: No conversation memory/context - properties: *ref_322 - required: *ref_323 - - type: object - description: Automatic context management properties: *ref_324 required: *ref_325 - type: object - description: Explicit message history + description: Automatic context management properties: *ref_326 required: *ref_327 + - type: object + description: Explicit message history + properties: *ref_328 + required: *ref_329 discriminator: propertyName: kind mapping: @@ -35839,62 +36868,62 @@ components: Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms - properties: *ref_328 - required: *ref_329 - - type: object - description: >- - Reference to an existing script by path. Use this when calling a - previously saved script instead of writing inline code properties: *ref_330 required: *ref_331 - type: object description: >- - Reference to an existing flow by path. Use this to call another flow - as a subflow + Reference to an existing script by path. Use this when calling a + previously saved script instead of writing inline code properties: *ref_332 required: *ref_333 + - type: object + description: >- + Reference to an existing flow by path. Use this to call another flow + as a subflow + properties: *ref_334 + required: *ref_335 - type: object description: >- Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations - properties: *ref_334 - required: *ref_335 + properties: *ref_336 + required: *ref_337 - type: object description: >- Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination - properties: *ref_336 - required: *ref_337 + properties: *ref_338 + required: *ref_339 - type: object description: >- Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes - properties: *ref_338 - required: *ref_339 + properties: *ref_340 + required: *ref_341 - type: object description: >- Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently - properties: *ref_340 - required: *ref_341 + properties: *ref_342 + required: *ref_343 - type: object description: >- Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder - properties: *ref_342 - required: *ref_343 + properties: *ref_344 + required: *ref_345 - type: object description: >- AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task - properties: *ref_344 - required: *ref_345 + properties: *ref_346 + required: *ref_347 discriminator: propertyName: type mapping: @@ -36300,8 +37329,8 @@ components: description: >- Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined - oneOf: *ref_346 - discriminator: *ref_347 + oneOf: *ref_348 + discriminator: *ref_349 output_type: allOf: - description: >- @@ -36354,8 +37383,8 @@ components: description: >- Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined - oneOf: *ref_348 - discriminator: *ref_349 + oneOf: *ref_350 + discriminator: *ref_351 output_schema: allOf: - description: >- @@ -36444,8 +37473,8 @@ components: description: >- A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool - properties: *ref_350 - required: *ref_351 + properties: *ref_352 + required: *ref_353 type: type: string enum: @@ -36485,8 +37514,8 @@ components: - type FlowStatus: type: object - properties: *ref_179 - required: *ref_180 + properties: *ref_181 + required: *ref_182 FlowStatusModule: type: object properties: @@ -36708,8 +37737,8 @@ components: - type CiTestResult: type: object - properties: *ref_112 - required: *ref_113 + properties: *ref_113 + required: *ref_114 HealthStatusResponse: type: object description: Health status response (cached with 5s TTL) @@ -36723,75 +37752,75 @@ components: HealthChecks: type: object description: Detailed health checks - required: *ref_352 - properties: *ref_353 + required: *ref_354 + properties: *ref_355 DatabaseHealth: type: object description: Database health status - required: *ref_354 - properties: *ref_355 + required: *ref_356 + properties: *ref_357 PoolStats: type: object description: Database connection pool statistics - required: *ref_356 - properties: *ref_357 + required: *ref_358 + properties: *ref_359 WorkersHealth: type: object description: Workers health status - required: *ref_358 - properties: *ref_359 + required: *ref_360 + properties: *ref_361 QueueHealth: type: object description: Job queue status - required: *ref_360 - properties: *ref_361 + required: *ref_362 + properties: *ref_363 ReadinessHealth: type: object description: Server readiness status - required: *ref_362 - properties: *ref_363 + required: *ref_364 + properties: *ref_365 AutoInviteConfig: type: object description: Configuration for auto-inviting users to the workspace - properties: *ref_364 + properties: *ref_366 ErrorHandlerConfig: type: object description: Configuration for the workspace error handler - properties: *ref_365 + properties: *ref_367 SuccessHandlerConfig: type: object description: Configuration for the workspace success handler - properties: *ref_366 + properties: *ref_368 EditErrorHandler: description: >- Request body for editing the workspace error handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: *ref_367 + oneOf: *ref_369 EditErrorHandlerNew: type: object description: New grouped format for editing error handler - properties: *ref_368 + properties: *ref_370 EditErrorHandlerLegacy: type: object description: >- Legacy flat format for editing error handler (deprecated, use new format) - properties: *ref_369 + properties: *ref_371 EditSuccessHandler: description: >- Request body for editing the workspace success handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: *ref_370 + oneOf: *ref_372 EditSuccessHandlerNew: type: object description: New grouped format for editing success handler - properties: *ref_371 + properties: *ref_373 EditSuccessHandlerLegacy: type: object description: >- Legacy flat format for editing success handler (deprecated, use new format) - properties: *ref_372 + properties: *ref_374 VaultSettings: type: object required: *ref_27 @@ -36806,28 +37835,28 @@ components: properties: *ref_34 SecretMigrationFailure: type: object - required: *ref_373 - properties: *ref_374 + required: *ref_375 + properties: *ref_376 SecretMigrationReport: type: object required: *ref_29 properties: *ref_30 JwksResponse: - type: object - required: *ref_375 - properties: *ref_376 - FlowConversation: type: object required: *ref_377 properties: *ref_378 - FlowConversationMessage: + FlowConversation: type: object required: *ref_379 properties: *ref_380 - EndpointTool: + FlowConversationMessage: type: object required: *ref_381 properties: *ref_382 + EndpointTool: + type: object + required: *ref_383 + properties: *ref_384 AIProvider: type: string enum: *ref_51 @@ -36840,112 +37869,112 @@ components: required: *ref_44 AIProviderConfig: type: object - properties: *ref_383 - required: *ref_384 + properties: *ref_385 + required: *ref_386 AIConfig: type: object properties: *ref_50 InstanceAIProviderSummary: type: object - properties: *ref_385 - required: *ref_386 + properties: *ref_387 + required: *ref_388 InstanceAISummary: type: object properties: *ref_52 required: *ref_53 Alert: type: object - properties: *ref_387 - required: *ref_388 + properties: *ref_389 + required: *ref_390 Configs: type: object nullable: true - properties: *ref_389 + properties: *ref_391 WorkspaceDependencies: type: object - properties: *ref_103 - required: *ref_104 + properties: *ref_104 + required: *ref_105 NewWorkspaceDependencies: - type: object - properties: *ref_390 - required: *ref_391 - Script: - type: object - properties: *ref_105 - required: *ref_106 - NewScript: type: object properties: *ref_392 required: *ref_393 + Script: + type: object + properties: *ref_106 + required: *ref_107 + NewScript: + type: object + properties: *ref_394 + required: *ref_395 ScriptHistory: type: object - properties: *ref_110 - required: *ref_111 + properties: *ref_111 + required: *ref_112 ScriptArgs: type: object description: The arguments to pass to the script or flow additionalProperties: true Input: type: object - properties: *ref_293 - required: *ref_294 + properties: *ref_295 + required: *ref_296 CreateInput: - type: object - properties: *ref_394 - required: *ref_395 - UpdateInput: type: object properties: *ref_396 required: *ref_397 + UpdateInput: + type: object + properties: *ref_398 + required: *ref_399 RunnableType: type: string - enum: *ref_398 + enum: *ref_400 QueuedJob: + type: object + properties: *ref_197 + required: *ref_198 + CompletedJob: type: object properties: *ref_195 required: *ref_196 - CompletedJob: - type: object - properties: *ref_193 - required: *ref_194 ExportableCompletedJob: type: object description: Completed job with full data for export/import operations - properties: *ref_182 - required: *ref_183 + properties: *ref_184 + required: *ref_185 ExportableQueuedJob: type: object description: Queued job with full data for export/import operations - properties: *ref_184 - required: *ref_185 + properties: *ref_186 + required: *ref_187 ObscuredJob: type: object - properties: *ref_399 + properties: *ref_401 Job: - oneOf: *ref_197 - discriminator: *ref_198 + oneOf: *ref_199 + discriminator: *ref_200 User: type: object properties: *ref_35 required: *ref_36 UserSource: type: object - properties: *ref_400 - required: *ref_401 + properties: *ref_402 + required: *ref_403 UserUsage: type: object - properties: *ref_402 + properties: *ref_404 Login: type: object - properties: *ref_403 - required: *ref_404 + properties: *ref_405 + required: *ref_406 PasswordResetResponse: type: object properties: *ref_7 required: *ref_8 EditWorkspaceUser: type: object - properties: *ref_405 + properties: *ref_407 OffboardAffectedPaths: type: object properties: *ref_11 @@ -36954,65 +37983,65 @@ components: properties: *ref_12 required: *ref_13 OffboardTokenInfo: - type: object - properties: *ref_406 - required: *ref_407 - OffboardRequest: type: object properties: *ref_408 required: *ref_409 + OffboardRequest: + type: object + properties: *ref_410 + required: *ref_411 OffboardResponse: type: object properties: *ref_14 OffboardSummary: - type: object - properties: *ref_410 - required: *ref_411 - GlobalOffboardPreview: type: object properties: *ref_412 required: *ref_413 - WorkspaceOffboardPreview: + GlobalOffboardPreview: type: object properties: *ref_414 required: *ref_415 - GlobalOffboardRequest: + WorkspaceOffboardPreview: type: object properties: *ref_416 + required: *ref_417 + GlobalOffboardRequest: + type: object + properties: *ref_418 WorkspaceReassignment: - type: object - properties: *ref_417 - required: *ref_418 - TruncatedToken: - type: object - properties: *ref_108 - required: *ref_109 - ExternalJwtToken: type: object properties: *ref_419 required: *ref_420 - NewToken: + TruncatedToken: + type: object + properties: *ref_109 + required: *ref_110 + ExternalJwtToken: type: object properties: *ref_421 + required: *ref_422 + NewToken: + type: object + properties: *ref_423 NewTokenImpersonate: type: object - properties: *ref_422 - required: *ref_423 + properties: *ref_424 + required: *ref_425 ListableVariable: type: object properties: *ref_61 required: *ref_62 ContextualVariable: - type: object - properties: *ref_424 - required: *ref_425 - CreateVariable: type: object properties: *ref_426 required: *ref_427 - EditVariable: + CreateVariable: type: object properties: *ref_428 + required: *ref_429 + EditVariable: + type: object + properties: *ref_430 AuditLog: type: object properties: *ref_5 @@ -37149,42 +38178,42 @@ components: - has_preprocessor ScriptLang: type: string - enum: *ref_99 + enum: *ref_100 ScriptModule: type: object description: An additional module file associated with a script - properties: *ref_101 - required: *ref_102 + properties: *ref_102 + required: *ref_103 Preview: type: object - properties: *ref_145 - required: *ref_146 + properties: *ref_147 + required: *ref_148 PreviewInline: - type: object - properties: *ref_429 - required: *ref_430 - InlineScriptArgs: - type: object - properties: *ref_144 - WorkflowTask: type: object properties: *ref_431 required: *ref_432 + InlineScriptArgs: + type: object + properties: *ref_146 + WorkflowTask: + type: object + properties: *ref_433 + required: *ref_434 WorkflowStatusRecord: type: object additionalProperties: type: object - properties: *ref_181 + properties: *ref_183 WorkflowStatus: type: object - properties: *ref_181 + properties: *ref_183 CreateResource: type: object - properties: *ref_433 - required: *ref_434 + properties: *ref_435 + required: *ref_436 EditResource: type: object - properties: *ref_435 + properties: *ref_437 Resource: type: object properties: @@ -37235,13 +38264,13 @@ components: required: *ref_83 EditResourceType: type: object - properties: *ref_436 + properties: *ref_438 Schedule: type: object - properties: *ref_202 - required: *ref_203 + properties: *ref_204 + required: *ref_205 ScheduleWJobs: - allOf: *ref_437 + allOf: *ref_439 ErrorHandler: type: string enum: @@ -37250,122 +38279,122 @@ components: - teams - email NewSchedule: - type: object - properties: *ref_438 - required: *ref_439 - EditSchedule: type: object properties: *ref_440 required: *ref_441 + EditSchedule: + type: object + properties: *ref_442 + required: *ref_443 JobTriggerKind: description: job trigger kind (schedule, http, websocket...) type: string - enum: *ref_177 + enum: *ref_179 TriggerMode: description: job trigger mode type: string - enum: *ref_211 + enum: *ref_213 TriggerExtraProperty: type: object - properties: *ref_219 - required: *ref_220 + properties: *ref_221 + required: *ref_222 AuthenticationMethod: type: string - enum: *ref_210 + enum: *ref_212 RunnableKind: type: string - enum: *ref_204 + enum: *ref_206 OpenapiSpecFormat: type: string - enum: *ref_442 + enum: *ref_444 OpenapiHttpRouteFilters: - type: object - properties: *ref_443 - required: *ref_444 - WebhookFilters: type: object properties: *ref_445 required: *ref_446 - OpenapiV3Info: + WebhookFilters: type: object properties: *ref_447 required: *ref_448 - GenerateOpenapiSpec: - type: object - properties: *ref_205 - HttpMethod: - type: string - enum: *ref_208 - HttpRequestType: - type: string - enum: *ref_209 - HttpTrigger: - allOf: *ref_213 - type: object - properties: *ref_214 - required: *ref_215 - NewHttpTrigger: - type: object - properties: *ref_206 - required: *ref_207 - EditHttpTrigger: + OpenapiV3Info: type: object properties: *ref_449 required: *ref_450 - TriggersCount: + GenerateOpenapiSpec: type: object - properties: *ref_129 - WebsocketHeartbeat: + properties: *ref_207 + HttpMethod: + type: string + enum: *ref_210 + HttpRequestType: + type: string + enum: *ref_211 + HttpTrigger: + allOf: *ref_215 type: object - properties: *ref_217 - required: *ref_218 - WebsocketTrigger: - allOf: *ref_221 + properties: *ref_216 + required: *ref_217 + NewHttpTrigger: type: object - properties: *ref_222 - required: *ref_223 - NewWebsocketTrigger: + properties: *ref_208 + required: *ref_209 + EditHttpTrigger: type: object properties: *ref_451 required: *ref_452 - EditWebsocketTrigger: + TriggersCount: + type: object + properties: *ref_130 + WebsocketHeartbeat: + type: object + properties: *ref_219 + required: *ref_220 + WebsocketTrigger: + allOf: *ref_223 + type: object + properties: *ref_224 + required: *ref_225 + NewWebsocketTrigger: type: object properties: *ref_453 required: *ref_454 + EditWebsocketTrigger: + type: object + properties: *ref_455 + required: *ref_456 WebsocketTriggerInitialMessage: - anyOf: *ref_216 + anyOf: *ref_218 MqttQoS: type: string - enum: *ref_455 + enum: *ref_457 MqttV3Config: type: object - properties: *ref_243 + properties: *ref_245 MqttV5Config: type: object - properties: *ref_244 + properties: *ref_246 MqttSubscribeTopic: type: object - properties: *ref_241 - required: *ref_242 + properties: *ref_243 + required: *ref_244 MqttClientVersion: type: string - enum: *ref_245 + enum: *ref_247 MqttTrigger: - allOf: *ref_246 + allOf: *ref_248 type: object - properties: *ref_247 - required: *ref_248 + properties: *ref_249 + required: *ref_250 NewMqttTrigger: - type: object - properties: *ref_456 - required: *ref_457 - EditMqttTrigger: type: object properties: *ref_458 required: *ref_459 + EditMqttTrigger: + type: object + properties: *ref_460 + required: *ref_461 DeliveryType: type: string - enum: *ref_251 + enum: *ref_253 description: >- Delivery mode for messages. 'push' for HTTP push delivery where messages are sent to a webhook endpoint, 'pull' for polling where the trigger @@ -37373,19 +38402,19 @@ components: PushConfig: type: object description: Configuration for push delivery mode. - properties: *ref_252 - required: *ref_253 + properties: *ref_254 + required: *ref_255 GcpTrigger: - allOf: *ref_255 + allOf: *ref_257 type: object description: >- A Google Cloud Pub/Sub trigger that executes a script or flow when messages are received. - properties: *ref_256 - required: *ref_257 + properties: *ref_258 + required: *ref_259 SubscriptionMode: type: string - enum: *ref_254 + enum: *ref_256 description: >- The mode of subscription. 'existing' means using an existing GCP subscription, while 'create_update' involves creating or updating a new @@ -37393,68 +38422,68 @@ components: GcpTriggerData: type: object description: Data for creating or updating a Google Cloud Pub/Sub trigger. - properties: *ref_249 - required: *ref_250 + properties: *ref_251 + required: *ref_252 GetAllTopicSubscription: - type: object - properties: *ref_460 - required: *ref_461 - DeleteGcpSubscription: type: object properties: *ref_462 required: *ref_463 + DeleteGcpSubscription: + type: object + properties: *ref_464 + required: *ref_465 AzureMode: type: string - enum: *ref_260 + enum: *ref_262 description: Azure Event Grid trigger mode. AzureArmResource: type: object description: An ARM resource the service principal can see. - properties: *ref_264 - required: *ref_265 + properties: *ref_266 + required: *ref_267 AzureDeleteSubscription: type: object - properties: *ref_464 - required: *ref_465 + properties: *ref_466 + required: *ref_467 AzureTrigger: - allOf: *ref_261 + allOf: *ref_263 type: object description: >- An Azure Event Grid trigger that executes a script or flow when events arrive. - properties: *ref_262 - required: *ref_263 + properties: *ref_264 + required: *ref_265 AzureTriggerData: type: object description: Data for creating or updating an Azure Event Grid trigger. - properties: *ref_258 - required: *ref_259 + properties: *ref_260 + required: *ref_261 TestAzureConnection: - type: object - properties: *ref_466 - required: *ref_467 - AzureListTopics: type: object properties: *ref_468 required: *ref_469 - AzureListSubscriptions: + AzureListTopics: type: object properties: *ref_470 required: *ref_471 + AzureListSubscriptions: + type: object + properties: *ref_472 + required: *ref_473 AwsAuthResourceType: type: string - enum: *ref_230 + enum: *ref_232 SqsTrigger: - allOf: *ref_231 + allOf: *ref_233 type: object - properties: *ref_232 - required: *ref_233 + properties: *ref_234 + required: *ref_235 LoggedWizardStatus: type: string enum: *ref_21 CustomInstanceDbLogs: type: object - properties: *ref_472 + properties: *ref_474 CustomInstanceDbTag: type: string enum: *ref_22 @@ -37463,109 +38492,109 @@ components: required: *ref_23 properties: *ref_24 NewSqsTrigger: - type: object - properties: *ref_473 - required: *ref_474 - EditSqsTrigger: type: object properties: *ref_475 required: *ref_476 - Slot: - type: object - properties: *ref_266 - SlotList: + EditSqsTrigger: type: object properties: *ref_477 + required: *ref_478 + Slot: + type: object + properties: *ref_268 + SlotList: + type: object + properties: *ref_479 PublicationData: type: object - properties: *ref_270 - required: *ref_271 + properties: *ref_272 + required: *ref_273 TableToTrack: type: array - items: *ref_478 + items: *ref_480 Relations: type: object - properties: *ref_267 - required: *ref_268 + properties: *ref_269 + required: *ref_270 Language: type: string - enum: *ref_479 + enum: *ref_481 TemplateScript: - type: object - properties: *ref_480 - required: *ref_481 - PostgresTrigger: - allOf: *ref_272 - type: object - properties: *ref_273 - required: *ref_274 - NewPostgresTrigger: type: object properties: *ref_482 required: *ref_483 - EditPostgresTrigger: + PostgresTrigger: + allOf: *ref_274 + type: object + properties: *ref_275 + required: *ref_276 + NewPostgresTrigger: type: object properties: *ref_484 required: *ref_485 - KafkaTrigger: - allOf: *ref_224 - type: object - properties: *ref_225 - required: *ref_226 - NewKafkaTrigger: + EditPostgresTrigger: type: object properties: *ref_486 required: *ref_487 - EditKafkaTrigger: + KafkaTrigger: + allOf: *ref_226 + type: object + properties: *ref_227 + required: *ref_228 + NewKafkaTrigger: type: object properties: *ref_488 required: *ref_489 - NatsTrigger: - allOf: *ref_227 - type: object - properties: *ref_228 - required: *ref_229 - NewNatsTrigger: + EditKafkaTrigger: type: object properties: *ref_490 required: *ref_491 - EditNatsTrigger: + NatsTrigger: + allOf: *ref_229 + type: object + properties: *ref_230 + required: *ref_231 + NewNatsTrigger: type: object properties: *ref_492 required: *ref_493 - EmailTrigger: - allOf: *ref_275 - type: object - properties: *ref_276 - required: *ref_277 - NewEmailTrigger: + EditNatsTrigger: type: object properties: *ref_494 required: *ref_495 - EditEmailTrigger: + EmailTrigger: + allOf: *ref_277 + type: object + properties: *ref_278 + required: *ref_279 + NewEmailTrigger: type: object properties: *ref_496 required: *ref_497 + EditEmailTrigger: + type: object + properties: *ref_498 + required: *ref_499 Group: type: object - properties: *ref_283 - required: *ref_284 + properties: *ref_285 + required: *ref_286 InstanceGroup: type: object - required: *ref_498 - properties: *ref_499 + required: *ref_500 + properties: *ref_501 InstanceGroupWithWorkspaces: type: object - required: *ref_279 - properties: *ref_280 + required: *ref_281 + properties: *ref_282 WorkspaceInfo: type: object - properties: *ref_500 - required: *ref_501 + properties: *ref_502 + required: *ref_503 Folder: type: object - properties: *ref_286 - required: *ref_287 + properties: *ref_288 + required: *ref_289 FolderDefaultPermissionedAs: description: > Ordered list of rules applied at create-time when admins or @@ -37573,19 +38602,19 @@ components: `path_glob` matches the item path (relative to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: *ref_285 + items: *ref_287 WorkerPing: - type: object - properties: *ref_502 - required: *ref_503 - UserWorkspaceList: type: object properties: *ref_504 required: *ref_505 - CreateWorkspace: + UserWorkspaceList: type: object properties: *ref_506 required: *ref_507 + CreateWorkspace: + type: object + properties: *ref_508 + required: *ref_509 CreateWorkspaceFork: type: object properties: *ref_19 @@ -37596,15 +38625,15 @@ components: required: *ref_16 DependencyMap: type: object - properties: *ref_508 + properties: *ref_510 DependencyDependent: - type: object - properties: *ref_509 - required: *ref_510 - DependentsAmount: type: object properties: *ref_511 required: *ref_512 + DependentsAmount: + type: object + properties: *ref_513 + required: *ref_514 WorkspaceInvite: type: object properties: *ref_41 @@ -37614,54 +38643,58 @@ components: properties: *ref_39 required: *ref_40 Flow: - allOf: *ref_128 + allOf: *ref_129 ExtraPerms: type: object - additionalProperties: *ref_513 + additionalProperties: *ref_515 FlowMetadata: - type: object - properties: *ref_514 - required: *ref_515 - OpenFlowWPath: - allOf: *ref_130 - FlowPreview: - type: object - properties: *ref_156 - required: *ref_157 - RestartedFrom: - type: object - properties: *ref_155 - Policy: - type: object - properties: *ref_131 - ListableApp: type: object properties: *ref_516 required: *ref_517 - ScopeDefinition: + OpenFlowWPath: + allOf: *ref_131 + FlowPreview: + type: object + properties: *ref_158 + required: *ref_159 + RestartedFrom: + type: object + properties: *ref_157 + Policy: + type: object + properties: *ref_132 + ListableApp: type: object properties: *ref_518 required: *ref_519 - ScopeDomain: + ScopeDefinition: type: object properties: *ref_520 required: *ref_521 - ListableRawApp: + ScopeDomain: type: object properties: *ref_522 required: *ref_523 + ListableRawApp: + type: object + properties: *ref_524 + required: *ref_525 AppWithLastVersion: type: object - properties: *ref_132 - required: *ref_133 + properties: *ref_133 + required: *ref_134 AppHistory: type: object - properties: *ref_134 - required: *ref_135 + properties: *ref_137 + required: *ref_138 + EmbedTokenResponse: + type: object + properties: *ref_135 + required: *ref_136 FlowVersion: type: object - properties: *ref_126 - required: *ref_127 + properties: *ref_127 + required: *ref_128 SlackToken: type: object properties: @@ -37685,11 +38718,11 @@ components: required: *ref_75 HubScriptKind: type: string - enum: *ref_98 + enum: *ref_99 PolarsClientKwargs: type: object - properties: *ref_297 - required: *ref_298 + properties: *ref_299 + required: *ref_300 LargeFileStorage: type: object properties: *ref_45 @@ -37702,36 +38735,36 @@ components: required: *ref_46 properties: *ref_47 DataTableSchema: - type: object - required: *ref_524 - properties: *ref_525 - DataTableTables: type: object required: *ref_526 properties: *ref_527 - DataTableTableSchema: + DataTableTables: type: object required: *ref_528 properties: *ref_529 + DataTableTableSchema: + type: object + required: *ref_530 + properties: *ref_531 DynamicInputData: type: object - properties: *ref_530 - required: *ref_531 + properties: *ref_532 + required: *ref_533 WindmillLargeFile: - type: object - properties: *ref_299 - required: *ref_300 - WindmillFileMetadata: - type: object - properties: *ref_303 - WindmillFilePreview: type: object properties: *ref_301 required: *ref_302 + WindmillFileMetadata: + type: object + properties: *ref_305 + WindmillFilePreview: + type: object + properties: *ref_303 + required: *ref_304 S3Resource: type: object - properties: *ref_295 - required: *ref_296 + properties: *ref_297 + required: *ref_298 WorkspaceGitSyncSettings: type: object properties: *ref_56 @@ -37743,48 +38776,48 @@ components: properties: *ref_59 S3PermissionRule: type: object - properties: *ref_532 - required: *ref_533 + properties: *ref_534 + required: *ref_535 GitRepositorySettings: type: object properties: *ref_57 required: *ref_58 MetricMetadata: - type: object - properties: *ref_534 - required: *ref_535 - ScalarMetric: type: object properties: *ref_536 required: *ref_537 - TimeseriesMetric: + ScalarMetric: type: object properties: *ref_538 required: *ref_539 - MetricDataPoint: + TimeseriesMetric: type: object properties: *ref_540 required: *ref_541 - RawScriptForDependencies: - type: object - properties: *ref_147 - required: *ref_148 - ConcurrencyGroup: + MetricDataPoint: type: object properties: *ref_542 required: *ref_543 - ExtendedJobs: + RawScriptForDependencies: + type: object + properties: *ref_149 + required: *ref_150 + ConcurrencyGroup: type: object properties: *ref_544 required: *ref_545 + ExtendedJobs: + type: object + properties: *ref_546 + required: *ref_547 ExportedUser: type: object properties: *ref_9 required: *ref_10 GlobalSetting: type: object - properties: *ref_546 - required: *ref_547 + properties: *ref_548 + required: *ref_549 InstanceConfig: type: object description: >- @@ -37793,52 +38826,52 @@ components: properties: *ref_26 Config: type: object - properties: *ref_548 - required: *ref_549 + properties: *ref_550 + required: *ref_551 ExportedInstanceGroup: type: object - properties: *ref_281 - required: *ref_282 + properties: *ref_283 + required: *ref_284 JobSearchHit: type: object - properties: *ref_550 + properties: *ref_552 LogSearchHit: type: object - properties: *ref_551 + properties: *ref_553 AutoscalingEvent: type: object - properties: *ref_552 + properties: *ref_554 CriticalAlert: type: object properties: *ref_63 CaptureTriggerKind: type: string - enum: *ref_288 + enum: *ref_290 Capture: type: object - properties: *ref_289 - required: *ref_290 + properties: *ref_291 + required: *ref_292 CaptureConfig: type: object - properties: *ref_553 - required: *ref_554 + properties: *ref_555 + required: *ref_556 OperatorSettings: nullable: true type: object required: *ref_37 properties: *ref_38 WorkspaceComparison: - type: object - required: *ref_555 - properties: *ref_556 - WorkspaceItemDiff: type: object required: *ref_557 properties: *ref_558 - CompareSummary: + WorkspaceItemDiff: type: object required: *ref_559 properties: *ref_560 + CompareSummary: + type: object + required: *ref_561 + properties: *ref_562 TeamInfo: type: object required: @@ -37859,12 +38892,12 @@ components: description: List of channels within the team items: type: object - required: &ref_561 + required: &ref_563 - channel_id - channel_name - tenant_id - service_url - properties: &ref_562 + properties: &ref_564 channel_id: type: string description: The unique identifier of the channel @@ -37884,11 +38917,11 @@ components: https://smba.trafficmanager.net/amer/12345678-1234-1234-1234-123456789012/ ChannelInfo: type: object - required: *ref_561 - properties: *ref_562 + required: *ref_563 + properties: *ref_564 GithubInstallations: type: array - items: *ref_563 + items: *ref_565 WorkspaceGithubInstallation: type: object properties: @@ -37901,8 +38934,8 @@ components: - installation_id S3Object: type: object - properties: *ref_138 - required: *ref_139 + properties: *ref_140 + required: *ref_141 TeamsChannel: type: object required: @@ -37929,14 +38962,14 @@ components: minLength: 1 AssetUsageKind: type: string - enum: *ref_308 + enum: *ref_310 AssetUsageAccessType: type: string - enum: *ref_307 + enum: *ref_309 nullable: true AssetKind: type: string - enum: *ref_306 + enum: *ref_308 Asset: type: object properties: @@ -37944,26 +38977,26 @@ components: type: string kind: type: string - enum: *ref_306 + enum: *ref_308 required: - path - kind Volume: type: object - required: *ref_564 - properties: *ref_565 + required: *ref_566 + properties: *ref_567 ProtectionRuleset: type: object description: A workspace protection rule defining restrictions and bypass permissions - required: *ref_566 - properties: *ref_567 + required: *ref_568 + properties: *ref_569 ProtectionRules: type: array description: Configuration of protection restrictions items: *ref_64 ProtectionRuleKind: type: string - enum: *ref_568 + enum: *ref_570 RuleBypasserGroups: type: array description: Groups that can bypass this ruleset @@ -37973,13 +39006,13 @@ components: description: Users that can bypass this ruleset items: *ref_66 DeploymentRequestEligibleDeployer: - type: object - required: *ref_569 - properties: *ref_570 - DeploymentRequestAssignee: type: object required: *ref_571 properties: *ref_572 + DeploymentRequestAssignee: + type: object + required: *ref_573 + properties: *ref_574 DeploymentRequestComment: type: object required: *ref_69 @@ -37994,27 +39027,27 @@ components: required: *ref_72 NativeServiceName: type: string - enum: *ref_234 + enum: *ref_236 NativeTrigger: type: object description: A native trigger stored in Windmill - properties: *ref_573 - required: *ref_574 + properties: *ref_575 + required: *ref_576 NativeTriggerWithExternal: type: object description: >- Full trigger response containing both Windmill data and external service data - properties: *ref_575 - required: *ref_576 - WorkspaceIntegrations: - type: object properties: *ref_577 required: *ref_578 + WorkspaceIntegrations: + type: object + properties: *ref_579 + required: *ref_580 WorkspaceOAuthConfig: type: object - properties: *ref_235 - required: *ref_236 + properties: *ref_237 + required: *ref_238 WebhookEvent: type: object properties: @@ -38025,7 +39058,7 @@ components: request_type: type: string description: The type of webhook request (define possible values here) - enum: &ref_579 + enum: &ref_581 - async - sync required: @@ -38034,21 +39067,21 @@ components: WebhookRequestType: type: string description: The type of webhook request (define possible values here) - enum: *ref_579 + enum: *ref_581 RedirectUri: type: object - properties: *ref_237 - required: *ref_238 + properties: *ref_239 + required: *ref_240 NativeTriggerData: type: object description: Data for creating or updating a native trigger - properties: *ref_239 - required: *ref_240 + properties: *ref_241 + required: *ref_242 CreateTriggerResponse: type: object description: Response returned when a native trigger is created - properties: *ref_580 - required: *ref_581 + properties: *ref_582 + required: *ref_583 SyncResult: type: object properties: @@ -38071,36 +39104,36 @@ components: - total_external - total_windmill NextCloudEventType: - type: object - properties: *ref_582 - required: *ref_583 - GoogleCalendarEntry: type: object properties: *ref_584 required: *ref_585 - GoogleDriveFile: + GoogleCalendarEntry: type: object properties: *ref_586 required: *ref_587 - GoogleDriveFilesResponse: + GoogleDriveFile: type: object properties: *ref_588 required: *ref_589 - SharedDriveEntry: + GoogleDriveFilesResponse: type: object properties: *ref_590 required: *ref_591 - GithubRepoEntry: + SharedDriveEntry: type: object properties: *ref_592 required: *ref_593 + GithubRepoEntry: + type: object + properties: *ref_594 + required: *ref_595 schemas-StaticTransform: type: object description: >- Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource' - properties: *ref_140 - required: *ref_141 + properties: *ref_142 + required: *ref_143 schemas-JavascriptTransform: type: object description: >- @@ -38129,22 +39162,22 @@ components: Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms - properties: *ref_328 - required: *ref_329 + properties: *ref_330 + required: *ref_331 schemas-PathScript: type: object description: >- Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code - properties: *ref_330 - required: *ref_331 + properties: *ref_332 + required: *ref_333 schemas-PathFlow: type: object description: >- Reference to an existing flow by path. Use this to call another flow as a subflow - properties: *ref_332 - required: *ref_333 + properties: *ref_334 + required: *ref_335 schemas-FlowModule: type: object description: A single step in a flow. Can be a script, subflow, loop, or branch @@ -38157,96 +39190,96 @@ components: 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations - properties: *ref_334 - required: *ref_335 + properties: *ref_336 + required: *ref_337 schemas-WhileloopFlow: type: object description: >- Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination - properties: *ref_336 - required: *ref_337 + properties: *ref_338 + required: *ref_339 schemas-BranchOne: type: object description: >- Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes - properties: *ref_338 - required: *ref_339 + properties: *ref_340 + required: *ref_341 schemas-BranchAll: type: object description: >- Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently - properties: *ref_340 - required: *ref_341 + properties: *ref_342 + required: *ref_343 schemas-Identity: type: object description: >- Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder - properties: *ref_342 - required: *ref_343 + properties: *ref_344 + required: *ref_345 AIProviderKind: type: string description: Supported AI provider types - enum: *ref_321 + enum: *ref_323 schemas-ProviderConfig: type: object description: >- Complete AI provider configuration with resource reference and model selection - properties: *ref_594 - required: *ref_595 + properties: *ref_596 + required: *ref_597 StaticProviderTransform: type: object description: Static provider configuration passed directly to the AI agent - properties: *ref_596 - required: *ref_597 + properties: *ref_598 + required: *ref_599 ProviderTransform: description: >- Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined - oneOf: *ref_346 - discriminator: *ref_347 + oneOf: *ref_348 + discriminator: *ref_349 MemoryOff: type: object description: No conversation memory/context - properties: *ref_322 - required: *ref_323 + properties: *ref_324 + required: *ref_325 MemoryAuto: type: object description: Automatic context management - properties: *ref_324 - required: *ref_325 + properties: *ref_326 + required: *ref_327 MemoryMessage: type: object description: A single message in conversation history - properties: *ref_598 - required: *ref_599 + properties: *ref_600 + required: *ref_601 MemoryManual: type: object description: Explicit message history - properties: *ref_326 - required: *ref_327 + properties: *ref_328 + required: *ref_329 schemas-MemoryConfig: description: Conversation memory configuration - oneOf: *ref_600 - discriminator: *ref_601 + oneOf: *ref_602 + discriminator: *ref_603 StaticMemoryTransform: type: object description: Static memory configuration passed directly to the AI agent - properties: *ref_602 - required: *ref_603 + properties: *ref_604 + required: *ref_605 MemoryTransform: description: >- Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined - oneOf: *ref_348 - discriminator: *ref_349 + oneOf: *ref_350 + discriminator: *ref_351 schemas-FlowModuleValue: description: >- The actual implementation of a flow step. Can be a script (inline or @@ -38257,41 +39290,41 @@ components: description: >- A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module - allOf: *ref_604 + allOf: *ref_606 McpToolValue: type: object description: >- Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers - properties: *ref_605 - required: *ref_606 + properties: *ref_607 + required: *ref_608 WebsearchToolValue: type: object description: >- A tool implemented as a websearch tool. The AI can call this like any other websearch tool - properties: *ref_607 - required: *ref_608 + properties: *ref_609 + required: *ref_610 ToolValue: description: >- The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference - oneOf: *ref_609 - discriminator: *ref_610 + oneOf: *ref_611 + discriminator: *ref_612 AgentTool: type: object description: >- A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool - properties: *ref_350 - required: *ref_351 + properties: *ref_352 + required: *ref_353 schemas-AiAgent: type: object description: >- AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task - properties: *ref_344 - required: *ref_345 + properties: *ref_346 + required: *ref_347 schemas-StopAfterIf: type: object description: Early termination condition for a module @@ -38300,17 +39333,17 @@ components: RetryIf: type: object description: Conditional retry based on error or result - properties: *ref_199 - required: *ref_200 + properties: *ref_201 + required: *ref_202 schemas-Retry: type: object description: Retry configuration for failed module executions - properties: *ref_320 + properties: *ref_322 schemas-FlowNote: type: object description: A sticky note attached to a flow for documentation and annotation - properties: *ref_149 - required: *ref_150 + properties: *ref_151 + required: *ref_152 FlowGroup: type: object description: >- @@ -38319,16 +39352,16 @@ components: flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id. - properties: *ref_151 - required: *ref_152 + properties: *ref_153 + required: *ref_154 schemas-FlowValue: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_611 - required: *ref_612 + properties: *ref_613 + required: *ref_614 schemas-FlowStatusModule: type: object - properties: *ref_158 - required: *ref_159 + properties: *ref_160 + required: *ref_161 diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 9d543633b8..eb500fdce0 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.734.0 + version: 1.742.0 title: Windmill API contact: @@ -127,40 +127,89 @@ paths: schema: type: string - /inkeep: - post: - summary: query Windmill AI documentation assistant (EE only) - operationId: queryDocumentation + /docs/search: + get: + summary: "Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more." + operationId: searchDocs x-mcp-tool: true tags: - documentation - requestBody: - description: query to send to the AI documentation assistant - required: true - content: - application/json: - schema: - type: object - properties: - query: - type: string - description: The documentation query to send to the AI assistant - required: - - query + parameters: + - name: query + description: 'Keywords to search for in the documentation body, e.g. "chromium worker tag" or "retry exponential backoff". Fewer, more distinctive words match better.' + in: query + required: true + schema: + type: string responses: "200": - description: AI documentation assistant response + description: matching documentation pages content: application/json: schema: type: object - description: Response from Inkeep service - "403": - description: Enterprise Edition required + properties: + text: + type: string + description: Model-ready rendering of the results + results: + type: array + items: + type: object + properties: + url: + type: string + title: + type: string + score: + type: integer + snippets: + type: array + items: + type: string + required: + - url + - title + - score + - snippets + required: + - text + - results + + /docs/page: + get: + summary: "Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section." + operationId: readDocsPage + x-mcp-tool: true + tags: + - documentation + parameters: + - name: url + description: "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + in: query + required: true + schema: + type: string + - name: section + description: "Optional. A heading title from the page outline to read just that section instead of the full page." + in: query + schema: + type: string + responses: + "200": + description: documentation page content content: - text/plain: + application/json: schema: - type: string + type: object + properties: + text: + type: string + source_url: + type: string + required: + - text + - source_url /openapi.yaml: get: @@ -997,6 +1046,39 @@ paths: schema: $ref: "#/components/schemas/UserWorkspaceList" + /workspaces/session_workspace_status: + post: + summary: get the lifecycle status of workspaces referenced by client-side sessions + operationId: getSessionWorkspaceStatus + tags: + - workspace + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + workspace_ids: + type: array + items: + type: string + required: + - workspace_ids + responses: + "200": + description: map of workspace id to status (active, archived, or deleted) + content: + application/json: + schema: + type: object + additionalProperties: + type: string + enum: + - active + - archived + - deleted + /w/{workspace}/workspaces/get_as_superadmin: get: summary: get workspace as super admin (require to be super admin) @@ -1655,6 +1737,9 @@ paths: s3_deleted: type: integer format: int64 + s3_not_found: + type: integer + format: int64 orphans_scanned: type: integer format: int64 @@ -1727,6 +1812,92 @@ paths: - last_run_exported - updated_at + /settings/audit_logs_s3_backfill: + post: + summary: start an opt-in historical backfill of audit logs to object storage + operationId: runAuditLogsS3Backfill + tags: + - setting + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + from: + type: string + format: date-time + description: inclusive lower bound of the window to export + to: + type: string + format: date-time + description: exclusive upper bound of the window to export + required: + - from + - to + responses: + "202": + description: backfill started + + /settings/audit_logs_s3_backfill_status: + get: + summary: get status of the audit-log object-store historical backfill + operationId: getAuditLogsS3BackfillStatus + tags: + - setting + responses: + "200": + description: current backfill status (null if never run) + content: + application/json: + schema: + nullable: true + type: object + properties: + running: + type: boolean + started_at: + type: string + format: date-time + finished_at: + type: string + format: date-time + nullable: true + phase: + type: string + from: + type: string + format: date-time + to: + type: string + format: date-time + rows_written: + type: integer + format: int64 + objects_written: + type: integer + format: int64 + last_ts: + type: string + format: date-time + nullable: true + errors: + type: integer + format: int64 + last_error: + type: string + nullable: true + required: + - running + - started_at + - phase + - from + - to + - rows_written + - objects_written + - errors + /settings/send_stats: post: summary: send stats @@ -6620,6 +6791,9 @@ paths: cc_instance: type: string description: "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side (client_credentials flow only). The token URL is never caller-supplied." + cc_token_url: + type: string + description: "Bring-your-own token endpoint override (client_credentials flow only). Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; ignored/rejected on the shared-instance path." mcp_server_url: type: string description: "MCP server URL for MCP OAuth token refresh" @@ -6675,6 +6849,9 @@ paths: cc_instance: type: string description: "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side. The token URL is never caller-supplied." + cc_token_url: + type: string + description: "Bring-your-own token endpoint override. Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; rejected on the shared-instance path." responses: "200": description: OAuth token response @@ -7821,6 +7998,22 @@ paths: workspace_id: type: string + /apps_u/embed_token_by_custom_path/{custom_path}: + get: + summary: get app embed token by custom path + operationId: getAppEmbedTokenByCustomPath + tags: + - app + parameters: + - $ref: "#/components/parameters/CustomPath" + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /scripts/hub/get/{path}: get: summary: get hub script content by path @@ -10751,6 +10944,133 @@ paths: schema: type: string + /w/{workspace}/ai_skills/list: + get: + summary: list the workspace AI chat skills (name + description only) + operationId: listAiSkills + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: skill listing + content: + application/json: + schema: + type: array + items: + type: object + required: + - name + - description + properties: + name: + type: string + description: + type: string + + /w/{workspace}/ai_skills/get/{name}: + get: + summary: get a workspace AI chat skill including its instructions + operationId: getAiSkill + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: name + in: path + required: true + schema: + type: string + responses: + "200": + description: skill + content: + application/json: + schema: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + description: + type: string + instructions: + type: string + + /w/{workspace}/ai_skills/upload: + post: + summary: upsert workspace AI chat skills (admin only) + operationId: uploadAiSkills + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - skills + properties: + skills: + type: array + maxItems: 50 + items: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + minLength: 1 + maxLength: 64 + pattern: "^[a-z0-9-]+$" + description: + type: string + minLength: 1 + maxLength: 1024 + instructions: + type: string + minLength: 1 + maxLength: 65536 + responses: + "200": + description: uploaded + content: + text/plain: + schema: + type: string + + /w/{workspace}/ai_skills/delete/{name}: + delete: + summary: delete a workspace AI chat skill (admin only) + operationId: deleteAiSkill + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: name + in: path + required: true + schema: + type: string + responses: + "200": + description: deleted + content: + text/plain: + schema: + type: string + /w/{workspace}/apps/get_data/v/{secretWithExtension}: get: summary: get raw app data by @@ -10762,6 +11082,10 @@ paths: - name: secretWithExtension in: path required: true + description: >- + App version secret suffixed with the requested file type extension. + Supported extensions are `.js` (JavaScript bundle), `.css` + (stylesheet), and `.html` (sandboxed wrapper document). schema: type: string responses: @@ -10771,6 +11095,12 @@ paths: text/javascript: schema: type: string + text/css: + schema: + type: string + text/html: + schema: + type: string /w/{workspace}/apps/list_search: get: @@ -11022,6 +11352,23 @@ paths: - $ref: "#/components/schemas/AppWithLastVersion" - $ref: "#/components/schemas/UserDraftOverlay" + /w/{workspace}/apps/embed_token/p/{path}: + get: + summary: get app embed token by path + operationId: getAppEmbedTokenByPath + tags: + - app + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/ScriptPath" + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /w/{workspace}/apps/get/lite/{path}: get: summary: get app lite by path @@ -11139,6 +11486,27 @@ paths: schema: $ref: "#/components/schemas/AppWithLastVersion" + /w/{workspace}/apps_u/embed_token/{secret}: + get: + summary: get app embed token by secret + operationId: getAppEmbedTokenBySecret + tags: + - app + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: secret + in: path + required: true + schema: + type: string + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /w/{workspace}/apps_u/public_resource/{path}: get: summary: get public resource @@ -11847,6 +12215,11 @@ paths: in: query schema: type: boolean + - name: timeout + description: custom timeout in seconds for this preview run + in: query + schema: + type: integer - $ref: "#/components/parameters/NewJobId" requestBody: @@ -13030,6 +13403,64 @@ paths: schema: type: string + /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}: + get: + summary: get all logs for a flow job in a structured format + operationId: getFlowAllLogsStructured + tags: + - job + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/JobId" + responses: + "200": + description: structured logs of all flow steps, one entry per job + content: + application/json: + schema: + type: array + items: + type: object + properties: + job_id: + type: string + label: + type: string + description: human-readable label describing the job's position in the flow tree + kind: + type: string + description: job kind (script, flow, forloopflow, ...) + flow_step_id: + type: string + nullable: true + step_path: + type: string + nullable: true + description: materialized step path (e.g. "a/b") + depth: + type: integer + description: depth in the flow tree (0 for the root flow job) + parent_module_type: + type: string + nullable: true + description: parent module type (forloopflow, branchall, ...) + sibling_index: + type: integer + description: 1-based index of this job among siblings sharing the same step + sibling_count: + type: integer + description: total number of siblings sharing the same step + logs: + type: string + required: + - job_id + - label + - kind + - depth + - sibling_index + - sibling_count + - logs + /w/{workspace}/jobs_u/get_completed_logs_tail/{id}: get: summary: get completed job logs tail @@ -23193,6 +23624,8 @@ components: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string required: @@ -23313,6 +23746,8 @@ components: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string required: @@ -23687,6 +24122,10 @@ components: type: array items: type: string + folders_read: + type: array + items: + type: string folders_owners: type: array items: @@ -23706,6 +24145,7 @@ components: - operator - disabled - folders + - folders_read - folders_owners UserSource: @@ -28021,6 +28461,13 @@ components: type: string on_behalf_of_email: type: string + sandbox: + type: boolean + description: > + Publisher opt-in to app sandbox isolation (alpha). When true the app + is isolated from each viewer's Windmill session. When false/absent + the app runs same-origin with the viewer's full session (the + default, pre-isolation behavior). ListableApp: type: object @@ -28240,6 +28687,36 @@ components: required: - version + EmbedTokenResponse: + type: object + properties: + token: + type: string + nullable: true + description: Narrowly-scoped embed token for the iframe. Absent for fully anonymous or raw apps, which load without a scoped token. + expiration: + type: string + format: date-time + nullable: true + description: Expiration of the embed token. + raw_app: + type: boolean + description: Raw apps render single-iframe and skip the opaque-viewer indirection and the embed token entirely. + sandbox: + type: boolean + description: Publisher opted this app into sandbox isolation. When false the viewer runs the app same-origin with its full session. + app_path: + type: string + nullable: true + description: The resolved app path; the embedder uses it to scope the app's backing localStorage per app. + workspace_id: + type: string + nullable: true + description: The resolved workspace; pairs with app_path so apps at the same path in different workspaces don't share a localStorage store. + required: + - raw_app + - sandbox + FlowVersion: type: object properties: diff --git a/backend/windmill-api/src/ai_skills.rs b/backend/windmill-api/src/ai_skills.rs new file mode 100644 index 0000000000..6deb8dc58f --- /dev/null +++ b/backend/windmill-api/src/ai_skills.rs @@ -0,0 +1,394 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2026 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +use crate::db::{ApiAuthed, DB}; +use std::collections::HashSet; +use axum::{ + extract::{Extension, Json, Path}, + routing::{delete, get, post}, + Router, +}; +use serde::{Deserialize, Serialize}; +use windmill_audit::audit_oss::audit_log; +use windmill_audit::ActionKind; +use windmill_common::{ + db::UserDB, + error::{Error, JsonResult, Result}, + utils::require_admin, +}; + +pub fn workspaced_service() -> Router { + Router::new() + .route("/list", get(list_skills)) + .route("/get/{name}", get(get_skill)) + .route("/upload", post(upload_skills)) + .route("/delete/{name}", delete(delete_skill)) +} + +/// Cheap listing surfaced in the AI chat system prompt — no `instructions` body. +#[derive(Serialize)] +pub struct SkillListItem { + pub name: String, + pub description: String, +} + +/// Full skill, including the SKILL.md body, fetched on demand by `read_skill`. +#[derive(Serialize)] +pub struct Skill { + pub name: String, + pub description: String, + pub instructions: String, +} + +#[derive(Deserialize)] +pub struct UploadSkills { + pub skills: Vec, +} + +#[derive(Deserialize)] +pub struct SkillUpload { + pub name: String, + pub description: String, + pub instructions: String, +} + +const MAX_SKILLS_PER_UPLOAD: usize = 50; +// Every stored skill's name + description is advertised in the global AI chat +// system prompt, so bound the total a workspace can accumulate across uploads. +const MAX_SKILLS_PER_WORKSPACE: usize = 100; +// `name` and `description` follow the Claude SKILL.md spec +// (https://platform.claude.com/docs/en/agents-and-tools/agent-skills): both are +// loaded into the AI chat system prompt and `name` is the model-facing skill id, +// so matching the upstream limits keeps skills portable with Claude Code. +const MAX_SKILL_NAME_CHARS: usize = 64; +const MAX_SKILL_DESCRIPTION_CHARS: usize = 1_024; +// Not a spec field — a payload bound on the SKILL.md body, so measured in bytes. +const MAX_SKILL_INSTRUCTIONS_BYTES: usize = 64 * 1024; + +fn validate_skill(skill: &SkillUpload) -> Result<()> { + let name = skill.name.trim(); + if name.is_empty() || name.chars().count() > MAX_SKILL_NAME_CHARS { + return Err(Error::BadRequest(format!( + "skill name must be between 1 and {MAX_SKILL_NAME_CHARS} characters, got {:?}", + skill.name + ))); + } + if !name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') + { + return Err(Error::BadRequest(format!( + "skill name {name:?} must only contain lowercase letters, digits or '-'" + ))); + } + if skill.description.trim().is_empty() { + return Err(Error::BadRequest(format!( + "skill {name:?} is missing a description (the SKILL.md frontmatter `description`)" + ))); + } + if skill.description.chars().count() > MAX_SKILL_DESCRIPTION_CHARS { + return Err(Error::BadRequest(format!( + "skill {name:?} description must be at most {MAX_SKILL_DESCRIPTION_CHARS} characters" + ))); + } + if skill.instructions.trim().is_empty() { + return Err(Error::BadRequest(format!( + "skill {name:?} has an empty SKILL.md body" + ))); + } + if skill.instructions.len() > MAX_SKILL_INSTRUCTIONS_BYTES { + return Err(Error::BadRequest(format!( + "skill {name:?} instructions must be at most {MAX_SKILL_INSTRUCTIONS_BYTES} bytes" + ))); + } + Ok(()) +} + +/// Collect the trimmed skill names, rejecting duplicates within a single upload. +/// The insert upserts by name, so a duplicate would silently keep only the last +/// and make the reported/audited count wrong. +fn collect_upload_names(skills: &[SkillUpload]) -> Result> { + let mut names = Vec::with_capacity(skills.len()); + let mut seen = HashSet::with_capacity(skills.len()); + for skill in skills { + let name = skill.name.trim().to_string(); + if !seen.insert(name.clone()) { + return Err(Error::BadRequest(format!( + "duplicate skill name {name:?} in upload" + ))); + } + names.push(name); + } + Ok(names) +} + +/// Reject an upload that would push the workspace past `MAX_SKILLS_PER_WORKSPACE`. +/// Uploads upsert, so names already present (`replacing`) don't count as new. +fn check_workspace_skill_capacity( + existing_total: i64, + replacing: i64, + upload_count: usize, +) -> Result<()> { + let new_count = upload_count as i64 - replacing; + if existing_total + new_count > MAX_SKILLS_PER_WORKSPACE as i64 { + return Err(Error::BadRequest(format!( + "workspace cannot store more than {MAX_SKILLS_PER_WORKSPACE} skills" + ))); + } + Ok(()) +} + +async fn list_skills( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = sqlx::query!( + "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name", + &w_id + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + + Ok(Json( + rows.into_iter() + .map(|r| SkillListItem { name: r.name, description: r.description }) + .collect(), + )) +} + +async fn get_skill( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, name)): Path<(String, String)>, +) -> JsonResult { + let mut tx = user_db.begin(&authed).await?; + let row = sqlx::query!( + "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2", + &w_id, + &name + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + + row.map(|r| { + Json(Skill { name: r.name, description: r.description, instructions: r.instructions }) + }) + .ok_or_else(|| Error::NotFound(format!("no skill named {name:?} in workspace {w_id}"))) +} + +/// Bulk upsert the uploaded skills by name. Existing skills not in the payload +/// are left untouched — removal goes through `delete_skill`. +async fn upload_skills( + authed: ApiAuthed, + Extension(db): Extension, + Path(w_id): Path, + Json(payload): Json, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + + if payload.skills.is_empty() { + return Err(Error::BadRequest("no skills to upload".to_string())); + } + if payload.skills.len() > MAX_SKILLS_PER_UPLOAD { + return Err(Error::BadRequest(format!( + "cannot upload more than {MAX_SKILLS_PER_UPLOAD} skills at a time" + ))); + } + for skill in &payload.skills { + validate_skill(skill)?; + } + let names = collect_upload_names(&payload.skills)?; + + let mut tx = db.begin().await?; + let counts = sqlx::query!( + r#"SELECT + COUNT(*)::bigint AS "total!", + COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS "replacing!" + FROM ai_skill + WHERE workspace_id = $1"#, + &w_id, + &names + ) + .fetch_one(&mut *tx) + .await?; + check_workspace_skill_capacity(counts.total, counts.replacing, names.len())?; + + for (skill, name) in payload.skills.iter().zip(names.iter()) { + sqlx::query!( + r#"INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by) + VALUES ($1, $2, $3, $4, now(), $5) + ON CONFLICT (workspace_id, name) DO UPDATE + SET description = EXCLUDED.description, + instructions = EXCLUDED.instructions, + edited_at = now(), + edited_by = EXCLUDED.edited_by"#, + &w_id, + name, + skill.description, + skill.instructions, + &authed.username, + ) + .execute(&mut *tx) + .await?; + } + + let audit_resource = names.join(","); + audit_log( + &mut *tx, + &authed, + "ai_skills.upload", + ActionKind::Update, + &w_id, + Some(&audit_resource), + Some([("skill_count", &names.len().to_string()[..])].into()), + ) + .await?; + tx.commit().await?; + + Ok(format!( + "Uploaded {} skill(s) to workspace {}", + payload.skills.len(), + &w_id + )) +} + +async fn delete_skill( + authed: ApiAuthed, + Extension(db): Extension, + Path((w_id, name)): Path<(String, String)>, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + + let mut tx = db.begin().await?; + let deleted = sqlx::query_scalar!( + "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name", + &w_id, + &name + ) + .fetch_optional(&mut *tx) + .await?; + + if deleted.is_none() { + tx.commit().await?; + return Err(Error::NotFound(format!( + "no skill named {name:?} in workspace {w_id}" + ))); + } + + audit_log( + &mut *tx, + &authed, + "ai_skills.delete", + ActionKind::Delete, + &w_id, + Some(&name), + None, + ) + .await?; + tx.commit().await?; + + Ok(format!("Deleted skill {name} from workspace {w_id}")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn skill() -> SkillUpload { + SkillUpload { + name: "test-skill".to_string(), + description: "Useful for tests".to_string(), + instructions: "# Test\n\nDo the thing.".to_string(), + } + } + + #[test] + fn validate_skill_rejects_oversized_description() { + let mut skill = skill(); + skill.description = "x".repeat(MAX_SKILL_DESCRIPTION_CHARS + 1); + + assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_)))); + } + + #[test] + fn validate_skill_rejects_oversized_instructions() { + let mut skill = skill(); + skill.instructions = "x".repeat(MAX_SKILL_INSTRUCTIONS_BYTES + 1); + + assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_)))); + } + + #[test] + fn validate_skill_rejects_oversized_name() { + let mut skill = skill(); + skill.name = "a".repeat(MAX_SKILL_NAME_CHARS + 1); + + assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_)))); + } + + #[test] + fn validate_skill_rejects_non_slug_name() { + // Uppercase, underscore, space and punctuation are all outside the + // Claude SKILL.md `[a-z0-9-]` name charset. + for bad in ["My-Skill", "my_skill", "my skill", "skill!"] { + let mut skill = skill(); + skill.name = bad.to_string(); + + assert!( + matches!(validate_skill(&skill), Err(Error::BadRequest(_))), + "{bad:?} should be rejected" + ); + } + } + + #[test] + fn validate_skill_counts_description_in_characters() { + // 1024 two-byte chars exceed the byte limit but sit exactly on the + // character limit, so they must be accepted. + let mut skill = skill(); + skill.description = "é".repeat(MAX_SKILL_DESCRIPTION_CHARS); + + assert!(validate_skill(&skill).is_ok()); + } + + #[test] + fn workspace_capacity_allows_replacement_at_cap() { + // Already at the cap, but the upload only replaces an existing skill. + let at_cap = MAX_SKILLS_PER_WORKSPACE as i64; + assert!(check_workspace_skill_capacity(at_cap, 1, 1).is_ok()); + } + + #[test] + fn workspace_capacity_rejects_new_skill_over_cap() { + let at_cap = MAX_SKILLS_PER_WORKSPACE as i64; + assert!(matches!( + check_workspace_skill_capacity(at_cap, 0, 1), + Err(Error::BadRequest(_)) + )); + } + + #[test] + fn collect_upload_names_trims_and_collects() { + let names = collect_upload_names(&[skill()]).unwrap(); + assert_eq!(names, vec!["test-skill".to_string()]); + } + + #[test] + fn collect_upload_names_rejects_duplicates() { + // Names are compared after trimming, so whitespace can't smuggle a dup in. + let dup = SkillUpload { name: " test-skill ".to_string(), ..skill() }; + assert!(matches!( + collect_upload_names(&[skill(), dup]), + Err(Error::BadRequest(_)) + )); + } +} diff --git a/backend/windmill-api/src/apps.rs b/backend/windmill-api/src/apps.rs index 0615735081..08a41e2d1f 100644 --- a/backend/windmill-api/src/apps.rs +++ b/backend/windmill-api/src/apps.rs @@ -75,6 +75,7 @@ use windmill_common::{ use windmill_object_store::object_store_reexports::{Attribute, Attributes}; use windmill_store::resources::get_resource_value_interpolated_internal; +use windmill_api_auth::{create_token_internal, ensure_scopes_within_caller, NewToken}; use windmill_git_sync::{handle_deployment_metadata, DeployedObject}; use windmill_queue::{push, PushArgs, PushArgsOwned, PushIsolationLevel}; @@ -90,6 +91,7 @@ pub fn workspaced_service(raw_app_body_limit: usize) -> Router { .route("/list", get(list_apps)) .route("/list_search", get(list_search_apps)) .route("/get/p/{*path}", get(get_app)) + .route("/embed_token/p/{*path}", get(get_app_embed_token_for_path)) .route("/get/lite/{*path}", get(get_app_lite)) .route("/secret_of/{*path}", get(get_secret_id)) .route( @@ -134,6 +136,7 @@ pub fn unauthed_service() -> Router { .route("/delete_s3_file", delete(delete_s3_file_from_app)) .route("/download_s3_file/{*path}", get(download_s3_file_from_app)) .route("/public_app/{secret}", get(get_public_app_by_secret)) + .route("/embed_token/{secret}", get(get_app_embed_token)) .route("/public_resource/{*path}", get(get_public_resource)) .route("/get_data/v/{*id}", get(get_raw_app_data)) } @@ -300,6 +303,13 @@ pub struct Policy { pub execution_mode: ExecutionMode, pub s3_inputs: Option>, pub allowed_s3_keys: Option>, + // WIN-2006: publisher opt-in to iframe sandbox isolation (alpha). When true the + // app is isolated from each viewer's Windmill session: low-code renders in an + // opaque-origin iframe with a scoped embed token, raw renders its bundle in an + // opaque iframe. Default/absent means unsandboxed — the app runs same-origin + // with the viewer's full session, the pre-isolation behavior. + #[serde(skip_serializing_if = "Option::is_none")] + pub sandbox: Option, } #[derive(Deserialize)] @@ -348,6 +358,12 @@ async fn list_search_apps( Path(w_id): Path, Extension(user_db): Extension, ) -> JsonResult> { + // Require domain-level read: this returns every visible app's full value (code). + // The route layer treats `apps:run` as satisfying read, so without this handler + // check a scoped embed token (apps:run + apps:read:) could read all + // apps' definitions. `check_scopes` uses ScopeDefinition::includes, where run + // does NOT include read, so it correctly denies such tokens. + check_scopes(&authed, || "apps:read".to_string())?; #[cfg(feature = "enterprise")] let n = 1000; @@ -379,6 +395,9 @@ async fn list_apps( Query(pagination): Query, Query(lq): Query, ) -> JsonResult> { + // Domain-level read (see list_search_apps): keeps a scoped embed token, whose + // `apps:run` only satisfies read at the route layer, from listing all apps. + check_scopes(&authed, || "apps:read".to_string())?; let (per_page, offset) = paginate(pagination); let mut sqlb = SqlBuilder::select_from("app") @@ -553,6 +572,7 @@ async fn list_apps( async fn get_raw_app_data( Path((w_id, secret_with_ext)): Path<(String, String)>, + Query(query): Query>, Extension(db): Extension, ) -> Result { #[cfg(all(feature = "enterprise", feature = "parquet"))] @@ -575,13 +595,52 @@ async fn get_raw_app_data( .await?; let file_type = splitted.next().unwrap_or(""); + + // Sandboxed wrapper document that hosts the bundle. Served from a real URL + // (not blob:/srcdoc) so we can attach `CSP: sandbox` as a response header, + // which forces an opaque origin even on direct navigation — a raw-app + // bundle can then never reach the authenticated Windmill origin (WIN-2006). + // The `.js`/`.css` are loaded as same-path subresources by this document. + if file_type == "html" { + // ALWAYS served with `CSP: sandbox`, which forces an opaque origin even on + // direct top-level navigation — so this real-origin URL can never be used + // to run a raw-app bundle with the viewer's session (WIN-2006). The + // unsandboxed (default) render is NOT applied here: it is handled entirely + // on the viewer side, which builds its own same-origin wrapper. Relaxing + // this header from a policy flag would let anyone with the share secret + // hand a logged-in victim a same-origin URL that runs the bundle with + // their session — so the standalone document stays sandboxed no matter how + // it is reached. + let html = raw_app_wrapper_html(secret_id); + let mut builder = Response::builder() + .header(http::header::CONTENT_TYPE, "text/html; charset=utf-8") + .header("X-Content-Type-Options", "nosniff") + .header("Cross-Origin-Resource-Policy", "cross-origin") + .header( + http::header::CONTENT_SECURITY_POLICY, + "sandbox allow-scripts allow-forms allow-popups \ + allow-popups-to-escape-sandbox allow-downloads allow-modals \ + allow-top-navigation", + ); + // When the public app page is embedded in a cross-origin-isolated page + // (`wm_coep` opt-in, COEP `require-corp`), this nested wrapper document + // must itself assert COEP to be allowed to load. Opt-in only — COEP + // restricts the bundle's own subresources to CORP'd/same-origin ones + // (e.g. external images would break), so it must not be always-on. The + // viewer propagates the flag from the page URL (see RawAppPreview). + if query.contains_key("wm_coep") { + builder = builder.header("Cross-Origin-Embedder-Policy", "require-corp"); + } + return Ok(builder.body(Body::from(html)).unwrap()); + } + let file_type = if file_type == "css" { "css" } else if file_type == "js" { "js" } else { return Err(Error::BadRequest( - "Invalid file type, only .css and .js are supported".to_string(), + "Invalid file type, only .css, .js and .html are supported".to_string(), )); }; // tracing::info!("file_type: {}", file_type); @@ -632,20 +691,128 @@ async fn get_raw_app_data( if let Some(body) = body { // let stream = tokio_util::io::ReaderStream::new(file); - let res = Response::builder().header( - http::header::CONTENT_TYPE, - if file_type == "css" { - "text/css" - } else { - "text/javascript" - }, - ); + let res = Response::builder() + .header( + http::header::CONTENT_TYPE, + if file_type == "css" { + "text/css" + } else { + "text/javascript" + }, + ) + // nosniff + CORP so the bundle loads correctly as a subresource of + // the opaque, sandboxed wrapper (incl. under a cross-origin-isolated + // / COEP `require-corp` embedder). + .header("X-Content-Type-Options", "nosniff") + .header("Cross-Origin-Resource-Policy", "cross-origin"); Ok(res.body(body).unwrap()) } else { return Err(Error::NotFound("File not found".to_string())); } } +/// HTML wrapper that hosts a raw-app bundle inside a sandboxed, opaque-origin +/// iframe. Served by [`get_raw_app_data`] for the `.html` "file type". It loads +/// the bundle `.js`/`.css` as same-path subresources, shims web storage (which +/// an opaque origin disallows), and waits for the embedder to hand it the user +/// context via `postMessage` before evaluating the bundle — so the bundle never +/// receives a credential and `window.ctx` is set synchronously when it runs. +fn raw_app_wrapper_html(secret: &str) -> String { + const TEMPLATE: &str = r##" + + + +App + + + + +
+ + +"##; + TEMPLATE.replace("__SECRET__", secret) +} + // async fn get_app_version( // authed: ApiAuthed, // Extension(user_db): Extension, @@ -937,6 +1104,17 @@ async fn get_public_app_by_secret( let mut app = not_found_if_none(app_o, "App", id.to_string())?; + // Confine the app embed token (the only credential handed to untrusted app JS, + // carrying the viewer's identity + `apps:read:`) to the app the secret + // resolves to: without this, app JS could reuse the viewer's identity to read any + // app it can see by secret via the RLS check below. Scoped to embed tokens only — + // other callers (anonymous, cookie, plain external JWT) keep their existing access. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:read:{}", app.path))?; + } + } + let policy = serde_json::from_str::(app.policy.0.get()).map_err(to_anyhow)?; if !matches!(policy.execution_mode, ExecutionMode::Anonymous) { @@ -971,6 +1149,300 @@ async fn get_public_app_by_secret( Ok(Json(app)) } +/// Scopes granted to a short-lived "app embed token". This is the token the +/// app-embedder page hands the (opaque-origin) app iframe at startup so the app +/// never receives the viewer's session cookie. Instead of restricting which +/// routes a *domain* may hit, we restrict which routes the *token* may hit, so +/// that even a malicious or compromised app document can only reach the +/// endpoints an app legitimately needs. The `app_embed` sentinel turns each of +/// these into a strict route allowlist (`app_embed_route_denied`): +/// - `jobs:read` → by-id job poll/cancel only; enumeration, counts, exports, +/// and `job_signature`/`resume_urls` are denied, and by-id +/// reads are confined to the app's own runs. +/// - `app_embed` → sentinel tagging this as an app embed token (grants nothing). +/// - `resources:run` → resource metadata only (pickers, type schemas), never values. +/// - `users:read` → `users/whoami` only. +/// - `folders:read` → `folders/listnames` only. +/// Plus two path-scoped scopes minted per app (see `mint_app_embed_token`): +/// - `apps:read:` → the app's own definition (`apps/get/p/`); no +/// `apps:write`, so management routes are unreachable. +/// - `apps:run:` → run THIS app's components (`execute_component`, which +/// re-checks the path); `apps_u/*` public-serving routes. +pub const APP_EMBED_SCOPES: [&str; 5] = [ + "jobs:read", + windmill_api_auth::scopes::APP_EMBED_SENTINEL, + "resources:run", + "users:read", + "folders:read", +]; + +/// How long an app embed token stays valid. The embedder re-mints on demand +/// (e.g. after a `401` from the iframe) so this can stay short. +const APP_EMBED_TOKEN_VALIDITY_HOURS: i64 = 12; + +#[derive(Serialize)] +pub struct EmbedTokenResponse { + /// Narrowly-scoped token for the iframe. `None` for fully anonymous access + /// (the iframe then calls the public endpoints anonymously). + pub token: Option, + pub expiration: Option>, + /// WIN-2006: raw apps render single-iframe (the bundle is already isolated in + /// its own opaque iframe), so the viewer skips the opaque-viewer indirection + /// and the embed token entirely — it loads the app with the page credential. + #[serde(default)] + pub raw_app: bool, + /// WIN-2006: publisher opted this app into sandbox isolation. When false the + /// viewer runs the app same-origin with its full session (the default, + /// pre-isolation behavior). + #[serde(default)] + pub sandbox: bool, + /// WIN-2006: the resolved app path. The embedder uses it (together with + /// `workspace_id`) to scope the app's backing `localStorage` per app (so + /// sandboxed apps don't share one store). Not a new disclosure — the viewer + /// already receives `path` when it loads the app (e.g. `get_public_app_by_secret`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub app_path: Option, + /// WIN-2006: the resolved workspace. Pairs with `app_path` for the per-app + /// `localStorage` key so two apps at the same path in different workspaces don't + /// share a store. For custom-path apps the viewer can't derive this itself. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub workspace_id: Option, +} + +/// Mint a short-lived, narrowly-scoped embed token for `app_path` when a caller +/// is authenticated. When `opt_authed` is `None` (anonymous access to an +/// anonymous app) no token is minted and the iframe relies on the public +/// endpoints. +/// +/// The CALLER MUST verify the viewer's access to `app_path` before calling: this +/// mints a token on behalf of `opt_authed` unconditionally (DB access remains +/// gated by the viewer's own RLS, but the token's existence is not access-checked +/// here). All current call sites (`get_app_embed_token`, +/// `get_app_embed_token_for_path`, and the EE custom-path variant) do this. +/// +/// Scope confinement IS enforced here: the minted scopes must be within the +/// caller's own (`ensure_scopes_within_caller`), so a scope-restricted bearer +/// token cannot bootstrap a broader-scoped embed token. For the normal caller — +/// an unscoped browser session — this is a no-op and the mint is purely +/// narrowing. +pub async fn mint_app_embed_token( + db: &DB, + w_id: &str, + app_path: &str, + opt_authed: Option<&ApiAuthed>, +) -> Result { + let token_and_exp = if let Some(authed) = opt_authed { + // An app embed token represents untrusted app JS in the sandboxed iframe; it + // must never reach this mint path to renew itself. The 12h expiry is the + // blast-radius cap on a leaked embed token, and `ensure_scopes_within_caller` + // below would pass a same-scoped renewal (the requested scopes equal the + // caller's own), making the credential indefinitely self-renewable. Refresh + // minting is the trusted embedder session/JWT's job. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotAuthorized( + "App embed tokens cannot mint or renew embed tokens".to_string(), + )); + } + let expiration = + chrono::Utc::now() + chrono::Duration::hours(APP_EMBED_TOKEN_VALIDITY_HOURS); + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Path-scoped read so the app can fetch its OWN definition (apps/get/p, + // which the in-workspace sandboxed viewer uses) — but no other app's. The + // public viewer fetches via apps_u/public_app and doesn't rely on this. + scopes.push(format!("apps:read:{app_path}")); + // Path-scoped run (NOT unqualified `apps:run`) so the token can only execute + // THIS app's components: `execute_component` re-checks `apps:run:` for + // the requested app, so the token can't drive another app's runnables. + scopes.push(format!("apps:run:{app_path}")); + // A scope-restricted caller token must not bootstrap a broader-scoped + // embed token (`create_token_internal` deliberately does not check this + // itself). No-op for unscoped sessions — the normal embed flow. + ensure_scopes_within_caller(authed, Some(&scopes))?; + let token_config = NewToken::new( + Some(format!("embed_app:{app_path}")), + Some(expiration), + None, + Some(scopes), + Some(w_id.to_string()), + // Never let an embed token gain write capability the caller's own + // session lacks. + Some(authed.read_only), + ); + let mut tx = db.begin().await?; + let token = create_token_internal(&mut *tx, db, authed, token_config).await?; + tx.commit().await?; + Some((token, expiration)) + } else { + None + }; + + Ok(EmbedTokenResponse { + token: token_and_exp.as_ref().map(|(t, _)| t.clone()), + expiration: token_and_exp.map(|(_, e)| e), + raw_app: false, + sandbox: false, + app_path: Some(app_path.to_string()), + workspace_id: Some(w_id.to_string()), + }) +} + +/// Issue an embed token for a public app addressed by its (secret) share id. +/// Mirrors the access check in [`get_public_app_by_secret`]: anonymous apps are +/// reachable without auth, otherwise the caller must be logged in and have read +/// access to the app. +async fn get_app_embed_token( + OptAuthed(opt_authed): OptAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, secret)): Path<(String, String)>, +) -> JsonResult { + let id = get_id_from_secret(&db, &w_id, secret, None).await?; + + let app = sqlx::query!( + "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.id = $1 AND a.workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + let app = not_found_if_none(app, "App", id.to_string())?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient field-level read instead of a strict `Policy` parse: a legacy app + // whose stored policy predates newer required fields must still resolve to + // its (unsandboxed) render here rather than erroring out of the viewer. + let policy = parse_embed_policy(&policy_str)?; + + let authed_for_token = if policy.anonymous_execution { + // Anonymous app: still mint a scoped token if the viewer happens to be + // logged in (so the app sees their identity), otherwise stay anonymous. + opt_authed + } else { + let authed = opt_authed.ok_or_else(|| { + Error::NotAuthorized( + "App visibility does not allow public access and you are not logged in".to_string(), + ) + })?; + let mut tx = user_db.begin(&authed).await?; + let is_visible = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM app WHERE id = $1 AND workspace_id = $2)", + id, + &w_id + ) + .fetch_one(&mut *tx) + .await?; + tx.commit().await?; + if !is_visible.unwrap_or(false) { + return Err(Error::NotAuthorized( + "App visibility does not allow public access and you are logged in but you have no read-access to that app".to_string(), + )); + } + Some(authed) + }; + + // The token is only consumed by the sandboxed low-code render. Raw apps + // render single-iframe with the page credential (WIN-2006 Variant A), and + // unsandboxed apps render same-origin with the viewer's own session — minting + // for those would write a useless token row per view and, worse, could fail + // the whole render for a scope-restricted caller (`ensure_scopes_within_caller`) + // even though no token is needed. The access check above still gates + // visibility in every case. + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, &app.path, authed_for_token.as_ref()).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(app.path); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + +/// Minimal, lenient view of an app policy for the embed-token endpoints +/// (WIN-2006). Reads only the fields the sandbox decision needs, via +/// `serde_json::Value`, so a legacy policy that no longer satisfies the strict +/// [`Policy`] struct (e.g. `triggerables_v2` entries predating now-required +/// fields) still renders instead of failing the viewer with "Not found". +/// A missing/unknown `execution_mode` is treated as NOT anonymous — the +/// strictest access interpretation. +pub struct EmbedPolicyView { + pub anonymous_execution: bool, + pub sandbox: bool, +} + +pub fn parse_embed_policy(policy_str: &str) -> Result { + let v: serde_json::Value = serde_json::from_str(policy_str).map_err(to_anyhow)?; + Ok(EmbedPolicyView { + anonymous_execution: v.get("execution_mode").and_then(|m| m.as_str()) == Some("anonymous"), + sandbox: v.get("sandbox").and_then(|b| b.as_bool()).unwrap_or(false), + }) +} + +/// Authenticated, path-based embed token for the in-workspace app viewer +/// (WIN-2006). Mirrors [`get_app_embed_token`] but keyed by app path and gated by +/// the caller's read access (RLS), so the logged-in `/apps/get` viewer can render +/// the app sandboxed — isolated from the member's full session — using the same +/// scoped token. Raw apps get no token (single-iframe with the page credential). +async fn get_app_embed_token_for_path( + authed: ApiAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, +) -> JsonResult { + let path = path.to_path(); + check_scopes(&authed, || format!("apps:read:{}", path))?; + // RLS: the caller must have read access to this app, otherwise it's not found. + let mut tx = user_db.begin(&authed).await?; + let app = sqlx::query!( + "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.path = $1 AND a.workspace_id = $2", + path, + &w_id + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let app = not_found_if_none(app, "App", path)?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient parse + mint only for the sandboxed low-code render — see + // [`get_app_embed_token`] for the rationale (identical here). + let policy = parse_embed_policy(&policy_str)?; + + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, path, Some(&authed)).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(path.to_string()); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + async fn get_id_from_secret( db: &DB, w_id: &str, @@ -1245,8 +1717,6 @@ async fn create_app_raw<'a>( ) .await?; - check_scopes(&authed, || format!("apps:write:{}", path))?; - webhook.send_message( w_id.clone(), WebhookMessage::CreateApp { workspace: w_id, path: path.clone() }, @@ -1290,7 +1760,6 @@ async fn create_app( )); } let path = app.path.clone(); - check_scopes(&authed, || format!("apps:write:{}", &path))?; if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, @@ -1304,6 +1773,7 @@ async fn create_app( return Err(Error::PermissionDenied(msg)); } + // scope is enforced inside create_app_internal, before any persistence. let (new_tx, _path, _id) = create_app_internal(authed, db, user_db, &w_id, false, app).await?; new_tx.commit().await?; @@ -1350,6 +1820,10 @@ async fn create_app_internal<'a>( raw_app: bool, mut app: CreateApp, ) -> Result<(sqlx::Transaction<'a, sqlx::Postgres>, String, i64)> { + // Enforce scope before any persistence: the raw-app create path commits + // inside process_app_multipart!, so checking after this call would leave a + // denied app committed in the DB. + check_scopes(&authed, || format!("apps:write:{}", &app.path))?; if *CLOUD_HOSTED { let nb_apps = sqlx::query_scalar!("SELECT COUNT(*) FROM app WHERE workspace_id = $1", &w_id) @@ -1367,7 +1841,9 @@ async fn create_app_internal<'a>( )); } } - let mut tx = user_db.clone().begin(&authed).await?; + // Resolve the on-behalf-of defaults on the (non-RLS) pool *before* opening + // the RLS transaction below: doing these lookups mid-transaction would hold + // a second simultaneous connection while `tx` is still checked out. let should_preserve = app.preserve_on_behalf_of.unwrap_or(false) && windmill_common::can_preserve_on_behalf_of(&authed) && app.policy.on_behalf_of.is_some(); @@ -1393,6 +1869,8 @@ async fn create_app_internal<'a>( app.policy.on_behalf_of_email = Some(authed.email.clone()); } } + + let mut tx = user_db.clone().begin(&authed).await?; let path = app.path.clone(); if &app.path == "" { return Err(Error::BadRequest("App path cannot be empty".to_string())); @@ -1891,6 +2369,13 @@ async fn update_app_internal<'a>( ns: EditApp, ) -> Result<(sqlx::Transaction<'a, sqlx::Postgres>, String, i64)> { use sql_builder::prelude::*; + + // A rename moves the app to ns.path, so the destination must also be within + // the token's write scope, not just the source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("apps:write:{}", npath))?; + } + let mut tx = user_db.clone().begin(&authed).await?; let mut preserved_on_behalf_of: Option = None; @@ -2276,6 +2761,17 @@ async fn execute_component( Path((w_id, path)): Path<(String, StripPath)>, Json(mut payload): Json, ) -> Result { + let path = path.to_path(); + // Authorize FIRST, before touching the payload: confine the app embed token (the + // only credential handed to untrusted app JS, carrying `apps:run:`) to + // the app it was minted for. The route layer can't path-check the apps domain, so + // enforce it here. Scoped to embed tokens only — other callers (anonymous, cookie, + // plain external JWT) keep their existing access; the run is still policy-gated. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path))?; + } + } // Only honor temp_script_refs for the inline-script preview path: // preview/editor mode (force_viewer_static_fields set, == `is_preview`), // raw_code present, and no deployed app_script id — i.e. `wmill app dev`. @@ -2298,7 +2794,6 @@ async fn execute_component( _ => {} }; - let path = path.to_path(); let (arc_policy, policy): (Arc, Policy); let policy_triggerables_default = Default::default(); // Preview mode means the request was issued from the editor; the editing @@ -2762,6 +3257,15 @@ async fn upload_s3_file_from_app( Query(query): Query, request: axum::extract::Request, ) -> JsonResult { + // Confine an app embed token (untrusted app JS) to uploading for its OWN app. + // The route is reachable with `apps:run` (RUN_PATH_ACTIONS), so without this a + // token minted for app A could drive app B's upload policy. Mirrors + // execute_component / download_s3_file; other callers are unaffected. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path.to_path()))?; + } + } let policy = if let Some(file_key_regex) = query.force_viewer_file_key_regex { // `force_viewer_*` lets the caller supply a synthetic upload policy that // bypasses the deployed app's file_key_regex / resource restrictions. @@ -2796,6 +3300,7 @@ async fn upload_s3_file_from_app( .unwrap_or_default(), }]), allowed_s3_keys: None, + sandbox: None, }) } else { let policy_o = sqlx::query_scalar!( @@ -3155,6 +3660,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: Some(force_allowed_s3_keys), + sandbox: None, } } else { // TODO: improve db query to not return uneeded fields @@ -3177,6 +3683,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: None, + sandbox: None, }) }; @@ -3220,34 +3727,46 @@ async fn check_if_allowed_to_access_s3_file_from_app( return Err(Error::InternalErr( "Internal error: signature validation is not supported in open source mode".to_string(), )); - } else if opt_authed.is_some() { + } else if opt_authed.as_ref().is_some_and(|authed| { + !windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) + }) { + // A normal logged-in caller (editor / full session) may fetch any file they + // can reach. An app embed token also carries an identity but represents + // untrusted app JS, so it falls through to the allowlist below instead of + // this bypass — otherwise the app could read arbitrary S3 keys the + // viewer/on-behalf identity can see, beyond its own declared keys/outputs. Ok(()) } else { - let allowed = policy - .allowed_s3_keys + // Anonymous viewer, or an app embed token: confine to the app's declared S3 + // keys, or files produced by THIS app's own component runs. The producing + // identity is the embed viewer for a token, else `anonymous`. + let creator = opt_authed .as_ref() - .unwrap() - .iter() - .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) - || { - sqlx::query_scalar!( - r#"SELECT EXISTS ( + .map(|authed| authed.username.clone()) + .unwrap_or_else(|| "anonymous".to_string()); + let allowed = policy.allowed_s3_keys.as_ref().is_some_and(|keys| { + keys.iter() + .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) + }) || { + sqlx::query_scalar!( + r#"SELECT EXISTS ( SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id) WHERE j.workspace_id = $2 AND (j.kind = 'appscript' OR j.kind = 'preview') - AND j.created_by = 'anonymous' + AND j.created_by = $4 AND c.started_at > now() - interval '3 hours' AND j.runnable_path LIKE $3 || '/%' AND c.result @> ('{"s3":"' || $1 || '"}')::jsonb )"#, - file_query.s3, - w_id, - path, - ) - .fetch_one(db) - .await? - .unwrap_or(false) - }; + file_query.s3, + w_id, + path, + creator, + ) + .fetch_one(db) + .await? + .unwrap_or(false) + }; if !allowed { Err(Error::BadRequest("File restricted".to_string())) @@ -3286,6 +3805,15 @@ async fn download_s3_file_from_app( let path = path.to_path(); + // Authorize the app path first: a scoped caller (notably an app embed token, + // which carries `apps:read:`) may only download files for the app it + // was minted for — otherwise it could read another app's S3 files via that app's + // on-behalf policy. Unscoped sessions / anonymous callers pass through (the + // latter still gated by the policy allowlist in `check_if_allowed_...`). + if let Some(authed) = opt_authed.as_ref() { + check_scopes(authed, || format!("apps:read:{}", path))?; + } + let force_viewer_allowed_s3_keys = if let Some(force_viewer_allowed_s3_keys) = query.force_viewer_allowed_s3_keys.clone() { @@ -3562,3 +4090,253 @@ async fn build_args( job_id, )) } + +#[cfg(test)] +mod embed_token_tests { + use super::APP_EMBED_SCOPES; + use windmill_api_auth::scopes::check_scopes_for_route; + + /// The embed token must reach exactly the endpoints an app needs and nothing + /// else. This locks the allow/deny matrix that confines a malicious or + /// compromised app to app-only routes (WIN-2006). + #[test] + fn embed_scopes_allow_app_routes_and_deny_the_rest() { + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Mirror mint_app_embed_token: the per-app path-scoped read + run. + scopes.push("apps:read:u/admin/app".to_string()); + scopes.push("apps:run:u/admin/app".to_string()); + let scopes = Some(scopes.as_slice()); + + // Allowed: the routes a running app legitimately calls. + let allowed = [ + // Own definition + the public app-serving / execution endpoints. + ("/api/w/test/apps/get/p/u/admin/app", "GET"), + ("/api/w/test/apps_u/public_app/secret", "GET"), + ("/api/w/test/apps_u/get_data/v/secret.js", "GET"), + ("/api/w/test/apps_u/public_resource/f/app_themes/t", "GET"), + ("/api/w/test/apps_u/execute_component/u/admin/app", "POST"), + // S3 file upload from the app's S3 File Input component: a `run` action + // (RUN_PATH_ACTIONS) so the embed token reaches it; the handler re-checks + // `apps:run:` to confine it to this app, like execute_component. + ("/api/w/test/apps_u/upload_s3_file/u/admin/app", "POST"), + // By-id job poll routes (the JobLoader surface) stay allowed. + ("/api/w/test/jobs_u/get/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate_sse/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_result/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_timing/some-uuid", "GET"), + // By-id cancel (POST): permitted at the route layer; the handler confines + // it to the app's own jobs (created_by == viewer). + ("/api/w/test/jobs_u/queue/cancel/some-uuid", "POST"), + ("/api/w/test/users/whoami", "GET"), + // Resource METADATA only (picker list + type schemas) — never values. + ("/api/w/test/resources/list", "GET"), + ("/api/w/test/resources/exists/u/admin/r", "GET"), + ("/api/w/test/resources/type/list", "GET"), + ("/api/w/test/folders/listnames", "GET"), + ]; + for (path, method) in allowed { + assert!( + check_scopes_for_route(scopes, path, method).is_ok(), + "embed token should allow {method} {path}" + ); + } + + // Denied: anything outside what an app needs, including app management + // (apps:write is intentionally withheld), resource VALUE reads (which can + // hold credentials), and other workspace domains. + let denied = [ + ("/api/w/test/apps/update/u/admin/app", "POST"), + ("/api/w/test/apps/delete/u/admin/app", "DELETE"), + // Workspace app inventory must NOT be reachable (Apps domain is + // default-denied for the embed sentinel; only own-def + apps_u/* allowed). + ("/api/w/test/apps/exists/u/admin/app", "GET"), + ("/api/w/test/apps/custom_path_exists/foo", "GET"), + ( + "/api/w/test/apps/list_paths_from_workspace_runnable/script/u/admin/x", + "GET", + ), + ("/api/w/test/apps/list", "GET"), + // The embed-token MINT endpoints are public app routes (`apps_u/`) but + // create credentials — denied so a captured embed token can't renew + // itself indefinitely past the 12h expiry (refresh is the embedder's job). + ("/api/w/test/apps_u/embed_token/secret", "GET"), + ("/api/w/test/apps_u/embed_token_by_custom_path/foo", "GET"), + ("/api/w/test/scripts/list", "GET"), + ("/api/w/test/variables/list", "GET"), + ("/api/w/test/resources/update/u/admin/r", "POST"), + // Resource value reads must NOT be reachable with the embed token. + ("/api/w/test/resources/get/u/admin/r", "GET"), + ("/api/w/test/resources/get_value/u/admin/r", "GET"), + ( + "/api/w/test/resources/get_value_interpolated/u/admin/r", + "GET", + ), + ("/api/w/test/resources/list_search", "GET"), + // Workspace-wide job enumeration/export must NOT be reachable — an app + // reads only jobs it launched, by id (blocked via the app_embed sentinel). + ("/api/w/test/jobs/list", "GET"), + ("/api/w/test/jobs/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/completed/list", "GET"), + ("/api/w/test/jobs/completed/export", "GET"), + ("/api/w/test/jobs/queue/list", "GET"), + ("/api/w/test/jobs/queue/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/queue/export", "GET"), + // Job counts (workspace-wide aggregates) and the capability-minting + // routes (signed resume/approval URLs) are NOT by-id polling — denied. + ("/api/w/test/jobs/completed/count", "GET"), + ("/api/w/test/jobs/completed/count_jobs", "GET"), + ("/api/w/test/jobs/queue/count", "GET"), + ("/api/w/test/jobs/job_signature/some-uuid/some-rid", "GET"), + ("/api/w/test/jobs/resume_urls/some-uuid/some-rid", "GET"), + // get_root_job_id has no access check in its handler and the app never + // calls it — denied so the token can't probe foreign jobs' flow lineage. + ("/api/w/test/jobs_u/get_root_job_id/some-uuid", "GET"), + // `users:read`/`folders:read` exist only for whoami/listnames — every + // other route in those domains is denied via the app_embed sentinel + // (the whole /users and /folders routers are CORS-enabled for the iframe). + ("/api/w/test/users/list", "GET"), + ("/api/w/test/users/list_usage", "GET"), + ("/api/w/test/users/username_to_email/admin", "GET"), + ("/api/w/test/folders/list", "GET"), + ("/api/w/test/folders/get/myfolder", "GET"), + ("/api/w/test/folders/getusage/myfolder", "GET"), + ]; + for (path, method) in denied { + assert!( + check_scopes_for_route(scopes, path, method).is_err(), + "embed token should deny {method} {path}" + ); + } + } + + /// `apps:run` satisfies read at the route layer, so `apps/list` / `apps/list_search` + /// pass the route check — that's why those handlers ALSO call + /// `check_scopes(apps:read)`, which uses `ScopeDefinition::includes` (where run + /// does NOT include read). Lock that: no embed scope, including the + /// dynamically-minted path-scoped read, satisfies a domain-level `apps:read`, so + /// the token cannot list all apps' definitions (their full `value`/code). + #[test] + fn embed_scopes_cannot_satisfy_domain_app_read() { + use windmill_api_auth::scopes::ScopeDefinition; + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // mint_app_embed_token also grants read scoped to the single app path: + scopes.push("apps:read:u/admin/app".to_string()); + let required = ScopeDefinition::from_scope_string("apps:read").unwrap(); + for s in &scopes { + // The `app_embed` sentinel intentionally doesn't parse as a domain:action + // scope (it grants nothing; it only drives the job-enumeration deny). + let Ok(def) = ScopeDefinition::from_scope_string(s) else { + continue; + }; + assert!( + !def.includes(&required), + "embed scope {s} must not satisfy domain-level apps:read (would leak apps/list[_search])" + ); + } + // Sanity: a genuine domain-level apps:read token does satisfy it. + assert!(ScopeDefinition::from_scope_string("apps:read") + .unwrap() + .includes(&required)); + } + + /// The token carries path-scoped `apps:run:` and `apps:read:` + /// (NOT unqualified `apps:run`). Every handler that resolves an app and acts on + /// its behalf re-checks the requested path via `ScopeDefinition::includes`, so the + /// token is confined to its OWN app: + /// - `apps:run:` — `execute_component`. + /// - `apps:read:` — `get_app` (apps/get/p), `get_public_app_by_secret`, + /// the EE custom-path `get_public_app_by_custom_path`, and + /// `download_s3_file_from_app`. + /// This blocks cross-app execution, definition reads (by secret / custom path), + /// and S3 file reads through another app's on-behalf policy. + #[test] + fn embed_run_scope_is_path_scoped_to_its_app() { + use windmill_api_auth::scopes::ScopeDefinition; + // The mint must not grant unqualified run (which would include any path). + assert!( + !APP_EMBED_SCOPES.contains(&"apps:run"), + "embed scopes must not include unqualified apps:run" + ); + for action in ["run", "read"] { + let own = + ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")).unwrap(); + assert!( + own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")) + .unwrap() + ), + "apps:{action} must grant its own app" + ); + assert!( + !own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/other")) + .unwrap() + ), + "apps:{action} must NOT grant another app (cross-app)" + ); + } + } + + /// `mint_app_embed_token` guards its `create_token_internal` call with + /// `ensure_scopes_within_caller`, so a scope-restricted bearer token cannot + /// bootstrap the broader embed-scope set. Lock that boundary on the exact + /// scope vec the mint builds: rejected for a path-scoped caller, no-op for + /// the unscoped browser session that is the normal embed flow. + #[test] + fn embed_token_mint_is_scope_bounded() { + use windmill_api_auth::{ensure_scopes_within_caller, ApiAuthed}; + + // Same scope set mint_app_embed_token assembles for an app. + let mut minted: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + minted.push("apps:read:u/admin/app".to_string()); + + // A caller restricted to a single app read must not widen to the full + // embed set (apps:run, jobs:read, resources:read, ...). + let restricted = ApiAuthed { + scopes: Some(vec!["apps:read:u/admin/app".to_string()]), + ..Default::default() + }; + assert!( + ensure_scopes_within_caller(&restricted, Some(&minted)).is_err(), + "a path-scoped caller must not mint the broader embed-scope set" + ); + + // An unscoped session (the normal embed flow) passes — the mint only + // narrows. + let unscoped = ApiAuthed { scopes: None, ..Default::default() }; + assert!(ensure_scopes_within_caller(&unscoped, Some(&minted)).is_ok()); + } + + /// The embed-token endpoints must keep working for legacy apps whose stored + /// policy no longer satisfies the strict `Policy` struct (pre-dating + /// now-required fields): `parse_embed_policy` reads only the sandbox-decision + /// fields, leniently, and treats a missing/unknown `execution_mode` as NOT + /// anonymous (the strictest access interpretation). + #[test] + fn embed_policy_parse_is_lenient() { + use super::parse_embed_policy; + + // Quirky legacy policy: triggerables_v2 entry missing required fields, + // no execution_mode at all — must still parse, and absent `sandbox` + // resolves to the unsandboxed default. + let p = parse_embed_policy(r#"{"triggerables_v2": {"x": {}}}"#).unwrap(); + assert!(!p.sandbox); + assert!( + !p.anonymous_execution, + "missing execution_mode must not grant anonymous access" + ); + + // Normal policies map field-for-field. + let p = parse_embed_policy(r#"{"execution_mode": "anonymous", "sandbox": true}"#).unwrap(); + assert!(p.anonymous_execution); + assert!(p.sandbox); + + // Unknown execution_mode value: lenient parse, but not anonymous. + let p = parse_embed_policy(r#"{"execution_mode": "weird"}"#).unwrap(); + assert!(!p.anonymous_execution); + + // Invalid JSON still errors. + assert!(parse_embed_policy("not json").is_err()); + } +} diff --git a/backend/windmill-api/src/db.rs b/backend/windmill-api/src/db.rs index 7b8d468da8..b81a9dbb84 100644 --- a/backend/windmill-api/src/db.rs +++ b/backend/windmill-api/src/db.rs @@ -84,6 +84,12 @@ lazy_static::lazy_static! { (20260228000000, include_str!( "../../migrations/20260228000000_v2_job_completed_failure_index.up.sql" ).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")), + (20260610151334, include_str!( + "../../migrations/20260610151334_folder_labels.up.sql" + ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()), + (20260614075900, include_str!( + "../../migrations/20260614075900_dedup_folder_labels.up.sql" + ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()), ].into_iter().collect(); } @@ -288,6 +294,12 @@ pub async fn migrate( // idempotent, so re-applying on an already-migrated DB is a no-op. 20260423050000, 20260523055641, + // Reworked to stop reading pg_authid (via pg_has_role) from an elevated + // context, which managed providers (e.g. Cloud SQL) forbid — the original + // aborted startup. The new file is idempotent (CREATE OR REPLACE + an + // epoch-guarded UPDATE that no-ops once anchored), so re-applying on an + // already-migrated DB is safe. + 20260626132251, ]; for m in migrator.migrations.iter() { if m.migration_type.is_down_migration() { diff --git a/backend/windmill-api/src/docs/corpus.rs b/backend/windmill-api/src/docs/corpus.rs new file mode 100644 index 0000000000..a797879cfe --- /dev/null +++ b/backend/windmill-api/src/docs/corpus.rs @@ -0,0 +1,86 @@ +//! The vendored documentation snapshot, embedded into the binary and parsed once. +//! +//! `docs_snapshot/*.gz` are refreshed by `docs_snapshot/fetch.sh`. Embedding them +//! lets docs search work with no runtime egress (including air-gapped instances). + +use std::io::Read; +use std::sync::OnceLock; + +use flate2::read::GzDecoder; + +use super::search::{ + canonical_docs_page_url, canonical_search_url, parse_docs_full_text, parse_docs_index, + DocsFullPage, DocsIndexEntry, +}; + +const LLMS_FULL_GZ: &[u8] = + include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs_snapshot/llms-full.txt.gz")); +const LLMS_INDEX_GZ: &[u8] = + include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs_snapshot/llms.txt.gz")); + +pub struct DocsCorpus { + /// Every docs page, keyed by its `Source:` URL (from llms-full.txt). + pub pages: Vec, + /// The curated page index with one-line descriptions (from llms.txt). + pub index: Vec, +} + +impl DocsCorpus { + /// Finds the page whose `Source:` URL matches a model/CLI-supplied path or URL + /// after canonicalization (origin re-anchored, `.md` and ordering prefixes + /// stripped). + pub fn find_page(&self, path: &str) -> Option<&DocsFullPage> { + let key = canonical_search_url(&canonical_docs_page_url(path)); + self.pages.iter().find(|p| canonical_search_url(&p.url) == key) + } +} + +static CORPUS: OnceLock = OnceLock::new(); + +fn decompress(bytes: &[u8]) -> String { + let mut out = String::new(); + if let Err(e) = GzDecoder::new(bytes).read_to_string(&mut out) { + // The embedded snapshot is valid gzip text; a decode failure is a + // build-time packaging error, so failing closed to an empty corpus + // (docs search returns "no matches") is acceptable. + tracing::error!("failed to decompress embedded docs snapshot: {e}"); + return String::new(); + } + out +} + +/// Returns the parsed docs corpus, decompressing and parsing the embedded +/// snapshot once on first access. +pub fn corpus() -> &'static DocsCorpus { + CORPUS.get_or_init(|| { + let pages = parse_docs_full_text(&decompress(LLMS_FULL_GZ)); + let index = parse_docs_index(&decompress(LLMS_INDEX_GZ)); + DocsCorpus { pages, index } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn embedded_corpus_parses_to_non_empty() { + let corpus = corpus(); + assert!(corpus.pages.len() > 50, "expected many pages, got {}", corpus.pages.len()); + assert!(corpus.index.len() > 50, "expected many index entries, got {}", corpus.index.len()); + assert!(corpus + .pages + .iter() + .all(|p| p.url.starts_with("https://www.windmill.dev/docs/") && !p.body.is_empty())); + } + + #[test] + fn find_page_matches_by_canonical_url() { + let corpus = corpus(); + // A page that is expected to exist in the published docs. + let by_path = corpus.find_page("/docs/core_concepts/worker_groups"); + let by_url = corpus.find_page("https://www.windmill.dev/docs/core_concepts/worker_groups.md"); + assert!(by_path.is_some()); + assert_eq!(by_path.map(|p| &p.url), by_url.map(|p| &p.url)); + } +} diff --git a/backend/windmill-api/src/docs/mod.rs b/backend/windmill-api/src/docs/mod.rs new file mode 100644 index 0000000000..cd903e593f --- /dev/null +++ b/backend/windmill-api/src/docs/mod.rs @@ -0,0 +1,124 @@ +//! Self-hosted documentation search. +//! +//! The backend embeds a vendored docs snapshot (see [`corpus`]) and exposes two +//! read-only endpoints over it, so docs search works with no runtime egress: +//! - `GET /api/docs/search?query=...` — full-text + index search +//! - `GET /api/docs/page?url=...§ion=...` — read one page (or a section) +//! +//! These back the AI chat `search_docs`/`read_docs_page` tools, the MCP +//! `searchDocs`/`readDocsPage` tools, and the `wmill docs` CLI. The routes are +//! nested behind the global authed service in `lib.rs`, so a valid token is +//! required but no workspace. + +mod corpus; +mod search; + +use axum::{extract::Query, routing::get, Json, Router}; +use serde::{Deserialize, Serialize}; +use windmill_common::error::JsonResult; + +use search::DocsSearchResult; + +pub fn global_service() -> Router { + Router::new() + .route("/search", get(search_docs)) + .route("/page", get(read_docs_page)) +} + +#[derive(Deserialize)] +struct SearchQuery { + query: String, +} + +#[derive(Serialize)] +struct SearchResponse { + /// Model-ready rendering of the results (the exact string the AI/MCP tool + /// returns). Built once here so every consumer is identical. + text: String, + /// Structured results for non-AI consumers (e.g. the CLI's pretty/`--json`). + results: Vec, +} + +#[derive(Deserialize)] +struct PageQuery { + /// A page's `Source` URL (as returned by the docs search tool); a bare `/docs/...` + /// path is also accepted and canonicalized before lookup. + url: String, + section: Option, +} + +#[derive(Serialize)] +struct PageResponse { + text: String, + source_url: String, +} + +async fn search_docs(Query(q): Query) -> JsonResult { + let query = q.query.trim().to_string(); + if query.is_empty() { + return Ok(Json(SearchResponse { + text: "No search query was provided. Provide a `query` of one or more keywords." + .to_string(), + results: Vec::new(), + })); + } + + // Lazy corpus init (gzip decompress + parse) and the per-query full-corpus scan + // are CPU-bound; keep them off the async runtime. + let (text, results) = tokio::task::spawn_blocking(move || { + let corpus = corpus::corpus(); + // Body grep first (concrete content hits), then index titles/descriptions to + // surface named features body grep misses; merge dedupes by canonical URL. + let body = search::search_docs_pages(&corpus.pages, &query, 5); + let index = search::search_docs_index(&corpus.index, &query, 4); + let results = search::merge_docs_search_results(body, index, search::SEARCH_MAX_PAGES); + let text = search::format_docs_search_results(&query, &results); + (text, results) + }) + .await + .map_err(|e| windmill_common::error::Error::InternalErr(format!("docs search task: {e}")))?; + + Ok(Json(SearchResponse { text, results })) +} + +async fn read_docs_page(Query(q): Query) -> JsonResult { + let url = q.url.trim(); + if url.is_empty() { + return Ok(Json(PageResponse { + text: "No documentation page URL was provided. Provide a `url` — e.g. a `Source` URL returned by the docs search tool.".to_string(), + source_url: String::new(), + })); + } + + let url = url.to_string(); + let section = q.section.filter(|s| !s.trim().is_empty()); + + // Corpus init + page sanitize/render are CPU-bound; keep them off the runtime. + let resp = tokio::task::spawn_blocking(move || { + let corpus = corpus::corpus(); + match corpus.find_page(&url) { + Some(page) => { + // Rewrite docusaurus source-file links to canonical published URLs + // so the model never echoes a broken `.mdx` path. + let sanitized = search::sanitize_docs_markdown_links(&page.body, &page.url); + let rendered = search::render_docs_page_result(&sanitized, section.as_deref()); + let text = format!( + "Source page — cite this URL when referencing this page: {}\n\n{}", + page.url, rendered + ); + PageResponse { text, source_url: page.url.clone() } + } + None => PageResponse { + text: format!( + "No documentation page found for \"{}\". Use the docs search tool to find the correct Source URL first.", + url + ), + source_url: search::canonical_docs_page_url(&url), + }, + } + }) + .await + .map_err(|e| windmill_common::error::Error::InternalErr(format!("docs page task: {e}")))?; + + Ok(Json(resp)) +} diff --git a/backend/windmill-api/src/docs/search.rs b/backend/windmill-api/src/docs/search.rs new file mode 100644 index 0000000000..733558ceb4 --- /dev/null +++ b/backend/windmill-api/src/docs/search.rs @@ -0,0 +1,729 @@ +//! Documentation search & page rendering. +//! +//! Direct port of the pure functions in the frontend's +//! `copilot/chat/docs/core.ts`, so the AI chat, the MCP `searchDocs`/`readDocsPage` +//! tools and the `wmill docs` CLI all return identical results from one place. +//! Operates over the vendored corpus parsed in [`super::corpus`]. + +use lazy_static::lazy_static; +use regex::Regex; +use serde::Serialize; +use std::collections::HashSet; + +pub const DOCS_ORIGIN: &str = "https://www.windmill.dev"; + +// Above this size, return an outline of the page's headings instead of the full +// content, prompting the model to request a specific section. +const FULL_PAGE_CHAR_LIMIT: usize = 20_000; + +// search result caps — keep the returned payload small (the whole point of search +// vs. dumping the index or full pages is token economy). +pub const SEARCH_MAX_PAGES: usize = 8; +const SEARCH_MAX_SNIPPETS_PER_PAGE: usize = 3; +const SEARCH_MAX_SNIPPET_CHARS: usize = 200; +// Each distinct query term triggers a full-corpus scan; cap it so a long, +// caller-controlled query can't multiply the scan cost without bound. Real +// queries are a handful of keywords, so this never truncates a useful search. +const MAX_QUERY_TERMS: usize = 24; + +// --------------------------------------------------------------------------- +// Corpus record types +// --------------------------------------------------------------------------- + +/// A single page extracted from llms-full.txt, keyed by its `Source:` URL. +pub struct DocsFullPage { + pub url: String, + pub title: String, + pub body: String, + /// `body` lowercased once at parse time, so the per-query full-corpus scan + /// doesn't re-allocate a lowercase copy of every page on each request. + pub body_lower: String, +} + +/// A line of the llms.txt index: title, URL and one-line description. +pub struct DocsIndexEntry { + pub title: String, + pub url: String, + pub description: String, + /// `title`/`description` lowercased once at parse time (same rationale as + /// `DocsFullPage::body_lower`). + pub title_lower: String, + pub description_lower: String, +} + +#[derive(Serialize, Clone)] +pub struct DocsSearchResult { + pub url: String, + pub title: String, + /// Higher = more relevant. Distinct query terms matched dominate raw occurrences. + pub score: i64, + pub snippets: Vec, +} + +// --------------------------------------------------------------------------- +// Corpus parsing +// --------------------------------------------------------------------------- + +lazy_static! { + // In llms-full.txt every page's `Source:` line is preceded by a category-header + // lead-in: `...page body...\n\n---\n\n## \n\nSource: `. Splitting + // on `Source:` lines leaves that lead-in on the *previous* page, so strip a + // trailing `---` + level-2-heading block to avoid mis-attributing the next + // page's category title to the previous page. + static ref TRAILING_LEAD_IN_RE: Regex = + Regex::new(r"\n+-{3,}[ \t]*\n+#{2}[ \t]+.*[ \t]*\n*$").unwrap(); + // A line in llms.txt: `- [Title](https://.../page.md): question-phrased description`. + static ref INDEX_ENTRY_RE: Regex = + Regex::new(r"^\s*-\s*\[([^\]]+)\]\(([^)\s]+)\)\s*:?\s*(.*)$").unwrap(); + static ref ORDERING_PREFIX_RE: Regex = Regex::new(r"^\d+[_-]").unwrap(); + // Markdown inline link `](target "optional title")`. + static ref MD_LINK_RE: Regex = Regex::new(r#"\]\(([^)\s]+?)(\s+"[^"]*")?\)"#).unwrap(); +} + +/// `^Source:\s*(\S+)\s*$` — returns the single non-whitespace URL token. +fn parse_source_line(line: &str) -> Option<&str> { + let rest = line.strip_prefix("Source:")?; + let trimmed = rest.trim(); + if trimmed.is_empty() || trimmed.contains(char::is_whitespace) { + return None; + } + Some(trimmed) +} + +/// Splits the llms-full.txt corpus into per-page records keyed by the `Source:` +/// URL. Content before the first `Source:` line (the corpus preamble) is dropped. +pub fn parse_docs_full_text(full_text: &str) -> Vec { + let mut pages = Vec::new(); + let mut url: Option = None; + let mut buffer: Vec<&str> = Vec::new(); + + for line in full_text.split('\n') { + if let Some(u) = parse_source_line(line) { + flush_page(&mut pages, &url, &buffer); + url = Some(u.to_string()); + buffer.clear(); + continue; + } + if url.is_some() { + buffer.push(line); + } + } + flush_page(&mut pages, &url, &buffer); + pages +} + +fn flush_page(pages: &mut Vec, url: &Option, buffer: &[&str]) { + let Some(url) = url else { + return; + }; + let joined = buffer.join("\n"); + let body = TRAILING_LEAD_IN_RE.replace(&joined, ""); + let body = body.trim(); + if !body.is_empty() { + let title = first_heading(body).unwrap_or_else(|| url.clone()); + let body = body.to_string(); + let body_lower = body.to_lowercase(); + pages.push(DocsFullPage { url: url.clone(), title, body, body_lower }); + } +} + +fn first_heading(body: &str) -> Option { + for line in body.split('\n') { + if let Some((_, title)) = parse_heading_line(line, 6) { + return Some(title); + } + } + None +} + +/// Parses the llms.txt index into per-page entries (title, URL, description). +pub fn parse_docs_index(index_text: &str) -> Vec { + let mut entries = Vec::new(); + for line in index_text.split('\n') { + if let Some(caps) = INDEX_ENTRY_RE.captures(line) { + let url = caps[2].trim(); + if !url.contains("/docs/") { + continue; + } + let title = caps[1].trim().to_string(); + let description = caps[3].trim().to_string(); + entries.push(DocsIndexEntry { + title_lower: title.to_lowercase(), + description_lower: description.to_lowercase(), + title, + url: url.to_string(), + description, + }); + } + } + entries +} + +// --------------------------------------------------------------------------- +// Headings, outline, section extraction +// --------------------------------------------------------------------------- + +pub struct DocsHeading { + pub level: usize, + pub title: String, + /// Byte offset of the start of the heading line within the document. + pub start_index: usize, +} + +/// `^(#{1,max_level})\s+(.*\S)\s*$` — heading level and trimmed title. +fn parse_heading_line(line: &str, max_level: usize) -> Option<(usize, String)> { + let hashes = line.bytes().take_while(|&b| b == b'#').count(); + if hashes < 1 || hashes > max_level { + return None; + } + let rest = &line[hashes..]; + // require at least one whitespace after the hashes (\s+) + if !rest.starts_with(|c: char| c.is_whitespace()) { + return None; + } + let title = rest.trim(); + if title.is_empty() { + return None; + } + Some((hashes, title.to_string())) +} + +fn match_fence(line: &str) -> Option { + let trimmed = line.trim_start(); + let first = trimmed.chars().next()?; + if first != '`' && first != '~' { + return None; + } + let count = trimmed.chars().take_while(|&c| c == first).count(); + if count >= 3 { + Some(std::iter::repeat(first).take(count).collect()) + } else { + None + } +} + +/// Parses the markdown headings (`#`–`####`) of a docs page, ignoring any +/// heading-like lines inside fenced code blocks (which are common in samples). +pub fn parse_docs_headings(content: &str) -> Vec { + let mut headings = Vec::new(); + let mut offset = 0usize; + let mut fence_marker: Option = None; + + for line in content.split('\n') { + if let Some(fence) = match_fence(line) { + match &fence_marker { + None => fence_marker = Some(fence), + Some(marker) if line.trim_start().starts_with(marker.as_str()) => { + fence_marker = None; + } + _ => {} + } + offset += line.len() + 1; + continue; + } + + if fence_marker.is_none() { + if let Some((level, title)) = parse_heading_line(line, 4) { + headings.push(DocsHeading { level, title, start_index: offset }); + } + } + offset += line.len() + 1; + } + headings +} + +fn section_end_index(content: &str, headings: &[DocsHeading], index: usize) -> usize { + let level = headings[index].level; + // A section ends at the next heading of the same or higher (shallower) level. + for h in &headings[index + 1..] { + if h.level <= level { + return h.start_index; + } + } + content.len() +} + +/// Builds a human-readable outline of a page's headings, with an approximate +/// size for each section. Used when a page is too large to return whole. +pub fn build_docs_outline(content: &str) -> String { + let headings = parse_docs_headings(content); + if headings.is_empty() { + return "(no markdown headings found on this page)".to_string(); + } + headings + .iter() + .enumerate() + .map(|(i, h)| { + let end = section_end_index(content, &headings, i); + let approx = end.saturating_sub(h.start_index); + let indent = " ".repeat(h.level.saturating_sub(1)); + format!("{}- {} (~{} chars)", indent, h.title, approx) + }) + .collect::>() + .join("\n") +} + +/// Normalizes a heading title for tolerant, case/punctuation-insensitive matching. +fn normalize_heading_title(title: &str) -> String { + // toLowerCase, replace /[^a-z0-9]+/g with ' ', trim + let mut out = String::new(); + let mut prev_space = false; + for ch in title.to_lowercase().chars() { + if ch.is_ascii_alphanumeric() { + out.push(ch); + prev_space = false; + } else if !prev_space { + out.push(' '); + prev_space = true; + } + } + out.trim().to_string() +} + +/// Extracts the content of the section whose heading matches `section` (from the +/// heading up to the next heading of the same/higher level). Case-insensitive and +/// tolerant of minor punctuation differences. `None` when no heading matches. +pub fn extract_docs_section(content: &str, section: &str) -> Option { + let headings = parse_docs_headings(content); + let target = normalize_heading_title(section); + if target.is_empty() { + return None; + } + let match_index = headings + .iter() + .position(|h| normalize_heading_title(&h.title) == target) + .or_else(|| { + // Fall back to a contains match so "Result streaming" matches "Result". + headings + .iter() + .position(|h| normalize_heading_title(&h.title).contains(&target)) + })?; + + let start = headings[match_index].start_index; + let end = section_end_index(content, &headings, match_index); + Some(content[start..end].trim().to_string()) +} + +/// Decides what to return for read_docs_page: a requested section, the full page, +/// or an outline asking the model to pick a section. +pub fn render_docs_page_result(content: &str, section: Option<&str>) -> String { + if let Some(section) = section { + if let Some(extracted) = extract_docs_section(content, section) { + return extracted; + } + return format!( + "No section matching \"{}\" was found on this page. Available sections:\n\n{}", + section, + build_docs_outline(content) + ); + } + + if content.len() <= FULL_PAGE_CHAR_LIMIT { + return content.to_string(); + } + + format!( + "This documentation page is large. Below is its list of sections with approximate sizes.\n\ + Call the docs page-reading tool again with the same `url` argument and a `section` set to one of these headings to read that section.\n\n{}", + build_docs_outline(content) + ) +} + +// --------------------------------------------------------------------------- +// Ranking +// --------------------------------------------------------------------------- + +/// Splits a query into distinct, lowercased, non-empty terms (insertion order), +/// capped at `MAX_QUERY_TERMS`. +fn tokenize_query(query: &str) -> Vec { + let mut seen = HashSet::new(); + let mut out = Vec::new(); + for term in query.to_lowercase().split_whitespace() { + if seen.insert(term.to_string()) { + out.push(term.to_string()); + if out.len() >= MAX_QUERY_TERMS { + break; + } + } + } + out +} + +fn count_occurrences(haystack: &str, needle: &str) -> usize { + if needle.is_empty() { + return 0; + } + haystack.matches(needle).count() +} + +struct Scored { + res: DocsSearchResult, + distinct_terms: usize, + order: usize, +} + +/// Prefer pages that cover every query term; sort by score desc then input order; +/// take the top `max_pages`. +fn finalize_pool(mut scored: Vec, term_count: usize, max_pages: usize) -> Vec { + let has_full = scored.iter().any(|s| s.distinct_terms == term_count); + if has_full { + scored.retain(|s| s.distinct_terms == term_count); + } + scored.sort_by(|a, b| b.res.score.cmp(&a.res.score).then(a.order.cmp(&b.order))); + scored.into_iter().take(max_pages).map(|s| s.res).collect() +} + +/// Ranks docs pages for a keyword query. Score is `distinctTermsMatched` +/// (dominant) then total occurrences. Each result carries up to +/// `SEARCH_MAX_SNIPPETS_PER_PAGE` of its most term-dense lines. +pub fn search_docs_pages(pages: &[DocsFullPage], query: &str, max_pages: usize) -> Vec { + let terms = tokenize_query(query); + if terms.is_empty() { + return Vec::new(); + } + + let mut scored = Vec::new(); + for (order, page) in pages.iter().enumerate() { + let mut distinct = 0usize; + let mut occurrences = 0usize; + for term in &terms { + let count = count_occurrences(&page.body_lower, term); + if count > 0 { + distinct += 1; + occurrences += count; + } + } + if distinct == 0 { + continue; + } + scored.push(Scored { + res: DocsSearchResult { + url: page.url.clone(), + title: page.title.clone(), + // distinctTerms dominates so a page matching all terms always + // outranks one matching fewer, regardless of raw occurrences. + score: distinct as i64 * 1_000_000 + occurrences as i64, + snippets: select_snippets( + &page.body, + &terms, + SEARCH_MAX_SNIPPETS_PER_PAGE, + SEARCH_MAX_SNIPPET_CHARS, + ), + }, + distinct_terms: distinct, + order, + }); + } + finalize_pool(scored, terms.len(), max_pages) +} + +/// Ranks index entries by matching query terms against each entry's title and +/// description (title matches weigh more). The description becomes the result's +/// single snippet. Recovers "named feature" discovery that full-text grep misses. +pub fn search_docs_index(entries: &[DocsIndexEntry], query: &str, max_pages: usize) -> Vec { + let terms = tokenize_query(query); + if terms.is_empty() { + return Vec::new(); + } + + let mut scored = Vec::new(); + for (order, entry) in entries.iter().enumerate() { + let mut distinct = 0usize; + let mut score = 0i64; + for term in &terms { + let in_title = entry.title_lower.contains(term.as_str()); + let in_desc = entry.description_lower.contains(term.as_str()); + if in_title || in_desc { + distinct += 1; + score += if in_title { 5 } else { 0 } + if in_desc { 1 } else { 0 }; + } + } + if distinct == 0 { + continue; + } + scored.push(Scored { + res: DocsSearchResult { + url: entry.url.clone(), + title: entry.title.clone(), + score: distinct as i64 * 1_000_000 + score, + snippets: if entry.description.is_empty() { + Vec::new() + } else { + vec![entry.description.clone()] + }, + }, + distinct_terms: distinct, + order, + }); + } + finalize_pool(scored, terms.len(), max_pages) +} + +/// Picks the most term-dense lines of a page body as snippets, in document order, +/// deduped, each trimmed to `max_chars` around the first matched term. +fn select_snippets(body: &str, terms: &[String], max_snippets: usize, max_chars: usize) -> Vec { + struct LineHit { + text: String, + distinct: usize, + order: usize, + } + let mut hits = Vec::new(); + for (order, line) in body.split('\n').enumerate() { + let lower = line.to_lowercase(); + let distinct = terms.iter().filter(|t| lower.contains(t.as_str())).count(); + if distinct == 0 { + continue; + } + let text = make_snippet(line, terms, max_chars); + if !text.is_empty() { + hits.push(LineHit { text, distinct, order }); + } + } + hits.sort_by(|a, b| b.distinct.cmp(&a.distinct).then(a.order.cmp(&b.order))); + + let mut seen = HashSet::new(); + let mut result = Vec::new(); + for hit in hits { + if !seen.insert(hit.text.clone()) { + continue; + } + result.push(hit.text); + if result.len() >= max_snippets { + break; + } + } + result +} + +/// Collapses a matched line to a single-line snippet of at most `max_chars`, +/// windowed around the first matched term (with ellipses) when the line is long. +/// Operates on `char`s so multibyte content can't split mid-codepoint. +fn make_snippet(line: &str, terms: &[String], max_chars: usize) -> String { + let collapsed = line.split_whitespace().collect::>().join(" "); + let chars: Vec = collapsed.chars().collect(); + if chars.len() <= max_chars { + return collapsed; + } + + let lower = collapsed.to_lowercase(); + let mut first_index: Option = None; + for term in terms { + if let Some(byte_idx) = lower.find(term.as_str()) { + let char_idx = lower[..byte_idx].chars().count(); + first_index = Some(first_index.map_or(char_idx, |f| f.min(char_idx))); + } + } + + let total = chars.len(); + match first_index { + None => { + let slice: String = chars[..max_chars].iter().collect(); + format!("{}…", slice.trim_end()) + } + Some(fi) => { + let start = fi.saturating_sub(max_chars / 3); + let end = (start + max_chars).min(total); + let prefix = if start > 0 { "…" } else { "" }; + let suffix = if end < total { "…" } else { "" }; + let slice: String = chars[start..end].iter().collect(); + format!("{}{}{}", prefix, slice.trim(), suffix) + } + } +} + +/// Strips the `.md` suffix and trailing slash so index/body URLs dedupe. +pub fn canonical_search_url(url: &str) -> String { + let stripped = strip_md_suffix(url); + stripped.strip_suffix('/').unwrap_or(stripped).to_string() +} + +fn strip_md_suffix(s: &str) -> &str { + if s.len() >= 3 && s[s.len() - 3..].eq_ignore_ascii_case(".md") { + &s[..s.len() - 3] + } else { + s + } +} + +/// Merges full-text (body) results with index-description results. Body matches +/// come first; index-only matches fill remaining slots — so a named feature +/// surfaced only by its index entry still appears even when body grep missed it. +pub fn merge_docs_search_results( + body_results: Vec, + index_results: Vec, + max_pages: usize, +) -> Vec { + let mut seen: HashSet = + body_results.iter().map(|r| canonical_search_url(&r.url)).collect(); + let mut merged = body_results; + for entry in index_results { + let key = canonical_search_url(&entry.url); + if seen.insert(key) { + merged.push(entry); + } + } + merged.truncate(max_pages); + merged +} + +/// Renders search results as the string returned to the model. +pub fn format_docs_search_results(query: &str, results: &[DocsSearchResult]) -> String { + if results.is_empty() { + return format!( + "No documentation pages matched \"{}\". Try fewer or more general keywords (a single distinctive term often works best).", + query + ); + } + + let blocks = results + .iter() + .map(|r| { + let mut lines = vec![format!("## {}", r.title), format!("Source: {}", r.url)]; + for snippet in &r.snippets { + lines.push(format!(" - {}", snippet)); + } + lines.join("\n") + }) + .collect::>() + .join("\n\n"); + + format!( + "Found {} documentation page(s) matching \"{}\", most relevant first:\n\n{}\n\n\ + Cite the exact \"Source\" URL when referencing a page. If these snippets are not enough, call the docs page-reading tool with a Source URL as its `url` argument to read the full page or a section.", + results.len(), + query, + blocks + ) +} + +// --------------------------------------------------------------------------- +// URL normalization & link sanitization +// --------------------------------------------------------------------------- + +/// Strips docusaurus numeric ordering prefixes (`13_`, `8-`) from each path +/// segment so it matches the published route. +fn strip_docs_path_prefixes(path: &str) -> String { + path.split('/') + .map(|seg| ORDERING_PREFIX_RE.replace(seg, "").into_owned()) + .collect::>() + .join("/") +} + +/// Normalizes a user/model-supplied docs reference to a fully-qualified `.md` URL +/// on the docs origin. Accepts a full URL, `/docs/...`, or `docs/...`. +pub fn normalize_docs_url(input: &str) -> String { + let mut value = input.trim().to_string(); + + if is_http_url(&value) { + // Strip the origin so we re-anchor to DOCS_ORIGIN and normalize the path. + if let Ok(parsed) = url::Url::parse(&value) { + value = parsed.path().to_string(); + } + } + + // Drop any query string or hash fragment. + value = value.split('#').next().unwrap().split('?').next().unwrap().to_string(); + + if !value.starts_with('/') { + value = format!("/{}", value); + } + // Strip a trailing slash (but keep the leading one). + if value.len() > 1 && value.ends_with('/') { + value.pop(); + } + + value = strip_docs_path_prefixes(&value); + if let Some(stripped) = value.strip_suffix(".mdx") { + value = format!("{}.md", stripped); + } + if !value.ends_with(".md") { + value = format!("{}.md", value); + } + + format!("{}{}", DOCS_ORIGIN, value) +} + +/// The canonical published URL a model should cite for a docs page (the `.md` +/// fetch URL without the suffix). +pub fn canonical_docs_page_url(path: &str) -> String { + strip_md_suffix(&normalize_docs_url(path)).to_string() +} + +fn is_http_url(s: &str) -> bool { + let lower = s.to_ascii_lowercase(); + lower.starts_with("http://") || lower.starts_with("https://") +} + +/// Rewrites relative/source-file doc links inside raw page markdown to canonical +/// published URLs, so the model never echoes a docusaurus source path into its +/// answer as a broken link. Non-doc links (external, images, anchors) and `../` +/// cross-directory links are left untouched. +pub fn sanitize_docs_markdown_links(content: &str, page_url: &str) -> String { + let base = url::Url::parse(page_url).ok(); + MD_LINK_RE + .replace_all(content, |caps: ®ex::Captures| { + let whole = caps.get(0).unwrap().as_str(); + let target = &caps[1]; + let title = caps.get(2).map(|m| m.as_str()).unwrap_or(""); + + // Only rewrite links to docusaurus source files (.md/.mdx); leave + // images, external URLs and bare anchors untouched. + if !is_md_target(target) { + return whole.to_string(); + } + // `../` cross-directory links are authored against the docusaurus + // source tree, whose depth differs from the published URL, so strict + // resolution is unreliable. Leave them for the canonical-URL header. + if has_parent_traversal(target) { + return whole.to_string(); + } + let Some(base) = &base else { + return whole.to_string(); + }; + let Ok(resolved) = base.join(target) else { + return whole.to_string(); + }; + if resolved.scheme() != "https" + || resolved.host_str() != Some("www.windmill.dev") + || !resolved.path().starts_with("/docs/") + { + return whole.to_string(); + } + let pathname = strip_docs_path_prefixes(resolved.path()); + let pathname = strip_md_or_mdx(&pathname); + let hash = resolved.fragment().map(|f| format!("#{}", f)).unwrap_or_default(); + format!("]({}{}{}{})", DOCS_ORIGIN, pathname, hash, title) + }) + .into_owned() +} + +/// `\.mdx?($|[#?])` — the target points at a markdown source file. +fn is_md_target(target: &str) -> bool { + for marker in [".md", ".mdx"] { + if let Some(idx) = target.to_ascii_lowercase().find(marker) { + let after = &target[idx + marker.len()..]; + // `.md` must not be a prefix of `.mdx` here: only accept when the + // extension is followed by end / `#` / `?`. + if after.is_empty() || after.starts_with('#') || after.starts_with('?') { + return true; + } + } + } + false +} + +/// `(^|/)\.\./` — the target contains a parent-directory traversal segment. +fn has_parent_traversal(target: &str) -> bool { + target == ".." || target.starts_with("../") || target.contains("/../") +} + +fn strip_md_or_mdx(path: &str) -> &str { + if let Some(stripped) = path.strip_suffix(".mdx") { + stripped + } else { + strip_md_suffix(path) + } +} + +#[cfg(test)] +mod tests; diff --git a/backend/windmill-api/src/docs/search/tests.rs b/backend/windmill-api/src/docs/search/tests.rs new file mode 100644 index 0000000000..84c422aa73 --- /dev/null +++ b/backend/windmill-api/src/docs/search/tests.rs @@ -0,0 +1,267 @@ +//! Parity tests ported from the frontend `copilot/chat/docs/core.test.ts`. + +use super::*; + +const SAMPLE: &str = "# Jobs\n\nIntro text about jobs.\n\n## Job kinds\n\nSome kinds.\n\n## Result\n\n### Result of jobs that failed\n\n```\n{ \"error\": \"boom\" }\n```\n\n### Result streaming\n\n#### Returning a stream directly\n\n```python\n# Returning a stream directly is a comment heading that must be ignored\ndef main():\n pass\n```\n\n## Retention policy\n\nFinal section.\n"; + +// Mirrors the llms-full.txt layout: a corpus preamble, then per-page blocks each +// introduced by a `---` + `## ` lead-in followed by a `Source:` line. +const SAMPLE_FULL: &str = "# Windmill\n\n> Preamble blurb that precedes the first Source line and must be ignored.\n\n## Browser automation\n\nSource: https://www.windmill.dev/docs/advanced/browser_automation\n\n# Browser automation\n\nBy default, a worker group named `reports` handles jobs with the `chromium` tag.\nThe chromium binary will be available on these workers at /usr/bin/chromium.\nYou can disable the sandbox by passing the --no-sandbox flag.\n\n---\n\n## Worker groups\n\nSource: https://www.windmill.dev/docs/core_concepts/worker_groups\n\n# Worker groups\n\nWorker groups let you assign tags to workers.\nSet the chromium tag on a worker so it can run browser jobs.\n\n---\n\n## Scheduling\n\nSource: https://www.windmill.dev/docs/core_concepts/scheduling\n\n# Scheduling\n\nUse cron expressions to schedule scripts and flows.\n"; + +#[test] +fn parses_headings_and_ignores_fenced_blocks() { + let titles: Vec = parse_docs_headings(SAMPLE) + .iter() + .map(|h| format!("{}:{}", h.level, h.title)) + .collect(); + assert_eq!( + titles, + vec![ + "1:Jobs", + "2:Job kinds", + "2:Result", + "3:Result of jobs that failed", + "3:Result streaming", + "4:Returning a stream directly", + "2:Retention policy", + ] + ); +} + +#[test] +fn heading_start_index_points_at_the_heading_line() { + for h in parse_docs_headings(SAMPLE) { + let at = &SAMPLE[h.start_index..]; + assert!(at.starts_with(&"#".repeat(h.level))); + assert!(at[h.level..].trim_start().starts_with(&h.title)); + } +} + +#[test] +fn handles_tilde_fences() { + let content = "# Title\n\n~~~\n# not a heading\n~~~\n\n## Real\n"; + let titles: Vec = parse_docs_headings(content).iter().map(|h| h.title.clone()).collect(); + assert_eq!(titles, vec!["Title", "Real"]); +} + +#[test] +fn extracts_section_up_to_next_same_or_higher_heading() { + let section = extract_docs_section(SAMPLE, "Result").unwrap(); + assert!(section.contains("## Result")); + assert!(section.contains("### Result of jobs that failed")); + assert!(section.contains("### Result streaming")); + assert!(!section.contains("## Retention policy")); +} + +#[test] +fn extract_section_is_case_and_punctuation_tolerant() { + let section = extract_docs_section(SAMPLE, "retention-policy!").unwrap(); + assert!(section.contains("## Retention policy")); + assert!(section.contains("Final section.")); +} + +#[test] +fn extract_section_returns_none_when_missing() { + assert!(extract_docs_section(SAMPLE, "Nonexistent section").is_none()); +} + +#[test] +fn build_outline_lists_headings_with_indent() { + let outline = build_docs_outline(SAMPLE); + assert!(outline.contains("- Jobs (~")); + assert!(outline.contains(" - Job kinds (~")); + assert!(outline.contains(" - Result of jobs that failed (~")); +} + +#[test] +fn build_outline_handles_no_headings() { + assert_eq!( + build_docs_outline("just some text\nwith no headings"), + "(no markdown headings found on this page)" + ); +} + +#[test] +fn render_page_returns_whole_small_page() { + assert_eq!(render_docs_page_result(SAMPLE, None), SAMPLE); +} + +#[test] +fn render_page_returns_outline_for_large_page() { + let large = format!("# Big\n\n{}\n\n## Tail\n\nmore", "x".repeat(25_000)); + let result = render_docs_page_result(&large, None); + assert!(result.contains("This documentation page is large")); + assert!(result.contains("same `url` argument")); + assert!(!result.contains("same path")); + assert!(!result.contains("read_docs_page")); + assert!(result.contains("- Big (~")); + assert!(result.contains("- Tail (~")); +} + +#[test] +fn render_page_returns_requested_section() { + let result = render_docs_page_result(SAMPLE, Some("Job kinds")); + assert!(result.contains("## Job kinds")); + assert!(result.contains("Some kinds.")); +} + +#[test] +fn render_page_missing_section_returns_outline_note() { + let result = render_docs_page_result(SAMPLE, Some("Does not exist")); + assert!(result.contains("No section matching \"Does not exist\" was found")); + assert!(result.contains("- Jobs (~")); +} + +#[test] +fn normalize_docs_url_cases() { + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("docs/core_concepts/jobs"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!( + normalize_docs_url("https://www.windmill.dev/docs/core_concepts/jobs#result?foo=bar"), + "https://www.windmill.dev/docs/core_concepts/jobs.md" + ); + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs.md"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs/"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("/docs/flows/13_flow_branches"), "https://www.windmill.dev/docs/flows/flow_branches.md"); + assert_eq!(normalize_docs_url("/docs/flows/13_flow_branches.mdx"), "https://www.windmill.dev/docs/flows/flow_branches.md"); +} + +#[test] +fn canonical_docs_page_url_cases() { + assert_eq!(canonical_docs_page_url("/docs/flows/flow_editor"), "https://www.windmill.dev/docs/flows/flow_editor"); + assert_eq!(canonical_docs_page_url("/docs/flows/14_retries.md"), "https://www.windmill.dev/docs/flows/retries"); +} + +#[test] +fn sanitize_markdown_links_cases() { + let page = "https://www.windmill.dev/docs/flows/flow_editor.md"; + assert_eq!( + sanitize_docs_markdown_links("See [retries](./14_retries.mdx) for more.", page), + "See [retries](https://www.windmill.dev/docs/flows/retries) for more." + ); + assert_eq!( + sanitize_docs_markdown_links("[handling](./8_error_handling.mdx)", page), + "[handling](https://www.windmill.dev/docs/flows/error_handling)" + ); + assert_eq!( + sanitize_docs_markdown_links("[branch all](./13_flow_branches.mdx#branch-all)", page), + "[branch all](https://www.windmill.dev/docs/flows/flow_branches#branch-all)" + ); + // images & external links untouched + let external = "![diagram](./assets/flow_example.png) and [site](https://example.com/page.md)"; + assert_eq!(sanitize_docs_markdown_links(external, page), external); + // bare anchor untouched + assert_eq!(sanitize_docs_markdown_links("[top](#introduction)", page), "[top](#introduction)"); + // ../ cross-directory links untouched + let parent = "[handling](../core_concepts/8_error_handling.mdx)"; + assert_eq!(sanitize_docs_markdown_links(parent, page), parent); + let parent2 = "[retries](../../flows/14_retries.md)"; + assert_eq!(sanitize_docs_markdown_links(parent2, page), parent2); +} + +#[test] +fn parse_full_text_splits_pages_and_drops_preamble() { + let pages = parse_docs_full_text(SAMPLE_FULL); + assert_eq!( + pages.iter().map(|p| p.url.clone()).collect::>(), + vec![ + "https://www.windmill.dev/docs/advanced/browser_automation", + "https://www.windmill.dev/docs/core_concepts/worker_groups", + "https://www.windmill.dev/docs/core_concepts/scheduling", + ] + ); + assert_eq!( + pages.iter().map(|p| p.title.clone()).collect::>(), + vec!["Browser automation", "Worker groups", "Scheduling"] + ); + // The next page's "## Worker groups" lead-in must not leak into this body. + let browser = pages.iter().find(|p| p.url.ends_with("/browser_automation")).unwrap(); + assert!(!browser.body.contains("Worker groups")); + assert!(!browser.body.contains("---")); +} + +#[test] +fn search_pages_ranks_more_occurrences_first() { + let pages = parse_docs_full_text(SAMPLE_FULL); + let results = search_docs_pages(&pages, "chromium", 5); + assert_eq!( + results.iter().map(|r| r.url.clone()).collect::>(), + vec![ + "https://www.windmill.dev/docs/advanced/browser_automation", + "https://www.windmill.dev/docs/core_concepts/worker_groups", + ] + ); + assert!(!results[0].snippets.is_empty()); + assert!(results[0].snippets.join("\n").contains("chromium")); +} + +#[test] +fn search_prefers_pages_covering_all_terms() { + let pages = parse_docs_full_text(SAMPLE_FULL); + // Only browser_automation contains "sandbox"; worker_groups has "chromium" but + // not "sandbox". A full-coverage page exists, so partial matches are dropped. + let results = search_docs_pages(&pages, "chromium sandbox", 5); + assert_eq!( + results.iter().map(|r| r.url.clone()).collect::>(), + vec!["https://www.windmill.dev/docs/advanced/browser_automation"] + ); +} + +#[test] +fn merge_dedupes_index_against_body_by_canonical_url() { + let body = vec![DocsSearchResult { + url: "https://www.windmill.dev/docs/a".to_string(), + title: "A".to_string(), + score: 10, + snippets: vec![], + }]; + let index = vec![ + DocsSearchResult { + url: "https://www.windmill.dev/docs/a.md".to_string(), + title: "A".to_string(), + score: 5, + snippets: vec![], + }, + DocsSearchResult { + url: "https://www.windmill.dev/docs/b.md".to_string(), + title: "B".to_string(), + score: 5, + snippets: vec![], + }, + ]; + let merged = merge_docs_search_results(body, index, SEARCH_MAX_PAGES); + assert_eq!( + merged.iter().map(|r| r.url.clone()).collect::>(), + vec!["https://www.windmill.dev/docs/a", "https://www.windmill.dev/docs/b.md"] + ); +} + +#[test] +fn empty_query_returns_no_results() { + let pages = parse_docs_full_text(SAMPLE_FULL); + assert!(search_docs_pages(&pages, " ", 5).is_empty()); +} + +#[test] +fn format_search_results_no_matches() { + let out = format_docs_search_results("zzz", &[]); + assert!(out.contains("No documentation pages matched \"zzz\"")); +} + +#[test] +fn format_search_results_uses_caller_neutral_followup_guidance() { + let out = format_docs_search_results( + "jobs", + &[DocsSearchResult { + url: "https://www.windmill.dev/docs/core_concepts/jobs".to_string(), + title: "Jobs".to_string(), + score: 1, + snippets: vec!["Jobs run scripts and flows.".to_string()], + }], + ); + + assert!(out.contains("Source: https://www.windmill.dev/docs/core_concepts/jobs")); + assert!(out.contains("Source URL as its `url` argument")); + assert!(!out.contains("read_docs_page")); + assert!(!out.contains("readDocsPage")); +} diff --git a/backend/windmill-api/src/health.rs b/backend/windmill-api/src/health.rs index fab9979c12..4dc8947de6 100644 --- a/backend/windmill-api/src/health.rs +++ b/backend/windmill-api/src/health.rs @@ -219,12 +219,16 @@ struct DatabaseCheckResult { async fn check_database_with_latency(db: &DB) -> DatabaseCheckResult { let start = std::time::Instant::now(); + // `pg_is_in_recovery()` is true on standbys/read-only replicas, so a primary + // returns true here. A read-only replica (e.g. after a failover where the + // primary became a secondary) reports unhealthy, letting liveness probes + // restart the pod instead of silently failing all writes. let healthy = tokio::time::timeout( HEALTH_CHECK_TIMEOUT, - sqlx::query_scalar!("SELECT 1").fetch_one(db), + sqlx::query_scalar!("SELECT NOT pg_is_in_recovery()").fetch_one(db), ) .await - .map(|r| r.is_ok()) + .map(|r| matches!(r, Ok(Some(true)))) .unwrap_or(false); let latency_ms = start.elapsed().as_millis() as i64; diff --git a/backend/windmill-api/src/inkeep_oss.rs b/backend/windmill-api/src/inkeep_oss.rs deleted file mode 100644 index a34ea91435..0000000000 --- a/backend/windmill-api/src/inkeep_oss.rs +++ /dev/null @@ -1,23 +0,0 @@ -#[cfg(feature = "private")] -#[allow(unused)] -pub use crate::inkeep_ee::*; - -#[cfg(not(feature = "private"))] -use axum::{routing::post, Router}; - -#[cfg(not(feature = "private"))] -use windmill_common::error::Error; - -#[cfg(not(feature = "private"))] -pub fn global_service() -> Router { - Router::new().route("/", post(inkeep_not_available)) -} - -#[cfg(not(feature = "private"))] -async fn inkeep_not_available() -> windmill_common::error::Result<()> { - Err(Error::Generic( - http::StatusCode::FORBIDDEN, - "Inkeep AI documentation assistant is only available in Windmill Enterprise Edition" - .to_string(), - )) -} diff --git a/backend/windmill-api/src/jobs.rs b/backend/windmill-api/src/jobs.rs index c6d978e0d6..5192d27724 100644 --- a/backend/windmill-api/src/jobs.rs +++ b/backend/windmill-api/src/jobs.rs @@ -390,6 +390,10 @@ pub fn workspace_unauthed_service() -> Router { .route("/get/{id}", get(get_job)) .route("/get_logs/{id}", get(get_job_logs)) .route("/get_flow_all_logs/{id}", get(get_flow_all_logs)) + .route( + "/get_flow_all_logs_structured/{id}", + get(get_flow_all_logs_structured), + ) .route( "/get_completed_logs_tail/{id}", get(get_completed_job_logs_tail), @@ -513,6 +517,26 @@ async fn cancel_job_api( Path((w_id, id)): Path<(String, Uuid)>, Json(CancelJob { reason }): Json, ) -> error::Result { + // App embed tokens (the sandboxed app iframe) may cancel ONLY jobs they launched + // — their app's component runs, stamped created_by == viewer. cancel_job_api has + // no other per-job ownership check, so without this an embed token (which carries + // the viewer's identity) could cancel any job by id. NotFound (not 403) so the + // untrusted app can't probe job existence. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + let created_by = sqlx::query_scalar!( + "SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + if created_by.as_deref() != Some(authed.username.as_str()) { + return Err(Error::NotFound(format!("Job {id} not found"))); + } + } + } + let tx = db.begin().await?; let audit_author: AuditAuthor = match opt_authed.as_ref() { @@ -1007,6 +1031,17 @@ async fn require_job_read_access( return Ok(()); } + // App embed tokens (the sandboxed app iframe) carry the viewer's identity so the + // app can read its own component runs — which are stamped `created_by == viewer` + // and so already returned above. They must NOT inherit the viewer's *broader* + // job access (share links, folder ACLs, admin RLS): user-authored app JS holds + // this token, and letting it reach any job merely visible to the viewer would + // expose unrelated runs' results/logs. Stop at the launched-by-viewer grant. + // NotFound (not PermissionDenied) so the untrusted app can't probe job existence. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotFound(format!("Job {job_id} not found"))); + } + // `username_override` is derived from the token *label* (`username_override_from_label`), // which is fully user-controlled with no uniqueness/ownership check (webhook-/http-/ // email-/ws- trigger tokens, `ephemeral-script-end-user-*`, and the generic `label-*` @@ -1388,6 +1423,7 @@ macro_rules! get_job_query { @impl "v2_job_completed", ($($opts)*), "v2_job_completed.duration_ms, v2_job_completed.completed_at, CASE WHEN status = 'success' OR status = 'skipped' THEN true ELSE false END as success, result_columns, deleted, status = 'skipped' as is_skipped, \ v2_job.labels, \ + EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry, \ CASE WHEN result is null or pg_column_size(result) < 90000 THEN result ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as result", "", ) @@ -1397,7 +1433,8 @@ macro_rules! get_job_query { @impl "v2_job_queue", ($($opts)*), "scheduled_for, running, ping as last_ping, suspend, suspend_until, same_worker, pre_run_error, visible_to_owner, \ flow_innermost_root_job AS root_job, flow_leaf_jobs AS leaf_jobs, concurrent_limit, concurrency_time_window_s, timeout, flow_step_id, cache_ttl, cache_ignore_s3_path, runnable_settings_handle, \ - script_entrypoint_override, v2_job.labels", + script_entrypoint_override, v2_job.labels, \ + EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", "LEFT JOIN v2_job_runtime ON v2_job_runtime.id = v2_job_queue.id LEFT JOIN v2_job_status ON v2_job_status.id = v2_job_queue.id", ) }; @@ -2177,14 +2214,40 @@ async fn resolve_logs_to_string( logs.to_string() } -async fn get_flow_all_logs( - OptViewToken(view_token): OptViewToken, - OptAuthed(opt_authed): OptAuthed, +/// A single job in a flow's execution tree, with its resolved logs and a +/// human-readable label describing its position (iteration, branch, subflow…). +#[derive(Serialize)] +struct FlowLogEntry { + job_id: String, + /// Human-readable label, e.g. "Step a (iteration 2/3)" or "Flow". + label: String, + /// Job kind (script, flow, forloopflow, …). + kind: String, + /// The flow step id this job corresponds to, if any. + flow_step_id: Option, + /// Materialized step path (e.g. "a/b") used to locate the step in the flow. + step_path: Option, + /// Depth in the flow tree (0 for the root flow job). + depth: i32, + /// The parent module type (forloopflow, branchall, …), if any. + parent_module_type: Option, + /// 1-based index of this job among its siblings sharing the same step. + sibling_index: i32, + /// Total number of siblings sharing the same step. + sibling_count: i32, + /// Resolved logs for this job (pulled from disk/object store as needed). + logs: String, +} + +async fn collect_flow_log_entries( + view_token: Option, + opt_authed: Option, opt_tokened: OptTokened, - Extension(db): Extension, - Extension(user_db): Extension, - Path((w_id, id)): Path<(String, Uuid)>, -) -> error::Result { + db: &DB, + user_db: &UserDB, + w_id: &str, + id: Uuid, +) -> error::Result> { let tags = opt_authed .as_ref() .map(|authed| get_scope_tags(authed).map(|v| v.iter().map(|s| s.to_string()).collect_vec())) @@ -2197,17 +2260,17 @@ async fn get_flow_all_logs( w_id, tags.as_ref().map(|v| v.as_slice()) ) - .fetch_optional(&db) + .fetch_optional(db) .await?; let root_job = not_found_if_none(root_job, "Job", id.to_string())?; if let Some(authed) = opt_authed.as_ref() { require_job_read_access( - &db, - &user_db, + db, + user_db, authed, - &w_id, + w_id, &id, &root_job.created_by, view_token.as_deref(), @@ -2220,10 +2283,10 @@ async fn get_flow_all_logs( } log_job_view( - &db, + db, opt_authed.as_ref(), opt_tokened.token.as_deref(), - &w_id, + w_id, &id, ) .await?; @@ -2290,10 +2353,10 @@ async fn get_flow_all_logs( w_id, id, ) - .fetch_all(&db) + .fetch_all(db) .await?; - let mut all_logs = String::new(); + let mut entries = Vec::with_capacity(records.len()); for record in &records { let kind = record.kind.as_deref().unwrap_or(""); @@ -2356,18 +2419,89 @@ async fn get_flow_all_logs( }; let job_id = record.id.map(|u| u.to_string()).unwrap_or_default(); - all_logs.push_str(&format!("\n=== {} (Job: {}) ===\n", label, job_id)); let logs = record.logs.as_deref().unwrap_or(""); let resolved = resolve_logs_to_string(record.log_offset.unwrap_or(0), logs, &record.log_file_index) .await; - all_logs.push_str(&resolved); + + entries.push(FlowLogEntry { + job_id, + label, + kind: kind.to_string(), + flow_step_id: record.flow_step_id.clone(), + step_path: record.path_label.clone(), + depth, + parent_module_type: if parent_module_type.is_empty() { + None + } else { + Some(parent_module_type.to_string()) + }, + sibling_index, + sibling_count, + logs: resolved, + }); + } + + Ok(entries) +} + +async fn get_flow_all_logs( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + opt_tokened: OptTokened, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> error::Result { + let entries = collect_flow_log_entries( + view_token, + opt_authed, + opt_tokened, + &db, + &user_db, + &w_id, + id, + ) + .await?; + + let mut all_logs = String::new(); + for entry in &entries { + all_logs.push_str(&format!( + "\n=== {} (Job: {}) ===\n", + entry.label, entry.job_id + )); + all_logs.push_str(&entry.logs); all_logs.push('\n'); } Ok(content_plain(Body::from(all_logs))) } +/// Structured alternative to `get_flow_all_logs`: returns the same flow log +/// tree as a JSON array of entries (one per job) instead of a flat text blob, +/// so callers can render or process logs per-step without parsing delimiters. +async fn get_flow_all_logs_structured( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + opt_tokened: OptTokened, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> JsonResult> { + let entries = collect_flow_log_entries( + view_token, + opt_authed, + opt_tokened, + &db, + &user_db, + &w_id, + id, + ) + .await?; + + Ok(Json(entries)) +} + async fn get_args( OptViewToken(view_token): OptViewToken, OptAuthed(opt_authed): OptAuthed, @@ -4022,11 +4156,17 @@ fn conditionally_require_authed_user( } pub async fn create_job_signature( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>, Query(approver): Query, ) -> error::Result { + // The HMAC is treated as full authority by the resume endpoints, so minting + // it requires run scope on the suspended job's flow — not merely any + // jobs:run scope. No-op for unscoped tokens (incl. the in-flow substep token + // used by wmill.get_resume_urls()). + let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?; + check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?; let key = get_workspace_key(&w_id, &db).await?; create_signature(key, job_id, resume_id, approver.approver) } @@ -4109,11 +4249,17 @@ fn build_resume_url( } pub async fn get_resume_urls( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>, Query(approver): Query, ) -> error::JsonResult { + // These URLs embed a resume signature (full resume capability), so a scoped + // token must hold run scope on the suspended job's flow. No-op for unscoped + // tokens (incl. the in-flow substep token). Trusted internal callers use + // get_resume_urls_internal directly and are unaffected. + let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?; + check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?; get_resume_urls_internal( Extension(db), Path((w_id, job_id, resume_id)), @@ -4180,6 +4326,46 @@ pub async fn get_resume_urls_internal( Ok(Json(res)) } +/// Resolve the runnable path of the flow a (possibly step) job belongs to, used +/// to scope-check resume-signature minting against `jobs:run:flows:`. +/// Returns an empty string when the path can't be resolved (e.g. previews or an +/// unknown job); an empty path only matters for path-restricted tokens, which +/// would not be running such a flow. Never hard-fails, so it can't break resume +/// for unscoped tokens (the in-flow `get_resume_urls()` path). +async fn resume_target_flow_path(db: &DB, w_id: &str, job_id: Uuid) -> error::Result { + let job = sqlx::query!( + r#"SELECT kind::text as "kind!", parent_job, runnable_path + FROM v2_job WHERE id = $1 AND workspace_id = $2"#, + job_id, + w_id + ) + .fetch_optional(db) + .await?; + let Some(job) = job else { + return Ok(String::new()); + }; + // All flow kinds: the job itself is the flow whose path scopes the resume. + if matches!( + job.kind.as_str(), + "flow" | "flowpreview" | "flownode" | "singlestepflow" + ) { + return Ok(job.runnable_path.unwrap_or_default()); + } + // Otherwise it's a step; its parent is the flow. + if let Some(parent) = job.parent_job { + return Ok(sqlx::query_scalar!( + "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2", + parent, + w_id + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or_default()); + } + Ok(job.runnable_path.unwrap_or_default()) +} + /// Get the flow ID for a job. If the job is a flow, returns the job_id. /// If the job is a step in a flow, returns the parent flow ID. async fn get_flow_id_for_job(db: &DB, job_id: Uuid) -> error::Result { @@ -9488,16 +9674,31 @@ mod approval_view_gate_tests { fn anonymous_cannot_view_when_auth_required() { // The regression: an unauthenticated holder of the approval token must see nothing. let c = Some(conds(true, vec![])); - assert!(!can_view(&None, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(!can_view( + &None, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); } #[test] fn anonymous_can_view_when_no_auth_required() { // Unchanged behaviour: token alone is sufficient when auth isn't required. let c = Some(conds(false, vec![])); - assert!(can_view(&None, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &None, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); // No approval conditions at all also allows token-only view. - assert!(can_view(&None, &None, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &None, + &None, + Some("f/team/flow"), + "trigger@example.com" + )); } #[test] @@ -9522,7 +9723,17 @@ mod approval_view_gate_tests { let member = Some(authed("carol", false, vec!["approvers".to_string()])); let outsider = Some(authed("dave", false, vec!["other".to_string()])); // Use a non-owned folder path so ownership doesn't short-circuit the check. - assert!(can_view(&member, &c, Some("f/team/flow"), "trigger@example.com")); - assert!(!can_view(&outsider, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &member, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); + assert!(!can_view( + &outsider, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); } } diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index 0f5f9c40ca..cbbb406db4 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -66,6 +66,7 @@ use crate::scim_oss::has_scim_token; use windmill_common::error::AppError; mod ai; +mod ai_skills; mod apps; pub mod args; mod audit; @@ -77,6 +78,7 @@ mod capture; mod concurrency_groups; mod db; mod db_health; +mod docs; mod drafts; #[cfg(feature = "private")] @@ -94,9 +96,6 @@ mod health; #[cfg(feature = "private")] pub mod indexer_ee; mod indexer_oss; -#[cfg(feature = "private")] -mod inkeep_ee; -mod inkeep_oss; mod integration; mod internal_db; mod live_migrations; @@ -545,7 +544,15 @@ pub async fn run_server( Router::new() // Reordered alphabetically .nest("/acls", granular_acls::workspaced_service()) - .nest("/apps", apps::workspaced_service(request_size_limit * 5)) + // CORS so the opaque-origin in-workspace app viewer (WIN-2006, + // sandboxed /apps/get) can read the app definition by path + // (apps/get/p, apps/embed_token/p) with a scoped embed token. + // Bearer-token-only (no cookies), consistent with the other + // workspaced services the iframe calls. + .nest( + "/apps", + apps::workspaced_service(request_size_limit * 5).layer(cors.clone()), + ) .nest("/assets", windmill_api_assets::workspaced_service()) .nest("/audit", audit::workspaced_service()) .nest("/capture", capture::workspaced_service()) @@ -565,7 +572,13 @@ pub async fn run_server( "/flow_conversations", windmill_api_flow_conversations::workspaced_service(), ) - .nest("/folders", folders::workspaced_service()) + // CORS so an opaque-origin app iframe (WIN-2006 embed, + // no separate domain) can read folders/listnames with a + // scoped embed token. Consistent with apps_u/jobs_u cors. + .nest( + "/folders", + folders::workspaced_service().layer(cors.clone()), + ) .nest("/folders_history", folder_history::workspaced_service()) .nest("/groups", groups::workspaced_service()) .nest("/groups_history", group_history::workspaced_service()) @@ -608,20 +621,30 @@ pub async fn run_server( Router::new() }) .nest("/ai", ai::workspaced_service()) + .nest("/ai_skills", ai_skills::workspaced_service()) .nest("/npm_proxy", windmill_api_npm_proxy::workspaced_service()) .nest( "/path_autocomplete", path_autocomplete::workspaced_service(), ) .nest("/raw_apps", raw_apps::workspaced_service()) - .nest("/resources", resources::workspaced_service()) + // CORS so the opaque-origin app iframe can read + // resources/list, resources/type/* with a scoped token. + .nest( + "/resources", + resources::workspaced_service().layer(cors.clone()), + ) .nest("/shared_ui", workspace_shared_ui::workspaced_service()) .nest("/schedules", windmill_api_schedule::workspaced_service()) .nest("/scripts", scripts::workspaced_service()) .nest("/trash", trash::workspaced_service()) .nest( "/users", - users::workspaced_service().layer(Extension(argon2.clone())), + // CORS so the opaque-origin app iframe can read + // users/whoami with a scoped embed token. + users::workspaced_service() + .layer(Extension(argon2.clone())) + .layer(cors.clone()), ) .nest("/variables", variables::workspaced_service()) .nest("/volumes", volumes_oss::workspaced_service()) @@ -662,7 +685,7 @@ pub async fn run_server( .nest("/schedules", windmill_api_schedule::global_service()) .nest("/embeddings", embeddings::global_service()) .nest("/ai", ai::global_service()) - .nest("/inkeep", inkeep_oss::global_service()) + .nest("/docs", docs::global_service()) .nest("/indexer", indexer_oss::management_service()) .nest("/mcp/w/{workspace_id}/list_tools", mcp_list_tools_service) .nest("/db_health", db_health::global_service()) @@ -727,7 +750,11 @@ pub async fn run_server( .nest("/apps_u", { #[cfg(feature = "enterprise")] { - apps_oss::global_unauthed_service() + // CORS so the opaque-origin app viewer (WIN-2006 embed, no + // separate domain) can load a custom-path public app via + // public_app_by_custom_path cross-origin. Consistent with + // the workspaced /w/{workspace_id}/apps_u mount below. + apps_oss::global_unauthed_service().layer(cors.clone()) } #[cfg(not(feature = "enterprise"))] diff --git a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs index 2e9b5babe7..d44484f9db 100644 --- a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs +++ b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs @@ -7,25 +7,53 @@ use windmill_mcp::server::EndpointTool; pub fn all_tools() -> Vec { vec![ EndpointTool { - name: Cow::Borrowed("queryDocumentation"), - description: Cow::Borrowed("query Windmill AI documentation assistant (EE only)"), + name: Cow::Borrowed("searchDocs"), + description: Cow::Borrowed("Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more."), instructions: Cow::Borrowed(""), - path: Cow::Borrowed("/inkeep"), - method: Cow::Borrowed("POST"), + path: Cow::Borrowed("/docs/search"), + method: Cow::Borrowed("GET"), path_params_schema: None, - query_params_schema: None, - body_schema: Some(serde_json::json!({ + query_params_schema: Some(serde_json::json!({ "type": "object", "properties": { "query": { "type": "string", - "description": "The documentation query to send to the AI assistant" + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better." } }, "required": [ "query" ] })), + body_schema: None, + path_field_renames: None, + query_field_renames: None, + body_field_renames: None, + }, + EndpointTool { + name: Cow::Borrowed("readDocsPage"), + description: Cow::Borrowed("Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section."), + instructions: Cow::Borrowed(""), + path: Cow::Borrowed("/docs/page"), + method: Cow::Borrowed("GET"), + path_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + }, + "section": { + "type": "string", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page." + } + }, + "required": [ + "url" + ] +})), + body_schema: None, path_field_renames: None, query_field_renames: None, body_field_renames: None, @@ -84,6 +112,9 @@ pub fn all_tools() -> Vec { "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -169,6 +200,9 @@ pub fn all_tools() -> Vec { "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the variable (body parameter)" @@ -210,6 +244,10 @@ pub fn all_tools() -> Vec { "include_encrypted": { "type": "boolean", "description": "ask to include the encrypted value if secret and decrypt secret is not true (default: false)\n" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -260,6 +298,10 @@ pub fn all_tools() -> Vec { "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft variables whose path has no\ndeployed variable. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge.\n" } }, "required": [] @@ -309,6 +351,9 @@ pub fn all_tools() -> Vec { "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -383,6 +428,9 @@ pub fn all_tools() -> Vec { "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the resource (body parameter)" @@ -414,7 +462,16 @@ pub fn all_tools() -> Vec { "path" ] })), - query_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +})), body_schema: None, path_field_renames: None, query_field_renames: None, @@ -469,6 +526,10 @@ pub fn all_tools() -> Vec { "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft resources whose path has\nno deployed resource. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] @@ -719,6 +780,10 @@ Creates a new version of an existing script when called with the same path and t "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -847,6 +912,10 @@ Creates a new version of an existing script when called with the same path and t "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -1184,6 +1253,14 @@ Creates a new version of an existing script when called with the same path and t "language" ] } + }, + "temp_script_refs": { + "type": "object", + "nullable": true, + "description": "Map of relative-import script path -> temp storage hash so the preview job resolves those imports from not-yet-deployed local content instead of the deployed script", + "additionalProperties": { + "type": "string" + } } }, "required": [ @@ -1456,6 +1533,10 @@ Creates a new version of an existing script when called with the same path and t "type": "boolean", "description": "filter on successful jobs" }, + "status": { + "type": "string", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.. Possible values: success, failure, canceled, skipped" + }, "all_workspaces": { "type": "boolean", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)" @@ -1464,6 +1545,10 @@ Creates a new version of an existing script when called with the same path and t "type": "boolean", "description": "is not a scheduled job" }, + "excludes_entrypoint_override": { + "type": "boolean", + "description": "exclude jobs that were started with a `_ENTRYPOINT_OVERRIDE` arg (e.g. dynamic-select helper runs and preprocessor previews)" + }, "broad_filter": { "type": "string", "description": "broad search across multiple fields (case-insensitive substring match on path, tag, schedule path, trigger kind, label)" @@ -1502,6 +1587,10 @@ Creates a new version of an existing script when called with the same path and t }, "no_code": { "type": "boolean" + }, + "approval_token": { + "type": "string", + "description": "Approval token granting read access to the job when not logged in. The token must be the one issued for this job's flow (i.e. the flow id used when generating the approval URL)." } }, "required": [] @@ -1559,7 +1648,7 @@ You should get the schema of the script or flow before creating the schedule to "properties": { "path": { "type": "string", - "description": "The unique path identifier for this schedule" + "description": "The unique Windmill path for this schedule. Must be of the form `u//` or `f//`." }, "schedule": { "type": "string", @@ -2002,7 +2091,16 @@ You should get the schema of the script or flow before updating the schedule to "path" ] })), - query_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +})), body_schema: None, path_field_renames: None, query_field_renames: None, @@ -2061,6 +2159,10 @@ You should get the schema of the script or flow before updating the schedule to "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft schedules whose path has\nno deployed schedule. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] diff --git a/backend/windmill-api/src/mcp/oauth_server.rs b/backend/windmill-api/src/mcp/oauth_server.rs index 7cfabd2319..c5142c3031 100644 --- a/backend/windmill-api/src/mcp/oauth_server.rs +++ b/backend/windmill-api/src/mcp/oauth_server.rs @@ -18,6 +18,7 @@ use windmill_common::{ }; use crate::db::ApiAuthed; +use windmill_mcp::parse_mcp_scopes; /// Token expiration for MCP OAuth tokens (1 week in seconds) const MCP_OAUTH_TOKEN_EXPIRATION_SECS: u64 = 7 * 24 * 60 * 60; @@ -585,6 +586,8 @@ async fn handle_refresh_token_grant( Some(&new_access_token) }; let new_refresh_token = rd_string(32); + // Re-issues the already-approved (hence already-contained) scopes verbatim; + // containment is enforced once at approval time, so no re-check here. let scopes = token_row.scopes; // Create new access token (rejects archived workspaces inline) @@ -820,6 +823,40 @@ async fn oauth_approve_inner( .map(|s| s.to_string()) .collect(); + // The approver's own token bounds what it may grant: a scope-restricted MCP + // token must not approve a broader one (e.g. mcp:scripts:f/x -> mcp:all). An + // unrestricted approver (interactive session, scopes None) grants freely, + // which is the normal consent flow. This is the legitimate MCP-narrowing + // path, so it uses MCP-pattern containment rather than the byte-identical + // rule ensure_scopes_within_caller applies on the generic token endpoints. + let caller_restricted = authed + .scopes + .as_deref() + .is_some_and(|s| s.iter().any(|x| !x.starts_with("if_jobs:filter_tags:"))); + if caller_restricted { + // An empty grant would mint a token the auth layer treats as unscoped + // (full privileges), so a restricted approver must not produce one. + if scopes.is_empty() { + return Err(Error::NotAuthorized( + "A scope-restricted token cannot approve an empty scope grant".to_string(), + )); + } + if scopes.iter().any(|s| !s.starts_with("mcp:")) { + return Err(Error::NotAuthorized( + "A scope-restricted token can only approve MCP (mcp:*) scopes".to_string(), + )); + } + let caller_config = parse_mcp_scopes(authed.scopes.as_deref().unwrap_or(&[])) + .map_err(|e| Error::InternalErr(format!("Failed to parse caller MCP scopes: {e}")))?; + let requested_config = parse_mcp_scopes(&scopes) + .map_err(|e| Error::BadRequest(format!("Failed to parse requested MCP scopes: {e}")))?; + if !caller_config.contains(&requested_config) { + return Err(Error::NotAuthorized( + "Requested scopes exceed the approving token's own MCP scopes".to_string(), + )); + } + } + sqlx::query!( "INSERT INTO mcp_oauth_server_code (code, client_id, user_email, workspace_id, scopes, redirect_uri, code_challenge, code_challenge_method) diff --git a/backend/windmill-api/src/mcp/utils.rs b/backend/windmill-api/src/mcp/utils.rs index 4073758ba9..2a0c34ae8d 100644 --- a/backend/windmill-api/src/mcp/utils.rs +++ b/backend/windmill-api/src/mcp/utils.rs @@ -455,9 +455,35 @@ pub async fn create_http_request( } }; + // Bound the minted JWT to exactly this proxied route so a scope-restricted + // MCP token can't be widened into a full-privilege blank check. The + // endpoint-name gate (in the MCP runner) already authorized *which* endpoint + // may be called; this constrains what the resulting request can do. Unscoped + // callers (cookie / full-privilege tokens) keep an unscoped JWT to preserve + // existing behavior. A scope-restricted caller whose route can't be resolved + // fails closed. + let caller_restricted = api_authed + .scopes + .as_deref() + .is_some_and(|s| s.iter().any(|x| !x.starts_with("if_jobs:filter_tags:"))); + let scopes = if caller_restricted { + let parsed = reqwest::Url::parse(url) + .map_err(|e| ErrorData::internal_error(format!("Invalid proxied URL: {}", e), None))?; + let scope = + windmill_api_auth::scopes::scope_for_route(method, parsed.path()).ok_or_else(|| { + ErrorData::internal_error( + "Could not derive route scope for proxied MCP endpoint".to_string(), + None, + ) + })?; + Some(vec![scope]) + } else { + None + }; + // Add authorization header let authed = Authed::from(api_authed.clone()); - let token = create_jwt_token(authed, workspace_id, 3600, None, None, None, None) + let token = create_jwt_token(authed, workspace_id, 3600, None, None, None, scopes) .await .map_err(|e| ErrorData::internal_error(e.to_string(), None))?; request_builder = request_builder.header("Authorization", format!("Bearer {}", token)); diff --git a/backend/windmill-api/src/offboarding.rs b/backend/windmill-api/src/offboarding.rs index 3d5342e3a5..24b6d3f10d 100644 --- a/backend/windmill-api/src/offboarding.rs +++ b/backend/windmill-api/src/offboarding.rs @@ -1,13 +1,13 @@ use std::collections::HashMap; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use axum::{ extract::{Extension, Path}, Json, }; use serde::{Deserialize, Serialize}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_api_users::users::delete_workspace_user_internal; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; @@ -483,11 +483,13 @@ pub(crate) async fn global_offboard_preview( pub(crate) async fn offboard_global_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Path(email): Path, Json(req): Json, ) -> Result> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let workspaces = sqlx::query!( "SELECT workspace_id, username FROM usr WHERE email = $1", diff --git a/backend/windmill-api/src/secret_backend_ext.rs b/backend/windmill-api/src/secret_backend_ext.rs index f76ba0ba64..fd17b8271e 100644 --- a/backend/windmill-api/src/secret_backend_ext.rs +++ b/backend/windmill-api/src/secret_backend_ext.rs @@ -8,245 +8,24 @@ //! Secret backend extension for the API layer //! -//! This module provides helper functions for integrating the SecretBackend -//! trait with variable operations in the API. +//! Backend resolution and read helpers live in +//! `windmill_common::secret_backend`; this module keeps the API-specific bulk +//! rename helper used when renaming users. //! //! Note: HashiCorp Vault integration requires Enterprise Edition. //! The OSS version only supports the database backend. -#[cfg(all(feature = "private", feature = "enterprise"))] -use std::sync::Arc; - use windmill_common::{db::DB, error::Result}; -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::error::Error; - -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::secret_backend::{database::DatabaseBackend, SecretBackend}; - #[cfg(all(feature = "private", feature = "enterprise"))] use windmill_common::{ - global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, + error::Error, secret_backend::{ - AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, - AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, + get_secret_backend, is_aws_sm_stored_value, is_azure_kv_stored_value, + is_external_stored_value, is_vault_backend_configured, }, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use tokio::sync::RwLock; - -// Cached Vault backend to avoid recreating it for every request -// This enables connection pooling and avoids repeated setup overhead -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedVaultBackend { - backend: Arc, - settings: VaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached Azure Key Vault backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAzureKvBackend { - backend: Arc, - settings: AzureKeyVaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached AWS Secrets Manager backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAwsSmBackend { - backend: Arc, - settings: AwsSecretsManagerSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -/// Get the current secret backend based on global settings (EE only) -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_secret_backend(db: &DB) -> Result> { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - match config { - SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), - SecretBackendConfig::HashiCorpVault(settings) => { - get_or_create_vault_backend(db, settings).await - } - SecretBackendConfig::AzureKeyVault(settings) => { - get_or_create_azure_kv_backend(db, settings).await - } - SecretBackendConfig::AwsSecretsManager(settings) => { - get_or_create_aws_sm_backend(db, settings).await - } - } -} - -/// Get a cached Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_vault_backend( - _db: &DB, - settings: VaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = VAULT_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = VAULT_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = { - #[cfg(feature = "openidconnect")] - if settings.token.is_none() { - Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) - } else { - Arc::new(VaultBackend::new(settings.clone())) - } - - #[cfg(not(feature = "openidconnect"))] - Arc::new(VaultBackend::new(settings.clone())) - }; - - // Cache it - *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached Azure Key Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_azure_kv_backend( - _db: &DB, - settings: AzureKeyVaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = AZURE_KV_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = AZURE_KV_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); - - // Cache it - *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached AWS SM backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_aws_sm_backend( - _db: &DB, - settings: AwsSecretsManagerSettings, -) -> Result> { - { - let cache = AWS_SM_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - let mut cache = AWS_SM_BACKEND_CACHE.write().await; - - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - let backend: Arc = - Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); - - *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Check if an external secret backend is currently configured (EE only) -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn is_vault_backend_configured(db: &DB) -> Result { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - Ok(matches!( - config, - SecretBackendConfig::HashiCorpVault(_) - | SecretBackendConfig::AzureKeyVault(_) - | SecretBackendConfig::AwsSecretsManager(_) - )) -} - -/// Check if a value is stored in Vault (indicated by the $vault: prefix) -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_vault_stored_value(value: &str) -> bool { - value.starts_with("$vault:") -} - -/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_azure_kv_stored_value(value: &str) -> bool { - value.starts_with("$azure_kv:") -} - -/// Check if a value is stored in AWS Secrets Manager -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_aws_sm_stored_value(value: &str) -> bool { - value.starts_with("$aws_sm:") -} - -/// Check if a value is stored in any external secret backend -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_external_stored_value(value: &str) -> bool { - is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) -} - /// Bulk rename secrets in Vault when a path prefix changes (e.g., user rename) /// EE only feature. /// diff --git a/backend/windmill-api/src/token.rs b/backend/windmill-api/src/token.rs index ac4da6497e..7a4a4887f8 100644 --- a/backend/windmill-api/src/token.rs +++ b/backend/windmill-api/src/token.rs @@ -98,6 +98,7 @@ fn build_standard_scope_domains() -> Vec { ("configs", "Configs", "Configuration management", false), ("oauth", "OAuth", "OAuth management", false), ("ai", "AI", "AI feature management", false), + ("ai_skills", "AI Skills", "AI skill management", false), ( "agent_workers", "Agent Workers", @@ -193,6 +194,19 @@ lazy_static! { ], }]; + // Read-only: `/api/docs/*` exposes only GET routes, so there is no + // `docs:write`. Kept out of build_standard_scope_domains (which mints a + // read+write pair) for that reason. + groups.push(ScopeDomain { + name: "Documentation".to_string(), + description: Some("Read-only documentation search".to_string()), + scopes: vec![ScopeOption { + value: "docs:read".to_string(), + label: "Read".to_string(), + requires_resource_path: false, + }], + }); + groups.extend(build_standard_scope_domains()); groups.extend(build_trigger_scope_domains()); @@ -207,3 +221,21 @@ pub fn global_service() -> Router { async fn get_all_available_scopes() -> JsonResult> { Ok(Json(ALL_SCOPES.clone())) } + +#[cfg(test)] +mod tests { + use super::*; + + /// The token-scope picker is driven by this catalog, so a scope that is + /// enforced but absent here can't be granted through the supported UI. + #[test] + fn docs_read_scope_is_exposed_read_only() { + let values: Vec<&str> = ALL_SCOPES + .iter() + .flat_map(|d| d.scopes.iter()) + .map(|s| s.value.as_str()) + .collect(); + assert!(values.contains(&"docs:read"), "docs:read must be selectable"); + assert!(!values.contains(&"docs:write"), "docs has no write surface"); + } +} diff --git a/backend/windmill-api/src/triggers/http/handler.rs b/backend/windmill-api/src/triggers/http/handler.rs index 0fd7717582..0508586182 100644 --- a/backend/windmill-api/src/triggers/http/handler.rs +++ b/backend/windmill-api/src/triggers/http/handler.rs @@ -216,7 +216,7 @@ async fn get_http_route_trigger( let email = windmill_common::users::get_email_from_permissioned_as( &trigger.permissioned_as, &trigger.workspace_id, - &db, + db, ) .await?; let authed = windmill_api_auth::fetch_api_authed_from_permissioned_as( diff --git a/backend/windmill-api/src/users.rs b/backend/windmill-api/src/users.rs index 0da987e675..36b0d6c734 100644 --- a/backend/windmill-api/src/users.rs +++ b/backend/windmill-api/src/users.rs @@ -11,7 +11,7 @@ pub use windmill_api_users::users::*; use std::sync::Arc; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use argon2::Argon2; use axum::{ @@ -21,7 +21,7 @@ use axum::{ }; use hyper::StatusCode; use serde::Deserialize; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; use windmill_common::audit::AuditAuthor; @@ -71,11 +71,13 @@ pub fn make_unauthed_service() -> Router { async fn create_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Extension(webhook): Extension, Extension(argon2): Extension>>, Json(nu): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::create_user(authed, db, webhook, argon2, nu).await } @@ -141,8 +143,10 @@ async fn set_password( Extension(db): Extension, Extension(argon2): Extension>>, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let email = authed.email.clone(); crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -152,9 +156,11 @@ async fn set_password_of_user( Extension(argon2): Extension>>, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -165,11 +171,13 @@ struct RenameUser { async fn rename_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(user_email): Path, Extension(db): Extension, Json(ru): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; diff --git a/backend/windmill-api/src/workspaces_export.rs b/backend/windmill-api/src/workspaces_export.rs index 8af8b4ebd7..fc2bfab8b4 100644 --- a/backend/windmill-api/src/workspaces_export.rs +++ b/backend/windmill-api/src/workspaces_export.rs @@ -12,6 +12,8 @@ use crate::db::ApiAuthed; use crate::{apps::AppWithLastVersion, db::DB, folders::Folder}; +use windmill_api_auth::check_scopes; + #[cfg(any( feature = "http_trigger", feature = "websocket", @@ -582,6 +584,18 @@ pub(crate) async fn tarball_workspace( skip_resources ); + // The route is gated by workspaces:read, but exporting DECRYPTED secrets is a + // variable-read capability beyond workspace metadata. Require variables:read + // only on the plaintext-secret path: ordinary tarball pulls (structure and + // encrypted-only values) keep working with workspaces:read, and the workspace + // key itself stays admin-only (include_key). No-op for unscoped tokens. + if plain_secret.or(plain_secrets).unwrap_or(false) + && !skip_secrets.unwrap_or(false) + && !skip_variables.unwrap_or(false) + { + check_scopes(&authed, || "variables:read".to_string())?; + } + // Opt-in behavior for surfacing per-resource ACLs on flow/app rows. // Folder and group rows have always carried `extra_perms` in source and // continue to do so unconditionally (`KeepEvenEmpty`) so existing @@ -592,8 +606,37 @@ pub(crate) async fn tarball_workspace( ExtraPermsBehavior::Drop }; + // Resolve workspace dependencies on the pool *before* opening the RLS + // transaction: fetching them mid-transaction would hold a second + // simultaneous connection while `tx` is still checked out. + let workspace_dependencies = if include_workspace_dependencies.unwrap_or(false) + && require_admin(authed.is_admin, &authed.username).is_ok() + { + Some(WorkspaceDependencies::list(&w_id, &db).await?) + } else { + None + }; + let mut tx = user_db.begin(&authed).await?; + // Exporting decrypted secrets in bulk is the same capability as a per-item + // secret read, so record it for parity with variables.decrypt_secret. + if plain_secret.or(plain_secrets).unwrap_or(false) + && !skip_variables.unwrap_or(false) + && !skip_secrets.unwrap_or(false) + { + windmill_audit::audit_oss::audit_log( + &mut *tx, + &authed, + "variables.decrypt_secret", + windmill_audit::ActionKind::Execute, + &w_id, + Some("workspace_tarball_export"), + None, + ) + .await?; + } + // Source-of-truth for fork-ness: the workspace's parent_workspace_id column. // The wm-fork-* prefix is a creation-time naming convention that could in // principle drift (rename, manual SQL); the column is the contract that @@ -851,11 +894,8 @@ pub(crate) async fn tarball_workspace( } } - if include_workspace_dependencies.unwrap_or(false) - && require_admin(authed.is_admin, &authed.username).is_ok() - { + if let Some(workspace_dependencies) = workspace_dependencies { tracing::info!("Including workspace dependencies in tarball export"); - let workspace_dependencies = WorkspaceDependencies::list(&w_id, &db).await?; tracing::info!( "Found {} workspace dependencies", workspace_dependencies.len() diff --git a/backend/windmill-common/src/assets.rs b/backend/windmill-common/src/assets.rs index 2b70227454..c186f59141 100644 --- a/backend/windmill-common/src/assets.rs +++ b/backend/windmill-common/src/assets.rs @@ -4,7 +4,10 @@ use sqlx::{PgExecutor, Postgres, Transaction}; use crate::{error, scripts::ScriptHash}; -pub use windmill_parser::asset_parser::{parse_pipeline_annotations, TriggerSpec, PARTITION_TOKEN}; +pub use windmill_parser::asset_parser::{ + merge_column_lineage, parse_pipeline_annotations, ColumnLineage, ColumnRef, DataTest, + PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN, +}; pub use windmill_types::assets::*; #[derive(sqlx::Type, Debug, Clone, Copy, PartialEq)] diff --git a/backend/windmill-common/src/auth.rs b/backend/windmill-common/src/auth.rs index 7fc6a0825a..5c1cedc6d8 100644 --- a/backend/windmill-common/src/auth.rs +++ b/backend/windmill-common/src/auth.rs @@ -287,7 +287,7 @@ impl From for Authed { } } -pub async fn is_super_admin_email(db: &DB, email: &str) -> Result { +pub async fn is_super_admin_email<'c>(db: impl sqlx::PgExecutor<'c>, email: &str) -> Result { if email == SUPERADMIN_SECRET_EMAIL || email == SUPERADMIN_NOTIFICATION_EMAIL { return Ok(true); } diff --git a/backend/windmill-common/src/bench.rs b/backend/windmill-common/src/bench.rs index 1ed2b9aa56..68c3cb5a31 100644 --- a/backend/windmill-common/src/bench.rs +++ b/backend/windmill-common/src/bench.rs @@ -258,6 +258,8 @@ pub async fn benchmark_init(benchmark_jobs: i32, db: &DB) { .execute(db) .await .unwrap_or_else(|e| panic!("failed to clean up concurrency_counter: {e:#}")); + // Benchmark jobs never produce dispatch_event / flow_conversation_message / + // zombie_job_counter rows, so this cleanup needs no side-table deletes (cf. delete_jobs). sqlx::query!("DELETE FROM v2_job WHERE workspace_id = 'admins'") .execute(db) .await diff --git a/backend/windmill-common/src/cache.rs b/backend/windmill-common/src/cache.rs index e39fd6ea43..6c34db8169 100644 --- a/backend/windmill-common/src/cache.rs +++ b/backend/windmill-common/src/cache.rs @@ -1189,11 +1189,25 @@ const _: () = { use std::fs::OpenOptions; use std::io::Write; + // Atomic write: truncate+write a uniquely-named temp file (UUID, not pid — pids + // collide across container PID namespaces on a shared cache volume), fsync, then + // rename(2) over the target. Without this a shorter overwrite leaves a stale tail + // and concurrent writers tear the file — a corrupt entry a reader would import. + let final_path = item.path(self); + let tmp_path = final_path.with_extension(format!("tmp.{}", Uuid::new_v4())); OpenOptions::new() .write(true) .create(true) - .open(item.path(self)) - .and_then(|mut file| file.write_all(data.as_ref())) + .truncate(true) + .open(&tmp_path) + .and_then(|mut file| { + file.write_all(data.as_ref())?; + file.sync_all() + }) + .and_then(|()| std::fs::rename(&tmp_path, &final_path)) + .inspect_err(|_| { + let _ = std::fs::remove_file(&tmp_path); + }) } } @@ -1237,6 +1251,33 @@ mod tests { use super::*; use serde_json::json; + #[test] + fn fs_cache_put_overwrites_without_stale_tail() { + // Regression for the non-truncating, non-atomic `put`: overwriting a value with a + // shorter one must not leave stale trailing bytes (which imported as corrupt/wrong + // content — the #9751 worker-cache hazard). + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + + root.put("k", b"a-long-cached-value-0123456789").unwrap(); + assert_eq!(root.get("k").unwrap(), b"a-long-cached-value-0123456789"); + + root.put("k", b"short").unwrap(); + assert_eq!( + root.get("k").unwrap(), + b"short", + "shorter overwrite must fully replace, no stale tail" + ); + + // No temp files left behind after a successful write. + let leftover: Vec<_> = std::fs::read_dir(root) + .unwrap() + .filter_map(|e| e.ok()) + .filter(|e| e.file_name().to_string_lossy().contains(".tmp.")) + .collect(); + assert!(leftover.is_empty(), "temp files must be renamed/cleaned up"); + } + #[test] fn flow_data_extras_preserves_notes_and_groups() { let raw = serde_json::value::to_raw_value(&json!({ diff --git a/backend/windmill-common/src/git_sync_oss.rs b/backend/windmill-common/src/git_sync_oss.rs index 4cd7a7c3d1..1e17532ec7 100644 --- a/backend/windmill-common/src/git_sync_oss.rs +++ b/backend/windmill-common/src/git_sync_oss.rs @@ -15,6 +15,29 @@ pub async fn get_github_app_token_internal( )); } +lazy_static::lazy_static! { + /// Matches a `user:password@` (or `user@`) userinfo component right after the URL scheme. + static ref GIT_URL_USERINFO_RE: regex::Regex = + regex::Regex::new(r"://[^/@]+@").unwrap(); +} + +/// Strip embedded credentials (the `user:password@` userinfo component) from a git URL so it can be +/// safely included in error messages and logs. Falls back to a regex when the URL does not parse. +pub fn sanitize_git_url(url: &str) -> String { + if let Ok(mut parsed) = Url::parse(url) { + if !parsed.username().is_empty() || parsed.password().is_some() { + // These setters only fail for cannot-be-a-base URLs, in which case we keep the parsed + // string as-is and let the regex fallback below handle stripping. + let _ = parsed.set_username(""); + let _ = parsed.set_password(None); + } + return GIT_URL_USERINFO_RE + .replace(parsed.as_str(), "://***@") + .into_owned(); + } + GIT_URL_USERINFO_RE.replace(url, "://***@").into_owned() +} + pub fn prepend_token_to_github_url( github_url: &str, installation_token: &str, @@ -32,3 +55,41 @@ pub fn prepend_token_to_github_url( url.path() )) } + +#[cfg(test)] +mod tests { + use super::sanitize_git_url; + + #[test] + fn strips_username_and_password() { + assert_eq!( + sanitize_git_url("https://user:p4ssw0rd@github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn strips_token_only_userinfo() { + assert_eq!( + sanitize_git_url("https://ghp_secrettoken@github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn leaves_credential_free_url_untouched() { + assert_eq!( + sanitize_git_url("https://github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn strips_credentials_from_unparseable_url() { + // scp-like syntax that `url::Url` cannot parse + assert_eq!( + sanitize_git_url("not a url://user:secret@host/repo"), + "not a url://***@host/repo" + ); + } +} diff --git a/backend/windmill-common/src/jobs.rs b/backend/windmill-common/src/jobs.rs index 78f5a6ee5a..2178d9a023 100644 --- a/backend/windmill-common/src/jobs.rs +++ b/backend/windmill-common/src/jobs.rs @@ -343,8 +343,8 @@ lazy_static::lazy_static! { ).unwrap_or(false); } -pub async fn check_tag_available_for_workspace_internal( - db: &DB, +pub async fn check_tag_available_for_workspace_internal<'c>( + db: impl sqlx::PgExecutor<'c>, w_id: &str, tag: &str, email: &str, @@ -458,6 +458,47 @@ pub struct WorkerInternalServerInlineUtils { pub static WORKER_INTERNAL_SERVER_INLINE_UTILS: OnceCell = OnceCell::new(); +/// Deletes the given jobs from `v2_job` together with the side tables that reference it +/// without an `ON DELETE CASCADE` foreign key. +/// +/// **Authorization contract:** this helper does NO authorization and NO workspace scoping — +/// it deletes exactly the `ids` passed, regardless of which workspace they belong to. Callers +/// MUST ensure `ids` only contains jobs the caller is allowed to delete (either a trusted +/// internal id set, e.g. a retention batch, or ids already filtered by `workspace_id`). +/// Passing user-supplied, unvalidated ids would reintroduce the cross-workspace side-row +/// deletion this centralizes. It is deliberately not workspace-scoped at the signature level +/// because its primary caller — retention — deletes expired jobs across every workspace at +/// once; a `workspace_id` parameter cannot express that. (This is the same trust model as the +/// `ON DELETE CASCADE` FK it replaces: given a job id, the row and its side rows go.) +/// +/// Those FKs were removed (migration `drop_v2_job_side_table_cascades`) because they turned +/// every bulk retention delete into a per-row RI trigger; for the unindexed +/// `flow_conversation_message.job_id` that was a sequential scan per deleted row. The +/// set-based deletes below cost one scan per table per call instead. Because the cascade no +/// longer fires, every code path that deletes from `v2_job` by id must go through this helper +/// (or delete these tables itself) or it will leave orphan rows behind. +pub async fn delete_jobs(conn: &mut sqlx::PgConnection, ids: &[uuid::Uuid]) -> error::Result<()> { + sqlx::query!( + "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)", + ids + ) + .execute(&mut *conn) + .await?; + sqlx::query!( + "DELETE FROM flow_conversation_message WHERE job_id = ANY($1)", + ids + ) + .execute(&mut *conn) + .await?; + sqlx::query!("DELETE FROM zombie_job_counter WHERE job_id = ANY($1)", ids) + .execute(&mut *conn) + .await?; + sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", ids) + .execute(&mut *conn) + .await?; + Ok(()) +} + #[cfg(test)] mod tests { use super::is_safe_log_file_path; diff --git a/backend/windmill-common/src/lib.rs b/backend/windmill-common/src/lib.rs index 7c2a475361..be336a84e1 100644 --- a/backend/windmill-common/src/lib.rs +++ b/backend/windmill-common/src/lib.rs @@ -285,6 +285,16 @@ lazy_static::lazy_static! { const LATEST_VERSION_ID_CACHE_TTL: std::time::Duration = std::time::Duration::from_secs(60); +/// Test hook: disables the process-global deployed-script hash/info caches so +/// every resolution reads the current DB. Integration tests use `#[sqlx::test]` +/// isolated DBs that share one workspace id and reuse script paths, so a cache +/// keyed by `(workspace, path)`/`(workspace, hash)` resolves a path to a hash +/// that lives in a *different* test's DB — and when the info cache misses for +/// that foreign hash the lookup 404s in the wrong DB. Always `false` in +/// production (the caches are TTL/LRU-bounded against real deploys). +pub static DEPLOYED_SCRIPT_CACHE_DISABLED: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + pub async fn shutdown_signal( tx: KillpillSender, mut rx: tokio::sync::broadcast::Receiver<()>, @@ -529,6 +539,31 @@ mod classify_python_logging_line_tests { } } +#[cfg(test)] +mod validate_dbname_tests { + use super::validate_dbname; + + #[test] + fn accepts_letters_digits_underscores_and_hyphens() { + assert!(validate_dbname("mydb").is_ok()); + assert!(validate_dbname("my_db").is_ok()); + assert!(validate_dbname("my-database").is_ok()); + assert!(validate_dbname("My-Db_1").is_ok()); + } + + #[test] + fn rejects_invalid_names() { + // Must start with a letter (hyphen/digit/underscore leads are rejected). + assert!(validate_dbname("-db").is_err()); + assert!(validate_dbname("1db").is_err()); + assert!(validate_dbname("_db").is_err()); + // No other special characters or whitespace. + assert!(validate_dbname("my db").is_err()); + assert!(validate_dbname("my;db").is_err()); + assert!(validate_dbname("").is_err()); + } +} + #[derive(Serialize, Debug)] pub struct PrepareQueryColumnInfo { pub name: String, @@ -812,7 +847,7 @@ impl PgDatabase { } /// Validate a database name to prevent SQL injection. -/// Must start with a letter, contain only alphanumeric characters or underscores, and be <= 63 chars. +/// Must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be <= 63 chars. pub fn validate_dbname(dbname: &str) -> error::Result<()> { let dbname = dbname.trim(); if dbname.is_empty() { @@ -836,10 +871,11 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> { } if !dbname .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '_') + .all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') { return Err(error::Error::BadRequest( - "Database name must contain only alphanumeric characters or underscores".to_string(), + "Database name must contain only alphanumeric characters, underscores, or hyphens" + .to_string(), )); } Ok(()) @@ -1279,8 +1315,12 @@ pub fn get_latest_deployed_hash_for_path<'e>( ) -> impl Future>> + Send + 'e { async move { let cache_key = (w_id.to_string(), script_path.to_string()); + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); let mut computed_hash = None; - let hash = match DEPLOYED_SCRIPT_HASH_CACHE.get(&cache_key) { + let hash = match DEPLOYED_SCRIPT_HASH_CACHE + .get(&cache_key) + .filter(|_| use_cache) + { Some(cached_hash) if cached_hash.expires_at > std::time::Instant::now() && db.as_ref().is_none_or(|x| { @@ -1323,13 +1363,15 @@ pub fn get_latest_deployed_hash_for_path<'e>( }; let hash = utils::not_found_if_none(hash, "script", script_path)?; - DEPLOYED_SCRIPT_HASH_CACHE.insert( - cache_key, - ExpiringLatestVersionId { - id: hash, - expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, - }, - ); + if use_cache { + DEPLOYED_SCRIPT_HASH_CACHE.insert( + cache_key, + ExpiringLatestVersionId { + id: hash, + expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, + }, + ); + } hash } @@ -1361,9 +1403,10 @@ pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( hash: i64, ) -> error::Result> { let key = (w_id.to_string(), hash); + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); let mut computed_hash = None; - match DEPLOYED_SCRIPT_INFO_CACHE.get(&key) { + match DEPLOYED_SCRIPT_INFO_CACHE.get(&key).filter(|_| use_cache) { Some(info) if db_authed.as_ref().is_none_or(|x| { let r = HASH_PERMS_CACHE.check_perms_in_cache(x.authed, scripts::ScriptHash(hash)); @@ -1392,7 +1435,9 @@ pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( let info = utils::not_found_if_none(info, "script", &hash.to_string())?; - DEPLOYED_SCRIPT_INFO_CACHE.insert(key, info.clone()); + if use_cache { + DEPLOYED_SCRIPT_INFO_CACHE.insert(key, info.clone()); + } Ok(info) } diff --git a/backend/windmill-common/src/materialization.rs b/backend/windmill-common/src/materialization.rs index ccc1f7782e..8d9e31899f 100644 --- a/backend/windmill-common/src/materialization.rs +++ b/backend/windmill-common/src/materialization.rs @@ -13,7 +13,8 @@ use chrono::{DateTime, Utc}; use serde::{Deserialize, Serialize}; -use sqlx::PgExecutor; +use sqlx::types::Json; +use sqlx::{PgExecutor, Postgres, Transaction}; use uuid::Uuid; use crate::assets::AssetKind; @@ -34,6 +35,17 @@ pub enum MaterializationStatus { Failed, } +/// One column of a captured asset output schema: its name and substrate type +/// (e.g. `{"name": "order_id", "type": "BIGINT"}`). `type` is the substrate's +/// own type spelling (DuckDB for ducklake) — kept verbatim so #2b can compare +/// declared vs. captured without a lossy normalization step. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SchemaColumn { + pub name: String, + #[serde(rename = "type")] + pub data_type: String, +} + /// The materialization outcome an agent worker (`Connection::Http`, no direct /// DB) sends to the API to be recorded. Mirrors the `record_materialization` /// args; the API handler unpacks it and calls that function with its own DB. @@ -47,6 +59,13 @@ pub struct RecordMaterializationRequest { pub row_count: Option, pub job_id: Option, pub error: Option, + /// Captured output schema of the materialized asset (`None` when the + /// substrate/run produced no schema, e.g. a failed run or a polyglot helper + /// that doesn't DESCRIBE). When present, the recorder also upserts a + /// `materialized_asset_schema` version. Defaults to `None` so older agents + /// stay wire-compatible. + #[serde(default)] + pub schema: Option>, } /// Upsert the latest materialization state for one (asset, partition) slice. @@ -133,3 +152,133 @@ pub async fn list_materialized_partitions<'e>( .await?; Ok(rows) } + +/// One captured schema version of an asset, newest first — the schema-evolution +/// history surfaced on the asset node and read by #2b contract enforcement. +#[derive(sqlx::FromRow, Debug, Clone, Serialize)] +pub struct AssetSchemaVersion { + pub version: i64, + pub columns: Json>, + pub snapshot_id: Option, + pub job_id: Option, + pub captured_at: DateTime, +} + +/// Record the captured output schema of a freshly-materialized asset. +/// +/// **Authorization:** like the sibling `record_materialization`, this performs +/// no access control of its own — it writes the row for whatever `workspace_id` +/// it is given. Callers MUST pass a workspace-authorized executor and a +/// `workspace_id` the caller is allowed to write: an RLS-scoped `user_db` +/// transaction for API / agent-worker entry points, or the trusted worker DB +/// pool for the in-worker recorder. Do not expose it to an unauthenticated path. +/// +/// Versioning across re-materializations: a new `version` row is inserted only +/// when `columns` differs from the latest stored version; an unchanged +/// re-materialize re-affirms the latest row in place (updates its +/// `snapshot_id`/`job_id`/`captured_at`). The result is a compact +/// schema-evolution history where `MAX(version)` is the current contract. +/// +/// Runs in a transaction guarded by a per-asset advisory lock so two concurrent +/// materializations of the same asset can't both insert the same next version +/// or interleave a stale comparison. Returns `true` if a new version was +/// inserted (the schema changed), `false` if the latest was re-affirmed. +pub async fn record_asset_schema( + tx: &mut Transaction<'_, Postgres>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, + columns: &[SchemaColumn], + snapshot_id: Option, + job_id: Option, +) -> Result { + // Serialize concurrent captures of the *same* asset; the lock auto-releases + // at tx end. Hash the identity into the bigint advisory-lock key space. + sqlx::query!( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0::int8))", + format!("materialized_asset_schema:{workspace_id}:{asset_kind:?}:{asset_path}"), + ) + .fetch_one(&mut **tx) + .await?; + + let latest = sqlx::query!( + r#"SELECT version, columns AS "columns: Json>" + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY version DESC + LIMIT 1"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_optional(&mut **tx) + .await?; + + let columns_json = Json(columns.to_vec()); + let next_version = match latest { + Some(latest) if latest.columns.0.as_slice() == columns => { + // Unchanged schema — re-affirm the latest version in place. + sqlx::query!( + "UPDATE materialized_asset_schema + SET snapshot_id = $5, job_id = $6, captured_at = now() + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + AND version = $4", + workspace_id, + asset_kind as AssetKind, + asset_path, + latest.version, + snapshot_id, + job_id, + ) + .execute(&mut **tx) + .await?; + return Ok(false); + } + Some(latest) => latest.version + 1, + None => 1, + }; + sqlx::query!( + "INSERT INTO materialized_asset_schema + (workspace_id, asset_kind, asset_path, version, columns, + snapshot_id, job_id, captured_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, now())", + workspace_id, + asset_kind as AssetKind, + asset_path, + next_version, + columns_json as Json>, + snapshot_id, + job_id, + ) + .execute(&mut **tx) + .await?; + Ok(true) +} + +/// All captured schema versions for one asset, newest version first. +/// +/// **Authorization:** performs no access control (mirrors +/// `list_materialized_partitions`); the caller must pass a workspace-authorized +/// executor (an RLS-scoped `user_db` transaction on the API read path) and a +/// `workspace_id` it is allowed to read. +pub async fn list_asset_schemas<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, +) -> Result> { + let rows = sqlx::query_as!( + AssetSchemaVersion, + r#"SELECT version, columns AS "columns: Json>", + snapshot_id, job_id, captured_at + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY version DESC"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_all(executor) + .await?; + Ok(rows) +} diff --git a/backend/windmill-common/src/query_builders.rs b/backend/windmill-common/src/query_builders.rs index 2c168cd433..ce7dbc2533 100644 --- a/backend/windmill-common/src/query_builders.rs +++ b/backend/windmill-common/src/query_builders.rs @@ -172,6 +172,19 @@ pub struct SimpleColumn { pub struct SelectOptions { pub limit: Option, pub offset: Option, + /// DuckLake time-travel: when set (DuckDB only), the read is pinned to this + /// catalog snapshot via `AT (VERSION => n)`. Ignored for other db types. + pub version: Option, +} + +/// DuckLake time-travel suffix appended after a table name in a FROM clause. +/// `n` is a server-controlled `i64` (a snapshot id), so inlining it is +/// injection-safe. Empty string when unpinned (reads the latest snapshot). +fn duckdb_version_suffix(version: Option) -> String { + match version { + Some(v) => format!(" AT (VERSION => {})", v), + None => String::new(), + } } // --------------------------------------------------------------------------- @@ -190,6 +203,8 @@ struct SelectPayload { #[serde(rename = "fixPgIntTypes")] fix_pg_int_types: Option, ducklake: Option, + /// DuckLake snapshot to time-travel the read to (DuckDB only). + version: Option, } #[derive(Deserialize)] @@ -200,6 +215,21 @@ struct CountPayload { #[serde(rename = "whereClause")] where_clause: Option, ducklake: Option, + /// DuckLake snapshot to time-travel the count to (DuckDB only). + version: Option, +} + +/// `WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS` payload — lists the time-travel history +/// of a ducklake table. DuckLake snapshots are catalog-wide commits, so without +/// a `table` this lists every commit; with one it is scoped to snapshots where +/// that table exists (see `expand_ducklake_snapshots`). +#[derive(Deserialize)] +struct DucklakeSnapshotsPayload { + ducklake: String, + /// Schema-qualified table name (e.g. `main.events_daily`) to scope the + /// history to. Snapshots predating the table's creation are excluded — a + /// time-travel read can't target a version where the table didn't exist. + table: Option, } #[derive(Deserialize)] @@ -304,6 +334,10 @@ pub fn try_expand_internal_db_query( expand_primary_key_constraint(json_str, db_type).map(ExpandedQuery::sql) } "SNOWFLAKE_PRIMARY_KEYS" => expand_snowflake_primary_keys(json_str).map(ExpandedQuery::sql), + // DuckLake time-travel: list a ducklake's snapshot history + "DUCKLAKE_SNAPSHOTS" => { + expand_ducklake_snapshots(json_str, db_type).map(ExpandedQuery::sql) + } _ => Err(format!("Unknown WM_INTERNAL_DB operation: {}", op)), }; @@ -324,7 +358,8 @@ fn expand_select(json_str: &str, db_type: DbType) -> Result { let payload: SelectPayload = serde_json::from_str(json_str).map_err(|e| format!("Invalid SELECT payload: {}", e))?; - let options = SelectOptions { limit: payload.limit, offset: payload.offset }; + let options = + SelectOptions { limit: payload.limit, offset: payload.offset, version: payload.version }; let breaking = payload .fix_pg_int_types .map(|v| BreakingFeatures { fix_pg_int_types: v }); @@ -350,11 +385,47 @@ fn expand_count(json_str: &str, db_type: DbType) -> Result { &payload.table, payload.where_clause.as_deref(), &payload.column_defs, + payload.version, )?; Ok(maybe_wrap_ducklake(query, payload.ducklake.as_deref())) } +/// Expand `DUCKLAKE_SNAPSHOTS` into the catalog's time-travel history. DuckLake +/// snapshots are catalog-wide commits, so `ducklake_snapshots('dl')` (the alias +/// `maybe_wrap_ducklake` attaches) lists every version any `AT (VERSION => n)` +/// read can target, newest first. +fn expand_ducklake_snapshots(json_str: &str, db_type: DbType) -> Result { + if db_type != DbType::Duckdb { + return Err("DUCKLAKE_SNAPSHOTS is only supported for DuckDB".to_string()); + } + let payload: DucklakeSnapshotsPayload = serde_json::from_str(json_str) + .map_err(|e| format!("Invalid DUCKLAKE_SNAPSHOTS payload: {}", e))?; + // `dl` is the alias `wrap_ducklake_query` attaches and `USE`s below. + let query = match &payload.table { + // Scope to snapshots from the table's first creation onward. A DuckLake + // table created at snapshot N can't be read before N (the catalog-wide + // list would otherwise offer impossible versions). The creation snapshot + // is the earliest whose `changes.tables_created` names the table; + // COALESCE to 0 (show all) if it is never found. + Some(table) => { + let table = escape_sql_literal(table); + format!( + "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') \ + WHERE snapshot_id >= COALESCE((\ + SELECT min(snapshot_id) FROM ducklake_snapshots('dl') \ + WHERE list_contains(changes.tables_created, '{table}')), 0) \ + ORDER BY snapshot_id DESC" + ) + } + None => { + "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') ORDER BY snapshot_id DESC" + .to_string() + } + }; + Ok(maybe_wrap_ducklake(query, Some(&payload.ducklake))) +} + /// Filter columns to primary keys only; fall back to all columns if none are marked. fn pk_columns_or_all(columns: &[ColumnDef]) -> Vec { let pks: Vec = columns.iter().filter(|c| c.isprimarykey).cloned().collect(); @@ -950,9 +1021,10 @@ pub fn make_select_query( ); query.push_str(&format!( - "SELECT {} FROM {}\n", + "SELECT {} FROM {}{}\n", filtered_columns.join(", "), - quote_table_name(table, db_type) + quote_table_name(table, db_type), + duckdb_version_suffix(options.and_then(|o| o.version)) )); query.push_str(&format!( " WHERE {} {}\n", @@ -977,6 +1049,8 @@ pub fn make_count_query( table: &str, where_clause: Option<&str>, column_defs: &[ColumnDef], + // DuckLake time-travel snapshot (DuckDB only); `None` counts the latest. + version: Option, ) -> Result { let where_prefix = " WHERE "; let and_condition = " AND "; @@ -1118,8 +1192,9 @@ pub fn make_count_query( quicksearch_condition.push_str(" ($quicksearch = '' OR 1 = 1)"); } query.push_str(&format!( - "SELECT COUNT(*) as count FROM {}", - quote_table_name(table, db_type) + "SELECT COUNT(*) as count FROM {}{}", + quote_table_name(table, db_type), + duckdb_version_suffix(version) )); } } @@ -2998,7 +3073,7 @@ mod tests { #[test] fn test_select_snowflake_custom_limit() { let cols = vec![col("id", "int")]; - let opts = SelectOptions { limit: Some(50), offset: Some(10) }; + let opts = SelectOptions { limit: Some(50), offset: Some(10), version: None }; let result = make_select_query( "my_table", &cols, @@ -3072,7 +3147,7 @@ mod tests { #[test] fn test_count_postgresql_basic() { let cols = vec![col("id", "int4"), col("name", "text")]; - let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- $1 quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\"")); @@ -3090,6 +3165,7 @@ mod tests { "my_table", Some("status = 'active'"), &cols, + None, ) .unwrap(); @@ -3105,7 +3181,7 @@ mod tests { c.ignored = Some(true); c }]; - let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("($1 = '' OR 1 = 1)")); } @@ -3116,7 +3192,7 @@ mod tests { #[test] fn test_count_mysql_basic() { let cols = vec![col("id", "int"), col("name", "varchar")]; - let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- :quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`")); @@ -3130,7 +3206,7 @@ mod tests { #[test] fn test_count_mssql_basic() { let cols = vec![col("id", "int"), col("name", "nvarchar")]; - let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap(); assert!(result.contains("SELECT COUNT(*) as count FROM [my_table]")); assert!(result.contains("(@p1 = '' OR CONCAT([id], [name]) LIKE '%' + @p1 + '%')")); @@ -3143,7 +3219,7 @@ mod tests { #[test] fn test_count_snowflake_basic() { let cols = vec![col("id", "int"), col("name", "text")]; - let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap(); // Two quicksearch params for snowflake with visible columns assert!(result.contains("-- ? quicksearch (text)\n-- ? quicksearch (text)")); @@ -3158,7 +3234,7 @@ mod tests { c.ignored = Some(true); c }]; - let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap(); // One quicksearch param let param_lines: Vec<&str> = result.lines().filter(|l| l.starts_with("-- ?")).collect(); assert_eq!(param_lines.len(), 1); @@ -3172,7 +3248,7 @@ mod tests { #[test] fn test_count_bigquery_basic() { let cols = vec![col("id", "INTEGER"), col("name", "STRING")]; - let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- @quicksearch (string)")); assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`")); @@ -3182,7 +3258,7 @@ mod tests { #[test] fn test_count_bigquery_json_type() { let cols = vec![col("id", "INTEGER"), col("data", "JSON")]; - let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap(); assert!(result.contains("TO_JSON_STRING(`data`)")); } @@ -3193,7 +3269,7 @@ mod tests { #[test] fn test_count_duckdb_basic() { let cols = vec![col("id", "int"), col("name", "text")]; - let result = make_count_query(DbType::Duckdb, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Duckdb, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- $quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\"")); @@ -3202,6 +3278,74 @@ mod tests { ); } + // ----------------------------------------------------------------------- + // DuckLake time-travel (AT VERSION) + snapshot history + // ----------------------------------------------------------------------- + + #[test] + fn test_select_duckdb_time_travel() { + let cols = vec![col("id", "int"), col("name", "text")]; + let opts = SelectOptions { limit: None, offset: None, version: Some(42) }; + let result = + make_select_query("orders", &cols, None, DbType::Duckdb, Some(&opts), None).unwrap(); + // Read is pinned to the catalog snapshot via AT (VERSION => n). + assert!(result.contains("FROM \"orders\" AT (VERSION => 42)\n")); + } + + #[test] + fn test_select_duckdb_no_version_unpinned() { + let cols = vec![col("id", "int")]; + let result = make_select_query("orders", &cols, None, DbType::Duckdb, None, None).unwrap(); + // Without a version the read targets the latest snapshot — no AT clause. + assert!(result.contains("FROM \"orders\"\n")); + assert!(!result.contains("AT (VERSION")); + } + + #[test] + fn test_count_duckdb_time_travel() { + let cols = vec![col("id", "int")]; + let result = make_count_query(DbType::Duckdb, "orders", None, &cols, Some(7)).unwrap(); + assert!(result.contains("FROM \"orders\" AT (VERSION => 7)")); + } + + #[test] + fn test_version_ignored_for_non_duckdb() { + // AT (VERSION) is DuckLake-only; other dialects must never emit it even + // if a version is somehow passed through. + let cols = vec![col("id", "int4")]; + let opts = SelectOptions { limit: None, offset: None, version: Some(5) }; + let result = + make_select_query("orders", &cols, None, DbType::Postgresql, Some(&opts), None) + .unwrap(); + assert!(!result.contains("AT (VERSION")); + } + + #[test] + fn test_expand_ducklake_snapshots() { + let json = r#"{"ducklake": "analytics"}"#; + let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap(); + assert!(result.contains("ATTACH 'ducklake://analytics' AS dl;USE dl;")); + assert!(result.contains("ducklake_snapshots('dl')")); + assert!(result.contains("ORDER BY snapshot_id DESC")); + // Unscoped: no per-table existence filter. + assert!(!result.contains("tables_created")); + } + + #[test] + fn test_expand_ducklake_snapshots_scoped_to_table() { + let json = r#"{"ducklake": "analytics", "table": "main.events_daily"}"#; + let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap(); + // Scoped to snapshots from the table's first creation onward. + assert!(result.contains("list_contains(changes.tables_created, 'main.events_daily')")); + assert!(result.contains("snapshot_id >= COALESCE")); + } + + #[test] + fn test_expand_ducklake_snapshots_non_duckdb_errors() { + let json = r#"{"ducklake": "analytics"}"#; + assert!(expand_ducklake_snapshots(json, DbType::Postgresql).is_err()); + } + // ----------------------------------------------------------------------- // DELETE - all DB types // ----------------------------------------------------------------------- @@ -3500,7 +3644,7 @@ mod tests { #[test] fn test_count_mssql_no_where() { let cols = vec![col("id", "int")]; - let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap(); // MSSQL uses WHERE directly (no AND replacement) assert!(result.contains("SELECT COUNT(*) as count FROM [my_table] WHERE ")); } @@ -3508,7 +3652,7 @@ mod tests { #[test] fn test_count_mysql_no_where_uses_where_keyword() { let cols = vec![col("id", "int")]; - let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap(); // The AND should be replaced with WHERE assert!(result.contains("FROM `my_table` WHERE ")); assert!(!result.contains("FROM `my_table` AND ")); diff --git a/backend/windmill-common/src/runnable_settings/mod.rs b/backend/windmill-common/src/runnable_settings/mod.rs index c5692d40bd..33d4c1a039 100644 --- a/backend/windmill-common/src/runnable_settings/mod.rs +++ b/backend/windmill-common/src/runnable_settings/mod.rs @@ -52,13 +52,10 @@ pub trait RunnableSettingsTrait: /// get [[Self]] from cache or fetch from db /// if not found, returns Error - fn get<'a>( + fn get<'e>( hash: i64, - db: &'a Pool, - ) -> impl Future> - where - Self: 'a, - { + db: impl sqlx::PgExecutor<'e>, + ) -> impl Future> { async move { let v = RUNNABLE_INDIVIDUAL_SETTINGS .get_or_insert_async(hash, async { diff --git a/backend/windmill-common/src/runnable_settings/settings.rs b/backend/windmill-common/src/runnable_settings/settings.rs index 04b62fbaab..b814646553 100644 --- a/backend/windmill-common/src/runnable_settings/settings.rs +++ b/backend/windmill-common/src/runnable_settings/settings.rs @@ -1,5 +1,3 @@ -use std::future::Future; - use sqlx::{Pool, Postgres}; use crate::{ @@ -38,29 +36,63 @@ pub async fn prefetch_cached_from_handle( prefetch_cached(&rs, db).await } +/// Like [`prefetch_cached`], but reuses a held transaction's connection instead +/// of checking out a second one from the pool. Use this on paths that already +/// hold an open `tx` to avoid dual-connection pool contention. +pub async fn prefetch_cached_tx( + rs: &RunnableSettings, + tx: &mut sqlx::Transaction<'_, Postgres>, +) -> error::Result<(DebouncingSettings, ConcurrencySettings)> { + Ok(( + if let Some(hash) = rs.debouncing_settings { + DebouncingSettings::get(hash, &mut **tx).await? + } else { + Default::default() + }, + if let Some(hash) = rs.concurrency_settings { + ConcurrencySettings::get(hash, &mut **tx).await? + } else { + Default::default() + }, + )) +} + +/// Resolve the retry policy (if any) for a job from its `runnable_settings_handle`. +/// Returns `None` when the job carries no retry policy. Read lazily on the +/// failure path only — never on the hot job-pull path. +pub async fn prefetch_retry_from_handle( + hash: Option, + db: &DB, +) -> error::Result> { + let rs = from_handle(hash, db).await?; + Ok(if let Some(hash) = rs.retry_settings { + Some(RetrySettings::get(hash, db).await?) + } else { + None + }) +} + /// Returns error if provided `hash` has no corresponding entry in db /// If `hash` is None, returns Default -pub fn from_handle<'a>( +pub async fn from_handle<'e>( hash: Option, - db: &'a DB, -) -> impl Future> + 'a { - async move { - if let Some(hash) = hash { - super::RUNNABLE_SETTINGS_REFERENCES - .get_or_insert_async(hash, async { - sqlx::query_as!( - RunnableSettings, - r#"SELECT concurrency_settings, debouncing_settings FROM runnable_settings WHERE hash = $1"#, - hash - ) - .fetch_one(db) - .await - .map_err(error::Error::from) - }) + db: impl sqlx::PgExecutor<'e>, +) -> error::Result { + if let Some(hash) = hash { + super::RUNNABLE_SETTINGS_REFERENCES + .get_or_insert_async(hash, async { + sqlx::query_as!( + RunnableSettings, + r#"SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1"#, + hash + ) + .fetch_one(db) .await - } else { - Ok(RunnableSettings::default()) - } + .map_err(error::Error::from) + }) + .await + } else { + Ok(RunnableSettings::default()) } } @@ -68,7 +100,9 @@ pub async fn insert_rs(rs: RunnableSettings, db: &Pool) -> error::Resu use std::hash::{Hash, Hasher}; if !min_version_supports_runnable_settings_v0().await - || (rs.debouncing_settings.is_none() && rs.concurrency_settings.is_none()) + || (rs.debouncing_settings.is_none() + && rs.concurrency_settings.is_none() + && rs.retry_settings.is_none()) { return Ok(None); } @@ -82,13 +116,14 @@ pub async fn insert_rs(rs: RunnableSettings, db: &Pool) -> error::Resu super::RUNNABLE_SETTINGS_REFERENCES .get_or_insert_async(hash, async { sqlx::query!( - "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings) - VALUES ($1, $2, $3) + "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings) + VALUES ($1, $2, $3, $4) ON CONFLICT (hash) DO NOTHING", hash, rs.debouncing_settings, - rs.concurrency_settings + rs.concurrency_settings, + rs.retry_settings ) .execute(db) .await?; @@ -132,3 +167,25 @@ impl super::private_mod::RunnableSettingsTraitInternal for ConcurrencySettings { } } impl super::RunnableSettingsTrait for ConcurrencySettings {} +impl super::private_mod::RunnableSettingsTraitInternal for RetrySettings { + const SETTINGS_NAME: &str = "retry_settings"; + const INCLUDE_FIELDS: &[&str] = &[ + "constant_attempts", + "constant_seconds", + "exponential_attempts", + "exponential_multiplier", + "exponential_seconds", + "exponential_random_factor", + "retry_if_expr", + ]; + fn bind_arguments<'a>(&'a self, q: Q<'a>) -> Q<'a> { + q.bind(&self.constant_attempts) + .bind(&self.constant_seconds) + .bind(&self.exponential_attempts) + .bind(&self.exponential_multiplier) + .bind(&self.exponential_seconds) + .bind(&self.exponential_random_factor) + .bind(&self.retry_if_expr) + } +} +impl super::RunnableSettingsTrait for RetrySettings {} diff --git a/backend/windmill-common/src/scripts.rs b/backend/windmill-common/src/scripts.rs index 0cbcb4a1c1..f3abf14e1c 100644 --- a/backend/windmill-common/src/scripts.rs +++ b/backend/windmill-common/src/scripts.rs @@ -345,10 +345,10 @@ pub async fn clone_script<'c>( ))); }; - let rs = - runnable_settings::from_handle(s.runnable_settings.runnable_settings_handle, db).await?; + let rs = runnable_settings::from_handle(s.runnable_settings.runnable_settings_handle, &mut *tx) + .await?; let (debouncing_settings, concurrency_settings) = - runnable_settings::prefetch_cached(&rs, db).await?; + runnable_settings::prefetch_cached_tx(&rs, &mut tx).await?; let ns = NewScript { path: s.path.clone(), diff --git a/backend/windmill-common/src/secret_backend/mod.rs b/backend/windmill-common/src/secret_backend/mod.rs index 71fa2ea999..596539cd08 100644 --- a/backend/windmill-common/src/secret_backend/mod.rs +++ b/backend/windmill-common/src/secret_backend/mod.rs @@ -13,6 +13,9 @@ //! vaults like HashiCorp Vault (Enterprise Edition). pub mod database; +pub mod resolver; + +pub use resolver::*; #[cfg(feature = "private")] pub mod vault_ee; diff --git a/backend/windmill-common/src/secret_backend/resolver.rs b/backend/windmill-common/src/secret_backend/resolver.rs new file mode 100644 index 0000000000..949300a3d6 --- /dev/null +++ b/backend/windmill-common/src/secret_backend/resolver.rs @@ -0,0 +1,324 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2024 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +//! Resolution of the configured secret backend. +//! +//! Lives in `windmill-common` (rather than the API/store crates) so that +//! lower-level helpers such as [`crate::variables::get_variable_or_self`] can +//! route secret reads through the configured backend. With an external backend +//! (Vault / Azure Key Vault / AWS Secrets Manager), the `variable.value` column +//! holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64 +//! ciphertext, so decrypting it directly fails — reads must go through the +//! backend instead. +//! +//! Note: external backends require Enterprise Edition. The OSS version only +//! supports the database backend. + +use std::sync::Arc; + +use crate::{ + db::DB, + error::{Error, Result}, + secret_backend::{database::DatabaseBackend, SecretBackend}, + variables::{build_crypt, decrypt}, +}; + +#[cfg(all(feature = "private", feature = "enterprise"))] +use crate::{ + global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, + secret_backend::{ + AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, + AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, + }, +}; + +#[cfg(all(feature = "private", feature = "enterprise"))] +use tokio::sync::RwLock; + +// Cached Vault backend to avoid recreating it for every request +// This enables connection pooling and avoids repeated setup overhead +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedVaultBackend { + backend: Arc, + settings: VaultSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedAzureKvBackend { + backend: Arc, + settings: AzureKeyVaultSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +// Cached AWS Secrets Manager backend +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedAwsSmBackend { + backend: Arc, + settings: AwsSecretsManagerSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +/// Get the current secret backend based on global settings +/// +/// OSS: Always returns DatabaseBackend +/// EE: Returns configured backend (Database or Vault) +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn get_secret_backend(db: &DB) -> Result> { + Ok(Arc::new(DatabaseBackend::new(db.clone()))) +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +pub async fn get_secret_backend(db: &DB) -> Result> { + let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { + Some(value) => serde_json::from_value::(value).unwrap_or_default(), + None => SecretBackendConfig::default(), + }; + + match config { + SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), + SecretBackendConfig::HashiCorpVault(settings) => { + get_or_create_vault_backend(db, settings).await + } + SecretBackendConfig::AzureKeyVault(settings) => { + get_or_create_azure_kv_backend(db, settings).await + } + SecretBackendConfig::AwsSecretsManager(settings) => { + get_or_create_aws_sm_backend(db, settings).await + } + } +} + +/// Get a cached Vault backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_vault_backend( + _db: &DB, + settings: VaultSettings, +) -> Result> { + // Check if we have a cached backend with matching settings (read lock) + { + let cache = VAULT_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + // Need to create a new backend - acquire write lock + let mut cache = VAULT_BACKEND_CACHE.write().await; + + // Double-check (another task may have created it while we waited) + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + // Create new backend + let backend: Arc = { + #[cfg(feature = "openidconnect")] + if settings.token.is_none() { + Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) + } else { + Arc::new(VaultBackend::new(settings.clone())) + } + + #[cfg(not(feature = "openidconnect"))] + Arc::new(VaultBackend::new(settings.clone())) + }; + + // Cache it + *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Get a cached Azure Key Vault backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_azure_kv_backend( + _db: &DB, + settings: AzureKeyVaultSettings, +) -> Result> { + // Check if we have a cached backend with matching settings (read lock) + { + let cache = AZURE_KV_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + // Need to create a new backend - acquire write lock + let mut cache = AZURE_KV_BACKEND_CACHE.write().await; + + // Double-check (another task may have created it while we waited) + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + // Create new backend + let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); + + // Cache it + *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Get a cached AWS SM backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_aws_sm_backend( + _db: &DB, + settings: AwsSecretsManagerSettings, +) -> Result> { + { + let cache = AWS_SM_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + let mut cache = AWS_SM_BACKEND_CACHE.write().await; + + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + let backend: Arc = + Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); + + *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Check if a Vault backend is currently configured +/// +/// OSS: Always returns false +/// EE: Checks global settings +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn is_vault_backend_configured(_db: &DB) -> Result { + Ok(false) +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +pub async fn is_vault_backend_configured(db: &DB) -> Result { + let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { + Some(value) => serde_json::from_value::(value).unwrap_or_default(), + None => SecretBackendConfig::default(), + }; + + Ok(matches!( + config, + SecretBackendConfig::HashiCorpVault(_) + | SecretBackendConfig::AzureKeyVault(_) + | SecretBackendConfig::AwsSecretsManager(_) + )) +} + +/// Get a secret value using the configured backend +/// +/// For database backend: decrypts `encrypted_value` using the workspace key +/// For external backends (EE only): fetches from the backend at `path`, +/// ignoring `encrypted_value` (which holds only a `$...:` marker) +pub async fn get_secret_value( + db: &DB, + workspace_id: &str, + path: &str, + encrypted_value: &str, +) -> Result { + let backend = get_secret_backend(db).await?; + + match backend.backend_name() { + "database" => { + // Use existing database decryption + let mc = build_crypt(db, workspace_id).await?; + decrypt(&mc, encrypted_value.to_string()).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) + }) + } + "hashicorp_vault" => { + // Fetch from Vault directly + backend.get_secret(workspace_id, path).await + } + "azure_key_vault" => backend.get_secret(workspace_id, path).await, + "aws_secrets_manager" => backend.get_secret(workspace_id, path).await, + _ => Err(Error::internal_err(format!( + "Unknown backend: {}", + backend.backend_name() + ))), + } +} + +/// Check if a value is stored in Vault (indicated by the $vault: prefix) +pub fn is_vault_stored_value(value: &str) -> bool { + value.starts_with("$vault:") +} + +/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) +pub fn is_azure_kv_stored_value(value: &str) -> bool { + value.starts_with("$azure_kv:") +} + +/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix) +pub fn is_aws_sm_stored_value(value: &str) -> bool { + value.starts_with("$aws_sm:") +} + +/// Check if a value is stored in any external secret backend +pub fn is_external_stored_value(value: &str) -> bool { + is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn external_markers_are_detected() { + assert!(is_external_stored_value("$vault:u/admin/secret")); + assert!(is_external_stored_value("$azure_kv:u/admin/secret")); + assert!(is_external_stored_value("$aws_sm:u/admin/secret")); + } + + #[test] + fn base64_ciphertext_is_not_treated_as_external() { + // A base64 magic_crypt blob must route through `decrypt`, never the + // external backend. The leading `$` is what distinguishes a marker from + // ciphertext; decrypting a marker fails with "Invalid byte 36" (`$`), + // which is the bug this gate prevents. + for v in [ + "bm90LWEtbWFya2Vy", + "AAAA1234+/abcd==", + "", + "$something_else", + ] { + assert!(!is_external_stored_value(v), "unexpected external: {v:?}"); + } + } +} diff --git a/backend/windmill-common/src/users.rs b/backend/windmill-common/src/users.rs index f91c21701a..d40b5cd19c 100644 --- a/backend/windmill-common/src/users.rs +++ b/backend/windmill-common/src/users.rs @@ -50,10 +50,10 @@ const EMAIL_CACHE_TTL_SECS: u64 = 60; /// - "u/{username}" → lookup email from usr table (cached) /// - "g/{group}" → "group-{group}@windmill.dev" /// - raw email → return as-is -pub async fn get_email_from_permissioned_as( +pub async fn get_email_from_permissioned_as<'c>( permissioned_as: &str, workspace_id: &str, - db: &sqlx::Pool, + db: impl sqlx::PgExecutor<'c>, ) -> crate::error::Result { if let Some(username) = permissioned_as.strip_prefix(PERMISSIONED_AS_USER_PREFIX) { let lookup = EmailCacheKey(workspace_id, username); diff --git a/backend/windmill-common/src/variables.rs b/backend/windmill-common/src/variables.rs index 0d42b95a5a..b43eb07e9c 100644 --- a/backend/windmill-common/src/variables.rs +++ b/backend/windmill-common/src/variables.rs @@ -9,6 +9,7 @@ use crate::db::{Authable, UserDB}; use crate::error::{self, Error}; use crate::scripts::ScriptHash; +use crate::secret_backend::{get_secret_value, is_external_stored_value}; use crate::utils::WarnAfterExt; use crate::worker::Connection; use crate::{worker::WORKER_GROUP, BASE_URL, DB}; @@ -234,13 +235,17 @@ pub async fn get_secret_value_as_admin( let r = if variable.is_secret { let value = variable.value; if !value.is_empty() { - let mc = build_crypt(db, w_id).await?; - decrypt(&mc, value).map_err(|e| { - crate::error::Error::internal_err(format!( - "Error decrypting variable {}: {}", - variable.path, e - )) - })? + if is_external_stored_value(&value) { + get_secret_value(db, w_id, &variable.path, &value).await? + } else { + let mc = build_crypt(db, w_id).await?; + decrypt(&mc, value).map_err(|e| { + crate::error::Error::internal_err(format!( + "Error decrypting variable {}: {}", + variable.path, e + )) + })? + } } else { "".to_string() } @@ -546,10 +551,14 @@ pub async fn get_variable_or_self( if let Some(record) = record { let mut value = record.value; if record.is_secret { - let mc = build_crypt(db, w_id).await?; - value = decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) - })?; + if is_external_stored_value(&value) { + value = get_secret_value(db, w_id, &path, &value).await?; + } else { + let mc = build_crypt(db, w_id).await?; + value = decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) + })?; + } } Ok(value) @@ -591,10 +600,14 @@ pub async fn get_variable_or_self_as( if let Some(record) = record { let mut value = record.value; if record.is_secret { - let mc = build_crypt(db, w_id).await?; - value = decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e)) - })?; + if is_external_stored_value(&value) { + value = get_secret_value(db, w_id, &var_path, &value).await?; + } else { + let mc = build_crypt(db, w_id).await?; + value = decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e)) + })?; + } } Ok(value) diff --git a/backend/windmill-common/src/worker.rs b/backend/windmill-common/src/worker.rs index e3bd7ef7ef..3a1469d5e3 100644 --- a/backend/windmill-common/src/worker.rs +++ b/backend/windmill-common/src/worker.rs @@ -693,8 +693,6 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error let full_path = job_dir.join(&user_path); - // let normalized_job_dir = std::fs::canonicalize(job_dir)?; - // let normalized_full_path = std::fs::canonicalize(&full_path)?; let normalized_job_dir = normalize_path(job_dir); let normalized_full_path = normalize_path(&full_path); @@ -706,6 +704,36 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error .into()); } + // The lexical check above cannot see symlinks: a symlink planted inside the + // job dir - e.g. by an earlier Ansible `git_repos` clone whose tracked + // content includes one - would let a later `git clone` or file write follow + // it out of the job dir while still passing the textual `starts_with` check. + // Walk the *normalized* relative path (`..`/`.` already collapsed) so each + // step matches the real on-disk resolution, and reject any existing component + // that is a symlink. Walking the raw user path would drift on an in-bounds + // `..` (e.g. `foo/../link`, which normalizes back inside the job dir) and miss + // the real symlinked component. Not-yet-existing components are safe: a path + // that does not exist cannot itself be a symlink. + let relative = normalized_full_path + .strip_prefix(&normalized_job_dir) + .unwrap_or(&normalized_full_path); + let mut current = normalized_job_dir.clone(); + for component in relative.components() { + if let Component::Normal(c) = component { + current.push(c); + if std::fs::symlink_metadata(¤t) + .map(|m| m.file_type().is_symlink()) + .unwrap_or(false) + { + return Err(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "Path traverses a symlink, which is not allowed.", + ) + .into()); + } + } + } + Ok(normalized_full_path) } @@ -2828,4 +2856,71 @@ mod tests { let _ = std::fs::remove_dir_all(&base); } + + #[test] + fn test_is_allowed_file_location_allows_plain_relative() { + let base = std::env::temp_dir().join(format!("wm_allowed_loc_ok_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + let out = is_allowed_file_location(job_dir_str, "repo/sub/playbook.yml").unwrap(); + assert_eq!(out, normalize_path(&job_dir.join("repo/sub/playbook.yml"))); + + let _ = std::fs::remove_dir_all(&base); + } + + #[test] + fn test_is_allowed_file_location_rejects_parent_and_absolute() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_esc_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + assert!(is_allowed_file_location(job_dir_str, "../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "a/../../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "/etc/passwd").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } + + // Regression for GHSA-v934-cvpf-6fjw: a symlink planted inside the job dir + // (e.g. by an earlier `git_repos` clone) must not let a later target traverse + // it out of the job dir, even though the lexical path stays "inside". + #[cfg(unix)] + #[test] + fn test_is_allowed_file_location_rejects_symlink_traversal() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_symlink_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + // Stand-in for the shared cache dir living outside the job dir. + let outside = base.join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + // Plant `job/repo` -> `../outside`, as a malicious first clone would. + let planted = job_dir.join("repo"); + std::os::unix::fs::symlink(&outside, &planted).unwrap(); + + // Both the symlink itself and any path traversing it are rejected. + assert!(is_allowed_file_location(job_dir_str, "repo").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/payload").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/sub/payload").is_err()); + + // An in-bounds `..` must not bypass the check: `foo/../repo/payload` + // normalizes back to `repo/payload` and still traverses the symlink. + assert!(is_allowed_file_location(job_dir_str, "foo/../repo/payload").is_err()); + std::fs::create_dir(job_dir.join("real")).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "real/../repo/payload").is_err()); + + // A dangling symlink (target does not exist yet) is still caught: + // `symlink_metadata` does not follow the link. + let dangling = job_dir.join("dangling"); + std::os::unix::fs::symlink(base.join("nonexistent"), &dangling).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "dangling/payload").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } } diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index 6a7ecb40a6..cebbf729be 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -8,6 +8,7 @@ use strum::AsRefStr; use crate::{ error::{self, to_anyhow, Error, Result}, get_database_url, + secret_backend::{get_secret_value, is_external_stored_value}, utils::get_custom_pg_instance_password, variables::{build_crypt, decrypt}, PgDatabase, DB, @@ -731,10 +732,14 @@ async fn transform_json_unchecked( .await .map_err(to_anyhow)?; let value = if is_secret { - let mc = build_crypt(&db, &w_id).await?; - decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", &s, e)) - })? + if is_external_stored_value(&value) { + get_secret_value(db, w_id, &s[5..], &value).await? + } else { + let mc = build_crypt(&db, &w_id).await?; + decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", &s, e)) + })? + } } else { value }; diff --git a/backend/windmill-common/tests/asset_schema_capture.rs b/backend/windmill-common/tests/asset_schema_capture.rs new file mode 100644 index 0000000000..f12ff3cc14 --- /dev/null +++ b/backend/windmill-common/tests/asset_schema_capture.rs @@ -0,0 +1,105 @@ +/*! + * Tests the schema-capture versioning contract (gap #2a): + * `record_asset_schema` inserts a new `materialized_asset_schema` version only + * when the captured column set changes, re-affirms the latest row in place when + * it doesn't, and `list_asset_schemas` returns the evolution history newest + * first. + */ + +use sqlx::{Pool, Postgres}; +use windmill_common::assets::AssetKind; +use windmill_common::materialization::{list_asset_schemas, record_asset_schema, SchemaColumn}; + +const WS: &str = "test-workspace"; +const PATH: &str = "analytics/orders"; + +fn col(name: &str, ty: &str) -> SchemaColumn { + SchemaColumn { name: name.to_string(), data_type: ty.to_string() } +} + +async fn record(db: &Pool, cols: &[SchemaColumn], snapshot_id: i64) -> bool { + let mut tx = db.begin().await.expect("begin"); + let inserted = record_asset_schema( + &mut tx, + WS, + AssetKind::Ducklake, + PATH, + cols, + Some(snapshot_id), + None, + ) + .await + .expect("record asset schema"); + tx.commit().await.expect("commit"); + inserted +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn first_capture_inserts_version_one(db: Pool) { + let cols = [col("order_id", "BIGINT"), col("status", "VARCHAR")]; + assert!( + record(&db, &cols, 10).await, + "first capture inserts a version" + ); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 1); + assert_eq!(versions[0].version, 1); + assert_eq!(versions[0].snapshot_id, Some(10)); + assert_eq!(versions[0].columns.0, cols); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn unchanged_schema_reaffirms_without_new_version(db: Pool) { + let cols = [col("order_id", "BIGINT")]; + record(&db, &cols, 10).await; + // Identical column set on a later snapshot: no new version, but the latest + // row's snapshot_id advances. + assert!( + !record(&db, &cols, 20).await, + "unchanged schema must not insert a new version" + ); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 1, "still a single version"); + assert_eq!(versions[0].version, 1); + assert_eq!(versions[0].snapshot_id, Some(20), "snapshot re-affirmed"); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn changed_schema_bumps_version_newest_first(db: Pool) { + record(&db, &[col("order_id", "BIGINT")], 10).await; + // A column added → schema changed → new version. + let evolved = [col("order_id", "BIGINT"), col("amount", "DOUBLE")]; + assert!(record(&db, &evolved, 20).await, "changed schema inserts v2"); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 2); + // Newest first. + assert_eq!(versions[0].version, 2); + assert_eq!(versions[0].columns.0, evolved); + assert_eq!(versions[1].version, 1); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn column_order_change_is_a_new_version(db: Pool) { + let a = [col("a", "BIGINT"), col("b", "VARCHAR")]; + let reordered = [col("b", "VARCHAR"), col("a", "BIGINT")]; + record(&db, &a, 10).await; + // Same columns, different physical order: the captured list is ordered, so a + // reorder is a real schema change (downstream `SELECT *` consumers see it). + assert!( + record(&db, &reordered, 20).await, + "column reorder is a distinct schema version" + ); + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 2); +} diff --git a/backend/windmill-mcp/src/common/scope.rs b/backend/windmill-mcp/src/common/scope.rs index 7da4e2cebc..f43095d740 100644 --- a/backend/windmill-mcp/src/common/scope.rs +++ b/backend/windmill-mcp/src/common/scope.rs @@ -38,6 +38,71 @@ impl McpScopeConfig { is_resource_allowed(path, patterns) } + + /// Directional subset check: does this config grant at least everything + /// `requested` grants? Used to enforce monotonic containment when an MCP + /// OAuth approval mints a token (the granted scopes must be within the + /// approving token's own scopes). + /// + /// Unlike `is_allowed` (which tests a single concrete path with OR + /// semantics), this requires every requested pattern to be covered by some + /// caller pattern — so `mcp:scripts:f/x` cannot widen into `mcp:scripts:*`. + pub fn contains(&self, requested: &McpScopeConfig) -> bool { + if self.all { + return true; + } + if requested.all { + return false; + } + if requested.favorites && !self.favorites { + return false; + } + if let Some(req_hub) = requested.hub_apps.as_ref() { + match self.hub_apps.as_ref() { + Some(caller_hub) => { + let caller_apps: std::collections::HashSet<&str> = + caller_hub.split(',').map(|s| s.trim()).collect(); + if !req_hub + .split(',') + .map(|s| s.trim()) + .all(|a| caller_apps.contains(a)) + { + return false; + } + } + None => return false, + } + } + resource_list_covers(&self.scripts, &requested.scripts) + && resource_list_covers(&self.flows, &requested.flows) + && resource_list_covers(&self.endpoints, &requested.endpoints) + } +} + +/// Every requested pattern must be covered by some caller pattern. +fn resource_list_covers(caller: &[String], requested: &[String]) -> bool { + requested + .iter() + .all(|req| caller.iter().any(|c| pattern_covers(c, req))) +} + +/// Directional: does the single caller pattern cover `requested`? `caller` may +/// be `*`, an exact path/name, or a `/*` subtree; `requested` may itself +/// be a subtree wildcard, in which case the whole requested subtree must fall +/// within the caller's. Mirrors the route-scope containment in windmill-api-auth. +fn pattern_covers(caller: &str, requested: &str) -> bool { + if caller == "*" || caller == requested { + return true; + } + // An exact caller pattern only covers itself (handled above); a wildcard + // requested can never be covered by a non-`*` exact caller. + let Some(prefix) = caller.strip_suffix("/*") else { + return false; + }; + let requested_base = requested.strip_suffix("/*").unwrap_or(requested); + requested_base == prefix + || (requested_base.starts_with(prefix) + && requested_base.as_bytes().get(prefix.len()) == Some(&b'/')) } /// Parse MCP scopes from token scope strings @@ -254,4 +319,51 @@ mod tests { assert!(config.is_allowed("flow", "f/automation/test")); assert!(!config.is_allowed("flow", "f/other/test")); } + + fn cfg(scopes: &[&str]) -> McpScopeConfig { + parse_mcp_scopes(&scopes.iter().map(|s| s.to_string()).collect::>()).unwrap() + } + + #[test] + fn test_contains_subset_and_widening() { + // mcp:all contains anything. + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:all"]))); + + // A wildcard caller covers narrower requests, but not other domains/all. + let star = cfg(&["mcp:scripts:*"]); + assert!(star.contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(star.contains(&cfg(&["mcp:scripts:*"]))); + assert!(!star.contains(&cfg(&["mcp:all"]))); + assert!(!star.contains(&cfg(&["mcp:flows:f/x"]))); + + // The core regression: a single-path caller must NOT widen into `*` or + // into another path. + let narrow = cfg(&["mcp:scripts:f/x"]); + assert!(narrow.contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(!narrow.contains(&cfg(&["mcp:scripts:*"]))); + assert!(!narrow.contains(&cfg(&["mcp:scripts:f/y"]))); + assert!(!narrow.contains(&cfg(&["mcp:all"]))); + + // Subtree wildcard covers paths within it but not a sibling subtree. + let subtree = cfg(&["mcp:scripts:f/team/*"]); + assert!(subtree.contains(&cfg(&["mcp:scripts:f/team/sub"]))); + assert!(subtree.contains(&cfg(&["mcp:scripts:f/team/sub/*"]))); + assert!(!subtree.contains(&cfg(&["mcp:scripts:f/other/x"]))); + } + + #[test] + fn test_contains_favorites_and_endpoints() { + assert!(cfg(&["mcp:favorites"]).contains(&cfg(&["mcp:favorites"]))); + // A caller without favorites cannot grant favorites. + assert!(!cfg(&["mcp:scripts:*"]).contains(&cfg(&["mcp:favorites"]))); + + // Endpoint names match exactly (or via `*`). + let ep = cfg(&["mcp:endpoints:getVariable"]); + assert!(ep.contains(&cfg(&["mcp:endpoints:getVariable"]))); + assert!(!ep.contains(&cfg(&["mcp:endpoints:getResource"]))); + assert!(!ep.contains(&cfg(&["mcp:all"]))); + // mcp:all grants all endpoints. + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:endpoints:getResource"]))); + } } diff --git a/backend/windmill-oauth/src/lib.rs b/backend/windmill-oauth/src/lib.rs index 2f28a6d4c7..e314f13d75 100644 --- a/backend/windmill-oauth/src/lib.rs +++ b/backend/windmill-oauth/src/lib.rs @@ -452,12 +452,12 @@ pub async fn build_client_credentials_oauth_client( let caller_supplied_creds = !client_id.is_empty() && !client_secret.is_empty(); - // Apply the server-resolved concrete token URL. Instance-templated providers - // (e.g. Coupa) carry an empty or `{instance}`-templated token URL in their - // registry config; the resolved value (host-pinned for bring-your-own, - // persisted on the row for refresh) is what completes it. The caller never - // supplies a free-form token URL: this value always comes from - // `resolve_cc_token_url_input` or a previously-resolved persisted URL. + // Apply the resolved concrete token URL. Instance-templated providers (e.g. + // Coupa) carry an empty or `{instance}`-templated token URL in their registry + // config; the resolved value (host-pinned for instance-name connections, + // persisted on the row for refresh) is what completes it. For bring-your-own + // connections this value may instead be a caller-supplied override — safe + // because only the caller's own credentials are ever sent to it. if let Some(url) = resolved_token_url { connect_config.token_url = url.to_string(); } @@ -652,6 +652,28 @@ pub fn resolve_cc_token_url_input( Ok(template.replace("{instance}", value)) } +/// Whether a built-in provider's client-credentials token URL is host-pinned via +/// an `{instance}` template (e.g. servicenow, snowflake, coupa). Such providers +/// only accept an instance name substituted into a fixed-host template, so a +/// free-form caller token URL override must be rejected for them — otherwise the +/// exchange host could be redirected, which is exactly what the template pins. +/// Fixed-host registry providers and custom (non-registry) providers return +/// `false`: an override is allowed there. +pub fn is_instance_templated_cc(connect_configs_json: &str, client_name: &str) -> bool { + serde_json::from_str::>(connect_configs_json) + .ok() + .and_then(|m| resolve_registry_config(&m, client_name)) + .map(|cfg| { + cfg.connect_config_template + .as_ref() + .map(|t| t.token_url.clone()) + .filter(|u| !u.is_empty()) + .unwrap_or(cfg.token_url) + .contains("{instance}") + }) + .unwrap_or(false) +} + /// Resolve the concrete bring-your-own client-credentials token URL for any /// provider, never from a caller-supplied URL: /// - **Built-in registry providers** resolve from the registry via @@ -1350,4 +1372,20 @@ mod tests { assert!(resolve_cc_token_url_input(CC_REGISTRY, "bad_host_tpl", Some("evil.com")).is_err()); assert!(resolve_cc_token_url_input(CC_REGISTRY, "bad_mid_tpl", Some("evil")).is_err()); } + + #[test] + fn instance_templated_cc_true_for_templated_providers() { + // Host-pinned via `{instance}`: a bring-your-own token URL override must be + // refused for these (only the instance-name path may set their URL). + assert!(is_instance_templated_cc(CC_REGISTRY, "coupa")); + assert!(is_instance_templated_cc(CC_REGISTRY, "servicenow")); + } + + #[test] + fn instance_templated_cc_false_for_fixed_host_and_unknown() { + // Fixed-host registry provider and custom (non-registry) provider both allow + // an override, so neither is reported as instance-templated. + assert!(!is_instance_templated_cc(CC_REGISTRY, "visma")); + assert!(!is_instance_templated_cc(CC_REGISTRY, "my_custom_thing")); + } } diff --git a/backend/windmill-queue/Cargo.toml b/backend/windmill-queue/Cargo.toml index ac956cb709..b9bd6fd28e 100644 --- a/backend/windmill-queue/Cargo.toml +++ b/backend/windmill-queue/Cargo.toml @@ -17,10 +17,14 @@ benchmark = ["windmill-common/benchmark"] failpoints = [] prometheus = ["dep:prometheus"] smtp = [] +# Enables native evaluation of `retry_if` expressions on the job-failure path. +# Without it, a `retry_if` gate cannot be evaluated and the job does not retry. +quickjs = ["dep:windmill-jseval", "windmill-jseval/quickjs"] [dependencies] windmill-audit.workspace = true windmill-common = { workspace = true, default-features = false } +windmill-jseval = { workspace = true, optional = true } anyhow.workspace = true hmac.workspace = true sql-builder.workspace = true diff --git a/backend/windmill-queue/src/asset_dispatch.rs b/backend/windmill-queue/src/asset_dispatch.rs index 7647ffc864..753614d07b 100644 --- a/backend/windmill-queue/src/asset_dispatch.rs +++ b/backend/windmill-queue/src/asset_dispatch.rs @@ -226,6 +226,15 @@ async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result if !is_eligible_kind(job) { return Ok(DispatchResult::default()); } + // A parented script is dispatch-eligible only as a native retry attempt — a + // re-run of the SAME runnable as its chain parent. Schedule/error/recovery + // handlers are also parented `Script` children but run a DIFFERENT script; + // excluding them stops a handler that happens to declare assets from + // triggering a cascade (the pre-native-retry `parent_job IS NULL` guard + // excluded every parented child). + if job.parent_job.is_some() && !is_native_retry_attempt(db, job).await? { + return Ok(DispatchResult::default()); + } let runnable_path = match job.runnable_path.as_deref() { Some(p) if !p.is_empty() => p, _ => return Ok(DispatchResult::default()), @@ -406,12 +415,28 @@ fn is_eligible_kind(job: &MiniCompletedJob) -> bool { if !matches!(job.kind, JobKind::Script | JobKind::Preview) { return false; } - if job.parent_job.is_some() || job.flow_step_id.is_some() { + // Flow steps (and sub-flow jobs) carry `flow_step_id` and are ineligible. + // Native script-retry attempts carry `parent_job` (the chain root) but no + // `flow_step_id`; whether a parented job is actually a retry attempt (vs a + // schedule/error handler child) is decided in `try_dispatch`. + if job.flow_step_id.is_some() { return false; } true } +// Native retry attempts carry an explicit `native_retry_attempt` marker; no +// other parented `Script` child (schedule handlers, WAC inline children, flow +// steps) does. One indexed point lookup, only for parented jobs. +async fn is_native_retry_attempt(db: &DB, job: &MiniCompletedJob) -> Result { + Ok(sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"", + job.id, + ) + .fetch_one(db) + .await?) +} + async fn fetch_args( db: &Pool, workspace_id: &str, @@ -618,14 +643,16 @@ async fn push_subscriber( script.runnable_settings.debouncing_settings, ); - // Retry is only available via the flow runtime — wrap the script in a - // one-step flow when the cascade declares one. No retry = - // unwrapped `ScriptHash` push. + // When the cascade declares a retry, hand `push` a one-step-flow request + // carrying the policy + `language`; `push` materializes it into a native + // retryable `Script` (not a flow), so a failed/recovered subscriber stays + // eligible to trigger its own downstream. No retry = plain `ScriptHash`. let payload = if let Some(retry) = crate::cascade::cascade_retry(retry_count, retry_delay_s) { JobPayload::SingleStepFlow { path: subscriber_path.to_string(), hash: Some(hash), flow_version: None, + language: Some(script.language), args: HashMap::new(), retry: Some(retry), error_handler_path: None, diff --git a/backend/windmill-queue/src/jobs.rs b/backend/windmill-queue/src/jobs.rs index 5cb8911c7e..c92cea9ef3 100644 --- a/backend/windmill-queue/src/jobs.rs +++ b/backend/windmill-queue/src/jobs.rs @@ -45,8 +45,8 @@ use windmill_common::min_version::{ MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2, }; use windmill_common::runnable_settings::{ - ConcurrencySettings, ConcurrencySettingsWithCustom, DebouncingSettings, RunnableSettings, - RunnableSettingsTrait, + ConcurrencySettings, ConcurrencySettingsWithCustom, DebouncingSettings, RetrySettings, + RunnableSettings, RunnableSettingsTrait, }; use windmill_common::triggers::TriggerMetadata; use windmill_common::utils::{calculate_hash, configure_client, now_from_db}; @@ -68,7 +68,7 @@ use windmill_common::{ }, flows::{ add_virtual_items_if_necessary, FlowModule, FlowModuleValue, FlowValue, InputTransform, - StopAfterIf, + Retry, StopAfterIf, }, jobs::{get_payload_tag_from_prefixed_path, JobKind, JobPayload, QueuedJob, RawCode}, min_version::{MIN_VERSION_IS_AT_LEAST_1_432, MIN_VERSION_IS_AT_LEAST_1_440}, @@ -958,6 +958,28 @@ pub async fn add_completed_job( )); } + // Native script retry: a failed `Script` job that carries a retry policy and + // has attempts left gets its next attempt enqueued here — before the queue + // row (which holds the attempt counter) is removed by commit. The failed + // attempt is still recorded as a completed job below. `maybe_enqueue_…` + // self-guards on kind/cancellation/policy, so the success path is unaffected. + let retry_pending = if !success && !skipped && !from_cache { + // Serialized lazily, and only when a `retry_if` policy actually needs it. + let result_fn = || serde_json::value::to_raw_value(&result).ok(); + match maybe_enqueue_native_script_retry(db, completed_job, &canceled_by, &result_fn).await { + Ok(enqueued) => enqueued, + Err(e) => { + tracing::error!( + "native retry enqueue failed for {}: {e:#}", + completed_job.id + ); + false + } + } + } else { + false + }; + let result_columns = result_columns.as_ref(); let (opt_uuid, duration, _skip_downstream_error_handlers, wac_job_ids) = (|| { commit_completed_job( @@ -972,6 +994,7 @@ pub async fn add_completed_job( flow_is_done, duration, from_cache, + retry_pending, ) .warn_after_seconds(10) }) @@ -1031,6 +1054,9 @@ async fn commit_completed_job( flow_is_done: bool, duration: Option, from_cache: bool, + // True when a native script retry was enqueued for this failed attempt, i.e. + // this is not the terminal attempt — schedule completion handlers must wait. + retry_pending: bool, ) -> windmill_common::error::Result<(Option, i64, bool, Option)> { // let start = std::time::Instant::now(); @@ -1050,6 +1076,19 @@ async fn commit_completed_job( return value; } + // Resolve the concurrency-limit settings on the pool *before* opening the + // completion transaction: doing it inside the tx would hold a second + // simultaneous connection from the small per-worker pool. + let has_concurrent_limit = completed_job.concurrent_limit.is_some() + || windmill_common::runnable_settings::prefetch_cached_from_handle( + completed_job.runnable_settings_handle, + db, + ) + .await? + .1 + .concurrent_limit + .is_some(); + let mut tx = db.begin().warn_after_seconds(10).await?; let duration = sqlx::query_scalar!( @@ -1258,11 +1297,17 @@ async fn commit_completed_job( // for flows, only try to schedule next tick here if flow failed and because first handle_flow failed (step = 0, modules[0] = {type: 'Failure', 'job': uuid::nil()}) // or job was cancelled before first handle_flow was called (step = 0, modules = [] OR modules[0].type == 'WaitingForPriorSteps') // otherwise flow rescheduling is done inside handle_flow - let schedule_next_tick = !completed_job.is_flow() - || from_cache - || !success - && sqlx::query_scalar!( - "SELECT + // Native retry attempts carry the schedule trigger (so the + // terminal attempt can drive handlers) but `parent_job` is set — + // they must not each push the next cron tick (the root already + // did), so gate next-tick on a top-level (`parent_job IS NULL`) + // occurrence. + let schedule_next_tick = completed_job.parent_job.is_none() + && (!completed_job.is_flow() + || from_cache + || !success + && sqlx::query_scalar!( + "SELECT flow_status->>'step' = '0' AND ( jsonb_array_length(flow_status->'modules') = 0 @@ -1273,15 +1318,15 @@ async fn commit_completed_job( ) ) FROM v2_job_completed WHERE id = $2 AND workspace_id = $3", - Uuid::nil().to_string(), - &completed_job.id, - &completed_job.workspace_id - ) - .fetch_optional(&mut *tx) - .warn_after_seconds(10) - .await? - .flatten() - .unwrap_or(false); + Uuid::nil().to_string(), + &completed_job.id, + &completed_job.workspace_id + ) + .fetch_optional(&mut *tx) + .warn_after_seconds(10) + .await? + .flatten() + .unwrap_or(false)); if schedule_next_tick { let (returned_tx, schedule_push_err) = @@ -1292,42 +1337,55 @@ async fn commit_completed_job( } } + // Defer schedule completion handlers (on_failure/on_success/ + // on_recovery) while a native retry is pending: only the terminal + // attempt should drive them. apply_schedule_handlers resolves + // per-occurrence failure/recovery status across the whole retry + // chain, so multi-count/exact handler policies work even though + // each attempt is its own completed job. #[cfg(all(feature = "enterprise", feature = "private"))] - if let Err(err) = crate::jobs_ee::apply_schedule_handlers( - db, - &schedule, - &script_path, - &completed_job.workspace_id, - success, - result, - job_id, - completed_job.started_at.unwrap_or(chrono::Utc::now()), - completed_job.priority, - ) - .warn_after_seconds(10) - .await - { - if !success { - tracing::error!("Could not apply schedule error handler: {}", err); - let base_url = windmill_common::BASE_URL.load(); - let w_id: &String = &completed_job.workspace_id; - if !matches!(err, Error::QuotaExceeded(_)) { - report_error_to_workspace_handler_or_critical_side_channel( - &completed_job, - db, - format!( - "Failed to push schedule error handler job to handle failed job ({base_url}/run/{}?workspace={w_id}): {}", - completed_job.id, - err - ), - ) - .warn_after_seconds(10) - .await; + if !retry_pending { + if let Err(err) = crate::jobs_ee::apply_schedule_handlers( + db, + &schedule, + &script_path, + &completed_job.workspace_id, + success, + result, + job_id, + // Current occurrence's root: the terminal native-retry + // attempt's parent, else the job itself. + completed_job.parent_job.unwrap_or(job_id), + completed_job.started_at.unwrap_or(chrono::Utc::now()), + completed_job.priority, + ) + .warn_after_seconds(10) + .await + { + if !success { + tracing::error!("Could not apply schedule error handler: {}", err); + let base_url = windmill_common::BASE_URL.load(); + let w_id: &String = &completed_job.workspace_id; + if !matches!(err, Error::QuotaExceeded(_)) { + report_error_to_workspace_handler_or_critical_side_channel( + &completed_job, + db, + format!( + "Failed to push schedule error handler job to handle failed job ({base_url}/run/{}?workspace={w_id}): {}", + completed_job.id, + err + ), + ) + .warn_after_seconds(10) + .await; + } + } else { + tracing::error!("Could not apply schedule recovery handler: {}", err); } - } else { - tracing::error!("Could not apply schedule recovery handler: {}", err); - } - }; + }; + } + #[cfg(not(all(feature = "enterprise", feature = "private")))] + let _ = retry_pending; } else { tracing::error!( "Schedule {schedule_path} in {} not found. Impossible to schedule again and apply schedule handlers", @@ -1337,16 +1395,7 @@ async fn commit_completed_job( } } - if completed_job.concurrent_limit.is_some() - || windmill_common::runnable_settings::prefetch_cached_from_handle( - completed_job.runnable_settings_handle, - db, - ) - .await? - .1 - .concurrent_limit - .is_some() - { + if has_concurrent_limit { let concurrency_key = sqlx::query_scalar!( "SELECT key FROM concurrency_key WHERE job_id = $1", &completed_job.id @@ -1589,6 +1638,267 @@ async fn restart_job_if_perpetual_inner( Ok(()) } +/// Evaluate a `retry_if` JS expression. `result`/`previous_result` are the +/// failure output and `flow_input` the job args. Defaults to retrying on eval +/// error (an unevaluable gate shouldn't silently swallow retries). +#[cfg(feature = "quickjs")] +async fn eval_retry_if( + expr: &str, + result: Option<&serde_json::value::RawValue>, + args: &HashMap>, +) -> bool { + let result_val = result + .and_then(|r| serde_json::from_str::(r.get()).ok()) + .unwrap_or(serde_json::Value::Null); + let mut globals = HashMap::new(); + globals.insert("result".to_string(), result_val.clone()); + globals.insert("previous_result".to_string(), result_val); + globals.insert( + "flow_input".to_string(), + serde_json::to_value(args).unwrap_or(serde_json::Value::Null), + ); + match windmill_jseval::eval_simple_js(format!("Boolean({expr})"), globals).await { + Ok(v) => v.get() == "true", + Err(e) => { + tracing::warn!("Failed to evaluate retry_if expression, retrying anyway: {e:#}"); + true + } + } +} + +/// `retry_if` is unsupported on a worker built without the `quickjs` feature +/// (the expression cannot be evaluated). Such a worker can't run JS jobs either, +/// so this is not reached in practice; we fail closed and do not retry. +#[cfg(not(feature = "quickjs"))] +async fn eval_retry_if( + _expr: &str, + _result: Option<&serde_json::value::RawValue>, + _args: &HashMap>, +) -> bool { + tracing::warn!("retry_if is unsupported without the quickjs feature; not retrying"); + false +} + +/// Native script retry. When a failed `Script` job carries a retry policy (via +/// `runnable_settings_handle`) and has attempts left, enqueue a fresh attempt of +/// the same script after the policy's backoff delay — instead of having wrapped +/// it in a one-step flow. Each attempt is a real `Script` job; the attempt +/// counter lives in the `native_retry_attempt` marker, written here and read only +/// on the next failure (never on the hot job-pull path). +/// +/// Returns `true` if a retry was enqueued. +/// +/// Authorization: this performs no auth check by design. It is `pub` only so the +/// integration test can reach it; the sole production caller is the worker +/// job-completion path (`add_completed_job`), which passes a `MiniCompletedJob` +/// built from a real, already-persisted completed job — its workspace/identity +/// fields come from the DB, not from request input. Callers MUST uphold this: +/// never invoke it with caller-supplied or unauthorized job identity. +pub async fn maybe_enqueue_native_script_retry( + db: &Pool, + job: &MiniCompletedJob, + canceled_by: &Option, + // Lazily serialize the failure result: only `retry_if` policies need it, so + // the common (no-retry_if) failure never pays the serialization cost. + result_fn: &(dyn Fn() -> Option> + Sync), +) -> Result { + // Only plain top-level scripts retry natively; cancellation always wins. + if canceled_by.is_some() || !matches!(job.kind, JobKind::Script) || job.is_flow_step() { + return Ok(false); + } + + let Some(retry_settings) = windmill_common::runnable_settings::prefetch_retry_from_handle( + job.runnable_settings_handle, + db, + ) + .await? + else { + return Ok(false); + }; + let policy: Retry = retry_settings.into(); + if !policy.has_attempts() { + return Ok(false); + } + + // Attempt counter for this job: its `native_retry_attempt` marker, or 0 for + // the first (un-marked) attempt. The marker is persistent (unlike the queue + // row), so it doubles as the explicit "this job is a retry attempt" signal + // consumers key off of. + let prev_attempts = sqlx::query_scalar!( + "SELECT attempt FROM native_retry_attempt WHERE job_id = $1", + job.id, + ) + .fetch_optional(db) + .await? + .unwrap_or(0) as u32; + let root = job.parent_job.unwrap_or(job.id); + // Scheduled chains keep the schedule trigger so the terminal attempt drives + // the schedule completion handlers (on_failure/on_success); `parent_job` + // keeps every retry out of the per-occurrence handler counting queries. + let trigger = job + .schedule_path() + .map(|sp| TriggerMetadata::new(Some(sp), JobTriggerKind::Schedule)); + + let Some(delay) = policy.interval(prev_attempts, false) else { + // Attempts exhausted — let the failure finalize normally. + return Ok(false); + }; + // Cap the backoff to match the flow-runtime retry path (evaluate_retry). + let delay = std::cmp::min(delay, MAX_RETRY_INTERVAL); + let scheduled_for = chrono::Utc::now() + + chrono::Duration::from_std(delay).unwrap_or_else(|_| chrono::Duration::zero()); + + let args = sqlx::query_scalar!( + "SELECT args as \"args: sqlx::types::Json>>\" FROM v2_job WHERE id = $1 AND workspace_id = $2", + job.id, + job.workspace_id, + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or_default(); + + // Optional `retry_if`: gate the retry on a JS expression over the failure + // `result` and `flow_input` (the job args). Evaluated by `eval_retry_if`, + // which on a worker built without the `quickjs` feature cannot evaluate the + // expression and fails closed (no retry). + if let Some(retry_if) = policy.retry_if.as_ref() { + let result = result_fn(); + if !eval_retry_if(&retry_if.expr, result.as_deref(), &args.0).await { + return Ok(false); + } + } + + // Re-push as a one-step-flow request; `push` materializes it back into a + // native retryable `Script` carrying the policy again. `parent_job = root` + // links the chain (and excludes retries from schedule-handler counting); the + // schedule trigger, when present, lets the terminal attempt fire handlers. + // + // Idempotent retry id: deterministic per (root, next attempt). If a worker + // dies between this push and the current attempt's finalization, the reaper + // re-handles the un-finalized attempt and we land here again with the same + // id, so the retry is enqueued exactly once (no double-retry). + let retry_job_id = { + use std::hash::{Hash, Hasher}; + let next_attempt = prev_attempts + 1; + let mut high = std::hash::DefaultHasher::new(); + root.hash(&mut high); + next_attempt.hash(&mut high); + let mut low = std::hash::DefaultHasher::new(); + "native-retry".hash(&mut low); + next_attempt.hash(&mut low); + root.hash(&mut low); + Uuid::from_u64_pair(high.finish(), low.finish()) + }; + // If that retry already exists (the crash-and-reaper-replay case above), + // report it as pending WITHOUT re-pushing. Deriving `retry_pending` from the + // push *result* would otherwise flip to false on the duplicate-id error and + // let the schedule completion handlers fire for this non-terminal attempt. + if sqlx::query_scalar!("SELECT 1 FROM v2_job WHERE id = $1", retry_job_id) + .fetch_optional(db) + .await? + .is_some() + { + return Ok(true); + } + // Carry forward the failed attempt's concurrency/debouncing settings (same + // runnable_settings_handle as the retry policy) so a retry of a concurrency- + // limited script still inserts its concurrency_key and respects the limit, + // rather than running unbounded with only the retry policy. + let (debouncing_settings, concurrency_settings) = + windmill_common::runnable_settings::prefetch_cached_from_handle( + job.runnable_settings_handle, + db, + ) + .await?; + let tx = PushIsolationLevel::IsolatedRoot(db.clone()); + let (new_id, mut tx) = match push( + db, + tx, + &job.workspace_id, + JobPayload::SingleStepFlow { + path: job.runnable_path.clone().unwrap_or_default(), + hash: job.runnable_id, + flow_version: None, + language: job.script_lang.clone(), + args: HashMap::new(), + retry: Some(policy), + error_handler_path: None, + error_handler_args: None, + skip_handler: None, + cache_ttl: job.cache_ttl, + cache_ignore_s3_path: job.cache_ignore_s3_path, + priority: job.priority, + tag_override: Some(job.tag.clone()), + trigger_path: None, + apply_preprocessor: false, + concurrency_settings, + debouncing_settings, + }, + PushArgs::from(&args.0), + &job.created_by, + &job.permissioned_as_email, + job.permissioned_as.clone(), + Some(&format!("retry.{}", job.id)), + Some(scheduled_for), + None, + Some(root), + None, + None, + Some(retry_job_id), + false, + false, + None, + true, + Some(job.tag.clone()), + None, + None, + job.priority, + None, + false, + None, + trigger, + None, + ) + .await + { + Ok(v) => v, + Err(e) => { + // Race with a concurrent completion of the same attempt: it may have + // inserted the deterministic retry id between our pre-check and this + // push, so the push fails on the duplicate id. The retry IS pending — + // re-check and report it as such instead of propagating the error + // (which would flip `retry_pending` to false and fire the schedule + // completion handlers for this non-terminal attempt). + if sqlx::query_scalar!("SELECT 1 FROM v2_job WHERE id = $1", retry_job_id) + .fetch_optional(db) + .await? + .is_some() + { + return Ok(true); + } + return Err(e); + } + }; + + sqlx::query!( + "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)", + new_id, + (prev_attempts + 1) as i32, + ) + .execute(&mut *tx) + .await?; + + tx.commit().await?; + tracing::info!( + "Native retry: enqueued attempt {} of script {:?} (root {root}) in {}s as {new_id}", + prev_attempts + 1, + job.runnable_path, + delay.as_secs(), + ); + Ok(true) +} + #[cfg(feature = "cloud")] fn apply_completed_job_cloud_usage( db: &Pool, @@ -1995,7 +2305,7 @@ pub async fn try_schedule_next_job<'c>( let email = match windmill_common::users::get_email_from_permissioned_as( &permissioned_as, &job.workspace_id, - db, + &mut *tx, ) .await { @@ -3172,108 +3482,189 @@ impl PulledJobResult { if let Some(args) = &mut j.args { args.remove(field_name); } + + // No accumulation on this path: just clean up the batch rows. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j_id, + ) + .execute(db) + .await?; } else if let Some(arg_name_to_accumulate) = // TODO: Maybe support multiple arguments in future debounce_args_to_accumulate.as_ref().and_then(|v| v.get(0)) { - tracing::debug!( - job_id = %j_id, - job_kind = ?kind, - arg_name = arg_name_to_accumulate, - "Accumulating debounced arguments from batch" - ); - let mut accumulated_arg: Vec> = vec![]; - for str_o in sqlx::query_scalar!( - "WITH ids AS ( - SELECT id as job_id FROM v2_job_debounce_batch WHERE debounce_batch = ( - SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 - ) - ) SELECT args->>$2 FROM ids LEFT JOIN v2_job ON v2_job.id = ids.job_id + // Claim this job's contribution to its debounce batch exactly once. + // Instead of deleting the batch rows, mark them consumed (stamping + // consumed_by = this job). A batch normally has a single survivor that + // sweeps every row; only a narrow push/pull race can leave two survivors + // on one batch. The claim lets the second survivor tell apart: + // - already swept in by the other survivor -> run empty (no duplicate), + // - its own earlier claim on re-pull -> keep its accumulated args, + // - never batched (CE / workers behind v2) -> keep its own args. + // Consumed rows are GC'd by the monitor. + // Claim + accumulate + persist atomically: a crash between stamping the + // batch rows consumed_by=self and persisting the merged args would + // otherwise let a zombie re-pull see its own prior claim and keep only + // its own args (dropping the siblings it had claimed). One transaction + // makes the claim and the merged-args write commit together (or neither). + let mut tx = db.begin().await?; + // Emitted AFTER the transaction commits — writing logs via a second pool + // connection while the claim tx + row locks are held risks pool-exhaustion + // stalls under concurrent debounced pulls. + let mut accumulation_log: Option = None; + let claim = sqlx::query!( + "WITH mine AS ( + SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1 + ), claimed AS ( + -- Claim the whole batch in ONE update so concurrent same-batch + -- survivors lock rows in identical scan order (no lock-ordering + -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual + -- and skips rows the other already took. A claim therefore consumes + -- every still-unclaimed row of the batch atomically. + UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1 + WHERE debounce_batch = (SELECT debounce_batch FROM mine) + AND consumed_at IS NULL + RETURNING id + ) + SELECT + EXISTS (SELECT 1 FROM mine) AS \"had_row!\", + (SELECT consumed_by FROM mine) AS prev_consumed_by, + ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\", + EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\" ", j_id, - arg_name_to_accumulate, ) - .fetch_all(db) - .await? - .into_iter() - { - if let Some(s) = str_o.as_ref() { - match serde_json::from_str::>>(s) { - Ok(ref mut vec) => accumulated_arg.append(vec), - Err(_) => { - // Value is not an array — wrap the scalar into a - // single-element array. This supports union types - // like T | T[] where the caller may pass a bare T. - match RawValue::from_string(s.to_string()) { - Ok(raw) => accumulated_arg.push(raw), - Err(e) => { - return Err(error::Error::ArgumentErr(format!("cannot consolidate argument `{arg_name_to_accumulate}`: value is neither a valid list nor a valid JSON value\nUnwrapped Error: {e}"))); + .fetch_one(&mut *tx) + .await?; + + if !claim.had_row { + // Never batched (CE / workers behind v2): keep the job's own args. + tracing::debug!( + job_id = %j_id, + "Debounce: no batch row, keeping original args" + ); + } else if claim.claimed_self { + // We claimed our own row; since a claim takes the whole batch, this also + // swept any not-yet-claimed siblings. Accumulate exactly the rows we own. + let ids = claim.claimed_ids; + + tracing::debug!( + job_id = %j_id, + job_kind = ?kind, + arg_name = arg_name_to_accumulate, + claimed = ids.len(), + "Accumulating debounced arguments from claimed batch rows" + ); + + let mut accumulated_arg: Vec> = vec![]; + for str_o in sqlx::query_scalar!( + "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)", + &ids, + arg_name_to_accumulate, + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + { + if let Some(s) = str_o.as_ref() { + match serde_json::from_str::>>(s) { + Ok(ref mut vec) => accumulated_arg.append(vec), + Err(_) => { + // Value is not an array — wrap the scalar into a + // single-element array. This supports union types + // like T | T[] where the caller may pass a bare T. + match RawValue::from_string(s.to_string()) { + Ok(raw) => accumulated_arg.push(raw), + Err(e) => { + return Err(error::Error::ArgumentErr(format!("cannot consolidate argument `{arg_name_to_accumulate}`: value is neither a valid list nor a valid JSON value\nUnwrapped Error: {e}"))); + } } } } } } - } - tracing::debug!( - job_id = %j_id, - arg_name = arg_name_to_accumulate, - accumulated_count = accumulated_arg.len(), - "Accumulated arguments from debounced jobs in batch" - ); + if !accumulated_arg.is_empty() { + let new_value = to_raw_value(&accumulated_arg); - // If the batch query returned no entries (e.g. CE where - // v2_job_debounce_batch is never populated), keep the - // original value unchanged instead of replacing it with []. - if !accumulated_arg.is_empty() { - let new_value = to_raw_value(&accumulated_arg); + let original_value = j + .args + .as_ref() + .and_then(|a| a.get(arg_name_to_accumulate)) + .map(|v| v.get().to_string()) + .unwrap_or_else(|| "null".to_string()); - let original_value = j - .args - .as_ref() - .and_then(|a| a.get(arg_name_to_accumulate)) - .map(|v| v.get().to_string()) - .unwrap_or_else(|| "null".to_string()); - - append_logs( - &j_id, - &j.workspace_id, - format!( + accumulation_log = Some(format!( "Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n", &new_value - ), - &(db.into()), - ) - .await; + )); + j.args + .get_or_insert(Json(Default::default())) + .as_mut() + .insert(arg_name_to_accumulate.to_owned(), new_value); + + // Persist accumulated args to v2_job so that flow steps + // re-reading from the DB (via get_mini_pulled_job) see them + if let Some(ref args) = j.args { + sqlx::query!( + "UPDATE v2_job SET args = $2 WHERE id = $1", + j_id, + args as &Json>>, + ) + .execute(&mut *tx) + .await?; + } + } + } else if claim.prev_consumed_by == Some(j_id) { + // Our own prior claim seen again on a re-pull (e.g. crash recovery): + // keep the args we already persisted on the first pull. + } else { + // Another survivor already accumulated this job's contribution + // (consumed_by a different job); run as a no-op so its items are not + // reprocessed. + tracing::info!( + job_id = %j_id, + arg_name = arg_name_to_accumulate, + "Debounce: contribution already consumed by a concurrent survivor, running empty" + ); j.args .get_or_insert(Json(Default::default())) .as_mut() - .insert(arg_name_to_accumulate.to_owned(), new_value); - - // Persist accumulated args to v2_job so that flow steps - // re-reading from the DB (via get_mini_pulled_job) see them + .insert( + arg_name_to_accumulate.to_owned(), + to_raw_value(&Vec::>::new()), + ); if let Some(ref args) = j.args { sqlx::query!( "UPDATE v2_job SET args = $2 WHERE id = $1", j_id, args as &Json>>, ) - .execute(db) + .execute(&mut *tx) .await?; } } - } + tx.commit().await?; - // Clean up the debounce batch entries now that the job has been pulled - sqlx::query!( - "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( - SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 - )", - j_id, - ) - .execute(db) - .await?; + if let Some(msg) = accumulation_log { + append_logs(&j_id, &j.workspace_id, msg, &(db.into())).await; + } + } else { + // Debounced but no args to accumulate (plain debounce / dependency job): + // consume the batch by removing this job's rows. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j_id, + ) + .execute(db) + .await?; + } // Handle dependency job debouncing cleanup when a job is pulled for execution if is_djob_to_debounce { @@ -4926,6 +5317,7 @@ async fn push_inner<'c, 'd>( _low_level_priority: Option, concurrency_settings: ConcurrencySettings, debouncing_settings: DebouncingSettings, + retry_settings: RetrySettings, labels: Option>, } let mut preprocessed = None; @@ -4944,6 +5336,7 @@ async fn push_inner<'c, 'd>( _low_level_priority, mut concurrency_settings, debouncing_settings, + retry_settings, labels, } = match job_payload { JobPayload::ScriptHash { @@ -5250,6 +5643,7 @@ async fn push_inner<'c, 'd>( path, hash, flow_version, + language, retry, error_handler_path, error_handler_args, @@ -5263,10 +5657,71 @@ async fn push_inner<'c, 'd>( apply_preprocessor, debouncing_settings, concurrency_settings, - } => { + } => 'ssf: { // Determine if this is a flow or a script let is_flow = flow_version.is_some(); + // Native retry: a bare script wrapped only to gain a retry (no + // skip/error-handler modules) is pushed as a real `Script` job that + // carries the policy in `runnable_settings`. This avoids spawning a + // one-step flow — and its extra job rows, flow_status, and UI + // projection — for the common schedule/pipeline retry case. The flow + // path below is kept only for handler-bearing or flow-wrapping cases. + // + // Gated on the runnable-settings min version: on a mixed-version + // fleet the policy can't be persisted (`insert_rs` would drop it), so + // we fall back to the flow wrapper to preserve retry semantics. + // `retry_if` is always materialized natively and evaluated on the + // failure path (see `eval_retry_if`). On a worker built without the + // `quickjs` feature it cannot be evaluated and fails closed (no retry); + // the flow path is not a fallback, since the flow runtime needs quickjs + // too. + let native_retry = !is_flow + && skip_handler.is_none() + && error_handler_path.is_none() + && hash.is_some() + && language.is_some() + && windmill_common::runnable_settings::min_version_supports_runnable_settings_v0() + .await; + if native_retry { + if apply_preprocessor { + preprocessed = Some(false); + } + // Preserve worker affinity: normal ScriptHash pushes carry the + // script's `dedicated_worker` (it drives the dedicated tag below), + // but the SingleStepFlow payload doesn't — resolve it from the + // script row so a dedicated-worker script keeps its dedicated pool. + let dedicated_worker = if let Some(h) = &hash { + // Read on the non-RLS pool: push_inner is also entered with RLS + // isolation variants under which the script row may be invisible, + // which would mis-resolve dedicated_worker routing. + sqlx::query_scalar::<_, Option>( + "SELECT dedicated_worker FROM script WHERE hash = $1 AND workspace_id = $2", + ) + .bind(h.0) + .bind(workspace_id) + .fetch_optional(db) + .await? + .flatten() + } else { + None + }; + break 'ssf JobPayloadUntagged { + runnable_id: hash.map(|h| h.0), + runnable_path: Some(path), + job_kind: JobKind::Script, + language, + dedicated_worker, + concurrency_settings, + debouncing_settings, + retry_settings: retry.as_ref().map(RetrySettings::from).unwrap_or_default(), + cache_ttl, + cache_ignore_s3_path, + _low_level_priority: priority, + ..Default::default() + }; + } + // Build modules list let mut modules = vec![]; @@ -5903,6 +6358,7 @@ async fn push_inner<'c, 'd>( RunnableSettings { debouncing_settings: debouncing_settings.insert_cached(db).await?, concurrency_settings: concurrency_settings.insert_cached(db).await?, + retry_settings: retry_settings.insert_cached(db).await?, }, db, ) diff --git a/backend/windmill-queue/src/schedule.rs b/backend/windmill-queue/src/schedule.rs index 3f6c587be2..8816eeae25 100644 --- a/backend/windmill-queue/src/schedule.rs +++ b/backend/windmill-queue/src/schedule.rs @@ -255,6 +255,7 @@ pub async fn push_scheduled_job<'c>( path: schedule.script_path.clone(), hash, flow_version, + language: None, args: args.clone(), retry, error_handler_path: None, @@ -352,6 +353,11 @@ pub async fn push_scheduled_job<'c>( .warn_after_seconds_with_sql(1, "get_latest_hash_for_path".to_string()) .await?; + // NB: read on the non-RLS pool (`db`), not `tx`. push_scheduled_job is + // also invoked with an RLS user_db transaction (api-schedule/api-flows), + // under which these lookups would resolve against the caller's row + // visibility rather than the full table. The dual-connection here is + // intentional and required for correctness. let (debouncing_settings, concurrency_settings) = windmill_common::runnable_settings::prefetch_cached_from_handle( runnable_settings_handle, @@ -371,12 +377,20 @@ pub async fn push_scheduled_job<'c>( for (arg_name, arg_value) in args.clone() { static_args.insert(arg_name, arg_value); } - // if retry is set, we wrap the script into a one step flow with a retry on the module + // A retry on a scheduled script is materialized into a native retry + // (see `push`): `Some(language)` opts in. Completion handlers are + // driven from the terminal attempt, and the per-occurrence + // failure/recovery counting queries (apply_schedule_handlers) resolve + // terminal status across the retry chain — so on_failure/on_recovery + // (incl. multi-count/exact) are all handled. A `retry_if` gate is + // evaluated at failure time; on a worker built without quickjs it + // cannot be evaluated and fails closed (no retry). ( JobPayload::SingleStepFlow { path: schedule.script_path.clone(), hash: Some(hash), flow_version: None, + language: Some(language), retry: Some(parsed_retry), error_handler_path: None, error_handler_args: None, @@ -388,8 +402,12 @@ pub async fn push_scheduled_job<'c>( tag_override: schedule.tag.clone(), trigger_path: None, apply_preprocessor: false, - concurrency_settings: ConcurrencySettings::default(), - debouncing_settings: DebouncingSettings::default(), + // Carry the script's concurrency/debounce settings (fetched + // above) into the native retry materialization, so a retrying + // concurrency-limited scheduled script still inserts its + // concurrency_key instead of running unbounded. + concurrency_settings, + debouncing_settings, }, if schedule.tag.as_ref().is_some_and(|x| x != "") { schedule.tag.clone() @@ -497,7 +515,7 @@ pub async fn push_scheduled_job<'c>( if let Some(tag) = tag.as_deref().filter(|t| !t.is_empty()) { check_tag_available_for_workspace_internal( - &db, + db, &schedule.workspace_id, &tag, &email, @@ -600,7 +618,9 @@ pub async fn clear_schedule<'c>( w_id: &str, ) -> Result<()> { tracing::info!("Clearing schedule {}", path); - sqlx::query!( + // Delete the queued jobs (cascading their v2_job_queue-keyed side tables), then route the + // freed ids through delete_jobs so v2_job and its no-longer-cascading side tables go too. + let deleted_ids: Vec = sqlx::query_scalar!( "WITH to_delete AS ( SELECT id FROM v2_job_queue JOIN v2_job j USING (id) @@ -610,15 +630,16 @@ pub async fn clear_schedule<'c>( AND flow_step_id IS NULL AND running = false FOR UPDATE - ), deleted AS ( - DELETE FROM v2_job_queue - WHERE id IN (SELECT id FROM to_delete) - RETURNING id - ) DELETE FROM v2_job WHERE id IN (SELECT id FROM deleted)", + ) + DELETE FROM v2_job_queue + WHERE id IN (SELECT id FROM to_delete) + RETURNING id", path, w_id ) - .execute(&mut **tx) + .fetch_all(&mut **tx) .await?; + + windmill_common::jobs::delete_jobs(&mut **tx, &deleted_ids).await?; Ok(()) } diff --git a/backend/windmill-queue/tests/debounce_test.rs b/backend/windmill-queue/tests/debounce_test.rs index a2c6971c1c..d74f20ff3f 100644 --- a/backend/windmill-queue/tests/debounce_test.rs +++ b/backend/windmill-queue/tests/debounce_test.rs @@ -3364,6 +3364,7 @@ mod debounce { let rs = RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: concurrency_hash, + retry_settings: None, }; let rs_handle = insert_rs(rs, &db).await?; @@ -3741,6 +3742,7 @@ mod debounce { RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: concurrency_hash, + retry_settings: None, }, db, ) @@ -3878,6 +3880,1577 @@ mod debounce { Ok(()) } + /// Helper: push a script job through push-time `maybe_debounce` with the given key. + /// Returns the args JSON it was pushed with. + async fn push_debounced_script( + db: &Pool, + id: Uuid, + items: Vec, + settings: &DebouncingSettings, + rs_handle: Option, + ) -> serde_json::Value { + let args_val = serde_json::json!({ "items": items }); + insert_script_job_with_args(db, id, "test-workspace", "f/test/script", &args_val).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(db) + .await + .unwrap(); + let args_hm: HashMap> = + serde_json::from_value(args_val.clone()).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &push_args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + args_val + } + + /// Regression test for the "running survivor" data-loss bug. + /// + /// When a debounce survivor has already been pulled and is executing (running), it + /// has committed its batch and can no longer accumulate later arrivals. The old + /// behavior superseded the running survivor anyway: it was completed/skipped + /// ("Debounced Running by ...") and deleted from the queue, silently dropping its + /// accumulated work, while the late arrival could not merge into it. + /// + /// Fix: a late arrival that finds the current survivor already running starts a + /// FRESH debounce window. The running survivor is left to finish with its own + /// accumulated batch; the late arrival accumulates only its own batch. No job is + /// killed and no item is dropped or double-run. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J1, J2 share a window; J2 is the survivor with batch {J1, J2}. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + assert!(is_completed(&db, &j1).await, "J1 should be debounced by J2"); + assert!(is_queued(&db, &j2).await, "J2 should be the survivor"); + + // Worker pulls J2 and marks it running: the window where the key still points + // to J2 and its batch is intact, but J2 can no longer accumulate new arrivals. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3 while J2 is running. + let j3 = Uuid::new_v4(); + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + + // The running survivor J2 must NOT be superseded: still queued, not completed. + assert!( + is_queued(&db, &j2).await, + "running survivor J2 must stay in the queue" + ); + assert!( + !is_completed(&db, &j2).await, + "running survivor J2 must not be completed/skipped" + ); + + // J3 must own the debounce key as the head of a FRESH window (no previous job). + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3, "J3 should hold the debounce key"); + assert!( + dk_prev.is_none(), + "J3 should start a fresh window with no previous job (got {dk_prev:?})" + ); + assert_eq!( + dk_times, 0, + "fresh window should reset debounced_times to 0" + ); + + // The running survivor J2 accumulates only its own committed batch: [1, 2]. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &serde_json::json!({"items": [2]}), + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert!( + j2_res.job.is_some(), + "running survivor J2 must still execute (not nulled out)" + ); + assert_accumulated_items(&j2_res, &[1, 2], "items"); + + // The late arrival J3 accumulates only its own batch: [3]. No overlap with J2. + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert!(j3_res.job.is_some(), "J3 must execute"); + assert_accumulated_items(&j3_res, &[3], "items"); + + Ok(()) + } + + /// A second arrival debouncing a NON-running survivor must keep accumulating into + /// the same batch (the normal debounce behavior must be unchanged by the + /// running-survivor guard). + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_queued_survivor_still_accumulates( + db: Pool, + ) -> anyhow::Result<()> { + let key = "queued_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Three arrivals, none running: classic debounce, all accumulate into J3. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + let j3 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + + assert!(is_completed(&db, &j1).await, "J1 debounced"); + assert!(is_completed(&db, &j2).await, "J2 debounced"); + assert!(is_queued(&db, &j3).await, "J3 is the survivor"); + + // Window keeps growing: J3 is the third arrival in the same batch. + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3); + assert_eq!(dk_prev, Some(j2), "previous job should be J2"); + assert_eq!(dk_times, 2, "debounced_times should keep incrementing"); + + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j3_res, &[1, 2, 3], "items"); + + Ok(()) + } + + /// The running-survivor guard must also apply to plain debounce (delay only, no + /// argument accumulation): a running survivor must never be completed/skipped. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_no_accumulation( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_no_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // No debounce_args_to_accumulate. + ..Default::default() + }; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + insert_noop_job(&db, j1, "test-workspace").await; + insert_noop_job(&db, j2, "test-workspace").await; + + let push = |id: Uuid| { + let settings = settings.clone(); + let db = db.clone(); + async move { + let args_hm = empty_args(); + let args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Noop, + id, + &args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + }; + + push(j1).await; + push(j2).await; + assert!(is_completed(&db, &j1).await, "J1 debounced by J2"); + + // J2 starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3. + let j3 = Uuid::new_v4(); + insert_noop_job(&db, j3, "test-workspace").await; + push(j3).await; + + // Running survivor J2 is preserved; J3 takes over a fresh window. + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3); + assert!(dk_prev.is_none(), "fresh window: no previous job"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + + Ok(()) + } + + /// Once a survivor has fully been pulled (batch + key consumed by + /// maybe_apply_debouncing) and is running, a later arrival naturally starts a new + /// window. This locks in that the committed-running case stays correct alongside + /// the in-flight-running guard. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_committed_running_survivor_independent( + db: Pool, + ) -> anyhow::Result<()> { + let key = "committed_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // J2 is pulled: accumulate its batch and consume key + batch. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j2_res, &[1, 2], "items"); + assert!( + get_debounce_key(&db, key).await.is_none(), + "key consumed when survivor pulled" + ); + + // J2 now running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3: fresh window, independent batch, J2 untouched. + let j3 = Uuid::new_v4(); + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + assert!(is_queued(&db, &j2).await, "running J2 untouched"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, _, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, j3); + assert_eq!(dk_times, 0); + + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j3_res, &[3], "items"); + + Ok(()) + } + + /// Flow post-preprocessing debounce must apply the same running-survivor guard: + /// a running flow survivor must not be completed/skipped by a late flow arrival. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_post_preprocessing_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "pp_running_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + ..Default::default() + }; + let args_hm = empty_args(); + + let flow1 = Uuid::new_v4(); + let flow2 = Uuid::new_v4(); + insert_flow_job(&db, flow1, "test-workspace", "f/test/flow").await; + insert_flow_job(&db, flow2, "test-workspace", "f/test/flow").await; + + let pp = |id: Uuid| { + let settings = settings.clone(); + let db = db.clone(); + let args_hm = args_hm.clone(); + async move { + let args = PushArgs::from(&args_hm); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow".to_string()), + "test-workspace", + id, + &args, + &db, + ) + .await + .unwrap() + } + }; + + pp(flow1).await; + pp(flow2).await; + assert!(is_completed(&db, &flow1).await, "flow1 debounced by flow2"); + + // flow2 (the survivor) starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + flow2 + ) + .execute(&db) + .await?; + + // Late flow3 arrival while flow2 is running. + let flow3 = Uuid::new_v4(); + insert_flow_job(&db, flow3, "test-workspace", "f/test/flow").await; + let sched = pp(flow3).await; + assert!(sched.is_some(), "flow3 should be debounced (fresh window)"); + + // Running flow2 must be preserved; flow3 owns a fresh window. + assert!(is_queued(&db, &flow2).await, "running flow2 stays queued"); + assert!( + !is_completed(&db, &flow2).await, + "running flow2 must not be completed" + ); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, flow3, "flow3 holds the key"); + assert!(dk_prev.is_none(), "fresh window: no previous job"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + + Ok(()) + } + + /// A running survivor resets the debounce window for the late arrival, so an + /// inherited high `debounced_times` cannot push the new arrival over + /// max_total_debounces_amount and force an immediate (un-debounced) run. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_resets_limit_window( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_limit_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // Limit of 3: J1, J2 stay debounced; a third arrival in the SAME window + // would trip the limit (current_amount + 1 >= 3) and fire immediately. + max_total_debounces_amount: Some(3), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Build up the window close to the limit: J1, J2 (debounced_times = 1 on J2). + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let (_, _, times_before) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(times_before, 1); + + // J2 starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // J3 arrives. Without the reset it would inherit debounced_times and could trip + // the max-count limit and fire immediately, killing running J2. With the guard + // it starts a fresh window (debounced_times = 0) and is debounced normally. + let j3 = Uuid::new_v4(); + let mut scheduled_for = None; + { + let args_val = serde_json::json!({ "items": [3] }); + insert_script_job_with_args(&db, j3, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j3, + ) + .execute(&db) + .await?; + let args_hm: HashMap> = serde_json::from_value(args_val).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + j3, + &push_args, + &mut tx, + ) + .await?; + tx.commit().await?; + } + + // J3 is debounced (scheduled_for set, not fired immediately) and J2 survives. + assert!( + scheduled_for.is_some(), + "J3 should be debounced, not fired immediately" + ); + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, j3); + assert!(dk_prev.is_none()); + assert_eq!(dk_times, 0, "fresh window resets the limit counter"); + + Ok(()) + } + + /// Concurrency regression: two late arrivals racing AFTER the survivor started + /// running must not both spawn independent windows. The running check reads the + /// post-conflict-lock holder (`debounce_key.job_id`), so the row lock serializes the + /// two upserts: the first observes the running survivor and opens a fresh window; + /// the second observes that fresh-window head (queued, not running) and debounces + /// into it. Exactly one late arrival survives, the other is debounced — never both. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_concurrent_arrivals_after_running_survivor( + db: Pool, + ) -> anyhow::Result<()> { + let key = "concurrent_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J2 is the survivor and starts running. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Insert the two late arrivals up front, then race only their maybe_debounce calls. + let j3 = Uuid::new_v4(); + let j4 = Uuid::new_v4(); + for (id, items) in [(j3, 3i64), (j4, 4i64)] { + let args_val = serde_json::json!({ "items": [items] }); + insert_script_job_with_args(&db, id, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(&db) + .await?; + } + + let race = |id: Uuid, items: i64| { + let db = db.clone(); + let settings = settings.clone(); + async move { + let args_hm: HashMap> = + serde_json::from_value(serde_json::json!({ "items": [items] })).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &push_args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + }; + tokio::join!(race(j3, 3), race(j4, 4)); + + // The running survivor is untouched. + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + + // Exactly one late arrival survives; the other is debounced into the same window. + let j3q = is_queued(&db, &j3).await; + let j4q = is_queued(&db, &j4).await; + let j3c = is_completed(&db, &j3).await; + let j4c = is_completed(&db, &j4).await; + assert!( + (j3q && j4c && !j4q && !j3c) || (j4q && j3c && !j3q && !j4c), + "exactly one late arrival must survive and the other be debounced \ + (not two independent windows); got j3 queued={j3q} completed={j3c}, \ + j4 queued={j4q} completed={j4c}" + ); + + // The surviving holder chained the debounced arrival into one window. + let (holder, prev, times) = get_debounce_key(&db, key).await.expect("key exists"); + let (survivor, debounced) = if j3q { (j3, j4) } else { (j4, j3) }; + assert_eq!(holder, survivor, "key points to the surviving late arrival"); + assert_eq!( + prev, + Some(debounced), + "the surviving window debounced the other late arrival" + ); + assert_eq!(times, 1, "single fresh window with one debounce"); + + // Both late arrivals must share a batch: when the survivor is pulled, its + // accumulation must include the debounced arrival's items, not just its own. + let survivor_args = serde_json::json!({ "items": [if j3q { 3 } else { 4 }] }); + let mut survivor_res = make_pulled_job_result( + survivor, + "test-workspace", + "f/test/script", + &survivor_args, + JobKind::Script, + "deno", + rs_handle, + ); + survivor_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&survivor_res, &[3, 4], "items"); + + Ok(()) + } + + /// Regression: a push that would chain onto a queued holder must not error if the + /// worker pull path concurrently deletes that holder's debounce_key + /// (`DELETE ... WHERE job_id = ...`, which does NOT take the push advisory lock). + /// The upsert is a single atomic `INSERT ... ON CONFLICT`, so a deleted holder simply + /// yields a fresh window rather than a "no row updated" failure. Races the two and + /// asserts the push always succeeds and leaves a consistent key. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_push_races_key_deletion_by_pull( + db: Pool, + ) -> anyhow::Result<()> { + let key = "push_vs_pull_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // 50 rounds to give the interleaving a chance to land in the read/write window + // that the old split read+UPDATE path would have failed on. + for round in 0..50 { + sqlx::query!("DELETE FROM debounce_key WHERE key = $1", key) + .execute(&db) + .await?; + let holder = Uuid::new_v4(); + push_debounced_script(&db, holder, vec![round], &settings, rs_handle).await; + + let late = Uuid::new_v4(); + let late_args = serde_json::json!({ "items": [round * 1000] }); + insert_script_job_with_args(&db, late, "test-workspace", "f/test/script", &late_args) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + late, + ) + .execute(&db) + .await?; + + // Race: push the late arrival (chains onto `holder`) against the worker pull + // cleanup deleting `holder`'s key. + let push = { + let db = db.clone(); + let settings = settings.clone(); + async move { + let args_hm: HashMap> = + serde_json::from_value(serde_json::json!({ "items": [round * 1000] })) + .unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + let res = windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + late, + &push_args, + &mut tx, + ) + .await; + if res.is_ok() { + tx.commit().await.unwrap(); + } + res + } + }; + let delete = { + let db = db.clone(); + async move { + sqlx::query!("DELETE FROM debounce_key WHERE job_id = $1", holder) + .execute(&db) + .await + } + }; + let (push_res, _) = tokio::join!(push, delete); + assert!( + push_res.is_ok(), + "round {round}: push must not error when the holder key is concurrently deleted: {push_res:?}" + ); + } + + Ok(()) + } + + /// Claim-based exactly-once: if two survivors end up on the same batch (only + /// possible in a narrow push/pull race), the args of each member are accumulated + /// into exactly ONE run. The survivor that claims the batch first accumulates + /// everyone; the second survivor finds its contribution already consumed and runs + /// empty — no item is dropped and none is processed twice. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_batch_consumed_exactly_once(db: Pool) -> anyhow::Result<()> { + let key = "exactly_once_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J1 superseded, J2 the (first) survivor of batch B = {J1, J2}. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // Simulate the race outcome: a second survivor J3 ended up on the SAME batch B. + let j3 = Uuid::new_v4(); + let j3_args = serde_json::json!({ "items": [3] }); + insert_script_job_with_args(&db, j3, "test-workspace", "f/test/script", &j3_args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j3, + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch) + SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + j3, + j2, + ) + .execute(&db) + .await?; + + // J2 pulled first: claims the whole batch, accumulates everyone's items. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert!(j2_res.job.is_some(), "J2 runs"); + assert_accumulated_items(&j2_res, &[1, 2, 3], "items"); + + // J3 pulled next: its contribution was already consumed by J2 -> runs empty, + // so [3] is not processed a second time. + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + let job = j3_res.job.as_ref().expect("J3 still runs (empty)"); + let items: Vec = + serde_json::from_str(job.job.args.as_ref().unwrap().get("items").unwrap().get())?; + assert!( + items.is_empty(), + "J3's items must be empty (already consumed by J2), got {items:?}" + ); + + Ok(()) + } + + /// A survivor re-pulled (e.g. crash recovery) must NOT mistake its own earlier + /// claim for a sibling's and wipe its accumulated args. consumed_by = self is + /// distinguished from consumed_by = another job. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_repull_keeps_accumulated(db: Pool) -> anyhow::Result<()> { + let key = "repull_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // First pull: J2 claims its batch and accumulates [1, 2]. + let mut first = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + first.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&first, &[1, 2], "items"); + + // Re-pull with the args persisted by the first pull: J2 sees its OWN prior claim + // (consumed_by = j2), so it keeps the accumulated args rather than running empty. + let persisted = first.job.as_ref().unwrap().job.args.as_ref().unwrap(); + let persisted_json = serde_json::to_value(persisted).unwrap(); + let mut second = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &persisted_json, + JobKind::Script, + "deno", + rs_handle, + ); + second.maybe_apply_debouncing(&db).await?; + assert!(second.job.is_some(), "re-pulled J2 still runs"); + assert_accumulated_items(&second, &[1, 2], "items"); + + Ok(()) + } + + /// Helper: insert a script job and put it on the SAME debounce batch as `of_job` + /// (simulating a chained survivor). Returns its args JSON. + async fn add_survivor_to_batch_of( + db: &Pool, + id: Uuid, + items: Vec, + of_job: Uuid, + rs_handle: Option, + ) -> serde_json::Value { + let args = serde_json::json!({ "items": items }); + insert_script_job_with_args(db, id, "test-workspace", "f/test/script", &args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(db) + .await + .unwrap(); + let inserted = sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch) + SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + id, + of_job, + ) + .execute(db) + .await + .unwrap(); + // `of_job` must already have a batch row, else this no-ops and the test would + // pass vacuously (the job would end up never-batched, keeping its own args). + assert_eq!( + inserted.rows_affected(), + 1, + "add_survivor_to_batch_of: {of_job} has no batch row to share" + ); + args + } + + /// Helper: read the accumulated `items` of a pulled job as a sorted Vec. + fn items_of(result: &windmill_queue::PulledJobResult) -> Vec { + let job = result.job.as_ref().expect("job present"); + let raw = job.job.args.as_ref().unwrap().get("items").unwrap(); + let mut v: Vec = serde_json::from_str::>(raw.get()) + .unwrap() + .iter() + .map(|x| x.as_i64().unwrap()) + .collect(); + v.sort(); + v + } + + /// Edge: an accumulate-debounced job that was NEVER batched (CE / workers behind v2: + /// no v2_job_debounce_batch row) must keep its own args, not be emptied. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_never_batched_keeps_own_args(db: Pool) -> anyhow::Result<()> { + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("never_batched_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Insert a job with the debounce handle but DO NOT push through maybe_debounce, + // so it has no batch row at all. + let j = Uuid::new_v4(); + let args = serde_json::json!({ "items": [7, 8] }); + insert_script_job_with_args(&db, j, "test-workspace", "f/test/script", &args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j, + ) + .execute(&db) + .await?; + + let mut res = make_pulled_job_result( + j, + "test-workspace", + "f/test/script", + &args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + assert!(res.job.is_some(), "never-batched job still runs"); + assert_accumulated_items(&res, &[7, 8], "items"); + Ok(()) + } + + /// Edge: two survivors of one batch pulled CONCURRENTLY. The atomic claim must + /// partition the batch disjointly — the union of what they each accumulate is the + /// full set, with NO item processed by both. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_concurrent_claim_disjoint(db: Pool) -> anyhow::Result<()> { + let key = "concurrent_claim_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; // superseded + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; // survivor 1 + let j3 = Uuid::new_v4(); + let j3_args = add_survivor_to_batch_of(&db, j3, vec![3], j2, rs_handle).await; // survivor 2 + + let pull = |id: Uuid, args: serde_json::Value| { + let db = db.clone(); + async move { + let mut res = make_pulled_job_result( + id, + "test-workspace", + "f/test/script", + &args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await.unwrap(); + res + } + }; + let (r2, r3) = tokio::join!(pull(j2, j2_args), pull(j3, j3_args)); + + let mut union = items_of(&r2); + union.extend(items_of(&r3)); + union.sort(); + assert_eq!( + union, + vec![1, 2, 3], + "every item accumulated exactly once across the two concurrent survivors" + ); + // disjoint: no overlap between the two survivors' items + let i2 = items_of(&r2); + let i3 = items_of(&r3); + assert!( + !i2.iter().any(|x| i3.contains(x)), + "no item processed by both survivors; got j2={i2:?} j3={i3:?}" + ); + Ok(()) + } + + /// Edge: three survivors on one batch pulled in sequence. The first claims the whole + /// batch; the rest find themselves consumed and run empty. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_three_survivors_first_takes_all( + db: Pool, + ) -> anyhow::Result<()> { + let key = "three_survivors_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let j3 = Uuid::new_v4(); + let j3_args = add_survivor_to_batch_of(&db, j3, vec![3], j2, rs_handle).await; + let j4 = Uuid::new_v4(); + let j4_args = add_survivor_to_batch_of(&db, j4, vec![4], j2, rs_handle).await; + + let mk = |id, args: &serde_json::Value| { + make_pulled_job_result( + id, + "test-workspace", + "f/test/script", + args, + JobKind::Script, + "deno", + rs_handle, + ) + }; + let (mut r2, mut r3, mut r4) = (mk(j2, &j2_args), mk(j3, &j3_args), mk(j4, &j4_args)); + r2.maybe_apply_debouncing(&db).await?; + r3.maybe_apply_debouncing(&db).await?; + r4.maybe_apply_debouncing(&db).await?; + + assert_eq!( + items_of(&r2), + vec![1, 2, 3, 4], + "first survivor takes the whole batch" + ); + assert!(items_of(&r3).is_empty(), "second survivor runs empty"); + assert!(items_of(&r4).is_empty(), "third survivor runs empty"); + Ok(()) + } + + /// Edge: plain debounce (no accumulate args) must HARD-DELETE its batch rows on pull + /// (not leave consumed rows lingering), so the non-accumulate path doesn't leak. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_non_accumulate_deletes_batch(db: Pool) -> anyhow::Result<()> { + let key = "non_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // no debounce_args_to_accumulate + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + // push_debounced_script sends {items:[...]} but with no accumulate arg configured, + // the batch is created yet never accumulated. + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + let batch_rows_before: i64 = + sqlx::query_scalar!("SELECT count(*) as \"c!\" FROM v2_job_debounce_batch") + .fetch_one(&db) + .await?; + assert!(batch_rows_before >= 2, "batch rows exist before pull"); + + let mut res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + + let remaining: i64 = + sqlx::query_scalar!("SELECT count(*) as \"c!\" FROM v2_job_debounce_batch") + .fetch_one(&db) + .await?; + assert_eq!( + remaining, 0, + "non-accumulate pull hard-deletes the batch rows" + ); + Ok(()) + } + + /// Edge: GC sweep deletes consumed rows past the grace period but keeps recently + /// consumed and not-yet-consumed rows. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_gc_consumed_batches(db: Pool) -> anyhow::Result<()> { + let old = Uuid::new_v4(); + let recent = Uuid::new_v4(); + let unconsumed = Uuid::new_v4(); + // A consumed-long-ago sibling that is STILL QUEUED (e.g. stuck behind a + // concurrency limit): its marker must survive GC so its eventual pull still sees + // "already consumed" and runs empty (no duplicate). + let queued_old = Uuid::new_v4(); + insert_script_job_with_args( + &db, + queued_old, + "test-workspace", + "f/test/script", + &serde_json::json!({ "items": [9] }), + ) + .await; + sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES + ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'), + ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'), + ($3, nextval('debounce_batch_seq'), NULL), + ($4, nextval('debounce_batch_seq'), now() - interval '20 minutes')", + old, + recent, + unconsumed, + queued_old, + ) + .execute(&db) + .await?; + + // Mirror the monitor GC sweep (age floor + only-if-no-longer-queued). + let deleted = sqlx::query_scalar!( + "WITH del AS ( + DELETE FROM v2_job_debounce_batch + WHERE consumed_at IS NOT NULL + AND consumed_at < now() - interval '10 minutes' + AND id NOT IN (SELECT id FROM v2_job_queue) + RETURNING 1 + ) SELECT count(*) as \"c!\" FROM del" + ) + .fetch_one(&db) + .await?; + assert_eq!( + deleted, 1, + "only the old, no-longer-queued consumed row is GC'd" + ); + + let exists = |id: Uuid, db: Pool| async move { + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"", + id + ) + .fetch_one(&db) + .await + .unwrap() + }; + assert!(!exists(old, db.clone()).await, "old consumed row gone"); + assert!( + exists(recent, db.clone()).await, + "recently consumed row kept" + ); + assert!(exists(unconsumed, db.clone()).await, "unconsumed row kept"); + assert!( + exists(queued_old, db.clone()).await, + "old consumed row whose job is still queued must be kept" + ); + Ok(()) + } + + /// Helper: mark a queued job as running (simulates a survivor that the + /// concurrency limiter has just started executing). + async fn set_running(db: &Pool, job_id: &Uuid) { + sqlx::query!( + "UPDATE v2_job_queue SET running = true WHERE id = $1", + job_id + ) + .execute(db) + .await + .expect("set running"); + } + + /// Regression (ref #9781): post-preprocessing debounce with + /// `debounce_args_to_accumulate` under a concurrency limit. A survivor accumulates + /// its own element and starts running; a later same-key message must start a NEW + /// batch (survive) rather than be folded into the running survivor and silently + /// dropped. Exercises the full EE path via `jobs_ee::maybe_debounce_post_preprocessing`. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_post_preprocessing_debounce_into_running_survivor_loses_message( + db: Pool, + ) -> anyhow::Result<()> { + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("ported_running_survivor_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // --- Wave 1: a single message becomes the survivor and starts running. --- + let survivor = Uuid::new_v4(); + let survivor_args = serde_json::json!({ "items": [1] }); + insert_flow_job_with_preprocessor( + &db, + survivor, + "test-workspace", + "f/test/flow_run", + true, + 0, + ) + .await; + sqlx::query!( + "UPDATE v2_job SET args = $2 WHERE id = $1", + survivor, + survivor_args + ) + .execute(&db) + .await?; + + let survivor_args_hm: HashMap> = + serde_json::from_value(survivor_args.clone()).unwrap(); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow_run".to_string()), + "test-workspace", + survivor, + &PushArgs::from(&survivor_args_hm), + &db, + ) + .await?; + assert!(is_queued(&db, &survivor).await, "survivor should be queued"); + + // Worker pulls the survivor: accumulate its own [1], consume the batch, run. + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + survivor, + ) + .execute(&db) + .await?; + let mut pulled = make_pulled_job_result( + survivor, + "test-workspace", + "f/test/flow_run", + &survivor_args, + JobKind::Flow, + "flow", + rs_handle, + ); + pulled.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&pulled, &[1], "items"); + set_running(&db, &survivor).await; // survivor is now RUNNING + + // --- Wave 2: a new message arrives while the survivor is running. --- + let late = Uuid::new_v4(); + let late_args = serde_json::json!({ "items": [2] }); + insert_flow_job_with_preprocessor(&db, late, "test-workspace", "f/test/flow_run", true, 0) + .await; + sqlx::query!("UPDATE v2_job SET args = $2 WHERE id = $1", late, late_args) + .execute(&db) + .await?; + + let late_args_hm: HashMap> = + serde_json::from_value(late_args.clone()).unwrap(); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow_run".to_string()), + "test-workspace", + late, + &PushArgs::from(&late_args_hm), + &db, + ) + .await?; + + // The late message must survive (new batch): still queued, and the debounce_key + // moved off the already-running survivor. + let late_survived = is_queued(&db, &late).await && !is_completed(&db, &late).await; + let dk = get_debounce_key(&db, "ported_running_survivor_key").await; + let key_moved_off_running_survivor = + dk.map(|(job_id, _, _)| job_id != survivor).unwrap_or(true); + assert!( + late_survived && key_moved_off_running_survivor, + "message arriving while the survivor is running must start a new batch \ + (survive), not be folded into the running survivor and dropped. \ + late_survived={late_survived}, key_moved_off={key_moved_off_running_survivor}" + ); + Ok(()) + } + + /// Flow-node debounce (third EE entry point, `jobs_ee::maybe_debounce_flow_node`): + /// a running survivor child must not be superseded by a later same-key child; the + /// late child starts a fresh window and the running child is left to finish. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_flow_node_debounce_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "flow_node_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + ..Default::default() + }; + let args_hm = empty_args(); + + let flow1 = Uuid::new_v4(); + let child1 = Uuid::new_v4(); + insert_flow_job(&db, flow1, "test-workspace", "f/test/my_flow").await; + insert_child_job_with_parent(&db, child1, flow1, "test-workspace").await; + + // child1 becomes the survivor, then starts running. + { + let args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce_flow_node( + &settings, + child1, + flow1, + "f/test/my_flow", + "step_a", + "test-workspace", + &args, + &mut tx, + &db, + ) + .await?; + tx.commit().await?; + } + set_running(&db, &child1).await; + + // child2 (later same-key child) arrives while child1 is running. + let flow2 = Uuid::new_v4(); + let child2 = Uuid::new_v4(); + insert_flow_job(&db, flow2, "test-workspace", "f/test/my_flow").await; + insert_child_job_with_parent(&db, child2, flow2, "test-workspace").await; + { + let args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce_flow_node( + &settings, + child2, + flow2, + "f/test/my_flow", + "step_a", + "test-workspace", + &args, + &mut tx, + &db, + ) + .await?; + tx.commit().await?; + } + + // The running child1 (and its parent flow1) must be left alone; child2 owns a + // fresh window. + assert!(is_queued(&db, &child1).await, "running child1 stays queued"); + assert!( + !is_completed(&db, &child1).await, + "running child1 not completed" + ); + assert!( + !is_completed(&db, &flow1).await, + "flow1 of running child not completed" + ); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, child2, "child2 holds the key"); + assert!(dk_prev.is_none(), "fresh window: no previous child"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + Ok(()) + } + + /// Edge: accumulate values that are bare scalars (not arrays) — the `T | T[]` union + /// case. Each scalar contribution must be wrapped into a single-element list so the + /// survivor accumulates them all. Exercises the non-array fallback in the claim path. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_accumulate_scalar_values(db: Pool) -> anyhow::Result<()> { + let key = "scalar_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Push two jobs whose `items` is a BARE SCALAR, not an array. + let push_scalar = |id: Uuid, v: i64, db: Pool, settings: DebouncingSettings| async move { + let args_val = serde_json::json!({ "items": v }); + insert_script_job_with_args(&db, id, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(&db) + .await + .unwrap(); + let hm: HashMap> = serde_json::from_value(args_val).unwrap(); + let mut sf = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut sf, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &PushArgs::from(&hm), + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + }; + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_scalar(j1, 1, db.clone(), settings.clone()).await; + push_scalar(j2, 2, db.clone(), settings.clone()).await; + + let mut res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &serde_json::json!({ "items": 2 }), + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + // Both bare scalars are wrapped and accumulated into a list. + assert_accumulated_items(&res, &[1, 2], "items"); + Ok(()) + } + + /// Edge: GC reclaiming a survivor's consumed row before a re-pull must NOT lose data — + /// the re-pull finds no row (had_row=false) and keeps its already-persisted accumulated + /// args, rather than running empty. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_repull_after_gc_keeps_accumulated( + db: Pool, + ) -> anyhow::Result<()> { + let key = "repull_gc_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + let mut first = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + first.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&first, &[1, 2], "items"); + + // Simulate the GC sweep reclaiming the (now consumed) batch rows for this batch. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j2, + ) + .execute(&db) + .await + .ok(); + // (and any that were already consumed elsewhere) + sqlx::query!("DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL") + .execute(&db) + .await?; + + // Re-pull with the args persisted on the first pull: no batch row now, so it must + // fall back to its own (already-accumulated) args — no loss. + let persisted = + serde_json::to_value(first.job.as_ref().unwrap().job.args.as_ref().unwrap()).unwrap(); + let mut second = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &persisted, + JobKind::Script, + "deno", + rs_handle, + ); + second.maybe_apply_debouncing(&db).await?; + assert!(second.job.is_some(), "re-pulled survivor still runs"); + assert_accumulated_items(&second, &[1, 2], "items"); + Ok(()) + } + + /// Throughput benchmark for the FULL debounce path (EE push + + /// `jobs_ee::maybe_debounce`/`complete_debounced_job`/`upsert_debounce_key`, then OSS + /// `maybe_apply_debouncing` claim/accumulate/consume). #[ignore]d — run manually: + /// cargo test -p windmill-queue --test debounce_test --features private,enterprise \ + /// bench_debounce_full_path -- --ignored --nocapture --test-threads=1 + /// Each cycle = a burst of BURST pushes to one key (debounced) + one survivor pull + /// (accumulate+consume), run across CONCURRENCY tasks. Compare before/after by running + /// it on each code revision. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + #[ignore] + async fn bench_debounce_full_path(db: Pool) -> anyhow::Result<()> { + const CONCURRENCY: usize = 4; + const CYCLES_PER_TASK: usize = 300; + const BURST: usize = 3; + + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("bench_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let start = std::time::Instant::now(); + let tasks: Vec<_> = (0..CONCURRENCY) + .map(|t| { + let db = db.clone(); + let settings = DebouncingSettings { + // distinct key space per task so bursts collapse independently + debounce_key: Some(format!("bench_key_{t}")), + ..settings.clone() + }; + tokio::spawn(async move { + for c in 0..CYCLES_PER_TASK { + // Fresh key per cycle so each cycle is one full collapse+pull. + let key = format!("bench_{t}_{c}"); + let settings = DebouncingSettings { + debounce_key: Some(key.clone()), + ..settings.clone() + }; + let mut survivor = Uuid::new_v4(); + for b in 0..BURST { + let id = Uuid::new_v4(); + survivor = id; + let args_val = serde_json::json!({ "items": [b as i64] }); + insert_script_job_with_args( + &db, id, "test-workspace", "f/test/script", &args_val, + ) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, id, + ) + .execute(&db) + .await + .unwrap(); + let hm: HashMap> = + serde_json::from_value(args_val).unwrap(); + let mut sf = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, &mut sf, &Some("f/test/script".to_string()), + "test-workspace", JobKind::Script, id, &PushArgs::from(&hm), &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + // Survivor pull: claim + accumulate + consume. + let mut res = make_pulled_job_result( + survivor, "test-workspace", "f/test/script", + &serde_json::json!({ "items": [] }), + JobKind::Script, "deno", rs_handle, + ); + res.maybe_apply_debouncing(&db).await.unwrap(); + } + }) + }) + .collect(); + for t in tasks { + t.await.unwrap(); + } + let elapsed = start.elapsed(); + let cycles = CONCURRENCY * CYCLES_PER_TASK; + let jobs = cycles * BURST; + eprintln!( + "BENCH full debounce path: {cycles} cycles ({jobs} pushed jobs + {cycles} pulls) in {:.2?} | {:.0} pushes/s | {:.0} pulls/s", + elapsed, + jobs as f64 / elapsed.as_secs_f64(), + cycles as f64 / elapsed.as_secs_f64(), + ); + Ok(()) + } + /// Test: Push-time (script) debounce with max_total_debounces_amount=2. /// 5 calls, each sending {x: [i]}. Expected: /// Call 1: debounced (scheduled_for set) diff --git a/backend/windmill-queue/tests/native_retry_test.rs b/backend/windmill-queue/tests/native_retry_test.rs new file mode 100644 index 0000000000..fb8046149b --- /dev/null +++ b/backend/windmill-queue/tests/native_retry_test.rs @@ -0,0 +1,376 @@ +// Integration tests for native single-script retry (no one-step-flow wrapping). +// +// These use the *runtime* sqlx API (`sqlx::query`/`query_as`, not the `!` macros) +// like schedule_push.rs, so they need no `.sqlx` offline cache entry. +mod native_retry { + use sqlx::{Pool, Postgres}; + use uuid::Uuid; + + use windmill_common::flows::{ConstantDelay, Retry}; + use windmill_common::jobs::{JobKind, JobTriggerKind}; + use windmill_common::runnable_settings::{ + from_handle, insert_rs, ConcurrencySettings, RetrySettings, RunnableSettings, + RunnableSettingsTrait, + }; + use windmill_common::scripts::{ScriptHash, ScriptLang}; + use windmill_common::users::username_to_permissioned_as; + use windmill_queue::jobs::{maybe_enqueue_native_script_retry, MiniCompletedJob}; + + const WS: &str = "test-workspace"; + const SCHED: &str = "f/system/test_schedule"; + const SCRIPT: &str = "f/system/test_script"; + + fn mini(id: Uuid, parent_job: Option, handle: Option) -> MiniCompletedJob { + MiniCompletedJob { + id, + workspace_id: WS.to_string(), + runnable_id: Some(ScriptHash(100001)), + scheduled_for: chrono::Utc::now(), + parent_job, + flow_innermost_root_job: None, + runnable_path: Some(SCRIPT.to_string()), + kind: JobKind::Script, + started_at: Some(chrono::Utc::now()), + permissioned_as: username_to_permissioned_as("test-user"), + created_by: "test-user".to_string(), + script_lang: Some(ScriptLang::Deno), + permissioned_as_email: "test@windmill.dev".to_string(), + flow_step_id: None, + trigger_kind: Some(JobTriggerKind::Schedule), + trigger: Some(SCHED.to_string()), + priority: None, + concurrent_limit: None, + tag: "deno".to_string(), + cache_ttl: None, + cache_ignore_s3_path: None, + runnable_settings_handle: handle, + } + } + + async fn count_retries(db: &Pool, root: Uuid) -> i64 { + sqlx::query_scalar::<_, i64>("SELECT count(*) FROM v2_job WHERE parent_job = $1") + .bind(root) + .fetch_one(db) + .await + .unwrap() + } + + /// The queued retry with the given attempt number, if any: (id, kind, parent_job, backoff_s, handle). + async fn retry_by_attempt( + db: &Pool, + root: Uuid, + attempt: i64, + ) -> Option<(Uuid, String, Option, f64, Option)> { + sqlx::query_as::<_, (Uuid, String, Option, Option, Option)>( + "SELECT j.id, j.kind::text, j.parent_job, + EXTRACT(EPOCH FROM (q.scheduled_for - now()))::float8, + q.runnable_settings_handle + FROM v2_job j + JOIN v2_job_queue q ON q.id = j.id + JOIN native_retry_attempt nra ON nra.job_id = j.id + WHERE j.parent_job = $1 AND nra.attempt = $2", + ) + .bind(root) + .bind(attempt) + .fetch_optional(db) + .await + .unwrap() + .map(|(id, kind, parent, backoff, handle)| { + (id, kind, parent, backoff.unwrap_or(0.0), handle) + }) + } + + fn no_result() -> Option> { + None + } + + // attempt0 -> retry1 -> retry2 -> exhausted, with crash-replay idempotency. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn chains_attempts_and_is_idempotent(db: Pool) -> anyhow::Result<()> { + // Policy: 2 constant attempts, 1s apart. + let retry = Retry { + constant: ConstantDelay { attempts: 2, seconds: 1 }, + exponential: Default::default(), + retry_if: None, + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: None, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + assert!(handle.is_some(), "a retry policy must produce a handle"); + + // attempt 0 (the schedule root); maybe_enqueue tolerates the absent queue row (counter -> 0). + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + + // First failure -> retry 1. + assert!( + maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?, + "first failure enqueues a retry" + ); + let (r1_id, kind, parent, backoff, r1_handle) = retry_by_attempt(&db, root_id, 1) + .await + .expect("retry attempt 1 exists"); + assert_eq!( + kind, "script", + "retry is a native Script, not a singlestepflow" + ); + assert_eq!(parent, Some(root_id), "retry links to the chain root"); + assert!( + r1_handle.is_some(), + "retry carries the policy for further chaining" + ); + assert!( + backoff > 0.0 && backoff <= 3.0, + "constant 1s backoff, got {backoff}s" + ); + + // Crash-replay: the SAME completion again must not double-enqueue, and must + // still report pending (so schedule handlers stay deferred). Regression for P1. + assert!( + maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?, + "replay still reports the retry as pending" + ); + assert_eq!( + count_retries(&db, root_id).await, + 1, + "no double retry on crash replay" + ); + + // retry 1 fails -> retry 2 (still within attempts = 2). + let r1 = mini(r1_id, Some(root_id), r1_handle); + assert!(maybe_enqueue_native_script_retry(&db, &r1, &None, &no_result).await?); + assert_eq!(count_retries(&db, root_id).await, 2); + + // retry 2 fails -> attempts exhausted, no retry 3. + let (r2_id, _, _, _, r2_handle) = retry_by_attempt(&db, root_id, 2) + .await + .expect("retry attempt 2 exists"); + let r2 = mini(r2_id, Some(root_id), r2_handle); + assert!( + !maybe_enqueue_native_script_retry(&db, &r2, &None, &no_result).await?, + "exhausted policy does not enqueue" + ); + assert_eq!( + count_retries(&db, root_id).await, + 2, + "no retry past max attempts" + ); + Ok(()) + } + + // Cancellation always wins over a pending retry. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn canceled_job_does_not_retry(db: Pool) -> anyhow::Result<()> { + let retry = Retry { + constant: ConstantDelay { attempts: 3, seconds: 1 }, + exponential: Default::default(), + retry_if: None, + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: None, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + let canceled = Some(windmill_queue::jobs::CanceledBy { + username: Some("test-user".to_string()), + reason: Some("manual".to_string()), + }); + assert!(!maybe_enqueue_native_script_retry(&db, &root, &canceled, &no_result).await?); + assert_eq!( + count_retries(&db, root_id).await, + 0, + "canceled job must not retry" + ); + Ok(()) + } + + // A concurrency-limited script that also retries must carry its concurrency + // settings into each retry, otherwise the retry runs unbounded. Regression: P1. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn retry_preserves_concurrency_settings(db: Pool) -> anyhow::Result<()> { + let retry = Retry { + constant: ConstantDelay { attempts: 1, seconds: 0 }, + exponential: Default::default(), + retry_if: None, + }; + let concurrency = ConcurrencySettings { + concurrency_key: Some("f/system/test_script".to_string()), + concurrent_limit: Some(1), + concurrency_time_window_s: Some(60), + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: concurrency.insert_cached(&db).await?, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + assert!(maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?); + + let (_id, _kind, _parent, _backoff, r1_handle) = retry_by_attempt(&db, root_id, 1) + .await + .expect("retry attempt 1 exists"); + // The retry's own handle must resolve to the same concurrency settings, not + // just the retry policy — otherwise it would run with no concurrency_key. + let rs = from_handle(r1_handle, &db).await?; + let resolved = ConcurrencySettings::get( + rs.concurrency_settings + .expect("retry must carry concurrency settings forward"), + &db, + ) + .await?; + assert_eq!( + resolved.concurrency_key.as_deref(), + Some("f/system/test_script") + ); + assert_eq!(resolved.concurrent_limit, Some(1)); + assert!( + rs.retry_settings.is_some(), + "retry policy is still carried for further chaining" + ); + Ok(()) + } + + // ------------------------------------------------------------------ + // Per-occurrence terminal status (drives on_failure_times / on_recovery). + // Mirrors the exact query in windmill-ee-private jobs_ee::apply_schedule_handlers. + // ------------------------------------------------------------------ + // `is_retry` marks the seeded job as a native retry attempt (the explicit + // native_retry_attempt marker), the same signal `apply_schedule_handlers` + // keys off. Handlers / WAC inline children are seeded without it. + async fn seed_job( + db: &Pool, + id: Uuid, + parent: Option, + is_retry: bool, + status: &str, + ) { + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, trigger_kind, trigger, parent_job) + VALUES ($1, $2, 'script', $3, 'schedule', $4, $5)", + ) + .bind(id) + .bind(WS) + .bind(SCRIPT) + .bind(SCHED) + .bind(parent) + .execute(db) + .await + .unwrap(); + sqlx::query( + "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status) + VALUES ($1, $2, 1, $3::job_status)", + ) + .bind(id) + .bind(WS) + .bind(status) + .execute(db) + .await + .unwrap(); + if is_retry { + sqlx::query("INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, 1)") + .bind(id) + .execute(db) + .await + .unwrap(); + } + } + + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn per_occurrence_status_counts_recovered_as_success(db: Pool) { + // A native retry attempt is marked (native_retry_attempt); handler and WAC + // inline children are parented Script children WITHOUT the marker. + // a: root fails, a marked retry succeeds -> RECOVERED -> success + // b: root fails, retry also fails -> failure + // c: root succeeds directly -> success + // e: root fails, only its on_failure HANDLER (unmarked) succeeds + // -> failure: handler child must NOT count as a recovery + // f: root fails, only a WAC inline child (unmarked) succeeds + // -> failure: WAC inline child must NOT count as a recovery + // d: the current occurrence (excluded by `j.id != $4`) + let (a, b, c, e, f, d) = ( + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + ); + seed_job(&db, a, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(a), true, "success").await; // a's retry attempt (marked) + seed_job(&db, b, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(b), true, "failure").await; // b's failed retry + seed_job(&db, c, None, false, "success").await; + seed_job(&db, e, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(e), false, "success").await; // e's handler child (unmarked) + seed_job(&db, f, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(f), false, "success").await; // f's WAC inline child (unmarked) + seed_job(&db, d, None, false, "failure").await; // current occurrence + + // Exact expression from jobs_ee::apply_schedule_handlers: the EXISTS counts + // only marked native retry children, so neither handler nor WAC inline + // children count as a recovery. + let rows = sqlx::query_as::<_, (Uuid, bool)>( + "SELECT j.id, (status = 'success' OR EXISTS ( + SELECT 1 FROM native_retry_attempt nra + JOIN v2_job jc ON jc.id = nra.job_id + JOIN v2_job_completed cc ON cc.id = nra.job_id + WHERE jc.parent_job = j.id AND cc.status = 'success' + )) + FROM v2_job j JOIN v2_job_completed USING (id) + WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2 + AND parent_job IS NULL AND runnable_path = $3 AND j.id != $4 + ORDER BY created_at DESC", + ) + .bind(WS) + .bind(SCHED) + .bind(SCRIPT) + .bind(d) + .fetch_all(&db) + .await + .unwrap(); + + let status: std::collections::HashMap = rows.into_iter().collect(); + // Retries/handlers/WAC children (parent_job set) are NOT occurrences, and the + // current one is excluded: exactly the five roots a, b, c, e, f remain. + assert_eq!( + status.len(), + 5, + "child jobs excluded from occurrence counting; current excluded" + ); + assert_eq!( + status[&a], true, + "recovered occurrence (same-runnable retry succeeded) = success" + ); + assert_eq!( + status[&b], false, + "all-attempts-failed occurrence = failure" + ); + assert_eq!(status[&c], true, "direct success"); + assert_eq!( + status[&e], false, + "on_failure handler success must NOT count as a recovery" + ); + assert_eq!( + status[&f], false, + "WAC inline child success must NOT count as a recovery" + ); + } +} diff --git a/backend/windmill-queue/tests/schedule_push.rs b/backend/windmill-queue/tests/schedule_push.rs index c0e09003f0..70a53763e0 100644 --- a/backend/windmill-queue/tests/schedule_push.rs +++ b/backend/windmill-queue/tests/schedule_push.rs @@ -3,6 +3,9 @@ mod schedule_push { use sqlx::{Pool, Postgres}; use windmill_common::db::Authed; use windmill_common::jobs::{JobKind, JobTriggerKind}; + use windmill_common::runnable_settings::{ + from_handle, insert_rs, ConcurrencySettings, RunnableSettings, RunnableSettingsTrait, + }; use windmill_common::schedule::Schedule; use windmill_common::scripts::ScriptHash; use windmill_common::users::username_to_permissioned_as; @@ -233,13 +236,78 @@ mod schedule_push { assert_eq!(count_queued_jobs(&db).await, 1); - // When retry is set, the job kind is singlescriptflow (SingleStepFlow wraps it) - let kind = sqlx::query_scalar::<_, String>( - "SELECT kind::text FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + // Native retry: a scheduled script with a retry policy is pushed as a plain + // Script carrying the policy via runnable_settings_handle — no SingleStepFlow. + let (kind, handle) = sqlx::query_as::<_, (String, Option)>( + "SELECT kind::text, q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", ) .fetch_one(&db) .await?; - assert_eq!(kind, "singlestepflow"); + assert_eq!(kind, "script"); + assert!( + handle.is_some(), + "retry policy carried via runnable_settings_handle" + ); + Ok(()) + } + + // A scheduled, concurrency-limited script with a retry policy: the materialized + // root attempt's handle must resolve to BOTH the retry policy and the script's + // concurrency settings — otherwise the retry chain runs unbounded. Regression: P1. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn test_push_script_with_retry_keeps_concurrency( + db: Pool, + ) -> anyhow::Result<()> { + let concurrency = ConcurrencySettings { + concurrency_key: Some("f/system/test_script".to_string()), + concurrent_limit: Some(1), + concurrency_time_window_s: Some(60), + }; + let script_handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: concurrency.insert_cached(&db).await?, + retry_settings: None, + }, + &db, + ) + .await?; + sqlx::query( + "UPDATE script SET runnable_settings_handle = $1 WHERE workspace_id = 'test-workspace' AND path = 'f/system/test_script'", + ) + .bind(script_handle) + .execute(&db) + .await?; + + let schedule = make_schedule(|s| { + s.retry = Some(serde_json::json!({ "constant": { "attempts": 3, "seconds": 10 } })); + }); + let authed = make_authed(); + let tx = db.begin().await?; + let tx = push_scheduled_job(&db, tx, &schedule, Some(&authed), None).await?; + tx.commit().await?; + + let handle = sqlx::query_scalar::<_, Option>( + "SELECT q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + ) + .fetch_one(&db) + .await?; + let rs = from_handle(handle, &db).await?; + assert!( + rs.retry_settings.is_some(), + "root attempt carries the retry policy" + ); + let resolved = ConcurrencySettings::get( + rs.concurrency_settings + .expect("root attempt must carry concurrency settings, not just retry"), + &db, + ) + .await?; + assert_eq!(resolved.concurrent_limit, Some(1)); + assert_eq!( + resolved.concurrency_key.as_deref(), + Some("f/system/test_script") + ); Ok(()) } @@ -779,7 +847,7 @@ mod schedule_push { } // ----------------------------------------------------------------------- - // try_schedule_next_job: script with retry wraps in SingleStepFlow + // try_schedule_next_job: script with retry is a native Script (no wrapping) // ----------------------------------------------------------------------- #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] @@ -798,12 +866,16 @@ mod schedule_push { assert!(err.is_none()); assert_eq!(count_queued_jobs(&db).await, 1); - let kind = sqlx::query_scalar::<_, String>( - "SELECT kind::text FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + let (kind, handle) = sqlx::query_as::<_, (String, Option)>( + "SELECT kind::text, q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", ) .fetch_one(&db) .await?; - assert_eq!(kind, "singlestepflow"); + assert_eq!(kind, "script"); + assert!( + handle.is_some(), + "retry policy carried via runnable_settings_handle" + ); Ok(()) } diff --git a/backend/windmill-store/Cargo.toml b/backend/windmill-store/Cargo.toml index b3aca5e669..fdd82a7e5d 100644 --- a/backend/windmill-store/Cargo.toml +++ b/backend/windmill-store/Cargo.toml @@ -53,3 +53,7 @@ futures.workspace = true chrono.workspace = true reqwest.workspace = true anyhow.workspace = true +base64.workspace = true + +[dev-dependencies] +magic-crypt.workspace = true diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index 44db0a1b68..7da6d75f4b 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -1634,6 +1634,12 @@ async fn update_resource( let path = path.to_path(); check_scopes(&authed, || format!("resources:write:{}", path))?; + // A rename moves the resource (and its linked variable) to ns.path, so the + // destination must also be within the token's write scope, not just the + // source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("resources:write:{}", npath))?; + } if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, AuditAuthorable::username(&authed), diff --git a/backend/windmill-store/src/secret_backend_ext.rs b/backend/windmill-store/src/secret_backend_ext.rs index 1ae32c80ce..d5272384c0 100644 --- a/backend/windmill-store/src/secret_backend_ext.rs +++ b/backend/windmill-store/src/secret_backend_ext.rs @@ -6,269 +6,27 @@ * LICENSE-AGPL for a copy of the license. */ -//! Secret backend extension for the API layer +//! Secret backend extension for the store layer //! -//! This module provides helper functions for integrating the SecretBackend -//! trait with variable operations in the API. +//! Write-side helpers for integrating the SecretBackend trait with variable +//! operations. Backend resolution and read helpers live in +//! `windmill_common::secret_backend` (so lower-level crates can resolve secrets +//! too) and are re-exported here for existing callers. //! //! Note: HashiCorp Vault integration requires Enterprise Edition. //! The OSS version only supports the database backend. -use std::sync::Arc; - use windmill_common::{ db::DB, error::{Error, Result}, - secret_backend::{database::DatabaseBackend, SecretBackend}, - variables::{build_crypt, decrypt, encrypt}, + variables::{build_crypt, encrypt}, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::{ - global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, - secret_backend::{ - AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, - AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, - }, +pub use windmill_common::secret_backend::{ + get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value, + is_external_stored_value, is_vault_backend_configured, is_vault_stored_value, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use tokio::sync::RwLock; - -// Cached Vault backend to avoid recreating it for every request -// This enables connection pooling and avoids repeated setup overhead -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedVaultBackend { - backend: Arc, - settings: VaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAzureKvBackend { - backend: Arc, - settings: AzureKeyVaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached AWS Secrets Manager backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAwsSmBackend { - backend: Arc, - settings: AwsSecretsManagerSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -/// Get the current secret backend based on global settings -/// -/// OSS: Always returns DatabaseBackend -/// EE: Returns configured backend (Database or Vault) -#[cfg(not(all(feature = "private", feature = "enterprise")))] -pub async fn get_secret_backend(db: &DB) -> Result> { - Ok(Arc::new(DatabaseBackend::new(db.clone()))) -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -pub async fn get_secret_backend(db: &DB) -> Result> { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - match config { - SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), - SecretBackendConfig::HashiCorpVault(settings) => { - get_or_create_vault_backend(db, settings).await - } - SecretBackendConfig::AzureKeyVault(settings) => { - get_or_create_azure_kv_backend(db, settings).await - } - SecretBackendConfig::AwsSecretsManager(settings) => { - get_or_create_aws_sm_backend(db, settings).await - } - } -} - -/// Get a cached Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_vault_backend( - _db: &DB, - settings: VaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = VAULT_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = VAULT_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = { - #[cfg(feature = "openidconnect")] - if settings.token.is_none() { - Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) - } else { - Arc::new(VaultBackend::new(settings.clone())) - } - - #[cfg(not(feature = "openidconnect"))] - Arc::new(VaultBackend::new(settings.clone())) - }; - - // Cache it - *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached Azure Key Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_azure_kv_backend( - _db: &DB, - settings: AzureKeyVaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = AZURE_KV_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = AZURE_KV_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); - - // Cache it - *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached AWS SM backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_aws_sm_backend( - _db: &DB, - settings: AwsSecretsManagerSettings, -) -> Result> { - { - let cache = AWS_SM_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - let mut cache = AWS_SM_BACKEND_CACHE.write().await; - - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - let backend: Arc = - Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); - - *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Check if a Vault backend is currently configured -/// -/// OSS: Always returns false -/// EE: Checks global settings -#[cfg(not(all(feature = "private", feature = "enterprise")))] -pub async fn is_vault_backend_configured(_db: &DB) -> Result { - Ok(false) -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -pub async fn is_vault_backend_configured(db: &DB) -> Result { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - Ok(matches!( - config, - SecretBackendConfig::HashiCorpVault(_) - | SecretBackendConfig::AzureKeyVault(_) - | SecretBackendConfig::AwsSecretsManager(_) - )) -} - -/// Get a secret value using the configured backend -/// -/// For database backend: decrypts using workspace key -/// For vault backend (EE only): fetches from Vault directly -pub async fn get_secret_value( - db: &DB, - workspace_id: &str, - path: &str, - encrypted_value: &str, -) -> Result { - let backend = get_secret_backend(db).await?; - - match backend.backend_name() { - "database" => { - // Use existing database decryption - let mc = build_crypt(db, workspace_id).await?; - decrypt(&mc, encrypted_value.to_string()).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) - }) - } - "hashicorp_vault" => { - // Fetch from Vault directly - backend.get_secret(workspace_id, path).await - } - "azure_key_vault" => backend.get_secret(workspace_id, path).await, - "aws_secrets_manager" => backend.get_secret(workspace_id, path).await, - _ => Err(Error::internal_err(format!( - "Unknown backend: {}", - backend.backend_name() - ))), - } -} - /// Store a secret value using the configured backend /// /// For database backend: encrypts using workspace key and returns encrypted value @@ -412,26 +170,6 @@ pub async fn delete_secret_from_backend(db: &DB, workspace_id: &str, path: &str) } } -/// Check if a value is stored in Vault (indicated by the $vault: prefix) -pub fn is_vault_stored_value(value: &str) -> bool { - value.starts_with("$vault:") -} - -/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) -pub fn is_azure_kv_stored_value(value: &str) -> bool { - value.starts_with("$azure_kv:") -} - -/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix) -pub fn is_aws_sm_stored_value(value: &str) -> bool { - value.starts_with("$aws_sm:") -} - -/// Check if a value is stored in any external secret backend -pub fn is_external_stored_value(value: &str) -> bool { - is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) -} - /// Rename a secret in Vault when a variable path changes (EE only) #[cfg(not(all(feature = "private", feature = "enterprise")))] pub async fn rename_vault_secret( diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index 72ed20cb71..55047ec0c2 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -14,8 +14,8 @@ use windmill_common::db::DB; use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use crate::secret_backend_ext::{ - delete_secret_from_backend, get_secret_value, is_vault_stored_value, rename_vault_secret, - store_secret_value, + delete_secret_from_backend, get_secret_value, is_external_stored_value, is_vault_stored_value, + rename_vault_secret, store_secret_value, }; use windmill_common::utils::{escape_ilike_pattern, BulkDeleteRequest}; use windmill_common::webhook::{WebhookMessage, WebhookShared}; @@ -25,6 +25,7 @@ use axum::{ routing::{delete, get, post}, Json, Router, }; +use base64::{engine::general_purpose::STANDARD, Engine as _}; use futures::future::try_join_all; use hyper::StatusCode; use serde_json::Value; @@ -535,6 +536,35 @@ async fn check_path_conflict(db: &DB, w_id: &str, path: &str) -> Result<()> { return Ok(()); } +/// Reject a secret value flagged as already-encrypted (`already_encrypted=true`) +/// that is not actually workspace-key ciphertext — e.g. plaintext mistakenly +/// pushed as encrypted. Storing plaintext in the encrypted `value` column +/// silently bricks the variable: every later read fails to decrypt it. +/// +/// The check is purely structural and never decrypts, so it cannot act as a +/// decryption/padding oracle for a caller who can write but not read secrets. +/// `encrypt` (AES-256-CBC) always yields standard base64 decoding to a non-zero +/// multiple of the 16-byte block size; anything else cannot be our ciphertext. +/// Values stored by an external backend ($vault:/$aws_sm:/$azure_kv: markers) +/// are not workspace ciphertext and are passed through untouched. +fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> { + if is_external_stored_value(value) { + return Ok(()); + } + let looks_like_ciphertext = STANDARD + .decode(value) + .map(|bytes| !bytes.is_empty() && bytes.len() % 16 == 0) + .unwrap_or(false); + if !looks_like_ciphertext { + return Err(Error::BadRequest(format!( + "Variable {path} was sent as already-encrypted (already_encrypted=true) but its \ + value is not valid workspace-encrypted ciphertext. To push a plaintext secret, \ + send it without already_encrypted (CLI: use --plain-secrets) so it gets encrypted." + ))); + } + Ok(()) +} + async fn create_variable( authed: ApiAuthed, Extension(db): Extension, @@ -585,6 +615,11 @@ async fn create_variable( // Use secret backend for encryption (supports both DB and Vault) store_secret_value(&db, &w_id, &variable.path, &plain).await? } else { + if variable.is_secret { + // already_encrypted == true: value is stored verbatim, so it must be + // ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(&variable.path, &variable.value)?; + } variable.value }; @@ -1037,6 +1072,12 @@ async fn update_variable( let path = path.to_path(); check_scopes(&authed, || format!("variables:write:{}", path))?; + // A rename moves the (possibly secret) variable to ns.path, so the + // destination must also be within the token's write scope, not just the + // source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("variables:write:{}", npath))?; + } let authed = maybe_refresh_folders(&path, &w_id, authed, &db).await; let mut sqlb = SqlBuilder::update_table("variable"); @@ -1076,6 +1117,11 @@ async fn update_variable( // Store at target_path (new path if renaming, otherwise current path) store_secret_value(&db, &w_id, target_path, &plain).await? } else { + if is_secret { + // already_encrypted == true: value is stored verbatim, so it must + // be ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(target_path, &nvalue)?; + } nvalue }; sqlb.set_str("value", &value); @@ -1507,3 +1553,61 @@ pub async fn get_value_internal<'a>( Ok(r) } + +#[cfg(test)] +mod tests { + use super::*; + use magic_crypt::MagicCryptTrait; + + #[test] + fn accepts_real_workspace_ciphertext() { + // The exact shape produced by `encrypt` (AES-256-CBC, base64). + let mc = magic_crypt::new_magic_crypt!("a-test-workspace-key", 256); + for plain in [ + "", + "original-secret", + "some: plaintext\n", + "a".repeat(500).as_str(), + ] { + let ciphertext = mc.encrypt_str_to_base64(plain); + assert!( + validate_already_encrypted_secret("f/x/cfg", &ciphertext).is_ok(), + "should accept genuine ciphertext for plaintext {plain:?}: {ciphertext}" + ); + } + } + + #[test] + fn rejects_plaintext_mislabeled_as_encrypted() { + // Plaintext mislabeled as encrypted: storing it verbatim would make the + // variable undecryptable on every read, so it must be rejected. + for plaintext in [ + "some: plaintext\n", + "original-secret", + "hunter2", + "{\"a\": 1}", + "not base64!!", + " leading-space", + ] { + assert!( + validate_already_encrypted_secret("f/x/cfg", plaintext).is_err(), + "should reject plaintext mislabeled as encrypted: {plaintext:?}" + ); + } + } + + #[test] + fn rejects_empty_and_non_block_aligned() { + // Valid base64 but not a whole number of AES blocks -> cannot be our ciphertext. + assert!(validate_already_encrypted_secret("p", "").is_err()); + assert!(validate_already_encrypted_secret("p", "dGVzdA==").is_err()); // "test" -> 4 bytes + } + + #[test] + fn passes_through_external_backend_markers() { + // External secret backends store $-prefixed markers, not workspace ciphertext. + for marker in ["$vault:f/x/cfg", "$aws_sm:f/x/cfg", "$azure_kv:f/x/cfg"] { + assert!(validate_already_encrypted_secret("f/x/cfg", marker).is_ok()); + } + } +} diff --git a/backend/windmill-trigger-http/src/handler.rs b/backend/windmill-trigger-http/src/handler.rs index 7b617e1b8d..17321b7775 100644 --- a/backend/windmill-trigger-http/src/handler.rs +++ b/backend/windmill-trigger-http/src/handler.rs @@ -7,7 +7,7 @@ use axum::{extract::Path, routing::post, Extension, Json, Router}; use http::StatusCode; use sqlx::PgConnection; use std::collections::HashSet; -use windmill_api_auth::ApiAuthed; +use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_audit::{audit_oss::audit_log, ActionKind}; use windmill_common::global_settings::HTTP_ROUTE_WORKSPACED_ROUTE; use windmill_common::{ @@ -262,6 +262,12 @@ pub async fn create_many_http_triggers( let mut route_path_keys = Vec::with_capacity(new_http_triggers.len()); for new_http_trigger in new_http_triggers.iter() { + // Per-item write scope, matching the single-create handler. The bulk + // endpoint must not let a path-scoped token create triggers outside it. + check_scopes(&authed, || { + format!("http_triggers:write:{}", &new_http_trigger.base.path) + })?; + handler .validate_new(&db, &w_id, &new_http_trigger.config) .await @@ -373,7 +379,8 @@ impl TriggerCrud for HttpTrigger { const TABLE_NAME: &'static str = "http_trigger"; const TRIGGER_TYPE: &'static str = "http"; - const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerHttp; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = + windmill_common::user_drafts::UserDraftItemKind::TriggerHttp; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/http_triggers"; diff --git a/backend/windmill-trigger-websocket/src/handler.rs b/backend/windmill-trigger-websocket/src/handler.rs index ed8e95b164..529e5902cc 100644 --- a/backend/windmill-trigger-websocket/src/handler.rs +++ b/backend/windmill-trigger-websocket/src/handler.rs @@ -4,7 +4,7 @@ use async_trait::async_trait; use itertools::Itertools; use serde_json::value::RawValue; use sqlx::{types::Json as SqlxJson, PgConnection}; -use windmill_api_auth::ApiAuthed; +use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_common::DB; use windmill_common::{ db::UserDB, @@ -15,10 +15,39 @@ use windmill_git_sync::DeployedObject; use windmill_trigger::{Trigger, TriggerCrud, TriggerData}; use super::{ - get_url_from_runnable_value, proxy::connect_async_with_proxy, validate_websocket_url_for_ssrf, - TestWebsocketConfig, WebsocketConfig, WebsocketConfigRequest, WebsocketTrigger, + get_url_from_runnable_value, listener::InitialMessage, proxy::connect_async_with_proxy, + validate_websocket_url_for_ssrf, TestWebsocketConfig, WebsocketConfig, WebsocketConfigRequest, + WebsocketTrigger, }; +/// A websocket_triggers:write token can configure secondary runnables that the +/// listener later executes under the trigger owner's identity: a `$flow:`/ +/// `$script:` URL resolver and `initial_messages` of kind `runnable_result`. +/// That execution happens in a background task where the reconstructed authed is +/// scopeless (so its check_scopes is a no-op), so enforce run scope here, at +/// create/update time, against the API caller's token. +fn check_secondary_runnable_scopes( + authed: &ApiAuthed, + config: &WebsocketConfigRequest, +) -> Result<()> { + if let Some(rest) = config.url.strip_prefix("$flow:") { + check_scopes(authed, || format!("jobs:run:flows:{}", rest))?; + } else if let Some(rest) = config.url.strip_prefix("$script:") { + check_scopes(authed, || format!("jobs:run:scripts:{}", rest))?; + } + if let Some(messages) = config.initial_messages.as_ref() { + for msg in messages { + if let Ok(InitialMessage::RunnableResult { path, is_flow, .. }) = + serde_json::from_value::(msg.clone()) + { + let kind = if is_flow { "flows" } else { "scripts" }; + check_scopes(authed, || format!("jobs:run:{}:{}", kind, path))?; + } + } + } + Ok(()) +} + #[async_trait] impl TriggerCrud for WebsocketTrigger { type TriggerConfig = WebsocketConfig; @@ -101,6 +130,7 @@ impl TriggerCrud for WebsocketTrigger { w_id: &str, trigger: TriggerData, ) -> Result<()> { + check_secondary_runnable_scopes(authed, &trigger.config)?; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); let filters = trigger @@ -178,6 +208,7 @@ impl TriggerCrud for WebsocketTrigger { path: &str, trigger: TriggerData, ) -> Result<()> { + check_secondary_runnable_scopes(authed, &trigger.config)?; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); let filters = trigger diff --git a/backend/windmill-trigger-websocket/src/listener.rs b/backend/windmill-trigger-websocket/src/listener.rs index 31e3f83995..d46faf1b1a 100644 --- a/backend/windmill-trigger-websocket/src/listener.rs +++ b/backend/windmill-trigger-websocket/src/listener.rs @@ -509,7 +509,7 @@ impl Clone for ReturnMessageChannels { } #[derive(Debug, Deserialize)] -enum InitialMessage { +pub(crate) enum InitialMessage { #[serde(rename = "raw_message")] RawMessage(String), #[serde(rename = "runnable_result")] diff --git a/backend/windmill-trigger/src/global_handler.rs b/backend/windmill-trigger/src/global_handler.rs index 3fbbcddd0e..98b689dda9 100644 --- a/backend/windmill-trigger/src/global_handler.rs +++ b/backend/windmill-trigger/src/global_handler.rs @@ -14,7 +14,7 @@ use windmill_api_jobs::execution::cancel_jobs; use windmill_common::{ db::{UserDB, DB}, error::{self, Error, Result}, - jobs::JobTriggerKind, + jobs::{delete_jobs, JobTriggerKind}, triggers::TriggerMetadata, }; @@ -262,9 +262,7 @@ pub async fn resume_suspended_trigger_jobs( .execute(&mut *tx) .await?; - sqlx::query!("DELETE FROM v2_job WHERE id = $1", job.id) - .execute(&mut *tx) - .await?; + delete_jobs(&mut *tx, &[job.id]).await?; } } diff --git a/backend/windmill-trigger/src/trigger_helpers.rs b/backend/windmill-trigger/src/trigger_helpers.rs index 44e5884420..1427f7f102 100644 --- a/backend/windmill-trigger/src/trigger_helpers.rs +++ b/backend/windmill-trigger/src/trigger_helpers.rs @@ -924,6 +924,8 @@ async fn trigger_script_with_retry_and_error_handler<'c>( path, hash: Some(hash), flow_version: None, + // Keep the flow path until native retry covers handler semantics. + language: None, args: HashMap::from(&push_args), retry, error_handler_path, diff --git a/backend/windmill-types/src/flows.rs b/backend/windmill-types/src/flows.rs index 2d1a538124..006185dbd9 100644 --- a/backend/windmill-types/src/flows.rs +++ b/backend/windmill-types/src/flows.rs @@ -155,6 +155,19 @@ fn validate_retry(retry: &Retry, module_id: &str) -> anyhow::Result<()> { Ok(()) } +/// Script/sub-flow step references must be workspace paths (`u/`, `f/`, `g/`) or a hub +/// reference (`hub/`). Empty is tolerated for intermediate/incomplete steps. This blocks +/// absolute or local filesystem paths (e.g. `/tmp/.../ops/scripts/...` baked in by a +/// `wmill sync push` from a feature-branch checkout) from being persisted into a flow, +/// where they silently mis-resolve to an unrelated script at runtime (#9751). +fn is_workspace_runnable_path(path: &str) -> bool { + path.is_empty() + || path.starts_with("u/") + || path.starts_with("f/") + || path.starts_with("g/") + || path.starts_with("hub/") +} + fn validate_flow_value<'de, D>(deserializer: D) -> Result, D::Error> where D: Deserializer<'de>, @@ -164,21 +177,38 @@ where let flow_value: FlowValue = serde_json::from_str(raw_value.get()) .map_err(|e| serde::de::Error::custom(format!("Invalid flow value: {}", e)))?; - FlowModule::traverse_modules(&flow_value.modules, &mut |module| { + let mut validate_module = |module: &FlowModule| -> anyhow::Result<()> { if let Some(ref retry) = module.retry { validate_retry(retry, &module.id)?; } - return Ok(()); - }) - .map_err(|e| serde::de::Error::custom(e.to_string()))?; + if let Ok(FlowModuleValue::Script { path, .. } | FlowModuleValue::Flow { path, .. }) = + module.get_value() + { + if !is_workspace_runnable_path(&path) { + return Err(anyhow::anyhow!( + "step '{}' references '{}', which is not a workspace path (expected u/, \ + f/, g/ or hub/). Absolute or local filesystem paths are not allowed in \ + flow steps.", + module.id, + path + )); + } + } + Ok(()) + }; - if let Some(ref _failure_module) = flow_value.failure_module { - //add validation logic here for failure module - } - - if let Some(ref _preprocessor_module) = flow_value.preprocessor_module { - //add validation logic here for preprocessor module - } + // The API is the authoritative guard (it can be called directly, bypassing the CLI), so + // it must cover every step that resolves a path: the main modules AND the failure / + // preprocessor modules (which can themselves be sub-flows/loops/branches). + let extra_modules: Vec = flow_value + .failure_module + .iter() + .chain(flow_value.preprocessor_module.iter()) + .map(|m| (**m).clone()) + .collect(); + FlowModule::traverse_modules(&flow_value.modules, &mut validate_module) + .and_then(|()| FlowModule::traverse_modules(&extra_modules, &mut validate_module)) + .map_err(|e| serde::de::Error::custom(e.to_string()))?; Ok(raw_value) } @@ -1228,6 +1258,108 @@ mod tests { assert_eq!(val.modules.len(), 1); } + #[test] + fn flow_rejects_absolute_step_path() { + // #9751: an absolute local path baked into a step must be rejected on deploy. + let bad = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "validate_onboard_target", + "value": { + "type": "script", + "path": "/tmp/tmp.X/f/ops/scripts/clean_device/pre_clean", + "input_transforms": {} + } + }]} + }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "unexpected error: {err}" + ); + assert!( + err.contains("validate_onboard_target"), + "error should name the step: {err}" + ); + } + + #[test] + fn flow_rejects_absolute_step_path_in_nested_module() { + let bad = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "loop", + "value": { + "type": "forloopflow", + "iterator": {"type": "javascript", "expr": "[1]"}, + "modules": [{ + "id": "inner", + "value": {"type": "script", "path": "/abs/path", "input_transforms": {}} + }] + } + }]} + }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "unexpected error: {err}" + ); + } + + #[test] + fn flow_rejects_absolute_path_in_failure_and_preprocessor_modules() { + for slot in ["failure_module", "preprocessor_module"] { + // Build the value with the slot as an explicit (interpolated) key. + let mut value = serde_json::Map::new(); + value.insert("modules".to_string(), json!([])); + value.insert( + slot.to_string(), + json!({ + "id": slot, + "value": {"type": "script", "path": "/abs/path", "input_transforms": {}} + }), + ); + let bad = json!({ "path": "f/test/flow", "summary": "", "value": value }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "{slot} should be validated, got: {err}" + ); + } + } + + #[test] + fn flow_accepts_workspace_step_paths() { + for p in [ + "f/ops/scripts/x", + "u/me/y", + "g/grp/z", + "hub/123/foo", + "", // tolerated for incomplete steps + ] { + let ok = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "a", + "value": {"type": "script", "path": p, "input_transforms": {}} + }]} + }); + assert!( + serde_json::from_value::(ok).is_ok(), + "path {p:?} should be accepted" + ); + } + } + #[test] fn ai_agent_omit_output_from_conversation_defaults_to_false() { let input = json!({ diff --git a/backend/windmill-types/src/jobs.rs b/backend/windmill-types/src/jobs.rs index ac15430759..448e6f4d01 100644 --- a/backend/windmill-types/src/jobs.rs +++ b/backend/windmill-types/src/jobs.rs @@ -232,6 +232,14 @@ pub struct QueuedJob { pub runnable_settings_handle: Option, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + // True when this job is a native retry attempt (has a native_retry_attempt + // marker). Lets the run-page chain distinguish real retries from other + // same-script children (e.g. WAC inline children). The list and single-job + // GET endpoints select it; `#[sqlx(default)]` lets any other query omit the + // column and default to None. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_retry: Option, } impl QueuedJob { @@ -307,6 +315,7 @@ impl Default for QueuedJob { preprocessed: None, runnable_settings_handle: None, labels: None, + is_retry: None, } } } @@ -362,6 +371,14 @@ pub struct CompletedJob { pub labels: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub preprocessed: Option, + // True when this job is a native retry attempt (has a native_retry_attempt + // marker). Lets the run-page chain distinguish real retries from other + // same-script children (e.g. WAC inline children). The list and single-job + // GET endpoints select it; `#[sqlx(default)]` lets any other query omit the + // column and default to None. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_retry: Option, } impl CompletedJob { @@ -473,6 +490,10 @@ pub enum JobPayload { path: String, hash: Option, flow_version: Option, + // Set when wrapping a script (not a flow). Lets `push` materialize a + // bare-script-with-retry as a native retryable `Script` job instead of + // spawning a one-step flow. + language: Option, args: HashMap>, retry: Option, error_handler_path: Option, diff --git a/backend/windmill-types/src/runnable_settings.rs b/backend/windmill-types/src/runnable_settings.rs index dfff0df79e..ea0fd30dc0 100644 --- a/backend/windmill-types/src/runnable_settings.rs +++ b/backend/windmill-types/src/runnable_settings.rs @@ -1,9 +1,75 @@ use serde::{Deserialize, Serialize}; +use crate::flows::{ConstantDelay, ExponentialDelay, Retry, RetryIf}; + #[derive(Deserialize, Clone, Copy, Serialize, Default, Hash)] pub struct RunnableSettings { pub debouncing_settings: Option, pub concurrency_settings: Option, + pub retry_settings: Option, +} + +/// Flattened, dedup-friendly representation of a [`Retry`] policy. Native script +/// retry stores the policy here (via `runnable_settings_handle`) instead of +/// wrapping the script in a one-step flow. +#[derive( + Debug, Default, Clone, Serialize, Deserialize, Hash, PartialEq, sqlx::FromRow, sqlx::Decode, +)] +pub struct RetrySettings { + #[serde(skip_serializing_if = "Option::is_none")] + pub constant_attempts: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub constant_seconds: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_attempts: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_multiplier: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_seconds: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_random_factor: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub retry_if_expr: Option, +} + +impl From<&Retry> for RetrySettings { + fn from(r: &Retry) -> Self { + Self { + // attempts are u32; saturate the narrowing to i32 (the seconds/ + // multiplier/random_factor fields are u16/i8 and can't overflow i32). + constant_attempts: Some(r.constant.attempts.min(i32::MAX as u32) as i32), + constant_seconds: Some(r.constant.seconds as i32), + exponential_attempts: Some(r.exponential.attempts.min(i32::MAX as u32) as i32), + exponential_multiplier: Some(r.exponential.multiplier as i32), + exponential_seconds: Some(r.exponential.seconds as i32), + exponential_random_factor: r.exponential.random_factor.map(|x| x as i32), + retry_if_expr: r.retry_if.as_ref().map(|x| x.expr.clone()), + } + } +} + +impl From for Retry { + fn from(s: RetrySettings) -> Self { + Retry { + constant: ConstantDelay { + attempts: s.constant_attempts.unwrap_or(0).max(0) as u32, + seconds: s.constant_seconds.unwrap_or(0).clamp(0, u16::MAX as i32) as u16, + }, + exponential: ExponentialDelay { + attempts: s.exponential_attempts.unwrap_or(0).max(0) as u32, + // Mirror ExponentialDelay::default().multiplier (1) when absent. + multiplier: s + .exponential_multiplier + .unwrap_or(1) + .clamp(0, u16::MAX as i32) as u16, + seconds: s.exponential_seconds.unwrap_or(0).clamp(0, u16::MAX as i32) as u16, + random_factor: s + .exponential_random_factor + .map(|x| x.clamp(i8::MIN as i32, i8::MAX as i32) as i8), + }, + retry_if: s.retry_if_expr.map(|expr| RetryIf { expr }), + } + } } // TODO: Add validation logic. @@ -111,3 +177,56 @@ impl From for ConcurrencySettings { } } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::flows::{ConstantDelay, ExponentialDelay, RetryIf}; + + #[test] + fn retry_settings_roundtrips_retry() { + let cases = [ + // constant only + Retry { + constant: ConstantDelay { attempts: 3, seconds: 5 }, + exponential: ExponentialDelay::default(), + retry_if: None, + }, + // exponential with jitter + Retry { + constant: ConstantDelay::default(), + exponential: ExponentialDelay { + attempts: 4, + multiplier: 2, + seconds: 3, + random_factor: Some(20), + }, + retry_if: None, + }, + // mixed + retry_if + max-ish narrowings + Retry { + constant: ConstantDelay { attempts: 1, seconds: u16::MAX }, + exponential: ExponentialDelay { + attempts: 2, + multiplier: u16::MAX, + seconds: 7, + random_factor: Some(i8::MIN), + }, + retry_if: Some(RetryIf { expr: "result.error.code != 'fatal'".to_string() }), + }, + ]; + for r in cases { + let back: Retry = RetrySettings::from(&r).into(); + assert_eq!(back, r, "RetrySettings round-trip must preserve {r:?}"); + } + } + + #[test] + fn retry_settings_default_maps_to_default_exponential() { + // All-None settings must mirror ExponentialDelay::default() (multiplier 1), + // so a row with no exponential values doesn't decode to a 0 multiplier. + let r: Retry = RetrySettings::default().into(); + assert_eq!(r, Retry::default()); + assert_eq!(r.exponential.multiplier, 1); + } +} diff --git a/backend/windmill-worker/Cargo.toml b/backend/windmill-worker/Cargo.toml index 14c3698162..14aeb734ed 100644 --- a/backend/windmill-worker/Cargo.toml +++ b/backend/windmill-worker/Cargo.toml @@ -21,6 +21,7 @@ bigquery = ["dep:gcp_auth"] benchmark = ["windmill-queue/benchmark", "windmill-common/benchmark"] parquet = ["windmill-common/parquet", "windmill-object-store/parquet"] flow_testing = [] +failpoints = [] cloud = [] sqlx = [] deno_core = ["dep:windmill-runtime-nativets"] @@ -38,7 +39,7 @@ java = ["dep:windmill-parser-java"] ruby = ["dep:windmill-parser-ruby"] rlang = ["dep:windmill-parser-r"] duckdb = ["dep:libloading"] -quickjs = ["windmill-jseval/quickjs"] +quickjs = ["windmill-jseval/quickjs", "windmill-queue/quickjs"] bedrock = ["windmill-ai/bedrock"] [dependencies] @@ -114,7 +115,8 @@ hmac.workspace = true pem = { workspace = true, optional = true } rsa = { workspace = true, optional = true } urlencoding.workspace = true -nix.workspace = true +# `fs` adds flock(2) for the cross-process Python install lock (shared cache mounts) +nix = { workspace = true, features = ["fs"] } bytes.workspace = true reqwest.workspace = true reqwest-middleware.workspace = true diff --git a/backend/windmill-worker/src/ansible_executor.rs b/backend/windmill-worker/src/ansible_executor.rs index dec0623e36..2758fe85ff 100644 --- a/backend/windmill-worker/src/ansible_executor.rs +++ b/backend/windmill-worker/src/ansible_executor.rs @@ -13,7 +13,7 @@ use tokio::process::Command; use uuid::Uuid; use windmill_common::{ error, - git_sync_oss::prepend_token_to_github_url, + git_sync_oss::{prepend_token_to_github_url, sanitize_git_url}, worker::{ is_allowed_file_location, split_python_requirements, to_raw_value, write_file, write_file_at_user_defined_location, Connection, PyVAlias, WORKER_CONFIG, @@ -847,7 +847,7 @@ pub async fn get_git_repo_full_head_commit_hash( .first() .ok_or(anyhow!( "The HEAD commit hash was not found for repo `{}`", - &repo.url + sanitize_git_url(&repo.url) ))? .split_whitespace() .next() @@ -1254,7 +1254,12 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } else { clone_repo( &repo, @@ -1269,7 +1274,12 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } append_logs( @@ -1316,7 +1326,7 @@ pub async fn handle_ansible_job( append_logs( &job.id, &job.workspace_id, - format!("\nCloning {}...\n", &repo.url), + format!("\nCloning {}...\n", sanitize_git_url(&repo.url)), conn, ) .await; @@ -1338,13 +1348,18 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } else { if req_lockfiles.is_some() { append_logs( &job.id, &job.workspace_id, - format!("Warning: `{}` is using latest commit because the lockfile didn't store a commit hash for this repo. Updates to the repo could break the deployed playbook.\n", &repo.url), + format!("Warning: `{}` is using latest commit because the lockfile didn't store a commit hash for this repo. Updates to the repo could break the deployed playbook.\n", sanitize_git_url(&repo.url)), conn, ) .await; @@ -1362,13 +1377,22 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } append_logs( &job.id, &job.workspace_id, - format!("Cloned {} into {}\n", &repo.url, &repo.target_path), + format!( + "Cloned {} into {}\n", + sanitize_git_url(&repo.url), + &repo.target_path + ), conn, ) .await; diff --git a/backend/windmill-worker/src/deno_executor.rs b/backend/windmill-worker/src/deno_executor.rs index c39e9e999e..a446548e1f 100644 --- a/backend/windmill-worker/src/deno_executor.rs +++ b/backend/windmill-worker/src/deno_executor.rs @@ -434,7 +434,11 @@ try {{ if let Some(ref npmrc_content) = npmrc { if !npmrc_content.trim().is_empty() { write_file(job_dir, ".npmrc", npmrc_content)?; - write_file(job_dir, "deno.json", "{}")?; + // minimumDependencyAge=0 opts out of Deno's supply-chain guard that rejects + // npm packages published within the last ~24h. Private/internal registries + // routinely serve just-published versions, so the guard would break them. + // Older Deno ignores the unknown field, so this is safe across versions. + write_file(job_dir, "deno.json", r#"{"minimumDependencyAge":"0"}"#)?; } } diff --git a/backend/windmill-worker/src/duckdb_executor.rs b/backend/windmill-worker/src/duckdb_executor.rs index 3b6a07f702..e4b94acabf 100644 --- a/backend/windmill-worker/src/duckdb_executor.rs +++ b/backend/windmill-worker/src/duckdb_executor.rs @@ -39,6 +39,47 @@ struct MaterializeExec { asset_kind: windmill_common::assets::AssetKind, asset_path: String, partition: String, + // Number of `// data_test` checks the codegen embedded. Enforcement recovers + // the per-test outcomes from the summary row; if it recovers fewer than this + // (e.g. an FFI serialization change drops the column), we fail loud rather + // than silently pass declared-but-unverified tests. + n_data_tests: usize, +} + +// Fetch and validate a custom data-test script's body. v1 custom tests are +// DuckDB scripts holding a single SELECT/CTE that returns the violating rows +// (dbt's singular-test convention); the worker embeds that query as a subquery +// check in the materialize connection (the single-statement constraint is +// enforced in sql_materialize.rs). Server workers only — agent (Http) workers +// have no script cache to read deployed content from. +async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Result { + let Connection::Sql(db) = conn else { + return Err(Error::ExecutionErr(format!( + "data_test custom `{path}`: custom tests require a server worker (not supported on \ + agent workers in v1)" + ))); + }; + let hash = windmill_common::get_latest_script_hash(db, path, w_id) + .await? + .ok_or_else(|| { + Error::ExecutionErr(format!( + "data_test custom `{path}`: no deployed script found at this path" + )) + })?; + let content = + crate::get_script_content_by_hash(&windmill_common::scripts::ScriptHash(hash), w_id, conn) + .await?; + if !matches!( + content.language, + Some(windmill_common::scripts::ScriptLang::DuckDb) + ) { + return Err(Error::ExecutionErr(format!( + "data_test custom `{path}`: must be a DuckDB script returning the violating rows \ + (got language {:?})", + content.language + ))); + } + Ok(content.content) } // If `query` declares `// materialize `, return what to record plus, @@ -46,22 +87,47 @@ struct MaterializeExec { // mode the script writes its own DDL, so the rewrite is `None`). The rewritten // SQL contains a synthetic `ATTACH 'ducklake://' AS _wm_target` that the // normal ATTACH-transform pass resolves to real credentials — the same path as -// the user's own ATTACH. Returns `None` when there is no materialize annotation -// or the target isn't a ducklake (only ducklake is materialized in v1). +// the user's own ATTACH. `// data_test` lines append verifier probes that run +// against the freshly-materialized target and raise (failing the run) on +// violation. Returns `None` when there is no materialize annotation or the +// target isn't a ducklake (only ducklake is materialized in v1). fn build_materialized_query( query: &str, partition_value: Option<&str>, + // Custom (`// data_test `) test bodies, pre-fetched by the caller + // (`fetch_custom_test_bodies`) so this stays pure/sync and unit-testable — + // the DB read is the only thing that needs a connection. Keyed by script path. + custom_test_bodies: &std::collections::HashMap, ) -> Result, MaterializeExec)>> { - use windmill_parser::asset_parser::{parse_pipeline_annotations, AssetKind as PAssetKind}; + use windmill_parser::asset_parser::{ + parse_pipeline_annotations, AssetKind as PAssetKind, DataTest, + }; use windmill_parser::sql_materialize::{ - build_wrap_blocks, classify_wrap, MaterializeStrategy, TARGET_ALIAS, + build_wrap_blocks, DataTestResolved, MaterializeStrategy, TARGET_ALIAS, }; let ann = parse_pipeline_annotations(query); + let has_tests = !ann.data_tests.is_empty(); let Some(m) = ann.materialize else { + // Data tests run *against the materialized asset*; without a + // `// materialize` target there is nothing to test. Fail loudly rather + // than silently skip the declared checks. + if has_tests { + return Err(Error::ExecutionErr( + "data_test: requires a `// materialize` target — data tests run against the \ + materialized asset" + .to_string(), + )); + } return Ok(None); }; if m.target_kind != PAssetKind::Ducklake { + if has_tests { + return Err(Error::ExecutionErr( + "data_test: only `ducklake://` materialization targets support data tests in v1" + .to_string(), + )); + } return Ok(None); } let partitioned = ann.partition.is_some(); @@ -86,10 +152,36 @@ fn build_materialized_query( asset_kind: windmill_common::assets::AssetKind::Ducklake, asset_path: m.target_path.clone(), partition: partition.clone(), + n_data_tests: ann.data_tests.len(), + }; + + // `{partition}` → escaped SQL literal substitution, applied to the managed + // SELECT, its setup, and any custom-test body so a partitioned test can + // filter by the active slice. Always a complete `'…'` literal (with `'` + // doubled) whether or not the author quoted it, so a run caller can't break + // out and alter statement boundaries. The pre-quoted `'{partition}'` form is + // matched first so it doesn't become `''…''`. No-op when unpartitioned. + let lit = format!("'{}'", partition.replace('\'', "''")); + let substitute = |s: &str| -> String { + if !partitioned { + return s.to_string(); + } + let tok = windmill_common::assets::PARTITION_TOKEN; + let quoted_tok = format!("'{tok}'"); + s.replace("ed_tok, &lit).replace(tok, &lit) }; if m.manual { - // Escape hatch: the script owns its DDL; we only record state. + // Escape hatch: the script owns its DDL. We can't reliably attach the + // managed target or know the partition column it wrote, so data tests + // are not generated for manual mode in v1. + if has_tests { + return Err(Error::ExecutionErr( + "data_test: not supported with `// materialize manual` in v1 — use managed \ + `// materialize`" + .to_string(), + )); + } return Ok(Some((None, meta))); } if table.is_empty() { @@ -98,24 +190,10 @@ fn build_materialized_query( m.target_path ))); } - let mut plan = classify_wrap(query).map_err(|e| Error::ExecutionErr(e.message()))?; - // Resolve the `{partition}` token (same token `// on` asset URIs use) to the - // current partition value everywhere in the managed script, so a partitioned - // materialize can filter its source by the active slice, e.g. - // `WHERE day = {partition}`. The token is always replaced by a *complete* - // escaped SQL literal (`'…'` with `'` doubled) whether or not the author - // quoted it — so a run caller can't pass metacharacters that break out of - // the literal and alter statement boundaries. The pre-quoted form - // `'{partition}'` is matched first so it doesn't become `''…''`. Only - // meaningful when partitioned. - if partitioned { - let lit = format!("'{}'", partition.replace('\'', "''")); - let tok = windmill_common::assets::PARTITION_TOKEN; - let quoted_tok = format!("'{tok}'"); - plan.output = plan.output.replace("ed_tok, &lit).replace(tok, &lit); - for s in plan.setup.iter_mut() { - *s = s.replace("ed_tok, &lit).replace(tok, &lit); - } + let mut plan = classify_wrap_or_err(query)?; + plan.output = substitute(&plan.output); + for s in plan.setup.iter_mut() { + *s = substitute(s); } let strategy = if m.append { MaterializeStrategy::Append @@ -126,8 +204,30 @@ fn build_materialized_query( }; // Inline the partition as an escaped SQL literal (DuckLake has no bind for // the partition column in our generated DDL). - let pval = format!("'{}'", partition.replace('\'', "''")); + let pval = lit.clone(); let synthetic_attach = format!("ATTACH 'ducklake://{ducklake_name}' AS {TARGET_ALIAS};"); + + // Resolve data tests (fetch + partition-substitute custom bodies) so codegen + // can embed every check's violating-row count in the materialize summary. + // The summary then carries the full per-test breakdown back to the worker, + // which runs them all and decides pass/fail (no abort-on-first). Empty when + // there are no `// data_test` lines. + let mut resolved = Vec::with_capacity(ann.data_tests.len()); + for test in &ann.data_tests { + match test { + DataTest::Custom { path } => { + let raw = custom_test_bodies.get(path).ok_or_else(|| { + Error::ExecutionErr(format!( + "data_test custom `{path}`: body not fetched before codegen (internal)" + )) + })?; + resolved + .push(DataTestResolved::Custom { path: path.clone(), body: substitute(raw) }); + } + other => resolved.push(DataTestResolved::BuiltIn(other.clone())), + } + } + let blocks = build_wrap_blocks( &plan, &synthetic_attach, @@ -137,10 +237,44 @@ fn build_materialized_query( &pval, partitioned, strategy, - ); + &resolved, + ) + .map_err(Error::ExecutionErr)?; + Ok(Some((Some(blocks.join("\n")), meta))) } +// Fetch the deployed body of every `// data_test ` custom test declared in +// `query`, keyed by path, so the sync `build_materialized_query` can splice them +// in. The DB read is the only part of materialize codegen that needs a +// connection; isolating it here keeps the codegen pure and unit-testable. +// Server workers only (`fetch_custom_test_body` errors on agent workers). Empty +// when there are no custom tests. +async fn fetch_custom_test_bodies( + query: &str, + conn: &Connection, + w_id: &str, +) -> Result> { + use windmill_parser::asset_parser::{parse_pipeline_annotations, DataTest}; + let ann = parse_pipeline_annotations(query); + let mut bodies = std::collections::HashMap::new(); + for test in &ann.data_tests { + if let DataTest::Custom { path } = test { + if !bodies.contains_key(path) { + let body = fetch_custom_test_body(conn, w_id, path).await?; + bodies.insert(path.clone(), body); + } + } + } + Ok(bodies) +} + +// classify_wrap with the spec's actionable message turned into an executor error. +fn classify_wrap_or_err(query: &str) -> Result { + windmill_parser::sql_materialize::classify_wrap(query) + .map_err(|e| Error::ExecutionErr(e.message())) +} + // Pull a named i64 field (`snapshot_id` / `rows`) out of the trailing summary // read — which in wrap mode is the job result. Shape-tolerant (object / array / // nested), returns None if absent (literal mode, or capture failed). @@ -156,6 +290,112 @@ fn extract_i64(result: &RawValue, field: &str) -> Option { find(&serde_json::from_str::(result.get()).ok()?, field) } +// One data test's outcome as carried by the materialize summary's `data_tests` +// column: its display name and how many rows violated it (0 = pass). +struct DataTestOutcome { + name: String, + violating: i64, +} + +// Pull the per-test breakdown out of the materialize summary result. The +// `data_tests` column is a DuckLake list-of-struct `[{test, violating}, …]`; +// the FFI may surface it as a nested JSON array or as a JSON string, so accept +// both. Returns empty when there are no tests (the column is absent). +fn extract_data_tests(result: &RawValue) -> Vec { + fn collect(v: &Value, out: &mut Vec) { + if let Value::Array(arr) = v { + for item in arr { + if let Value::Object(o) = item { + if let Some(Value::String(name)) = o.get("test") { + let violating = o + .get("violating") + .and_then(|x| x.as_i64().or_else(|| x.as_f64().map(|f| f as i64))) + .unwrap_or(0); + out.push(DataTestOutcome { name: name.clone(), violating }); + } + } + } + } + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("data_tests"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let mut out = Vec::new(); + let Ok(root) = serde_json::from_str::(result.get()) else { + return out; + }; + match find_field(&root) { + Some(arr @ Value::Array(_)) => collect(arr, &mut out), + // FFI serialized the list-of-struct as a JSON string — parse it. + Some(Value::String(s)) => { + if let Ok(parsed) = serde_json::from_str::(s) { + collect(&parsed, &mut out); + } + } + _ => {} + } + out +} + +// Pull the captured output schema out of the materialize summary's +// `output_schema` column (gap #2a): a list-of-struct `[{name, type}, …]` the +// codegen built from a `DESCRIBE`. Like `data_tests`, the FFI may surface it as +// a nested JSON array or a JSON string — accept both. Returns `None` when the +// column is absent (literal mode, manual mode, or capture failed) so the worker +// records the run without a schema rather than an empty one. +fn extract_schema( + result: &RawValue, +) -> Option> { + use windmill_common::materialization::SchemaColumn; + fn collect(v: &Value) -> Option> { + let Value::Array(arr) = v else { return None }; + let mut out = Vec::with_capacity(arr.len()); + for item in arr { + let o = item.as_object()?; + let name = o.get("name")?.as_str()?.to_string(); + let data_type = o.get("type")?.as_str()?.to_string(); + out.push(SchemaColumn { name, data_type }); + } + Some(out) + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("output_schema"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let root = serde_json::from_str::(result.get()).ok()?; + match find_field(&root)? { + arr @ Value::Array(_) => collect(arr), + // FFI serialized the list-of-struct as a JSON string — parse it. + Value::String(s) => collect(&serde_json::from_str::(s).ok()?), + _ => None, + } +} + +// Render the full pass/fail breakdown for a failed data-test run — every test, +// not just the first failure, so the user sees the whole picture in one place. +fn format_data_test_breakdown(asset_path: &str, tests: &[DataTestOutcome]) -> String { + let failed = tests.iter().filter(|t| t.violating > 0).count(); + let mut lines = vec![format!( + "data tests failed on {asset_path} ({failed}/{} failed):", + tests.len() + )]; + for t in tests { + if t.violating > 0 { + lines.push(format!(" ✗ {} — {} violating row(s)", t.name, t.violating)); + } else { + lines.push(format!(" ✓ {}", t.name)); + } + } + lines.join("\n") +} + // Best-effort record of a materialization outcome. On a Sql connection it writes // the row directly; on an agent worker (Http, no direct DB) it posts to the API // so state lands the same way. Never fails the job — a lost row degrades the @@ -168,6 +408,9 @@ async fn record_mat( status: windmill_common::materialization::MaterializationStatus, snapshot_id: Option, row_count: Option, + // Captured output schema (gap #2a). Only set on a successful materialize; + // when present, also upserts a `materialized_asset_schema` version. + schema: Option>, error: Option<&str>, ) { let req = windmill_common::materialization::RecordMaterializationRequest { @@ -179,22 +422,44 @@ async fn record_mat( row_count, job_id: Some(job_id), error: error.map(|e| e.to_string()), + schema: schema.clone(), }; let res: anyhow::Result<()> = match conn { - Connection::Sql(db) => windmill_common::materialization::record_materialization( - db, - w_id, - req.asset_kind, - &req.asset_path, - &req.partition, - req.status, - req.snapshot_id, - req.row_count, - req.job_id, - req.error.as_deref(), - ) - .await - .map_err(|e| anyhow::anyhow!("{e:#}")), + Connection::Sql(db) => { + let partition_res = windmill_common::materialization::record_materialization( + db, + w_id, + req.asset_kind, + &req.asset_path, + &req.partition, + req.status, + req.snapshot_id, + req.row_count, + req.job_id, + req.error.as_deref(), + ) + .await + .map_err(|e| anyhow::anyhow!("{e:#}")); + // Schema capture is a separate, independently best-effort write (its + // own transaction for the per-asset advisory lock); a failure here + // must not lose the partition row above. + if let Some(cols) = schema.as_ref() { + if let Err(e) = record_asset_schema_best_effort( + db, + w_id, + meta.asset_kind, + &meta.asset_path, + cols, + snapshot_id, + job_id, + ) + .await + { + tracing::warn!("failed to record captured asset schema: {e:#}"); + } + } + partition_res + } Connection::Http(client) => { crate::agent_workers::record_materialization_from_agent_http(client, w_id, &req).await } @@ -204,6 +469,33 @@ async fn record_mat( } } +// Open a short transaction (needed for the per-asset advisory lock) and upsert +// the captured schema version. Isolated so its tx lifetime doesn't entangle the +// partition write. +async fn record_asset_schema_best_effort( + db: &windmill_common::DB, + w_id: &str, + asset_kind: windmill_common::assets::AssetKind, + asset_path: &str, + columns: &[windmill_common::materialization::SchemaColumn], + snapshot_id: Option, + job_id: Uuid, +) -> anyhow::Result<()> { + let mut tx = db.begin().await?; + windmill_common::materialization::record_asset_schema( + &mut tx, + w_id, + asset_kind, + asset_path, + columns, + snapshot_id, + Some(job_id), + ) + .await?; + tx.commit().await?; + Ok(()) +} + pub async fn do_duckdb( job: &MiniPulledJob, client: &AuthedClient, @@ -250,11 +542,23 @@ pub async fn do_duckdb( .and_then(|a| a.0.get(windmill_common::partition::PARTITION_ARG)) .and_then(|rv| serde_json::from_str::(rv.get()).ok()) .filter(|s| !s.is_empty()); - let materialize = if query.contains("materialize") { - build_materialized_query(query, partition_value.as_deref())? + let materialize = if query.contains("materialize") || query.contains("data_test") { + // Custom-test bodies need a DB read; fetch them first so the codegen + // itself stays pure/sync. + let custom_test_bodies = + fetch_custom_test_bodies(query, conn, &job.workspace_id).await?; + build_materialized_query(query, partition_value.as_deref(), &custom_test_bodies)? } else { None }; + // Parse the signature from the ORIGINAL script: managed materialize wraps + // the trailing SELECT and strips line comments, which drops the + // `-- $name (type)` arg declarations while their `$name` references + // survive in the embedded SELECT. Parsing args here (pre-wrap) keeps them + // declared so they are still bound — and s3object args translated to + // `s3://` URIs — at run time. + let sig = parse_duckdb_sig(query)?.args; + let materialized_query; let query: &str = match &materialize { Some((Some(rewritten), _)) => { @@ -264,7 +568,18 @@ pub async fn do_duckdb( _ => query, }; - let sig = parse_duckdb_sig(query)?.args; + // Managed materialize generates its own trailing summary row (asset / + // rows / snapshot_id / data_tests), and data-test enforcement below reads + // the `data_tests` column off that row. The row shape is ours, not the + // user's — so force the full-last-row strategy regardless of any + // `// result_collection` annotation, which would otherwise reshape it + // (e.g. a scalar mode drops every column but the first) and silently + // bypass test enforcement. + let collection_strategy = if matches!(&materialize, Some((Some(_), _))) { + SqlResultCollectionStrategy::LastStatementAllRows + } else { + collection_strategy + }; let mut job_args = build_args_values(job, client, conn).await?; let reserved_variables = @@ -404,6 +719,7 @@ pub async fn do_duckdb( windmill_common::materialization::MaterializationStatus::Failed, None, None, + None, Some(&e.to_string()), ) .await; @@ -421,9 +737,70 @@ pub async fn do_duckdb( if let Some((_, meta)) = &materialize { // In wrap mode the job result is the summary read (snapshot_id + - // rows); in literal mode there is none, so both stay None. + // rows + the per-test breakdown); in literal mode there is none. let snapshot_id = extract_i64(&result, "snapshot_id"); let row_count = extract_i64(&result, "rows"); + // Data tests all ran (every check counted in one query); decide + // pass/fail here. Any violation fails the run — the write is already + // committed (like dbt), so the slice is recorded `Failed` and the + // cascade stops. The error lists *every* test so the user sees the + // whole picture, not just the first failure. + let tests = extract_data_tests(&result); + // Captured output schema (gap #2a) — recorded only on the successful + // path below, not on the failure paths (a failed run shouldn't + // advance the asset's recorded schema version). Managed mode ONLY: + // in `// materialize manual` the result is the user's own query + // output (we generate no summary), so an `output_schema` field there + // is caller-shaped and must not be trusted — `materialize` is + // `Some((Some(_), _))` for managed, `Some((None, _))` for manual. + let is_managed = matches!(&materialize, Some((Some(_), _))); + let schema = if is_managed { + extract_schema(&result) + } else { + None + }; + // Defense-in-depth: codegen embedded `n_data_tests` checks, so the + // summary row must carry that many outcomes. Recovering fewer means + // the `data_tests` column was dropped/reshaped before we read it — + // fail loud rather than silently pass unverified tests. + if tests.len() < meta.n_data_tests { + let msg = format!( + "data tests on {}: expected {} test outcome(s) but recovered {} from the \ + result — aborting to avoid a silent pass", + meta.asset_path, + meta.n_data_tests, + tests.len() + ); + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + snapshot_id, + row_count, + None, + Some(&msg), + ) + .await; + return Err(Error::ExecutionErr(msg)); + } + if tests.iter().any(|t| t.violating > 0) { + let breakdown = format_data_test_breakdown(&meta.asset_path, &tests); + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + snapshot_id, + row_count, + None, + Some(&breakdown), + ) + .await; + return Err(Error::ExecutionErr(breakdown)); + } record_mat( conn, &job.workspace_id, @@ -432,6 +809,7 @@ pub async fn do_duckdb( windmill_common::materialization::MaterializationStatus::Materialized, snapshot_id, row_count, + schema, None, ) .await; @@ -1106,6 +1484,42 @@ mod tests { ); } + // Managed `// materialize` may take SQL args (e.g. an s3object uploaded on + // the run form). The wrap strips line comments — including the + // `-- $name (type)` declarations — so the executor parses the signature from + // the original script (done above, before the rewrite) while the `$name` + // references survive inside the wrapped SELECT. This pins both halves of that + // contract so a regression that drops either is caught. + #[test] + fn materialize_preserves_sql_args() { + let script = "-- materialize ducklake://main/rows\n\ + -- $file (s3object)\n\ + SELECT * FROM read_json_auto($file)"; + + // The signature is recoverable from the original (un-wrapped) script. + let sig = parse_duckdb_sig(script).expect("sig parses").args; + let file_arg = sig + .iter() + .find(|a| a.name == "file") + .expect("`$file` declared"); + assert_eq!(file_arg.otyp.as_deref(), Some("s3object")); + + // The wrapped query still references `$file`, so the parsed sig binds it. + // No custom data tests here, so no fetched bodies are needed. + let (rewritten, _) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("$file"), + "wrapped query must keep the `$file` reference, got:\n{rewritten}" + ); + // The declaration comment is gone (wrap strips line comments) — which is + // exactly why the sig must come from the original, not the rewrite. + assert!(!rewritten.contains("-- $file")); + } + // Tests for parse_attach_db_resource function #[test] fn test_parse_attach_db_resource_postgres_res_prefix() { @@ -1464,4 +1878,79 @@ mod tests { let serialized = serde_json::to_string(&arg).unwrap(); assert!(serialized.contains("\"json_value\":{\"key\":\"value\"}")); } + + fn raw(s: &str) -> Box { + serde_json::from_str(s).unwrap() + } + + #[test] + fn extract_data_tests_parses_nested_array() { + // The real result shape: an array of one summary row carrying a nested + // `data_tests` array (how the FFI serialises the list-of-struct). + let r = raw( + r#"[{"rows":3,"snapshot_id":17,"materialized":"ducklake://a/b", + "data_tests":[{"test":"unique(order_id)","violating":0}, + {"test":"accepted_values(status)","violating":2}]}]"#, + ); + let out = extract_data_tests(&r); + assert_eq!(out.len(), 2); + assert_eq!(out[0].name, "unique(order_id)"); + assert_eq!(out[0].violating, 0); + assert_eq!(out[1].name, "accepted_values(status)"); + assert_eq!(out[1].violating, 2); + } + + #[test] + fn extract_data_tests_handles_string_encoded_and_absent() { + // Fallback: some serialisations surface the list-of-struct as a JSON string. + let s = raw(r#"{"data_tests":"[{\"test\":\"not_null(x)\",\"violating\":1}]"}"#); + let out = extract_data_tests(&s); + assert_eq!(out.len(), 1); + assert_eq!(out[0].name, "not_null(x)"); + assert_eq!(out[0].violating, 1); + // Absent column (no tests) -> empty, no panic. + assert!(extract_data_tests(&raw(r#"[{"rows":3}]"#)).is_empty()); + } + + #[test] + fn extract_schema_parses_nested_and_string_encoded() { + // Real shape: the summary row carries a nested `output_schema` + // list-of-struct from the DESCRIBE fold. + let r = raw( + r#"[{"materialized":"ducklake://a/b","rows":3,"snapshot_id":17, + "output_schema":[{"name":"order_id","type":"BIGINT"}, + {"name":"status","type":"VARCHAR"}]}]"#, + ); + let cols = extract_schema(&r).expect("schema present"); + assert_eq!(cols.len(), 2); + assert_eq!(cols[0].name, "order_id"); + assert_eq!(cols[0].data_type, "BIGINT"); + assert_eq!(cols[1].name, "status"); + assert_eq!(cols[1].data_type, "VARCHAR"); + // Fallback: FFI serialised the list-of-struct as a JSON string. + let s = raw(r#"{"output_schema":"[{\"name\":\"x\",\"type\":\"INTEGER\"}]"}"#); + let cols = extract_schema(&s).expect("schema present"); + assert_eq!(cols.len(), 1); + assert_eq!(cols[0].name, "x"); + assert_eq!(cols[0].data_type, "INTEGER"); + // Absent column (literal/manual mode) -> None, no panic. + assert!(extract_schema(&raw(r#"[{"rows":3}]"#)).is_none()); + } + + #[test] + fn format_data_test_breakdown_lists_all_with_marks() { + let tests = vec![ + DataTestOutcome { name: "unique(order_id)".into(), violating: 1 }, + DataTestOutcome { name: "not_null(user_id)".into(), violating: 0 }, + DataTestOutcome { name: "accepted_values(status)".into(), violating: 2 }, + ]; + let msg = format_data_test_breakdown("analytics/orders", &tests); + assert_eq!( + msg, + "data tests failed on analytics/orders (2/3 failed):\n \ + ✗ unique(order_id) — 1 violating row(s)\n \ + ✓ not_null(user_id)\n \ + ✗ accepted_values(status) — 2 violating row(s)" + ); + } } diff --git a/backend/windmill-worker/src/python_executor.rs b/backend/windmill-worker/src/python_executor.rs index 409dc362af..9ca79c6852 100644 --- a/backend/windmill-worker/src/python_executor.rs +++ b/backend/windmill-worker/src/python_executor.rs @@ -37,8 +37,8 @@ use windmill_common::{ scripts::ScriptLang, utils::calculate_hash, worker::{ - copy_dir_recursively, pad_string, split_python_requirements, write_file, Connection, - PyVAlias, PythonAnnotations, WORKER_CONFIG, + copy_dir_recursively, is_allowed_file_location, pad_string, split_python_requirements, + write_file, Connection, PyVAlias, PythonAnnotations, WORKER_CONFIG, }, }; @@ -62,6 +62,13 @@ lazy_static::lazy_static! { static ref PY_CONCURRENT_DOWNLOADS: usize = var("PY_CONCURRENT_DOWNLOADS").ok().map(|flag| flag.parse().unwrap_or(20)).unwrap_or(20); + // uv's HTTP request timeout (seconds). spawn_uv_install uses env_clear(), so a + // UV_HTTP_TIMEOUT set on the worker is dropped unless forwarded explicitly. + // Only forwarded when set; otherwise uv keeps its own default. Lets operators + // raise it for slow/contended private registries ("operation timed out"). + static ref UV_HTTP_TIMEOUT: Option = + var("UV_HTTP_TIMEOUT").ok().filter(|v| !v.is_empty()); + static ref NON_ALPHANUM_CHAR: Regex = regex::Regex::new(r"[^0-9A-Za-z=.-]").unwrap(); @@ -664,10 +671,16 @@ pub fn compute_python_module_dir(script_path: &str) -> String { .replace("-", "_") .replace("@", "."); if dirs_full.len() > 0 { - dirs_full - .strip_prefix("/") - .unwrap_or(&dirs_full) - .to_string() + let dirs = dirs_full.strip_prefix("/").unwrap_or(&dirs_full); + // This directory is appended to job_dir and written to. Neutralize any + // `.`/`..` segment so the result stays a relative path inside job_dir: a + // Preview path is request-supplied and skips the DB `proper_id` CHECK that + // deployed runnables get, and the `@`->`.` rewrite above can also turn a + // segment like `@.` into `..`. + dirs.split('/') + .map(|seg| if seg == "." || seg == ".." { "_" } else { seg }) + .collect::>() + .join("/") } else { "tmp".to_string() } @@ -1668,6 +1681,10 @@ async fn prepare_wrapper( last }; let module_dir = format!("{}/{}", job_dir, dirs); + // Defense-in-depth: `dirs`/`last` derive from the (request-supplied for + // previews) script path. compute_python_module_dir already neutralizes `..`, + // but assert containment here too so the write can never escape job_dir. + is_allowed_file_location(job_dir, &format!("{dirs}/{last}.py"))?; tokio::fs::create_dir_all(format!("{module_dir}/")).await?; let _ = write_file(&module_dir, &format!("{last}.py"), inner_content)?; @@ -2040,6 +2057,33 @@ Returned from server: py_version - {:?}, py_version_v2 - {:?} lazy_static::lazy_static! { static ref PIP_SECRET_VARIABLE: Regex = Regex::new(r"\$\{PIP_SECRET:([^\s\}]+)\}").unwrap(); + + /// venv paths whose wheel RECORD this process has already verified against + /// disk. A cache entry is only damaged out-of-band (disk-pressure eviction, + /// an interrupted extraction on a shared cache volume, or a corrupt entry + /// that predates this worker), never spontaneously while we keep running, so + /// re-verifying it once per process is enough — every later reuse trusts the + /// in-memory marker and pays only the original single stat. + static ref VERIFIED_VENVS: tokio::sync::Mutex> = + tokio::sync::Mutex::new(HashSet::new()); + + // In-process locks serializing concurrent installs into the same shared + // `venv_p` cache dir; `uv --reinstall` removes a package's .dist-info/RECORD + // before rewriting it, so a sibling install/verify racing it corrupts the + // dir. Keyed by venv_p so distinct deps still install in parallel. + static ref PY_INSTALL_LOCKS: tokio::sync::Mutex>>> = + tokio::sync::Mutex::new(std::collections::HashMap::new()); +} + +/// Returns the in-process install lock for a given target cache dir, creating it +/// on first use. Idle entries (only the map holds a reference) are pruned each +/// call so the map stays bounded by the number of in-flight installs. +async fn get_venv_install_lock(venv_p: &str) -> Arc> { + let mut map = PY_INSTALL_LOCKS.lock().await; + map.retain(|_, v| Arc::strong_count(v) > 1); + map.entry(venv_p.to_string()) + .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(()))) + .clone() } /// Spawn process of uv install @@ -2085,6 +2129,9 @@ async fn spawn_uv_install( if *NATIVE_CERT { vars.push(("UV_NATIVE_TLS", "true")); } + if let Some(timeout) = UV_HTTP_TIMEOUT.as_ref() { + vars.push(("UV_HTTP_TIMEOUT", timeout.as_str())); + } let _owner; if let Some(py_path) = py_path.as_ref() { @@ -2188,6 +2235,9 @@ async fn spawn_uv_install( let mut envs = vec![("PATH", PATH_ENV.as_str())]; envs.push(("HOME", HOME_ENV.as_str())); envs.push(("UV_INDEX_STRATEGY", uv_index_strategy)); + if let Some(timeout) = UV_HTTP_TIMEOUT.as_ref() { + envs.push(("UV_HTTP_TIMEOUT", timeout.as_str())); + } if let Some(mirror) = uv_python_install_mirror.as_ref() { envs.push(("UV_PYTHON_INSTALL_MIRROR", mirror)); } @@ -2479,8 +2529,53 @@ pub async fn handle_python_reqs( req.replace(' ', "").replace('/', "").replace(':', "") ); if metadata(venv_p.clone() + "/.valid.windmill").await.is_ok() { - req_paths.push(venv_p); - in_cache.push(req.to_string()); + // The .valid.windmill marker is written once at creation time, after + // verify_wheel_record passes on the install/pull paths. It is an empty + // file with no binding to the directory contents, so a file dropped + // out-of-band afterwards (disk-pressure eviction, interrupted tar + // extraction on a shared cache volume, or a corrupt entry that + // predates this worker) leaves the marker intact while the wheel is + // incomplete. Re-verify the RECORD once per process so such an entry + // is repaired rather than trusted; VERIFIED_VENVS makes every later + // reuse skip the scan and pay only the single stat above. + let already_verified = VERIFIED_VENVS.lock().await.contains(&venv_p); + let verify_res = if already_verified { + Ok(()) + } else { + verify_wheel_record(&venv_p).await + }; + match verify_res { + Ok(()) => { + if !already_verified { + VERIFIED_VENVS.lock().await.insert(venv_p.clone()); + } + req_paths.push(venv_p); + in_cache.push(req.to_string()); + } + Err(verify_err) => { + tracing::warn!( + workspace_id = %w_id, + job_id = %job_id, + "Local cache for {venv_p} failed wheel RECORD verification, will reinstall: {verify_err}" + ); + append_logs( + &job_id, + w_id, + format!( + "\n[!] cached wheel for {req} failed integrity check, reinstalling: {verify_err}\n" + ), + conn, + ) + .await; + if let Err(rm_err) = tokio::fs::remove_dir_all(&venv_p).await { + tracing::warn!( + workspace_id = %w_id, + "could not remove broken cache dir {venv_p}: {rm_err}" + ); + } + req_with_penv.push((req.to_string(), venv_p)); + } + } } else { // There is no valid or no wheel at all. Regardless of if there is content or not, we will overwrite it with --reinstall flag req_with_penv.push((req.to_string(), venv_p)); @@ -2716,6 +2811,96 @@ pub async fn handle_python_reqs( ); let start = std::time::Instant::now(); + + // Lock the shared target dir (see PY_INSTALL_LOCKS). In-process lock + // first; only one task per dir then contends the cross-process file + // lock below. Both guards drop on every return path. + let venv_lock = get_venv_install_lock(&venv_p).await; + let _venv_guard = tokio::select! { + _ = kill_rx.recv() => { + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Err(Error::from(anyhow::anyhow!( + "install of {venv_p} canceled while waiting for venv lock" + ))); + } + guard = venv_lock.lock_owned() => guard, + }; + + // Cross-process advisory lock. Best-effort: if the filesystem doesn't + // support flock we log and proceed — verify_wheel_record + job retry + // still guard correctness, just without the dedup. + #[cfg(unix)] + let _venv_file_lock: Option = { + use std::os::unix::io::AsRawFd; + let lock_path = format!("{venv_p}.lock"); + if let Some(parent) = std::path::Path::new(&lock_path).parent() { + let _ = std::fs::create_dir_all(parent); + } + match std::fs::OpenOptions::new().create(true).write(true).open(&lock_path) { + Ok(f) => { + // Bounded wait: a holder that crashes releases the lock (the + // kernel drops it on fd close), but a live-but-stuck holder + // (e.g. uv wedged on a hung mount) would otherwise block us + // forever. After the cap, proceed degraded rather than hang — + // verify_wheel_record + retry still guard correctness. + const MAX_WAIT: std::time::Duration = std::time::Duration::from_secs(300); + let waited_since = std::time::Instant::now(); + loop { + match nix::fcntl::flock(f.as_raw_fd(), nix::fcntl::FlockArg::LockExclusiveNonblock) { + Ok(()) => break Some(f), + // EWOULDBLOCK == EAGAIN on Linux: another holder has the lock. + Err(nix::errno::Errno::EWOULDBLOCK) => { + if waited_since.elapsed() >= MAX_WAIT { + tracing::warn!( + workspace_id = %w_id, + "venv install lock {lock_path} still held after {}s, proceeding without cross-process install lock", + MAX_WAIT.as_secs() + ); + break Some(f); + } + tokio::select! { + _ = kill_rx.recv() => { + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Err(Error::from(anyhow::anyhow!( + "install of {venv_p} canceled while waiting for venv file lock" + ))); + } + _ = tokio::time::sleep(std::time::Duration::from_millis(200)) => {} + } + } + Err(e) => { + tracing::warn!( + workspace_id = %w_id, + "could not flock {lock_path}, proceeding without cross-process install lock: {e}" + ); + break Some(f); + } + } + } + } + Err(e) => { + tracing::warn!( + workspace_id = %w_id, + "could not open install lock file {lock_path}, proceeding without cross-process install lock: {e}" + ); + None + } + } + }; + + // Double-checked: another job (this process or another sharing the + // mount) may have installed this exact dep while we waited on the + // locks. Reuse it instead of reinstalling. + if metadata(format!("{venv_p}/.valid.windmill")).await.is_ok() { + print_success( + false, false, &job_id, &w_id, &req, req_tl, counter_arc, + total_to_install, start, &conn, + ) + .await; + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Ok(()); + } + #[cfg(all(feature = "enterprise", feature = "parquet"))] if is_not_pro { if let Some(os) = windmill_object_store::get_object_store().await { @@ -3357,6 +3542,17 @@ mod tests { assert_eq!(compute_python_module_dir("f/in/script"), "f/_in"); } + #[test] + fn test_compute_python_module_dir_neutralizes_traversal() { + // A Preview path skips the DB `proper_id` CHECK, so it can carry `..`. + // `..`/`.` segments must be neutralized so the dir stays inside job_dir. + let dirs = compute_python_module_dir("u/x/../../../../tmp/evil/payload"); + assert!(!dirs.split('/').any(|s| s == ".." || s == ".")); + assert_eq!(dirs, "u/x/_/_/_/_/tmp/evil"); + // The `@`->`.` rewrite must not be able to synthesize a `..` segment. + assert_eq!(compute_python_module_dir("u/@./script"), "u/_"); + } + #[test] fn test_compute_py_codegen_basic_args() { let code = "def main(x: str, y: int):\n return x\n"; @@ -3446,4 +3642,179 @@ mod tests { assert_eq!(kept, lines(&["# py: 3.11", "requests==2.0"])); assert_eq!(ignored, lines(&["pyyaml==6.0"])); } + + /// Materialize a fake installed wheel: every file in `files` is created, and + /// `record_entries` is written verbatim as the RECORD (so a test can list a + /// path in RECORD without creating it, to simulate out-of-band loss). + fn write_fake_wheel(root: &std::path::Path, files: &[&str], record_entries: &[&str]) { + for f in files { + let full = root.join(f); + std::fs::create_dir_all(full.parent().unwrap()).unwrap(); + std::fs::write(full, b"x").unwrap(); + } + let dist_info = root.join("pkg-1.0.0.dist-info"); + std::fs::create_dir_all(&dist_info).unwrap(); + std::fs::write(dist_info.join("RECORD"), record_entries.join("\n") + "\n").unwrap(); + } + + #[tokio::test] + async fn test_verify_wheel_record_ok_when_all_present() { + let dir = tempfile::tempdir().unwrap(); + write_fake_wheel( + dir.path(), + &["pkg/__init__.py", "pkg/mod.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "pkg/mod.py,sha256=bbb,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_ok()); + } + + #[tokio::test] + async fn test_verify_wheel_record_err_when_file_missing() { + let dir = tempfile::tempdir().unwrap(); + // RECORD lists pkg/mod.py but we never create it: the exact failure mode + // the customer hit (wmill/s3_reader.py present in RECORD, gone on disk). + write_fake_wheel( + dir.path(), + &["pkg/__init__.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "pkg/mod.py,sha256=bbb,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + let err = verify_wheel_record(dir.path().to_str().unwrap()) + .await + .unwrap_err(); + assert!(err.contains("pkg/mod.py"), "unexpected error: {err}"); + } + + #[tokio::test] + async fn test_verify_wheel_record_err_when_no_dist_info() { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("loose.py"), b"x").unwrap(); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_err()); + } + + #[tokio::test] + async fn test_verify_wheel_record_skips_absolute_and_escaping_entries() { + let dir = tempfile::tempdir().unwrap(); + // Absolute and `..` RECORD entries are not package-relative and must be + // skipped rather than reported missing. + write_fake_wheel( + dir.path(), + &["pkg/__init__.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "/etc/passwd,sha256=ccc,1", + "../outside.py,sha256=ddd,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_ok()); + } + + // Regression tests for the concurrent-install guard. Two jobs installing the + // same uncached dep into the shared `venv_p` used to race uv's `--reinstall`, + // corrupting the on-disk wheel and failing with "Env installation did not + // succeed". The guard serializes those installs. + + #[tokio::test] + async fn test_venv_install_lock_serializes_same_path() { + use std::sync::atomic::{AtomicUsize, Ordering}; + // Same target path => one shared lock => no two tasks install at once. + let active = Arc::new(AtomicUsize::new(0)); + let max_seen = Arc::new(AtomicUsize::new(0)); + let mut handles = vec![]; + for _ in 0..8 { + let active = active.clone(); + let max_seen = max_seen.clone(); + handles.push(tokio::spawn(async move { + let lock = get_venv_install_lock("/cache/py/3.11/samedep==1.0").await; + let _g = lock.lock_owned().await; + let cur = active.fetch_add(1, Ordering::SeqCst) + 1; + max_seen.fetch_max(cur, Ordering::SeqCst); + // Yield so any concurrency would be observed by another task. + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + active.fetch_sub(1, Ordering::SeqCst); + })); + } + for h in handles { + h.await.unwrap(); + } + assert_eq!( + max_seen.load(Ordering::SeqCst), + 1, + "installs into the same target dir must be serialized" + ); + } + + #[tokio::test] + async fn test_venv_install_lock_distinct_paths_are_independent() { + // Different target paths get different locks and never block each other. + let a = get_venv_install_lock("/cache/py/3.11/depA==1.0").await; + let b = get_venv_install_lock("/cache/py/3.11/depB==1.0").await; + let _ga = a.lock_owned().await; + // Holding depA's lock must not prevent acquiring depB's. + assert!( + b.try_lock().is_ok(), + "distinct deps must install in parallel" + ); + // Same path returns the same underlying lock. + let a2 = get_venv_install_lock("/cache/py/3.11/depA==1.0").await; + assert!( + a2.try_lock().is_err(), + "same target dir must map to the same lock" + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn test_venv_file_lock_excludes_across_descriptions() { + // The cross-process layer: flock on a sibling `.lock` excludes a second + // independent open file description (i.e. another worker process) while + // held, and frees it on close. Mirrors the loop in handle_python_reqs. + use nix::fcntl::{flock, FlockArg}; + use std::os::unix::io::AsRawFd; + + let dir = std::env::temp_dir().join("wm_venv_lock_test"); + std::fs::create_dir_all(&dir).unwrap(); + let lock_path = dir.join("dep==1.0.lock"); + + let f1 = std::fs::OpenOptions::new() + .create(true) + .write(true) + .open(&lock_path) + .unwrap(); + flock(f1.as_raw_fd(), FlockArg::LockExclusiveNonblock).unwrap(); + + // A second descriptor (stand-in for another process) cannot take it. + let f2 = std::fs::OpenOptions::new() + .create(true) + .write(true) + .open(&lock_path) + .unwrap(); + assert_eq!( + flock(f2.as_raw_fd(), FlockArg::LockExclusiveNonblock), + Err(nix::errno::Errno::EWOULDBLOCK), + "a second holder must be blocked while the lock is held" + ); + + // Releasing the first lets the second acquire it. + drop(f1); + flock(f2.as_raw_fd(), FlockArg::LockExclusiveNonblock) + .expect("lock must be acquirable once the holder releases it"); + + drop(f2); + let _ = std::fs::remove_file(&lock_path); + } } diff --git a/backend/windmill-worker/src/python_versions.rs b/backend/windmill-worker/src/python_versions.rs index 0ae6b3462a..3a1f35a302 100644 --- a/backend/windmill-worker/src/python_versions.rs +++ b/backend/windmill-worker/src/python_versions.rs @@ -551,6 +551,13 @@ impl PyV { .stdout(Stdio::piped()) .stderr(Stdio::piped()); + if let Some(cert_path) = INDEX_CERT.as_ref() { + child_cmd.env("SSL_CERT_FILE", cert_path); + } + if *NATIVE_CERT { + child_cmd.env("UV_NATIVE_TLS", "true"); + } + if let Some(mirror) = UV_PYTHON_INSTALL_MIRROR.read().await.as_ref() { child_cmd.env("UV_PYTHON_INSTALL_MIRROR", mirror); } diff --git a/backend/windmill-worker/src/result_processor.rs b/backend/windmill-worker/src/result_processor.rs index ab2cad4da9..26fbdb21af 100644 --- a/backend/windmill-worker/src/result_processor.rs +++ b/backend/windmill-worker/src/result_processor.rs @@ -1295,7 +1295,7 @@ pub async fn handle_job_error( db, &parent_job, mem_peak, - canceled_by, + canceled_by.clone(), e, worker_name, false, diff --git a/backend/windmill-worker/src/worker.rs b/backend/windmill-worker/src/worker.rs index 45ae8c3c25..2335a78453 100644 --- a/backend/windmill-worker/src/worker.rs +++ b/backend/windmill-worker/src/worker.rs @@ -44,7 +44,7 @@ use windmill_common::{ schema::{should_validate_schema, SchemaValidator}, utils::{create_directory_async, WarnAfterExt}, worker::{ - make_pull_query, write_file, Connection, HttpClient, MAX_TIMEOUT, + is_allowed_file_location, make_pull_query, write_file, Connection, HttpClient, MAX_TIMEOUT, MIN_PERIODIC_SCRIPT_INTERVAL_SECONDS, ROOT_CACHE_DIR, ROOT_CACHE_NOMOUNT_DIR, WINDMILL_DIR, }, worker_group_job_stats::JobStatsMap, @@ -4630,44 +4630,148 @@ async fn handle_code_execution_job( .await } +/// True when `path` contains only `Normal`/`CurDir` components, i.e. it cannot +/// escape the directory it is joined onto (no `..`, no absolute root, no Windows +/// drive prefix). +fn is_contained_relative_path(path: &str) -> bool { + use std::path::Component; + std::path::Path::new(path) + .components() + .all(|c| matches!(c, Component::Normal(_) | Component::CurDir)) +} + pub async fn write_module_files( job_dir: &str, modules: &std::collections::HashMap, base_dir: Option<&str>, ) -> error::Result<()> { + // base_dir is derived from the runnable path, which on a preview run can + // carry `..` traversal (it is not the validated module-map key). Reject it + // before it is used to build any write target, otherwise a module could + // escape job_dir and write arbitrary files. + if let Some(dir) = base_dir { + if !is_contained_relative_path(dir) { + return Err(error::Error::BadRequest(format!( + "Invalid module base directory (path traversal): {dir}" + ))); + } + } for (relpath, module) in modules { - // Reject path traversal attempts in module paths - if relpath.contains("..") { + // Reject path traversal attempts in module paths (the module-map key). + if !is_contained_relative_path(relpath) { tracing::warn!("Skipping module with path traversal: {relpath}"); continue; } - let full_path = match base_dir { - Some(dir) => format!("{}/{}/{}", job_dir, dir, relpath), - None => format!("{}/{}", job_dir, relpath), + let relpath_from_job_dir = match base_dir { + Some(dir) => format!("{}/{}", dir, relpath), + None => relpath.to_string(), }; - if let Some(parent) = std::path::Path::new(&full_path).parent() { + // Authoritative guard: resolve the path and assert it stays inside job_dir. + let full_path = is_allowed_file_location(job_dir, &relpath_from_job_dir)?; + if let Some(parent) = full_path.parent() { tokio::fs::create_dir_all(parent).await?; } // For Python modules, create __init__.py in each intermediate directory // between base_dir and the module's parent so that relative imports work. if let Some(dir) = base_dir { - let rel = std::path::Path::new(relpath); - let base = std::path::Path::new(job_dir).join(dir); - let mut current = base.clone(); - for component in rel.parent().into_iter().flat_map(|p| p.components()) { + let mut current = std::path::PathBuf::from(dir); + for component in std::path::Path::new(relpath) + .parent() + .into_iter() + .flat_map(|p| p.components()) + { current = current.join(component); - let init_py = current.join("__init__.py"); + let init_py = is_allowed_file_location( + job_dir, + ¤t.join("__init__.py").to_string_lossy(), + )?; if !init_py.exists() { tokio::fs::write(&init_py, "").await?; } } } - tracing::debug!("Writing module file: {full_path}"); + tracing::debug!("Writing module file: {}", full_path.display()); tokio::fs::write(&full_path, &module.content).await?; } Ok(()) } +#[cfg(test)] +mod write_module_files_tests { + use super::*; + use std::collections::HashMap; + use windmill_common::scripts::ScriptLang; + + fn module(content: &str) -> ScriptModule { + ScriptModule { content: content.to_string(), language: ScriptLang::Python3, lock: None } + } + + #[test] + fn contained_relative_path_rejects_traversal_and_absolute() { + assert!(is_contained_relative_path("u/admin/pkg")); + assert!(is_contained_relative_path("./pkg/sub")); + // A `..` in a filename is a valid name, not a traversal. + assert!(is_contained_relative_path("weird..name")); + + assert!(!is_contained_relative_path("u/x/../../../etc")); + assert!(!is_contained_relative_path("../escape")); + assert!(!is_contained_relative_path("/etc/cron.d/wm")); + } + + #[tokio::test] + async fn base_dir_traversal_is_rejected_and_writes_nothing() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + // Sentinel just outside job_dir that a successful traversal would create. + let outside = job.path().parent().unwrap().join("wm_escaped_marker"); + + let mut modules = HashMap::new(); + modules.insert( + "wm_escaped_marker".to_string(), + module("* * * * * root id\n"), + ); + + // base_dir derived from a preview path carrying `..` traversal. + let res = write_module_files(job_dir, &modules, Some("u/x/../../../../../..")).await; + assert!(res.is_err(), "traversal base_dir must be rejected"); + assert!(!outside.exists(), "no file may be written outside job_dir"); + } + + #[tokio::test] + async fn relpath_traversal_is_skipped() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + let outside = job.path().parent().unwrap().join("wm_relpath_escape.py"); + + let mut modules = HashMap::new(); + modules.insert("../wm_relpath_escape.py".to_string(), module("x = 1")); + + write_module_files(job_dir, &modules, None).await.unwrap(); + assert!(!outside.exists()); + } + + #[tokio::test] + async fn legitimate_modules_are_written_with_init_py() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + + let mut modules = HashMap::new(); + modules.insert("pkg/sub/mod.py".to_string(), module("VALUE = 42")); + + write_module_files(job_dir, &modules, Some("u/admin")) + .await + .unwrap(); + + let base = job.path().join("u/admin"); + assert_eq!( + std::fs::read_to_string(base.join("pkg/sub/mod.py")).unwrap(), + "VALUE = 42" + ); + assert!(base.join("pkg/__init__.py").exists()); + assert!(base.join("pkg/sub/__init__.py").exists()); + } +} + pub async fn run_language_executor( job: &MiniPulledJob, conn: &Connection, diff --git a/backend/windmill-worker/src/worker_flow.rs b/backend/windmill-worker/src/worker_flow.rs index 7f90f9240e..10e8218a77 100644 --- a/backend/windmill-worker/src/worker_flow.rs +++ b/backend/windmill-worker/src/worker_flow.rs @@ -28,7 +28,7 @@ use serde::{Deserialize, Serialize}; use serde_json::value::RawValue; use serde_json::{json, Value}; use sqlx::types::Json; -use sqlx::{FromRow, Postgres, Transaction}; +use sqlx::{Acquire, FromRow, Postgres, Transaction}; use tracing::instrument; use uuid::Uuid; use windmill_common::auth::get_job_perms; @@ -332,14 +332,14 @@ fn get_stop_after_if_data(stop_after_if: Option<&StopAfterIf>) -> (bool, Option< return (false, None, false); } -async fn get_id_ctx_for_expr( +async fn get_id_ctx_for_expr<'c>( expr: &str, flow: uuid::Uuid, - db: &DB, + e: impl sqlx::PgExecutor<'c>, status: &FlowStatus, ) -> error::Result> { if expr.contains("results.") || expr.contains("results[") || expr.contains("results?.") { - let flow_job = get_mini_pulled_job(db, &flow).await?; + let flow_job = get_mini_pulled_job(e, &flow).await?; if let Some(flow_job) = flow_job { Ok(Some(get_transform_context(&flow_job, "", &status))) } else { @@ -351,7 +351,7 @@ async fn get_id_ctx_for_expr( } async fn evaluate_stop_after_all_iters_if( - db: &DB, + tx: &mut Transaction<'_, Postgres>, stop_after_all_iters_if: &StopAfterIf, module_status: &FlowStatusModule, w_id: &str, @@ -366,9 +366,20 @@ async fn evaluate_stop_after_all_iters_if( flow: uuid::Uuid, status: &FlowStatus, ) -> error::Result<()> { + // Test hook (see test_stop_after_all_iters_if_db_error_isolated_by_savepoint): + // run a query that aborts this (savepoint) transaction so the test can verify the + // caller's savepoint keeps the outer status-update transaction committable. + #[cfg(feature = "failpoints")] + if stop_after_all_iters_if.expr == "__wm_failpoint_abort_tx__" { + sqlx::query("SELECT 1/0") + .execute(&mut **tx) + .await + .map_err(|e| Error::internal_err(format!("failpoint abort_tx: {e:#}")))?; + } + let iters_result = match &module_status { FlowStatusModule::InProgress { flow_jobs: Some(flow_jobs), .. } => { - Arc::new(retrieve_flow_jobs_results(db, w_id, flow_jobs).await?) + Arc::new(retrieve_flow_jobs_results(&mut **tx, w_id, flow_jobs).await?) } _ => { return Err(Error::internal_err(format!( @@ -379,7 +390,8 @@ async fn evaluate_stop_after_all_iters_if( *nresult = Some(iters_result.clone()); // as an optimization, we store the result of all jobs as when stop_early_after_all_iters evaluates to false, it would have to be computed (finished loop/branchall) - let id_ctx = get_id_ctx_for_expr(&stop_after_all_iters_if.expr, flow, db, status).await?; + let id_ctx = + get_id_ctx_for_expr(&stop_after_all_iters_if.expr, flow, &mut **tx, status).await?; let stop_early_after_all_iters = compute_bool_from_expr( &stop_after_all_iters_if.expr, @@ -976,8 +988,15 @@ pub async fn update_flow_status_after_job_completion_internal( .and_then(|x| x.stop_after_all_iters_if.as_ref()) { let args = from_result_to_args(args.as_ref().await.get_ref())?; - if let Err(e) = evaluate_stop_after_all_iters_if( - db, + // Isolate the reads in a savepoint on the same connection: the + // caller below swallows our error and keeps using `tx`, so a DB + // read failure must not leave the outer transaction aborted (that + // would fail the later commit). On error we roll back to the + // savepoint, matching the previous pool-read behaviour where a + // failed read left `tx` usable and the iteration was marked failed. + let mut sp = tx.begin().await?; + let eval_res = evaluate_stop_after_all_iters_if( + &mut sp, stop_after_all_iters_if, module_status, w_id, @@ -992,15 +1011,19 @@ pub async fn update_flow_status_after_job_completion_internal( flow, &old_status, ) - .await - { - tracing::error!("error evaluating stop_after_all_iters_if: {e:#}"); - stop_early = true; - skip_if_stop_early = false; - stop_early_err_msg = Some(format!( - "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", - stop_after_all_iters_if.expr - )); + .await; + match eval_res { + Ok(()) => sp.commit().await?, + Err(e) => { + let _ = sp.rollback().await; + tracing::error!("error evaluating stop_after_all_iters_if: {e:#}"); + stop_early = true; + skip_if_stop_early = false; + stop_early_err_msg = Some(format!( + "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", + stop_after_all_iters_if.expr + )); + } } } @@ -1054,7 +1077,7 @@ pub async fn update_flow_status_after_job_completion_internal( let r = sqlx::query_scalar!( "DELETE FROM parallel_monitor_lock WHERE parent_flow_id = $1 RETURNING last_ping", flow, - ).fetch_optional(db).await.map_err(|e| { + ).fetch_optional(&mut *tx).await.map_err(|e| { Error::internal_err(format!( "error while deleting parallel_monitor_lock: {e:#}" )) @@ -1198,8 +1221,12 @@ pub async fn update_flow_status_after_job_completion_internal( { let args = from_result_to_args(args.as_ref().await.get_ref())?; - if let Err(e) = evaluate_stop_after_all_iters_if( - db, + // See the matching savepoint comment above: isolate the reads so a + // DB read failure (whose error the caller swallows) cannot abort + // the outer transaction and break the later commit. + let mut sp = tx.begin().await?; + let eval_res = evaluate_stop_after_all_iters_if( + &mut sp, stop_after_all_iters_if, module_status, w_id, @@ -1214,14 +1241,18 @@ pub async fn update_flow_status_after_job_completion_internal( flow, &old_status, ) - .await - { - stop_early = true; - skip_if_stop_early = false; - stop_early_err_msg = Some(format!( - "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", - stop_after_all_iters_if.expr - )); + .await; + match eval_res { + Ok(()) => sp.commit().await?, + Err(e) => { + let _ = sp.rollback().await; + stop_early = true; + skip_if_stop_early = false; + stop_early_err_msg = Some(format!( + "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", + stop_after_all_iters_if.expr + )); + } } } } @@ -1462,7 +1493,7 @@ pub async fn update_flow_status_after_job_completion_internal( match &new_status { Some(FlowStatusModule::Success { flow_jobs: Some(jobs), .. }) | Some(FlowStatusModule::Failure { flow_jobs: Some(jobs), .. }) => { - Arc::new(retrieve_flow_jobs_results(db, w_id, jobs).await?) + Arc::new(retrieve_flow_jobs_results(&mut *tx, w_id, jobs).await?) } _ => result.clone(), } @@ -1604,6 +1635,7 @@ pub async fn update_flow_status_after_job_completion_internal( windmill_common::runnable_settings::RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: None, + retry_settings: None, }, db, ) @@ -2273,8 +2305,8 @@ async fn set_success_and_duration_in_flow_job_success<'c>( Ok(()) } -async fn retrieve_flow_jobs_results( - db: &DB, +async fn retrieve_flow_jobs_results<'c>( + e: impl sqlx::PgExecutor<'c>, w_id: &str, job_uuids: &Vec, ) -> error::Result> { @@ -2285,7 +2317,7 @@ async fn retrieve_flow_jobs_results( job_uuids.as_slice(), w_id ) - .fetch_all(db) + .fetch_all(e) .await? .into_iter() .map(|br| (br.id, br.result)) @@ -4326,7 +4358,7 @@ async fn push_next_flow_job( .filter(|t| !t.is_empty() && *t != flow_job.tag.as_str()) { check_tag_available_for_workspace_internal( - &db, + db, &flow_job.workspace_id, tag_str, email, @@ -6179,8 +6211,15 @@ fn needs_resume(flow: &FlowValue, status: &FlowStatus) -> Option<(Suspend, Uuid) return None; } - if let &FlowStatusModule::Success { job, .. } = status.modules.get(prev)? { - Some((suspend.unwrap(), job)) + if let &FlowStatusModule::Success { job, skipped, .. } = status.modules.get(prev)? { + // A step skipped via skip_if never ran, so its suspend/approval was never + // armed and no resume event will ever arrive. Gating the next step on it + // would park the flow forever. + if skipped { + None + } else { + Some((suspend.unwrap(), job)) + } } else { None } diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 243366b147..f8cc748bdb 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.734.0"; +export const VERSION = "v1.742.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/src/commands/docs/docs.ts b/cli/src/commands/docs/docs.ts index fed91201e0..4ba7ae95d5 100644 --- a/cli/src/commands/docs/docs.ts +++ b/cli/src/commands/docs/docs.ts @@ -7,24 +7,16 @@ import { GlobalOptions } from "../../types.ts"; import { getHeaders } from "../../utils/utils.ts"; import { detectAuthGatewayChallenge } from "../../utils/http_guards.ts"; -interface DocContentItem { - title: string; +interface DocsSearchResult { url: string; - source?: { - content?: Array<{ text: string }>; - }; + title: string; + score: number; + snippets: string[]; } -interface InkeepResponse { - choices?: Array<{ - message?: { - content?: string; - }; - }>; -} - -interface ParsedContent { - content?: DocContentItem[]; +interface DocsSearchResponse { + text: string; + results: DocsSearchResult[]; } async function docs( @@ -34,7 +26,9 @@ async function docs( await requireLogin(opts); const workspace = await resolveWorkspace(opts); - const url = `${workspace.remote}api/inkeep`; + // The backend self-hosts the docs corpus and does the search, so this works + // against any instance (no windmill.dev egress required). + const url = `${workspace.remote}api/docs/search?query=${encodeURIComponent(query)}`; console.log(colors.bold(`\nSearching Windmill docs...\n`)); @@ -42,13 +36,11 @@ async function docs( let res: Response; try { res = await fetch(url, { - method: "POST", + method: "GET", headers: { - "Content-Type": "application/json", Authorization: `Bearer ${workspace.token}`, ...extraHeaders, }, - body: JSON.stringify({ query }), }); } catch (e) { throw new Error(`Network error connecting to ${workspace.remote}: ${e}`); @@ -56,54 +48,31 @@ async function docs( await detectAuthGatewayChallenge(res, url); - if (res.status === 403) { - log.info( - "Windmill documentation search is an Enterprise Edition feature. Please upgrade to use this command." - ); - return; - } - if (!res.ok) { throw new Error( `Documentation search failed: ${res.status} ${res.statusText}\n${await res.text()}` ); } - const data = (await res.json()) as InkeepResponse; - const raw = data.choices?.[0]?.message?.content; - - if (!raw) { - log.info("No documentation found for this query."); - return; - } - - let parsed: ParsedContent; - try { - parsed = JSON.parse(raw); - } catch { - throw new Error("Failed to parse documentation response."); - } - - const items = parsed.content ?? []; - - if (items.length === 0) { - log.info("No documentation found for this query."); - return; - } + const data = (await res.json()) as DocsSearchResponse; + const items = data.results ?? []; if (opts.json) { console.log(JSON.stringify(items, null, 2)); return; } + if (items.length === 0) { + log.info("No documentation found for this query."); + return; + } + for (const item of items) { console.log(colors.bold(colors.cyan(`📄 ${item.title}`))); if (item.url) { console.log(` ${colors.underline(item.url)}`); } - const text = item.source?.content?.[0]?.text; - if (text) { - const snippet = text.length > 500 ? text.slice(0, 500) + "..." : text; + for (const snippet of item.snippets ?? []) { console.log(` ${snippet}`); } console.log(); diff --git a/cli/src/commands/flow/flow.ts b/cli/src/commands/flow/flow.ts index c0e94c478d..71d90af6fe 100644 --- a/cli/src/commands/flow/flow.ts +++ b/cli/src/commands/flow/flow.ts @@ -135,6 +135,31 @@ function warnAboutLocalPathScriptDivergence( const alreadySynced: string[] = []; +// Collect every script/sub-flow step path in a flow value — recursively through loops, +// branches, and the failure/preprocessor modules — for workspace-path validation. Unlike +// `collectPathScriptPaths` this also includes `type: "flow"` sub-flow steps. +function collectStepPaths(flowValue: any): string[] { + const paths: string[] = []; + const walk = (modules: any[] | undefined) => { + for (const m of modules ?? []) { + const v = m?.value; + if (!v) continue; + if ((v.type === "script" || v.type === "flow") && typeof v.path === "string") { + paths.push(v.path); + } + walk(v.modules); + walk(v.default); + for (const b of v.branches ?? []) walk(b?.modules); + // AI-agent tools are step-like and can carry script paths too. + walk(v.tools); + } + }; + walk(flowValue?.modules); + if (flowValue?.failure_module) walk([flowValue.failure_module]); + if (flowValue?.preprocessor_module) walk([flowValue.preprocessor_module]); + return paths; +} + export async function pushFlow( workspace: string, remotePath: string, @@ -190,6 +215,21 @@ export async function pushFlow( ); } + // Reject script/sub-flow steps whose path is not a workspace path (u/, f/, g/ or hub/). + // A flow.yaml generated from a feature-branch checkout can carry absolute local paths + // (e.g. /tmp/.../ops/scripts/...); pushed, they silently mis-resolve at runtime (#9751). + // The backend re-validates the same rule for every step type, so this is a fail-fast. + const badStepPaths = collectStepPaths(localFlow.value).filter( + (p) => p !== "" && !/^(u|f|g|hub)\//.test(p) + ); + if (badStepPaths.length > 0) { + throw new Error( + `Cannot push flow ${remotePath}: step(s) reference non-workspace path(s): ${badStepPaths.join(", ")}. ` + + `Flow step paths must be workspace paths (u/, f/, g/ or hub/), not absolute or local filesystem paths. ` + + `This usually means flow.yaml was generated with paths from a checkout directory.` + ); + } + const hasOnBehalfOf = (localFlow as any).has_on_behalf_of ?? !!localFlow.on_behalf_of_email; delete (localFlow as any).has_on_behalf_of; diff --git a/cli/src/commands/pipeline/boundedCascade.ts b/cli/src/commands/pipeline/boundedCascade.ts new file mode 100644 index 0000000000..27380d799a --- /dev/null +++ b/cli/src/commands/pipeline/boundedCascade.ts @@ -0,0 +1,241 @@ +// Bounded-cascade graph engine for `wmill pipeline run --to`. +// +// MIRROR of frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts — +// the two have no shared import path (frontend is a separate package), so keep +// them in sync. The grammar is intentionally tiny: there is no dbt-style +// `--select` string. The user names a start (a schedule / manual root) and one +// or more end nodes; the run is the "path between" them: +// +// descendants(start) ∩ (ancestors(ends) ∪ ends) ∪ {start} +// +// Node ids: assets `${kind}:${path}` (e.g. `datatable:main/raw`); runnables +// `script:${path}`. Operates on the asset-graph payload shape used by +// pipeline.ts. + +export type BCGraph = { + runnables: { path: string; usage_kind: "script" | "flow" | "job" }[]; + assets: { kind: string; path: string }[]; + edges: { + runnable_kind: string; + runnable_path: string; + asset_kind: string; + asset_path: string; + access_type?: "r" | "w" | "rw"; + }[]; + triggers: ( + | { + trigger_kind: "asset"; + asset_kind: string; + asset_path: string; + runnable_kind: string; + runnable_path: string; + } + | { trigger_kind: string; runnable_kind: string; runnable_path: string } + )[]; +}; + +export const SCRIPT_PREFIX = "script:"; +export const scriptNodeId = (path: string): string => `${SCRIPT_PREFIX}${path}`; +export const isScriptNode = (id: string): boolean => id.startsWith(SCRIPT_PREFIX); +export const scriptPathOf = (id: string): string => id.slice(SCRIPT_PREFIX.length); +const assetNodeId = (kind: string, path: string): string => `${kind}:${path}`; + +// Native trigger kinds that fan out per-event — never bounded-run starts. +// `webhook` / `data_upload` have no trigger row in the graph payload, so a root +// whose only entry is one of those reads as a manual root. +const EVENT_TRIGGER_KINDS = new Set([ + "kafka", + "mqtt", + "nats", + "postgres", + "sqs", + "gcp", + "email", +]); + +/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */ +export function assetUriToNodeId(uri: string): string | undefined { + const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i); + if (!m) return undefined; + const prefix = m[1].toLowerCase(); + const kind = prefix === "s3" ? "s3object" : prefix; + return `${kind}:${m[2]}`; +} + +export type LineageDag = { + down: Map>; + up: Map>; + nodes: Set; +}; + +function addEdge(dag: LineageDag, a: string, b: string) { + if (a === b) return; + dag.nodes.add(a); + dag.nodes.add(b); + (dag.down.get(a) ?? dag.down.set(a, new Set()).get(a)!).add(b); + (dag.up.get(b) ?? dag.up.set(b, new Set()).get(b)!).add(a); +} + +/** Unified upstream→downstream lineage DAG over scripts ∪ assets. */ +export function buildLineageDag(g: BCGraph): LineageDag { + const dag: LineageDag = { down: new Map(), up: new Map(), nodes: new Set() }; + for (const r of g.runnables ?? []) { + if (r.usage_kind === "script") dag.nodes.add(scriptNodeId(r.path)); + } + for (const a of g.assets ?? []) dag.nodes.add(assetNodeId(a.kind, a.path)); + for (const e of g.edges ?? []) { + if (e.runnable_kind !== "script") continue; + const aid = assetNodeId(e.asset_kind, e.asset_path); + const access = e.access_type ?? "r"; + if (access === "w" || access === "rw") { + addEdge(dag, scriptNodeId(e.runnable_path), aid); // producer + } else if (access === "r") { + addEdge(dag, aid, scriptNodeId(e.runnable_path)); // pure reader + } + } + for (const t of g.triggers ?? []) { + if (t.trigger_kind !== "asset" || t.runnable_kind !== "script") continue; + const at = t as Extract; + addEdge(dag, assetNodeId(at.asset_kind, at.asset_path), scriptNodeId(at.runnable_path)); + } + return dag; +} + +function closure(adj: Map>, start: string): Set { + const seen = new Set(); + const queue = [start]; + while (queue.length > 0) { + const cur = queue.shift()!; + for (const n of adj.get(cur) ?? []) { + if (seen.has(n)) continue; + seen.add(n); + queue.push(n); + } + } + // A cycle back to `start` would have re-added it; the contract excludes + // the node itself. + seen.delete(start); + return seen; +} + +export const descendants = (dag: LineageDag, n: string): Set => closure(dag.down, n); +export const ancestors = (dag: LineageDag, n: string): Set => closure(dag.up, n); + +export type BoundedResult = { + nodes: Set; + reachableEnds: string[]; + droppedEnds: string[]; +}; + +/** Path-between node set for `start` and `ends` (inclusive). */ +export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult { + const downSet = new Set(descendants(dag, start)); + downSet.add(start); + const reachableEnds = ends.filter((e) => downSet.has(e)); + const droppedEnds = ends.filter((e) => !downSet.has(e)); + if (reachableEnds.length === 0) { + return { nodes: new Set([start]), reachableEnds, droppedEnds }; + } + const upClosure = new Set(); + for (const e of reachableEnds) { + upClosure.add(e); + for (const a of ancestors(dag, e)) upClosure.add(a); + } + const nodes = new Set(); + for (const n of downSet) if (upClosure.has(n)) nodes.add(n); + nodes.add(start); + return { nodes, reachableEnds, droppedEnds }; +} + +/** Script node ids eligible to start a bounded run. */ +export function validStarts(g: BCGraph): Set { + const subscribers = new Set(); + const scheduleScripts = new Set(); + const eventScripts = new Set(); + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== "script") continue; + if (t.trigger_kind === "asset") subscribers.add(t.runnable_path); + else if (t.trigger_kind === "schedule") scheduleScripts.add(t.runnable_path); + else if (EVENT_TRIGGER_KINDS.has(t.trigger_kind)) eventScripts.add(t.runnable_path); + } + const out = new Set(); + for (const r of g.runnables ?? []) { + if (r.usage_kind !== "script") continue; + const p = r.path; + if (scheduleScripts.has(p)) out.add(scriptNodeId(p)); + else if (!subscribers.has(p) && !eventScripts.has(p)) out.add(scriptNodeId(p)); + } + return out; +} + +/** Project a node-id set to the script paths it contains. */ +export function scriptsOf(nodes: Iterable): string[] { + const out: string[] = []; + for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id)); + return out; +} + +/** + * Resolve a CLI `--to` / `--from` token to a node id, or undefined if it + * matches nothing. Asset URIs (`kind://path`) resolve to the asset node; a bare + * token matches a runnable by exact path or by short (last-segment) name. + */ +export function resolveToken(g: BCGraph, token: string): string | undefined { + if (token.includes("://")) { + const id = assetUriToNodeId(token); + return id && g.assets.some((a) => `${a.kind}:${a.path}` === id) ? id : undefined; + } + const scripts = (g.runnables ?? []).filter((r) => r.usage_kind === "script"); + const exact = scripts.find((r) => r.path === token); + if (exact) return scriptNodeId(exact.path); + const byShort = scripts.filter((r) => (r.path.split("/").pop() ?? r.path) === token); + return byShort.length === 1 ? scriptNodeId(byShort[0].path) : undefined; +} + +/** + * Topological order of `scripts` over the in-set producer→subscriber edges + * (assets collapsed). Scripts on a cycle are returned in `cyclic` and excluded + * from `order`. Serial-run friendly: every script comes after its in-set + * upstreams. + */ +export function topoOrder( + g: BCGraph, + scripts: Set, +): { order: string[]; cyclic: string[] } { + const dag = buildLineageDag(g); + const down = new Map>(); + const indegree = new Map(); + for (const s of scripts) indegree.set(s, 0); + // One-hop (through a single asset) script→script edges, restricted to the set. + for (const s of scripts) { + const sid = scriptNodeId(s); + const oneHop = new Set(); + for (const asset of dag.down.get(sid) ?? []) { + for (const sub of dag.down.get(asset) ?? []) { + if (isScriptNode(sub)) { + const p = scriptPathOf(sub); + if (p !== s && scripts.has(p)) oneHop.add(p); + } + } + } + if (oneHop.size > 0) { + down.set(s, oneHop); + for (const p of oneHop) indegree.set(p, (indegree.get(p) ?? 0) + 1); + } + } + const ready = [...scripts].filter((s) => (indegree.get(s) ?? 0) === 0); + const remaining = new Map(indegree); + const order: string[] = []; + while (ready.length > 0) { + const n = ready.shift()!; + order.push(n); + for (const p of down.get(n) ?? []) { + const d = (remaining.get(p) ?? 0) - 1; + remaining.set(p, d); + if (d === 0) ready.push(p); + } + } + const orderedSet = new Set(order); + const cyclic = [...scripts].filter((s) => !orderedSet.has(s)); + return { order, cyclic }; +} diff --git a/cli/src/commands/pipeline/pipeline.ts b/cli/src/commands/pipeline/pipeline.ts index b4e08f6416..ac01da29a1 100644 --- a/cli/src/commands/pipeline/pipeline.ts +++ b/cli/src/commands/pipeline/pipeline.ts @@ -8,6 +8,18 @@ import { requireLogin } from "../../core/auth.ts"; import { resolveWorkspace } from "../../core/context.ts"; import * as log from "../../core/log.ts"; import { GlobalOptions } from "../../types.ts"; +import { + type BCGraph, + boundedSet, + buildLineageDag, + descendants, + resolveToken, + scriptNodeId, + scriptPathOf, + scriptsOf, + topoOrder, + validStarts, +} from "./boundedCascade.ts"; // Mirrors the asset-graph endpoint payload (backend/windmill-api-assets). // TODO: the checked-in generated client (cli/gen, last regenerated 2025-04) @@ -269,6 +281,196 @@ async function show( console.log(lines.join("\n")); } +// Poll a launched job to a terminal state. Modest fixed cadence; capped so a +// wedged job can't hang the CLI forever. +async function waitJob(workspace: string, id: string): Promise { + const MAX_RETRIES = 6000; // ~10min at 100ms + for (let i = 0; i < MAX_RETRIES; i++) { + try { + const r = await wmill.getCompletedJobResultMaybe({ + workspace, + id, + getStarted: false, + }); + // A completed job without an explicit `success: true` is a failure + // (mirrors the frontend `waitJobTerminal`): the cascade only advances on + // a confirmed success. + if (r.completed) return r.success === true; + } catch { + // transient — retry + } + await new Promise((res) => setTimeout(res, 100)); + } + throw new Error(`Timed out waiting for job ${id}`); +} + +// Bounded-cascade run: start at a schedule / manual root, fan downstream, but +// stop at the `--to` end node(s). Scripts run in topological order; each is +// launched with `_wmill_skip_asset_dispatch` so the CLI owns the whole closure +// (the backend dispatcher never double-fires the deployed part). With no +// `--to`, runs the full read-aware downstream of `--from` (every descendant in +// the lineage DAG, pure readers included — broader than the canvas cascade, +// which dispatches subscribers only). +async function run( + opts: GlobalOptions & { + from?: string; + to?: string[]; + dryRun?: boolean; + json?: boolean; + }, + folder: string, +) { + if (opts.json) log.setSilent(true); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const f = folder.replace(/^f\//, "").replace(/\/$/, ""); + const graph = await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ); + + // Resolve the start: explicit --from (must be a valid start) or the folder's + // sole valid start. + const starts = validStarts(graph); + let start: string; + if (opts.from) { + const resolved = resolveToken(graph, opts.from); + if (!resolved) { + // Distinguish "no match" from "ambiguous short name" (resolveToken + // returns undefined for both) so the hint is actionable. + const matches = graph.runnables.filter( + (r) => r.usage_kind === "script" && (r.path.split("/").pop() ?? r.path) === opts.from, + ); + if (matches.length > 1) { + throw new Error( + `--from '${opts.from}' matches multiple scripts (${matches.map((r) => r.path).sort().join(", ")}) — pass the full path.`, + ); + } + throw new Error(`--from '${opts.from}' matched no script in f/${f}.`); + } + if (!starts.has(resolved)) { + throw new Error( + `--from '${opts.from}' is not a valid bounded-run start. Starts must be schedule-triggered or manual roots; row-backed event triggers (kafka/mqtt/nats/postgres/sqs/gcp/email) fan out per-event and can't be bounded.`, + ); + } + start = resolved; + } else if (starts.size === 1) { + start = [...starts][0]; + } else if (starts.size === 0) { + throw new Error( + `No schedule or manual root in f/${f} to start a bounded run from.`, + ); + } else { + throw new Error( + `f/${f} has ${starts.size} possible starts — pass --from %sveltekit.head% @@ -56,7 +156,7 @@ /> + {:else} + {/if} {/if} diff --git a/frontend/src/lib/components/CompareDrafts.svelte b/frontend/src/lib/components/CompareDrafts.svelte index aca63e4b32..867268cd1a 100644 --- a/frontend/src/lib/components/CompareDrafts.svelte +++ b/frontend/src/lib/components/CompareDrafts.svelte @@ -192,7 +192,7 @@ // side to diff the name against. Raw apps are fetched via the apps endpoint too // (it auto-detects raw from the deployed row and overlays the raw_app draft). const summaryCache = $state< - Record + Record >({}) async function fetchDraftSummary(item: Row) { @@ -216,9 +216,27 @@ path: item.path, getDraft: true }))) as any + // A draft is stale when the version it forked from no longer matches the + // current deployed head: a newer version was deployed after the draft began. + // Scripts compare `parent_hash` vs the deployed `hash`; flows the pinned + // `version_id` vs the deployed head `version_id`; apps the pinned + // `parent_version` vs the deployed head (`versions[last]`). + const draftBlob = r.draft as any + const appHead = Array.isArray(r.versions) ? r.versions[r.versions.length - 1] : undefined + const stale = + item.draftKind === 'script' + ? !!r.hash && !!draftBlob?.parent_hash && draftBlob.parent_hash !== r.hash + : item.draftKind === 'flow' + ? r.version_id != null && + draftBlob?.version_id != null && + draftBlob.version_id !== r.version_id + : appHead != null && + draftBlob?.parent_version != null && + draftBlob.parent_version !== appHead summaryCache[item.key] = { deployed: r.summary, - draft: (r.draft as any)?.summary, + draft: draftBlob?.summary, + stale, loading: false } } catch (error) { @@ -341,13 +359,10 @@ let deployedAny = false for (const item of toDeploy) { deploymentStatus[item.key] = { status: 'loading' } - const res = await deployDraft( - item.draftKind, - item.path, - currentWorkspaceId, - item.draft_only, - item.raw_app - ) + const res = await deployDraft(item.draftKind, item.path, currentWorkspaceId, { + draftOnly: item.draft_only, + rawApp: item.raw_app + }) if (res.success) { deploymentStatus[item.key] = { status: 'deployed' } deployedAny = true @@ -621,6 +636,19 @@ {/snippet} {/if} + {#if draftItem.mine && summaryCache[draftItem.key]?.stale} + + {#snippet trigger()} + + {/snippet} + {#snippet content()} +
+ Started from an older deployed version. A newer version was deployed after this + draft began. Review the latest deploy before deploying. +
+ {/snippet} + + {/if} {#if deploymentStatus[draftItem.key]?.status !== 'deployed'} {#if draftItem.draftKind === 'data_pipeline'} +
+ {#if item.shortcut} + {item.shortcut} + {/if} + {#if item.selected} + + {/if} +
+ {/if} + {#if item.tooltip && !item.disabled} + + + {#snippet text()} + {item.tooltip} + {/snippet} + + {/if} + +{/snippet} + {#if computedItems}
{#each computedItems ?? [] as item} @@ -36,52 +88,14 @@ {/if} {#if item.submenuItems && builders} + {:else if item.disabled && item.tooltip} + +
+ {@render menuItem(item)} +
{:else} - item?.action?.(e)} - href={item?.href} - target={item?.hrefTarget} - disabled={item?.disabled} - class={twMerge( - 'px-4 py-2 text-primary font-normal hover:bg-surface-hover cursor-pointer text-xs transition-colors w-full', - 'data-[highlighted]:bg-surface-hover', - 'flex flex-row gap-2 items-center rounded-sm', - item?.disabled && 'text-disabled cursor-not-allowed', - item?.type === 'delete' && - !item?.disabled && - 'text-red-600 dark:text-red-400 data-[highlighted]:bg-red-500/10 dark:data-[highlighted]:bg-red-900/80 dark:data-[highlighted]:text-red-300 ' - )} - item={meltItem} - aiId={`${aiId ? `${aiId}-${item.displayName}` : undefined}`} - aiDescription={item.displayName} - > - {#if item.icon} - - {/if} -

- {item.displayName} -

- {@render item.extra?.()} - {#if item.shortcut || item.selected} - -
- {#if item.shortcut} - {item.shortcut} - {/if} - {#if item.selected} - - {/if} -
- {/if} - {#if item.tooltip} - - {#snippet text()} - {item.tooltip} - {/snippet} - - {/if} - + {@render menuItem(item)} {/if} {/each}
diff --git a/frontend/src/lib/components/Editor.svelte b/frontend/src/lib/components/Editor.svelte index d0cd91e2fd..af114965b6 100644 --- a/frontend/src/lib/components/Editor.svelte +++ b/frontend/src/lib/components/Editor.svelte @@ -159,11 +159,6 @@ preparedAssetsSqlQueries?: InferAssetsSqlQueryDetails[] | undefined // To execute preview scripts with the right worker group customTag?: string - // Opt-in: reflect external `code` prop mutations back into Monaco (see - // the effect below). One-way `code={...}` callers that need live - // external updates — e.g. the inline flow rawscript — set this. Off by - // default so every other caller's behavior is unchanged. - syncExternalCode?: boolean } let { @@ -196,8 +191,7 @@ enablePreprocessorSnippet = false, rawAppRunnableKey = undefined, preparedAssetsSqlQueries, - customTag, - syncExternalCode = false + customTag }: Props = $props() $effect.pre(() => { @@ -396,23 +390,12 @@ code = ncode } - if (noHistory) { - editor?.setValue(ncode) - } else { - if (editor?.getModel()) { - // editor.setValue(ncode) - editor.pushUndoStop() - - editor.executeEdits('set', [ - { - range: editor.getModel()!.getFullModelRange(), // full range - text: ncode - } - ]) - - editor.pushUndoStop() - } - } + // setCode is an authoritative overwrite (reset, AI apply, module switch). + // Cancel any in-flight keystroke debounce first: otherwise alignCodeWithEditor + // skips on the `timeoutModel` guard (leaving Monaco stale), and the pending + // updateCode later reads the old buffer and writes it back over `ncode`. + cancelPendingChanges() + alignCodeWithEditor(!noHistory) // Dispatch change immediately when code actually changed. This ensures // callers like the Reset button and copilot trigger on:change handlers. // The debounced onDidChangeModelContent handler will no-op since code @@ -446,6 +429,7 @@ return } code = ncode + lastEditorCode = ncode dispatch('change', ncode) } @@ -457,12 +441,19 @@ * see it. Clears the chain state so the next keystroke after this * flush is a fresh leading fire. */ export function flushPendingChanges(): void { + cancelPendingChanges() + updateCode() + } + + /** Discard any in-flight keystroke debounce without materializing it, so a + * deferred updateCode can't fire later. Resets chain state to a fresh leading + * fire on the next keystroke. */ + function cancelPendingChanges(): void { if (timeoutModel !== undefined) { clearTimeout(timeoutModel) timeoutModel = undefined } changeChainStart = undefined - updateCode() } export function append(code: string): void { @@ -1923,29 +1914,6 @@ lang = scriptLangToEditorLang(scriptLang) }) - // Opt-in (syncExternalCode): reflect external `code` prop mutations into - // Monaco's model. Parents that pass `code={...}` one-way (no bind) — e.g. - // the inline rawscript in the flow editor — otherwise mutate the prop - // without Monaco ever showing the change (the AI chat editing a flow - // module's content in a session is the motivating case). Gated off by - // default: Editor is sensitive and most callers either bind:code (and - // carry their own external-sync) or treat code as init-only, so a blanket - // setValue would risk clobbering them. The `getValue() !== code` guard - // keeps the caret intact when the change originated from typing inside - // Monaco (which round-trips code back via `$bindable`, re-firing this - // effect with `code === getValue()`). - let lastExternalCodeSync = code - $effect(() => { - if (!syncExternalCode) return - if (code === lastExternalCodeSync) return - lastExternalCodeSync = code - if (!editor) return - untrack(() => { - if (editor!.getValue() !== code) { - editor!.setValue(code ?? '') - } - }) - }) $effect(() => { filePath = computePath(path) }) @@ -2033,25 +2001,50 @@ }) }) - // External `code` prop changes should flow into the Monaco editor. The - // `untrack` block reads/writes Monaco without subscribing — only the - // prop read above is tracked — so the editor's own change handler - // (`updateCode`) re-running with the same value short-circuits and we - // don't loop. - $effect(() => { - const next = code ?? '' + let applyExternalCode = useDebounce(() => alignCodeWithEditor(true), 800) + + // Last `code` value the editor itself produced or aligned to. Used to tell an + // echo (the bindable changed because the user typed — Monaco is already + // ahead) from a genuine external write. Without this, a typing burst longer + // than the debounce window would sync the lagging `code` back over newer + // keystrokes. Must be kept in step with every editor↔`code` sync point. + let lastEditorCode = code + + function alignCodeWithEditor(history: boolean) { const ed = editor if (!ed) return - untrack(() => { - if (ed.getValue() === next) return - const model = ed.getModel() - if (!model) return + const next = code ?? '' + const value = ed.getValue() + const model = ed.getModel() + // Some keystrokes are still being debounced, don't overwrite them. + // When the debounce is done, updateCode will be called and the code will be aligned with the editor. + if (timeoutModel !== undefined) return + if (!model) return + lastEditorCode = next + if (value === next) return + if (history) { ed.pushUndoStop() ed.executeEdits('external', [{ range: model.getFullModelRange(), text: next }]) ed.pushUndoStop() + } else { + ed.setValue(next) + } + } + + // External `code` prop changes should flow into the Monaco editor. Skip + // echoes: when `code` matches what the editor last produced (`updateCode`) + // or aligned to, the change came from the editor itself, so syncing back + // would clobber input typed since. Only genuine external writes — where + // `code` diverges from `lastEditorCode` — schedule a sync. The `untrack` + // block reads/writes Monaco without subscribing, so we don't loop. + $effect(() => { + ;[code, editor] + if (!editor) return + untrack(() => { + if (code === lastEditorCode) return + applyExternalCode() }) }) - let isTsWorkerInitialized = resource([() => lang, () => initialized], async () => { if (lang !== 'typescript' || !initialized) return false // Use the stable model URI (computed once at mount), not filePath which changes on rename diff --git a/frontend/src/lib/components/FlowBuilder.svelte b/frontend/src/lib/components/FlowBuilder.svelte index 4ba68d6586..2955e4d596 100644 --- a/frontend/src/lib/components/FlowBuilder.svelte +++ b/frontend/src/lib/components/FlowBuilder.svelte @@ -113,6 +113,7 @@ disabledFlowInputs = false, savedPrimarySchedule = undefined, version = undefined, + draftBaseVersion = undefined, draftTriggersFromUrl = undefined, selectedTriggerIndexFromUrl = undefined, children, @@ -220,7 +221,12 @@ } let onLatest = true async function compareVersions() { - if (version === undefined) { + // Compare the draft's pinned fork base against the current head when editing + // a draft, else the load-time head. This catches both a concurrent deploy + // (head moved since open) AND a stale draft reopened after a deploy (head == + // load-time head, but the draft was forked from an older version). + const base = draftBaseVersion ?? version + if (base === undefined) { return } try { @@ -230,7 +236,7 @@ path: initialPath }) - onLatest = version === flowVersion?.id + onLatest = base === flowVersion?.id } else { onLatest = true } diff --git a/frontend/src/lib/components/FlowStatusViewerInner.svelte b/frontend/src/lib/components/FlowStatusViewerInner.svelte index ed1c536ff5..0bc47e4437 100644 --- a/frontend/src/lib/components/FlowStatusViewerInner.svelte +++ b/frontend/src/lib/components/FlowStatusViewerInner.svelte @@ -1180,15 +1180,25 @@ export type FlowModuleForTimeline = { id: string type: FlowModuleValue['type'] + suspend?: boolean } function allModulesForTimeline( modules: FlowModule[], expandedSubflows: Record ): FlowModuleForTimeline[] { - const ids = dfs(modules, (x) => ({ id: x.id, type: x.value.type }) as FlowModuleForTimeline, { - skipToolNodes: true - }) + const ids = dfs( + modules, + (x) => + ({ + id: x.id, + type: x.value.type, + suspend: x.suspend != undefined + }) as FlowModuleForTimeline, + { + skipToolNodes: true + } + ) function rec( ids: FlowModuleForTimeline[], @@ -1208,7 +1218,8 @@ fms, (x) => ({ id: x.id.startsWith('subflow:') ? x.id : buildSubflowKey(x.id, nprefix), - type: x.value.type + type: x.value.type, + suspend: x.suspend != undefined }), { skipToolNodes: true } ), diff --git a/frontend/src/lib/components/FlowTimeline.svelte b/frontend/src/lib/components/FlowTimeline.svelte index f1610c874a..4ed13c1df1 100644 --- a/frontend/src/lib/components/FlowTimeline.svelte +++ b/frontend/src/lib/components/FlowTimeline.svelte @@ -72,6 +72,66 @@ } const barHeight = 32 + + // Whole approval-wait machinery below is inert unless a step actually has a suspend config, + // so large suspend-free flows pay nothing for the flatten/sort and the per-tick recompute. + const hasSuspendModule = $derived(flowModules.some((m) => m.suspend)) + + // Push times of every job on the timeline, ascending. Used to locate when the step + // that follows an approval step started — i.e. the moment the approval was granted. + const allCreatedAts = $derived( + hasSuspendModule + ? Object.values(items ?? {}) + .flat() + .map((j) => j.created_at) + .filter((t): t is number => t != undefined) + .sort((a, b) => a - b) + : [] + ) + + // Heuristic: the grant moment is approximated by the next job pushed anywhere on the + // timeline. Exact for sequential flows; for an approval step inside one branch of a + // parallel branchall a concurrent sibling job can land first and understate the wait. + function nextCreatedAtAfter(t: number): number | undefined { + return allCreatedAts.find((c) => c > t) + } + + // For a completed suspend/approval step, the time spent waiting for the approval is the + // gap between the step finishing and the next step being pushed (or now, if still waiting). + function approvalWait(b: { + started_at?: number + duration_ms?: number + }): { start: number; len: number; running: boolean } | undefined { + if (b.started_at == undefined || b.duration_ms == undefined) { + return undefined + } + const end = b.started_at + b.duration_ms + const next = nextCreatedAtAfter(end) + const waitEnd = next ?? (flowDone ? undefined : now) + if (waitEnd == undefined) { + return undefined + } + const len = waitEnd - end + if (len < 100) { + return undefined + } + return { start: end, len, running: next == undefined } + } + + // Approval wait per module id, computed once and consumed by both the rows and the legend. + const approvalWaitByModule = $derived.by(() => { + const result: Record = {} + for (const m of flowModules) { + if (!m.suspend) continue + const sub = (items?.[m.id] ?? []).filter((x) => x.created_at && x.started_at) + if (sub.length !== 1) continue + const aw = approvalWait(sub[0]) + if (aw) result[m.id] = aw + } + return result + }) + + const hasApprovalWait = $derived(Object.keys(approvalWaitByModule).length > 0) Execution + {#if hasApprovalWait} +
+
+ Approval wait +
+ {/if} {#if max && min} {msToSec(max - min, 1)}s {/if} @@ -113,7 +179,7 @@ /> {/if} - {#each flowModules as { id: k, type: typ } (k)} + {#each flowModules as { id: k, type: typ, suspend: isSuspend } (k)} {@const subItems = items?.[k]?.filter((x) => x.created_at && x.started_at)}
@@ -161,6 +227,7 @@ ? 0 : now - b?.created_at : 0} + {@const aw = isSuspend ? approvalWaitByModule[k] : undefined}
{#if b.started_at} {/if} + {#if aw} + + {/if}
{:else}
@@ -196,7 +277,6 @@
{/each}
- {:else} {/if} diff --git a/frontend/src/lib/components/ForkWorkspaceBanner.svelte b/frontend/src/lib/components/ForkWorkspaceBanner.svelte index 56418c1988..76740163ed 100644 --- a/frontend/src/lib/components/ForkWorkspaceBanner.svelte +++ b/frontend/src/lib/components/ForkWorkspaceBanner.svelte @@ -12,10 +12,14 @@ let comparison: WorkspaceComparison | undefined = $state(undefined) let error: string | undefined = $state(undefined) - let isFork = $derived($workspaceStore?.startsWith('wm-fork-') ?? false) let currentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === $workspaceStore)) let parentWorkspaceId = $derived(currentWorkspaceData?.parent_workspace_id) let parentWorkspaceData = $derived($userWorkspaces.find((w) => w.id === parentWorkspaceId)) + // A fork must have a parent to compare/merge against. Treating the wm-fork- + // prefix alone as "is a fork" renders a parentless "Fork of ()" banner when + // the parent linkage was dropped (e.g. by a workspace id change), so require + // both, matching the forks/compare page. + let isFork = $derived(($workspaceStore?.startsWith('wm-fork-') ?? false) && !!parentWorkspaceId) // Drafts in this fork. When the fork is otherwise in sync with its parent, a // user with only pending drafts should still get the draft CTA (mirrors the diff --git a/frontend/src/lib/components/InstanceSettings.svelte b/frontend/src/lib/components/InstanceSettings.svelte index 1e1416295a..b3d0ee1446 100644 --- a/frontend/src/lib/components/InstanceSettings.svelte +++ b/frontend/src/lib/components/InstanceSettings.svelte @@ -1063,6 +1063,14 @@
  • AI chat usage (provider, model, mode, session count, message count — last 30 days)
    • +
  • resource counts (workspaces, scripts per language, flows, workflows as code, low-code + apps, raw apps)
    • +
  • infrastructure info (container runtime, managed database provider, database version, + size and cluster size, max and active connections, object storage backend)

    • For air-gapped instances, you can download the telemetry data and send it manually. @@ -1101,6 +1109,10 @@
  • AI chat usage (provider, model, mode, session count, message count — last 30 days)
    • +
  • resource counts (workspaces, scripts per language, flows, workflows as code, low-code + apps, raw apps)
    • {/if} diff --git a/frontend/src/lib/components/JobLoader.svelte b/frontend/src/lib/components/JobLoader.svelte index b0d7dae0e0..a32eecf93a 100644 --- a/frontend/src/lib/components/JobLoader.svelte +++ b/frontend/src/lib/components/JobLoader.svelte @@ -336,12 +336,14 @@ callbacks?: Callbacks, flowPath?: string, modules?: Record | null, - tempScriptRefs?: Record + tempScriptRefs?: Record, + timeout?: number ): Promise { return abstractRun( () => JobService.runScriptPreview({ workspace: $workspaceStore!, + timeout, requestBody: { path, content: code, diff --git a/frontend/src/lib/components/ModuleTest.svelte b/frontend/src/lib/components/ModuleTest.svelte index 02f51ae187..785e111527 100644 --- a/frontend/src/lib/components/ModuleTest.svelte +++ b/frontend/src/lib/components/ModuleTest.svelte @@ -56,6 +56,16 @@ runTestWithStepArgs() } + // A step's timeout is an InputTransform. Only a static numeric value can be applied + // to a single-step preview; dynamic expressions are evaluated server-side and only + // take effect when running the full flow. + function staticTimeout(timeout: FlowModule['timeout']): number | undefined { + if (timeout?.type === 'static' && typeof timeout.value === 'number') { + return timeout.value + } + return undefined + } + export async function runTest(args: any) { // Not defined if JobProgressBar not loaded if (jobProgressReset) jobProgressReset() @@ -68,6 +78,7 @@ } const val = mod.value + const timeout = staticTimeout(mod.timeout) // let jobId: string | undefined = undefined let callbacks: Callbacks = { done: (x) => { @@ -86,7 +97,8 @@ callbacks, $pathStore, undefined, - devTempScriptRefs?.() + devTempScriptRefs?.(), + timeout ) } else if (val.type == 'script') { const script = val.hash @@ -101,7 +113,10 @@ script.lock, val.hash ?? script.hash, callbacks, - $pathStore + $pathStore, + undefined, + undefined, + timeout ) } else if (val.type == 'flow') { await jobLoader?.runFlowByPath(val.path, args, callbacks) diff --git a/frontend/src/lib/components/ObjectStoreConfigSettings.svelte b/frontend/src/lib/components/ObjectStoreConfigSettings.svelte index 5c96007d33..1896f5696e 100644 --- a/frontend/src/lib/components/ObjectStoreConfigSettings.svelte +++ b/frontend/src/lib/components/ObjectStoreConfigSettings.svelte @@ -1,5 +1,5 @@ @@ -37,20 +43,26 @@ {/if} {#snippet text()} - {id} + {#if tooltip} + {tooltip} + {:else} + {id} + {/if} {/snippet} {#if len > 0} {@const narrow = len / total < 0.09} diff --git a/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte b/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte index 2ee01b4607..90abb0ebf2 100644 --- a/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte +++ b/frontend/src/lib/components/WorkspaceItemDiffViewer.svelte @@ -4,6 +4,9 @@ Inline diff renderer for a single workspace item. Mirrors the per-kind rendering that DiffDrawer does in its body (`DiffDrawer.svelte:181-271`): - `flow` → `` (its own Graph / YAML toggle inside) +- `raw_app_file` → `` (one synthesized raw-app file item: a + single diff with a per-file size guard; the metadata item adds a full-app + YAML expand). Raw apps are exploded into these items by `rawAppDiffToItems`. - has `content` (scripts) → Tabs(Content | Metadata) with two Monaco diffs - everything else (apps, resources, variables, schedules, triggers…) → a single Monaco YAML diff over the metadata @@ -18,12 +21,15 @@ doesn't reflow the parent. import Tabs from './common/tabs/Tabs.svelte' import Tab from './common/tabs/Tab.svelte' import FlowDiffViewer from './FlowDiffViewer.svelte' + import RawAppFileDiff from './raw_apps/RawAppFileDiff.svelte' + import type { RawAppFileItem } from './raw_apps/rawAppDiffUtils' import { Loader2 } from 'lucide-svelte' import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils' import { scriptLangToEditorLang } from '$lib/scripts' interface Props { - /** Any WorkspaceItemDiff['kind'] — used only to special-case `flow`. */ + /** Any WorkspaceItemDiff['kind'], plus the synthetic `raw_app_file`. + * `flow` and `raw_app_file` are special-cased. */ kind: string /** Raw value from `getItemValue(kind, path, parentWorkspace)`. Undefined * for "added" items (don't exist in the parent). */ @@ -33,9 +39,11 @@ doesn't reflow the parent. currentRaw?: unknown /** Force unified diff (Monaco renderSideBySide=false). Default false. */ inlineDiff?: boolean + /** For `raw_app_file`: the synthesized per-file diff item to render. */ + rawFile?: RawAppFileItem } - let { kind, originalRaw, currentRaw, inlineDiff = false }: Props = $props() + let { kind, originalRaw, currentRaw, inlineDiff = false, rawFile }: Props = $props() type Prepared = { lang?: string; content?: string; metadata: string } @@ -102,6 +110,16 @@ doesn't reflow the parent. {inlineDiff} /> +{:else if kind === 'raw_app_file' && rawFile} + {:else if hasContent}
    diff --git a/frontend/src/lib/components/WorkspaceItemRow.svelte b/frontend/src/lib/components/WorkspaceItemRow.svelte index fe923b12ca..6e5900594f 100644 --- a/frontend/src/lib/components/WorkspaceItemRow.svelte +++ b/frontend/src/lib/components/WorkspaceItemRow.svelte @@ -3,7 +3,9 @@ Visual row for a workspace item (script / flow / app / resource / schedule / trigger / …). Matches the leaf-row layout used by WorkspaceItemDrillPicker: RowIcon + summary line on top with mono path -beneath, or just the mono path when there's no summary. +beneath, or just the mono path when there's no summary. With `singleLine`, +both collapse to one row showing `summary ?? secondary` (summary in normal +text, the mono path as the fallback) — denser, for the diff tree. Pure presentation — the caller controls highlighting / current state via props, supplies the onclick/onmouseenter handlers, and can pass an @@ -26,6 +28,9 @@ doesn't steal focus from a sibling search input (matches the picker). /** For `kind: 'trigger'`, specifies the concrete trigger subtype. * Forwarded to RowIcon. */ triggerKind?: string + /** For `kind: 'raw_app_file'`, the file name/path — forwarded to RowIcon + * to pick an extension-specific icon. */ + iconPath?: string /** Optional summary text shown above the path. */ summary?: string /** Mono path (or any secondary identifier). When summary is empty @@ -47,6 +52,9 @@ doesn't steal focus from a sibling search input (matches the picker). /** Reserve two lines of height and vertically center the content so * summary and summary-less rows are the same height (diff viewer). */ uniformHeight?: boolean + /** Collapse to a single line showing `summary ?? secondary` (summary in + * normal text, secondary in mono) instead of stacking both. */ + singleLine?: boolean /** Extra left padding (px) for tree-view indentation. Adds to the * default `px-3` horizontal padding. */ indent?: number @@ -69,6 +77,7 @@ doesn't steal focus from a sibling search input (matches the picker). let { kind, triggerKind, + iconPath, summary, secondary, highlighted = false, @@ -82,7 +91,8 @@ doesn't steal focus from a sibling search input (matches the picker). onclick, onmouseenter, extras, - uniformHeight = false + uniformHeight = false, + singleLine = false }: Props = $props() const rootClass = $derived( @@ -96,6 +106,37 @@ doesn't steal focus from a sibling search input (matches the picker). ) +{#snippet body()} + +
    + {#if singleLine} +
    + {summary ?? secondary} +
    + {:else if summary} +
    {summary}
    +
    + {secondary} +
    + {:else} +
    + {secondary} +
    + {/if} +
    + {#if extras} +
    + {@render extras()} +
    + {/if} +{/snippet} + {#if href} - -
    - {#if summary} -
    {summary}
    -
    - {secondary} -
    - {:else} -
    - {secondary} -
    - {/if} -
    - {#if extras} -
    - {@render extras()} -
    - {/if} + {@render body()}     {:else} {/if} diff --git a/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte b/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte index 85f0e3193e..14832214b0 100644 --- a/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte +++ b/frontend/src/lib/components/apps/components/display/AppNavbarItem.svelte @@ -13,7 +13,7 @@ import { twMerge } from 'tailwind-merge' import type { Output } from '../../rx' import ResolveNavbarItemPath from './ResolveNavbarItemPath.svelte' - import { urlParamsToObject } from '$lib/utils' + import { urlParamsToObject, WINDMILL_RESERVED_QUERY_PARAMS } from '$lib/utils' interface Props { navbarItem: NavbarItem @@ -54,8 +54,22 @@ let resolvedHidden: boolean | undefined = $state(undefined) function extractPathDetails() { - const url = window.location.pathname + window.location.search + window.location.hash - const processedUrl = url.replace('/apps/edit/', '').replace('/apps/get/', '') + // Drop Windmill transport params (wm_embed, …) so they don't poison the + // comparison against the item's resolved path. + const params = new URLSearchParams(window.location.search) + const reserved: string[] = [] + params.forEach((_v, k) => { + if (WINDMILL_RESERVED_QUERY_PARAMS.has(k)) reserved.push(k) + }) + reserved.forEach((k) => params.delete(k)) + const qs = params.toString() + const url = window.location.pathname + (qs ? `?${qs}` : '') + window.location.hash + // `/app_embed/{workspace}/` is the opaque in-workspace viewer route + // (WIN-2006) — same app-path suffix as `/apps/get/`. + const processedUrl = url + .replace('/apps/edit/', '') + .replace('/apps/get/', '') + .replace(/^\/app_embed\/[^/]+\//, '') return processedUrl } diff --git a/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte b/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte index ba17f6c7c3..8e068274b8 100644 --- a/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte +++ b/frontend/src/lib/components/apps/components/display/dbtable/AppDbExplorer.svelte @@ -86,7 +86,10 @@ } const resolvedConfig = $state( - initConfig(components['dbexplorercomponent'].initialData.configuration, untrack(() => configuration)) + initConfig( + components['dbexplorercomponent'].initialData.configuration, + untrack(() => configuration) + ) ) let timeoutInput: number | undefined = undefined @@ -180,18 +183,22 @@ ) } - let outputs = initOutput($worldStore, untrack(() => id), { - selectedRowIndex: 0, - selectedRow: {}, - selectedRows: [] as any[], - result: [] as any[], - inputs: {}, - loading: false, - page: 0, - newChange: { row: 0, column: '', value: undefined }, - ready: undefined as boolean | undefined, - openedModalRow: {} - }) + let outputs = initOutput( + $worldStore, + untrack(() => id), + { + selectedRowIndex: 0, + selectedRow: {}, + selectedRows: [] as any[], + result: [] as any[], + inputs: {}, + loading: false, + page: 0, + newChange: { row: 0, column: '', value: undefined }, + ready: undefined as boolean | undefined, + openedModalRow: {} + } + ) let lastResource: string | undefined = undefined @@ -260,9 +267,7 @@ resolvedConfig.type, { table: { - selectOptions: dbSchemas - ? await getTablesByResource(dbSchemas, dbtype, dbPath, $workspaceStore!) - : [], + selectOptions: dbSchemas ? await getTablesByResource(dbSchemas, dbtype) : [], loading: false } } diff --git a/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts b/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts index 3ec21a3801..1d0fa63342 100644 --- a/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts +++ b/frontend/src/lib/components/apps/components/display/dbtable/metadata.ts @@ -1,4 +1,4 @@ -import { JobService, ResourceService } from '$lib/gen' +import { JobService } from '$lib/gen' import { runScriptAndPollResult } from '$lib/components/jobs/utils' import type { DbInput } from '$lib/components/dbTypes' @@ -39,17 +39,9 @@ export async function loadTableMetaData( const ducklake = input.type === 'ducklake' ? input.ducklake : undefined const dbArg = getDatabaseArg(input) - // MySQL needs the database name for metadata queries - let databaseName: string | undefined - if (input.type === 'database' && input.resourceType === 'mysql') { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: input.resourcePath - })) as any - databaseName = resourceObj?.database - } - - const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table, databaseName }, ducklake) + // MySQL: the metadata query resolves the database name server-side (it falls + // back to `DATABASE()`), so we don't read the resource value client-side for it. + const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table }, ducklake) const job = await JobService.runScriptPreview({ workspace, @@ -106,22 +98,10 @@ export async function loadAllTablesMetaData( const dbArg = getDatabaseArg(input) const ducklake = input.type === 'ducklake' ? input.ducklake : undefined - // MySQL needs the database name for metadata queries - let databaseName: string | undefined - if (input.type === 'database' && input.resourceType === 'mysql') { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: input.resourcePath - })) as any - databaseName = resourceObj?.database - } - const language = getLanguageByResourceType(dbType) - const content = makeMetadataMarker( - 'LOAD_TABLE_METADATA', - { table: undefined, databaseName }, - ducklake - ) + // MySQL db name is resolved server-side via `DATABASE()` (see loadTableMetaData); + // no client-side resource-value read. + const content = makeMetadataMarker('LOAD_TABLE_METADATA', { table: undefined }, ducklake) let result = (await runScriptAndPollResult({ workspace, @@ -259,7 +239,11 @@ export async function getDbSchemas( const dbSchema = { lang: resourceTypeToLang(resourceType) as SQLSchema['lang'], schema, - publicOnly: !!schema.public || !!schema.PUBLIC || !!schema.dbo + publicOnly: !!schema.public || !!schema.PUBLIC || !!schema.dbo, + // MySQL introspection selects `DATABASE() AS default_db_name`; carry it + // so the table picker can tell the default db apart from other visible + // schemas. Other dbs don't return it (stays undefined). + defaultDb: Array.isArray(result) ? (result[0] as any)?.default_db_name : undefined } return { ...dbSchema, stringified: stringifySchema(dbSchema) } } else { @@ -283,9 +267,7 @@ export async function getDbSchemas( export async function getTablesByResource( schema: Partial>, - dbType: DbType | undefined, - dbPath: string, - workspace: string + dbType: DbType | undefined ): Promise { const s = Object.values(schema)?.[0] switch (dbType) { @@ -301,14 +283,15 @@ export async function getTablesByResource( return paths } case 'mysql': { - const resourceObj = (await ResourceService.getResourceValue({ - workspace, - path: dbPath.split('$res:')[1] - })) as any + // MySQL introspection lists DATABASE() plus any other visible non-system + // schemas. Show the default db's tables unprefixed and the rest as + // `db.table` — matching the pre-removal behavior (which matched the + // resource's `database`); `defaultDb` is the connection's DATABASE(). + const defaultDb = s && 'defaultDb' in s ? s.defaultDb : undefined const paths: string[] = [] for (const key in s?.schema) { for (const subKey in s.schema[key]) { - if (key === resourceObj?.database) { + if (key === defaultDb) { paths.push(`${subKey}`) } else { paths.push(`${key}.${subKey}`) diff --git a/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte b/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte index dd294be6af..c7c4e7d160 100644 --- a/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte +++ b/frontend/src/lib/components/apps/components/helpers/RunnableWrapper.svelte @@ -3,7 +3,7 @@ import type { AppInput } from '../../inputType' import type { Output } from '../../rx' import type { AppViewerContext, ListContext } from '../../types' - import { isScriptByNameDefined, isScriptByPathDefined } from '../../utils' + import { appNavigateSameWindow, isScriptByNameDefined, isScriptByPathDefined } from '../../utils' import NonRunnableComponent from './NonRunnableComponent.svelte' import RunnableComponent from './RunnableComponent.svelte' import { sendUserToast } from '$lib/toast' @@ -261,7 +261,9 @@ if (newTab) { window.open(gotoUrl, '_blank') } else { - window.location.href = gotoUrl + // Top-level load; inside the opaque viewer iframe this targets the + // top page (pre-sandbox behavior) instead of the cookieless frame. + appNavigateSameWindow(gotoUrl) } break diff --git a/frontend/src/lib/components/apps/components/helpers/eval.ts b/frontend/src/lib/components/apps/components/helpers/eval.ts index 86523b2959..bacfb65b2d 100644 --- a/frontend/src/lib/components/apps/components/helpers/eval.ts +++ b/frontend/src/lib/components/apps/components/helpers/eval.ts @@ -2,6 +2,8 @@ import type { World } from '../../rx' import { sendUserToast } from '$lib/toast' import { waitJob } from '$lib/components/waitJob' import { base } from '$lib/base' +import { appNavigateSameWindow } from '../../utils' +import { OpenAPI } from '$lib/gen/core/OpenAPI' export function computeGlobalContext( world: World | undefined, @@ -200,7 +202,9 @@ export async function eval_like( } window.open(x, '_blank') } else { - window.location.href = x + // Top-level load; inside the opaque viewer iframe this targets the + // top page (pre-sandbox behavior) instead of the cookieless frame. + appNavigateSameWindow(x) } }, (id, index) => { @@ -292,10 +296,30 @@ export async function eval_like( if (typeof input === 'object' && input.s3) { const workspaceId = ((context ?? {}) as any).ctx?.workspace - const s3href = `${base}/api/w/${workspaceId}/job_helpers/download_s3_file?file_key=${encodeURIComponent( - input?.s3 ?? '' - )}${input?.storage ? `&storage=${input.storage}` : ''}` - downloadFile(s3href, filename || input.s3) + const appPath = ((context ?? {}) as any).ctx?.app_path + let inSandbox = false + try { + inSandbox = + window.parent !== window && + new URLSearchParams(window.location.search).get('wm_embed') === '1' + } catch (_) {} + if (inSandbox && appPath && typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + // Sandboxed viewer: the opaque iframe carries no cookie, so the + // cookie-authed job_helpers download fails. Route through the + // app-policy-confined apps_u endpoint with the embed token in the + // query (like the image/file components), scoped to this app's path. + const params = new URLSearchParams() + params.append('s3', input.s3 ?? '') + if (input.storage) params.append('storage', input.storage) + params.append('token', OpenAPI.TOKEN) + const s3href = `${base}/api/w/${workspaceId}/apps_u/download_s3_file/${appPath}?${params.toString()}` + downloadFile(s3href, filename || input.s3) + } else { + const s3href = `${base}/api/w/${workspaceId}/job_helpers/download_s3_file?file_key=${encodeURIComponent( + input?.s3 ?? '' + )}${input?.storage ? `&storage=${input.storage}` : ''}` + downloadFile(s3href, filename || input.s3) + } } else if (typeof input === 'string') { if (input.startsWith('data:')) { downloadFile(input, filename) diff --git a/frontend/src/lib/components/apps/editor/AppEditor.svelte b/frontend/src/lib/components/apps/editor/AppEditor.svelte index 0cffdc2400..64cc94f551 100644 --- a/frontend/src/lib/components/apps/editor/AppEditor.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditor.svelte @@ -69,6 +69,7 @@ path, policy, summary, + labels, deployedBaseline = undefined, fromHub = false, diffDrawer = undefined, @@ -83,7 +84,8 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore }: AppEditorProps = $props() migrateApp(untrack(() => app)) @@ -885,12 +887,13 @@ void @@ -112,6 +115,10 @@ loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void } let { @@ -125,6 +132,7 @@ bottomPanelHidden = false, newApp, newPath = '', + labels: initialLabels = undefined, userDraftPath = '', onSavedNewAppPath, onShowLeftPanel, @@ -137,7 +145,8 @@ onResetToDeployed, loadedFromDraft = false, othersDraftsCount = 0, - onOpenOthersDrafts + onOpenOthersDrafts, + onRestore }: Props = $props() /** Mirror of the path the user is editing in the pen popover. Initialized @@ -256,7 +265,8 @@ policy, deployment_message: deploymentMsg, custom_path: customPath, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels } }) // New path now exists server-side — drop the autocomplete cache so @@ -267,7 +277,8 @@ value: structuredClone($state.snapshot($app)), path: path, policy: policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } closeSaveDrawer() sendUserToast('App deployed successfully') @@ -310,7 +321,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels }) ) ) { @@ -361,7 +373,8 @@ // it also means that customPath needs to be set to '' instead of undefined to unset it (when admin) custom_path: $userStore?.is_admin || $userStore?.is_super_admin ? (customPath ?? '') : undefined, - preserve_on_behalf_of: preserveOnBehalfOf || undefined + preserve_on_behalf_of: preserveOnBehalfOf || undefined, + labels } }) invalidateWorkspacePaths($workspaceStore!) @@ -370,13 +383,19 @@ value: structuredClone($state.snapshot($app)), path: npath, policy, - custom_path: customPath + custom_path: customPath, + labels: $state.snapshot(labels) } const appHistory = await AppService.getAppHistoryByPath({ workspace: $workspaceStore!, path: npath }) version = appHistory[0]?.version + // Re-pin the fork base to the just-deployed head: the editor stays open, so a + // follow-up deploy (or a new edit) would otherwise compare against the now- + // superseded base and falsely warn. parent_version is in + // DRAFT_COMPARE_IGNORED_FIELDS, so this write can't spawn a spurious draft. + if ($app) $app.parent_version = version closeSaveDrawer() sendUserToast('App deployed successfully') @@ -395,14 +414,16 @@ } } - async function setPublishState() { + async function setPublishState(message?: string) { policy = await updatePolicy($app, policy) await AppService.updateApp({ workspace: $workspaceStore!, path: $appPath, requestBody: { policy } }) - if (policy.execution_mode == 'anonymous') { + if (message) { + sendUserToast(message) + } else if (policy.execution_mode == 'anonymous') { sendUserToast('App require no login to be accessed') } else { sendUserToast('App require login and read-access') @@ -419,7 +440,12 @@ let onLatest = $state(true) async function compareVersions() { - if (version === undefined) { + // Compare the draft's pinned fork base (`$app.parent_version`) against the + // current head when editing a draft, else the load-time head. Catches both a + // concurrent deploy (head moved since open) AND a stale draft reopened after a + // deploy (head == load-time head, but the draft was forked from an older one). + const base = $app?.parent_version ?? version + if (base === undefined) { return } try { @@ -427,7 +453,7 @@ workspace: $workspaceStore!, path: $appPath }) - onLatest = appVersion?.version === undefined || version === appVersion?.version + onLatest = appVersion?.version === undefined || base === appVersion?.version } catch (e) { console.error('Error comparing versions', e) onLatest = true @@ -612,7 +638,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels } }) }, @@ -711,6 +738,7 @@ }) let customPath = $state(savedApp?.custom_path) + let labels = $state(untrack(() => initialLabels)) $effect(() => { if ($openDebugRun == undefined) { @@ -740,7 +768,8 @@ value: $app, path: newEditedPath || savedApp?.path, policy, - custom_path: customPath + custom_path: customPath, + labels }} /> @@ -784,7 +813,8 @@ value: $app, path: newEditedPath || savedApp.path, policy, - custom_path: customPath + custom_path: customPath, + labels }, button: { text: 'Looks good, deploy', @@ -837,6 +867,7 @@ bind:pathError bind:newEditedPath bind:preserveOnBehalfOf + bind:labels hideSecretUrl={false} /> @@ -850,7 +881,7 @@ (historyBrowserDrawerOpen = false)}> - + onRestore?.(e.detail)} appPath={$appPath} /> diff --git a/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte b/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte index c344114404..01c0cb3028 100644 --- a/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte +++ b/frontend/src/lib/components/apps/editor/AppEditorHeaderDeploy.svelte @@ -1,5 +1,6 @@ + + { + refresh = requestTokenRefresh + loadApp() + }} +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + + +{#if canWriteApp && !hideEditBtn} +
    + +
    +{/if} diff --git a/frontend/src/lib/components/apps/editor/PublicApp.svelte b/frontend/src/lib/components/apps/editor/PublicApp.svelte index fef6ba8fa3..92cde8788b 100644 --- a/frontend/src/lib/components/apps/editor/PublicApp.svelte +++ b/frontend/src/lib/components/apps/editor/PublicApp.svelte @@ -7,8 +7,13 @@ import { isCloudHosted } from '$lib/cloud' import { Alert, Skeleton } from '$lib/components/common' import { WindmillIcon } from '$lib/components/icons' - import { onMount, setContext } from 'svelte' - import { IS_APP_PUBLIC_CONTEXT_KEY, type EditorBreakpoint } from '../types' + import { getContext, onMount, setContext } from 'svelte' + import { + EMBED_NAV_CONTEXT_KEY, + IS_APP_PUBLIC_CONTEXT_KEY, + type EditorBreakpoint, + type EmbedNav + } from '../types' import { UserService, type AppWithLastVersion, type GlobalWhoamiResponse } from '$lib/gen' import { urlParamsToObject } from '$lib/utils' import { goto } from '$app/navigation' @@ -24,7 +29,9 @@ jwtError, onLoginSuccess, app, - workspace + workspace, + inWorkspace = false, + hideRefreshBar = false }: { notExists: boolean noPermission: boolean @@ -32,12 +39,27 @@ onLoginSuccess: () => void app: (AppWithLastVersion & { value: any; workspace_id?: string }) | undefined workspace: string | undefined + /** + * In-workspace rendering (`/apps/get`, `/app_embed`): keep exact parity + * with the pre-sandbox member viewer — no "Powered by Windmill" badge, no + * user overlay, no HTML-result approval gate, column flex wrapper. + */ + inWorkspace?: boolean + hideRefreshBar?: boolean } = $props() // Use workspace from props or from app.workspace_id (for custom path responses) let effectiveWorkspace = $derived(workspace ?? app?.workspace_id) - setContext(IS_APP_PUBLIC_CONTEXT_KEY, true) + // HTML results from runnables only need viewer approval on the public + // surfaces (untrusted distribution); the in-workspace viewer never gated them. + setContext(IS_APP_PUBLIC_CONTEXT_KEY, !inWorkspace) + + // WIN-2006: inside the opaque viewer iframe, navigations to other routes + // (navbar "app" items) must happen on the TOP page — the iframe is cookieless, + // so navigating it would just show a login screen. PublicAppFrame provides the + // relay; outside the opaque viewer this is undefined and goto works directly. + const embedNav = getContext(EMBED_NAV_CONTEXT_KEY) const breakpoint = writable('lg') @@ -71,27 +93,29 @@ }) -
    - Powered by   Windmill -
    + Powered by   Windmill +
    -{#snippet userInfo(child)} -
    {child}
    -{/snippet} + {#snippet userInfo(child)} +
    {child}
    + {/snippet} -
    {#if $userStore} - {@render userInfo($userStore.username)} - {:else if globalUser} - {@render userInfo(globalUser.email)} - {:else}{/if} -
    +
    {#if $userStore} + {@render userInfo($userStore.username)} + {:else if globalUser} + {@render userInfo(globalUser.email)} + {:else}{/if} +
    +{/if} {#if notExists}
    goto(path)} - gotoFn={(path, opt) => goto(path, opt)} + gotoFn={(path, opt) => (embedNav ? embedNav.navigateTop(path) : goto(path, opt))} />
    {/if} diff --git a/frontend/src/lib/components/apps/editor/PublicAppFrame.svelte b/frontend/src/lib/components/apps/editor/PublicAppFrame.svelte new file mode 100644 index 0000000000..43dc937651 --- /dev/null +++ b/frontend/src/lib/components/apps/editor/PublicAppFrame.svelte @@ -0,0 +1,427 @@ + + +{#if isViewer} + {#if viewerReady} + {@render viewer()} + {:else if viewerOrphaned} +
    + + This is a Windmill app viewer and must be loaded by Windmill. If you embedded it in your own + page, use the app's public URL without the wm_embed parameter. + +
    + {:else} + + {/if} +{:else if status === 'loading'} + +{:else if status === 'notExists'} +
    + + There was an error loading the app, is the url correct? + Go to Windmill + +
    +{:else if status === 'noPermission'} + +
    This app requires read access
    +
    + initEmbedder()} + popup + rd={page.url.pathname + page.url.search + page.url.hash} + /> +
    +{:else if unsandboxed} + + {@render viewer()} +{:else if isRaw} + + {@render viewer()} +{:else} + + +{/if} diff --git a/frontend/src/lib/components/apps/editor/appPolicy.ts b/frontend/src/lib/components/apps/editor/appPolicy.ts index 8c54926f1b..b6e35bd65e 100644 --- a/frontend/src/lib/components/apps/editor/appPolicy.ts +++ b/frontend/src/lib/components/apps/editor/appPolicy.ts @@ -180,12 +180,13 @@ export async function updatePolicy(app: App, currentPolicy: Policy | undefined): }) .filter(Boolean) as { s3_path: string; storage?: string | undefined }[] - return { + const next = { ...(currentPolicy ?? {}), allowed_s3_keys: s3FileKeys, s3_inputs, triggerables_v2: ntriggerables } + return next } export async function processRunnable( diff --git a/frontend/src/lib/components/apps/types.ts b/frontend/src/lib/components/apps/types.ts index e4e030de67..960e273ac3 100644 --- a/frontend/src/lib/components/apps/types.ts +++ b/frontend/src/lib/components/apps/types.ts @@ -144,6 +144,8 @@ export interface AppEditorProps { path: string policy: Policy summary: string + /** Initial labels for the app, threaded from the loaded app data. */ + labels?: string[] /** Deployed app value the autosave `discardIf` compares against, so an * edit reverting to deployed clears the draft instead of leaving a no-op. * `undefined` for draft-only paths (no deployed baseline). */ @@ -157,6 +159,7 @@ export interface AppEditorProps { summary: string policy: any custom_path?: string + labels?: string[] } | undefined version?: number | undefined @@ -176,6 +179,10 @@ export interface AppEditorProps { loadedFromDraft?: boolean othersDraftsCount?: number onOpenOthersDrafts?: () => void + // Restoring an older deployment from the history drawer. Threaded through + // AppEditorHeader as a callback prop rather than `on:restore` forwarding, + // which does not propagate through these runes-mode components. + onRestore?: (restoredApp: any) => void } export type App = { @@ -198,6 +205,13 @@ export type App = { hideLegacyTopBar?: boolean | undefined mobileViewOnSmallerScreens?: boolean | undefined version?: number + /** + * Fork base for the stale-draft check: the deployed app version this draft + * was started from, pinned at fork. Stamped on the draft seed in the editor; + * compared against the deployed head (`versions[last]`) in the compare view. + * In DRAFT_COMPARE_IGNORED_FIELDS so it never trips the autosave no-op check. + */ + parent_version?: number /** * User-typed path persisted on the autosaved App when it differs from * the deployed/seeded baseline. The home list renders it so a friendly @@ -370,6 +384,12 @@ export type EditorBreakpoint = 'sm' | 'lg' export const IS_APP_PUBLIC_CONTEXT_KEY = 'isAppPublicContext' as const +// Set by PublicAppFrame in opaque-viewer mode (WIN-2006). Lets the app relay +// top-level navigations (e.g. navbar links to another app) to the embedder, +// since navigating inside the opaque iframe would load the SPA cookieless. +export const EMBED_NAV_CONTEXT_KEY = 'appEmbedNav' as const +export type EmbedNav = { navigateTop: (href: string) => void } + type ComponentID = string export type ContextPanelContext = { diff --git a/frontend/src/lib/components/apps/utils.ts b/frontend/src/lib/components/apps/utils.ts index 85bdd774eb..2dfdd42e6b 100644 --- a/frontend/src/lib/components/apps/utils.ts +++ b/frontend/src/lib/components/apps/utils.ts @@ -20,6 +20,31 @@ import type { } from './types' import { allItems, BG_PREFIX } from './editor/appUtilsCore' +/** + * Same-window navigation for app code (frontend-script `goto`, button + * `onSuccess: gotoUrl`). Inside the opaque viewer iframe (WIN-2006, + * `wm_embed=1`), navigating the current window would load the target inside + * the cookieless frame — so the navigation is relayed to the embedder page, + * which navigates itself (`wm_embed_navigate` in PublicAppFrame). That matches + * the pre-sandbox behavior exactly: the app used to run ON the embedder page, + * including when that page is itself inside a third-party iframe (where the + * embedder — not the third party's top — was what `window.location` changed). + * Outside the opaque viewer it keeps navigating the current window as before. + */ +export function appNavigateSameWindow(url: string) { + try { + const params = new URLSearchParams(window.location.search) + if (window.parent !== window && params.get('wm_embed') === '1') { + window.parent.postMessage( + { type: 'wm_embed_navigate', href: url }, + params.get('wm_embedder_origin') ?? '*' + ) + return + } + } catch (_) {} + window.location.href = url +} + // `migrateApp` moved to its own light module so non-editor callers can reuse it // without pulling the whole `apps/utils` graph; re-exported here for existing // `from '../utils'` importers. diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte index c9cc92f3b8..c13e9abc10 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphCanvas.svelte @@ -13,10 +13,12 @@ import RunnableNode from './RunnableNode.svelte' import TriggerNode, { type TriggerNodeKind } from './TriggerNode.svelte' import AddNode from './AddNode.svelte' + import DataTestNode from './DataTestNode.svelte' import AssetGraphEdge from './AssetGraphEdge.svelte' import PanToNode from './PanToNode.svelte' import { layoutAssetGraph } from './assetGraphLayout' import { buildDownstreamMap } from './graphTraversal' + import { buildLineageDownstreamMap } from './boundedCascade' import type { AssetGraphResponse, AssetGraphSelection, NativeTriggerKind } from './types' import type { RunnableRunState } from './activeRunnables.svelte' import type { AssetKind } from '$lib/gen' @@ -139,6 +141,24 @@ // editor passes it, so the asset-graph page is unaffected. The page // clears it once the pan has had time to settle. panToNodeId?: string | undefined + // Script paths eligible to *start* a bounded-cascade run (schedule + // roots + manual roots — see boundedCascade.validStarts). When a path is + // in this set and `onStartBoundedRun` is wired, its node's cascade menu + // gains a "Run downstream up to…" entry. + validStartPaths?: ReadonlySet + onStartBoundedRun?: (startPath: string) => void + // Active bounded-run pick mode. Ids are in *canvas* space (`script:path` + // for runnables, `asset:${kind}:${path}` for assets). When set the canvas + // dims nodes outside `eligible ∪ {start}`, rings the `bounded` set, marks + // `ends`, and routes clicks on eligible nodes to `onPickEnd` instead of + // selecting. + boundPick?: { + start: string + eligible: ReadonlySet + ends: ReadonlySet + bounded: ReadonlySet + } + onPickEnd?: (canvasNodeId: string) => void } let { graph, @@ -160,7 +180,11 @@ onOpenDataUpload, hoveredPaths, selectedRunPaths, - panToNodeId + panToNodeId, + validStartPaths, + onStartBoundedRun, + boundPick, + onPickEnd }: Props = $props() // `${kind}:${path}` ids for the hovered / pinned runs (both script and flow @@ -180,12 +204,24 @@ // the DAG. They force sugiyama to put + at layer 0 (top) and center // it horizontally over the roots — same mechanism the flow editor // uses for its Trigger node. Filtered out of rendered edges. - kind: 'lineage-write' | 'lineage-read' | 'trigger-asset' | 'trigger-native' | 'add-anchor' + kind: + | 'lineage-write' + | 'lineage-read' + | 'trigger-asset' + | 'trigger-native' + | 'add-anchor' + | 'data-test' unsaved?: boolean // Edge from a missing-trigger placeholder — styled red dashed to // signal "this script declared `// on kafka` but no trigger row // targets it; create one or remove the annotation". missing?: boolean + // Producer's `// data_test` checks, on the write-edge to the + // materialized asset — rendered as a flask badge on the link. + data_tests?: NonNullable + // Producer's `// column` declared lineage, on the same write-edge — + // rendered as a columns badge on the link. + column_lineage?: NonNullable } // Graph-id of the script the user just launched (zero-latency hint), @@ -200,10 +236,14 @@ function build(g: AssetGraphResponse) { const nodes: Array<{ id: string - type: 'asset' | 'runnable' | 'trigger' | 'add' + type: 'asset' | 'runnable' | 'trigger' | 'add' | 'data-test' data: any }> = [] const edges: BuiltEdge[] = [] + // Custom (`// data_test `) tests are deployed scripts, so we + // draw each as its own node hanging off the asset it validates (deduped + // by node id across producers). + const addedTestNodes = new Set() const hasAddNode = onAddPipelineScript != null if (hasAddNode) { @@ -259,6 +299,13 @@ downstreamByScript.set(path, set.size) downstreamUnsavedByScript.set(path, [...set].filter((s) => unsavedRunnables.has(s)).length) } + // Read-aware downstream presence for the bounded-run gate. The bounded + // engine treats pure-read `asset → script` edges as downstream, but the + // subscriber-only `downstreamByScript` above does not — so a start whose + // only downstream is a pure reader would otherwise be denied the menu + // item even though its bounded set is non-empty. Mirror of the engine's + // own adjacency (boundedCascade.buildLineageDownstreamMap). + const hasLineageDownstream = new Set(buildLineageDownstreamMap(g).keys()) for (const a of g.assets) { const assetId = `asset:${a.kind}:${a.path}` @@ -284,6 +331,40 @@ for (const r of g.runnables) { if (r.unsaved) unsavedRunnablePaths.add(r.path) } + // Producer → its `// data_test` checks, keyed by runnable id, so the + // write-edge to the materialized asset can carry the test badge: the + // edge *is* the transformation, and the tests assert on what it produces. + const producerTests = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.data_tests && r.data_tests.length > 0) { + producerTests.set(`${r.usage_kind}:${r.path}`, r.data_tests) + } + } + // Producer → its `// column` declared lineage, keyed by runnable id, so + // the write-edge to the materialized asset can carry the columns badge — + // the edge *is* the transformation, and the lineage describes its output. + const producerColumnLineage = new Map< + string, + NonNullable + >() + // The `// materialize` target the lineage describes, so the badge lands + // only on that write-edge (a multi-output script writes several ducklake + // tables) — mirrors the anchor logic in `buildColumnGraph`. + const producerMaterializeTarget = new Map< + string, + NonNullable + >() + for (const r of g.runnables) { + if (r.column_lineage && r.column_lineage.length > 0) { + producerColumnLineage.set(`${r.usage_kind}:${r.path}`, r.column_lineage) + } + if (r.materialize_target) { + producerMaterializeTarget.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + } for (const r of g.runnables) { const rid = `${r.usage_kind}:${r.path}` // Optimistic badge: the moment a run is launched from this view @@ -325,6 +406,16 @@ downstreamCount: downstreamByScript.get(r.path) ?? 0, downstreamUnsavedCount: downstreamUnsavedByScript.get(r.path) ?? 0, runState, + // Bounded-cascade entrypoint: only valid starts (schedule / + // manual roots) with downstream get the "Run downstream up + // to…" menu item. + onStartBoundedRun: + r.usage_kind === 'script' && + onStartBoundedRun && + validStartPaths?.has(r.path) && + hasLineageDownstream.has(r.path) + ? () => onStartBoundedRun(r.path) + : undefined, onRequestRemove: onRunnableMenuRemove ? () => onRunnableMenuRemove({ @@ -342,13 +433,46 @@ const assetId = `asset:${e.asset_kind}:${e.asset_path}` const access = e.access_type ?? 'r' if (access === 'w' || access === 'rw') { + // Data tests assert on the `// materialize` target, which is always + // a ducklake asset (v1 enforces this), so only the ducklake + // write-edge carries the badge / custom-test nodes — a producer's + // other (e.g. S3/datatable) outputs must not show them. + const edgeTests = e.asset_kind === 'ducklake' ? producerTests.get(runnableId) : undefined + // Column lineage describes one materialized output, so the badge + // lands only on that asset's write-edge: the declared `// materialize` + // target when known (a multi-output script writes several ducklake + // tables), else the ducklake write-edge as the unambiguous fallback. + const matTarget = producerMaterializeTarget.get(runnableId) + const isOutputEdge = matTarget + ? e.asset_kind === matTarget.kind && e.asset_path === matTarget.path + : e.asset_kind === 'ducklake' + const edgeColumnLineage = isOutputEdge ? producerColumnLineage.get(runnableId) : undefined edges.push({ id: `prod:${runnableId}->${assetId}`, source: runnableId, target: assetId, kind: 'lineage-write', - unsaved: e.unsaved + unsaved: e.unsaved, + data_tests: edgeTests, + column_lineage: edgeColumnLineage }) + // Each custom (`// data_test `) test → its own node + // below the asset it validates, with a dashed "tests" edge. + for (const t of edgeTests ?? []) { + if (t.type !== 'custom') continue + const testNodeId = `datatest:${assetId}:${t.path}` + if (!addedTestNodes.has(testNodeId)) { + addedTestNodes.add(testNodeId) + nodes.push({ id: testNodeId, type: 'data-test', data: { path: t.path } }) + } + edges.push({ + id: `test:${assetId}->${testNodeId}`, + source: assetId, + target: testNodeId, + kind: 'data-test', + unsaved: e.unsaved + }) + } } if (access === 'r' || access === 'rw') { edges.push({ @@ -375,7 +499,13 @@ kind: TriggerNodeKind ref: string missing: boolean + // First target script (drives the per-script create/edit flows). runnable_path?: string + // Every target script: a single (kind, ref) — e.g. one schedule — + // can be shared across scripts and dedupes to one node with N + // edges. The bounded-run action must consider all of them, not + // just the first. + runnable_paths: string[] } >() function recordSourceTrigger( @@ -388,9 +518,19 @@ ) { const prev = triggerSourceNodes.get(id) if (!prev) { - triggerSourceNodes.set(id, { allUnsaved: unsaved, kind, ref, missing, runnable_path }) + triggerSourceNodes.set(id, { + allUnsaved: unsaved, + kind, + ref, + missing, + runnable_path, + runnable_paths: runnable_path ? [runnable_path] : [] + }) } else { prev.allUnsaved = prev.allUnsaved && unsaved + if (runnable_path && !prev.runnable_paths.includes(runnable_path)) { + prev.runnable_paths.push(runnable_path) + } } } @@ -449,6 +589,14 @@ }) } for (const [id, info] of triggerSourceNodes) { + // Bounded-run targets among this trigger's scripts: valid starts that + // also have read-aware downstream. A shared schedule may have several + // targets; only offer the action when exactly one qualifies, so the + // run starts from an unambiguous script (rather than the arbitrary + // first-seen one). Multi-eligible nodes suppress it rather than guess. + const eligibleStarts = onStartBoundedRun + ? info.runnable_paths.filter((p) => validStartPaths?.has(p) && hasLineageDownstream.has(p)) + : [] nodes.push({ id, type: 'trigger', @@ -465,7 +613,14 @@ onEditTrigger, onDeleteTrigger, onOpenWebhook, - onOpenDataUpload + onOpenDataUpload, + // View-mode bounded-run entry: offer it on the trigger node + // when exactly one target script is a valid start with + // downstream (see eligibleStarts above). + onStartBoundedRun: + onStartBoundedRun && eligibleStarts.length === 1 + ? () => onStartBoundedRun(eligibleStarts[0]) + : undefined } }) } @@ -573,12 +728,22 @@ : assetEmph === 'input' ? 'wm-asset-input' : undefined + // Bounded-run pick overlay takes precedence over the activity rings + // (it's a transient, modal selection). start ring > end mark > in-set + // ring > eligible (clickable, no style) > dimmed (out of reach). + let boundClass: string | undefined + if (boundPick && n.type !== 'add' && n.type !== 'trigger') { + if (n.id === boundPick.start) boundClass = 'wm-bound-start' + else if (boundPick.ends.has(n.id)) boundClass = 'wm-bound-end' + else if (boundPick.bounded.has(n.id)) boundClass = 'wm-bound-in' + else if (!boundPick.eligible.has(n.id)) boundClass = 'wm-bound-dim' + } return { id: n.id, type: n.type, position: { x: p.x + xCenter + xShift, y: p.y + 40 }, data: n.data, - class: runClass ?? assetClass, + class: boundClass ?? runClass ?? assetClass, selected: n.id === selectedId, // All nodes non-draggable: the layout is sugiyama-computed, // dragging would fight the reactive re-layout. Selection is @@ -701,6 +866,13 @@ style = 'stroke: rgb(156 163 175); stroke-width: 1.25px;' animated = flowAnimated break + case 'data-test': + // Asset → its custom-test script: dashed, muted, no run + // animation (the test isn't a producing step). + style = 'stroke: rgb(156 163 175); stroke-width: 1.25px;' + strokeDasharray = '4 3' + markerColor = 'rgb(156 163 175)' + break case 'trigger-asset': style = 'stroke: rgb(107 114 128); stroke-width: 2px;' animated = flowAnimated @@ -755,7 +927,17 @@ source: e.source, target: e.target, type: 'asset', - data: { detourX: detourForEdge(e.source, e.target) }, + data: { + detourX: detourForEdge(e.source, e.target), + // Data-test badge on the producer→asset write-edge. The + // producer's last-run status (the script fails if any test + // fails) tints it green/red; neutral until it has run. + data_tests: e.data_tests, + testsRunStatus: e.data_tests?.length ? runStates?.get(e.source)?.status : undefined, + // Column-lineage badge on the same write-edge (the link is the + // transformation whose output columns the lineage describes). + column_lineage: e.column_lineage + }, animated, label, labelStyle, @@ -784,7 +966,8 @@ asset: AssetNode as any, runnable: RunnableNode as any, trigger: TriggerNode as any, - add: AddNode as any + add: AddNode as any, + 'data-test': DataTestNode as any } const edgeTypes = { @@ -792,12 +975,26 @@ } function handleNodeClick({ node }: { node: Node }) { + // Bounded-run pick mode intercepts clicks: an eligible (downstream) + // node toggles as an end bound; the start, dimmed nodes, and + // triggers/+ are inert. Selection (details pane) is suppressed so the + // modal pick stays focused. + if (boundPick && onPickEnd) { + if (node.type !== 'asset' && node.type !== 'runnable') return + if (node.id === boundPick.start) return + if (!boundPick.eligible.has(node.id)) return + onPickEnd(node.id) + return + } if (!onselect) return const data = node.data as any if (node.type === 'asset') { onselect({ kind: 'asset', asset_kind: data.asset_kind, path: data.path }) } else if (node.type === 'runnable') { onselect({ kind: 'runnable', runnable_kind: data.runnable_kind, path: data.path }) + } else if (node.type === 'data-test') { + // A custom test is a deployed script — open it like any runnable. + onselect({ kind: 'runnable', runnable_kind: 'script', path: data.path }) } // 'schedule' doesn't produce a selection. } @@ -870,4 +1067,20 @@ :global(.svelte-flow__node.wm-asset-input .drop-shadow-sm) { @apply outline outline-2 outline-gray-400; } + /* Bounded-run pick mode. The start is a solid blue ring; chosen end + bounds get a thicker amber ring; nodes inside the path-between set ring + blue; everything out of reach fades back so the matched subset reads at + a glance. */ + :global(.svelte-flow__node.wm-bound-start .drop-shadow-sm) { + @apply outline outline-2 outline-blue-500; + } + :global(.svelte-flow__node.wm-bound-in .drop-shadow-sm) { + @apply outline outline-2 outline-blue-400/70; + } + :global(.svelte-flow__node.wm-bound-end .drop-shadow-sm) { + @apply outline outline-[3px] outline-amber-500; + } + :global(.svelte-flow__node.wm-bound-dim) { + @apply opacity-30; + } diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte index 6073ee392c..46ebaaf624 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphDetailsPane.svelte @@ -25,11 +25,17 @@ import type { Schema } from '$lib/common' import type { AssetGraphSelection, PipelineMode } from './types' import PipelineScriptView from './PipelineScriptView.svelte' - import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations' + import { + parsePipelineAnnotations, + type ColumnLineage, + type PipelineAnnotations + } from './parsePipelineAnnotations' + import ColumnLineageTrace from './ColumnLineageTrace.svelte' + import { assetColumnNodes, type ColumnLineageGraph } from './columnLineageGraph' import SummaryPathDisplay from '$lib/components/SummaryPathDisplay.svelte' import S3FilePreview from '$lib/components/S3FilePreview.svelte' import DataTablePreview from './DataTablePreview.svelte' - import PartitionStatusGrid from './PartitionStatusGrid.svelte' + import DucklakeAssetPanel from './DucklakeAssetPanel.svelte' import AssetRunsPanel from './AssetRunsPanel.svelte' import { Pane, Splitpanes } from 'svelte-splitpanes' import { fade } from 'svelte/transition' @@ -72,7 +78,11 @@ // edges + synthesize asset nodes for drafts whose body has been // edited past the seeded template. Fires on every keystroke that // changes the inferred set. - onAssetsChange?: (scriptPath: string | undefined, assets: AssetWithAltAccessType[]) => void + onAssetsChange?: ( + scriptPath: string | undefined, + assets: AssetWithAltAccessType[], + columnLineage?: ColumnLineage[] + ) => void // Emits the live editor buffer on every keystroke so the parent can // autosave the in-flight content WITHOUT waiting for the pane teardown // (`onDraftPersist`). `onDraftPersist` stays the authoritative commit on @@ -114,6 +124,14 @@ path: string unsaved?: boolean }> + // Pipeline-wide column-lineage graph (built by the parent page from the + // resolved graph). Drives the transitive column-lineage trace shown for a + // selected materialized asset. + selectionColumnGraph?: ColumnLineageGraph + // Whether the selected ducklake asset's schema can evolve (whole-table + // `replace` producer). Forwarded to the Schema tab: version history when + // true, a single fixed-schema view when false. Defaults to true (unknown). + schemaCanEvolve?: boolean // Bumped by the parent after dispatching a run so the runs panel // re-fetches the listing immediately (rather than waiting on its // background poll tick). @@ -165,6 +183,11 @@ // cascade option when > 0. The page computes this from the graph // edges + triggers + currently-open path. downstreamSubscribers?: number + // Pipeline-only: set when the currently-open script is a valid + // bounded-run start (schedule / manual root). Surfaces a "Run downstream + // up to…" entry on the Test split's caret that enters the canvas + // end-node pick mode rooted at this script. Undefined → no entry. + onStartBoundedRun?: () => void // Sister to `requestRunSignal`. When bumped, the bridge calls // `ScriptEditor.runTest({ cascade: true })` — used by the canvas // runnable menu's "Run + trigger N downstream" item when the chosen @@ -209,6 +232,8 @@ onScriptRenamed, onScriptRemoved, selectionProducers = [], + selectionColumnGraph, + schemaCanEvolve = true, runsRefreshKey, runsPendingJobId, onRunCompleted, @@ -219,6 +244,7 @@ onDraftPathChange, requestRemoveSignal, downstreamSubscribers = 0, + onStartBoundedRun, requestRunCascadeSignal, focusUploadSignal, mode = 'edit', @@ -349,6 +375,10 @@ // edges as the user edits the body (e.g. renaming a CREATE TABLE // target updates the output asset node in real time). let liveBodyAssets = $state(undefined) + // Body-inferred column lineage (DuckDB SQL AST), bound out of ScriptEditor + // alongside `liveBodyAssets` and forwarded so the live graph can show + // inferred column lineage on the edited script before it deploys. + let liveColumnLineage = $state(undefined) // Bumped when the runs panel reports a watched job has reached a // terminal state. Drives S3FilePreview's refreshKey so the preview @@ -529,7 +559,9 @@ : { inPipeline: false, triggerAssets: [], - nativeTriggers: [] + nativeTriggers: [], + dataTests: [], + columnLineage: [] } ) $effect(() => { @@ -542,7 +574,7 @@ }) $effect(() => { if (readOnly) return - onAssetsChange?.(script?.path, liveBodyAssets ?? []) + onAssetsChange?.(script?.path, liveBodyAssets ?? [], liveColumnLineage) }) $effect(() => { if (readOnly) return @@ -984,7 +1016,25 @@ refreshKey={previewRefreshKey} /> {:else if selection.asset_kind === 'ducklake'} - + + {#key selection.path} +
    + {#if selectionColumnGraph && assetColumnNodes(selectionColumnGraph, selection.asset_kind, selection.path).length > 0} +
    + +
    + {/if} +
    + +
    +
    + {/key} {:else}
    No inline preview yet for {selection.asset_kind}. Use the producer/consumer arrows @@ -1082,6 +1132,7 @@ argsAboveLogs: true, logsResultSideBySide: true, downstreamSubscribers, + onBoundedRun: onStartBoundedRun, // Selecting a script node should immediately show // "what happened last time it ran" — pulling the // latest top-level completed job into the preview @@ -1092,6 +1143,7 @@ bind:code={script.content} bind:schema={script.schema} bind:assets={liveBodyAssets} + bind:inferredColumnLineage={liveColumnLineage} {onTestStateChange} {args} /> diff --git a/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte b/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte index 0c66513559..c4bef2dd1f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/AssetGraphEdge.svelte @@ -1,6 +1,8 @@ + +
    +
    + + Column lineage + {#if targetLabel} + + {targetLabel} + {/if} + + {#if selected} + + {:else if component.size > 1} + click a column to trace + {/if} +
    + + {#if component.size === 0} + No column lineage for this asset. + {:else} +
    + + {#each edges as e (`${e.from}->${e.to}`)} + {@const hot = traced !== undefined && traced.has(e.from) && traced.has(e.to)} + {@const cold = traced !== undefined && !hot} + + {/each} + + + {#each [...component] as id (id)} + {@const n = graph.nodes.get(id)} + {@const p = pos.get(id)} + {#if n && p} + {@const isSeed = seedSet.has(id)} + + {/if} + {/each} +
    + {/if} +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/DataTestNode.svelte b/frontend/src/lib/components/assets/AssetGraph/DataTestNode.svelte new file mode 100644 index 0000000000..9c46fd1795 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DataTestNode.svelte @@ -0,0 +1,31 @@ + + + +
    + + test + + {data.path} + +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeAssetPanel.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeAssetPanel.svelte new file mode 100644 index 0000000000..5f7ad050e4 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeAssetPanel.svelte @@ -0,0 +1,123 @@ + + +
    + {#if !qualifiedTable} + + + {:else} +
    + (tab = e.detail)}> + {#snippet children({ item })} + + + + {/snippet} + +
    + +
    + {#if tab === 'partitions'} + + {:else if tab === 'schema'} + + {:else} +
    + + + snapshots.refetch()} + selectedVersion={effectiveVersion} + onSelect={(v) => (selectedVersion = v)} + /> + + +
    + +
    +     +     +
    + {/if} +
    + {/if} +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeSnapshotHistory.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeSnapshotHistory.svelte new file mode 100644 index 0000000000..e95be118ee --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeSnapshotHistory.svelte @@ -0,0 +1,79 @@ + + +
    +
    + Snapshot history +
    + +
    + {#if loading && !items.length} +
    + Loading snapshots… +
    + {:else if error} +

    Failed to load: {error}

    + {:else if !items.length} +

    + No snapshots yet. DuckLake records one on every // materialize write; each becomes a version you can time-travel to. +

    + {:else} +
    + {#each items as s (s.snapshot_id)} + {@const selected = s.snapshot_id === selectedVersion} + + {/each} +
    + {/if} +
    +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/DucklakeVersionPreview.svelte b/frontend/src/lib/components/assets/AssetGraph/DucklakeVersionPreview.svelte new file mode 100644 index 0000000000..5afb51fe66 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/DucklakeVersionPreview.svelte @@ -0,0 +1,133 @@ + + +
    + {#if version == undefined} +
    + Select a snapshot from the list to preview the table as of that version. +
    + {:else} + {#if exampleSql} +
    + + {exampleSql} + +
    + {/if} + {#if ready && dbTableOps} + {#key version} +
    + +
    + {/key} + {:else if columns.error} +
    + + Couldn't load this table at version {version}. +
    + {:else} +
    + +
    + {/if} + {/if} +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte b/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte index 93eeb04163..c709866a5f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/RunnableNode.svelte @@ -11,6 +11,7 @@ Play, RotateCw, Tag, + Target, Timer, Trash2, XCircle, @@ -67,6 +68,10 @@ // "Discard" for drafts, "Delete…" (which the page maps to its // archive/delete confirmation flow) for persisted scripts. onRequestRemove?: () => void + // Wired only for valid bounded-run starts (schedule / manual roots + // with downstream). Enters the page's end-node pick mode for a + // bounded cascade rooted at this script. + onStartBoundedRun?: () => void } // SvelteFlow injects this when the user clicks the node. Combined with // hover state to drive the run-button visibility (same pattern as @@ -113,9 +118,9 @@ } } - // Cascade option is surfaced directly on the Run button (via a caret + - // popover) when `downstreamCount > 0`, so the kebab menu stays focused - // on lifecycle actions only. + // Cascade + bounded-run options live on the Run button's caret popover + // (whenever there's a cascade OR a bounded-run start — see `hasCaret` + // below), so the kebab menu stays focused on lifecycle actions only. let menuItems: Item[] = $derived( data.onRequestRemove ? [ @@ -238,6 +243,11 @@ + trigger N downstream". Matches the editor Test split button so the affordance is identical on both surfaces. --> {@const hasCascade = (data.downstreamCount ?? 0) > 0} + + {@const hasCaret = hasCascade || !!data.onStartBoundedRun}
    - {#if hasCascade} + {#if hasCaret}
    - + + Let the asset-trigger cascade fan out to the {data.downstreamCount} + subscribed script{data.downstreamCount === 1 ? '' : 's'} after this run succeeds. + + {#if data.unsaved || (data.downstreamUnsavedCount ?? 0) > 0} + + {#if data.unsaved} + Unsaved chain — runs as previews in dependency order. + {:else} + {data.downstreamUnsavedCount} unsaved — chain runs as previews in dependency + order. + {/if} + Deploy to enable automatic triggering. + + {/if} +
    + + {/if} + {#if data.onStartBoundedRun} + + {/if} {/snippet}     diff --git a/frontend/src/lib/components/assets/AssetGraph/SchemaHistoryPanel.svelte b/frontend/src/lib/components/assets/AssetGraph/SchemaHistoryPanel.svelte new file mode 100644 index 0000000000..fbccccf860 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/SchemaHistoryPanel.svelte @@ -0,0 +1,171 @@ + + +{#snippet columnsTable(cols: SchemaColumn[])} +
    + + + + + + + + {#each cols as col (col.name)} + + + + + {/each} + +
    ColumnType
    {col.name}{col.type}
    +{/snippet} + +
    +
    + Captured schema +
    + +
    + {#if schemas.loading} +
    + Loading schema… +
    + {:else if schemas.error} +

    Failed to load: {schemas.error.message}

    + {:else if !schemas.current?.length} +

    + No schema captured yet. The output schema is recorded automatically after a // materialize run. +

    + {:else if !canEvolve} + +
    +
    + + This asset's schema is fixed — an append / merge / partitioned materialize INSERTs into + a fixed-schema table, so the columns can't change run-to-run. +
    + {#if selected} + {@render columnsTable(selected.columns)} + {/if} +
    + {:else} +
    + + +
    + {#each schemas.current as s, i (s.version)} + + {/each} +
    +     + +
    + {#if selected} + {@render columnsTable(selected.columns)} + {/if} +
    +     +     +
    + {/if} +
    +
    diff --git a/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte b/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte index 6d8f1224ae..caf7d90482 100644 --- a/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte +++ b/frontend/src/lib/components/assets/AssetGraph/TriggerNode.svelte @@ -60,7 +60,7 @@ import { Handle, Position } from '@xyflow/svelte' import { NODE } from '$lib/components/graph/util' import { twMerge } from 'tailwind-merge' - import { AlertTriangle, EllipsisVertical, Trash2 } from 'lucide-svelte' + import { AlertTriangle, EllipsisVertical, Target, Trash2 } from 'lucide-svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { stopPropagation, preventDefault } from 'svelte/legacy' import type { Item } from '$lib/utils' @@ -113,6 +113,12 @@ // trigger. Confirmation is the caller's responsibility — the // node just exposes the entry point on the kebab menu. onDeleteTrigger?: (kind: NativeTriggerKind, triggerPath: string) => void + // Wired by the canvas only when this trigger's target script is a + // valid bounded-run start (schedule / manual root) with downstream. + // Entering the page's end-node pick mode rooted at that script — the + // View-mode entry point for bounded runs (the script's own Run-button + // caret is Edit-only). + onStartBoundedRun?: () => void } } let { data }: Props = $props() @@ -173,8 +179,17 @@ !!data.onDeleteTrigger ) - let menuItems: Item[] = $derived( - canDelete + let menuItems: Item[] = $derived([ + ...(data.onStartBoundedRun + ? [ + { + displayName: 'Run downstream up to…', + icon: Target, + action: () => data.onStartBoundedRun?.() + } + ] + : []), + ...(canDelete ? [ { displayName: 'Delete…', @@ -182,12 +197,12 @@ type: 'delete' as const, action: () => { if (!data.ref || !data.onDeleteTrigger) return - data.onDeleteTrigger(data.kind as NativeTriggerKind, data.ref) + data.onDeleteTrigger?.(data.kind as NativeTriggerKind, data.ref) } } ] - : [] - ) + : []) + ]) function handleMissingClick() { if (!canCreate || !data.runnable_path || !data.onCreateMissingTrigger) return diff --git a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts new file mode 100644 index 0000000000..fa735682b7 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.test.ts @@ -0,0 +1,283 @@ +import { describe, expect, it } from 'vitest' +import type { AssetGraphResponse, AssetGraphTrigger, NativeTriggerKind } from './types' +import { + ancestors, + assetUriToNodeId, + boundedSet, + buildLineageDag, + buildLineageDownstreamMap, + descendants, + scriptNodeId, + scriptsOf, + validStarts +} from './boundedCascade' +import { computeInducedSchedule } from './graphTraversal' + +type W = [script: string, asset: string] // producer write edge (datatable) +type R = [script: string, asset: string] // pure-read edge (datatable) +type S = [script: string, asset: string] // `// on ` subscription + +function graph(opts: { + scripts?: string[] + writes?: W[] + reads?: R[] + subs?: S[] + native?: Array<[kind: NativeTriggerKind, script: string]> +}): AssetGraphResponse { + const { scripts = [], writes = [], reads = [], subs = [], native = [] } = opts + const triggers: AssetGraphTrigger[] = [ + ...subs.map( + ([s, a]) => + ({ + trigger_kind: 'asset', + asset_kind: 'datatable', + asset_path: a, + runnable_kind: 'script', + runnable_path: s + }) as const + ), + ...native.map( + ([kind, s]) => + ({ trigger_kind: kind, runnable_kind: 'script', runnable_path: s }) as AssetGraphTrigger + ) + ] + return { + assets: [], + runnables: scripts.map((p) => ({ path: p, usage_kind: 'script' as const })), + edges: [ + ...writes.map(([s, a]) => ({ + runnable_path: s, + runnable_kind: 'script' as const, + asset_kind: 'datatable' as const, + asset_path: a, + access_type: 'w' as const + })), + ...reads.map(([s, a]) => ({ + runnable_path: s, + runnable_kind: 'script' as const, + asset_kind: 'datatable' as const, + asset_path: a, + access_type: 'r' as const + })) + ], + triggers + } +} + +const sn = scriptNodeId +const asset = (p: string) => `datatable:${p}` + +describe('buildLineageDag', () => { + it('links producer → asset → subscriber and asset → reader', () => { + // a writes x; b subscribes to x; c reads x. + const g = graph({ writes: [['a', 'x']], subs: [['b', 'x']], reads: [['c', 'x']] }) + const dag = buildLineageDag(g) + expect([...(dag.down.get(sn('a')) ?? [])]).toEqual([asset('x')]) + expect([...(dag.down.get(asset('x')) ?? [])].sort()).toEqual([sn('b'), sn('c')]) + }) + + it('treats rw as production only (no self-cycle through the asset)', () => { + const g: AssetGraphResponse = { + assets: [], + runnables: [{ path: 'u', usage_kind: 'script' }], + edges: [ + { + runnable_path: 'u', + runnable_kind: 'script', + asset_kind: 'datatable', + asset_path: 'x', + access_type: 'rw' + } + ], + triggers: [] + } + const dag = buildLineageDag(g) + expect([...(dag.down.get(sn('u')) ?? [])]).toEqual([asset('x')]) + expect(dag.up.get(sn('u'))).toBeUndefined() // asset is not upstream of its own writer + }) +}) + +describe('ancestors / descendants', () => { + it('walks transitively over scripts and assets', () => { + // a → x → b → y → c + const g = graph({ + writes: [ + ['a', 'x'], + ['b', 'y'] + ], + subs: [ + ['b', 'x'], + ['c', 'y'] + ] + }) + const dag = buildLineageDag(g) + expect(descendants(dag, sn('a'))).toEqual(new Set([asset('x'), sn('b'), asset('y'), sn('c')])) + expect(ancestors(dag, sn('c'))).toEqual(new Set([asset('y'), sn('b'), asset('x'), sn('a')])) + }) + + it('excludes the start node even on a cycle back to it', () => { + // a → x → b → y → a (cycle): descendants(a) must not contain a. + const g = graph({ + writes: [ + ['a', 'x'], + ['b', 'y'] + ], + subs: [ + ['b', 'x'], + ['a', 'y'] + ] + }) + const dag = buildLineageDag(g) + expect(descendants(dag, sn('a')).has(sn('a'))).toBe(false) + expect(ancestors(dag, sn('a')).has(sn('a'))).toBe(false) + }) +}) + +describe('boundedSet', () => { + // a → x → b → y → c → z → d (linear chain through assets) + const chain = () => + graph({ + writes: [ + ['a', 'x'], + ['b', 'y'], + ['c', 'z'] + ], + subs: [ + ['b', 'x'], + ['c', 'y'], + ['d', 'z'] + ] + }) + + it('stops at a single end node (script)', () => { + const dag = buildLineageDag(chain()) + const res = boundedSet(dag, sn('a'), [sn('c')]) + // path a..c includes a,x,b,y,c — not z or d. + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b', 'c']) + expect(res.nodes.has(asset('z'))).toBe(false) + expect(res.droppedEnds).toEqual([]) + }) + + it('supports an asset as the end bound', () => { + const dag = buildLineageDag(chain()) + const res = boundedSet(dag, sn('a'), [asset('y')]) + // up to datatable://y → a, b produced it. + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b']) + }) + + it('unions multiple ends', () => { + // diamond: a → b and a → c, both → d. Bound to {b, c} excludes d. + const g = graph({ + writes: [ + ['a', 'xa'], + ['b', 'xb'], + ['c', 'xc'] + ], + subs: [ + ['b', 'xa'], + ['c', 'xa'], + ['d', 'xb'], + ['d', 'xc'] + ] + }) + const dag = buildLineageDag(g) + const res = boundedSet(dag, sn('a'), [sn('b'), sn('c')]) + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b', 'c']) + }) + + it('drops ends not downstream of start', () => { + const dag = buildLineageDag(chain()) + const res = boundedSet(dag, sn('c'), [sn('a')]) + expect(res.droppedEnds).toEqual([sn('a')]) + expect(res.reachableEnds).toEqual([]) + expect([...res.nodes]).toEqual([sn('c')]) + }) + + it('is cycle-safe', () => { + // b ↔ c cycle downstream of a; bounding to c terminates. + const g = graph({ + writes: [ + ['a', 'x'], + ['b', 'y'], + ['c', 'z'] + ], + subs: [ + ['b', 'x'], + ['c', 'y'], + ['b', 'z'] + ] + }) + const dag = buildLineageDag(g) + const res = boundedSet(dag, sn('a'), [sn('c')]) + expect(scriptsOf(res.nodes).sort()).toEqual(['a', 'b', 'c']) + }) +}) + +describe('validStarts', () => { + it('includes schedule-rooted scripts', () => { + const g = graph({ scripts: ['s'], native: [['schedule', 's']] }) + expect(validStarts(g)).toEqual(new Set([sn('s')])) + }) + + it('includes manual roots (no trigger, not a subscriber)', () => { + const g = graph({ scripts: ['m'] }) + expect(validStarts(g)).toEqual(new Set([sn('m')])) + }) + + it('excludes event-only roots', () => { + const g = graph({ scripts: ['k'], native: [['kafka', 'k']] }) + expect(validStarts(g).has(sn('k'))).toBe(false) + }) + + it('excludes pure asset subscribers but keeps a schedule subscriber', () => { + // sub is `// on x`; sched is both a subscriber and schedule-triggered. + const g = graph({ + scripts: ['a', 'sub', 'sched'], + writes: [['a', 'x']], + subs: [ + ['sub', 'x'], + ['sched', 'x'] + ], + native: [['schedule', 'sched']] + }) + const starts = validStarts(g) + expect(starts.has(sn('a'))).toBe(true) // manual root + expect(starts.has(sn('sub'))).toBe(false) // event-less but a subscriber + expect(starts.has(sn('sched'))).toBe(true) // schedule overrides subscriber + }) +}) + +describe('buildLineageDownstreamMap (read-aware scheduling)', () => { + // a writes x; c only *reads* x (no `// on x`). c must still run after a. + const g = graph({ + scripts: ['a', 'c'], + writes: [['a', 'x']], + reads: [['c', 'x']] + }) + + it('links a producer to a pure reader of its asset', () => { + const map = buildLineageDownstreamMap(g) + expect([...(map.get('a') ?? [])]).toEqual(['c']) + }) + + it('orders the reader after the producer through computeInducedSchedule', () => { + const selected = new Set(['a', 'c']) + // Subscriber-only map (default) misses the read dep → c is a stray root. + const subscriberOnly = computeInducedSchedule(g, selected) + expect(subscriberOnly.roots.sort()).toEqual(['a', 'c']) + // Read-aware map orders c strictly after a. + const readAware = computeInducedSchedule(g, selected, buildLineageDownstreamMap(g)) + expect(readAware.roots).toEqual(['a']) + expect(readAware.indegree.get('c')).toBe(1) + expect(readAware.nodes).toEqual(['a', 'c']) + }) +}) + +describe('assetUriToNodeId', () => { + it('maps s3 prefix to the s3object kind, others verbatim', () => { + expect(assetUriToNodeId('s3://bucket/key')).toBe('s3object:bucket/key') + expect(assetUriToNodeId('datatable://prod/users')).toBe('datatable:prod/users') + expect(assetUriToNodeId('ducklake://lake/t')).toBe('ducklake:lake/t') + expect(assetUriToNodeId('not-a-uri')).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts new file mode 100644 index 0000000000..44a26d9c71 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts @@ -0,0 +1,235 @@ +import type { AssetGraphResponse, NativeTriggerKind } from './types' +import { assetKey, isWriteEdge } from './lib' + +// Bounded-cascade selective execution: instead of a dbt-style `--select` +// grammar (a compile-time file selector dbt needs because it has no runtime +// DAG), Windmill already runs a live cascade. So the primitive here is to +// *bound* that cascade — start at a pipeline entrypoint, fan downstream, but +// stop at one or more chosen end node(s). The matched set is the "path +// between" start and the ends: +// +// descendants(start) ∩ (ancestors(ends) ∪ ends) ∪ {start} +// +// Pure module (no Svelte runes) so the graph algebra is unit-testable and can +// be ported verbatim to the CLI. See `boundedCascade.test.ts`, and the mirror +// at `cli/src/commands/pipeline/boundedCascade.ts` (keep them in sync). + +// Unified lineage-DAG node ids. Assets use `${asset_kind}:${asset_path}` +// (identical to `assetKey`); runnables are prefixed so a script path can never +// collide with an asset key. Flows are excluded — they aren't cascade members +// (mirrors graphTraversal/asset_dispatch). +export const SCRIPT_PREFIX = 'script:' + +export function scriptNodeId(path: string): string { + return `${SCRIPT_PREFIX}${path}` +} +export function isScriptNode(id: string): boolean { + return id.startsWith(SCRIPT_PREFIX) +} +export function scriptPathOf(id: string): string { + return id.slice(SCRIPT_PREFIX.length) +} + +/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */ +export function assetUriToNodeId(uri: string): string | undefined { + const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i) + if (!m) return undefined + const prefix = m[1].toLowerCase() + // `s3` is the URI prefix for the `s3object` asset kind (mirrors the CLI + // `assetUri` and the canvas). All other kinds use their name verbatim. + const kind = prefix === 's3' ? 's3object' : prefix + return `${kind}:${m[2]}` +} + +// Native trigger kinds that fan out *per event*: a single event always flows +// through the whole reactive downstream, so "run up to X now" is not a +// meaningful gesture and these are never offered as bounded-run starts. +// `webhook`/`data_upload` have no trigger row in `/assets/graph`, so a root +// whose only entry is one of those reads as a *manual* root below. +const EVENT_TRIGGER_KINDS: ReadonlySet = new Set([ + 'kafka', + 'mqtt', + 'nats', + 'postgres', + 'sqs', + 'gcp', + 'email' +]) + +export type LineageDag = { + /** upstream node id → set of direct downstream node ids. */ + down: Map> + /** downstream node id → set of direct upstream node ids. */ + up: Map> + /** Every node id (scripts + assets), including isolated ones. */ + nodes: Set +} + +/** + * Build the directed upstream→downstream lineage DAG over scripts ∪ assets: + * - producer script → asset (write / rw edges) + * - asset → reader script (pure-read edges — a data dependency) + * - asset → subscriber script (`// on ` triggers) + * + * An `rw` edge is treated as production only (script → asset); emitting the + * reverse asset → script too would make every upsert a 2-cycle through its own + * asset. + */ +export function buildLineageDag(g: AssetGraphResponse): LineageDag { + const down = new Map>() + const up = new Map>() + const nodes = new Set() + + const addEdge = (a: string, b: string) => { + if (a === b) return + nodes.add(a) + nodes.add(b) + ;(down.get(a) ?? down.set(a, new Set()).get(a)!).add(b) + ;(up.get(b) ?? up.set(b, new Set()).get(b)!).add(a) + } + + // Register every node up front so isolated scripts/assets still appear. + for (const r of g.runnables ?? []) { + if (r.usage_kind === 'script') nodes.add(scriptNodeId(r.path)) + } + for (const a of g.assets ?? []) nodes.add(`${a.kind}:${a.path}`) + + for (const e of g.edges ?? []) { + if (e.runnable_kind !== 'script') continue + const aid = assetKey(e) + if (isWriteEdge(e)) { + addEdge(scriptNodeId(e.runnable_path), aid) + } else if ((e.access_type ?? 'r') === 'r') { + addEdge(aid, scriptNodeId(e.runnable_path)) + } + } + for (const t of g.triggers ?? []) { + if (t.trigger_kind !== 'asset' || t.runnable_kind !== 'script') continue + addEdge(assetKey(t), scriptNodeId(t.runnable_path)) + } + + return { down, up, nodes } +} + +/** BFS closure over an adjacency map, excluding `start`. */ +function closure(adj: Map>, start: string): Set { + const seen = new Set() + const queue = [start] + while (queue.length > 0) { + const cur = queue.shift()! + for (const n of adj.get(cur) ?? []) { + if (seen.has(n)) continue + seen.add(n) + queue.push(n) + } + } + // A cycle back to `start` would have re-added it; the contract is to + // exclude the node itself. + seen.delete(start) + return seen +} + +/** Transitive downstream of `n` (excludes `n`). Cycle-safe. */ +export function descendants(dag: LineageDag, n: string): Set { + return closure(dag.down, n) +} +/** Transitive upstream of `n` (excludes `n`). Cycle-safe. */ +export function ancestors(dag: LineageDag, n: string): Set { + return closure(dag.up, n) +} + +export type BoundedResult = { + /** Path-between node set (scripts + assets), always including `start`. */ + nodes: Set + /** Ends that are actually reachable downstream of `start`. */ + reachableEnds: string[] + /** Ends passed in that are not downstream of `start` (ignored, surfaced). */ + droppedEnds: string[] +} + +/** + * Nodes on any path from `start` to any of `ends` (inclusive of both). Ends + * not reachable from `start` are dropped and reported. With no reachable end + * the result is just `{start}` — the caller decides whether to warn or fall + * back to the full downstream. + */ +export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult { + const desc = descendants(dag, start) + const downSet = new Set(desc) + downSet.add(start) + + const reachableEnds = ends.filter((e) => downSet.has(e)) + const droppedEnds = ends.filter((e) => !downSet.has(e)) + if (reachableEnds.length === 0) { + return { nodes: new Set([start]), reachableEnds, droppedEnds } + } + + const upClosure = new Set() + for (const e of reachableEnds) { + upClosure.add(e) + for (const a of ancestors(dag, e)) upClosure.add(a) + } + const nodes = new Set() + for (const n of downSet) if (upClosure.has(n)) nodes.add(n) + nodes.add(start) + return { nodes, reachableEnds, droppedEnds } +} + +/** + * Script node ids eligible to *start* a bounded run: schedule-triggered + * scripts, or manual roots (not an asset subscriber and not event-triggered). + * Event-driven entrypoints are excluded — they have no "run up to X" gesture. + */ +export function validStarts(g: AssetGraphResponse): Set { + const subscribers = new Set() + const scheduleScripts = new Set() + const eventScripts = new Set() + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== 'script') continue + if (t.trigger_kind === 'asset') subscribers.add(t.runnable_path) + else if (t.trigger_kind === 'schedule') scheduleScripts.add(t.runnable_path) + else if (EVENT_TRIGGER_KINDS.has(t.trigger_kind)) eventScripts.add(t.runnable_path) + } + + const out = new Set() + for (const r of g.runnables ?? []) { + if (r.usage_kind !== 'script') continue + const p = r.path + if (scheduleScripts.has(p)) out.add(scriptNodeId(p)) + else if (!subscribers.has(p) && !eventScripts.has(p)) out.add(scriptNodeId(p)) + } + return out +} + +/** Project a node-id set to the script paths it contains (run targets). */ +export function scriptsOf(nodes: Iterable): string[] { + const out: string[] = [] + for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id)) + return out +} + +/** + * Producer→consumer adjacency (script path → downstream script paths) over the + * lineage DAG: one hop through a single asset, following BOTH `// on` + * subscribers *and* pure-read data dependencies. Unlike + * `graphTraversal.buildDownstreamMap` (subscriber-only, which models the + * production dispatch), this orders a script that merely reads an upstream + * asset after its producer — so a bounded selection containing such a reader + * schedules correctly. Mirror of the CLI `topoOrder` adjacency. + */ +export function buildLineageDownstreamMap(g: AssetGraphResponse): Map> { + const dag = buildLineageDag(g) + const map = new Map>() + for (const id of dag.nodes) { + if (!isScriptNode(id)) continue + const s = scriptPathOf(id) + const subs = new Set() + for (const asset of dag.down.get(id) ?? []) { + for (const sub of dag.down.get(asset) ?? []) { + if (isScriptNode(sub) && scriptPathOf(sub) !== s) subs.add(scriptPathOf(sub)) + } + } + if (subs.size > 0) map.set(s, subs) + } + return map +} diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts index 5509839dab..92d7a977fe 100644 --- a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.test.ts @@ -1,6 +1,6 @@ import { describe, expect, it } from 'vitest' -import { runCascade } from './cascadeOrchestrator' -import type { DownstreamClosure } from './graphTraversal' +import { runCascade, runSelection } from './cascadeOrchestrator' +import type { DownstreamClosure, InducedSchedule } from './graphTraversal' function closure(edges: Array<[from: string, to: string]>, nodes: string[]): DownstreamClosure { const e = new Map>() @@ -161,3 +161,60 @@ describe('runCascade', () => { expect(res.statuses.get('c')?.status).toBe('skipped') }) }) + +function schedule( + edges: Array<[from: string, to: string]>, + nodes: string[], + roots: string[] +): InducedSchedule { + const e = new Map>() + const indegree = new Map() + for (const n of nodes) indegree.set(n, 0) + for (const [from, to] of edges) { + const set = e.get(from) ?? new Set() + set.add(to) + e.set(from, set) + indegree.set(to, (indegree.get(to) ?? 0) + 1) + } + return { nodes, edges: e, indegree, roots, cyclic: [] } +} + +describe('runSelection', () => { + it('runs every selected node, seeding all roots', async () => { + // two independent roots a, b each → c. + const sched = schedule( + [ + ['a', 'c'], + ['b', 'c'] + ], + ['a', 'b', 'c'], + ['a', 'b'] + ) + const r = fakeRunner() + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(true) + // Sort a copy for the membership check — mutating `r.launched` here + // would invalidate the launch-order assertions below. + expect([...r.launched].sort()).toEqual(['a', 'b', 'c']) + // c only after both upstreams. + expect(r.launched.indexOf('c')).toBeGreaterThan(r.launched.indexOf('a')) + expect(r.launched.indexOf('c')).toBeGreaterThan(r.launched.indexOf('b')) + }) + + it('stops scheduling after a failure', async () => { + const sched = schedule([['a', 'b']], ['a', 'b'], ['a']) + const r = fakeRunner({ a: 'failure' }) + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(false) + expect(r.launched).toEqual(['a']) + expect(res.statuses.get('b')?.status).toBe('skipped') + }) + + it('a single-node selection runs that node', async () => { + const sched = schedule([], ['a'], ['a']) + const r = fakeRunner() + const res = await runSelection({ schedule: sched, ...r }) + expect(res.ok).toBe(true) + expect(r.launched).toEqual(['a']) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts index 7e6a6029ed..16d357af00 100644 --- a/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts +++ b/frontend/src/lib/components/assets/AssetGraph/cascadeOrchestrator.ts @@ -1,4 +1,4 @@ -import type { DownstreamClosure } from './graphTraversal' +import type { DownstreamClosure, InducedSchedule } from './graphTraversal' // Client-side orchestration of a pipeline chain dev-run ("Run + downstream" // over a graph that contains drafts). The backend asset-trigger dispatcher @@ -112,3 +112,82 @@ export async function runCascade(opts: CascadeRunOptions): Promise s.status === 'success') return { ok, statuses } } + +export type SelectionRunOptions = { + /** Multi-root induced schedule of the selected scripts (computeInducedSchedule). */ + schedule: InducedSchedule + /** Launch one script (preview for drafts, by-path for deployed); returns the job id. */ + launch: (path: string) => Promise + /** Resolve once the job reaches a terminal state. */ + waitTerminal: (jobId: string) => Promise<'success' | 'failure'> + /** Snapshot of all node states, emitted on every transition. */ + onUpdate?: (statuses: Map) => void +} + +/** + * Execute an arbitrary selected set of scripts (e.g. a bounded-cascade + * selection) in topological order. Unlike `runCascade` there is no single + * privileged root — every `schedule.roots` entry is seeded at once and a node + * runs as soon as its in-set upstreams all succeed. Failure stops *scheduling* + * (in-flight jobs finish); everything not yet started ends 'skipped'. + */ +export async function runSelection(opts: SelectionRunOptions): Promise { + const { schedule, launch, waitTerminal, onUpdate } = opts + const statuses = new Map() + for (const n of schedule.nodes) statuses.set(n, { status: 'pending' }) + const remaining = new Map(schedule.indegree) + let failed = false + const inFlight = new Set>() + + const emit = () => onUpdate?.(new Map(statuses)) + + function schedule_(path: string) { + const p = runNode(path).finally(() => inFlight.delete(p)) + inFlight.add(p) + } + + async function runNode(path: string): Promise { + statuses.set(path, { status: 'running' }) + emit() + let jobId: string | undefined + try { + jobId = await launch(path) + statuses.set(path, { status: 'running', jobId }) + emit() + const term = await waitTerminal(jobId) + statuses.set(path, { status: term, jobId }) + emit() + if (term === 'failure') { + failed = true + return + } + } catch (e) { + statuses.set(path, { + status: 'failure', + jobId, + error: e instanceof Error ? e.message : String(e) + }) + emit() + failed = true + return + } + for (const s of schedule.edges.get(path) ?? []) { + const d = (remaining.get(s) ?? 0) - 1 + remaining.set(s, d) + if (d === 0 && !failed) schedule_(s) + } + } + + for (const r of schedule.roots) schedule_(r) + while (inFlight.size > 0) { + await Promise.race(inFlight) + } + + for (const [n, st] of statuses) { + if (st.status === 'pending') statuses.set(n, { status: 'skipped' }) + } + emit() + + const ok = !failed && [...statuses.values()].every((s) => s.status === 'success') + return { ok, statuses } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts new file mode 100644 index 0000000000..6842dcfd86 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.test.ts @@ -0,0 +1,162 @@ +import { describe, expect, it } from 'vitest' +import type { AssetGraphResponse } from './types' +import { + buildColumnGraph, + colNodeId, + traceColumn, + connectedComponent, + assetColumnNodes, + computeDepths +} from './columnLineageGraph' + +// Two scripts chained through an intermediate ducklake table: +// s1: orders.amount -> staging.amt +// s2: staging.amt -> daily.total +// s2: customers.name -> daily.cust (a second source into the sink) +function chainGraph(): AssetGraphResponse { + return { + assets: [], + triggers: [], + runnables: [ + { + path: 's1', + usage_kind: 'script', + column_lineage: [ + { + column: 'amt', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/orders', from_column: 'amount' }] + } + ] + }, + { + path: 's2', + usage_kind: 'script', + column_lineage: [ + { + column: 'total', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/staging', from_column: 'amt' }] + }, + { + column: 'cust', + inputs: [{ from_kind: 'ducklake', from_path: 'wh/customers', from_column: 'name' }] + } + ] + } + ], + edges: [ + { + runnable_path: 's1', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/staging', + access_type: 'w' + }, + { + runnable_path: 's2', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/daily', + access_type: 'w' + } + ] + } +} + +const ORDERS_AMOUNT = colNodeId('ducklake', 'wh/orders', 'amount') +const STAGING_AMT = colNodeId('ducklake', 'wh/staging', 'amt') +const DAILY_TOTAL = colNodeId('ducklake', 'wh/daily', 'total') +const DAILY_CUST = colNodeId('ducklake', 'wh/daily', 'cust') +const CUSTOMERS_NAME = colNodeId('ducklake', 'wh/customers', 'name') + +describe('buildColumnGraph', () => { + it('stitches per-script lineage into a transitive graph via shared columns', () => { + const g = buildColumnGraph(chainGraph()) + // orders.amount feeds staging.amt feeds daily.total + expect(g.up.get(STAGING_AMT)).toEqual(new Set([ORDERS_AMOUNT])) + expect(g.up.get(DAILY_TOTAL)).toEqual(new Set([STAGING_AMT])) + expect(g.down.get(ORDERS_AMOUNT)).toEqual(new Set([STAGING_AMT])) + expect(g.down.get(STAGING_AMT)).toEqual(new Set([DAILY_TOTAL])) + }) + + it('anchors to the // materialize target, not a guessed write-edge', () => { + const graph = chainGraph() + // s1 declares its materialize target and also has unordered extra + // ducklake writes; the lineage must anchor to the declared target. + const s1 = graph.runnables.find((r) => r.path === 's1')! + s1.materialize_target = { kind: 'ducklake', path: 'wh/staging' } + graph.edges.unshift({ + runnable_path: 's1', + runnable_kind: 'script', + asset_kind: 'ducklake', + asset_path: 'wh/other', + access_type: 'w' + }) + const g = buildColumnGraph(graph) + expect(g.nodes.has(STAGING_AMT)).toBe(true) // anchored to the declared target + expect(g.nodes.has(colNodeId('ducklake', 'wh/other', 'amt'))).toBe(false) + }) + + it('falls back to a ducklake write-edge when there is no materialize target', () => { + // chainGraph's runnables carry no materialize_target, so s1's lineage is + // anchored via its (single) ducklake write-edge. + const g = buildColumnGraph(chainGraph()) + expect(g.nodes.has(STAGING_AMT)).toBe(true) + }) + + it('node ids are collision-proof across `#` / `:` in paths and columns', () => { + // A delimiter-concatenated id would merge these; the JSON-encoded id must not. + expect(colNodeId('ducklake', 'a#b', 'c')).not.toBe(colNodeId('ducklake', 'a', 'b#c')) + expect(colNodeId('ducklake', 'a:b', 'c')).not.toBe(colNodeId('ducklake', 'a', 'b:c')) + }) + + it('skips producers with no ducklake output asset (columns unanchorable)', () => { + const graph = chainGraph() + graph.edges = graph.edges.filter((e) => e.runnable_path !== 's1') // s1 loses its output edge + const g = buildColumnGraph(graph) + // staging.amt is no longer produced as a node by s1... + expect(g.up.has(STAGING_AMT)).toBe(false) + // ...but s2 still anchors daily.total ← staging.amt (staging.amt as a source). + expect(g.up.get(DAILY_TOTAL)).toEqual(new Set([STAGING_AMT])) + }) +}) + +describe('traceColumn', () => { + it('returns the full upstream + downstream impact set of a source column', () => { + const g = buildColumnGraph(chainGraph()) + // from the root source, the whole chain downstream is impacted + expect(traceColumn(ORDERS_AMOUNT, g)).toEqual( + new Set([ORDERS_AMOUNT, STAGING_AMT, DAILY_TOTAL]) + ) + }) + + it('traces backward from a sink to every contributing source', () => { + const g = buildColumnGraph(chainGraph()) + expect(traceColumn(DAILY_TOTAL, g)).toEqual(new Set([DAILY_TOTAL, STAGING_AMT, ORDERS_AMOUNT])) + // the sibling output `cust` and its source are NOT in total's trace + expect(traceColumn(DAILY_TOTAL, g).has(DAILY_CUST)).toBe(false) + expect(traceColumn(DAILY_TOTAL, g).has(CUSTOMERS_NAME)).toBe(false) + }) + + it('traces an intermediate column both directions', () => { + const g = buildColumnGraph(chainGraph()) + expect(traceColumn(STAGING_AMT, g)).toEqual(new Set([STAGING_AMT, ORDERS_AMOUNT, DAILY_TOTAL])) + }) +}) + +describe('connectedComponent + depths', () => { + it('collects the neighborhood of an asset and lays it out by hop depth', () => { + const g = buildColumnGraph(chainGraph()) + const seeds = assetColumnNodes(g, 'ducklake', 'wh/daily') // the sink asset + expect(new Set(seeds)).toEqual(new Set([DAILY_TOTAL, DAILY_CUST])) + const comp = connectedComponent(seeds, g) + expect(comp).toEqual( + new Set([DAILY_TOTAL, DAILY_CUST, STAGING_AMT, ORDERS_AMOUNT, CUSTOMERS_NAME]) + ) + const depths = computeDepths(comp, g) + expect(depths.get(ORDERS_AMOUNT)).toBe(0) + expect(depths.get(STAGING_AMT)).toBe(1) + expect(depths.get(DAILY_TOTAL)).toBe(2) + expect(depths.get(CUSTOMERS_NAME)).toBe(0) + expect(depths.get(DAILY_CUST)).toBe(1) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts new file mode 100644 index 0000000000..50fa1c4e7b --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/columnLineageGraph.ts @@ -0,0 +1,169 @@ +import type { AssetKind } from '$lib/gen' +import type { AssetGraphResponse } from './types' + +// A node in the column-level lineage graph: one column of one asset. +export type ColumnNode = { kind: AssetKind; path: string; column: string } +export type ColumnNodeId = string + +// Collision-proof node id — JSON-encoded tuple, so a `#`/`:` inside a path or +// (quoted) column name can't merge two distinct columns into one node. +export function colNodeId(kind: AssetKind, path: string, column: string): ColumnNodeId { + return JSON.stringify([kind, path, column]) +} + +// The pipeline-wide column-lineage graph, stitched across every producer. Each +// producer's `column_lineage` contributes single-hop edges (its output column ← +// its source columns); shared (asset,column) nodes chain those hops into the +// full transitive graph (`orders.amount → staging.amt → daily.total`). +export type ColumnLineageGraph = { + nodes: Map + // outputColumn → the source columns it derives from (walk upstream). + up: Map> + // sourceColumn → the output columns derived from it (walk downstream). + down: Map> +} + +// Build the column graph from a resolved asset graph. A producer's +// `column_lineage` describes the columns of the asset it materializes; that +// output asset is the ducklake target it writes (v1 materialize target), found +// from its write-edge. Producers without a known ducklake output are skipped +// (their columns can't be anchored to an asset node). +export function buildColumnGraph(graph: AssetGraphResponse): ColumnLineageGraph { + const nodes = new Map() + const up = new Map>() + const down = new Map>() + + const addNode = (n: ColumnNode): ColumnNodeId => { + const id = colNodeId(n.kind, n.path, n.column) + if (!nodes.has(id)) nodes.set(id, n) + return id + } + const addEdge = (src: ColumnNodeId, out: ColumnNodeId) => { + if (src === out) return + ;(up.get(out) ?? up.set(out, new Set()).get(out)!).add(src) + ;(down.get(src) ?? down.set(src, new Set()).get(src)!).add(out) + } + + // The output asset a runnable's `column_lineage` describes. The declared + // `// materialize` target is authoritative (a multi-output script writes + // several ducklake tables, and the deployed write-edges are unordered, so + // picking "a" write-edge can anchor to the wrong asset). Fall back to a + // ducklake write-edge only for producers with no materialize annotation + // (e.g. a literal single-output CTAS). + const outputAsset = new Map() + for (const r of graph.runnables ?? []) { + if (r.materialize_target) { + outputAsset.set(`${r.usage_kind}:${r.path}`, r.materialize_target) + } + } + for (const e of graph.edges ?? []) { + const access = e.access_type ?? 'r' + const key = `${e.runnable_kind}:${e.runnable_path}` + if ( + (access === 'w' || access === 'rw') && + e.asset_kind === 'ducklake' && + !outputAsset.has(key) + ) { + outputAsset.set(key, { kind: e.asset_kind, path: e.asset_path }) + } + } + + for (const r of graph.runnables ?? []) { + const lineage = r.column_lineage + if (!lineage || lineage.length === 0) continue + const out = outputAsset.get(`${r.usage_kind}:${r.path}`) + if (!out) continue + for (const cl of lineage) { + const outId = addNode({ kind: out.kind, path: out.path, column: cl.column }) + for (const inp of cl.inputs) { + const srcId = addNode({ + kind: inp.from_kind, + path: inp.from_path, + column: inp.from_column + }) + addEdge(srcId, outId) + } + } + } + + return { nodes, up, down } +} + +// Every node reachable from `start` by following `adj` (transitive closure, +// excluding `start` itself). Iterative to avoid deep-recursion limits. +function reach(start: ColumnNodeId, adj: Map>): Set { + const seen = new Set() + const stack = [start] + while (stack.length) { + const n = stack.pop()! + for (const m of adj.get(n) ?? []) { + if (!seen.has(m)) { + seen.add(m) + stack.push(m) + } + } + } + return seen +} + +// The full transitive trace of a column: itself + all upstream ancestors + all +// downstream descendants. This is the impact set — "everything that feeds, or +// is fed by, this column". +export function traceColumn(id: ColumnNodeId, g: ColumnLineageGraph): Set { + const out = new Set([id]) + for (const a of reach(id, g.up)) out.add(a) + for (const d of reach(id, g.down)) out.add(d) + return out +} + +// The connected neighborhood of a set of seed columns (an asset's columns): +// the seeds plus everything upstream and downstream of any of them. This is the +// subgraph the trace view renders around a selected asset. +export function connectedComponent( + seeds: ColumnNodeId[], + g: ColumnLineageGraph +): Set { + const out = new Set() + for (const s of seeds) { + if (!g.nodes.has(s)) continue + out.add(s) + for (const a of reach(s, g.up)) out.add(a) + for (const d of reach(s, g.down)) out.add(d) + } + return out +} + +// All column-node ids belonging to one asset (its seed set for a trace). +export function assetColumnNodes( + g: ColumnLineageGraph, + kind: AssetKind, + path: string +): ColumnNodeId[] { + const ids: ColumnNodeId[] = [] + for (const [id, n] of g.nodes) if (n.kind === kind && n.path === path) ids.push(id) + return ids +} + +// Longest-path depth of each node within `ids`, sources at depth 0 and depth +// increasing downstream — so a left→right layout reads upstream→downstream. +// Cycle-guarded (lineage is a DAG, but be defensive). +export function computeDepths( + ids: Set, + g: ColumnLineageGraph +): Map { + const depth = new Map() + const visiting = new Set() + const d = (id: ColumnNodeId): number => { + const memo = depth.get(id) + if (memo !== undefined) return memo + if (visiting.has(id)) return 0 + visiting.add(id) + let m = 0 + for (const u of g.up.get(id) ?? []) if (ids.has(u)) m = Math.max(m, d(u) + 1) + visiting.delete(id) + depth.set(id, m) + return m + } + for (const id of ids) d(id) + return depth +} diff --git a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts index 08da865d61..197d7eca6c 100644 --- a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.test.ts @@ -1,6 +1,10 @@ import { describe, expect, it } from 'vitest' import type { AssetGraphResponse } from './types' -import { buildDownstreamMap, computeDownstreamClosure } from './graphTraversal' +import { + buildDownstreamMap, + computeDownstreamClosure, + computeInducedSchedule +} from './graphTraversal' /** Direct (one-hop) subscriber script paths of `scriptPath`. */ function downstreamSubscribers(g: AssetGraphResponse, scriptPath: string): string[] { @@ -170,3 +174,54 @@ describe('computeDownstreamClosure', () => { expect(cl.cyclic).toEqual([]) }) }) + +describe('computeInducedSchedule', () => { + // a → b → c → d linear chain through assets. + const chain = () => + graph( + [ + ['a', 'xa'], + ['b', 'xb'], + ['c', 'xc'] + ], + [ + ['b', 'xa'], + ['c', 'xb'], + ['d', 'xc'] + ] + ) + + it('orders a subset topologically and keeps only in-set edges', () => { + const sched = computeInducedSchedule(chain(), new Set(['a', 'b', 'c'])) + expect(sched.nodes).toEqual(['a', 'b', 'c']) + expect(sched.roots).toEqual(['a']) + expect(sched.indegree.get('b')).toBe(1) + expect([...(sched.edges.get('c') ?? [])]).toEqual([]) // d is out of set + expect(sched.cyclic).toEqual([]) + }) + + it('a gap in the selected set produces two independent roots', () => { + // pick a and c (skip b): with b excluded there is no in-set edge, so both + // are roots. + const sched = computeInducedSchedule(chain(), new Set(['a', 'c'])) + expect(sched.roots.sort()).toEqual(['a', 'c']) + expect(sched.indegree.get('c')).toBe(0) + }) + + it('reports cyclic members instead of hanging', () => { + // b ↔ c cycle. + const g = graph( + [ + ['b', 'y'], + ['c', 'z'] + ], + [ + ['c', 'y'], + ['b', 'z'] + ] + ) + const sched = computeInducedSchedule(g, new Set(['b', 'c'])) + expect(sched.nodes).toEqual([]) + expect(sched.cyclic.sort()).toEqual(['b', 'c']) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts index 6b53020ddb..b89569e754 100644 --- a/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts +++ b/frontend/src/lib/components/assets/AssetGraph/graphTraversal.ts @@ -117,3 +117,87 @@ export function computeDownstreamClosure(g: AssetGraphResponse, root: string): D } return { nodes: ordered, edges: cleanEdges, indegree: cleanIndegree, cyclic } } + +export type InducedSchedule = { + /** Selected scripts that are schedulable, in a topological order. */ + nodes: string[] + /** In-set dependency edges: `edges.get(p)` = in-set subscribers of `p`. */ + edges: Map> + /** In-set upstream count per node. */ + indegree: Map + /** Schedulable scripts with no in-set upstream — the schedule's seeds. */ + roots: string[] + /** + * Selected scripts on (or fed only through) a cycle. Excluded from + * `nodes`/`edges`/`indegree`/`roots`; surfaced so callers can warn. + */ + cyclic: string[] +} + +/** + * Topological schedule of an arbitrary *set* of scripts (e.g. a bounded-cascade + * selection), respecting only the dependency edges that fall *inside* the set. + * Multi-root: every selected script with no in-set upstream is a seed. + * Cycle-safe in the same way as `computeDownstreamClosure`. + * + * `oneHop` is the producer→consumer adjacency (script path → downstream script + * paths). It defaults to the `// on` subscriber map — but a bounded run passes + * a *read-aware* map (boundedCascade.buildLineageDownstreamMap) so a selected + * script that merely *reads* an upstream asset still runs after its producer, + * matching the CLI's `topoOrder`. Keep the default subscriber-only: it mirrors + * the production asset-trigger dispatch the unbounded cascade simulates. + */ +export function computeInducedSchedule( + g: AssetGraphResponse, + selected: Set, + oneHop: Map> = buildDownstreamMap(g) +): InducedSchedule { + // In-set edges + indegrees. + const edges = new Map>() + const indegree = new Map() + for (const n of selected) indegree.set(n, 0) + for (const p of selected) { + const subs = new Set() + for (const s of oneHop.get(p) ?? []) { + if (!selected.has(s)) continue + subs.add(s) + indegree.set(s, (indegree.get(s) ?? 0) + 1) + } + if (subs.size > 0) edges.set(p, subs) + } + + // Kahn from every indegree-0 node. + const seeds = [...selected].filter((n) => (indegree.get(n) ?? 0) === 0) + const remaining = new Map(indegree) + const ready = [...seeds] + const ordered: string[] = [] + while (ready.length > 0) { + const n = ready.shift()! + ordered.push(n) + for (const s of edges.get(n) ?? []) { + const d = (remaining.get(s) ?? 0) - 1 + remaining.set(s, d) + if (d === 0) ready.push(s) + } + } + + const orderedSet = new Set(ordered) + const cyclic = [...selected].filter((n) => !orderedSet.has(n)) + if (cyclic.length === 0) { + return { nodes: ordered, edges, indegree, roots: seeds, cyclic } + } + + // Strip cyclic nodes from the schedulable structures. + const cyclicSet = new Set(cyclic) + const cleanEdges = new Map>() + const cleanIndegree = new Map() + for (const n of ordered) cleanIndegree.set(n, 0) + for (const [p, subs] of edges) { + if (cyclicSet.has(p)) continue + const kept = new Set([...subs].filter((s) => !cyclicSet.has(s))) + if (kept.size > 0) cleanEdges.set(p, kept) + for (const s of kept) cleanIndegree.set(s, (cleanIndegree.get(s) ?? 0) + 1) + } + const roots = ordered.filter((n) => (cleanIndegree.get(n) ?? 0) === 0) + return { nodes: ordered, edges: cleanEdges, indegree: cleanIndegree, roots, cyclic } +} diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts index 98f8edb523..3eb1e4e574 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.parity.test.ts @@ -18,7 +18,9 @@ const ASSERTED_TS_FIELDS: Record = { freshness: true, tag: true, retry: true, - materialize: true + materialize: true, + dataTests: true, + columnLineage: true } // Parser-parity guard: this TS parser (drives the live graph preview) and @@ -66,6 +68,13 @@ type Fixture = { append?: boolean unique_key?: string | null } | null + // Snake_case form matching the Rust `DataTest` serde output, so the one + // corpus drives both sides. The TS parser emits this shape verbatim + // (snake_case fields), so the comparison is 1:1. Absent === []. + data_tests?: Array> + // Snake_case `ColumnLineage` serde shape — TS parser emits it verbatim, + // so the comparison is 1:1. Absent === []. + column_lineage?: Array> } } @@ -154,6 +163,10 @@ describe('parsePipelineAnnotations matches the shared Rust fixture corpus', () = f.expected.materialize.unique_key ?? undefined ) } + + expect(got.dataTests, 'data tests').toEqual(f.expected.data_tests ?? []) + + expect(got.columnLineage, 'column lineage').toEqual(f.expected.column_lineage ?? []) }) } }) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts index f73819b959..5f4b063e3f 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.test.ts @@ -1,5 +1,9 @@ import { describe, expect, it } from 'vitest' -import { parsePipelineAnnotations } from './parsePipelineAnnotations' +import { + mergeColumnLineage, + parsePipelineAnnotations, + type ColumnLineage +} from './parsePipelineAnnotations' // Unit-test the TS mirror of the backend `parse_pipeline_annotations`. The // two implementations MUST stay behaviorally identical — these tests are @@ -25,6 +29,38 @@ describe('parsePipelineAnnotations: tag', () => { const out = parsePipelineAnnotations('// tagged heavy') expect(out.tag).toBeUndefined() }) + + it('skips a tag value containing whitespace (regular comment false-positive)', () => { + const out = parsePipelineAnnotations('# tag this function so we remember to refactor it later') + expect(out.tag).toBeUndefined() + }) + + it('skips a tag value longer than 50 chars', () => { + const out = parsePipelineAnnotations('// tag ' + 'x'.repeat(51)) + expect(out.tag).toBeUndefined() + }) +}) + +describe('parsePipelineAnnotations: header scan', () => { + it('ignores annotations in the body once code has started', () => { + const code = [ + 'import pandas as pd', + '', + 'def main():', + ' # tag each row with its source so downstream steps can filter', + ' # on s3://should/not/parse', + ' return pd.DataFrame()' + ].join('\n') + const out = parsePipelineAnnotations(code) + expect(out.tag).toBeUndefined() + expect(out.triggerAssets).toHaveLength(0) + }) + + it('tolerates blank lines before code but stops at the first code line', () => { + const code = ['#!/usr/bin/env python', '', '# tag heavy', 'import os', '# tag light'].join('\n') + const out = parsePipelineAnnotations(code) + expect(out.tag).toBe('heavy') + }) }) describe('parsePipelineAnnotations: retry', () => { @@ -99,3 +135,29 @@ describe('parsePipelineAnnotations: data_upload', () => { expect(out.nativeTriggers).toEqual([]) }) }) + +describe('mergeColumnLineage', () => { + const ref = (path: string, col: string): ColumnLineage['inputs'][number] => ({ + from_kind: 'ducklake', + from_path: path, + from_column: col + }) + + it('annotation wins per output column; inferred fills the rest (mirrors Rust)', () => { + const inferred: ColumnLineage[] = [ + { column: 'total', inputs: [ref('w/o', 'amount')] }, + { column: 'qty', inputs: [ref('w/o', 'qty')] } + ] + const annotated: ColumnLineage[] = [{ column: 'total', inputs: [ref('w/manual', 'grand')] }] + const merged = mergeColumnLineage(inferred, annotated) + expect(merged).toEqual([ + { column: 'total', inputs: [ref('w/manual', 'grand')] }, // annotation, first + authoritative + { column: 'qty', inputs: [ref('w/o', 'qty')] } // inferred, not overridden + ]) + }) + + it('returns annotations unchanged when there is no inferred lineage', () => { + const annotated: ColumnLineage[] = [{ column: 'a', inputs: [ref('w/o', 'a')] }] + expect(mergeColumnLineage([], annotated)).toEqual(annotated) + }) +}) diff --git a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts index 700513e39a..ccbc746b04 100644 --- a/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts +++ b/frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts @@ -89,6 +89,68 @@ export type MaterializeSpec = { uniqueKey?: string } +// `// data_test …` — a data-quality assertion run against the +// freshly-materialized asset, failing the run on violation. See backend +// `DataTest`. The first extensible annotation family: a keyword head selects +// the variant, the rest is parsed per-variant; a sibling family (e.g. +// column-lineage) follows the same shape. Built-ins mirror dbt's generic data +// tests; `custom` is dbt's singular-test escape hatch (a DuckDB script path). +// Keyword is `data_test`, NOT `test`, to stay clear of the unrelated `// test:` +// CI-test annotation. +// Field names are snake_case to match the Rust `DataTest` serde output verbatim +// — the same type is populated both by this parser (live drafts) and by the +// backend graph endpoint (deployed nodes), so the two must be wire-identical. +export type DataTest = + | { type: 'unique'; column: string } + | { type: 'not_null'; column: string } + | { type: 'accepted_values'; column: string; values: string[] } + | { + type: 'relationships' + column: string + to_kind: AssetKind + to_path: string + to_column: string + } + | { type: 'custom'; path: string } + +// `// column <- .[, …]` — declared column-level +// lineage: one output column and the upstream source columns it derives from. +// See backend `ColumnLineage`. A sibling of `DataTest` in the extensible +// annotation family — same parse shape, accumulating (one line per output +// column) — but pure metadata: drives the column-lineage graph view, runs no +// probe. Field names are snake_case to match the Rust `ColumnLineage` serde +// output verbatim, so the live-draft parse and the backend graph endpoint +// (deployed nodes) produce wire-identical shapes. +export type ColumnRef = { + from_kind: AssetKind + from_path: string + from_column: string +} +export type ColumnLineage = { + column: string + inputs: ColumnRef[] +} + +// Combine body-inferred column lineage with `// column` annotations, the +// annotation winning per output column. Mirrors the Rust `merge_column_lineage` +// (`asset_parser.rs`) so the live-draft preview matches what deploys: the +// backend already merges inferred + annotated server-side, and the live graph +// must apply the same precedence to the WASM-inferred lineage. +export function mergeColumnLineage( + inferred: ColumnLineage[], + annotated: ColumnLineage[] +): ColumnLineage[] { + const seen = new Set(annotated.map((c) => c.column)) + const out = [...annotated] + for (const c of inferred) { + if (!seen.has(c.column)) { + seen.add(c.column) + out.push(c) + } + } + return out +} + export type PipelineAnnotations = { inPipeline: boolean triggerAssets: PipelineTriggerAsset[] @@ -99,6 +161,10 @@ export type PipelineAnnotations = { retry?: RetrySpec // `// materialize [manual] [append] [key=]` — target + strategy. materialize?: MaterializeSpec + // `// data_test …` — accumulating data-quality checks (multiple lines). + dataTests: DataTest[] + // `// column <- .[, …]` — accumulating column lineage. + columnLineage: ColumnLineage[] } // Tokenize a `key=value [key="quoted value"] ...` option string. Bare @@ -182,6 +248,121 @@ function parseMaterializeSpec(s: string): MaterializeSpec | undefined { return { targetKind: asset.kind, targetPath: asset.path, manual, append, uniqueKey } } +// A single bare identifier token (column name). Rejects empty / multi-token +// input. Mirrors Rust `single_ident`. +function singleIdent(s: string): string | undefined { + const t = s.trim() + if (t === '' || t.split(/\s+/).length !== 1) return undefined + return t +} + +// Strip one layer of matching surrounding single or double quotes. +function unquote(s: string): string { + if (s.length >= 2 && (s[0] === '"' || s[0] === "'") && s[s.length - 1] === s[0]) { + return s.slice(1, -1) + } + return s +} + +// ` = a,b,c`. Mirrors Rust `parse_accepted_values`. +function parseAcceptedValues(s: string): DataTest | undefined { + const eq = s.indexOf('=') + if (eq < 0) return undefined + const column = singleIdent(s.slice(0, eq)) + if (!column) return undefined + const values = s + .slice(eq + 1) + .split(',') + .map((v) => unquote(v.trim())) + .filter((v) => v !== '') + if (values.length === 0) return undefined + return { type: 'accepted_values', column, values } +} + +// ` -> .`. Mirrors Rust `parse_relationships`. +function parseRelationships(s: string): DataTest | undefined { + const arrow = s.indexOf('->') + if (arrow < 0) return undefined + const column = singleIdent(s.slice(0, arrow)) + if (!column) return undefined + const target = s.slice(arrow + 2).trim() + const dot = target.lastIndexOf('.') + if (dot < 0) return undefined + const toColumn = singleIdent(target.slice(dot + 1)) + if (!toColumn) return undefined + const asset = parseAssetSyntaxDefault(target.slice(0, dot).trim()) + if (!asset || asset.path === '') return undefined + return { + type: 'relationships', + column, + to_kind: asset.kind, + to_path: asset.path, + to_column: toColumn + } +} + +// `.` upstream reference. The column is the segment after the +// final `.`; the rest is the asset URI (default-syntax shorthands enabled). +// Mirrors Rust `parse_column_ref`. +function parseColumnRef(s: string): ColumnRef | undefined { + const dot = s.lastIndexOf('.') + if (dot < 0) return undefined + const fromColumn = singleIdent(s.slice(dot + 1)) + if (!fromColumn) return undefined + const asset = parseAssetSyntaxDefault(s.slice(0, dot).trim()) + if (!asset || asset.path === '') return undefined + return { from_kind: asset.kind, from_path: asset.path, from_column: fromColumn } +} + +// ` <- [, …]`. Individually malformed refs are dropped; the +// line is kept iff ≥1 ref parses (mirrors `parseAcceptedValues`). A missing +// `<-`, a non-ident output column, or zero valid refs drops the line. +// Mirrors Rust `parse_column_lineage_spec`. +function parseColumnLineageSpec(s: string): ColumnLineage | undefined { + const arrow = s.indexOf('<-') + if (arrow < 0) return undefined + const column = singleIdent(s.slice(0, arrow)) + if (!column) return undefined + const inputs = s + .slice(arrow + 2) + .split(',') + .map((r) => parseColumnRef(r.trim())) + .filter((r): r is ColumnRef => r !== undefined) + if (inputs.length === 0) return undefined + return { column, inputs } +} + +// Parse a `// data_test …` right-hand side into one `DataTest`. The +// leading token selects the variant; anything not a built-in keyword is the +// `custom` escape hatch (a single script-path token). Returns `undefined` for +// malformed input so a typo fails safe. Mirrors Rust `parse_data_test_spec`. +function parseDataTestSpec(s: string): DataTest | undefined { + const trimmed = s.trim() + if (trimmed === '') return undefined + const m = trimmed.match(/^(\S+)(?:\s+([\s\S]*))?$/) + if (!m) return undefined + const head = m[1] + const rest = (m[2] ?? '').trim() + switch (head) { + case 'unique': { + const column = singleIdent(rest) + return column ? { type: 'unique', column } : undefined + } + case 'not_null': { + const column = singleIdent(rest) + return column ? { type: 'not_null', column } : undefined + } + case 'accepted_values': + return parseAcceptedValues(rest) + case 'relationships': + return parseRelationships(rest) + default: + // Custom escape hatch: the whole right-hand side must be one path + // token. Trailing content after a non-built-in head is rejected. + return rest === '' ? { type: 'custom', path: head } : undefined + } +} + type ParsedTriggerSpec = | { kind: 'asset'; value: PipelineTriggerAsset } | { kind: 'native'; value: PipelineNativeTrigger } @@ -287,12 +468,19 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { const out: PipelineAnnotations = { inPipeline: false, triggerAssets: [], - nativeTriggers: [] + nativeTriggers: [], + dataTests: [], + columnLineage: [] } for (const rawLine of code.split('\n')) { + // Annotations live in the leading comment header: skip blank lines but + // stop at the first line of actual code, so comments inside the body + // (e.g. a regular `# tag ...` prose comment) can't false-positive. + // Mirrors the Rust parse_pipeline_annotations header scan. + if (rawLine.trim() === '') continue const rest = stripCommentPrefix(rawLine) - if (rest === undefined) continue + if (rest === undefined) break const inner = rest.trimStart() const afterPipeline = consumeKeyword(inner, 'pipeline') @@ -323,7 +511,10 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { const afterTag = consumeKeyword(inner, 'tag') if (afterTag !== undefined) { const name = afterTag.trim() - if (name && !out.tag) { + // Worker tags are single-word identifiers; a value with whitespace + // or beyond the script.tag column width is almost certainly a + // regular comment starting with "# tag ...". + if (name && !out.tag && !/\s/.test(name) && name.length <= 50) { out.tag = name } continue @@ -347,6 +538,24 @@ export function parsePipelineAnnotations(code: string): PipelineAnnotations { continue } + // `data_test` is a complete word (so it never collides with the `// test:` + // CI annotation, which has no whitespace after `test`). Accumulates. + const afterDataTest = consumeKeyword(inner, 'data_test') + if (afterDataTest !== undefined) { + const spec = parseDataTestSpec(afterDataTest.trim()) + if (spec) out.dataTests.push(spec) + continue + } + + // `column` is a complete word; a body comment that merely starts with + // `column` has no `<-` and is dropped fail-safe. Accumulates. + const afterColumn = consumeKeyword(inner, 'column') + if (afterColumn !== undefined) { + const spec = parseColumnLineageSpec(afterColumn.trim()) + if (spec) out.columnLineage.push(spec) + continue + } + const afterOn = consumeKeyword(inner, 'on') if (afterOn !== undefined) { const specText = afterOn.trim() diff --git a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts index 380e4300a3..cbb5476516 100644 --- a/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts +++ b/frontend/src/lib/components/assets/AssetGraph/pipelineTemplates.ts @@ -546,6 +546,14 @@ function bodyDuckdb(ctx: TemplateContext): string { } if (ducklakeDb) { lines.push(`ATTACH 'ducklake://${ducklakeDb}' AS lake;`) + if (input?.kind === 'ducklake') { + // Discoverability hint: every materialize records a DuckLake snapshot, + // so a consumer can pin its read to a past version. Snapshot ids live + // in the asset's History tab. + lines.push( + `-- time-travel: read a past snapshot with \`FROM ${`lake.${catalogTableRef(input.path)}`} AT (VERSION => 42)\`` + ) + } } if (datatableDb || ducklakeDb) lines.push('') diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts index 78199a3871..dd4c64b225 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.test.ts @@ -8,6 +8,7 @@ const ann = (over: Partial = {}): PipelineAnnotations => ({ inPipeline: false, triggerAssets: [], nativeTriggers: [], + dataTests: [], ...over }) diff --git a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts index 6e4f512000..b7c82822fa 100644 --- a/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts +++ b/frontend/src/lib/components/assets/AssetGraph/resolveGraph.ts @@ -1,5 +1,10 @@ import type { AssetGraphResponse, NativeTriggerKind } from './types' -import { parsePipelineAnnotations, type PipelineAnnotations } from './parsePipelineAnnotations' +import { + mergeColumnLineage, + parsePipelineAnnotations, + type ColumnLineage, + type PipelineAnnotations +} from './parsePipelineAnnotations' import { extractWrites, extractReads, @@ -20,7 +25,12 @@ export type ResolveGraphInput = { /** In-flight drafts keyed by script path. */ drafts: Map /** Body assets inferred for the currently-open script (live keystrokes). */ - liveBodyAssets: { scriptPath: string | undefined; assets: AssetWithAltAccessType[] } + liveBodyAssets: { + scriptPath: string | undefined + assets: AssetWithAltAccessType[] + /** Body-inferred column lineage (DuckDB SQL AST) for the open script. */ + columnLineage?: ColumnLineage[] + } /** Pipeline annotations parsed from the currently-open buffer. */ liveAnnotations: { scriptPath: string | undefined; annotations: PipelineAnnotations } /** Sticky session caches of inferred body writes/reads per script path. */ @@ -173,6 +183,13 @@ function makeContext(input: ResolveGraphInput): ResolveContext { if (liveAnnotations.scriptPath === openPath) { for (const a of liveAnnotations.annotations.triggerAssets) liveRefKeys.add(`${a.kind}:${a.path}`) + // The `// materialize ` target is a declared *output*, but it + // lives in an annotation (not the SQL body), so neither triggerAssets + // (inputs) nor the body-inferred assets cover it. Without this its + // persisted write-edge is judged stale and dropped the moment the + // script is selected/edited — leaving the output asset unlinked. + const m = liveAnnotations.annotations.materialize + if (m) liveRefKeys.add(`${m.targetKind}:${m.targetPath}`) } for (const a of liveBodyAssets.assets) liveRefKeys.add(`${a.kind}:${a.path}`) } @@ -215,6 +232,19 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { for (const [path, d] of drafts) { const parsed = parsePipelineAnnotations(d.script.content) + // For the open script, fold in the WASM-inferred column lineage (DuckDB + // SQL AST) under the same annotation-wins precedence the backend applies + // on deploy, so the live preview matches what deploys. Only the open + // script carries live inference (`liveBodyAssets`); other drafts stay + // annotation-only until they deploy (the backend infers then). + const inferredCL = + path === liveBodyAssets.scriptPath ? (liveBodyAssets.columnLineage ?? []) : [] + const mergedCL = mergeColumnLineage(inferredCL, parsed.columnLineage) + // The `// materialize` target this draft's column lineage describes, so + // the column graph anchors to it rather than guessing a write-edge. + const materializeTarget = parsed.materialize + ? { kind: parsed.materialize.targetKind, path: parsed.materialize.targetPath } + : undefined // A draft can coexist with a base entry — during save the refetch // lands before drafts cleanup, and a user re-editing a deployed // script also produces both. In that case we mutate the existing @@ -232,10 +262,23 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { freshness: parsed.freshness?.duration, tag: parsed.tag, retry: parsed.retry, + data_tests: parsed.dataTests.length > 0 ? parsed.dataTests : undefined, + column_lineage: mergedCL.length > 0 ? mergedCL : undefined, + materialize_target: materializeTarget, unsaved: true }) } else { - runnables[baseIdx] = { ...runnables[baseIdx], unsaved: true } + // Refresh annotation-derived badges from the live parse too, so + // adding/removing `// data_test` / `// column` lines on an + // already-deployed script updates the badge immediately (not only + // after redeploy/refetch). + runnables[baseIdx] = { + ...runnables[baseIdx], + data_tests: parsed.dataTests.length > 0 ? parsed.dataTests : undefined, + column_lineage: mergedCL.length > 0 ? mergedCL : undefined, + materialize_target: materializeTarget, + unsaved: true + } } // Output asset(s): three-tier resolution. // 1. Active draft (the body the user is editing right now): @@ -261,6 +304,16 @@ function seedDraftOverlays(acc: Accumulator, input: ResolveGraphInput) { if (writeOuts.length === 0 && d.outputAsset) { writeOuts.push(d.outputAsset) } + // `// materialize ` declares a write output via annotation, not + // the SQL body, so the body-inference tiers above miss it. Add it from + // the live-parsed annotations so an edited materialize script keeps its + // output edge (the loop below dedups against existing assets/edges). + if (parsed.materialize) { + writeOuts.push({ + kind: parsed.materialize.targetKind, + path: parsed.materialize.targetPath + }) + } for (const out of writeOuts) { const hasAsset = assets.some((a) => a.kind === out.kind && a.path === out.path) if (!hasAsset) assets.push({ kind: out.kind, path: out.path }) diff --git a/frontend/src/lib/components/assets/AssetGraph/types.ts b/frontend/src/lib/components/assets/AssetGraph/types.ts index 5f3aa4b52d..9276f500af 100644 --- a/frontend/src/lib/components/assets/AssetGraph/types.ts +++ b/frontend/src/lib/components/assets/AssetGraph/types.ts @@ -1,4 +1,5 @@ import type { AssetKind } from '$lib/gen' +import type { ColumnLineage, DataTest } from './parsePipelineAnnotations' export type GraphUsageKind = 'script' | 'flow' @@ -28,6 +29,23 @@ export interface AssetGraphRunnableNode { // duration string (`"5s"`, `"30s"`); absent = back-to-back. Surfaced as // a badge so retry-enabled scripts are visible without opening the pane. retry?: { count: number; delay?: string } + // `// data_test …` data-quality checks run against the materialized + // asset. Surfaced as a count badge (with a per-test breakdown in the title) + // so test coverage is visible on the node without opening the pane. + data_tests?: DataTest[] + // `// column <- .` declared column-level lineage for this + // script's materialized output. Surfaced as a count badge on the write-edge + // and as a column-to-column diagram in the asset details pane. + column_lineage?: ColumnLineage[] + // `// materialize ` target — the asset `column_lineage` describes. + // Lets the column graph anchor lineage to the exact output instead of + // guessing a ducklake write-edge (a multi-output script writes several). + materialize_target?: { kind: AssetKind; path: string } + // Managed `// materialize` write strategy. Absent for non-materializing or + // `manual` scripts. Used (with `partition_kind`) to decide whether a + // produced asset's schema can evolve: only whole-table `replace` can, since + // `append`/`merge`/partitioned writes INSERT into a fixed-schema table. + materialize_strategy?: 'replace' | 'append' | 'merge' // Synthesized by the page from a local draft; the script doesn't exist // in the DB yet. Drives a dashed/lower-opacity rendering to mirror how // unsaved triggers are styled — visually distinct from persisted nodes. diff --git a/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte b/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte index 2d7a9cd3ff..26366325b5 100644 --- a/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte +++ b/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte @@ -38,6 +38,13 @@ draftSavedAt?: string | undefined /** ISO timestamp of the latest deploy at this path. */ deployedAt?: string | undefined + /** Precise staleness inputs (flows/apps): the deployed version the draft was + * forked from, and the current deployed head. When both are set they drive + * `isStale` and the dedup key instead of the timestamps — exact, and stable + * across autosaves (the timestamp drifts past `deployedAt` as you keep + * editing). Absent (pre-feature drafts, scripts) ⇒ timestamp fallback. */ + draftBaseVersion?: number | undefined + deployedHeadVersion?: number | undefined /** Discard the draft and reload deployed (same as "Reset to deployed"). */ onLoadLatestDeploy?: () => void | Promise /** Defaults to true; set to false to suppress all modals. */ @@ -56,6 +63,8 @@ othersModalOpen = $bindable(), draftSavedAt = undefined, deployedAt = undefined, + draftBaseVersion = undefined, + deployedHeadVersion = undefined, onLoadLatestDeploy, enabled = true }: Props = $props() @@ -65,13 +74,27 @@ let staleAlertKey = $state(undefined) let staleModalOpen = $state(false) + // Prefer the exact version comparison (flows/apps) over the timestamp: the + // draft's pinned fork base never drifts, whereas `draftSavedAt` advances past + // `deployedAt` once you keep editing a stale draft, hiding the staleness. + const useVersion = $derived(draftBaseVersion != null && deployedHeadVersion != null) const isStale = $derived( - !!draftSavedAt && - !!deployedAt && - !!onLoadLatestDeploy && - new Date(draftSavedAt).getTime() < new Date(deployedAt).getTime() + !!onLoadLatestDeploy && + (useVersion + ? draftBaseVersion !== deployedHeadVersion + : !!draftSavedAt && + !!deployedAt && + new Date(draftSavedAt).getTime() < new Date(deployedAt).getTime()) + ) + // Key on the versions (not `draftSavedAt`) in the version path, else every + // autosave would mint a new key and re-pop the modal mid-edit. + const currentKey = $derived( + isStale + ? useVersion + ? `${path}|v|${draftBaseVersion}|${deployedHeadVersion}` + : `${path}|${draftSavedAt}|${deployedAt}` + : undefined ) - const currentKey = $derived(isStale ? `${path}|${draftSavedAt}|${deployedAt}` : undefined) $effect(() => { const key = currentKey diff --git a/frontend/src/lib/components/common/table/RawAppRow.svelte b/frontend/src/lib/components/common/table/RawAppRow.svelte index e759a1d9fe..3f74356506 100644 --- a/frontend/src/lib/components/common/table/RawAppRow.svelte +++ b/frontend/src/lib/components/common/table/RawAppRow.svelte @@ -33,7 +33,7 @@ {:else if effectiveKind === 'app' || effectiveKind === 'raw_app'} + {:else if effectiveKind === 'raw_app_file'} + {:else if effectiveKind === 'script'} {:else if effectiveKind === 'variable'} diff --git a/frontend/src/lib/components/copilot/chat/AIChat.svelte b/frontend/src/lib/components/copilot/chat/AIChat.svelte index add2c7b44f..17578d6e0e 100644 --- a/frontend/src/lib/components/copilot/chat/AIChat.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChat.svelte @@ -163,7 +163,8 @@ {headerLeft} hasDiff={aiChatManager.scriptEditorOptions && !!aiChatManager.scriptEditorOptions.lastDeployedCode && - aiChatManager.scriptEditorOptions.lastDeployedCode !== aiChatManager.scriptEditorOptions.code} + aiChatManager.scriptEditorOptions.lastDeployedCode !== + aiChatManager.scriptEditorOptions.getCode()} diffMode={aiChatManager.scriptEditorOptions?.diffMode ?? false} {disabled} {disabledMessage} diff --git a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte index e9ac5eed16..3a53aceb1a 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChatDisplay.svelte @@ -10,6 +10,8 @@ ChevronDown, ChevronsRight, CheckIcon, + FileText, + Folder, Hand, HistoryIcon, Hourglass, @@ -26,9 +28,8 @@ import { isActiveUserQuestion, type DisplayMessage } from './shared' import type { ContextElement } from './context' import ChatQuickActions from './ChatQuickActions.svelte' - import ProviderModelSelector from './ProviderModelSelector.svelte' import ContextUsageIndicator from './ContextUsageIndicator.svelte' - import AIChatSettingsMenu from './AIChatSettingsMenu.svelte' + import AIChatModelSettings from './AIChatModelSettings.svelte' import ChatMode from './ChatMode.svelte' import DatatableCreationPolicy from './DatatableCreationPolicy.svelte' import Tooltip from '$lib/components/meltComponents/Tooltip.svelte' @@ -41,6 +42,15 @@ import QueuedMessageChip from './QueuedMessageChip.svelte' import { getModifierKey } from '$lib/utils' import type { SelectedContext } from './app/core' + import AttachedFilesBar from './files/AttachedFilesBar.svelte' + import { type FileToAttach } from './files/attachedFiles.svelte' + import { + hasFileSystemAccess, + pickDirectory, + handlesFromDataTransfer, + readDroppedEntries + } from './files/fsAccess' + import { sendUserToast } from '$lib/toast' const MAX_YOLO_TOOLTIP_TOOLS = 8 const aiChatManager = getAiChatManager() @@ -242,16 +252,142 @@ const showTypingIndicator = $derived(aiChatManager.loading) - // `@` context picker is offered in modes that accept workspace/script/flow - // references (SCRIPT, FLOW, GLOBAL → workspace items + code blocks) or in - // APP mode (datatables, frontend files, etc.). Other modes (NAVIGATOR, - // ASK, API) don't accept @-context. + // The manual `@` context-picker button. Shown in SCRIPT/FLOW (workspace items + + // code blocks) and APP (datatables, frontend files). Hidden in GLOBAL — there + // `@`-context is still invoked inline by typing `@` in the input, so the button + // is redundant. NAVIGATOR/ASK/API don't take @-context at all. const showContextPicker = $derived( aiChatManager.mode === AIMode.SCRIPT || aiChatManager.mode === AIMode.FLOW || - aiChatManager.mode === AIMode.GLOBAL || aiChatManager.mode === AIMode.APP ) + + // File attachment is GLOBAL-mode only. + const canAttachFiles = $derived(aiChatManager.mode === AIMode.GLOBAL && !disabled) + // Steers the OS file picker toward text formats (soft hint; content sniff is authoritative). + const TEXT_FILE_ACCEPT = + 'text/*,.txt,.csv,.tsv,.json,.jsonl,.ndjson,.md,.markdown,.log,.yaml,.yml,.toml,.ini,.cfg,.conf,.env,.xml,.html,.htm,.css,.js,.mjs,.cjs,.ts,.tsx,.jsx,.py,.rb,.rs,.go,.java,.kt,.c,.h,.cpp,.cc,.cs,.php,.sh,.bash,.zsh,.sql,.svelte,.vue,.dockerfile' + let fileInputEl = $state(null) + let folderInputEl = $state(null) + let dragDepth = $state(0) + const isDraggingFiles = $derived(dragDepth > 0) + // File System Access API → live re-grantable folder handles (refreshed each turn). + // Otherwise folders are snapshotted into the browser (via webkitdirectory / dropped-entry + // walk), same as files. Either way folders display identically. + const canUseFsAccess = hasFileSystemAccess() + + function reportAddResult(added: string[], rejected: { name: string; reason: string }[]) { + if (rejected.length === 0) return + // Single rejected file (e.g. one dropped image): show the precise reason. + if (added.length === 0 && rejected.length === 1) { + sendUserToast(`Could not attach "${rejected[0].name}": ${rejected[0].reason}`, true) + return + } + // Otherwise (folders / multi-select): summarize to avoid a flood of toasts. The only + // per-file rejection left is non-text content (binary files are skipped). + const lead = added.length + ? `Attached ${added.length}, skipped ${rejected.length}` + : `Skipped ${rejected.length} file${rejected.length === 1 ? '' : 's'}` + sendUserToast(`${lead} (non-text).`, added.length === 0) + } + + async function handleAddFiles(files: FileList | FileToAttach[]) { + const { added, rejected } = await aiChatManager.attachedFiles.addFiles(files) + reportAddResult(added, rejected) + } + + async function addDirHandle(dir: FileSystemDirectoryHandle) { + const { added, rejected } = await aiChatManager.attachedFiles.addFolder(dir) + reportAddResult(added, rejected) + } + + function linkFiles() { + // Files are always snapshotted (every browser), so the universal picker is fine. + fileInputEl?.click() + } + + async function linkFolder() { + if (!canUseFsAccess) { + // No File System Access API → pick a folder via the directory input; its files are + // snapshotted into the browser (no live handle), grouped under the folder name. + folderInputEl?.click() + return + } + let dir: FileSystemDirectoryHandle | undefined + try { + dir = await pickDirectory() + } catch (e) { + // The picker threw instead of opening — surface why (e.g. a browser/enterprise + // policy blocking the File System Access API) rather than appearing to do nothing. + sendUserToast( + `Couldn't open the folder picker: ${e instanceof Error ? e.message : String(e)}`, + true + ) + return + } + if (dir) await addDirHandle(dir) + } + + function dragHasFiles(e: DragEvent): boolean { + return Array.from(e.dataTransfer?.types ?? []).includes('Files') + } + + function onPanelDragEnter(e: DragEvent) { + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + dragDepth++ + } + function onPanelDragOver(e: DragEvent) { + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + if (e.dataTransfer) e.dataTransfer.dropEffect = 'copy' + } + function onPanelDragLeave(_e: DragEvent) { + if (!canAttachFiles) return + dragDepth = Math.max(0, dragDepth - 1) + } + async function onPanelDrop(e: DragEvent) { + dragDepth = 0 + if (!canAttachFiles || !dragHasFiles(e)) return + e.preventDefault() + const dt = e.dataTransfer + if (!dt) return + if (canUseFsAccess) { + // getAsFileSystemHandle calls are kicked off synchronously inside this call. + const handles = await handlesFromDataTransfer(dt) + for (const h of handles) { + if (h.kind === 'directory') { + // Folders link as a live handle. + await addDirHandle(h as FileSystemDirectoryHandle) + } else { + // Files are always snapshotted (handle discarded). + await handleAddFiles([{ file: await (h as FileSystemFileHandle).getFile() }]) + } + } + } else { + // Fallback (no File System Access API): snapshot dropped files AND folders by walking + // the legacy webkitGetAsEntry tree. readDroppedEntries reads the entries synchronously + // (they're only valid during this event) before its first await; if it yields nothing + // (no entry API), fall back to the flat dt.files. + const entries = await readDroppedEntries(Array.from(dt.items ?? [])) + if (entries.length > 0) await handleAddFiles(entries) + else if (dt.files.length > 0) await handleAddFiles(dt.files) + } + } + + function onFileInputChange(e: Event) { + const input = e.currentTarget as HTMLInputElement + if (input.files && input.files.length > 0) void handleAddFiles(input.files) + input.value = '' // allow re-selecting the same file + } + + function onFolderInputChange(e: Event) { + const input = e.currentTarget as HTMLInputElement + // webkitdirectory files carry webkitRelativePath (`folder/sub/file`); addFiles groups + // them under the folder and skips junk paths. Snapshot, like a dropped folder. + if (input.files && input.files.length > 0) void handleAddFiles(input.files) + input.value = '' + } const availableAutonomyModeOptions = $derived.by(() => autonomyModeOptions.filter((option) => isAutonomyModeAvailable( @@ -339,7 +475,28 @@ -
    +
    + {#if isDraggingFiles} +
    +
    + + Drop files to attach +
    +
    + {/if} {#if !hideHeader}
    {/if}
    + {#if aiChatManager.mode === AIMode.GLOBAL} + + + {/if} {#if inputPreface} {@render inputPreface()} {/if} @@ -544,6 +707,7 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. --> bind:this={aiChatInput} bind:selectedContext {availableContext} + showContext={aiChatManager.mode !== AIMode.GLOBAL} disabled={disabled || hasActiveUserQuestion} isFirstMessage={messages.length === 0} /> @@ -595,11 +759,76 @@ the panel, or the Escape-to-stop focus check would wrongly reject them. --> setShowing={(showing) => { if (!showing) close() }} + onSelectFile={(name) => { + aiChatInput?.insertFileMention(name) + close() + }} /> {/if} {/snippet} {/if} + {#if canAttachFiles} + [ + { displayName: 'Attach file', icon: FileText, action: () => linkFiles() }, + { + // A real (live) link needs the File System Access API; without it the + // folder is only snapshotted, so call it "Add folder", not "Link folder". + displayName: canUseFsAccess ? 'Link folder' : 'Add folder', + icon: Folder, + tooltip: canUseFsAccess + ? 'Linked live — the assistant reads the folder’s current files from disk and refreshes each turn.' + : 'Loaded as a snapshot — the folder’s files are copied into your browser (they won’t auto-update). For a live link that refreshes from disk, use a Chromium-based browser (Chrome, Edge).', + action: () => linkFolder() + } + ]} + placement="bottom-start" + fixedHeight={false} + > + {#snippet buttonReplacement()} + +
    {/if}
    -
    - -
    {#if (aiChatManager.mode === AIMode.NAVIGATOR || aiChatManager.mode === AIMode.ASK) && suggestions.length > 0 && messages.filter((m) => m.role === 'user').length === 0 && !disabled}
    diff --git a/frontend/src/lib/components/copilot/chat/AIChatInput.svelte b/frontend/src/lib/components/copilot/chat/AIChatInput.svelte index 3d9e5c550f..cd735b85ce 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatInput.svelte +++ b/frontend/src/lib/components/copilot/chat/AIChatInput.svelte @@ -6,6 +6,7 @@ import type { ContextElement } from './context' import { AIMode } from './AIChatManager.svelte' import { CHAT_INPUT_PADDING, getAiChatManager } from './aiChatManagerContext' + import { formatMention } from './mention' import { twMerge } from 'tailwind-merge' import { tick, untrack, type Snippet } from 'svelte' import Portal from '$lib/components/Portal.svelte' @@ -196,6 +197,13 @@ focusInput() } + /** Insert a plain @filename mention for an attached file (used by the @ menu Files category). */ + export function insertFileMention(name: string) { + const sep = instructions.length === 0 || instructions.endsWith(' ') ? '' : ' ' + instructions = `${instructions}${sep}${formatMention(name)} ` + focusInput() + } + function clickOutside(node: HTMLElement) { function handleClick(event: MouseEvent) { if (node && !node.contains(event.target as Node)) { @@ -222,7 +230,9 @@ // Workspace items are fetched on-demand and not in availableContext, // so skip the availableContext check for them const isWorkspaceItem = - contextElement.type === 'workspace_script' || contextElement.type === 'workspace_flow' + contextElement.type === 'workspace_script' || + contextElement.type === 'workspace_flow' || + contextElement.type === 'workspace_app' if ( !isWorkspaceItem && !availableContext.find( diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index 0eb9a79707..fdba95ca7f 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -73,18 +73,25 @@ import { getCurrentModel, tryGetCurrentModel, getCombinedCustomPrompt, + getCustomPromptParts, + getUserCustomPrompts, + setUserCustomPrompts, isWebSearchEnabledForProvider } from '$lib/aiStore' import type { WorkspaceMutationTarget } from './workspaceTools' import { globalToolsFor, + loadWorkspaceSkills, prepareGlobalSystemMessage, prepareGlobalUserMessage, + type AiSkillListItem, type GlobalToolHelpers } from './global/core' import { isGlobalAiEnabled } from './global/gate' import { scopedKey, onUserChange, migrateLegacyLocalStorage } from '$lib/userScopedStorage' import { getLocalSetting, storeLocalSetting } from '$lib/utils' +import { AttachedFilesStore } from './files/attachedFiles.svelte' +import { appendAttachedFilesRoster } from './files/fileTools' // Compaction of the stored history: once the projected request size // (contextTokens — the provider's report when current, a fresh chars/4 @@ -109,6 +116,15 @@ const MAX_CONSECUTIVE_COMPACTION_FAILURES = 3 // (panel teardown, save-and-clear) pass their own reason, so the queued-message // flush can tell "the user wants to move on" from "the turn was torn down". const USER_CANCEL_REASON = 'user_cancelled' +// Built-in `/compact` session command — summarizes the conversation locally +// instead of sending a turn to the model. Matched on the whole input so a +// regular message that merely mentions "/compact" mid-sentence is unaffected. +const COMPACT_COMMAND_NAME = 'compact' +const COMPACT_COMMAND_RE = /^\/compact\s*$/ +// Built-in `/clear` session command — saves the conversation to history and +// resets to a fresh chat (the "New chat" action), instead of sending a turn. +const CLEAR_COMMAND_NAME = 'clear' +const CLEAR_COMMAND_RE = /^\/clear\s*$/ const AI_AUTONOMY_MODE_STORAGE_KEY = 'ai-chat-autonomy-mode' const LEGACY_AUTO_ACCEPT_TOOL_CONFIRMATIONS_STORAGE_KEY = 'ai-chat-yolo-mode' const WEB_SEARCH_ERROR_HINT = @@ -230,6 +246,8 @@ function getSendRequestErrorMessage(err: unknown, webSearchUnavailable: boolean) export class AIChatManager { contextManager = new ContextManager() historyManager = new HistoryManager() + /** Files the user attached to the current GLOBAL-mode conversation. */ + attachedFiles = new AttachedFilesStore() abortController: AbortController | undefined = undefined inlineAbortController: AbortController | undefined = undefined // Flag to skip Responses API if it's not available (e.g., Azure region doesn't support it) @@ -318,6 +336,31 @@ export class AIChatManager { // session rather than the UI-active one — keeps backgrounded sessions isolated. sessionId: string | undefined = undefined + // Workspace AI skills (name + description) advertised in the GLOBAL system + // prompt and surfaced as slash commands in session chat. Loaded + // asynchronously when entering GLOBAL mode; the system message is rebuilt + // once they resolve. + globalSkills = $state([]) + private globalSkillsRefreshId = 0 + + // Built-in session-chat slash commands, listed in the command picker + // alongside workspace skills. Unlike a skill, these run locally and never + // reach the model; the submit path intercepts them first, so they shadow any + // workspace skill of the same name. + readonly sessionBuiltinCommands: AiSkillListItem[] = [ + { name: COMPACT_COMMAND_NAME, description: 'Summarize the conversation to free up context' }, + { name: CLEAR_COMMAND_NAME, description: 'Clear the conversation and start a new chat' } + ] + + // Built-ins followed by workspace skills, with any skill whose name collides + // with a built-in dropped: the picker keys leaves by name, so a duplicate + // would break its keyed list and ambiguous-resolve nav. Built-ins win — they + // already shadow same-named skills at execution (the submit interception). + sessionCommands: AiSkillListItem[] = $derived([ + ...this.sessionBuiltinCommands, + ...this.globalSkills.filter((s) => !this.sessionBuiltinCommands.some((b) => b.name === s.name)) + ]) + allowedModes: Record = $derived({ script: this.flowAiChatHelpers === undefined && @@ -419,6 +462,64 @@ export class AIChatManager { return freed } + /** + * Core summarize + rewrite, shared by automatic and manual compaction. Sends + * the prefix to the summarizer, then replaces the summarized prefix with a + * single summary message in `messages` (as a user message) and + * `displayMessages` (as a `summary` boundary). Surviving tail user messages + * have their restart `index` re-based onto the new history: the summary + * occupies slot 0, so a tail user message that was at `keepFrom` lands at slot + * 1. `displayKeepFrom` is where the kept tail begins in `displayMessages`. + * + * Owns only the `compacting` flag and the history rewrite; callers own trigger + * policy (circuit breaker, gates) and persistence. Returns the outcome — + * 'aborted' is a user Stop (history left untouched), distinct from 'error'. + */ + private runSummarization = async ( + prefix: ChatCompletionMessageParam[], + tail: ChatCompletionMessageParam[], + keepFrom: number, + displayKeepFrom: number, + abortController: AbortController + ): Promise<'ok' | 'empty' | 'aborted' | 'error'> => { + this.compacting = true + try { + const raw = await getNonStreamingCompletion( + [...prefix, { role: 'user', content: getCompactionSummaryPrompt() }], + abortController + ) + const formatted = formatCompactSummary(raw ?? '') + if (!formatted) { + return 'empty' + } + + this.messages = [{ role: 'user', content: buildSummaryMessageContent(formatted) }, ...tail] + + // Replace the summarized display prefix with the boundary marker and + // re-base the surviving tail's restart indices (the summary occupies + // slot 0, so the tail now starts at slot 1). + this.displayMessages = [ + { role: 'summary', content: formatted }, + ...this.displayMessages + .slice(displayKeepFrom) + .map((m) => (m.role === 'user' ? { ...m, index: m.index - keepFrom + 1 } : m)) + ] + + // The provider report described the pre-compaction history; the new + // history is much smaller, so clear it and let readers re-estimate. + this.contextUsage = undefined + return 'ok' + } catch (err) { + if (abortController.signal.aborted) { + return 'aborted' + } + console.error('Conversation summarization failed', err) + return 'error' + } finally { + this.compacting = false + } + } + /** * Summary-based partial compaction. Summarizes the older PREFIX of the stored * history into a single user message and keeps the recent tail verbatim, @@ -493,45 +594,94 @@ export class AIChatManager { return false } - this.compacting = true - try { - const raw = await getNonStreamingCompletion( - [...prefix, { role: 'user', content: getCompactionSummaryPrompt() }], - abortController - ) - const formatted = formatCompactSummary(raw ?? '') - if (!formatted) { - this.consecutiveCompactionFailures++ - return false - } - - this.messages = [{ role: 'user', content: buildSummaryMessageContent(formatted) }, ...tail] - - // Replace the summarized display prefix with the boundary marker and - // re-base the surviving tail's restart indices (the summary occupies - // slot 0, so the tail now starts at slot 1). - this.displayMessages = [ - { role: 'summary', content: formatted }, - ...this.displayMessages - .slice(displayKeepFrom) - .map((m) => (m.role === 'user' ? { ...m, index: m.index - keepFrom + 1 } : m)) - ] - - // The provider report described the pre-compaction history; the new - // history is much smaller, so clear it and let readers re-estimate. - this.contextUsage = undefined + const result = await this.runSummarization( + prefix, + tail, + keepFrom, + displayKeepFrom, + abortController + ) + if (result === 'ok') { this.consecutiveCompactionFailures = 0 return true - } catch (err) { - // A user Stop aborts the in-flight summary — that's a turn cancel, not a - // compaction failure, so it doesn't count toward the circuit breaker. - if (!abortController.signal.aborted) { - console.error('Conversation summarization failed', err) - this.consecutiveCompactionFailures++ + } + // 'aborted' is a user Stop during the in-flight summary — a turn cancel, not + // a compaction failure, so it doesn't count toward the circuit breaker. + if (result === 'empty' || result === 'error') { + this.consecutiveCompactionFailures++ + } + return false + } + + /** + * Manual compaction (the `/compact` session command): summarize the ENTIRE + * stored history into a single summary message and keep nothing verbatim, so + * the next message continues from the summary alone. Unlike the automatic + * trigger it ignores the context-window budget, the circuit breaker, and the + * prefix-size gate — the user asked for it explicitly — and runs on its own + * abort controller so the Stop button (`cancel`) can interrupt the in-flight + * summary, leaving history untouched. + */ + compactManually = async (): Promise => { + if (this.loading) { + return + } + // A summary round-trip only pays off once there's a prior exchange to fold + // in; a single message (or none) has nothing to compact. + if (this.messages.length < 2) { + sendUserToast('Nothing to compact yet.') + return + } + + const abortController = new AbortController() + this.abortController = abortController + this.loading = true + let result: 'ok' | 'empty' | 'aborted' | 'error' = 'error' + try { + // Everything is the prefix, nothing is kept verbatim: keepFrom and + // displayKeepFrom point past the end so the kept tail is empty. + result = await this.runSummarization( + [...this.messages], + [], + this.messages.length, + this.displayMessages.length, + abortController + ) + switch (result) { + case 'ok': + await this.historyManager.saveChat( + this.displayMessages, + this.messages, + this.contextUsage + ) + sendUserToast('Conversation compacted.') + break + case 'empty': + sendUserToast( + 'Compaction produced an empty summary — conversation left unchanged.', + true + ) + break + case 'error': + sendUserToast('Failed to compact the conversation.', true) + break + // 'aborted' (user Stop): history untouched, no toast. } - return false } finally { - this.compacting = false + this.loading = false + } + + // Flush a message typed while compaction ran. Mirrors the send-turn + // epilogue (loading gated its capture): auto-send after a successful + // compaction or a deliberate user cancel — the user is ready to move on — + // while a failed/empty compaction or a programmatic cancel leaves it queued. + if ((result === 'ok' || this.wasCancelledByUser()) && this.queuedMessage) { + const next = this.queuedMessage + this.queuedMessage = '' + const accepted = await this.sendRequest({ instructions: next }) + if (accepted === false) { + this.queuedMessage = next + } } } @@ -760,7 +910,7 @@ export class AIChatManager { this.helpers = { getScriptOptions: () => { return { - code: this.scriptEditorOptions?.code ?? '', + code: this.scriptEditorOptions?.getCode() ?? '', lang: lang, path: this.scriptEditorOptions?.path ?? '', args: this.scriptEditorOptions?.args ?? {} @@ -808,15 +958,29 @@ export class AIChatManager { this.tools = [searchDocsTool, readDocsPageTool, ...this.apiTools] this.helpers = {} } else if (mode === AIMode.GLOBAL) { - const customPrompt = getCombinedCustomPrompt(mode) - this.systemMessage = prepareGlobalSystemMessage(customPrompt, { - previewTools: this.isSessionChat + this.systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(mode), { + previewTools: this.isSessionChat, + skills: this.globalSkills }) this.tools = globalToolsFor({ sessionPreview: this.isSessionChat }) this.helpers = { ...(this.isSessionChat ? { sessionId: this.sessionId } : {}), - testActiveFlow: async (args?: Record) => this.flowAiChatHelpers?.testFlow(args) + testActiveFlow: async (args?: Record) => + this.flowAiChatHelpers?.testFlow(args), + attachedFiles: this.attachedFiles, + getUserInstructions: () => getUserCustomPrompts()[AIMode.GLOBAL] ?? '', + setUserInstructions: (instructions: string) => { + const prompts = getUserCustomPrompts() + if (instructions.trim()) { + prompts[AIMode.GLOBAL] = instructions + } else { + delete prompts[AIMode.GLOBAL] + } + setUserCustomPrompts(prompts) + this.rebuildGlobalSystemMessage() + } } satisfies GlobalToolHelpers + void this.refreshGlobalSkills() } else if (mode === AIMode.APP) { const customPrompt = getCombinedCustomPrompt(mode) this.systemMessage = prepareAppSystemMessage(customPrompt) @@ -825,6 +989,53 @@ export class AIChatManager { } } + // Fetch the workspace's AI skills and, if GLOBAL mode is still active, rebuild + // the system message so the next chat-loop iteration advertises them. Ignore + // stale resolves so workspace changes cannot overwrite newer skills. + refreshGlobalSkills = async (workspace = get(workspaceStore) ?? '') => { + const refreshId = ++this.globalSkillsRefreshId + const skills = await loadWorkspaceSkills(workspace) + if (refreshId !== this.globalSkillsRefreshId) { + return + } + this.globalSkills = skills + if (this.mode === AIMode.GLOBAL) { + this.systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { + previewTools: this.isSessionChat, + skills + }) + } + } + + // Rebuild the GLOBAL system message in place so an updated user instruction (persisted by + // the update_user_instructions tool) is picked up on the next chat-loop iteration, which + // re-reads this.systemMessage via a getter. + rebuildGlobalSystemMessage = () => { + if (this.mode !== AIMode.GLOBAL) { + return + } + this.systemMessage = prepareGlobalSystemMessage(getCustomPromptParts(AIMode.GLOBAL), { + previewTools: this.isSessionChat, + skills: this.globalSkills + }) + } + + private expandGlobalSkillCommand = (instructions: string): string => { + if (!this.isSessionChat || this.mode !== AIMode.GLOBAL || !instructions.startsWith('/')) { + return instructions + } + const match = /^\/([a-z0-9-]+)(?:\s+([\s\S]*))?$/.exec(instructions) + if (!match) { + return instructions + } + const skill = this.globalSkills.find((s) => s.name === match[1]) + if (!skill) { + return instructions + } + const rest = match[2]?.trim() + return rest ? `Use the "${skill.name}" skill. ${rest}` : `Use the "${skill.name}" skill.` + } + canApplyCode = $derived(this.allowedModes.script && this.mode === AIMode.SCRIPT) private changeModeTool = { @@ -1015,7 +1226,13 @@ export class AIChatManager { messages, addedMessages, get systemMessage() { - return systemMessageOverride ?? self.systemMessage + const base = systemMessageOverride ?? self.systemMessage + // Inject the attached-files roster at request time (re-read each iteration) + // so it always reflects the live file list without reactive bookkeeping. + if (self.mode === AIMode.GLOBAL && self.attachedFiles.count > 0) { + return appendAttachedFilesRoster(base, self.attachedFiles) + } + return base }, get tools() { return self.tools @@ -1189,9 +1406,11 @@ export class AIChatManager { isPreprocessor?: boolean } = {} ) => { - // Returns whether the message was actually turned into a chat turn — - // the queue flush uses this to restore messages dropped by an early - // return instead of silently losing them. + // Returns whether the input was consumed: true when it was sent as a chat + // turn OR handled as a local built-in command, false when it was dropped + // without being acted on (mode hidden, empty, beforeSend failed). The + // queue flush restores the queued message only on false, so a consumed + // command isn't re-queued and re-fired into the next conversation. const requestedMode = options.mode ?? this.mode if (!isAIModeVisible(requestedMode)) { return false @@ -1206,6 +1425,40 @@ export class AIChatManager { if (!this.instructions.trim()) { return false } + // Built-in session commands run locally instead of becoming a chat turn. + // Intercepted here — before the beforeSend workspace commit, file regrants, + // and skill expansion. Scoped to session chat GLOBAL mode, where the + // slash-command UI lives. Return true (consumed, not dropped) so that a + // command flushed from the queue isn't restored and re-fired into the next + // conversation. + if (this.isSessionChat && this.mode === AIMode.GLOBAL) { + const trimmed = this.instructions.trim() + // `/compact`: summarize the conversation locally to free up context. + if (COMPACT_COMMAND_RE.test(trimmed)) { + this.instructions = '' + await this.compactManually() + return true + } + // `/clear`: save the conversation to history and start a fresh chat. + if (CLEAR_COMMAND_RE.test(trimmed)) { + this.instructions = '' + await this.saveAndClear() + return true + } + } + // Re-grant any locked File System Access handles within this send gesture, so the + // file tools can read the live files. requestPermission() needs a user gesture, and + // this runs before the first await/network call while the Send click is still active. + // Attachment upkeep must never block the send — affected files just stay locked/stale + // and the tools report their status to the model. + try { + await this.attachedFiles.regrantLocked() + // Re-enumerate linked folders so on-disk changes (renamed/added/removed/edited + // files) are reflected in the roster + indexes before this turn runs. + await this.attachedFiles.refreshFolders() + } catch (e) { + console.error('Attached-files upkeep failed before send', e) + } if (this.beforeSend) { try { await this.beforeSend() @@ -1224,6 +1477,11 @@ export class AIChatManager { return false } } + // Session chats commit their workspace in beforeSend; skills must match the + // committed workspace before the system prompt is sent. + if (this.mode === AIMode.GLOBAL) { + await this.refreshGlobalSkills(get(workspaceStore) ?? '') + } const isFirstUserTurn = !this.displayMessages.some((message) => message.role === 'user') // Declared outside `try` so the catch can recover what the loop produced // before a failure: the structured messages and the latest streamed text @@ -1297,6 +1555,10 @@ export class AIChatManager { // The LLM gets the full pasted content; the display message above keeps // the compact tokens + registry so the bubble can render/expand chips. const oldInstructions = expanded(chatDraft(this.instructions, pastes)) + const modelInstructions = + this.mode === AIMode.GLOBAL + ? this.expandGlobalSkillCommand(oldInstructions) + : oldInstructions this.instructions = '' if (this.mode === AIMode.SCRIPT && !this.scriptEditorOptions && !options.lang) { @@ -1329,7 +1591,7 @@ export class AIChatManager { userMessage = prepareApiUserMessage(oldInstructions) break case AIMode.GLOBAL: - userMessage = prepareGlobalUserMessage(oldInstructions, oldSelectedContext, { + userMessage = prepareGlobalUserMessage(modelInstructions, oldSelectedContext, { workspace: get(workspaceStore) }) break @@ -1714,6 +1976,11 @@ export class AIChatManager { this.displayMessages = [] this.messages = [] this.contextUsage = undefined + // In an AI session, linked files are session-scoped: they persist across conversations + // (cleared only when the session is deleted). The ephemeral global side-panel chat has no + // session, so "New chat" must clear them — otherwise the next, unrelated conversation + // would still get the previous file roster and could read/search it. + if (!this.isSessionChat) this.attachedFiles.clear() } loadPastChat = async (id: string) => { @@ -1722,6 +1989,9 @@ export class AIChatManager { // Drop any message queued in the current conversation so it doesn't // auto-send into the loaded one or linger as a card across the switch. this.queuedMessage = '' + // Same isolation as saveAndClear: the ephemeral global chat's attachments belong to + // the conversation being left, not the one being loaded; sessions keep them. + if (!this.isSessionChat) this.attachedFiles.clear() this.displayMessages = chat.displayMessages this.messages = chat.actualMessages this.contextUsage = normalizeContextUsage(chat.contextUsage) @@ -1856,14 +2126,13 @@ export class AIChatManager { lastDeployedCode: undefined, lastSavedCode: undefined } - return { args: moduleState?.previewArgs ?? {}, error: moduleState && !moduleState.previewSuccess ? getStringError(moduleState.previewResult) : undefined, - code: module.value.content, + getCode: () => module.value.type === 'rawscript' ? module.value.content : '', lang: module.value.language, path: module.id, ...editorRelated diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts index 16a08d7344..d23cbf96a1 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.test.ts @@ -15,7 +15,9 @@ const mocks = vi.hoisted(() => ({ getOpenaiClient: vi.fn(), getAnthropicClient: vi.fn(), getNonStreamingCompletion: vi.fn(), - runChatLoop: vi.fn() + runChatLoop: vi.fn(), + listAiSkills: vi.fn(), + workspace: 'test_workspace' as string | undefined })) vi.mock('monaco-editor', () => ({ @@ -24,7 +26,8 @@ vi.mock('monaco-editor', () => ({ vi.mock('$lib/gen', () => ({ WorkspaceService: { - logAiChat: mocks.logAiChat + logAiChat: mocks.logAiChat, + listAiSkills: mocks.listAiSkills }, ScriptService: {}, FlowService: {}, @@ -36,7 +39,12 @@ vi.mock('$lib/gen', () => ({ const TEST_EMAIL = 'admin@test' vi.mock('$lib/stores', () => ({ - workspaceStore: { subscribe: () => () => undefined }, + workspaceStore: { + subscribe: (run: (value: string | undefined) => void) => { + run(mocks.workspace) + return () => undefined + } + }, userStore: { subscribe: (run: (value: { username: string; email: string }) => void) => { run({ username: 'admin', email: 'admin@test' }) @@ -53,6 +61,9 @@ vi.mock('$lib/aiStore', () => ({ getCurrentModel: mocks.getCurrentModel, tryGetCurrentModel: mocks.tryGetCurrentModel, getCombinedCustomPrompt: () => '', + getCustomPromptParts: () => ({}), + getUserCustomPrompts: () => ({}), + setUserCustomPrompts: () => {}, isWebSearchEnabledForProvider: mocks.isWebSearchEnabledForProvider })) @@ -95,6 +106,8 @@ beforeEach(() => { mocks.logAiChat.mockResolvedValue(undefined) mocks.getOpenaiClient.mockReturnValue({}) mocks.getAnthropicClient.mockReturnValue({}) + mocks.listAiSkills.mockResolvedValue([]) + mocks.workspace = 'test_workspace' mocks.runChatLoop.mockResolvedValue({ addedMessages: [], tokenUsage: { prompt: 0, completion: 0, total: 0 }, @@ -185,6 +198,82 @@ describe('AIChatManager request errors', () => { }) }) +describe('AIChatManager global skills', () => { + const model = { provider: 'openai', model: 'gpt-4o' } + + beforeEach(() => { + localStorage.clear() + mocks.getCurrentModel.mockReturnValue(model) + mocks.tryGetCurrentModel.mockReturnValue(model) + }) + + it('loads skills after beforeSend commits the session workspace', async () => { + let resolveParentSkills: ((skills: { name: string; description: string }[]) => void) | undefined + const parentSkills = new Promise<{ name: string; description: string }[]>((resolve) => { + resolveParentSkills = resolve + }) + mocks.workspace = 'parent' + mocks.listAiSkills.mockImplementation(({ workspace }: { workspace: string }) => { + if (workspace === 'parent') { + return parentSkills + } + return Promise.resolve([{ name: 'child-skill', description: 'child workspace skill' }]) + }) + mocks.runChatLoop.mockImplementation(async (config: any) => { + expect(config.workspace).toBe('child') + expect(config.systemMessage.content).toContain('child-skill') + expect(config.systemMessage.content).not.toContain('parent-skill') + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + + const manager = new AIChatManager() + manager.isSessionChat = true + manager.beforeSend = () => { + mocks.workspace = 'child' + } + + await manager.sendRequest({ instructions: 'first', mode: AIMode.GLOBAL }) + resolveParentSkills?.([{ name: 'parent-skill', description: 'parent workspace skill' }]) + await Promise.resolve() + + expect(mocks.listAiSkills).toHaveBeenCalledWith({ workspace: 'parent' }) + expect(mocks.listAiSkills).toHaveBeenCalledWith({ workspace: 'child' }) + expect(manager.systemMessage.content).toContain('child-skill') + expect(manager.systemMessage.content).not.toContain('parent-skill') + }) + + it('expands a leading slash skill command for the model while preserving the displayed text', async () => { + mocks.listAiSkills.mockResolvedValue([ + { name: 'review-code', description: 'review code for bugs' } + ]) + mocks.runChatLoop.mockImplementation(async (config: any) => { + const userMessage = config.messages[config.messages.length - 1] + expect(userMessage.content).toContain('Use the "review-code" skill. find bugs') + expect(userMessage.content).not.toContain('/review-code find bugs') + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + + const manager = new AIChatManager() + manager.isSessionChat = true + + await manager.sendRequest({ instructions: '/review-code find bugs', mode: AIMode.GLOBAL }) + + expect(manager.displayMessages[0]?.content).toBe('/review-code find bugs') + }) +}) + describe('AIChatManager autonomy mode', () => { beforeEach(() => { localStorage.clear() @@ -514,6 +603,37 @@ describe('AIChatManager queued messages', () => { await manager.loadPastChat('chat-b') expect(manager.queuedMessage).toBe('') }) + + it('clears attachments on New chat / load past chat (non-session), keeps them in a session', async () => { + const txt = (n: string) => new File(['hello\n'], n, { type: 'text/plain' }) + + // Non-session global chat: New chat must clear the previous conversation's attachments. + const manager = createManager(createInputMock()) + await manager.attachedFiles.addFiles([txt('a.txt')]) + expect(manager.attachedFiles.count).toBe(1) + await manager.saveAndClear() + expect(manager.attachedFiles.count).toBe(0) + + // ...and loading a past chat clears them too. + vi.spyOn(manager.historyManager, 'loadPastChat').mockReturnValue({ + id: 'chat-c', + title: 'Chat C', + displayMessages: [], + actualMessages: [], + lastModified: 0 + } as unknown as ReturnType) + await manager.attachedFiles.addFiles([txt('c.txt')]) + expect(manager.attachedFiles.count).toBe(1) + await manager.loadPastChat('chat-c') + expect(manager.attachedFiles.count).toBe(0) + + // Session chat: attachments are session-scoped — they survive New chat. + const session = createManager(createInputMock()) + session.isSessionChat = true + await session.attachedFiles.addFiles([txt('b.txt')]) + await session.saveAndClear() + expect(session.attachedFiles.count).toBe(1) + }) }) describe('AIChatManager context compaction', () => { @@ -823,7 +943,7 @@ describe('AIChatManager context compaction', () => { // The request that went out begins with the summary user message, then the // recent tail verbatim, then the new question. - const sent = mocks.runChatLoop.mock.calls[0][0].messages + const sent = mocks.runChatLoop.mock.calls[mocks.runChatLoop.mock.calls.length - 1][0].messages expect(sent).toHaveLength(4) expect(sent[0].role).toBe('user') expect(sent[0].content).toContain('SUMMARY TEXT') @@ -885,12 +1005,10 @@ describe('AIChatManager context compaction', () => { mocks.tryGetCurrentModel.mockReturnValue(gpt4oModel) // The user hits Stop while the summary request is in flight: it aborts the // turn's controller and rejects. - mocks.getNonStreamingCompletion.mockImplementation( - async (_msgs: any, ac: AbortController) => { - ac.abort('user_cancelled') - throw new Error('aborted') - } - ) + mocks.getNonStreamingCompletion.mockImplementation(async (_msgs: any, ac: AbortController) => { + ac.abort('user_cancelled') + throw new Error('aborted') + }) // With the controller already aborted, the real request returns nothing; // mirror that so the turn takes the cancel/rollback path. mocks.runChatLoop.mockImplementation(async () => ({ @@ -914,6 +1032,244 @@ describe('AIChatManager context compaction', () => { }) }) +describe('AIChatManager manual compaction', () => { + const model = { provider: 'openai', model: 'gpt-4o' } + + beforeEach(() => { + localStorage.clear() + vi.clearAllMocks() + mocks.getCurrentModel.mockReturnValue(model) + mocks.tryGetCurrentModel.mockReturnValue(model) + // changeMode(GLOBAL) refreshes workspace skills; keep it a no-op here. + mocks.listAiSkills.mockResolvedValue([]) + }) + + function seedExchange(manager: AIChatManager) { + manager.messages = [ + { role: 'user', content: 'q1' }, + { role: 'assistant', content: 'a1' }, + { role: 'user', content: 'q2' }, + { role: 'assistant', content: 'a2' } + ] + manager.displayMessages = [ + { role: 'user', content: 'q1', index: 0 }, + { role: 'assistant', content: 'a1' }, + { role: 'user', content: 'q2', index: 2 }, + { role: 'assistant', content: 'a2' } + ] + } + + it('folds the whole history into a single summary boundary, keeping nothing verbatim', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('MANUAL SUMMARY') + const manager = new AIChatManager() + seedExchange(manager) + manager.contextUsage = 123 + const saveChat = vi.spyOn(manager.historyManager, 'saveChat') + + await manager.compactManually() + + // The summarizer saw the entire history, then the summary instruction. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + const summaryReq = mocks.getNonStreamingCompletion.mock.calls[0][0] + expect(summaryReq).toHaveLength(5) + expect(summaryReq[0].content).toBe('q1') + expect(summaryReq[3].content).toBe('a2') + expect(summaryReq[4].content).toContain('detailed summary') + + // Nothing kept verbatim: messages collapse to just the summary user message. + expect(manager.messages).toHaveLength(1) + expect(manager.messages[0].role).toBe('user') + expect(manager.messages[0].content).toContain('MANUAL SUMMARY') + expect(manager.messages[0].content).toContain('continued from a previous conversation') + expect(manager.messages[0].content).not.toContain('') + + // The transcript shows one summary boundary in place of the old bubbles. + expect(manager.displayMessages).toHaveLength(1) + expect(manager.displayMessages[0]).toMatchObject({ role: 'summary', content: 'MANUAL SUMMARY' }) + + expect(manager.contextUsage).toBeUndefined() + expect(saveChat).toHaveBeenCalled() + expect(mocks.sendUserToast).toHaveBeenCalledWith('Conversation compacted.') + expect(manager.loading).toBe(false) + expect(manager.compacting).toBe(false) + }) + + it('no-ops with a toast when there is nothing worth compacting', async () => { + const manager = new AIChatManager() + manager.messages = [{ role: 'user', content: 'only one' }] + + await manager.compactManually() + + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + expect(mocks.sendUserToast).toHaveBeenCalledWith('Nothing to compact yet.') + expect(manager.messages).toHaveLength(1) + }) + + it('leaves history untouched when the user stops mid-summary', async () => { + mocks.getNonStreamingCompletion.mockImplementation(async (_msgs: any, ac: AbortController) => { + ac.abort('user_cancelled') + throw new Error('aborted') + }) + const manager = new AIChatManager() + seedExchange(manager) + + await manager.compactManually() + + expect(manager.messages).toHaveLength(4) + expect(manager.displayMessages.some((m) => m.role === 'summary')).toBe(false) + // An abort is a user cancel, not a failure — no toast, no destructive change. + expect(mocks.sendUserToast).not.toHaveBeenCalled() + expect(manager.loading).toBe(false) + }) + + it('routes the /compact session command to manual compaction instead of the model', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('VIA COMMAND') + const manager = new AIChatManager() + manager.isSessionChat = true + seedExchange(manager) + + const sent = await manager.sendRequest({ instructions: '/compact', mode: AIMode.GLOBAL }) + + // Consumed as a local command (true so the queue flush won't re-fire it), + // without ever reaching the model loop... + expect(sent).toBe(true) + expect(mocks.runChatLoop).not.toHaveBeenCalled() + // ...it ran the summarizer and compacted in place, clearing the composer. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + expect(manager.displayMessages[0]).toMatchObject({ role: 'summary', content: 'VIA COMMAND' }) + expect(manager.instructions).toBe('') + }) + + it('auto-sends a message queued while compaction was running', async () => { + mocks.getNonStreamingCompletion.mockResolvedValue('S') + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = true + manager.changeMode(AIMode.GLOBAL) + seedExchange(manager) + // A message typed while loading was true gets queued, not sent. + manager.queuedMessage = 'follow-up question' + + await manager.compactManually() + + // Compaction ran once, then the queued message went out as a real turn. + expect(mocks.getNonStreamingCompletion).toHaveBeenCalledTimes(1) + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + const sent = mocks.runChatLoop.mock.calls[0][0].messages + expect(sent[sent.length - 1].content).toContain('follow-up question') + expect(manager.queuedMessage).toBe('') + }) + + it('routes the /clear session command to a fresh chat instead of the model', async () => { + const manager = new AIChatManager() + manager.isSessionChat = true + seedExchange(manager) + + const sent = await manager.sendRequest({ instructions: '/clear', mode: AIMode.GLOBAL }) + + // Consumed as a local command (true so the queue flush won't re-fire it), + // without ever reaching the model... + expect(sent).toBe(true) + expect(mocks.runChatLoop).not.toHaveBeenCalled() + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + // ...it reset the conversation and cleared the composer. + expect(manager.displayMessages).toEqual([]) + expect(manager.messages).toEqual([]) + expect(manager.instructions).toBe('') + }) + + it('consumes a /clear flushed from the queue without re-queuing it', async () => { + // A normal turn that commits cleanly, so its epilogue flushes the queue. + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = true + manager.changeMode(AIMode.GLOBAL) + seedExchange(manager) + // `/clear` typed while the turn was streaming gets queued, not sent. + manager.queuedMessage = '/clear' + + await manager.sendRequest({ instructions: 'a normal message', mode: AIMode.GLOBAL }) + + // The committed turn's flush ran `/clear` (resetting the conversation) and, + // because the command reports itself as consumed, did NOT restore it — so a + // stale `/clear` can't re-fire and wipe the next conversation. + expect(manager.queuedMessage).toBe('') + expect(manager.displayMessages).toEqual([]) + expect(manager.messages).toEqual([]) + }) + + it('does not intercept /clear outside session chat', async () => { + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = false + + await manager.sendRequest({ instructions: '/clear', mode: AIMode.GLOBAL }) + + // Without the session-chat command surface, /clear is a normal message. + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + }) + + it('does not intercept /compact outside session chat', async () => { + mocks.runChatLoop.mockImplementation(async (config: any) => { + const message = { role: 'assistant' as const, content: 'done' } + config.addedMessages?.push(message) + return { + addedMessages: [message], + tokenUsage: { prompt: 0, completion: 0, total: 0 }, + hitMaxIterations: false + } + }) + const manager = new AIChatManager() + manager.isSessionChat = false + + await manager.sendRequest({ instructions: '/compact', mode: AIMode.GLOBAL }) + + // Without the session-chat command surface, /compact is a normal message. + expect(mocks.runChatLoop).toHaveBeenCalledTimes(1) + expect(mocks.getNonStreamingCompletion).not.toHaveBeenCalled() + }) + + it('shadows a workspace skill that collides with a built-in command', () => { + const manager = new AIChatManager() + manager.globalSkills = [ + { name: 'compact', description: 'a workspace skill that happens to be named compact' }, + { name: 'review-code', description: 'review code for bugs' } + ] + + // Built-ins come first and the colliding skill is dropped, so the picker + // never renders two leaves with the same `skill:compact` key. + const names = manager.sessionCommands.map((c) => c.name) + expect(names).toEqual(['compact', 'clear', 'review-code']) + expect(manager.sessionCommands[0].description).toBe( + 'Summarize the conversation to free up context' + ) + }) +}) + const assistantToolCall = (id: string): ChatCompletionMessageParam => ({ role: 'assistant', content: '', diff --git a/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte new file mode 100644 index 0000000000..0a911134b6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte @@ -0,0 +1,456 @@ + + +{#snippet externalLinkIcon()} + +{/snippet} + + + {#snippet buttonReplacement()} +
    + +
    + {/snippet} + {#snippet menu({ item, builders, close })} +
    + + + +
    +
    Model
    +
    + {#each models as m (m.provider + m.model)} + selectModel(m)} + > + {m.model} + {#if m.model === providerModel.model && m.provider === providerModel.provider} + + {/if} + + {/each} +
    + +
    + {#if capability.supported} + + +
    + Thinking + {currentStop} +
    + {#if stops.length > 1} + +
    + selectReasoning(stops[+e.currentTarget.value])} + use:isolatePointer + class="lean-range no-default-style w-full" + aria-label="Reasoning effort" + /> +
    + {/if} +     + {:else} + +
    +
    Thinking
    +
    Not supported by this model
    +
    + {/if} +
    + {/snippet} +     + + + + diff --git a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte b/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte deleted file mode 100644 index 683bf3c586..0000000000 --- a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte +++ /dev/null @@ -1,186 +0,0 @@ - - -{#snippet externalLinkIcon()} - -{/snippet} - - - {#snippet buttonReplacement()} - + {file.name} +
    diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte new file mode 100644 index 0000000000..50b961c090 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte @@ -0,0 +1,86 @@ + + +{#snippet chip(card: Card)} + {#if card.kind === 'folder'} + removeCard(card)} /> + {:else} + removeCard(card)} /> + {/if} +{/snippet} + +{#if cards.length > 0} +
    + {#each visible as card (card.key)} + {@render chip(card)} + {/each} + + {#if overflow.length > 0} + + {#snippet trigger()} +
    + +{overflow.length} +
    + {/snippet} + {#snippet content()} +
    + {#each overflow as card (card.key)} + {@render chip(card)} + {/each} +
    + {/snippet} +     + {/if} + + {#if lockedCount > 0} + + {/if} +
    +{/if} diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte new file mode 100644 index 0000000000..068447a878 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte @@ -0,0 +1,48 @@ + + +
    (showDelete = true)} + onmouseleave={() => (showDelete = false)} + role="listitem" + title={hoverList} +> + + {folder.name} +
    diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts new file mode 100644 index 0000000000..124b390ec2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts @@ -0,0 +1,602 @@ +/** + * Session-scoped store of files/folders the user has linked to the GLOBAL AI chat. + * + * Persistence model (survives reload, keyed by session in ./attachedFilesDB): + * - FILES are always stored as a full-byte Blob snapshot — same on every browser, + * no permission re-grant, never "locked". + * - FOLDERS link as a live File System Access directory handle where the API exists + * (one record, re-enumerated live on restore — folder files are read through the + * handle, not copied). Where it doesn't (Firefox/Safari), a dropped/picked folder's + * files are snapshotted individually (each carrying its `folder`/`relPath`) so they + * regroup into the same folder chip on restore. + * + * Storage is bounded by the real browser quota (writes that exceed it are caught and + * the item simply isn't persisted — it stays usable for the session). Persistence is + * gated on the session being persisted (non-transient); links in a transient session + * are buffered and flushed on the first send. + */ +import { createLongHash } from '$lib/editorLangUtils' +import { buildLineIndex, isTextFile, type FileEntry } from './fileEngine' +import { + putItem, + deleteItem, + getItemsForSession, + ensurePersistentStorage, + type PersistedAttachedItem +} from './attachedFilesDB' +import { enumerateDir, isIgnoredPath, queryReadPermission, requestReadPermission } from './fsAccess' + +export type AttachedFileStatus = 'indexing' | 'ready' | 'error' | 'locked' | 'unavailable' + +export interface AttachedFile extends FileEntry { + size: number + status: AttachedFileStatus + error?: string + /** Top-level folder this file came from (first path segment), if part of a folder. */ + folder?: string + /** Persisted source-record id. Folder children share the folder's record id. */ + sourceId: string + /** Parent directory handle (folder children only) — used to re-grant / re-enumerate. */ + handle?: FileSystemDirectoryHandle + /** Relative path within the folder (folder children only) — stable key for refresh diffing. */ + relPath?: string + /** + * Internal: a single placeholder row standing in for a not-yet-expanded folder + * (locked/unavailable). Consumers should read `store.folders` instead of testing this. + */ + isFolderRoot?: boolean +} + +/** A linked folder as a first-class object — consumers read this instead of re-grouping rows. */ +export interface AttachedFolder { + name: string + /** Aggregate status (locked > unavailable > indexing > error > ready). */ + status: AttachedFileStatus + /** Child files; empty while the folder is locked/unavailable after a reload. */ + files: AttachedFile[] +} + +/** Aggregate a folder's rows (children + a possible placeholder) into one status. */ +function folderStatus(rows: AttachedFile[]): AttachedFileStatus { + for (const status of ['locked', 'unavailable', 'indexing', 'error'] as const) { + if (rows.some((f) => f.status === status)) return status + } + return 'ready' +} + +export interface AddFilesResult { + added: string[] + rejected: { name: string; reason: string }[] +} + +/** A file to link: a raw File, or `{ file, path? }` (path = relative display name). */ +export type FileToAttach = File | { file: File; path?: string } + +const EMPTY = new Blob([]) + +export class AttachedFilesStore { + files = $state([]) + + /** Session context, set by the runtime; persistence writes are gated on `#persisted`. */ + sessionId: string | undefined = undefined + #persisted = false + /** Records buffered while the session is transient (flushed on first send). */ + #pending: PersistedAttachedItem[] = [] + + list(): AttachedFile[] { + return this.files + } + get(name: string): AttachedFile | undefined { + // Resolve to a real file — a folder-root placeholder may share the folder's name. + return this.files.find((f) => f.name === name && !f.isFolderRoot) + } + readyFiles(): AttachedFile[] { + // Folder-root placeholders aren't real files — never expose them to the read/search tools. + return this.files.filter((f) => f.status === 'ready' && !f.isFolderRoot) + } + get count(): number { + return this.files.length + } + + /** Linked folders, children grouped and status aggregated (placeholder rows hidden). */ + folders: AttachedFolder[] = $derived.by(() => { + const byName = new Map() + for (const f of this.files) { + if (!f.folder) continue + const rows = byName.get(f.folder) + if (rows) rows.push(f) + else byName.set(f.folder, [f]) + } + return [...byName.entries()].map(([name, rows]) => ({ + name, + status: folderStatus(rows), + files: rows.filter((f) => !f.isFolderRoot) + })) + }) + + /** Files linked on their own (not as part of a folder). */ + standalone: AttachedFile[] = $derived.by(() => this.files.filter((f) => !f.folder)) + + /** Number of locked folders needing a re-grant. */ + get lockedCount(): number { + return this.folders.filter((f) => f.status === 'locked').length + } + + clear(): void { + this.files = [] + this.#pending = [] + } + + removeFile(name: string): void { + // Target the real file only — never a folder-root placeholder that happens to share + // the name (those are managed via removeFolder), else removing a same-named standalone + // file would also drop the folder's placeholder. + const f = this.files.find((x) => x.name === name && !x.isFolderRoot) + if (!f) return + this.files = this.files.filter((x) => !(x.name === name && !x.isFolderRoot)) + void this.#deleteRecord(f.sourceId) + } + + /** Remove every file linked as part of the given folder (and its persisted record). */ + removeFolder(folder: string): void { + const ids = new Set(this.files.filter((f) => f.folder === folder).map((f) => f.sourceId)) + this.files = this.files.filter((f) => f.folder !== folder) + for (const id of ids) void this.#deleteRecord(id) + } + + // ---------------------------------------------------------------- linking + + /** + * Link individual files — always stored as a Blob snapshot. Items carrying a folder + * path (`folder/sub/file`, from a dropped/picked folder) are grouped into a folder and + * have their junk paths (node_modules/.git/dotfiles) skipped; a loose single file is + * kept as-is (so an explicitly attached `.env` isn't filtered out). + */ + async addFiles(input: FileList | FileToAttach[]): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + + for (const item of Array.from(input as ArrayLike)) { + const file = item instanceof File ? item : item.file + const desired = + (item instanceof File ? '' : (item.path ?? '')) || + (file as File & { webkitRelativePath?: string }).webkitRelativePath || + file.name || + 'file' + + const folder = desired.includes('/') ? desired.split('/')[0] : undefined + if (folder && isIgnoredPath(desired)) continue // skip junk inside folders + + if (this.#isDuplicate(desired, file)) continue // silent no-op on re-link + + const reason = await this.#preflight(file) + if (reason) { + result.rejected.push({ name: desired, reason }) + continue + } + + const name = this.#uniqueName(desired) + const relPath = folder ? desired : undefined + const sourceId = createLongHash() + + this.#pushIndexing({ name, file, folder, sourceId, relPath }) + result.added.push(name) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'snapshot', + name, + folder, + relPath, + blob: file, + size: file.size, + lastModified: file.lastModified, + addedAt: Date.now() + }) + } + + return result + } + + /** + * Link a folder via a live directory handle (File System Access path only). + * Enumerates the handle internally (junk-filtered, capped) — the same walk used + * on restore and refresh, so callers never pre-enumerate. + */ + async addFolder(dirHandle: FileSystemDirectoryHandle): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + const folder = dirHandle.name + const existing = this.files.filter((f) => f.folder === folder) + if (existing.length > 0) { + const placeholder = existing.length === 1 ? existing.find((f) => f.isFolderRoot) : undefined + if (placeholder) { + // Re-picking a folder that sits locked/unavailable after a reload is a natural + // recovery gesture — replace the stale link with the freshly-granted handle. + this.files = this.files.filter((f) => f.sourceId !== placeholder.sourceId) + void this.#deleteRecord(placeholder.sourceId) + } else { + // Same basename, possibly a different directory — surface it instead of a silent no-op. + result.rejected.push({ name: folder, reason: 'A folder with this name is already linked' }) + return result + } + } + + const files = await enumerateDir(dirHandle) + const sourceId = createLongHash() + for (const { file, path } of files) { + if (!(await this.#sniffText(file))) { + result.rejected.push({ name: path, reason: 'Not a text file' }) + continue + } + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + result.added.push(name) + } + // Keep the folder represented even when it links empty (or all-binary): a placeholder + // carries the handle so the chip stays and refreshFolders picks up files added later. + // Persist unconditionally so an empty-at-link folder also survives a reload. + this.#ensureFolderRow(sourceId, folder, dirHandle) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'dir-handle', + name: folder, + folder, + handle: dirHandle, + addedAt: Date.now() + }) + return result + } + + // ------------------------------------------------------------- persistence + + /** Set session context and load any persisted items for it (called on activation). */ + async restore(sessionId: string, persisted: boolean): Promise { + this.sessionId = sessionId + this.#persisted = persisted + this.files = [] + this.#pending = [] + + const items = await getItemsForSession(sessionId) + for (const item of items) { + try { + if (item.kind === 'snapshot') { + if (!item.blob) { + this.#pushPlaceholder(item, 'unavailable') + continue + } + this.#pushIndexing({ + name: item.name, + file: item.blob, + folder: item.folder, + relPath: item.relPath, + sourceId: item.id + }) + } else { + // dir-handle (folder) + const handle = item.handle as FileSystemDirectoryHandle + if ((await queryReadPermission(handle)) === 'granted') { + await this.#expandFolder(handle, item.id) + } else { + this.#pushPlaceholder(item, 'locked', true) + } + } + } catch { + this.#pushPlaceholder(item, 'unavailable', item.kind === 'dir-handle') + } + } + } + + /** Re-grant any locked folder handles. MUST be called within a user gesture (e.g. on send). */ + async regrantLocked(): Promise { + const sources = new Map() + for (const f of this.files) { + if (f.status === 'locked' && f.handle) sources.set(f.sourceId, f) + } + if (sources.size === 0) return + + // Kick off all permission requests within the gesture, then process. A rejected + // request (requestReadPermission never rejects, but stay defensive) counts as denied. + const decided = await Promise.all( + [...sources.values()].map((f) => + requestReadPermission(f.handle!).then( + (perm) => ({ f, perm }), + () => ({ f, perm: 'denied' as PermissionState }) + ) + ) + ) + for (const { f, perm } of decided) { + if (perm !== 'granted') continue + try { + await this.#expandFolder(f.handle as FileSystemDirectoryHandle, f.sourceId) + // Children are in — drop the locked placeholder row, then restore a ready + // placeholder if the folder came back empty/all-binary (else dropping the only + // handle-bearing row would unlink the folder and stop it ever refreshing). + this.files = this.files.filter((x) => !(x.sourceId === f.sourceId && x.isFolderRoot)) + this.#ensureFolderRow(f.sourceId, f.folder ?? f.name, f.handle as FileSystemDirectoryHandle) + } catch { + // Enumeration failed (folder moved/deleted on disk): drop any partially-added + // children and keep the placeholder so the chip shows "unavailable". + this.files = this.files.filter((x) => x.sourceId !== f.sourceId || x.isFolderRoot) + this.#patchSource(f.sourceId, { status: 'unavailable' }) + } + } + } + + /** Flush buffered links once the session becomes persistent (first send). */ + async flushPending(): Promise { + this.#persisted = true + if (!this.sessionId) return + const pending = this.#pending + this.#pending = [] + if (pending.length === 0) return + void ensurePersistentStorage() + for (const item of pending) { + try { + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file', e) + } + } + } + + async #persist(item: PersistedAttachedItem): Promise { + if (this.#persisted && this.sessionId) { + void ensurePersistentStorage() + try { + // A QuotaExceededError just means it won't survive a reload — the item + // stays usable for this session. Swallow + log rather than fail the link. + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file (kept for this session)', e) + } + } else { + this.#pending.push(item) + } + } + + async #deleteRecord(sourceId: string): Promise { + this.#pending = this.#pending.filter((p) => p.id !== sourceId) + if (this.#persisted) { + try { + await deleteItem(sourceId) + } catch { + /* ignore */ + } + } + } + + // ------------------------------------------------------------- internals + + /** Identical re-link (same name, or same File identity) → silent no-op. */ + #isDuplicate(desired: string, file: File): boolean { + return this.files.some( + (f) => + // Folder-root placeholders aren't real files — they must not block attaching a + // standalone file that happens to share the folder's name. + !f.isFolderRoot && + (f.name === desired || + // Identical re-drop at the SAME relative path (its row name may have been + // auto-suffixed). Keyed on the path, NOT the basename — otherwise two distinct + // files sharing a basename under different folder subdirs (proj/a/index.ts vs + // proj/b/index.ts) would be wrongly deduped and silently dropped. + ((f.relPath ?? f.name) === desired && + f.size === file.size && + f.file instanceof File && + f.file.lastModified === file.lastModified)) + ) + } + + /** Returns a rejection reason, or undefined if the file may be linked. */ + async #preflight(file: File): Promise { + if (!(await this.#sniffText(file))) return 'Not a text file' + return undefined + } + + async #sniffText(file: Blob): Promise { + try { + return await isTextFile(file) + } catch { + return false + } + } + + #pushIndexing(p: { + name: string + file: File | Blob + folder?: string + sourceId: string + handle?: FileSystemDirectoryHandle + relPath?: string + }): void { + this.files = [ + ...this.files, + { + name: p.name, + file: p.file, + size: p.file.size, + lineIndex: [], + lineCount: 0, + status: 'indexing', + folder: p.folder, + sourceId: p.sourceId, + handle: p.handle, + relPath: p.relPath + } + ] + void this.#indexFile(p.name, p.file) + } + + #pushPlaceholder( + item: PersistedAttachedItem, + status: AttachedFileStatus, + isFolderRoot = false + ): void { + this.files = [ + ...this.files, + { + name: item.name, + file: EMPTY, + size: item.size ?? 0, + lineIndex: [], + lineCount: 0, + status, + folder: item.folder, + sourceId: item.id, + handle: item.handle as FileSystemDirectoryHandle | undefined, + isFolderRoot + } + ] + } + + async #expandFolder(dirHandle: FileSystemDirectoryHandle, sourceId: string): Promise { + const folder = dirHandle.name + const children = await enumerateDir(dirHandle) + for (const { file, path } of children) { + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + } + this.#ensureFolderRow(sourceId, folder, dirHandle) + } + + /** + * Re-enumerate granted folder handles to reflect on-disk changes since they were + * linked/last refreshed: added/removed/renamed files and content edits. Called on + * each send so the AI sees the folder's current state. Unchanged files are left as-is + * (diffed by relative path + lastModified); only changed files are re-indexed. + */ + async refreshFolders(): Promise { + const sources = new Map() + for (const f of this.files) { + // Include folder-root placeholders (an emptied folder keeps only its placeholder), + // else the source is lost and the folder never re-enumerates again. + if (f.folder && f.handle) { + sources.set(f.sourceId, { handle: f.handle, folder: f.folder }) + } + } + for (const [sourceId, { handle, folder }] of sources) { + try { + if ((await queryReadPermission(handle)) !== 'granted') continue + const children = await enumerateDir(handle) + await this.#reconcileFolder(sourceId, folder, handle, children) + } catch { + this.#patchSource(sourceId, { status: 'unavailable' }) + } + } + } + + async #reconcileFolder( + sourceId: string, + folder: string, + handle: FileSystemDirectoryHandle, + children: { file: File; path: string }[] + ): Promise { + const existing = new Map() + for (const f of this.files) if (f.sourceId === sourceId && f.relPath) existing.set(f.relPath, f) + const seen = new Set() + + for (const { file, path } of children) { + seen.add(path) + const cur = existing.get(path) + if (!cur) { + // newly added on disk + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle, relPath: path }) + } else { + const curMod = cur.file instanceof File ? cur.file.lastModified : undefined + if (file.size !== cur.size || file.lastModified !== curMod) { + // content changed → re-read + re-index + this.#patch(cur.name, { file, size: file.size, status: 'indexing' }) + void this.#indexFile(cur.name, file) + } + } + } + // removed/renamed-away on disk → drop from memory + const removed = [...existing.values()].filter((f) => f.relPath && !seen.has(f.relPath)) + if (removed.length > 0) { + const names = new Set(removed.map((f) => f.name)) + this.files = this.files.filter((f) => !names.has(f.name)) + } + this.#ensureFolderRow(sourceId, folder, handle) + } + + /** + * Keep a linked folder represented even with no readable children: leave one + * handle-carrying placeholder row so the chip stays visible AND `refreshFolders` + * keeps the live source (without it, an emptied folder vanishes and never + * re-enumerates). Drop the placeholder as soon as real children exist again. + */ + #ensureFolderRow(sourceId: string, folder: string, handle: FileSystemDirectoryHandle): void { + const hasChild = this.files.some((f) => f.sourceId === sourceId && !f.isFolderRoot) + const hasPlaceholder = this.files.some((f) => f.sourceId === sourceId && f.isFolderRoot) + if (!hasChild && !hasPlaceholder) { + this.files = [ + ...this.files, + { + name: folder, + file: EMPTY, + size: 0, + lineIndex: [], + lineCount: 0, + status: 'ready', + folder, + sourceId, + handle, + isFolderRoot: true + } + ] + } else if (hasChild && hasPlaceholder) { + this.files = this.files.filter((f) => !(f.sourceId === sourceId && f.isFolderRoot)) + } + } + + async #indexFile(name: string, file: File | Blob): Promise { + try { + const { lineIndex, lineCount } = await buildLineIndex(file) + this.#patchFile(name, file, { lineIndex, lineCount, status: 'ready' }) + } catch (e) { + this.#patchFile(name, file, { + status: 'error', + error: e instanceof Error ? e.message : String(e) + }) + } + } + + #patch(name: string, changes: Partial): void { + this.files = this.files.map((f) => (f.name === name ? { ...f, ...changes } : f)) + } + /** + * Patch the row for `name` ONLY while it still holds the exact `file` we indexed. + * `buildLineIndex` is async and unawaited; between its start and finish the row's + * file can be swapped (remove + re-add a same-named file, or a folder refresh + * re-indexing an edited file). Without the identity check a stale completion would + * stamp the wrong lineIndex/lineCount on the new file, and read_file would then slice + * the new Blob with the old offsets. + */ + #patchFile(name: string, file: File | Blob, changes: Partial): void { + this.files = this.files.map((f) => + f.name === name && f.file === file ? { ...f, ...changes } : f + ) + } + #patchSource(sourceId: string, changes: Partial): void { + this.files = this.files.map((f) => (f.sourceId === sourceId ? { ...f, ...changes } : f)) + } + + #uniqueName(original: string): string { + // Uniqueness is only among real files — folder-root placeholders may share a name + // with a standalone file and must not push it to a "(2)" suffix. + const taken = (n: string) => this.files.some((f) => f.name === n && !f.isFolderRoot) + if (!taken(original)) return original + const dot = original.lastIndexOf('.') + const base = dot > 0 ? original.slice(0, dot) : original + const ext = dot > 0 ? original.slice(dot) : '' + let n = 2 + let candidate = `${base} (${n})${ext}` + while (taken(candidate)) { + n++ + candidate = `${base} (${n})${ext}` + } + return candidate + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts new file mode 100644 index 0000000000..c81bc1f9c4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts @@ -0,0 +1,476 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// Mock persistence + File System Access so we exercise the in-memory store logic. +vi.mock('./attachedFilesDB', () => ({ + putItem: vi.fn(async () => {}), + deleteItem: vi.fn(async () => {}), + getItemsForSession: vi.fn(async () => []), + ensurePersistentStorage: vi.fn(async () => {}) +})) + +const enumerateDirMock = vi.fn<(h: unknown) => Promise<{ file: File; path: string }[]>>() +vi.mock('./fsAccess', () => ({ + enumerateDir: (h: unknown) => enumerateDirMock(h), + isIgnoredPath: (p: string) => + p.split('/').some((s) => s.startsWith('.') || ['node_modules', 'dist', '.git'].includes(s)), + queryReadPermission: vi.fn(async () => 'granted'), + requestReadPermission: vi.fn(async () => 'granted') +})) + +// buildLineIndex is real by default; a single test flips to 'manual' to control +// completion ordering and exercise the stale-index race guard. +type BuildResult = { lineIndex: number[]; lineCount: number } +const buildDeferreds: Array<{ file: Blob; resolve: (r: BuildResult) => void }> = [] +let buildMode: 'real' | 'manual' = 'real' +vi.mock('./fileEngine', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + buildLineIndex: (file: Blob) => + buildMode === 'real' + ? actual.buildLineIndex(file) + : new Promise((resolve) => buildDeferreds.push({ file, resolve })) + } +}) + +import { AttachedFilesStore } from './attachedFiles.svelte' + +function file(name: string, content: string, lastModified = 1): File { + return new File([content], name, { type: 'text/plain', lastModified }) +} + +const dir = { kind: 'directory', name: 'proj' } as unknown as FileSystemDirectoryHandle + +async function settle(store: AttachedFilesStore) { + for (let i = 0; i < 100 && store.list().some((f) => f.status === 'indexing'); i++) { + await new Promise((r) => setTimeout(r, 2)) + } +} + +const names = (store: AttachedFilesStore) => + store + .list() + .map((f) => f.name) + .sort() + +describe('AttachedFilesStore', () => { + let store: AttachedFilesStore + beforeEach(async () => { + store = new AttachedFilesStore() + await store.restore('s1', false) + }) + + it('links and indexes individual files as snapshots', async () => { + await store.addFiles([file('a.txt', 'one\ntwo\n')]) + await settle(store) + const f = store.get('a.txt') + expect(f?.status).toBe('ready') + expect(f?.lineCount).toBe(2) + expect(f?.handle).toBeUndefined() // files never carry a handle + }) + + it('removes a file', async () => { + await store.addFiles([file('a.txt', 'x')]) + store.removeFile('a.txt') + expect(store.count).toBe(0) + }) + + it('links a folder via a directory handle (enumerating it internally)', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n'), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + expect(enumerateDirMock).toHaveBeenCalledWith(dir) + expect(names(store)).toEqual(['proj/app.ts', 'proj/old.ts']) + expect(store.get('proj/app.ts')?.folder).toBe('proj') + }) + + it('refreshFolders detects rename, add, edit, and delete', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n', 1), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n', 1), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + + // On disk: app.ts edited (mtime bumped), old.ts renamed → new.ts, readme.md added. + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\nedited\n', 2), path: 'proj/app.ts' }, + { file: file('new.ts', 'y\n', 1), path: 'proj/new.ts' }, + { file: file('readme.md', '# hi\n', 1), path: 'proj/readme.md' } + ]) + await store.refreshFolders() + await settle(store) + + // old.ts dropped (renamed away); new.ts + readme.md added; app.ts kept. + expect(names(store)).toEqual(['proj/app.ts', 'proj/new.ts', 'proj/readme.md']) + // edited file re-indexed to its new content (2 lines). + expect(store.get('proj/app.ts')?.status).toBe('ready') + expect(store.get('proj/app.ts')?.lineCount).toBe(2) + }) + + it('exposes folders and standalone as structured views', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await store.addFolder(dir) + await store.addFiles([file('solo.txt', 'one\n')]) + await settle(store) + + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + expect(store.folders[0].status).toBe('ready') + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/app.ts', + 'proj/sub/b.ts' + ]) + expect(store.standalone.map((f) => f.name)).toEqual(['solo.txt']) + expect(store.lockedCount).toBe(0) + }) + + it('a locked folder surfaces as one folder with no files', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + + expect(s2.folders).toEqual([{ name: 'proj', status: 'locked', files: [] }]) + expect(s2.standalone).toEqual([]) + expect(s2.lockedCount).toBe(1) + }) + + it('re-picking a locked folder relinks it instead of silently no-oping', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.folders[0]?.status).toBe('locked') + + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + const result = await s2.addFolder(dir) + await settle(s2) + expect(result.added).toEqual(['proj/app.ts']) + expect(s2.folders).toHaveLength(1) + expect(s2.folders[0].status).toBe('ready') + }) + + it('rejects linking a second folder with the same name (visible, not silent)', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + const result = await store.addFolder(dir) + expect(result.added).toEqual([]) + expect(result.rejected[0]?.reason).toMatch(/already linked/) + expect(store.folders).toHaveLength(1) + }) + + it('regrant keeps the folder visible as unavailable when enumeration fails', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Permission re-granted, but the directory is gone from disk. + enumerateDirMock.mockRejectedValueOnce(new Error('directory removed')) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'unavailable', files: [] }]) + }) + + it('regrant of an empty folder keeps it linked and refreshing (not unlinked)', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Access re-granted, but the folder is currently empty — it must stay linked (ready + // placeholder), not vanish when the locked placeholder is dropped. + enumerateDirMock.mockResolvedValueOnce([]) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(s2.lockedCount).toBe(0) + + // A file added afterward is picked up — the handle survived. + enumerateDirMock.mockResolvedValueOnce([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s2.refreshFolders() + await settle(s2) + expect(s2.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) + + it('removeFolder drops all of a folder’s files', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/b.ts' } + ]) + await store.addFolder(dir) + store.removeFolder('proj') + expect(store.count).toBe(0) + }) + + it('snapshots a folder via addFiles (paths), grouping it and persisting folder + relPath', async () => { + const { putItem } = await import('./attachedFilesDB') + // A persisted (non-transient) session writes through to IndexedDB immediately. + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + expect(s.folders.map((f) => f.name)).toEqual(['proj']) + expect(s.folders[0].files.map((f) => f.relPath).sort()).toEqual(['proj/a.ts', 'proj/sub/b.ts']) + expect(s.standalone).toEqual([]) + const rec = (putItem as ReturnType).mock.calls + .map((c) => c[0]) + .find((r) => r.name === 'proj/a.ts') + expect(rec).toMatchObject({ kind: 'snapshot', folder: 'proj', relPath: 'proj/a.ts' }) + }) + + it('keeps same-basename files from different folder subdirs (dedup by path, not basename)', async () => { + // Two distinct files with the same basename, size and lastModified, different subdirs. + const res = await store.addFiles([ + { file: file('index.ts', 'a\n', 5), path: 'proj/a/index.ts' }, + { file: file('index.ts', 'a\n', 5), path: 'proj/b/index.ts' } + ]) + await settle(store) + expect(res.added.sort()).toEqual(['proj/a/index.ts', 'proj/b/index.ts']) + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/a/index.ts', + 'proj/b/index.ts' + ]) + }) + + it('skips junk paths (node_modules/.git/dotfiles) inside a snapshotted folder', async () => { + const res = await store.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('dep.js', 'z\n'), path: 'proj/node_modules/dep.js' }, + { file: file('cfg', 'w\n'), path: 'proj/.git/config' } + ]) + await settle(store) + expect(res.added).toEqual(['proj/a.ts']) + expect(store.folders[0].files).toHaveLength(1) + }) + + it('keeps an explicitly attached standalone dotfile (filter is folder-only)', async () => { + const res = await store.addFiles([file('.env', 'SECRET=1\n')]) + await settle(store) + expect(res.added).toEqual(['.env']) + expect(store.standalone.map((f) => f.name)).toEqual(['.env']) + }) + + it('restores a snapshot folder grouped from its persisted folder/relPath', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 's-a', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/a.ts', + folder: 'proj', + relPath: 'proj/a.ts', + blob: file('a.ts', 'x\n'), + addedAt: 0 + }, + { + id: 's-b', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/b.ts', + folder: 'proj', + relPath: 'proj/b.ts', + blob: file('b.ts', 'y\n'), + addedAt: 0 + } + ]) + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + await settle(s2) + expect(s2.folders.map((f) => f.name)).toEqual(['proj']) + expect(s2.folders[0].files).toHaveLength(2) + expect(s2.standalone).toEqual([]) + }) + + it('imposes no file-count cap on a folder', async () => { + enumerateDirMock.mockResolvedValue( + Array.from({ length: 150 }, (_, i) => ({ + file: file(`f${i}.ts`, 'x\n'), + path: `proj/f${i}.ts` + })) + ) + await store.addFolder(dir) + await settle(store) + expect(store.folders[0].files.length).toBe(150) + }) + + it('removeFolder deletes every snapshot record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + const ids = s + .list() + .filter((f) => f.folder === 'proj') + .map((f) => f.sourceId) + expect(ids.length).toBe(2) + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + const deleted = (deleteItem as ReturnType).mock.calls.map((c) => c[0]) + for (const id of ids) expect(deleted).toContain(id) + }) + + it('a stale index completion does not corrupt a re-added same-named file', async () => { + buildMode = 'manual' + try { + const A = file('a.txt', 'AAA\n') + const B = file('a.txt', 'BBB\nBBB\nBBB\n') + await store.addFiles([A]) // row 'a.txt' (file A) → buildLineIndex(A) pending + store.removeFile('a.txt') + await store.addFiles([B]) // new row 'a.txt' (file B) → buildLineIndex(B) pending + + // The old (stale) index for A resolves last — it must NOT touch the row now holding B. + buildDeferreds.find((d) => d.file === A)!.resolve({ lineIndex: [0], lineCount: 99 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('indexing') + expect(store.get('a.txt')?.lineCount).not.toBe(99) + + // B's own index applies normally. + buildDeferreds.find((d) => d.file === B)!.resolve({ lineIndex: [0, 4, 8], lineCount: 3 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('ready') + expect(store.get('a.txt')?.lineCount).toBe(3) + } finally { + buildMode = 'real' + buildDeferreds.length = 0 + } + }) + + it('removeFolder deletes the live folder record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.addFolder(dir) + await settle(s) + const sourceId = s.list().find((f) => f.folder === 'proj')?.sourceId + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + expect((deleteItem as ReturnType).mock.calls.map((c) => c[0])).toContain(sourceId) + }) + + it('an emptied live folder stays visible and refreshes when files return', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Folder emptied on disk → the last child is removed but the folder persists (placeholder). + enumerateDirMock.mockResolvedValue([]) + await store.refreshFolders() + await settle(store) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(store.get('proj/app.ts')).toBeUndefined() + expect(store.readyFiles()).toEqual([]) // placeholder is never a tool target + + // A file added back on disk is picked up — the live source survived the empty state. + enumerateDirMock.mockResolvedValue([{ file: file('new.ts', 'y\n'), path: 'proj/new.ts' }]) + await store.refreshFolders() + await settle(store) + expect(store.folders[0].files.map((f) => f.relPath)).toEqual(['proj/new.ts']) + }) + + it('an empty-folder placeholder does not collide with a same-named standalone file', async () => { + enumerateDirMock.mockResolvedValue([]) // empty folder "proj" → creates a placeholder named "proj" + await store.addFolder(dir) + await settle(store) + + // A standalone file literally named "proj" must NOT be deduped by the placeholder. + const res = await store.addFiles([file('proj', 'hello\n')]) + await settle(store) + expect(res.added).toEqual(['proj']) + expect(store.standalone.map((f) => f.name)).toEqual(['proj']) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Removing that standalone leaves the folder's placeholder intact. + store.removeFile('proj') + expect(store.standalone).toEqual([]) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + }) + + it('links an initially empty live folder (kept visible, persisted, refreshes)', async () => { + const { putItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([]) // folder is empty at link time + const res = await s.addFolder(dir) + await settle(s) + expect(res.added).toEqual([]) + expect(s.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + // persisted as a dir-handle so it survives a reload despite being empty + const persisted = (putItem as ReturnType).mock.calls.map((c) => c[0]) + expect(persisted.some((r) => r.kind === 'dir-handle' && r.folder === 'proj')).toBe(true) + + // a file added later is picked up — the source existed from the start. + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.refreshFolders() + await settle(s) + expect(s.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts new file mode 100644 index 0000000000..9673e29edc --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from 'vitest' +import { + getItemsForSession, + putItem, + deleteItem, + deleteItemsForSession, + ensurePersistentStorage +} from './attachedFilesDB' + +// IndexedDB is unavailable in the node test env. The module must degrade gracefully +// (open fails → reads return [], writes/deletes are no-ops) rather than throwing. +describe('attachedFilesDB without IndexedDB', () => { + it('returns [] for reads', async () => { + expect(await getItemsForSession('s1')).toEqual([]) + }) + + it('does not throw on writes/deletes', async () => { + await expect( + putItem({ id: 'a', sessionId: 's1', kind: 'snapshot', name: 'x.txt', addedAt: 0 }) + ).resolves.toBeUndefined() + await expect(deleteItem('a')).resolves.toBeUndefined() + await expect(deleteItemsForSession('s1')).resolves.toBeUndefined() + }) + + it('does not throw when requesting persistent storage', async () => { + await expect(ensurePersistentStorage()).resolves.toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts new file mode 100644 index 0000000000..36f9ac4fbe --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts @@ -0,0 +1,118 @@ +/** + * IndexedDB persistence for AI-chat linked files, keyed by session id. + * + * Two kinds of records survive a reload (see the persistence plan): + * - handle records ('file-handle' / 'dir-handle'): a re-grantable File System + * Access handle is stored (structured-clone), re-read live on restore. + * - 'snapshot' records: a full-byte Blob copy (fallback when the File System + * Access API is unavailable). + * + * Mirrors the `idb` usage in HistoryManager.svelte.ts. + */ +import { openDB, type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' + +export type AttachedItemKind = 'snapshot' | 'dir-handle' + +export interface PersistedAttachedItem { + /** Stable record id. */ + id: string + sessionId: string + /** 'snapshot' = a file copied into IndexedDB; 'dir-handle' = a live folder handle. */ + kind: AttachedItemKind + /** Display name: relative path for files, folder name for dir-handle records. */ + name: string + /** Top-level folder (for grouping); equals `name` for dir-handle records. */ + folder?: string + /** Folder-relative path (snapshot folder children) — restores the folder grouping/tree. */ + relPath?: string + /** Live directory handle (for 'dir-handle'). */ + handle?: FileSystemDirectoryHandle + /** Full-content copy (for 'snapshot'). */ + blob?: Blob + size?: number + lastModified?: number + addedAt: number +} + +interface AttachedFilesSchema extends IDBSchema { + items: { + key: string + value: PersistedAttachedItem + indexes: { 'by-session': string } + } +} + +let dbPromise: Promise | undefined> | undefined + +function getDB(): Promise | undefined> { + if (!dbPromise) { + try { + dbPromise = openDB('copilot-attached-files', 1, { + upgrade(db) { + if (!db.objectStoreNames.contains('items')) { + const store = db.createObjectStore('items', { keyPath: 'id' }) + store.createIndex('by-session', 'sessionId') + } + } + }).catch((err) => { + console.error('Could not open attached-files database', err) + return undefined + }) + } catch (err) { + // IndexedDB unavailable (e.g. private mode / no DOM) — degrade gracefully. + console.error('Could not open attached-files database', err) + dbPromise = Promise.resolve(undefined) + } + } + return dbPromise +} + +export async function putItem(item: PersistedAttachedItem): Promise { + const db = await getDB() + await db?.put('items', item) +} + +export async function getItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return [] + try { + return await db.getAllFromIndex('items', 'by-session', sessionId) + } catch (err) { + console.error('Could not read attached files', err) + return [] + } +} + +export async function deleteItem(id: string): Promise { + const db = await getDB() + await db?.delete('items', id) +} + +export async function deleteItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return + try { + const tx = db.transaction('items', 'readwrite') + const index = tx.store.index('by-session') + let cursor = await index.openCursor(sessionId) + while (cursor) { + await cursor.delete() + cursor = await cursor.continue() + } + await tx.done + } catch (err) { + console.error('Could not delete attached files for session', err) + } +} + +/** Ask the browser to keep our storage from being evicted (best-effort, once). */ +let persistRequested = false +export async function ensurePersistentStorage(): Promise { + if (persistRequested) return + persistRequested = true + try { + await navigator.storage?.persist?.() + } catch { + // best-effort; ignore + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts new file mode 100644 index 0000000000..8ef6d5b827 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts @@ -0,0 +1,286 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + buildLineIndex, + readFile, + searchFiles, + searchFilesInWorker, + isTextFile, + numberLines, + type FileEntry +} from './fileEngine' + +function makeFile(content: string | Uint8Array, name = 'f.txt'): File { + return new File([content as BlobPart], name) +} + +async function makeEntry(content: string, name = 'f.txt'): Promise { + const file = makeFile(content, name) + const { lineIndex, lineCount } = await buildLineIndex(file) + return { name, file, lineIndex, lineCount } +} + +describe('buildLineIndex', () => { + it('counts lines without a trailing newline', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('does not count a single trailing newline as an extra line', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb\n')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('handles an empty file', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('')) + expect(lineCount).toBe(0) + expect(lineIndex).toEqual([]) + }) + + it('handles CRLF line endings (offsets by byte)', async () => { + // bytes: a=0 \r=1 \n=2 b=3 → line starts at 0 and 3 + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\r\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 3]) + }) + + it('counts lines correctly across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => `line ${i}`) + const { lineCount } = await buildLineIndex(makeFile(lines.join('\n'))) + expect(lineCount).toBe(5000) + }) +}) + +describe('readFile', () => { + it('reads a bounded window and reports pagination', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 2 }) + expect(res.text).toBe('l1\nl2\n') + expect(res.startLine).toBe(1) + expect(res.endLine).toBe(2) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=3') + }) + + it('reads the final line to end of file', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 3 }) + expect(res.text).toBe('l3') + expect(res.truncated).toBe(false) + }) + + it('clamps the window to maxLines', async () => { + const entry = await makeEntry(Array.from({ length: 100 }, (_, i) => `l${i}`).join('\n')) + const res = await readFile(entry, { startLine: 1, maxLines: 10 }) + expect(res.endLine).toBe(10) + expect(res.truncated).toBe(true) + }) + + it('char-caps a degenerate single long line', async () => { + const entry = await makeEntry('x'.repeat(10000)) + const res = await readFile(entry, { maxChars: 8000 }) + expect(res.text.length).toBe(8000) + expect(res.truncated).toBe(true) + expect(res.note).toContain('8000 characters') + }) + + it('bounds the byte decode on a newline-sparse window (never reads past maxChars*4 bytes)', async () => { + // One 200k-char line; with maxChars=1000 only ≤4000 bytes are sliced before decode. + const entry = await makeEntry('y'.repeat(200_000)) + const res = await readFile(entry, { maxChars: 1000 }) + expect(res.text).toBe('y'.repeat(1000)) + expect(res.truncated).toBe(true) + }) + + it('clamps a start line beyond the end', async () => { + const entry = await makeEntry('l1\nl2') + const res = await readFile(entry, { startLine: 99 }) + expect(res.startLine).toBe(2) + expect(res.text).toBe('l2') + }) + + it('returns empty for an empty file', async () => { + const entry = await makeEntry('') + const res = await readFile(entry) + expect(res.text).toBe('') + expect(res.totalLines).toBe(0) + }) + + it('char-truncation inside the first line resumes the note at the next line', async () => { + // Line 1 exceeds maxChars; lines 2-3 follow. The window only returns line 1's prefix, + // so the note must report line 1 and point the model at line 2 (not claim lines 1-3). + const entry = await makeEntry('x'.repeat(20000) + '\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 3, maxChars: 5000 }) + expect(res.endLine).toBe(1) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=2') + }) + + it('char-truncation after some whole lines resumes at the truncated line', async () => { + const entry = await makeEntry('a\nb\n' + 'x'.repeat(20000) + '\nd') + const res = await readFile(entry, { startLine: 1, endLine: 4, maxChars: 5000 }) + expect(res.endLine).toBe(2) // a, b whole; line 3 (xxx) cut + expect(res.note).toContain('start_line=3') + // the partial line 3 must NOT leak into the body — it would contradict the note + expect(res.text).toBe('a\nb\n') + expect(numberLines(res.text, res.startLine)).toBe('1→a\n2→b') + }) +}) + +describe('searchFiles', () => { + it('finds matches with 1-based line numbers', async () => { + const entry = await makeEntry('alpha\nbeta\ngamma beta') + const res = await searchFiles([entry], 'beta') + expect(res.error).toBeUndefined() + expect(res.hits).toEqual([ + { file: 'f.txt', line: 2, text: 'beta' }, + { file: 'f.txt', line: 3, text: 'gamma beta' } + ]) + }) + + it('searches across multiple files', async () => { + const a = await makeEntry('needle here', 'a.txt') + const b = await makeEntry('nope\nneedle', 'b.txt') + const res = await searchFiles([a, b], 'needle') + expect(res.hits.map((h) => `${h.file}:${h.line}`)).toEqual(['a.txt:1', 'b.txt:2']) + }) + + it('restricts to a single file with pathFilter', async () => { + const a = await makeEntry('needle', 'a.txt') + const b = await makeEntry('needle', 'b.txt') + const res = await searchFiles([a, b], 'needle', { pathFilter: 'b.txt' }) + expect(res.hits).toEqual([{ file: 'b.txt', line: 1, text: 'needle' }]) + }) + + it('reports an unknown pathFilter as an error', async () => { + const a = await makeEntry('needle', 'a.txt') + const res = await searchFiles([a], 'needle', { pathFilter: 'missing.txt' }) + expect(res.error).toContain('missing.txt') + }) + + it('truncates at maxHits', async () => { + const entry = await makeEntry(Array.from({ length: 10 }, () => 'match').join('\n')) + const res = await searchFiles([entry], 'match', { maxHits: 3 }) + expect(res.hits.length).toBe(3) + expect(res.truncated).toBe(true) + }) + + it('supports case-insensitive flags', async () => { + const entry = await makeEntry('Hello\nworld') + const res = await searchFiles([entry], 'hello', { flags: 'i' }) + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'Hello' }]) + }) + + it('strips trailing CR from matched CRLF lines', async () => { + const entry = await makeEntry('foo\r\nbar') + const res = await searchFiles([entry], 'foo') + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'foo' }]) + }) + + it('returns a friendly error for an invalid regex', async () => { + const entry = await makeEntry('anything') + const res = await searchFiles([entry], '(') + expect(res.error).toContain('Invalid regex') + expect(res.hits).toEqual([]) + }) + + it('matches across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => (i === 4999 ? 'TARGET' : `line ${i}`)) + const entry = await makeEntry(lines.join('\n')) + const res = await searchFiles([entry], 'TARGET') + expect(res.hits).toEqual([{ file: 'f.txt', line: 5000, text: 'TARGET' }]) + }) + + it('a global flag does not drop matches via a stale lastIndex', async () => { + // `.test()` is stateful under the `g` flag; without resetting lastIndex, lines after + // the first match would be tested from a stale offset and silently miss. + const entry = await makeEntry('match\nmatch\nmatch') + const res = await searchFiles([entry], 'match', { flags: 'g' }) + expect(res.hits.map((h) => h.line)).toEqual([1, 2, 3]) + }) +}) + +describe('searchFilesInWorker', () => { + // A controllable stand-in for the search Worker — `reply`, `error`, or `hang` (never responds). + class MockWorker { + onmessage: ((e: MessageEvent) => void) | null = null + onerror: ((e: unknown) => void) | null = null + static mode: 'reply' | 'error' | 'hang' = 'reply' + static reply: unknown = { hits: [], truncated: false } + static terminated = false + constructor(_url: URL | string, _opts?: unknown) {} + postMessage(): void { + if (MockWorker.mode === 'reply') + queueMicrotask(() => this.onmessage?.({ data: MockWorker.reply } as MessageEvent)) + else if (MockWorker.mode === 'error') queueMicrotask(() => this.onerror?.({})) + // 'hang' → never responds, exercising the timeout path. + } + terminate(): void { + MockWorker.terminated = true + } + } + + beforeEach(() => { + MockWorker.terminated = false + vi.stubGlobal('Worker', MockWorker) + }) + afterEach(() => vi.unstubAllGlobals()) + + const entry: FileEntry = { name: 'a.txt', file: makeFile('x'), lineIndex: [], lineCount: 0 } + + it('resolves with the worker result and terminates the worker', async () => { + MockWorker.mode = 'reply' + MockWorker.reply = { hits: [{ file: 'a.txt', line: 1, text: 'x' }], truncated: false } + const res = await searchFilesInWorker([entry], 'x') + expect(res.hits).toEqual([{ file: 'a.txt', line: 1, text: 'x' }]) + expect(MockWorker.terminated).toBe(true) + }) + + it('times out on a non-responding (pathological) pattern and terminates the worker', async () => { + MockWorker.mode = 'hang' + const res = await searchFilesInWorker([entry], '^(a+)+$', {}, 20) + expect(res.error).toContain('timed out') + expect(MockWorker.terminated).toBe(true) + }) +}) + +describe('isTextFile', () => { + it('accepts UTF-8 text', async () => { + expect(await isTextFile(makeFile('hello © world'))).toBe(true) + }) + + it('accepts an empty file', async () => { + expect(await isTextFile(makeFile(''))).toBe(true) + }) + + it('rejects content with NUL bytes', async () => { + expect(await isTextFile(makeFile(new Uint8Array([104, 0, 105]), 'b.bin'))).toBe(false) + }) +}) + +describe('numberLines', () => { + it('prefixes each line with its absolute 1-based number', () => { + expect(numberLines('l3\nl4\n', 3)).toBe('3→l3\n4→l4') + }) + + it('right-aligns numbers to a common width', () => { + expect(numberLines('a\nb', 9)).toBe(' 9→a\n10→b') + }) + + it('numbers a final line that has no trailing newline', () => { + expect(numberLines('only', 5)).toBe('5→only') + }) + + it('matches a readFile window (numbers the returned lines, no phantom line)', async () => { + const file = makeFile('l1\nl2\nl3') + const { lineIndex, lineCount } = await buildLineIndex(file) + const res = await readFile( + { name: 'f.txt', file, lineIndex, lineCount }, + { startLine: 1, endLine: 2 } + ) + expect(numberLines(res.text, res.startLine)).toBe('1→l1\n2→l2') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts new file mode 100644 index 0000000000..5dbceb77e2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts @@ -0,0 +1,430 @@ +/** + * Storage-agnostic streaming engine for reading and searching attached files. + * + * Files are kept as `File` handles (lazy references to bytes on disk). Nothing is + * decoded into the JS heap wholesale: we stream in chunks, so a large file never + * freezes the tab or blows up memory. The only per-file state held in RAM is a + * line-offset index (a flat array of byte offsets, ~8 bytes per line). + * + * Line semantics match `String.split('\n')` except a single trailing newline does + * NOT add an empty final line (so "a\nb\n" is 2 lines, like `wc -l`). Lines are + * 1-based in the public read/search API. + */ + +/** Minimal shape the engine needs. The attached-files store extends this with reactive status. */ +export interface FileEntry { + name: string + /** A File (live link) or a Blob (restored snapshot) — both stream/slice identically. */ + file: File | Blob + lineIndex: number[] + lineCount: number +} + +const CHUNK_NEWLINE = 0x0a // '\n' — in UTF-8 this byte never appears inside a multibyte sequence + +export const DEFAULT_READ_MAX_LINES = 200 +export const DEFAULT_READ_MAX_CHARS = 8000 +export const DEFAULT_SEARCH_MAX_HITS = 50 +/** + * Per-line cap on how much of a degenerate long line the regex is tested against. + * This bounds work for linear-time patterns; it does NOT prevent catastrophic + * backtracking — a nested-quantifier pattern can still go exponential within the + * capped prefix (search runs on the main thread, so that is a self-DoS of the tab). + */ +export const DEFAULT_SEARCH_LINE_SCAN_CAP = 100_000 +/** How much of a matching line we echo back, to keep search results bounded. */ +export const DEFAULT_SEARCH_LINE_ECHO_CAP = 500 + +/** + * Stream the file once and record the byte offset at which each line starts. + * Scans raw bytes for '\n' (no decode needed — 0x0A is unambiguous in UTF-8). + */ +export async function buildLineIndex( + file: Blob +): Promise<{ lineIndex: number[]; lineCount: number }> { + const fileSize = file.size + if (fileSize === 0) { + return { lineIndex: [], lineCount: 0 } + } + + const lineIndex: number[] = [0] + let offset = 0 + const reader = file.stream().getReader() + try { + while (true) { + const { done, value } = await reader.read() + if (done) break + const chunk = value as Uint8Array + for (let i = 0; i < chunk.length; i++) { + if (chunk[i] === CHUNK_NEWLINE) { + lineIndex.push(offset + i + 1) + } + } + offset += chunk.length + } + } finally { + reader.releaseLock() + } + + // A trailing newline points one past the end (a phantom empty line) — drop it. + if (lineIndex.length > 1 && lineIndex[lineIndex.length - 1] === fileSize) { + lineIndex.pop() + } + + return { lineIndex, lineCount: lineIndex.length } +} + +export interface ReadResult { + text: string + startLine: number + endLine: number + totalLines: number + truncated: boolean + note: string +} + +/** + * Read a bounded window of lines, reading only the relevant byte range from disk. + * Clamps the window to `maxLines` and the returned text to `maxChars` (protects + * against degenerate single-line files). Returns a self-describing pagination note. + */ +export async function readFile( + entry: FileEntry, + opts: { + startLine?: number + endLine?: number + maxLines?: number + maxChars?: number + } = {} +): Promise { + const totalLines = entry.lineCount + const maxLines = opts.maxLines ?? DEFAULT_READ_MAX_LINES + const maxChars = opts.maxChars ?? DEFAULT_READ_MAX_CHARS + + if (totalLines === 0) { + return { + text: '', + startLine: 0, + endLine: 0, + totalLines: 0, + truncated: false, + note: 'File is empty.' + } + } + + let start = opts.startLine ?? 1 + if (start < 1) start = 1 + if (start > totalLines) start = totalLines + + const requestedEnd = opts.endLine ?? start + maxLines - 1 + let end = requestedEnd + if (end < start) end = start + const cappedByLines = end - start + 1 > maxLines + if (cappedByLines) end = start + maxLines - 1 + if (end > totalLines) end = totalLines + + const byteStart = entry.lineIndex[start - 1] + const byteEnd = end < totalLines ? entry.lineIndex[end] : entry.file.size + // Bound the decode for newline-sparse files (minified JS, single-line JSONL): the + // window can span the whole file, but we only ever return maxChars characters, and + // a UTF-8 character is at most 4 bytes — so never materialize more than that. + const byteCap = byteStart + maxChars * 4 + const byteCapped = byteCap < byteEnd + + let text: string + try { + text = await entry.file.slice(byteStart, byteCapped ? byteCap : byteEnd).text() + } catch (e) { + throw new FileReadError(entry.name, e instanceof Error ? e.message : String(e)) + } + + let cappedByChars = byteCapped + if (text.length > maxChars) { + text = text.slice(0, maxChars) + cappedByChars = true + } + + // When the char cap truncates the window short of `end`, the text holds fewer lines + // than requested — so the note must report the last line actually returned and resume + // at the next unread one (otherwise it claims lines it didn't return and skips them). + let lastLine = end + let resumeAt: number | undefined = end < totalLines ? end + 1 : undefined + if (cappedByChars) { + const completeLines = (text.match(/\n/g) || []).length + if (completeLines >= 1) { + // lines start..start+completeLines-1 are whole; the next line was cut mid-content. + // Trim that partial line off the returned text so the body matches the note (and + // the model doesn't see a line the note says it'll get on the next read). + lastLine = start + completeLines - 1 + resumeAt = start + completeLines + text = text.slice(0, text.lastIndexOf('\n') + 1) + } else { + // the cap fell inside line `start` itself — it can't be returned in full, so + // advance past it rather than re-truncating the same line forever. + lastLine = start + resumeAt = start + 1 + } + if (resumeAt > totalLines) resumeAt = undefined + } + + const truncated = cappedByChars || resumeAt !== undefined + + let note = `Showing lines ${start}-${lastLine} of ${totalLines}.` + if (cappedByChars) { + note += ` Output truncated to ${maxChars} characters (line(s) very long).` + } + if (resumeAt !== undefined) { + note += ` Call read_file again with start_line=${resumeAt} for more.` + } + + return { text, startLine: start, endLine: lastLine, totalLines, truncated, note } +} + +/** + * Prefix each line of a read window with its absolute 1-based number (`→`), + * so the model can quote/reference exact lines. `startLine` is the window's first line. + */ +export function numberLines(text: string, startLine: number): string { + const lines = text.split('\n') + // readFile's window ends with the trailing newline of its last line when more lines + // follow, so split yields a phantom empty element — drop it before numbering. + if (lines.length > 1 && lines[lines.length - 1] === '') lines.pop() + const width = String(startLine + lines.length - 1).length + return lines.map((l, i) => `${String(startLine + i).padStart(width)}→${l}`).join('\n') +} + +export interface SearchHit { + file: string + line: number + text: string +} + +export interface SearchResult { + hits: SearchHit[] + truncated: boolean + error?: string +} + +/** + * Run a regex across one or more files, streaming each (no full-file load), and + * return matching lines with 1-based line numbers. Stops at `maxHits`. + */ +export async function searchFiles( + entries: FileEntry[], + pattern: string, + opts: { + flags?: string + pathFilter?: string + maxHits?: number + lineScanCap?: number + lineEchoCap?: number + } = {} +): Promise { + const maxHits = opts.maxHits ?? DEFAULT_SEARCH_MAX_HITS + const lineScanCap = opts.lineScanCap ?? DEFAULT_SEARCH_LINE_SCAN_CAP + const lineEchoCap = opts.lineEchoCap ?? DEFAULT_SEARCH_LINE_ECHO_CAP + + let regex: RegExp + try { + regex = new RegExp(pattern, opts.flags ?? '') + } catch (e) { + return { + hits: [], + truncated: false, + error: `Invalid regex: ${e instanceof Error ? e.message : String(e)}` + } + } + + const targets = opts.pathFilter ? entries.filter((e) => e.name === opts.pathFilter) : entries + if (opts.pathFilter && targets.length === 0) { + return { hits: [], truncated: false, error: `No attached file named "${opts.pathFilter}".` } + } + + const hits: SearchHit[] = [] + let truncated = false + + for (const entry of targets) { + if (hits.length >= maxHits) { + truncated = true + break + } + try { + await streamLines(entry.file, (line, lineNo) => { + // Bound backtracking on pathological long lines by only testing a prefix. + const scanned = line.length > lineScanCap ? line.slice(0, lineScanCap) : line + // `regex` may carry a caller-supplied `g`/`y` flag, which makes `.test()` + // stateful (it advances `lastIndex`) — reset so each line matches from 0. + regex.lastIndex = 0 + if (regex.test(scanned)) { + hits.push({ + file: entry.name, + line: lineNo, + text: line.length > lineEchoCap ? line.slice(0, lineEchoCap) + '…' : line + }) + } + return hits.length < maxHits // continue? + }) + } catch (e) { + return { + hits, + truncated, + error: `Error reading "${entry.name}": ${e instanceof Error ? e.message : String(e)}` + } + } + if (hits.length >= maxHits) { + truncated = true + break + } + } + + return { hits, truncated } +} + +/** + * Run `searchFiles` off the main thread. A model-supplied regex can backtrack + * catastrophically (e.g. /^(a+)+$/) and `regex.test()` can't be interrupted — so we run + * it in a Worker and `terminate()` it on a timeout, keeping the tab responsive instead of + * frozen. Falls back to a main-thread search where Workers aren't available (best effort). + */ +export function searchFilesInWorker( + entries: FileEntry[], + pattern: string, + opts: { flags?: string; pathFilter?: string; maxHits?: number } = {}, + timeoutMs = 3000 +): Promise { + let worker: Worker + try { + worker = new Worker(new URL('./searchWorker.ts', import.meta.url), { type: 'module' }) + } catch { + return searchFiles(entries, pattern, opts) + } + return new Promise((resolve) => { + const finish = (r: SearchResult) => { + clearTimeout(timer) + worker.terminate() + resolve(r) + } + const timer = setTimeout( + () => + finish({ + hits: [], + truncated: false, + error: 'Search timed out — the pattern is too expensive. Try a simpler regex.' + }), + timeoutMs + ) + worker.onmessage = (e: MessageEvent) => finish(e.data) + worker.onerror = () => { + // Worker script failed to load/run — fall back to a main-thread search. + clearTimeout(timer) + worker.terminate() + searchFiles(entries, pattern, opts).then(resolve) + } + worker.postMessage({ + files: entries.map((e) => ({ name: e.name, file: e.file })), + pattern, + flags: opts.flags, + pathFilter: opts.pathFilter, + maxHits: opts.maxHits + }) + }) +} + +export class FileReadError extends Error { + constructor( + public fileName: string, + message: string + ) { + super(message) + this.name = 'FileReadError' + } +} + +/** + * Max characters buffered for a single line while streaming. A newline-less file + * (e.g. minified JS) would otherwise accumulate wholesale in `buffer`; past this + * cap excess characters are dropped (the line's intact prefix is preserved) — + * harmless for search, which only tests/echoes a prefix far smaller than this. + */ +const MAX_LINE_BUFFER_CHARS = 1_000_000 + +/** + * Stream a file and invoke `onLine` for each line (1-based). A trailing newline + * does not produce an empty final line. `onLine` returns false to stop early. + * A trailing '\r' (CRLF) is stripped before the callback. Overlong lines are + * passed with at least their first MAX_LINE_BUFFER_CHARS characters intact; + * content beyond the cap may be dropped. + */ +async function streamLines( + file: Blob, + onLine: (line: string, lineNo: number) => boolean +): Promise { + const reader = file.stream().getReader() + const decoder = new TextDecoder('utf-8') + let buffer = '' + let lineNo = 0 + try { + while (true) { + const { done, value } = await reader.read() + if (done) { + buffer += decoder.decode() + break + } + buffer += decoder.decode(value as Uint8Array, { stream: true }) + let start = 0 + let nlIdx: number + while ((nlIdx = buffer.indexOf('\n', start)) !== -1) { + let line = buffer.slice(start, nlIdx) + if (line.endsWith('\r')) line = line.slice(0, -1) + start = nlIdx + 1 + lineNo++ + if (!onLine(line, lineNo)) return + } + buffer = buffer.slice(start) + // The remainder holds no newline — cap how much of an overlong line we keep. + // Dropped characters are line content only, so newline detection and line + // numbering in later chunks are unaffected. + if (buffer.length > MAX_LINE_BUFFER_CHARS) { + buffer = buffer.slice(0, MAX_LINE_BUFFER_CHARS) + } + } + if (buffer.length > 0) { + let line = buffer + if (line.endsWith('\r')) line = line.slice(0, -1) + lineNo++ + onLine(line, lineNo) + } + } finally { + reader.releaseLock() + } +} + +/** + * Sniff the first bytes of a file to decide whether it is text (UTF-8 decodable, + * no NUL bytes). Used to reject binary files at attach time. + */ +export async function isTextFile(file: Blob, sampleBytes = 8192): Promise { + if (file.size === 0) return true + const slice = file.slice(0, Math.min(sampleBytes, file.size)) + const buf = new Uint8Array(await slice.arrayBuffer()) + for (let i = 0; i < buf.length; i++) { + if (buf[i] === 0) return false // NUL byte → binary + } + try { + // `fatal` throws on invalid UTF-8. We may cut a multibyte char at the sample + // boundary, so only treat it as binary if the error is not at the very end. + new TextDecoder('utf-8', { fatal: true }).decode(buf) + return true + } catch { + // Could be a truncated trailing multibyte sequence — retry on a trimmed buffer. + if (buf.length >= 4) { + try { + new TextDecoder('utf-8', { fatal: true }).decode(buf.slice(0, buf.length - 3)) + return true + } catch { + return false + } + } + return false + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts new file mode 100644 index 0000000000..0bd45153af --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts @@ -0,0 +1,66 @@ +import { describe, expect, it, vi } from 'vitest' + +// '../shared' transitively imports monaco (CSS) which the node test env can't load. +// fileTools only needs createToolDef from it (called at module load), so stub it. +vi.mock('../shared', () => ({ + createToolDef: (_schema: unknown, name: string, description: string) => ({ name, description }) +})) + +import { searchFilesTool } from './fileTools' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Minimal store stub: searchFilesTool's empty-ready path only reads count/readyFiles/list. */ +function fakeStore(rows: Array>): AttachedFilesStore { + const files = rows as AttachedFile[] + return { + get count() { + return files.length + }, + readyFiles: () => files.filter((f) => f.status === 'ready' && !f.isFolderRoot), + list: () => files + } as unknown as AttachedFilesStore +} + +async function runSearch(store: AttachedFilesStore): Promise { + const res = await searchFilesTool.fn({ + args: { pattern: 'x' }, + helpers: { attachedFiles: store }, + toolId: 't', + toolCallbacks: { setToolStatus: () => {} } + } as any) + return res as string +} + +describe('search_files — attachments present but nothing readable', () => { + it('reports no searchable text for an empty/binary-only linked folder (only ready placeholders)', async () => { + // An empty/all-binary folder leaves a single `ready` placeholder row, filtered out of readyFiles(). + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'ready', isFolderRoot: true }])) + expect(msg).toMatch(/no searchable text files/i) + expect(msg).not.toMatch(/indexed/i) + }) + + it('tells the user to restore access when a restored folder is locked', async () => { + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'locked', isFolderRoot: true }])) + expect(msg).toMatch(/restore access/i) + }) + + it('tells the user to re-link when files are unavailable', async () => { + const msg = await runSearch(fakeStore([{ name: 'gone.txt', status: 'unavailable' }])) + expect(msg).toMatch(/re-link/i) + }) + + it('still reports indexing while a file is genuinely indexing', async () => { + const msg = await runSearch(fakeStore([{ name: 'a.txt', status: 'indexing' }])) + expect(msg).toMatch(/still being indexed/i) + }) + + it('prefers the indexing message when an indexing file coexists with an empty folder', async () => { + const msg = await runSearch( + fakeStore([ + { name: 'proj', status: 'ready', isFolderRoot: true }, + { name: 'a.txt', status: 'indexing' } + ]) + ) + expect(msg).toMatch(/still being indexed/i) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.ts new file mode 100644 index 0000000000..982b858154 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.ts @@ -0,0 +1,226 @@ +/** + * AI tools and system-prompt roster for files attached to the GLOBAL chat. + * + * The model is made aware of attached files via a metadata-only roster appended to + * the system message (see `appendAttachedFilesRoster`). Their contents are NEVER + * inlined — the model pulls only the slices it needs through these two read-only + * tools, which stream from disk via ./fileEngine. + */ +import { z } from 'zod' +import type { ChatCompletionSystemMessageParam } from 'openai/resources/chat/completions.mjs' +import { createToolDef, type Tool } from '../shared' +import { + readFile, + searchFilesInWorker, + numberLines, + FileReadError, + type SearchHit +} from './fileEngine' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Slice of the GLOBAL tool helpers that exposes the attached-files store. */ +export interface AttachedFilesHelper { + attachedFiles?: AttachedFilesStore +} + +function storeFrom(helpers: unknown): AttachedFilesStore | undefined { + return (helpers as AttachedFilesHelper | undefined)?.attachedFiles +} + +/** + * For a specifically requested attached file, a message describing why it can't be read / + * searched yet (still indexing, locked, unavailable, errored) or that it isn't attached — + * or undefined when it's `ready`. Shared by read_file and search_files so both report the + * same accurate status instead of search_files claiming a non-ready file isn't attached. + */ +function notReadyMessage(store: AttachedFilesStore, file: string): string | undefined { + const entry = store.get(file) + if (entry?.status === 'ready') return undefined + if (entry?.status === 'indexing') + return `File "${file}" is still being indexed. Try again shortly.` + if (entry?.status === 'locked') + return `File "${file}" is locked after a reload. Ask the user to restore access (send a message, or click "Restore access").` + if (entry?.status === 'unavailable') + return `File "${file}" is no longer available (moved, deleted, or its local copy was evicted). Ask the user to re-link it.` + if (entry?.status === 'error') + return `File "${file}" failed to load: ${entry.error ?? 'unknown error'}.` + const names = store + .list() + .map((f) => f.name) + .join(', ') + return `No attached file named "${file}". Attached files: ${names || '(none)'}.` +} + +/** + * When attachments exist but none expose a readable target (`readyFiles()` is empty), + * explain the actual reason instead of always claiming files are still indexing. Empty + * or binary-only linked folders leave only `ready` placeholder rows (filtered out of + * `readyFiles`), while a locked/unavailable restore surfaces those statuses on the rows. + */ +function noReadyFilesMessage(store: AttachedFilesStore): string { + const statuses = new Set(store.list().map((f) => f.status)) + if (statuses.has('indexing')) return 'Attached files are still being indexed. Try again shortly.' + if (statuses.has('locked')) + return 'The attached files are locked after a reload. Ask the user to restore access (send a message, or click "Restore access").' + if (statuses.has('unavailable')) + return 'The attached files are no longer available (moved, deleted, or their local copies were evicted). Ask the user to re-link them.' + if (statuses.has('error')) return 'The attached files failed to load.' + return 'No searchable text files are attached (a linked folder may be empty or contain only non-text files).' +} + +function humanSize(bytes: number): string { + if (bytes < 1024) return `${bytes} B` + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB` + return `${(bytes / (1024 * 1024)).toFixed(1)} MB` +} + +const searchFilesSchema = z.object({ + pattern: z.string().describe('JavaScript regular expression to search for.'), + file: z + .string() + .optional() + .describe( + 'Optional exact filename (as listed under "Attached files") to restrict the search to. Omit to search across all attached files.' + ), + ignore_case: z.boolean().optional().describe('Case-insensitive matching. Defaults to false.') +}) + +const searchFilesToolDef = createToolDef( + searchFilesSchema, + 'search_files', + 'Search the user-attached files with a regular expression and return matching lines with their line numbers. Use this to locate content before reading a specific window with read_file.' +) + +export const searchFilesTool: Tool<{}> = { + def: searchFilesToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = searchFilesSchema.parse(args) + // Validate a specifically requested file against the full store first, so a non-ready + // target reports its real status (indexing/locked/…) instead of "not attached". + if (parsed.file) { + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + } + const ready = store.readyFiles() + if (ready.length === 0) { + return noReadyFilesMessage(store) + } + toolCallbacks.setToolStatus(toolId, { + content: `Searching attached files for /${parsed.pattern}/...` + }) + + // Run in a Worker so a pathological model-supplied regex can't freeze the tab. + const result = await searchFilesInWorker(ready, parsed.pattern, { + flags: parsed.ignore_case ? 'i' : '', + pathFilter: parsed.file + }) + if (result.error) { + return `Error: ${result.error}` + } + const scope = parsed.file ? `"${parsed.file}"` : `${ready.length} file(s)` + if (result.hits.length === 0) { + return `No matches for /${parsed.pattern}/ in ${scope}.` + } + const body = result.hits.map((h: SearchHit) => `${h.file}:${h.line}: ${h.text}`).join('\n') + const header = `Found ${result.hits.length} match(es) in ${scope}:` + const footer = result.truncated + ? '\n\n(Stopped at the result limit — refine your pattern or pass a `file` to narrow the search.)' + : '' + return `${header}\n${body}${footer}` + } +} + +const readFileSchema = z.object({ + file: z.string().describe('Exact filename to read, as listed under "Attached files".'), + start_line: z.number().int().optional().describe('1-based first line to read. Defaults to 1.'), + end_line: z + .number() + .int() + .optional() + .describe('1-based last line to read. The window is capped at 200 lines.') +}) + +const readFileToolDef = createToolDef( + readFileSchema, + 'read_file', + 'Read a bounded window of lines from a user-attached file. Returns each line prefixed with its 1-based number (`→`) plus a pagination note. Files are not in context, so use this to inspect their contents.' +) + +export const readFileTool: Tool<{}> = { + def: readFileToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = readFileSchema.parse(args) + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + const entry = store.get(parsed.file)! + toolCallbacks.setToolStatus(toolId, { content: `Reading "${parsed.file}"...` }) + + try { + const res = await readFile(entry, { + startLine: parsed.start_line, + endLine: parsed.end_line + }) + return res.text ? `${res.note}\n\n${numberLines(res.text, res.startLine)}` : res.note + } catch (e) { + if (e instanceof FileReadError) { + return `Could not read "${parsed.file}": ${e.message}. The file may have been moved or deleted since it was attached.` + } + return `Error reading "${parsed.file}": ${e instanceof Error ? e.message : String(e)}` + } + } +} + +export const fileTools: Tool<{}>[] = [searchFilesTool, readFileTool] + +function rosterLine(f: AttachedFile): string { + if (f.status === 'indexing') return `- ${f.name} (indexing…)` + if (f.status === 'locked') return `- ${f.name} (locked — needs the user to restore access)` + if (f.status === 'unavailable') return `- ${f.name} (unavailable)` + if (f.status === 'error') return `- ${f.name} (failed to load)` + return `- ${f.name} — ${f.lineCount} lines, ${humanSize(f.size)}` +} + +/** Build the `## Attached files` system-prompt section (metadata only, never content). */ +export function buildAttachedFilesRoster(store: AttachedFilesStore): string { + const lines: string[] = [] + for (const folder of store.folders) { + // A locked/unavailable folder has no readable children — one line for the whole folder. + if (folder.status === 'locked') { + lines.push(`- ${folder.name} (locked — needs the user to restore access)`) + } else if (folder.status === 'unavailable') { + lines.push(`- ${folder.name} (unavailable)`) + } else { + lines.push(...folder.files.map(rosterLine)) + } + } + lines.push(...store.standalone.map(rosterLine)) + if (lines.length === 0) return '' + return [ + '## Attached files', + 'The user has attached the following files to this conversation. Their contents are NOT included here.', + 'Use the `search_files` tool to find content with a regex, and `read_file` to read a bounded window of lines.', + '', + lines.join('\n') + ].join('\n') +} + +/** + * Return a copy of the system message with the attached-files roster appended. + * Always derives from the provided base so the roster never accumulates across turns. + */ +export function appendAttachedFilesRoster( + base: ChatCompletionSystemMessageParam, + store: AttachedFilesStore +): ChatCompletionSystemMessageParam { + const roster = buildAttachedFilesRoster(store) + if (!roster || typeof base.content !== 'string') return base + return { ...base, content: `${base.content}\n\n${roster}` } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts new file mode 100644 index 0000000000..e4536b6eac --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts @@ -0,0 +1,49 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { hasFileSystemAccess, isIgnoredPath, pickDirectory } from './fsAccess' + +describe('hasFileSystemAccess', () => { + it('is false when the File System Access API is absent (node / Firefox / Safari today)', () => { + // The test env exposes none of showOpenFilePicker / showDirectoryPicker / + // DataTransferItem.getAsFileSystemHandle, so the gate must report false. + // The positive path (all three present) is exercised via browser verification. + expect(hasFileSystemAccess()).toBe(false) + }) +}) + +describe('pickDirectory', () => { + afterEach(() => { + delete (window as { showDirectoryPicker?: unknown }).showDirectoryPicker + }) + + it('returns undefined when the user dismisses the picker (AbortError)', async () => { + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('aborted', 'AbortError') + } + await expect(pickDirectory()).resolves.toBeUndefined() + }) + + it('rethrows any non-abort failure instead of silently no-oping', async () => { + // e.g. a policy that blocks the File System Access API, or a lost user-activation. + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('blocked by policy', 'SecurityError') + } + await expect(pickDirectory()).rejects.toThrow(/blocked by policy/) + }) +}) + +describe('isIgnoredPath', () => { + it('keeps normal source paths', () => { + expect(isIgnoredPath('myproj/src/app.ts')).toBe(false) + expect(isIgnoredPath('README.md')).toBe(false) + }) + it('skips ignored directories', () => { + expect(isIgnoredPath('myproj/node_modules/lib/index.js')).toBe(true) + expect(isIgnoredPath('myproj/dist/bundle.js')).toBe(true) + expect(isIgnoredPath('a/target/x')).toBe(true) + }) + it('skips dotfiles and dotdirs', () => { + expect(isIgnoredPath('myproj/.env')).toBe(true) + expect(isIgnoredPath('myproj/.git/config')).toBe(true) + expect(isIgnoredPath('.DS_Store')).toBe(true) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts new file mode 100644 index 0000000000..2c741c0988 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts @@ -0,0 +1,192 @@ +/** + * Thin wrappers over the File System Access API, used when available so linked + * files/folders can be re-read live after a reload (re-grantable handles). + * Capability is feature-detected (never browser-sniffed): the day Firefox/Safari + * ship the API, the handle path lights up automatically. + */ +const IGNORED_DIRS = new Set([ + 'node_modules', + 'dist', + 'build', + 'out', + 'target', + 'vendor', + 'coverage', + '__pycache__', + '.git', + '.svelte-kit', + '.next', + '.nuxt', + '.venv', + 'venv', + '.idea', + '.vscode', + '.turbo', + '.cache' +]) + +function isIgnoredSegment(name: string): boolean { + return name.startsWith('.') || IGNORED_DIRS.has(name) +} + +/** True if any segment of a relative path is a dotfile/dotdir or an ignored directory. */ +export function isIgnoredPath(path: string): boolean { + return path.split('/').some(isIgnoredSegment) +} + +type FSWindow = Window & { + showDirectoryPicker?: (opts?: { + mode?: 'read' | 'readwrite' + }) => Promise +} + +type FSDataTransferItem = DataTransferItem & { + getAsFileSystemHandle?: () => Promise +} + +/** + * True when the File System Access API needed for FOLDER linking is usable: + * the directory picker plus drag-drop handles. (Files never use the API — they're + * always snapshotted — so showOpenFilePicker is intentionally not required.) + */ +export function hasFileSystemAccess(): boolean { + return ( + typeof window !== 'undefined' && + 'showDirectoryPicker' in window && + typeof DataTransferItem !== 'undefined' && + 'getAsFileSystemHandle' in DataTransferItem.prototype + ) +} + +/** + * Open the directory picker. Returns undefined if the user dismisses it. + * Any other failure (a policy that blocks the File System Access API, a lost + * user-activation, etc.) is rethrown — swallowing it makes the picker silently + * never open, which is indistinguishable from a no-op and impossible to debug. + */ +export async function pickDirectory(): Promise { + const w = window as FSWindow + if (!w.showDirectoryPicker) return undefined + try { + return await w.showDirectoryPicker({ mode: 'read' }) + } catch (e) { + // AbortError means the user dismissed the dialog (and, under browser automation, + // that CDP intercepted the chooser) — a no-op, not a failure. + if (e instanceof DOMException && e.name === 'AbortError') return undefined + throw e + } +} + +/** + * Resolve File System Access handles from a drop's items. The `getAsFileSystemHandle` + * calls are kicked off synchronously (items are only valid during the drop event); + * the returned promise resolves the handles. + */ +export function handlesFromDataTransfer(dt: DataTransfer): Promise { + const pending = Array.from(dt.items) + .filter((it) => it.kind === 'file') + .map((it) => (it as FSDataTransferItem).getAsFileSystemHandle?.() ?? Promise.resolve(null)) + return Promise.all(pending).then((handles) => handles.filter((h): h is FileSystemHandle => !!h)) +} + +export function isFileHandle(h: FileSystemHandle): h is FileSystemFileHandle { + return h.kind === 'file' +} +export function isDirectoryHandle(h: FileSystemHandle): h is FileSystemDirectoryHandle { + return h.kind === 'directory' +} + +/** + * Recursively read a directory handle into a flat list of files with relative paths, + * skipping junk (dotfiles/dotdirs, node_modules, …). No file-count cap — the browser's + * memory/quota are the only limit. Used on link and on live re-enumeration after a reload. + */ +export async function enumerateDir( + dir: FileSystemDirectoryHandle +): Promise<{ file: File; path: string }[]> { + const out: { file: File; path: string }[] = [] + + async function walk(handle: FileSystemDirectoryHandle, prefix: string): Promise { + // @ts-ignore - values() is an async iterator in the File System Access API + for await (const entry of handle.values() as AsyncIterable) { + const path = `${prefix}/${entry.name}` + if (isIgnoredPath(path)) continue + if (isFileHandle(entry)) { + out.push({ file: await entry.getFile(), path }) + } else if (isDirectoryHandle(entry)) { + await walk(entry, path) + } + } + } + + await walk(dir, dir.name) + return out +} + +/** + * Recursively read dropped files AND folders via the legacy `webkitGetAsEntry` API — + * the fallback for browsers without the File System Access API (Firefox/Safari). Folder + * contents are snapshotted into the browser (no live handle). Each result `path` is + * folder-relative (`folder/sub/file` for a dropped folder, bare name for a loose file), + * junk paths skipped, no file-count cap. + * + * `webkitGetAsEntry()` is only valid synchronously during the drop event, so this MUST be + * called from the drop handler — its `.map(...)` runs before the first `await`, capturing + * the entries while the items are still live. + */ +export async function readDroppedEntries( + items: DataTransferItem[] +): Promise<{ file: File; path: string }[]> { + const roots = items + .map((it) => it.webkitGetAsEntry?.() ?? null) + .filter((e): e is FileSystemEntry => !!e) + const out: { file: File; path: string }[] = [] + for (const root of roots) await walkDropEntry(root, out) + return out +} + +async function walkDropEntry( + entry: FileSystemEntry, + out: { file: File; path: string }[] +): Promise { + const path = entry.fullPath.replace(/^\//, '') + if (isIgnoredPath(path)) return + if (entry.isFile) { + const fileEntry = entry as FileSystemFileEntry + const file = await new Promise((res, rej) => fileEntry.file(res, rej)) + out.push({ file, path }) + } else if (entry.isDirectory) { + const reader = (entry as FileSystemDirectoryEntry).createReader() + // readEntries yields in batches and returns [] once exhausted — loop until empty. + while (true) { + const batch = await new Promise((res, rej) => reader.readEntries(res, rej)) + if (batch.length === 0) break + for (const child of batch) await walkDropEntry(child, out) + } + } +} + +/** queryPermission without a user gesture; 'granted' | 'prompt' | 'denied'. Never rejects. */ +export async function queryReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - queryPermission is part of the File System Access API + return (await handle.queryPermission?.({ mode: 'read' })) ?? 'prompt' + } catch { + return 'prompt' + } +} + +/** + * requestPermission — MUST be called within a user gesture. Never rejects: the spec + * rejects with SecurityError when user activation is missing (e.g. a second prompt + * after the first consumed the gesture) — that maps to 'denied' here so callers can + * treat it as "still locked" instead of blowing up the send path. + */ +export async function requestReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - requestPermission is part of the File System Access API + return (await handle.requestPermission?.({ mode: 'read' })) ?? 'denied' + } catch { + return 'denied' + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/searchWorker.ts b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts new file mode 100644 index 0000000000..68960801da --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts @@ -0,0 +1,38 @@ +/** + * Web Worker that runs `search_files` off the main thread. + * + * The regex is model-supplied and `RegExp.prototype.test()` can't be interrupted, so a + * catastrophic-backtracking pattern (e.g. /^(a+)+$/) would otherwise hang the whole tab. + * Running it here lets the caller `terminate()` this worker on a timeout instead. The + * matching itself reuses `searchFiles` from the engine (single source of truth). + */ +import { searchFiles, type FileEntry } from './fileEngine' + +interface SearchRequest { + files: { name: string; file: Blob }[] + pattern: string + flags?: string + pathFilter?: string + maxHits?: number +} + +self.onmessage = async (e: MessageEvent) => { + const { files, pattern, flags, pathFilter, maxHits } = e.data + // searchFiles only reads `name` + `file` (it streams); the index fields are unused here. + const entries: FileEntry[] = files.map((f) => ({ + name: f.name, + file: f.file, + lineIndex: [], + lineCount: 0 + })) + try { + const result = await searchFiles(entries, pattern, { flags, pathFilter, maxHits }) + ;(self as unknown as Worker).postMessage(result) + } catch (err) { + ;(self as unknown as Worker).postMessage({ + hits: [], + truncated: false, + error: err instanceof Error ? err.message : String(err) + }) + } +} diff --git a/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte b/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte index 37db9a90ba..b2496bbcda 100644 --- a/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte +++ b/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte @@ -135,7 +135,8 @@ await acceptPendingFlowEditsIfEnabled() }, getFlowInputsSchema: async () => { - return flowStore.val.schema ?? {} + const s = flowStore.val.schema ?? {} + return { type: 'object', properties: {}, required: [], ...s } }, updateExprsToSet: (id: string, inputTransforms: Record) => { diff --git a/frontend/src/lib/components/copilot/chat/global/core.test.ts b/frontend/src/lib/components/copilot/chat/global/core.test.ts index 50ad2547f0..a3b9c22499 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.test.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.test.ts @@ -64,6 +64,9 @@ vi.mock('$lib/gen', async () => { getScriptByPath: vi.fn(async () => { throw new Error('getScriptByPath mock not configured') }), + getScriptByHash: vi.fn(async () => { + throw new Error('getScriptByHash mock not configured') + }), queryHubScripts: vi.fn(async () => []), getHubScriptContentByPath: vi.fn(async () => ''), listScripts: vi.fn(async () => []) @@ -119,6 +122,9 @@ vi.mock('$lib/gen', async () => { getFlowByPath: vi.fn(async () => { throw new Error('getFlowByPath mock not configured') }), + getFlowVersion: vi.fn(async () => { + throw new Error('getFlowVersion mock not configured') + }), getFlowLatestVersion: vi.fn(async () => ({ id: 1 })), listFlows: vi.fn(async () => []) }), @@ -141,6 +147,9 @@ vi.mock('$lib/gen', async () => { getAppByPath: vi.fn(async () => { throw new Error('getAppByPath mock not configured') }), + getAppByVersion: vi.fn(async () => { + throw new Error('getAppByVersion mock not configured') + }), listApps: vi.fn(async () => []) }), ResourceService: wrapService(actual.ResourceService, { @@ -157,6 +166,9 @@ vi.mock('$lib/gen', async () => { createVariable: vi.fn(async () => 'created'), updateVariable: vi.fn(async () => 'updated') }), + FolderService: wrapService(actual.FolderService, { + createFolder: vi.fn(async () => 'created') + }), DraftService: wrapService(actual.DraftService, { updateDraft: vi.fn(async ({ kind, path, requestBody }: any) => { const key = `${kind}:${path}` @@ -209,6 +221,13 @@ vi.mock('./rawAppBundlerBridge', () => ({ })) })) +vi.mock('$lib/infer', async () => ({ + ...(await vi.importActual('$lib/infer')), + // Avoid the wasm parser in unit tests: the script deploy path infers the arg + // schema but tolerates failure, and these tests don't assert on the schema. + inferArgs: vi.fn(async () => {}) +})) + import { globalTools, globalToolsFor, @@ -233,6 +252,7 @@ import { bundleRawAppDraft } from './rawAppBundlerBridge' import { AppService, FlowService, + FolderService, HttpTriggerService, JobService, ResourceService, @@ -240,6 +260,8 @@ import { ScriptService, VariableService } from '$lib/gen' +import { userStore } from '$lib/stores' +import { get } from 'svelte/store' import type { Tool, ToolCallbacks } from '../shared' const WORKSPACE = 'global-core-test' @@ -605,6 +627,178 @@ describe('global AI tools', () => { expect(VariableService.updateVariable).not.toHaveBeenCalled() }) + it('deploys every field of a script draft (not just content/summary)', async () => { + // The deploy delegates to the shared deployer, which reads the full persisted + // draft via getScriptByPath(getDraft) and deploys all of it. Config fields + // (tag/priority/schema/description/concurrency) were previously dropped, + // sourced from the deployed version instead. + seedBackendDraft( + 'script', + 'f/scripts/full', + { path: 'f/scripts/full', content: 'export async function main() {}', language: 'bun' }, + { workspace: WORKSPACE } + ) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ + hash: 1234, + path: 'f/scripts/full', + summary: 'Full script', + description: 'desc', + content: 'export async function main() {}', + schema: { foo: 'bar' }, + language: 'bun', + kind: 'script', + tag: 'custom-tag', + priority: 7, + concurrent_limit: 3, + draft_only: true + } as any) + + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/full' }) + + expect(ScriptService.createScript).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'f/scripts/full', + content: 'export async function main() {}', + summary: 'Full script', + description: 'desc', + schema: { foo: 'bar' }, + language: 'bun', + tag: 'custom-tag', + priority: 7, + concurrent_limit: 3, + parent_hash: 1234 + }) + }) + // Editor-only / server-managed draft keys must not leak into the deploy body. + const calls = vi.mocked(ScriptService.createScript).mock.calls + const body = calls[calls.length - 1][0].requestBody as any + expect(body.draft_only).toBeUndefined() + }) + + it('deploys every config field of a flow draft via createFlow', async () => { + seedBackendDraft( + 'flow', + 'f/flows/full', + { summary: 'Full flow', description: 'flow desc', value: { modules: [] }, schema: {} }, + { workspace: WORKSPACE } + ) + vi.mocked(FlowService.getFlowByPath).mockResolvedValueOnce({ + path: 'f/flows/full', + summary: 'Full flow', + description: 'flow desc', + value: { modules: [] }, + schema: { x: 1 }, + tag: 'flow-tag', + dedicated_worker: true + } as any) + + await callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/full' }) + + // No deployed flow row (existsFlowByPath defaults to false) → create. + expect(FlowService.createFlow).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'f/flows/full', + summary: 'Full flow', + description: 'flow desc', + value: { modules: [] }, + schema: { x: 1 }, + tag: 'flow-tag', + dedicated_worker: true + }) + }) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + }) + + it('deploys an editor draft_only script at its chosen path, not its synthetic storage key', async () => { + // A new script created in the editor lives at a synthetic `u/{user}/draft_{uuid}` + // storage key while its chosen path is in the draft value. The chat addresses + // it by the chosen (display) path; deploy must resolve to the storage key so the + // shared deployer can read the draft via getScriptByPath, then deploy at the + // chosen path. Reading at the chosen path would 404. + const storageKey = 'u/admin/draft_abc123' + const chosenPath = 'f/team/chosen_path' + seedBackendDraft( + 'script', + storageKey, + { + path: chosenPath, + summary: 'New script', + description: '', + content: 'export async function main() {}', + schema: {}, + is_template: false, + language: 'bun', + kind: 'script' + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'script', + storagePath: storageKey, + effectivePath: chosenPath + }) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ + path: chosenPath, + summary: 'New script', + description: '', + content: 'export async function main() {}', + schema: {}, + language: 'bun', + kind: 'script' + } as any) + + const flushSpy = vi.spyOn(UserDraftDbSyncer, 'flush') + + await callGlobalTool('deploy_workspace_item', { type: 'script', path: chosenPath }) + + // Any pending editor autosave is flushed at the storage key before delegating, + // so the shared deployer reads the latest value, not a stale persisted draft. + expect(flushSpy).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, itemKind: 'script', path: storageKey }) + ) + // The draft is read at the STORAGE key (the chosen path would 404)… + expect(ScriptService.getScriptByPath).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, path: storageKey, getDraft: true }) + ) + // …and deployed at the chosen path. + expect(ScriptService.createScript).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ path: chosenPath }) + }) + }) + + it('aborts deploy when the pre-deploy draft flush hit a conflict', async () => { + // flush() resolves even when the save recorded a conflict; deploy must abort + // rather than publish the stale persisted draft. + seedBackendDraft( + 'script', + 'f/scripts/conflicted', + { + path: 'f/scripts/conflicted', + summary: '', + description: '', + content: 'export async function main() {}', + schema: {}, + is_template: false, + language: 'bun', + kind: 'script' + }, + { workspace: WORKSPACE } + ) + const conflictSpy = vi + .spyOn(UserDraftDbSyncer, 'getConflict') + .mockReturnValue({ conflict: { serverTimestamp: '2026', localLastSync: null } } as any) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/conflicted' }) + ).rejects.toThrow(/conflicting/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + conflictSpy.mockRestore() + }) + it('writes script drafts into UserDraft', async () => { const content = 'export async function main() {\n\treturn "hello"\n}' @@ -1009,6 +1203,260 @@ describe('global AI tools', () => { }) }) + describe('stale-draft deploy guard and rebase', () => { + // The suite's beforeEach only clears mock calls (not implementations), so + // restore the script-service mocks these tests override back to their factory + // defaults; otherwise a persistent resolved value leaks into later tests. + afterEach(() => { + vi.mocked(ScriptService.existsScriptByPath).mockResolvedValue(false) + vi.mocked(ScriptService.getScriptByPath).mockImplementation(async () => { + throw new Error('getScriptByPath mock not configured') + }) + vi.mocked(ScriptService.getScriptByHash).mockImplementation(async () => { + throw new Error('getScriptByHash mock not configured') + }) + vi.mocked(FlowService.existsFlowByPath).mockResolvedValue(false) + vi.mocked(FlowService.getFlowByPath).mockImplementation(async () => { + throw new Error('getFlowByPath mock not configured') + }) + vi.mocked(FlowService.getFlowVersion).mockImplementation(async () => { + throw new Error('getFlowVersion mock not configured') + }) + vi.mocked(FlowService.getFlowLatestVersion).mockResolvedValue({ id: 1 } as any) + vi.mocked(AppService.existsApp).mockResolvedValue(false) + vi.mocked(AppService.getAppByPath).mockImplementation(async () => { + throw new Error('getAppByPath mock not configured') + }) + vi.mocked(AppService.getAppByVersion).mockImplementation(async () => { + throw new Error('getAppByVersion mock not configured') + }) + }) + + function seedStaleScriptDraft(path: string, parentHash: string, content = 'draft content') { + seedBackendDraft('script', path, { + path, + summary: 's', + description: '', + content, + language: 'bun', + kind: 'script', + parent_hash: parentHash, + schema: {} + }) + } + + function mockDeployedScript(path: string, hash: string, content = 'latest deployed') { + vi.mocked(ScriptService.existsScriptByPath).mockResolvedValue(true) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValue({ + path, + hash, + content, + language: 'bun', + summary: 's' + } as any) + } + + it('blocks deploying a script draft started from an older deployed version', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash') + mockDeployedScript('f/scripts/stale', 'new-hash') + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + }) + + it('deploys a stale script draft when force is set', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash') + mockDeployedScript('f/scripts/stale', 'new-hash') + + const result = JSON.parse( + await callGlobalTool('deploy_workspace_item', { + type: 'script', + path: 'f/scripts/stale', + force: true + }) + ) + expect(result.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + it('deploys a script draft that is based on the current deployed head', async () => { + seedStaleScriptDraft('f/scripts/fresh', 'head-hash') + mockDeployedScript('f/scripts/fresh', 'head-hash') + + const result = JSON.parse( + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/fresh' }) + ) + expect(result.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + it('rebase_draft discards the stale draft and surfaces the changes to replay', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash', 'base content\nmy added line\n') + mockDeployedScript('f/scripts/stale', 'new-hash', 'latest deployed content\n') + vi.mocked(ScriptService.getScriptByHash).mockResolvedValue({ + hash: 'base-hash', + content: 'base content\n', + language: 'bun' + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'script', path: 'f/scripts/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_hash).toBe('new-hash') + // The diff surfaces the draft's own change over its fork base. + expect(result.your_changes).toContain('my added line') + + // The stale draft is discarded (not reset), so a premature deploy fails + // cleanly rather than silently shipping the latest unchanged. + expect(getBackendDraft('script', 'f/scripts/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + + // Re-applying re-bases onto the current head; the deploy then passes. + await callGlobalTool('write_script', { + path: 'f/scripts/stale', + summary: 's', + language: 'bun', + content: 'latest deployed content\nmy added line\n' + }) + const deploy = JSON.parse( + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ) + expect(deploy.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + function seedStaleFlowDraft(path: string, versionId: number, modules: any[] = []) { + seedBackendDraft('flow', path, { + path, + summary: 'f', + description: '', + version_id: versionId, + value: { modules }, + schema: {} + }) + } + + function mockDeployedFlow(path: string, versionId: number) { + vi.mocked(FlowService.existsFlowByPath).mockResolvedValue(true) + vi.mocked(FlowService.getFlowByPath).mockResolvedValue({ + path, + summary: 'f', + version_id: versionId, + value: { modules: [] }, + schema: {} + } as any) + } + + it('blocks deploying a flow draft started from an older deployed version', async () => { + seedStaleFlowDraft('f/flows/stale', 1) + mockDeployedFlow('f/flows/stale', 2) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + expect(FlowService.createFlow).not.toHaveBeenCalled() + }) + + it('rebase_draft discards the stale flow draft and surfaces the changes to replay', async () => { + seedStaleFlowDraft('f/flows/stale', 1, [{ id: 'a', value: { type: 'identity' } }]) + mockDeployedFlow('f/flows/stale', 2) + vi.mocked(FlowService.getFlowVersion).mockResolvedValue({ + value: { modules: [] } + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'flow', path: 'f/flows/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_version).toBe(2) + expect(result.your_changes).toContain('identity') + + // The stale draft is discarded, so a premature deploy fails cleanly. + expect(getBackendDraft('flow', 'f/flows/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + }) + + function seedStaleAppDraft(path: string, parentVersion: number, file = 'old') { + seedBackendDraft('raw_app', path, { + summary: 'a', + files: { '/index.tsx': file }, + runnables: {}, + data: { tables: [] }, + parent_version: parentVersion + }) + } + + function mockDeployedApp(path: string, versionId: number, file = 'latest') { + vi.mocked(AppService.existsApp).mockResolvedValue(true) + vi.mocked(AppService.getAppByPath).mockResolvedValue({ + path, + summary: 'a', + versions: [versionId], + value: { files: { '/index.tsx': file }, runnables: {}, data: { tables: [] } }, + policy: { execution_mode: 'publisher' } + } as any) + } + + it('grafts the fork-base version onto a new app draft and keeps it through the save whitelist', async () => { + // No draft yet: the first app edit projects the deployed app into a draft. + // This exercises the runtime path types can't catch — the graft in + // appSourceToDraftValue AND survival through normalizeAppDraftValue's whitelist. + mockDeployedApp('f/apps/fresh', 5) + + await callGlobalTool('write_app_file', { + path: 'f/apps/fresh', + file_path: '/src/New.tsx', + content: 'export default function New() { return null }' + }) + + const draft = getBackendDraft('raw_app', 'f/apps/fresh', { workspace: WORKSPACE }) + expect(draft.parent_version).toBe(5) + }) + + it('blocks deploying an app draft started from an older deployed version', async () => { + seedStaleAppDraft('f/apps/stale', 1) + mockDeployedApp('f/apps/stale', 2) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(AppService.createAppRaw).not.toHaveBeenCalled() + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + + it('rebase_draft discards the stale app draft and surfaces the changes to replay', async () => { + seedStaleAppDraft('f/apps/stale', 1, 'my-change') + mockDeployedApp('f/apps/stale', 2, 'latest-deployed') + vi.mocked(AppService.getAppByVersion).mockResolvedValue({ + value: { files: { '/index.tsx': 'base' }, runnables: {}, data: { tables: [] } } + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'app', path: 'f/apps/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_version).toBe(2) + expect(result.your_changes).toContain('my-change') + + // The stale draft is discarded, so a premature deploy fails cleanly. + expect(getBackendDraft('raw_app', 'f/apps/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + }) + it('preserves existing flow metadata and seeds freshness on first flow write', async () => { vi.mocked(FlowService.existsFlowByPath).mockResolvedValueOnce(true) vi.mocked(FlowService.getFlowLatestVersion).mockResolvedValueOnce({ id: 42 } as any) @@ -1383,9 +1831,7 @@ describe('global AI tools', () => { file_path: '/min.tsx' }) - expect(result).toContain( - 'lines 1-3 of 3, truncated to the first 50000 of 90002 chars.' - ) + expect(result).toContain('lines 1-3 of 3, truncated to the first 50000 of 90002 chars.') expect(result).toContain('the file is likely minified') expect(result.split('\n\n')[1]).toHaveLength(50_000) }) @@ -1401,9 +1847,7 @@ describe('global AI tools', () => { file_path: '/generated.js' }) - expect(result).toContain( - 'lines 1-1 of 1, truncated to the first 50000 of 60000 chars.' - ) + expect(result).toContain('lines 1-1 of 1, truncated to the first 50000 of 60000 chars.') expect(result).toContain('re-read with a smaller limit') expect(result.split('\n\n')[1]).toBe('x'.repeat(50_000)) }) @@ -1488,8 +1932,7 @@ describe('global AI tools', () => { versions: [5], value: { files: { - '/lib/aggregations.ts': - 'export function computeRevenue(o) {\n return o.unitPrice\n}\n', + '/lib/aggregations.ts': 'export function computeRevenue(o) {\n return o.unitPrice\n}\n', '/components/SummaryPanel.tsx': 'import { computeRevenue } from "../lib/aggregations"\nconst total = computeRevenue(order)\n', '/components/OrdersTable.tsx': 'const r = computeRevenue(row)\n// renders revenue\n', @@ -1787,6 +2230,9 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + // getAppByPath resolves with no draft_path → deploy at the item's own path. + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + const raw = await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/report', @@ -1832,6 +2278,93 @@ describe('global AI tools', () => { }) }) + it('deploys an editor raw app draft at its draft_path, not its synthetic storage key', async () => { + // An editor-created draft_only raw app lives at a synthetic storage key with + // its chosen path in `draft_path`; deploy must resolve to the storage key, + // read draft_path, and create the app there — not at the synthetic key. + const storageKey = 'u/admin/draft_app999' + const chosenPath = 'f/team/chosen_app' + seedBackendDraft( + 'raw_app', + storageKey, + { + summary: 'Editor app', + files: { '/App.tsx': 'export default () => null' }, + runnables: {}, + data: { tables: [] }, + draft_path: chosenPath + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: storageKey, + effectivePath: chosenPath + }) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ + draft: { draft_path: chosenPath } + } as any) + const flushSpy = vi.spyOn(UserDraftDbSyncer, 'flush') + + await callGlobalTool('deploy_workspace_item', { type: 'app', path: chosenPath }) + + // The draft is flushed at the storage key before the draft_path read, so a + // not-yet-saved editor rename isn't read stale. + expect(flushSpy).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, itemKind: 'raw_app', path: storageKey }) + ) + // draft_path is read from the backend draft at the storage key… + expect(AppService.getAppByPath).toHaveBeenCalledWith( + expect.objectContaining({ + workspace: WORKSPACE, + path: storageKey, + getDraft: true, + rawApp: true + }) + ) + // …and the app is created at the chosen path, not the synthetic key. + expect(AppService.createAppRaw).toHaveBeenCalledWith( + expect.objectContaining({ + formData: expect.objectContaining({ + app: expect.objectContaining({ path: chosenPath }) + }) + }) + ) + }) + + it('aborts a raw app deploy when the draft_path lookup fails (non-404)', async () => { + // A real lookup failure (network/5xx) must abort, not silently fall back to the + // storage path and deploy there. Only a 404 justifies the storage-path fallback. + seedBackendDraft( + 'raw_app', + 'u/admin/draft_appfail', + { + summary: 'Editor app', + files: { '/App.tsx': 'export default () => null' }, + runnables: {}, + data: { tables: [] }, + draft_path: 'f/team/chosen_app' + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: 'u/admin/draft_appfail', + effectivePath: 'f/team/chosen_app' + }) + vi.mocked(AppService.getAppByPath).mockRejectedValueOnce( + Object.assign(new Error('server error'), { status: 500 }) + ) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/team/chosen_app' }) + ).rejects.toThrow() + expect(AppService.createAppRaw).not.toHaveBeenCalled() + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + it('deploys an existing raw app draft by bundling files and updating the raw app', async () => { vi.mocked(AppService.existsApp).mockResolvedValueOnce(true) seedBackendDraft( @@ -1848,6 +2381,8 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/report' @@ -1876,6 +2411,40 @@ describe('global AI tools', () => { expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() }) + it('forwards preserve_on_behalf_of when the deployed policy carries an on_behalf_of', async () => { + // Without the flag the backend resets the policy's on_behalf_of to the + // deploying user; this chat path has no on-behalf-of selector, so it must + // preserve whatever the carried policy already holds. + vi.mocked(AppService.existsApp).mockResolvedValueOnce(true) + seedBackendDraft( + 'raw_app', + 'f/apps/obo', + { + summary: 'On-behalf app', + files: { '/index.tsx': 'console.log("obo")' }, + runnables: {}, + data: { tables: [] }, + policy: { + execution_mode: 'publisher', + on_behalf_of: 'u/alice', + on_behalf_of_email: 'alice@windmill.dev' + } + }, + { workspace: WORKSPACE } + ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + + await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/obo' }) + + expect(AppService.updateAppRaw).toHaveBeenCalledWith( + expect.objectContaining({ + formData: expect.objectContaining({ + app: expect.objectContaining({ preserve_on_behalf_of: true }) + }) + }) + ) + }) + it('notifies the session preview (as raw_app) after deploying a raw app', async () => { const onDeployed = vi.fn() setDeployedInSessionHandler(onDeployed) @@ -1892,6 +2461,8 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + await callGlobalTool( 'deploy_workspace_item', { type: 'app', path: 'f/apps/report' }, @@ -2477,6 +3048,54 @@ describe('global AI tools', () => { }) }) +describe('folder tools', () => { + beforeEach(() => { + vi.clearAllMocks() + userStore.set(undefined) + }) + afterEach(() => { + userStore.set(undefined) + }) + + it('create_folder requires confirmation', () => { + const tool = getGlobalTool('create_folder') + expect(tool.requiresConfirmation).toBe(true) + expect(tool.confirmationMessage).toBe('Create folder') + }) + + it('create_folder creates the folder and reflects it in the path context', async () => { + userStore.set({ username: 'bob', is_admin: false, folders: ['existing'] } as any) + const raw = await callGlobalTool('create_folder', { name: 'analytics', summary: 'team data' }) + + expect(vi.mocked(FolderService.createFolder)).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: { name: 'analytics', summary: 'team data' } + }) + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(true) + expect(parsed.message).toContain('f/analytics') + expect((get(userStore) as any)?.folders).toContain('analytics') + }) + + it('create_folder rejects an invalid name without calling the API', async () => { + const raw = await callGlobalTool('create_folder', { name: 'bad name!' }) + expect(vi.mocked(FolderService.createFolder)).not.toHaveBeenCalled() + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(false) + expect(parsed.error).toContain('alphanumeric') + }) + + it('create_folder surfaces a backend error (e.g. name conflict)', async () => { + vi.mocked(FolderService.createFolder).mockRejectedValueOnce( + new Error('Folder already exists') + ) + const raw = await callGlobalTool('create_folder', { name: 'taken' }) + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(false) + expect(parsed.error).toContain('Folder already exists') + }) +}) + describe('prepareGlobalSystemMessage', () => { it('keeps global chat draft instructions concise and user-facing', () => { const message = prepareGlobalSystemMessage() @@ -2496,6 +3115,104 @@ describe('prepareGlobalSystemMessage', () => { expect(content).not.toContain('frontend AI draft store') }) + it('honors user-supplied shared folder paths without asking first', () => { + const content = prepareGlobalSystemMessage(undefined, { + user: { username: 'admin', is_admin: true, folders: ['evals'] } + }).content as string + + expect(content).toContain( + 'If the user supplies a fully qualified `f//...` path, use that exact path' + ) + expect(content).toContain('Do not ask for folder confirmation') + expect(content).toContain('substitute a `u/admin/...` path unless a tool rejects it') + }) + + it('tells the model to create a folder only when the user explicitly asks', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).toContain( + 'create one with `create_folder` only when the user explicitly asks for a new folder' + ) + }) + + describe('folder guidance', () => { + const guidanceOf = (user: { + username: string + is_admin?: boolean + folders?: string[] + folders_read?: string[] + }) => prepareGlobalSystemMessage(undefined, { user }).content as string + + it('lists the writable folders for a non-admin', () => { + const content = guidanceOf({ + username: 'bob', + is_admin: false, + folders: ['marketing', 'data_engineering'], + folders_read: ['marketing', 'data_engineering'] + }) + expect(content).toContain( + 'Folders you can write to in this workspace: `f/marketing`, `f/data_engineering`.' + ) + expect(content).not.toContain('You can see but NOT write to') + }) + + it('flags read-only folders a non-admin cannot write to', () => { + const content = guidanceOf({ + username: 'bob', + is_admin: false, + folders: ['team_a'], + folders_read: ['team_a', 'team_b'] + }) + expect(content).toContain('Folders you can write to in this workspace: `f/team_a`.') + expect(content).toContain( + 'You can see but NOT write to: `f/team_b` — never create or deploy items there.' + ) + }) + + it('points a non-admin with no writable folders at the personal scope', () => { + const content = guidanceOf({ username: 'bob', is_admin: false, folders: [] }) + expect(content).toContain( + 'You have no shared folders you can write to in this workspace, so use `u/bob/`.' + ) + }) + + it('gives an admin permission-agnostic guidance with a non-exhaustive hint', () => { + const content = guidanceOf({ + username: 'admin', + is_admin: true, + folders: ['marketing', 'data_engineering'] + }) + expect(content).toContain('As a workspace admin you can write to any existing folder.') + expect(content).toContain( + 'Folders here include `f/marketing`, `f/data_engineering` (you can also write to others not listed).' + ) + expect(content).toContain( + 'If the user names a folder, use it; if they explicitly ask for a new folder, create it with `create_folder`; otherwise ask them which folder to use rather than guessing or creating one unprompted.' + ) + expect(content).not.toContain('Folders you can write to in this workspace') + }) + + it('omits the folder hint for an admin with no associated folders', () => { + const content = guidanceOf({ username: 'admin', is_admin: true, folders: [] }) + expect(content).toContain( + '- As a workspace admin you can write to any existing folder. If the user names a folder, use it; if they explicitly ask for a new folder, create it with `create_folder`; otherwise ask them which folder to use rather than guessing or creating one unprompted.' + ) + expect(content).not.toContain('Folders here include') + }) + + it('caps the folder list and notes the remainder', () => { + const folders = Array.from({ length: 45 }, (_, i) => `f${i}`) + const content = guidanceOf({ username: 'bob', is_admin: false, folders }) + expect(content).toContain('(+5 more)') + }) + + it('emits no folder guidance when no user is available', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).not.toContain('Folders you can write to in this workspace') + expect(content).not.toContain('As a workspace admin you can write to any existing folder') + expect(content).not.toContain('You have no shared folders you can write to') + }) + }) + it('exposes separate tools for discarding drafts and deleting workspace items', () => { const discard = getGlobalTool('discard_local_draft') const deleteItem = getGlobalTool('delete_workspace_item') @@ -2668,6 +3385,156 @@ describe('session-only preview tools gating', () => { expect(on).toContain('get_app_runtime_logs') expect(on).toContain('list_app_runs') }) + + // The instruction headers are matched by their distinctive parenthetical so the + // guidance bullet (which references both block names) doesn't false-positive. + const WS_HEADER = 'WORKSPACE INSTRUCTIONS (configured by a workspace admin' + const USER_HEADER = "USER INSTRUCTIONS (this user's personal instructions" + + it('renders only the workspace block when given workspace instructions', () => { + const content = prepareGlobalSystemMessage({ workspace: 'Always be terse.' }).content as string + expect(content).toContain(WS_HEADER) + expect(content).toContain('Always be terse.') + expect(content).not.toContain(USER_HEADER) + }) + + it('renders the user block with the edit-tool mention when given user instructions', () => { + const content = prepareGlobalSystemMessage({ user: 'Prefer Bun for new scripts.' }) + .content as string + expect(content).toContain(USER_HEADER) + expect(content).toContain('update_user_instructions') + expect(content).toContain('Prefer Bun for new scripts.') + expect(content).not.toContain(WS_HEADER) + }) + + it('renders the workspace block before the user block when both are present', () => { + const content = prepareGlobalSystemMessage({ workspace: 'WS rule.', user: 'User rule.' }) + .content as string + expect(content).toContain(WS_HEADER) + expect(content.indexOf(USER_HEADER)).toBeGreaterThan(content.indexOf(WS_HEADER)) + }) + + it('omits both instruction headers when none are provided', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).not.toContain(WS_HEADER) + expect(content).not.toContain(USER_HEADER) + }) +}) + +describe('update_user_instructions', () => { + function makeHelpers(initial = '') { + let value = initial + return { + getUserInstructions: () => value, + setUserInstructions: vi.fn((v: string) => { + value = v + }) + } + } + + it('appends to empty instructions', async () => { + const helpers = makeHelpers('') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Prefer Bun for new scripts.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Prefer Bun for new scripts.') + expect(res).toContain('Added a personal instruction') + }) + + it('appends to existing instructions joined by a blank line', async () => { + const helpers = makeHelpers('Existing rule.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Another rule.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Existing rule.\n\nAnother rule.') + }) + + it('returns only a short confirmation, not the resulting instructions', async () => { + const helpers = makeHelpers('Existing rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Another rule.' }, + toolCallbacks, + helpers + ) + expect(res).not.toContain('Existing rule.') + expect(res).not.toContain('Another rule.') + }) + + it('replaces an exact match', async () => { + const helpers = makeHelpers('Prefer Bun.\n\nUse tabs.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'Prefer Bun.', new_string: 'Prefer Deno.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Prefer Deno.\n\nUse tabs.') + }) + + it('removes the matched text when new_string is empty', async () => { + const helpers = makeHelpers('Keep this.\n\nDrop this.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: '\n\nDrop this.', new_string: '' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Keep this.') + }) + + it('clears all instructions when the whole text is replaced with empty', async () => { + const helpers = makeHelpers('Only rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'Only rule.', new_string: '' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('') + expect(res).toContain('Cleared your personal instructions') + }) + + it('errors without writing when old_string is not found, and echoes the current text for recovery', async () => { + const helpers = makeHelpers('Existing rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'missing', new_string: 'x' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).not.toHaveBeenCalled() + expect(res).toContain('not found') + expect(res).toContain('Existing rule.') + }) + + it('rejects a result over the length cap without writing', async () => { + const helpers = makeHelpers('') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'a'.repeat(5001) }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).not.toHaveBeenCalled() + expect(res).toContain('over the 5000') + }) + + it('fails gracefully when the context does not provide instruction helpers', async () => { + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'x' }, + toolCallbacks, + {} + ) + expect(res).toContain('cannot modify user instructions') + }) }) describe('prepareGlobalUserMessage', () => { @@ -2705,15 +3572,23 @@ describe('prepareGlobalUserMessage', () => { path: 'f/flows/reporting', title: 'f/flows/reporting', summary: 'Reporting flow' + }, + { + type: 'workspace_app', + path: 'f/apps/dashboard', + title: 'f/apps/dashboard', + summary: 'Dashboard raw app' } ]) expect(message.content).toContain('## SELECTED CONTEXT') expect(message.content).toContain('- type: script, path: f/scripts/report') expect(message.content).toContain('- type: flow, path: f/flows/reporting') + expect(message.content).toContain('- type: raw_app, path: f/apps/dashboard') expect(message.content).toContain('## INSTRUCTIONS:\nUpdate these items') expect(message.content).not.toContain('Report script') expect(message.content).not.toContain('Reporting flow') + expect(message.content).not.toContain('Dashboard raw app') }) it('omits selected context section when no workspace item is selected', () => { diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index 90925255d2..b85e271632 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -2,6 +2,7 @@ import { AppService, AzureTriggerService, FlowService, + FolderService, GcpTriggerService, HttpTriggerService, JobService, @@ -14,8 +15,10 @@ import { ScriptService, SqsTriggerService, VariableService, - WebsocketTriggerService + WebsocketTriggerService, + WorkspaceService } from '$lib/gen' +import { createTwoFilesPatch } from 'diff' import { $ScriptLang } from '$lib/gen/schemas.gen' import type { AppWithLastVersion, @@ -76,6 +79,8 @@ import { import { searchDocsTool, readDocsPageTool } from '../docs/core' import type { ContextElement } from '../context' import { getDatatableTools } from '../datatableTools' +import { fileTools } from '../files/fileTools' +import type { AttachedFilesStore } from '../files/attachedFiles.svelte' import { UserDraft } from '$lib/userDraft.svelte' import { emptySchema } from '$lib/utils' import { inferArgs } from '$lib/infer' @@ -97,9 +102,10 @@ import { type WorkspaceItem, type WorkspaceItemType } from './workspaceItems' -import { buildFlowDeployRequestBody, buildScriptDeployRequestBody } from './deployRequests' import { userStore } from '$lib/stores' import { get } from 'svelte/store' +import { deployDraft as deployDraftToWorkspace } from '$lib/utils_draft_deploy' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { bundleRawAppDraft } from './rawAppBundlerBridge' import { clearEphemeralSecretVariableDraftValue, @@ -186,6 +192,43 @@ const askUserQuestionSchema = z.object({ .describe('Two to ten mutually exclusive proposed answer strings.') }) +// Matches the per-mode cap enforced by the prompt-settings UI (AIPromptsModal) and +// the backend workspace prompt (MAX_CUSTOM_PROMPT_LENGTH). +export const MAX_USER_INSTRUCTIONS_LENGTH = 5000 + +const updateUserInstructionsSchema = z.object({ + operation: z + .enum(['append', 'replace']) + .describe( + 'What to do with your personal Global instructions. \'append\' adds a new instruction to the end (use for "remember this" / "always do X"). \'replace\' performs an exact find-and-replace to edit or remove existing text (use for "change X" / "stop doing Y"); set new_string to "" to remove.' + ), + text: z + .string() + .min(1) + .optional() + .describe("Required when operation is 'append': the instruction to add. Ignored for 'replace'."), + old_string: z + .string() + .min(1) + .optional() + .describe( + "Required when operation is 'replace': exact text to find in your current personal instructions. Ignored for 'append'." + ), + new_string: z + .string() + .optional() + .describe( + "Required when operation is 'replace': replacement text. Use an empty string to delete the matched text. Ignored for 'append'." + ), + replace_all: z + .boolean() + .optional() + .default(false) + .describe( + "For operation 'replace': when true, replace every exact match; when false, old_string must match exactly once." + ) +}) + const listWorkspaceItemsSchema = z.object({ types: z .array(itemTypeSchema) @@ -413,7 +456,20 @@ const deployWorkspaceItemSchema = z.object({ deployment_message: z .string() .optional() - .describe('Optional deployment message recorded with the change.') + .describe('Optional deployment message recorded with the change.'), + force: z + .boolean() + .optional() + .describe( + 'Deploy even if the draft was started from an older deployed version, overwriting the version deployed since. Defaults to false; prefer calling rebase_draft first to keep the newer changes.' + ) +}) + +const rebaseDraftSchema = z.object({ + type: itemTypeSchema, + path: z + .string() + .describe('Workspace path of the draft to rebase onto the latest deployed version.') }) const editScriptSchema = z.object({ @@ -678,22 +734,87 @@ const initAppSchema = z.object({ ) }) +// Mirrors the backend VALID_FOLDER_NAME check so an invalid name fails before the +// network round-trip (the server enforces the same rule). +const VALID_FOLDER_NAME = /^[a-zA-Z_0-9-]+$/ + +const createFolderSchema = z.object({ + name: z + .string() + .describe( + 'Folder name — letters, digits, underscores or hyphens only. The folder becomes addressable as `f//`.' + ), + summary: z.string().optional().describe('Optional human-readable description of the folder.') +}) + +type FolderPromptContext = { folders: string[]; foldersRead: string[]; isAdmin: boolean } + +// Renders the folders the current user can act on into the system prompt so the +// model can pick an `f//...` path without a discovery round-trip (there +// is no folder-listing tool). For a non-admin, `folders` (from whoami) is exactly +// the writable set, so read-only folders are listed separately as off-limits. +// Admins bypass folder ACLs and can write anywhere, but `folders` only carries +// their explicitly-permissioned subset, so for admins it is offered as a +// non-exhaustive hint alongside permission-agnostic guidance (the complete set +// needs a folder-listing tool — follow-up). +// Capped so a folder-heavy workspace can't dominate the prompt. +function buildFolderGuidance(username: string, ctx?: FolderPromptContext): string { + if (!ctx) return '' + const MAX = 40 + const writable = ctx.folders ?? [] + const fmt = (names: string[]) => { + const shown = names + .slice(0, MAX) + .map((n) => `\`f/${n}\``) + .join(', ') + return names.length > MAX ? `${shown} (+${names.length - MAX} more)` : shown + } + if (ctx.isAdmin) { + const known = + writable.length > 0 + ? ` Folders here include ${fmt(writable)} (you can also write to others not listed).` + : '' + return `- As a workspace admin you can write to any existing folder.${known} If the user names a folder, use it; if they explicitly ask for a new folder, create it with \`create_folder\`; otherwise ask them which folder to use rather than guessing or creating one unprompted.` + } + const readOnly = (ctx.foldersRead ?? []).filter((f) => !writable.includes(f)) + const lines: string[] = [] + if (writable.length > 0) { + lines.push( + `- Folders you can write to in this workspace: ${fmt(writable)}. For shared/team work, pick the one whose purpose matches the request; if none clearly fits, ask which folder to use (askUserQuestion) rather than inventing a path. Use \`create_folder\` only when the user explicitly asks for a new folder.` + ) + } else { + lines.push( + `- You have no shared folders you can write to in this workspace, so use \`u/${username}/\`. If the user explicitly asks for a shared folder, create one with \`create_folder\` (you become an owner); otherwise ask before placing shared work rather than inventing an \`f//...\` path.` + ) + } + if (readOnly.length > 0) { + lines.push( + `- You can see but NOT write to: ${fmt(readOnly)} — never create or deploy items there.` + ) + } + return lines.join('\n') +} + const buildGlobalSystemPrompt = ( username: string, - previewTools: boolean -) => `You are Windmill's global workspace assistant. + previewTools: boolean, + folderCtx?: FolderPromptContext, + skills: AiSkillListItem[] = [] +) => { + const folderGuidance = buildFolderGuidance(username, folderCtx) + const folderGuidanceBlock = folderGuidance ? `\n${folderGuidance}` : '' + return `You are Windmill's global workspace assistant. The current user's workspace username is "${username}". Use tools to inspect workspace items and create per-user drafts (saved server-side, visible only to this user — not deployed) for scripts, flows, schedules, triggers, resources, variables, and raw apps. Path conventions: -- Every workspace path has exactly three segments and starts with one of two namespaces: - - \`u/${username}/\` — the current user's personal scope. Default for ad-hoc, exploratory, or scratch work. - - \`f//\` — a shared folder scope. The folder must already exist; bare \`f/\` is INVALID and will fail. -- When the user gives a bare name without a namespace prefix (e.g. "create a flow called myflow"), default to \`u/${username}/\`. Do NOT invent \`f/\` — that is a structurally invalid path. -- If the request implies shared / team work but doesn't name a specific folder (e.g. "the marketing flow"), ask which folder to use rather than guessing. Call \`list_workspace_items\` with \`type: ['folder']\` (or rely on the user's hint) before assuming a folder exists. -- Only use an \`f//\` path when the user explicitly named the folder or you confirmed it exists. +- A workspace path starts with one of two namespaces; its trailing may itself contain "/", so a path has three or more segments: + - \`u/${username}/\` — your personal scope. Default for ad-hoc, exploratory, or scratch work. + - \`f//\` — a shared folder scope; the must already exist (a bare \`f/\` with no folder segment is INVALID and will fail). +- If the user supplies a fully qualified \`f//...\` path, use that exact path; they have already chosen the folder. Do not ask for folder confirmation or substitute a \`u/${username}/...\` path unless a tool rejects it. +- Default a bare name with no namespace prefix (e.g. "create a flow called myflow") to \`u/${username}/\`. Never invent an \`f//...\` path for a folder that does not exist; create one with \`create_folder\` only when the user explicitly asks for a new folder.${folderGuidanceBlock} Rules: - Draft tools create or update drafts only; they do not deploy or mutate deployed workspace items. @@ -707,14 +828,15 @@ Rules: - After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step with representative args before reporting that it works. These tools prefer drafts, so testing does not require deployment. - Use list_runs to find recent runs (optionally filtered by path, creator, label, or status), then get_job_logs with a returned id to inspect a specific run's logs — without starting a new test run. - When a required decision is ambiguous, use askUserQuestion with two to ten clear proposed answer strings instead of guessing. The user can also type a custom answer when none of the proposed answers fit. +- When the user asks you to remember a lasting preference, always/never do something, or change/stop a behavior going forward, call update_user_instructions to persist it. It edits only the USER INSTRUCTIONS block (not WORKSPACE INSTRUCTIONS). Keep each instruction concise; do not use it for one-off requests scoped to the current task. - Keep context targeted.${ - previewTools - ? ` + previewTools + ? ` - After writing or substantially editing a script / flow / app draft, show it via open_preview(kind, path) so the user sees the editor and live preview right next to the chat. First check whether it is already shown: if unsure, call get_preview_status. Only call open_preview (or offer to) when no preview is open or it is showing a different item — don't re-open a preview already showing the item you just edited. - When debugging a running raw app, call get_app_runtime_logs to read the live preview's browser console output. It needs the raw app preview open (open_preview kind="raw_app"). - get_app_runtime_logs only shows the app's browser console. For the server-side logs of a backend runnable the app invoked (a backend. call), call list_app_runs to get that run's job_id from the live preview, then get_job_logs with it. Use this when a backend call errors or returns something unexpected.` - : '' -} + : '' + } Documentation: - Use search_docs to look up how a Windmill feature works in the official documentation (a flag, concept, function, or "does Windmill support X") instead of guessing about product behavior. It returns matching doc snippets with their Source URL; call read_docs_page with a Source URL to read the full page (or a section, if it returns headings). Cite the Source URL when you rely on it. @@ -740,7 +862,17 @@ Data Tables: - Use list_datatables to discover the available datatables and their tables. Reuse an existing table rather than creating a duplicate. If list_datatables reports none, this is a blocking prerequisite — tell the user to set up a datatable in their workspace settings and stop; do not assume a "main" datatable exists or call exec_datatable_sql. - Use get_datatable_table_schema only when you need a table's column names/types; list_datatables is enough for table-list or availability summaries. - Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement — it appears in list_datatables afterward, with no registration step. -- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.` +- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.${ + skills.length > 0 + ? ` + +Skills: +- Skills are reusable instruction sets curated for this workspace, each covering a specific kind of task. The available skills are listed below by name and description. +- When a user's request matches a skill's description, call read_skill with its exact name to load the full instructions BEFORE acting, then follow them. +${skills.map((s) => `- ${s.name}: ${s.description}`).join('\n')}` + : '' + }` +} const DEFAULT_LIST_TYPES = ['script', 'flow'] as const satisfies readonly WorkspaceItemType[] @@ -1491,7 +1623,51 @@ function getInstructions(subject: InstructionSubject, language?: ScriptLang): st } } +export type AiSkillListItem = { name: string; description: string } + +/** Fetch the workspace's AI skills (name + description) for the global system prompt. */ +export async function loadWorkspaceSkills(workspace: string): Promise { + if (!workspace) return [] + try { + return await WorkspaceService.listAiSkills({ workspace }) + } catch (e) { + console.error('Failed to load AI skills', e) + return [] + } +} + +const readSkillSchema = z.object({ + name: z + .string() + .describe('The exact skill name as listed in the Skills section of the system prompt.') +}) + +export const readSkillTool: Tool<{}> = { + def: createToolDef( + readSkillSchema, + 'read_skill', + 'Load the full instructions for a workspace AI skill by name. Skills are listed in the system prompt under "Skills"; call this before acting on a task a skill covers, then follow its instructions.' + ), + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = readSkillSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Reading skill "${parsed.name}"...` }) + try { + const skill = await WorkspaceService.getAiSkill({ workspace, name: parsed.name }) + toolCallbacks.setToolStatus(toolId, { content: `Read skill "${parsed.name}"` }) + return `Skill: ${skill.name}\nDescription: ${skill.description}\n\nInstructions:\n${skill.instructions}` + } catch (e) { + const msg = e instanceof Error ? e.message : String(e) + toolCallbacks.setToolStatus(toolId, { + content: `Error reading skill "${parsed.name}"`, + error: msg + }) + return `Failed to read skill "${parsed.name}": ${msg}. Check the name against the Skills list in the system prompt.` + } + } +} + export const globalTools: Tool<{}>[] = [ + readSkillTool, { def: createToolDef( getInstructionsSchema, @@ -1565,6 +1741,79 @@ export const globalTools: Tool<{}>[] = [ return selectedChoice } }, + { + def: createToolDef( + updateUserInstructionsSchema, + 'update_user_instructions', + 'Modify your own persistent personal instructions for the Global assistant. Use when the user asks you to remember a preference, always/never do something, or change/stop a behavior. Edits only the user-level instructions (the USER INSTRUCTIONS block, not workspace-level); changes persist in this browser and take effect on your next message.' + ), + showDetails: true, + fn: async ({ args, toolId, toolCallbacks, helpers }) => { + const parsed = updateUserInstructionsSchema.parse(args) + const h = helpers as GlobalToolHelpers + if (!h?.getUserInstructions || !h?.setUserInstructions) { + const message = 'This chat context cannot modify user instructions.' + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + + const current = h.getUserInstructions() + let next: string + if (parsed.operation === 'append') { + const text = parsed.text?.trim() + if (!text) { + const message = "operation 'append' requires a non-empty text." + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + next = current.trim() ? `${current.trim()}\n\n${text}` : text + } else { + if (parsed.old_string === undefined || parsed.new_string === undefined) { + const message = "operation 'replace' requires old_string and new_string." + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + try { + next = findAndReplace( + current, + parsed.old_string, + parsed.new_string, + parsed.replace_all ?? false, + 'personal instructions' + ).trim() + } catch (e) { + const detail = e instanceof Error ? e.message : String(e) + // Echo the current text on this rare recovery path so the model can rebuild a + // correct old_string. The system prompt's USER INSTRUCTIONS block can be stale + // within a turn that already made a successful edit, but `current` is always live. + const hint = current.trim() + ? `Your current personal instructions are:\n${current.trim()}` + : "You have no personal instructions yet; use operation 'append' to add one." + const message = `${detail} ${hint}` + toolCallbacks.setToolStatus(toolId, { content: detail, error: detail }) + return message + } + } + + if (next.length > MAX_USER_INSTRUCTIONS_LENGTH) { + const message = `Resulting instructions would be ${next.length} characters, over the ${MAX_USER_INSTRUCTIONS_LENGTH} limit. Make them more concise.` + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + + h.setUserInstructions(next) + + const summary = next + ? parsed.operation === 'append' + ? 'Added a personal instruction' + : 'Updated your personal instructions' + : 'Cleared your personal instructions' + toolCallbacks.setToolStatus(toolId, { content: summary, result: summary }) + // Return only a short confirmation. The updated text is re-injected into the system + // prompt on the next iteration, so echoing it back here would waste context. + return `${summary}. It takes effect from your next message and is editable in the Global chat settings.` + } + }, { def: createToolDef( listWorkspaceItemsSchema, @@ -1636,6 +1885,46 @@ export const globalTools: Tool<{}>[] = [ return JSON.stringify(serializeWorkspaceItemForRead(item), null, 2) } }, + { + def: createToolDef( + createFolderSchema, + 'create_folder', + 'Create a new shared folder (addressable as f//) in the workspace. The current user is added as an owner.' + ), + requiresConfirmation: true, + confirmationMessage: 'Create folder', + showDetails: true, + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = createFolderSchema.parse(args) + if (!VALID_FOLDER_NAME.test(parsed.name)) { + const error = + 'Folder name can only contain alphanumeric characters, underscores, and hyphens.' + toolCallbacks.setToolStatus(toolId, { content: error, error }) + return JSON.stringify({ success: false, error }) + } + toolCallbacks.setToolStatus(toolId, { content: `Creating folder \`f/${parsed.name}\`...` }) + try { + await FolderService.createFolder({ + workspace, + requestBody: { name: parsed.name, summary: parsed.summary } + }) + // Reflect the new folder in the path-convention context for the rest of this + // session, matching FolderPicker's local update (avoids userStore.set()). + const user = get(userStore) + if (user) { + if (!user.folders) user.folders = [] + if (!user.folders.includes(parsed.name)) user.folders.push(parsed.name) + } + const message = `Created folder \`f/${parsed.name}\`. You can now write items to \`f/${parsed.name}/\`.` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ success: true, message }) + } catch (e) { + const error = e instanceof Error ? e.message : String(e) + toolCallbacks.setToolStatus(toolId, { content: `Error: ${error}`, error }) + return JSON.stringify({ success: false, error }) + } + } + }, { def: createToolDef(writeScriptSchema, 'write_script', 'Create or overwrite a draft script.'), showDetails: true, @@ -1840,6 +2129,20 @@ export const globalTools: Tool<{}>[] = [ return deployDraft(parsed, { ...ctx, sessionId: sessionIdFromCtx(ctx) }) } }, + { + def: createToolDef( + rebaseDraftSchema, + 'rebase_draft', + 'Discard a stale script, flow, or app draft and return your changes as a diff to re-apply on the latest deployed version. Use when deploy_workspace_item reports the draft was started from an older deployed version.', + { strict: false } + ), + showDetails: true, + showFade: true, + fn: async (ctx) => { + const parsed = rebaseDraftSchema.parse(ctx.args) + return rebaseDraft(parsed, ctx) + } + }, { def: createToolDef( deleteWorkspaceItemSchema, @@ -2112,7 +2415,9 @@ export const globalTools: Tool<{}>[] = [ } }, // Workspace-scoped datatable tools (unrestricted: no whitelist, no creation policy) - ...getDatatableTools() + ...getDatatableTools(), + // Read-only tools over files the user attached to the conversation + ...fileTools ] // Tools that only make sense inside an AI session (they drive the session's @@ -2160,6 +2465,12 @@ export type SessionToolHelpers = { sessionId?: string } export type GlobalToolHelpers = SessionToolHelpers & { testActiveFlow?: (args?: Record) => Promise + attachedFiles?: AttachedFilesStore + // Read/write the user-level Global instructions. `setUserInstructions` persists the + // value and rebuilds the system message so the change applies on the next chat-loop + // iteration. Backed by the update_user_instructions tool. + getUserInstructions?: () => string + setUserInstructions?: (instructions: string) => void } function sessionIdFromCtx(ctx: { helpers?: unknown }): string | undefined { @@ -2500,7 +2811,7 @@ function finishDraftWrite( type WriteSpec = { probe: (workspace: string, path: string) => Promise fetchDeployed: (workspace: string, path: string) => Promise - buildDraft: (base: T | undefined, args: A, path: string) => T + buildDraft: (base: T | undefined, args: A, path: string) => T | Promise beforePersist?: (workspace: string, args: A) => void } @@ -2523,7 +2834,7 @@ async function writeDraft( existed = true } - const draft = spec.buildDraft(base, args, path) + const draft = await spec.buildDraft(base, args, path) spec.beforePersist?.(workspace, args) const result = await persistGlobalDraft(workspace, type, path, draft, { @@ -2547,8 +2858,8 @@ const SCRIPT_SPEC: WriteSpec = { const existing = await ScriptService.getScriptByPath({ workspace, path }) return { ...(existing as unknown as NewScript), parent_hash: existing.hash } }, - buildDraft: (base, args, path) => - base + buildDraft: async (base, args, path) => { + const draft: NewScript = base ? { ...structuredClone(base), path, @@ -2566,6 +2877,18 @@ const SCRIPT_SPEC: WriteSpec = { language: args.language, kind: 'script' } + // Infer the arg schema from the content at save time, like the editor does, + // so the persisted draft is the single source of truth at deploy. Keep the + // previous schema (or empty) on failure rather than blanking it. + try { + const schema = emptySchema() + await inferArgs(draft.language, draft.content, schema) + draft.schema = schema + } catch (e) { + console.error('Failed to infer script schema before saving draft', e) + } + return draft + } } function writeScriptDraft(args: ScriptDraftArgs, ctx: WriteDraftCtx): Promise { @@ -3253,7 +3576,9 @@ async function searchApp( const header = `${totalMatchCount} match${ totalMatchCount === 1 ? '' : 'es' } in ${fileCount} file${fileCount === 1 ? '' : 's'}${ - truncated ? ` (showing the first ${maxMatches}; narrow with file_glob or a more specific query)` : '' + truncated + ? ` (showing the first ${maxMatches}; narrow with file_glob or a more specific query)` + : '' }` const out: string[] = [header] @@ -3575,17 +3900,310 @@ async function discardLocalDraft( ) } +// A draft started from an older deploy would silently overwrite whatever was +// deployed since. Block the deploy and point the model at rebase_draft, unless it +// explicitly forces the overwrite. `base`/`head` undefined ⇒ can't tell ⇒ allow. +function assertDraftBasedOnLatest( + type: WorkspaceItemType, + path: string, + base: string | number | undefined, + head: string | number | undefined, + force: boolean | undefined +): void { + if (force || base == null || head == null || base === head) return + throw new Error( + `This ${type} draft "${path}" was started from an older deployed version (forked from ${base}, ` + + `latest is ${head}). Deploying now would overwrite the version deployed since. Call rebase_draft to ` + + `discard the stale draft and get your changes back as a diff, then re-apply them (the new draft ` + + `re-bases onto the latest version) and deploy. To deploy as-is and replace the newer version, call ` + + `deploy_workspace_item again with force: true.` + ) +} + +// Discard a stale draft and return its own changes (vs the fork base) as a diff so +// the model can re-apply them on the latest deploy. Discarding rather than resetting +// keeps the base pointer honest (the next write re-bases on the current head) and +// fails safe: a premature deploy hits "no draft" instead of silently shipping the +// latest unchanged. +async function rebaseDraft( + args: { type: WorkspaceItemType; path: string }, + ctx: WriteDraftCtx +): Promise { + switch (args.type) { + case 'script': + return rebaseScriptDraft(args.path, ctx) + case 'flow': + return rebaseFlowDraft(args.path, ctx) + case 'app': + return rebaseAppDraft(args.path, ctx) + default: + throw new Error('rebase_draft currently supports scripts, flows, and apps.') + } +} + +async function rebaseScriptDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'script', path) + if (!draft || typeof draft.value !== 'string' || !draft.language) { + throw new Error(`No script draft found for "${path}".`) + } + if (!(await ScriptService.existsScriptByPath({ workspace, path }))) { + throw new Error(`Script "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const latest = await ScriptService.getScriptByPath({ workspace, path }) + const baseHash = draft.parentHash + if (baseHash && baseHash === latest.hash) { + const message = `Draft "${path}" is already based on the latest deployed version (${latest.hash}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ success: true, alreadyLatest: true, latest_hash: latest.hash, message }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // Capture the draft's own changes (vs its fork base) BEFORE discarding — this + // diff is the only clean record of what to replay. Best-effort: if the base + // version is gone, diff against empty so the full draft is surfaced. + let baseContent = '' + if (baseHash) { + try { + baseContent = (await ScriptService.getScriptByHash({ workspace, hash: baseHash })).content + } catch (e) { + console.error(`rebase_draft: could not fetch base version ${baseHash} for "${path}"`, e) + } + } + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + baseContent, + draft.value, + '', + '' + ) + + // Discard the stale draft rather than resetting it to latest: the next write + // re-bases on the current head, and a premature deploy fails cleanly ("no + // draft") instead of silently shipping the latest unchanged and losing the work. + await deleteGlobalDraft(workspace, 'script', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a diff against the ` + + `version you forked from). Re-apply them with edit_script / write_script — the new draft will be ` + + `based on the latest deployed version (hash ${latest.hash}) — then deploy.`, + latest_hash: latest.hash, + your_changes: yourChanges + }, + null, + 2 + ) +} + +async function rebaseFlowDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'flow', path) + if (!draft || draft.value === undefined || typeof draft.value === 'string') { + throw new Error(`No flow draft found for "${path}".`) + } + if (!(await FlowService.existsFlowByPath({ workspace, path }))) { + throw new Error(`Flow "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const latest = await FlowService.getFlowByPath({ workspace, path }) + const baseVersion = draft.parentVersionId + if (baseVersion != null && baseVersion === latest.version_id) { + const message = `Draft "${path}" is already based on the latest deployed version (${latest.version_id}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ + success: true, + alreadyLatest: true, + latest_version: latest.version_id, + message + }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // The draft's own changes vs its fork-base flow value, as a JSON diff for the + // model to replay. Best-effort: skip if the base version can't be fetched. + let baseValue: unknown = {} + if (baseVersion != null) { + try { + baseValue = (await FlowService.getFlowVersion({ workspace, version: baseVersion })).value + } catch (e) { + console.error( + `rebase_draft: could not fetch base flow version ${baseVersion} for "${path}"`, + e + ) + } + } + const oursValue = (draft.value as FlowDraftValue).value + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + JSON.stringify(baseValue, null, 2), + JSON.stringify(oursValue, null, 2), + '', + '' + ) + + // Discard the stale draft (see rebaseScriptDraft): the next write re-bases on + // the current head, and a premature deploy fails cleanly instead of shipping + // the latest unchanged. + await deleteGlobalDraft(workspace, 'flow', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a JSON diff against ` + + `the version you forked from). Re-apply them with the flow edit tools — the new draft will be ` + + `based on the latest deployed version (version ${latest.version_id}) — then deploy.`, + latest_version: latest.version_id, + your_changes: yourChanges + }, + null, + 2 + ) +} + +async function rebaseAppDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'app', path) + if (!draft || !draft.value || typeof draft.value === 'string' || !('files' in draft.value)) { + throw new Error(`No app draft found for "${path}".`) + } + if (!(await AppService.existsApp({ workspace, path }))) { + throw new Error(`App "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const deployed = await AppService.getAppByPath({ workspace, path }) + const headVersion = deployed.versions?.[deployed.versions.length - 1] + const baseVersion = draft.parentVersionId + if (baseVersion != null && baseVersion === headVersion) { + const message = `Draft "${path}" is already based on the latest deployed version (${headVersion}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ + success: true, + alreadyLatest: true, + latest_version: headVersion, + message + }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // The draft's own changes vs its fork-base app source, as a JSON diff for the + // model to replay. Best-effort: skip if the base version can't be fetched. + const oursValue = draft.value as AppDraftValue + let baseSource: Pick = { + files: {}, + runnables: {}, + data: undefined + } + if (baseVersion != null) { + try { + const baseApp = await AppService.getAppByVersion({ workspace, id: baseVersion }) + const base = appSourceToDraftValue(baseApp, baseApp) + baseSource = { files: base.files, runnables: base.runnables, data: base.data } + } catch (e) { + console.error( + `rebase_draft: could not fetch base app version ${baseVersion} for "${path}"`, + e + ) + } + } + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + JSON.stringify(baseSource, null, 2), + JSON.stringify( + { files: oursValue.files, runnables: oursValue.runnables, data: oursValue.data }, + null, + 2 + ), + '', + '' + ) + + // Discard the stale draft (see rebaseScriptDraft): the next write re-projects + // the deployed app into a fresh draft (re-pinning parent_version to the head), + // and a premature deploy fails cleanly instead of shipping the latest unchanged. + await deleteGlobalDraft(workspace, 'app', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a JSON diff against ` + + `the version you forked from). Re-apply them with the app edit tools — the new draft will be based ` + + `on the latest deployed version (version ${headVersion}) — then deploy.`, + latest_version: headVersion, + your_changes: yourChanges + }, + null, + 2 + ) +} + +// Flush a draft's pending editor autosave, then verify it actually landed before +// the caller re-reads the persisted draft. `flush()` resolves even when the save +// recorded a conflict (server has a newer version) or failed (network/5xx) — it +// does not throw — so without this check a deploy could publish a stale/conflicting +// draft. Abort with a clear message instead. +async function flushDraftOrThrow( + query: Parameters[0], + label: string +): Promise { + await UserDraftDbSyncer.flush(query) + if (UserDraftDbSyncer.getConflict(query).conflict) { + throw new Error( + `Cannot deploy ${label}: the draft has a conflicting newer version on the server. Open it in the editor and resolve the conflict first.` + ) + } + const { state, failureMessage } = UserDraftDbSyncer.getState(query) + if (state === 'failed') { + throw new Error( + `Cannot deploy ${label}: saving the latest draft failed (${failureMessage ?? 'unknown error'}). Retry once the draft saves.` + ) + } +} + async function deployDraft( args: { type: WorkspaceItemType path: string trigger_kind?: TriggerKind deployment_message?: string + force?: boolean }, ctx: WriteDraftCtx ): Promise { const { workspace, toolId, toolCallbacks, sessionId } = ctx - const { type, path, trigger_kind: triggerKind, deployment_message: deploymentMessage } = args + const { + type, + path, + trigger_kind: triggerKind, + deployment_message: deploymentMessage, + force + } = args if (type === 'trigger' && !triggerKind) { throw new Error('trigger_kind is required when deploying a trigger.') @@ -3605,169 +4223,241 @@ async function deployDraft( let actions: ToolDisplayAction[] | undefined - switch (type) { - case 'script': { + if (type === 'script' || type === 'flow') { + // Promote the full persisted draft via the shared deploy module — the same + // "promote a draft to deployed" code the compare page's Review & Deploy uses. + // It deploys every field of the draft; the previous local builders dropped + // most config fields (tag, priority, schema, description, concurrency…), + // reading them from the already-deployed version instead. The other kinds + // below already deploy their draft value directly, so only script/flow need + // this. Scripts always create (with parent_hash); a flow on a deployed item + // updates, a draft-only flow (no flow row) is created. + // Address the draft by its STORAGE path: a draft_only item created in the + // editor lives at a synthetic `u/{user}/draft_{uuid}` key while its chosen + // path is held in the draft value. The shared deployer reads the draft via + // getScriptByPath/getFlowByPath at the path we pass (then deploys at the + // draft's own `path`), so passing the display/chosen path would 404. For a + // draft on a deployed item the storage path is just the item path. + const storagePath = getGlobalDraftStoragePath(workspace, type, path, triggerKind) + // The shared deployer re-reads the persisted DB draft, but an open editor's + // edit may still be parked in a debounced/disabled autosave. Flush it first so + // we deploy the latest value (not a stale persisted one) — and so the + // post-deploy draft delete doesn't drop an unsaved edit. `flush` always saves + // the parked value (like Ctrl/Cmd+S), since the user explicitly asked to deploy. + await flushDraftOrThrow({ workspace, itemKind: type, path: storagePath }, `${type} "${path}"`) + // Stale-draft guard: block when the draft was forked from an older deploy than + // the current head (unless force), pointing the model at rebase_draft. + if (type === 'script') { const existing = (await ScriptService.existsScriptByPath({ workspace, path })) ? await ScriptService.getScriptByPath({ workspace, path }) : undefined - const requestBody = buildScriptDeployRequestBody(path, draft, existing, deploymentMessage) - // Infer the arg schema from the content so it matches the code, like the editor does. - try { - const schema = emptySchema() - await inferArgs(requestBody.language, requestBody.content, schema) - requestBody.schema = schema - } catch (e) { - console.error('Failed to infer script schema before deploy', e) - } - await ScriptService.createScript({ workspace, requestBody }) - break - } - case 'flow': { - const flowDraft = draft.value as FlowDraftValue + assertDraftBasedOnLatest('script', path, draft.parentHash, existing?.hash, force) + } else { const existing = (await FlowService.existsFlowByPath({ workspace, path })) ? await FlowService.getFlowByPath({ workspace, path }) : undefined - const requestBody = buildFlowDeployRequestBody( - path, - draft.summary, - flowDraft, - existing, - deploymentMessage - ) - if (existing) { - await FlowService.updateFlow({ workspace, path, requestBody }) - } else { - await FlowService.createFlow({ workspace, requestBody }) - } - break + assertDraftBasedOnLatest('flow', path, draft.parentVersionId, existing?.version_id, force) } - case 'schedule': { - const requestBody = draft.value as any - if (await ScheduleService.existsSchedule({ workspace, path })) { - await ScheduleService.updateSchedule({ workspace, path, requestBody }) - } else { - await ScheduleService.createSchedule({ workspace, requestBody }) - } - actions = [createOpenScheduleAction(path, requestBody.is_flow ? 'flow' : 'script')] - break + const draftOnly = + type === 'flow' + ? !(await FlowService.existsFlowByPath({ workspace, path: storagePath })) + : false + const result = await deployDraftToWorkspace(type, storagePath, workspace, { + draftOnly, + deploymentMessage + }) + if (!result.success) { + throw new Error(result.error ?? `Failed to deploy ${type} "${path}".`) } - case 'trigger': { - const service = triggerServices[triggerKind!] - const requestBody = draft.value as { is_flow?: boolean } - if (await service.exists({ workspace, path })) { - await service.update({ workspace, path, requestBody }) - } else { - await service.create({ workspace, requestBody }) - } - actions = [ - createOpenTriggerAction(triggerKind!, path, requestBody.is_flow ? 'flow' : 'script') - ] - break - } - case 'resource': { - const requestBody = draft.value as any - if (await ResourceService.existsResource({ workspace, path })) { - await ResourceService.updateResource({ workspace, path, requestBody }) - } else { - await ResourceService.createResource({ workspace, requestBody }) - } - actions = [createOpenResourceAction(path)] - break - } - case 'variable': { - const requestBody = buildVariableDeployRequestBody( - workspace, - path, - draft.value as CreateVariable - ) - if (await VariableService.existsVariable({ workspace, path })) { - await VariableService.updateVariable({ workspace, path, requestBody }) - } else { - await VariableService.createVariable({ workspace, requestBody }) - } - actions = [createOpenVariableAction(path)] - break - } - case 'app': { - const appDraft = draft.value as AppDraftValue - const appValue: AppDraftValue = { - ...appDraft, - files: { ...(appDraft.files ?? {}) }, - runnables: { ...(appDraft.runnables ?? {}) }, - data: appDraft.data ?? { ...DEFAULT_RAW_APP_DATA } - } - await recomputeAppPolicy(appValue) - const policy = appValue.policy - if (!policy) { - throw new Error(`Draft app "${path}" has no policy to deploy.`) - } - - toolCallbacks.setToolStatus(toolId, { - content: `Bundling app "${path}"...` - }) - const bundle = await bundleRawAppDraft({ - workspace, - files: appValue.files, - onLog: (delta) => { - const lines = delta - .split('\n') - .map((line) => line.trim()) - .filter(Boolean) - const latest = lines[lines.length - 1] - if (latest) { - toolCallbacks.setToolStatus(toolId, { - content: `Bundling app "${path}"... ${latest}` - }) - } + } else { + switch (type) { + case 'schedule': { + const requestBody = draft.value as any + if (await ScheduleService.existsSchedule({ workspace, path })) { + await ScheduleService.updateSchedule({ workspace, path, requestBody }) + } else { + await ScheduleService.createSchedule({ workspace, requestBody }) } - }) - - toolCallbacks.setToolStatus(toolId, { - content: `Deploying app "${path}"...` - }) - const rawAppValue = { - files: appValue.files, - runnables: appValue.runnables, - data: appValue.data ?? { ...DEFAULT_RAW_APP_DATA } + actions = [createOpenScheduleAction(path, requestBody.is_flow ? 'flow' : 'script')] + break } - const summary = appValue.summary ?? draft.summary ?? '' - if (await AppService.existsApp({ workspace, path })) { - // Omit custom_path on update for now. The backend preserves it when absent, while - // sending it requires admin privileges; this chat deploy path does not yet mirror - // the raw app editor's user/admin-specific custom_path handling. - await AppService.updateAppRaw({ + case 'trigger': { + const service = triggerServices[triggerKind!] + const requestBody = draft.value as { is_flow?: boolean } + if (await service.exists({ workspace, path })) { + await service.update({ workspace, path, requestBody }) + } else { + await service.create({ workspace, requestBody }) + } + actions = [ + createOpenTriggerAction(triggerKind!, path, requestBody.is_flow ? 'flow' : 'script') + ] + break + } + case 'resource': { + const requestBody = draft.value as any + if (await ResourceService.existsResource({ workspace, path })) { + await ResourceService.updateResource({ workspace, path, requestBody }) + } else { + await ResourceService.createResource({ workspace, requestBody }) + } + actions = [createOpenResourceAction(path)] + break + } + case 'variable': { + // The chat keeps secret draft values only in memory (the DB draft + // stores `''`); buildVariableDeployRequestBody re-injects the ephemeral + // secret, so this can't go through the DB-reading shared deployer. + const requestBody = buildVariableDeployRequestBody( workspace, path, - formData: { - app: { - path, - value: rawAppValue, - summary, - policy, - deployment_message: deploymentMessage - }, - js: bundle.js, - css: bundle.css - } + draft.value as CreateVariable + ) + if (await VariableService.existsVariable({ workspace, path })) { + await VariableService.updateVariable({ workspace, path, requestBody }) + } else { + await VariableService.createVariable({ workspace, requestBody }) + } + actions = [createOpenVariableAction(path)] + break + } + case 'app': { + // Raw apps store a flat AppDraftValue (files/runnables at top level), + // not the deployed app's nested `value` shape the shared raw-app + // deployer reads, so they deploy through the chat's own bundle path. + const appDraft = draft.value as AppDraftValue + // Stale-draft guard: only fetch the deployed head when the draft records + // a fork base to compare against (pre-feature drafts have none). + if (draft.parentVersionId != null) { + const deployedApp = (await AppService.existsApp({ workspace, path })) + ? await AppService.getAppByPath({ workspace, path }) + : undefined + assertDraftBasedOnLatest( + 'app', + path, + draft.parentVersionId, + deployedApp?.versions?.[deployedApp.versions.length - 1], + force + ) + } + const appValue: AppDraftValue = { + ...appDraft, + files: { ...(appDraft.files ?? {}) }, + runnables: { ...(appDraft.runnables ?? {}) }, + data: appDraft.data ?? { ...DEFAULT_RAW_APP_DATA } + } + await recomputeAppPolicy(appValue) + const policy = appValue.policy + if (!policy) { + throw new Error(`Draft app "${path}" has no policy to deploy.`) + } + + toolCallbacks.setToolStatus(toolId, { + content: `Bundling app "${path}"...` }) - } else { - await AppService.createAppRaw({ + const bundle = await bundleRawAppDraft({ workspace, - formData: { - app: { - path, - value: rawAppValue, - summary, - policy, - deployment_message: deploymentMessage, - custom_path: appValue.custom_path - }, - js: bundle.js, - css: bundle.css + files: appValue.files, + onLog: (delta) => { + const lines = delta + .split('\n') + .map((line) => line.trim()) + .filter(Boolean) + const latest = lines[lines.length - 1] + if (latest) { + toolCallbacks.setToolStatus(toolId, { + content: `Bundling app "${path}"... ${latest}` + }) + } } + }) + + toolCallbacks.setToolStatus(toolId, { + content: `Deploying app "${path}"...` }) + const rawAppValue = { + files: appValue.files, + runnables: appValue.runnables, + data: appValue.data ?? { ...DEFAULT_RAW_APP_DATA } + } + const summary = appValue.summary ?? draft.summary ?? '' + // Deploy at the draft's chosen path. A draft_only raw app created in the + // editor lives at a synthetic `u/{user}/draft_{uuid}` storage key with its + // chosen path in the raw_app draft's `draft_path`; the chat's AppDraftValue + // doesn't carry it, so read it from the backend draft. For a chat-created app + // (real path, no draft_path) or a draft on a deployed app, the storage path + // is the deploy path. Same storage-path resolution as script/flow. + const storagePath = getGlobalDraftStoragePath(workspace, 'app', path) + // `draft_path` is read from the persisted backend draft below, but an + // editor rename may still be parked in a debounced/disabled autosave. + // Flush first (like script/flow) so we read the latest chosen path. + await flushDraftOrThrow( + { workspace, itemKind: 'raw_app', path: storagePath }, + `app "${path}"` + ) + let targetPath = storagePath + try { + const row = (await AppService.getAppByPath({ + workspace, + path: storagePath, + getDraft: true, + rawApp: true + })) as { draft?: { draft_path?: string; path?: string }; draft_path?: string } + targetPath = row?.draft?.draft_path ?? row?.draft?.path ?? row?.draft_path ?? storagePath + } catch (e) { + // Only a missing item (404) justifies falling back to the storage path; + // a real lookup failure (network/5xx) must abort rather than silently + // deploy to the wrong path. + if ((e as { status?: number } | null | undefined)?.status === 404) { + targetPath = storagePath + } else { + throw e + } + } + if (await AppService.existsApp({ workspace, path: targetPath })) { + // Omit custom_path on update for now. The backend preserves it when absent, while + // sending it requires admin privileges; this chat deploy path does not yet mirror + // the raw app editor's user/admin-specific custom_path handling. + await AppService.updateAppRaw({ + workspace, + path: targetPath, + formData: { + app: { + path: targetPath, + value: rawAppValue, + summary, + policy, + deployment_message: deploymentMessage, + // Preserve the policy's on_behalf_of: this chat deploy path has no + // on-behalf-of selector, so without the flag the backend resets it to + // the deploying user (gated server-side by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined + }, + js: bundle.js, + css: bundle.css + } + }) + } else { + await AppService.createAppRaw({ + workspace, + formData: { + app: { + path: targetPath, + value: rawAppValue, + summary, + policy, + deployment_message: deploymentMessage, + custom_path: appValue.custom_path, + // Preserve the policy's on_behalf_of (see update branch above). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined + }, + js: bundle.js, + css: bundle.css + } + }) + } + break } - break } } @@ -3861,13 +4551,36 @@ async function deleteWorkspaceItem( } export function prepareGlobalSystemMessage( - customPrompt?: string, - opts?: { previewTools?: boolean } + instructions?: { workspace?: string; user?: string }, + opts?: { + previewTools?: boolean + // Identity the path-convention guidance is built from. Production omits it + // (read from userStore); callers that must not touch the process-global + // store (the eval harness) pass it explicitly instead. + user?: { username: string; is_admin?: boolean; folders?: string[]; folders_read?: string[] } + skills?: AiSkillListItem[] + } ): ChatCompletionSystemMessageParam { - const username = get(userStore)?.username ?? '' - let content = buildGlobalSystemPrompt(username, opts?.previewTools ?? false) - if (customPrompt?.trim()) { - content = `${content}\n\nUSER GIVEN INSTRUCTIONS:\n${customPrompt.trim()}` + const user = opts?.user ?? get(userStore) + const username = user?.username ?? '' + const folderCtx: FolderPromptContext | undefined = user + ? { + folders: user.folders ?? [], + foldersRead: user.folders_read ?? user.folders ?? [], + isAdmin: user.is_admin ?? false + } + : undefined + let content = buildGlobalSystemPrompt( + username, + opts?.previewTools ?? false, + folderCtx, + opts?.skills ?? [] + ) + if (instructions?.workspace?.trim()) { + content = `${content}\n\nWORKSPACE INSTRUCTIONS (configured by a workspace admin, shared by everyone in this workspace — you cannot modify these):\n${instructions.workspace.trim()}` + } + if (instructions?.user?.trim()) { + content = `${content}\n\nUSER INSTRUCTIONS (this user's personal instructions — update them with the update_user_instructions tool when the user asks you to remember, change, or stop something):\n${instructions.user.trim()}` } return { @@ -3893,7 +4606,10 @@ export function prepareGlobalUserMessage( options: GlobalUserMessageOptions = {} ): ChatCompletionUserMessageParam { const selectedWorkspaceItems = selectedContext.filter( - (context) => context.type === 'workspace_script' || context.type === 'workspace_flow' + (context) => + context.type === 'workspace_script' || + context.type === 'workspace_flow' || + context.type === 'workspace_app' ) const activeEditor = options.activeEditor ?? @@ -3910,7 +4626,13 @@ export function prepareGlobalUserMessage( if (selectedWorkspaceItems.length > 0) { content += '## SELECTED CONTEXT\n' for (const context of selectedWorkspaceItems) { - content += `- type: ${context.type === 'workspace_script' ? 'script' : 'flow'}, path: ${context.path}\n` + const itemType = + context.type === 'workspace_script' + ? 'script' + : context.type === 'workspace_flow' + ? 'flow' + : 'raw_app' + content += `- type: ${itemType}, path: ${context.path}\n` } content += '\n' } diff --git a/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts b/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts deleted file mode 100644 index 61a6827cbf..0000000000 --- a/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts +++ /dev/null @@ -1,190 +0,0 @@ -import { describe, expect, it } from 'vitest' -import type { Flow, NewScript, Script } from '$lib/gen/types.gen' -import { buildFlowDeployRequestBody, buildScriptDeployRequestBody } from './deployRequests' -import type { WorkspaceItem } from './workspaceItems' - -describe('global AI deploy request builders', () => { - it('preserves existing script metadata while replacing draft-controlled fields', () => { - const existing = { - hash: 'parent-hash', - path: 'f/demo/script', - summary: 'existing summary', - description: 'existing description', - content: 'old content', - schema: { properties: { name: { type: 'string' } } }, - is_template: true, - language: 'bun', - kind: 'script', - tag: 'node', - envs: ['ENV_A'], - concurrent_limit: 3, - concurrency_time_window_s: 60, - concurrency_key: 'key', - debounce_key: 'debounce', - debounce_delay_s: 5, - debounce_args_to_accumulate: ['ids'], - max_total_debouncing_time: 120, - max_total_debounces_amount: 4, - cache_ttl: 30, - cache_ignore_s3_path: true, - dedicated_worker: true, - ws_error_handler_muted: true, - priority: 9, - restart_unless_cancelled: true, - timeout: 300, - delete_after_secs: 600, - visible_to_runner_only: true, - auto_kind: 'script', - codebase: 'repo', - has_preprocessor: true, - on_behalf_of_email: 'deployer@example.com', - assets: [{ path: 's3://bucket/key', kind: 's3object' }], - modules: { 'helper.ts': { content: 'export const helper = 1', language: 'bun' } }, - labels: ['prod'], - lock: 'stale lock' - } as unknown as Script & Partial - const draft: WorkspaceItem = { - type: 'script', - path: 'f/demo/script', - summary: 'draft summary', - language: 'bun', - value: 'new content', - isDraft: true - } - - const requestBody = buildScriptDeployRequestBody('f/demo/script', draft, existing, 'ai deploy') - - expect(requestBody).toMatchObject({ - path: 'f/demo/script', - parent_hash: 'parent-hash', - summary: 'draft summary', - description: 'existing description', - content: 'new content', - schema: existing.schema, - tag: 'node', - envs: ['ENV_A'], - concurrent_limit: 3, - concurrency_time_window_s: 60, - concurrency_key: 'key', - debounce_key: 'debounce', - debounce_delay_s: 5, - debounce_args_to_accumulate: ['ids'], - max_total_debouncing_time: 120, - max_total_debounces_amount: 4, - cache_ttl: 30, - cache_ignore_s3_path: true, - dedicated_worker: true, - ws_error_handler_muted: true, - priority: 9, - restart_unless_cancelled: true, - timeout: 300, - delete_after_secs: 600, - visible_to_runner_only: true, - auto_kind: 'script', - codebase: 'repo', - has_preprocessor: true, - on_behalf_of_email: 'deployer@example.com', - preserve_on_behalf_of: true, - assets: existing.assets, - modules: existing.modules, - labels: ['prod'], - deployment_message: 'ai deploy' - }) - expect(requestBody.lock).toBeUndefined() - }) - - it('preserves existing flow metadata and uses draft value/schema overrides', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - value: { modules: [] }, - schema: { required: ['name'] }, - tag: 'python', - ws_error_handler_muted: true, - priority: 7, - dedicated_worker: true, - timeout: 60, - visible_to_runner_only: true, - on_behalf_of_email: 'deployer@example.com', - labels: ['critical'] - } as unknown as Flow - const draftValue = { - value: { modules: [{ id: 'step', value: { type: 'identity' } }] }, - schema: { properties: { name: { type: 'string' } } }, - groups: [{ start_id: 'step', end_id: 'step', summary: 'Group' }] - } - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - undefined, - draftValue as any, - existing, - 'ai deploy' - ) - - expect(requestBody).toMatchObject({ - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - schema: draftValue.schema, - tag: 'python', - ws_error_handler_muted: true, - priority: 7, - dedicated_worker: true, - timeout: 60, - visible_to_runner_only: true, - on_behalf_of_email: 'deployer@example.com', - preserve_on_behalf_of: true, - labels: ['critical'], - deployment_message: 'ai deploy' - }) - expect(requestBody.value.modules).toHaveLength(1) - expect(requestBody.value.groups).toEqual(draftValue.groups) - }) - - it('deploys a draft-set flow description, overriding the existing one', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - value: { modules: [] }, - schema: {} - } as unknown as Flow - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - undefined, - { - value: { modules: [] }, - schema: null, - groups: null, - description: 'draft-set description' - } as any, - existing, - undefined - ) - - expect(requestBody.description).toBe('draft-set description') - }) - - it('falls back to existing flow schema when the draft has no schema', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - value: { modules: [] }, - schema: { properties: { existing: { type: 'boolean' } } } - } as unknown as Flow - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - 'draft summary', - { value: { modules: [] }, schema: null, groups: null }, - existing, - undefined - ) - - expect(requestBody.summary).toBe('draft summary') - expect(requestBody.schema).toBe(existing.schema) - }) -}) diff --git a/frontend/src/lib/components/copilot/chat/global/deployRequests.ts b/frontend/src/lib/components/copilot/chat/global/deployRequests.ts deleted file mode 100644 index 5fe44566c9..0000000000 --- a/frontend/src/lib/components/copilot/chat/global/deployRequests.ts +++ /dev/null @@ -1,101 +0,0 @@ -import type { Flow, NewScript, OpenFlowWPath, Script } from '$lib/gen/types.gen' -import type { FlowDraftValue, WorkspaceItem } from './workspaceItems' - -type ScriptWithDeployMetadata = Script & Partial> - -export type FlowDeployRequestBody = OpenFlowWPath & { - deployment_message?: string -} - -function preserveOnBehalfOf(email: string | undefined): true | undefined { - return email ? true : undefined -} - -export function buildScriptDeployRequestBody( - path: string, - draft: WorkspaceItem, - existing: Script | undefined, - deploymentMessage: string | undefined -): NewScript { - if (typeof draft.value !== 'string' || !draft.language) { - throw new Error(`Draft script "${path}" is missing content or language.`) - } - - const existingWithMetadata = existing as ScriptWithDeployMetadata | undefined - - return { - path, - summary: draft.summary ?? existing?.summary ?? '', - description: existing?.description ?? '', - content: draft.value, - parent_hash: existing?.hash, - schema: existing?.schema, - is_template: existing?.is_template, - language: draft.language, - kind: existing?.kind, - tag: existing?.tag, - envs: existing?.envs, - concurrent_limit: existing?.concurrent_limit, - concurrency_time_window_s: existing?.concurrency_time_window_s, - debounce_key: existing?.debounce_key, - debounce_delay_s: existing?.debounce_delay_s, - debounce_args_to_accumulate: existing?.debounce_args_to_accumulate, - max_total_debouncing_time: existing?.max_total_debouncing_time, - max_total_debounces_amount: existing?.max_total_debounces_amount, - cache_ttl: existing?.cache_ttl, - cache_ignore_s3_path: existingWithMetadata?.cache_ignore_s3_path, - dedicated_worker: existing?.dedicated_worker, - ws_error_handler_muted: existing?.ws_error_handler_muted, - priority: existing?.priority, - restart_unless_cancelled: existing?.restart_unless_cancelled, - timeout: existing?.timeout, - delete_after_secs: existing?.delete_after_secs, - deployment_message: deploymentMessage, - concurrency_key: existing?.concurrency_key, - visible_to_runner_only: existing?.visible_to_runner_only, - auto_kind: existing?.auto_kind, - codebase: existing?.codebase, - has_preprocessor: existing?.has_preprocessor, - on_behalf_of_email: existing?.on_behalf_of_email, - preserve_on_behalf_of: preserveOnBehalfOf(existing?.on_behalf_of_email), - assets: existingWithMetadata?.assets, - modules: existing?.modules, - labels: existing?.labels - } -} - -function flowValueWithDraftGroups(flowDraft: FlowDraftValue): FlowDraftValue['value'] { - if (flowDraft.groups === undefined) { - return flowDraft.value - } - return { - ...flowDraft.value, - groups: flowDraft.groups ?? undefined - } -} - -export function buildFlowDeployRequestBody( - path: string, - draftSummary: string | undefined, - flowDraft: FlowDraftValue, - existing: Flow | undefined, - deploymentMessage: string | undefined -): FlowDeployRequestBody { - return { - path, - summary: draftSummary ?? existing?.summary ?? '', - description: flowDraft.description ?? existing?.description ?? '', - value: flowValueWithDraftGroups(flowDraft), - schema: flowDraft.schema ?? existing?.schema ?? {}, - tag: existing?.tag, - ws_error_handler_muted: existing?.ws_error_handler_muted, - priority: existing?.priority, - dedicated_worker: existing?.dedicated_worker, - timeout: existing?.timeout, - visible_to_runner_only: existing?.visible_to_runner_only, - on_behalf_of_email: existing?.on_behalf_of_email, - preserve_on_behalf_of: preserveOnBehalfOf(existing?.on_behalf_of_email), - labels: existing?.labels, - deployment_message: deploymentMessage - } -} diff --git a/frontend/src/lib/components/copilot/chat/global/gate.ts b/frontend/src/lib/components/copilot/chat/global/gate.ts index 664f06ff39..df8035de87 100644 --- a/frontend/src/lib/components/copilot/chat/global/gate.ts +++ b/frontend/src/lib/components/copilot/chat/global/gate.ts @@ -11,8 +11,8 @@ * When the mode is ready to ship to everyone, replace every call to * `isGlobalAiEnabled()` with `true` and delete this file. The references are * intentionally narrow (chat mode visibility, custom prompt settings, the - * `change_mode` tool enum, and the `/global_drafts` dev route) so the rip-out - * is a small grep. + * `change_mode` tool enum, the AI skills workspace settings tab, and the + * `/global_drafts` dev route) so the rip-out is a small grep. */ const STORAGE_KEY = 'wm_dev_global_ai' diff --git a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts index 009050342e..87ca767904 100644 --- a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts +++ b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts @@ -66,7 +66,10 @@ function normalizeAppDraftValue(value: AppDraftValue): AppDraftValue { runnables: { ...(value.runnables ?? {}) }, data: value.data ?? { ...DEFAULT_RAW_APP_DATA }, policy: value.policy === undefined ? undefined : clone(value.policy), - custom_path: value.custom_path + custom_path: value.custom_path, + // Carry the fork-base version through the whitelist — it is dropped on every + // save otherwise, which would defeat the stale-draft check. + parent_version: value.parent_version } } @@ -135,6 +138,7 @@ function scriptDraftToWorkspaceItem(path: string, draft: NewScript): WorkspaceIt summary: draft.summary, language: draft.language, value: draft.content, + parentHash: draft.parent_hash, isDraft: true } } @@ -144,6 +148,10 @@ function flowDraftToWorkspaceItem(path: string, draft: Flow): WorkspaceItem { type: 'flow', path, summary: draft.summary, + // The persisted flow draft carries `version_id` (the deployed head it was + // forked from, pinned at fork by writeDraft/the editor) — the flow analog + // of a script's parent_hash. + parentVersionId: draft.version_id, value: { value: draft.value, schema: draft.schema ?? null, @@ -160,6 +168,7 @@ function appDraftToWorkspaceItem(path: string, draft: AppDraftValue): WorkspaceI type: 'app', path, summary: value.summary, + parentVersionId: value.parent_version, value, isDraft: true } diff --git a/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts b/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts index 09bbc12da5..a8196a5054 100644 --- a/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts +++ b/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts @@ -72,6 +72,9 @@ export type AppDraftValue = { data?: any policy?: Policy custom_path?: string + // Fork base: the deployed app version this draft was started from, pinned at + // fork. The app analog of a script's parent_hash / a flow's version_id. + parent_version?: number } export type ResourceDraftState = { @@ -99,6 +102,11 @@ export type WorkspaceItem = { summary?: string language?: ScriptLang triggerKind?: TriggerKind + // Fork base: the deployed version this draft was started from, compared against + // the current deployed head to detect a stale draft. `parentHash` for scripts + // (script hash), `parentVersionId` for flows (flow_version id). + parentHash?: string + parentVersionId?: number value?: | string | FlowDraftValue diff --git a/frontend/src/lib/components/copilot/chat/mention.test.ts b/frontend/src/lib/components/copilot/chat/mention.test.ts new file mode 100644 index 0000000000..8d379ce3b4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest' +import { MENTION_RE, mentionTitle, formatMention } from './mention' + +describe('formatMention', () => { + it('leaves a simple name bare', () => { + expect(formatMention('app.ts')).toBe('@app.ts') + expect(formatMention('proj/sub/a.ts')).toBe('@proj/sub/a.ts') + }) + it('brackets a name containing whitespace', () => { + expect(formatMention('my file.txt')).toBe('@[my file.txt]') + expect(formatMention('my folder/a b.ts')).toBe('@[my folder/a b.ts]') + }) + it('brackets names with HTML-sensitive chars, parens, brackets', () => { + expect(formatMention('R&D notes.md')).toBe('@[R&D notes.md]') + expect(formatMention('a.txt')).toBe('@[a.txt]') + expect(formatMention('report(final).csv')).toBe('@[report(final).csv]') + }) +}) + +describe('mentionTitle', () => { + it('strips the @ from a bare mention', () => { + expect(mentionTitle('@app.ts')).toBe('app.ts') + }) + it('strips the @[ ] from a bracketed mention', () => { + expect(mentionTitle('@[my file.txt]')).toBe('my file.txt') + }) +}) + +describe('MENTION_RE', () => { + it('captures a bracketed (spaced) mention whole alongside bare ones', () => { + const tokens = [...'see @app.ts and @[my file.txt] ok'.matchAll(MENTION_RE)].map((m) => m[0]) + expect(tokens).toEqual(['@app.ts', '@[my file.txt]']) + expect(tokens.map(mentionTitle)).toEqual(['app.ts', 'my file.txt']) + }) + it('round-trips formatMention → MENTION_RE → mentionTitle for a spaced name', () => { + const name = 'my notes (v2).md' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips a name containing both whitespace and a closing bracket', () => { + const name = 'notes ] draft.md' + expect(formatMention(name)).toBe('@[notes \\] draft.md]') + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(m[0]).toBe('@[notes \\] draft.md]') + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips an HTML-sensitive name (highlighter handles HTML-escaping separately)', () => { + const name = 'a & c].txt' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/mention.ts b/frontend/src/lib/components/copilot/chat/mention.ts new file mode 100644 index 0000000000..65a2b3b7c6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.ts @@ -0,0 +1,31 @@ +/** + * `@mention` formatting shared between the chat input (which inserts mentions) and the + * textarea highlighter (which parses them) so the two never disagree. + * + * A simple name is inserted bare (`@app.ts`); a name containing whitespace is bracketed + * (`@[my file.txt]`) so it's captured whole instead of truncating at the first space. + */ + +/** + * Matches a mention token: a bracketed `@[name with spaces]` (where `\]` and `\\` are + * escaped, so a `]` inside the name doesn't end the token early) first, then a bare `@name`. + */ +export const MENTION_RE = /@\[(?:\\.|[^\]\\\r\n])*\]|@[\w/.\-\[\]]+/g + +/** The title of a mention token (`@name` or `@[name]`), brackets stripped and unescaped. */ +export function mentionTitle(token: string): string { + if (token.startsWith('@[') && token.endsWith(']')) { + return token.slice(2, -1).replace(/\\(.)/g, '$1') + } + return token.slice(1) +} + +/** Chars the bare `@name` regex matches without truncating; anything else needs brackets. */ +const BARE_SAFE = /^[\w/.\-]+$/ + +/** Format a name as a mention token. A bare `@name` only survives for simple names; anything + * with whitespace, HTML-sensitive chars (`< > &`), brackets, parens, etc. is bracketed (with + * `\` and `]` escaped) so the token is captured whole and round-trips through the parser. */ +export function formatMention(name: string): string { + return BARE_SAFE.test(name) ? `@${name}` : `@[${name.replace(/[\\\]]/g, '\\$&')}]` +} diff --git a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts index dfe3eb7565..1600d8fe9a 100644 --- a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts +++ b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts @@ -129,9 +129,17 @@ export class AIChatEditorHandler { const deletedChange = group.changes[0] const addedChange = group.changes[1] if (deletedChange.type === 'deleted' && addedChange.type === 'added_block') { - applyChange(this.editor, deletedChange) - addedChange.position.afterLineNumber = deletedChange.range.startLine - 1 - applyChange(this.editor, addedChange) + this.editor.executeEdits('chat', [ + { + range: { + startLineNumber: deletedChange.range.startLine, + startColumn: 1, + endLineNumber: deletedChange.range.endLine + 1, + endColumn: 0 + }, + text: addedChange.value + '\n' + } + ]) } else { throw new Error('Invalid group') } @@ -284,7 +292,7 @@ export class AIChatEditorHandler { }) if (!opts?.applyAll) { - ;({ collection, ids } = await displayVisualChanges( + ; ({ collection, ids } = await displayVisualChanges( 'editor-windmill-chat-style', this.editor, changes, diff --git a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte index 3709eda53c..d0a4cba462 100644 --- a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte @@ -17,6 +17,7 @@ import { AIMode } from '../AIChatManager.svelte' import { getAiChatManager } from '../aiChatManagerContext' import { Check, Play } from 'lucide-svelte' + import MermaidDisplay from './MermaidDisplay.svelte' const aiChatManager = getAiChatManager() @@ -90,7 +91,7 @@ if ( aiChatManager.mode !== AIMode.SCRIPT || !aiChatManager.scriptEditorApplyCode || - code === aiChatManager.scriptEditorOptions?.code + code === aiChatManager.scriptEditorOptions?.getCode() ) { return false } @@ -108,14 +109,18 @@
    - + {#if language === 'mermaid'} + + {:else} + + {/if}
    diff --git a/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte new file mode 100644 index 0000000000..06bcb63d98 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte @@ -0,0 +1,62 @@ + + +{#if showSvg} +
    + + {@html svg} +
    +{:else} + + 
    {code}
    +{/if} diff --git a/frontend/src/lib/components/copilot/chat/shared.test.ts b/frontend/src/lib/components/copilot/chat/shared.test.ts index 56b6bc4d2b..9e53e3c40c 100644 --- a/frontend/src/lib/components/copilot/chat/shared.test.ts +++ b/frontend/src/lib/components/copilot/chat/shared.test.ts @@ -136,14 +136,22 @@ describe('buildContextString', () => { path: 'f/flows/reporting', title: 'f/flows/reporting', summary: 'Reporting flow' + }, + { + type: 'workspace_app', + path: 'f/apps/dashboard', + title: 'f/apps/dashboard', + summary: 'Dashboard raw app' } ]) expect(context).toContain('SELECTED WORKSPACE ITEMS:') expect(context).toContain('- type: script, path: f/scripts/report') expect(context).toContain('- type: flow, path: f/flows/reporting') + expect(context).toContain('- type: raw_app, path: f/apps/dashboard') expect(context).not.toContain('Report script') expect(context).not.toContain('Reporting flow') + expect(context).not.toContain('Dashboard raw app') expect(context).not.toContain('Code:') expect(context).not.toContain('Value:') }) diff --git a/frontend/src/lib/components/copilot/chat/shared.ts b/frontend/src/lib/components/copilot/chat/shared.ts index aecea1ddfc..b958283972 100644 --- a/frontend/src/lib/components/copilot/chat/shared.ts +++ b/frontend/src/lib/components/copilot/chat/shared.ts @@ -422,6 +422,11 @@ export function buildContextString(selectedContext: ContextElement[]): string { workspaceItemsContext = 'SELECTED WORKSPACE ITEMS:\n' } workspaceItemsContext += `- type: flow, path: ${context.path}\n` + } else if (context.type === 'workspace_app') { + if (!workspaceItemsContext) { + workspaceItemsContext = 'SELECTED WORKSPACE ITEMS:\n' + } + workspaceItemsContext += `- type: raw_app, path: ${context.path}\n` } } @@ -936,7 +941,9 @@ export async function buildSchemaForTool( throw new Error(`Invalid flow inputs schema: ${invalidProperties.join(', ')}`) } - toolDef.function.parameters = { ...schema, additionalProperties: false } + // Anthropic requires input_schema.type to be present; flows with no inputs + // can produce a sparse schema (e.g. { order: [] }) lacking it. + toolDef.function.parameters = { type: 'object', ...schema, additionalProperties: false } // recursively normalize provider-incompatible schema fragments normalizeToolParameterSchema(toolDef.function.parameters) diff --git a/frontend/src/lib/components/custom_ui.ts b/frontend/src/lib/components/custom_ui.ts index 68ecf43a57..a04c9ea4b5 100644 --- a/frontend/src/lib/components/custom_ui.ts +++ b/frontend/src/lib/components/custom_ui.ts @@ -94,6 +94,11 @@ export type PreviewPanelUi = { // "what happened the last time this ran". No-op when a test is already // in progress (we don't clobber a live run). loadLastRunOnMount?: boolean + // Pipeline-only: when set, the Test split's caret popover gains a "Run + // downstream up to…" entry that calls this, letting the user bound the + // cascade from the script they're editing. Wired by the pipeline details + // pane only when the open script is a valid bounded-run start. + onBoundedRun?: () => void } export type EditorBarUi = { diff --git a/frontend/src/lib/components/dbOps.ts b/frontend/src/lib/components/dbOps.ts index 3a61e06d8b..f804bc14a8 100644 --- a/frontend/src/lib/components/dbOps.ts +++ b/frontend/src/lib/components/dbOps.ts @@ -1,5 +1,6 @@ import { getLanguageByResourceType, + ColumnIdentity, type ColumnDef, type TableMetadata } from './apps/components/display/dbtable/utils' @@ -48,7 +49,8 @@ export function dbTableOpsWithPreviewScripts({ tableKey, colDefs, workspace, - whereClause + whereClause, + version }: { input: DbInput tableKey: string @@ -57,6 +59,9 @@ export function dbTableOpsWithPreviewScripts({ // Optional raw SQL predicate AND-ed into the read queries (count + rows). // Caller-trusted — build it with escaped values. whereClause?: string + // DuckLake time-travel: when set, reads are pinned to this catalog snapshot + // via `AT (VERSION => n)` (DuckDB/ducklake only). Read-only by nature. + version?: number }): IDbTableOps { const dbType = getDbType(input) const language = getLanguageByResourceType(dbType) @@ -76,7 +81,8 @@ export function dbTableOpsWithPreviewScripts({ const content = makeMarker('COUNT', { table: tableKey, columnDefs: colDefs, - ...(whereClause ? { whereClause } : {}) + ...(whereClause ? { whereClause } : {}), + ...(version != undefined ? { version } : {}) }) const result = await runScriptAndPollResult({ workspace, @@ -90,7 +96,8 @@ export function dbTableOpsWithPreviewScripts({ table: tableKey, columnDefs: colDefs, fixPgIntTypes: true, - ...(whereClause ? { whereClause } : {}) + ...(whereClause ? { whereClause } : {}), + ...(version != undefined ? { version } : {}) }) let items = (await runScriptAndPollResult({ workspace, @@ -133,6 +140,90 @@ export function dbTableOpsWithPreviewScripts({ } } +export type DucklakeSnapshot = { + snapshot_id: number + // DuckLake returns this as microseconds-since-epoch serialized as a string + // (TIMESTAMP); callers must convert before formatting. + snapshot_time: string | number +} + +/** + * Column metadata of a ducklake table *at a specific snapshot*. The catalog's + * `information_schema` only reflects the current schema, so a time-travel read + * pinned to an older version must enumerate the columns that existed *then* — + * otherwise a column added in a later snapshot would break the `SELECT … AT + * (VERSION => n)`. `DESCRIBE SELECT * FROM … AT (VERSION => n)` gives exactly + * that. Returns minimal `ColumnDef`s (field + datatype) — enough for the + * read-only preview's SELECT/COUNT and grid headers. + */ +export async function fetchDucklakeColumnsAtVersion({ + workspace, + ducklake, + tableKey, + version +}: { + workspace: string + ducklake: string + tableKey: string + version: number +}): Promise { + // Quote each identifier part (schema.table) so a dotted/odd table name can't + // break the statement, and double single-quotes in the catalog name so it + // can't break out of the ATTACH string literal (mirrors the backend's + // `escape_sql_literal`). `version` is a number — injection-safe. + const quoted = tableKey + .split('.') + .map((p) => `"${p.replace(/"/g, '""')}"`) + .join('.') + const ducklakeLit = ducklake.replace(/'/g, "''") + const content = + `ATTACH 'ducklake://${ducklakeLit}' AS __dlv__; USE __dlv__; ` + + `DESCRIBE SELECT * FROM ${quoted} AT (VERSION => ${version});` + const rows = (await runScriptAndPollResult({ + workspace, + requestBody: { args: {}, language: 'duckdb', content } + })) as { column_name: string; column_type: string }[] + if (!Array.isArray(rows)) return [] + return rows.map((r) => ({ + field: r.column_name, + datatype: r.column_type, + defaultvalue: '', + isprimarykey: false, + isidentity: ColumnIdentity.No, + isnullable: 'YES' as const, + isenum: false + })) +} + +/** + * List a ducklake table's time-travel history, newest first. DuckLake snapshots + * are catalog-wide commits; passing `table` (schema-qualified, e.g. + * `main.events_daily`) scopes the list to snapshots where the table exists — + * otherwise an `AT (VERSION => n)` read could target a version predating the + * table's creation and error. Runs the `DUCKLAKE_SNAPSHOTS` marker as a duckdb + * preview job (server-side SQL build + ATTACH), so no raw SQL is constructed in + * the client. + */ +export async function fetchDucklakeSnapshots({ + workspace, + ducklake, + table +}: { + workspace: string + ducklake: string + table?: string +}): Promise { + const content = `-- WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS ${JSON.stringify({ + ducklake, + ...(table ? { table } : {}) + })}` + const rows = await runScriptAndPollResult({ + workspace, + requestBody: { args: {}, language: 'duckdb', content } + }) + return Array.isArray(rows) ? (rows as DucklakeSnapshot[]) : [] +} + export type IDbSchemaOps = { onDelete: (params: { tableKey: string; schema?: string }) => Promise onCreate: (params: { values: TableEditorValues; schema?: string }) => Promise diff --git a/frontend/src/lib/components/flow_builder.ts b/frontend/src/lib/components/flow_builder.ts index 1fe043707b..48340e937e 100644 --- a/frontend/src/lib/components/flow_builder.ts +++ b/frontend/src/lib/components/flow_builder.ts @@ -24,6 +24,9 @@ export type FlowBuilderProps = { disabledFlowInputs?: boolean savedPrimarySchedule?: ScheduleTrigger | undefined // used to set the primary schedule in the legacy primaryScheduleStore version?: number | undefined + /** flow_version the draft was forked from; when set, the deploy-time staleness + * check compares it (not the load-time head `version`) against the latest. */ + draftBaseVersion?: number | undefined draftTriggersFromUrl?: Trigger[] | undefined selectedTriggerIndexFromUrl?: number | undefined children?: import('svelte').Snippet diff --git a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte index 1bf578a081..46a5c78d32 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte @@ -867,7 +867,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { @@ -931,7 +930,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { diff --git a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte index 39aaaa66f2..62a3432995 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte @@ -108,9 +108,14 @@ {/if} -
    - -

    The timeout will be ignored when running "Test this step"

    -     -
    + {#if flowModule.timeout && flowModule.timeout.type !== 'static'} +
    + +

    + A dynamic timeout expression is evaluated when running the full flow. It is ignored when + running "Test this step" — only a static timeout value applies there. +

    +     +
    + {/if} diff --git a/frontend/src/lib/components/flows/flowModuleNextId.test.ts b/frontend/src/lib/components/flows/flowModuleNextId.test.ts new file mode 100644 index 0000000000..dcf007e925 --- /dev/null +++ b/frontend/src/lib/components/flows/flowModuleNextId.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it } from 'vitest' + +import type { OpenFlow } from '$lib/gen' +import type { FlowState } from './flowState' +import { nextId } from './flowModuleNextId' + +function flowWith(ids: string[]): OpenFlow { + return { + summary: '', + value: { + modules: ids.map((id) => ({ id, value: { type: 'identity' } as any })) + } + } as OpenFlow +} + +function stateWith(keys: string[]): FlowState { + return Object.fromEntries(keys.map((k) => [k, {}])) as FlowState +} + +describe('nextId', () => { + it('produces a, b, c, ... for a fresh flow', () => { + expect(nextId(stateWith(['failure']), flowWith([]))).toBe('a') + expect(nextId(stateWith(['a', 'failure']), flowWith(['a']))).toBe('b') + expect(nextId(stateWith(['a', 'b', 'c', 'failure']), flowWith(['a', 'b', 'c']))).toBe('d') + }) + + it('ignores the reserved failure/preprocessor keys always present in flowState', () => { + expect(nextId(stateWith(['failure', 'preprocessor']), flowWith([]))).toBe('a') + }) + + // Regression: copy ids ("z2"), subflow result keys and other non-canonical keys land in + // flowState; charsToNumber on them used to leak into the max and made new steps jump to + // garbage ids like "bzw". + it('is not poisoned by copy ids', () => { + const ids = ['a', 'b', 'c'] + const state = stateWith([...ids, 'c2', 'a2', 'z2', 'c10', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('d') + }) + + it('is not poisoned by subflow result keys', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'subflow:abcd', 'Result', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) + + // A step renamed to a long lowercase word ("process") is a valid base-26 string and would + // otherwise inflate the max; the length cutoff keeps such renames out of the sequence. + it('is not poisoned by renames to long lowercase words or underscored ids', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'process', 'my_step', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) +}) diff --git a/frontend/src/lib/components/flows/flowModuleNextId.ts b/frontend/src/lib/components/flows/flowModuleNextId.ts index e10c900982..48b2eb5ac2 100644 --- a/frontend/src/lib/components/flows/flowModuleNextId.ts +++ b/frontend/src/lib/components/flows/flowModuleNextId.ts @@ -1,19 +1,35 @@ import type { OpenFlow } from '$lib/gen' import { dfs } from './dfs' import type { FlowState } from './flowState' -import { charsToNumber, numberToChars } from './idUtils' +import { charsToNumber, forbiddenIds, numberToChars } from './idUtils' + +const reservedIds = new Set(forbiddenIds) + +// Returns the base-26 value of a key only if it is a short, auto-generated step id +// (a, b, ..., z, aa, ...). flowState/module-id keys also include copy ids ("a2"), subflow +// result keys ("subflow:..."), reserved keys and user-renamed ids; feeding those through +// charsToNumber yields meaningless (often huge) numbers that would poison id generation and +// make new steps jump to ids like "bzw". Short non-canonical keys are rejected via a +// round-trip check; longer keys are skipped entirely, which also leaves user renames to long +// lowercase words (e.g. "process") out of the sequence. +function autoIdNumber(key: string): number | undefined { + if (key.length >= 4 || reservedIds.has(key)) { + return undefined + } + const num = charsToNumber(key) + if (num < 0 || numberToChars(num) !== key) { + return undefined + } + return num +} // Computes the next available id export function nextId(flowState: FlowState, fullFlow: OpenFlow): string { const allIds = dfs(fullFlow.value.modules, (fm) => fm.id) const max = allIds.concat(Object.keys(flowState)).reduce((acc, key) => { - if (key.length >= 4) { - return acc - } else { - const num = charsToNumber(key) - return Math.max(acc, num + 1) - } + const num = autoIdNumber(key) + return num === undefined ? acc : Math.max(acc, num + 1) }, 0) return numberToChars(max) } diff --git a/frontend/src/lib/components/flows/map/VirtualItem.svelte b/frontend/src/lib/components/flows/map/VirtualItem.svelte index f7a08f1933..652cf4041d 100644 --- a/frontend/src/lib/components/flows/map/VirtualItem.svelte +++ b/frontend/src/lib/components/flows/map/VirtualItem.svelte @@ -9,7 +9,11 @@ import { Database, Square } from 'lucide-svelte' import FlowGraphPreviewButton from './FlowGraphPreviewButton.svelte' import type { Job } from '$lib/gen' - import { getNodeColorClasses, aiActionToNodeState } from '$lib/components/graph' + import { + getNodeColorClasses, + aiActionToNodeState, + type FlowNodeState + } from '$lib/components/graph' import { getGraphContext } from '$lib/components/graph/graphContext' interface Props { @@ -39,6 +43,9 @@ job?: Job showJobStatus?: boolean flowHasChanged?: boolean + /** When set, overrides the node outline with this run-state's colored outline. + * Used to mark the branch taken at runtime on branchone/branchall nodes. */ + borderState?: FlowNodeState } let { @@ -67,14 +74,13 @@ individualStepTests = false, job, showJobStatus = false, - flowHasChanged = false + flowHasChanged = false, + borderState = undefined }: Props = $props() const flowGraphContext = getGraphContext() - let isMultiSelected = $derived( - (flowGraphContext?.selectionManager?.selectedIds?.length ?? 0) > 1 - ) + let isMultiSelected = $derived((flowGraphContext?.selectionManager?.selectedIds?.length ?? 0) > 1) const outputPickerVisible = $derived( (nodeKind || (inputJson && Object.keys(inputJson).length > 0)) && editMode @@ -96,6 +102,10 @@ // AI action colors take priority over execution state, fallback to _VirtualItem const effectiveState = $derived(aiActionToNodeState(action) ?? outputType ?? '_VirtualItem') let colorClasses = $derived(getNodeColorClasses(effectiveState, selected)) + // The branch taken at runtime keeps its outline regardless of selection so it stays visible. + let outlineClasses = $derived( + borderState ? getNodeColorClasses(borderState, true).outline : colorClasses.outline + )
    diff --git a/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts b/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts index 06a4972ee5..8aa7a1f69f 100644 --- a/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts +++ b/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts @@ -100,9 +100,11 @@ export class StepsInputArgs { if (modules.length < 1) { return } + // dfs returns [step, immediate parent, ..., root]; the prop picker needs the + // immediate parent so nested loops resolve flow_input.iter to the innermost loop. let parentModule: FlowModule | undefined = undefined if (modules.length > 1) { - parentModule = modules[modules.length - 1] + parentModule = modules[1] } const stepPropPicker = getStepPropPicker( flowState, @@ -175,9 +177,11 @@ export class StepsInputArgs { if (modules.length < 1) { return } + // dfs returns [step, immediate parent, ..., root]; the prop picker needs the + // immediate parent so nested loops resolve flow_input.iter to the innermost loop. let parentModule: FlowModule | undefined = undefined if (modules.length > 1) { - parentModule = modules[modules.length - 1] + parentModule = modules[1] } const stepPropPicker = getStepPropPicker( flowState, diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte index 1f2623e552..9682d941b6 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchAllStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchAllStartN['data'] id: string @@ -14,6 +15,10 @@ let { data, id }: Props = $props() const { selectionManager } = getGraphContext() + + let borderStatus = $derived( + computeBorderStatus(data.branchIndex, 'branchall', data.flowModuleState) + ) @@ -22,6 +27,7 @@ label={data.label} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data.eventHandlers.select(data.id)) }} diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte index 6333d32845..9902582ee0 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchOneStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchOneStartN['data'] id: string @@ -13,6 +14,12 @@ const { selectionManager } = getGraphContext() let { data, id }: Props = $props() + + // branchIndex is -1 for the default branch and 0-based for explicit branches; + // branchChosen is 0 for default and 1-based, hence the +1. + let borderStatus = $derived( + computeBorderStatus(data.branchIndex + 1, 'branchone', data.flowModuleState) + ) @@ -22,11 +29,12 @@ preLabel={data.preLabel} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data?.eventHandlers?.select(data.id)) }} /> - {#if data.insertable} + {#if data.insertable && data.branchIndex >= 0} + +
    + {/if} + {#if guarded} +
    + + Large file — {lineCount.toLocaleString()} lines. + + +
    + {:else} + {#await import('$lib/components/DiffEditor.svelte')} +
    + {:then Module} +
    + +
    + {/await} + {/if} + diff --git a/frontend/src/lib/components/raw_apps/RawAppPreview.svelte b/frontend/src/lib/components/raw_apps/RawAppPreview.svelte index 296590a0e4..9a70b162d7 100644 --- a/frontend/src/lib/components/raw_apps/RawAppPreview.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppPreview.svelte @@ -2,8 +2,8 @@ import { type UserExt } from '$lib/stores' import RawAppBackgroundRunner from './RawAppBackgroundRunner.svelte' import type { Runnable } from './rawAppPolicy' - import { htmlContent } from './utils' - import { onMount, untrack } from 'svelte' + import { getContext, onMount, untrack } from 'svelte' + import { unsandboxedRawAppHtml } from './utils' interface Props { workspace: string @@ -17,43 +17,190 @@ let iframe = $state() as HTMLIFrameElement | undefined - // Get initial hash from parent URL to pass to iframe - let initialHash = $state('') + // Get initial hash from parent URL to pass to the iframe + let initialHash = '' - onMount(() => { - initialHash = window.location.hash || '' + // WIN-2006: unless the publisher opted into sandbox isolation, run the bundle + // same-origin with full access (the default); otherwise the opaque-origin sandbox. + const unsandboxedCtx = getContext<{ value: boolean }>('IS_APP_UNSANDBOXED') + let unsandboxed = $derived(unsandboxedCtx?.value ?? false) + // Unsandboxed (the default) must match the pre-isolation viewer exactly: NO + // sandbox attribute (a same-origin blob with full session — an attribute would + // only break leftover features like unsandboxed popups for OAuth flows, while + // adding no isolation). The sandboxed path keeps the restrictive attribute; the + // wrapper document's `CSP: sandbox` response header enforces the opaque origin + // regardless. + let sandboxAttr = $derived( + unsandboxed + ? undefined + : 'allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-downloads allow-modals allow-top-navigation' + ) + + // WIN-2006: source of the bundle iframe. + // - DEFAULT (isolated): a real API URL serving a sandboxed, opaque-origin + // document (`CSP: sandbox` response header + the iframe sandbox attribute), + // so a malicious bundle can never reach the authenticated Windmill origin + // (no cookie, no window.parent, no token). Root-relative so it resolves + // against the real host even when this component itself runs inside an opaque + // viewer (where `location.origin` is "null"). Context is handed over via + // postMessage — never baked into the document, never a credential. + // - UNSANDBOXED (the default — publisher did not opt into isolation): a + // client-built blob: wrapper (same-origin with the SPA) loaded with `allow-same-origin`, + // so relative `fetch('/api/...')` and the session cookie work. The backend + // `.html` is ALWAYS sandboxed, so we must build the same-origin wrapper here + // rather than relax a real-origin endpoint a victim could be linked to. + let iframeSrc = $derived.by(() => { + if (!secret || typeof window === 'undefined') return undefined + if (unsandboxed) { + // untrack(user) so userStore refreshes don't regenerate the blob URL and + // reload the iframe (losing state); ctx is only needed for initial render. + // Always pass the wrapper object — pre-sandbox bundles rely on + // `window.ctx.workspace` even for anonymous viewers (ctx.ctx undefined). + const u = untrack(() => user) + const html = unsandboxedRawAppHtml( + workspace, + secret, + { ctx: u, workspace }, + window.location.origin, + window.location.hash || '' + ) + return URL.createObjectURL(new Blob([html], { type: 'text/html' })) + } + // `wm_coep` (embed-in-cross-origin-isolated-page opt-in) must be propagated + // to the wrapper document: under a COEP `require-corp` embedder, a nested + // document is only allowed to load if it asserts COEP itself, so the + // backend adds the header when the flag is present. + const coep = new URLSearchParams(window.location.search).has('wm_coep') ? '?wm_coep=1' : '' + return `/api/w/${workspace}/apps_u/get_data/v/${secret}.html${coep}` }) - // Use blob URL instead of srcDoc to give the iframe a proper origin. - // srcDoc iframes have "null" origin which breaks URL constructor in routers. - // untrack(user) so that userStore refreshes don't regenerate the blob URL - // and cause the iframe to fully reload (losing all state). - // The user context is only needed for initial render. - let blobUrl = $derived.by(() => { - if (!secret) return undefined - const u = untrack(() => user) - const baseUrl = typeof window !== 'undefined' ? window.location.origin : '' - const html = htmlContent(workspace, secret, { ctx: u, workspace }, baseUrl, initialHash) - const blob = new Blob([html], { type: 'text/html' }) - return URL.createObjectURL(blob) - }) - - // Cleanup blob URL when it changes or component unmounts + // Revoke blob: URLs (unsandboxed path) when they change or on unmount. $effect(() => { - const url = blobUrl + const url = iframeSrc return () => { - if (url) URL.revokeObjectURL(url) + if (url && url.startsWith('blob:')) URL.revokeObjectURL(url) + } + }) + + // Persistence for the bundle's (opaque-origin) localStorage, backed by a store + // scoped PER APP (keyed by workspace + app path) so one sandboxed app can't read + // or clobber another's (even two apps at the same path in different workspaces). On a real origin (workspace viewer, public page — even when + // that page sits inside someone else's iframe) it reads/writes real localStorage + // directly. Only inside an opaque frame (the Windmill embed viewer), where Web + // Storage throws, does it relay per-key ops up to the embedder, the persistence + // authority. `framed` therefore probes storage rather than just `window.parent`: + // an externally-embedded public page is framed too, but its parent is not the + // Windmill embedder and would never answer the relay (leaving the bundle without + // ctx). The snapshot is handed to the bundle before it evaluates so its + // localStorage is hydrated synchronously. + const SHARED_LS_KEY = `wm_apps_localstorage:${workspace}:${path}` + function storageAccessible(): boolean { + try { + localStorage.getItem(SHARED_LS_KEY) + return true + } catch (_) { + return false + } + } + const framed = typeof window !== 'undefined' && window.parent !== window && !storageAccessible() + let bundleStorage: Record | undefined = undefined + let pendingReady = false + + function readDirect(): Record { + try { + return JSON.parse(localStorage.getItem(SHARED_LS_KEY) || '{}') + } catch (_) { + return {} + } + } + + function applyDirectOp(d: any) { + try { + const s = readDirect() + if (d.op === 'set') s[d.key] = String(d.value) + else if (d.op === 'remove') delete s[d.key] + else if (d.op === 'clear') for (const k in s) delete s[k] + localStorage.setItem(SHARED_LS_KEY, JSON.stringify(s)) + } catch (_) {} + } + + function respondCtx() { + iframe?.contentWindow?.postMessage( + { + type: 'windmill:ctx', + // Same shape as the unsandboxed wrapper: always the object, so + // `window.ctx.workspace` works for anonymous viewers too. + ctx: { ctx: user, workspace }, + initialHash, + storage: { local: bundleStorage ?? {}, session: {} } + }, + '*' + ) + } + + onMount(() => { + initialHash = window.location.hash || '' + if (framed) { + // Pre-fetch the shared store from the embedder. + try { + window.parent.postMessage({ type: 'wm_ls_req' }, '*') + } catch (_) {} + // If the parent never answers (it isn't the Windmill embedder, e.g. an + // opaque context created by a third party), don't hold the bundle's ctx + // hostage: proceed with empty storage. Must beat the backend wrapper's + // own 1.5s no-ctx fallback. + const fallback = setTimeout(() => { + if (bundleStorage === undefined) { + bundleStorage = {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + } + }, 750) + return () => clearTimeout(fallback) } }) - // Listen for hash changes from iframe and update parent URL $effect(() => { function handleMessage(event: MessageEvent) { - console.log('[Parent] Received message:', event.data) - if (event.data?.type === 'windmill:hashchange') { - const newHash = event.data.hash || '' - console.log('[Parent] Updating hash to:', newHash) - // Update parent URL without triggering navigation + const data = event.data + // Shared-store hydration from the embedder (public mode only). + if (framed && event.source === window.parent && data?.type === 'wm_ls_hydrate') { + bundleStorage = data.data || {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + return + } + // Everything else must come from the bundle iframe. + if (event.source !== iframe?.contentWindow) return + if (data?.type === 'windmill:ready') { + // Hand the bundle its context + shared storage before it evaluates. + if (!framed) { + bundleStorage = readDirect() + respondCtx() + } else if (bundleStorage !== undefined) { + respondCtx() + } else { + pendingReady = true + } + } else if (data?.type === 'wm_ls_op') { + // The bundle mutated localStorage — apply it to the shared store. + if (!framed) { + applyDirectOp(data) + } else { + try { + window.parent.postMessage( + { type: 'wm_ls_op', op: data.op, key: data.key, value: data.value }, + '*' + ) + } catch (_) {} + } + } else if (data?.type === 'windmill:hashchange') { + // Keep the parent URL hash in sync for shareable URLs. + const newHash = data.hash || '' if (window.location.hash !== newHash) { history.replaceState(null, '', newHash || window.location.pathname) } @@ -65,13 +212,29 @@ }) - + -{#if blobUrl} +{#if iframeSrc} + + {/if} diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts new file mode 100644 index 0000000000..300b5dde1e --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts @@ -0,0 +1,299 @@ +import { describe, expect, it } from 'vitest' +import { + parseRawAppDiff, + rawAppDiffToItems, + RAW_APP_METADATA_PATH, + type RawAppDiffEntry +} from './rawAppDiffUtils' + +function byPath(entries: RawAppDiffEntry[], path: string): RawAppDiffEntry | undefined { + return entries.find((e) => e.path === path) +} + +describe('parseRawAppDiff — files', () => { + it('detects added, removed and modified files, omits unchanged', () => { + const original = { + files: { 'index.html': '

    hi

    ', 'styles.css': 'body{}', 'gone.js': 'x' } + } + const current = { + files: { 'index.html': '

    hello

    ', 'styles.css': 'body{}', 'new.ts': 'y' } + } + const entries = parseRawAppDiff(original, current) + const paths = entries.map((e) => e.path).sort() + // styles.css unchanged → omitted + expect(paths).toEqual(['gone.js', 'index.html', 'new.ts']) + + expect(byPath(entries, 'index.html')?.status).toBe('modified') + expect(byPath(entries, 'index.html')?.lang).toBe('html') + expect(byPath(entries, 'gone.js')?.status).toBe('removed') + expect(byPath(entries, 'new.ts')?.status).toBe('added') + expect(byPath(entries, 'new.ts')?.lang).toBe('typescript') + }) + + it('carries original/current content on the right sides', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': 'old', 'b.txt': 'keep' } }, + { files: { 'a.txt': 'new', 'b.txt': 'keep' } } + ) + const a = byPath(entries, 'a.txt')! + expect(a.original).toBe('old') + expect(a.current).toBe('new') + }) +}) + +describe('parseRawAppDiff — runnables', () => { + it('emits per-runnable leaves for add/remove/modify', () => { + const original = { + runnables: { a: { path: 'u/x/a' }, b: { path: 'u/x/b' }, same: { path: 'u/x/same' } } + } + const current = { + runnables: { a: { path: 'u/x/a2' }, c: { path: 'u/x/c' }, same: { path: 'u/x/same' } } + } + const entries = parseRawAppDiff(original, current) + expect(byPath(entries, 'runnables/a')?.status).toBe('modified') + expect(byPath(entries, 'runnables/a')?.lang).toBe('yaml') + expect(byPath(entries, 'runnables/b')?.status).toBe('removed') + expect(byPath(entries, 'runnables/c')?.status).toBe('added') + // identical runnable omitted + expect(byPath(entries, 'runnables/same')).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — metadata', () => { + it('collapses summary/data/policy/custom_path into one app.yaml leaf', () => { + const entries = parseRawAppDiff( + { summary: 'old summary', custom_path: 'foo' }, + { summary: 'new summary', custom_path: 'foo' } + ) + const meta = byPath(entries, RAW_APP_METADATA_PATH) + expect(meta?.status).toBe('modified') + expect(meta?.lang).toBe('yaml') + expect(meta?.original).toContain('old summary') + expect(meta?.current).toContain('new summary') + }) + + it('omits app.yaml when no metadata field changed', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': '1' }, summary: 's' }, + { files: { 'a.txt': '2' }, summary: 's' } + ) + expect(byPath(entries, RAW_APP_METADATA_PATH)).toBeUndefined() + expect(entries).toHaveLength(1) + }) +}) + +describe('parseRawAppDiff — whole app added / removed', () => { + it('marks everything added when the original side is absent', () => { + const current = { + files: { 'index.html': '

    hi

    ' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'brand new' + } + const entries = parseRawAppDiff(undefined, current) + expect(entries.every((e) => e.status === 'added')).toBe(true) + expect(byPath(entries, 'index.html')?.original).toBeUndefined() + expect(byPath(entries, RAW_APP_METADATA_PATH)?.status).toBe('added') + }) + + it('marks everything removed when the current side is absent', () => { + const original = { + files: { 'index.html': '

    hi

    ' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'going away' + } + const entries = parseRawAppDiff(original, undefined) + expect(entries.every((e) => e.status === 'removed')).toBe(true) + expect(byPath(entries, 'index.html')?.current).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — collisions', () => { + it('disambiguates synthesized paths against real files', () => { + const original = { files: { 'app.yaml': 'real-old' }, summary: 'sa' } + const current = { files: { 'app.yaml': 'real-new' }, summary: 'sb' } + const entries = parseRawAppDiff(original, current) + // The real file keeps the natural path. + const realFile = byPath(entries, 'app.yaml')! + expect(realFile.original).toBe('real-old') + // The metadata leaf is pushed to a non-colliding path. + const meta = byPath(entries, 'app.yaml~2')! + expect(meta.original).toContain('sa') + expect(meta.current).toContain('sb') + // No two entries share a path. + const paths = entries.map((e) => e.path) + expect(new Set(paths).size).toBe(paths.length) + }) +}) + +describe('parseRawAppDiff — input shapes', () => { + // getItemValue returns the deployed app row: files/runnables/data live under + // `value`; summary/policy/custom_path at the top level. + it('reads files/runnables/data from the `value` wrapper (app-row shape)', () => { + const original = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

    a

    ' }, runnables: {}, data: {} } + } + const current = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

    b

    ' }, runnables: {}, data: {} } + } + const entries = parseRawAppDiff(original, current) + const file = byPath(entries, 'index.html') + expect(file?.status).toBe('modified') + expect(file?.original).toBe('

    a

    ') + expect(file?.current).toBe('

    b

    ') + }) + + it('still handles the flat draft shape (files at top level)', () => { + const entries = parseRawAppDiff({ files: { 'a.ts': 'x' } }, { files: { 'a.ts': 'y' } }) + expect(byPath(entries, 'a.ts')?.status).toBe('modified') + }) + + it('prefers `value` over a stray top-level files key', () => { + const entries = parseRawAppDiff( + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'a' } } }, + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'b' } } } + ) + expect(byPath(entries, 'real.ts')?.status).toBe('modified') + expect(byPath(entries, 'top.ts')).toBeUndefined() + }) +}) + +describe('rawAppDiffToItems', () => { + const appPath = 'u/admin/classy_app' + + it('produces composite-pathed items with status-derived exists flags', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'a', '/gone.ts': 'x' }, runnables: { r: { p: 1 } } } }, + { value: { files: { '/App.tsx': 'b', '/new.ts': 'y' }, runnables: {} } } + ) + const byPath = (p: string) => items.find((i) => i.path === p) + + const app = byPath('u/admin/classy_app/App.tsx')! + expect(app.kind).toBe('raw_app_file') + expect(app.status).toBe('modified') + expect(app.exists_in_source).toBe(true) + expect(app.exists_in_fork).toBe(true) + expect(app.appPath).toBe(appPath) + expect((app as any).lang).toBe('typescript') + + const gone = byPath('u/admin/classy_app/gone.ts')! + expect(gone.status).toBe('removed') + expect(gone.exists_in_source).toBe(true) + expect(gone.exists_in_fork).toBe(false) + + const added = byPath('u/admin/classy_app/new.ts')! + expect(added.status).toBe('added') + expect(added.exists_in_source).toBe(false) + expect(added.exists_in_fork).toBe(true) + + // Runnables render as script/flow rows, not file leaves. + const runnable = byPath('u/admin/classy_app/runnables/r')! + expect(runnable.kind).toBe('script') + expect(runnable.status).toBe('removed') + }) + + it('renders an inline-script runnable as a script row with code hoisted', () => { + const mk = (code: string) => ({ + value: { + files: {}, + runnables: { + a: { name: 'a', type: 'inline', inlineScript: { content: code, language: 'bun' } } + } + } + }) + const items = rawAppDiffToItems(appPath, mk('old code'), mk('new code')) + const r = items.find((i) => i.path === `${appPath}/runnables/a`) + expect(r?.kind).toBe('script') + expect(r?.status).toBe('modified') + // content + language hoisted to the top level for the script-style viewer + const cur = (r as any).currentRaw + expect(cur.content).toBe('new code') + expect(cur.language).toBe('bun') + expect(cur.inlineScript.content).toBeUndefined() + }) + + it('picks the flow kind for a runType:flow runnable', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: {}, runnables: {} } }, + { value: { files: {}, runnables: { f: { runType: 'flow', path: 'u/x/f' } } } } + ) + const r = items.find((i) => i.path === `${appPath}/runnables/f`) + expect(r?.kind).toBe('flow') + expect(r?.status).toBe('added') + }) + + it('flags the metadata item and attaches whole-app YAML for the expand view', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: {} } }, + { summary: 'new', value: { files: {} } } + ) + const meta = items.find((i) => i.path === `${appPath}/${RAW_APP_METADATA_PATH}`) as any + expect(meta.kind).toBe('raw_app_file') + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlOriginal).toContain('old') + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named app.yaml', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { 'app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { 'app.yaml': 'real-new' } } } + ) + // Real file keeps the natural path and is NOT the metadata item. + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + expect(realFile.isMetadata).toBe(false) + expect(realFile.fullYamlCurrent).toBeUndefined() + // Synthesized metadata moved to app.yaml~2 but still carries the flag + YAML. + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named /app.yaml (leading slash)', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { '/app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { '/app.yaml': 'real-new' } } } + ) + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(realFile.isMetadata).toBe(false) + expect(meta.isMetadata).toBe(true) + // distinct composite paths → distinct row keys + expect(realFile.path).not.toBe(meta.path) + }) + + it('treats a file keyed /App.tsx on one side and App.tsx on the other as one item', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'old' } } }, + { value: { files: { 'App.tsx': 'new' } } } + ) + const files = items.filter((i) => i.path === `${appPath}/App.tsx`) + expect(files).toHaveLength(1) + expect(files[0].status).toBe('modified') + expect((files[0] as any).original).toBe('old') + expect((files[0] as any).current).toBe('new') + }) + + it('dedups a runnable leaf against a real file named runnables/', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/runnables/foo': 'old' }, runnables: { foo: { p: 1 } } } }, + { value: { files: { '/runnables/foo': 'new' }, runnables: { foo: { p: 2 } } } } + ) + const realFile = items.find((i) => i.kind === 'raw_app_file')! + const runnable = items.find((i) => i.kind === 'script')! + expect(realFile.path).toBe(`${appPath}/runnables/foo`) + // Reserved away from the real file's composite path (slash-normalized). + expect(runnable.path).toBe(`${appPath}/runnables/foo~2`) + expect(runnable.path).not.toBe(realFile.path) + }) +}) diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts new file mode 100644 index 0000000000..444e03a0a3 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts @@ -0,0 +1,383 @@ +import { extToLang } from '$lib/editorLangUtils' +import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils' + +// A raw app rendered as a *folder of files* for diffing. Each entry is one +// virtual file: real `files` keep their natural path, runnables become +// `runnables/`, and the remaining app metadata collapses into a single +// `app.yaml` leaf. Only changed entries are emitted (unchanged are omitted). +export type RawAppDiffStatus = 'added' | 'removed' | 'modified' + +export interface RawAppDiffEntry { + /** Tree path. Real file path, `runnables/`, or `app.yaml`. */ + path: string + status: RawAppDiffStatus + /** Original (parent) side content. Undefined when status is `added`. */ + original?: string + /** Current (fork) side content. Undefined when status is `removed`. */ + current?: string + /** Monaco language id for the per-file diff editor. */ + lang: string + /** True for the synthesized metadata leaf. Tracked as a flag (not by matching + * `path === 'app.yaml'`) so it survives `reserveUnique` moving the leaf to + * `app.yaml~2` when a real file is literally named `app.yaml`. */ + isMetadata?: boolean +} + +// Loose shape — diff inputs come from `getItemValue` / drafts and aren't +// strictly typed, and arrive in TWO shapes: +// - the deployed app row (getAppByPath): `{ summary, policy, custom_path?, +// value: { files, runnables, data } }` — files/runnables/data nested under +// `value`. +// - the flat draft/runtime shape (RawAppDraft / RuntimeRawApp): everything at +// the top level. +// `normalizeRawApp` coalesces both. Anything not present is empty/absent. +export interface RawAppish { + value?: Record + files?: Record + runnables?: Record + summary?: unknown + data?: unknown + policy?: unknown + custom_path?: unknown + [k: string]: unknown +} + +interface NormalizedRawApp { + files: Record + runnables: Record + summary: unknown + data: unknown + policy: unknown + custom_path: unknown +} + +export const RAW_APP_METADATA_PATH = 'app.yaml' +const RUNNABLES_PREFIX = 'runnables/' +const METADATA_FIELDS = ['summary', 'data', 'policy', 'custom_path'] as const + +// Real file keys may carry a leading slash (`/App.tsx`) which `joinAppPath` +// strips. Collision reservation must compare in the stripped space, else a real +// `/app.yaml` and the synthetic `app.yaml` leaf both become `/app.yaml`. +const stripLeadingSlash = (p: string) => p.replace(/^\/+/, '') + +function isObject(v: unknown): v is Record { + return !!v && typeof v === 'object' +} + +function asFileMap(f: unknown): Record { + if (!isObject(f)) return {} + const out: Record = {} + for (const [k, v] of Object.entries(f)) { + // Canonicalize the key (strip the leading slash `joinAppPath` would strip + // anyway) so the same file keyed `/App.tsx` on one side and `App.tsx` on the + // other is treated as one file — not two leaves colliding at the same + // composite path. Coerce non-string content so the diff editor gets a string. + out[stripLeadingSlash(k)] = typeof v === 'string' ? v : String(v ?? '') + } + return out +} + +// Coalesce the two raw-app shapes (app row with a `value` wrapper vs flat draft) +// into a canonical view. Returns undefined when the whole app is absent. +// +// Per-field precedence is deliberately asymmetric and mirrors the app-row shape: +// the editor payload (`files`/`runnables`/`data`) lives under `value`, so those +// prefer `value` first; the row-level metadata (`summary`/`policy`/`custom_path`) +// lives at the top level, so those prefer `raw` first. Each falls back to the +// other side so a flat draft (everything top-level) still normalizes correctly. +function normalizeRawApp(raw: RawAppish | undefined): NormalizedRawApp | undefined { + if (!isObject(raw)) return undefined + const value = isObject(raw.value) ? raw.value : undefined + return { + files: asFileMap(value?.files ?? raw.files), + runnables: isObject(value?.runnables ?? raw.runnables) + ? ((value?.runnables ?? raw.runnables) as Record) + : {}, + summary: raw.summary ?? value?.summary, + data: value?.data ?? raw.data, + policy: raw.policy ?? value?.policy, + custom_path: raw.custom_path ?? value?.custom_path + } +} + +// The serialized metadata blob for one side, or undefined when the whole app +// is absent (so the `app.yaml` leaf reads as added/removed). +function metadataYaml(app: NormalizedRawApp | undefined): string | undefined { + if (!app) return undefined + const meta: Record = {} + for (const field of METADATA_FIELDS) { + if (app[field] !== undefined) meta[field] = app[field] + } + return orderedYamlStringify(meta) +} + +function extOf(path: string): string { + const base = path.split('/').pop() ?? path + const dot = base.lastIndexOf('.') + return dot >= 0 ? base.slice(dot + 1).toLowerCase() : '' +} + +function diffStatus(o: string | undefined, c: string | undefined): RawAppDiffStatus | undefined { + if (o === undefined && c !== undefined) return 'added' + if (o !== undefined && c === undefined) return 'removed' + if (o !== c) return 'modified' + return undefined +} + +// Reserve a synthesized path, disambiguating against already-taken paths so a +// real file literally named `app.yaml` or under `runnables/` never collides +// with a synthesized leaf. +function reserveUnique(path: string, taken: Set): string { + if (!taken.has(path)) { + taken.add(path) + return path + } + let i = 2 + while (taken.has(`${path}~${i}`)) i++ + const p = `${path}~${i}` + taken.add(p) + return p +} + +/** + * Diff two raw-app objects into a flat list of changed virtual files. Either + * side may be undefined (whole app added or removed). Consumers build a tree + * from the returned paths with `buildFileTree`. + */ +export function parseRawAppDiff( + original: RawAppish | undefined, + current: RawAppish | undefined, + opts: { includeRunnables?: boolean } = {} +): RawAppDiffEntry[] { + const { includeRunnables = true } = opts + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oFiles = oApp?.files ?? {} + const cFiles = cApp?.files ?? {} + const oRunnables = oApp?.runnables ?? {} + const cRunnables = cApp?.runnables ?? {} + + // Reserve every real file path (changed or not) up front so synthesized + // runnable/metadata paths can dodge collisions — slash-normalized so a real + // `/app.yaml` or `/runnables/x` is seen as colliding with the synthetic leaf. + const taken = new Set( + [...Object.keys(oFiles), ...Object.keys(cFiles)].map(stripLeadingSlash) + ) + + const entries: RawAppDiffEntry[] = [] + + // Real files. + for (const path of [...new Set([...Object.keys(oFiles), ...Object.keys(cFiles)])].sort()) { + const o = Object.prototype.hasOwnProperty.call(oFiles, path) ? oFiles[path] : undefined + const c = Object.prototype.hasOwnProperty.call(cFiles, path) ? cFiles[path] : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ path, status, original: o, current: c, lang: extToLang(extOf(path)) }) + } + + // Runnables → one YAML leaf each under `runnables/`. Consumers that render + // runnables as script/flow rows (rawAppDiffToItems) skip these and diff the + // runnable objects themselves. + const runnableNames = includeRunnables + ? [...new Set([...Object.keys(oRunnables), ...Object.keys(cRunnables)])].sort() + : [] + for (const name of runnableNames) { + const inO = Object.prototype.hasOwnProperty.call(oRunnables, name) + const inC = Object.prototype.hasOwnProperty.call(cRunnables, name) + const o = inO ? orderedYamlStringify(oRunnables[name]) : undefined + const c = inC ? orderedYamlStringify(cRunnables[name]) : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ + path: reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken), + status, + original: o, + current: c, + lang: 'yaml' + }) + } + + // Remaining metadata → single `app.yaml` leaf. + const oMeta = metadataYaml(oApp) + const cMeta = metadataYaml(cApp) + const metaStatus = diffStatus(oMeta, cMeta) + if (metaStatus) { + entries.push({ + path: reserveUnique(RAW_APP_METADATA_PATH, taken), + status: metaStatus, + original: oMeta, + current: cMeta, + lang: 'yaml', + isMetadata: true + }) + } + + return entries +} + +// A raw-app file rendered as a standalone diff item, shaped like a +// WorkspaceItemDiff (so it flows through the existing list / tree / search / +// count machinery) plus the embedded diff payload. `kind: 'raw_app_file'` +// keeps it distinct from the backend kinds. The composite `path` +// (`/`) nests it under the app's folder in the tree. +export interface RawAppFileItem { + kind: 'raw_app_file' + path: string + /** Friendly composite path (`/`) for tree display only; + * `path` stays storage-keyed so a never-deployed draft still loads/edits via + * its `…/draft_` path. Defaults to `path` when no friendly path differs. */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + /** Workspace path of the owning raw app (for the edit link). */ + appPath: string + status: RawAppDiffStatus + original?: string + current?: string + lang: string + /** True for the synthesized `app.yaml` metadata item. */ + isMetadata: boolean + /** Whole serialized app (both sides) — only on the metadata item, for its + * optional "expand to full YAML" view. */ + fullYamlOriginal?: string + fullYamlCurrent?: string +} + +// A raw-app runnable rendered as a script/flow item (what it actually is). It +// carries the reshaped runnable object per side so the normal script-style +// diff (Content + Metadata) renders it — inline code shows with proper syntax +// highlighting instead of a YAML blob. `kind` drives the row icon/label +// (script vs flow); the body is always rendered script-style. +export interface RawAppRunnableItem { + kind: 'script' | 'flow' + path: string + /** Friendly composite path for tree display only (see RawAppFileItem). */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + appPath: string + status: RawAppDiffStatus + /** Reshaped runnable (content/language hoisted) for the diff viewer. */ + originalRaw?: unknown + currentRaw?: unknown +} + +export type RawAppSyntheticItem = RawAppFileItem | RawAppRunnableItem + +function joinAppPath(appPath: string, filePath: string): string { + // File keys may carry a leading slash (e.g. `/App.tsx`); strip it so the + // composite path has single separators and splits cleanly in the tree. + return `${appPath}/${filePath.replace(/^\/+/, '')}` +} + +// The whole serialized app for one side, matching the previous YAML escape +// hatch (cleaned + ordered). Undefined when the app is absent on that side. +function wholeAppYaml(raw: RawAppish | undefined): string | undefined { + if (!isObject(raw)) return undefined + return orderedYamlStringify(cleanValueProperties(replaceFalseWithUndefined(raw))) +} + +// Hoist an inline runnable's code + language to the top level so the +// script-style diff viewer (which reads top-level `content`/`language`) shows +// the code in the Content tab and the rest as Metadata. Path-referencing +// runnables (no inline script) just fall through to a YAML metadata diff. +function reshapeRunnable(runnable: unknown): unknown { + if (!isObject(runnable)) return runnable + const inline = isObject(runnable.inlineScript) ? runnable.inlineScript : undefined + if (!inline) return runnable + return { + ...runnable, + content: inline.content, + language: inline.language, + // Drop the now-hoisted code so it isn't duplicated in the Metadata tab. + inlineScript: { ...inline, content: undefined } + } +} + +// Runnables become script/flow rows; `runType: 'flow'` picks the flow icon. +function runnableKind(...sides: unknown[]): 'script' | 'flow' { + return sides.some((s) => isObject(s) && s.runType === 'flow') ? 'flow' : 'script' +} + +/** + * Expand a raw-app diff into standalone items for `appPath`: + * - files + the `app.yaml` metadata leaf → `RawAppFileItem`s (the metadata item + * also carries the whole-app YAML for its expand view); + * - runnables → `RawAppRunnableItem`s (script/flow rows). + */ +export function rawAppDiffToItems( + appPath: string, + original: RawAppish | undefined, + current: RawAppish | undefined, + displayAppPath: string = appPath +): RawAppSyntheticItem[] { + // Files + metadata (runnables handled separately as script/flow rows). + const fileItems = parseRawAppDiff(original, current, { includeRunnables: false }).map( + (e): RawAppFileItem => ({ + kind: 'raw_app_file', + path: joinAppPath(appPath, e.path), + displayPath: joinAppPath(displayAppPath, e.path), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: e.status !== 'added', + exists_in_fork: e.status !== 'removed', + appPath, + status: e.status, + original: e.original, + current: e.current, + lang: e.lang, + isMetadata: e.isMetadata ?? false + }) + ) + const metaItem = fileItems.find((i) => i.isMetadata) + if (metaItem) { + metaItem.fullYamlOriginal = wholeAppYaml(original) + metaItem.fullYamlCurrent = wholeAppYaml(current) + } + + // Runnables → script/flow rows, diffed on their object value. + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oRun = oApp?.runnables ?? {} + const cRun = cApp?.runnables ?? {} + // Reserve each runnable's composite leaf against the real file paths, mirroring + // parseRawAppDiff, so a real file literally named `runnables/` can't yield + // a second leaf at the same composite path. Normalize the leading slash (which + // joinAppPath strips) so `/runnables/x` and `runnables/x` are seen as equal. + const taken = new Set( + [...Object.keys(oApp?.files ?? {}), ...Object.keys(cApp?.files ?? {})].map(stripLeadingSlash) + ) + const runnableItems: RawAppRunnableItem[] = [] + for (const name of [...new Set([...Object.keys(oRun), ...Object.keys(cRun)])].sort()) { + const inO = Object.prototype.hasOwnProperty.call(oRun, name) + const inC = Object.prototype.hasOwnProperty.call(cRun, name) + const oStr = inO ? orderedYamlStringify(oRun[name]) : undefined + const cStr = inC ? orderedYamlStringify(cRun[name]) : undefined + const status = diffStatus(oStr, cStr) + if (!status) continue + const rel = reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken) + runnableItems.push({ + kind: runnableKind(oRun[name], cRun[name]), + path: joinAppPath(appPath, rel), + displayPath: joinAppPath(displayAppPath, rel), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: status !== 'added', + exists_in_fork: status !== 'removed', + appPath, + status, + originalRaw: inO ? reshapeRunnable(oRun[name]) : undefined, + currentRaw: inC ? reshapeRunnable(cRun[name]) : undefined + }) + } + + return [...fileItems, ...runnableItems] +} diff --git a/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts b/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts index b99704436b..35ff0805d3 100644 --- a/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts +++ b/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts @@ -47,6 +47,12 @@ export function appSourceToDraftValue(app: any, fallback?: any): AppDraftValue { runnables: { ...(value.runnables ?? {}) }, data: normalizeRawAppData(value), policy: app.policy ?? fallback?.policy, - custom_path: app.custom_path ?? fallback?.custom_path + custom_path: app.custom_path ?? fallback?.custom_path, + // Pin the fork base: a deployed app exposes `versions` (head = last); an + // existing draft already carries `parent_version` — preserve it. + parent_version: + app.parent_version ?? + (Array.isArray(app.versions) ? app.versions[app.versions.length - 1] : undefined) ?? + fallback?.parent_version } } diff --git a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts index f33c6e7eb6..16f588c993 100644 --- a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts +++ b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts @@ -19,10 +19,11 @@ export async function updateRawAppPolicy( ) ).filter((entry): entry is [string, TriggerableV2] => entry != null) const triggerables_v2 = Object.fromEntries(entries) - return { + const next: Policy = { ...currentPolicy, triggerables_v2 } + return next } type RunnableWithInlineScript = RunnableWithFields & { diff --git a/frontend/src/lib/components/raw_apps/utils.test.ts b/frontend/src/lib/components/raw_apps/utils.test.ts index b8ca6d207d..7d087a61ec 100644 --- a/frontend/src/lib/components/raw_apps/utils.test.ts +++ b/frontend/src/lib/components/raw_apps/utils.test.ts @@ -1,9 +1,11 @@ import { describe, expect, it } from 'vitest' import { + canonicalRawAppDiffValue, formatRuntimeLogsForChat, genWmillTs, normalizeRawAppRuntimeLogs, + stripRawAppDiffNoise, type Runnable } from './utils' @@ -61,3 +63,100 @@ describe('normalizeRawAppRuntimeLogs', () => { expect(formatRuntimeLogsForChat(entries)).toBe('[06:13:20.000] LOG: ready') }) }) + +// A deployed raw-app row as returned by getAppByPath: nested `value`, plus the +// server-managed columns and a recomputed inline-script lock. +function deployedRow() { + return { + id: 42, + raw_app: true, + is_draft: false, + created_at: '2024-01-01', + created_by: 'admin', + versions: [1, 2], + extra_perms: { 'u/admin': true }, + summary: 'app', + path: 'u/admin/app', + policy: { execution_mode: 'publisher' }, + value: { + files: { '/App.tsx': 'export default 1' }, + runnables: { + a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun', lock: 'deps\n' } } + } + } + } +} + +describe('stripRawAppDiffNoise', () => { + it('drops server-managed columns, nulls inline locks and canonicalizes data', () => { + const cleaned = stripRawAppDiffNoise(deployedRow()) + + for (const key of [ + 'raw_app', + 'id', + 'created_at', + 'created_by', + 'versions', + 'extra_perms', + 'is_draft' + ]) { + expect(cleaned).not.toHaveProperty(key) + } + expect(cleaned.value.runnables.a.inlineScript.lock).toBeUndefined() + // absent `data` is canonicalized to the default empty shape + expect(cleaned.value.data).toEqual({ tables: [], datatable: undefined, schema: undefined }) + }) + + it('does not mutate the input (live editor state)', () => { + const input = deployedRow() + stripRawAppDiffNoise(input) + expect(input.raw_app).toBe(true) + expect(input.value.runnables.a.inlineScript.lock).toBe('deps\n') + }) + + it('handles the flat editor/draft shape (files/runnables top-level)', () => { + const flat = { + summary: 'app', + files: { '/App.tsx': 'x' }, + runnables: { + a: { type: 'inline', inlineScript: { content: 'm()', language: 'bun', lock: 'l' } } + } + } + const cleaned = stripRawAppDiffNoise(flat) + expect(cleaned.runnables.a.inlineScript.lock).toBeUndefined() + expect(cleaned.data).toEqual({ tables: [], datatable: undefined, schema: undefined }) + }) +}) + +describe('canonicalRawAppDiffValue', () => { + it('collapses a nested deployed row and a flat draft to an identical value when content matches', () => { + const deployed = deployedRow() + // The flat draft shape a raw app autosaves: top-level files/runnables/data, + // no server columns, lock cleared on edit. + const draft = { + summary: 'app', + files: { '/App.tsx': 'export default 1' }, + runnables: { a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun' } } }, + data: { tables: [] }, + policy: { execution_mode: 'publisher' } + } + + expect(canonicalRawAppDiffValue(deployed)).toEqual(canonicalRawAppDiffValue(draft)) + }) + + it('still surfaces a real change (summary edit)', () => { + const deployed = deployedRow() + const draft = { + summary: 'app EDITED', + files: { '/App.tsx': 'export default 1' }, + runnables: { a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun' } } }, + data: { tables: [] } + } + + const a = canonicalRawAppDiffValue(deployed) + const b = canonicalRawAppDiffValue(draft) + expect(a).not.toEqual(b) + expect(a.summary).toBe('app') + expect(b.summary).toBe('app EDITED') + }) +}) diff --git a/frontend/src/lib/components/raw_apps/utils.ts b/frontend/src/lib/components/raw_apps/utils.ts index 53d84fd97c..c0fba48e18 100644 --- a/frontend/src/lib/components/raw_apps/utils.ts +++ b/frontend/src/lib/components/raw_apps/utils.ts @@ -3,6 +3,8 @@ import type { Schema } from '../../common' import { schemaToTsType } from '../../schema' import { isRunnableByName, isRunnableByPath, type RunnableWithFields } from '../apps/inputType' import type { InlineScript } from '../apps/sharedTypes' +import { stateSnapshot } from '$lib/svelte5Utils.svelte' +import { appSourceToDraftValue, normalizeRawAppData } from './rawAppDraftValue' // export type RunnableWithFields = any @@ -15,6 +17,66 @@ export type RawApp = { files: string[] } +// Server-managed columns the deployed app row (getAppByPath) carries but the +// editor's current value never does — leaving them in renders as spurious diff. +const RAW_APP_DEPLOYED_METADATA_KEYS = [ + 'raw_app', + 'id', + 'created_at', + 'created_by', + 'versions', + 'extra_perms', + 'is_draft' +] as const + +/** + * Normalize a raw-app value before a deployed-vs-current diff (or unsaved-change + * comparison). Three sources of spurious post-deploy diff: + * - the deployed row carries server-managed columns (`raw_app`, timestamps, …) + * that the editor's current value lacks; + * - inline-script `lock`s are recomputed server-side at every deploy and the + * editor clears them on edit, so the editor value and the freshly deployed + * one always diverge on `lock` even though the user changed nothing there; + * - the deployed value omits an empty `data` while the editor always carries + * the default `{ tables: [] }`, so even an untouched app reads as changed. + * All three must be neutralized symmetrically on both sides. Returns a deep + * clone; never mutates the input (the current side is live editor state). + */ +export function stripRawAppDiffNoise>(value: T): T { + const cloned = structuredClone(stateSnapshot(value)) as Record + for (const key of RAW_APP_DEPLOYED_METADATA_KEYS) { + delete cloned[key] + } + // Runnables/data live under `.value` on a deployed row and on the editor's + // diff value alike, but a flat draft shape carries them top-level. + const source = cloned.value ?? cloned + const runnables = source.runnables + if (runnables && typeof runnables === 'object') { + for (const k of Object.keys(runnables)) { + const inlineScript = runnables[k]?.inlineScript + if (inlineScript && inlineScript.lock != undefined) { + inlineScript.lock = undefined + } + } + } + // Canonicalize `data` so an absent and a default-empty `data` compare equal. + source.data = normalizeRawAppData(source) + return cloned as T +} + +/** + * Canonical raw-app value for diffing a *draft* against a *deployed* row. On top + * of the noise stripped by stripRawAppDiffNoise, the two also differ in shape: a + * deployed row nests its source under `value`, whereas a draft carries + * `files`/`runnables`/`data` at the top level. `appSourceToDraftValue` collapses + * both onto the same flat field set first. Use this for the session/compare + * draft diff so it matches the editor's Diff button (which shares + * stripRawAppDiffNoise). + */ +export function canonicalRawAppDiffValue(source: Record) { + return stripRawAppDiffNoise(appSourceToDraftValue(source)) +} + export type RawAppRuntimeLogLevel = 'log' | 'info' | 'warn' | 'error' | 'debug' export type RawAppRuntimeLogEntry = { level: RawAppRuntimeLogLevel @@ -79,65 +141,50 @@ export function formatAppRunsForChat(runs: RawAppRunSummary[]): string { return JSON.stringify(runs, null, 2) } -export function htmlContent( +// The sandboxed (isolated) raw-app wrapper is generated server-side and served as +// a sandboxed, opaque-origin document (see `get_raw_app_data` in the backend +// `apps.rs`, WIN-2006) — a blob: URL cannot carry the `CSP: sandbox` response +// header that enforces isolation, so the wrapper must come from the backend. +// +// The function below is used ONLY for the unsandboxed path (the default — the +// publisher did not opt into sandbox isolation). It is loaded as a blob: URL — +// same-origin with the SPA — so, with `allow-same-origin`, the bundle runs with +// the viewer's full session. Crucially this is an in-memory blob, not a +// real-origin endpoint, so it is not a URL an attacker can navigate a logged-in +// victim to in order to gain isolation-bypassing access — the backend `.html` +// document stays sandboxed whenever the publisher did opt in. +export function unsandboxedRawAppHtml( workspace: string, - secret: string | undefined, + secret: string, ctx: any, - baseUrl: string = '', - initialHash: string = '' + baseUrl: string, + initialHash: string ) { return ` - App Preview + App diff --git a/frontend/src/lib/components/runs/RunsQueue.svelte b/frontend/src/lib/components/runs/RunsQueue.svelte index b0c5ed5adb..2cd2914878 100644 --- a/frontend/src/lib/components/runs/RunsQueue.svelte +++ b/frontend/src/lib/components/runs/RunsQueue.svelte @@ -1,7 +1,7 @@ + +{#if self} +
    +
    {HANDLER_LABEL[self.kind]}
    +
    + {#if self.handledJob} + + {/if} + {#if self.schedulePath} + + for schedule {self.schedulePath} + + {/if} +
    +
    +{:else} + {#if retries.length > 1} +
    +
    Retries ({retries.length - 1})
    +
    + {#each retries as attempt, i (attempt.id)} + + {/each} +
    +
    + {/if} + {#if handlers.length > 0} +
    +
    Handlers
    +
    + {#each handlers as handler (handler.id)} + + {/each} +
    +
    + {/if} +{/if} diff --git a/frontend/src/lib/components/sessions/SessionDraftBar.svelte b/frontend/src/lib/components/sessions/SessionDraftBar.svelte index dcc5640f2c..d23070151a 100644 --- a/frontend/src/lib/components/sessions/SessionDraftBar.svelte +++ b/frontend/src/lib/components/sessions/SessionDraftBar.svelte @@ -7,6 +7,7 @@ import DraftDiffDrawer from './DraftDiffDrawer.svelte' import SessionDiffButton from './SessionDiffButton.svelte' import { useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' + import Tooltip from '$lib/components/meltComponents/Tooltip.svelte' let { session }: { session: Session } = $props() @@ -58,7 +59,13 @@ >
    - {count} draft{count === 1 ? '' : 's'} + + {count} draft{count === 1 ? '' : 's'} + {#snippet text()} + Tracks all unsaved draft changes in this workspace — including edits made outside this + chat (e.g. in the editor), not only changes made by the assistant. + {/snippet} +
    drawer?.open()} /> diff --git a/frontend/src/lib/components/sessions/SessionFilterMenu.svelte b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte new file mode 100644 index 0000000000..e5e67d1567 --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte @@ -0,0 +1,77 @@ + + + + +{#if $subOpen} +
    + +
    +
    + + Include sessions from every workspace. +
    +
    + + {#if archivedCount > 0} + + {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + + {/if} +
    +
    +
    +{/if} diff --git a/frontend/src/lib/components/sessions/SessionForkBar.svelte b/frontend/src/lib/components/sessions/SessionForkBar.svelte index b1e74b43eb..18d36a755c 100644 --- a/frontend/src/lib/components/sessions/SessionForkBar.svelte +++ b/frontend/src/lib/components/sessions/SessionForkBar.svelte @@ -68,6 +68,10 @@ const forkStatus = $derived(deriveForkStatus(session, $userWorkspaces, comparison)) const isUnavailable = $derived(forkStatus === 'unavailable') + // The committed workspace is gone, so we can't read its parent to know if + // it was a fork — fall back to the fork-id convention for the wording. + const committedIsFork = $derived(committedId?.startsWith('wm-fork-') ?? false) + $effect(() => { if (!runtime || !committedId || !parentWorkspaceId) return void runtime.ensureForkComparison(parentWorkspaceId, committedId) @@ -111,15 +115,19 @@ {#if committedId && isUnavailable} - + chat input is disabled by SessionWrapper while this is shown. Shown even + for an archived session — unarchiving in place can't help when the + workspace is gone, so move/discard is the only real recovery path. -->
    - The fork has been archived or deleted + The {committedIsFork ? 'fork' : 'workspace'} has been archived or deleted Move this session to another workspace, or discard it. {committedId} diff --git a/frontend/src/lib/components/sessions/SessionPicker.svelte b/frontend/src/lib/components/sessions/SessionPicker.svelte index 55683b4d96..a3273ddceb 100644 --- a/frontend/src/lib/components/sessions/SessionPicker.svelte +++ b/frontend/src/lib/components/sessions/SessionPicker.svelte @@ -22,8 +22,9 @@ import { createSession, deriveForkStatus, - getEffectiveWorkspaceId, + deleteSessionsForWorkspace, isForkSession, + reconcileAfterWorkspaceChange, renameSession, selectSession, sessionState, @@ -41,14 +42,15 @@ removeSession } from './sessionRuntime.svelte' import SessionStatusDot from './SessionStatusDot.svelte' + import SessionFilterMenu from './SessionFilterMenu.svelte' import { Menu, Menubar, MenuItem } from '$lib/components/meltComponents' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' - import { visibleWorkspaceIds } from './sessionScope.svelte' import { isGlobalAiEnabled } from '$lib/components/copilot/chat/global/gate' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' + import { currentWorkspaceRootId, workspaceRootId } from './sessionScope.svelte' // Look up the cached fork comparison for a session through its runtime // (if any). The deriveForkStatus helper handles the "no runtime yet" @@ -101,40 +103,81 @@ 'boolean' ) const showArchived = useLocalStorageValue('windmill_sessions_show_archived', false, 'boolean') + // Off by default: the list is scoped to the current workspace family. Turn on + // to include sessions from every workspace (grouped by family) — handy when + // switching sessions across workspaces without switching workspace first. + const showAllWorkspaces = useLocalStorageValue( + 'windmill_sessions_show_all_workspaces', + false, + 'boolean' + ) let listRoot: HTMLDivElement | undefined = $state() - // Sessions visible in the current workspace (active workspace + its - // forks). Drafts (no committed workspace) are scoped by their - // pending workspace pick — set at create time to the workspace the - // user was in. Archived sessions are filtered out unless the user - // has opted in via the filter popover. + // A session's family root: the stored grouping id, else derived live. + function sessionRootOf(s: Session): string | undefined { + return ( + s.workspace_root_id ?? + workspaceRootId(s.workspace_id ?? s.pending_workspace_id, $userWorkspaces) + ) + } + + // Flat list passing the archive + scope filters. Grouping for display happens + // in `sessionGroups`; this flat view drives the runtime / fork-comparison + // effects, the unread total, and keyboard navigation. const visibleSessions = $derived( sessionState.sessions.filter((s) => { - // Transient (not-yet-sent) sessions live as their own page but - // don't clutter the sidebar list. if (s.transient) return false + // The open session always stays in the list, ignoring both filters. + if (s.id === sessionState.currentSessionId) return true if (s.archived && !showArchived.val) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - // Unavailable sessions (committed workspace was deleted / - // archived / access revoked) stay visible everywhere so the - // user can resolve them — move, archive, or delete. They'd - // otherwise be permanently hidden the moment their workspace - // disappeared. - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false + if (!showAllWorkspaces.val) { + const currentRoot = $currentWorkspaceRootId + if (currentRoot && sessionRootOf(s) !== currentRoot) return false + } + return true }) ) + + // Sessions grouped by workspace family for display, each group newest-first. + // Family order is stable (by most-recent activity) and deliberately NOT tied + // to the current workspace: pinning the active family first reshuffled the + // whole list on every workspace switch, which is disorienting. + const sessionGroups = $derived.by(() => { + const byRoot = new Map() + for (const s of visibleSessions) { + const root = sessionRootOf(s) ?? s.workspace_id ?? s.pending_workspace_id ?? '' + const arr = byRoot.get(root) + if (arr) arr.push(s) + else byRoot.set(root, [s]) + } + const groups = [...byRoot.entries()].map(([rootId, sessions]) => { + sessions.sort((a, b) => b.createdAt - a.createdAt) + return { + rootId, + name: $userWorkspaces.find((w) => w.id === rootId)?.name || rootId || 'Workspace', + sessions, + mostRecent: sessions[0]?.createdAt ?? 0 + } + }) + groups.sort((a, b) => b.mostRecent - a.mostRecent) + return groups + }) + + // Family labels are redundant when scoped to the current workspace (a single + // family) — show them when including all workspaces, and also if the + // active-session override surfaces a second family while scoped (avoids + // ambiguity). + const showGroupHeaders = $derived(showAllWorkspaces.val || sessionGroups.length > 1) + const archivedCount = $derived( sessionState.sessions.filter((s) => { if (!s.archived || s.transient) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false + if (showAllWorkspaces.val) return true + const currentRoot = $currentWorkspaceRootId + return ( + !currentRoot || sessionRootOf(s) === currentRoot || s.id === sessionState.currentSessionId + ) }).length ) @@ -267,8 +310,9 @@ if (forkToDelete) { try { await WorkspaceService.deleteWorkspace({ workspace: forkToDelete }) + await deleteSessionsForWorkspace(forkToDelete) sendUserToast(`Deleted forked workspace ${forkToDelete}`) - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to delete fork ${forkToDelete}: ${e?.body ?? e}`, true) } @@ -326,7 +370,7 @@
    {#snippet children({ createMenu })} - + {#snippet triggr({ trigger })}
    {/snippet} - {#snippet children({ item })} + {#snippet children({ item, builders })}
    @@ -355,47 +399,66 @@
    - {#each visibleSessions as session (session.id)} - {@const runtime = getRuntime(session.id)} - {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} - {@const isSelected = - onSessionsPage && session.id === sessionState.currentSessionId} - {@const unread = unreadFor(session)} - {@const draft = hasDraft(session)} - activate(session)} - {item} - > - - 0 ? 'font-semibold text-primary' : '' - )} + +
    +
    + {#each sessionGroups as group (group.rootId)} + {#if showGroupHeaders} +
    - {session.summary ?? 'Untitled session'} - - {#if draft || unread > 0} - - {#if draft} - - {/if} - {#if unread > 0} - - {unread > 9 ? '9+' : unread} - - {/if} + {group.name} +
    + {/if} + {#each group.sessions as session (session.id)} + {@const runtime = getRuntime(session.id)} + {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} + {@const isSelected = + onSessionsPage && session.id === sessionState.currentSessionId} + {@const unread = unreadFor(session)} + {@const draft = hasDraft(session)} + activate(session)} + {item} + > + + 0 ? 'font-semibold text-primary' : '' + )} + > + {session.summary ?? 'Untitled session'} - {/if} - + {#if draft || unread > 0} + + {#if draft} + + {/if} + {#if unread > 0} + + {unread > 9 ? '9+' : unread} + + {/if} + + {/if} + + {/each} {/each}
    @@ -436,7 +499,8 @@ type="button" title="Filter sessions" aria-label="Filter sessions" - class="inline-flex items-center justify-center w-5 h-5 rounded text-tertiary hover:bg-surface-hover hover:text-primary {showArchived.val + class="inline-flex items-center justify-center w-5 h-5 rounded text-tertiary hover:bg-surface-hover hover:text-primary {showArchived.val || + showAllWorkspaces.val ? 'text-emphasis' : ''}" > @@ -445,18 +509,30 @@ {/snippet} {#snippet content()}
    - - {#if archivedCount > 0} +
    + - {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + Include sessions from every workspace. - {/if} +
    +
    + + {#if archivedCount > 0} + + {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + + {/if} +
    {/snippet} @@ -479,119 +555,137 @@ role="listbox" tabindex="-1" > - {#each visibleSessions as session (session.id)} - {@const runtime = getRuntime(session.id)} - {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} - {@const isSelected = onSessionsPage && session.id === sessionState.currentSessionId} - {@const isEditing = editingId === session.id} - {@const unread = unreadFor(session)} - {@const draft = hasDraft(session)} -
    - {#if isEditing} - - - - { - if (e.key === 'Enter') commitRename() - else if (e.key === 'Escape') cancelRename() - }} - onblur={commitRename} - placeholder="Untitled session" - autofocus - spellcheck="false" - class="flex-1 min-w-0 bg-transparent border-0 outline-none text-xs font-normal text-primary" - /> - - {:else} - -
    - startRename(session) - }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => setSessionArchived(session.id, true) - }, - { - displayName: 'Delete', - icon: Trash2, - type: 'delete', - action: () => (pendingDelete = session) - } - ]} + {#each sessionGroups as group (group.rootId)} + {#if showGroupHeaders} +
    + {group.name} +
    + {/if} + {#each group.sessions as session (session.id)} + {@const runtime = getRuntime(session.id)} + {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} + {@const isSelected = onSessionsPage && session.id === sessionState.currentSessionId} + {@const isEditing = editingId === session.id} + {@const unread = unreadFor(session)} + {@const draft = hasDraft(session)} +
    + {#if isEditing} + + + + { + if (e.key === 'Enter') commitRename() + else if (e.key === 'Escape') cancelRename() + }} + onblur={commitRename} + placeholder="Untitled session" + autofocus + spellcheck="false" + class="flex-1 min-w-0 bg-transparent border-0 outline-none text-xs font-normal text-primary" + /> + + {:else} +
    - {/if} -
    + {/if} + +
    + startRename(session) + }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailableFork(session) + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => setSessionArchived(session.id, true) + } + ]), + { + displayName: 'Delete', + icon: Trash2, + type: 'delete', + action: () => (pendingDelete = session) + } + ]} + > + {#snippet buttonReplacement()} + + + + {/snippet} + +
    + {/if} +
    + {/each} {/each}
    {/if} diff --git a/frontend/src/lib/components/sessions/SessionWrapper.svelte b/frontend/src/lib/components/sessions/SessionWrapper.svelte index c74bf294e6..027089d4ae 100644 --- a/frontend/src/lib/components/sessions/SessionWrapper.svelte +++ b/frontend/src/lib/components/sessions/SessionWrapper.svelte @@ -7,7 +7,7 @@ import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { AIChatManager } from '$lib/components/copilot/chat/AIChatManager.svelte' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' import Toggle from '$lib/components/Toggle.svelte' @@ -32,9 +32,11 @@ import SessionDraftBar from './SessionDraftBar.svelte' import { createSession, + deleteSessionsForWorkspace, getEffectiveWorkspaceId, moveSessionToNewFork, moveSessionToWorkspace, + reconcileAfterWorkspaceChange, renameSession, selectSession, sessionState, @@ -101,16 +103,6 @@ let archiveConfirmOpen = $state(false) let archiveAlsoFork = $state(false) - async function refreshWorkspaceList() { - // Match the SidebarContent.deleteFork pattern: replace the in-memory - // list rather than nulling it. See B1 fix. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } - } - async function handleConfirmedDelete() { deleteConfirmOpen = false if (!session) return @@ -125,8 +117,9 @@ if (forkToDelete) { try { await WorkspaceService.deleteWorkspace({ workspace: forkToDelete }) + await deleteSessionsForWorkspace(forkToDelete) sendUserToast(`Deleted forked workspace ${forkToDelete}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to delete fork ${forkToDelete}: ${e?.body ?? e}`, true) } @@ -150,7 +143,7 @@ try { await WorkspaceService.archiveWorkspace({ workspace: forkToArchive }) sendUserToast(`Archived forked workspace ${forkToArchive}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to archive fork ${forkToArchive}: ${e?.body ?? e}`, true) } @@ -280,6 +273,29 @@ is position:fixed, so it doesn't count as a flex item — no stray gap when only one bar shows. -->
    + {#if session.archived && !isUnavailable} + +
    +
    + + This session is archived +
    + +
    + {/if} moveAndActivate(workspaceId)} @@ -320,17 +336,25 @@ icon: Pencil, action: () => summaryInput?.edit() }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => archiveAndReset() - }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailable + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => archiveAndReset() + } + ]), { displayName: 'Delete', icon: Trash2, @@ -403,10 +427,12 @@ hideHeader hideModeSelector wideLayout - forceDisabled={isUnavailable} + forceDisabled={isUnavailable || !!session.archived} forceDisabledMessage={isUnavailable ? 'This session is linked to a workspace that no longer exists. Move it or discard it from the banner above to keep working.' - : ''} + : session.archived + ? 'This session is archived. Unarchive it from the banner above to keep working.' + : ''} emptyHint={sessionEmptyHint} {inputPreface} /> diff --git a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte index 472ea1515b..c4099edd91 100644 --- a/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/WorkspaceDiffDrawer.svelte @@ -19,7 +19,7 @@ diff --git a/frontend/src/lib/components/workspaceSettings/AISettings.svelte b/frontend/src/lib/components/workspaceSettings/AISettings.svelte index 2080ecd332..f44059e395 100644 --- a/frontend/src/lib/components/workspaceSettings/AISettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/AISettings.svelte @@ -13,6 +13,8 @@ import { supportsAutocomplete } from '../copilot/utils' import TestAiKey from '../copilot/TestAIKey.svelte' import Label from '../Label.svelte' + import AiSkillsSettings from './AiSkillsSettings.svelte' + import { isGlobalAiEnabled } from '../copilot/chat/global/gate' import SettingsPageHeader from '../settings/SettingsPageHeader.svelte' import ResourcePicker from '../ResourcePicker.svelte' import Toggle from '../Toggle.svelte' @@ -587,6 +589,10 @@
    {/if} + + {#if promptScope === 'workspace' && isGlobalAiEnabled()} + + {/if}
    + import { onMount } from 'svelte' + import YAML from 'yaml' + import Button from '../common/button/Button.svelte' + import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte' + import SettingCard from '../instanceSettings/SettingCard.svelte' + import Label from '../Label.svelte' + import autosize from '$lib/autosize' + import { workspaceStore } from '$lib/stores' + import { sendUserToast } from '$lib/toast' + import { WorkspaceService } from '$lib/gen' + import { FolderUp, Plus, Trash2 } from 'lucide-svelte' + + type SkillListItem = { name: string; description: string } + type SkillUpload = { name: string; description: string; instructions: string } + + // `//SKILL.md` is 3 path segments; SKILL.md files nested deeper + // are likely vendored/incidental and are skipped so importing a parent dir + // doesn't sweep in unrelated skills. + const MAX_SKILL_DEPTH = 3 + const MAX_SKILLS_PER_IMPORT = 50 + const MAX_SKILLS_PER_WORKSPACE = 100 + // `name` + `description` mirror the Claude SKILL.md spec (counted in + // characters); the body is a byte-bounded payload. Keep these in sync with + // backend `validate_skill`. + const MAX_SKILL_NAME_LENGTH = 64 + const MAX_SKILL_DESCRIPTION_LENGTH = 1_024 + const MAX_SKILL_INSTRUCTIONS_LENGTH = 64 * 1024 + const SKILL_NAME_PATTERN = /^[a-z0-9-]+$/ + const textEncoder = new TextEncoder() + const SAMPLE_SKILL_PLACEHOLDER = + '---\nname: my-skill\ndescription: what this skill helps with\n---\n\n# My skill\n\nInstructions for the assistant…' + + let skills: SkillListItem[] = $state([]) + let uploading: boolean = $state(false) + let pasteContent: string = $state('') + let dirInput: HTMLInputElement | undefined = $state(undefined) + let toDelete: string | undefined = $state(undefined) + let pendingImport: SkillUpload[] | undefined = $state(undefined) + let pendingSkipped: string[] = $state([]) + let listRequestId = 0 + + let pendingNamesPreview = $derived.by(() => { + const p = pendingImport ?? [] + const shown = p + .slice(0, 12) + .map((s) => s.name) + .join(', ') + return p.length > 12 ? `${shown}, … (+${p.length - 12} more)` : shown + }) + + async function loadList(workspace: string | undefined) { + const requestId = ++listRequestId + if (!workspace) { + skills = [] + return + } + try { + const loaded = await WorkspaceService.listAiSkills({ workspace }) + if (requestId === listRequestId && workspace === $workspaceStore) { + skills = loaded + } + } catch (e) { + if (requestId === listRequestId && workspace === $workspaceStore) { + sendUserToast(`Failed to load skills: ${e}`, true) + } + } + } + + /** Split a SKILL.md into its frontmatter `name`/`description` and the markdown body. */ + function parseSkillMd(raw: string): { + name: string | undefined + description: string | undefined + instructions: string + } { + const text = raw.replace(/^/, '') + const fm = /^---\s*\r?\n([\s\S]*?)\r?\n---\s*\r?\n?/.exec(text) + if (!fm) { + return { name: undefined, description: undefined, instructions: text.trim() } + } + let name: string | undefined + let description: string | undefined + try { + const data = YAML.parse(fm[1]) ?? {} + if (typeof data?.name === 'string') name = data.name.trim() + if (typeof data?.description === 'string') description = data.description.trim() + } catch { + // Malformed frontmatter — fall through so the skill is reported as + // invalid rather than silently dropped. + } + return { name, description, instructions: text.slice(fm[0].length).trim() } + } + + function validateParsedSkill(skill: SkillUpload): string | undefined { + if ([...skill.name].length > MAX_SKILL_NAME_LENGTH) { + return `name is longer than ${MAX_SKILL_NAME_LENGTH} characters` + } + if (!SKILL_NAME_PATTERN.test(skill.name)) { + return `name ${JSON.stringify(skill.name)} must only contain lowercase letters, digits or '-'` + } + if ([...skill.description].length > MAX_SKILL_DESCRIPTION_LENGTH) { + return `description is longer than ${MAX_SKILL_DESCRIPTION_LENGTH} characters` + } + if (textEncoder.encode(skill.instructions).byteLength > MAX_SKILL_INSTRUCTIONS_LENGTH) { + return `body is longer than ${MAX_SKILL_INSTRUCTIONS_LENGTH} bytes` + } + } + + /** + * Turn a map of `relativePath -> content` (from an imported folder) into skills. + * A skill is any `SKILL.md`; its id is the name of the folder holding it. + */ + function collectSkills(files: Record): { + skills: SkillUpload[] + skipped: string[] + } { + const collected: SkillUpload[] = [] + const skipped: string[] = [] + for (const [path, content] of Object.entries(files)) { + const segments = path.split('/') + if (segments[segments.length - 1]?.toLowerCase() !== 'skill.md') continue + const name = segments.length >= 2 ? segments[segments.length - 2] : '' + const { description, instructions } = parseSkillMd(content) + if (!name) { + skipped.push(`${path} (SKILL.md must live in a named folder)`) + } else if (!description) { + skipped.push(`${name} (missing frontmatter description)`) + } else if (!instructions) { + skipped.push(`${name} (empty body)`) + } else { + const parsed = { name, description, instructions } + const validationError = validateParsedSkill(parsed) + if (validationError) { + skipped.push(`${name} (${validationError})`) + } else { + collected.push(parsed) + } + } + } + return { skills: collected, skipped } + } + + async function uploadSkills(parsed: SkillUpload[], skipped: string[] = []) { + const workspace = $workspaceStore + if (!workspace || parsed.length === 0) { + sendUserToast( + `No valid skill found.${skipped.length ? ` Skipped: ${skipped.join(', ')}` : ''}`, + true + ) + return false + } + if (parsed.length > MAX_SKILLS_PER_IMPORT) { + sendUserToast(`Cannot add more than ${MAX_SKILLS_PER_IMPORT} skills at a time.`, true) + return false + } + // Uploads upsert, so only names not already stored count toward the cap. + const existingNames = new Set(skills.map((s) => s.name)) + const newCount = parsed.filter((s) => !existingNames.has(s.name)).length + if (skills.length + newCount > MAX_SKILLS_PER_WORKSPACE) { + sendUserToast(`This workspace can store at most ${MAX_SKILLS_PER_WORKSPACE} skills.`, true) + return false + } + uploading = true + try { + await WorkspaceService.uploadAiSkills({ + workspace, + requestBody: { skills: parsed } + }) + let message = `Added ${parsed.length} skill(s)` + if (skipped.length) message += `; skipped ${skipped.length}: ${skipped.join(', ')}` + sendUserToast(message) + await loadList(workspace) + return true + } catch (e) { + sendUserToast(`Failed to add skills: ${e}`, true) + return false + } finally { + uploading = false + } + } + + async function addPastedSkill() { + const { name, description, instructions } = parseSkillMd(pasteContent) + if (!name) { + sendUserToast('The pasted SKILL.md needs a `name` in its frontmatter.', true) + return + } + if (!description) { + sendUserToast('The pasted SKILL.md needs a `description` in its frontmatter.', true) + return + } + if (!instructions) { + sendUserToast('The pasted SKILL.md has an empty body.', true) + return + } + const parsed = { name, description, instructions } + const validationError = validateParsedSkill(parsed) + if (validationError) { + sendUserToast(`The pasted SKILL.md ${validationError}.`, true) + return + } + if (await uploadSkills([parsed])) { + pasteContent = '' + } + } + + async function onDirSelected(event: Event) { + const target = event.target as HTMLInputElement + const files = Array.from(target.files ?? []) + // Reset early so re-selecting the same folder re-fires `change`. + if (dirInput) dirInput.value = '' + + // Pick SKILL.md files within the depth limit BEFORE reading any content, + // so a huge tree never gets read in full. + const skipped: string[] = [] + const eligible: File[] = [] + for (const f of files) { + const path = f.webkitRelativePath || f.name + const segments = path.split('/') + if (segments[segments.length - 1]?.toLowerCase() !== 'skill.md') continue + if (segments.length > MAX_SKILL_DEPTH) { + skipped.push(`${path} (nested deeper than ${MAX_SKILL_DEPTH} folder levels)`) + continue + } + eligible.push(f) + } + + if (eligible.length === 0) { + sendUserToast( + `No SKILL.md found within ${MAX_SKILL_DEPTH} folder levels.${ + skipped.length ? ` Skipped ${skipped.length} deeper file(s).` : '' + }`, + true + ) + return + } + if (eligible.length > MAX_SKILLS_PER_IMPORT) { + sendUserToast( + `Found ${eligible.length} skills in this folder; imports are limited to ${MAX_SKILLS_PER_IMPORT} at a time.`, + true + ) + return + } + + const map: Record = {} + for (const f of eligible) { + map[f.webkitRelativePath || f.name] = await f.text() + } + const { skills: parsed, skipped: parseSkipped } = collectSkills(map) + const allSkipped = [...skipped, ...parseSkipped] + if (parsed.length === 0) { + sendUserToast( + `No valid skill found.${allSkipped.length ? ` Skipped: ${allSkipped.join(', ')}` : ''}`, + true + ) + return + } + // Confirm before writing — the import can pull in several skills at once. + pendingSkipped = allSkipped + pendingImport = parsed + } + + async function deleteSkill(name: string) { + const workspace = $workspaceStore + if (!workspace) return + try { + await WorkspaceService.deleteAiSkill({ workspace, name }) + sendUserToast(`Deleted skill ${name}`) + await loadList(workspace) + } catch (e) { + sendUserToast(`Failed to delete skill: ${e}`, true) + } + } + + onMount(() => { + return workspaceStore.subscribe((workspace) => { + toDelete = undefined + pendingImport = undefined + pendingSkipped = [] + void loadList(workspace) + }) + }) + + + +
    + + + + + {#if skills.length > 0} +
    + {#each skills as skill (skill.name)} +
    +
    +
    {skill.name}
    +
    {skill.description}
    +
    +
    + {/each} +
    + {/if} +
    +     + + { + const toImport = pendingImport + const skipped = pendingSkipped + pendingImport = undefined + pendingSkipped = [] + if (toImport) await uploadSkills(toImport, skipped) + }} + onCanceled={() => { + pendingImport = undefined + pendingSkipped = [] + }} +> + + Add {pendingImport?.length} skill(s) to the AI chat? + {pendingNamesPreview} + {#if pendingSkipped.length} +
    {pendingSkipped.length} file(s) will be skipped. + {/if} +     +     + + { + const name = toDelete + toDelete = undefined + if (name) await deleteSkill(name) + }} + onCanceled={() => (toDelete = undefined)} +> + + Delete the skill {toDelete}? The AI chat will no longer be able to use it. + + diff --git a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte index 024d94e8e6..30b050a9f6 100644 --- a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte +++ b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte @@ -22,6 +22,7 @@ import { sendUserToast } from '$lib/toast' import TestAIKey from '$lib/components/copilot/TestAIKey.svelte' import { switchWorkspace } from '$lib/storeUtils' + import { deleteSessionsForWorkspace } from '$lib/components/sessions/sessionState.svelte' import { isCloudHosted } from '$lib/cloud' import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte' import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte' @@ -100,6 +101,14 @@ deletingExistingFork = true try { await WorkspaceService.deleteWorkspace({ workspace: prefixedId }) + // Drop local sessions bound to this id so they don't resurface (or + // auto-unarchive) against a new fork recreated under the same id. + // Fire-and-forget: neither a slow nor a failing IndexedDB op should + // block the delete/reuse flow (cleanup completes long before the UI + // could create a session in a recreated fork). + void deleteSessionsForWorkspace(prefixedId).catch((e) => + console.error(`Session cleanup for reused fork id ${prefixedId} failed`, e) + ) sendUserToast(`Permanently deleted workspace ${prefixedId}`) deleteExistingForkOpen = false await validateName(id) diff --git a/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte b/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte index eab008fef6..156d11ce95 100644 --- a/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte +++ b/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte @@ -146,7 +146,7 @@ title: 'Database name is valid', status: status?.logs.valid_dbname, description: - 'The database name must be alphanumeric (underscores allowed) and cannot be named the same as the Windmill database (usually "windmill")' + 'The database name must be alphanumeric (underscores and hyphens allowed) and cannot be named the same as the Windmill database (usually "windmill")' }, { title: diff --git a/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte b/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte index 490f8fcb89..17aab3b32e 100644 --- a/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte @@ -41,8 +41,6 @@ } return s } - - let DEFAULT_DATATABLE_DB_NAME = 'datatable_db' -{#if app} - {#key app} -
    - { - goto(path) - }} - gotoFn={(path, opt) => { - goto(path, opt) - }} - /> - {#if can_write && !hideEditBtn} -
    - -
    - {/if} -
    + +{#if workspace && path} + + {#key `${workspace}/${path}`} + {/key} {:else} diff --git a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js deleted file mode 100644 index 5b680f2fd0..0000000000 --- a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `App ${params.path}` } - } -} diff --git a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte index db7b7da001..dbba9ce790 100644 --- a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte @@ -1,41 +1,17 @@ - -
    - -{#if !loaded} - -{/if} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte index 94c6d72e71..6227776ec4 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte @@ -57,6 +57,7 @@ // let lastVersion = 0 let policy: any = $state({}) let summary = $state('') + let labels = $state(undefined) /** User-typed path from `RawAppEditorHeader` when it differs from * `savedApp.path`; mirrored into the draft below as `draft_path` for the * home list's friendly name. */ @@ -72,6 +73,7 @@ summary: string policy: any custom_path?: string + labels?: string[] no_deployed?: boolean } | undefined = $state(undefined) @@ -140,6 +142,7 @@ if (extractedData) data = extractedData files = app.value.files summary = app.summary + labels = app.labels // lastVersion = app.version policy = app.policy // Prefer the saved `draft_path` so the topbar shows the pending name, not @@ -179,6 +182,10 @@ loadedFromDraft = false draftSavedAt = undefined deployedAt = undefined + // `labels` is route-level state; reset it too so a fresh draft doesn't + // inherit (and then deploy) the previously-opened app's labels. The + // import branch re-seeds it via extractRawApp below. + labels = undefined // Brand-new raw app: no deployed baseline, so never discard-on-equal. deployedBaseline = undefined // Suspend autosave across the bootstrap: the seed template and the @@ -355,6 +362,7 @@ path: backendApp_.path, policy: backendApp_.policy, custom_path: backendApp_.custom_path, + labels: backendApp_.labels, no_deployed: backendApp_.no_deployed } // Extract the effective raw app into the editor's local pieces. The bundle @@ -449,16 +457,17 @@ let diffDrawer: DiffDrawer | undefined = $state(undefined) - function onRestore(ev: any) { + function onRestore(restoredApp: any) { sendUserToast('App restored from previous deployment') - let prev = ev.detail + let prev = restoredApp extractRawApp(prev) savedApp = { summary: prev.summary, value: structuredClone(stateSnapshot(prev.value)), path: prev.path, policy: structuredClone(stateSnapshot(policy)), - custom_path: prev.custom_path + custom_path: prev.custom_path, + labels: prev.labels } redraw++ } @@ -536,18 +545,19 @@ {#key redraw}
    { + onSavedNewAppPath={(savedPath) => { draftSync.remove() - goto(`/apps_raw/edit/${event.detail}`) - newPath = event.detail + goto(`/apps_raw/edit/${savedPath}`) + newPath = savedPath }} - on:restore={onRestore} + {onRestore} bind:files bind:runnables bind:data bind:summary bind:pendingDraftPath {newPath} + {labels} path={page.params.path ?? ''} liveEditorDraftStoragePath={path} {policy} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte index 9fdf4c5c7d..990e74ed8c 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte @@ -1,65 +1,26 @@ -
    - {#if !$workspaceStore || !$userStore || !app} - - {:else} - - {/if} - {#if can_write && !hideEditBtn} -
    - -
    - {/if} -
    +{#if workspace && path} + + {#key `${workspace}/${path}`} + + {/key} +{:else} + +{/if} diff --git a/frontend/src/routes/(root)/(logged)/assets/+page.svelte b/frontend/src/routes/(root)/(logged)/assets/+page.svelte index 5c6948994a..2873140fea 100644 --- a/frontend/src/routes/(root)/(logged)/assets/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/assets/+page.svelte @@ -40,8 +40,7 @@ import { untrack } from 'svelte' import { VolumeService } from '$lib/gen' import VolumesDrawer from '$lib/components/assets/VolumesDrawer.svelte' - import { HardDriveIcon, NetworkIcon } from 'lucide-svelte' - import { base } from '$lib/base' + import { HardDriveIcon } from 'lucide-svelte' interface AssetCursor { created_at?: string @@ -162,18 +161,7 @@ title="Assets" tooltip="Assets show up here whenever you use them in Windmill." documentationLink="https://www.windmill.dev/docs/core_concepts/assets" - > -
    - -
    - + />
    diff --git a/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte index f06fa7265b..e73af57ee2 100644 --- a/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte @@ -56,6 +56,9 @@ let othersModalOpen = $state(false) let draftSavedAt = $state(undefined) let deployedAt = $state(undefined) + // The flow_version the draft was forked from (pinned, doesn't drift), for the + // precise staleness check in DraftEditorModals + FlowBuilder's deploy guard. + let draftBaseVersion = $state(undefined) // Editor-displayed path; defaults to the URL path. Cleared to '' in the // `new_draft` branch so the Path widget's `initPath` seeds the friendly name. let flowInitialPath = $state(page.params.path ?? '') @@ -152,6 +155,11 @@ loadedFromDraft = false draftSavedAt = undefined deployedAt = undefined + // New-draft skips the deployed/draft fetches, so the version-staleness + // inputs are never reassigned — clear the previous flow's values, else they + // bleed across the reused route and falsely trip the stale-draft modal. + version = undefined + draftBaseVersion = undefined // Brand-new flow: no deployed baseline, so never discard-on-equal. deployedBaseline = undefined // Suspend autosave around the bootstrap cascade: the Path widget's @@ -357,6 +365,9 @@ // Layer the draft (`.draft`, if any) over the deployed payload at the field // level. See /scripts/edit's loader for the rationale. const { draft: draftFromBackend, ...deployedFlow } = backendFlow as any + // `version_id` rides on the persisted draft (pinned at fork); undefined for a + // pre-feature draft or when editing the deployed flow directly (no draft). + draftBaseVersion = draftFromBackend?.version_id as number | undefined const effectiveFlow: Flow = draftFromBackend ? ({ ...deployedFlow, ...draftFromBackend } as Flow) : (deployedFlow as Flow) @@ -501,6 +512,8 @@ bind:othersModalOpen {draftSavedAt} {deployedAt} + {draftBaseVersion} + deployedHeadVersion={version} onLoadLatestDeploy={async () => { // stopSync-bracketed; see /scripts/edit's restoreDeployed for the race. if (!$workspaceStore) return @@ -563,6 +576,7 @@ {draftTriggersFromUrl} {selectedTriggerIndexFromUrl} {version} + {draftBaseVersion} {loadedFromHistoryFromUrl} /> {/if} diff --git a/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte b/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte index 2710167325..624db4061e 100644 --- a/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte @@ -2,9 +2,14 @@ import CompareWorkspaces from '$lib/components/CompareWorkspaces.svelte' import CompareDrafts from '$lib/components/CompareDrafts.svelte' import { WorkspaceService, type WorkspaceComparison } from '$lib/gen' + import { + archiveSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' import { useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' import { page } from '$app/state' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { onDestroy, untrack } from 'svelte' import CenteredPage from '$lib/components/CenteredPage.svelte' import PageHeader from '$lib/components/PageHeader.svelte' @@ -143,14 +148,9 @@ let acting = $state(false) async function afterForkGone() { - // Mirror SidebarContent.deleteFork (B1): refresh the workspace list - // rather than letting `clearStores()` null it, then land the user on - // the parent if still accessible. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } + // The workspace list was already refreshed by reconcileAfterWorkspaceChange + // (so the just-removed fork is gone from it); land the user on the parent if + // it's still accessible. if (parentWorkspaceId && $userWorkspaces.find((w) => w.id === parentWorkspaceId)) { switchWorkspace(parentWorkspaceId) await goto('/') @@ -166,6 +166,15 @@ try { await WorkspaceService.archiveWorkspace({ workspace: currentWorkspaceId }) sendUserToast(`Archived fork ${currentWorkspaceId}`) + // Client session cleanup is best-effort: a local IndexedDB failure must + // not falsely report the (already successful) archive as failed, nor + // block navigation away from the now-archived fork. + try { + await archiveSessionsForWorkspace(currentWorkspaceId) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after fork archive failed', e) + } await afterForkGone() } catch (e: any) { sendUserToast(`Failed to archive fork: ${e?.body ?? e}`, true) @@ -181,6 +190,15 @@ try { await WorkspaceService.deleteWorkspace({ workspace: currentWorkspaceId }) sendUserToast(`Deleted fork ${currentWorkspaceId}`) + // Client session cleanup is best-effort: a local IndexedDB failure must + // not abort the redirect after a successful delete, leaving the user on + // the now-deleted workspace path. + try { + await deleteSessionsForWorkspace(currentWorkspaceId) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after fork delete failed', e) + } await afterForkGone() } catch (e: any) { sendUserToast(`Failed to delete fork: ${e?.body ?? e}`, true) diff --git a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte index c98b11139f..3056245e51 100644 --- a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte @@ -33,11 +33,29 @@ import PipelineModeToggle from '$lib/components/assets/AssetGraph/PipelineModeToggle.svelte' import { parsePipelineAnnotations, + type ColumnLineage, type PipelineAnnotations } from '$lib/components/assets/AssetGraph/parsePipelineAnnotations' + import { + buildColumnGraph, + type ColumnLineageGraph + } from '$lib/components/assets/AssetGraph/columnLineageGraph' import { resolveGraph } from '$lib/components/assets/AssetGraph/resolveGraph' - import { computeDownstreamClosure } from '$lib/components/assets/AssetGraph/graphTraversal' - import { runCascade } from '$lib/components/assets/AssetGraph/cascadeOrchestrator' + import { + computeDownstreamClosure, + computeInducedSchedule + } from '$lib/components/assets/AssetGraph/graphTraversal' + import { runCascade, runSelection } from '$lib/components/assets/AssetGraph/cascadeOrchestrator' + import { + boundedSet, + buildLineageDag, + buildLineageDownstreamMap, + descendants, + isScriptNode, + scriptNodeId, + scriptsOf, + validStarts + } from '$lib/components/assets/AssetGraph/boundedCascade' import { diffDeployedGraph, extractCascadeFacts, @@ -65,8 +83,10 @@ History, Loader2, NetworkIcon, + Play, RefreshCw, Save, + Target, Telescope } from 'lucide-svelte' import { @@ -456,7 +476,9 @@ annotations: { inPipeline: false, triggerAssets: [], - nativeTriggers: [] + nativeTriggers: [], + dataTests: [], + columnLineage: [] } }) @@ -469,6 +491,7 @@ let liveBodyAssets = $state<{ scriptPath: string | undefined assets: AssetWithAltAccessType[] + columnLineage?: ColumnLineage[] }>({ scriptPath: undefined, assets: [] }) // The open draft's live editor buffer, emitted by the pane on every @@ -486,7 +509,13 @@ const EMPTY_LIVE_ASSETS = { scriptPath: undefined, assets: [] } const EMPTY_LIVE_ANNOTATIONS = { scriptPath: undefined, - annotations: { inPipeline: false, triggerAssets: [], nativeTriggers: [] } + annotations: { + inPipeline: false, + triggerAssets: [], + nativeTriggers: [], + dataTests: [], + columnLineage: [] + } } // Reset every live editor overlay (annotations / body assets / content) @@ -1174,14 +1203,18 @@ ) { liveAnnotations = { scriptPath, annotations } } - function handleAssetsChange(scriptPath: string | undefined, assets: AssetWithAltAccessType[]) { + function handleAssetsChange( + scriptPath: string | undefined, + assets: AssetWithAltAccessType[], + columnLineage?: ColumnLineage[] + ) { // Single update site for the live overlay. `inferredWritesByPath` // / `inferredReadsByPath` are now derived from `liveBodyAssets` // (for the open script) + `inferredAssetsByPath` (prefetched // snapshot for every other script), so we don't have to write // into those caches here — the derive picks up our update on the // next reactive tick. - liveBodyAssets = { scriptPath, assets } + liveBodyAssets = { scriptPath, assets, columnLineage } } function handleContentChange(scriptPath: string | undefined, content: string) { liveContent = { scriptPath, content } @@ -1659,7 +1692,11 @@ // edits is not picked up — same as production dispatch). async function launchCascadeScript(path: string): Promise { if (!$workspaceStore) throw new Error('no workspace') - const draft = drafts.get(path) + // Only run draft content when the displayed graph actually includes + // drafts (same condition as `displayGraph`). Otherwise — View mode with + // drafts hidden — a bounded run must execute the *deployed* scripts the + // user is looking at, not preview jobs from hidden local drafts. + const draft = mode === 'edit' || includeDrafts ? drafts.get(path) : undefined if (draft) { if (!draft.script.content || !draft.script.language) { throw new Error(`draft ${path} has no content/language`) @@ -1761,6 +1798,142 @@ } return rootJobId } + + // ── Bounded-cascade selective execution ────────────────────────────── + // "Run downstream up to…" lets the user run a *prefix* of a cascade: start + // at a schedule/manual root, fan downstream, but stop at chosen end node(s). + // The matched set is the path-between of start and ends over the lineage DAG. + // Pick state is in engine node-id space (`script:path`, `${kind}:${path}`); + // it is converted to canvas ids only at the canvas boundary. + let boundPickStart = $state(undefined) + let boundPickEnds = $state>(new Set()) + + // Script paths eligible to start a bounded run, for the canvas menu gate. + let validStartPaths = $derived(new Set(scriptsOf(validStarts(displayGraph)))) + // Scripts with read-aware downstream — the same gate the canvas applies + // (AssetGraphCanvas `hasLineageDownstream`). A valid start with no downstream + // has no end to pick, so the bounded-run entry is suppressed everywhere, + // including the details-pane (ScriptEditor) Test caret. + let lineageDownstreamPaths = $derived(new Set(buildLineageDownstreamMap(displayGraph).keys())) + + // Rebuilt only while a pick is active (cheap to skip otherwise). + let boundDag = $derived(boundPickStart ? buildLineageDag(displayGraph) : undefined) + let boundEligible = $derived( + boundDag && boundPickStart ? descendants(boundDag, boundPickStart) : new Set() + ) + let boundResult = $derived( + boundDag && boundPickStart + ? boundedSet(boundDag, boundPickStart, [...boundPickEnds]) + : undefined + ) + let boundScripts = $derived(boundResult ? scriptsOf(boundResult.nodes) : []) + + // Engine id → canvas id: scripts keep `script:path`; assets gain the + // canvas's `asset:` prefix (AssetGraphCanvas node ids). + const toCanvasId = (eid: string): string => (isScriptNode(eid) ? eid : `asset:${eid}`) + const fromCanvasId = (cid: string): string => + cid.startsWith('asset:') ? cid.slice('asset:'.length) : cid + // Short label for a `script:f/folder/name` start id (last path segment). + const shortPath = (scriptId: string): string => { + const p = isScriptNode(scriptId) ? scriptId.slice('script:'.length) : scriptId + return p.split('/').pop() ?? p + } + let boundPick = $derived( + boundPickStart && boundDag + ? { + start: boundPickStart, + eligible: new Set([...boundEligible].map(toCanvasId)), + ends: new Set([...boundPickEnds].map(toCanvasId)), + bounded: new Set([...(boundResult?.nodes ?? [])].map(toCanvasId)) + } + : undefined + ) + + function startBoundedRun(path: string) { + boundPickStart = scriptNodeId(path) + boundPickEnds = new Set() + } + function pickBoundEnd(canvasNodeId: string) { + const eid = fromCanvasId(canvasNodeId) + if (eid === boundPickStart) return + const next = new Set(boundPickEnds) + if (next.has(eid)) next.delete(eid) + else next.add(eid) + boundPickEnds = next + } + function cancelBoundedRun() { + boundPickStart = undefined + boundPickEnds = new Set() + } + async function confirmBoundedRun() { + const scripts = boundScripts + cancelBoundedRun() + await runBoundedCascade(scripts) + } + // Run an arbitrary selected set of scripts in topological order. Same + // per-hop launch + poll as the draft-aware cascade (it skips the backend + // dispatcher so the page owns the whole closure), but multi-root: every + // selected script with no in-set upstream is seeded at once. + async function runBoundedCascade(scripts: string[]): Promise { + if (scripts.length === 0) { + sendUserToast('No scripts to run in this selection', true) + return + } + if (cascadeRunningRoot) { + sendUserToast(`A chain run from ${cascadeRunningRoot} is still in progress`, true) + return + } + // Read-aware adjacency so a pure-reader member runs after its producer + // (parity with the CLI `topoOrder`); see buildLineageDownstreamMap. + const schedule = computeInducedSchedule( + displayGraph, + new Set(scripts), + buildLineageDownstreamMap(displayGraph) + ) + if (schedule.cyclic.length > 0) { + sendUserToast( + `Not running ${schedule.cyclic.length} script(s) on a dependency cycle: ${schedule.cyclic.join(', ')}`, + true + ) + } + if (schedule.nodes.length === 0) { + sendUserToast('No runnable scripts in this selection', true) + return + } + cascadeRunningRoot = schedule.roots[0] ?? scripts[0] + let firstJobId: string | undefined + try { + const res = await runSelection({ + schedule, + launch: async (path) => { + const jobId = await launchCascadeScript(path) + activeRunnables.arm(`script:${path}`) + if (firstJobId === undefined) { + firstJobId = jobId + runsPendingJobId = jobId + runsRefreshKey++ + } + return jobId + }, + waitTerminal: waitJobTerminal + }) + const n = res.statuses.size + if (res.ok) { + sendUserToast(`Bounded run complete — ${n} script${n === 1 ? '' : 's'} succeeded`) + } else { + const failed = [...res.statuses.entries()].filter(([, s]) => s.status === 'failure') + const skipped = [...res.statuses.values()].filter((s) => s.status === 'skipped').length + sendUserToast( + `Bounded run failed at ${failed.map(([p]) => p).join(', ')}` + + (skipped > 0 ? ` — ${skipped} downstream skipped` : ''), + true + ) + } + } finally { + cascadeRunningRoot = undefined + } + } + // Counter bumped when the canvas Run button targets the currently-open // script — the pane intercepts and routes through ScriptEditor.runTest // so logs/result/cancel land in the test panel instead of going off @@ -1802,6 +1975,48 @@ .map((e) => ({ kind: e.runnable_kind, path: e.runnable_path, unsaved: e.unsaved })) }) + // Empty graph reused when the trace isn't shown (no ducklake-asset selection, + // or a draft is actively edited) so the pane blanks out like the other + // selection overlays and `buildColumnGraph` doesn't run. + const EMPTY_COLUMN_GRAPH: ColumnLineageGraph = { + nodes: new Map(), + up: new Map(), + down: new Map() + } + // Pipeline-wide column-lineage graph, stitched across every producer's + // (inferred + annotated) `column_lineage` and the asset write-edges. Drives + // the transitive column trace in the details pane. Built from `displayGraph` + // — the exact graph the canvas renders — so the trace matches it: draft + // overlays in edit / show-drafts, deployed-only in plain View. Gated to a + // ducklake-asset selection so it isn't rebuilt on every editor keystroke when + // the trace UI isn't even shown. + let columnGraph = $derived( + selection?.kind === 'asset' && selection.asset_kind === 'ducklake' + ? buildColumnGraph(displayGraph) + : EMPTY_COLUMN_GRAPH + ) + + // Whether the selected ducklake asset's captured schema can *evolve* (drives + // the asset panel's Schema tab: version history vs. a single fixed schema). + // Only a whole-table `replace` producer (CREATE OR REPLACE) can change + // columns run-to-run; `append`/`merge`/partitioned writes INSERT into a + // fixed-schema table, so their schema is pinned at first materialize. + // + // Fail open: show the fixed view only when we're *sure* — every producer is a + // known insert-style write. A producer with no `materialize_strategy` + // metadata (e.g. a draft-overlay runnable, which the graph synthesizes + // without it) is treated as unknown → evolvable, so captured history is never + // hidden behind a stale "fixed" verdict. + let schemaCanEvolve = $derived.by(() => { + const sel = selection + if (!sel || sel.kind !== 'asset' || sel.asset_kind !== 'ducklake') return true + const producerPaths = new Set(selectionProducers.map((p) => p.path)) + const producers = graphWithDraft.runnables.filter((r) => producerPaths.has(r.path)) + const knownFixed = (r: (typeof producers)[number]) => + !!r.materialize_strategy && !(r.materialize_strategy === 'replace' && !r.partition_kind) + return producers.length === 0 || !producers.every(knownFixed) + }) + // Downstream subscriber count for the currently-edited script. Drives // the Test button's cascade UX: when > 0, ScriptEditor renders a split // button exposing "just this step" (default, with `_wmill_skip_asset_dispatch`) @@ -2278,8 +2493,42 @@ onAddPipelineScript={mode === 'edit' ? handleAddPipelineScript : undefined} onRunnableMenuRemove={mode === 'edit' ? handleRunnableMenuRemove : undefined} onRunProducer={mode === 'edit' ? handleRunProducer : undefined} + validStartPaths={isOperator ? undefined : validStartPaths} + onStartBoundedRun={isOperator ? undefined : startBoundedRun} + {boundPick} + onPickEnd={pickBoundEnd} {panToNodeId} /> + {#if boundPick} + +
    + +
    + + {boundPickEnds.size === 0 + ? 'Click end node(s) to bound the run' + : `${boundScripts.length} script${boundScripts.length === 1 ? '' : 's'} up to ${boundPickEnds.size} end${boundPickEnds.size === 1 ? '' : 's'}`} + + + from {boundPickStart ? shortPath(boundPickStart) : ''} + +
    + + +
    + {/if} {#if mode === 'edit'} @@ -2341,10 +2590,17 @@ onRunByPath={runByPathLegit} selection={activeDraft ? undefined : selection} selectionProducers={activeDraft ? [] : selectionProducers} + selectionColumnGraph={activeDraft ? EMPTY_COLUMN_GRAPH : columnGraph} + {schemaCanEvolve} {runsRefreshKey} {runsPendingJobId} {activeRunnable} downstreamSubscribers={editedScriptDownstreamCount} + onStartBoundedRun={openScriptPath && + validStartPaths.has(openScriptPath) && + lineageDownstreamPaths.has(openScriptPath) + ? () => startBoundedRun(openScriptPath!) + : undefined} onRunCompleted={() => { activeRunnable = undefined activeRunnableJobId = undefined diff --git a/frontend/src/routes/(root)/(logged)/run/[...run]/+page.svelte b/frontend/src/routes/(root)/(logged)/run/[...run]/+page.svelte index 7913d48180..a859345fe7 100644 --- a/frontend/src/routes/(root)/(logged)/run/[...run]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/run/[...run]/+page.svelte @@ -56,6 +56,7 @@ import LogViewer from '$lib/components/LogViewer.svelte' import { ActionRow, Button, Skeleton, Tab, Alert, DrawerContent } from '$lib/components/common' import JobDetailHeader from '$lib/components/runs/JobDetailHeader.svelte' + import ScriptRetryChain from '$lib/components/runs/ScriptRetryChain.svelte' import FlowExecutionStatus from '$lib/components/runs/FlowExecutionStatus.svelte' import JobArgs from '$lib/components/JobArgs.svelte' import FlowProgressBar from '$lib/components/flows/FlowProgressBar.svelte' @@ -868,6 +869,10 @@
    + {#if job} + + {/if} + {#if isNotFlow(job?.job_kind)} {#if ['python3', 'bun', 'deno'].includes(job?.language ?? '') && (job?.job_kind == 'script' || isScriptPreview(job?.job_kind))} diff --git a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte index a733c325d6..1d6f3f45ec 100644 --- a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte @@ -7,7 +7,6 @@ import SessionWrapper from '$lib/components/sessions/SessionWrapper.svelte' import { createSession, - getEffectiveWorkspaceId, selectSession, sessionState, syncWorkspaceTo @@ -19,7 +18,6 @@ promoteEditorWarm } from '$lib/components/sessions/sessionRuntime.svelte' import { markSessionSeen } from '$lib/components/sessions/sessionUnread.svelte' - import { visibleWorkspaceIds } from '$lib/components/sessions/sessionScope.svelte' import { isGlobalAiEnabled } from '$lib/components/copilot/chat/global/gate' import { userWorkspaces } from '$lib/stores' @@ -46,21 +44,10 @@ untrack(() => syncWorkspaceTo(ws)) }) - // Resolve the active session if its effective workspace is in scope - // (active workspace + its forks). Unavailable sessions — committed to - // a workspace that no longer exists — also resolve so the user can - // land on the move/discard banner instead of hitting "Session not - // found". - const activeSession = $derived( - sessionState.sessions.find((s) => { - if (s.name !== sessionName) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false - }) - ) + // sessionState.sessions holds every local session for the user. Resolve by + // name without applying the sidebar root filter so an open chat survives + // workspace switches. + const activeSession = $derived(sessionState.sessions.find((s) => s.name === sessionName)) // Touch the runtime for the active session so it gets created on first visit // and the pane shows up. Subsequent renders find it via listRuntimes(). @@ -94,20 +81,15 @@ }) }) - // Warm = has a live runtime (module-scoped) AND its workspace is in - // scope (or its workspace is unavailable — those sessions still need - // to render the move/discard banner instead of vanishing on us). + // Warm = sessions that currently have a live (module-scoped) runtime. The + // picker eagerly creates runtimes for its visible sessions, so this tracks + // whatever the picker shows — the current family, or every family when + // "Show all workspaces" is on. Runtimes whose session record isn't loaded + // resolve to undefined here and drop out. const warmSessions = $derived( listRuntimes() .map((r) => sessionState.sessions.find((s) => s.id === r.sessionId)) .filter((s): s is NonNullable => s != null) - .filter((s) => { - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false - }) ) // Promote the active session in the LRU. Mutations untracked so the effect diff --git a/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte b/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte index e71f0bb71e..f4f36eeb4d 100644 --- a/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/workspace_settings/+page.svelte @@ -68,6 +68,13 @@ type DucklakeSettingsType } from '$lib/components/workspaceSettings/DucklakeSettings.svelte' import UnsavedConfirmationModal from '$lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte' + import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' + import { + archiveSessionsForWorkspace, + countSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' import TextInput from '$lib/components/text_input/TextInput.svelte' import CollapseLink from '$lib/components/CollapseLink.svelte' import { validateWebhookUrl, validateEncryptionKey } from '$lib/validators/workspaceSettings' @@ -87,6 +94,65 @@ let slack_team_name: string | undefined = $state() let teams_team_id: string | undefined = $state() let teams_team_name: string | undefined = $state() + + // Workspace archive/delete cascade to the workspace's client-side AI sessions + // (archive → archive, delete → delete) via reconcileSessionsLifecycle. The + // confirmation modals warn how many sessions are affected first. + let archiveConfirmOpen = $state(false) + let deleteConfirmOpen = $state(false) + let affectedSessionCount = $state(0) + + async function openArchiveConfirm() { + affectedSessionCount = await countSessionsForWorkspace($workspaceStore ?? '') + archiveConfirmOpen = true + } + async function openDeleteConfirm() { + affectedSessionCount = await countSessionsForWorkspace($workspaceStore ?? '') + deleteConfirmOpen = true + } + async function doArchiveWorkspace() { + const ws = $workspaceStore ?? '' + // Land on the parent workspace if this is a fork and the parent is still + // accessible — otherwise fall back to the workspace picker. + const parentId = $userWorkspaces.find((w) => w.id === ws)?.parent_workspace_id + const parentStillAccessible = !!(parentId && $userWorkspaces.find((w) => w.id === parentId)) + await WorkspaceService.archiveWorkspace({ workspace: ws }) + sendUserToast(`Archived workspace ${ws}`) + // Best-effort client cleanup: a local IndexedDB failure must not strand the + // user on the just-archived workspace. The reconcile also refreshes the + // workspace list (dropping the archived one) so the parent-accessible check + // below — captured before the archive — still routes correctly. + try { + await archiveSessionsForWorkspace(ws) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after workspace archive failed', e) + } + if (parentStillAccessible && parentId) { + switchWorkspace(parentId) + await goto('/') + } else { + workspaceStore.set(undefined) + usersWorkspaceStore.set(undefined) + await goto('/user/workspaces') + } + } + async function doDeleteWorkspace() { + const ws = $workspaceStore ?? '' + await WorkspaceService.deleteWorkspace({ workspace: ws }) + sendUserToast(`Deleted workspace ${ws}`) + // Best-effort client cleanup — must not block navigation off the deleted + // workspace if a local IndexedDB op throws. + try { + await deleteSessionsForWorkspace(ws) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after workspace delete failed', e) + } + workspaceStore.set(undefined) + usersWorkspaceStore.set(undefined) + await goto('/user/workspaces') + } let useCustomSlackApp: boolean = $state(false) let slackAppType: 'instance' | 'workspace' = $state('instance') @@ -1543,34 +1609,7 @@ disabled={$workspaceStore === 'admins' || $workspaceStore === 'starter'} unifiedSize="md" btnClasses="mt-2" - on:click={async () => { - const ws = $workspaceStore ?? '' - // Land on the parent workspace if this is a fork and the - // parent is still accessible — otherwise fall back to the - // workspace picker. - const parentId = $userWorkspaces.find((w) => w.id === ws)?.parent_workspace_id - const parentStillAccessible = !!( - parentId && $userWorkspaces.find((w) => w.id === parentId) - ) - await WorkspaceService.archiveWorkspace({ workspace: ws }) - sendUserToast(`Archived workspace ${ws}`) - if (parentStillAccessible && parentId) { - // Refresh the list so the just-archived workspace drops out before - // we land on the parent. Guarded: a refresh failure must not block - // the switch (the list reloads on next page load). - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces after archive', e) - } - switchWorkspace(parentId) - await goto('/') - } else { - workspaceStore.set(undefined) - usersWorkspaceStore.set(undefined) - await goto('/user/workspaces') - } - }} + on:click={openArchiveConfirm} > Archive workspace @@ -1581,18 +1620,51 @@ disabled={$workspaceStore === 'admins' || $workspaceStore === 'starter'} size="sm" btnClasses="mt-2" - on:click={async () => { - await WorkspaceService.deleteWorkspace({ workspace: $workspaceStore ?? '' }) - sendUserToast(`Deleted workspace ${$workspaceStore}`) - workspaceStore.set(undefined) - usersWorkspaceStore.set(undefined) - goto('/user/workspaces') - }} + on:click={openDeleteConfirm} > Delete workspace (superadmin) {/if}
    + + { + archiveConfirmOpen = false + await doArchiveWorkspace() + }} + onCanceled={() => (archiveConfirmOpen = false)} + > +
    + + Archiving this workspace also archives its AI sessions{affectedSessionCount > 0 + ? ` (${affectedSessionCount})` + : ''}. Unarchiving the workspace restores them. + +
    +     + + { + deleteConfirmOpen = false + await doDeleteWorkspace() + }} + onCanceled={() => (deleteConfirmOpen = false)} + > +
    + + Permanently deleting this workspace also permanently deletes its AI sessions{affectedSessionCount > + 0 + ? ` (${affectedSessionCount})` + : ''}. This cannot be undone. + +
    +     {:else if tab == 'webhook'} void) | undefined + + // Embedder side: validate access (main session cookie or shared JWT) and mint + // a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { if (parsedCustomPath.jwt) { - const token = 'jwt_ext_' + parsedCustomPath.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false + OpenAPI.TOKEN = 'jwt_ext_' + parsedCustomPath.jwt } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/apps_u/embed_token_by_custom_path/${parsedCustomPath.path}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. + async function loadApp() { try { app = await AppService.getPublicAppByCustomPath({ customPath: parsedCustomPath.path @@ -62,9 +91,13 @@ workspaceStore.set(app.workspace_id) noPermission = false notExists = false + jwtError = false try { userStore.set(await getUserExt(app.workspace_id)) + // A JWT in the custom path that fails to resolve a user is surfaced as a + // toast (matches the pre-sandbox custom-path viewer) rather than silently + // falling through to anonymous. if (!$userStore && parsedCustomPath.jwt) { jwtError = true sendUserToast('Could not authentify user with jwt token', true) @@ -74,7 +107,8 @@ } } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -83,17 +117,25 @@ if (BROWSER) { setLicense() - loadApp() } - { + { + refresh = requestTokenRefresh loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte new file mode 100644 index 0000000000..d0af10b433 --- /dev/null +++ b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte @@ -0,0 +1,99 @@ + + + { + refresh = requestTokenRefresh + loadApp() + }} +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js deleted file mode 100644 index 42a8b51427..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `Public App` } - } -} diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte deleted file mode 100644 index 38563cfbd9..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte +++ /dev/null @@ -1,11 +0,0 @@ - - - diff --git a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte index c661aa48c6..089b95c97d 100644 --- a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte +++ b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte @@ -4,21 +4,20 @@ import { AppService, OpenAPI, type AppWithLastVersion } from '$lib/gen' import { userStore } from '$lib/stores' - import { setContext } from 'svelte' import { setLicense } from '$lib/enterpriseUtils' import { getUserExt } from '$lib/user' - import { sendUserToast } from '$lib/toast' import { page } from '$app/state' + import { base } from '$lib/base' import PublicApp from '$lib/components/apps/editor/PublicApp.svelte' + import PublicAppFrame from '$lib/components/apps/editor/PublicAppFrame.svelte' let app: (AppWithLastVersion & { value: any }) | undefined = $state(undefined) let notExists = $state(false) let noPermission = $state(false) - let jwtError = $state(false) - function parseSecret(secret: string): { secret: string; jwt: string } { + function parseSecret(secret: string): { secret: string; jwt: string | undefined } { const parts = secret.split('/') return { secret: parts[0], @@ -27,18 +26,59 @@ } const parsedSecret = parseSecret(page.params.secret ?? '') + const workspace = page.params.workspace ?? '' + // URL for the opaque viewer iframe: the share URL WITHOUT the trailing JWT + // segment. The JWT is a viewer credential (broader and longer-lived than the + // scoped embed token) consumed here on the embedder side only — it must never + // appear in the iframe's own location, where app-authored code could read it. + // Captured once (not reactively): the embedder mirrors the app's hash/query + // back onto this page's URL, and re-deriving the src from it would reload the + // app on its every navigation. + const viewerUrl = `${base}/public/${workspace}/${parsedSecret.secret}${page.url.search}${page.url.hash}` + + let refresh: (() => void) | undefined + + // Embedder side: validate access (using the main session cookie or the shared + // JWT) and mint a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { + if (parsedSecret.jwt) { + OpenAPI.TOKEN = 'jwt_ext_' + parsedSecret.jwt + } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/w/${workspace}/apps_u/embed_token/${parsedSecret.secret}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. async function loadApp() { + try { + userStore.set(await getUserExt(workspace)) + } catch (e) { + console.warn('Anonymous user') + } try { app = await AppService.getPublicAppBySecret({ - workspace: page.params.workspace ?? '', + workspace, path: parsedSecret.secret }) noPermission = false notExists = false } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -47,42 +87,25 @@ if (BROWSER) { setLicense() - loadAll() - } - - function loadAll() { - console.log('loadAll') - loadUser().then(() => { - loadApp() - }) - } - - async function loadUser() { - if (parsedSecret.jwt) { - const token = 'jwt_ext_' + parsedSecret.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false - } - try { - userStore.set(await getUserExt(page.params.workspace ?? '')) - if (!$userStore && parsedSecret.jwt) { - jwtError = true - sendUserToast('Could not authentify user with jwt token', true) - } - } catch (e) { - console.warn('Anonymous user') - } } - { - loadAll() + { + refresh = requestTokenRefresh + loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/vite.config.js b/frontend/vite.config.js index 989fc5cbe2..ac927801dc 100644 --- a/frontend/vite.config.js +++ b/frontend/vite.config.js @@ -1,5 +1,5 @@ import { sveltekit } from '@sveltejs/kit/vite' -import { readFileSync } from 'fs' +import { existsSync, readFileSync } from 'fs' import { fileURLToPath } from 'url' import mkcert from 'vite-plugin-mkcert' @@ -7,6 +7,15 @@ const file = fileURLToPath(new URL('package.json', import.meta.url)) const json = readFileSync(file, 'utf8') const version = JSON.parse(json) +// The postinstall downloads the pinned UI Builder artifact into static/ui_builder, +// which SvelteKit serves at /ui_builder. Serve that directly; only proxy to a +// live UI Builder dev server on :4000 when the bundle is absent (mirrors the +// backend's static-vs-:4000 fallback). Delete static/ui_builder to develop the +// builder against :4000. +const uiBuilderStaticPresent = existsSync( + fileURLToPath(new URL('static/ui_builder/app-preview.html', import.meta.url)) +) + const remoteUrl = process.env.REMOTE ?? (process.env.BACKEND_PORT @@ -84,15 +93,19 @@ const config = { changeOrigin: true, ws: true }, - '^/ui_builder/.*': { - target: 'http://localhost:4000', - changeOrigin: true, - headers: { - 'Cross-Origin-Opener-Policy': 'same-origin', - 'Cross-Origin-Embedder-Policy': 'require-corp', - 'Cross-Origin-Resource-Policy': 'cross-origin' - } - } + ...(uiBuilderStaticPresent + ? {} + : { + '^/ui_builder/.*': { + target: 'http://localhost:4000', + changeOrigin: true, + headers: { + 'Cross-Origin-Opener-Policy': 'same-origin', + 'Cross-Origin-Embedder-Policy': 'require-corp', + 'Cross-Origin-Resource-Policy': 'cross-origin' + } + } + }) } }, preview: { port: 3001 }, diff --git a/integration_tests/ai_agent_tests/conftest.py b/integration_tests/ai_agent_tests/conftest.py index 681304e614..5e1e2cbf9f 100644 --- a/integration_tests/ai_agent_tests/conftest.py +++ b/integration_tests/ai_agent_tests/conftest.py @@ -18,6 +18,80 @@ load_dotenv(Path(__file__).parent / ".env") TEST_IMAGE_PATH = Path(__file__).parent / "test_image.webp" TEST_IMAGE_S3_KEY = "test_images/test_image.webp" +# Env vars each provider needs before its cases can run. Cases for a provider +# whose keys are absent are skipped instead of failed, so a CI run (or a local +# dev) can exercise only the providers it has credentials for. +PROVIDER_ENV_REQUIREMENTS: dict[str, list[str]] = { + "openai": ["OPENAI_API_KEY"], + "azure_openai": ["AZURE_OPENAI_API_KEY", "AZURE_OPENAI_BASE_URL"], + "anthropic": ["ANTHROPIC_API_KEY"], + "google_ai": ["GOOGLE_AI_API_KEY"], + "openrouter": ["OPENROUTER_API_KEY"], + "bedrock": ["BEDROCK_API_KEY"], + "bedrock_api_key": ["BEDROCK_API_KEY"], + "bedrock_iam": ["BEDROCK_IAM_ACCESS_KEY_ID", "BEDROCK_IAM_SECRET_ACCESS_KEY"], + "bedrock_iam_session": [ + "BEDROCK_SESSION_ACCESS_KEY_ID", + "BEDROCK_SESSION_SECRET_ACCESS_KEY", + "BEDROCK_SESSION_TOKEN", + ], + "bedrock_env": ["AWS_ACCESS_KEY_ID", "AWS_SECRET_ACCESS_KEY"], +} + + +def pytest_configure(config): + config.addinivalue_line( + "markers", + "requires_provider(provider): skip test when the provider's credentials are absent", + ) + + +def _provider_name_from_item(item) -> str | None: + callspec = getattr(item, "callspec", None) + if callspec is None: + marker = item.get_closest_marker("requires_provider") + if marker is None: + return None + provider = marker.args[0] if marker.args else marker.kwargs.get("provider") + else: + provider = callspec.params.get("provider_config") + + if isinstance(provider, str): + return provider + if isinstance(provider, dict) and isinstance(provider.get("name"), str): + return provider["name"] + return None + + +def _provider_skip_reason(provider_name: str) -> str | None: + required = PROVIDER_ENV_REQUIREMENTS.get(provider_name, []) + missing = [name for name in required if not os.environ.get(name)] + if missing: + return f"{provider_name}: missing {', '.join(missing)}" + return None + + +def pytest_collection_modifyitems(config, items): + for item in items: + provider_name = _provider_name_from_item(item) + if provider_name is None: + continue + + reason = _provider_skip_reason(provider_name) + if reason is not None: + item.add_marker(pytest.mark.skip(reason=reason)) + + +@pytest.fixture(autouse=True) +def skip_provider_without_credentials(request): + """Skip dynamic provider cases when that provider's keys are not set.""" + provider_name = _provider_name_from_item(request.node) + if provider_name is None: + return + reason = _provider_skip_reason(provider_name) + if reason is not None: + pytest.skip(reason) + class AIAgentTestClient: """HTTP client for testing AI agents via the preview_flow endpoint.""" diff --git a/integration_tests/ai_agent_tests/test_completion_params.py b/integration_tests/ai_agent_tests/test_completion_params.py index c937c540a6..af53ce4ce0 100644 --- a/integration_tests/ai_agent_tests/test_completion_params.py +++ b/integration_tests/ai_agent_tests/test_completion_params.py @@ -5,7 +5,7 @@ Tests that AI agents correctly handle temperature and max_completion_tokens: - Default parameters (undefined) - Low temperature (0.0 - deterministic) - High temperature (0.9 - more random) -- Low max_completion_tokens (10 - short response) +- Low max_completion_tokens (16 - short response; OpenAI's minimum) - High max_completion_tokens (4096 - longer response allowed) - Combined parameters """ @@ -132,14 +132,15 @@ class TestCompletionParams: provider_config, ): """ - Test with max_completion_tokens=10 (short response). - The response should be truncated or very short. + Test with a low max_completion_tokens (16 — short response). + The response should be truncated or very short. 16 is OpenAI's minimum + for max_output_tokens; lower values (e.g. 10) are rejected with a 400. """ flow_value = create_ai_agent_flow( provider_input_transform=provider_config["input_transform"], system_prompt="You are a helpful assistant.", output_type="text", - max_completion_tokens=10, + max_completion_tokens=16, ) result = client.run_preview_flow( @@ -153,7 +154,7 @@ class TestCompletionParams: result_str = str(result) assert len(result_str) > 0, f"Expected non-empty result: {result}" - print(f"Low max_tokens (10) result from {provider_config['name']}: {result}") + print(f"Low max_tokens (16) result from {provider_config['name']}: {result}") @pytest.mark.parametrize( "provider_config", diff --git a/integration_tests/ai_agent_tests/test_tool_calling.py b/integration_tests/ai_agent_tests/test_tool_calling.py index 7ac559250a..c80158fd3a 100644 --- a/integration_tests/ai_agent_tests/test_tool_calling.py +++ b/integration_tests/ai_agent_tests/test_tool_calling.py @@ -119,6 +119,7 @@ class TestToolCalling: assert "23" in result_str, f"Expected '23' in result: {result}" print(f"Workspace script tool result from {provider_config['name']}: {result}") + @pytest.mark.requires_provider("google_ai") def test_nested_ai_agent_tool_with_gemini_3( self, client: AIAgentTestClient, diff --git a/lsp/Pipfile b/lsp/Pipfile index c8e82b7653..7f9d00903a 100644 --- a/lsp/Pipfile +++ b/lsp/Pipfile @@ -4,7 +4,7 @@ verify_ssl = true name = "pypi" [packages] -wmill = ">=1.734.0" +wmill = ">=1.742.0" sendgrid = "*" mysql-connector-python = "*" pymongo = "*" diff --git a/openflow.openapi.yaml b/openflow.openapi.yaml index 55ec973252..f4342930e9 100644 --- a/openflow.openapi.yaml +++ b/openflow.openapi.yaml @@ -1,7 +1,7 @@ openapi: '3.0.3' info: - version: 1.734.0 + version: 1.742.0 title: OpenFlow Spec contact: name: Ruben Fiszel diff --git a/powershell-client/WindmillClient/WindmillClient.psd1 b/powershell-client/WindmillClient/WindmillClient.psd1 index 74118fc367..e5b553e6f3 100644 --- a/powershell-client/WindmillClient/WindmillClient.psd1 +++ b/powershell-client/WindmillClient/WindmillClient.psd1 @@ -12,7 +12,7 @@ RootModule = 'WindmillClient.psm1' # Version number of this module. - ModuleVersion = '1.734.0' + ModuleVersion = '1.742.0' # Supported PSEditions # CompatiblePSEditions = @() diff --git a/powershell-client/WindmillClient/WindmillClient.psm1 b/powershell-client/WindmillClient/WindmillClient.psm1 index 9f0aca4b1a..f2d241ddc7 100644 --- a/powershell-client/WindmillClient/WindmillClient.psm1 +++ b/powershell-client/WindmillClient/WindmillClient.psm1 @@ -258,14 +258,15 @@ function Invoke-WindmillScript { [string] $Hash = $null, [Hashtable] $Arguments = @{}, [boolean] $AssertResultIsNotNull = $true, - [int] $Timeout = $null + [int] $Timeout = $null, + [string] $Tag = $null ) if (-not $script:WindmillConnection) { throw "Windmill connection not established. Run Connect-Windmill first." } - $jobId = Start-WindmillScript -Path $Path -Hash $Hash -Arguments $Arguments + $jobId = Start-WindmillScript -Path $Path -Hash $Hash -Arguments $Arguments -Tag $Tag $until = if ($Timeout) { (Get-Date).AddSeconds($Timeout) } else { [DateTime]::MaxValue } return $script:WindmillConnection.WaitJob($jobId, $until, $AssertResultIsNotNull) } @@ -279,14 +280,15 @@ function Start-WindmillScript { [string] $Path = $null, [string] $Hash = $null, [Hashtable] $Arguments = @{}, - [int] $ScheduledInSecs = $null + [int] $ScheduledInSecs = $null, + [string] $Tag = $null ) if (-not $script:WindmillConnection) { throw "Windmill connection not established. Run Connect-Windmill first." } - return $script:WindmillConnection.RunScriptAsync($Path, $Hash, $Arguments, $ScheduledInSecs) + return $script:WindmillConnection.RunScriptAsync($Path, $Hash, $Arguments, $ScheduledInSecs, $Tag) } <# @@ -297,14 +299,15 @@ function Start-WindmillFlow { param( [string] $Path = $null, [Hashtable] $Arguments = @{}, - [int] $ScheduledInSecs = $null + [int] $ScheduledInSecs = $null, + [string] $Tag = $null ) if (-not $script:WindmillConnection) { throw "Windmill connection not established. Run Connect-Windmill first." } - return $script:WindmillConnection.RunFlowAsync($Path, $Arguments, $ScheduledInSecs) + return $script:WindmillConnection.RunFlowAsync($Path, $Arguments, $ScheduledInSecs, $Tag) } <# @@ -596,7 +599,13 @@ class Windmill { return $result } + # Preserve the original 4-arg arity (PowerShell class dispatch is by exact + # argument count, so existing direct callers would otherwise break). [PSCustomObject] RunScriptAsync([string] $Path, [string] $Hash, [Hashtable] $Arguments, [int] $ScheduledInSecs) { + return $this.RunScriptAsync($Path, $Hash, $Arguments, $ScheduledInSecs, $null) + } + + [PSCustomObject] RunScriptAsync([string] $Path, [string] $Hash, [Hashtable] $Arguments, [int] $ScheduledInSecs, [string] $Tag) { $params = @{} if ($Path -and $Hash) { @@ -607,6 +616,10 @@ class Windmill { $params["scheduled_in_secs"] = $ScheduledInSecs } + if ($Tag) { + $params["tag"] = $Tag + } + if ($env:WM_JOB_ID) { $params["parent_job"] = $env:WM_JOB_ID } @@ -631,13 +644,23 @@ class Windmill { return $this.Post($endpoint, $Arguments, $true).Content } + # Preserve the original 3-arg arity (PowerShell class dispatch is by exact + # argument count, so existing direct callers would otherwise break). [string] RunFlowAsync([string] $Path, [Hashtable] $Arguments, [int] $ScheduledInSecs) { + return $this.RunFlowAsync($Path, $Arguments, $ScheduledInSecs, $null) + } + + [string] RunFlowAsync([string] $Path, [Hashtable] $Arguments, [int] $ScheduledInSecs, [string] $Tag) { $params = @{} if ($ScheduledInSecs -ne $null) { $params["scheduled_in_secs"] = $ScheduledInSecs } + if ($Tag) { + $params["tag"] = $Tag + } + # TODO: Figure out why this fails when we set parent_job (at least for HN Discord Feed) if ($env:WM_JOB_ID) { $params["parent_job"] = $env:WM_JOB_ID diff --git a/python-client/wmill/pyproject.toml b/python-client/wmill/pyproject.toml index 314edea15b..5310b72896 100644 --- a/python-client/wmill/pyproject.toml +++ b/python-client/wmill/pyproject.toml @@ -1,6 +1,6 @@ [tool.poetry] name = "wmill" -version = "1.734.0" +version = "1.742.0" description = "A client library for accessing Windmill server wrapping the Windmill client API" license = "Apache-2.0" homepage = "https://windmill.dev" diff --git a/python-client/wmill/wmill/client.py b/python-client/wmill/wmill/client.py index 597514a5e4..e1a59b312d 100644 --- a/python-client/wmill/wmill/client.py +++ b/python-client/wmill/wmill/client.py @@ -168,16 +168,17 @@ class Windmill: hash_: str = None, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job and return its job id. - + .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. """ logging.warning( "run_script_async is deprecated. Use run_script_by_path_async or run_script_by_hash_async instead.", ) assert not (path and hash_), "path and hash_ are mutually exclusive" - return self._run_script_async_internal(path=path, hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs) + return self._run_script_async_internal(path=path, hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs, tag=tag) def _run_script_async_internal( self, @@ -185,10 +186,13 @@ class Windmill: hash_: str = None, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Internal helper for running scripts asynchronously.""" args = args or {} params = {"scheduled_in_secs": scheduled_in_secs} if scheduled_in_secs else {} + if tag: + params["tag"] = tag if os.environ.get("WM_JOB_ID"): params["parent_job"] = os.environ.get("WM_JOB_ID") if os.environ.get("WM_ROOT_FLOW_JOB_ID"): @@ -208,18 +212,20 @@ class Windmill: path: str, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job by path and return its job id.""" - return self._run_script_async_internal(path=path, args=args, scheduled_in_secs=scheduled_in_secs) + return self._run_script_async_internal(path=path, args=args, scheduled_in_secs=scheduled_in_secs, tag=tag) def run_script_by_hash_async( self, hash_: str, args: dict = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job by hash and return its job id.""" - return self._run_script_async_internal(hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs) + return self._run_script_async_internal(hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs, tag=tag) def run_flow_async( self, @@ -230,10 +236,13 @@ class Windmill: # as otherwise the child flow and its own child will store their state in the parent job which will # lead to incorrectness and failures do_not_track_in_parent: bool = True, + tag: str = None, ) -> str: """Create a flow job and return its job id.""" args = args or {} params = {"scheduled_in_secs": scheduled_in_secs} if scheduled_in_secs else {} + if tag: + params["tag"] = tag if not do_not_track_in_parent: if os.environ.get("WM_JOB_ID"): params["parent_job"] = os.environ.get("WM_JOB_ID") @@ -254,9 +263,10 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Run script synchronously and return its result. - + .. deprecated:: Use run_script_by_path or run_script_by_hash instead. """ logging.warning( @@ -265,7 +275,7 @@ class Windmill: assert not (path and hash_), "path and hash_ are mutually exclusive" return self._run_script_internal( path=path, hash_=hash_, args=args, timeout=timeout, verbose=verbose, - cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none + cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none, tag=tag ) def _run_script_internal( @@ -277,6 +287,7 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Internal helper for running scripts synchronously.""" args = args or {} @@ -290,7 +301,7 @@ class Windmill: if isinstance(timeout, dt.timedelta): timeout = timeout.total_seconds() - job_id = self._run_script_async_internal(path=path, hash_=hash_, args=args) + job_id = self._run_script_async_internal(path=path, hash_=hash_, args=args, tag=tag) return self.wait_job( job_id, timeout, verbose, cleanup, assert_result_is_not_none ) @@ -303,11 +314,12 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Run script by path synchronously and return its result.""" return self._run_script_internal( path=path, args=args, timeout=timeout, verbose=verbose, - cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none + cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none, tag=tag ) def run_script_by_hash( @@ -318,11 +330,12 @@ class Windmill: verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, + tag: str = None, ) -> Any: """Run script by hash synchronously and return its result.""" return self._run_script_internal( hash_=hash_, args=args, timeout=timeout, verbose=verbose, - cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none + cleanup=cleanup, assert_result_is_not_none=assert_result_is_not_none, tag=tag ) def run_inline_script_preview( @@ -1453,6 +1466,7 @@ def run_script_async( hash_or_path: str, args: Dict[str, Any] = None, scheduled_in_secs: int = None, + tag: str = None, ) -> str: """Create a script job and return its job ID. @@ -1460,6 +1474,7 @@ def run_script_async( hash_or_path: Script hash or path (determined by presence of '/') args: Script arguments scheduled_in_secs: Delay before execution in seconds + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1472,6 +1487,7 @@ def run_script_async( path=path, args=args, scheduled_in_secs=scheduled_in_secs, + tag=tag, ) @@ -1484,6 +1500,7 @@ def run_flow_async( # as otherwise the child flow and its own child will store their state in the parent job which will # lead to incorrectness and failures do_not_track_in_parent: bool = True, + tag: str = None, ) -> str: """Create a flow job and return its job ID. @@ -1492,6 +1509,7 @@ def run_flow_async( args: Flow arguments scheduled_in_secs: Delay before execution in seconds do_not_track_in_parent: Whether to track in parent job (default: True) + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1501,6 +1519,7 @@ def run_flow_async( args=args, scheduled_in_secs=scheduled_in_secs, do_not_track_in_parent=do_not_track_in_parent, + tag=tag, ) @@ -1512,6 +1531,7 @@ def run_script_sync( assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, + tag: str = None, ) -> Any: """Run a script synchronously by hash and return its result. @@ -1522,6 +1542,7 @@ def run_script_sync( assert_result_is_not_none: Raise exception if result is None cleanup: Register cleanup handler to cancel job on exit timeout: Maximum time to wait + tag: Override the worker tag the job runs on Returns: Script result @@ -1533,6 +1554,7 @@ def run_script_sync( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -1541,6 +1563,7 @@ def run_script_by_path_async( path: str, args: Dict[str, Any] = None, scheduled_in_secs: Union[None, int] = None, + tag: str = None, ) -> str: """Create a script job by path and return its job ID. @@ -1548,6 +1571,7 @@ def run_script_by_path_async( path: Script path args: Script arguments scheduled_in_secs: Delay before execution in seconds + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1556,6 +1580,7 @@ def run_script_by_path_async( path=path, args=args, scheduled_in_secs=scheduled_in_secs, + tag=tag, ) @@ -1564,6 +1589,7 @@ def run_script_by_hash_async( hash_: str, args: Dict[str, Any] = None, scheduled_in_secs: Union[None, int] = None, + tag: str = None, ) -> str: """Create a script job by hash and return its job ID. @@ -1571,6 +1597,7 @@ def run_script_by_hash_async( hash_: Script hash args: Script arguments scheduled_in_secs: Delay before execution in seconds + tag: Override the worker tag the job runs on Returns: Job ID string @@ -1579,6 +1606,7 @@ def run_script_by_hash_async( hash_=hash_, args=args, scheduled_in_secs=scheduled_in_secs, + tag=tag, ) @@ -1590,6 +1618,7 @@ def run_script_by_path_sync( assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, + tag: str = None, ) -> Any: """Run a script synchronously by path and return its result. @@ -1600,6 +1629,7 @@ def run_script_by_path_sync( assert_result_is_not_none: Raise exception if result is None cleanup: Register cleanup handler to cancel job on exit timeout: Maximum time to wait + tag: Override the worker tag the job runs on Returns: Script result @@ -1611,6 +1641,7 @@ def run_script_by_path_sync( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -2062,9 +2093,10 @@ def run_script( verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = True, + tag: str = None, ) -> Any: """Run script synchronously and return its result. - + .. deprecated:: Use run_script_by_path or run_script_by_hash instead. """ return _client.run_script( @@ -2075,6 +2107,7 @@ def run_script( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -2086,6 +2119,7 @@ def run_script_by_path( verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = True, + tag: str = None, ) -> Any: """Run script by path synchronously and return its result.""" return _client.run_script_by_path( @@ -2095,6 +2129,7 @@ def run_script_by_path( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @@ -2106,6 +2141,7 @@ def run_script_by_hash( verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = True, + tag: str = None, ) -> Any: """Run script by hash synchronously and return its result.""" return _client.run_script_by_hash( @@ -2115,6 +2151,7 @@ def run_script_by_hash( assert_result_is_not_none=assert_result_is_not_none, cleanup=cleanup, timeout=timeout, + tag=tag, ) @init_global_client diff --git a/rust-client/src/client.rs b/rust-client/src/client.rs index 4fc9eca192..451dc58d85 100644 --- a/rust-client/src/client.rs +++ b/rust-client/src/client.rs @@ -629,7 +629,43 @@ impl Windmill { ret!(async move { let job_id = self - .run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs) + .run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, None) + .await?; + self.wait_job_inner( + &job_id.to_string(), + timeout_secs, + verbose, + assert_result_is_not_none, + ) + .await + }); + } + + /// Same as [`Windmill::run_script_sync`] but allows overriding the worker `tag` + /// the job runs on. + /// + /// # Parameters + /// In addition to the parameters of [`Windmill::run_script_sync`]: + /// - `tag`: Optional worker tag override (the job is dispatched to workers + /// listening on this tag instead of the script's default tag) + pub fn run_script_sync_with_tag<'a>( + &'a self, + ident: &'a str, + ident_is_hash: bool, + args: Value, + scheduled_in_secs: Option, + timeout_secs: Option, + verbose: bool, + assert_result_is_not_none: bool, + tag: Option<&'a str>, + ) -> MaybeFuture<'a, Result> { + if verbose { + println!("running `{ident}` synchronously with {:?}", &args); + } + + ret!(async move { + let job_id = self + .run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, tag) .await?; self.wait_job_inner( &job_id.to_string(), @@ -690,7 +726,25 @@ impl Windmill { args: Value, scheduled_in_secs: Option, ) -> MaybeFuture<'a, Result> { - ret!(self.run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs)); + ret!(self.run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, None)); + } + + /// Same as [`Windmill::run_script_async`] but allows overriding the worker `tag` + /// the job runs on. + /// + /// # Arguments + /// In addition to the arguments of [`Windmill::run_script_async`]: + /// * `tag` - Optional worker tag override (the job is dispatched to workers + /// listening on this tag instead of the script's default tag) + pub fn run_script_async_with_tag<'a>( + &'a self, + ident: &'a str, + ident_is_hash: bool, + args: Value, + scheduled_in_secs: Option, + tag: Option<&'a str>, + ) -> MaybeFuture<'a, Result> { + ret!(self.run_script_async_inner(ident, ident_is_hash, args, scheduled_in_secs, tag)); } async fn run_script_async_inner<'a>( @@ -699,6 +753,7 @@ impl Windmill { ident_is_hash: bool, mut args: Value, scheduled_in_secs: Option, + tag: Option<&'a str>, ) -> Result { if let Ok(parent_job) = var("WM_JOB_ID") { args["parent_job"] = json!(parent_job); @@ -712,6 +767,9 @@ impl Windmill { args["scheduled_in_secs"] = json!(scheduled_in_secs); } + // The `None`s below map positionally to the query params of the generated + // job API. The 5th one is the worker `tag` override (after scheduled_for, + // scheduled_in_secs, skip_preprocessor and parent_job). let uuid = if ident_is_hash { job_api::run_script_by_hash( &self.client_config, @@ -722,7 +780,7 @@ impl Windmill { None, None, None, - None, + tag, None, None, None, @@ -746,7 +804,7 @@ impl Windmill { None, None, None, - None, + tag, None, None, None, diff --git a/system_prompts/auto-generated/cli/cli-commands.md b/system_prompts/auto-generated/cli/cli-commands.md index f42aebc2d4..bcaa7ecc14 100644 --- a/system_prompts/auto-generated/cli/cli-commands.md +++ b/system_prompts/auto-generated/cli/cli-commands.md @@ -426,6 +426,11 @@ inspect asset-driven pipelines (scripts marked `// pipeline`, wired by `// on ` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal - `--json` - Output the raw asset graph as JSON +- `pipeline run ` - run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s) + - `--from ` - Start script (short name or path). Defaults to the folder's sole schedule/manual root. + - `--to ` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream. + - `--dry-run` - Print the topological run plan without executing. + - `--json` - Output the plan as JSON (for piping to jq). ### protection-rules diff --git a/system_prompts/auto-generated/prompts.d.ts b/system_prompts/auto-generated/prompts.d.ts index 32ddcdc4bd..46ab41d819 100644 --- a/system_prompts/auto-generated/prompts.d.ts +++ b/system_prompts/auto-generated/prompts.d.ts @@ -1,34 +1,36 @@ -export declare const SCRIPT_BASE = "# Windmill Script Writing Guide\n\n## General Principles\n\n- Scripts must export a main function (do not call it)\n- Libraries are installed automatically - do not show installation instructions\n- Credentials and configuration are stored in resources and passed as parameters\n- The windmill client (`wmill`) provides APIs for interacting with the platform\n\n## Function Naming\n\n- Main function: `main` (or `preprocessor` for preprocessor scripts)\n- Must be async for TypeScript variants\n\n## Return Values\n\n- Scripts can return any JSON-serializable value\n- Return values become available to subsequent flow steps via `results.step_id`\n\n## Preprocessor Scripts\n\nPreprocessor scripts process raw trigger data from various sources (webhook, custom HTTP route, SQS, WebSocket, Kafka, NATS, MQTT, Postgres, or email) before passing it to the flow. This separates the trigger logic from the flow logic and keeps the auto-generated UI clean.\n\nThe returned object determines the parameter values passed to the flow.\ne.g., `{ b: 1, a: 2 }` calls the flow with `a = 2` and `b = 1`, assuming the flow has two inputs called `a` and `b`.\n\nThe preprocessor receives a single parameter called `event`.\n"; -export declare const FLOW_BASE = "# Windmill Flow Building Guide\n\n## OpenFlow Schema\n\nThe OpenFlow schema (openflow.openapi.yaml) is the source of truth for flow structure. Refer to OPENFLOW_SCHEMA for the complete type definitions.\n\n## Reserved Module IDs\n\n- `failure` - Reserved for failure handler module\n- `preprocessor` - Reserved for preprocessor module\n- `Input` - Reserved for flow input reference\n\n## Hard Structural Rules\n\nThese are strict Windmill schema rules. Follow them exactly.\n\n- `value.modules` is only for normal sequential steps\n- `value.preprocessor_module` and `value.failure_module` are special top-level fields inside `value`, not entries in `value.modules`\n- If a flow needs a preprocessor, create `value.preprocessor_module` with `id: preprocessor`\n- If a flow needs a failure handler, create `value.failure_module` with `id: failure`\n- Do NOT create regular modules inside `value.modules` named `preprocessor` or `failure`\n- `preprocessor_module` and `failure_module` only support `script` or `rawscript`\n- `preprocessor_module` runs before normal modules and cannot reference `results.*`\n- `failure_module` can use the `error` object with `error.message`, `error.step_id`, `error.name`, and `error.stack`\n\nCorrect shape:\n\n```yaml\nvalue:\n preprocessor_module:\n id: preprocessor\n value:\n type: rawscript\n ...\n failure_module:\n id: failure\n value:\n type: rawscript\n ...\n modules:\n - id: process_event\n value:\n type: rawscript\n ...\n```\n\nIncorrect shape:\n\n```yaml\nvalue:\n modules:\n - id: preprocessor\n ...\n - id: process_event\n ...\n - id: failure\n ...\n```\n\n## Module ID Rules\n\n- Must be unique across the entire flow\n- Use underscores, not spaces (e.g., `fetch_data` not `fetch data`)\n- Use descriptive names that reflect the step's purpose\n\n## Common Mistakes to Avoid\n\n- Missing `input_transforms` - Rawscript parameters won't receive values without them\n- Referencing future steps - `results.step_id` only works for steps that execute before the current one\n- Duplicate module IDs - Each module ID must be unique in the flow\n\n## Data Flow Between Steps\n\n- `flow_input.property` - Access flow input parameters\n- `results.step_id` - Access output from a previous step only when that step result is in scope\n- `results.step_id.property` - Access specific property from a previous step output only when that step result is in scope\n- `flow_input.iter.value` - Current iteration value when inside a loop (`forloopflow` or `whileloopflow`)\n- `flow_input.iter.index` - Current loop index when inside a loop (`forloopflow` or `whileloopflow`)\n\n## Loop Structure Rules\n\n- For `whileloopflow`, use module-level `stop_after_if` on the loop module itself when the loop should stop after an iteration result\n- Do NOT put `stop_after_if` inside `value` of a `whileloopflow`\n- `stop_after_all_iters_if` is for checks after the whole loop finishes, not the normal per-iteration break condition\n- When a `whileloopflow` carries state forward between iterations, use `flow_input.iter.value` as the current loop value and provide an explicit first-iteration fallback when needed\n- Use `flow_input.iter.index` only when the loop logic is truly based on the iteration index, not as a replacement for the current loop value\n- If the user asks for a final scalar/object after a loop, add a normal step after the loop that extracts the final value from the loop result instead of returning the whole loop result array\n\nCorrect `whileloopflow` shape:\n\n```yaml\n- id: loop_until_done\n stop_after_if:\n expr: result.done === true\n skip_if_stopped: false\n value:\n type: whileloopflow\n skip_failures: false\n modules:\n - id: advance_state\n value:\n type: rawscript\n input_transforms:\n state:\n type: javascript\n expr: flow_input.iter && flow_input.iter.value !== undefined ? flow_input.iter.value : flow_input.initial_state\n- id: return_final_state\n value:\n type: rawscript\n input_transforms:\n final_state:\n type: javascript\n expr: results.loop_until_done[results.loop_until_done.length - 1]\n```\n\nIncorrect `whileloopflow` patterns:\n\n```yaml\n- id: loop_until_done\n value:\n type: whileloopflow\n stop_after_if:\n expr: result.done === true\n```\n\n```yaml\ninput_transforms:\n state:\n type: javascript\n expr: flow_input.iter.index\n```\n\n```yaml\ninput_transforms:\n final_state:\n type: javascript\n expr: results.loop_until_done\n```\n\n## Approval / Suspend Structure\n\n- `suspend` belongs on the flow module object itself, as a sibling of `id` and `value`\n- Never put `suspend` inside `value`\n\nCorrect shape:\n\n```yaml\n- id: request_approval\n suspend:\n required_events: 1\n resume_form:\n schema:\n type: object\n properties:\n comment:\n type: string\n required: [comment]\n value:\n type: identity\n```\n\nIncorrect shape:\n\n```yaml\n- id: request_approval\n value:\n type: rawscript\n suspend:\n required_events: 1\n```\n\n## Branch Result Scope Rules\n\n- Inside a branch, you may reference earlier outer steps and earlier steps in the same branch\n- Outside a `branchone`, do NOT reference ids of steps that only exist inside its branches or default branch. Use `results.` instead\n- Outside a `branchall`, do NOT reference ids of steps inside its branches. Use `results.` instead\n- If downstream steps need a stable shape after a branch, make each branch return the same fields\n- When needed, add a normalization step immediately after the branch and consume `results.` there\n\nCorrect after `branchone`:\n\n```yaml\n- id: route_order\n value:\n type: branchone\n ...\n- id: send_confirmation\n value:\n input_transforms:\n routed:\n type: javascript\n expr: results.route_order\n```\n\nIncorrect after `branchone`:\n\n```yaml\nexpr: results.create_shipment\nexpr: results.create_backorder\n```\n\nCorrect after `branchall`:\n\n```yaml\n- id: enrich_parallel\n value:\n type: branchall\n parallel: true\n ...\n- id: combine_data\n value:\n input_transforms:\n enrichments:\n type: javascript\n expr: results.enrich_parallel\n```\n\n## Input Transforms\n\nEvery rawscript module needs `input_transforms` to map function parameters to values:\n\nStatic transform (fixed value):\n{\"param_name\": {\"type\": \"static\", \"value\": \"fixed_string\"}}\n\nJavaScript transform (dynamic expression):\n{\"param_name\": {\"type\": \"javascript\", \"expr\": \"results.previous_step.data\"}}\n\n## Resource References\n\n- For flow inputs: Use type `\"object\"` with format `\"resource-{type}\"` (e.g., `\"resource-postgresql\"`)\n- For step inputs: Use static value `\"$res:path/to/resource\"`\n\n## Final Structural Self-Check\n\nBefore finalizing a flow, verify:\n\n- any preprocessor is in `value.preprocessor_module`\n- any failure handler is in `value.failure_module`\n- any approval step has module-level `suspend`\n- no downstream step references inner branch step ids from outside the branch\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations.\n\nTo accept an S3 object as flow input:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"file\": {\n \"type\": \"object\",\n \"format\": \"resource-s3_object\",\n \"description\": \"File to process\"\n }\n }\n}\n```\n\n## Using Resources in Flows\n\nOn Windmill, credentials and configuration are stored in resources. Resource types define the format of the resource.\n\n### As Flow Input\n\nIn the flow schema, set the property type to `\"object\"` with format `\"resource-{type}\"`:\n\n```json\n{\n \"type\": \"object\",\n \"properties\": {\n \"database\": {\n \"type\": \"object\",\n \"format\": \"resource-postgresql\",\n \"description\": \"Database connection\"\n }\n }\n}\n```\n\n### As Step Input (Static Reference)\n\nReference a specific resource using `$res:` prefix:\n\n```json\n{\n \"database\": {\n \"type\": \"static\",\n \"value\": \"$res:f/folder/my_database\"\n }\n}\n```\n"; -export declare const RESOURCES_BASE = "# Windmill Resources\n\nResources store credentials and configuration for external services.\n\n## File Format\n\nResource files use the pattern: `{path}.resource.json`\n\nExample: `f/databases/postgres_prod.resource.json`\n\n## Resource Structure\n\n```json\n{\n \"value\": {\n \"host\": \"db.example.com\",\n \"port\": 5432,\n \"user\": \"admin\",\n \"password\": \"$var:g/all/db_password\",\n \"dbname\": \"production\"\n },\n \"description\": \"Production PostgreSQL database\",\n \"resource_type\": \"postgresql\"\n}\n```\n\n## Required Fields\n\n- `value` - Object containing the resource configuration\n- `resource_type` - Name of the resource type (e.g., \"postgresql\", \"slack\")\n\n## Variable References\n\nReference variables in resource values:\n\n```json\n{\n \"value\": {\n \"api_key\": \"$var:g/all/api_key\",\n \"secret\": \"$var:u/admin/secret\"\n }\n}\n```\n\n**Reference formats:**\n- `$var:g/all/name` - Global variable\n- `$var:u/username/name` - User variable\n- `$var:f/folder/name` - Folder variable\n\n## Resource References\n\nReference other resources:\n\n```json\n{\n \"value\": {\n \"database\": \"$res:f/databases/postgres\"\n }\n}\n```\n\n## Common Resource Types\n\n### PostgreSQL\n```json\n{\n \"resource_type\": \"postgresql\",\n \"value\": {\n \"host\": \"localhost\",\n \"port\": 5432,\n \"user\": \"postgres\",\n \"password\": \"$var:g/all/pg_password\",\n \"dbname\": \"windmill\",\n \"sslmode\": \"prefer\"\n }\n}\n```\n\n### MySQL\n```json\n{\n \"resource_type\": \"mysql\",\n \"value\": {\n \"host\": \"localhost\",\n \"port\": 3306,\n \"user\": \"root\",\n \"password\": \"$var:g/all/mysql_password\",\n \"database\": \"myapp\"\n }\n}\n```\n\n### Slack\n```json\n{\n \"resource_type\": \"slack\",\n \"value\": {\n \"token\": \"$var:g/all/slack_token\"\n }\n}\n```\n\n### AWS S3\n```json\n{\n \"resource_type\": \"s3\",\n \"value\": {\n \"bucket\": \"my-bucket\",\n \"region\": \"us-east-1\",\n \"accessKeyId\": \"$var:g/all/aws_access_key\",\n \"secretAccessKey\": \"$var:g/all/aws_secret_key\"\n }\n}\n```\n\n### HTTP/API\n```json\n{\n \"resource_type\": \"http\",\n \"value\": {\n \"baseUrl\": \"https://api.example.com\",\n \"headers\": {\n \"Authorization\": \"Bearer $var:g/all/api_token\"\n }\n }\n}\n```\n\n### Kafka\n```json\n{\n \"resource_type\": \"kafka\",\n \"value\": {\n \"brokers\": \"broker1:9092,broker2:9092\",\n \"sasl_mechanism\": \"PLAIN\",\n \"security_protocol\": \"SASL_SSL\",\n \"username\": \"$var:g/all/kafka_user\",\n \"password\": \"$var:g/all/kafka_password\"\n }\n}\n```\n\n### NATS\n```json\n{\n \"resource_type\": \"nats\",\n \"value\": {\n \"servers\": [\"nats://localhost:4222\"],\n \"user\": \"$var:g/all/nats_user\",\n \"password\": \"$var:g/all/nats_password\"\n }\n}\n```\n\n### MQTT\n```json\n{\n \"resource_type\": \"mqtt\",\n \"value\": {\n \"host\": \"mqtt.example.com\",\n \"port\": 8883,\n \"username\": \"$var:g/all/mqtt_user\",\n \"password\": \"$var:g/all/mqtt_password\",\n \"tls\": true\n }\n}\n```\n\n## Custom Resource Types\n\nCreate custom resource types with JSON Schema:\n\n```json\n{\n \"name\": \"custom_api\",\n \"schema\": {\n \"type\": \"object\",\n \"properties\": {\n \"base_url\": {\"type\": \"string\", \"format\": \"uri\"},\n \"api_key\": {\"type\": \"string\"},\n \"timeout\": {\"type\": \"integer\", \"default\": 30}\n },\n \"required\": [\"base_url\", \"api_key\"]\n },\n \"description\": \"Custom API connection\"\n}\n```\n\nSave as: `custom_api.resource-type.json`\n\n## OAuth Resources\n\nOAuth resources are managed through the Windmill UI and marked:\n\n```json\n{\n \"is_oauth\": true,\n \"account\": 123\n}\n```\n\nOAuth tokens are automatically refreshed by Windmill.\n\n## Using Resources in Scripts\n\n### TypeScript (Bun/Deno)\n```typescript\nexport async function main(db: RT.Postgresql) {\n // db contains the resource values\n const { host, port, user, password, dbname } = db;\n}\n```\n\n### Python\n```python\nclass postgresql(TypedDict):\n host: str\n port: int\n user: str\n password: str\n dbname: str\n\ndef main(db: postgresql):\n # db contains the resource values\n pass\n```\n\n## CLI Commands\n\n```bash\n# List resources\nwmill resource list\n\n# List resource types with schemas\nwmill resource-type list --schema\n\n# Get specific resource type schema\nwmill resource-type get postgresql\n\n# Push resources (tell the user to run this, do NOT run it yourself)\nwmill sync push\n```\n"; -export declare const RAW_APP_BASE = "# Windmill Raw Apps\n\nRaw apps let you build custom frontends with React, Svelte, or Vue that connect to Windmill backend runnables and datatables.\n\n## App shape\n\nA raw app has three logical parts:\n\n- **Frontend** \u2014 bundled with esbuild from `index.tsx` as the entrypoint. Files include the entrypoint, components (`App.tsx`), styles, etc.\n- **Backend runnables** \u2014 server-side scripts the frontend calls, each addressed by a unique key.\n- **Data** \u2014 optional whitelisted datatables (managed PostgreSQL) that the backend runnables can query. The frontend never queries the database directly; backend runnables are the only bridge.\n\n## Frontend\n\n### Entrypoint\n\n`index.tsx` is the bundling entrypoint. It typically renders a top-level `App` component. The bundler is esbuild.\n\n### Generated bindings (`wmill.d.ts` / `wmill.ts`)\n\nThe frontend imports a generated module that mirrors the backend runnables. **Never write to it directly** \u2014 it gets regenerated whenever backend runnables change. Modifying it by hand will be overwritten.\n\n### Calling backend runnables\n\nImport the generated bindings and call the runnable like a function:\n\n```typescript\nimport { backend } from './wmill';\n\n// Call a backend runnable\nconst user = await backend.get_user({ user_id: '123' });\n```\n\nThe frontend cannot reach datatables, workspace items, or external services on its own \u2014 it goes through `backend.(args)` for everything server-side.\n\n## Backend runnables\n\nEach runnable has a unique key (used to call it from the frontend) and one of four types:\n\n| Type | What it is |\n|---|---|\n| `inline` | Custom code stored on the app itself. Most common for app-specific logic. |\n| `script` | Reference to an existing workspace script by path. |\n| `flow` | Reference to an existing workspace flow by path. |\n| `hubscript` | Reference to a hub script by path. |\n\n### Inline runnables\n\nInline runnables carry their own source code. For file-based raw apps, the runnable language is determined by the backend file extension. The script must expose a `main` function as its entrypoint.\n\n**TypeScript example** (`backend/get_user.ts`):\n\n```typescript\nimport * as wmill from 'windmill-client';\n\nexport async function main(user_id: string) {\n const sql = wmill.datatable();\n const user = await sql`SELECT * FROM users WHERE id = ${user_id}`.fetchOne();\n return user;\n}\n```\n\n**Python example** (`backend/get_user.py`):\n\n```python\nimport wmill\n\ndef main(user_id: str):\n db = wmill.datatable()\n user = db.query('SELECT * FROM users WHERE id = $1', user_id).fetch_one()\n return user\n```\n\n### Path runnables (script / flow / hubscript)\n\nWhen `type` is `script`, `flow`, or `hubscript`, the runnable just stores a `path` to an existing workspace or hub item \u2014 no inline code. The referenced item's input/output schema becomes the runnable's surface.\n\n### Static inputs\n\n`staticInputs` is an optional `Record` for arguments not overridable from the frontend. Useful with path runnables to pre-fill some args while leaving the rest to the frontend caller.\n\n## Data Tables\n\nData tables are PostgreSQL databases managed by Windmill. Backend runnables query them via the `wmill` client; the frontend never queries them directly.\n\n### Critical rules\n\n1. **Whitelisted tables only**: a runnable can only query tables listed in the app's `data.tables` config. Tables not in this list are not accessible.\n2. **Add tables before using**: queries against unlisted tables fail at runtime. When you introduce a new table, register it in `data.tables` first.\n3. **Use the configured datatable/schema**: the app's `data` config sets the default datatable and schema; reference them consistently across runnables.\n\n### Querying in TypeScript (Bun/Deno)\n\n```typescript\nimport * as wmill from 'windmill-client';\n\nexport async function main(user_id: string) {\n const sql = wmill.datatable(); // Or: wmill.datatable('other_datatable')\n\n // Parameterized queries (safe from SQL injection)\n const user = await sql`SELECT * FROM users WHERE id = ${user_id}`.fetchOne();\n const users = await sql`SELECT * FROM users WHERE active = ${true}`.fetch();\n\n // Insert/Update\n await sql`INSERT INTO users (name, email) VALUES (${name}, ${email})`;\n await sql`UPDATE users SET name = ${newName} WHERE id = ${user_id}`;\n\n return user;\n}\n```\n\n### Querying in Python\n\n```python\nimport wmill\n\ndef main(user_id: str):\n db = wmill.datatable() # Or: wmill.datatable('other_datatable')\n\n # Use $1, $2, etc. for parameters\n user = db.query('SELECT * FROM users WHERE id = $1', user_id).fetch_one()\n users = db.query('SELECT * FROM users WHERE active = $1', True).fetch()\n\n # Insert/Update\n db.query('INSERT INTO users (name, email) VALUES ($1, $2)', name, email)\n db.query('UPDATE users SET name = $1 WHERE id = $2', new_name, user_id)\n\n return user\n```\n\n## Best Practices\n\n1. **Check existing tables** before creating new ones \u2014 reuse beats schema growth.\n2. **Use parameterized queries** \u2014 never concatenate user input into SQL.\n3. **Keep runnables focused** \u2014 one function per runnable; small surface area.\n4. **Use descriptive keys** \u2014 `get_user`, not `a`.\n5. **Always whitelist tables** \u2014 adding a runnable that queries a new table requires the table to be in `data.tables` first.\n"; -export declare const WORKFLOW_AS_CODE_BASE = "# Windmill Workflow-as-Code Writing Guide\n\n## Scope\n\nUse this guide when writing or modifying Windmill Workflow-as-Code (WAC) scripts.\nWAC is authored as a Windmill script and deployed with the normal script workflow. It is not an OpenFlow YAML flow.\n\nSupported WAC authoring targets:\n- Bun TypeScript scripts that import from `windmill-client`\n- Python 3 scripts that import from `wmill`\n\n## File Shape\n\nBun TypeScript:\n\n```typescript\nimport {\n task,\n taskScript,\n taskFlow,\n step,\n sleep,\n waitForApproval,\n getResumeUrls,\n parallel,\n workflow,\n} from \"windmill-client\";\n\nconst process = task(async (x: string): Promise => {\n return `processed: ${x}`;\n});\n\nexport const main = workflow(async (x: string) => {\n const result = await process(x);\n return { result };\n});\n```\n\nPython:\n\n```python\nfrom wmill import task, task_script, task_flow, step, sleep, wait_for_approval, get_resume_urls, parallel, workflow\n\n@task()\nasync def process(x: str) -> str:\n return f\"processed: {x}\"\n\n@workflow\nasync def main(x: str):\n result = await process(x)\n return {\"result\": result}\n```\n\nRules:\n- Do not call `main`.\n- Bun TypeScript should export the workflow entrypoint, preferably `export const main = workflow(async (...) => { ... })`.\n- Python must use `@workflow` on an async top-level function, usually `main`.\n- Define task functions and `taskScript`/`task_script` or `taskFlow`/`task_flow` assignments at module top level with stable names.\n- Use the exact SDK names. Do not alias `workflow`, `task`, `taskScript`, `taskFlow`, `step`, `sleep`, `waitForApproval`, `task_script`, `task_flow`, or `wait_for_approval`; the WAC parser recognizes these names directly.\n\n## Checkpoint And Replay Model\n\nThe parent workflow may rerun from the top after any suspension, retry, approval, or child task completion. Completed durable steps are replayed from the checkpoint.\n\nPut every side effect or non-deterministic value behind a durable WAC boundary:\n- Use `task()` / `@task()` for substantial work that should run as its own child job.\n- Use `taskScript()` / `task_script()` for an existing script or a relative module file.\n- Use `taskFlow()` / `task_flow()` for an existing Windmill flow.\n- Use `step(name, fn)` for lightweight inline work whose result must be checkpointed.\n- Use `sleep(seconds)` for server-side sleeps that do not hold a worker.\n- Use `waitForApproval()` / `wait_for_approval()` for external approval suspension.\n\nNever put API calls, database writes, notifications, random values, timestamps, or irreversible changes directly in the top-level workflow body. The workflow body can be rerun. Put those operations in a task or in `step()`.\n\nBranching on task or step results is safe because those results are checkpointed. Branching on current time, random data, environment reads, or external state is unsafe unless the value is first captured with `step()`.\n\n## Tasks\n\nUse `task()` / `@task()` for inline functions that become workflow steps:\n\n```typescript\nconst enrich = task(async (customerId: string) => {\n return await fetchCustomer(customerId);\n});\n```\n\n```python\n@task(timeout=600, tag=\"etl\")\nasync def enrich(customer_id: str):\n return await fetch_customer(customer_id)\n```\n\nIn TypeScript, prefer assigning each task to a named top-level const. In Python, prefer top-level async functions decorated with `@task()` or `@task`.\n\nFor existing scripts:\n\n```typescript\nconst helper = taskScript(\"./helper.ts\");\nconst existing = taskScript(\"f/data/extract\", { timeout: 600 });\nconst value = await helper({ input: x });\n```\n\n```python\nhelper = task_script(\"./helper.py\")\nexisting = task_script(\"f/data/extract\", timeout=600)\nvalue = await helper(input=x)\n```\n\nFor existing flows:\n\n```typescript\nconst pipeline = taskFlow(\"f/etl/pipeline\");\nconst output = await pipeline({ input: data });\n```\n\n```python\npipeline = task_flow(\"f/etl/pipeline\")\noutput = await pipeline(input=data)\n```\n\n## Inline Steps\n\nUse `step()` for lightweight inline values that must not change during replay:\n\n```typescript\nconst urls = await step(\"get_urls\", () => getResumeUrls());\nconst startedAt = await step(\"started_at\", () => new Date().toISOString());\n```\n\n```python\nurls = await step(\"get_urls\", lambda: get_resume_urls())\n```\n\nUse stable, descriptive step names. Do not generate step names dynamically.\n\n## Parallelism\n\nTo run independent work in parallel, start task promises/coroutines before awaiting them together:\n\n```typescript\nconst [a, b] = await Promise.all([process(\"a\"), process(\"b\")]);\nconst many = await parallel(items, process, { concurrency: 5 });\n```\n\n```python\nimport asyncio\n\na, b = await asyncio.gather(process(\"a\"), process(\"b\"))\nmany = await parallel(items, process, concurrency=5)\n```\n\nOnly parallelize independent steps. Do not read the result of a task before it is awaited.\n\n## Approvals\n\nGenerate resume URLs inside `step()` before sending them:\n\n```typescript\nconst urls = await step(\"get_urls\", () => getResumeUrls());\nawait step(\"notify\", () => sendApprovalEmail(urls.approvalPage));\nconst approval = await waitForApproval({ timeout: 3600 });\n```\n\n```python\nurls = await step(\"get_urls\", lambda: get_resume_urls())\nawait step(\"notify\", lambda: send_approval_email(urls[\"approvalPage\"]))\napproval = await wait_for_approval(timeout=3600)\n```\n\n`selfApproval: false` and `self_approval=False` are Enterprise-only approval behavior. Do not use them unless the user asks for that behavior.\n\n## Error Handling\n\nLet task errors fail the workflow unless the user asks for recovery logic.\n\nPython: `except Exception` is safe around WAC calls because internal suspension inherits from `BaseException`. Avoid bare `except:` in workflow code. If the user asks for recovery logic around failed child work, catch `TaskError` from `wmill` for task failures.\n\nTypeScript: avoid broad `try/catch` around WAC SDK calls. The SDK uses an internal suspension error during initial dispatch; catching it can break workflow suspension. If a broad catch is unavoidable, rethrow internal suspension errors before handling business errors.\n"; -export declare const FLOW_CHAT_SPECIAL_MODULES = "## Special Modules\n\n- Use `set_preprocessor_module` to add, replace, or remove the top-level `value.preprocessor_module`\n- Use `set_failure_module` to add, replace, or remove the top-level `value.failure_module`\n- Use `set_flow_json` only when you are replacing the whole flow, including normal modules and optional special modules\n\n**Example - Update only the special modules:**\n```javascript\nset_preprocessor_module({\n module: JSON.stringify({\n id: \"preprocessor\",\n value: {\n type: \"rawscript\",\n language: \"bun\",\n content: \"export async function preprocessor(payload: string) { const trimmed = payload.trim(); if (!trimmed) { throw new Error('payload must not be empty'); } return { payload: trimmed }; }\",\n input_transforms: {\n payload: { type: \"javascript\", expr: \"flow_input.payload\" }\n }\n }\n })\n})\n\nset_failure_module({\n module: JSON.stringify({\n id: \"failure\",\n value: {\n type: \"rawscript\",\n language: \"bun\",\n content: \"export async function main(message: string, name: string, step_id: string) { return { message, name, step_id }; }\",\n input_transforms: {\n message: { type: \"javascript\", expr: \"error.message\" },\n name: { type: \"javascript\", expr: \"error.name\" },\n step_id: { type: \"javascript\", expr: \"error.step_id\" }\n }\n }\n })\n})\n```\n"; -export declare const SDK_TYPESCRIPT = "# TypeScript SDK (windmill-client)\n\nImport: import * as wmill from 'windmill-client'\n\nworkerHasInternalServer(): boolean\n\n/**\n * Initialize the Windmill client with authentication token and base URL\n * @param token - Authentication token (defaults to WM_TOKEN env variable)\n * @param baseUrl - API base URL (defaults to BASE_INTERNAL_URL or BASE_URL env variable)\n */\nsetClient(token?: string, baseUrl?: string): void\n\n/**\n * Create a client configuration from env variables\n * @returns client configuration\n */\ngetWorkspace(): string\n\n/**\n * Get a resource value by path\n * @param path path of the resource, default to internal state path\n * @param undefinedIfEmpty if the resource does not exist, return undefined instead of throwing an error\n * @returns resource value\n */\nasync getResource(path?: string, undefinedIfEmpty?: boolean): Promise\n\n/**\n * Get the true root job id\n * @param jobId job id to get the root job id from (default to current job)\n * @returns root job id\n */\nasync getRootJobId(jobId?: string): Promise\n\n/**\n * @deprecated Use runScriptByPath or runScriptByHash instead\n */\nasync runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Run a script synchronously by its path and wait for the result\n * @param path - Script path in Windmill\n * @param args - Arguments to pass to the script\n * @param verbose - Enable verbose logging\n * @returns Script execution result\n */\nasync runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Run a script synchronously by its hash and wait for the result\n * @param hash_ - Script hash in Windmill\n * @param args - Arguments to pass to the script\n * @param verbose - Enable verbose logging\n * @returns Script execution result\n */\nasync runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Append a text to the result stream\n * @param text text to append to the result stream\n */\nappendToResultStream(text: string): void\n\n/**\n * Stream to the result stream\n * @param stream stream to stream to the result stream\n */\nasync streamResult(stream: AsyncIterable): Promise\n\n/**\n * Run a flow synchronously by its path and wait for the result\n * @param path - Flow path in Windmill\n * @param args - Arguments to pass to the flow\n * @param verbose - Enable verbose logging\n * @returns Flow execution result\n */\nasync runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise\n\n/**\n * Wait for a job to complete and return its result\n * @param jobId - ID of the job to wait for\n * @param verbose - Enable verbose logging\n * @returns Job result when completed\n */\nasync waitJob(jobId: string, verbose: boolean = false): Promise\n\n/**\n * Get the result of a completed job\n * @param jobId - ID of the completed job\n * @returns Job result\n */\nasync getResult(jobId: string): Promise\n\n/**\n * Get the result of a job if completed, or its current status\n * @param jobId - ID of the job\n * @returns Object with started, completed, success, and result properties\n */\nasync getResultMaybe(jobId: string): Promise\n\n/**\n * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead\n */\nasync runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise\n\n/**\n * Run a script asynchronously by its path\n * @param path - Script path in Windmill\n * @param args - Arguments to pass to the script\n * @param scheduledInSeconds - Schedule execution for a future time (in seconds)\n * @returns Job ID of the created job\n */\nasync runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise\n\n/**\n * Run a script asynchronously by its hash\n * @param hash_ - Script hash in Windmill\n * @param args - Arguments to pass to the script\n * @param scheduledInSeconds - Schedule execution for a future time (in seconds)\n * @returns Job ID of the created job\n */\nasync runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise\n\n/**\n * Run a flow asynchronously by its path\n * @param path - Flow path in Windmill\n * @param args - Arguments to pass to the flow\n * @param scheduledInSeconds - Schedule execution for a future time (in seconds)\n * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job)\n * @returns Job ID of the created job\n */\nasync runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise\n\n/**\n * Resolve a resource value in case the default value was picked because the input payload was undefined\n * @param obj resource value or path of the resource under the format `$res:path`\n * @returns resource value\n */\nasync resolveDefaultResource(obj: any): Promise\n\n/**\n * Get the state file path from environment variables\n * @returns State path string\n */\ngetStatePath(): string\n\n/**\n * Set a resource value by path\n * @param path path of the resource to set, default to state path\n * @param value new value of the resource to set\n * @param initializeToTypeIfNotExist if the resource does not exist, initialize it with this type\n */\nasync setResource(value: any, path?: string, initializeToTypeIfNotExist?: string): Promise\n\n/**\n * Set the state\n * @param state state to set\n * @deprecated use setState instead\n */\nasync setInternalState(state: any): Promise\n\n/**\n * Set the state\n * @param state state to set\n * @param path Optional state resource path override. Defaults to `getStatePath()`.\n */\nasync setState(state: any, path?: string): Promise\n\n/**\n * Set the progress\n * Progress cannot go back and limited to 0% to 99% range\n * @param percent Progress to set in %\n * @param jobId? Job to set progress for\n */\nasync setProgress(percent: number, jobId?: any): Promise\n\n/**\n * Get the progress\n * @param jobId? Job to get progress from\n * @returns Optional clamped between 0 and 100 progress value\n */\nasync getProgress(jobId?: any): Promise\n\n/**\n * Set a flow user state\n * @param key key of the state\n * @param value value of the state\n */\nasync setFlowUserState(key: string, value: any, errorIfNotPossible?: boolean): Promise\n\n/**\n * Get a flow user state\n * @param path path of the variable\n */\nasync getFlowUserState(key: string, errorIfNotPossible?: boolean): Promise\n\n/**\n * Get the internal state\n * @deprecated use getState instead\n */\nasync getInternalState(): Promise\n\n/**\n * Get the state shared across executions\n * @param path Optional state resource path override. Defaults to `getStatePath()`.\n */\nasync getState(path?: string): Promise\n\n/**\n * Get a variable by path\n * @param path path of the variable\n * @returns variable value\n */\nasync getVariable(path: string): Promise\n\n/**\n * Set a variable by path, create if not exist\n * @param path path of the variable\n * @param value value of the variable\n * @param isSecretIfNotExist if the variable does not exist, create it as secret or not (default: false)\n * @param descriptionIfNotExist if the variable does not exist, create it with this description (default: \"\")\n */\nasync setVariable(path: string, value: string, isSecretIfNotExist?: boolean, descriptionIfNotExist?: string): Promise\n\n/**\n * Build a PostgreSQL connection URL from a database resource\n * @param path - Path to the database resource\n * @returns PostgreSQL connection URL string\n */\nasync databaseUrlFromResource(path: string): Promise\n\nasync polarsConnectionSettings(s3_resource_path: string | undefined): Promise\n\nasync duckdbConnectionSettings(s3_resource_path: string | undefined): Promise\n\n/**\n * Get S3 client settings from a resource or workspace default\n * @param s3_resource_path - Path to S3 resource (uses workspace default if undefined)\n * @returns S3 client configuration settings\n */\nasync denoS3LightClientSettings(s3_resource_path: string | undefined): Promise\n\n/**\n * Load the content of a file stored in S3. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.\n * \n * ```typescript\n * let fileContent = await wmill.loadS3FileContent(inputFile)\n * // if the file is a raw text file, it can be decoded and printed directly:\n * const text = new TextDecoder().decode(fileContentStream)\n * console.log(text);\n * ```\n */\nasync loadS3File(s3object: S3Object, s3ResourcePath: string | undefined = undefined): Promise\n\n/**\n * Load the content of a file stored in S3 as a stream. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.\n * \n * ```typescript\n * let fileContentBlob = await wmill.loadS3FileStream(inputFile)\n * // if the content is plain text, the blob can be read directly:\n * console.log(await fileContentBlob.text());\n * ```\n */\nasync loadS3FileStream(s3object: S3Object, s3ResourcePath: string | undefined = undefined): Promise\n\n/**\n * Persist a file to the S3 bucket. If the s3ResourcePath is undefined, it will default to the workspace S3 resource.\n * \n * ```typescript\n * const s3object = await writeS3File(s3Object, \"Hello Windmill!\")\n * const fileContentAsUtf8Str = (await s3object.toArray()).toString('utf-8')\n * console.log(fileContentAsUtf8Str)\n * ```\n */\nasync writeS3File(s3object: S3Object | undefined, fileContent: string | Blob, s3ResourcePath: string | undefined = undefined, contentType: string | undefined = undefined, contentDisposition: string | undefined = undefined): Promise\n\n/**\n * Sign S3 objects to be used by anonymous users in public apps\n * @param s3objects s3 objects to sign\n * @returns signed s3 objects\n */\nasync signS3Objects(s3objects: S3Object[]): Promise\n\n/**\n * Sign S3 object to be used by anonymous users in public apps\n * @param s3object s3 object to sign\n * @returns signed s3 object\n */\nasync signS3Object(s3object: S3Object): Promise\n\n/**\n * Generate a presigned public URL for an array of S3 objects.\n * If an S3 object is not signed yet, it will be signed first.\n * @param s3Objects s3 objects to sign\n * @returns list of signed public URLs\n */\nasync getPresignedS3PublicUrls(s3Objects: S3Object[], { baseUrl }: { baseUrl?: string } = {}): Promise\n\n/**\n * Generate a presigned public URL for an S3 object. If the S3 object is not signed yet, it will be signed first.\n * @param s3Object s3 object to sign\n * @returns signed public URL\n */\nasync getPresignedS3PublicUrl(s3Objects: S3Object, { baseUrl }: { baseUrl?: string } = {}): Promise\n\n/**\n * Get URLs needed for resuming a flow after this step\n * @param approver approver name\n * @param flowLevel if true, generate resume URLs for the parent flow instead of the specific step.\n * This allows pre-approvals that can be consumed by any later suspend step in the same flow.\n * @returns approval page UI URL, resume and cancel API URLs for resuming the flow\n */\nasync getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{\n approvalPage: string;\n resume: string;\n cancel: string;\n}>\n\n/**\n * @deprecated use getResumeUrls instead\n */\ngetResumeEndpoints(approver?: string): Promise<{\n approvalPage: string;\n resume: string;\n cancel: string;\n}>\n\n/**\n * Get an OIDC jwt token for auth to external services (e.g: Vault, AWS) (ee only)\n * @param audience audience of the token\n * @param expiresIn Optional number of seconds until the token expires\n * @returns jwt token\n */\nasync getIdToken(audience: string, expiresIn?: number): Promise\n\n/**\n * Convert a base64-encoded string to Uint8Array\n * @param data - Base64-encoded string\n * @returns Decoded Uint8Array\n */\nbase64ToUint8Array(data: string): Uint8Array\n\n/**\n * Convert a Uint8Array to base64-encoded string\n * @param arrayBuffer - Uint8Array to encode\n * @returns Base64-encoded string\n */\nuint8ArrayToBase64(arrayBuffer: Uint8Array): string\n\n/**\n * Get email from workspace username\n * This method is particularly useful for apps that require the email address of the viewer.\n * Indeed, in the viewer context, WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app.\n * @param username\n * @returns email address\n */\nasync usernameToEmail(username: string): Promise\n\n/**\n * Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields.\n * \n * **[Enterprise Edition Only]** To include form fields in the Slack approval request, go to **Advanced -> Suspend -> Form**\n * and define a form. Learn more at [Windmill Documentation](https://www.windmill.dev/docs/flows/flow_approval#form).\n * \n * @param {Object} options - The configuration options for the Slack approval request.\n * @param {string} options.slackResourcePath - The path to the Slack resource in Windmill.\n * @param {string} options.channelId - The Slack channel ID where the approval request will be sent.\n * @param {string} [options.message] - Optional custom message to include in the Slack approval request.\n * @param {string} [options.approver] - Optional user ID or name of the approver for the request.\n * @param {DefaultArgs} [options.defaultArgsJson] - Optional object defining or overriding the default arguments to a form field.\n * @param {Enums} [options.dynamicEnumsJson] - Optional object overriding the enum default values of an enum form field.\n * @param {string} [options.resumeButtonText] - Optional text for the resume button.\n * @param {string} [options.cancelButtonText] - Optional text for the cancel button.\n * \n * @returns {Promise} Resolves when the Slack approval request is successfully sent.\n * \n * @throws {Error} If the function is not called within a flow or flow preview.\n * @throws {Error} If the `JobService.getSlackApprovalPayload` call fails.\n * \n * **Usage Example:**\n * ```typescript\n * await requestInteractiveSlackApproval({\n * slackResourcePath: \"/u/alex/my_slack_resource\",\n * channelId: \"admins-slack-channel\",\n * message: \"Please approve this request\",\n * approver: \"approver123\",\n * defaultArgsJson: { key1: \"value1\", key2: 42 },\n * dynamicEnumsJson: { foo: [\"choice1\", \"choice2\"], bar: [\"optionA\", \"optionB\"] },\n * resumeButtonText: \"Resume\",\n * cancelButtonText: \"Cancel\",\n * });\n * ```\n * \n * **Note:** This function requires execution within a Windmill flow or flow preview.\n */\nasync requestInteractiveSlackApproval({ slackResourcePath, channelId, message, approver, defaultArgsJson, dynamicEnumsJson, resumeButtonText, cancelButtonText, }: SlackApprovalOptions): Promise\n\n/**\n * Sends an interactive approval request via Teams, allowing optional customization of the message, approver, and form fields.\n * \n * **[Enterprise Edition Only]** To include form fields in the Teams approval request, go to **Advanced -> Suspend -> Form**\n * and define a form. Learn more at [Windmill Documentation](https://www.windmill.dev/docs/flows/flow_approval#form).\n * \n * @param {Object} options - The configuration options for the Teams approval request.\n * @param {string} options.teamName - The Teams team name where the approval request will be sent.\n * @param {string} options.channelName - The Teams channel name where the approval request will be sent.\n * @param {string} [options.message] - Optional custom message to include in the Teams approval request.\n * @param {string} [options.approver] - Optional user ID or name of the approver for the request.\n * @param {DefaultArgs} [options.defaultArgsJson] - Optional object defining or overriding the default arguments to a form field.\n * @param {Enums} [options.dynamicEnumsJson] - Optional object overriding the enum default values of an enum form field.\n * \n * @returns {Promise} Resolves when the Teams approval request is successfully sent.\n * \n * @throws {Error} If the function is not called within a flow or flow preview.\n * @throws {Error} If the `JobService.getTeamsApprovalPayload` call fails.\n * \n * **Usage Example:**\n * ```typescript\n * await requestInteractiveTeamsApproval({\n * teamName: \"admins-teams\",\n * channelName: \"admins-teams-channel\",\n * message: \"Please approve this request\",\n * approver: \"approver123\",\n * defaultArgsJson: { key1: \"value1\", key2: 42 },\n * dynamicEnumsJson: { foo: [\"choice1\", \"choice2\"], bar: [\"optionA\", \"optionB\"] },\n * });\n * ```\n * \n * **Note:** This function requires execution within a Windmill flow or flow preview.\n */\nasync requestInteractiveTeamsApproval({ teamName, channelName, message, approver, defaultArgsJson, dynamicEnumsJson, }: TeamsApprovalOptions): Promise\n\n/**\n * Parse an S3 object from URI string or record format\n * @param s3Object - S3 object as URI string (s3://storage/key) or record\n * @returns S3 object record with storage and s3 key\n */\nparseS3Object(s3Object: S3Object): S3ObjectRecord\n\nsetWorkflowCtx(ctx: WorkflowCtx | null): void\n\nasync sleep(seconds: number): Promise\n\nasync step(name: string, fn: () => T | Promise): Promise\n\n/**\n * Create a task that dispatches to a separate Windmill script.\n * \n * @example\n * const extract = taskScript(\"f/data/extract\");\n * // inside workflow: await extract({ url: \"https://...\" })\n */\ntaskScript(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Create a task that dispatches to a separate Windmill flow.\n * \n * @example\n * const pipeline = taskFlow(\"f/etl/pipeline\");\n * // inside workflow: await pipeline({ input: data })\n */\ntaskFlow(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Mark an async function as a workflow-as-code entry point.\n * \n * The function must be **deterministic**: given the same inputs it must call\n * tasks in the same order on every replay. Branching on task results is fine\n * (results are replayed from checkpoint), but branching on external state\n * (current time, random values, external API calls) must use `step()` to\n * checkpoint the value so replays see the same result.\n */\nworkflow(fn: (...args: any[]) => Promise): void\n\n/**\n * Suspend the workflow and wait for an external approval.\n * \n * Use `getResumeUrls()` (wrapped in `step()`) to obtain resume/cancel/approvalPage\n * URLs before calling this function.\n * \n * @example\n * const urls = await step(\"urls\", () => getResumeUrls());\n * await step(\"notify\", () => sendEmail(urls.approvalPage));\n * const { value, approver } = await waitForApproval({ timeout: 3600 });\n */\nwaitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }>\n\n/**\n * Process items in parallel with optional concurrency control.\n * \n * Each item is processed by calling `fn(item)`, which should be a task().\n * Items are dispatched in batches of `concurrency` (default: all at once).\n * \n * @example\n * const process = task(async (item: string) => { ... });\n * const results = await parallel(items, process, { concurrency: 5 });\n */\nasync parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: { concurrency?: number },): Promise\n\n/**\n * Commit Kafka offsets for a trigger with auto_commit disabled.\n * @param triggerPath - Path to the Kafka trigger (from event.wm_trigger.trigger_path)\n * @param topic - Kafka topic name (from event.topic)\n * @param partition - Partition number (from event.partition)\n * @param offset - Message offset to commit (from event.offset)\n */\nasync commitKafkaOffsets(triggerPath: string, topic: string, partition: number, offset: number,): Promise\n\n/**\n * Create a SQL template function for PostgreSQL/datatable queries\n * @param name - Database/datatable name (default: \"main\")\n * @returns SQL template function for building parameterized queries\n * @example\n * let sql = wmill.datatable()\n * let name = 'Robin'\n * let age = 21\n * await sql`\n * SELECT * FROM friends\n * WHERE name = ${name} AND age = ${age}::int\n * `.fetch()\n */\ndatatable(name: string = \"main\"): DatatableSqlTemplateFunction\n\n/**\n * Create a SQL template function for DuckDB/ducklake queries\n * @param name - DuckDB database name (default: \"main\")\n * @returns SQL template function for building parameterized queries\n * @example\n * let sql = wmill.ducklake()\n * let name = 'Robin'\n * let age = 21\n * await sql`\n * SELECT * FROM friends\n * WHERE name = ${name} AND age = ${age}\n * `.fetch()\n */\nducklake(name: string = \"main\"): SqlTemplateFunction\n"; -export declare const SDK_PYTHON = "# Python SDK (wmill)\n\nImport: import wmill\n\ndef worker_has_internal_server() -> bool\n\ndef get_mocked_api() -> Optional[dict]\n\n# Get the HTTP client instance.\n# \n# Returns:\n# Configured httpx.Client for API requests\ndef get_client() -> httpx.Client\n\n# Make an HTTP GET request to the Windmill API.\n# \n# Args:\n# endpoint: API endpoint path\n# raise_for_status: Whether to raise an exception on HTTP errors\n# **kwargs: Additional arguments passed to httpx.get\n# \n# Returns:\n# HTTP response object\ndef get(endpoint, raise_for_status = True, **kwargs) -> httpx.Response\n\n# Make an HTTP POST request to the Windmill API.\n# \n# Args:\n# endpoint: API endpoint path\n# raise_for_status: Whether to raise an exception on HTTP errors\n# **kwargs: Additional arguments passed to httpx.post\n# \n# Returns:\n# HTTP response object\ndef post(endpoint, raise_for_status = True, **kwargs) -> httpx.Response\n\n# Create a new authentication token.\n# \n# Args:\n# duration: Token validity duration (default: 1 day)\n# \n# Returns:\n# New authentication token string\ndef create_token(duration = dt.timedelta(days=1)) -> str\n\n# Create a script job and return its job id.\n# \n# .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead.\ndef run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str\n\n# Create a script job by path and return its job id.\ndef run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str\n\n# Create a script job by hash and return its job id.\ndef run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str\n\n# Create a flow job and return its job id.\ndef run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str\n\n# Run script synchronously and return its result.\n# \n# .. deprecated:: Use run_script_by_path or run_script_by_hash instead.\ndef run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any\n\n# Run script by path synchronously and return its result.\ndef run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any\n\n# Run script by hash synchronously and return its result.\ndef run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any\n\n# Run a script on the current worker without creating a job.\n# \n# On agent workers (no internal server), falls back to running a normal\n# preview job and waiting for the result.\ndef run_inline_script_preview(content: str, language: str, args: dict = None) -> Any\n\n# Wait for a job to complete and return its result.\n# \n# Args:\n# job_id: ID of the job to wait for\n# timeout: Maximum time to wait (seconds or timedelta)\n# verbose: Enable verbose logging\n# cleanup: Register cleanup handler to cancel job on exit\n# assert_result_is_not_none: Raise exception if result is None\n# \n# Returns:\n# Job result when completed\n# \n# Raises:\n# TimeoutError: If timeout is reached\n# Exception: If job fails\ndef wait_job(job_id, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False)\n\n# Cancel a specific job by ID.\n# \n# Args:\n# job_id: UUID of the job to cancel\n# reason: Optional reason for cancellation\n# \n# Returns:\n# Response message from the cancel endpoint\ndef cancel_job(job_id: str, reason: str = None) -> str\n\n# Cancel currently running executions of the same script.\ndef cancel_running() -> dict\n\n# Get job details by ID.\n# \n# Args:\n# job_id: UUID of the job\n# \n# Returns:\n# Job details dictionary\ndef get_job(job_id: str) -> dict\n\n# Get the root job ID for a flow hierarchy.\n# \n# Args:\n# job_id: Job ID (defaults to current WM_JOB_ID)\n# \n# Returns:\n# Root job ID\ndef get_root_job_id(job_id: str | None = None) -> dict\n\n# Get an OIDC JWT token for authentication to external services.\n# \n# Args:\n# audience: Token audience (e.g., \"vault\", \"aws\")\n# expires_in: Optional expiration time in seconds\n# \n# Returns:\n# JWT token string\ndef get_id_token(audience: str, expires_in: int | None = None) -> str\n\n# Get the status of a job.\n# \n# Args:\n# job_id: UUID of the job\n# \n# Returns:\n# Job status: \"RUNNING\", \"WAITING\", or \"COMPLETED\"\ndef get_job_status(job_id: str) -> JobStatus\n\n# Get the result of a completed job.\n# \n# Args:\n# job_id: UUID of the completed job\n# assert_result_is_not_none: Raise exception if result is None\n# \n# Returns:\n# Job result\ndef get_result(job_id: str, assert_result_is_not_none: bool = True) -> Any\n\n# Get a variable value by path.\n# \n# Args:\n# path: Variable path in Windmill\n# \n# Returns:\n# Variable value as string\ndef get_variable(path: str) -> str\n\n# Set a variable value by path, creating it if it doesn't exist.\n# \n# Args:\n# path: Variable path in Windmill\n# value: Variable value to set\n# is_secret: Whether the variable should be secret (default: False)\ndef set_variable(path: str, value: str, is_secret: bool = False) -> None\n\n# Get a resource value by path.\n# \n# Args:\n# path: Resource path in Windmill\n# none_if_undefined: Return None instead of raising if not found\n# interpolated: if variables and resources are fully unrolled\n# \n# Returns:\n# Resource value dictionary or None\ndef get_resource(path: str, none_if_undefined: bool = False, interpolated: bool = True) -> dict | None\n\n# Set a resource value by path, creating it if it doesn't exist.\n# \n# Args:\n# value: Resource value to set\n# path: Resource path in Windmill\n# resource_type: Resource type for creation\ndef set_resource(value: Any, path: str, resource_type: str)\n\n# List resources from Windmill workspace.\n# \n# Args:\n# resource_type: Optional resource type to filter by (e.g., \"postgresql\", \"mysql\", \"s3\")\n# page: Optional page number for pagination\n# per_page: Optional number of results per page\n# \n# Returns:\n# List of resource dictionaries\ndef list_resources(resource_type: str = None, page: int = None, per_page: int = None) -> list[dict]\n\n# Set the workflow state.\n# \n# Args:\n# value: State value to set\n# path: Optional state resource path override.\ndef set_state(value: Any, path: str | None = None) -> None\n\n# Get the workflow state.\n# \n# Args:\n# path: Optional state resource path override.\n# \n# Returns:\n# State value or None if not set\ndef get_state(path: str | None = None) -> Any\n\n# Set job progress percentage (0-99).\n# \n# Args:\n# value: Progress percentage\n# job_id: Job ID (defaults to current WM_JOB_ID)\ndef set_progress(value: int, job_id: Optional[str] = None)\n\n# Get job progress percentage.\n# \n# Args:\n# job_id: Job ID (defaults to current WM_JOB_ID)\n# \n# Returns:\n# Progress value (0-100) or None if not set\ndef get_progress(job_id: Optional[str] = None) -> Any\n\n# Set the user state of a flow at a given key\ndef set_flow_user_state(key: str, value: Any) -> None\n\n# Get the user state of a flow at a given key\ndef get_flow_user_state(key: str) -> Any\n\n# Get the Windmill server version.\n# \n# Returns:\n# Version string\ndef version()\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from DuckDB\ndef get_duckdb_connection_settings(s3_resource_path: str = '') -> DuckDbConnectionSettings | None\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from Polars\ndef get_polars_connection_settings(s3_resource_path: str = '') -> PolarsConnectionSettings\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection using boto3\ndef get_boto3_connection_settings(s3_resource_path: str = '') -> Boto3ConnectionSettings\n\n# Load a file from the workspace s3 bucket and returns its content as bytes.\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# my_obj_content = client.load_s3_file(s3_obj)\n# file_content = my_obj_content.decode(\"utf-8\")\n# '''\ndef load_s3_file(s3object: S3Object | str, s3_resource_path: str | None) -> bytes\n\n# Load a file from the workspace s3 bucket and returns the bytes stream.\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# with wmill.load_s3_file_reader(s3object, s3_resource_path) as file_reader:\n# print(file_reader.read())\n# '''\ndef load_s3_file_reader(s3object: S3Object | str, s3_resource_path: str | None) -> BufferedReader\n\n# Write a file to the workspace S3 bucket\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# \n# # for an in memory bytes array:\n# file_content = b'Hello Windmill!'\n# client.write_s3_file(s3_obj, file_content)\n# \n# # for a file:\n# with open(\"my_file.txt\", \"rb\") as my_file:\n# client.write_s3_file(s3_obj, my_file)\n# '''\ndef write_s3_file(s3object: S3Object | str | None, file_content: BufferedReader | bytes, s3_resource_path: str | None, content_type: str | None = None, content_disposition: str | None = None) -> S3Object\n\n# Permanently delete a file from the workspace S3 bucket.\n# \n# '''python\n# from wmill import S3Object\n# \n# s3_obj = S3Object(s3=\"/path/to/my_file.txt\")\n# client.delete_s3_object(s3_obj)\n# '''\ndef delete_s3_object(s3object: S3Object | str, s3_resource_path: str | None = None) -> None\n\n# Sign S3 objects for use by anonymous users in public apps.\n# \n# Args:\n# s3_objects: List of S3 objects to sign\n# \n# Returns:\n# List of signed S3 objects\ndef sign_s3_objects(s3_objects: list[S3Object | str]) -> list[S3Object]\n\n# Sign a single S3 object for use by anonymous users in public apps.\n# \n# Args:\n# s3_object: S3 object to sign\n# \n# Returns:\n# Signed S3 object\ndef sign_s3_object(s3_object: S3Object | str) -> S3Object\n\n# Generate presigned public URLs for an array of S3 objects.\n# If an S3 object is not signed yet, it will be signed first.\n# \n# Args:\n# s3_objects: List of S3 objects to sign\n# base_url: Optional base URL for the presigned URLs (defaults to WM_BASE_URL)\n# \n# Returns:\n# List of signed public URLs\n# \n# Example:\n# >>> s3_objs = [S3Object(s3=\"/path/to/file1.txt\"), S3Object(s3=\"/path/to/file2.txt\")]\n# >>> urls = client.get_presigned_s3_public_urls(s3_objs)\ndef get_presigned_s3_public_urls(s3_objects: list[S3Object | str], base_url: str | None = None) -> list[str]\n\n# Generate a presigned public URL for an S3 object.\n# If the S3 object is not signed yet, it will be signed first.\n# \n# Args:\n# s3_object: S3 object to sign\n# base_url: Optional base URL for the presigned URL (defaults to WM_BASE_URL)\n# \n# Returns:\n# Signed public URL\n# \n# Example:\n# >>> s3_obj = S3Object(s3=\"/path/to/file.txt\")\n# >>> url = client.get_presigned_s3_public_url(s3_obj)\ndef get_presigned_s3_public_url(s3_object: S3Object | str, base_url: str | None = None) -> str\n\n# Get the current user information.\n# \n# Returns:\n# User details dictionary\ndef whoami() -> dict\n\n# Get the current user information (alias for whoami).\n# \n# Returns:\n# User details dictionary\ndef user() -> dict\n\n# Get the state resource path from environment.\n# \n# Returns:\n# State path string\ndef state_path() -> str\n\n# Get the workflow state.\n# \n# Returns:\n# State value or None if not set\ndef state() -> Any\n\n# Set the state in the shared folder using pickle\ndef set_shared_state_pickle(value: Any, path: str = 'state.pickle') -> None\n\n# Get the state in the shared folder using pickle\ndef get_shared_state_pickle(path: str = 'state.pickle') -> Any\n\n# Set the state in the shared folder using pickle\ndef set_shared_state(value: Any, path: str = 'state.json') -> None\n\n# Get the state in the shared folder using pickle\ndef get_shared_state(path: str = 'state.json') -> None\n\n# Get URLs needed for resuming a flow after suspension.\n# \n# Args:\n# approver: Optional approver name\n# flow_level: If True, generate resume URLs for the parent flow instead of the\n# specific step. This allows pre-approvals that can be consumed by any later\n# suspend step in the same flow.\n# \n# Returns:\n# Dictionary with approvalPage, resume, and cancel URLs\ndef get_resume_urls(approver: str = None, flow_level: bool = None) -> dict\n\n# Sends an interactive approval request via Slack, allowing optional customization of the message, approver, and form fields.\n# \n# **[Enterprise Edition Only]** To include form fields in the Slack approval request, use the \"Advanced -> Suspend -> Form\" functionality.\n# Learn more at: https://www.windmill.dev/docs/flows/flow_approval#form\n# \n# :param slack_resource_path: The path to the Slack resource in Windmill.\n# :type slack_resource_path: str\n# :param channel_id: The Slack channel ID where the approval request will be sent.\n# :type channel_id: str\n# :param message: Optional custom message to include in the Slack approval request.\n# :type message: str, optional\n# :param approver: Optional user ID or name of the approver for the request.\n# :type approver: str, optional\n# :param default_args_json: Optional dictionary defining or overriding the default arguments for form fields.\n# :type default_args_json: dict, optional\n# :param dynamic_enums_json: Optional dictionary overriding the enum default values of enum form fields.\n# :type dynamic_enums_json: dict, optional\n# \n# :raises Exception: If the function is not called within a flow or flow preview.\n# :raises Exception: If the required flow job or flow step environment variables are not set.\n# \n# :return: None\n# \n# **Usage Example:**\n# >>> client.request_interactive_slack_approval(\n# ... slack_resource_path=\"/u/alex/my_slack_resource\",\n# ... channel_id=\"admins-slack-channel\",\n# ... message=\"Please approve this request\",\n# ... approver=\"approver123\",\n# ... default_args_json={\"key1\": \"value1\", \"key2\": 42},\n# ... dynamic_enums_json={\"foo\": [\"choice1\", \"choice2\"], \"bar\": [\"optionA\", \"optionB\"]},\n# ... )\n# \n# **Notes:**\n# - This function must be executed within a Windmill flow or flow preview.\n# - The function checks for required environment variables (`WM_FLOW_JOB_ID`, `WM_FLOW_STEP_ID`) to ensure it is run in the appropriate context.\ndef request_interactive_slack_approval(slack_resource_path: str, channel_id: str, message: str = None, approver: str = None, default_args_json: dict = None, dynamic_enums_json: dict = None) -> None\n\n# Get email from workspace username\n# This method is particularly useful for apps that require the email address of the viewer.\n# Indeed, in the viewer context WM_USERNAME is set to the username of the viewer but WM_EMAIL is set to the email of the creator of the app.\ndef username_to_email(username: str) -> str\n\n# Send a message to a Microsoft Teams conversation with conversation_id, where success is used to style the message\ndef send_teams_message(conversation_id: str, text: str, success: bool = True, card_block: dict = None)\n\n# Get a DataTable client for SQL queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# \n# Returns:\n# DataTableClient instance\ndef datatable(name: str = 'main')\n\n# Get a DuckLake client for DuckDB queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# \n# Returns:\n# DucklakeClient instance\ndef ducklake(name: str = 'main')\n\ndef init_global_client(f)\n\ndef deprecate(in_favor_of: str)\n\n# Get the current workspace ID.\n# \n# Returns:\n# Workspace ID string\ndef get_workspace() -> str\n\ndef get_version() -> str\n\n# Run a script synchronously by hash and return its result.\n# \n# Args:\n# hash: Script hash\n# args: Script arguments\n# verbose: Enable verbose logging\n# assert_result_is_not_none: Raise exception if result is None\n# cleanup: Register cleanup handler to cancel job on exit\n# timeout: Maximum time to wait\n# \n# Returns:\n# Script result\ndef run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any\n\n# Run a script synchronously by path and return its result.\n# \n# Args:\n# path: Script path\n# args: Script arguments\n# verbose: Enable verbose logging\n# assert_result_is_not_none: Raise exception if result is None\n# cleanup: Register cleanup handler to cancel job on exit\n# timeout: Maximum time to wait\n# \n# Returns:\n# Script result\ndef run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from DuckDB\ndef duckdb_connection_settings(s3_resource_path: str = '') -> DuckDbConnectionSettings\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection from Polars\ndef polars_connection_settings(s3_resource_path: str = '') -> PolarsConnectionSettings\n\n# Convenient helpers that takes an S3 resource as input and returns the settings necessary to\n# initiate an S3 connection using boto3\ndef boto3_connection_settings(s3_resource_path: str = '') -> Boto3ConnectionSettings\n\n# Get the state resource path from environment.\n# \n# Returns:\n# State path string\ndef get_state_path() -> str\n\n# Parse resource syntax from string.\ndef parse_resource_syntax(s: str) -> Optional[str]\n\n# Parse S3 object from string or S3Object format.\ndef parse_s3_object(s3_object: S3Object | str) -> S3Object\n\n# Parse variable syntax from string.\ndef parse_variable_syntax(s: str) -> Optional[str]\n\n# Append a text to the result stream.\n# \n# Args:\n# text: text to append to the result stream\ndef append_to_result_stream(text: str) -> None\n\n# Stream to the result stream.\n# \n# Args:\n# stream: stream to stream to the result stream\ndef stream_result(stream) -> None\n\n# Execute a SQL query against the DataTable.\n# \n# Args:\n# sql: SQL query string with $1, $2, etc. placeholders\n# *args: Positional arguments to bind to query placeholders\n# \n# Returns:\n# SqlQuery instance for fetching results\ndef query(sql: str, *args) -> SqlQuery\n\n# Execute query and fetch results.\n# \n# Args:\n# result_collection: Optional result collection mode\n# \n# Returns:\n# Query results\ndef fetch(result_collection: str | None = None)\n\n# Execute query and fetch first row of results.\n# \n# Returns:\n# First row of query results\ndef fetch_one()\n\n# Execute query and fetch first row of results. Return result as a scalar value.\n# \n# Returns:\n# First row of query result as a scalar value\ndef fetch_one_scalar()\n\n# Execute query and don't return any results.\n# \ndef execute()\n\n# DuckDB executor requires explicit argument types at declaration\n# These types exist in both DuckDB and Postgres\n# Check that the types exist if you plan to extend this function for other SQL engines.\ndef infer_sql_type(value) -> str\n\ndef parse_sql_client_name(name: str) -> tuple[str, Optional[str]]\n\n# Decorator that marks a function as a workflow task.\n# \n# Works in both WAC v1 (sync, HTTP-based dispatch) and WAC v2\n# (async, checkpoint/replay) modes:\n# \n# - **v2 (inside @workflow)**: dispatches as a checkpoint step.\n# - **v1 (WM_JOB_ID set, no @workflow)**: dispatches via HTTP API.\n# - **Standalone**: executes the function body directly.\n# \n# Usage::\n# \n# @task\n# async def extract_data(url: str): ...\n# \n# @task(path=\"f/external_script\", timeout=600, tag=\"gpu\")\n# async def run_external(x: int): ...\ndef task(_func = None, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill script.\n# \n# Usage::\n# \n# extract = task_script(\"f/data/extract\", timeout=600)\n# \n# @workflow\n# async def main():\n# data = await extract(url=\"https://...\")\ndef task_script(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill flow.\n# \n# Usage::\n# \n# pipeline = task_flow(\"f/etl/pipeline\", priority=10)\n# \n# @workflow\n# async def main():\n# result = await pipeline(input=data)\ndef task_flow(path: str, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Decorator marking an async function as a workflow-as-code entry point.\n# \n# The function must be **deterministic**: given the same inputs it must call\n# tasks in the same order on every replay. Branching on task results is fine\n# (results are replayed from checkpoint), but branching on external state\n# (current time, random values, external API calls) must use ``step()`` to\n# checkpoint the value so replays see the same result.\ndef workflow(func)\n\n# Execute ``fn`` inline and checkpoint the result.\n# \n# On replay the cached value is returned without re-executing ``fn``.\n# Use for lightweight deterministic operations (timestamps, random IDs,\n# config reads) that should not incur the overhead of a child job.\nasync def step(name: str, fn)\n\n# Server-side sleep \u2014 suspend the workflow for the given duration without holding a worker.\n# \n# Inside a @workflow, the parent job suspends and auto-resumes after ``seconds``.\n# Outside a workflow, falls back to ``asyncio.sleep``.\nasync def sleep(seconds: int)\n\n# Suspend the workflow and wait for an external approval.\n# \n# Use ``get_resume_urls()`` (wrapped in ``step()``) to obtain\n# resume/cancel/approval URLs before calling this function.\n# \n# Returns a dict with ``value`` (form data), ``approver``, and ``approved``.\n# \n# Args:\n# timeout: Approval timeout in seconds (default 1800).\n# form: Optional form schema for the approval page.\n# self_approval: Whether the user who triggered the flow can approve it (default True).\n# \n# Example::\n# \n# urls = await step(\"urls\", lambda: get_resume_urls())\n# await step(\"notify\", lambda: send_email(urls[\"approvalPage\"]))\n# result = await wait_for_approval(timeout=3600)\nasync def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_approval: bool = True) -> dict\n\n# Process items in parallel with optional concurrency control.\n# \n# Each item is processed by calling ``fn(item)``, which should be a @task.\n# Items are dispatched in batches of ``concurrency`` (default: all at once).\n# \n# Example::\n# \n# @task\n# async def process(item: str):\n# ...\n# \n# results = await parallel(items, process, concurrency=5)\nasync def parallel(items, fn, concurrency: Optional[int] = None)\n\n# Commit Kafka offsets for a trigger with auto_commit disabled.\n# \n# Args:\n# trigger_path: Path to the Kafka trigger (from event['wm_trigger']['trigger_path'])\n# topic: Kafka topic name (from event['topic'])\n# partition: Partition number (from event['partition'])\n# offset: Message offset to commit (from event['offset'])\ndef commit_kafka_offsets(trigger_path: str, topic: str, partition: int, offset: int) -> None\n\n"; -export declare const WAC_SDK_TYPESCRIPT = "## TypeScript Workflow-as-Code API (windmill-client)\n\nImport: `import { workflow, task, taskScript, taskFlow, step, sleep, waitForApproval, getResumeUrls, parallel } from \"windmill-client\"`\n\n```typescript\nexport interface TaskOptions {\n timeout?: number;\n tag?: string;\n cache_ttl?: number;\n priority?: number;\n concurrency_limit?: number;\n concurrency_key?: string;\n concurrency_time_window_s?: number;\n}\n\n/**\n * Get URLs needed for resuming a flow after this step\n * @param approver approver name\n * @param flowLevel if true, generate resume URLs for the parent flow instead of the specific step.\n * This allows pre-approvals that can be consumed by any later suspend step in the same flow.\n * @returns approval page UI URL, resume and cancel API URLs for resuming the flow\n */\nexport async function getResumeUrls(approver?: string, flowLevel?: boolean): Promise<{ approvalPage: string; resume: string; cancel: string; }>\n\n/**\n * Wrap an async function as a workflow task.\n *\n * @example\n * const extract_data = task(async (url: string) => { ... });\n * const run_external = task(\"f/external_script\", async (x: number) => { ... });\n *\n * Inside a `workflow()`, calling a task dispatches it as a step.\n * Outside a workflow, the function body executes directly.\n */\nexport function task Promise>(fnOrPath: T | string, maybeFnOrOptions?: T | TaskOptions, maybeOptions?: TaskOptions,): T\n\n/**\n * Create a task that dispatches to a separate Windmill script.\n *\n * @example\n * const extract = taskScript(\"f/data/extract\");\n * // inside workflow: await extract({ url: \"https://...\" })\n */\nexport function taskScript(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Create a task that dispatches to a separate Windmill flow.\n *\n * @example\n * const pipeline = taskFlow(\"f/etl/pipeline\");\n * // inside workflow: await pipeline({ input: data })\n */\nexport function taskFlow(path: string, options?: TaskOptions): (...args: any[]) => PromiseLike\n\n/**\n * Mark an async function as a workflow-as-code entry point.\n *\n * The function must be **deterministic**: given the same inputs it must call\n * tasks in the same order on every replay. Branching on task results is fine\n * (results are replayed from checkpoint), but branching on external state\n * (current time, random values, external API calls) must use `step()` to\n * checkpoint the value so replays see the same result.\n */\nexport function workflow(fn: (...args: any[]) => Promise)\n\nexport async function step(name: string, fn: () => T | Promise): Promise\n\nexport async function sleep(seconds: number): Promise\n\n/**\n * Suspend the workflow and wait for an external approval.\n *\n * Use `getResumeUrls()` (wrapped in `step()`) to obtain resume/cancel/approvalPage\n * URLs before calling this function.\n *\n * @example\n * const urls = await step(\"urls\", () => getResumeUrls());\n * await step(\"notify\", () => sendEmail(urls.approvalPage));\n * const { value, approver } = await waitForApproval({ timeout: 3600 });\n */\nexport function waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }>\n\n/**\n * Process items in parallel with optional concurrency control.\n *\n * Each item is processed by calling `fn(item)`, which should be a task().\n * Items are dispatched in batches of `concurrency` (default: all at once).\n *\n * @example\n * const process = task(async (item: string) => { ... });\n * const results = await parallel(items, process, { concurrency: 5 });\n */\nexport async function parallel(items: T[], fn: (item: T) => PromiseLike | R, options?: { concurrency?: number },): Promise\n```\n"; -export declare const WAC_SDK_PYTHON = "## Python Workflow-as-Code API (wmill)\n\nImport: `from wmill import workflow, task, task_script, task_flow, step, sleep, wait_for_approval, get_resume_urls, parallel, TaskError`\n\n```python\n# Raised when a WAC task step failed.\n#\n# Attributes:\n# step_key: The checkpoint key of the failed step.\n# child_job_id: The UUID of the failed child job.\n# result: The error result from the child job.\nclass TaskError(Exception):\n def __init__(self, message: str, *, step_key: str = '', child_job_id: str = '', result = None)\n\n# Get URLs needed for resuming a flow after suspension.\n#\n# Args:\n# approver: Optional approver name\n# flow_level: If True, generate resume URLs for the parent flow instead of the\n# specific step. This allows pre-approvals that can be consumed by any later\n# suspend step in the same flow.\n#\n# Returns:\n# Dictionary with approvalPage, resume, and cancel URLs\ndef get_resume_urls(approver: str = None, flow_level: bool = None) -> dict\n\n# Decorator that marks a function as a workflow task.\n#\n# Works in both WAC v1 (sync, HTTP-based dispatch) and WAC v2\n# (async, checkpoint/replay) modes:\n#\n# - **v2 (inside @workflow)**: dispatches as a checkpoint step.\n# - **v1 (WM_JOB_ID set, no @workflow)**: dispatches via HTTP API.\n# - **Standalone**: executes the function body directly.\n#\n# Usage::\n#\n# @task\n# async def extract_data(url: str): ...\n#\n# @task(path=\"f/external_script\", timeout=600, tag=\"gpu\")\n# async def run_external(x: int): ...\ndef task(_func = None, *, path: Optional[str] = None, tag: Optional[str] = None, timeout: Optional[int] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill script.\n#\n# Usage::\n#\n# extract = task_script(\"f/data/extract\", timeout=600)\n#\n# @workflow\n# async def main():\n# data = await extract(url=\"https://...\")\ndef task_script(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Create a task that dispatches to a separate Windmill flow.\n#\n# Usage::\n#\n# pipeline = task_flow(\"f/etl/pipeline\", priority=10)\n#\n# @workflow\n# async def main():\n# result = await pipeline(input=data)\ndef task_flow(path: str, *, timeout: Optional[int] = None, tag: Optional[str] = None, cache_ttl: Optional[int] = None, priority: Optional[int] = None, concurrency_limit: Optional[int] = None, concurrency_key: Optional[str] = None, concurrency_time_window_s: Optional[int] = None)\n\n# Decorator marking an async function as a workflow-as-code entry point.\n#\n# The function must be **deterministic**: given the same inputs it must call\n# tasks in the same order on every replay. Branching on task results is fine\n# (results are replayed from checkpoint), but branching on external state\n# (current time, random values, external API calls) must use ``step()`` to\n# checkpoint the value so replays see the same result.\ndef workflow(func)\n\n# Execute ``fn`` inline and checkpoint the result.\n#\n# On replay the cached value is returned without re-executing ``fn``.\n# Use for lightweight deterministic operations (timestamps, random IDs,\n# config reads) that should not incur the overhead of a child job.\nasync def step(name: str, fn)\n\n# Server-side sleep \u2014 suspend the workflow for the given duration without holding a worker.\n#\n# Inside a @workflow, the parent job suspends and auto-resumes after ``seconds``.\n# Outside a workflow, falls back to ``asyncio.sleep``.\nasync def sleep(seconds: int)\n\n# Suspend the workflow and wait for an external approval.\n#\n# Use ``get_resume_urls()`` (wrapped in ``step()``) to obtain\n# resume/cancel/approval URLs before calling this function.\n#\n# Returns a dict with ``value`` (form data), ``approver``, and ``approved``.\n#\n# Args:\n# timeout: Approval timeout in seconds (default 1800).\n# form: Optional form schema for the approval page.\n# self_approval: Whether the user who triggered the flow can approve it (default True).\n#\n# Example::\n#\n# urls = await step(\"urls\", lambda: get_resume_urls())\n# await step(\"notify\", lambda: send_email(urls[\"approvalPage\"]))\n# result = await wait_for_approval(timeout=3600)\nasync def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_approval: bool = True) -> dict\n\n# Process items in parallel with optional concurrency control.\n#\n# Each item is processed by calling ``fn(item)``, which should be a @task.\n# Items are dispatched in batches of ``concurrency`` (default: all at once).\n#\n# Example::\n#\n# @task\n# async def process(item: str):\n# ...\n#\n# results = await parallel(items, process, concurrency=5)\nasync def parallel(items, fn, *, concurrency: Optional[int] = None)\n```\n"; -export declare const DATATABLE_SDK_TYPESCRIPT = "## TypeScript Datatable API (windmill-client)\n\nImport: `import * as wmill from 'windmill-client'`\n\nSQL statement object with query content, arguments, and execution methods\n```typescript\ntype SqlStatement = {\n /** Raw SQL content with formatted arguments */\n content: string;\n\n /** Argument values keyed by parameter name */\n args: Record;\n\n /**\n * Execute the SQL query and return results\n * @param params - Optional parameters including result collection mode\n * @returns Query results based on the result collection mode\n */\n fetch(\n params?: FetchParams // The union is for auto-completion\n ): Promise>;\n\n /**\n * Execute the SQL query and return only the first row\n * @param params - Optional parameters\n * @returns First row of the query result\n */\n fetchOne(\n params?: Omit, \"resultCollection\">\n ): Promise>;\n\n /**\n * Execute the SQL query and return only the first row as a scalar value\n * @param params - Optional parameters\n * @returns First row of the query result\n */\n fetchOneScalar(\n params?: Omit<\n FetchParams<\"last_statement_first_row_scalar\">,\n \"resultCollection\"\n >\n ): Promise>;\n\n /**\n * Execute the SQL query without fetching rows\n * @param params - Optional parameters\n */\n execute(\n params?: Omit, \"resultCollection\">\n ): Promise;\n};\n```\n\n```typescript\n// Template tag function: sql`SELECT * FROM table WHERE id = ${id}`.fetch()\ninterface DatatableSqlTemplateFunction {\n // Tagged template usage:\n (strings: TemplateStringsArray, ...values: any[]): SqlStatement;\n query(sql: string, ...params: any[]): SqlStatement;\n};\n```\n\nCreate a SQL template function for PostgreSQL/datatable queries\n@param name - Database/datatable name (default: \"main\")\n@returns SQL template function for building parameterized queries\n@example\nlet sql = wmill.datatable()\nlet name = 'Robin'\nlet age = 21\nawait sql`\n SELECT * FROM friends\n WHERE name = ${name} AND age = ${age}::int\n`.fetch()\n```typescript\nfunction datatable(name: string = \"main\"): DatatableSqlTemplateFunction\n```\n"; -export declare const DATATABLE_SDK_PYTHON = "## Python Datatable API (wmill)\n\nImport: `import wmill`\n\n# Get a DataTable client for SQL queries.\n# \n# Args:\n# name: Database name (default: \"main\")\n# \n# Returns:\n# DataTableClient instance\ndef datatable(name: str = 'main') -> DataTableClient\n\n# Client for executing SQL queries against Windmill DataTables.\nclass DataTableClient:\n # Initialize DataTableClient.\n # \n # Args:\n # client: Windmill client instance\n # name: DataTable name\n def __init__(client: Windmill, name: str)\n\n # Execute a SQL query against the DataTable.\n # \n # Args:\n # sql: SQL query string with $1, $2, etc. placeholders\n # *args: Positional arguments to bind to query placeholders\n # \n # Returns:\n # SqlQuery instance for fetching results\n def query(sql: str, *args) -> SqlQuery\n\n\n# Query result handler for DataTable and DuckLake queries.\nclass SqlQuery:\n # Initialize SqlQuery.\n # \n # Args:\n # sql: SQL query string\n # fetch_fn: Function to execute the query\n def __init__(sql: str, fetch_fn)\n\n # Execute query and fetch results.\n # \n # Args:\n # result_collection: Optional result collection mode\n # \n # Returns:\n # Query results\n def fetch(result_collection: str | None = None)\n\n # Execute query and fetch first row of results.\n # \n # Returns:\n # First row of query results\n def fetch_one()\n\n # Execute query and fetch first row of results. Return result as a scalar value.\n # \n # Returns:\n # First row of query result as a scalar value\n def fetch_one_scalar()\n\n # Execute query and don't return any results.\n # \n def execute()\n\n\n"; -export declare const OPENFLOW_SCHEMA = "## OpenFlow Schema\n\n{\"OpenFlow\":{\"type\":\"object\",\"description\":\"Top-level flow definition containing metadata, configuration, and the flow structure\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Short description of what this flow does\"},\"description\":{\"type\":\"string\",\"description\":\"Detailed documentation for this flow\"},\"value\":{\"$ref\":\"#/components/schemas/FlowValue\"},\"schema\":{\"type\":\"object\",\"description\":\"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')\"},\"on_behalf_of_email\":{\"type\":\"string\",\"description\":\"The flow will be run with the permissions of the user with this email.\"}},\"required\":[\"summary\",\"value\"]},\"FlowValue\":{\"type\":\"object\",\"description\":\"The flow structure containing modules and optional preprocessor/failure handlers\",\"properties\":{\"modules\":{\"type\":\"array\",\"description\":\"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"failure_module\":{\"description\":\"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types\",\"$ref\":\"#/components/schemas/FlowModule\"},\"preprocessor_module\":{\"description\":\"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results\",\"$ref\":\"#/components/schemas/FlowModule\"},\"same_worker\":{\"type\":\"boolean\",\"description\":\"If true, all steps run on the same worker for better performance\"},\"concurrent_limit\":{\"type\":\"number\",\"description\":\"Maximum number of concurrent executions of this flow\"},\"concurrency_key\":{\"type\":\"string\",\"description\":\"Expression to group concurrent executions (e.g., by user ID)\"},\"concurrency_time_window_s\":{\"type\":\"number\",\"description\":\"Time window in seconds for concurrent_limit\"},\"debounce_delay_s\":{\"type\":\"integer\",\"description\":\"Delay in seconds to debounce flow executions\"},\"debounce_key\":{\"type\":\"string\",\"description\":\"Expression to group debounced executions\"},\"debounce_args_to_accumulate\":{\"type\":\"array\",\"description\":\"Arguments to accumulate across debounced executions\",\"items\":{\"type\":\"string\"}},\"max_total_debouncing_time\":{\"type\":\"integer\",\"description\":\"Maximum total time in seconds that a job can be debounced\"},\"max_total_debounces_amount\":{\"type\":\"integer\",\"description\":\"Maximum number of times a job can be debounced\"},\"skip_expr\":{\"type\":\"string\",\"description\":\"JavaScript expression to conditionally skip the entire flow\"},\"cache_ttl\":{\"type\":\"number\",\"description\":\"Cache duration in seconds for flow results\"},\"cache_ignore_s3_path\":{\"type\":\"boolean\"},\"delete_after_secs\":{\"type\":\"integer\",\"description\":\"If set, delete the flow job's args, result and logs after this many seconds following job completion\"},\"flow_env\":{\"type\":\"object\",\"description\":\"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).\",\"additionalProperties\":{}},\"priority\":{\"type\":\"number\",\"description\":\"Execution priority (higher numbers run first)\"},\"early_return\":{\"type\":\"string\",\"description\":\"JavaScript expression to return early from the flow\"},\"chat_input_enabled\":{\"type\":\"boolean\",\"description\":\"Whether this flow accepts chat-style input\"},\"notes\":{\"type\":\"array\",\"description\":\"Sticky notes attached to the flow\",\"items\":{\"$ref\":\"#/components/schemas/FlowNote\"}},\"groups\":{\"type\":\"array\",\"description\":\"Semantic groups of modules for organizational purposes\",\"items\":{\"$ref\":\"#/components/schemas/FlowGroup\"}}},\"required\":[\"modules\"]},\"Retry\":{\"type\":\"object\",\"description\":\"Retry configuration for failed module executions\",\"properties\":{\"constant\":{\"type\":\"object\",\"description\":\"Retry with constant delay between attempts\",\"properties\":{\"attempts\":{\"type\":\"integer\",\"description\":\"Number of retry attempts\"},\"seconds\":{\"type\":\"integer\",\"description\":\"Seconds to wait between retries\"}}},\"exponential\":{\"type\":\"object\",\"description\":\"Retry with exponential backoff (delay doubles each time)\",\"properties\":{\"attempts\":{\"type\":\"integer\",\"description\":\"Number of retry attempts\"},\"multiplier\":{\"type\":\"integer\",\"description\":\"Multiplier for exponential backoff\"},\"seconds\":{\"type\":\"integer\",\"minimum\":1,\"description\":\"Initial delay in seconds\"},\"random_factor\":{\"type\":\"integer\",\"minimum\":0,\"maximum\":100,\"description\":\"Random jitter percentage (0-100) to avoid thundering herd\"}}},\"retry_if\":{\"$ref\":\"#/components/schemas/RetryIf\"}}},\"FlowNote\":{\"type\":\"object\",\"description\":\"A sticky note attached to a flow for documentation and annotation\",\"properties\":{\"id\":{\"type\":\"string\",\"description\":\"Unique identifier for the note\"},\"text\":{\"type\":\"string\",\"description\":\"Content of the note\"},\"position\":{\"type\":\"object\",\"description\":\"Position of the note in the flow editor\",\"properties\":{\"x\":{\"type\":\"number\",\"description\":\"X coordinate\"},\"y\":{\"type\":\"number\",\"description\":\"Y coordinate\"}},\"required\":[\"x\",\"y\"]},\"size\":{\"type\":\"object\",\"description\":\"Size of the note in the flow editor\",\"properties\":{\"width\":{\"type\":\"number\",\"description\":\"Width in pixels\"},\"height\":{\"type\":\"number\",\"description\":\"Height in pixels\"}},\"required\":[\"width\",\"height\"]},\"color\":{\"type\":\"string\",\"description\":\"Color of the note (e.g., \\\"yellow\\\", \\\"#ffff00\\\")\"},\"type\":{\"type\":\"string\",\"enum\":[\"free\",\"group\"],\"description\":\"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes\"},\"locked\":{\"type\":\"boolean\",\"default\":false,\"description\":\"Whether the note is locked and cannot be edited or moved\"},\"contained_node_ids\":{\"type\":\"array\",\"items\":{\"type\":\"string\"},\"description\":\"For group notes, the IDs of nodes contained within this group\"}},\"required\":[\"id\",\"text\",\"color\",\"type\"]},\"FlowGroup\":{\"type\":\"object\",\"description\":\"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Display name for this group\"},\"note\":{\"type\":\"string\",\"description\":\"Markdown note shown below the group header\"},\"autocollapse\":{\"type\":\"boolean\",\"default\":false,\"description\":\"If true, this group is collapsed by default in the flow editor. UI hint only.\"},\"start_id\":{\"type\":\"string\",\"description\":\"ID of the first flow module in this group (topological entry point)\"},\"end_id\":{\"type\":\"string\",\"description\":\"ID of the last flow module in this group (topological exit point)\"},\"color\":{\"type\":\"string\",\"description\":\"Color for the group in the flow editor\"}},\"required\":[\"start_id\",\"end_id\"]},\"RetryIf\":{\"type\":\"object\",\"description\":\"Conditional retry based on error or result\",\"properties\":{\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables\"}},\"required\":[\"expr\"]},\"StopAfterIf\":{\"type\":\"object\",\"description\":\"Early termination condition for a module\",\"properties\":{\"skip_if_stopped\":{\"type\":\"boolean\",\"description\":\"If true, following steps are skipped when this condition triggers\"},\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop\"},\"error_message\":{\"type\":\"string\",\"nullable\":true,\"description\":\"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.\"}},\"required\":[\"expr\"]},\"FlowModule\":{\"type\":\"object\",\"description\":\"A single step in a flow. Can be a script, subflow, loop, or branch\",\"properties\":{\"id\":{\"type\":\"string\",\"description\":\"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)\"},\"value\":{\"$ref\":\"#/components/schemas/FlowModuleValue\"},\"stop_after_if\":{\"description\":\"Early termination condition evaluated after this step completes\",\"$ref\":\"#/components/schemas/StopAfterIf\"},\"stop_after_all_iters_if\":{\"description\":\"For loops only - early termination condition evaluated after all iterations complete\",\"$ref\":\"#/components/schemas/StopAfterIf\"},\"skip_if\":{\"type\":\"object\",\"description\":\"Conditionally skip this step based on previous results or flow inputs\",\"properties\":{\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'\"}},\"required\":[\"expr\"]},\"sleep\":{\"description\":\"Delay before executing this step (in seconds or as expression)\",\"$ref\":\"#/components/schemas/InputTransform\"},\"cache_ttl\":{\"type\":\"number\",\"description\":\"Cache duration in seconds for this step's results\"},\"cache_ignore_s3_path\":{\"type\":\"boolean\"},\"timeout\":{\"description\":\"Maximum execution time in seconds (static value or expression)\",\"$ref\":\"#/components/schemas/InputTransform\"},\"delete_after_secs\":{\"type\":\"integer\",\"description\":\"If set, delete the step's args, result and logs after this many seconds following job completion\"},\"summary\":{\"type\":\"string\",\"description\":\"Short description of what this step does\"},\"mock\":{\"type\":\"object\",\"description\":\"Mock configuration for testing without executing the actual step\",\"properties\":{\"enabled\":{\"type\":\"boolean\",\"description\":\"If true, return mock value instead of executing\"},\"return_value\":{\"description\":\"Value to return when mocked\"}}},\"suspend\":{\"type\":\"object\",\"description\":\"Configuration for approval/resume steps that wait for user input\",\"properties\":{\"required_events\":{\"type\":\"integer\",\"description\":\"Number of approvals required before continuing\"},\"timeout\":{\"type\":\"integer\",\"description\":\"Timeout in seconds before auto-continuing or canceling\"},\"resume_form\":{\"type\":\"object\",\"description\":\"Form schema for collecting input when resuming\",\"properties\":{\"schema\":{\"type\":\"object\",\"description\":\"JSON Schema for the resume form\"}}},\"user_auth_required\":{\"type\":\"boolean\",\"description\":\"If true, only authenticated users can approve\"},\"user_groups_required\":{\"description\":\"Expression or list of groups that can approve\",\"$ref\":\"#/components/schemas/InputTransform\"},\"self_approval_disabled\":{\"type\":\"boolean\",\"description\":\"If true, the user who started the flow cannot approve\"},\"hide_cancel\":{\"type\":\"boolean\",\"description\":\"If true, hide the cancel button on the approval form\"},\"continue_on_disapprove_timeout\":{\"type\":\"boolean\",\"description\":\"If true, continue flow on timeout instead of canceling\"}}},\"priority\":{\"type\":\"number\",\"description\":\"Execution priority for this step (higher numbers run first)\"},\"continue_on_error\":{\"type\":\"boolean\",\"description\":\"If true, flow continues even if this step fails\"},\"retry\":{\"description\":\"Retry configuration if this step fails\",\"$ref\":\"#/components/schemas/Retry\"},\"debouncing\":{\"description\":\"Debounce configuration for this step (EE only)\",\"type\":\"object\",\"properties\":{\"debounce_delay_s\":{\"type\":\"integer\",\"description\":\"Delay in seconds to debounce this step's executions across flow runs\"},\"debounce_key\":{\"type\":\"string\",\"description\":\"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-\"},\"debounce_args_to_accumulate\":{\"type\":\"array\",\"description\":\"Array-type arguments to accumulate across debounced executions\",\"items\":{\"type\":\"string\"}},\"max_total_debouncing_time\":{\"type\":\"integer\",\"description\":\"Maximum total time in seconds before forced execution\"},\"max_total_debounces_amount\":{\"type\":\"integer\",\"description\":\"Maximum number of debounces before forced execution\"}}}},\"required\":[\"value\",\"id\"]},\"InputTransform\":{\"description\":\"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs\",\"oneOf\":[{\"$ref\":\"#/components/schemas/StaticTransform\"},{\"$ref\":\"#/components/schemas/JavascriptTransform\"},{\"$ref\":\"#/components/schemas/AiTransform\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"static\":\"#/components/schemas/StaticTransform\",\"javascript\":\"#/components/schemas/JavascriptTransform\",\"ai\":\"#/components/schemas/AiTransform\"}}},\"StaticTransform\":{\"type\":\"object\",\"description\":\"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'\",\"properties\":{\"value\":{\"description\":\"The static value. For resources, use format '$res:path/to/resource'\"},\"type\":{\"type\":\"string\",\"enum\":[\"static\"]}},\"required\":[\"type\"]},\"JavascriptTransform\":{\"type\":\"object\",\"description\":\"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value\",\"properties\":{\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)\"},\"type\":{\"type\":\"string\",\"enum\":[\"javascript\"]}},\"required\":[\"expr\",\"type\"]},\"AiTransform\":{\"type\":\"object\",\"description\":\"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"ai\"]}},\"required\":[\"type\"]},\"AIProviderKind\":{\"type\":\"string\",\"description\":\"Supported AI provider types\",\"enum\":[\"openai\",\"azure_openai\",\"anthropic\",\"mistral\",\"deepseek\",\"googleai\",\"groq\",\"openrouter\",\"togetherai\",\"customai\",\"aws_bedrock\"]},\"ProviderConfig\":{\"type\":\"object\",\"description\":\"Complete AI provider configuration with resource reference and model selection\",\"properties\":{\"kind\":{\"$ref\":\"#/components/schemas/AIProviderKind\"},\"resource\":{\"type\":\"string\",\"description\":\"Resource reference in format '$res:{resource_path}' pointing to provider credentials\"},\"model\":{\"type\":\"string\",\"description\":\"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')\"}},\"required\":[\"kind\",\"resource\",\"model\"]},\"StaticProviderTransform\":{\"type\":\"object\",\"description\":\"Static provider configuration passed directly to the AI agent\",\"properties\":{\"value\":{\"$ref\":\"#/components/schemas/ProviderConfig\"},\"type\":{\"type\":\"string\",\"enum\":[\"static\"]}},\"required\":[\"type\",\"value\"]},\"ProviderTransform\":{\"description\":\"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined\",\"oneOf\":[{\"$ref\":\"#/components/schemas/StaticProviderTransform\"},{\"$ref\":\"#/components/schemas/JavascriptTransform\"},{\"$ref\":\"#/components/schemas/AiTransform\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"static\":\"#/components/schemas/StaticProviderTransform\",\"javascript\":\"#/components/schemas/JavascriptTransform\",\"ai\":\"#/components/schemas/AiTransform\"}}},\"MemoryOff\":{\"type\":\"object\",\"description\":\"No conversation memory/context\",\"properties\":{\"kind\":{\"type\":\"string\",\"enum\":[\"off\"]}},\"required\":[\"kind\"]},\"MemoryAuto\":{\"type\":\"object\",\"description\":\"Automatic context management\",\"properties\":{\"kind\":{\"type\":\"string\",\"enum\":[\"auto\"]},\"context_length\":{\"type\":\"integer\",\"description\":\"Maximum number of messages to retain in context\"},\"memory_id\":{\"type\":\"string\",\"description\":\"Identifier for persistent memory across agent invocations\"}},\"required\":[\"kind\"]},\"MemoryMessage\":{\"type\":\"object\",\"description\":\"A single message in conversation history\",\"properties\":{\"role\":{\"type\":\"string\",\"enum\":[\"user\",\"assistant\",\"system\"]},\"content\":{\"type\":\"string\"}},\"required\":[\"role\",\"content\"]},\"MemoryManual\":{\"type\":\"object\",\"description\":\"Explicit message history\",\"properties\":{\"kind\":{\"type\":\"string\",\"enum\":[\"manual\"]},\"messages\":{\"type\":\"array\",\"items\":{\"$ref\":\"#/components/schemas/MemoryMessage\"}}},\"required\":[\"kind\",\"messages\"]},\"MemoryConfig\":{\"description\":\"Conversation memory configuration\",\"oneOf\":[{\"$ref\":\"#/components/schemas/MemoryOff\"},{\"$ref\":\"#/components/schemas/MemoryAuto\"},{\"$ref\":\"#/components/schemas/MemoryManual\"}],\"discriminator\":{\"propertyName\":\"kind\",\"mapping\":{\"off\":\"#/components/schemas/MemoryOff\",\"auto\":\"#/components/schemas/MemoryAuto\",\"manual\":\"#/components/schemas/MemoryManual\"}}},\"StaticMemoryTransform\":{\"type\":\"object\",\"description\":\"Static memory configuration passed directly to the AI agent\",\"properties\":{\"value\":{\"$ref\":\"#/components/schemas/MemoryConfig\"},\"type\":{\"type\":\"string\",\"enum\":[\"static\"]}},\"required\":[\"type\",\"value\"]},\"MemoryTransform\":{\"description\":\"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined\",\"oneOf\":[{\"$ref\":\"#/components/schemas/StaticMemoryTransform\"},{\"$ref\":\"#/components/schemas/JavascriptTransform\"},{\"$ref\":\"#/components/schemas/AiTransform\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"static\":\"#/components/schemas/StaticMemoryTransform\",\"javascript\":\"#/components/schemas/JavascriptTransform\",\"ai\":\"#/components/schemas/AiTransform\"}}},\"FlowModuleValue\":{\"description\":\"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type\",\"oneOf\":[{\"$ref\":\"#/components/schemas/RawScript\"},{\"$ref\":\"#/components/schemas/PathScript\"},{\"$ref\":\"#/components/schemas/PathFlow\"},{\"$ref\":\"#/components/schemas/ForloopFlow\"},{\"$ref\":\"#/components/schemas/WhileloopFlow\"},{\"$ref\":\"#/components/schemas/BranchOne\"},{\"$ref\":\"#/components/schemas/BranchAll\"},{\"$ref\":\"#/components/schemas/Identity\"},{\"$ref\":\"#/components/schemas/AiAgent\"}],\"discriminator\":{\"propertyName\":\"type\",\"mapping\":{\"rawscript\":\"#/components/schemas/RawScript\",\"script\":\"#/components/schemas/PathScript\",\"flow\":\"#/components/schemas/PathFlow\",\"forloopflow\":\"#/components/schemas/ForloopFlow\",\"whileloopflow\":\"#/components/schemas/WhileloopFlow\",\"branchone\":\"#/components/schemas/BranchOne\",\"branchall\":\"#/components/schemas/BranchAll\",\"identity\":\"#/components/schemas/Identity\",\"aiagent\":\"#/components/schemas/AiAgent\"}}},\"RawScript\":{\"type\":\"object\",\"description\":\"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments\",\"additionalProperties\":{\"$ref\":\"#/components/schemas/InputTransform\"}},\"content\":{\"type\":\"string\",\"description\":\"The script source code. Should export a 'main' function\"},\"language\":{\"type\":\"string\",\"description\":\"Programming language for this script\",\"enum\":[\"deno\",\"bun\",\"python3\",\"go\",\"bash\",\"powershell\",\"postgresql\",\"mysql\",\"bigquery\",\"snowflake\",\"mssql\",\"oracledb\",\"graphql\",\"nativets\",\"php\",\"rust\",\"ansible\",\"csharp\",\"nu\",\"java\",\"ruby\",\"rlang\",\"duckdb\"]},\"path\":{\"type\":\"string\",\"description\":\"Optional path for saving this script\"},\"lock\":{\"type\":\"string\",\"description\":\"Lock file content for dependencies\"},\"type\":{\"type\":\"string\",\"enum\":[\"rawscript\"]},\"tag\":{\"type\":\"string\",\"description\":\"Worker group tag for execution routing\"},\"concurrent_limit\":{\"type\":\"number\",\"description\":\"Maximum concurrent executions of this script\"},\"concurrency_time_window_s\":{\"type\":\"number\",\"description\":\"Time window for concurrent_limit\"},\"custom_concurrency_key\":{\"type\":\"string\",\"description\":\"Custom key for grouping concurrent executions\"},\"is_trigger\":{\"type\":\"boolean\",\"description\":\"If true, this script is a trigger that can start the flow\"},\"assets\":{\"type\":\"array\",\"description\":\"External resources this script accesses (S3 objects, resources, etc.)\",\"items\":{\"type\":\"object\",\"required\":[\"path\",\"kind\"],\"properties\":{\"path\":{\"type\":\"string\",\"description\":\"Path to the asset\"},\"kind\":{\"type\":\"string\",\"description\":\"Type of asset\",\"enum\":[\"s3object\",\"resource\",\"ducklake\",\"datatable\",\"volume\"]},\"access_type\":{\"type\":\"string\",\"nullable\":true,\"description\":\"Access level for this asset\",\"enum\":[\"r\",\"w\",\"rw\"]},\"alt_access_type\":{\"type\":\"string\",\"nullable\":true,\"description\":\"Alternative access level\",\"enum\":[\"r\",\"w\",\"rw\"]}}}}},\"required\":[\"type\",\"content\",\"language\",\"input_transforms\"]},\"PathScript\":{\"type\":\"object\",\"description\":\"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments\",\"additionalProperties\":{\"$ref\":\"#/components/schemas/InputTransform\"}},\"path\":{\"type\":\"string\",\"description\":\"Path to the script in the workspace (e.g., 'f/scripts/send_email')\"},\"hash\":{\"type\":\"string\",\"description\":\"Optional specific version hash of the script to use\"},\"type\":{\"type\":\"string\",\"enum\":[\"script\"]},\"tag_override\":{\"type\":\"string\",\"description\":\"Override the script's default worker group tag\"},\"is_trigger\":{\"type\":\"boolean\",\"description\":\"If true, this script is a trigger that can start the flow\"}},\"required\":[\"type\",\"path\",\"input_transforms\"]},\"PathFlow\":{\"type\":\"object\",\"description\":\"Reference to an existing flow by path. Use this to call another flow as a subflow\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments\",\"additionalProperties\":{\"$ref\":\"#/components/schemas/InputTransform\"}},\"path\":{\"type\":\"string\",\"description\":\"Path to the flow in the workspace (e.g., 'f/flows/process_user')\"},\"type\":{\"type\":\"string\",\"enum\":[\"flow\"]}},\"required\":[\"type\",\"path\",\"input_transforms\"]},\"ForloopFlow\":{\"type\":\"object\",\"description\":\"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations\",\"properties\":{\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"iterator\":{\"description\":\"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'\",\"$ref\":\"#/components/schemas/InputTransform\"},\"skip_failures\":{\"type\":\"boolean\",\"description\":\"If true, iteration failures don't stop the loop. Failed iterations return null\"},\"type\":{\"type\":\"string\",\"enum\":[\"forloopflow\"]},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency\"},\"parallelism\":{\"description\":\"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression\",\"$ref\":\"#/components/schemas/InputTransform\"},\"squash\":{\"type\":\"boolean\"}},\"required\":[\"modules\",\"iterator\",\"skip_failures\",\"type\"]},\"WhileloopFlow\":{\"type\":\"object\",\"description\":\"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination\",\"properties\":{\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute in each iteration. Use stop_after_if to control when the loop ends\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"skip_failures\":{\"type\":\"boolean\",\"description\":\"If true, iteration failures don't stop the loop. Failed iterations return null\"},\"type\":{\"type\":\"string\",\"enum\":[\"whileloopflow\"]},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, iterations run concurrently (use with caution in while loops)\"},\"parallelism\":{\"description\":\"Maximum number of concurrent iterations when parallel=true\",\"$ref\":\"#/components/schemas/InputTransform\"},\"squash\":{\"type\":\"boolean\"}},\"required\":[\"modules\",\"skip_failures\",\"type\"]},\"BranchOne\":{\"type\":\"object\",\"description\":\"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes\",\"properties\":{\"branches\":{\"type\":\"array\",\"description\":\"Array of branches to evaluate in order. The first branch with expr evaluating to true executes\",\"items\":{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Short description of this branch condition\"},\"expr\":{\"type\":\"string\",\"description\":\"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins\"},\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute if this branch's expr is true\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}}},\"required\":[\"modules\",\"expr\"]}},\"default\":{\"type\":\"array\",\"description\":\"Steps to execute if no branch expressions match\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}},\"type\":{\"type\":\"string\",\"enum\":[\"branchone\"]}},\"required\":[\"branches\",\"default\",\"type\"]},\"BranchAll\":{\"type\":\"object\",\"description\":\"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently\",\"properties\":{\"branches\":{\"type\":\"array\",\"description\":\"Array of branches that all execute (either in parallel or sequentially)\",\"items\":{\"type\":\"object\",\"properties\":{\"summary\":{\"type\":\"string\",\"description\":\"Short description of this branch's purpose\"},\"skip_failure\":{\"type\":\"boolean\",\"description\":\"If true, failure in this branch doesn't fail the entire flow\"},\"modules\":{\"type\":\"array\",\"description\":\"Steps to execute in this branch\",\"items\":{\"$ref\":\"#/components/schemas/FlowModule\"}}},\"required\":[\"modules\"]}},\"type\":{\"type\":\"string\",\"enum\":[\"branchall\"]},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, all branches execute concurrently. If false, they execute sequentially\"}},\"required\":[\"branches\",\"type\"]},\"AgentTool\":{\"type\":\"object\",\"description\":\"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool\",\"properties\":{\"id\":{\"type\":\"string\",\"description\":\"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')\"},\"summary\":{\"type\":\"string\",\"description\":\"Short description of what this tool does (shown to the AI)\"},\"value\":{\"$ref\":\"#/components/schemas/ToolValue\"}},\"required\":[\"id\",\"value\"]},\"ToolValue\":{\"description\":\"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference\",\"oneOf\":[{\"$ref\":\"#/components/schemas/FlowModuleTool\"},{\"$ref\":\"#/components/schemas/McpToolValue\"},{\"$ref\":\"#/components/schemas/WebsearchToolValue\"}],\"discriminator\":{\"propertyName\":\"tool_type\",\"mapping\":{\"flowmodule\":\"#/components/schemas/FlowModuleTool\",\"mcp\":\"#/components/schemas/McpToolValue\",\"websearch\":\"#/components/schemas/WebsearchToolValue\"}}},\"FlowModuleTool\":{\"description\":\"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module\",\"allOf\":[{\"type\":\"object\",\"properties\":{\"tool_type\":{\"type\":\"string\",\"enum\":[\"flowmodule\"]}},\"required\":[\"tool_type\"]},{\"$ref\":\"#/components/schemas/FlowModuleValue\"}]},\"WebsearchToolValue\":{\"type\":\"object\",\"description\":\"A tool implemented as a websearch tool. The AI can call this like any other websearch tool\",\"properties\":{\"tool_type\":{\"type\":\"string\",\"enum\":[\"websearch\"]}},\"required\":[\"tool_type\"]},\"McpToolValue\":{\"type\":\"object\",\"description\":\"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers\",\"properties\":{\"tool_type\":{\"type\":\"string\",\"enum\":[\"mcp\"]},\"resource_path\":{\"type\":\"string\",\"description\":\"Path to the MCP resource/server configuration\"},\"include_tools\":{\"type\":\"array\",\"description\":\"Whitelist of specific tools to include from this MCP server\",\"items\":{\"type\":\"string\"}},\"exclude_tools\":{\"type\":\"array\",\"description\":\"Blacklist of tools to exclude from this MCP server\",\"items\":{\"type\":\"string\"}}},\"required\":[\"tool_type\",\"resource_path\"]},\"AiAgent\":{\"type\":\"object\",\"description\":\"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task\",\"properties\":{\"input_transforms\":{\"type\":\"object\",\"description\":\"Input parameters for the AI agent mapped to their values\",\"properties\":{\"provider\":{\"$ref\":\"#/components/schemas/ProviderTransform\"},\"output_type\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n\"},\"user_message\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax.\"},\"system_prompt\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"System instructions that guide the AI's behavior, persona, and response style. Optional.\"},\"streaming\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n\"},\"memory\":{\"$ref\":\"#/components/schemas/MemoryTransform\"},\"output_schema\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n\"},\"user_attachments\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n\"},\"max_completion_tokens\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n\"},\"temperature\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n\"},\"max_iterations\":{\"allOf\":[{\"$ref\":\"#/components/schemas/InputTransform\"}],\"description\":\"Number. Limits how many times the agent can loop through reasoning and tool use.\\nRange: 1-1000.\\n\"}},\"required\":[\"provider\",\"user_message\",\"output_type\"]},\"tools\":{\"type\":\"array\",\"description\":\"Array of tools the agent can use. The agent decides which tools to call based on the task\",\"items\":{\"$ref\":\"#/components/schemas/AgentTool\"}},\"type\":{\"type\":\"string\",\"enum\":[\"aiagent\"]},\"omit_output_from_conversation\":{\"type\":\"boolean\",\"default\":false,\"description\":\"If true, this AI agent step does not persist its assistant or tool messages to the flow conversation when chat mode is enabled.\"},\"parallel\":{\"type\":\"boolean\",\"description\":\"If true, the agent can execute multiple tool calls in parallel\"}},\"required\":[\"tools\",\"type\",\"input_transforms\"]},\"Identity\":{\"type\":\"object\",\"description\":\"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"identity\"]},\"flow\":{\"type\":\"boolean\",\"description\":\"If true, marks this as a flow identity (special handling)\"}},\"required\":[\"type\"]},\"FlowStatus\":{\"type\":\"object\",\"properties\":{\"step\":{\"type\":\"integer\"},\"modules\":{\"type\":\"array\",\"items\":{\"$ref\":\"#/components/schemas/FlowStatusModule\"}},\"user_states\":{\"additionalProperties\":true},\"preprocessor_module\":{\"allOf\":[{\"$ref\":\"#/components/schemas/FlowStatusModule\"}]},\"failure_module\":{\"allOf\":[{\"$ref\":\"#/components/schemas/FlowStatusModule\"},{\"type\":\"object\",\"properties\":{\"parent_module\":{\"type\":\"string\"}}}]},\"retry\":{\"type\":\"object\",\"properties\":{\"fail_count\":{\"type\":\"integer\"},\"failed_jobs\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"format\":\"uuid\"}}}}},\"required\":[\"step\",\"modules\",\"failure_module\"]},\"FlowStatusModule\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"WaitingForPriorSteps\",\"WaitingForEvents\",\"WaitingForExecutor\",\"InProgress\",\"Success\",\"Failure\"]},\"id\":{\"type\":\"string\"},\"job\":{\"type\":\"string\",\"format\":\"uuid\"},\"count\":{\"type\":\"integer\"},\"progress\":{\"type\":\"integer\"},\"iterator\":{\"type\":\"object\",\"properties\":{\"index\":{\"type\":\"integer\"},\"itered\":{\"type\":\"array\",\"items\":{}},\"itered_len\":{\"type\":\"integer\"},\"args\":{}}},\"flow_jobs\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"flow_jobs_success\":{\"type\":\"array\",\"items\":{\"type\":\"boolean\"}},\"flow_jobs_duration\":{\"type\":\"object\",\"properties\":{\"started_at\":{\"type\":\"array\",\"items\":{\"type\":\"string\"}},\"duration_ms\":{\"type\":\"array\",\"items\":{\"type\":\"integer\"}}}},\"branch_chosen\":{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"branch\",\"default\"]},\"branch\":{\"type\":\"integer\"}},\"required\":[\"type\"]},\"branchall\":{\"type\":\"object\",\"properties\":{\"branch\":{\"type\":\"integer\"},\"len\":{\"type\":\"integer\"}},\"required\":[\"branch\",\"len\"]},\"approvers\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"resume_id\":{\"type\":\"integer\"},\"approver\":{\"type\":\"string\"}},\"required\":[\"resume_id\",\"approver\"]}},\"failed_retries\":{\"type\":\"array\",\"items\":{\"type\":\"string\",\"format\":\"uuid\"}},\"skipped\":{\"type\":\"boolean\"},\"agent_actions\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"oneOf\":[{\"type\":\"object\",\"properties\":{\"job_id\":{\"type\":\"string\",\"format\":\"uuid\"},\"function_name\":{\"type\":\"string\"},\"type\":{\"type\":\"string\",\"enum\":[\"tool_call\"]},\"module_id\":{\"type\":\"string\"}},\"required\":[\"job_id\",\"function_name\",\"type\",\"module_id\"]},{\"type\":\"object\",\"properties\":{\"call_id\":{\"type\":\"string\",\"format\":\"uuid\"},\"function_name\":{\"type\":\"string\"},\"resource_path\":{\"type\":\"string\"},\"type\":{\"type\":\"string\",\"enum\":[\"mcp_tool_call\"]},\"arguments\":{\"type\":\"object\"}},\"required\":[\"call_id\",\"function_name\",\"resource_path\",\"type\"]},{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"web_search\"]}},\"required\":[\"type\"]},{\"type\":\"object\",\"properties\":{\"type\":{\"type\":\"string\",\"enum\":[\"message\"]}},\"required\":[\"content\",\"type\"]}]}},\"agent_actions_success\":{\"type\":\"array\",\"items\":{\"type\":\"boolean\"}}},\"required\":[\"type\"]}}"; -export declare const CLI_COMMANDS = "# Windmill CLI Commands\n\nThe Windmill CLI (`wmill`) provides commands for managing scripts, flows, apps, and other resources.\n\n## Global Options\n\n- `--workspace ` - Specify the target workspace. This overrides the default workspace.\n- `--debug --verbose` - Show debug/verbose logs\n- `--show-diffs` - Show diff informations when syncing (may show sensitive informations)\n- `--token ` - Specify an API token. This will override any stored token.\n- `--base-url ` - Specify the base URL of the API. If used, --token and --workspace are required and no local remote/workspace already set will be used.\n- `--config-dir ` - Specify a custom config directory. Overrides WMILL_CONFIG_DIR environment variable and default ~/.config location.\n\n## Commands\n\n### app\n\napp related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `app list` - list all apps\n - `--json` - Output as JSON (for piping to jq)\n- `app get ` - get an app's details\n - `--json` - Output as JSON (for piping to jq)\n- `app push [file_path:string] [remote_path:string]` - push a local app. With no args, infers the app from the current directory and the remote path from its location relative to wmill.yaml.\n- `app dev [app_folder:string]` - Start a development server for building apps with live reload and hot module replacement\n - `--port ` - Port to run the dev server on (will find next available port if occupied)\n - `--host ` - Host to bind the dev server to\n - `--entry ` - Entry point file (default: index.ts for Svelte/Vue, index.tsx otherwise)\n - `--no-open` - Don't automatically open the browser\n- `app lint [app_folder:string]` - Lint a raw app folder to validate structure and buildability\n - `--fix` - Attempt to fix common issues (not implemented yet)\n- `app new` - create a new raw app from a template\n - `--summary ` - App summary (short description). Skips the prompt when provided. Triggers non-interactive mode.\n - `--path ` - App path (e.g., f/folder/my_app or u/username/my_app). Skips the prompt when provided. Triggers non-interactive mode.\n - `--framework ` - Framework template: react19 | react18 | svelte5 | vue. Skips the prompt when provided. Triggers non-interactive mode.\n - `--datatable ` - Datatable to wire up. Without this flag in non-interactive mode, no datatable is configured.\n - `--schema ` - Schema to use with --datatable. Created (CREATE SCHEMA IF NOT EXISTS) if it doesn't already exist.\n - `--overwrite` - Overwrite the target directory if it already exists, without prompting.\n - `--no-open-in-desktop` - Do not prompt to open the new app in Claude Desktop.\n- `app generate-agents [app_folder:string]` - regenerate AGENTS.md and DATATABLES.md from remote workspace\n- `app set-permissioned-as ` - Set the on_behalf_of_email for an app (requires admin or wm_deployers group)\n\n### audit\n\nView audit logs (requires admin)\n\n**Subcommands:**\n\n- `audit list` - List audit log entries\n- `audit get ` - Get a specific audit log entry\n - `--json` - Output as JSON (for piping to jq)\n\n### config\n\nShow all available wmill.yaml configuration options\n\n**Options:**\n- `--json` - Output as JSON for programmatic consumption\n\n**Subcommands:**\n\n- `config migrate` - Migrate wmill.yaml from gitBranches/environments to workspaces format\n\n### datatable\n\ndatatable related commands\n\n**Subcommands:**\n\n- `datatable list` - list all datatables in the workspace\n - `--json` - Output as JSON (for piping to jq)\n- `datatable run ` - run a SQL query on a datatable\n - `-n --name ` - Datatable name (default: main)\n - `-s --silent` - Output only the final result as JSON. Useful for scripting.\n- `datatable serve` - Serve all datatables as a Postgres-wire endpoint (psql, DBeaver, pgAdmin); the client picks the datatable via the database name in its connection string\n - `--port ` - Port to listen on (default: first free port in 5433-5500)\n - `--host ` - Bind address (default: 127.0.0.1)\n - `--password ` - Password for Postgres clients (default: generate a random password at startup)\n- `datatable psql` - Start a serve listener and launch psql connected to it\n - `-n --name ` - Datatable to connect psql to (default: main)\n - `--port ` - Port the proxy listens on (default: first free port in 5433-5500)\n - `--host ` - Bind address for the proxy (default: 127.0.0.1)\n - `--password ` - Password for the temporary Postgres proxy (default: generate a random password at startup)\n\n### dependencies\n\nworkspace dependencies related commands\n\n**Alias:** `deps`\n\n**Subcommands:**\n\n- `dependencies push ` - Push workspace dependencies from a local file\n\n### dev\n\nWatch local file changes and live-reload the dev page for preview. Does NOT deploy to the remote workspace \u2014 use wmill sync push for that.\n\n**Options:**\n- `--includes ` - Filter paths given a glob pattern or path\n- `--proxy-port ` - Port for a localhost reverse proxy to the remote Windmill server\n- `--path ` - Watch a specific windmill path (e.g., u/admin/my_script or f/my_flow)\n- `--no-open` - Do not open the browser automatically\n\n### docs\n\nSearch Windmill documentation.\n\n**Arguments:** ``\n\n**Options:**\n- `--json` - Output results as JSON.\n\n### ducklake\n\nducklake related commands\n\n**Subcommands:**\n\n- `ducklake list` - list all ducklakes in the workspace\n - `--json` - Output as JSON (for piping to jq)\n- `ducklake run ` - run a SQL query on a ducklake\n - `-n --name ` - Ducklake name (default: main)\n - `-s --silent` - Output only the final result as JSON. Useful for scripting.\n\n### flow\n\nflow related commands\n\n**Options:**\n- `--show-archived` - Enable archived flows in output\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `flow list` - list all flows\n - `--show-archived` - Enable archived flows in output\n - `--json` - Output as JSON (for piping to jq)\n- `flow get ` - get a flow's details\n - `--json` - Output as JSON (for piping to jq)\n- `flow push ` - push a local flow spec. This overrides any remote versions.\n - `--message ` - Deployment message\n- `flow run ` - run a flow by path.\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not ouput anything other then the final output. Useful for scripting.\n- `flow preview ` - preview a local flow without deploying it. Runs the flow definition from local files and uses local PathScripts by default.\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not output anything other then the final output. Useful for scripting.\n - `--remote` - Use deployed workspace scripts for PathScript steps instead of local files.\n- `flow new ` - create a new empty flow\n - `--summary ` - flow summary\n - `--description ` - flow description\n- `flow bootstrap ` - create a new empty flow (alias for new)\n - `--summary ` - flow summary\n - `--description ` - flow description\n- `flow history ` - Show version history for a flow\n - `--json` - Output as JSON (for piping to jq)\n- `flow show-version ` - Show a specific version of a flow\n - `--json` - Output as JSON (for piping to jq)\n- `flow set-permissioned-as ` - Set the on_behalf_of_email for a flow (requires admin or wm_deployers group)\n\n### folder\n\nfolder related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `folder list` - list all folders\n - `--json` - Output as JSON (for piping to jq)\n- `folder get ` - get a folder's details\n - `--json` - Output as JSON (for piping to jq)\n- `folder new ` - create a new folder locally\n - `--summary ` - folder summary\n- `folder push ` - push a local folder to the remote by name. This overrides any remote versions.\n- `folder add-missing` - create default folder.meta.yaml for all subdirectories of f/ that are missing one\n - `-y, --yes` - skip confirmation prompt\n- `folder show-rules ` - Show default_permissioned_as rules for a folder. Use --test-path to see which rule matches a given item path.\n - `--test-path ` - Test which rule matches this item path (e.g. f/prod/jobs/my_script)\n - `--json` - Output as JSON\n\n### generate-metadata\n\nGenerate metadata (locks, schemas) for all scripts, flows, and apps\n\n**Arguments:** `[folder:string]`\n\n**Options:**\n- `--yes` - Skip confirmation prompt\n- `--dry-run` - Show what would be updated without making changes\n- `--lock-only` - Re-generate only the lock files\n- `--schema-only` - Re-generate only script schemas (skips flows and apps)\n- `--skip-scripts` - Skip processing scripts\n- `--skip-flows` - Skip processing flows\n- `--skip-apps` - Skip processing apps\n- `--strict-folder-boundaries` - Only update items inside the specified folder (requires folder argument)\n- `--parallel ` - Number of items to process in parallel\n- `-i --includes ` - Comma separated patterns to specify which files to include\n- `-e --excludes ` - Comma separated patterns to specify which files to exclude\n\n**Subcommands:**\n\n- `generate-metadata rehash [folder:string]`\n - `--skip-scripts` - Skip processing scripts\n - `--skip-flows` - Skip processing flows\n - `--skip-apps` - Skip processing apps\n - `--parallel ` - Number of items to process in parallel\n - `-i --includes ` - Comma separated patterns to specify which files to include\n - `-e --excludes ` - Comma separated patterns to specify which files to exclude\n\n### gitsync-settings\n\nManage git-sync settings between local wmill.yaml and Windmill backend\n\n**Subcommands:**\n\n- `gitsync-settings pull` - Pull git-sync settings from Windmill backend to local wmill.yaml\n - `--repository ` - Specify repository path (e.g., u/user/repo)\n - `--default` - Write settings to top-level defaults instead of overrides\n - `--replace` - Replace existing settings (non-interactive mode)\n - `--override` - Add branch-specific override (non-interactive mode)\n - `--diff` - Show differences without applying changes\n - `--json-output` - Output in JSON format\n - `--with-backend-settings ` - Use provided JSON settings instead of querying backend (for testing)\n - `--yes` - Skip interactive prompts and use default behavior\n - `--promotion ` - Use promotionOverrides from the specified branch instead of regular overrides\n- `gitsync-settings push` - Push git-sync settings from local wmill.yaml to Windmill backend\n - `--repository ` - Specify repository path (e.g., u/user/repo)\n - `--diff` - Show what would be pushed without applying changes\n - `--json-output` - Output in JSON format\n - `--with-backend-settings ` - Use provided JSON settings instead of querying backend (for testing)\n - `--yes` - Skip interactive prompts and use default behavior\n - `--promotion ` - Use promotionOverrides from the specified branch instead of regular overrides\n\n### group\n\nManage workspace groups\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `group list` - List all groups in the workspace\n - `--json` - Output as JSON (for piping to jq)\n- `group get ` - Get group details and members\n - `--json` - Output as JSON (for piping to jq)\n- `group create ` - Create a new group\n - `--summary ` - Group summary/description\n- `group delete ` - Delete a group\n- `group add-user ` - Add a user to a group\n- `group remove-user ` - Remove a user from a group\n\n### hub\n\nHub related commands. EXPERIMENTAL. INTERNAL USE ONLY.\n\n**Subcommands:**\n\n- `hub pull` - pull any supported definitions. EXPERIMENTAL.\n\n### init\n\nBootstrap a windmill project with a wmill.yaml file\n\n**Options:**\n- `--use-default` - Use default settings without checking backend\n- `--use-backend` - Use backend git-sync settings if available\n- `--repository ` - Specify repository path (e.g., u/user/repo) when using backend settings\n- `--bind-profile` - Automatically bind active workspace profile to current Git branch\n- `--no-bind-profile` - Skip workspace profile binding prompt\n\n### instance\n\nsync local with a remote instance or the opposite (push or pull)\n\n**Subcommands:**\n\n- `instance add [instance_name:string] [remote:string] [token:string]` - Add a new instance\n- `instance remove ` - Remove an instance\n- `instance switch ` - Switch the current instance\n- `instance pull` - Pull instance settings, users, configs, instance groups and overwrite local\n - `--yes` - Pull without needing confirmation\n - `--dry-run` - Perform a dry run without making changes\n - `--skip-users` - Skip pulling users\n - `--skip-settings` - Skip pulling settings\n - `--skip-configs` - Skip pulling configs (worker groups)\n - `--skip-groups` - Skip pulling instance groups\n - `--include-workspaces` - Also pull workspaces\n - `--folder-per-instance` - Create a folder per instance\n - `--instance ` - Name of the instance to pull from, override the active instance\n - `--prefix ` - Prefix of the local workspaces to pull, used to create the folders when using --include-workspaces\n - `--prefix-settings` - Store instance yamls inside prefixed folders when using --prefix and --folder-per-instance\n- `instance push` - Push instance settings, users, configs, group and overwrite remote\n - `--yes` - Push without needing confirmation\n - `--dry-run` - Perform a dry run without making changes\n - `--skip-users` - Skip pushing users\n - `--skip-settings` - Skip pushing settings\n - `--skip-configs` - Skip pushing configs (worker groups)\n - `--skip-groups` - Skip pushing instance groups\n - `--include-workspaces` - Also push workspaces\n - `--folder-per-instance` - Create a folder per instance\n - `--instance ` - Name of the instance to push to, override the active instance\n - `--prefix ` - Prefix of the local workspaces folders to push\n - `--prefix-settings` - Store instance yamls inside prefixed folders when using --prefix and --folder-per-instance\n- `instance whoami` - Display information about the currently logged-in user\n- `instance get-config` - Dump the current instance config (global settings + worker configs) as YAML\n - `-o, --output-file ` - Write YAML to a file instead of stdout\n - `--show-secrets` - Include sensitive fields (license key, JWT secret) without prompting\n - `--instance ` - Name of the instance, override the active instance\n- `instance connect-slack`\n - `--bot-token ` - Slack bot token (xoxb-...)\n - `--team-id ` - Slack team id\n - `--team-name ` - Slack team name\n - `--instance ` - Instance profile to connect against (defaults to the active instance)\n\n### job\n\nManage jobs (list, inspect, cancel)\n\n**Subcommands:**\n\n- `job list` - List recent jobs\n- `job get ` - Get job details. For flows: shows step tree with sub-job IDs\n - `--json` - Output as JSON (for piping to jq)\n- `job result ` - Get the result of a completed job (machine-friendly)\n- `job logs ` - Get job logs. For flows: aggregates all step logs\n- `job cancel ` - Cancel a running or queued job\n - `--reason ` - Reason for cancellation\n- `job rerun ` - Re-run a completed job with the same args. Prints the new job UUID on stdout.\n- `job restart ` - Restart a completed flow at a given top-level step. Prints the new flow job UUID on stdout.\n - `--step ` - Top-level step id to restart the flow from\n - `--iteration ` - For a top-level branchall or for-loop step, the iteration to restart at\n\n### jobs\n\nPull completed and queued jobs from workspace\n\n**Arguments:** `[workspace:string]`\n\n**Options:**\n- `-c, --completed-output ` - Completed jobs output file (default: completed_jobs.json)\n- `-q, --queued-output ` - Queued jobs output file (default: queued_jobs.json)\n- `--skip-worker-check` - Skip checking for active workers before export\n\n**Subcommands:**\n\n- `jobs pull`\n- `jobs push`\n\n### lint\n\nValidate Windmill flow, schedule, and trigger YAML files in a directory\n\n**Arguments:** `[directory:string]`\n\n**Options:**\n- `--json` - Output results in JSON format\n- `--fail-on-warn` - Exit with code 1 when warnings are emitted\n- `--locks-required` - Fail if scripts or flow inline scripts that need locks have no locks\n- `-w, --watch` - Watch for file changes and re-lint automatically\n\n### protection-rules\n\n**Subcommands:**\n\n- `protection-rules pull [workspace:string]` - Pull protection rules from Windmill into protection-rules.yaml for a workspace\n - `--all` - Pull every workspace defined in wmill.yaml\n - `--dry-run` - Show what would change without writing the file\n - `--json-output` - Output in JSON format\n- `protection-rules push [workspace:string]` - Push protection rules from protection-rules.yaml to Windmill for a workspace (full reconcile: creates, updates, and deletes)\n - `--all` - Push every workspace defined in protection-rules.yaml\n - `--dry-run` - Show what would change without applying\n - `--json-output` - Output in JSON format\n - `--yes` - Skip the confirmation prompt (including deletions)\n\n### queues\n\nList all queues with their metrics\n\n**Arguments:** `[workspace:string] the optional workspace to filter by (default to all workspaces)`\n\n**Options:**\n- `--instance [instance]` - Name of the instance to push to, override the active instance\n- `--base-url [baseUrl]` - If used with --token, will be used as the base url for the instance\n\n### refresh\n\nRefresh wmill-managed project files (AGENTS.cli.md and skills)\n\n**Subcommands:**\n\n- `refresh prompts` - Refresh AGENTS.cli.md and managed skills. User-owned AGENTS.md and CLAUDE.md are never overwritten unless you opt in.\n - `--yes` - Non-interactive: skip the migration prompt for existing AGENTS.md / CLAUDE.md without the expected include; defaults to appending the include.\n\n### resource\n\nresource related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `resource list` - list all resources\n - `--json` - Output as JSON (for piping to jq)\n- `resource get ` - get a resource's details\n - `--json` - Output as JSON (for piping to jq)\n- `resource new ` - create a new resource locally\n- `resource push ` - push a local resource spec. This overrides any remote versions.\n\n### resource-type\n\nresource type related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `resource-type list` - list all resource types\n - `--schema` - Show schema in the output\n - `--json` - Output as JSON (for piping to jq)\n- `resource-type get ` - get a resource type's details\n - `--json` - Output as JSON (for piping to jq)\n- `resource-type new ` - create a new resource type locally\n- `resource-type push ` - push a local resource spec. This overrides any remote versions.\n- `resource-type generate-namespace` - Create a TypeScript definition file with the RT namespace generated from the resource types\n\n### schedule\n\nschedule related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `schedule list` - list all schedules\n - `--json` - Output as JSON (for piping to jq)\n- `schedule get ` - get a schedule's details\n - `--json` - Output as JSON (for piping to jq)\n- `schedule new ` - create a new schedule locally\n- `schedule push ` - push a local schedule spec. This overrides any remote versions.\n- `schedule enable ` - Enable a schedule\n - `--force` - Bypass the fork-conflict warning when the parent workspace has the same schedule (acknowledges that both crons will fire)\n- `schedule disable ` - Disable a schedule\n- `schedule set-permissioned-as ` - Set the email (run-as user) for a schedule (requires admin or wm_deployers group)\n\n### script\n\nscript related commands\n\n**Options:**\n- `--show-archived` - Show archived scripts instead of active ones\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `script list` - list all scripts\n - `--show-archived` - Show archived scripts instead of active ones\n - `--json` - Output as JSON (for piping to jq)\n- `script push ` - push a local script spec. This overrides any remote versions. Use the script file (.ts, .js, .py, .sh)\n - `--message ` - Deployment message\n- `script get ` - get a script's details\n - `--json` - Output as JSON (for piping to jq)\n- `script show ` - show a script's content (alias for get)\n- `script run ` - run a script by path\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not output anything other then the final output. Useful for scripting.\n- `script preview ` - preview a local script without deploying it. Supports both regular and codebase scripts.\n - `-d --data ` - Inputs specified as a JSON string or a file using @ or stdin using @-.\n - `-s --silent` - Do not output anything other than the final output. Useful for scripting.\n- `script new ` - create a new script\n - `--summary ` - script summary\n - `--description ` - script description\n- `script bootstrap ` - create a new script (alias for new)\n - `--summary ` - script summary\n - `--description ` - script description\n- `script set-permissioned-as ` - Set the on_behalf_of_email for a script (requires admin or wm_deployers group)\n- `script history ` - show version history for a script\n - `--json` - Output as JSON (for piping to jq)\n\n### sync\n\nsync local with a remote workspaces or the opposite (push or pull)\n\n**Subcommands:**\n\n- `sync pull` - Pull any remote changes and apply them locally.\n - `--yes` - Pull without needing confirmation\n - `--dry-run` - Show changes that would be pulled without actually pushing\n - `--plain-secrets` - Pull secrets as plain text\n - `--json` - Use JSON instead of YAML\n - `--skip-variables` - Skip syncing variables (including secrets)\n - `--skip-secrets` - Skip syncing only secrets variables\n - `--include-secrets` - Include secrets in sync (overrides skipSecrets in wmill.yaml)\n - `--skip-resources` - Skip syncing resources\n - `--skip-resource-types` - Skip syncing resource types\n - `--skip-scripts` - Skip syncing scripts\n - `--skip-flows` - Skip syncing flows\n - `--skip-apps` - Skip syncing apps\n - `--skip-folders` - Skip syncing folders\n - `--skip-workspace-dependencies` - Skip syncing workspace dependencies\n - `--skip-scripts-metadata` - Skip syncing scripts metadata, focus solely on logic\n - `--include-schedules` - Include syncing schedules\n - `--include-triggers` - Include syncing triggers\n - `--include-users` - Include syncing users\n - `--include-groups` - Include syncing groups\n - `--include-settings` - Include syncing workspace settings\n - `--include-key` - Include workspace encryption key\n - `--skip-branch-validation` - Skip git branch validation and prompts\n - `--json-output` - Output results in JSON format\n - `-i --includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Overrides wmill.yaml includes\n - `-e --excludes ` - Comma separated patterns to specify which file to NOT take into account. Overrides wmill.yaml excludes\n - `--extra-includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Useful to still take wmill.yaml into account and act as a second pattern to satisfy\n - `--repository ` - Specify repository path (e.g., u/user/repo) when multiple repositories exist\n - `--promotion ` - Use promotionOverrides from the specified branch instead of regular overrides\n - `--branch, --env ` - [Deprecated: use --workspace] Override the current git branch/environment\n- `sync push` - Push any local changes and apply them remotely.\n - `--yes` - Push without needing confirmation\n - `--dry-run` - Show changes that would be pushed without actually pushing\n - `--plain-secrets` - Push secrets as plain text\n - `--json` - Use JSON instead of YAML\n - `--skip-variables` - Skip syncing variables (including secrets)\n - `--skip-secrets` - Skip syncing only secrets variables\n - `--include-secrets` - Include secrets in sync (overrides skipSecrets in wmill.yaml)\n - `--skip-resources` - Skip syncing resources\n - `--skip-resource-types` - Skip syncing resource types\n - `--skip-scripts` - Skip syncing scripts\n - `--skip-flows` - Skip syncing flows\n - `--skip-apps` - Skip syncing apps\n - `--skip-folders` - Skip syncing folders\n - `--skip-workspace-dependencies` - Skip syncing workspace dependencies\n - `--skip-scripts-metadata` - Skip syncing scripts metadata, focus solely on logic\n - `--include-schedules` - Include syncing schedules\n - `--include-triggers` - Include syncing triggers\n - `--include-users` - Include syncing users\n - `--include-groups` - Include syncing groups\n - `--include-settings` - Include syncing workspace settings\n - `--include-key` - Include workspace encryption key\n - `--skip-branch-validation` - Skip git branch validation and prompts\n - `--json-output` - Output results in JSON format\n - `-i --includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string)\n - `-e --excludes ` - Comma separated patterns to specify which file to NOT take into account.\n - `--extra-includes ` - Comma separated patterns to specify which file to take into account (among files that are compatible with windmill). Patterns can include * (any string until '/') and ** (any string). Useful to still take wmill.yaml into account and act as a second pattern to satisfy\n - `--message ` - Include a message that will be added to all scripts/flows/apps updated during this push\n - `--parallel ` - Number of changes to process in parallel\n - `--repository ` - Specify repository path (e.g., u/user/repo) when multiple repositories exist\n - `--branch, --env ` - [Deprecated: use --workspace] Override the current git branch/environment\n - `--lint` - Run lint validation before pushing\n - `--locks-required` - Fail if scripts or flow inline scripts that need locks have no locks\n - `--auto-metadata` - Automatically regenerate stale metadata (locks and schemas) before pushing\n - `--accept-overriding-permissioned-as-with-self` - Accept that items with a different permissioned_as will be updated with your own user\n\n### token\n\nManage API tokens\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `token list` - List API tokens\n - `--json` - Output as JSON (for piping to jq)\n- `token create` - Create a new API token\n - `--label ` - Token label\n - `--expiration ` - Token expiration (ISO 8601 timestamp)\n- `token delete ` - Delete a token by its prefix\n\n### trigger\n\ntrigger related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `trigger list` - list all triggers\n - `--json` - Output as JSON (for piping to jq)\n- `trigger get ` - get a trigger's details\n - `--json` - Output as JSON (for piping to jq)\n - `--kind ` - Trigger kind (http, websocket, kafka, nats, postgres, mqtt, sqs, gcp, azure, email). Recommended for faster lookup\n- `trigger new ` - create a new trigger locally\n - `--kind ` - Trigger kind (required: http, websocket, kafka, nats, postgres, mqtt, sqs, gcp, azure, email)\n- `trigger push ` - push a local trigger spec. This overrides any remote versions.\n- `trigger set-permissioned-as ` - Set the email (run-as user) for a trigger (requires admin or wm_deployers group)\n - `--kind ` - Trigger kind (required: http, websocket, kafka, nats, postgres, mqtt, sqs, gcp, azure, email)\n\n### user\n\nuser related commands\n\n**Subcommands:**\n\n- `user add [password:string]` - Create a user\n - `--superadmin` - Specify to make the new user superadmin.\n - `--company ` - Specify to set the company of the new user.\n - `--name ` - Specify to set the name of the new user.\n- `user remove ` - Delete a user\n- `user create-token` - Create a new API token for the authenticated user\n - `--email ` - Specify credentials to use for authentication. This will not be stored. It will only be used to exchange for a token with the API server, which will not be stored either.\n - `--password ` - Specify credentials to use for authentication. This will not be stored. It will only be used to exchange for a token with the API server, which will not be stored either.\n\n### variable\n\nvariable related commands\n\n**Options:**\n- `--json` - Output as JSON (for piping to jq)\n\n**Subcommands:**\n\n- `variable list` - list all variables\n - `--json` - Output as JSON (for piping to jq)\n- `variable get ` - get a variable's details\n - `--json` - Output as JSON (for piping to jq)\n- `variable new ` - create a new variable locally\n- `variable push ` - Push a local variable spec. This overrides any remote versions.\n - `--plain-secrets` - Push secrets as plain text\n- `variable add ` - Create a new variable on the remote. This will update the variable if it already exists.\n - `--plain-secrets` - Push secrets as plain text\n - `--public` - Legacy option, use --plain-secrets instead\n\n### version\n\nShow version information\n\n### worker-groups\n\ndisplay worker groups, pull and push worker groups configs\n\n**Subcommands:**\n\n- `worker-groups pull` - Pull worker groups (similar to `wmill instance pull --skip-users --skip-settings --skip-groups`)\n - `--instance` - Name of the instance to push to, override the active instance\n - `--base-url` - Base url to be passed to the instance settings instead of the local one\n - `--yes` - Pull without needing confirmation\n- `worker-groups push` - Push worker groups (similar to `wmill instance push --skip-users --skip-settings --skip-groups`)\n - `--instance [instance]` - Name of the instance to push to, override the active instance\n - `--base-url [baseUrl]` - If used with --token, will be used as the base url for the instance\n - `--yes` - Push without needing confirmation\n\n### workers\n\nList all workers grouped by worker groups\n\n**Options:**\n- `--instance [instance]` - Name of the instance to push to, override the active instance\n- `--base-url [baseUrl]` - If used with --token, will be used as the base url for the instance\n\n### workspace\n\nworkspace related commands\n\n**Alias:** `profile`\n\n**Subcommands:**\n\n- `workspace switch ` - Switch to another workspace\n- `workspace add [workspace_name:string] [workspace_id:string] [remote:string]` - Add a workspace\n - `-c --create` - Create the workspace if it does not exist\n - `--create-workspace-name ` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id.\n - `--create-username ` - Specify your own username in the newly created workspace. Ignored if --create is not specified, the workspace already exists or automatic username creation is enabled on the instance.\n- `workspace remove ` - Remove a workspace\n- `workspace whoami` - Show the currently active user\n- `workspace list` - List local workspace profiles\n- `workspace list-remote` - List workspaces on the remote server that you have access to\n - `--as-superadmin` - List ALL workspaces on the instance (requires the token to belong to a superadmin/devops user)\n- `workspace list-forks` - List forked workspaces on the remote server\n- `workspace bind` - Create or update a workspace entry in wmill.yaml from the active profile\n - `--workspace ` - Workspace name (default: current branch or workspaceId)\n - `--branch ` - Git branch to associate (default: workspace name)\n- `workspace unbind` - Remove baseUrl and workspaceId from a workspace entry\n - `--workspace ` - Workspace to unbind\n- `workspace fork [workspace_name:string] [workspace_id:string]` - Create a forked workspace\n - `--create-workspace-name ` - Specify the workspace name. Ignored if --create is not specified or the workspace already exists. Will default to the workspace id.\n - `--color ` - Workspace color (hex code, e.g. #ff0000)\n - `--datatable-behavior ` - How to handle datatables: skip, schema_only, or schema_and_data (default: interactive prompt)\n - `-y --yes` - Skip interactive prompts (defaults datatable behavior to 'skip')\n- `workspace delete-fork ` - Delete a forked workspace and git branch\n - `-y --yes` - Skip confirmation prompt\n- `workspace merge` - Compare and deploy changes between a fork and its parent workspace\n - `--direction ` - Deploy direction: to-parent or to-fork\n - `--all` - Deploy all changed items including conflicts\n - `--skip-conflicts` - Skip items modified in both workspaces\n - `--include ` - Comma-separated kind:path items to include (e.g. script:f/test/main,flow:f/my/flow)\n - `--exclude ` - Comma-separated kind:path items to exclude\n - `--preserve-on-behalf-of` - Preserve original on_behalf_of/permissioned_as values\n - `-y --yes` - Non-interactive mode (deploy without prompts)\n- `workspace connect-slack` - Non-interactively connect Slack to the active workspace using a pre-minted bot token (xoxb-...). Produces the same artifacts as the UI OAuth flow: workspace_settings fields, g/slack group, f/slack_bot folder, and the encrypted bot token variable + resource at f/slack_bot/bot_token.\n - `--bot-token ` - Slack bot token (xoxb-...)\n - `--team-id ` - Slack team id\n - `--team-name ` - Slack team name\n- `workspace disconnect-slack`\n\n"; -export declare const LANG_BASH = "# Bash\n\n## Structure\n\nDo not include `#!/bin/bash`. Arguments are obtained as positional parameters:\n\n```bash\n# Get arguments\nvar1=\"$1\"\nvar2=\"$2\"\n\necho \"Processing $var1 and $var2\"\n\n# Return JSON by echoing to stdout\necho \"{\\\"result\\\": \\\"$var1\\\", \\\"count\\\": $var2}\"\n```\n\n**Important:**\n- Do not include shebang (`#!/bin/bash`)\n- Arguments are always strings\n- Access with `$1`, `$2`, etc.\n\n## Output\n\nThe script output is captured as the result. For structured data, output valid JSON:\n\n```bash\nname=\"$1\"\ncount=\"$2\"\n\n# Output JSON result\ncat << EOF\n{\n \"name\": \"$name\",\n \"count\": $count,\n \"timestamp\": \"$(date -Iseconds)\"\n}\nEOF\n```\n\n## Environment Variables\n\nEnvironment variables set in Windmill are available:\n\n```bash\n# Access environment variable\necho \"Workspace: $WM_WORKSPACE\"\necho \"Job ID: $WM_JOB_ID\"\n```\n"; -export declare const LANG_BIGQUERY = "# BigQuery\n\nArguments use `@name` syntax.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- @name1 (string)\n-- @name2 (int64) = 0\nSELECT * FROM users WHERE name = @name1 AND age > @name2;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as a `STRING` JSON parameter \u2014 Parquet/CSV\nfiles are decoded server-side into a JSON array of records, JSON/JSONL pass\nthrough. Consume with `JSON_EXTRACT_ARRAY` / `JSON_VALUE`:\n\n```sql\n-- @file (s3object)\nSELECT\n CAST(JSON_VALUE(row, '$.id') AS INT64) AS id,\n JSON_VALUE(row, '$.name') AS name\nFROM UNNEST(JSON_EXTRACT_ARRAY(@file)) AS row;\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered, bypassing the 10000-row return cap.\n"; -export declare const LANG_BUN = "# TypeScript (Bun)\n\nBun runtime with full npm ecosystem and fastest execution.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function. Libraries are installed automatically.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n```typescript\nimport Stripe from \"stripe\";\nimport { someFunction } from \"some-package\";\n```\n\n## Windmill Client\n\nImport the windmill client for platform interactions:\n\n```typescript\nimport * as wmill from \"windmill-client\";\n```\n\nSee the SDK documentation for available methods.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id,\n };\n}\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form \u2014 always use it instead of redefining your own.\n\n### Receiving an S3Object as a script parameter\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\nexport async function main(file: wmill.S3Object) {\n const content = await wmill.loadS3File(file);\n // ...\n}\n```\n\n### S3 operations\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\n// Load file content from S3\nconst content: Uint8Array = await wmill.loadS3File(s3object);\n\n// Load file as stream\nconst blob: Blob = await wmill.loadS3FileStream(s3object);\n\n// Write file to S3\nconst result: wmill.S3Object = await wmill.writeS3File(\n s3object, // Target path (or undefined to auto-generate)\n fileContent, // string or Blob\n s3ResourcePath // Optional: specific S3 resource to use\n);\n```\n"; -export declare const LANG_BUNNATIVE = "# TypeScript (Bun Native)\n\nNative TypeScript execution with fetch only - no external imports allowed.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n**No imports allowed.** Use the globally available `fetch` function:\n\n```typescript\nexport async function main(url: string) {\n const response = await fetch(url);\n return await response.json();\n}\n```\n\n## Windmill Client\n\nThe windmill client is not available in native TypeScript mode. Use fetch to call APIs directly.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id,\n };\n}\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form \u2014 always use it instead of redefining your own.\n\n### Receiving an S3Object as a script parameter\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\nexport async function main(file: wmill.S3Object) {\n const content = await wmill.loadS3File(file);\n // ...\n}\n```\n\n### S3 operations\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\n// Load file content from S3\nconst content: Uint8Array = await wmill.loadS3File(s3object);\n\n// Load file as stream\nconst blob: Blob = await wmill.loadS3FileStream(s3object);\n\n// Write file to S3\nconst result: wmill.S3Object = await wmill.writeS3File(\n s3object, // Target path (or undefined to auto-generate)\n fileContent, // string or Blob\n s3ResourcePath // Optional: specific S3 resource to use\n);\n```\n"; -export declare const LANG_CSHARP = "# C#\n\nThe script must contain a public static `Main` method inside a class:\n\n```csharp\npublic class Script\n{\n public static object Main(string name, int count)\n {\n return new { Name = name, Count = count };\n }\n}\n```\n\n**Important:**\n- Class name is irrelevant\n- Method must be `public static`\n- Return type can be `object` or specific type\n\n## NuGet Packages\n\nAdd packages using the `#r` directive at the top:\n\n```csharp\n#r \"nuget: Newtonsoft.Json, 13.0.3\"\n#r \"nuget: RestSharp, 110.2.0\"\n\nusing Newtonsoft.Json;\nusing RestSharp;\n\npublic class Script\n{\n public static object Main(string url)\n {\n var client = new RestClient(url);\n var request = new RestRequest();\n var response = client.Get(request);\n return JsonConvert.DeserializeObject(response.Content);\n }\n}\n```\n"; -export declare const LANG_DENO = "# TypeScript (Deno)\n\nDeno runtime with npm support via `npm:` prefix and native Deno libraries.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function. Libraries are installed automatically.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n```typescript\n// npm packages use npm: prefix\nimport Stripe from \"npm:stripe\";\nimport { someFunction } from \"npm:some-package\";\n\n// Deno standard library\nimport { serve } from \"https://deno.land/std/http/server.ts\";\n```\n\n## Windmill Client\n\nImport the windmill client for platform interactions:\n\n```typescript\nimport * as wmill from \"windmill-client\";\n```\n\nSee the SDK documentation for available methods.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id,\n };\n}\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations. The `wmill.S3Object` type covers both the `s3://storage/key` URI form (`s3:///key` for the workspace default storage) and the `{ s3, storage? }` record form \u2014 always use it instead of redefining your own.\n\n### Receiving an S3Object as a script parameter\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\nexport async function main(file: wmill.S3Object) {\n const content = await wmill.loadS3File(file);\n // ...\n}\n```\n\n### S3 operations\n\n```typescript\nimport * as wmill from \"windmill-client\";\n\n// Load file content from S3\nconst content: Uint8Array = await wmill.loadS3File(s3object);\n\n// Load file as stream\nconst blob: Blob = await wmill.loadS3FileStream(s3object);\n\n// Write file to S3\nconst result: wmill.S3Object = await wmill.writeS3File(\n s3object, // Target path (or undefined to auto-generate)\n fileContent, // string or Blob\n s3ResourcePath // Optional: specific S3 resource to use\n);\n```\n"; -export declare const LANG_DUCKDB = "# DuckDB\n\nArguments are defined with comments and used with `$name` syntax:\n\n```sql\n-- $name (text) = default\n-- $age (integer)\nSELECT * FROM users WHERE name = $name AND age > $age;\n```\n\n## Ducklake Integration\n\nAttach Ducklake for data lake operations:\n\n```sql\n-- Main ducklake\nATTACH 'ducklake' AS dl;\n\n-- Named ducklake\nATTACH 'ducklake://my_lake' AS dl;\n\n-- Then query\nSELECT * FROM dl.schema.table;\n```\n\n## External Database Connections\n\nConnect to external databases using resources:\n\n```sql\nATTACH '$res:path/to/resource' AS db (TYPE postgres);\nSELECT * FROM db.schema.table;\n```\n\n## S3 File Operations\n\nRead files from S3 storage:\n\n```sql\n-- Default storage\nSELECT * FROM read_csv('s3:///path/to/file.csv');\n\n-- Named storage\nSELECT * FROM read_csv('s3://storage_name/path/to/file.csv');\n\n-- Parquet files\nSELECT * FROM read_parquet('s3:///path/to/file.parquet');\n\n-- JSON files\nSELECT * FROM read_json('s3:///path/to/file.json');\n```\n\n### Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for it\nand binds the arg as the bare `s3://storage/key` URI, which DuckDB's reader\nfunctions consume directly:\n\n```sql\n-- $file (s3object)\nSELECT * FROM read_parquet($file);\n```\n\nWorks with any DuckDB reader: `read_csv($file)`, `read_json($file)`, etc.\n\n### Writing query results to S3\n\nDuckDB writes to S3 natively via `COPY ... TO`:\n\n```sql\nCOPY (SELECT * FROM users) TO 's3:///exports/users.parquet' (FORMAT PARQUET);\n```\n\nUse this instead of the `-- s3` streaming directive supported by the other SQL\ndialects \u2014 that directive is not available in DuckDB.\n"; -export declare const LANG_GO = "# Go\n\n## Structure\n\nThe file package must be `inner` and export a function called `main`:\n\n```go\npackage inner\n\nfunc main(param1 string, param2 int) (map[string]interface{}, error) {\n return map[string]interface{}{\n \"result\": param1,\n \"count\": param2,\n }, nil\n}\n```\n\n**Important:**\n- Package must be `inner`\n- Return type must be `({return_type}, error)`\n- Function name is `main` (lowercase)\n\n## Return Types\n\nThe return type can be any Go type that can be serialized to JSON:\n\n```go\npackage inner\n\ntype Result struct {\n Name string `json:\"name\"`\n Count int `json:\"count\"`\n}\n\nfunc main(name string, count int) (Result, error) {\n return Result{\n Name: name,\n Count: count,\n }, nil\n}\n```\n\n## Error Handling\n\nReturn errors as the second return value:\n\n```go\npackage inner\n\nimport \"errors\"\n\nfunc main(value int) (string, error) {\n if value < 0 {\n return \"\", errors.New(\"value must be positive\")\n }\n return \"success\", nil\n}\n```\n"; -export declare const LANG_GRAPHQL = "# GraphQL\n\n## Structure\n\nWrite GraphQL queries or mutations. Arguments can be added as query parameters:\n\n```graphql\nquery GetUser($id: ID!) {\n user(id: $id) {\n id\n name\n email\n }\n}\n```\n\n## Variables\n\nVariables are passed as script arguments and automatically bound to the query:\n\n```graphql\nquery SearchProducts($query: String!, $limit: Int = 10) {\n products(search: $query, first: $limit) {\n edges {\n node {\n id\n name\n price\n }\n }\n }\n}\n```\n\n## Mutations\n\n```graphql\nmutation CreateUser($input: CreateUserInput!) {\n createUser(input: $input) {\n id\n name\n createdAt\n }\n}\n```\n"; -export declare const LANG_JAVA = "# Java\n\nThe script must contain a Main public class with a `public static main()` method:\n\n```java\npublic class Main {\n public static Object main(String name, int count) {\n java.util.Map result = new java.util.HashMap<>();\n result.put(\"name\", name);\n result.put(\"count\", count);\n return result;\n }\n}\n```\n\n**Important:**\n- Class must be named `Main`\n- Method must be `public static Object main(...)`\n- Return type is `Object` or `void`\n\n## Maven Dependencies\n\nAdd dependencies using comments at the top:\n\n```java\n//requirements:\n//com.google.code.gson:gson:2.10.1\n//org.apache.httpcomponents:httpclient:4.5.14\n\nimport com.google.gson.Gson;\n\npublic class Main {\n public static Object main(String input) {\n Gson gson = new Gson();\n return gson.fromJson(input, Object.class);\n }\n}\n```\n"; -export declare const LANG_MSSQL = "# Microsoft SQL Server (MSSQL)\n\nArguments use `@P1`, `@P2`, etc.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- @P1 name1 (varchar)\n-- @P2 name2 (int) = 0\nSELECT * FROM users WHERE name = @P1 AND age > @P2;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as `nvarchar(max)` JSON text \u2014 Parquet/CSV\nfiles are decoded server-side into a JSON array of records, JSON/JSONL pass\nthrough. Consume with `OPENJSON`:\n\n```sql\n-- @P1 file (s3object)\nSELECT id, name\nFROM OPENJSON(@P1)\nWITH (id INT, name NVARCHAR(200));\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered as the script return value.\n"; -export declare const LANG_MYSQL = "# MySQL\n\nArguments use `?` placeholders.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- ? name1 (text)\n-- ? name2 (int) = 0\nSELECT * FROM users WHERE name = ? AND age > ?;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as JSON text \u2014 Parquet/CSV files are\ndecoded server-side into a JSON array of records, JSON/JSONL pass through.\nConsume with `JSON_TABLE`:\n\n```sql\n-- ? file (s3object)\nSELECT id, name\nFROM JSON_TABLE(?, '$[*]'\n COLUMNS (id INT PATH '$.id', name VARCHAR(200) PATH '$.name')\n) AS r;\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered as the script return value.\n"; -export declare const LANG_NATIVETS = "# TypeScript (Native)\n\nNative TypeScript execution with fetch only - no external imports allowed.\n\n## Structure\n\nExport a single **async** function called `main`:\n\n```typescript\nexport async function main(param1: string, param2: number) {\n // Your code here\n return { result: param1, count: param2 };\n}\n```\n\nDo not call the main function.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nUse the `RT` namespace for resource types:\n\n```typescript\nexport async function main(stripe: RT.Stripe) {\n // stripe contains API key and config from the resource\n}\n```\n\nOnly use resource types if you need them to satisfy the instructions. Always use the RT namespace.\n\nBefore using a resource type, check the `rt.d.ts` file in the project root to see all available resource types and their fields. This file is generated by `wmill resource-type generate-namespace`.\n\n## Imports\n\n**No imports allowed.** Use the globally available `fetch` function:\n\n```typescript\nexport async function main(url: string) {\n const response = await fetch(url);\n return await response.json();\n}\n```\n\n## Windmill Client\n\nThe windmill client is not available in native TypeScript mode. Use fetch to call APIs directly.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```typescript\ntype Event = {\n kind:\n | \"webhook\"\n | \"http\"\n | \"websocket\"\n | \"kafka\"\n | \"email\"\n | \"nats\"\n | \"postgres\"\n | \"sqs\"\n | \"mqtt\"\n | \"gcp\";\n body: any;\n headers: Record;\n query: Record;\n};\n\nexport async function preprocessor(event: Event) {\n return {\n param1: event.body.field1,\n param2: event.query.id\n };\n}\n```\n"; -export declare const LANG_PHP = "# PHP\n\n## Structure\n\nThe script must start with ` $param1, \"count\" => $param2];\n}\n```\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nYou need to **redefine** the type of the resources that are needed before the main function. Always check if the class already exists using `class_exists`:\n\n```php\n $2::INT;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as a `jsonb` parameter \u2014 Parquet/CSV files\nare decoded server-side into a JSON array of records, JSON/JSONL pass through.\nConsume with `jsonb_to_recordset` (or any `jsonb` API):\n\n```sql\n-- $1 file (s3object)\nSELECT *\nFROM jsonb_to_recordset($1::jsonb) AS r(id INT, name TEXT);\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered as the script return value.\n"; -export declare const LANG_POWERSHELL = "# PowerShell\n\n## Structure\n\nArguments are obtained by calling the `param` function on the first line:\n\n```powershell\nparam($Name, $Count = 0, [int]$Age)\n\n# Your code here\nWrite-Output \"Processing $Name, count: $Count, age: $Age\"\n\n# Return object\n@{\n name = $Name\n count = $Count\n age = $Age\n}\n```\n\n## Parameter Types\n\nYou can specify types for parameters:\n\n```powershell\nparam(\n [string]$Name,\n [int]$Count = 0,\n [bool]$Enabled = $true,\n [array]$Items\n)\n\n@{\n name = $Name\n count = $Count\n enabled = $Enabled\n items = $Items\n}\n```\n\n## Return Values\n\nReturn values by outputting them at the end of the script:\n\n```powershell\nparam($Input)\n\n$result = @{\n processed = $true\n data = $Input\n timestamp = Get-Date -Format \"o\"\n}\n\n$result\n```\n"; -export declare const LANG_PYTHON3 = "# Python\n\n## Structure\n\nThe script must contain at least one function called `main`:\n\n```python\ndef main(param1: str, param2: int):\n # Your code here\n return {\"result\": param1, \"count\": param2}\n```\n\nDo not call the main function. Libraries are installed automatically.\n\n## Resource Types\n\nOn Windmill, credentials and configuration are stored in resources and passed as parameters to main.\n\nYou need to **redefine** the type of the resources that are needed before the main function as TypedDict:\n\n```python\nfrom typing import TypedDict\n\nclass postgresql(TypedDict):\n host: str\n port: int\n user: str\n password: str\n dbname: str\n\ndef main(db: postgresql):\n # db contains the database connection details\n pass\n```\n\n**Important rules:**\n\n- The resource type name must be **IN LOWERCASE**\n- Only include resource types if they are actually needed\n- If an import conflicts with a resource type name, **rename the imported object, not the type name**\n- Make sure to import TypedDict from typing **if you're using it**\n\n## Imports\n\nLibraries are installed automatically. Do not show installation instructions.\n\n```python\nimport requests\nimport pandas as pd\nfrom datetime import datetime\n```\n\nIf an import name conflicts with a resource type:\n\n```python\n# Wrong - don't rename the type\nimport stripe as stripe_lib\nclass stripe_type(TypedDict): ...\n\n# Correct - rename the import\nimport stripe as stripe_sdk\nclass stripe(TypedDict):\n api_key: str\n```\n\n## Windmill Client\n\nImport the windmill client for platform interactions:\n\n```python\nimport wmill\n```\n\nSee the SDK documentation for available methods.\n\n## Preprocessor Scripts\n\nFor preprocessor scripts, the function should be named `preprocessor` and receives an `event` parameter:\n\n```python\nfrom typing import TypedDict, Literal, Any\n\nclass Event(TypedDict):\n kind: Literal[\"webhook\", \"http\", \"websocket\", \"kafka\", \"email\", \"nats\", \"postgres\", \"sqs\", \"mqtt\", \"gcp\"]\n body: Any\n headers: dict[str, str]\n query: dict[str, str]\n\ndef preprocessor(event: Event):\n # Transform the event into flow input parameters\n return {\n \"param1\": event[\"body\"][\"field1\"],\n \"param2\": event[\"query\"][\"id\"]\n }\n```\n\n## S3 Object Operations\n\nWindmill provides built-in support for S3-compatible storage operations.\n\n### Receiving an S3Object as a script parameter\n\nTo accept a file from S3 as input to a script, type the parameter with `S3Object` (imported from `wmill`):\n\n```python\nimport wmill\nfrom wmill import S3Object\n\ndef main(file: S3Object):\n content = wmill.load_s3_file(file)\n # ...\n```\n\n### S3 operations\n\n```python\nimport wmill\n\n# Load file content from S3\ncontent: bytes = wmill.load_s3_file(s3object)\n\n# Load file as stream reader\nreader: BufferedReader = wmill.load_s3_file_reader(s3object)\n\n# Write file to S3\nresult: S3Object = wmill.write_s3_file(\n s3object, # Target path (or None to auto-generate)\n file_content, # bytes or BufferedReader\n s3_resource_path, # Optional: specific S3 resource\n content_type, # Optional: MIME type\n content_disposition # Optional: Content-Disposition header\n)\n```\n"; -export declare const LANG_RLANG = "# R\n\n## Structure\n\nDefine a `main` function using `<-` or `=` assignment. Parameters become the script inputs:\n\n```r\nlibrary(dplyr)\nlibrary(jsonlite)\n\nmain <- function(x, name = \"default\", flag = TRUE) {\n df <- tibble(x = x, name = name)\n result <- df %>% mutate(greeting = paste(\"Hello\", name))\n return(toJSON(result, auto_unbox = TRUE))\n}\n```\n\n**Important:**\n- The `main` function is required\n- Use `library()` to load packages \u2014 they are resolved and installed automatically\n- `jsonlite` is always available (used internally for argument parsing)\n- Return values must be JSON-serializable\n\n## Parameters\n\nR types map to Windmill types:\n- `numeric` \u2192 float/int\n- `character` \u2192 string\n- `logical` \u2192 bool (use `TRUE`/`FALSE`)\n- `list` \u2192 object/dict\n- `NULL` \u2192 null\n\nDefault values are inferred from the function signature:\n\n```r\nmain <- function(\n name, # required string\n count = 10, # optional int, default 10\n verbose = FALSE # optional bool, default FALSE\n) {\n # ...\n}\n```\n\n## Resources and Variables\n\nUse the built-in Windmill helpers (no import needed):\n\n```r\nmain <- function() {\n # Get a variable\n api_key <- get_variable(\"f/my_folder/api_key\")\n\n # Get a resource (returns a list)\n db <- get_resource(\"f/my_folder/postgres_config\")\n host <- db$host\n port <- db$port\n\n return(list(host = host, port = port))\n}\n```\n\n## Output\n\nReturn any JSON-serializable value from `main`. The return value becomes the step result:\n\n```r\nmain <- function(x) {\n # Return a scalar\n return(x + 1)\n\n # Or a list (becomes JSON object)\n return(list(result = x + 1, status = \"ok\"))\n}\n```\n\n## Annotations\n\nControl execution behavior with comment annotations:\n\n```r\n#renv_verbose = true # Show verbose renv output during resolution\n#renv_install_verbose = true # Show verbose output during package installation\n#sandbox = true # Run in nsjail sandbox (requires nsjail)\n```\n"; -export declare const LANG_RUST = "# Rust\n\n## Structure\n\nThe script must contain a function called `main` with proper return type:\n\n```rust\nuse anyhow::anyhow;\nuse serde::Serialize;\n\n#[derive(Serialize, Debug)]\nstruct ReturnType {\n result: String,\n count: i32,\n}\n\nfn main(param1: String, param2: i32) -> anyhow::Result {\n Ok(ReturnType {\n result: param1,\n count: param2,\n })\n}\n```\n\n**Important:**\n- Arguments should be owned types\n- Return type must be serializable (`#[derive(Serialize)]`)\n- Return type is `anyhow::Result`\n\n## Dependencies\n\nPackages must be specified with a partial cargo.toml at the beginning of the script:\n\n```rust\n//! ```cargo\n//! [dependencies]\n//! anyhow = \"1.0.86\"\n//! reqwest = { version = \"0.11\", features = [\"json\"] }\n//! tokio = { version = \"1\", features = [\"full\"] }\n//! ```\n\nuse anyhow::anyhow;\n// ... rest of the code\n```\n\n**Note:** Serde is already included, no need to add it again.\n\n## Async Functions\n\nIf you need to handle async functions (e.g., using tokio), keep the main function sync and create the runtime inside:\n\n```rust\n//! ```cargo\n//! [dependencies]\n//! anyhow = \"1.0.86\"\n//! tokio = { version = \"1\", features = [\"full\"] }\n//! reqwest = { version = \"0.11\", features = [\"json\"] }\n//! ```\n\nuse anyhow::anyhow;\nuse serde::Serialize;\n\n#[derive(Serialize, Debug)]\nstruct Response {\n data: String,\n}\n\nfn main(url: String) -> anyhow::Result {\n let rt = tokio::runtime::Runtime::new()?;\n rt.block_on(async {\n let resp = reqwest::get(&url).await?.text().await?;\n Ok(Response { data: resp })\n })\n}\n```\n"; -export declare const LANG_SNOWFLAKE = "# Snowflake\n\nArguments use `?` placeholders.\n\nName the parameters by adding comments before the statement:\n\n```sql\n-- ? name1 (text)\n-- ? name2 (number) = 0\nSELECT * FROM users WHERE name = ? AND age > ?;\n```\n\n## Receiving an S3Object as a script parameter\n\nDeclare the arg with type `(s3object)`. Windmill renders an S3 file picker for\nit, downloads the file, and binds it as JSON text \u2014 Parquet/CSV files are\ndecoded server-side into a JSON array of records, JSON/JSONL pass through.\nWrap the bind with `PARSE_JSON(?)` and walk it with `LATERAL FLATTEN`:\n\n```sql\n-- ? file (s3object)\nSELECT\n v.value:id::NUMBER AS id,\n v.value:name::STRING AS name\nFROM LATERAL FLATTEN(input => PARSE_JSON(?)) v;\n```\n\n## Streaming query results to S3\n\nAdd a `-- s3` directive at the top of the script to stream the result set to S3\ninstead of returning rows. Windmill writes the file and returns its `S3Object`\nas the script result.\n\n```sql\n-- s3 prefix=exports/users format=parquet\nSELECT id, name FROM users;\n```\n\nAll keys are optional: `prefix` (object key prefix), `storage` (named storage \u2014\nomit to use the workspace default), `format` (`json` (default), `parquet`, or\n`csv`). Use this for large result sets \u2014 rows stream directly to S3 instead of\nbeing buffered, bypassing the 10000-row return cap.\n"; +// Auto-generated by generate.py - DO NOT EDIT + +export declare const SCRIPT_BASE: string; +export declare const FLOW_BASE: string; +export declare const RESOURCES_BASE: string; +export declare const RAW_APP_BASE: string; +export declare const WORKFLOW_AS_CODE_BASE: string; +export declare const FLOW_CHAT_SPECIAL_MODULES: string; +export declare const SDK_TYPESCRIPT: string; +export declare const SDK_PYTHON: string; +export declare const WAC_SDK_TYPESCRIPT: string; +export declare const WAC_SDK_PYTHON: string; +export declare const DATATABLE_SDK_TYPESCRIPT: string; +export declare const DATATABLE_SDK_PYTHON: string; +export declare const OPENFLOW_SCHEMA: string; +export declare const CLI_COMMANDS: string; +export declare const LANG_ANSIBLE: string; +export declare const LANG_BASH: string; +export declare const LANG_BIGQUERY: string; +export declare const LANG_BUN: string; +export declare const LANG_BUNNATIVE: string; +export declare const LANG_CSHARP: string; +export declare const LANG_DENO: string; +export declare const LANG_DUCKDB: string; +export declare const LANG_GO: string; +export declare const LANG_GRAPHQL: string; +export declare const LANG_JAVA: string; +export declare const LANG_MSSQL: string; +export declare const LANG_MYSQL: string; +export declare const LANG_PHP: string; +export declare const LANG_POSTGRESQL: string; +export declare const LANG_POWERSHELL: string; +export declare const LANG_PYTHON3: string; +export declare const LANG_RLANG: string; +export declare const LANG_RUST: string; +export declare const LANG_SNOWFLAKE: string; diff --git a/system_prompts/auto-generated/prompts.ts b/system_prompts/auto-generated/prompts.ts index c767151586..8c9ab65969 100644 --- a/system_prompts/auto-generated/prompts.ts +++ b/system_prompts/auto-generated/prompts.ts @@ -965,25 +965,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -1002,9 +1004,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -1031,25 +1034,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -1057,9 +1062,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -1560,27 +1566,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -1998,10 +2004,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -2012,10 +2019,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB @@ -3024,6 +3032,11 @@ inspect asset-driven pipelines (scripts marked \`// pipeline\`, wired by \`// on - \`--json\` - Output as JSON (for piping to jq) - \`pipeline show \` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal - \`--json\` - Output the raw asset graph as JSON +- \`pipeline run \` - run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s) + - \`--from \` - Start script (short name or path). Defaults to the folder's sole schedule/manual root. + - \`--to \` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream. + - \`--dry-run\` - Print the topological run plan without executing. + - \`--json\` - Output the plan as JSON (for piping to jq). ### protection-rules diff --git a/system_prompts/auto-generated/script.md b/system_prompts/auto-generated/script.md index bdf8cbdaaa..bdfec21036 100644 --- a/system_prompts/auto-generated/script.md +++ b/system_prompts/auto-generated/script.md @@ -1468,25 +1468,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -1505,9 +1507,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -1534,25 +1537,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -1560,9 +1565,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined @@ -2063,27 +2069,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -2501,10 +2507,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -2515,10 +2522,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB diff --git a/system_prompts/auto-generated/sdks/python.md b/system_prompts/auto-generated/sdks/python.md index e76bbe25a6..e6bb79d8d8 100644 --- a/system_prompts/auto-generated/sdks/python.md +++ b/system_prompts/auto-generated/sdks/python.md @@ -46,27 +46,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -484,10 +484,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -498,10 +499,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB diff --git a/system_prompts/auto-generated/sdks/typescript.md b/system_prompts/auto-generated/sdks/typescript.md index d9d9e531db..8557995d55 100644 --- a/system_prompts/auto-generated/sdks/typescript.md +++ b/system_prompts/auto-generated/sdks/typescript.md @@ -35,25 +35,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -72,9 +74,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -101,25 +104,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -127,9 +132,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined diff --git a/system_prompts/auto-generated/skills/cli-commands/SKILL.md b/system_prompts/auto-generated/skills/cli-commands/SKILL.md index c186e518f4..8f7486d8e9 100644 --- a/system_prompts/auto-generated/skills/cli-commands/SKILL.md +++ b/system_prompts/auto-generated/skills/cli-commands/SKILL.md @@ -431,6 +431,11 @@ inspect asset-driven pipelines (scripts marked `// pipeline`, wired by `// on ` - render a pipeline folder's DAG (sources, lineage, subscriptions) in the terminal - `--json` - Output the raw asset graph as JSON +- `pipeline run ` - run a bounded cascade: from a schedule/manual root, fan downstream up to the --to end node(s) + - `--from ` - Start script (short name or path). Defaults to the folder's sole schedule/manual root. + - `--to ` - End node(s) to stop at — script names/paths or asset URIs (e.g. datatable://main/staged). Repeatable or comma-separated. Omit to run the full downstream. + - `--dry-run` - Print the topological run plan without executing. + - `--json` - Output the plan as JSON (for piping to jq). ### protection-rules diff --git a/system_prompts/auto-generated/skills/write-script-bun/SKILL.md b/system_prompts/auto-generated/skills/write-script-bun/SKILL.md index c9ae25fc18..e6b7e3273e 100644 --- a/system_prompts/auto-generated/skills/write-script-bun/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-bun/SKILL.md @@ -206,25 +206,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -243,9 +245,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -272,25 +275,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -298,9 +303,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined diff --git a/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md b/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md index 47b49bd5b1..8ef23a996c 100644 --- a/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-bunnative/SKILL.md @@ -206,25 +206,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -243,9 +245,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -272,25 +275,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -298,9 +303,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined diff --git a/system_prompts/auto-generated/skills/write-script-deno/SKILL.md b/system_prompts/auto-generated/skills/write-script-deno/SKILL.md index ffcb2df6f7..9b1b04fa89 100644 --- a/system_prompts/auto-generated/skills/write-script-deno/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-deno/SKILL.md @@ -206,25 +206,27 @@ async getRootJobId(jobId?: string): Promise /** * @deprecated Use runScriptByPath or runScriptByHash instead */ -async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runScript(path: string | null = null, hash_: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its path and wait for the result * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByPath(path: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Run a script synchronously by its hash and wait for the result * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ -async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false): Promise +async runScriptByHash(hash_: string, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Append a text to the result stream @@ -243,9 +245,10 @@ async streamResult(stream: AsyncIterable): Promise * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ -async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false): Promise +async runFlow(path: string | null = null, args: Record | null = null, verbose: boolean = false, tag: string | null = null): Promise /** * Wait for a job to complete and return its result @@ -272,25 +275,27 @@ async getResultMaybe(jobId: string): Promise /** * @deprecated Use runScriptByPathAsync or runScriptByHashAsync instead */ -async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null): Promise +async runScriptAsync(path: string | null, hash_: string | null, args: Record | null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its path * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByPathAsync(path: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a script asynchronously by its hash * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null): Promise +async runScriptByHashAsync(hash_: string, args: Record | null = null, scheduledInSeconds: number | null = null, tag: string | null = null): Promise /** * Run a flow asynchronously by its path @@ -298,9 +303,10 @@ async runScriptByHashAsync(hash_: string, args: Record | null = nul * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ -async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true): Promise +async runFlowAsync(path: string | null, args: Record | null, scheduledInSeconds: number | null = null, // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures doNotTrackInParent: boolean = true, tag: string | null = null): Promise /** * Resolve a resource value in case the default value was picked because the input payload was undefined diff --git a/system_prompts/auto-generated/skills/write-script-python3/SKILL.md b/system_prompts/auto-generated/skills/write-script-python3/SKILL.md index 3601b7f8b3..4aee54e29a 100644 --- a/system_prompts/auto-generated/skills/write-script-python3/SKILL.md +++ b/system_prompts/auto-generated/skills/write-script-python3/SKILL.md @@ -231,27 +231,27 @@ def create_token(duration = dt.timedelta(days=1)) -> str # Create a script job and return its job id. # # .. deprecated:: Use run_script_by_path_async or run_script_by_hash_async instead. -def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_async(path: str = None, hash_: str = None, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by path and return its job id. -def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_path_async(path: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a script job by hash and return its job id. -def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None) -> str +def run_script_by_hash_async(hash_: str, args: dict = None, scheduled_in_secs: int = None, tag: str = None) -> str # Create a flow job and return its job id. -def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True) -> str +def run_flow_async(path: str, args: dict = None, scheduled_in_secs: int = None, do_not_track_in_parent: bool = True, tag: str = None) -> str # Run script synchronously and return its result. # # .. deprecated:: Use run_script_by_path or run_script_by_hash instead. -def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script(path: str = None, hash_: str = None, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by path synchronously and return its result. -def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_path(path: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run script by hash synchronously and return its result. -def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False) -> Any +def run_script_by_hash(hash_: str, args: dict = None, timeout: dt.timedelta | int | float | None = None, verbose: bool = False, cleanup: bool = True, assert_result_is_not_none: bool = False, tag: str = None) -> Any # Run a script on the current worker without creating a job. # @@ -669,10 +669,11 @@ def get_version() -> str # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Run a script synchronously by path and return its result. # @@ -683,10 +684,11 @@ def run_script_sync(hash: str, args: Dict[str, Any] = None, verbose: bool = Fals # assert_result_is_not_none: Raise exception if result is None # cleanup: Register cleanup handler to cancel job on exit # timeout: Maximum time to wait +# tag: Override the worker tag the job runs on # # Returns: # Script result -def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None) -> Any +def run_script_by_path_sync(path: str, args: Dict[str, Any] = None, verbose: bool = False, assert_result_is_not_none: bool = True, cleanup: bool = True, timeout: dt.timedelta = None, tag: str = None) -> Any # Convenient helpers that takes an S3 resource as input and returns the settings necessary to # initiate an S3 connection from DuckDB diff --git a/system_prompts/generate.py b/system_prompts/generate.py index ecae4b3bb0..6ee1971f43 100644 --- a/system_prompts/generate.py +++ b/system_prompts/generate.py @@ -689,6 +689,21 @@ def generate_ts_exports(prompts: dict[str, str]) -> str: return ts +def generate_ts_declarations(prompts: dict[str, str]) -> str: + """Generate the .d.ts for prompts.ts. + + Each export is declared as a plain `string` rather than a string-literal + type so the declaration file does not embed (and drift against) the prompt + contents — those live only in prompts.ts. + """ + dts = "// Auto-generated by generate.py - DO NOT EDIT\n\n" + + for name in prompts.keys(): + dts += f"export declare const {name}: string;\n" + + return dts + + # ============================================================================= # Schema File Generation # ============================================================================= @@ -2447,6 +2462,7 @@ def main(): # Generate TypeScript exports ts_exports = generate_ts_exports(prompts) (OUTPUT_GENERATED_DIR / "prompts.ts").write_text(ts_exports) + (OUTPUT_GENERATED_DIR / "prompts.d.ts").write_text(generate_ts_declarations(prompts)) # Generate complete script.md (all languages combined) script_md_parts = [script_base] @@ -2615,6 +2631,7 @@ export declare function getWorkflowAsCodePrompt(language?: string): string; print(f" - auto-generated/sdks/wac-python.md") print(f" - auto-generated/cli/cli-commands.md (auto-generated from CLI source)") print(f" - auto-generated/prompts.ts") + print(f" - auto-generated/prompts.d.ts") print(f" - auto-generated/index.ts") print(f" - auto-generated/script.md") print(f" - auto-generated/flow.md") diff --git a/typescript-client/client.d.ts b/typescript-client/client.d.ts index 1370a48658..1fa1ed1ca6 100644 --- a/typescript-client/client.d.ts +++ b/typescript-client/client.d.ts @@ -60,7 +60,8 @@ export declare function runScript( path?: string | null, hash_?: string | null, args?: Record | null, - verbose?: boolean + verbose?: boolean, + tag?: string | null ): Promise; export declare function waitJob(jobId: string, verbose?: boolean): Promise; export declare function getResult(jobId: string): Promise; @@ -70,7 +71,8 @@ export declare function runScriptAsync( path: string | null, hash_: string | null, args: Record | null, - scheduledInSeconds?: number | null + scheduledInSeconds?: number | null, + tag?: string | null ): Promise; /** * Resolve a resource value in case the default value was picked because the input payload was undefined diff --git a/typescript-client/client.ts b/typescript-client/client.ts index 37c1b2b35f..8b6117fef1 100644 --- a/typescript-client/client.ts +++ b/typescript-client/client.ts @@ -153,7 +153,8 @@ export async function runScript( path: string | null = null, hash_: string | null = null, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { console.warn( "runScript is deprecated. Use runScriptByPath or runScriptByHash instead." @@ -161,14 +162,15 @@ export async function runScript( if (path && hash_) { throw new Error("path and hash_ are mutually exclusive"); } - return _runScriptInternal(path, hash_, args, verbose); + return _runScriptInternal(path, hash_, args, verbose, tag); } async function _runScriptInternal( path: string | null = null, hash_: string | null = null, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { args = args || {}; @@ -183,7 +185,7 @@ async function _runScriptInternal( } } - const jobId = await _runScriptAsyncInternal(path, hash_, args); + const jobId = await _runScriptAsyncInternal(path, hash_, args, null, tag); return await waitJob(jobId, verbose); } @@ -192,14 +194,16 @@ async function _runScriptInternal( * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ export async function runScriptByPath( path: string, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { - return _runScriptInternal(path, null, args, verbose); + return _runScriptInternal(path, null, args, verbose, tag); } /** @@ -207,14 +211,16 @@ export async function runScriptByPath( * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Script execution result */ export async function runScriptByHash( hash_: string, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { - return _runScriptInternal(null, hash_, args, verbose); + return _runScriptInternal(null, hash_, args, verbose, tag); } /** @@ -240,12 +246,14 @@ export async function streamResult(stream: AsyncIterable) { * @param path - Flow path in Windmill * @param args - Arguments to pass to the flow * @param verbose - Enable verbose logging + * @param tag - Override the worker tag the job runs on * @returns Flow execution result */ export async function runFlow( path: string | null = null, args: Record | null = null, - verbose: boolean = false + verbose: boolean = false, + tag: string | null = null ): Promise { args = args || {}; @@ -253,7 +261,7 @@ export async function runFlow( console.info(`running \`${path}\` synchronously with args:`, args); } - const jobId = await runFlowAsync(path, args, null, false); + const jobId = await runFlowAsync(path, args, null, false, tag); return await waitJob(jobId, verbose); } @@ -368,7 +376,8 @@ export async function runScriptAsync( path: string | null, hash_: string | null, args: Record | null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { console.warn( "runScriptAsync is deprecated. Use runScriptByPathAsync or runScriptByHashAsync instead." @@ -377,14 +386,15 @@ export async function runScriptAsync( if (path && hash_) { throw new Error("path and hash_ are mutually exclusive"); } - return _runScriptAsyncInternal(path, hash_, args, scheduledInSeconds); + return _runScriptAsyncInternal(path, hash_, args, scheduledInSeconds, tag); } async function _runScriptAsyncInternal( path: string | null = null, hash_: string | null = null, args: Record | null = null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { // Create a script job and return its job id. args = args || {}; @@ -394,6 +404,10 @@ async function _runScriptAsyncInternal( params["scheduled_in_secs"] = scheduledInSeconds; } + if (tag) { + params["tag"] = tag; + } + let parentJobId = getEnv("WM_JOB_ID"); if (parentJobId !== undefined) { params["parent_job"] = parentJobId; @@ -431,14 +445,16 @@ async function _runScriptAsyncInternal( * @param path - Script path in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ export async function runScriptByPathAsync( path: string, args: Record | null = null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { - return _runScriptAsyncInternal(path, null, args, scheduledInSeconds); + return _runScriptAsyncInternal(path, null, args, scheduledInSeconds, tag); } /** @@ -446,14 +462,16 @@ export async function runScriptByPathAsync( * @param hash_ - Script hash in Windmill * @param args - Arguments to pass to the script * @param scheduledInSeconds - Schedule execution for a future time (in seconds) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ export async function runScriptByHashAsync( hash_: string, args: Record | null = null, - scheduledInSeconds: number | null = null + scheduledInSeconds: number | null = null, + tag: string | null = null ): Promise { - return _runScriptAsyncInternal(null, hash_, args, scheduledInSeconds); + return _runScriptAsyncInternal(null, hash_, args, scheduledInSeconds, tag); } /** @@ -462,6 +480,7 @@ export async function runScriptByHashAsync( * @param args - Arguments to pass to the flow * @param scheduledInSeconds - Schedule execution for a future time (in seconds) * @param doNotTrackInParent - If false, tracks state in parent job (only use when fully awaiting the job) + * @param tag - Override the worker tag the job runs on * @returns Job ID of the created job */ export async function runFlowAsync( @@ -471,7 +490,8 @@ export async function runFlowAsync( // can only be set to false if this the job will be fully await and not concurrent with any other job // as otherwise the child flow and its own child will store their state in the parent job which will // lead to incorrectness and failures - doNotTrackInParent: boolean = true + doNotTrackInParent: boolean = true, + tag: string | null = null ): Promise { // Create a script job and return its job id. @@ -482,6 +502,10 @@ export async function runFlowAsync( params["scheduled_in_secs"] = scheduledInSeconds; } + if (tag) { + params["tag"] = tag; + } + if (!doNotTrackInParent) { let parentJobId = getEnv("WM_JOB_ID"); if (parentJobId !== undefined) { diff --git a/typescript-client/jsr.json b/typescript-client/jsr.json index 497435b25d..7ae2023a7c 100644 --- a/typescript-client/jsr.json +++ b/typescript-client/jsr.json @@ -1,6 +1,6 @@ { "name": "@windmill/windmill", - "version": "1.734.0", + "version": "1.742.0", "exports": "./src/index.ts", "publish": { "exclude": ["!src", "./s3Types.ts", "./sqlUtils.ts", "./client.ts"] diff --git a/typescript-client/package.json b/typescript-client/package.json index 15ed703f8e..8edc248c45 100644 --- a/typescript-client/package.json +++ b/typescript-client/package.json @@ -1,7 +1,7 @@ { "name": "windmill-client", "description": "Windmill SDK client for browsers and Node.js", - "version": "1.734.0", + "version": "1.742.0", "author": "Ruben Fiszel", "license": "Apache 2.0", "homepage": "https://github.com/windmill-labs/windmill/tree/main/typescript-client#readme", diff --git a/version.txt b/version.txt index da12150d8d..bee795f5f0 100644 --- a/version.txt +++ b/version.txt @@ -1 +1 @@ -1.734.0 +1.742.0