diff --git a/.agents/skills/ai-chat/SKILL.md b/.agents/skills/ai-chat/SKILL.md new file mode 100644 index 0000000000..f670b2ffb7 --- /dev/null +++ b/.agents/skills/ai-chat/SKILL.md @@ -0,0 +1,40 @@ +--- +name: ai-chat +description: Guidance for improving the Windmill AI chat (copilot), especially global mode — tools, prompts, and context-window discipline. Use when editing chat tools, system prompts, or tool-result shapes under frontend/src/lib/components/copilot/chat, or when changing how the chat manages its context window. +--- + +## Always benchmark before and after + +No context or behavior change ships without an `ai_evals` A/B on the affected mode. +Add or adjust cases for exactly what you changed — see the `ai-evals` skill for +authoring and the full run reference. + +Run the affected mode **before** your change and **after**, same model(s), same cases. + +## Measure the window first, and cumulative second + +Optimize **`finalContextTokens`** (window occupancy — what drives overflow and +compaction), then cumulative prompt tokens. + +## Context discipline + +The dominant fixed cost is per-iteration overhead: the system prompt **plus every +tool schema** is re-sent on every loop iteration. So: + +- **Every tool and every parameter is a permanent tax.** Justify each one and measure + it; an extra "locate" round-trip can cost more than the reads it saves. Strip dead + params rather than leaving them in the schema. +- **Tool results return the minimum.** Never echo content the model already has. The + canonical mistake: a write tool that returns the whole edited artifact right after + the model authored it — return `{ success, message }` instead. When you touch a + *shared* write helper (e.g. `finishAppDraftWrite` in `global/core.ts`), re-check + this invariant for **all** the write tools routing through it — the echo has + regressed before via a shared refactor. + +## Prompts and tool descriptions are part of the surface + +The system prompt and tool descriptions steer behavior as much as the tools +themselves, and are benchmarkable the same way. A description that advertises +truncation makes the model self-limit; the path-conventions block changes where +drafts land. Treat prompt/description edits as real changes and A/B them — a +pure-prompt change is a legitimate, measurable improvement. \ No newline at end of file diff --git a/.agents/skills/ai-evals/SKILL.md b/.agents/skills/ai-evals/SKILL.md new file mode 100644 index 0000000000..ad334e2c6b --- /dev/null +++ b/.agents/skills/ai-evals/SKILL.md @@ -0,0 +1,87 @@ +--- +name: ai-evals +description: Author and run black-box benchmark cases for the Windmill AI generation modes (flow/app/script/cli/global) in ai_evals/. Use when adding or changing eval cases, or when running before/after benchmarks for AI chat / copilot changes. +--- + +# AI evals — authoring and running benchmark cases + +`ai_evals/` is a black-box benchmark runner for the Windmill AI generation modes: +`flow`, `app`, `script`, `cli`, `global`. It always tests the **current** production +prompts, tools, and guidance in this checkout. Each attempt runs the real production +path, deterministic validation, then LLM judging. + +The goal is to test current production guidance with realistic user requests — **not** +to pin one exact implementation shape. + +## Running benchmarks + +```bash +cd ai_evals +bun install # first time; frontend modes also need `cd frontend && bun install` +bun run cli -- models # list model aliases +bun run cli -- cases global # list cases for a mode +bun run cli -- run global global-test1-script-create --model sonnet +``` + +Frontend modes (`flow`/`script`/`app`/`global`) route model calls through a Windmill +backend's `/api/w//ai/proxy`, so you need **any** reachable backend: + +```bash +WMILL_AI_EVAL_BACKEND_URL=http://127.0.0.1: WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests \ + bun run cli -- run global --models sonnet,gpt-5.5,gemini-3.1-pro-preview +``` + +- **Reuse an existing workspace.** CE builds cap workspaces, so temp-workspace + creation 400s ("reached workspace limit"). Always set + `WMILL_AI_EVAL_BACKEND_WORKSPACE=integration-tests` (or any existing workspace) to + reuse one. The only side effect of a run is upserting an `f/evals/ai/` + resource there. +- Provider keys live in `ai_evals/.env` and are auto-loaded by bun. The judge is a + separate Anthropic call (default `claude-sonnet-4-6`) regardless of the model under + test. + +## Authoring core rules + +1. Write prompts like a real user request. +2. Prefer behavior, inputs, constraints, and outcomes over internal implementation. +3. Keep deterministic validation narrow and hard. +4. Put semantic expectations in `judgeChecklist`. +5. Use `expected` fixtures only when exact structure really matters. + +### Prompt writing + +Prompts should sound like something a user would naturally ask. Do not write prompts +as if the user knows Windmill internals unless the case explicitly tests a power-user +workflow. + +Good: +- "Create a flow that routes support requests based on customer tier." +- "Add a reset button that sets the counter back to 0." +- "Create a flow that reuses the existing greeting script instead of duplicating the logic." + +Bad: +- "Use `branchone` with 3 branches and a default branch." +- "Create a `rawscript` step with this exact topology." +- "This is a benchmark harness." + +### Deterministic validation + +Use deterministic checks only for hard failures: missing required files; unexpected +extra files when the prompt says not to create them; syntax errors; unresolved flow +refs; missing required special modules or suspend config; obvious corruption. + +Do **not** encode one preferred implementation. Bad hard checks: exact step topology +for a creation flow; exact branch structure when the prompt only asked for routing; +exact input shape when multiple reasonable shapes are acceptable. + +### Judge checklist + +Every non-trivial case should have a `judgeChecklist` capturing user-visible behavior +that must be present, important constraints, and key completion criteria — not +low-level implementation details unless truly required. + +Good: "the flow calculates the order total with 8% tax"; "the flow reuses the existing +workspace script instead of rewriting the logic". Bad: "uses `branchone`"; "contains a +`rawscript` node". + +See `ai_evals/README.md` for the full case format, fields, and fixture details. \ No newline at end of file diff --git a/.claude/skills/ai-chat/SKILL.md b/.claude/skills/ai-chat/SKILL.md new file mode 120000 index 0000000000..f8c527f3c9 --- /dev/null +++ b/.claude/skills/ai-chat/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-chat/SKILL.md \ No newline at end of file diff --git a/.claude/skills/ai-evals/SKILL.md b/.claude/skills/ai-evals/SKILL.md new file mode 120000 index 0000000000..8b714cb50d --- /dev/null +++ b/.claude/skills/ai-evals/SKILL.md @@ -0,0 +1 @@ +../../../.agents/skills/ai-evals/SKILL.md \ No newline at end of file diff --git a/.github/DockerfileBackendTests b/.github/DockerfileBackendTests index 88275f204b..9b29f72a64 100644 --- a/.github/DockerfileBackendTests +++ b/.github/DockerfileBackendTests @@ -28,7 +28,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /usr/local/cargo/bin/uv /usr/local/bin/uv ENV TZ=Etc/UTC diff --git a/.github/workflows/ai-agent-tests.yml b/.github/workflows/ai-agent-tests.yml new file mode 100644 index 0000000000..3a5c51f153 --- /dev/null +++ b/.github/workflows/ai-agent-tests.yml @@ -0,0 +1,132 @@ +name: AI Agent Integration Tests + +# Exercises the AI agent flow path (preview_flow with `aiagent` modules) against +# real LLM providers. Runs only when AI-agent backend code or the tests change, +# because each run makes real (paid) LLM calls. To avoid spending on every commit, +# the PR side triggers only when a PR is marked ready for review (out of draft) — +# not on `synchronize` — plus push to main and manual dispatch. +on: + workflow_dispatch: + push: + branches: [main] + paths: + - "integration_tests/ai_agent_tests/**" + - "backend/windmill-ai/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-worker/src/ai_executor.rs" + - "backend/windmill-worker/src/ai/**" + - "backend/windmill-worker/src/memory_common.rs" + - "backend/windmill-common/src/flow_conversations.rs" + - ".github/workflows/ai-agent-tests.yml" + pull_request: + types: [opened, reopened, ready_for_review] + paths: + - "integration_tests/ai_agent_tests/**" + - "backend/windmill-ai/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-worker/src/ai_executor.rs" + - "backend/windmill-worker/src/ai/**" + - "backend/windmill-worker/src/memory_common.rs" + - "backend/windmill-common/src/flow_conversations.rs" + - ".github/workflows/ai-agent-tests.yml" + +concurrency: + group: ai-agent-tests-${{ github.ref }} + cancel-in-progress: true + +jobs: + ai_agent_e2e: + # Skip draft PRs; the `opened`/`reopened` types would otherwise fire while + # still a draft. `ready_for_review` always arrives non-draft. + if: github.event_name != 'pull_request' || github.event.pull_request.draft == false + runs-on: ubicloud-standard-16 + services: + postgres: + image: postgres:16 + ports: + - 5432:5432 + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache-workspaces: backend + toolchain: 1.93.0 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.10 + + - uses: actions/setup-node@v4 + with: + node-version: "20" + + - uses: actions/setup-python@v5 + with: + python-version: "3.11" + + # CE build (no enterprise/license needed for AI agents). `quickjs` powers + # flow input-transform JS eval; `mcp` is required by the deepwiki MCP tool + # test. Bun tool scripts run via the always-on worker (BUN_PATH). + - name: Build Windmill + working-directory: ./backend + env: + SQLX_OFFLINE: true + CARGO_BUILD_JOBS: 12 + RUSTFLAGS: "" + run: cargo build --features quickjs,mcp + + - name: Start Windmill + working-directory: ./backend + env: + DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill + BUN_PATH: bun + NODE_BIN_PATH: node + RUST_LOG: info + run: | + mkdir -p ../integration_tests/logs + ./target/debug/windmill > ../integration_tests/logs/windmill.log 2>&1 & + echo "Waiting for Windmill to be ready..." + for i in $(seq 1 60); do + if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then + echo "Windmill is ready" + break + fi + sleep 2 + done + curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../integration_tests/logs/windmill.log; exit 1; } + + - name: Run AI agent integration tests + timeout-minutes: 20 + working-directory: ./integration_tests/ai_agent_tests + env: + WINDMILL_URL: http://localhost:8000 + # Only the providers we have org secrets for. Other providers + # (Azure, Bedrock, OpenRouter) are skipped by conftest when their + # keys are absent — see skip_provider_without_credentials. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GOOGLE_AI_API_KEY: ${{ secrets.GOOGLE_API_KEY }} + run: | + python -m venv .venv + .venv/bin/pip install -r requirements.txt + # The S3/vision-attachment tests need MinIO large-file storage and + # image-capable provider setup; out of scope for this cost-controlled + # smoke. Add MinIO secrets + a storage service to enable them. + .venv/bin/python -m pytest -v \ + --ignore=test_user_attachments.py \ + --ignore=test_user_images.py \ + --ignore=test_image_output.py + + - name: Archive Windmill logs + uses: actions/upload-artifact@v4 + if: always() + with: + name: ai-agent-tests-windmill-logs + path: integration_tests/logs diff --git a/.github/workflows/ai-evals-test.yml b/.github/workflows/ai-evals-test.yml new file mode 100644 index 0000000000..a255a4df94 --- /dev/null +++ b/.github/workflows/ai-evals-test.yml @@ -0,0 +1,166 @@ +name: AI Evals (global mode) + +# Smoke-tests the production global AI chat proxy/frontend execution path via +# the ai_evals harness, one case across one cheap model per provider. Runs only +# when the eval harness or the global chat code change, since each run makes real +# (paid) LLM calls. The backend is built from source purely as the AI proxy the +# harness routes model calls through; the global tools/drafts run in-process in +# the Vitest bridge against production frontend code. To avoid spending on every +# commit, the PR side triggers only when a PR is marked ready for review (out of +# draft) — not on `synchronize` — plus push to main and manual dispatch. +on: + workflow_dispatch: + push: + branches: [main] + paths: + - "ai_evals/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-ai/**" + - "frontend/src/lib/components/copilot/**" + # The eval harness runs production frontend code in-process; these are the + # AI/draft-specific deps outside copilot/ that the global smoke exercises. + - "frontend/src/lib/userDraft.svelte.ts" + - "frontend/src/lib/userDraftDbSyncer.svelte.ts" + - "frontend/src/lib/infer.ts" + - ".github/workflows/ai-evals-test.yml" + pull_request: + types: [opened, reopened, ready_for_review] + paths: + - "ai_evals/**" + - "backend/windmill-api/src/ai.rs" + - "backend/windmill-ai/**" + - "frontend/src/lib/components/copilot/**" + # The eval harness runs production frontend code in-process; these are the + # AI/draft-specific deps outside copilot/ that the global smoke exercises. + - "frontend/src/lib/userDraft.svelte.ts" + - "frontend/src/lib/userDraftDbSyncer.svelte.ts" + - "frontend/src/lib/infer.ts" + - ".github/workflows/ai-evals-test.yml" + +concurrency: + group: ai-evals-test-${{ github.ref }} + cancel-in-progress: true + +jobs: + ai_evals_global: + # Provider secrets are unavailable to forked and Dependabot PRs. + if: >- + github.event_name != 'pull_request' || + ( + github.event.pull_request.draft == false && + github.event.pull_request.head.repo.full_name == github.repository && + github.event.pull_request.user.login != 'dependabot[bot]' + ) + runs-on: ubicloud-standard-16 + services: + postgres: + image: postgres:16 + ports: + - 5432:5432 + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + steps: + - uses: actions/checkout@v4 + + - uses: actions-rust-lang/setup-rust-toolchain@v1 + with: + cache-workspaces: backend + toolchain: 1.93.0 + + - uses: oven-sh/setup-bun@v2 + with: + bun-version: 1.3.10 + + - uses: actions/setup-node@v4 + with: + # Node 22.19+ is required by the frontend's undici 8.x, which the + # Vitest bridge loads; Node 20 fails with markAsUncloneable. + node-version: "22" + + # CE build used only as the AI proxy (login, workspace, provider resource, + # /ai/proxy). No worker execution or MCP needed — global tools/drafts run + # in the Vitest bridge. quickjs matches the standard CE feature set. + - name: Build Windmill (AI proxy) + working-directory: ./backend + env: + SQLX_OFFLINE: true + CARGO_BUILD_JOBS: 12 + RUSTFLAGS: "" + run: cargo build --features quickjs + + - name: Start Windmill + working-directory: ./backend + env: + DATABASE_URL: postgres://postgres:changeme@localhost:5432/windmill + RUST_LOG: info + run: | + mkdir -p ../ai_evals/logs + ./target/debug/windmill > ../ai_evals/logs/windmill.log 2>&1 & + echo "Waiting for Windmill to be ready..." + for i in $(seq 1 60); do + if curl -sf http://localhost:8000/api/version > /dev/null 2>&1; then + echo "Windmill is ready" + break + fi + sleep 2 + done + curl -sf http://localhost:8000/api/version > /dev/null || { echo "Windmill failed to start"; tail -50 ../ai_evals/logs/windmill.log; exit 1; } + + - name: Install frontend deps + generate client + working-directory: ./frontend + run: | + npm ci + npm run generate-backend-client + + - name: Run global AI evals + timeout-minutes: 20 + working-directory: ./ai_evals + env: + WMILL_AI_EVAL_BACKEND_URL: http://localhost:8000 + WMILL_AI_EVAL_BACKEND_WORKSPACE: integration-tests + # Anthropic backs the haiku model. Google AI uses GEMINI_API_KEY. + ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }} + OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }} + GEMINI_API_KEY: ${{ secrets.GOOGLE_API_KEY }} + DEEPSEEK_API_KEY: ${{ secrets.DEEPSEEK_API_KEY }} + run: | + bun install + mkdir -p results + # One cheap model per provider (anthropic/openai/googleai/deepseek). + fail=0 + for m in haiku 4o gemini-3-flash-preview deepseek-v4-flash; do + echo "::group::global-test1-script-create ($m)" + if ! bun run cli -- run global global-test1-script-create \ + --model "$m" --execution-only --output "$PWD/results/ci-$m.json"; then + echo "$m: harness/proxy errored" + fail=1 + echo "::endgroup::" + continue + fi + # The CLI exits 0 when the harness records failed attempts, so gate + # on execution-only pass counts while ignoring model output quality. + if jq -e \ + '.attemptCount > 0 and .passedAttempts == .attemptCount' \ + "results/ci-$m.json" > /dev/null; then + echo "$m: OK — proxy/frontend execution completed" + else + echo "$m: FAILED proxy/frontend execution" + jq -c '.cases[0].attempts[0].checks' "results/ci-$m.json" || true + fail=1 + fi + echo "::endgroup::" + done + [ "$fail" = 0 ] || { echo "ai_evals global smoke failed"; exit 1; } + + - name: Archive logs and results + uses: actions/upload-artifact@v4 + if: always() + with: + name: ai-evals-global-logs + path: | + ai_evals/logs + ai_evals/results diff --git a/.github/workflows/backend-test-windows.yml b/.github/workflows/backend-test-windows.yml index 1c73e5d429..7065547b28 100644 --- a/.github/workflows/backend-test-windows.yml +++ b/.github/workflows/backend-test-windows.yml @@ -58,7 +58,9 @@ jobs: - uses: denoland/setup-deno@v2 with: - deno-version: v2.x + # Pin to the Deno version shipped in the runtime image (Dockerfile) so CI + # tests what production runs, instead of floating on the latest v2.x. + deno-version: 2.2.1 - uses: actions/setup-go@v2 with: @@ -74,7 +76,7 @@ jobs: - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: diff --git a/.github/workflows/backend-test.yml b/.github/workflows/backend-test.yml index 8f1f15447c..0be727d4bd 100644 --- a/.github/workflows/backend-test.yml +++ b/.github/workflows/backend-test.yml @@ -50,7 +50,9 @@ jobs: dotnet-version: "9.0.x" - uses: denoland/setup-deno@v2 with: - deno-version: v2.x + # Pin to the Deno version shipped in the runtime image (Dockerfile) so CI + # tests what production runs, instead of floating on the latest v2.x. + deno-version: 2.2.1 - uses: actions/setup-go@v2 with: go-version: 1.21.5 @@ -62,7 +64,7 @@ jobs: node-version: "20" - uses: astral-sh/setup-uv@v6.2.1 with: - version: "0.9.25" + version: "0.11.24" - uses: shivammathur/setup-php@v2 with: php-version: "8.3" diff --git a/.github/workflows/claude-plan.yml b/.github/workflows/claude-plan.yml index 3721ab0f17..c63e3fe1aa 100644 --- a/.github/workflows/claude-plan.yml +++ b/.github/workflows/claude-plan.yml @@ -45,7 +45,7 @@ jobs: allowed_bots: 'windmill-internal-app[bot]' trigger_phrase: '/plan' claude_args: | - --model claude-fable-5 + --model claude-opus-4-8 --system-prompt "# Claude Planning Mode You are operating in PLANNING MODE ONLY. Your role is to create detailed, structured plans without making any code changes. diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index 387485af5e..f5db652084 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -37,6 +37,44 @@ jobs: with: fetch-depth: 1 + # Make the EE source (the *_ee.rs files in the companion repo) available so the + # reviewer can see EE-only code (e.g. windmill-queue/src/jobs_ee.rs), not just the + # CE surface. The EE ref is read from the PR head's backend/ee-repo-ref.txt (via the + # API, so it reflects the PR's EE pin regardless of which ref is checked out here). + - name: Check EE access + id: ee + env: + EE_TOKEN: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} + GH_TOKEN: ${{ github.token }} + PR_NUMBER: ${{ github.event.issue.number || github.event.pull_request.number }} + run: | + if [ -z "$EE_TOKEN" ] || [ -z "$PR_NUMBER" ]; then + echo "available=false" >> "$GITHUB_OUTPUT" + exit 0 + fi + HEAD_SHA=$(gh api "repos/${{ github.repository }}/pulls/$PR_NUMBER" --jq .head.sha) + REF=$(gh api "repos/${{ github.repository }}/contents/backend/ee-repo-ref.txt?ref=$HEAD_SHA" --jq .content | base64 -d | tr -d '[:space:]') + if [ -z "$REF" ]; then + echo "available=false" >> "$GITHUB_OUTPUT" + else + echo "available=true" >> "$GITHUB_OUTPUT" + echo "ee_repo_ref=$REF" >> "$GITHUB_OUTPUT" + fi + + - name: Checkout EE repository + if: steps.ee.outputs.available == 'true' + uses: actions/checkout@v4 + with: + repository: windmill-labs/windmill-ee-private + path: ./windmill-ee-private + ref: ${{ steps.ee.outputs.ee_repo_ref }} + token: ${{ secrets.WINDMILL_EE_PRIVATE_ACCESS }} + fetch-depth: 1 + + - name: Substitute EE code + if: steps.ee.outputs.available == 'true' + run: ./backend/substitute_ee_code.sh --copy --dir ./windmill-ee-private + - name: Run Claude PR Action uses: anthropics/claude-code-action@v1 with: @@ -51,4 +89,4 @@ jobs: } claude_args: | --allowedTools "Bash,WebFetch,WebSearch" - --model claude-fable-5 + --model claude-opus-4-8 diff --git a/.github/workflows/pr-ready-review.yml b/.github/workflows/pr-ready-review.yml index f2462a5600..4d722df4af 100644 --- a/.github/workflows/pr-ready-review.yml +++ b/.github/workflows/pr-ready-review.yml @@ -160,4 +160,4 @@ jobs: ${{ env.REVIEW_PROMPT }} claude_args: | --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)" - --model claude-fable-5 + --model claude-opus-4-8 diff --git a/.github/workflows/refresh-docs-snapshot.yml b/.github/workflows/refresh-docs-snapshot.yml new file mode 100644 index 0000000000..a000f44df8 --- /dev/null +++ b/.github/workflows/refresh-docs-snapshot.yml @@ -0,0 +1,52 @@ +name: Refresh docs snapshot + +# The backend embeds a vendored docs snapshot (backend/windmill-api/docs_snapshot/*.gz) +# so in-product docs search works with no runtime egress. This job re-fetches it from +# windmill.dev on a schedule and opens a PR when it changed, keeping the embedded docs +# fresh independently of the release cadence (the binary embeds whatever is on the +# source tree at build time, so a merged refresh rides into the next release build). +on: + schedule: + - cron: "0 6 * * 1" # Mondays 06:00 UTC + workflow_dispatch: + +jobs: + refresh: + runs-on: ubicloud + permissions: + contents: write + pull-requests: write + steps: + - uses: actions/create-github-app-token@v2 + id: app + with: + app-id: ${{ vars.INTERNAL_APP_ID }} + private-key: ${{ secrets.INTERNAL_APP_KEY }} + - uses: actions/checkout@v4 + with: + token: ${{ steps.app.outputs.token }} + - name: Fetch + re-gzip docs snapshot + run: cd backend/windmill-api/docs_snapshot && ./fetch.sh + - name: Sanity-check the fetched corpus + # curl -f in fetch.sh rejects HTTP errors, but not a valid-but-garbage 200 + # (truncated file, error page). Guard against embedding a broken snapshot. + run: | + cd backend/windmill-api/docs_snapshot + test "$(wc -c < llms-full.txt.gz)" -gt 100000 + test "$(wc -c < llms.txt.gz)" -gt 1000 + pages=$(gzip -dc llms-full.txt.gz | grep -c '^Source:' || true) + echo "pages in snapshot: $pages" + test "${pages:-0}" -ge 200 + - uses: peter-evans/create-pull-request@v6 + with: + token: ${{ steps.app.outputs.token }} + branch: chore/refresh-docs-snapshot + add-paths: backend/windmill-api/docs_snapshot/*.gz + commit-message: "chore: refresh vendored docs snapshot" + title: "chore: refresh vendored docs snapshot" + body: | + Automated refresh of the embedded docs snapshot + (`backend/windmill-api/docs_snapshot/*.gz`) from windmill.dev. + + Review the diff for unexpected churn (a bad upstream docs deploy would + show up as a large drop in pages or content) before merging. diff --git a/.gitignore b/.gitignore index 5f733611de..e80b4f32ca 100644 --- a/.gitignore +++ b/.gitignore @@ -33,4 +33,5 @@ backend/chrome_profiler.json .fast-check/ __pycache__/ .playwright-mcp/ -.codex \ No newline at end of file +.codex +.claude/scheduled_tasks.lock diff --git a/AGENTS.md b/AGENTS.md index a294d47667..aead237e59 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -109,4 +109,5 @@ $NAV --root backend callees "X" # what does X call? - Search for existing code to reuse before writing new code - Follow established patterns in the codebase - Keep changes focused — don't refactor beyond what's asked +- **Comments record constraints, not narration.** Write a comment only for what the code can't show: why a non-obvious approach is required, what breaks if it's "simplified" away. State each invariant once, at the place where someone would break it, in ≤4 lines. Don't describe what the next line does, don't repeat the same rationale at multiple sites, and don't address the PR reviewer (justifying a change belongs in the PR description, not the code). Describe the code as it is, never its drafting history: "we no longer do X", "unchanged behavior", "instead of the previous approach" are meaningless to a reader who never saw the earlier iteration — before finishing, reread your comments as if the current state is the only state that ever existed. - **Never attribute work to a specific customer, account, or "requested by a customer" in repo-tracked content** (PR descriptions, commit messages, code comments, docs). Describe changes by their technical motivation instead. diff --git a/CHANGELOG.md b/CHANGELOG.md index d5145bcd26..361b6fbd91 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,419 @@ # Changelog +## [1.742.0](https://github.com/windmill-labs/windmill/compare/v1.741.0...v1.742.0) (2026-06-28) + + +### Features + +* **apps:** add labels input to app editor deploy drawer ([#9828](https://github.com/windmill-labs/windmill/issues/9828)) ([da45e69](https://github.com/windmill-labs/windmill/commit/da45e699c8aefeede172c90769ef4f4b182fec0c)) +* column-level lineage for DuckLake pipelines (SQL-AST inferred + traceable) ([#9814](https://github.com/windmill-labs/windmill/issues/9814)) ([003a262](https://github.com/windmill-labs/windmill/commit/003a262a4e9d6c2a63ada01aa8429aea1fbb6031)) + + +### Bug Fixes + +* **audit:** don't read pg_authid from an elevated context in S3 export migration ([#9832](https://github.com/windmill-labs/windmill/issues/9832)) ([75ba81b](https://github.com/windmill-labs/windmill/commit/75ba81b2d27fb0722095780312064cb93d20287e)) +* close unauthenticated DAP debugger program-mode launch bypass ([#9829](https://github.com/windmill-labs/windmill/issues/9829)) ([c0768de](https://github.com/windmill-labs/windmill/commit/c0768de0acdf63eaba5fb97d04bfc64f2f03b93d)) +* redeploy older app version from deployment history ([#9826](https://github.com/windmill-labs/windmill/issues/9826)) ([c479afa](https://github.com/windmill-labs/windmill/commit/c479afab8ebceccbee050e923dc5c27a6712ea62)) + +## [1.741.0](https://github.com/windmill-labs/windmill/compare/v1.740.0...v1.741.0) (2026-06-26) + + +### Features + +* **ai-chat:** add create_folder tool to global chat ([#9819](https://github.com/windmill-labs/windmill/issues/9819)) ([44c25de](https://github.com/windmill-labs/windmill/commit/44c25de418612ab98341adb15d5671222b54367e)) +* **ai-chat:** hint /compact in context usage tooltip ([#9777](https://github.com/windmill-labs/windmill/issues/9777)) ([aadfb62](https://github.com/windmill-labs/windmill/commit/aadfb620c0b7dcd7e94367b761875e14ef9abe69)) +* **ai-chat:** let global chat edit the user's personal instructions ([#9771](https://github.com/windmill-labs/windmill/issues/9771)) ([3be2752](https://github.com/windmill-labs/windmill/commit/3be27521b05de33e48582e80c6651071f889f048)) +* **ai-chat:** surface raw apps in the @-mention context picker ([#9800](https://github.com/windmill-labs/windmill/issues/9800)) ([1602244](https://github.com/windmill-labs/windmill/commit/16022447c7b445be753b9545b10b4c67da0893d5)) +* capture managed-materialize output schema as asset metadata ([#2](https://github.com/windmill-labs/windmill/issues/2)a) ([#9812](https://github.com/windmill-labs/windmill/issues/9812)) ([ade74b2](https://github.com/windmill-labs/windmill/commit/ade74b297f6a03441e700a20ffc2d7291c8a85fd)) +* **sdk:** allow overriding worker tag when running jobs (WIN-2105) ([#9807](https://github.com/windmill-labs/windmill/issues/9807)) ([52fc7bf](https://github.com/windmill-labs/windmill/commit/52fc7bf94cf3f87f68d9dba9884944d87e7d5d57)) + + +### Bug Fixes + +* apply step timeout to 'Test this step' preview ([#9810](https://github.com/windmill-labs/windmill/issues/9810)) ([d04062b](https://github.com/windmill-labs/windmill/commit/d04062bff58c9c4c79ce542a4321e71bcbcf0e98)) +* **flows:** reject corrupt step paths at deploy + atomic cache writes ([#9751](https://github.com/windmill-labs/windmill/issues/9751)) ([#9813](https://github.com/windmill-labs/windmill/issues/9813)) ([3cda447](https://github.com/windmill-labs/windmill/commit/3cda44762148bcd2ee5c0ea821db884950376ead)) +* **frontend:** clarify instance data table unavailable on cloud ([#9806](https://github.com/windmill-labs/windmill/issues/9806)) ([c3e8c78](https://github.com/windmill-labs/windmill/commit/c3e8c789ac05c9c28991d9ab6f2358f61fa87971)) +* hide GCS service account key behind a reveal in object storage settings ([#9815](https://github.com/windmill-labs/windmill/issues/9815)) ([0ec5061](https://github.com/windmill-labs/windmill/commit/0ec5061270749ed078e01f5a4bc7397a1755ca32)) +* ping job during volume setup to prevent false zombie restarts ([#9803](https://github.com/windmill-labs/windmill/issues/9803)) ([43bb676](https://github.com/windmill-labs/windmill/commit/43bb676dc5652cb06fe1414b8d3aacf295bae36b)) +* skipped suspend step no longer parks the flow forever ([#9821](https://github.com/windmill-labs/windmill/issues/9821)) ([40110bc](https://github.com/windmill-labs/windmill/commit/40110bc7158bc42c3d84bd4637a12b82fcd72a9a)) + + +### Performance Improvements + +* **audit:** re-anchor S3 audit export on enable + opt-in backfill ([#9818](https://github.com/windmill-labs/windmill/issues/9818)) ([577ceee](https://github.com/windmill-labs/windmill/commit/577ceeee8679f054c6898d1a7889df30ab830f8f)) + +## [1.740.0](https://github.com/windmill-labs/windmill/compare/v1.739.0...v1.740.0) (2026-06-25) + + +### Features + +* **api:** add structured endpoint for flow logs ([#9797](https://github.com/windmill-labs/windmill/issues/9797)) ([ba768fe](https://github.com/windmill-labs/windmill/commit/ba768fee888682cb50142d6e76c0422c40307f46)) +* bounded-cascade selective execution for pipelines (UI + CLI) ([#9695](https://github.com/windmill-labs/windmill/issues/9695)) ([248540a](https://github.com/windmill-labs/windmill/commit/248540ac4d6e4ee9ee7c3e6f2cc822c63cc6426e)) +* data tests for ducklake pipeline materialization ([#9708](https://github.com/windmill-labs/windmill/issues/9708)) ([f6998ec](https://github.com/windmill-labs/windmill/commit/f6998ec54cba2507703790bf33427e7567d42c4b)) +* detect and guard against deploying stale drafts ([#9768](https://github.com/windmill-labs/windmill/issues/9768)) ([d865518](https://github.com/windmill-labs/windmill/commit/d8655189347f58df9d17e83dc55798baf7964279)) +* ducklake time-travel UX (snapshot history + AT VERSION reads) ([#9709](https://github.com/windmill-labs/windmill/issues/9709)) ([d131d75](https://github.com/windmill-labs/windmill/commit/d131d754e1fc9674abf5de383d2bc93596df9bd1)) +* self-host docs search for chat, mcp, cli; drop inkeep ([#9772](https://github.com/windmill-labs/windmill/issues/9772)) ([9d61e4e](https://github.com/windmill-labs/windmill/commit/9d61e4e59e4101de84217f7c7846f1aa94e84d89)) + + +### Bug Fixes + +* allow hyphens in postgresql database name validation ([#9782](https://github.com/windmill-labs/windmill/issues/9782)) ([170cd79](https://github.com/windmill-labs/windmill/commit/170cd79aaf92152fc3c0f675f155853c7f0e5b25)) +* **debounce:** never supersede a running debounce survivor ([#9780](https://github.com/windmill-labs/windmill/issues/9780)) ([5549bdc](https://github.com/windmill-labs/windmill/commit/5549bdc67a5559a764616c44b1018543bc0568fe)) +* decrypt secret variables via external backend in common resolvers ([#9784](https://github.com/windmill-labs/windmill/issues/9784)) ([cd42c6c](https://github.com/windmill-labs/windmill/commit/cd42c6ca18261328055554788932c3fe876a4a5b)) +* enforce containment of python module dir for preview jobs ([#9704](https://github.com/windmill-labs/windmill/issues/9704)) ([88fca6a](https://github.com/windmill-labs/windmill/commit/88fca6a8c130b9e3b0f0cd410e422d4e074fc11f)) +* **frontend:** apply script editor timeout to preview/Test runs ([#9794](https://github.com/windmill-labs/windmill/issues/9794)) ([6664ce6](https://github.com/windmill-labs/windmill/commit/6664ce6dc0c5fbc283303148de06d6bb85e4acf7)) +* **frontend:** nested-loop "Test this step" resolves iter to innermost loop ([#9778](https://github.com/windmill-labs/windmill/issues/9778)) ([74ebfc6](https://github.com/windmill-labs/windmill/commit/74ebfc67f069047875db738926865bd4bd6fe9e9)) +* opt out of Deno minimum-dependency-age for private npm registries ([#9802](https://github.com/windmill-labs/windmill/issues/9802)) ([b28f974](https://github.com/windmill-labs/windmill/commit/b28f974e5069f635419d9ea56fad6a0e417894e8)) +* pass SSL cert env vars to `uv python install` ([#9790](https://github.com/windmill-labs/windmill/issues/9790)) ([962758c](https://github.com/windmill-labs/windmill/commit/962758c02de5f6d962c681fe9c39769b99429e8d)) +* **python:** re-verify wheel RECORD on local cache reuse (once per worker) ([#9775](https://github.com/windmill-labs/windmill/issues/9775)) ([6c71c33](https://github.com/windmill-labs/windmill/commit/6c71c33470e3ea547f3b994db829eb4d04882443)) +* **python:** serialize concurrent installs into shared wheel cache dir ([#9787](https://github.com/windmill-labs/windmill/issues/9787)) ([11d83ab](https://github.com/windmill-labs/windmill/commit/11d83ab1ec559be5d3263010228e6db65358e04b)) +* re-pin stale-draft fork base when restoring an app deployment ([#9792](https://github.com/windmill-labs/windmill/issues/9792)) ([b9711e5](https://github.com/windmill-labs/windmill/commit/b9711e5ace8585315a1c2b85bb25ac8dd7832d6f)) +* restore libargon2-1 for PHP runtime in server image ([#9795](https://github.com/windmill-labs/windmill/issues/9795)) ([e9cb806](https://github.com/windmill-labs/windmill/commit/e9cb80639b2dec63ede69fc3a4e3720bb1a3c319)) +* use transaction for parallel_monitor_lock DELETE in last-iteration path ([#9789](https://github.com/windmill-labs/windmill/issues/9789)) ([754cae9](https://github.com/windmill-labs/windmill/commit/754cae956ac8d431ddeb055453835e249cdd07b7)) + + +### Performance Improvements + +* drop v2_job side-table ON DELETE CASCADE FKs to speed retention deletes ([#9786](https://github.com/windmill-labs/windmill/issues/9786)) ([aa098c7](https://github.com/windmill-labs/windmill/commit/aa098c70c0271b2b1917749d1f607c0559cf04de)) +* eliminate dual-connection DB pool contention across worker, queue, and api ([#9798](https://github.com/windmill-labs/windmill/issues/9798)) ([0dbd9c1](https://github.com/windmill-labs/windmill/commit/0dbd9c1231b00d4693af68835fe1d9e7c8869b43)) + +## [1.739.0](https://github.com/windmill-labs/windmill/compare/v1.738.0...v1.739.0) (2026-06-24) + + +### Features + +* add /compact session chat command ([#9764](https://github.com/windmill-labs/windmill/issues/9764)) ([83cc553](https://github.com/windmill-labs/windmill/commit/83cc5533ee92e59356a117eefeaf42dad23287f6)) +* add session chat slash commands ([#9748](https://github.com/windmill-labs/windmill/issues/9748)) ([24b95e9](https://github.com/windmill-labs/windmill/commit/24b95e9fe12ba4abdfe1ff6e9f9fe42cb2ded011)) +* **ai-chat:** add /clear session command to start a fresh conversation ([#9769](https://github.com/windmill-labs/windmill/issues/9769)) ([3fafac2](https://github.com/windmill-labs/windmill/commit/3fafac275d2100a6f89924040650cf959945d209)) +* **ai-chat:** context usage gauge + unified model settings menu ([#9763](https://github.com/windmill-labs/windmill/issues/9763)) ([2e020b2](https://github.com/windmill-labs/windmill/commit/2e020b2ccc7a649a5923bff72a98f07d4fc85381)) +* **apps:** show raw-app fork diffs as per-file tree items ([#9491](https://github.com/windmill-labs/windmill/issues/9491)) ([e98df38](https://github.com/windmill-labs/windmill/commit/e98df38ac43823ee85209a4b09cd70690469302d)) +* **frontend:** add filter submenu to collapsed AI sessions popover ([#9757](https://github.com/windmill-labs/windmill/issues/9757)) ([3d48ba7](https://github.com/windmill-labs/windmill/commit/3d48ba7738c3d3356539b5fc44a871f6b7f9d548)) +* **frontend:** restore raw app 'open preview in separate window' ([#9765](https://github.com/windmill-labs/windmill/issues/9765)) ([a116715](https://github.com/windmill-labs/windmill/commit/a116715c418c39d48a91e6c0b4484a31537dff38)) +* **frontend:** show approval wait as a distinct segment in flow timeline ([#9756](https://github.com/windmill-labs/windmill/issues/9756)) ([2a70ccc](https://github.com/windmill-labs/windmill/commit/2a70ccc38675c7c2353807a4f85764a8a35224e2)) +* scope AI sessions per workspace root with lifecycle reconcile ([#9734](https://github.com/windmill-labs/windmill/issues/9734)) ([42c5e7a](https://github.com/windmill-labs/windmill/commit/42c5e7a3fc9b74256de8806ec0d7b62e8bbf029c)) + + +### Bug Fixes + +* **ai-chat:** strip unclosed <summary> tag leaking into compaction summary ([#9750](https://github.com/windmill-labs/windmill/issues/9750)) ([250a05f](https://github.com/windmill-labs/windmill/commit/250a05f544ae397bb91af5fc83bf408cfe1c554d)) +* **apps:** realign legacy raw-app drafts to raw_app draft kind ([#9761](https://github.com/windmill-labs/windmill/issues/9761)) ([288318a](https://github.com/windmill-labs/windmill/commit/288318ac269714fc03b15622dbb86b1c28268a36)) +* **backend:** resolve folder_labels search_path on non-public (PG_SCHEMA) schemas ([#9758](https://github.com/windmill-labs/windmill/issues/9758)) ([f582878](https://github.com/windmill-labs/windmill/commit/f5828780fd6a8be070b2933ebd41ee6dff98a9e1)) +* forbid superadmin job tokens from global user and token management ([#9715](https://github.com/windmill-labs/windmill/issues/9715)) ([043c2c0](https://github.com/windmill-labs/windmill/commit/043c2c05b7678c49faca0ccb28e5f6393567ba4d)) +* **frontend:** highlight the runtime-chosen branch in flow graph viewer ([#9755](https://github.com/windmill-labs/windmill/issues/9755)) ([de6192b](https://github.com/windmill-labs/windmill/commit/de6192bec1695883a07452f7db2fb51c94dbfd43)) +* **frontend:** keep #content portal target present on AI-session route ([#9754](https://github.com/windmill-labs/windmill/issues/9754)) ([5e09c50](https://github.com/windmill-labs/windmill/commit/5e09c501713ebbe05b28ce0084eca641f0dbe95c)) +* **frontend:** show AI skills settings only when global mode enabled ([#9747](https://github.com/windmill-labs/windmill/issues/9747)) ([c017f7f](https://github.com/windmill-labs/windmill/commit/c017f7f8919a51292ddf01574961d1774bc1ba23)) +* **frontend:** stop flow step id generation from being poisoned by non-canonical keys ([#9766](https://github.com/windmill-labs/windmill/issues/9766)) ([4dbf873](https://github.com/windmill-labs/windmill/commit/4dbf8737238ccc4dc2c67365e6d43f04f46c75b5)) +* persist on-behalf-of user across app deploy paths ([#9773](https://github.com/windmill-labs/windmill/issues/9773)) ([f99781c](https://github.com/windmill-labs/windmill/commit/f99781ca5f77248206c951935cc44acfa5f072eb)) +* reject symlink traversal in job-dir path validation ([#9713](https://github.com/windmill-labs/windmill/issues/9713)) ([b5bd824](https://github.com/windmill-labs/windmill/commit/b5bd8245d81b84fc14d3ea955bf1e66ac576bf37)) + + +### Performance Improvements + +* **audit:** adaptive timestamp floor for S3 audit-log export ([#9752](https://github.com/windmill-labs/windmill/issues/9752)) ([55bed4a](https://github.com/windmill-labs/windmill/commit/55bed4abcfce2a611b16054573980d2eb613ccb3)) +* **monitor:** vacuum job_perms/job_result_stream right after each orphan sweep ([#9753](https://github.com/windmill-labs/windmill/issues/9753)) ([8912e21](https://github.com/windmill-labs/windmill/commit/8912e21d1571e57b5cf21b7d4d9520e20a28e70d)) + +## [1.738.0](https://github.com/windmill-labs/windmill/compare/v1.737.0...v1.738.0) (2026-06-23) + + +### Features + +* add resource and infrastructure telemetry ([#9737](https://github.com/windmill-labs/windmill/issues/9737)) ([9793d01](https://github.com/windmill-labs/windmill/commit/9793d01575415963a89609a1baf2cd64f0d050cc)) +* render mermaid diagrams in chat code blocks ([#9738](https://github.com/windmill-labs/windmill/issues/9738)) ([cfb9f1d](https://github.com/windmill-labs/windmill/commit/cfb9f1dbc23110ecf8f91bb3c8c81fc6e35dc09b)) + + +### Bug Fixes + +* **ai-chat:** Fix incorrect editor edits from ai chat [#1](https://github.com/windmill-labs/windmill/issues/1) ([#9741](https://github.com/windmill-labs/windmill/issues/9741)) ([fc797a3](https://github.com/windmill-labs/windmill/commit/fc797a35fe7885630c81453df0fc94769e73873a)) +* allow object storage test for non-super-admins, harden on cloud ([#9739](https://github.com/windmill-labs/windmill/issues/9739)) ([24446e8](https://github.com/windmill-labs/windmill/commit/24446e80093ade349f7fbf65063d2d1cb5551c1e)) +* **frontend:** debounce external code→Monaco sync in Editor ([#9743](https://github.com/windmill-labs/windmill/issues/9743)) ([29c67ce](https://github.com/windmill-labs/windmill/commit/29c67ced97bf2919584986f9d9eceb4337c34ad9)) +* **frontend:** preserve editor content when closing instance settings drawer ([#9740](https://github.com/windmill-labs/windmill/issues/9740)) ([11d0e65](https://github.com/windmill-labs/windmill/commit/11d0e65f3af9a048bc1921bbdd3d676a07483a57)) +* pipeline annotation false-positives from body comments ([#9736](https://github.com/windmill-labs/windmill/issues/9736)) ([984ea72](https://github.com/windmill-labs/windmill/commit/984ea728d98649b66b1cae899bdab9af3176caa7)) +* preserve fork parent linkage on workspace id change ([#9716](https://github.com/windmill-labs/windmill/issues/9716)) ([cbf54d4](https://github.com/windmill-labs/windmill/commit/cbf54d4eb432638e27f67c4c8b879cbcc0291da3)) +* prevent variable push from corrupting is_secret variables ([#9705](https://github.com/windmill-labs/windmill/issues/9705)) ([ba4b368](https://github.com/windmill-labs/windmill/commit/ba4b368706e95e22f346a10e5fe145b0795ac3f6)) + + +### Performance Improvements + +* **monitor:** skip protected prefix in retention delete via cross-batch watermark (WIN-2088) ([#9744](https://github.com/windmill-labs/windmill/issues/9744)) ([e90b2be](https://github.com/windmill-labs/windmill/commit/e90b2be8fade1eb78cd685890291f5a4553a6a10)) + +## [1.737.0](https://github.com/windmill-labs/windmill/compare/v1.736.0...v1.737.0) (2026-06-23) + + +### Features + +* **apps:** opt-in sandbox isolation for published & raw apps (alpha) ([#9420](https://github.com/windmill-labs/windmill/issues/9420)) ([2879cbb](https://github.com/windmill-labs/windmill/commit/2879cbb65a4122c86b4a472206d74d9009b07904)) + + +### Bug Fixes + +* allow SQL args in managed // materialize scripts ([#9733](https://github.com/windmill-labs/windmill/issues/9733)) ([fa35968](https://github.com/windmill-labs/windmill/commit/fa3596885bf2d7ee8859f295e820ee362c756911)) +* bound orphan-cleanup drain rate with capped multi-batch loop ([#9730](https://github.com/windmill-labs/windmill/issues/9730)) ([31d9215](https://github.com/windmill-labs/windmill/commit/31d9215e5a61f19662cc87be8147007e1d47ebb6)) +* **ext-jwt:** reject external JWT auth for non-existent workspaces ([#9723](https://github.com/windmill-labs/windmill/issues/9723)) ([c644311](https://github.com/windmill-labs/windmill/commit/c644311eca4bcaf4b68058cf1d5d79d4078aee1a)) +* optimize cleanup_job_perms_orphaned and job_result_stream cleanup queries ([#9727](https://github.com/windmill-labs/windmill/issues/9727)) ([6d94865](https://github.com/windmill-labs/windmill/commit/6d9486510933af9109f52011d93b13847dbdbb39)) +* prevent silent audit-partition outage via monitor watchdog + alert ([#9729](https://github.com/windmill-labs/windmill/issues/9729)) ([8dea383](https://github.com/windmill-labs/windmill/commit/8dea38383f884f59b2956c39f1424005a21265bd)) + + +### Performance Improvements + +* **monitor:** hash active-root exclusion in retention delete (WIN-2088) ([#9732](https://github.com/windmill-labs/windmill/issues/9732)) ([75bafab](https://github.com/windmill-labs/windmill/commit/75bafabeeec76cf6da33eef41f588e37071df011)) + +## [1.736.0](https://github.com/windmill-labs/windmill/compare/v1.735.0...v1.736.0) (2026-06-23) + + +### Features + +* **ai-chat:** workspace AI chat skills (SKILL.md upload + read_skill tool) ([#9648](https://github.com/windmill-labs/windmill/issues/9648)) ([6f4017d](https://github.com/windmill-labs/windmill/commit/6f4017d694494a158ebd0579c93336c378cba0fd)) + + +### Bug Fixes + +* **drafts:** stop mis-filing workspace-blind legacy drafts on migration ([#9725](https://github.com/windmill-labs/windmill/issues/9725)) ([3bf5b72](https://github.com/windmill-labs/windmill/commit/3bf5b72afab3241ea41a261a40c2434764bdaf72)) +* **frontend:** destroy old WebsocketProvider on workspace switch in MultiplayerMenu ([#9719](https://github.com/windmill-labs/windmill/issues/9719)) ([6e96f90](https://github.com/windmill-labs/windmill/commit/6e96f90065dfe2f6ccc5eb4f85f4facd3515c70c)) +* **frontend:** ensure type:object in test_run_flow tool schema for Anthropic ([#9721](https://github.com/windmill-labs/windmill/issues/9721)) ([d5cb944](https://github.com/windmill-labs/windmill/commit/d5cb944cf92f074b2ee42c876595eacdfa2f4d76)) +* **health:** detect read-only replica via pg_is_in_recovery() ([#9722](https://github.com/windmill-labs/windmill/issues/9722)) ([e16061d](https://github.com/windmill-labs/windmill/commit/e16061df06babeae935a9396de5bdcd46e8119a9)) +* re-enforce scoped API token boundaries across handlers ([#9712](https://github.com/windmill-labs/windmill/issues/9712)) ([e19594d](https://github.com/windmill-labs/windmill/commit/e19594df2ad015a0336ade95e04562f5562ec3f6)) + +## [1.735.0](https://github.com/windmill-labs/windmill/compare/v1.734.0...v1.735.0) (2026-06-22) + + +### Features + +* clarify session draft bar tracks all workspace draft changes ([#9714](https://github.com/windmill-labs/windmill/issues/9714)) ([ed016a5](https://github.com/windmill-labs/windmill/commit/ed016a5edb4527877bf7e6bf92feafc2710669c2)) +* **copilot:** improve global-mode path selection + add path-selection evals ([#9698](https://github.com/windmill-labs/windmill/issues/9698)) ([74a2329](https://github.com/windmill-labs/windmill/commit/74a2329d2e8395141807c43acef07ef132490039)) +* link files & folders to the global AI chat ([#9520](https://github.com/windmill-labs/windmill/issues/9520)) ([84cc043](https://github.com/windmill-labs/windmill/commit/84cc043406d63a1e1472165cf24ce8c09905fc5f)) +* scope default instance db name to workspace (dt_/dl_) ([#9699](https://github.com/windmill-labs/windmill/issues/9699)) ([4a8a724](https://github.com/windmill-labs/windmill/commit/4a8a724895dcecb835e1eb1e4fd7d1bbc8b3e0fb)) + + +### Bug Fixes + +* enforce job_dir containment when writing module files ([#9703](https://github.com/windmill-labs/windmill/issues/9703)) ([e403f92](https://github.com/windmill-labs/windmill/commit/e403f92d7e84cebc78709dce1a0928048ba2506d)) +* **frontend:** deploy full script/flow draft from AI chat via shared module ([#9642](https://github.com/windmill-labs/windmill/issues/9642)) ([23bf6bf](https://github.com/windmill-labs/windmill/commit/23bf6bf3da01d552d2dfab6dbcfd30f758ed34d2)) +* **frontend:** strip raw-app post-deploy diff noise (raw_app/lock/data) ([#9706](https://github.com/windmill-labs/windmill/issues/9706)) ([e20a277](https://github.com/windmill-labs/windmill/commit/e20a27745a08d552e6d2c5a8bbaf08ccfe89c68f)) +* ignore NotFound errors when deleting log files from object store ([#9707](https://github.com/windmill-labs/windmill/issues/9707)) ([8a0b0ab](https://github.com/windmill-labs/windmill/commit/8a0b0abead71320c4f69eb3007739a19f76d4126)) +* **oauth:** restore bring-your-own CC token URL override ([#9711](https://github.com/windmill-labs/windmill/issues/9711)) ([ef4962e](https://github.com/windmill-labs/windmill/commit/ef4962e52aba0bc79bf72523de9101853c660654)) +* sanitize git credentials from ansible executor errors and logs ([#9697](https://github.com/windmill-labs/windmill/issues/9697)) ([ace7b68](https://github.com/windmill-labs/windmill/commit/ace7b68a28b00d715298ffcb6ae907c1974a74b8)) + +## [1.734.0](https://github.com/windmill-labs/windmill/compare/v1.733.1...v1.734.0) (2026-06-20) + + +### Features + +* ducklake materialization for data pipelines ([#9689](https://github.com/windmill-labs/windmill/issues/9689)) ([3ebf243](https://github.com/windmill-labs/windmill/commit/3ebf24359d66048d6361ce65cd879cdc04b737ed)) + + +### Bug Fixes + +* **frontend:** clear branch step state when switching outer loop iterations ([#9650](https://github.com/windmill-labs/windmill/issues/9650)) ([09a8004](https://github.com/windmill-labs/windmill/commit/09a80040ca268a4379d5302e3401435ba93247e0)) + +## [1.733.1](https://github.com/windmill-labs/windmill/compare/v1.733.0...v1.733.1) (2026-06-19) + + +### Bug Fixes + +* **backend:** validate ansible vault_id entries before config generation ([#9681](https://github.com/windmill-labs/windmill/issues/9681)) ([c1f31c0](https://github.com/windmill-labs/windmill/commit/c1f31c0e4777bf0cfed0dd7f03249e9a61cd8cb9)) +* **frontend:** group live pipeline runs in the activity panel ([#9684](https://github.com/windmill-labs/windmill/issues/9684)) ([1be4df9](https://github.com/windmill-labs/windmill/commit/1be4df9acb935250d4cc12e83cf67e366d870d5a)) +* require super admin for object storage config test endpoint ([#9683](https://github.com/windmill-labs/windmill/issues/9683)) ([fb44fe7](https://github.com/windmill-labs/windmill/commit/fb44fe7af2b8ebe8ef64ffb0e5acce8580bf4200)) +* validate websocket trigger urls and gate trigger test route ([#9682](https://github.com/windmill-labs/windmill/issues/9682)) ([c39ee07](https://github.com/windmill-labs/windmill/commit/c39ee07c0bcd2249dd19ffa5cd988125eefc6c9f)) + +## [1.733.0](https://github.com/windmill-labs/windmill/compare/v1.732.0...v1.733.0) (2026-06-19) + + +### Features + +* **ai-chat:** cap read_app_file + search_app grep tool to bound context in large raw apps ([#9653](https://github.com/windmill-labs/windmill/issues/9653)) ([4296a6a](https://github.com/windmill-labs/windmill/commit/4296a6ae1f73564de4df54fe1df0a03c1df05dfd)) +* **python, windows:** enable S3 to cache wheels ([#5199](https://github.com/windmill-labs/windmill/issues/5199)) ([ab3bc97](https://github.com/windmill-labs/windmill/commit/ab3bc97cd92b6480327029bcf018280442462af7)) + + +### Bug Fixes + +* allow users to always discard their own drafts without write permission ([#9659](https://github.com/windmill-labs/windmill/issues/9659)) ([6833a55](https://github.com/windmill-labs/windmill/commit/6833a554aeddb3e63173d3c3140b490c0bf2822b)) +* **backend:** clean up unique_ext_jwt_token on workspace deletion ([#9676](https://github.com/windmill-labs/windmill/issues/9676)) ([9add719](https://github.com/windmill-labs/windmill/commit/9add719d936cdcfb2c4062629e3e1f792694dafe)) +* **backend:** strip NUL bytes from draft values on write ([#9673](https://github.com/windmill-labs/windmill/issues/9673)) ([924f9c7](https://github.com/windmill-labs/windmill/commit/924f9c7e8d8863d9af40aee246a519b4be0e1ea2)) +* **python:** split PIP_TRUSTED_HOST by whitespace to support multiple hosts ([#9675](https://github.com/windmill-labs/windmill/issues/9675)) ([cafb473](https://github.com/windmill-labs/windmill/commit/cafb473494d9cff3a8b2aeaf9f18b015f966e7b3)) + +## [1.732.0](https://github.com/windmill-labs/windmill/compare/v1.731.0...v1.732.0) (2026-06-19) + + +### Features + +* **ansible:** add AI chat and editor bar buttons for ansible ([#9671](https://github.com/windmill-labs/windmill/issues/9671)) ([017c3d3](https://github.com/windmill-labs/windmill/commit/017c3d3343c2577501103be4b2dd8dac9727d80d)) + + +### Bug Fixes + +* **ai:** emit token usage in gemini proxy streaming translation ([#9669](https://github.com/windmill-labs/windmill/issues/9669)) ([0cc2257](https://github.com/windmill-labs/windmill/commit/0cc2257596a3965cf6db21a0090edcce6e1b8419)) +* **backend:** grant script_trigger access to windmill roles ([#9674](https://github.com/windmill-labs/windmill/issues/9674)) ([3361736](https://github.com/windmill-labs/windmill/commit/33617367d09537667d2ab3f91135c736194b9e7e)) +* **frontend:** ignore hash/assets in script diffs and drafts (WIN-2071) ([#9664](https://github.com/windmill-labs/windmill/issues/9664)) ([3371265](https://github.com/windmill-labs/windmill/commit/33712653821e83f2562dd5f271dbec0188d5d2f8)) + +## [1.731.0](https://github.com/windmill-labs/windmill/compare/v1.730.0...v1.731.0) (2026-06-19) + + +### Features + +* **backend:** auto-reconnect postgres trigger listener with backoff (WIN-2073) ([#9666](https://github.com/windmill-labs/windmill/issues/9666)) ([a425431](https://github.com/windmill-labs/windmill/commit/a425431e9067bcf85474fdc7b7ef7f73e41b9071)) + + +### Bug Fixes + +* **backend:** grant notify_event access to windmill roles ([#9665](https://github.com/windmill-labs/windmill/issues/9665)) ([a682d02](https://github.com/windmill-labs/windmill/commit/a682d02311a2110bfc0d5e0a5b52e96147fe0dd7)) +* **mcp:** repair invalid type keywords in tool JSON schemas ([#9667](https://github.com/windmill-labs/windmill/issues/9667)) ([c30bdec](https://github.com/windmill-labs/windmill/commit/c30bdecea77ff9b4d74d52961f3101201099b683)) +* trigger flow error handler on unrecoverable (OOM/zombie) step failures ([#9662](https://github.com/windmill-labs/windmill/issues/9662)) ([7e4df02](https://github.com/windmill-labs/windmill/commit/7e4df02bd60c4d6ee8c92d3dfd19f4e587ff9632)) + +## [1.730.0](https://github.com/windmill-labs/windmill/compare/v1.729.0...v1.730.0) (2026-06-18) + + +### Features + +* **ai-chat:** summary-based conversation compaction ([#9645](https://github.com/windmill-labs/windmill/issues/9645)) ([5d553b8](https://github.com/windmill-labs/windmill/commit/5d553b81c06664aab61131a93b198575c088d12d)) +* Data Pipelines alpha ([#9193](https://github.com/windmill-labs/windmill/issues/9193)) ([7155a0b](https://github.com/windmill-labs/windmill/commit/7155a0bb96cf30bd878272a0f4c3c3b02341b261)) + + +### Bug Fixes + +* **ai-chat:** stop echoing app draft value in global chat write tool results ([#9658](https://github.com/windmill-labs/windmill/issues/9658)) ([2fed808](https://github.com/windmill-labs/windmill/commit/2fed808b9e716d9a44b34c7a073ec0d37374be05)) +* **backend:** include raw_app drafts in list_apps draft_users ([#9647](https://github.com/windmill-labs/windmill/issues/9647)) ([19bc005](https://github.com/windmill-labs/windmill/commit/19bc0052f1069d732231950a0ec958f675d57417)) +* **frontend:** keep ?new_draft flag until first save is confirmed ([#9656](https://github.com/windmill-labs/windmill/issues/9656)) ([9b6b7c3](https://github.com/windmill-labs/windmill/commit/9b6b7c3862d9988e5e91eaab2b967a23f41cdc0d)) +* **frontend:** re-key raw-app autosave on post-deploy navigation ([#9646](https://github.com/windmill-labs/windmill/issues/9646)) ([1058bde](https://github.com/windmill-labs/windmill/commit/1058bdeccdc4c403ef4599db0ee74a65a66c715f)) +* gate agent-worker global setting reads with a blocklist ([#9623](https://github.com/windmill-labs/windmill/issues/9623)) ([fdd82f0](https://github.com/windmill-labs/windmill/commit/fdd82f0c48f29805cd9e219649f27fba45c7fd92)) +* **workspaces:** add instance setting to disable workspace invite/add emails ([#9643](https://github.com/windmill-labs/windmill/issues/9643)) ([796230d](https://github.com/windmill-labs/windmill/commit/796230d90a7e6d1debc15e139ab708881e527862)) + +## [1.729.0](https://github.com/windmill-labs/windmill/compare/v1.728.1...v1.729.0) (2026-06-18) + + +### Features + +* add ducklake schema support to the database manager ([#9633](https://github.com/windmill-labs/windmill/issues/9633)) ([3eeccaf](https://github.com/windmill-labs/windmill/commit/3eeccaf9682b7803fdf5be8dcbc4d243e0ba2e49)) +* **ai-chat:** self-hosted docs tools via windmill.dev llms.txt + ask benchmark ([#9578](https://github.com/windmill-labs/windmill/issues/9578)) ([f4425fc](https://github.com/windmill-labs/windmill/commit/f4425fca9fb0d02b845bd72888ade54905c5a30b)) +* **frontend:** View Diff and in-place Load for other users' drafts ([#9621](https://github.com/windmill-labs/windmill/issues/9621)) ([5508f1d](https://github.com/windmill-labs/windmill/commit/5508f1da9cd04c2583eb3f7ee6bce19d067f2227)) +* per-user draft review & deploy page (gating, badges, rename, raw-app deploy fixes) ([#9625](https://github.com/windmill-labs/windmill/issues/9625)) ([e09cd58](https://github.com/windmill-labs/windmill/commit/e09cd5862cb636e143027fe8d9a5be9c7097b031)) +* queue messages typed while ai chat is streaming ([#9525](https://github.com/windmill-labs/windmill/issues/9525)) ([51bd869](https://github.com/windmill-labs/windmill/commit/51bd8692a482850f7ac8b04dd16db5876336b5b9)) +* zero-setup oauth client credentials for registry providers ([#9559](https://github.com/windmill-labs/windmill/issues/9559)) ([e26a923](https://github.com/windmill-labs/windmill/commit/e26a9239a62a25abf90ef06ade4dde7f36e791bb)) + + +### Bug Fixes + +* **ai_evals:** adapt global eval harness to DB-backed user drafts ([#9641](https://github.com/windmill-labs/windmill/issues/9641)) ([e87ff79](https://github.com/windmill-labs/windmill/commit/e87ff79ecf6a6e0958916ed1b3756fb3addf719f)) +* **drafts:** preserve original timestamp when migrating localStorage drafts ([#9638](https://github.com/windmill-labs/windmill/issues/9638)) ([8021775](https://github.com/windmill-labs/windmill/commit/8021775f5f961ef6fd01b022639b85855326a1da)) +* **frontend:** don't save drafts on leave when auto-save is off, warn instead ([#9630](https://github.com/windmill-labs/windmill/issues/9630)) ([2523465](https://github.com/windmill-labs/windmill/commit/252346500945a9571af744c839ac0c7d6870504f)) +* **frontend:** render Modal2 dialogs above the AI chat panel ([#9636](https://github.com/windmill-labs/windmill/issues/9636)) ([b67c8cf](https://github.com/windmill-labs/windmill/commit/b67c8cf42b477575fc1bc448058ec0d3b7e54fee)) +* **frontend:** show AI sessions when AI unconfigured, with disabled chat ([#9644](https://github.com/windmill-labs/windmill/issues/9644)) ([ba69d81](https://github.com/windmill-labs/windmill/commit/ba69d8147b615e160cf3d2885fc65a0777b78b71)) +* **git-sync:** bump default sync script to hub/28719 (windmill-cli 1.728.1) for WAC modules ([#9649](https://github.com/windmill-labs/windmill/issues/9649)) ([3c0e38b](https://github.com/windmill-labs/windmill/commit/3c0e38b5890d77983cb5cf5f422a62a73e7a4f22)) + +## [1.728.1](https://github.com/windmill-labs/windmill/compare/v1.728.0...v1.728.1) (2026-06-17) + + +### Bug Fixes + +* **backend:** purge workspace_diff cache on workspace delete ([#9627](https://github.com/windmill-labs/windmill/issues/9627)) ([8a3f69d](https://github.com/windmill-labs/windmill/commit/8a3f69dda8f2088fb859ed8ed6e54458940423d0)) +* **cli:** fall back to esbuild-wasm on native host/binary mismatch ([#9629](https://github.com/windmill-labs/windmill/issues/9629)) ([86d1d16](https://github.com/windmill-labs/windmill/commit/86d1d160f0d3bd9faabdafada07e2956dd98445d)) +* **frontend:** persist session-editor draft path/summary edits + per-line diff tooltips ([#9622](https://github.com/windmill-labs/windmill/issues/9622)) ([e4bfeb2](https://github.com/windmill-labs/windmill/commit/e4bfeb29bc4e89669863b5f6396904a331167658)) + +## [1.728.0](https://github.com/windmill-labs/windmill/compare/v1.727.0...v1.728.0) (2026-06-16) + + +### Features + +* **frontend:** adapt AI-chat/sessions drafts to DB-backed model ([#9601](https://github.com/windmill-labs/windmill/issues/9601)) ([611c70a](https://github.com/windmill-labs/windmill/commit/611c70acd211cf4b8f8308da4a264c670a2f5f43)) +* **frontend:** consolidate draft-migration errors into a single toast + modal ([#9612](https://github.com/windmill-labs/windmill/issues/9612)) ([bc0d5bf](https://github.com/windmill-labs/windmill/commit/bc0d5bf241df3633921bd9d43d171e91034fbfcf)) +* **frontend:** dedup user drafts against the deployed baseline ([#9618](https://github.com/windmill-labs/windmill/issues/9618)) ([a2ce446](https://github.com/windmill-labs/windmill/commit/a2ce44645fdbfa98bf250fac2d15d2b5b26c4b47)) + + +### Bug Fixes + +* **frontend:** reset deleteWorkspaceForkModal on confirm in SidebarContent ([#9619](https://github.com/windmill-labs/windmill/issues/9619)) ([7cb5c6e](https://github.com/windmill-labs/windmill/commit/7cb5c6e749b2020dee5ee1499f0dc69c5109a6d8)) +* **frontend:** session Drafts drawer uses raw_app kind for the raw-app diff ([#9617](https://github.com/windmill-labs/windmill/issues/9617)) ([46288b6](https://github.com/windmill-labs/windmill/commit/46288b6143efae4dfdf6fe068b97a1e8831fce6a)) +* **nativets:** respect custom CA certs in in-process fetch runtime ([#9615](https://github.com/windmill-labs/windmill/issues/9615)) ([41562c7](https://github.com/windmill-labs/windmill/commit/41562c7d7c708d7d056d9b3d0c39b994a6f4a016)) +* **ResourceForm:** initialize JSON editor when resource type schema is unavailable ([#9611](https://github.com/windmill-labs/windmill/issues/9611)) ([5a24057](https://github.com/windmill-labs/windmill/commit/5a2405743b4622fc1021109114d007057abd5dfd)) +* show folder labels in the folder list table ([#9620](https://github.com/windmill-labs/windmill/issues/9620)) ([651fa13](https://github.com/windmill-labs/windmill/commit/651fa13ee80ff76e5a53ef1ed545b03ce6792294)) +* show last updated date per user in other-users-drafts modal ([#9614](https://github.com/windmill-labs/windmill/issues/9614)) ([f6104ce](https://github.com/windmill-labs/windmill/commit/f6104ce05c4005ffb9fe8112782d1ef6d3065300)) + +## [1.727.0](https://github.com/windmill-labs/windmill/compare/v1.726.1...v1.727.0) (2026-06-16) + + +### Features + +* support temp_script_refs in wmill dev for local relative imports ([#9554](https://github.com/windmill-labs/windmill/issues/9554)) ([33ac287](https://github.com/windmill-labs/windmill/commit/33ac287065742df53f363a5fe09f54f5584a85a6)) + + +### Bug Fixes + +* **cli:** harden legacy flow lock migration ordering and collision guard ([#9557](https://github.com/windmill-labs/windmill/issues/9557)) ([cd09870](https://github.com/windmill-labs/windmill/commit/cd098700c2cd8d7e9150f760938f4eaf34d188ec)) +* **cli:** include __mod/ folder in gitSyncIncludePattern for scripts ([#9606](https://github.com/windmill-labs/windmill/issues/9606)) ([252c1b3](https://github.com/windmill-labs/windmill/commit/252c1b35fc716c3486109d89615127c588bbe90a)) +* **frontend:** allow same-origin redirects in isValidLogoutRedirect ([#9568](https://github.com/windmill-labs/windmill/issues/9568)) ([8500435](https://github.com/windmill-labs/windmill/commit/8500435e82231e13a1b8a874fd0545f0a0a73fee)) +* **frontend:** make UserDraft read-after-write work without live entry ([#9609](https://github.com/windmill-labs/windmill/issues/9609)) ([51e82d7](https://github.com/windmill-labs/windmill/commit/51e82d7c6d30c66c84236feb743c09929934e564)) +* **frontend:** seed detached user-draft handles so new-item drawers render ([#9608](https://github.com/windmill-labs/windmill/issues/9608)) ([9e3c0de](https://github.com/windmill-labs/windmill/commit/9e3c0decf95378c66055d82215c15cd3bf4a69cb)) +* **frontend:** strip server-managed fields from value diffs ([#9599](https://github.com/windmill-labs/windmill/issues/9599)) ([c213801](https://github.com/windmill-labs/windmill/commit/c213801b5aee54d801c14b9eb31422f2a312ef7e)) + +## [1.726.1](https://github.com/windmill-labs/windmill/compare/v1.726.0...v1.726.1) (2026-06-15) + + +### Bug Fixes + +* **apps:** prevent decision tree graph editor crash on missing graph context ([#9602](https://github.com/windmill-labs/windmill/issues/9602)) ([24f3259](https://github.com/windmill-labs/windmill/commit/24f32596e9ca39c963c19af4a8b14fbcd04e3a78)) +* db-backed draft fixes — review-page UX, legacy drafts, session restore ([#9600](https://github.com/windmill-labs/windmill/issues/9600)) ([4e4b224](https://github.com/windmill-labs/windmill/commit/4e4b2247ef471dada1b8c894974fe921d14b3947)) + +## [1.726.0](https://github.com/windmill-labs/windmill/compare/v1.725.1...v1.726.0) (2026-06-15) + + +### Features + +* **audit:** record workspace archive/unarchive/delete in instance audit log ([#9596](https://github.com/windmill-labs/windmill/issues/9596)) ([9de5708](https://github.com/windmill-labs/windmill/commit/9de57086086bb5626d175c7f926915d1d6ac67ca)) +* **frontend:** add user-level toggle to disable Windmill AI ([#9585](https://github.com/windmill-labs/windmill/issues/9585)) ([5709a56](https://github.com/windmill-labs/windmill/commit/5709a564fbafd9aa91943572ecd8c3e0c45c20b1)) + + +### Bug Fixes + +* **embeddings:** retry HuggingFace model downloads with backoff ([#9597](https://github.com/windmill-labs/windmill/issues/9597)) ([6a62959](https://github.com/windmill-labs/windmill/commit/6a6295921d681359155d814507908792be405679)) +* resolve release CI failures (pypi bundle, flow serde test, cli windows) ([#9595](https://github.com/windmill-labs/windmill/issues/9595)) ([5ccaae8](https://github.com/windmill-labs/windmill/commit/5ccaae8ab36f2be18b67863ea069763455908029)) + +## [1.725.1](https://github.com/windmill-labs/windmill/compare/v1.725.0...v1.725.1) (2026-06-15) + + +### Bug Fixes + +* **apps:** apply scope-path predicate to app list/search endpoints ([#9581](https://github.com/windmill-labs/windmill/issues/9581)) ([3bf6e10](https://github.com/windmill-labs/windmill/commit/3bf6e102afbdad41e558617bc812012eaaaecd9b)) +* **auth:** add scope checks to scripts/flows list_tokens endpoints ([#9582](https://github.com/windmill-labs/windmill/issues/9582)) ([36c9f86](https://github.com/windmill-labs/windmill/commit/36c9f8612b5778aa2c981454729590b71671ce8d)) +* **cli:** preserve committed script.lock on transient NULL lock during git-sync deploy ([#9593](https://github.com/windmill-labs/windmill/issues/9593)) ([6b916ac](https://github.com/windmill-labs/windmill/commit/6b916ac688e0305284e6cf819bf28803bcca0118)) +* expose parent_hash in MCP createScript tool for updates ([#9586](https://github.com/windmill-labs/windmill/issues/9586)) ([a69505d](https://github.com/windmill-labs/windmill/commit/a69505df9bf25d7c4f11d0528a7450c08dbb422c)) +* **flows:** stop serializing default retry/stop_after_if fields ([#9583](https://github.com/windmill-labs/windmill/issues/9583)) ([e1e2a24](https://github.com/windmill-labs/windmill/commit/e1e2a24b6a6752b3ac779cb0db38061cbc54425e)) +* **security:** sanitize dependency names & connection strings against command/SQL injection ([#9590](https://github.com/windmill-labs/windmill/issues/9590)) ([aff0a4e](https://github.com/windmill-labs/windmill/commit/aff0a4ec189cd8e315282e878bb858ef00635b90)) + +## [1.725.0](https://github.com/windmill-labs/windmill/compare/v1.724.0...v1.725.0) (2026-06-15) + + +### Features + +* Db-backed user drafts ([#9351](https://github.com/windmill-labs/windmill/issues/9351)) ([1fc3557](https://github.com/windmill-labs/windmill/commit/1fc355709c025fd256c5a4035356e15a5a05b23d)) +* scope AI session storage per user, session list in IndexedDB ([#9518](https://github.com/windmill-labs/windmill/issues/9518)) ([aa26c4d](https://github.com/windmill-labs/windmill/commit/aa26c4d9b22b3a353a6c0605eb9a4193e34aa18c)) + + +### Bug Fixes + +* **powershell:** sanitize module names to prevent command injection (CWE-78) ([#9587](https://github.com/windmill-labs/windmill/issues/9587)) ([6acce7a](https://github.com/windmill-labs/windmill/commit/6acce7a88733683153db534cb18752b31d93af82)) + +## [1.724.0](https://github.com/windmill-labs/windmill/compare/v1.723.0...v1.724.0) (2026-06-15) + + +### Features + +* **cli:** add --yes, --secret/--no-secret and --description to variable add ([#9548](https://github.com/windmill-labs/windmill/issues/9548)) ([4e9e0c0](https://github.com/windmill-labs/windmill/commit/4e9e0c024b4b95f9676b1646591d8f0c662e84ab)) +* **frontend:** improve AI chat cancel and interrupted-turn handling ([#9539](https://github.com/windmill-labs/windmill/issues/9539)) ([114c412](https://github.com/windmill-labs/windmill/commit/114c41251a8c738b58a1a3dd9434d09d33feb6f1)) +* **frontend:** precise AI chat context usage tracking + indicator ([#9551](https://github.com/windmill-labs/windmill/issues/9551)) ([2b47180](https://github.com/windmill-labs/windmill/commit/2b471805bf1c92bb210cfacda217a4341e1f989c)) +* wire chat reasoning effort through gemini and bedrock proxies ([#9545](https://github.com/windmill-labs/windmill/issues/9545)) ([aaf0563](https://github.com/windmill-labs/windmill/commit/aaf05635cedadc73455dc522474b673719f9fd5c)) + + +### Bug Fixes + +* actually isolate windows job children from CTRL_BREAK_EVENT + reap on worker death ([#9563](https://github.com/windmill-labs/windmill/issues/9563)) ([61f3291](https://github.com/windmill-labs/windmill/commit/61f3291b240bdb5c26bee8947351a9590bc3bd45)) +* **ai:** enforce resource authz when loading MCP tools in agent worker ([#9571](https://github.com/windmill-labs/windmill/issues/9571)) ([317a862](https://github.com/windmill-labs/windmill/commit/317a8629d1c8436d2a6f3443bd25b81d606ce283)) +* append system CA bundle to tracing proxy cert file ([#9549](https://github.com/windmill-labs/windmill/issues/9549)) ([3cf4083](https://github.com/windmill-labs/windmill/commit/3cf40839602e5c3d1df51f0a29b01736bade09da)) +* **cli:** consistent flow inline lock filenames for compound extensions ([#9555](https://github.com/windmill-labs/windmill/issues/9555)) ([f0659a7](https://github.com/windmill-labs/windmill/commit/f0659a755a161420833e3bfdbe04befc6ebeb977)) +* **flows:** skip_if evaluates wrong previous_result during retry ([#9547](https://github.com/windmill-labs/windmill/issues/9547)) ([2aab352](https://github.com/windmill-labs/windmill/commit/2aab35245c362c2f911c60ea435f29bfb1369ebf)) +* **folders:** allow hyphens in folder names ([#9566](https://github.com/windmill-labs/windmill/issues/9566)) ([84df111](https://github.com/windmill-labs/windmill/commit/84df11177f2009bff007e9b722b55a9a5a63c06a)), closes [#8474](https://github.com/windmill-labs/windmill/issues/8474) +* **frontend:** load resource value in JSON editor when resource type is missing ([#9574](https://github.com/windmill-labs/windmill/issues/9574)) ([251266c](https://github.com/windmill-labs/windmill/commit/251266cd8119dbef314314daed43aa43ab92f1c9)) +* isolate windows job children from worker CTRL_BREAK_EVENT ([#9562](https://github.com/windmill-labs/windmill/issues/9562)) ([1d6191e](https://github.com/windmill-labs/windmill/commit/1d6191ebb75843917eec6c76a4be347f9ac4cb72)) +* stop sending temperature for AI chat across all providers ([#9553](https://github.com/windmill-labs/windmill/issues/9553)) ([3585716](https://github.com/windmill-labs/windmill/commit/358571687296fbe5c378533b3c1662707955c64a)) + ## [1.723.0](https://github.com/windmill-labs/windmill/compare/v1.722.0...v1.723.0) (2026-06-11) diff --git a/Dockerfile b/Dockerfile index d327c9b394..df03ffbdc0 100644 --- a/Dockerfile +++ b/Dockerfile @@ -163,14 +163,14 @@ ENV PATH /usr/local/bin:/root/.local/bin:/tmp/.local/bin:$PATH RUN apt-get update \ - && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 software-properties-common tini gnupg lsb-release \ + && apt-get install -y --no-install-recommends netbase tzdata ca-certificates wget curl jq unzip build-essential unixodbc xmlsec1 tini gnupg libargon2-1 \ && if echo "$features" | grep -q "ee"; then apt-get install -y --no-install-recommends libsasl2-modules-gssapi-mit krb5-user; fi \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* # Install latest PostgreSQL client (pg_dump) from official PostgreSQL apt repository RUN curl -fsSL https://www.postgresql.org/media/keys/ACCC4CF8.asc | gpg --dearmor -o /usr/share/keyrings/postgresql-archive-keyring.gpg \ - && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ + && echo "deb [signed-by=/usr/share/keyrings/postgresql-archive-keyring.gpg] https://apt.postgresql.org/pub/repos/apt $(. /etc/os-release; echo "$VERSION_CODENAME")-pgdg main" > /etc/apt/sources.list.d/pgdg.list \ && apt-get update \ && apt-get install -y --no-install-recommends postgresql-client \ && apt-get clean \ @@ -184,11 +184,11 @@ RUN if [ "$WITH_GIT" = "true" ]; then \ else echo 'Building the image without git'; fi; RUN if [ "$WITH_POWERSHELL" = "true" ]; then \ - if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \ + if [ "$TARGETPLATFORM" = "linux/amd64" ]; then apt-get update -y && apt install libicu72 -y && wget -O 'pwsh.deb' "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell_${POWERSHELL_DEB_VERSION}.deb_amd64.deb" && apt-get clean \ && rm -rf /var/lib/apt/lists/* && \ dpkg --install 'pwsh.deb' && \ rm 'pwsh.deb'; \ - elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu-dev -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \ + elif [ "$TARGETPLATFORM" = "linux/arm64" ]; then apt-get update -y && apt install libicu72 -y && wget -O powershell.tar.gz "https://github.com/PowerShell/PowerShell/releases/download/v${POWERSHELL_VERSION}/powershell-${POWERSHELL_VERSION}-linux-arm64.tar.gz" && apt-get clean \ && rm -rf /var/lib/apt/lists/* && \ mkdir -p /opt/microsoft/powershell/7 && \ tar zxf powershell.tar.gz -C /opt/microsoft/powershell/7 && \ @@ -233,7 +233,7 @@ ENV PATH="${PATH}:/usr/local/go/bin" ENV GO_PATH=/usr/local/go/bin/go # Install UV -RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.9.25/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv +RUN curl --proto '=https' --tlsv1.2 -LsSf https://github.com/astral-sh/uv/releases/download/0.11.24/uv-installer.sh | sh && mv /root/.local/bin/uv /usr/local/bin/uv # Preinstall python runtimes to temp build location (will copy with world-writable perms later) # --compile-bytecode precompiles the stdlib to .pyc so jobs don't recompile it on every run diff --git a/ai_evals/AGENTS.md b/ai_evals/AGENTS.md index af5427abae..6e63c4037e 100644 --- a/ai_evals/AGENTS.md +++ b/ai_evals/AGENTS.md @@ -1,208 +1,14 @@ -# AI Evals Authoring Guide +# AI Evals -This folder contains black-box benchmark cases for: +Black-box benchmark cases for the Windmill AI generation modes (`flow`, `app`, +`script`, `cli`, `global`). -- `flow` -- `app` -- `script` -- `cli` -- `global` +**Authoring and running cases is documented in the `ai-evals` skill** — load it +before adding/changing a case or running a benchmark. Claude Code reads +`.claude/skills/ai-evals/SKILL.md`; Codex and Pi read +`.agents/skills/ai-evals/SKILL.md` (same canonical file). Invoke with `/ai-evals` in +Claude Code, `$ai-evals` in Codex, or `pi --skill ai-evals`. -The goal is to test the current production prompts and guidance with realistic user requests, not to test one exact implementation shape. +For AI chat / copilot changes that these evals measure, see the `ai-chat` skill. -## Core rules - -1. Write prompts like a real user request. -2. Prefer behavior, inputs, constraints, and outcomes over internal implementation details. -3. Keep deterministic validation narrow and hard. -4. Put semantic expectations in `judgeChecklist`. -5. Use `expected` fixtures only when exact structure really matters. - -## Prompt writing - -Prompts should sound like something a user would naturally ask. - -Good: - -- "Create a flow that routes support requests based on customer tier." -- "Add a reset button that sets the counter back to 0." -- "Create a flow that reuses the existing greeting script instead of duplicating the logic." - -Bad: - -- "Use `branchone` with 3 branches and a default branch." -- "Create a `rawscript` step with this exact topology." -- "This is a benchmark harness." - -Do not write prompts as if the user knows Windmill internals unless the case is explicitly testing a power-user workflow. - -## Flow-specific rules - -This is the main principle you asked for: - -- flow prompts should read like requests from a user who does not know the product internals -- the user should ask for behavior, not for `branchone`, `branchall`, `rawscript`, `preprocessor_module`, `failure_module`, exact graph topology, or other internal constructs - -That means: - -- creation cases should describe the business behavior and expected result -- modification cases may mention existing step names, because the user can see the current flow -- only mention special Windmill constructs when the case is explicitly about those constructs - -Examples: - -- acceptable creation prompt: - "Create a purchase approval flow that pauses for approval and asks the approver for a comment." -- avoid: - "Create a suspend step with one required event and a resume form." - -For flow cases, do not fail a case just because the model chose a different valid topology. - -## App-specific rules - -App prompts should focus on user-visible behavior: - -- what the UI should let the user do -- what should persist -- what backend behavior is needed - -Avoid prompting in terms of React structure, component names, or implementation unless the case is specifically about editing an existing app. - -## CLI-specific rules - -CLI prompts can be more explicit about paths and file names because real CLI users often do specify them. - -Still, avoid benchmark phrasing. The prompt should read like a repo task, not a harness instruction. - -When relevant, ask the assistant to tell the user which `wmill` commands to run next. That is part of the benchmarked behavior. - -## Global-specific rules - -Global prompts should exercise workspace-level drafting behavior: - -- inspecting existing scripts, flows, apps, schedules, triggers, resources, and variables when relevant -- writing AI drafts rather than saving or deploying by default -- producing coherent multi-artifact changes when the request crosses artifact boundaries - -Keep deterministic validation focused on the draft contract: required draft type/path, required content snippets, forbidden draft paths, and forbidden mutating tools such as deploy/delete unless the case explicitly asks for them. - -Datatable cases should set `skipJudge: true` and validate through tool-use -(`requiredToolsUsed` / `forbiddenToolsUsed`) and SQL-argument assertions -(`toolCallArgs` with `stringIncludesAnyOf`, e.g. `['select']`, `['create table']`, -`['update', 'insert into']`). Two reasons the judge is unreliable here: - -- `list_datatables`, `get_datatable_table_schema`, and `exec_datatable_sql` - produce no drafts, and the global judge only sees the drafts artifact — it - scores a no-draft conversational answer as empty (same as the - `askUserQuestion` cases). -- Even a case that *does* produce a draft (a script reading the data table via - `wmill.datatable()` at runtime) is mis-judged: the judge has no datatable SDK - reference and penalizes correct `wmill.datatable()` usage as wrong. Verify the - SDK call deterministically instead — `requiredDrafts.valueIncludes: ['wmill.datatable(']` - plus forbidding `exec_datatable_sql` (keeping chat-time SQL distinct from - runtime SDK use). - -`stringIncludesAnyOf` is existential over calls (at least one matching call), so a -mutation case still passes when the model mixes its UPDATE/INSERT with -verification SELECTs. The in-memory engine (`datatableSqlEngine.ts`) is stateful -within a case — writes persist, so a model that re-queries to verify its -CREATE/UPDATE sees the change and does not loop. But the engine is best-effort -(SELECT returns all rows of the referenced/first table with no WHERE/projection), -so still never assert specific returned row values. Seed data via -`workspace.datatables` in the `initial` fixture (see README). - -## Deterministic validation - -Use deterministic validation only for hard failures such as: - -- missing required files -- unexpected extra files when the prompt says not to create them -- syntax errors -- unresolved flow refs -- missing required special modules or suspend config -- obvious artifact corruption - -Do not use deterministic validation to enforce one preferred implementation for broad creation tasks. - -Examples of bad hard checks: - -- exact step topology for a creation flow -- exact branch structure when the prompt only asked for routing behavior -- exact input shape when multiple reasonable shapes are acceptable - -## Judge checklist - -Every non-trivial case should have a `judgeChecklist`. - -The checklist should capture: - -- the user-visible behavior that must be present -- important constraints -- key completion criteria - -The checklist should not duplicate low-level implementation details unless they are truly required by the task. - -Good checklist items: - -- "the flow calculates the order total with 8% tax" -- "the app persists recipes appropriately for a raw Windmill app" -- "the flow reuses the existing workspace script instead of rewriting the logic" - -Bad checklist items: - -- "uses `branchone`" -- "contains a `rawscript` node" - -## When to use `expected` - -Use `expected` fixtures when the case is structure-sensitive, for example: - -- exact file creation -- exact script content -- modification cases where a specific file must change in a specific way -- cases where preserving an existing structure is part of the requirement - -Do not use a full `expected` artifact as the semantic oracle for broad creation tasks when multiple valid outputs should pass. - -## When to use `initial` - -Use `initial` when the benchmark is about: - -- editing an existing artifact -- reusing existing workspace assets -- preserving existing behavior while adding a change - -If the case is greenfield, prefer no `initial`. - -## Case design ladder - -Prefer suites that get gradually harder: - -1. trivial create case -2. realistic create case -3. reuse-existing-assets case -4. modification case -5. refactor case -6. edge-case or niche product behavior - -The last cases in a suite should cover unusual or product-specific behavior. - -## Anti-patterns - -Avoid these: - -- benchmark framing in prompts -- over-specified internal topology for creation tasks -- judge checklists that just restate implementation details -- deterministic validation that encodes one preferred solution -- fixtures that are so minimal or brittle that they create false negatives - -## Before adding a case - -Ask: - -1. Would a real user plausibly write this prompt? -2. If the model solves it in a different valid way, would the case still pass? -3. Are the hard deterministic checks only catching objectively broken output? -4. Does the `judgeChecklist` describe the real success criteria? -5. If this case fails, will the reason be understandable from the saved artifacts? +The full case format, fields, and fixture details remain in `ai_evals/README.md`. diff --git a/ai_evals/README.md b/ai_evals/README.md index 88825f4ee7..33bb47ae46 100644 --- a/ai_evals/README.md +++ b/ai_evals/README.md @@ -75,6 +75,8 @@ Public CLI surface: - `--model `: choose the model under test - `--models `: run the same cases sequentially against several model aliases - `--verbose`: stream assistant output for frontend runs +- `--skip-judge`: skip LLM judge scoring for the run +- `--execution-only`: only require the model/proxy/frontend loop to complete; skip validators, tool expectations, backend artifact validation, and judge scoring - `--record`: append a compact tracked summary line to `ai_evals/history/.jsonl` for full-suite runs only - `--backend-validation `: optional backend smoke validation (`off` or `preview`) for `script` and `flow` evals @@ -99,7 +101,7 @@ Notes: - the command also prints accepted alias spellings such as `gpt-4o`, `gpt-55`, `claude-opus-4.6`, and `claude-haiku-4.5` - frontend modes (`flow`, `script`, `app`, `global`) can use Anthropic, OpenAI, Gemini, and DeepSeek-backed aliases - `cli` mode always uses the Anthropic agent SDK, so only Anthropic aliases are valid there -- the judge model is separate and currently defaults to `claude-sonnet-4-6` +- the judge model is separate and currently defaults to `claude-sonnet-4-6`; use `--skip-judge` for deterministic-only runs ## Case Format diff --git a/ai_evals/adapters/cli/runtime.ts b/ai_evals/adapters/cli/runtime.ts index 2bcc0abc04..b45df70011 100644 --- a/ai_evals/adapters/cli/runtime.ts +++ b/ai_evals/adapters/cli/runtime.ts @@ -16,6 +16,9 @@ export interface PromptRunResult { output: string; durationMs: number; tokenUsage: BenchmarkTokenUsage | null; + // Input tokens on the last assistant turn. The SDK `result` message reports + // usage cumulatively, so the final context size comes from per-turn usage. + finalContextTokens: number | null; trace: CliTrace; } @@ -144,6 +147,7 @@ export async function runPromptAndCapture( let output = ""; let assistantMessageCount = 0; let tokenUsage: BenchmarkTokenUsage | null = null; + let finalContextTokens: number | null = null; const startedAt = Date.now(); const stubBinDir = join(cwd, WMILL_STUB_DIR_NAME); const wmillLogPath = join(cwd, WMILL_LOG_FILE_NAME); @@ -166,6 +170,12 @@ export async function runPromptAndCapture( for await (const message of query({ prompt, options })) { if (message.type === "assistant") { assistantMessageCount += 1; + const turnContext = anthropicUsageToBenchmarkTokenUsage( + message.message?.usage + )?.prompt; + if (turnContext && turnContext > 0) { + finalContextTokens = turnContext; + } const content = message.message?.content; if (Array.isArray(content)) { for (const block of content) { @@ -210,6 +220,7 @@ export async function runPromptAndCapture( output, durationMs: Date.now() - startedAt, tokenUsage, + finalContextTokens, trace: { toolsUsed, skillsInvoked, diff --git a/ai_evals/adapters/frontend/benchmarkRunner.ts b/ai_evals/adapters/frontend/benchmarkRunner.ts index 1729df7170..b45f880699 100644 --- a/ai_evals/adapters/frontend/benchmarkRunner.ts +++ b/ai_evals/adapters/frontend/benchmarkRunner.ts @@ -25,6 +25,12 @@ export async function runFrontendBenchmarkFromEnv(): Promise ); const emitProgress = process.env.WMILL_FRONTEND_AI_EVAL_PROGRESS === "1"; const verbose = process.env.WMILL_FRONTEND_AI_EVAL_VERBOSE === "1"; + const executionOnly = + process.env.WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY === "1"; + const judgeModel = + process.env.WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE === "1" || executionOnly + ? null + : DEFAULT_JUDGE_MODEL; const model = resolveEvalModel( mode, process.env.WMILL_FRONTEND_AI_EVAL_MODEL, @@ -48,7 +54,8 @@ export async function runFrontendBenchmarkFromEnv(): Promise cases: selectedCases, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, + executionOnly, concurrency: verbose ? 1 : undefined, verbose, onProgress: emitProgress @@ -60,7 +67,7 @@ export async function runFrontendBenchmarkFromEnv(): Promise mode, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, caseResults, }); } @@ -96,7 +103,12 @@ async function getModeRunner( } function parseMode(value: string | undefined): FrontendBenchmarkMode { - if (value === "flow" || value === "app" || value === "script" || value === "global") { + if ( + value === "flow" || + value === "app" || + value === "script" || + value === "global" + ) { return value; } throw new Error(`Unsupported frontend benchmark mode: ${String(value)}`); diff --git a/ai_evals/adapters/frontend/core/app/appEvalRunner.ts b/ai_evals/adapters/frontend/core/app/appEvalRunner.ts index 16543b28de..fa52e6e826 100644 --- a/ai_evals/adapters/frontend/core/app/appEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/app/appEvalRunner.ts @@ -38,6 +38,7 @@ export interface AppEvalResult { toolCallCount: number; toolsUsed: string[]; tokenUsage: TokenUsage; + finalContextTokens: number | null; } export interface AppEvalOptions { @@ -113,6 +114,7 @@ export async function runAppEval( toolCallCount: rawResult.toolCallsCount, toolsUsed: rawResult.toolsCalled, tokenUsage: rawResult.tokenUsage, + finalContextTokens: rawResult.finalContextTokens, }; } finally { await cleanup(); diff --git a/ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts b/ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts index 0cd25f5787..f533595ed2 100644 --- a/ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/flow/flowEvalRunner.ts @@ -39,6 +39,7 @@ export interface FlowEvalResult { toolsUsed: string[]; toolCallDetails: ToolCallDetail[]; tokenUsage: TokenUsage; + finalContextTokens: number | null; } export interface FlowEvalOptions { @@ -113,6 +114,7 @@ export async function runFlowEval( toolsUsed: rawResult.toolsCalled, toolCallDetails: rawResult.toolCallDetails, tokenUsage: rawResult.tokenUsage, + finalContextTokens: rawResult.finalContextTokens, }; } finally { await cleanup(); diff --git a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts index 058adc3644..e553d15f56 100644 --- a/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/global/globalEvalRunner.ts @@ -9,6 +9,7 @@ import { } from "../../../../../frontend/src/lib/components/copilot/chat/global/core"; import { clearGlobalDrafts, + getGlobalDraft, listGlobalDrafts, } from "../../../../../frontend/src/lib/components/copilot/chat/global/userDraftAdapter"; import type { Tool as ProductionTool } from "../../../../../frontend/src/lib/components/copilot/chat/shared"; @@ -18,6 +19,7 @@ import type { GlobalDraftState } from "../../../../core/validators"; import type { WindmillBackendSettings } from "../../../../core/windmillBackendSettings"; import { registerBenchmarkWorkspaceRunnables, + seedBenchmarkDraft, unregisterBenchmarkWorkspaceRunnables, type BenchmarkWorkspaceRunnables, } from "../../mockBackend"; @@ -30,6 +32,10 @@ const MUTATING_GLOBAL_TOOLS = new Set([ ]); const DISABLE_ACTIVE_EDITOR_CONTEXT_ENV = "WMILL_AI_EVAL_DISABLE_ACTIVE_EDITOR_CONTEXT"; +// A/B gate for the search_app read tool: set to "1" to run the baseline arm +// (toolset without search_app) so its token cost can be compared against the arm +// that offers it. +const DISABLE_SEARCH_APP_ENV = "WMILL_AI_EVAL_DISABLE_SEARCH_APP"; const LIVE_EDITOR_ITEM_KINDS = { script: "script", @@ -44,6 +50,20 @@ export interface GlobalLiveEditorDraftFixture { value?: unknown; } +// Identity the global system prompt builds paths from. Production reads +// `userStore` (whoami) to fill `u/{username}/...`; the eval harness never logs +// in, so without this the prompt sees an empty username (`u//...`) and no +// path-selection case is meaningful. Seeded per-case via the initial fixture and +// passed straight to `prepareGlobalSystemMessage` (no global-store mutation). +export interface GlobalUserFixture { + username: string; + is_admin?: boolean; + /** Folders the user can write to (the writable set whoami returns). */ + folders?: string[]; + /** Folders the user can read; read-only folders = folders_read \ folders. */ + folders_read?: string[]; +} + export interface GlobalEvalResult { success: boolean; state: GlobalDraftState; @@ -53,11 +73,13 @@ export interface GlobalEvalResult { toolsUsed: string[]; toolCallDetails: ToolCallDetail[]; tokenUsage: TokenUsage; + finalContextTokens: number | null; } export interface GlobalEvalOptions { workspaceFixtures?: BenchmarkWorkspaceRunnables; liveEditorDrafts?: GlobalLiveEditorDraftFixture[]; + user?: GlobalUserFixture; model?: string; maxIterations?: number; provider?: AIProvider; @@ -83,9 +105,11 @@ export async function runGlobalEval( const model = options.model ?? "claude-haiku-4-5-20251001"; const injectActiveEditorContext = process.env[DISABLE_ACTIVE_EDITOR_CONTEXT_ENV] !== "1"; + // Pass the seeded identity straight to the prompt builder rather than mutating + // the process-global `userStore`, so concurrent cases never race on it. const rawResult = await runEval({ userPrompt, - systemMessage: prepareGlobalSystemMessage(), + systemMessage: prepareGlobalSystemMessage(undefined, { user: options.user }), userMessage: prepareGlobalUserMessage( userPrompt, [], @@ -94,7 +118,7 @@ export async function runGlobalEval( tools: getGlobalEvalTools(), helpers: {}, apiKey, - getOutput: () => ({ drafts: listGlobalDrafts(workspaceRoot) }), + getOutput: () => collectGlobalDraftState(workspaceRoot), onAssistantMessageStart: options.runContext?.onAssistantMessageStart, onAssistantToken: options.runContext?.onAssistantChunk, onAssistantMessageEnd: options.runContext?.onAssistantMessageEnd, @@ -119,6 +143,7 @@ export async function runGlobalEval( toolsUsed: rawResult.toolsCalled, toolCallDetails: rawResult.toolCallDetails, tokenUsage: rawResult.tokenUsage, + finalContextTokens: rawResult.finalContextTokens, }; } finally { clearGlobalDrafts(workspaceRoot); @@ -130,6 +155,32 @@ export async function runGlobalEval( } } +// Build the harness output from the DB-backed drafts. `listGlobalDrafts` returns +// metadata-only rows for backend drafts (the model's `write_script` etc. persist +// straight to the backend with no in-tab editor cell), so re-read each such row +// with `getGlobalDraft` to attach the full value the validators assert on. A row +// that already carries a value (the production in-tab cell overlay) is kept as-is. +async function collectGlobalDraftState( + workspace: string, +): Promise { + const items = await listGlobalDrafts(workspace); + const drafts = await Promise.all( + items.map(async (item) => { + if (item.value !== undefined) { + return item; + } + const full = await getGlobalDraft( + workspace, + item.type, + item.path, + item.triggerKind, + ); + return full ?? item; + }), + ); + return { drafts: drafts as GlobalDraftState["drafts"] }; +} + function seedLiveEditorDrafts( workspace: string, fixtures: GlobalLiveEditorDraftFixture[], @@ -138,7 +189,9 @@ function seedLiveEditorDrafts( const itemKind = LIVE_EDITOR_ITEM_KINDS[fixture.type]; const storagePath = fixture.storagePath ?? fixture.effectivePath ?? ""; if (fixture.value !== undefined) { - UserDraft.save(itemKind, storagePath, fixture.value, { workspace }); + // Seed as a backend draft row, not an in-tab cell: a cell would shadow the + // model's DB-backed edit when the output is read back via listGlobalDrafts. + seedBenchmarkDraft(workspace, itemKind, storagePath, fixture.value); } UserDraft.setLiveEditorDraft({ workspace, @@ -161,25 +214,28 @@ function clearLiveEditorDrafts( } function getGlobalEvalTools(): ProductionTool<{}>[] { - return (globalTools as ProductionTool<{}>[]).map((tool) => { - if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) { - return tool; - } + const disableSearchApp = process.env[DISABLE_SEARCH_APP_ENV] === "1"; + return (globalTools as ProductionTool<{}>[]) + .filter((tool) => !(disableSearchApp && tool.def.function.name === "search_app")) + .map((tool) => { + if (!MUTATING_GLOBAL_TOOLS.has(tool.def.function.name)) { + return tool; + } - return { - ...tool, - requiresConfirmation: false, - validateBeforeConfirmation: undefined, - fn: async () => - JSON.stringify( - { - success: false, - error: - "This mutating workspace tool is disabled during ai_evals global mode.", - }, - null, - 2, - ), - }; - }); + return { + ...tool, + requiresConfirmation: false, + validateBeforeConfirmation: undefined, + fn: async () => + JSON.stringify( + { + success: false, + error: + "This mutating workspace tool is disabled during ai_evals global mode.", + }, + null, + 2, + ), + }; + }); } diff --git a/ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts b/ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts index 95ce6555e6..b9c80f3554 100644 --- a/ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/script/scriptEvalRunner.ts @@ -25,6 +25,7 @@ export interface ScriptEvalResult { toolsUsed: string[]; toolCallDetails: ToolCallDetail[]; tokenUsage: TokenUsage; + finalContextTokens: number | null; } export interface ScriptEvalOptions { @@ -111,6 +112,7 @@ export async function runScriptEval( toolsUsed: rawResult.toolsCalled, toolCallDetails: rawResult.toolCallDetails, tokenUsage: rawResult.tokenUsage, + finalContextTokens: rawResult.finalContextTokens, }; } finally { await cleanup(); diff --git a/ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts b/ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts index 5f4ea2c307..f787743dfa 100644 --- a/ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts +++ b/ai_evals/adapters/frontend/core/shared/baseEvalRunner.ts @@ -38,8 +38,9 @@ export interface RunEvalParams { helpers: THelpers; /** API key for the provider */ apiKey: string; - /** Function to get the current output state */ - getOutput: () => TOutput; + /** Function to get the current output state. May be async — global mode reads + * DB-backed drafts back through the (mocked) backend to build its output. */ + getOutput: () => TOutput | Promise; /** Model and Windmill backend configuration */ options: EvalRunnerOptions; onAssistantMessageStart?: () => void; @@ -154,9 +155,10 @@ export async function runEval( if (result.hitMaxIterations) { return { success: false, - output: getOutput(), + output: (await getOutput()) as TOutput, error: `Reached max turns (${maxIterations})`, tokenUsage: result.tokenUsage, + finalContextTokens: result.lastIterationUsage?.prompt ?? null, toolCallsCount, toolsCalled, toolCallDetails, @@ -170,8 +172,9 @@ export async function runEval( return { success: true, - output: getOutput(), + output: (await getOutput()) as TOutput, tokenUsage: result.tokenUsage, + finalContextTokens: result.lastIterationUsage?.prompt ?? null, toolCallsCount, toolsCalled, toolCallDetails, @@ -191,9 +194,10 @@ export async function runEval( return { success: false, - output: getOutput(), + output: (await getOutput()) as TOutput, error: errorMessage, tokenUsage: { prompt: 0, completion: 0, total: 0 }, + finalContextTokens: null, toolCallsCount, toolsCalled, toolCallDetails, diff --git a/ai_evals/adapters/frontend/core/shared/types.ts b/ai_evals/adapters/frontend/core/shared/types.ts index f081fe398a..c7af6c43e7 100644 --- a/ai_evals/adapters/frontend/core/shared/types.ts +++ b/ai_evals/adapters/frontend/core/shared/types.ts @@ -28,6 +28,8 @@ export interface RawEvalResult { output: TOutput; error?: string; tokenUsage: TokenUsage; + /** Input tokens on the last model request of the loop (see BenchmarkAttemptResult.finalContextTokens). */ + finalContextTokens: number | null; toolCallsCount: number; toolsCalled: string[]; toolCallDetails: ToolCallDetail[]; diff --git a/ai_evals/adapters/frontend/mockBackend.ts b/ai_evals/adapters/frontend/mockBackend.ts index 622c79bf46..379eb6d447 100644 --- a/ai_evals/adapters/frontend/mockBackend.ts +++ b/ai_evals/adapters/frontend/mockBackend.ts @@ -1,9 +1,20 @@ import { randomUUID } from 'node:crypto' -import type { CompletedJob, Flow, Job, Script } from '../../../frontend/src/lib/gen' +import type { + AppWithLastVersion, + CompletedJob, + Flow, + Job, + ListableApp, + Script +} from '../../../frontend/src/lib/gen' import type { DataTableTables, DataTableTableSchema, - ScriptLang + GetDraftForUserResponse, + ListDraftsResponse, + ScriptLang, + UpdateDraftResponse, + UserDraftItemKind } from '../../../frontend/src/lib/gen/types.gen' import { buildScriptLintResult } from './core/script/preview' import { applyDatatableSql, type BenchmarkDatatableSeed } from './datatableSqlEngine' @@ -29,6 +40,18 @@ export interface BenchmarkWorkspaceFlow { value: Flow['value'] } +export interface BenchmarkWorkspaceApp { + path: string + summary: string + value: { + files: Record + runnables: Record + data?: unknown + policy?: unknown + custom_path?: unknown + } +} + export interface BenchmarkWorkspaceJob { /** Stable id so a case prompt can reference a specific run (e.g. for get_job_logs). */ id?: string @@ -43,6 +66,7 @@ export interface BenchmarkWorkspaceJob { export interface BenchmarkWorkspaceRunnables { scripts?: BenchmarkWorkspaceScript[] flows?: BenchmarkWorkspaceFlow[] + apps?: BenchmarkWorkspaceApp[] datatables?: BenchmarkDatatableSeed[] jobs?: BenchmarkWorkspaceJob[] } @@ -63,6 +87,14 @@ export function resetBenchmarkMockBackend(): void { benchmarkWorkspaces.clear() benchmarkWorkspaceRunnables.clear() benchmarkJobs.clear() + benchmarkDrafts.clear() +} + +// Stand-in for FolderService.createFolder so the global create_folder tool runs in +// memory instead of mutating the real backend. Folders aren't otherwise modelled +// (no folder-listing in evals), so this just echoes the created name. +export function createBenchmarkFolder(_workspace: string, name: string): string { + return name } export function registerBenchmarkWorkspace(workspace: string): void { @@ -74,6 +106,8 @@ export function registerBenchmarkWorkspaceRunnables( runnables: BenchmarkWorkspaceRunnables ): void { benchmarkWorkspaces.add(workspace) + // Fresh case: drop any drafts left from a prior run on this workspace id. + clearBenchmarkDrafts(workspace) // Datatables are mutated in place by exec_datatable_sql (a write must be visible // to later reads), so store an isolated deep copy — never mutate the caller's seed. benchmarkWorkspaceRunnables.set(workspace, { @@ -98,6 +132,7 @@ export function registerBenchmarkWorkspaceRunnables( export function unregisterBenchmarkWorkspace(workspace: string): void { benchmarkWorkspaces.delete(workspace) benchmarkWorkspaceRunnables.delete(workspace) + clearBenchmarkDrafts(workspace) for (const [jobId, entry] of benchmarkJobs.entries()) { if (entry.workspace === workspace) { benchmarkJobs.delete(jobId) @@ -153,6 +188,22 @@ export function getBenchmarkFlowByPath(workspace: string, path: string): Flow | return flow ? buildBenchmarkFlow(flow) : null } +export function listBenchmarkApps(workspace: string): ListableApp[] | null { + const runnables = benchmarkWorkspaceRunnables.get(workspace) + if (!runnables) { + return null + } + return (runnables.apps ?? []).map(buildBenchmarkListableApp) +} + +export function getBenchmarkAppByPath(workspace: string, path: string): AppWithLastVersion | null { + const app = benchmarkWorkspaceRunnables + .get(workspace) + ?.apps?.find((entry) => entry.path === path) + + return app ? buildBenchmarkApp(app) : null +} + export function createBenchmarkCompletedJob(input: { workspace: string jobKind: CompletedJob['job_kind'] @@ -238,6 +289,110 @@ export function getBenchmarkJobLogs(workspace: string, jobId: string): string { return job.logs ?? '' } +// ============= Drafts (per-user, DB-backed in production) ============= + +/** + * In-memory stand-in for the per-user draft backend (`DraftService`). The global + * AI chat now persists and reads drafts through the backend DB instead of an + * in-tab `UserDraft` cell, so the eval mocks the three draft endpoints it + * exercises (`updateDraft` / `getDraftForUser` / `listDrafts`) and keeps the + * saved values here, keyed by workspace + draft kind + storage path. Mirrors the + * semantics of the production unit test's mock in + * `frontend/src/lib/components/copilot/chat/global/core.test.ts`. + */ +const benchmarkDrafts = new Map< + string, + { workspace: string; kind: UserDraftItemKind; path: string; value: unknown } +>() + +// Fixed timestamp so artifacts stay deterministic. No eval simulates a +// concurrent writer, so every save is accepted and the conflict branch is +// never taken — the syncer just records this as its `last_sync` baseline. +const BENCHMARK_DRAFT_TIMESTAMP = '1970-01-01T00:00:00.000Z' + +function benchmarkDraftKey(workspace: string, kind: string, path: string): string { + return `${workspace}::${kind}::${path}` +} + +export function clearBenchmarkDrafts(workspace: string): void { + for (const [key, entry] of benchmarkDrafts.entries()) { + if (entry.workspace === workspace) { + benchmarkDrafts.delete(key) + } + } +} + +/** + * Seed a draft straight into the store — used by the eval's live-editor draft + * fixtures, which model "the user already has this draft open/saved". Writing it + * here (instead of through `UserDraft.save`) keeps it a backend draft row with no + * shadowing in-tab cell, so a model edit that persists to the backend is what the + * output read-back captures — not the stale seed. + */ +export function seedBenchmarkDraft( + workspace: string, + kind: UserDraftItemKind, + path: string, + value: unknown +): void { + benchmarkDrafts.set(benchmarkDraftKey(workspace, kind, path), { + workspace, + kind, + path, + value + }) +} + +/** Mirror `DraftService.updateDraft`: a `null`/omitted value deletes the row. */ +export function updateBenchmarkDraft(input: { + workspace: string + kind: UserDraftItemKind + path: string + requestBody?: { value?: unknown } +}): UpdateDraftResponse { + const key = benchmarkDraftKey(input.workspace, input.kind, input.path) + const value = input.requestBody?.value + if (value == null) { + benchmarkDrafts.delete(key) + } else { + benchmarkDrafts.set(key, { + workspace: input.workspace, + kind: input.kind, + path: input.path, + value + }) + } + return { status: 'saved', current_timestamp: BENCHMARK_DRAFT_TIMESTAMP } +} + +/** Mirror `DraftService.getDraftForUser`: 404-shaped throw when absent so the + * adapter's narrowed catch treats it as "no draft" instead of re-throwing. */ +export function getBenchmarkDraftForUser(input: { + workspace: string + kind: UserDraftItemKind + path: string +}): GetDraftForUserResponse { + const entry = benchmarkDrafts.get(benchmarkDraftKey(input.workspace, input.kind, input.path)) + if (!entry) { + throw Object.assign(new Error(`no draft for "${input.path}"`), { status: 404 }) + } + return { value: entry.value, created_at: BENCHMARK_DRAFT_TIMESTAMP } +} + +/** Mirror `DraftService.listDrafts`: metadata rows (no value) for a workspace. */ +export function listBenchmarkDrafts(workspace: string): ListDraftsResponse { + return [...benchmarkDrafts.values()] + .filter((entry) => entry.workspace === workspace) + .map((entry) => ({ + kind: entry.kind, + path: entry.path, + summary: (entry.value as { summary?: string } | null)?.summary, + draft_only: true, + legacy_draft: false, + created_at: BENCHMARK_DRAFT_TIMESTAMP + })) +} + // ============= Datatables (best-effort in-memory SQL) ============= /** @@ -492,3 +647,35 @@ function buildBenchmarkFlow(flow: BenchmarkWorkspaceFlow): Flow { extra_perms: {} } as Flow } + +function buildBenchmarkListableApp(app: BenchmarkWorkspaceApp): ListableApp { + return { + id: 0, + workspace_id: 'benchmark', + path: app.path, + summary: app.summary, + version: 1, + extra_perms: {}, + edited_at: BENCHMARK_TIMESTAMP, + execution_mode: 'viewer', + raw_app: true + } +} + +function buildBenchmarkApp(app: BenchmarkWorkspaceApp): AppWithLastVersion { + return { + id: 0, + workspace_id: 'benchmark', + path: app.path, + summary: app.summary, + versions: [1], + created_by: 'benchmark', + created_at: BENCHMARK_TIMESTAMP, + value: app.value, + policy: (app.value.policy ?? {}) as AppWithLastVersion['policy'], + execution_mode: 'viewer', + extra_perms: {}, + custom_path: app.value.custom_path as string | undefined, + raw_app: true + } +} diff --git a/ai_evals/adapters/frontend/mockBackendDrafts.test.ts b/ai_evals/adapters/frontend/mockBackendDrafts.test.ts new file mode 100644 index 0000000000..a720de5e43 --- /dev/null +++ b/ai_evals/adapters/frontend/mockBackendDrafts.test.ts @@ -0,0 +1,94 @@ +import { afterEach, beforeEach, describe, expect, it } from 'bun:test' +import { + clearBenchmarkDrafts, + getBenchmarkDraftForUser, + listBenchmarkDrafts, + resetBenchmarkMockBackend, + seedBenchmarkDraft, + updateBenchmarkDraft +} from './mockBackend' + +const WORKSPACE = 'benchmark-drafts-ws' + +// Drives the in-memory stand-in for the per-user draft backend (`DraftService`) +// that the global AI-chat eval round-trips its drafts through. Mirrors the +// production-unit-test mock in +// `frontend/src/lib/components/copilot/chat/global/core.test.ts`. +describe('mockBackend drafts', () => { + beforeEach(() => resetBenchmarkMockBackend()) + afterEach(() => resetBenchmarkMockBackend()) + + it('round-trips a saved draft through update / get / list', () => { + const value = { summary: 'Greet a user', content: 'export async function main() {}' } + const res = updateBenchmarkDraft({ + workspace: WORKSPACE, + kind: 'script', + path: 'f/evals/greet', + requestBody: { value } + }) + expect(res.status).toBe('saved') + + expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/greet' }).value).toEqual( + value + ) + + const rows = listBenchmarkDrafts(WORKSPACE) + expect(rows).toHaveLength(1) + expect(rows[0]).toMatchObject({ kind: 'script', path: 'f/evals/greet', summary: 'Greet a user', draft_only: true }) + }) + + it('treats a null value as a delete', () => { + updateBenchmarkDraft({ + workspace: WORKSPACE, + kind: 'variable', + path: 'f/evals/token', + requestBody: { value: { summary: 'token' } } + }) + updateBenchmarkDraft({ + workspace: WORKSPACE, + kind: 'variable', + path: 'f/evals/token', + requestBody: { value: null } + }) + + expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0) + expect(() => getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'variable', path: 'f/evals/token' })).toThrow() + }) + + it('throws a 404-shaped error when no draft exists', () => { + try { + getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/missing' }) + throw new Error('expected a throw') + } catch (e) { + expect((e as { status?: number }).status).toBe(404) + } + }) + + it('seeds a draft as a backend row that a later edit overwrites', () => { + seedBenchmarkDraft(WORKSPACE, 'script', 'f/evals/current', { content: 'seed' }) + expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({ + content: 'seed' + }) + + // A model edit persists the same path and must win over the seed. + updateBenchmarkDraft({ + workspace: WORKSPACE, + kind: 'script', + path: 'f/evals/current', + requestBody: { value: { content: 'edited' } } + }) + expect(getBenchmarkDraftForUser({ workspace: WORKSPACE, kind: 'script', path: 'f/evals/current' }).value).toEqual({ + content: 'edited' + }) + }) + + it('clears only the targeted workspace', () => { + seedBenchmarkDraft(WORKSPACE, 'script', 'f/a', { content: 'a' }) + seedBenchmarkDraft('other-ws', 'script', 'f/b', { content: 'b' }) + + clearBenchmarkDrafts(WORKSPACE) + + expect(listBenchmarkDrafts(WORKSPACE)).toHaveLength(0) + expect(listBenchmarkDrafts('other-ws')).toHaveLength(1) + }) +}) diff --git a/ai_evals/adapters/frontend/runtime.ts b/ai_evals/adapters/frontend/runtime.ts index 347e15191c..9a54e86084 100644 --- a/ai_evals/adapters/frontend/runtime.ts +++ b/ai_evals/adapters/frontend/runtime.ts @@ -24,6 +24,8 @@ export async function runFrontendBenchmarkAdapter(input: { runs: number; model?: string; verbose?: boolean; + skipJudge?: boolean; + executionOnly?: boolean; backendValidation?: string; }): Promise { const tempDir = await mkdtemp( @@ -40,6 +42,9 @@ export async function runFrontendBenchmarkAdapter(input: { WMILL_FRONTEND_AI_EVAL_MODEL: input.model ?? "", WMILL_FRONTEND_AI_EVAL_PROGRESS: "1", WMILL_FRONTEND_AI_EVAL_VERBOSE: input.verbose ? "1" : "0", + WMILL_FRONTEND_AI_EVAL_SKIP_JUDGE: + input.skipJudge || input.executionOnly ? "1" : "0", + WMILL_FRONTEND_AI_EVAL_EXECUTION_ONLY: input.executionOnly ? "1" : "0", WMILL_FRONTEND_AI_EVAL_BACKEND_VALIDATION: input.backendValidation ?? "", }; diff --git a/ai_evals/adapters/frontend/vitestAdapter.test.ts b/ai_evals/adapters/frontend/vitestAdapter.test.ts index ebbbac8d11..dcaa1d2ca4 100644 --- a/ai_evals/adapters/frontend/vitestAdapter.test.ts +++ b/ai_evals/adapters/frontend/vitestAdapter.test.ts @@ -33,24 +33,30 @@ vi.mock('$lib/components/vscode', () => ({})) vi.mock('$lib/gen', async () => { const actual = await vi.importActual('$lib/gen') const { + getBenchmarkAppByPath, getBenchmarkCompletedJob, getBenchmarkCompletedJobResultMaybe, getBenchmarkDatatableSchema, + getBenchmarkDraftForUser, getBenchmarkFlowByPath, getBenchmarkJobLogs, getBenchmarkScriptByHash, getBenchmarkScriptByPath, hasBenchmarkWorkspace, + listBenchmarkApps, listBenchmarkDatatables, + listBenchmarkDrafts, listBenchmarkFlows, listBenchmarkJobs, listBenchmarkScripts, + createBenchmarkFolder, createBenchmarkHttpTrigger, createBenchmarkSchedule, previewBenchmarkSchedule, runBenchmarkDatatableSql, runBenchmarkFlowByPath, - runBenchmarkScriptPreview + runBenchmarkScriptPreview, + updateBenchmarkDraft } = await import('./mockBackend') function wrapService(target: T, overrides: Record): T { @@ -66,6 +72,31 @@ vi.mock('$lib/gen', async () => { return { ...actual, + DraftService: wrapService(actual.DraftService, { + updateDraft: async (data: { + workspace: string + kind: any + path: string + requestBody?: { value?: unknown } + }) => + hasBenchmarkWorkspace(data.workspace) + ? updateBenchmarkDraft(data) + : actual.DraftService.updateDraft(data), + getDraftForUser: async (data: { workspace: string; kind: any; path: string }) => + hasBenchmarkWorkspace(data.workspace) + ? getBenchmarkDraftForUser(data) + : actual.DraftService.getDraftForUser(data), + listDrafts: async (data: { workspace: string }) => + hasBenchmarkWorkspace(data.workspace) + ? listBenchmarkDrafts(data.workspace) + : actual.DraftService.listDrafts(data) + }), + FolderService: wrapService(actual.FolderService, { + createFolder: async (data: { workspace: string; requestBody: { name: string } }) => + hasBenchmarkWorkspace(data.workspace) + ? createBenchmarkFolder(data.workspace, data.requestBody.name) + : actual.FolderService.createFolder(data) + }), ScriptService: wrapService(actual.ScriptService, { listScripts: async (data: { workspace: string }) => hasBenchmarkWorkspace(data.workspace) @@ -277,12 +308,20 @@ vi.mock('$lib/gen', async () => { }), AppService: wrapService(actual.AppService, { existsApp: async (data: { workspace: string; path: string }) => - hasBenchmarkWorkspace(data.workspace) ? false : actual.AppService.existsApp(data), + hasBenchmarkWorkspace(data.workspace) + ? Boolean(getBenchmarkAppByPath(data.workspace, data.path)) + : actual.AppService.existsApp(data), listApps: async (data: { workspace: string }) => - hasBenchmarkWorkspace(data.workspace) ? [] : actual.AppService.listApps(data), + hasBenchmarkWorkspace(data.workspace) + ? (listBenchmarkApps(data.workspace) ?? []) + : actual.AppService.listApps(data), getAppByPath: async (data: { workspace: string; path: string }) => { if (hasBenchmarkWorkspace(data.workspace)) { - throw new Error(`App "${data.path}" not found in benchmark workspace`) + const app = getBenchmarkAppByPath(data.workspace, data.path) + if (!app) { + throw new Error(`App "${data.path}" not found in benchmark workspace`) + } + return app } return actual.AppService.getAppByPath(data) } @@ -434,5 +473,6 @@ benchmarkIt( resetBenchmarkMockBackend() } }, - 600_000 + // Full-suite runs (30+ cases at concurrency 2-3) routinely exceed 10 minutes. + 7_200_000 ) diff --git a/ai_evals/cases/global.yaml b/ai_evals/cases/global.yaml index 28839b0594..31b097867b 100644 --- a/ai_evals/cases/global.yaml +++ b/ai_evals/cases/global.yaml @@ -3,8 +3,9 @@ Create a draft Bun script at `f/evals/global/greet_user`. It should take a string `name` input and return `Hello, ${name}!`. Leave it as an AI draft only; do not deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json runtime: - maxTurns: 8 + maxTurns: 10 validate: draftCountExactly: 1 requiredDrafts: @@ -870,3 +871,380 @@ judgeChecklist: - fetches the logs for the requested job id - explains the failure from the returned logs (connection refused to the upstream API) + +# --- Documentation search (search_docs) --- +# Pure product-knowledge questions: the assistant should consult the docs via +# search_docs and answer conversationally, not draft or mutate anything. No +# draft is produced, so the global judge is skipped and we validate tool use. + +- id: global-docs-ai-agent-step + prompt: |- + Does Windmill support a flow step where an LLM decides which of my scripts to call based on the input? + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - search_docs + forbiddenToolsUsed: + - write_script + - write_flow + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + +- id: global-docs-retry-step + prompt: |- + How does automatic retry work for a flow step that calls a flaky API? + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - search_docs + forbiddenToolsUsed: + - write_script + - write_flow + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + +- id: global-docs-key-value-store + prompt: |- + Can I use a Redis-style key-value store from my Windmill scripts, and how? + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - search_docs + forbiddenToolsUsed: + - write_script + - write_flow + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + +- id: global-docs-cron-schedule-format + prompt: |- + How do Windmill's cron schedules work, and what format does the schedule expression use? + runtime: + maxTurns: 6 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - search_docs + forbiddenToolsUsed: + - write_script + - write_flow + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + +# --- Raw app on a large project (context-usage benchmark) --- +# These cases run against the deliberately large `analytics_dashboard` raw-app +# fixture (~20 frontend files incl. a 5k-line data module, plus backend runnables). +# They exist to measure how much context the global chat consumes when working in a +# big raw app: test29 is a read-heavy debugging hunt, test30 is a small edit baseline. +# tokenUsage is recorded per run, so the same cases re-run after a read-tool change +# (the read_app_file cap + offset/limit paging) quantify the optimization. skipJudge: +# the judge only sees the drafts artifact and cannot run the app, so we validate +# deterministically. + +- id: global-test29-raw-app-debug-large + prompt: |- + The analytics dashboard app at `f/evals/global/analytics_dashboard` has a bug: + the Revenue Summary tile shows a total that is lower than the per-order line + totals and the per-region breakdown. Track down what is computing revenue + incorrectly and fix it. Keep the change as an AI draft only; do not deploy or + save it. + initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard + runtime: + maxTurns: 20 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: app + path: f/evals/global/analytics_dashboard + valueIncludes: + - "return order.unitPrice * order.quantity" + toolExpect: + requiredToolsAnyOf: + # Inspecting the app's files is satisfied by either reading them directly + # or grepping for the revenue calculation. + - [read_app_file, search_app] + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - inspects the dashboard app's files to locate the revenue calculation + - fixes the per-order revenue so it multiplies unit price by quantity + - leaves the result as an AI draft and does not deploy or save it + +- id: global-test30-raw-app-small-edit-large + prompt: |- + In the dashboard app at `f/evals/global/analytics_dashboard`, change the main + page heading from "Operations Console" to "Revenue Overview". Leave everything + else unchanged. Keep it as an AI draft only; do not deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: app + path: f/evals/global/analytics_dashboard + valueIncludes: + - "Revenue Overview" + toolExpect: + requiredToolsAnyOf: + # Inspecting the app's files is satisfied by reading them directly or + # grepping for the target with search_app. + - [read_app_file, search_app] + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - renames the main page heading to Revenue Overview + - does not change other dashboard behavior + - leaves the result as an AI draft only + +- id: global-test31-raw-app-debug-inspect-data + prompt: |- + The raw app dashboard at `f/evals/global/analytics_dashboard` is reporting + revenue totals that look too low. Inspect the app's files — both the sample + order data module and the revenue calculation — to work out whether the bug is + in the data or in the calculation, then fix the actual cause. Keep the change as + an AI draft only; do not deploy or save it. + initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard + runtime: + maxTurns: 22 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: app + path: f/evals/global/analytics_dashboard + valueIncludes: + - "return order.unitPrice * order.quantity" + toolExpect: + requiredToolsAnyOf: + # Inspecting the app's files is satisfied by reading them directly or + # grepping for the target with search_app. + - [read_app_file, search_app] + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - inspects both the sample order data module and the revenue aggregation logic + - identifies the per-order revenue bug and fixes it to multiply unit price by quantity + - leaves the result as an AI draft only + +- id: global-test32-raw-app-cross-file-consistency + prompt: |- + The raw app dashboard at `f/evals/global/analytics_dashboard` shows revenue + totals that disagree between the Revenue Summary tile, the orders table, and the + regional breakdown. Investigate how each of those computes revenue, work out + which calculation is wrong, and fix it. Keep the change as an AI draft only; do + not deploy or save it. + # Cross-file investigation: forces the model through several overlapping files + # (the summary's aggregation helper, the orders table, the regional breakdown) — + # a realistic multi-file read load that exercises the read_app_file cap. + initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard + runtime: + maxTurns: 24 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: app + path: f/evals/global/analytics_dashboard + valueIncludes: + - "return order.unitPrice * order.quantity" + toolExpect: + requiredToolsAnyOf: + # Inspecting the app's files is satisfied by reading them directly or + # grepping for the target with search_app. + - [read_app_file, search_app] + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - inspects the revenue calculation behind the summary tile, the orders table, and the regional breakdown + - identifies that the per-order revenue helper omits quantity and fixes it to multiply unit price by quantity + - leaves the result as an AI draft only + +- id: global-test33-raw-app-rename-across-files + prompt: |- + In the dashboard app at `f/evals/global/analytics_dashboard`, rename the + `formatCurrency` helper to `formatMoney` everywhere it is defined, imported, and + called. Leave the separate `formatCurrencyPrecise` helper exactly as it is. Keep + the change as an AI draft only; do not deploy or save it. + # Find-all-usages rename: formatCurrency is defined once and called in 6 places + # spread across 4 component files (and imported in 4). Locating every usage is the + # exact task search_app is meant to make cheap — one grep returns all file:line + # rows instead of reading each component whole. valueExcludes "formatCurrency(" + # asserts the definition and all call sites were renamed while tolerating the + # preserved formatCurrencyPrecise (which is never followed by "("). + initial: ai_evals/fixtures/frontend/global/initial/analytics_dashboard + runtime: + maxTurns: 22 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: app + path: f/evals/global/analytics_dashboard + valueIncludes: + - "export function formatMoney" + - "formatMoney(" + valueExcludes: + - "formatCurrency(" + toolExpect: + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - renames the formatCurrency definition, imports, and all call sites to formatMoney + - leaves the unrelated formatCurrencyPrecise helper unchanged + - leaves the result as an AI draft only + +# --- Path selection (u/ vs f/) --- +# These cases assert how the assistant picks a workspace path when the user gives +# none: a bare name defaults to the personal scope `u//`, an existing folder +# whose purpose matches is used, a non-admin targets a writable folder and never a +# read-only one, and shared intent with no matching folder asks rather than invents. +# Each depends on the seeded `user` fixture (username / is_admin / folders / +# folders_read) so the prompt's folder guidance and `u/{username}` are well-formed. + +- id: global-path1-bare-name-defaults-to-personal + prompt: |- + Stage a quick draft helper that takes a string and returns it trimmed of + leading and trailing whitespace. Just keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + pathStartsWith: u/admin/ + toolExpect: + requiredToolsUsed: + - write_script + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - stages a single script draft for a trim helper + - defaults the path to the current user's personal scope (u/admin/...) since no path or folder was given + - does not invent an f/ path + - leaves the result as a draft only + +- id: global-path2-match-existing-folder + prompt: |- + Draft a flow for the marketing team's weekly campaign report. + It should take a week number and return a short summary string. + Keep it as a draft only. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: flow + pathStartsWith: f/marketing/ + toolExpect: + requiredToolsUsed: + - write_flow + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - drafts a flow for the marketing campaign report + - places it in the existing marketing folder (f/marketing/...) rather than the personal scope or an invented folder + - leaves the result as a draft only + +- id: global-path3-shared-intent-unknown-folder-asks + prompt: |- + Put together a draft onboarding checklist flow for the People Ops team to use + when a new hire joins. Keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_folders.json + runtime: + maxTurns: 8 + validate: + draftCountExactly: 0 + toolExpect: + requiredToolsUsed: + - askUserQuestion + forbiddenToolsUsed: + - write_flow + - write_script + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - recognizes the request implies shared/team work but names no existing folder (none of marketing/data_engineering/shared_utils fit People Ops) + - asks which folder to use instead of guessing or inventing one + - does not create a draft until the folder is known + +- id: global-path4-nonadmin-avoids-readonly-folder + prompt: |- + Draft a small flow that returns today's date as an ISO string, and stage it in + one of our shared team folders. Keep it as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: flow + pathStartsWith: f/team_a/ + forbiddenDrafts: + - type: flow + pathStartsWith: f/team_b/ + toolExpect: + requiredToolsUsed: + - write_flow + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - drafts a flow that returns the current date as an ISO string + - places it in team_a (writable by this non-admin user) and not team_b (read-only) + - leaves the result as a draft only + +- id: global-path5-create-folder-then-draft + prompt: |- + Create a new shared folder called "analytics" for our data work, then draft a + script in it that returns the current timestamp as an ISO string. Keep the + script as a draft. + initial: ai_evals/fixtures/frontend/global/initial/user_admin_empty.json + runtime: + maxTurns: 10 + validate: + draftCountExactly: 1 + requiredDrafts: + - type: script + pathStartsWith: f/analytics/ + toolExpect: + requiredToolsUsed: + - create_folder + - write_script + forbiddenToolsUsed: + - deploy_workspace_item + - delete_workspace_item + skipJudge: true + judgeChecklist: + - creates a new shared folder named "analytics" via create_folder + - drafts a script placed in that folder (f/analytics/...) returning an ISO timestamp + - leaves the script as a draft only diff --git a/ai_evals/cli/index.ts b/ai_evals/cli/index.ts index f92d6d7027..504b8a8d13 100644 --- a/ai_evals/cli/index.ts +++ b/ai_evals/cli/index.ts @@ -25,7 +25,9 @@ import { import { runSuite } from "../core/runSuite"; import { EVAL_MODES, type EvalMode } from "../core/types"; import { DEFAULT_JUDGE_MODEL } from "../core/judge"; -import { createCliModeRunner } from "../modes/cli"; +// createCliModeRunner is imported lazily in runCliBenchmark so the non-cli modes +// (global/flow/script/app) don't pull in the wmill CLI toolchain and its JSR deps +// (e.g. @cliffy/*) just to load this entrypoint. import { runFrontendBenchmarkAdapter } from "../adapters/frontend/runtime"; import { resolveWindmillBackendSettings } from "../core/windmillBackendSettings"; import { assertWindmillBackendReachable } from "../adapters/frontend/windmillBackend"; @@ -97,6 +99,11 @@ async function main() { "comma-separated model aliases to run sequentially", ) .option("--verbose", "stream assistant output during frontend runs") + .option("--skip-judge", "skip LLM judge scoring for this run") + .option( + "--execution-only", + "only require the model/proxy/frontend loop to complete", + ) .option( "--record", "append a compact summary line to ai_evals/history/.jsonl", @@ -115,6 +122,8 @@ async function main() { model?: string; models?: string; verbose?: boolean; + skipJudge?: boolean; + executionOnly?: boolean; record?: boolean; backendValidation?: string; }, @@ -127,6 +136,8 @@ async function main() { model: options.model, models: options.models, verbose: options.verbose ?? false, + skipJudge: options.skipJudge ?? false, + executionOnly: options.executionOnly ?? false, record: options.record ?? false, backendValidation: options.backendValidation, }); @@ -175,6 +186,8 @@ async function handleRun(input: { model?: string; models?: string; verbose: boolean; + skipJudge: boolean; + executionOnly: boolean; record: boolean; backendValidation?: string; }) { @@ -230,6 +243,8 @@ async function handleRun(input: { input.runs, getCliEvalModel(model), runModel, + input.skipJudge, + input.executionOnly, ) : await runFrontendBenchmarkAdapter({ mode: input.mode, @@ -237,6 +252,8 @@ async function handleRun(input: { runs: input.runs, model: model.id, verbose: input.verbose, + skipJudge: input.skipJudge, + executionOnly: input.executionOnly, backendValidation, }); @@ -278,20 +295,25 @@ async function runCliBenchmark( runs: number, model: ReturnType, runModel: string, + skipJudge: boolean, + executionOnly: boolean, ) { + const { createCliModeRunner } = await import("../modes/cli"); + const judgeModel = skipJudge || executionOnly ? null : DEFAULT_JUDGE_MODEL; const caseResults = await runSuite({ modeRunner: createCliModeRunner(model), cases, runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, + executionOnly, }); return buildRunResult({ mode: "cli", runs, runModel, - judgeModel: DEFAULT_JUDGE_MODEL, + judgeModel, caseResults, }); } diff --git a/ai_evals/core/cases.test.ts b/ai_evals/core/cases.test.ts index 05e2f1527b..5d6e3245db 100644 --- a/ai_evals/core/cases.test.ts +++ b/ai_evals/core/cases.test.ts @@ -212,6 +212,9 @@ describe("loadCases", () => { }, ], }); + expect(caseEntry?.initialPath).toContain( + "ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json" + ); expect(caseEntry?.toolExpect).toMatchObject({ requiredToolsUsed: ["write_script"], forbiddenToolsUsed: ["deploy_workspace_item", "delete_workspace_item"], @@ -246,6 +249,21 @@ describe("loadCases", () => { }); }); + it("loads global docs-search cases as tool-use checks", async () => { + const globalCases = await loadCases("global"); + const docsCases = globalCases.filter((entry) => + entry.id.startsWith("global-docs-"), + ); + expect(docsCases.length).toBeGreaterThanOrEqual(3); + + // Each docs case verifies the assistant reaches for search_docs and does not + // draft anything; with no draft, the global judge is skipped. + for (const entry of docsCases) { + expect(entry.skipJudge).toBe(true); + expect(entry.toolExpect?.requiredToolsUsed).toContain("search_docs"); + } + }); + it("loads tool expectations for workspace mutation cases", async () => { const scriptCases = await loadCases("script"); const caseEntry = scriptCases.find( diff --git a/ai_evals/core/results.test.ts b/ai_evals/core/results.test.ts index 2d6077c5bd..a5406f987f 100644 --- a/ai_evals/core/results.test.ts +++ b/ai_evals/core/results.test.ts @@ -92,6 +92,71 @@ describe("benchmark results", () => { expect(summary).toContain("Average duration (all attempts): 550ms"); }); + it("aggregates final context size over passed attempts only", () => { + const result = buildRunResult({ + mode: "global", + runs: 1, + runModel: "model-under-test", + judgeModel: "judge-model", + caseResults: [ + caseResult([ + { + attempt: 1, + passed: true, + durationMs: 1000, + assistantMessageCount: 1, + toolCallCount: 1, + toolsUsed: ["edit_script"], + skillsInvoked: [], + checks: [{ name: "edited", passed: true }], + judgeScore: 100, + judgeSummary: "ok", + error: null, + tokenUsage: { prompt: 12000, completion: 200, total: 12200 }, + finalContextTokens: 5000, + }, + { + attempt: 2, + passed: true, + durationMs: 1100, + assistantMessageCount: 1, + toolCallCount: 1, + toolsUsed: ["edit_script"], + skillsInvoked: [], + checks: [{ name: "edited", passed: true }], + judgeScore: 100, + judgeSummary: "ok", + error: null, + tokenUsage: { prompt: 18000, completion: 300, total: 18300 }, + finalContextTokens: 7000, + }, + { + attempt: 3, + passed: false, + durationMs: 100, + assistantMessageCount: 1, + toolCallCount: 0, + toolsUsed: [], + skillsInvoked: [], + checks: [{ name: "edited", passed: false }], + judgeScore: 10, + judgeSummary: "missed", + error: "failed", + tokenUsage: { prompt: 20000, completion: 100, total: 20100 }, + finalContextTokens: 9000, + }, + ]), + ], + }); + + // Final context size stays below cumulative prompt and ignores the failed attempt. + expect(result.averageFinalContextTokensPassed).toBe(6000); + expect(result.maxFinalContextTokensPassed).toBe(7000); + expect(formatRunSummary(result)).toContain( + "Final context size (passed): 6000 tokens (max 7000)", + ); + }); + it("reports passed averages as unavailable when no attempt passes", () => { const result = buildRunResult({ mode: "global", diff --git a/ai_evals/core/results.ts b/ai_evals/core/results.ts index 0b84497165..a9807d7d16 100644 --- a/ai_evals/core/results.ts +++ b/ai_evals/core/results.ts @@ -16,6 +16,9 @@ type AttemptAggregate = { durationTotal: number; tokenUsageAttemptCount: number; tokenUsageTotal: BenchmarkTokenUsage | null; + finalContextAttemptCount: number; + finalContextTotal: number; + finalContextMax: number | null; }; export async function writeRunResult( @@ -117,6 +120,8 @@ export function buildRunResult(input: { passedAttemptAggregate, passedAttempts, ), + averageFinalContextTokensPassed: averageFinalContext(passedAttemptAggregate), + maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax, cases: input.caseResults, }; } @@ -133,6 +138,11 @@ export function formatRunSummary(result: BenchmarkRunResult): string { `Average tokens (passed): ${formatTokenUsage(result.averageTokenUsagePerPassedAttempt)}`, ); } + if (result.averageFinalContextTokensPassed != null) { + lines.push( + `Final context size (passed): ${Math.round(result.averageFinalContextTokensPassed)} tokens (max ${Math.round(result.maxFinalContextTokensPassed ?? 0)})`, + ); + } if (result.passedAttempts < result.attemptCount) { lines.push( `Average duration (all attempts): ${Math.round(result.averageDurationMs)}ms`, @@ -181,10 +191,21 @@ function aggregateAttempts(attempts: BenchmarkAttemptResult[]): AttemptAggregate durationTotal: 0, tokenUsageAttemptCount: 0, tokenUsageTotal: null, + finalContextAttemptCount: 0, + finalContextTotal: 0, + finalContextMax: null, }; for (const attempt of attempts) { aggregate.durationTotal += attempt.durationMs; + if (typeof attempt.finalContextTokens === "number") { + aggregate.finalContextAttemptCount += 1; + aggregate.finalContextTotal += attempt.finalContextTokens; + aggregate.finalContextMax = Math.max( + aggregate.finalContextMax ?? 0, + attempt.finalContextTokens, + ); + } if (!attempt.tokenUsage) { continue; } @@ -204,6 +225,12 @@ function averageDuration(aggregate: AttemptAggregate): number | null { : aggregate.durationTotal / aggregate.attemptCount; } +function averageFinalContext(aggregate: AttemptAggregate): number | null { + return aggregate.finalContextAttemptCount === 0 + ? null + : aggregate.finalContextTotal / aggregate.finalContextAttemptCount; +} + function averageTokenUsage( aggregate: AttemptAggregate, denominator: number, @@ -318,6 +345,9 @@ function toHistoryRecord(result: BenchmarkRunResult) { averageTokenUsagePerAttempt: result.averageTokenUsagePerAttempt ?? null, averageTokenUsagePerPassedAttempt: result.averageTokenUsagePerPassedAttempt ?? null, + averageFinalContextTokensPassed: + result.averageFinalContextTokensPassed ?? null, + maxFinalContextTokensPassed: result.maxFinalContextTokensPassed ?? null, failedCaseIds: Array.from( new Set( result.cases @@ -361,6 +391,10 @@ function toHistoryRecord(result: BenchmarkRunResult) { passedAttemptAggregate, passedAttempts, ), + averageFinalContextTokensPassed: averageFinalContext( + passedAttemptAggregate, + ), + maxFinalContextTokensPassed: passedAttemptAggregate.finalContextMax, }; }), }; diff --git a/ai_evals/core/runSuite.test.ts b/ai_evals/core/runSuite.test.ts new file mode 100644 index 0000000000..26f300a5aa --- /dev/null +++ b/ai_evals/core/runSuite.test.ts @@ -0,0 +1,102 @@ +import { describe, expect, it } from "bun:test"; +import { runSuite } from "./runSuite"; +import type { ModeRunner } from "./types"; + +const modeRunner: ModeRunner = { + mode: "global", + concurrency: 1, + loadInitial: async () => undefined, + loadExpected: async () => undefined, + run: async () => ({ + success: true, + actual: { ok: true }, + assistantMessageCount: 1, + toolCallCount: 0, + toolsUsed: [], + skillsInvoked: [], + tokenUsage: null, + }), + validate: () => [], +}; + +describe("runSuite", () => { + it("skips judge checks when the run disables judge scoring", async () => { + const [caseResult] = await runSuite({ + modeRunner, + cases: [ + { + id: "case-1", + prompt: "Create a draft script", + judgeChecklist: ["the output satisfies the prompt"], + }, + ], + runs: 1, + runModel: "model-under-test", + judgeModel: null, + }); + + const [attempt] = caseResult.attempts; + expect(attempt.passed).toBe(true); + expect(attempt.judgeScore).toBeNull(); + expect(attempt.judgeSummary).toBeNull(); + expect(attempt.checks.map((check) => check.name)).toEqual([ + "run succeeded", + ]); + }); + + it("only requires run success when execution-only is enabled", async () => { + let loadExpectedCalls = 0; + let validateCalls = 0; + let backendValidateCalls = 0; + + const executionOnlyRunner: ModeRunner< + undefined, + undefined, + { ok: boolean } + > = { + ...modeRunner, + loadExpected: async () => { + loadExpectedCalls++; + return undefined; + }, + validate: () => { + validateCalls++; + return [{ name: "validator failed", passed: false }]; + }, + backendValidate: async () => { + backendValidateCalls++; + return { + checks: [{ name: "backend validation failed", passed: false }], + }; + }, + }; + + const [caseResult] = await runSuite({ + modeRunner: executionOnlyRunner, + cases: [ + { + id: "case-1", + prompt: "Create a draft script", + expectedPath: "fixtures/expected.json", + toolExpect: { requiredToolsUsed: ["write_script"] }, + judgeChecklist: ["the output satisfies the prompt"], + }, + ], + runs: 1, + runModel: "model-under-test", + judgeModel: "judge-model", + executionOnly: true, + }); + + const [attempt] = caseResult.attempts; + expect(attempt.passed).toBe(true); + expect(attempt.judgeScore).toBeNull(); + expect(attempt.judgeSummary).toBeNull(); + expect(attempt.checks.map((check) => check.name)).toEqual([ + "run succeeded", + ]); + expect(loadExpectedCalls).toBe(0); + expect(validateCalls).toBe(0); + expect(backendValidateCalls).toBe(0); + }); +}); diff --git a/ai_evals/core/runSuite.ts b/ai_evals/core/runSuite.ts index ed82d841cb..4a8c9dab7b 100644 --- a/ai_evals/core/runSuite.ts +++ b/ai_evals/core/runSuite.ts @@ -15,11 +15,13 @@ export async function runSuite(input: { runs: number; runModel: string | null; judgeModel?: string | null; + executionOnly?: boolean; concurrency?: number; verbose?: boolean; onProgress?: (event: FrontendBenchmarkProgressEvent) => void; }): Promise { - const judgeModel = input.judgeModel ?? DEFAULT_JUDGE_MODEL; + const judgeModel = + input.judgeModel === undefined ? DEFAULT_JUDGE_MODEL : input.judgeModel; const concurrency = Math.max(1, input.concurrency ?? input.modeRunner.concurrency); const results = new Array(input.cases.length); let cursor = 0; @@ -52,6 +54,7 @@ export async function runSuite(input: { runs: input.runs, judgeModel, judgeThreshold: input.modeRunner.judgeThreshold ?? 80, + executionOnly: input.executionOnly ?? false, modeRunner: input.modeRunner, totalCases: input.cases.length, verbose: input.verbose ?? false, @@ -72,8 +75,9 @@ async function runCaseAttempts(input: { caseIndex: number; evalCase: EvalCase; runs: number; - judgeModel: string; + judgeModel: string | null; judgeThreshold: number; + executionOnly: boolean; modeRunner: ModeRunner; totalCases: number; verbose: boolean; @@ -99,7 +103,9 @@ async function runCaseAttempts(input: { try { const initial = await input.modeRunner.loadInitial(input.evalCase.initialPath); - const expected = await input.modeRunner.loadExpected(input.evalCase.expectedPath); + const expected = input.executionOnly + ? undefined + : await input.modeRunner.loadExpected(input.evalCase.expectedPath); const run = await input.modeRunner.run(input.evalCase.prompt, initial, { evalCase: input.evalCase, caseId: input.evalCase.id, @@ -162,22 +168,30 @@ async function runCaseAttempts(input: { }); const checks: BenchmarkCheck[] = [ buildCheck("run succeeded", run.success, run.error), - ...input.modeRunner.validate({ - evalCase: input.evalCase, - prompt: input.evalCase.prompt, - initial, - expected, - actual: run.actual, - run, - }), - ...validateToolExpectations({ - run, - toolExpect: input.evalCase.toolExpect, - }), ]; + if (!input.executionOnly) { + checks.push( + ...input.modeRunner.validate({ + evalCase: input.evalCase, + prompt: input.evalCase.prompt, + initial, + expected, + actual: run.actual, + run, + }), + ...validateToolExpectations({ + run, + toolExpect: input.evalCase.toolExpect, + }) + ); + } const artifactFiles = input.modeRunner.buildArtifacts?.(run.actual) ?? []; - if (run.success && input.modeRunner.backendValidate) { + if ( + run.success && + !input.executionOnly && + input.modeRunner.backendValidate + ) { try { const backendValidation = await input.modeRunner.backendValidate({ evalCase: input.evalCase, @@ -218,14 +232,21 @@ async function runCaseAttempts(input: { let judgeScore: number | null = null; let judgeSummary: string | null = null; - if (run.success && !input.evalCase.skipJudge) { + if ( + run.success && + !input.executionOnly && + input.judgeModel !== null && + !input.evalCase.skipJudge + ) { const judge = await judgeOutput({ mode: input.modeRunner.mode, prompt: input.evalCase.prompt, checklist: input.evalCase.judgeChecklist, initial, expected: input.modeRunner.mode === "cli" ? undefined : expected, - actual: run.actual, + actual: input.modeRunner.prepareJudgeActual + ? input.modeRunner.prepareJudgeActual(run.actual) + : run.actual, model: input.judgeModel, }); @@ -255,6 +276,7 @@ async function runCaseAttempts(input: { judgeSummary, error: run.error ?? null, tokenUsage: run.tokenUsage ?? null, + finalContextTokens: run.finalContextTokens ?? null, artifactsPath: null, artifactFiles, }; @@ -291,6 +313,7 @@ async function runCaseAttempts(input: { judgeSummary: null, error: message, tokenUsage: null, + finalContextTokens: null, }; if (surface) { input.onProgress?.({ diff --git a/ai_evals/core/types.ts b/ai_evals/core/types.ts index 27c2fcddac..52667fa321 100644 --- a/ai_evals/core/types.ts +++ b/ai_evals/core/types.ts @@ -168,11 +168,21 @@ export interface ToolCallArgumentRule { export interface ToolValidationSpec { requiredToolsUsed?: string[]; + /** + * Each inner array is an alternatives group: the check passes when at least + * one tool in the group was used. Use when several tools satisfy the same + * intent so a model that picks any valid path passes — e.g. inspecting an + * app's files via either `read_app_file` or `search_app`. + */ + requiredToolsAnyOf?: string[][]; forbiddenToolsUsed?: string[]; toolCallArgs?: ToolCallArgumentRule[]; } -export type EvalValidationSpec = FlowValidationSpec | AppValidationSpec | GlobalValidationSpec; +export type EvalValidationSpec = + | FlowValidationSpec + | AppValidationSpec + | GlobalValidationSpec; export interface EvalCase { id: string; @@ -249,6 +259,12 @@ export interface ModeRunOutput { toolCallDetails?: ToolCallDetail[]; skillsInvoked: string[]; tokenUsage?: BenchmarkTokenUsage | null; + /** + * Total input tokens occupying the context window on the LAST model request + * of the agentic loop (input + cache-creation + cache-read). Complements the + * cumulative `tokenUsage.prompt`, which sums every iteration's input. + */ + finalContextTokens?: number | null; } export interface ModeRunContext { @@ -294,6 +310,12 @@ export interface ModeRunner { context: ModeRunContext; }): Promise; buildArtifacts?(actual: TActual): BenchmarkArtifactFile[]; + /** + * Optional transform applied to `actual` before it is handed to the LLM judge. + * Use it to strip fields the judge must stay blind to (e.g. which docs-tool + * arm produced an answer). When omitted, the judge receives `actual` as-is. + */ + prepareJudgeActual?(actual: TActual): unknown; } export interface BenchmarkAttemptResult { @@ -310,6 +332,7 @@ export interface BenchmarkAttemptResult { judgeSummary: string | null; error: string | null; tokenUsage?: BenchmarkTokenUsage | null; + finalContextTokens?: number | null; artifactsPath?: string | null; artifactFiles?: BenchmarkArtifactFile[]; } @@ -340,6 +363,8 @@ export interface BenchmarkRunResult { totalPassedTokenUsage?: BenchmarkTokenUsage | null; averageTokenUsagePerAttempt?: BenchmarkTokenUsage | null; averageTokenUsagePerPassedAttempt?: BenchmarkTokenUsage | null; + averageFinalContextTokensPassed?: number | null; + maxFinalContextTokensPassed?: number | null; artifactsPath?: string | null; cases: BenchmarkCaseResult[]; } diff --git a/ai_evals/core/validators.test.ts b/ai_evals/core/validators.test.ts index 7f0e841366..5183a20226 100644 --- a/ai_evals/core/validators.test.ts +++ b/ai_evals/core/validators.test.ts @@ -245,6 +245,49 @@ describe("validateToolExpectations", () => { 'accepted substrings: insert into, update; values: "DROP TABLE orders"', }); }); + + it("passes requiredToolsAnyOf when any alternative in the group is used", () => { + const checks = validateToolExpectations({ + run: { + success: true, + actual: {}, + assistantMessageCount: 1, + toolCallCount: 1, + toolsUsed: ["search_app", "patch_app_file"], + skillsInvoked: [], + }, + toolExpect: { + requiredToolsAnyOf: [["read_app_file", "search_app"]], + }, + }); + + expect(checks).toContainEqual({ + name: "uses one of read_app_file, search_app", + passed: true, + }); + }); + + it("fails requiredToolsAnyOf when no alternative in the group is used", () => { + const checks = validateToolExpectations({ + run: { + success: true, + actual: {}, + assistantMessageCount: 1, + toolCallCount: 1, + toolsUsed: ["patch_app_file"], + skillsInvoked: [], + }, + toolExpect: { + requiredToolsAnyOf: [["read_app_file", "search_app"]], + }, + }); + + expect(checks).toContainEqual({ + name: "uses one of read_app_file, search_app", + passed: false, + details: "tools used: patch_app_file", + }); + }); }); describe("validateGlobalState", () => { diff --git a/ai_evals/core/validators.ts b/ai_evals/core/validators.ts index e7a7641c00..23a6709f9b 100644 --- a/ai_evals/core/validators.ts +++ b/ai_evals/core/validators.ts @@ -169,6 +169,16 @@ export function validateToolExpectations(input: { ); } + for (const group of expect.requiredToolsAnyOf ?? []) { + checks.push( + check( + `uses one of ${group.join(", ")}`, + group.some((toolName) => input.run.toolsUsed.includes(toolName)), + `tools used: ${input.run.toolsUsed.join(", ") || "none"}` + ) + ); + } + for (const toolName of expect.forbiddenToolsUsed ?? []) { checks.push( check( diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts new file mode 100644 index 0000000000..b3c67cfdc7 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/main.ts @@ -0,0 +1,68 @@ +type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled' + +interface Order { + id: string + region: string + quantity: number + unitPrice: number + status: OrderStatus + placedAt: string +} + +// Server-side revenue rollup. Mirrors the client aggregation but is computed +// from the authoritative mocked order book so it can be used to cross-check +// the dashboard and to back the export. +const orders: Order[] = [ + { id: 'ORD-10001', region: 'North America', quantity: 3, unitPrice: 1195, status: 'delivered', placedAt: '2024-05-02' }, + { id: 'ORD-10002', region: 'EMEA', quantity: 5, unitPrice: 880, status: 'shipped', placedAt: '2024-05-03' }, + { id: 'ORD-10003', region: 'APAC', quantity: 2, unitPrice: 640, status: 'paid', placedAt: '2024-05-05' }, + { id: 'ORD-10004', region: 'LATAM', quantity: 7, unitPrice: 315, status: 'delivered', placedAt: '2024-05-07' }, + { id: 'ORD-10005', region: 'North America', quantity: 4, unitPrice: 150, status: 'refunded', placedAt: '2024-05-09' }, + { id: 'ORD-10006', region: 'EMEA', quantity: 6, unitPrice: 220, status: 'shipped', placedAt: '2024-05-12' }, + { id: 'ORD-10007', region: 'APAC', quantity: 1, unitPrice: 980, status: 'pending', placedAt: '2024-05-15' }, + { id: 'ORD-10008', region: 'North America', quantity: 8, unitPrice: 1100, status: 'delivered', placedAt: '2024-05-18' }, + { id: 'ORD-10009', region: 'EMEA', quantity: 2, unitPrice: 860, status: 'cancelled', placedAt: '2024-05-22' }, + { id: 'ORD-10010', region: 'LATAM', quantity: 9, unitPrice: 290, status: 'paid', placedAt: '2024-05-26' } +] + +const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered'] + +export async function main({ + from, + to, + region +}: { + from: string + to: string + region: string +}): Promise<{ + totalRevenue: number + netRevenue: number + totalOrders: number + averageOrderValue: number + unitsSold: number + refundedRevenue: number + currency: string +}> { + let scoped = orders.filter((order) => order.placedAt >= from && order.placedAt <= to) + if (region && region !== 'all') { + scoped = scoped.filter((order) => order.region === region) + } + + const booked = scoped.filter((order) => REVENUE_STATUSES.includes(order.status)) + const totalRevenue = booked.reduce((acc, order) => acc + order.unitPrice * order.quantity, 0) + const unitsSold = booked.reduce((acc, order) => acc + order.quantity, 0) + const refundedRevenue = scoped + .filter((order) => order.status === 'refunded') + .reduce((acc, order) => acc + order.unitPrice * order.quantity, 0) + + return { + totalRevenue, + netRevenue: totalRevenue - refundedRevenue, + totalOrders: booked.length, + averageOrderValue: booked.length === 0 ? 0 : Math.round(totalRevenue / booked.length), + unitsSold, + refundedRevenue, + currency: 'USD' + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json new file mode 100644 index 0000000000..ab2e5537f8 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/computeSummary/meta.json @@ -0,0 +1,4 @@ +{ + "name": "Compute Summary", + "language": "bun" +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts new file mode 100644 index 0000000000..ec10c1d951 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/main.ts @@ -0,0 +1,51 @@ +// Builds a downloadable report for the current dashboard view. Returns a data +// URL the browser can open directly so the export works without object storage. +export async function main({ + from, + to, + region, + format +}: { + from: string + to: string + region: string + format: 'csv' | 'json' +}): Promise<{ url: string; rows: number; filename: string }> { + const summary = { + from, + to, + region: region || 'all', + generatedAt: new Date().toISOString(), + rows: [ + { region: 'North America', revenue: 211_400, orders: 168 }, + { region: 'EMEA', revenue: 142_900, orders: 121 }, + { region: 'APAC', revenue: 86_500, orders: 78 }, + { region: 'LATAM', revenue: 41_500, orders: 45 } + ] + } + + const scoped = + region && region !== 'all' + ? summary.rows.filter((row) => row.region === region) + : summary.rows + + let body: string + let mime: string + if (format === 'csv') { + const header = 'region,revenue,orders' + const lines = scoped.map((row) => `${row.region},${row.revenue},${row.orders}`) + body = [header, ...lines].join('\n') + mime = 'text/csv' + } else { + body = JSON.stringify({ ...summary, rows: scoped }, null, 2) + mime = 'application/json' + } + + const encoded = Buffer.from(body, 'utf-8').toString('base64') + const filename = `revenue-report-${from}_${to}.${format}` + return { + url: `data:${mime};base64,${encoded}`, + rows: scoped.length, + filename + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json new file mode 100644 index 0000000000..8f198d716f --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/exportReport/meta.json @@ -0,0 +1,4 @@ +{ + "name": "Export Report", + "language": "bun" +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts new file mode 100644 index 0000000000..4ada29ad22 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/main.ts @@ -0,0 +1,40 @@ +interface MetricCardData { + id: string + label: string + value: number + unit: 'currency' | 'count' | 'percent' + delta: number + hint: string +} + +// Returns the headline metric cards for the selected range and region. Values +// are mocked but internally consistent (revenue / orders ≈ avg order value). +const baseByRegion: Record = { + all: { revenue: 482_300, orders: 412, units: 1840, refunds: 11_900 }, + 'North America': { revenue: 211_400, orders: 168, units: 770, refunds: 4_200 }, + EMEA: { revenue: 142_900, orders: 121, units: 560, refunds: 3_500 }, + APAC: { revenue: 86_500, orders: 78, units: 340, refunds: 2_600 }, + LATAM: { revenue: 41_500, orders: 45, units: 170, refunds: 1_600 } +} + +export async function main({ + from, + to, + region +}: { + from: string + to: string + region: string +}): Promise<{ cards: MetricCardData[]; generatedAt: string }> { + const base = baseByRegion[region] ?? baseByRegion.all + const aov = base.orders === 0 ? 0 : Math.round(base.revenue / base.orders) + const cards: MetricCardData[] = [ + { id: 'revenue', label: 'Total Revenue', value: base.revenue, unit: 'currency', delta: 0.082, hint: `Booked revenue ${from} – ${to}` }, + { id: 'orders', label: 'Orders', value: base.orders, unit: 'count', delta: 0.041, hint: 'Revenue-bearing orders in range' }, + { id: 'aov', label: 'Avg Order Value', value: aov, unit: 'currency', delta: -0.013, hint: 'Total revenue / order count' }, + { id: 'units', label: 'Units Sold', value: base.units, unit: 'count', delta: 0.067, hint: 'Total units in range' }, + { id: 'refunds', label: 'Refunded', value: base.refunds, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds' }, + { id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Sessions that became orders' } + ] + return { cards, generatedAt: new Date().toISOString() } +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json new file mode 100644 index 0000000000..1fbc3337c9 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadMetrics/meta.json @@ -0,0 +1,4 @@ +{ + "name": "Load Metrics", + "language": "bun" +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts new file mode 100644 index 0000000000..27a5396f2e --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/main.ts @@ -0,0 +1,54 @@ +type OrderStatus = 'paid' | 'shipped' | 'delivered' | 'pending' | 'refunded' | 'cancelled' + +interface Order { + id: string + placedAt: string + customer: string + product: string + sku: string + region: string + channel: string + rep: string + quantity: number + unitPrice: number + status: OrderStatus +} + +// Mocked order book. In a real deployment this would query the orders table; +// here it returns a representative slice so the table renders in preview. +const orders: Order[] = [ + { id: 'ORD-10001', placedAt: '2024-05-02T09:14:00Z', customer: 'Contoso Ltd', product: 'Aurora Analytics Suite', sku: 'ANL-100', region: 'North America', channel: 'direct', rep: 'Dana Wills', quantity: 3, unitPrice: 1195, status: 'delivered' }, + { id: 'ORD-10002', placedAt: '2024-05-03T11:42:00Z', customer: 'Fabrikam Inc', product: 'Borealis CRM', sku: 'CRM-210', region: 'EMEA', channel: 'partner', rep: 'Lena Fischer', quantity: 5, unitPrice: 880, status: 'shipped' }, + { id: 'ORD-10003', placedAt: '2024-05-05T15:03:00Z', customer: 'Tailspin Toys', product: 'Cascade Data Pipeline', sku: 'PIPE-330', region: 'APAC', channel: 'self-serve', rep: 'Sora Tanaka', quantity: 2, unitPrice: 640, status: 'paid' }, + { id: 'ORD-10004', placedAt: '2024-05-07T08:21:00Z', customer: 'Proseware Inc', product: 'Delta Insights', sku: 'INS-440', region: 'LATAM', channel: 'marketplace', rep: 'Diego Marin', quantity: 7, unitPrice: 315, status: 'delivered' }, + { id: 'ORD-10005', placedAt: '2024-05-09T13:58:00Z', customer: 'Litware Inc', product: 'Echo Monitoring', sku: 'MON-550', region: 'North America', channel: 'direct', rep: 'Owen Pratt', quantity: 4, unitPrice: 150, status: 'refunded' }, + { id: 'ORD-10006', placedAt: '2024-05-12T10:30:00Z', customer: 'Fourth Coffee', product: 'Helix Identity', sku: 'IDN-880', region: 'EMEA', channel: 'partner', rep: 'Aisha Khan', quantity: 6, unitPrice: 220, status: 'shipped' }, + { id: 'ORD-10007', placedAt: '2024-05-15T17:11:00Z', customer: 'Coho Vineyard', product: 'Kelvin Forecasting', sku: 'FCT-202', region: 'APAC', channel: 'direct', rep: 'Priya Nair', quantity: 1, unitPrice: 980, status: 'pending' }, + { id: 'ORD-10008', placedAt: '2024-05-18T12:05:00Z', customer: 'Alpine Ski House', product: 'Nimbus Compute', sku: 'CMP-505', region: 'North America', channel: 'self-serve', rep: 'Hugo Bernard', quantity: 8, unitPrice: 1100, status: 'delivered' }, + { id: 'ORD-10009', placedAt: '2024-05-22T14:47:00Z', customer: 'Trey Research', product: 'Onyx Security', sku: 'SEC-606', region: 'EMEA', channel: 'direct', rep: 'Sven Olsen', quantity: 2, unitPrice: 860, status: 'cancelled' }, + { id: 'ORD-10010', placedAt: '2024-05-26T16:39:00Z', customer: 'Blue Yonder Airlines', product: 'Polaris Reporting', sku: 'RPT-707', region: 'LATAM', channel: 'partner', rep: 'Mateo Russo', quantity: 9, unitPrice: 290, status: 'paid' } +] + +export async function main({ + from, + to, + region, + status +}: { + from: string + to: string + region: string + status: string +}): Promise<{ orders: Order[]; total: number }> { + let filtered = orders.filter((order) => { + const day = order.placedAt.slice(0, 10) + return day >= from && day <= to + }) + if (region && region !== 'all') { + filtered = filtered.filter((order) => order.region === region) + } + if (status && status !== 'all') { + filtered = filtered.filter((order) => order.status === status) + } + return { orders: filtered, total: filtered.length } +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json new file mode 100644 index 0000000000..098e077560 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/backend/loadOrders/meta.json @@ -0,0 +1,4 @@ +{ + "name": "Load Orders", + "language": "bun" +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx new file mode 100644 index 0000000000..f0a76a09f3 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/DateRangePicker.tsx @@ -0,0 +1,45 @@ +import React from 'react' +import type { DateRange } from '../lib/api' +import { rangeForPreset } from '../lib/api' +import { formatDateShort } from '../lib/format' + +interface DateRangePickerProps { + preset: string + range: DateRange + onPresetChange: (preset: string, range: DateRange) => void +} + +const PRESETS: { id: string; label: string }[] = [ + { id: '7d', label: 'Last 7 days' }, + { id: '14d', label: 'Last 14 days' }, + { id: '30d', label: 'Last 30 days' }, + { id: 'qtd', label: 'Quarter to date' } +] + +export const DateRangePicker: React.FC = ({ + preset, + range, + onPresetChange +}) => { + return ( +
+ + + {formatDateShort(range.from)} – {formatDateShort(range.to)} + +
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx new file mode 100644 index 0000000000..f4dc840c60 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/EmptyState.tsx @@ -0,0 +1,28 @@ +import React from 'react' + +interface EmptyStateProps { + title: string + description?: string + icon?: string + action?: React.ReactNode +} + +export const EmptyState: React.FC = ({ + title, + description, + icon = '📊', + action +}) => { + return ( +
+
+ {icon} +
+

{title}

+ {description ? ( +

{description}

+ ) : null} + {action ?
{action}
: null} +
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx new file mode 100644 index 0000000000..6fc9b5adde --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/ExportButton.tsx @@ -0,0 +1,51 @@ +import React, { useState } from 'react' +import { requestExport } from '../lib/api' +import type { DateRange } from '../lib/api' + +interface ExportButtonProps { + range: DateRange + region: string +} + +export const ExportButton: React.FC = ({ range, region }) => { + const [busy, setBusy] = useState(false) + const [error, setError] = useState(null) + + const handleExport = async (format: 'csv' | 'json') => { + setBusy(true) + setError(null) + try { + const result = await requestExport(range, region, format) + const anchor = document.createElement('a') + anchor.href = result.url + anchor.download = `revenue-report.${format}` + anchor.click() + } catch (err) { + setError(err instanceof Error ? err.message : 'Export failed') + } finally { + setBusy(false) + } + } + + return ( +
+ + + {error ? {error} : null} +
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx new file mode 100644 index 0000000000..38de9d6a9b --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/FilterBar.tsx @@ -0,0 +1,59 @@ +import React from 'react' +import type { DateRange } from '../lib/api' +import type { OrderStatus } from '../data/seedData' +import { REGIONS, ORDER_STATUSES, STATUS_LABELS } from '../data/seedData' +import { DateRangePicker } from './DateRangePicker' +import { ExportButton } from './ExportButton' + +interface FilterBarProps { + region: string + status: string + preset: string + range: DateRange + onRegionChange: (region: string) => void + onStatusChange: (status: string) => void + onPresetChange: (preset: string, range: DateRange) => void +} + +export const FilterBar: React.FC = ({ + region, + status, + preset, + range, + onRegionChange, + onStatusChange, + onPresetChange +}) => { + return ( +
+
+ + + +
+ +
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx new file mode 100644 index 0000000000..23dcd90123 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricCard.tsx @@ -0,0 +1,40 @@ +import React from 'react' +import type { MetricCardData } from '../data/seedData' +import { formatCurrency, formatNumber, formatPercent, formatSignedPercent } from '../lib/format' + +interface MetricCardProps { + metric: MetricCardData + loading?: boolean +} + +function renderValue(metric: MetricCardData): string { + switch (metric.unit) { + case 'currency': + return formatCurrency(metric.value) + case 'percent': + return formatPercent(metric.value) + case 'count': + default: + return formatNumber(metric.value) + } +} + +export const MetricCard: React.FC = ({ metric, loading }) => { + const positive = metric.delta >= 0 + return ( +
+
+ {metric.label} + + {formatSignedPercent(metric.delta)} + +
+
+ {loading ? … : renderValue(metric)} +
+

{metric.hint}

+
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx new file mode 100644 index 0000000000..c42a8aa733 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/MetricGrid.tsx @@ -0,0 +1,18 @@ +import React from 'react' +import type { MetricCardData } from '../data/seedData' +import { MetricCard } from './MetricCard' + +interface MetricGridProps { + metrics: MetricCardData[] + loading?: boolean +} + +export const MetricGrid: React.FC = ({ metrics, loading }) => { + return ( +
+ {metrics.map((metric) => ( + + ))} +
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx new file mode 100644 index 0000000000..fa0411c371 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/OrdersTable.tsx @@ -0,0 +1,117 @@ +import React, { useMemo, useState } from 'react' +import type { Order } from '../data/seedData' +import { StatusBadge } from './StatusBadge' +import { EmptyState } from './EmptyState' +import { formatCurrencyPrecise, formatDate, formatNumber, truncate } from '../lib/format' + +interface OrdersTableProps { + orders: Order[] + loading?: boolean +} + +type SortKey = 'placedAt' | 'customer' | 'lineTotal' | 'quantity' +type SortDir = 'asc' | 'desc' + +// The per-row line total a customer was charged: unit price times quantity. +function lineTotal(order: Order): number { + return order.quantity * order.unitPrice +} + +export const OrdersTable: React.FC = ({ orders, loading }) => { + const [sortKey, setSortKey] = useState('placedAt') + const [sortDir, setSortDir] = useState('desc') + + const sorted = useMemo(() => { + const copy = [...orders] + copy.sort((a, b) => { + let comparison = 0 + switch (sortKey) { + case 'customer': + comparison = a.customer.localeCompare(b.customer) + break + case 'lineTotal': + comparison = lineTotal(a) - lineTotal(b) + break + case 'quantity': + comparison = a.quantity - b.quantity + break + case 'placedAt': + default: + comparison = a.placedAt.localeCompare(b.placedAt) + break + } + return sortDir === 'asc' ? comparison : -comparison + }) + return copy + }, [orders, sortKey, sortDir]) + + const toggleSort = (key: SortKey) => { + if (key === sortKey) { + setSortDir((dir) => (dir === 'asc' ? 'desc' : 'asc')) + } else { + setSortKey(key) + setSortDir('desc') + } + } + + if (!loading && orders.length === 0) { + return ( + + ) + } + + const arrow = (key: SortKey) => (key === sortKey ? (sortDir === 'asc' ? '▲' : '▼') : '') + + return ( +
+ + + + + + + + + + + + + + + {sorted.map((order) => ( + + + + + + + + + + + ))} + +
toggleSort('placedAt')}> + Date {arrow('placedAt')} + toggleSort('customer')}> + Customer {arrow('customer')} + ProductRegion toggleSort('quantity')}> + Qty {arrow('quantity')} + Unit Price toggleSort('lineTotal')}> + Line Total {arrow('lineTotal')} + Status
{formatDate(order.placedAt)} + {truncate(order.customer, 24)} + {order.product}{order.region}{formatNumber(order.quantity)} + {formatCurrencyPrecise(order.unitPrice)} + + {formatCurrencyPrecise(lineTotal(order))} + + +
+
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx new file mode 100644 index 0000000000..41e47430b9 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RegionTable.tsx @@ -0,0 +1,52 @@ +import React, { useMemo } from 'react' +import type { Order } from '../data/seedData' +import { breakdownByRegion } from '../lib/aggregations' +import { formatCurrency, formatNumber, formatPercent } from '../lib/format' +import { EmptyState } from './EmptyState' + +interface RegionTableProps { + orders: Order[] +} + +export const RegionTable: React.FC = ({ orders }) => { + const rows = useMemo(() => breakdownByRegion(orders), [orders]) + const total = useMemo(() => rows.reduce((acc, row) => acc + row.revenue, 0), [rows]) + + if (rows.length === 0) { + return ( + + ) + } + + return ( +
+

Revenue by Region

+ + + + + + + + + + + {rows.map((row) => ( + + + + + + + ))} + +
RegionOrdersRevenueShare
{row.region}{formatNumber(row.orders)}{formatCurrency(row.revenue)} + {formatPercent(total === 0 ? 0 : row.revenue / total)} +
+
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx new file mode 100644 index 0000000000..f16fe779fe --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/RevenueChart.tsx @@ -0,0 +1,49 @@ +import React, { useMemo } from 'react' +import type { Order } from '../data/seedData' +import { dailyRevenue } from '../lib/aggregations' +import { formatCompact, formatDateShort } from '../lib/format' +import { EmptyState } from './EmptyState' + +interface RevenueChartProps { + orders: Order[] +} + +// Lightweight inline bar chart for daily revenue. Avoids a charting dependency +// by sizing flexed columns relative to the busiest day in the window. +export const RevenueChart: React.FC = ({ orders }) => { + const points = useMemo(() => dailyRevenue(orders), [orders]) + const max = useMemo(() => points.reduce((acc, point) => Math.max(acc, point.revenue), 0), [points]) + + if (points.length === 0) { + return ( + + ) + } + + return ( +
+

Daily Revenue

+
+ {points.map((point) => { + const heightPct = max === 0 ? 0 : Math.round((point.revenue / max) * 100) + return ( +
+
+ + {formatDateShort(point.date)} + +
+ ) + })} +
+
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx new file mode 100644 index 0000000000..5e92f4719b --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/Sidebar.tsx @@ -0,0 +1,50 @@ +import React from 'react' + +export type DashboardView = 'overview' | 'orders' | 'regions' | 'products' + +interface SidebarProps { + active: DashboardView + onSelect: (view: DashboardView) => void +} + +const NAV_ITEMS: { id: DashboardView; label: string; icon: string }[] = [ + { id: 'overview', label: 'Overview', icon: '📈' }, + { id: 'orders', label: 'Orders', icon: '🧾' }, + { id: 'regions', label: 'Regions', icon: '🌍' }, + { id: 'products', label: 'Products', icon: '📦' } +] + +export const Sidebar: React.FC = ({ active, onSelect }) => { + return ( + + ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx new file mode 100644 index 0000000000..114d2a7073 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/StatusBadge.tsx @@ -0,0 +1,26 @@ +import React from 'react' +import type { OrderStatus } from '../data/seedData' +import { STATUS_LABELS } from '../data/seedData' + +interface StatusBadgeProps { + status: OrderStatus +} + +const STATUS_STYLES: Record = { + paid: 'bg-blue-100 text-blue-700', + shipped: 'bg-indigo-100 text-indigo-700', + delivered: 'bg-emerald-100 text-emerald-700', + pending: 'bg-amber-100 text-amber-700', + refunded: 'bg-rose-100 text-rose-700', + cancelled: 'bg-gray-200 text-gray-600' +} + +export const StatusBadge: React.FC = ({ status }) => { + return ( + + {STATUS_LABELS[status]} + + ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx new file mode 100644 index 0000000000..0ec6a345bc --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/SummaryPanel.tsx @@ -0,0 +1,51 @@ +import React, { useMemo } from 'react' +import type { Order } from '../data/seedData' +import { summarizeRevenue } from '../lib/aggregations' +import { formatCurrency, formatCurrencyPrecise, formatNumber } from '../lib/format' + +interface SummaryPanelProps { + orders: Order[] + loading?: boolean +} + +// Headline revenue panel. It re-aggregates the orders client-side via +// summarizeRevenue so the totals stay in sync with whatever filter the user +// has applied, without waiting for another backend round trip. +export const SummaryPanel: React.FC = ({ orders, loading }) => { + const summary = useMemo(() => summarizeRevenue(orders), [orders]) + + const tiles = [ + { label: 'Total Revenue', value: formatCurrency(summary.totalRevenue), emphasis: true }, + { label: 'Net Revenue', value: formatCurrency(summary.netRevenue) }, + { label: 'Orders', value: formatNumber(summary.totalOrders) }, + { label: 'Avg Order Value', value: formatCurrencyPrecise(summary.averageOrderValue) }, + { label: 'Units Sold', value: formatNumber(summary.unitsSold) }, + { label: 'Refunded', value: formatCurrency(summary.refundedRevenue) } + ] + + return ( +
+
+

Revenue Summary

+ {loading ? Refreshing… : null} +
+
+ {tiles.map((tile) => ( +
+
+ {tile.label} +
+
+ {tile.value} +
+
+ ))} +
+
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx new file mode 100644 index 0000000000..9876e9701d --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/components/TopProducts.tsx @@ -0,0 +1,55 @@ +import React, { useMemo } from 'react' +import type { Order } from '../data/seedData' +import { topProducts } from '../lib/aggregations' +import { formatCurrency } from '../lib/format' +import { EmptyState } from './EmptyState' + +interface TopProductsProps { + orders: Order[] + limit?: number +} + +export const TopProducts: React.FC = ({ orders, limit = 5 }) => { + const products = useMemo(() => topProducts(orders, limit), [orders, limit]) + const max = useMemo( + () => products.reduce((acc, item) => Math.max(acc, item.revenue), 0), + [products] + ) + + if (products.length === 0) { + return ( + + ) + } + + return ( +
+

Top Products

+
    + {products.map((item, index) => { + const widthPct = max === 0 ? 0 : Math.round((item.revenue / max) * 100) + return ( +
  • +
    + + {index + 1}. {item.product} + + {formatCurrency(item.revenue)} +
    +
    +
    +
    +
  • + ) + })} +
+
+ ) +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/data/seedData.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/data/seedData.ts new file mode 100644 index 0000000000..742a83be3d --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/data/seedData.ts @@ -0,0 +1,5051 @@ +// Synthetic seed data for the analytics dashboard. +// +// This module is the app's offline data source: the backend runnables filter +// and aggregate these rows, and a few components fall back to them when a +// backend call is unavailable in preview. The array is intentionally large and +// varied so the dashboard renders realistic totals, regional splits, and time +// series. It is plain committed source — do not regenerate it at load time. + +export type OrderStatus = + | 'paid' + | 'shipped' + | 'delivered' + | 'pending' + | 'refunded' + | 'cancelled' + +export interface Order { + id: string + placedAt: string + customer: string + product: string + sku: string + region: string + channel: string + rep: string + quantity: number + unitPrice: number + status: OrderStatus +} + +export interface MetricCardData { + id: string + label: string + value: number + unit: 'currency' | 'count' | 'percent' + delta: number + hint: string +} + +export const PRODUCT_CATALOG: { name: string; sku: string; listPrice: number }[] = [ + { name: "Aurora Analytics Suite", sku: "ANL-100", listPrice: 1200 }, + { name: "Borealis CRM", sku: "CRM-210", listPrice: 890 }, + { name: "Cascade Data Pipeline", sku: "PIPE-330", listPrice: 640 }, + { name: "Delta Insights", sku: "INS-440", listPrice: 320 }, + { name: "Echo Monitoring", sku: "MON-550", listPrice: 150 }, + { name: "Fjord Storage", sku: "STO-660", listPrice: 75 }, + { name: "Glacier Backup", sku: "BAK-770", listPrice: 45 }, + { name: "Helix Identity", sku: "IDN-880", listPrice: 220 }, + { name: "Ion Messaging", sku: "MSG-990", listPrice: 60 }, + { name: "Juniper Workflow", sku: "WFL-101", listPrice: 410 }, + { name: "Kelvin Forecasting", sku: "FCT-202", listPrice: 980 }, + { name: "Lumen Dashboards", sku: "DSH-303", listPrice: 520 }, + { name: "Meridian ETL", sku: "ETL-404", listPrice: 730 }, + { name: "Nimbus Compute", sku: "CMP-505", listPrice: 1100 }, + { name: "Onyx Security", sku: "SEC-606", listPrice: 860 }, + { name: "Polaris Reporting", sku: "RPT-707", listPrice: 290 }, +] + +export const REGIONS: string[] = [ + "North America", + "EMEA", + "APAC", + "LATAM", +] + +export const ORDER_STATUSES: OrderStatus[] = [ + "paid", + "shipped", + "delivered", + "pending", + "refunded", + "cancelled", +] + +export const STATUS_LABELS: Record = { + paid: 'Paid', + shipped: 'Shipped', + delivered: 'Delivered', + pending: 'Pending', + refunded: 'Refunded', + cancelled: 'Cancelled' +} + +export const seedOrders: Order[] = [ + { + id: "ORD-10070", + placedAt: "2024-05-01T02:15:00Z", + customer: "Wingtip Toys", + product: "Fjord Storage", + sku: "STO-660", + region: "EMEA", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 71, + status: "pending" + }, + { + id: "ORD-10245", + placedAt: "2024-05-01T03:12:00Z", + customer: "Adventure Works", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "partner", + rep: "Priya Nair", + quantity: 8, + unitPrice: 161, + status: "shipped" + }, + { + id: "ORD-10368", + placedAt: "2024-05-01T06:05:00Z", + customer: "Blue Yonder Airlines", + product: "Polaris Reporting", + sku: "RPT-707", + region: "EMEA", + channel: "direct", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 295, + status: "paid" + }, + { + id: "ORD-10235", + placedAt: "2024-05-01T07:51:00Z", + customer: "Contoso Ltd", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "partner", + rep: "Priya Nair", + quantity: 7, + unitPrice: 997, + status: "shipped" + }, + { + id: "ORD-10142", + placedAt: "2024-05-01T08:59:00Z", + customer: "Lucerne Publishing", + product: "Nimbus Compute", + sku: "CMP-505", + region: "LATAM", + channel: "marketplace", + rep: "Priya Nair", + quantity: 6, + unitPrice: 1094, + status: "refunded" + }, + { + id: "ORD-10300", + placedAt: "2024-05-01T12:08:00Z", + customer: "Wide World Importers", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "EMEA", + channel: "self-serve", + rep: "Priya Nair", + quantity: 7, + unitPrice: 516, + status: "shipped" + }, + { + id: "ORD-10229", + placedAt: "2024-05-01T12:26:00Z", + customer: "Lucerne Publishing", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "marketplace", + rep: "Sven Olsen", + quantity: 4, + unitPrice: 147, + status: "refunded" + }, + { + id: "ORD-10091", + placedAt: "2024-05-01T14:29:00Z", + customer: "Wingtip Toys", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "LATAM", + channel: "direct", + rep: "Diego Marin", + quantity: 2, + unitPrice: 984, + status: "delivered" + }, + { + id: "ORD-10329", + placedAt: "2024-05-01T19:07:00Z", + customer: "City Power & Light", + product: "Ion Messaging", + sku: "MSG-990", + region: "EMEA", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 5, + unitPrice: 59, + status: "delivered" + }, + { + id: "ORD-10200", + placedAt: "2024-05-01T20:25:00Z", + customer: "Fourth Coffee", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 223, + status: "delivered" + }, + { + id: "ORD-10177", + placedAt: "2024-05-01T21:13:00Z", + customer: "Wingtip Toys", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "partner", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 1188, + status: "paid" + }, + { + id: "ORD-10101", + placedAt: "2024-05-01T21:21:00Z", + customer: "Fabrikam Inc", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "marketplace", + rep: "Dana Wills", + quantity: 1, + unitPrice: 155, + status: "shipped" + }, + { + id: "ORD-10148", + placedAt: "2024-05-01T21:24:00Z", + customer: "Coho Vineyard", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "partner", + rep: "Sven Olsen", + quantity: 7, + unitPrice: 313, + status: "refunded" + }, + { + id: "ORD-10358", + placedAt: "2024-05-02T00:56:00Z", + customer: "Blue Yonder Airlines", + product: "Fjord Storage", + sku: "STO-660", + region: "EMEA", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 63, + status: "cancelled" + }, + { + id: "ORD-10059", + placedAt: "2024-05-02T06:34:00Z", + customer: "Coho Vineyard", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "partner", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 990, + status: "refunded" + }, + { + id: "ORD-10279", + placedAt: "2024-05-02T13:07:00Z", + customer: "Wide World Importers", + product: "Glacier Backup", + sku: "BAK-770", + region: "North America", + channel: "direct", + rep: "Priya Nair", + quantity: 7, + unitPrice: 47, + status: "delivered" + }, + { + id: "ORD-10215", + placedAt: "2024-05-02T14:05:00Z", + customer: "Adventure Works", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "direct", + rep: "Sora Tanaka", + quantity: 1, + unitPrice: 29, + status: "pending" + }, + { + id: "ORD-10341", + placedAt: "2024-05-02T15:19:00Z", + customer: "Northwind Traders", + product: "Echo Monitoring", + sku: "MON-550", + region: "North America", + channel: "direct", + rep: "Sora Tanaka", + quantity: 5, + unitPrice: 160, + status: "paid" + }, + { + id: "ORD-10039", + placedAt: "2024-05-02T15:24:00Z", + customer: "Alpine Ski House", + product: "Glacier Backup", + sku: "BAK-770", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 4, + unitPrice: 46, + status: "paid" + }, + { + id: "ORD-10380", + placedAt: "2024-05-02T18:33:00Z", + customer: "Margies Travel", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "self-serve", + rep: "Sora Tanaka", + quantity: 5, + unitPrice: 537, + status: "shipped" + }, + { + id: "ORD-10083", + placedAt: "2024-05-02T21:21:00Z", + customer: "Fabrikam Inc", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "APAC", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 634, + status: "shipped" + }, + { + id: "ORD-10233", + placedAt: "2024-05-02T22:38:00Z", + customer: "Contoso Ltd", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 71, + status: "delivered" + }, + { + id: "ORD-10243", + placedAt: "2024-05-03T03:42:00Z", + customer: "Northwind Traders", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "self-serve", + rep: "Diego Marin", + quantity: 2, + unitPrice: 634, + status: "paid" + }, + { + id: "ORD-10269", + placedAt: "2024-05-03T03:43:00Z", + customer: "City Power & Light", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 745, + status: "shipped" + }, + { + id: "ORD-10230", + placedAt: "2024-05-03T07:54:00Z", + customer: "Adventure Works", + product: "Fjord Storage", + sku: "STO-660", + region: "APAC", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 73, + status: "shipped" + }, + { + id: "ORD-10354", + placedAt: "2024-05-03T10:30:00Z", + customer: "Humongous Insurance", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "partner", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 879, + status: "refunded" + }, + { + id: "ORD-10256", + placedAt: "2024-05-03T12:01:00Z", + customer: "Tailspin Toys", + product: "Polaris Reporting", + sku: "RPT-707", + region: "LATAM", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 1, + unitPrice: 298, + status: "delivered" + }, + { + id: "ORD-10146", + placedAt: "2024-05-03T14:03:00Z", + customer: "Fourth Coffee", + product: "Borealis CRM", + sku: "CRM-210", + region: "APAC", + channel: "marketplace", + rep: "Sven Olsen", + quantity: 8, + unitPrice: 895, + status: "cancelled" + }, + { + id: "ORD-10162", + placedAt: "2024-05-03T14:09:00Z", + customer: "Margies Travel", + product: "Borealis CRM", + sku: "CRM-210", + region: "APAC", + channel: "self-serve", + rep: "Sora Tanaka", + quantity: 2, + unitPrice: 887, + status: "shipped" + }, + { + id: "ORD-10313", + placedAt: "2024-05-03T21:27:00Z", + customer: "Fourth Coffee", + product: "Ion Messaging", + sku: "MSG-990", + region: "EMEA", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 1, + unitPrice: 46, + status: "refunded" + }, + { + id: "ORD-10282", + placedAt: "2024-05-03T22:46:00Z", + customer: "Lucerne Publishing", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "self-serve", + rep: "Diego Marin", + quantity: 7, + unitPrice: 405, + status: "delivered" + }, + { + id: "ORD-10224", + placedAt: "2024-05-03T23:05:00Z", + customer: "Contoso Ltd", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "self-serve", + rep: "Diego Marin", + quantity: 8, + unitPrice: 308, + status: "shipped" + }, + { + id: "ORD-10372", + placedAt: "2024-05-03T23:35:00Z", + customer: "Lucerne Publishing", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "marketplace", + rep: "Priya Nair", + quantity: 2, + unitPrice: 314, + status: "delivered" + }, + { + id: "ORD-10073", + placedAt: "2024-05-04T02:12:00Z", + customer: "City Power & Light", + product: "Ion Messaging", + sku: "MSG-990", + region: "North America", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 55, + status: "delivered" + }, + { + id: "ORD-10323", + placedAt: "2024-05-04T06:09:00Z", + customer: "Litware Inc", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "direct", + rep: "Priya Nair", + quantity: 1, + unitPrice: 635, + status: "shipped" + }, + { + id: "ORD-10012", + placedAt: "2024-05-04T06:18:00Z", + customer: "Proseware Inc", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "North America", + channel: "marketplace", + rep: "Dana Wills", + quantity: 4, + unitPrice: 526, + status: "shipped" + }, + { + id: "ORD-10124", + placedAt: "2024-05-04T06:28:00Z", + customer: "Litware Inc", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 2, + unitPrice: 503, + status: "refunded" + }, + { + id: "ORD-10338", + placedAt: "2024-05-04T06:32:00Z", + customer: "Northwind Traders", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "direct", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 895, + status: "cancelled" + }, + { + id: "ORD-10194", + placedAt: "2024-05-04T07:15:00Z", + customer: "Northwind Traders", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "direct", + rep: "Dana Wills", + quantity: 6, + unitPrice: 881, + status: "delivered" + }, + { + id: "ORD-10006", + placedAt: "2024-05-04T11:55:00Z", + customer: "Litware Inc", + product: "Fjord Storage", + sku: "STO-660", + region: "EMEA", + channel: "partner", + rep: "Sven Olsen", + quantity: 6, + unitPrice: 85, + status: "delivered" + }, + { + id: "ORD-10183", + placedAt: "2024-05-04T14:27:00Z", + customer: "Humongous Insurance", + product: "Glacier Backup", + sku: "BAK-770", + region: "EMEA", + channel: "partner", + rep: "Diego Marin", + quantity: 1, + unitPrice: 62, + status: "paid" + }, + { + id: "ORD-10079", + placedAt: "2024-05-04T19:21:00Z", + customer: "Litware Inc", + product: "Onyx Security", + sku: "SEC-606", + region: "EMEA", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 840, + status: "delivered" + }, + { + id: "ORD-10304", + placedAt: "2024-05-04T20:22:00Z", + customer: "Graphic Design Institute", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "direct", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 287, + status: "shipped" + }, + { + id: "ORD-10020", + placedAt: "2024-05-04T20:56:00Z", + customer: "School of Fine Art", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "direct", + rep: "Priya Nair", + quantity: 3, + unitPrice: 324, + status: "delivered" + }, + { + id: "ORD-10133", + placedAt: "2024-05-05T02:28:00Z", + customer: "Coho Vineyard", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "partner", + rep: "Dana Wills", + quantity: 1, + unitPrice: 152, + status: "refunded" + }, + { + id: "ORD-10028", + placedAt: "2024-05-05T02:50:00Z", + customer: "Northwind Traders", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "LATAM", + channel: "marketplace", + rep: "Dana Wills", + quantity: 7, + unitPrice: 513, + status: "delivered" + }, + { + id: "ORD-10097", + placedAt: "2024-05-05T08:20:00Z", + customer: "Lucerne Publishing", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "North America", + channel: "partner", + rep: "Dana Wills", + quantity: 7, + unitPrice: 1218, + status: "delivered" + }, + { + id: "ORD-10117", + placedAt: "2024-05-05T08:35:00Z", + customer: "Northwind Traders", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "marketplace", + rep: "Diego Marin", + quantity: 5, + unitPrice: 160, + status: "paid" + }, + { + id: "ORD-10109", + placedAt: "2024-05-05T09:18:00Z", + customer: "Fourth Coffee", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 4, + unitPrice: 734, + status: "delivered" + }, + { + id: "ORD-10285", + placedAt: "2024-05-05T11:54:00Z", + customer: "Alpine Ski House", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "self-serve", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 711, + status: "shipped" + }, + { + id: "ORD-10107", + placedAt: "2024-05-05T12:04:00Z", + customer: "Margies Travel", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "EMEA", + channel: "partner", + rep: "Diego Marin", + quantity: 1, + unitPrice: 986, + status: "cancelled" + }, + { + id: "ORD-10366", + placedAt: "2024-05-05T17:27:00Z", + customer: "Blue Yonder Airlines", + product: "Nimbus Compute", + sku: "CMP-505", + region: "APAC", + channel: "partner", + rep: "Sven Olsen", + quantity: 6, + unitPrice: 1108, + status: "shipped" + }, + { + id: "ORD-10356", + placedAt: "2024-05-05T19:37:00Z", + customer: "Lucerne Publishing", + product: "Delta Insights", + sku: "INS-440", + region: "LATAM", + channel: "marketplace", + rep: "Dana Wills", + quantity: 2, + unitPrice: 328, + status: "refunded" + }, + { + id: "ORD-10052", + placedAt: "2024-05-05T20:26:00Z", + customer: "Adventure Works", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "direct", + rep: "Mateo Russo", + quantity: 2, + unitPrice: 312, + status: "delivered" + }, + { + id: "ORD-10367", + placedAt: "2024-05-05T23:03:00Z", + customer: "Litware Inc", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 5, + unitPrice: 879, + status: "pending" + }, + { + id: "ORD-10234", + placedAt: "2024-05-05T23:09:00Z", + customer: "Margies Travel", + product: "Juniper Workflow", + sku: "WFL-101", + region: "EMEA", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 415, + status: "delivered" + }, + { + id: "ORD-10280", + placedAt: "2024-05-06T03:20:00Z", + customer: "Wide World Importers", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "partner", + rep: "Priya Nair", + quantity: 6, + unitPrice: 205, + status: "shipped" + }, + { + id: "ORD-10221", + placedAt: "2024-05-06T07:55:00Z", + customer: "Litware Inc", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "partner", + rep: "Owen Pratt", + quantity: 1, + unitPrice: 718, + status: "refunded" + }, + { + id: "ORD-10332", + placedAt: "2024-05-06T08:38:00Z", + customer: "Lucerne Publishing", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "marketplace", + rep: "Dana Wills", + quantity: 1, + unitPrice: 504, + status: "delivered" + }, + { + id: "ORD-10121", + placedAt: "2024-05-06T11:55:00Z", + customer: "Wide World Importers", + product: "Ion Messaging", + sku: "MSG-990", + region: "North America", + channel: "marketplace", + rep: "Dana Wills", + quantity: 5, + unitPrice: 50, + status: "delivered" + }, + { + id: "ORD-10128", + placedAt: "2024-05-06T13:54:00Z", + customer: "City Power & Light", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "marketplace", + rep: "Mateo Russo", + quantity: 7, + unitPrice: 299, + status: "delivered" + }, + { + id: "ORD-10238", + placedAt: "2024-05-06T15:53:00Z", + customer: "Wide World Importers", + product: "Nimbus Compute", + sku: "CMP-505", + region: "EMEA", + channel: "partner", + rep: "Sora Tanaka", + quantity: 2, + unitPrice: 1090, + status: "shipped" + }, + { + id: "ORD-10193", + placedAt: "2024-05-06T15:55:00Z", + customer: "Trey Research", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "EMEA", + channel: "marketplace", + rep: "Mateo Russo", + quantity: 6, + unitPrice: 1198, + status: "delivered" + }, + { + id: "ORD-10111", + placedAt: "2024-05-06T16:37:00Z", + customer: "Fourth Coffee", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "self-serve", + rep: "Aisha Khan", + quantity: 7, + unitPrice: 868, + status: "delivered" + }, + { + id: "ORD-10220", + placedAt: "2024-05-06T16:57:00Z", + customer: "Margies Travel", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "LATAM", + channel: "self-serve", + rep: "Sora Tanaka", + quantity: 6, + unitPrice: 525, + status: "pending" + }, + { + id: "ORD-10076", + placedAt: "2024-05-06T17:25:00Z", + customer: "Tailspin Toys", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "EMEA", + channel: "partner", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 500, + status: "refunded" + }, + { + id: "ORD-10369", + placedAt: "2024-05-06T17:43:00Z", + customer: "Coho Vineyard", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "LATAM", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 1190, + status: "shipped" + }, + { + id: "ORD-10378", + placedAt: "2024-05-06T19:28:00Z", + customer: "Fourth Coffee", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 7, + unitPrice: 397, + status: "delivered" + }, + { + id: "ORD-10283", + placedAt: "2024-05-06T20:02:00Z", + customer: "Margies Travel", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "APAC", + channel: "self-serve", + rep: "Dana Wills", + quantity: 4, + unitPrice: 983, + status: "shipped" + }, + { + id: "ORD-10277", + placedAt: "2024-05-06T20:35:00Z", + customer: "Blue Yonder Airlines", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "partner", + rep: "Hugo Bernard", + quantity: 6, + unitPrice: 149, + status: "delivered" + }, + { + id: "ORD-10212", + placedAt: "2024-05-07T02:01:00Z", + customer: "Wide World Importers", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 4, + unitPrice: 337, + status: "pending" + }, + { + id: "ORD-10340", + placedAt: "2024-05-07T03:23:00Z", + customer: "Blue Yonder Airlines", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "marketplace", + rep: "Dana Wills", + quantity: 7, + unitPrice: 332, + status: "paid" + }, + { + id: "ORD-10044", + placedAt: "2024-05-07T04:10:00Z", + customer: "Proseware Inc", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "LATAM", + channel: "direct", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 526, + status: "delivered" + }, + { + id: "ORD-10293", + placedAt: "2024-05-07T05:52:00Z", + customer: "Fourth Coffee", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "partner", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 166, + status: "shipped" + }, + { + id: "ORD-10058", + placedAt: "2024-05-07T06:38:00Z", + customer: "Litware Inc", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "direct", + rep: "Aisha Khan", + quantity: 2, + unitPrice: 423, + status: "delivered" + }, + { + id: "ORD-10298", + placedAt: "2024-05-07T06:47:00Z", + customer: "Wide World Importers", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "partner", + rep: "Owen Pratt", + quantity: 1, + unitPrice: 413, + status: "shipped" + }, + { + id: "ORD-10259", + placedAt: "2024-05-07T11:51:00Z", + customer: "Trey Research", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "direct", + rep: "Dana Wills", + quantity: 5, + unitPrice: 627, + status: "cancelled" + }, + { + id: "ORD-10295", + placedAt: "2024-05-07T14:13:00Z", + customer: "Humongous Insurance", + product: "Glacier Backup", + sku: "BAK-770", + region: "LATAM", + channel: "marketplace", + rep: "Dana Wills", + quantity: 8, + unitPrice: 60, + status: "delivered" + }, + { + id: "ORD-10219", + placedAt: "2024-05-07T14:15:00Z", + customer: "Northwind Traders", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 4, + unitPrice: 983, + status: "refunded" + }, + { + id: "ORD-10043", + placedAt: "2024-05-07T15:24:00Z", + customer: "Humongous Insurance", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "APAC", + channel: "direct", + rep: "Lena Fischer", + quantity: 1, + unitPrice: 978, + status: "delivered" + }, + { + id: "ORD-10271", + placedAt: "2024-05-07T15:28:00Z", + customer: "Margies Travel", + product: "Onyx Security", + sku: "SEC-606", + region: "APAC", + channel: "self-serve", + rep: "Aisha Khan", + quantity: 7, + unitPrice: 849, + status: "delivered" + }, + { + id: "ORD-10050", + placedAt: "2024-05-07T19:40:00Z", + customer: "Wide World Importers", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 5, + unitPrice: 888, + status: "delivered" + }, + { + id: "ORD-10167", + placedAt: "2024-05-07T21:03:00Z", + customer: "Humongous Insurance", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "partner", + rep: "Aisha Khan", + quantity: 8, + unitPrice: 49, + status: "pending" + }, + { + id: "ORD-10328", + placedAt: "2024-05-08T00:23:00Z", + customer: "Wingtip Toys", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "direct", + rep: "Mateo Russo", + quantity: 7, + unitPrice: 204, + status: "shipped" + }, + { + id: "ORD-10089", + placedAt: "2024-05-08T01:57:00Z", + customer: "Adventure Works", + product: "Ion Messaging", + sku: "MSG-990", + region: "EMEA", + channel: "direct", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 71, + status: "delivered" + }, + { + id: "ORD-10166", + placedAt: "2024-05-08T02:21:00Z", + customer: "Margies Travel", + product: "Fjord Storage", + sku: "STO-660", + region: "LATAM", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 5, + unitPrice: 86, + status: "cancelled" + }, + { + id: "ORD-10067", + placedAt: "2024-05-08T03:31:00Z", + customer: "Fourth Coffee", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "LATAM", + channel: "self-serve", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 645, + status: "delivered" + }, + { + id: "ORD-10004", + placedAt: "2024-05-08T08:06:00Z", + customer: "Fourth Coffee", + product: "Delta Insights", + sku: "INS-440", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 8, + unitPrice: 326, + status: "paid" + }, + { + id: "ORD-10174", + placedAt: "2024-05-08T16:11:00Z", + customer: "Adventure Works", + product: "Nimbus Compute", + sku: "CMP-505", + region: "EMEA", + channel: "partner", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 1113, + status: "refunded" + }, + { + id: "ORD-10204", + placedAt: "2024-05-08T18:09:00Z", + customer: "Contoso Ltd", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "North America", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 1, + unitPrice: 510, + status: "delivered" + }, + { + id: "ORD-10173", + placedAt: "2024-05-08T20:58:00Z", + customer: "School of Fine Art", + product: "Meridian ETL", + sku: "ETL-404", + region: "North America", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 6, + unitPrice: 715, + status: "paid" + }, + { + id: "ORD-10297", + placedAt: "2024-05-08T22:20:00Z", + customer: "Tailspin Toys", + product: "Ion Messaging", + sku: "MSG-990", + region: "APAC", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 5, + unitPrice: 56, + status: "delivered" + }, + { + id: "ORD-10144", + placedAt: "2024-05-08T22:40:00Z", + customer: "Lucerne Publishing", + product: "Polaris Reporting", + sku: "RPT-707", + region: "EMEA", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 6, + unitPrice: 287, + status: "paid" + }, + { + id: "ORD-10123", + placedAt: "2024-05-09T00:54:00Z", + customer: "Tailspin Toys", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 8, + unitPrice: 966, + status: "cancelled" + }, + { + id: "ORD-10038", + placedAt: "2024-05-09T02:54:00Z", + customer: "Fourth Coffee", + product: "Fjord Storage", + sku: "STO-660", + region: "LATAM", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 92, + status: "paid" + }, + { + id: "ORD-10330", + placedAt: "2024-05-09T04:15:00Z", + customer: "Margies Travel", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "partner", + rep: "Owen Pratt", + quantity: 1, + unitPrice: 417, + status: "paid" + }, + { + id: "ORD-10098", + placedAt: "2024-05-09T05:21:00Z", + customer: "Humongous Insurance", + product: "Borealis CRM", + sku: "CRM-210", + region: "LATAM", + channel: "partner", + rep: "Aisha Khan", + quantity: 6, + unitPrice: 907, + status: "delivered" + }, + { + id: "ORD-10377", + placedAt: "2024-05-09T06:21:00Z", + customer: "City Power & Light", + product: "Ion Messaging", + sku: "MSG-990", + region: "APAC", + channel: "partner", + rep: "Sora Tanaka", + quantity: 7, + unitPrice: 77, + status: "paid" + }, + { + id: "ORD-10239", + placedAt: "2024-05-09T07:32:00Z", + customer: "Tailspin Toys", + product: "Onyx Security", + sku: "SEC-606", + region: "APAC", + channel: "marketplace", + rep: "Diego Marin", + quantity: 1, + unitPrice: 848, + status: "paid" + }, + { + id: "ORD-10337", + placedAt: "2024-05-09T14:16:00Z", + customer: "City Power & Light", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 6, + unitPrice: 1191, + status: "shipped" + }, + { + id: "ORD-10251", + placedAt: "2024-05-09T15:22:00Z", + customer: "Lucerne Publishing", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "LATAM", + channel: "marketplace", + rep: "Priya Nair", + quantity: 5, + unitPrice: 976, + status: "shipped" + }, + { + id: "ORD-10305", + placedAt: "2024-05-09T19:38:00Z", + customer: "Margies Travel", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "direct", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 1191, + status: "shipped" + }, + { + id: "ORD-10240", + placedAt: "2024-05-09T20:34:00Z", + customer: "Contoso Ltd", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 4, + unitPrice: 301, + status: "shipped" + }, + { + id: "ORD-10090", + placedAt: "2024-05-09T23:38:00Z", + customer: "Contoso Ltd", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 424, + status: "pending" + }, + { + id: "ORD-10060", + placedAt: "2024-05-10T01:13:00Z", + customer: "Alpine Ski House", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "direct", + rep: "Dana Wills", + quantity: 8, + unitPrice: 532, + status: "delivered" + }, + { + id: "ORD-10203", + placedAt: "2024-05-10T02:13:00Z", + customer: "Northwind Traders", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "APAC", + channel: "direct", + rep: "Dana Wills", + quantity: 6, + unitPrice: 985, + status: "shipped" + }, + { + id: "ORD-10056", + placedAt: "2024-05-10T02:17:00Z", + customer: "Wingtip Toys", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 222, + status: "shipped" + }, + { + id: "ORD-10362", + placedAt: "2024-05-10T02:34:00Z", + customer: "Humongous Insurance", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "direct", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 413, + status: "shipped" + }, + { + id: "ORD-10241", + placedAt: "2024-05-10T04:29:00Z", + customer: "Margies Travel", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "North America", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 1189, + status: "pending" + }, + { + id: "ORD-10344", + placedAt: "2024-05-10T09:07:00Z", + customer: "Alpine Ski House", + product: "Helix Identity", + sku: "IDN-880", + region: "LATAM", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 1, + unitPrice: 208, + status: "delivered" + }, + { + id: "ORD-10066", + placedAt: "2024-05-10T09:13:00Z", + customer: "School of Fine Art", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 909, + status: "shipped" + }, + { + id: "ORD-10084", + placedAt: "2024-05-10T10:14:00Z", + customer: "Coho Vineyard", + product: "Delta Insights", + sku: "INS-440", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 325, + status: "delivered" + }, + { + id: "ORD-10349", + placedAt: "2024-05-10T10:19:00Z", + customer: "Fourth Coffee", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "partner", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 721, + status: "paid" + }, + { + id: "ORD-10262", + placedAt: "2024-05-10T13:05:00Z", + customer: "Fabrikam Inc", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "partner", + rep: "Sora Tanaka", + quantity: 6, + unitPrice: 90, + status: "shipped" + }, + { + id: "ORD-10376", + placedAt: "2024-05-10T14:01:00Z", + customer: "Northwind Traders", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 5, + unitPrice: 226, + status: "delivered" + }, + { + id: "ORD-10278", + placedAt: "2024-05-10T15:14:00Z", + customer: "Proseware Inc", + product: "Fjord Storage", + sku: "STO-660", + region: "APAC", + channel: "partner", + rep: "Lena Fischer", + quantity: 5, + unitPrice: 76, + status: "delivered" + }, + { + id: "ORD-10211", + placedAt: "2024-05-10T15:38:00Z", + customer: "Adventure Works", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "partner", + rep: "Sven Olsen", + quantity: 2, + unitPrice: 622, + status: "paid" + }, + { + id: "ORD-10151", + placedAt: "2024-05-10T16:46:00Z", + customer: "Alpine Ski House", + product: "Glacier Backup", + sku: "BAK-770", + region: "LATAM", + channel: "direct", + rep: "Priya Nair", + quantity: 5, + unitPrice: 56, + status: "delivered" + }, + { + id: "ORD-10063", + placedAt: "2024-05-10T19:38:00Z", + customer: "Adventure Works", + product: "Onyx Security", + sku: "SEC-606", + region: "APAC", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 872, + status: "shipped" + }, + { + id: "ORD-10347", + placedAt: "2024-05-10T21:40:00Z", + customer: "Adventure Works", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "EMEA", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 8, + unitPrice: 983, + status: "refunded" + }, + { + id: "ORD-10034", + placedAt: "2024-05-10T22:55:00Z", + customer: "City Power & Light", + product: "Borealis CRM", + sku: "CRM-210", + region: "North America", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 880, + status: "delivered" + }, + { + id: "ORD-10131", + placedAt: "2024-05-11T02:18:00Z", + customer: "Blue Yonder Airlines", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "direct", + rep: "Diego Marin", + quantity: 2, + unitPrice: 642, + status: "shipped" + }, + { + id: "ORD-10218", + placedAt: "2024-05-11T02:18:00Z", + customer: "Fourth Coffee", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "direct", + rep: "Priya Nair", + quantity: 1, + unitPrice: 393, + status: "delivered" + }, + { + id: "ORD-10361", + placedAt: "2024-05-11T02:44:00Z", + customer: "Northwind Traders", + product: "Ion Messaging", + sku: "MSG-990", + region: "APAC", + channel: "self-serve", + rep: "Sora Tanaka", + quantity: 1, + unitPrice: 43, + status: "pending" + }, + { + id: "ORD-10199", + placedAt: "2024-05-11T04:13:00Z", + customer: "Alpine Ski House", + product: "Glacier Backup", + sku: "BAK-770", + region: "EMEA", + channel: "partner", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 39, + status: "shipped" + }, + { + id: "ORD-10127", + placedAt: "2024-05-11T08:16:00Z", + customer: "Adventure Works", + product: "Onyx Security", + sku: "SEC-606", + region: "LATAM", + channel: "marketplace", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 877, + status: "refunded" + }, + { + id: "ORD-10355", + placedAt: "2024-05-11T10:52:00Z", + customer: "Humongous Insurance", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "direct", + rep: "Aisha Khan", + quantity: 1, + unitPrice: 631, + status: "delivered" + }, + { + id: "ORD-10320", + placedAt: "2024-05-11T13:30:00Z", + customer: "Northwind Traders", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 3, + unitPrice: 286, + status: "delivered" + }, + { + id: "ORD-10273", + placedAt: "2024-05-11T16:17:00Z", + customer: "Fabrikam Inc", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "North America", + channel: "self-serve", + rep: "Priya Nair", + quantity: 5, + unitPrice: 1209, + status: "shipped" + }, + { + id: "ORD-10263", + placedAt: "2024-05-11T17:44:00Z", + customer: "Fabrikam Inc", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "marketplace", + rep: "Priya Nair", + quantity: 6, + unitPrice: 30, + status: "refunded" + }, + { + id: "ORD-10172", + placedAt: "2024-05-11T18:43:00Z", + customer: "Adventure Works", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "LATAM", + channel: "marketplace", + rep: "Diego Marin", + quantity: 7, + unitPrice: 518, + status: "shipped" + }, + { + id: "ORD-10255", + placedAt: "2024-05-11T21:49:00Z", + customer: "City Power & Light", + product: "Onyx Security", + sku: "SEC-606", + region: "APAC", + channel: "direct", + rep: "Dana Wills", + quantity: 8, + unitPrice: 858, + status: "shipped" + }, + { + id: "ORD-10275", + placedAt: "2024-05-11T22:08:00Z", + customer: "Adventure Works", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "direct", + rep: "Hugo Bernard", + quantity: 8, + unitPrice: 631, + status: "delivered" + }, + { + id: "ORD-10048", + placedAt: "2024-05-12T08:34:00Z", + customer: "Trey Research", + product: "Polaris Reporting", + sku: "RPT-707", + region: "LATAM", + channel: "partner", + rep: "Sora Tanaka", + quantity: 8, + unitPrice: 276, + status: "paid" + }, + { + id: "ORD-10141", + placedAt: "2024-05-12T12:59:00Z", + customer: "Graphic Design Institute", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "direct", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 710, + status: "shipped" + }, + { + id: "ORD-10319", + placedAt: "2024-05-12T14:48:00Z", + customer: "Contoso Ltd", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "partner", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 869, + status: "paid" + }, + { + id: "ORD-10197", + placedAt: "2024-05-12T15:48:00Z", + customer: "Proseware Inc", + product: "Echo Monitoring", + sku: "MON-550", + region: "LATAM", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 157, + status: "delivered" + }, + { + id: "ORD-10281", + placedAt: "2024-05-12T19:15:00Z", + customer: "City Power & Light", + product: "Ion Messaging", + sku: "MSG-990", + region: "North America", + channel: "direct", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 64, + status: "paid" + }, + { + id: "ORD-10205", + placedAt: "2024-05-12T20:32:00Z", + customer: "Fourth Coffee", + product: "Meridian ETL", + sku: "ETL-404", + region: "EMEA", + channel: "direct", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 732, + status: "delivered" + }, + { + id: "ORD-10085", + placedAt: "2024-05-13T00:31:00Z", + customer: "Blue Yonder Airlines", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "direct", + rep: "Sora Tanaka", + quantity: 7, + unitPrice: 162, + status: "paid" + }, + { + id: "ORD-10373", + placedAt: "2024-05-13T00:58:00Z", + customer: "City Power & Light", + product: "Echo Monitoring", + sku: "MON-550", + region: "North America", + channel: "direct", + rep: "Owen Pratt", + quantity: 5, + unitPrice: 142, + status: "shipped" + }, + { + id: "ORD-10138", + placedAt: "2024-05-13T02:08:00Z", + customer: "City Power & Light", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "partner", + rep: "Sven Olsen", + quantity: 8, + unitPrice: 410, + status: "delivered" + }, + { + id: "ORD-10370", + placedAt: "2024-05-13T02:27:00Z", + customer: "City Power & Light", + product: "Borealis CRM", + sku: "CRM-210", + region: "LATAM", + channel: "direct", + rep: "Sora Tanaka", + quantity: 3, + unitPrice: 870, + status: "pending" + }, + { + id: "ORD-10005", + placedAt: "2024-05-13T05:43:00Z", + customer: "Margies Travel", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "direct", + rep: "Sven Olsen", + quantity: 5, + unitPrice: 154, + status: "shipped" + }, + { + id: "ORD-10049", + placedAt: "2024-05-13T08:32:00Z", + customer: "Trey Research", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "partner", + rep: "Sven Olsen", + quantity: 1, + unitPrice: 1211, + status: "paid" + }, + { + id: "ORD-10253", + placedAt: "2024-05-13T09:49:00Z", + customer: "Margies Travel", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "self-serve", + rep: "Aisha Khan", + quantity: 1, + unitPrice: 727, + status: "delivered" + }, + { + id: "ORD-10178", + placedAt: "2024-05-13T15:00:00Z", + customer: "Litware Inc", + product: "Borealis CRM", + sku: "CRM-210", + region: "North America", + channel: "marketplace", + rep: "Diego Marin", + quantity: 6, + unitPrice: 886, + status: "shipped" + }, + { + id: "ORD-10031", + placedAt: "2024-05-13T16:08:00Z", + customer: "Tailspin Toys", + product: "Onyx Security", + sku: "SEC-606", + region: "LATAM", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 845, + status: "delivered" + }, + { + id: "ORD-10257", + placedAt: "2024-05-13T17:06:00Z", + customer: "Tailspin Toys", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "partner", + rep: "Dana Wills", + quantity: 3, + unitPrice: 1180, + status: "paid" + }, + { + id: "ORD-10007", + placedAt: "2024-05-13T18:03:00Z", + customer: "Adventure Works", + product: "Glacier Backup", + sku: "BAK-770", + region: "North America", + channel: "partner", + rep: "Priya Nair", + quantity: 4, + unitPrice: 63, + status: "delivered" + }, + { + id: "ORD-10345", + placedAt: "2024-05-14T01:10:00Z", + customer: "Tailspin Toys", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "direct", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 68, + status: "paid" + }, + { + id: "ORD-10081", + placedAt: "2024-05-14T03:48:00Z", + customer: "Graphic Design Institute", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "partner", + rep: "Priya Nair", + quantity: 8, + unitPrice: 1199, + status: "delivered" + }, + { + id: "ORD-10333", + placedAt: "2024-05-14T05:55:00Z", + customer: "Alpine Ski House", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 8, + unitPrice: 744, + status: "paid" + }, + { + id: "ORD-10051", + placedAt: "2024-05-14T06:54:00Z", + customer: "Northwind Traders", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 2, + unitPrice: 656, + status: "delivered" + }, + { + id: "ORD-10182", + placedAt: "2024-05-14T09:17:00Z", + customer: "Contoso Ltd", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "partner", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 59, + status: "refunded" + }, + { + id: "ORD-10156", + placedAt: "2024-05-14T10:58:00Z", + customer: "Trey Research", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "EMEA", + channel: "direct", + rep: "Diego Marin", + quantity: 7, + unitPrice: 500, + status: "paid" + }, + { + id: "ORD-10032", + placedAt: "2024-05-14T14:27:00Z", + customer: "Graphic Design Institute", + product: "Polaris Reporting", + sku: "RPT-707", + region: "EMEA", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 285, + status: "cancelled" + }, + { + id: "ORD-10186", + placedAt: "2024-05-14T14:28:00Z", + customer: "Fourth Coffee", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "marketplace", + rep: "Diego Marin", + quantity: 4, + unitPrice: 399, + status: "pending" + }, + { + id: "ORD-10169", + placedAt: "2024-05-14T15:55:00Z", + customer: "Adventure Works", + product: "Ion Messaging", + sku: "MSG-990", + region: "North America", + channel: "marketplace", + rep: "Diego Marin", + quantity: 7, + unitPrice: 78, + status: "shipped" + }, + { + id: "ORD-10316", + placedAt: "2024-05-14T16:28:00Z", + customer: "Humongous Insurance", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 7, + unitPrice: 523, + status: "shipped" + }, + { + id: "ORD-10314", + placedAt: "2024-05-14T18:38:00Z", + customer: "Proseware Inc", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "partner", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 420, + status: "cancelled" + }, + { + id: "ORD-10136", + placedAt: "2024-05-14T19:22:00Z", + customer: "Alpine Ski House", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 217, + status: "paid" + }, + { + id: "ORD-10272", + placedAt: "2024-05-14T20:20:00Z", + customer: "Adventure Works", + product: "Polaris Reporting", + sku: "RPT-707", + region: "LATAM", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 1, + unitPrice: 276, + status: "pending" + }, + { + id: "ORD-10363", + placedAt: "2024-05-14T22:03:00Z", + customer: "Fabrikam Inc", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "EMEA", + channel: "direct", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 974, + status: "shipped" + }, + { + id: "ORD-10299", + placedAt: "2024-05-14T22:21:00Z", + customer: "Humongous Insurance", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "EMEA", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 995, + status: "delivered" + }, + { + id: "ORD-10202", + placedAt: "2024-05-14T23:23:00Z", + customer: "Proseware Inc", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 7, + unitPrice: 403, + status: "cancelled" + }, + { + id: "ORD-10013", + placedAt: "2024-05-14T23:34:00Z", + customer: "Fabrikam Inc", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "direct", + rep: "Diego Marin", + quantity: 1, + unitPrice: 738, + status: "pending" + }, + { + id: "ORD-10122", + placedAt: "2024-05-15T03:55:00Z", + customer: "Graphic Design Institute", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "partner", + rep: "Dana Wills", + quantity: 3, + unitPrice: 404, + status: "cancelled" + }, + { + id: "ORD-10210", + placedAt: "2024-05-15T04:01:00Z", + customer: "Fourth Coffee", + product: "Borealis CRM", + sku: "CRM-210", + region: "North America", + channel: "self-serve", + rep: "Aisha Khan", + quantity: 2, + unitPrice: 905, + status: "shipped" + }, + { + id: "ORD-10029", + placedAt: "2024-05-15T04:05:00Z", + customer: "Alpine Ski House", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 1, + unitPrice: 712, + status: "delivered" + }, + { + id: "ORD-10322", + placedAt: "2024-05-15T07:53:00Z", + customer: "Blue Yonder Airlines", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 896, + status: "delivered" + }, + { + id: "ORD-10209", + placedAt: "2024-05-15T08:38:00Z", + customer: "Adventure Works", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "North America", + channel: "partner", + rep: "Priya Nair", + quantity: 4, + unitPrice: 1208, + status: "cancelled" + }, + { + id: "ORD-10045", + placedAt: "2024-05-15T10:12:00Z", + customer: "Graphic Design Institute", + product: "Meridian ETL", + sku: "ETL-404", + region: "EMEA", + channel: "partner", + rep: "Dana Wills", + quantity: 5, + unitPrice: 734, + status: "refunded" + }, + { + id: "ORD-10306", + placedAt: "2024-05-15T18:47:00Z", + customer: "Margies Travel", + product: "Borealis CRM", + sku: "CRM-210", + region: "APAC", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 6, + unitPrice: 902, + status: "shipped" + }, + { + id: "ORD-10021", + placedAt: "2024-05-15T23:25:00Z", + customer: "Contoso Ltd", + product: "Echo Monitoring", + sku: "MON-550", + region: "LATAM", + channel: "self-serve", + rep: "Diego Marin", + quantity: 2, + unitPrice: 142, + status: "delivered" + }, + { + id: "ORD-10011", + placedAt: "2024-05-16T00:55:00Z", + customer: "Graphic Design Institute", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "LATAM", + channel: "direct", + rep: "Priya Nair", + quantity: 8, + unitPrice: 972, + status: "pending" + }, + { + id: "ORD-10334", + placedAt: "2024-05-16T00:55:00Z", + customer: "City Power & Light", + product: "Nimbus Compute", + sku: "CMP-505", + region: "North America", + channel: "direct", + rep: "Mateo Russo", + quantity: 6, + unitPrice: 1113, + status: "delivered" + }, + { + id: "ORD-10222", + placedAt: "2024-05-16T02:01:00Z", + customer: "School of Fine Art", + product: "Nimbus Compute", + sku: "CMP-505", + region: "APAC", + channel: "marketplace", + rep: "Dana Wills", + quantity: 2, + unitPrice: 1109, + status: "delivered" + }, + { + id: "ORD-10116", + placedAt: "2024-05-16T02:04:00Z", + customer: "Wide World Importers", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "self-serve", + rep: "Sora Tanaka", + quantity: 2, + unitPrice: 300, + status: "paid" + }, + { + id: "ORD-10026", + placedAt: "2024-05-16T05:17:00Z", + customer: "Proseware Inc", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 2, + unitPrice: 393, + status: "shipped" + }, + { + id: "ORD-10009", + placedAt: "2024-05-16T06:11:00Z", + customer: "Coho Vineyard", + product: "Ion Messaging", + sku: "MSG-990", + region: "EMEA", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 46, + status: "delivered" + }, + { + id: "ORD-10155", + placedAt: "2024-05-16T07:24:00Z", + customer: "Fabrikam Inc", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "APAC", + channel: "direct", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 983, + status: "delivered" + }, + { + id: "ORD-10102", + placedAt: "2024-05-16T10:39:00Z", + customer: "Lucerne Publishing", + product: "Fjord Storage", + sku: "STO-660", + region: "APAC", + channel: "direct", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 90, + status: "delivered" + }, + { + id: "ORD-10302", + placedAt: "2024-05-16T12:23:00Z", + customer: "Humongous Insurance", + product: "Nimbus Compute", + sku: "CMP-505", + region: "EMEA", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 1110, + status: "refunded" + }, + { + id: "ORD-10237", + placedAt: "2024-05-16T15:29:00Z", + customer: "Northwind Traders", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "direct", + rep: "Owen Pratt", + quantity: 6, + unitPrice: 739, + status: "paid" + }, + { + id: "ORD-10046", + placedAt: "2024-05-16T15:57:00Z", + customer: "Tailspin Toys", + product: "Nimbus Compute", + sku: "CMP-505", + region: "LATAM", + channel: "partner", + rep: "Owen Pratt", + quantity: 6, + unitPrice: 1113, + status: "paid" + }, + { + id: "ORD-10163", + placedAt: "2024-05-16T16:50:00Z", + customer: "Fourth Coffee", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "LATAM", + channel: "direct", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 635, + status: "delivered" + }, + { + id: "ORD-10196", + placedAt: "2024-05-16T17:46:00Z", + customer: "Fabrikam Inc", + product: "Delta Insights", + sku: "INS-440", + region: "North America", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 2, + unitPrice: 332, + status: "paid" + }, + { + id: "ORD-10024", + placedAt: "2024-05-16T18:20:00Z", + customer: "Litware Inc", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "partner", + rep: "Sven Olsen", + quantity: 6, + unitPrice: 229, + status: "cancelled" + }, + { + id: "ORD-10339", + placedAt: "2024-05-16T20:20:00Z", + customer: "Adventure Works", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 2, + unitPrice: 659, + status: "pending" + }, + { + id: "ORD-10082", + placedAt: "2024-05-16T23:02:00Z", + customer: "Humongous Insurance", + product: "Borealis CRM", + sku: "CRM-210", + region: "LATAM", + channel: "partner", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 890, + status: "delivered" + }, + { + id: "ORD-10274", + placedAt: "2024-05-16T23:56:00Z", + customer: "Contoso Ltd", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "marketplace", + rep: "Sven Olsen", + quantity: 8, + unitPrice: 874, + status: "pending" + }, + { + id: "ORD-10002", + placedAt: "2024-05-17T04:16:00Z", + customer: "Wide World Importers", + product: "Borealis CRM", + sku: "CRM-210", + region: "North America", + channel: "direct", + rep: "Priya Nair", + quantity: 2, + unitPrice: 908, + status: "shipped" + }, + { + id: "ORD-10170", + placedAt: "2024-05-17T05:32:00Z", + customer: "Trey Research", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "partner", + rep: "Dana Wills", + quantity: 1, + unitPrice: 402, + status: "delivered" + }, + { + id: "ORD-10086", + placedAt: "2024-05-17T05:54:00Z", + customer: "City Power & Light", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 68, + status: "cancelled" + }, + { + id: "ORD-10288", + placedAt: "2024-05-17T11:21:00Z", + customer: "Alpine Ski House", + product: "Polaris Reporting", + sku: "RPT-707", + region: "LATAM", + channel: "partner", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 298, + status: "shipped" + }, + { + id: "ORD-10106", + placedAt: "2024-05-17T13:49:00Z", + customer: "Blue Yonder Airlines", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "partner", + rep: "Sora Tanaka", + quantity: 1, + unitPrice: 410, + status: "shipped" + }, + { + id: "ORD-10201", + placedAt: "2024-05-17T14:23:00Z", + customer: "Coho Vineyard", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "marketplace", + rep: "Dana Wills", + quantity: 4, + unitPrice: 71, + status: "shipped" + }, + { + id: "ORD-10160", + placedAt: "2024-05-17T18:43:00Z", + customer: "Wide World Importers", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "direct", + rep: "Hugo Bernard", + quantity: 8, + unitPrice: 309, + status: "delivered" + }, + { + id: "ORD-10187", + placedAt: "2024-05-17T21:51:00Z", + customer: "Humongous Insurance", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "self-serve", + rep: "Dana Wills", + quantity: 3, + unitPrice: 987, + status: "delivered" + }, + { + id: "ORD-10359", + placedAt: "2024-05-17T22:44:00Z", + customer: "City Power & Light", + product: "Glacier Backup", + sku: "BAK-770", + region: "EMEA", + channel: "partner", + rep: "Hugo Bernard", + quantity: 8, + unitPrice: 25, + status: "shipped" + }, + { + id: "ORD-10078", + placedAt: "2024-05-17T23:04:00Z", + customer: "Wide World Importers", + product: "Nimbus Compute", + sku: "CMP-505", + region: "North America", + channel: "self-serve", + rep: "Diego Marin", + quantity: 1, + unitPrice: 1093, + status: "paid" + }, + { + id: "ORD-10071", + placedAt: "2024-05-18T00:32:00Z", + customer: "Alpine Ski House", + product: "Glacier Backup", + sku: "BAK-770", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 57, + status: "shipped" + }, + { + id: "ORD-10348", + placedAt: "2024-05-18T04:23:00Z", + customer: "School of Fine Art", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "LATAM", + channel: "self-serve", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 538, + status: "delivered" + }, + { + id: "ORD-10093", + placedAt: "2024-05-18T08:16:00Z", + customer: "Litware Inc", + product: "Meridian ETL", + sku: "ETL-404", + region: "EMEA", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 3, + unitPrice: 728, + status: "shipped" + }, + { + id: "ORD-10114", + placedAt: "2024-05-18T08:56:00Z", + customer: "Tailspin Toys", + product: "Borealis CRM", + sku: "CRM-210", + region: "APAC", + channel: "partner", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 893, + status: "shipped" + }, + { + id: "ORD-10019", + placedAt: "2024-05-18T09:30:00Z", + customer: "Northwind Traders", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 4, + unitPrice: 658, + status: "refunded" + }, + { + id: "ORD-10321", + placedAt: "2024-05-18T12:26:00Z", + customer: "Alpine Ski House", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "LATAM", + channel: "marketplace", + rep: "Mateo Russo", + quantity: 7, + unitPrice: 1203, + status: "delivered" + }, + { + id: "ORD-10292", + placedAt: "2024-05-18T13:07:00Z", + customer: "Northwind Traders", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "direct", + rep: "Sven Olsen", + quantity: 7, + unitPrice: 312, + status: "shipped" + }, + { + id: "ORD-10055", + placedAt: "2024-05-18T14:49:00Z", + customer: "Proseware Inc", + product: "Glacier Backup", + sku: "BAK-770", + region: "North America", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 62, + status: "delivered" + }, + { + id: "ORD-10158", + placedAt: "2024-05-18T15:39:00Z", + customer: "Northwind Traders", + product: "Nimbus Compute", + sku: "CMP-505", + region: "North America", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 7, + unitPrice: 1113, + status: "shipped" + }, + { + id: "ORD-10150", + placedAt: "2024-05-18T19:45:00Z", + customer: "City Power & Light", + product: "Fjord Storage", + sku: "STO-660", + region: "LATAM", + channel: "direct", + rep: "Sora Tanaka", + quantity: 7, + unitPrice: 68, + status: "shipped" + }, + { + id: "ORD-10331", + placedAt: "2024-05-18T20:29:00Z", + customer: "Coho Vineyard", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 978, + status: "paid" + }, + { + id: "ORD-10072", + placedAt: "2024-05-19T02:24:00Z", + customer: "Proseware Inc", + product: "Helix Identity", + sku: "IDN-880", + region: "LATAM", + channel: "partner", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 235, + status: "refunded" + }, + { + id: "ORD-10035", + placedAt: "2024-05-19T02:52:00Z", + customer: "School of Fine Art", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 1, + unitPrice: 621, + status: "delivered" + }, + { + id: "ORD-10326", + placedAt: "2024-05-19T02:58:00Z", + customer: "Alpine Ski House", + product: "Fjord Storage", + sku: "STO-660", + region: "LATAM", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 70, + status: "pending" + }, + { + id: "ORD-10145", + placedAt: "2024-05-19T03:36:00Z", + customer: "Wingtip Toys", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "LATAM", + channel: "partner", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 1200, + status: "refunded" + }, + { + id: "ORD-10276", + placedAt: "2024-05-19T04:22:00Z", + customer: "City Power & Light", + product: "Delta Insights", + sku: "INS-440", + region: "North America", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 339, + status: "cancelled" + }, + { + id: "ORD-10250", + placedAt: "2024-05-19T04:52:00Z", + customer: "School of Fine Art", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 3, + unitPrice: 416, + status: "delivered" + }, + { + id: "ORD-10264", + placedAt: "2024-05-19T07:33:00Z", + customer: "Northwind Traders", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "marketplace", + rep: "Priya Nair", + quantity: 5, + unitPrice: 213, + status: "refunded" + }, + { + id: "ORD-10130", + placedAt: "2024-05-19T11:38:00Z", + customer: "Lucerne Publishing", + product: "Borealis CRM", + sku: "CRM-210", + region: "LATAM", + channel: "self-serve", + rep: "Diego Marin", + quantity: 7, + unitPrice: 876, + status: "pending" + }, + { + id: "ORD-10181", + placedAt: "2024-05-19T12:32:00Z", + customer: "Coho Vineyard", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "direct", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 141, + status: "pending" + }, + { + id: "ORD-10228", + placedAt: "2024-05-19T13:22:00Z", + customer: "Proseware Inc", + product: "Delta Insights", + sku: "INS-440", + region: "North America", + channel: "partner", + rep: "Dana Wills", + quantity: 4, + unitPrice: 334, + status: "delivered" + }, + { + id: "ORD-10206", + placedAt: "2024-05-19T13:57:00Z", + customer: "Alpine Ski House", + product: "Nimbus Compute", + sku: "CMP-505", + region: "North America", + channel: "direct", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 1114, + status: "refunded" + }, + { + id: "ORD-10364", + placedAt: "2024-05-19T20:03:00Z", + customer: "School of Fine Art", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "partner", + rep: "Diego Marin", + quantity: 5, + unitPrice: 527, + status: "paid" + }, + { + id: "ORD-10132", + placedAt: "2024-05-19T22:57:00Z", + customer: "Fabrikam Inc", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "partner", + rep: "Lena Fischer", + quantity: 8, + unitPrice: 337, + status: "paid" + }, + { + id: "ORD-10242", + placedAt: "2024-05-20T00:28:00Z", + customer: "Alpine Ski House", + product: "Borealis CRM", + sku: "CRM-210", + region: "LATAM", + channel: "direct", + rep: "Sora Tanaka", + quantity: 8, + unitPrice: 874, + status: "shipped" + }, + { + id: "ORD-10099", + placedAt: "2024-05-20T01:51:00Z", + customer: "Contoso Ltd", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "EMEA", + channel: "self-serve", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 650, + status: "paid" + }, + { + id: "ORD-10248", + placedAt: "2024-05-20T01:53:00Z", + customer: "Trey Research", + product: "Helix Identity", + sku: "IDN-880", + region: "LATAM", + channel: "partner", + rep: "Mateo Russo", + quantity: 7, + unitPrice: 215, + status: "delivered" + }, + { + id: "ORD-10113", + placedAt: "2024-05-20T02:06:00Z", + customer: "Contoso Ltd", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "partner", + rep: "Diego Marin", + quantity: 1, + unitPrice: 1201, + status: "pending" + }, + { + id: "ORD-10287", + placedAt: "2024-05-20T02:20:00Z", + customer: "Blue Yonder Airlines", + product: "Onyx Security", + sku: "SEC-606", + region: "EMEA", + channel: "partner", + rep: "Diego Marin", + quantity: 8, + unitPrice: 864, + status: "shipped" + }, + { + id: "ORD-10350", + placedAt: "2024-05-20T06:01:00Z", + customer: "Margies Travel", + product: "Nimbus Compute", + sku: "CMP-505", + region: "APAC", + channel: "direct", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 1085, + status: "refunded" + }, + { + id: "ORD-10353", + placedAt: "2024-05-20T11:32:00Z", + customer: "Coho Vineyard", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "EMEA", + channel: "direct", + rep: "Dana Wills", + quantity: 1, + unitPrice: 1195, + status: "delivered" + }, + { + id: "ORD-10137", + placedAt: "2024-05-20T11:52:00Z", + customer: "Northwind Traders", + product: "Ion Messaging", + sku: "MSG-990", + region: "North America", + channel: "self-serve", + rep: "Dana Wills", + quantity: 7, + unitPrice: 70, + status: "pending" + }, + { + id: "ORD-10069", + placedAt: "2024-05-20T14:29:00Z", + customer: "Litware Inc", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "direct", + rep: "Sora Tanaka", + quantity: 7, + unitPrice: 165, + status: "delivered" + }, + { + id: "ORD-10015", + placedAt: "2024-05-20T14:34:00Z", + customer: "Tailspin Toys", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 868, + status: "paid" + }, + { + id: "ORD-10217", + placedAt: "2024-05-20T14:41:00Z", + customer: "Northwind Traders", + product: "Ion Messaging", + sku: "MSG-990", + region: "EMEA", + channel: "direct", + rep: "Priya Nair", + quantity: 8, + unitPrice: 62, + status: "shipped" + }, + { + id: "ORD-10214", + placedAt: "2024-05-20T22:10:00Z", + customer: "Lucerne Publishing", + product: "Fjord Storage", + sku: "STO-660", + region: "EMEA", + channel: "marketplace", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 68, + status: "shipped" + }, + { + id: "ORD-10265", + placedAt: "2024-05-20T22:31:00Z", + customer: "Proseware Inc", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "partner", + rep: "Dana Wills", + quantity: 1, + unitPrice: 58, + status: "shipped" + }, + { + id: "ORD-10195", + placedAt: "2024-05-21T02:12:00Z", + customer: "Proseware Inc", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "LATAM", + channel: "partner", + rep: "Aisha Khan", + quantity: 6, + unitPrice: 645, + status: "delivered" + }, + { + id: "ORD-10225", + placedAt: "2024-05-21T07:35:00Z", + customer: "Fabrikam Inc", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 1189, + status: "delivered" + }, + { + id: "ORD-10180", + placedAt: "2024-05-21T08:15:00Z", + customer: "Wingtip Toys", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "direct", + rep: "Owen Pratt", + quantity: 4, + unitPrice: 337, + status: "shipped" + }, + { + id: "ORD-10047", + placedAt: "2024-05-21T09:50:00Z", + customer: "Wide World Importers", + product: "Onyx Security", + sku: "SEC-606", + region: "EMEA", + channel: "direct", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 855, + status: "shipped" + }, + { + id: "ORD-10315", + placedAt: "2024-05-21T11:07:00Z", + customer: "Wide World Importers", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "LATAM", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 975, + status: "delivered" + }, + { + id: "ORD-10308", + placedAt: "2024-05-21T14:41:00Z", + customer: "Coho Vineyard", + product: "Delta Insights", + sku: "INS-440", + region: "North America", + channel: "marketplace", + rep: "Priya Nair", + quantity: 2, + unitPrice: 339, + status: "refunded" + }, + { + id: "ORD-10352", + placedAt: "2024-05-21T14:46:00Z", + customer: "Alpine Ski House", + product: "Polaris Reporting", + sku: "RPT-707", + region: "APAC", + channel: "partner", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 271, + status: "delivered" + }, + { + id: "ORD-10270", + placedAt: "2024-05-21T15:15:00Z", + customer: "Fourth Coffee", + product: "Nimbus Compute", + sku: "CMP-505", + region: "APAC", + channel: "direct", + rep: "Diego Marin", + quantity: 3, + unitPrice: 1089, + status: "cancelled" + }, + { + id: "ORD-10324", + placedAt: "2024-05-21T15:21:00Z", + customer: "Tailspin Toys", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 320, + status: "delivered" + }, + { + id: "ORD-10291", + placedAt: "2024-05-21T17:15:00Z", + customer: "Graphic Design Institute", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "APAC", + channel: "marketplace", + rep: "Priya Nair", + quantity: 8, + unitPrice: 629, + status: "delivered" + }, + { + id: "ORD-10054", + placedAt: "2024-05-21T19:36:00Z", + customer: "Wide World Importers", + product: "Fjord Storage", + sku: "STO-660", + region: "APAC", + channel: "direct", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 88, + status: "cancelled" + }, + { + id: "ORD-10227", + placedAt: "2024-05-22T00:21:00Z", + customer: "Wingtip Toys", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "self-serve", + rep: "Dana Wills", + quantity: 8, + unitPrice: 640, + status: "pending" + }, + { + id: "ORD-10213", + placedAt: "2024-05-22T04:25:00Z", + customer: "Margies Travel", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 161, + status: "shipped" + }, + { + id: "ORD-10246", + placedAt: "2024-05-22T08:25:00Z", + customer: "Tailspin Toys", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 4, + unitPrice: 64, + status: "paid" + }, + { + id: "ORD-10294", + placedAt: "2024-05-22T08:30:00Z", + customer: "Lucerne Publishing", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 87, + status: "pending" + }, + { + id: "ORD-10188", + placedAt: "2024-05-22T10:03:00Z", + customer: "Tailspin Toys", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "marketplace", + rep: "Priya Nair", + quantity: 3, + unitPrice: 536, + status: "delivered" + }, + { + id: "ORD-10075", + placedAt: "2024-05-22T10:04:00Z", + customer: "Fourth Coffee", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "EMEA", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 962, + status: "paid" + }, + { + id: "ORD-10312", + placedAt: "2024-05-22T10:13:00Z", + customer: "Trey Research", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "partner", + rep: "Sven Olsen", + quantity: 7, + unitPrice: 215, + status: "shipped" + }, + { + id: "ORD-10165", + placedAt: "2024-05-22T11:39:00Z", + customer: "Margies Travel", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "partner", + rep: "Hugo Bernard", + quantity: 1, + unitPrice: 157, + status: "delivered" + }, + { + id: "ORD-10318", + placedAt: "2024-05-22T13:57:00Z", + customer: "Margies Travel", + product: "Nimbus Compute", + sku: "CMP-505", + region: "EMEA", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 1114, + status: "cancelled" + }, + { + id: "ORD-10190", + placedAt: "2024-05-22T15:49:00Z", + customer: "Lucerne Publishing", + product: "Nimbus Compute", + sku: "CMP-505", + region: "North America", + channel: "partner", + rep: "Priya Nair", + quantity: 2, + unitPrice: 1112, + status: "cancelled" + }, + { + id: "ORD-10100", + placedAt: "2024-05-22T18:37:00Z", + customer: "School of Fine Art", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "self-serve", + rep: "Aisha Khan", + quantity: 5, + unitPrice: 334, + status: "delivered" + }, + { + id: "ORD-10010", + placedAt: "2024-05-22T23:45:00Z", + customer: "Wide World Importers", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "partner", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 417, + status: "paid" + }, + { + id: "ORD-10236", + placedAt: "2024-05-23T00:12:00Z", + customer: "Contoso Ltd", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "direct", + rep: "Diego Marin", + quantity: 5, + unitPrice: 506, + status: "delivered" + }, + { + id: "ORD-10261", + placedAt: "2024-05-23T00:43:00Z", + customer: "Tailspin Toys", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 2, + unitPrice: 139, + status: "shipped" + }, + { + id: "ORD-10309", + placedAt: "2024-05-23T01:47:00Z", + customer: "Wingtip Toys", + product: "Echo Monitoring", + sku: "MON-550", + region: "LATAM", + channel: "self-serve", + rep: "Diego Marin", + quantity: 7, + unitPrice: 150, + status: "shipped" + }, + { + id: "ORD-10105", + placedAt: "2024-05-23T01:54:00Z", + customer: "Wide World Importers", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "direct", + rep: "Lena Fischer", + quantity: 5, + unitPrice: 61, + status: "shipped" + }, + { + id: "ORD-10104", + placedAt: "2024-05-23T02:03:00Z", + customer: "Wide World Importers", + product: "Helix Identity", + sku: "IDN-880", + region: "LATAM", + channel: "partner", + rep: "Owen Pratt", + quantity: 2, + unitPrice: 201, + status: "paid" + }, + { + id: "ORD-10157", + placedAt: "2024-05-23T08:09:00Z", + customer: "Northwind Traders", + product: "Meridian ETL", + sku: "ETL-404", + region: "LATAM", + channel: "partner", + rep: "Diego Marin", + quantity: 7, + unitPrice: 723, + status: "shipped" + }, + { + id: "ORD-10208", + placedAt: "2024-05-23T08:34:00Z", + customer: "Northwind Traders", + product: "Polaris Reporting", + sku: "RPT-707", + region: "APAC", + channel: "self-serve", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 286, + status: "pending" + }, + { + id: "ORD-10096", + placedAt: "2024-05-23T09:41:00Z", + customer: "Coho Vineyard", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 2, + unitPrice: 283, + status: "paid" + }, + { + id: "ORD-10374", + placedAt: "2024-05-23T10:17:00Z", + customer: "Coho Vineyard", + product: "Fjord Storage", + sku: "STO-660", + region: "EMEA", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 76, + status: "delivered" + }, + { + id: "ORD-10249", + placedAt: "2024-05-23T12:41:00Z", + customer: "Wingtip Toys", + product: "Ion Messaging", + sku: "MSG-990", + region: "APAC", + channel: "marketplace", + rep: "Diego Marin", + quantity: 1, + unitPrice: 62, + status: "shipped" + }, + { + id: "ORD-10296", + placedAt: "2024-05-23T13:39:00Z", + customer: "Wingtip Toys", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "marketplace", + rep: "Diego Marin", + quantity: 2, + unitPrice: 222, + status: "paid" + }, + { + id: "ORD-10129", + placedAt: "2024-05-23T14:38:00Z", + customer: "Contoso Ltd", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "North America", + channel: "partner", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 1181, + status: "shipped" + }, + { + id: "ORD-10135", + placedAt: "2024-05-23T15:31:00Z", + customer: "Fabrikam Inc", + product: "Glacier Backup", + sku: "BAK-770", + region: "LATAM", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 7, + unitPrice: 32, + status: "refunded" + }, + { + id: "ORD-10154", + placedAt: "2024-05-23T22:44:00Z", + customer: "Blue Yonder Airlines", + product: "Juniper Workflow", + sku: "WFL-101", + region: "LATAM", + channel: "partner", + rep: "Dana Wills", + quantity: 1, + unitPrice: 403, + status: "shipped" + }, + { + id: "ORD-10307", + placedAt: "2024-05-23T23:25:00Z", + customer: "Fabrikam Inc", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "APAC", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 8, + unitPrice: 636, + status: "pending" + }, + { + id: "ORD-10103", + placedAt: "2024-05-23T23:51:00Z", + customer: "Alpine Ski House", + product: "Glacier Backup", + sku: "BAK-770", + region: "North America", + channel: "marketplace", + rep: "Priya Nair", + quantity: 5, + unitPrice: 44, + status: "delivered" + }, + { + id: "ORD-10198", + placedAt: "2024-05-24T00:34:00Z", + customer: "City Power & Light", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "partner", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 67, + status: "paid" + }, + { + id: "ORD-10053", + placedAt: "2024-05-24T05:36:00Z", + customer: "Proseware Inc", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "direct", + rep: "Diego Marin", + quantity: 7, + unitPrice: 155, + status: "cancelled" + }, + { + id: "ORD-10342", + placedAt: "2024-05-24T12:10:00Z", + customer: "Litware Inc", + product: "Fjord Storage", + sku: "STO-660", + region: "EMEA", + channel: "direct", + rep: "Sven Olsen", + quantity: 8, + unitPrice: 58, + status: "delivered" + }, + { + id: "ORD-10379", + placedAt: "2024-05-24T12:33:00Z", + customer: "City Power & Light", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "LATAM", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 974, + status: "delivered" + }, + { + id: "ORD-10042", + placedAt: "2024-05-24T12:49:00Z", + customer: "Northwind Traders", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "partner", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 403, + status: "shipped" + }, + { + id: "ORD-10317", + placedAt: "2024-05-24T13:49:00Z", + customer: "Fourth Coffee", + product: "Meridian ETL", + sku: "ETL-404", + region: "North America", + channel: "marketplace", + rep: "Dana Wills", + quantity: 8, + unitPrice: 713, + status: "delivered" + }, + { + id: "ORD-10357", + placedAt: "2024-05-24T15:20:00Z", + customer: "Fourth Coffee", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "direct", + rep: "Sven Olsen", + quantity: 1, + unitPrice: 134, + status: "refunded" + }, + { + id: "ORD-10092", + placedAt: "2024-05-24T15:52:00Z", + customer: "Wingtip Toys", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 1, + unitPrice: 512, + status: "delivered" + }, + { + id: "ORD-10095", + placedAt: "2024-05-24T16:06:00Z", + customer: "Humongous Insurance", + product: "Onyx Security", + sku: "SEC-606", + region: "APAC", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 5, + unitPrice: 862, + status: "pending" + }, + { + id: "ORD-10226", + placedAt: "2024-05-24T18:48:00Z", + customer: "Wingtip Toys", + product: "Borealis CRM", + sku: "CRM-210", + region: "EMEA", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 890, + status: "pending" + }, + { + id: "ORD-10003", + placedAt: "2024-05-24T22:27:00Z", + customer: "Litware Inc", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "partner", + rep: "Hugo Bernard", + quantity: 2, + unitPrice: 650, + status: "paid" + }, + { + id: "ORD-10365", + placedAt: "2024-05-25T00:27:00Z", + customer: "Blue Yonder Airlines", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 3, + unitPrice: 732, + status: "pending" + }, + { + id: "ORD-10325", + placedAt: "2024-05-25T01:59:00Z", + customer: "Wingtip Toys", + product: "Echo Monitoring", + sku: "MON-550", + region: "EMEA", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 155, + status: "refunded" + }, + { + id: "ORD-10143", + placedAt: "2024-05-25T03:36:00Z", + customer: "Lucerne Publishing", + product: "Onyx Security", + sku: "SEC-606", + region: "LATAM", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 857, + status: "delivered" + }, + { + id: "ORD-10134", + placedAt: "2024-05-25T04:25:00Z", + customer: "Humongous Insurance", + product: "Fjord Storage", + sku: "STO-660", + region: "LATAM", + channel: "self-serve", + rep: "Priya Nair", + quantity: 3, + unitPrice: 57, + status: "paid" + }, + { + id: "ORD-10301", + placedAt: "2024-05-25T04:58:00Z", + customer: "City Power & Light", + product: "Meridian ETL", + sku: "ETL-404", + region: "North America", + channel: "self-serve", + rep: "Sora Tanaka", + quantity: 2, + unitPrice: 724, + status: "refunded" + }, + { + id: "ORD-10244", + placedAt: "2024-05-25T05:20:00Z", + customer: "Proseware Inc", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "partner", + rep: "Lena Fischer", + quantity: 5, + unitPrice: 326, + status: "pending" + }, + { + id: "ORD-10061", + placedAt: "2024-05-25T16:28:00Z", + customer: "Adventure Works", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 8, + unitPrice: 720, + status: "shipped" + }, + { + id: "ORD-10087", + placedAt: "2024-05-25T16:41:00Z", + customer: "Adventure Works", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "self-serve", + rep: "Diego Marin", + quantity: 7, + unitPrice: 40, + status: "shipped" + }, + { + id: "ORD-10001", + placedAt: "2024-05-25T19:39:00Z", + customer: "Fabrikam Inc", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "North America", + channel: "self-serve", + rep: "Aisha Khan", + quantity: 7, + unitPrice: 1195, + status: "cancelled" + }, + { + id: "ORD-10030", + placedAt: "2024-05-25T21:06:00Z", + customer: "Litware Inc", + product: "Nimbus Compute", + sku: "CMP-505", + region: "EMEA", + channel: "marketplace", + rep: "Sven Olsen", + quantity: 3, + unitPrice: 1115, + status: "cancelled" + }, + { + id: "ORD-10115", + placedAt: "2024-05-25T21:27:00Z", + customer: "Lucerne Publishing", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "LATAM", + channel: "direct", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 649, + status: "refunded" + }, + { + id: "ORD-10336", + placedAt: "2024-05-25T22:58:00Z", + customer: "Tailspin Toys", + product: "Polaris Reporting", + sku: "RPT-707", + region: "APAC", + channel: "direct", + rep: "Aisha Khan", + quantity: 3, + unitPrice: 281, + status: "paid" + }, + { + id: "ORD-10216", + placedAt: "2024-05-25T23:02:00Z", + customer: "Graphic Design Institute", + product: "Helix Identity", + sku: "IDN-880", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 6, + unitPrice: 228, + status: "delivered" + }, + { + id: "ORD-10080", + placedAt: "2024-05-26T01:16:00Z", + customer: "Proseware Inc", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "partner", + rep: "Priya Nair", + quantity: 6, + unitPrice: 289, + status: "pending" + }, + { + id: "ORD-10161", + placedAt: "2024-05-26T02:41:00Z", + customer: "School of Fine Art", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "LATAM", + channel: "direct", + rep: "Owen Pratt", + quantity: 8, + unitPrice: 1185, + status: "pending" + }, + { + id: "ORD-10343", + placedAt: "2024-05-26T08:07:00Z", + customer: "Wingtip Toys", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "self-serve", + rep: "Mateo Russo", + quantity: 2, + unitPrice: 45, + status: "pending" + }, + { + id: "ORD-10077", + placedAt: "2024-05-26T08:37:00Z", + customer: "Contoso Ltd", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 5, + unitPrice: 727, + status: "delivered" + }, + { + id: "ORD-10327", + placedAt: "2024-05-26T09:58:00Z", + customer: "School of Fine Art", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "self-serve", + rep: "Priya Nair", + quantity: 3, + unitPrice: 39, + status: "pending" + }, + { + id: "ORD-10057", + placedAt: "2024-05-26T10:39:00Z", + customer: "Northwind Traders", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "partner", + rep: "Dana Wills", + quantity: 7, + unitPrice: 53, + status: "shipped" + }, + { + id: "ORD-10088", + placedAt: "2024-05-26T16:48:00Z", + customer: "Coho Vineyard", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "marketplace", + rep: "Diego Marin", + quantity: 5, + unitPrice: 218, + status: "shipped" + }, + { + id: "ORD-10286", + placedAt: "2024-05-26T21:13:00Z", + customer: "Wingtip Toys", + product: "Nimbus Compute", + sku: "CMP-505", + region: "LATAM", + channel: "direct", + rep: "Diego Marin", + quantity: 2, + unitPrice: 1092, + status: "shipped" + }, + { + id: "ORD-10140", + placedAt: "2024-05-26T22:31:00Z", + customer: "Tailspin Toys", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "North America", + channel: "direct", + rep: "Mateo Russo", + quantity: 5, + unitPrice: 520, + status: "paid" + }, + { + id: "ORD-10254", + placedAt: "2024-05-26T23:02:00Z", + customer: "Northwind Traders", + product: "Nimbus Compute", + sku: "CMP-505", + region: "EMEA", + channel: "self-serve", + rep: "Sven Olsen", + quantity: 3, + unitPrice: 1099, + status: "delivered" + }, + { + id: "ORD-10108", + placedAt: "2024-05-26T23:31:00Z", + customer: "Blue Yonder Airlines", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "APAC", + channel: "partner", + rep: "Priya Nair", + quantity: 6, + unitPrice: 504, + status: "paid" + }, + { + id: "ORD-10303", + placedAt: "2024-05-27T00:21:00Z", + customer: "Fourth Coffee", + product: "Onyx Security", + sku: "SEC-606", + region: "APAC", + channel: "marketplace", + rep: "Dana Wills", + quantity: 6, + unitPrice: 855, + status: "shipped" + }, + { + id: "ORD-10258", + placedAt: "2024-05-27T04:22:00Z", + customer: "Alpine Ski House", + product: "Borealis CRM", + sku: "CRM-210", + region: "LATAM", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 3, + unitPrice: 908, + status: "delivered" + }, + { + id: "ORD-10267", + placedAt: "2024-05-27T06:00:00Z", + customer: "Coho Vineyard", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 3, + unitPrice: 966, + status: "delivered" + }, + { + id: "ORD-10179", + placedAt: "2024-05-27T06:50:00Z", + customer: "City Power & Light", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "North America", + channel: "marketplace", + rep: "Dana Wills", + quantity: 5, + unitPrice: 644, + status: "paid" + }, + { + id: "ORD-10184", + placedAt: "2024-05-27T10:30:00Z", + customer: "Tailspin Toys", + product: "Helix Identity", + sku: "IDN-880", + region: "North America", + channel: "partner", + rep: "Diego Marin", + quantity: 5, + unitPrice: 208, + status: "pending" + }, + { + id: "ORD-10023", + placedAt: "2024-05-27T12:22:00Z", + customer: "City Power & Light", + product: "Glacier Backup", + sku: "BAK-770", + region: "APAC", + channel: "direct", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 31, + status: "pending" + }, + { + id: "ORD-10037", + placedAt: "2024-05-27T16:22:00Z", + customer: "Alpine Ski House", + product: "Echo Monitoring", + sku: "MON-550", + region: "APAC", + channel: "marketplace", + rep: "Priya Nair", + quantity: 3, + unitPrice: 141, + status: "refunded" + }, + { + id: "ORD-10247", + placedAt: "2024-05-27T22:46:00Z", + customer: "Trey Research", + product: "Glacier Backup", + sku: "BAK-770", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 7, + unitPrice: 60, + status: "delivered" + }, + { + id: "ORD-10027", + placedAt: "2024-05-27T23:51:00Z", + customer: "Fabrikam Inc", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "North America", + channel: "partner", + rep: "Sven Olsen", + quantity: 3, + unitPrice: 979, + status: "cancelled" + }, + { + id: "ORD-10064", + placedAt: "2024-05-28T02:28:00Z", + customer: "Fabrikam Inc", + product: "Polaris Reporting", + sku: "RPT-707", + region: "APAC", + channel: "self-serve", + rep: "Priya Nair", + quantity: 7, + unitPrice: 273, + status: "delivered" + }, + { + id: "ORD-10022", + placedAt: "2024-05-28T03:38:00Z", + customer: "Fourth Coffee", + product: "Fjord Storage", + sku: "STO-660", + region: "APAC", + channel: "partner", + rep: "Sora Tanaka", + quantity: 4, + unitPrice: 56, + status: "refunded" + }, + { + id: "ORD-10014", + placedAt: "2024-05-28T05:33:00Z", + customer: "Adventure Works", + product: "Nimbus Compute", + sku: "CMP-505", + region: "LATAM", + channel: "direct", + rep: "Owen Pratt", + quantity: 6, + unitPrice: 1100, + status: "paid" + }, + { + id: "ORD-10074", + placedAt: "2024-05-28T05:54:00Z", + customer: "Margies Travel", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 2, + unitPrice: 419, + status: "delivered" + }, + { + id: "ORD-10147", + placedAt: "2024-05-28T07:03:00Z", + customer: "Lucerne Publishing", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 1, + unitPrice: 652, + status: "delivered" + }, + { + id: "ORD-10371", + placedAt: "2024-05-28T07:48:00Z", + customer: "City Power & Light", + product: "Cascade Data Pipeline", + sku: "PIPE-330", + region: "APAC", + channel: "self-serve", + rep: "Dana Wills", + quantity: 8, + unitPrice: 655, + status: "shipped" + }, + { + id: "ORD-10375", + placedAt: "2024-05-28T08:26:00Z", + customer: "Fabrikam Inc", + product: "Glacier Backup", + sku: "BAK-770", + region: "EMEA", + channel: "partner", + rep: "Hugo Bernard", + quantity: 2, + unitPrice: 43, + status: "delivered" + }, + { + id: "ORD-10118", + placedAt: "2024-05-28T08:32:00Z", + customer: "Humongous Insurance", + product: "Fjord Storage", + sku: "STO-660", + region: "North America", + channel: "marketplace", + rep: "Diego Marin", + quantity: 1, + unitPrice: 81, + status: "paid" + }, + { + id: "ORD-10065", + placedAt: "2024-05-28T08:58:00Z", + customer: "Graphic Design Institute", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "EMEA", + channel: "direct", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 1203, + status: "delivered" + }, + { + id: "ORD-10289", + placedAt: "2024-05-28T10:03:00Z", + customer: "Lucerne Publishing", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "EMEA", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 1199, + status: "refunded" + }, + { + id: "ORD-10252", + placedAt: "2024-05-28T11:06:00Z", + customer: "Alpine Ski House", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "EMEA", + channel: "partner", + rep: "Priya Nair", + quantity: 8, + unitPrice: 513, + status: "shipped" + }, + { + id: "ORD-10119", + placedAt: "2024-05-28T12:50:00Z", + customer: "Lucerne Publishing", + product: "Glacier Backup", + sku: "BAK-770", + region: "North America", + channel: "marketplace", + rep: "Sora Tanaka", + quantity: 1, + unitPrice: 41, + status: "pending" + }, + { + id: "ORD-10016", + placedAt: "2024-05-28T13:48:00Z", + customer: "School of Fine Art", + product: "Polaris Reporting", + sku: "RPT-707", + region: "North America", + channel: "partner", + rep: "Lena Fischer", + quantity: 2, + unitPrice: 283, + status: "shipped" + }, + { + id: "ORD-10168", + placedAt: "2024-05-28T14:05:00Z", + customer: "Tailspin Toys", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 7, + unitPrice: 213, + status: "shipped" + }, + { + id: "ORD-10094", + placedAt: "2024-05-28T15:55:00Z", + customer: "Alpine Ski House", + product: "Nimbus Compute", + sku: "CMP-505", + region: "APAC", + channel: "self-serve", + rep: "Hugo Bernard", + quantity: 8, + unitPrice: 1086, + status: "delivered" + }, + { + id: "ORD-10036", + placedAt: "2024-05-28T19:44:00Z", + customer: "Trey Research", + product: "Delta Insights", + sku: "INS-440", + region: "LATAM", + channel: "self-serve", + rep: "Dana Wills", + quantity: 2, + unitPrice: 327, + status: "shipped" + }, + { + id: "ORD-10335", + placedAt: "2024-05-28T20:08:00Z", + customer: "Contoso Ltd", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "direct", + rep: "Diego Marin", + quantity: 6, + unitPrice: 841, + status: "delivered" + }, + { + id: "ORD-10025", + placedAt: "2024-05-28T20:19:00Z", + customer: "Proseware Inc", + product: "Ion Messaging", + sku: "MSG-990", + region: "North America", + channel: "self-serve", + rep: "Priya Nair", + quantity: 1, + unitPrice: 59, + status: "shipped" + }, + { + id: "ORD-10126", + placedAt: "2024-05-28T22:23:00Z", + customer: "Wingtip Toys", + product: "Nimbus Compute", + sku: "CMP-505", + region: "North America", + channel: "partner", + rep: "Priya Nair", + quantity: 5, + unitPrice: 1110, + status: "delivered" + }, + { + id: "ORD-10112", + placedAt: "2024-05-28T22:43:00Z", + customer: "Lucerne Publishing", + product: "Polaris Reporting", + sku: "RPT-707", + region: "EMEA", + channel: "partner", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 285, + status: "pending" + }, + { + id: "ORD-10068", + placedAt: "2024-05-28T23:49:00Z", + customer: "Fabrikam Inc", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "direct", + rep: "Owen Pratt", + quantity: 8, + unitPrice: 305, + status: "delivered" + }, + { + id: "ORD-10310", + placedAt: "2024-05-29T00:02:00Z", + customer: "Wingtip Toys", + product: "Fjord Storage", + sku: "STO-660", + region: "APAC", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 92, + status: "paid" + }, + { + id: "ORD-10232", + placedAt: "2024-05-29T02:04:00Z", + customer: "Graphic Design Institute", + product: "Helix Identity", + sku: "IDN-880", + region: "EMEA", + channel: "partner", + rep: "Hugo Bernard", + quantity: 2, + unitPrice: 200, + status: "paid" + }, + { + id: "ORD-10231", + placedAt: "2024-05-29T02:08:00Z", + customer: "Alpine Ski House", + product: "Glacier Backup", + sku: "BAK-770", + region: "EMEA", + channel: "partner", + rep: "Mateo Russo", + quantity: 2, + unitPrice: 46, + status: "delivered" + }, + { + id: "ORD-10351", + placedAt: "2024-05-29T02:55:00Z", + customer: "Margies Travel", + product: "Onyx Security", + sku: "SEC-606", + region: "LATAM", + channel: "self-serve", + rep: "Lena Fischer", + quantity: 4, + unitPrice: 876, + status: "delivered" + }, + { + id: "ORD-10185", + placedAt: "2024-05-29T04:16:00Z", + customer: "City Power & Light", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "direct", + rep: "Aisha Khan", + quantity: 6, + unitPrice: 73, + status: "paid" + }, + { + id: "ORD-10360", + placedAt: "2024-05-29T09:18:00Z", + customer: "Humongous Insurance", + product: "Helix Identity", + sku: "IDN-880", + region: "LATAM", + channel: "partner", + rep: "Sven Olsen", + quantity: 2, + unitPrice: 239, + status: "delivered" + }, + { + id: "ORD-10175", + placedAt: "2024-05-29T10:33:00Z", + customer: "Proseware Inc", + product: "Onyx Security", + sku: "SEC-606", + region: "LATAM", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 7, + unitPrice: 842, + status: "pending" + }, + { + id: "ORD-10176", + placedAt: "2024-05-29T11:19:00Z", + customer: "Northwind Traders", + product: "Polaris Reporting", + sku: "RPT-707", + region: "LATAM", + channel: "direct", + rep: "Mateo Russo", + quantity: 8, + unitPrice: 295, + status: "delivered" + }, + { + id: "ORD-10290", + placedAt: "2024-05-29T12:43:00Z", + customer: "Fabrikam Inc", + product: "Borealis CRM", + sku: "CRM-210", + region: "APAC", + channel: "direct", + rep: "Mateo Russo", + quantity: 4, + unitPrice: 890, + status: "shipped" + }, + { + id: "ORD-10040", + placedAt: "2024-05-29T12:44:00Z", + customer: "Northwind Traders", + product: "Helix Identity", + sku: "IDN-880", + region: "North America", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 226, + status: "shipped" + }, + { + id: "ORD-10191", + placedAt: "2024-05-29T13:11:00Z", + customer: "Humongous Insurance", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "marketplace", + rep: "Sven Olsen", + quantity: 2, + unitPrice: 868, + status: "refunded" + }, + { + id: "ORD-10346", + placedAt: "2024-05-29T14:13:00Z", + customer: "School of Fine Art", + product: "Juniper Workflow", + sku: "WFL-101", + region: "North America", + channel: "self-serve", + rep: "Diego Marin", + quantity: 8, + unitPrice: 429, + status: "refunded" + }, + { + id: "ORD-10110", + placedAt: "2024-05-29T14:21:00Z", + customer: "Graphic Design Institute", + product: "Nimbus Compute", + sku: "CMP-505", + region: "LATAM", + channel: "direct", + rep: "Aisha Khan", + quantity: 8, + unitPrice: 1110, + status: "shipped" + }, + { + id: "ORD-10223", + placedAt: "2024-05-29T18:50:00Z", + customer: "Blue Yonder Airlines", + product: "Onyx Security", + sku: "SEC-606", + region: "EMEA", + channel: "partner", + rep: "Priya Nair", + quantity: 1, + unitPrice: 869, + status: "refunded" + }, + { + id: "ORD-10149", + placedAt: "2024-05-29T19:52:00Z", + customer: "Lucerne Publishing", + product: "Echo Monitoring", + sku: "MON-550", + region: "LATAM", + channel: "marketplace", + rep: "Dana Wills", + quantity: 5, + unitPrice: 154, + status: "shipped" + }, + { + id: "ORD-10017", + placedAt: "2024-05-29T21:10:00Z", + customer: "Trey Research", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "partner", + rep: "Aisha Khan", + quantity: 5, + unitPrice: 1210, + status: "shipped" + }, + { + id: "ORD-10120", + placedAt: "2024-05-29T23:32:00Z", + customer: "Trey Research", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 2, + unitPrice: 210, + status: "pending" + }, + { + id: "ORD-10062", + placedAt: "2024-05-30T00:04:00Z", + customer: "Adventure Works", + product: "Nimbus Compute", + sku: "CMP-505", + region: "LATAM", + channel: "self-serve", + rep: "Dana Wills", + quantity: 2, + unitPrice: 1107, + status: "paid" + }, + { + id: "ORD-10207", + placedAt: "2024-05-30T04:36:00Z", + customer: "Litware Inc", + product: "Onyx Security", + sku: "SEC-606", + region: "North America", + channel: "marketplace", + rep: "Lena Fischer", + quantity: 3, + unitPrice: 879, + status: "shipped" + }, + { + id: "ORD-10033", + placedAt: "2024-05-30T05:10:00Z", + customer: "Fourth Coffee", + product: "Aurora Analytics Suite", + sku: "ANL-100", + region: "APAC", + channel: "self-serve", + rep: "Dana Wills", + quantity: 6, + unitPrice: 1192, + status: "delivered" + }, + { + id: "ORD-10189", + placedAt: "2024-05-30T07:29:00Z", + customer: "Fabrikam Inc", + product: "Meridian ETL", + sku: "ETL-404", + region: "EMEA", + channel: "marketplace", + rep: "Owen Pratt", + quantity: 2, + unitPrice: 746, + status: "shipped" + }, + { + id: "ORD-10266", + placedAt: "2024-05-30T09:26:00Z", + customer: "Proseware Inc", + product: "Juniper Workflow", + sku: "WFL-101", + region: "APAC", + channel: "partner", + rep: "Sora Tanaka", + quantity: 6, + unitPrice: 398, + status: "shipped" + }, + { + id: "ORD-10159", + placedAt: "2024-05-30T11:55:00Z", + customer: "City Power & Light", + product: "Onyx Security", + sku: "SEC-606", + region: "LATAM", + channel: "self-serve", + rep: "Owen Pratt", + quantity: 7, + unitPrice: 840, + status: "paid" + }, + { + id: "ORD-10008", + placedAt: "2024-05-30T13:33:00Z", + customer: "Northwind Traders", + product: "Helix Identity", + sku: "IDN-880", + region: "North America", + channel: "marketplace", + rep: "Hugo Bernard", + quantity: 6, + unitPrice: 239, + status: "pending" + }, + { + id: "ORD-10152", + placedAt: "2024-05-30T14:48:00Z", + customer: "Litware Inc", + product: "Helix Identity", + sku: "IDN-880", + region: "APAC", + channel: "marketplace", + rep: "Aisha Khan", + quantity: 5, + unitPrice: 210, + status: "delivered" + }, + { + id: "ORD-10153", + placedAt: "2024-05-30T19:31:00Z", + customer: "Fabrikam Inc", + product: "Ion Messaging", + sku: "MSG-990", + region: "LATAM", + channel: "direct", + rep: "Priya Nair", + quantity: 8, + unitPrice: 59, + status: "shipped" + }, + { + id: "ORD-10260", + placedAt: "2024-05-30T21:08:00Z", + customer: "Tailspin Toys", + product: "Delta Insights", + sku: "INS-440", + region: "APAC", + channel: "self-serve", + rep: "Dana Wills", + quantity: 6, + unitPrice: 303, + status: "paid" + }, + { + id: "ORD-10018", + placedAt: "2024-05-30T21:18:00Z", + customer: "Litware Inc", + product: "Borealis CRM", + sku: "CRM-210", + region: "APAC", + channel: "marketplace", + rep: "Dana Wills", + quantity: 3, + unitPrice: 888, + status: "refunded" + }, + { + id: "ORD-10041", + placedAt: "2024-05-31T02:23:00Z", + customer: "Contoso Ltd", + product: "Ion Messaging", + sku: "MSG-990", + region: "EMEA", + channel: "direct", + rep: "Lena Fischer", + quantity: 6, + unitPrice: 74, + status: "paid" + }, + { + id: "ORD-10284", + placedAt: "2024-05-31T03:38:00Z", + customer: "Margies Travel", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "North America", + channel: "direct", + rep: "Owen Pratt", + quantity: 5, + unitPrice: 527, + status: "refunded" + }, + { + id: "ORD-10171", + placedAt: "2024-05-31T04:26:00Z", + customer: "Northwind Traders", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "EMEA", + channel: "marketplace", + rep: "Dana Wills", + quantity: 1, + unitPrice: 981, + status: "delivered" + }, + { + id: "ORD-10268", + placedAt: "2024-05-31T13:21:00Z", + customer: "Lucerne Publishing", + product: "Lumen Dashboards", + sku: "DSH-303", + region: "LATAM", + channel: "partner", + rep: "Aisha Khan", + quantity: 7, + unitPrice: 501, + status: "paid" + }, + { + id: "ORD-10311", + placedAt: "2024-05-31T13:24:00Z", + customer: "Blue Yonder Airlines", + product: "Glacier Backup", + sku: "BAK-770", + region: "EMEA", + channel: "direct", + rep: "Aisha Khan", + quantity: 2, + unitPrice: 53, + status: "shipped" + }, + { + id: "ORD-10164", + placedAt: "2024-05-31T13:27:00Z", + customer: "Lucerne Publishing", + product: "Delta Insights", + sku: "INS-440", + region: "EMEA", + channel: "direct", + rep: "Aisha Khan", + quantity: 4, + unitPrice: 305, + status: "delivered" + }, + { + id: "ORD-10139", + placedAt: "2024-05-31T14:15:00Z", + customer: "Lucerne Publishing", + product: "Kelvin Forecasting", + sku: "FCT-202", + region: "APAC", + channel: "marketplace", + rep: "Dana Wills", + quantity: 3, + unitPrice: 976, + status: "delivered" + }, + { + id: "ORD-10192", + placedAt: "2024-05-31T17:42:00Z", + customer: "Litware Inc", + product: "Polaris Reporting", + sku: "RPT-707", + region: "APAC", + channel: "direct", + rep: "Diego Marin", + quantity: 4, + unitPrice: 290, + status: "shipped" + }, + { + id: "ORD-10125", + placedAt: "2024-05-31T20:45:00Z", + customer: "Fabrikam Inc", + product: "Meridian ETL", + sku: "ETL-404", + region: "APAC", + channel: "partner", + rep: "Mateo Russo", + quantity: 3, + unitPrice: 742, + status: "refunded" + }, +] + +// Pre-baked headline metric cards shown before/independently of order rows. +// Values here are illustrative placeholders refined by the backend runnables. +export const seedMetricCards: MetricCardData[] = [ + { id: 'revenue', label: 'Total Revenue', value: 0, unit: 'currency', delta: 0.082, hint: 'Booked revenue across paid, shipped and delivered orders' }, + { id: 'orders', label: 'Orders', value: 0, unit: 'count', delta: 0.041, hint: 'Count of revenue-bearing orders in range' }, + { id: 'aov', label: 'Avg Order Value', value: 0, unit: 'currency', delta: -0.013, hint: 'Total revenue divided by order count' }, + { id: 'units', label: 'Units Sold', value: 0, unit: 'count', delta: 0.067, hint: 'Total units across revenue-bearing orders' }, + { id: 'refunds', label: 'Refunded', value: 0, unit: 'currency', delta: -0.021, hint: 'Revenue lost to refunds in range' }, + { id: 'conversion', label: 'Conversion', value: 0.187, unit: 'percent', delta: 0.009, hint: 'Share of sessions that became orders' } +] + +export function ordersInRange(orders: Order[], from: string, to: string): Order[] { + return orders.filter((order) => { + const day = order.placedAt.slice(0, 10) + return day >= from && day <= to + }) +} + +export function ordersForRegion(orders: Order[], region: string): Order[] { + if (region === 'all') { + return orders + } + return orders.filter((order) => order.region === region) +} + diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx new file mode 100644 index 0000000000..e0b11ea465 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/index.tsx @@ -0,0 +1,164 @@ +import React, { useEffect, useMemo, useState } from 'react' +import { Sidebar, type DashboardView } from './components/Sidebar' +import { FilterBar } from './components/FilterBar' +import { MetricGrid } from './components/MetricGrid' +import { SummaryPanel } from './components/SummaryPanel' +import { RevenueChart } from './components/RevenueChart' +import { OrdersTable } from './components/OrdersTable' +import { RegionTable } from './components/RegionTable' +import { TopProducts } from './components/TopProducts' +import { EmptyState } from './components/EmptyState' +import { fetchMetrics, fetchOrders, rangeForPreset, type DateRange } from './lib/api' +import { + seedOrders, + seedMetricCards, + ordersInRange, + ordersForRegion, + type Order, + type MetricCardData +} from './data/seedData' + +const App = () => { + const [view, setView] = useState('overview') + const [preset, setPreset] = useState('30d') + const [range, setRange] = useState(rangeForPreset('30d')) + const [region, setRegion] = useState('all') + const [status, setStatus] = useState('all') + + const [metrics, setMetrics] = useState(seedMetricCards) + const [orders, setOrders] = useState(seedOrders) + const [loadingMetrics, setLoadingMetrics] = useState(true) + const [loadingOrders, setLoadingOrders] = useState(true) + const [errored, setErrored] = useState(false) + + useEffect(() => { + let cancelled = false + setLoadingMetrics(true) + fetchMetrics(range, region) + .then((result) => { + if (!cancelled) { + setMetrics(result.cards) + } + }) + .catch(() => { + if (!cancelled) { + setMetrics(seedMetricCards) + } + }) + .finally(() => { + if (!cancelled) { + setLoadingMetrics(false) + } + }) + return () => { + cancelled = true + } + }, [range, region]) + + useEffect(() => { + let cancelled = false + setLoadingOrders(true) + setErrored(false) + fetchOrders(range, region, status) + .then((result) => { + if (!cancelled) { + setOrders(result.orders) + } + }) + .catch(() => { + if (!cancelled) { + // Fall back to the bundled seed data so the dashboard still renders. + const scoped = ordersForRegion( + ordersInRange(seedOrders, range.from, range.to), + region + ).filter((order) => status === 'all' || order.status === status) + setOrders(scoped) + setErrored(true) + } + }) + .finally(() => { + if (!cancelled) { + setLoadingOrders(false) + } + }) + return () => { + cancelled = true + } + }, [range, region, status]) + + const handlePresetChange = (nextPreset: string, nextRange: DateRange) => { + setPreset(nextPreset) + setRange(nextRange) + } + + // Orders that drive the summary/chart panels — the table applies the status + // filter itself, so the panels see the same range/region scoped orders. + const scopedOrders = useMemo(() => orders, [orders]) + + const renderView = () => { + switch (view) { + case 'orders': + return + case 'regions': + return + case 'products': + return + case 'overview': + default: + return ( +
+ + +
+ + +
+ +
+ ) + } + } + + return ( +
+ +
+
+

+ Acme Inc +

+

Operations Console

+

+ Revenue, orders, and regional performance at a glance. +

+
+ +
+ {errored ? ( +
+ Showing locally bundled data — the live feed is unavailable. +
+ ) : null} + {scopedOrders.length === 0 && !loadingOrders ? ( + + ) : ( + renderView() + )} +
+
+
+ ) +} + +export default App diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts new file mode 100644 index 0000000000..8c6754b029 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/aggregations.ts @@ -0,0 +1,149 @@ +// Aggregation helpers that turn raw order/metric rows into the numbers the +// dashboard renders. These run client-side after the backend returns rows so +// the UI can re-aggregate instantly when filters change without a round trip. + +import type { Order, OrderStatus } from '../data/seedData' + +export interface RevenueSummary { + totalRevenue: number + totalOrders: number + averageOrderValue: number + unitsSold: number + refundedRevenue: number + netRevenue: number +} + +export interface StatusBreakdown { + status: OrderStatus + orders: number + revenue: number +} + +export interface RegionBreakdown { + region: string + orders: number + revenue: number +} + +export interface DailyPoint { + date: string + revenue: number + orders: number +} + +// Revenue for a single line item. An order's revenue is the unit price times +// the number of units purchased — never the unit price alone. +export function orderRevenue(order: Order): number { + return order.unitPrice +} + +// The statuses that count toward realized (booked) revenue. Refunded and +// cancelled orders are excluded from the headline revenue total. +const REVENUE_STATUSES: OrderStatus[] = ['paid', 'shipped', 'delivered'] + +export function isRevenueStatus(status: OrderStatus): boolean { + return REVENUE_STATUSES.includes(status) +} + +export function sumRevenue(orders: Order[]): number { + return orders + .filter((order) => isRevenueStatus(order.status)) + .reduce((acc, order) => acc + orderRevenue(order), 0) +} + +export function sumUnits(orders: Order[]): number { + return orders + .filter((order) => isRevenueStatus(order.status)) + .reduce((acc, order) => acc + order.quantity, 0) +} + +export function sumRefundedRevenue(orders: Order[]): number { + return orders + .filter((order) => order.status === 'refunded') + .reduce((acc, order) => acc + order.unitPrice * order.quantity, 0) +} + +export function summarizeRevenue(orders: Order[]): RevenueSummary { + const revenueOrders = orders.filter((order) => isRevenueStatus(order.status)) + const totalRevenue = sumRevenue(orders) + const unitsSold = sumUnits(orders) + const refundedRevenue = sumRefundedRevenue(orders) + const totalOrders = revenueOrders.length + return { + totalRevenue, + totalOrders, + averageOrderValue: totalOrders === 0 ? 0 : totalRevenue / totalOrders, + unitsSold, + refundedRevenue, + netRevenue: totalRevenue - refundedRevenue + } +} + +export function breakdownByStatus(orders: Order[]): StatusBreakdown[] { + const map = new Map() + for (const order of orders) { + const existing = map.get(order.status) ?? { + status: order.status, + orders: 0, + revenue: 0 + } + existing.orders += 1 + existing.revenue += order.unitPrice * order.quantity + map.set(order.status, existing) + } + return [...map.values()].sort((a, b) => b.revenue - a.revenue) +} + +export function breakdownByRegion(orders: Order[]): RegionBreakdown[] { + const map = new Map() + for (const order of orders) { + if (!isRevenueStatus(order.status)) { + continue + } + const existing = map.get(order.region) ?? { + region: order.region, + orders: 0, + revenue: 0 + } + existing.orders += 1 + existing.revenue += order.unitPrice * order.quantity + map.set(order.region, existing) + } + return [...map.values()].sort((a, b) => b.revenue - a.revenue) +} + +export function dailyRevenue(orders: Order[]): DailyPoint[] { + const map = new Map() + for (const order of orders) { + if (!isRevenueStatus(order.status)) { + continue + } + const day = order.placedAt.slice(0, 10) + const existing = map.get(day) ?? { date: day, revenue: 0, orders: 0 } + existing.revenue += order.unitPrice * order.quantity + existing.orders += 1 + map.set(day, existing) + } + return [...map.values()].sort((a, b) => a.date.localeCompare(b.date)) +} + +export function topProducts(orders: Order[], limit: number = 5): { product: string; revenue: number }[] { + const map = new Map() + for (const order of orders) { + if (!isRevenueStatus(order.status)) { + continue + } + map.set(order.product, (map.get(order.product) ?? 0) + order.unitPrice * order.quantity) + } + return [...map.entries()] + .map(([product, revenue]) => ({ product, revenue })) + .sort((a, b) => b.revenue - a.revenue) + .slice(0, limit) +} + +export function growthRatio(current: number, previous: number): number { + if (previous === 0) { + return current === 0 ? 0 : 1 + } + return (current - previous) / previous +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts new file mode 100644 index 0000000000..a02edc6c73 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/api.ts @@ -0,0 +1,79 @@ +// Thin wrappers around the app's backend runnables. Centralizing the calls +// here keeps the components free of `backend.*` plumbing and gives one place to +// normalize the request/response shapes. + +import { backend } from 'wmill' +import type { Order, MetricCardData } from '../data/seedData' + +export interface DateRange { + from: string + to: string +} + +export interface MetricsResponse { + cards: MetricCardData[] + generatedAt: string +} + +export interface OrdersResponse { + orders: Order[] + total: number +} + +export interface SummaryResponse { + totalRevenue: number + netRevenue: number + totalOrders: number + averageOrderValue: number + unitsSold: number + refundedRevenue: number + currency: string +} + +export async function fetchMetrics(range: DateRange, region: string): Promise { + return backend.loadMetrics({ from: range.from, to: range.to, region }) +} + +export async function fetchOrders( + range: DateRange, + region: string, + status: string +): Promise { + return backend.loadOrders({ + from: range.from, + to: range.to, + region, + status + }) +} + +export async function fetchSummary(range: DateRange, region: string): Promise { + return backend.computeSummary({ from: range.from, to: range.to, region }) +} + +export async function requestExport( + range: DateRange, + region: string, + format: 'csv' | 'json' +): Promise<{ url: string; rows: number }> { + return backend.exportReport({ from: range.from, to: range.to, region, format }) +} + +export function defaultRange(): DateRange { + return { from: '2024-05-01', to: '2024-05-31' } +} + +export function rangeForPreset(preset: string): DateRange { + switch (preset) { + case '7d': + return { from: '2024-05-25', to: '2024-05-31' } + case '14d': + return { from: '2024-05-18', to: '2024-05-31' } + case '30d': + return { from: '2024-05-01', to: '2024-05-31' } + case 'qtd': + return { from: '2024-04-01', to: '2024-05-31' } + default: + return defaultRange() + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts new file mode 100644 index 0000000000..99df7ef98e --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/analytics_dashboard/frontend/lib/format.ts @@ -0,0 +1,91 @@ +// Presentation-layer formatting helpers shared across the dashboard. +// Pure functions only — no React, no data fetching. + +export function formatCurrency(amount: number, currency: string = 'USD'): string { + if (!Number.isFinite(amount)) { + return '—' + } + return new Intl.NumberFormat('en-US', { + style: 'currency', + currency, + maximumFractionDigits: 0 + }).format(amount) +} + +export function formatCurrencyPrecise(amount: number, currency: string = 'USD'): string { + if (!Number.isFinite(amount)) { + return '—' + } + return new Intl.NumberFormat('en-US', { + style: 'currency', + currency, + minimumFractionDigits: 2, + maximumFractionDigits: 2 + }).format(amount) +} + +export function formatNumber(value: number): string { + if (!Number.isFinite(value)) { + return '—' + } + return new Intl.NumberFormat('en-US').format(value) +} + +export function formatCompact(value: number): string { + if (!Number.isFinite(value)) { + return '—' + } + return new Intl.NumberFormat('en-US', { + notation: 'compact', + maximumFractionDigits: 1 + }).format(value) +} + +export function formatPercent(ratio: number, digits: number = 1): string { + if (!Number.isFinite(ratio)) { + return '—' + } + return `${(ratio * 100).toFixed(digits)}%` +} + +export function formatSignedPercent(ratio: number, digits: number = 1): string { + const sign = ratio > 0 ? '+' : '' + return `${sign}${formatPercent(ratio, digits)}` +} + +export function formatDate(iso: string): string { + const date = new Date(iso) + if (Number.isNaN(date.getTime())) { + return iso + } + return date.toLocaleDateString('en-US', { + year: 'numeric', + month: 'short', + day: 'numeric' + }) +} + +export function formatDateShort(iso: string): string { + const date = new Date(iso) + if (Number.isNaN(date.getTime())) { + return iso + } + return date.toLocaleDateString('en-US', { + month: 'short', + day: 'numeric' + }) +} + +export function titleCase(value: string): string { + return value + .split(/[\s_-]+/) + .map((part) => part.charAt(0).toUpperCase() + part.slice(1).toLowerCase()) + .join(' ') +} + +export function truncate(value: string, max: number = 32): string { + if (value.length <= max) { + return value + } + return `${value.slice(0, max - 1)}…` +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json b/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json new file mode 100644 index 0000000000..8c81142fb0 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_empty.json @@ -0,0 +1,6 @@ +{ + "user": { + "username": "admin", + "is_admin": true + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json new file mode 100644 index 0000000000..236f091bc2 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_evals_folder.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["evals"], + "folders_read": ["evals"] + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json b/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json new file mode 100644 index 0000000000..7f7e237f36 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_admin_folders.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "admin", + "is_admin": true, + "folders": ["marketing", "data_engineering", "shared_utils"], + "folders_read": ["marketing", "data_engineering", "shared_utils"] + } +} diff --git a/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json b/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json new file mode 100644 index 0000000000..6320cef4d3 --- /dev/null +++ b/ai_evals/fixtures/frontend/global/initial/user_bob_nonadmin_teams.json @@ -0,0 +1,8 @@ +{ + "user": { + "username": "bob", + "is_admin": false, + "folders": ["team_a"], + "folders_read": ["team_a", "team_b"] + } +} diff --git a/ai_evals/modes/app.ts b/ai_evals/modes/app.ts index af9d7c667b..f4d0a23d13 100644 --- a/ai_evals/modes/app.ts +++ b/ai_evals/modes/app.ts @@ -48,6 +48,7 @@ export function createAppModeRunner( toolsUsed: result.toolsUsed, skillsInvoked: [], tokenUsage: result.tokenUsage, + finalContextTokens: result.finalContextTokens, }; }, validate({ evalCase, actual, initial, expected, run }) { diff --git a/ai_evals/modes/cli.ts b/ai_evals/modes/cli.ts index 3eef7d049d..3ce7cf1c16 100644 --- a/ai_evals/modes/cli.ts +++ b/ai_evals/modes/cli.ts @@ -106,6 +106,7 @@ export function createCliModeRunner( toolsUsed: run.trace.toolsUsed.map((entry) => entry.tool), skillsInvoked: run.trace.skillsInvoked, tokenUsage: run.tokenUsage ?? null, + finalContextTokens: run.finalContextTokens ?? null, }; } catch (error) { const message = error instanceof Error ? error.message : String(error); @@ -122,6 +123,7 @@ export function createCliModeRunner( toolsUsed: [], skillsInvoked: [], tokenUsage: null, + finalContextTokens: null, }; } finally { await rm(workspaceDir, { recursive: true, force: true }); diff --git a/ai_evals/modes/flow.ts b/ai_evals/modes/flow.ts index e40a495573..1b3781c02d 100644 --- a/ai_evals/modes/flow.ts +++ b/ai_evals/modes/flow.ts @@ -61,6 +61,7 @@ export function createFlowModeRunner( toolCallDetails: result.toolCallDetails, skillsInvoked: [], tokenUsage: result.tokenUsage, + finalContextTokens: result.finalContextTokens, }; }, validate({ evalCase, actual, initial, expected }) { diff --git a/ai_evals/modes/global.ts b/ai_evals/modes/global.ts index f3cbf6fd86..050a4caad9 100644 --- a/ai_evals/modes/global.ts +++ b/ai_evals/modes/global.ts @@ -1,7 +1,10 @@ -import { readFile } from "node:fs/promises"; +import { readFile, stat } from "node:fs/promises"; +import { basename } from "node:path"; +import { loadAppFixtureForEval } from "../adapters/frontend/core/app/appFixtureLoader"; import { runGlobalEval, type GlobalLiveEditorDraftFixture, + type GlobalUserFixture, } from "../adapters/frontend/core/global/globalEvalRunner"; import type { BenchmarkWorkspaceRunnables } from "../adapters/frontend/mockBackend"; import type { FrontendEvalModelConfig } from "../core/models"; @@ -13,6 +16,7 @@ import { getFrontendApiKey } from "./frontendCommon"; export interface GlobalInitialFixture { workspace?: BenchmarkWorkspaceRunnables; liveEditorDrafts?: GlobalLiveEditorDraftFixture[]; + user?: GlobalUserFixture; } export function createGlobalModeRunner( @@ -36,6 +40,7 @@ export function createGlobalModeRunner( { workspaceFixtures: initial?.workspace, liveEditorDrafts: initial?.liveEditorDrafts, + user: initial?.user, maxIterations: context.evalCase?.runtime?.maxTurns, provider: modelConfig.provider, model: modelConfig.model, @@ -54,6 +59,7 @@ export function createGlobalModeRunner( toolCallDetails: result.toolCallDetails, skillsInvoked: [], tokenUsage: result.tokenUsage, + finalContextTokens: result.finalContextTokens, }; }, validate({ evalCase, actual, expected }) { @@ -75,10 +81,33 @@ export function createGlobalModeRunner( } async function loadGlobalInitialFixture(path: string): Promise { + if ((await stat(path)).isDirectory()) { + const { initialFrontend, initialBackend, initialDatatables } = + await loadAppFixtureForEval(path); + const name = basename(path); + return { + workspace: { + apps: [ + { + path: `f/evals/global/${name}`, + summary: name, + value: { + files: initialFrontend, + runnables: initialBackend, + data: initialDatatables, + }, + }, + ], + }, + liveEditorDrafts: [], + }; + } + const parsed = JSON.parse(await readFile(path, "utf8")) as GlobalInitialFixture; return { workspace: parsed.workspace ?? {}, liveEditorDrafts: parsed.liveEditorDrafts ?? [], + user: parsed.user, }; } diff --git a/ai_evals/modes/script.ts b/ai_evals/modes/script.ts index 0c49b05d7d..0407caf208 100644 --- a/ai_evals/modes/script.ts +++ b/ai_evals/modes/script.ts @@ -52,6 +52,7 @@ export function createScriptModeRunner( toolCallDetails: result.toolCallDetails, skillsInvoked: [], tokenUsage: result.tokenUsage, + finalContextTokens: result.finalContextTokens, }; }, validate({ actual, initial, expected }) { diff --git a/backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json b/backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json new file mode 100644 index 0000000000..abae0ba456 --- /dev/null +++ b/backend/.sqlx/query-002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n COUNT(*)::bigint AS \"total!\",\n COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS \"replacing!\"\n FROM ai_skill\n WHERE workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "total!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "replacing!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "TextArray" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "002a606e71364b0581dbc496bf4337f276861dc71d2e277a7aef711543eb14d7" +} diff --git a/backend/.sqlx/query-0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941.json b/backend/.sqlx/query-0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941.json new file mode 100644 index 0000000000..cde17acf76 --- /dev/null +++ b/backend/.sqlx/query-0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '10 minutes'\n RETURNING 1\n ) SELECT count(*) FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "0035bf99ce6fc00c7338bebfeb7e79bb9e7bc3d216b84279dee0018603965941" +} diff --git a/backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json b/backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json deleted file mode 100644 index 461d1afb14..0000000000 --- a/backend/.sqlx/query-0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328.json +++ /dev/null @@ -1,94 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled, NULL::text as workspace_id FROM password WHERE email = $1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "email", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "login_type", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "super_admin", - "type_info": "Bool" - }, - { - "ordinal": 3, - "name": "devops", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "verified", - "type_info": "Bool" - }, - { - "ordinal": 5, - "name": "name", - "type_info": "Varchar" - }, - { - "ordinal": 6, - "name": "company", - "type_info": "Varchar" - }, - { - "ordinal": 7, - "name": "username", - "type_info": "Varchar" - }, - { - "ordinal": 8, - "name": "operator_only", - "type_info": "Bool" - }, - { - "ordinal": 9, - "name": "first_time_user", - "type_info": "Bool" - }, - { - "ordinal": 10, - "name": "role_source", - "type_info": "Varchar" - }, - { - "ordinal": 11, - "name": "disabled", - "type_info": "Bool" - }, - { - "ordinal": 12, - "name": "workspace_id", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - false, - null, - false, - false, - false, - true, - true, - true, - null, - false, - false, - false, - null - ] - }, - "hash": "0142d9dc9c1b57487dd5709a0376794f18d33e5bd6340c0189be7818cda64328" -} diff --git a/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json b/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json new file mode 100644 index 0000000000..a0d8b66a26 --- /dev/null +++ b/backend/.sqlx/query-01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd.json @@ -0,0 +1,33 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE materialized_asset_schema\n SET snapshot_id = $5, job_id = $6, captured_at = now()\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n AND version = $4", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text", + "Int8", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "01732ca02b1888145c48c4e51e5b5829657224a743af9c0b2d5a140ad70e13dd" +} diff --git a/backend/.sqlx/query-03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72.json b/backend/.sqlx/query-03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72.json new file mode 100644 index 0000000000..71f0c1e2d8 --- /dev/null +++ b/backend/.sqlx/query-03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n FROM flow\n WHERE path = $2 AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "03547bf921bbd4342dc8604277057336378f9e85baf14d1dc34ed7e9b42e5e72" +} diff --git a/backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json b/backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json new file mode 100644 index 0000000000..f3bc714a3e --- /dev/null +++ b/backend/.sqlx/query-03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "03c8a797ae734ff76e227259ae011ef6d35f892fe95448c58ec13dea58eee3fa" +} diff --git a/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json b/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json new file mode 100644 index 0000000000..173fe9d05c --- /dev/null +++ b/backend/.sqlx/query-074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT SUM(pg_database_size(datname))::BIGINT AS \"v!\" FROM pg_database", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "074dd26f6427f4ff97e92c35163ad042144e656adcca56a4936a2eb196d3f48c" +} diff --git a/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json b/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json index 79ef0c0a81..a779aa0e95 100644 --- a/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json +++ b/backend/.sqlx/query-089d7bc7acdbb97cf477159e111bc7e9ee85289ff5c52af43166928337c257e7.json @@ -34,7 +34,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0.json b/backend/.sqlx/query-09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0.json new file mode 100644 index 0000000000..d114d21bf0 --- /dev/null +++ b/backend/.sqlx/query-09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job WHERE workspace_id = $1 AND trigger_kind = 'asset'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "09095af7cad650fb10781d9e39b0dad250c59ed0fca6cab7b5be4ee2516275d0" +} diff --git a/backend/.sqlx/query-0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034.json b/backend/.sqlx/query-0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034.json deleted file mode 100644 index 2dcdba586b..0000000000 --- a/backend/.sqlx/query-0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Int8", - "Int8", - "Text" - ] - }, - "nullable": [] - }, - "hash": "0ca770234f3e38be3fb1c280d82e9c06440168806fe605e38808bdcb400d4034" -} diff --git a/backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json b/backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json new file mode 100644 index 0000000000..ee50840b07 --- /dev/null +++ b/backend/.sqlx/query-0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "?column?", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "UuidArray", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "0d6c3399bc637e2c599534c2cafacf623cd6e5fde2d075a4f1f0d4852efcedca" +} diff --git a/backend/.sqlx/query-0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f.json b/backend/.sqlx/query-0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f.json new file mode 100644 index 0000000000..e1386cf4d6 --- /dev/null +++ b/backend/.sqlx/query-0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO join_pending_inputs\n (workspace_id, subscriber_path, partition, trigger_ref)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "0e7fe0e1d7aa2072a3431d081080bbc18da7e1ed758cab017fba2598c9467b7f" +} diff --git a/backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json b/backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json deleted file mode 100644 index d2c85b0e53..0000000000 --- a/backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json +++ /dev/null @@ -1,89 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT $1 OFFSET $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "email", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "login_type", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "verified", - "type_info": "Bool" - }, - { - "ordinal": 3, - "name": "super_admin", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "devops", - "type_info": "Bool" - }, - { - "ordinal": 5, - "name": "name", - "type_info": "Varchar" - }, - { - "ordinal": 6, - "name": "company", - "type_info": "Varchar" - }, - { - "ordinal": 7, - "name": "username", - "type_info": "Varchar" - }, - { - "ordinal": 8, - "name": "operator_only", - "type_info": "Bool" - }, - { - "ordinal": 9, - "name": "first_time_user", - "type_info": "Bool" - }, - { - "ordinal": 10, - "name": "role_source", - "type_info": "Varchar" - }, - { - "ordinal": 11, - "name": "disabled", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8" - ] - }, - "nullable": [ - false, - null, - false, - false, - false, - true, - true, - true, - null, - false, - false, - false - ] - }, - "hash": "115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc" -} diff --git a/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json new file mode 100644 index 0000000000..cef272c219 --- /dev/null +++ b/backend/.sqlx/query-11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT language AS \"language!: _\", COUNT(*)::BIGINT AS \"count!\"\n FROM script\n WHERE archived = false AND deleted = false AND kind = 'script'\n AND (auto_kind IS NULL OR auto_kind <> 'wac')\n GROUP BY language\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "language!: _", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null + ] + }, + "hash": "11813108dbf6b104eba968c3609c74ac5a589542d3e76b1e756a82fb19d49ee8" +} diff --git a/backend/.sqlx/query-126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096.json b/backend/.sqlx/query-126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096.json new file mode 100644 index 0000000000..d59b38203d --- /dev/null +++ b/backend/.sqlx/query-126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) FROM workspace_diff\n WHERE source_workspace_id = 'wm-fork-test-workspace'\n OR fork_workspace_id = 'wm-fork-test-workspace'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "126cf6d54f8d2c916cd799f6663892119a94d66637062d1bf5a5fe97d89f8096" +} diff --git a/backend/.sqlx/query-16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37.json b/backend/.sqlx/query-16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37.json deleted file mode 100644 index 023eaaa010..0000000000 --- a/backend/.sqlx/query-16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37.json +++ /dev/null @@ -1,95 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT email as \"email!\", login_type::text, verified as \"verified!\", super_admin as \"super_admin!\", devops as \"devops!\", name, company, username, NULL::bool as operator_only, first_time_user as \"first_time_user!\", role_source as \"role_source!\", disabled as \"disabled!\", NULL::text as workspace_id FROM password\n UNION ALL\n SELECT email as \"email!\", 'service_account'::text as login_type, true as \"verified!\", false as \"super_admin!\", false as \"devops!\", NULL::text as name, NULL::text as company, username, true as operator_only, false as \"first_time_user!\", 'service_account'::text as \"role_source!\", disabled as \"disabled!\", workspace_id\n FROM usr\n WHERE is_service_account IS true\n ORDER BY \"super_admin!\" DESC, \"devops!\" DESC, \"email!\"\n LIMIT $1 OFFSET $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "email!", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "login_type", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "verified!", - "type_info": "Bool" - }, - { - "ordinal": 3, - "name": "super_admin!", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "devops!", - "type_info": "Bool" - }, - { - "ordinal": 5, - "name": "name", - "type_info": "Varchar" - }, - { - "ordinal": 6, - "name": "company", - "type_info": "Varchar" - }, - { - "ordinal": 7, - "name": "username", - "type_info": "Varchar" - }, - { - "ordinal": 8, - "name": "operator_only", - "type_info": "Bool" - }, - { - "ordinal": 9, - "name": "first_time_user!", - "type_info": "Bool" - }, - { - "ordinal": 10, - "name": "role_source!", - "type_info": "Varchar" - }, - { - "ordinal": 11, - "name": "disabled!", - "type_info": "Bool" - }, - { - "ordinal": 12, - "name": "workspace_id", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8" - ] - }, - "nullable": [ - null, - null, - null, - null, - null, - null, - null, - null, - null, - null, - null, - null, - null - ] - }, - "hash": "16b4496c21d0619dab4521dca22e5fe144c59156a8f06d8592291684c49b2f37" -} diff --git a/backend/.sqlx/query-1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f.json b/backend/.sqlx/query-1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f.json new file mode 100644 index 0000000000..d67ddf711b --- /dev/null +++ b/backend/.sqlx/query-1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path)\n path,\n value as \"value!: sqlx::types::Json>\",\n created_at,\n typ::text as \"typ!\"\n FROM draft\n WHERE workspace_id = $1\n AND typ IN ('app', 'raw_app')\n AND (email = $2 OR email IS NULL)\n AND NOT EXISTS (\n SELECT 1 FROM app a\n WHERE a.workspace_id = draft.workspace_id\n AND a.path = draft.path\n )\n ORDER BY path, (email IS NULL), created_at DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value!: sqlx::types::Json>", + "type_info": "Json" + }, + { + "ordinal": 2, + "name": "created_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 3, + "name": "typ!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + null + ] + }, + "hash": "1860c102a5309a8e1bb7288d0616f48e4330af7fbe61426e999c6073f630d88f" +} diff --git a/backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json b/backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json index 8944a6001f..dfc8540468 100644 --- a/backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json +++ b/backend/.sqlx/query-19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319.json @@ -5,7 +5,7 @@ "columns": [ { "ordinal": 0, - "name": "id", + "name": "id!", "type_info": "Uuid" } ], @@ -16,7 +16,7 @@ ] }, "nullable": [ - false + null ] }, "hash": "19513c4158267cc7fe10d999ad571052c112e6bbb3cf834f16176cbb7e1ac319" diff --git a/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json b/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json new file mode 100644 index 0000000000..e576cf8fae --- /dev/null +++ b/backend/.sqlx/query-19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT kind::text as \"kind!\", parent_job, runnable_path\n FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "kind!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "parent_job", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + null, + true, + true + ] + }, + "hash": "19f1cb1c7a1974920549917a6392ff0d56f31ca5662062f4a71fed0ccf859cc4" +} diff --git a/backend/.sqlx/query-1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d.json b/backend/.sqlx/query-1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d.json new file mode 100644 index 0000000000..381a2b8ae4 --- /dev/null +++ b/backend/.sqlx/query-1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d.json @@ -0,0 +1,27 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3\n RETURNING usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n WHERE $3 = 'script'\n AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "asset_usage_kind", + "kind": { + "Enum": [ + "script", + "flow", + "job" + ] + } + } + } + ] + }, + "nullable": [] + }, + "hash": "1acfeed9c7a5b1e3d2da262d338655dba6e43067a9912cc2b775830856390c5d" +} diff --git a/backend/.sqlx/query-1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043.json b/backend/.sqlx/query-1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043.json new file mode 100644 index 0000000000..152a88d082 --- /dev/null +++ b/backend/.sqlx/query-1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script'\n AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)\n RETURNING usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "1f375b37ff9f6f01972e284e84a7b2f9d2d323a3da55f20ff6671e8eba510043" +} diff --git a/backend/.sqlx/query-1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2.json b/backend/.sqlx/query-1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2.json new file mode 100644 index 0000000000..900cbdae4d --- /dev/null +++ b/backend/.sqlx/query-1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO draft (workspace_id, path, typ, value, created_at, email)\n SELECT $2, path, typ, value, created_at, email\n FROM draft\n WHERE workspace_id = $1 AND (email = $3 OR email IS NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "1f486036b2902a0ffa0ce15f665d82c85129e2407819477adf7e87f097276ae2" +} diff --git a/backend/.sqlx/query-1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650.json b/backend/.sqlx/query-1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650.json new file mode 100644 index 0000000000..9d727dfc6c --- /dev/null +++ b/backend/.sqlx/query-1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, args, created_by,\n permissioned_as, permissioned_as_email, tag, script_lang)\n VALUES ($1, $2, 'script'::job_kind, $3, $4, 'test-user',\n 'u/test-user', 'test@windmill.dev', 'deno', 'bash'::script_lang)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Varchar", + "Varchar", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "1fc04d31ae69dbb1df9c63cb69e83e8f8e6770b78f6ed052b99afed6cea28650" +} diff --git a/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json new file mode 100644 index 0000000000..e3f72d9c3a --- /dev/null +++ b/backend/.sqlx/query-231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT version, columns AS \"columns: Json>\"\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC\n LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "columns: Json>", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "231475dc825518aa88f562698f8061d2562c3ac4af8d088f9eea9f0bc11d5fe7" +} diff --git a/backend/.sqlx/query-2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366.json b/backend/.sqlx/query-2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366.json new file mode 100644 index 0000000000..e350cc8233 --- /dev/null +++ b/backend/.sqlx/query-2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT kind, path, script_path, is_flow FROM (\n SELECT 'schedule' AS kind, path, script_path, is_flow FROM schedule\n WHERE workspace_id = $1\n AND script_path IS NOT NULL\n UNION ALL\n SELECT 'email', path, script_path, is_flow FROM email_trigger\n WHERE workspace_id = $1\n UNION ALL\n SELECT 'kafka', path, script_path, is_flow FROM kafka_trigger\n WHERE workspace_id = $1\n UNION ALL\n SELECT 'mqtt', path, script_path, is_flow FROM mqtt_trigger\n WHERE workspace_id = $1\n UNION ALL\n SELECT 'nats', path, script_path, is_flow FROM nats_trigger\n WHERE workspace_id = $1\n UNION ALL\n SELECT 'postgres', path, script_path, is_flow FROM postgres_trigger\n WHERE workspace_id = $1\n UNION ALL\n SELECT 'sqs', path, script_path, is_flow FROM sqs_trigger\n WHERE workspace_id = $1\n UNION ALL\n SELECT 'gcp', path, script_path, is_flow FROM gcp_trigger\n WHERE workspace_id = $1\n ) t\n WHERE ($2::text IS NULL OR script_path LIKE $2)\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "kind", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "script_path", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "is_flow", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "2484323d94f249be30f4472ece89659c1e6a24d5454a691e31e0179f58c24366" +} diff --git a/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json b/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json new file mode 100644 index 0000000000..3e617e3f0c --- /dev/null +++ b/backend/.sqlx/query-255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "255310c81beab0bc4cff13e966c84af7833264f208a94d20a5c8f216040c43cd" +} diff --git a/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json new file mode 100644 index 0000000000..36ea3d7fb2 --- /dev/null +++ b/backend/.sqlx/query-25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_perms\n WHERE ctid IN (\n SELECT jp.ctid FROM job_perms jp\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id)\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "25ecae25ebc03d6296b0e72482a9201c2bffb0f0f7419b0b598b2786bdb326ab" +} diff --git a/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json b/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json new file mode 100644 index 0000000000..02a419de06 --- /dev/null +++ b/backend/.sqlx/query-26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900.json @@ -0,0 +1,97 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n subscriber_path AS \"subscriber_path!\",\n asset_kind AS \"asset_kind!: windmill_common::assets::AssetKind\",\n asset_path AS \"asset_path!\",\n outcome::text AS \"outcome!\",\n child_job_id,\n partition,\n received_inputs,\n required_inputs,\n debounce_s,\n reason,\n created_at AS \"created_at!\"\n FROM dispatch_event\n WHERE producer_job_id = $1 AND workspace_id = $2\n ORDER BY id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "subscriber_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "asset_kind!: windmill_common::assets::AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 2, + "name": "asset_path!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "outcome!", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "child_job_id", + "type_info": "Uuid" + }, + { + "ordinal": 5, + "name": "partition", + "type_info": "Text" + }, + { + "ordinal": 6, + "name": "received_inputs", + "type_info": "Int4" + }, + { + "ordinal": 7, + "name": "required_inputs", + "type_info": "Int4" + }, + { + "ordinal": 8, + "name": "debounce_s", + "type_info": "Int4" + }, + { + "ordinal": 9, + "name": "reason", + "type_info": "Text" + }, + { + "ordinal": 10, + "name": "created_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + false, + false, + false, + null, + true, + true, + true, + true, + true, + true, + false + ] + }, + "hash": "26e63135fcd8e7d48e25de190a2f72ece70ec92c5b04baad2622639850445900" +} diff --git a/backend/.sqlx/query-825ca00bd011b220f47da175d1d6e0783acf9bdc1a6e058060bd4a1703f747c3.json b/backend/.sqlx/query-26fb727ef7d50c93c1c8ce33356ef6feafc448e70c30ba7e7c835f5e3167f3a5.json similarity index 79% rename from backend/.sqlx/query-825ca00bd011b220f47da175d1d6e0783acf9bdc1a6e058060bd4a1703f747c3.json rename to backend/.sqlx/query-26fb727ef7d50c93c1c8ce33356ef6feafc448e70c30ba7e7c835f5e3167f3a5.json index 876bc39b48..2d22e40cc7 100644 --- a/backend/.sqlx/query-825ca00bd011b220f47da175d1d6e0783acf9bdc1a6e058060bd4a1703f747c3.json +++ b/backend/.sqlx/query-26fb727ef7d50c93c1c8ce33356ef6feafc448e70c30ba7e7c835f5e3167f3a5.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT id, workspace_id, path, summary, policy, versions, extra_perms, draft_only, custom_path\n FROM app\n WHERE workspace_id = $1", + "query": "SELECT id, workspace_id, path, summary, policy, versions, extra_perms, custom_path\n FROM app\n WHERE workspace_id = $1", "describe": { "columns": [ { @@ -40,11 +40,6 @@ }, { "ordinal": 7, - "name": "draft_only", - "type_info": "Bool" - }, - { - "ordinal": 8, "name": "custom_path", "type_info": "Text" } @@ -62,9 +57,8 @@ false, false, false, - true, true ] }, - "hash": "825ca00bd011b220f47da175d1d6e0783acf9bdc1a6e058060bd4a1703f747c3" + "hash": "26fb727ef7d50c93c1c8ce33356ef6feafc448e70c30ba7e7c835f5e3167f3a5" } diff --git a/backend/.sqlx/query-27b0c827467cc92979f094620957bc0edfa295d6c2292e509a5536765d120bd8.json b/backend/.sqlx/query-27b0c827467cc92979f094620957bc0edfa295d6c2292e509a5536765d120bd8.json deleted file mode 100644 index 6d127f4c83..0000000000 --- a/backend/.sqlx/query-27b0c827467cc92979f094620957bc0edfa295d6c2292e509a5536765d120bd8.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT draft_only FROM app WHERE path = $1 AND workspace_id = $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "draft_only", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - true - ] - }, - "hash": "27b0c827467cc92979f094620957bc0edfa295d6c2292e509a5536765d120bd8" -} diff --git a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json similarity index 59% rename from backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json rename to backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json index 0769d083d6..4e5f9f6ed7 100644 --- a/backend/.sqlx/query-e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5.json +++ b/backend/.sqlx/query-282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf.json @@ -1,12 +1,12 @@ { "db_name": "PostgreSQL", - "query": "SELECT 1", + "query": "SELECT NOT pg_is_in_recovery()", "describe": { "columns": [ { "ordinal": 0, "name": "?column?", - "type_info": "Int4" + "type_info": "Bool" } ], "parameters": { @@ -16,5 +16,5 @@ null ] }, - "hash": "e004ebd5b5532a4b85984a62f8ad48a81aa3460c1ca07701f386135d72cdecf5" + "hash": "282b56cfb8504312ac586cd4c1f3f914cf1651c08b8edd1b06d9a6454ed779bf" } diff --git a/backend/.sqlx/query-285c136fc92ce63417e4c65e657d914a6e158d636b854248e4028ed35326f3c6.json b/backend/.sqlx/query-285c136fc92ce63417e4c65e657d914a6e158d636b854248e4028ed35326f3c6.json deleted file mode 100644 index 8b353e43e1..0000000000 --- a/backend/.sqlx/query-285c136fc92ce63417e4c65e657d914a6e158d636b854248e4028ed35326f3c6.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT\n path\n FROM\n flow_version\n WHERE\n id = $1 AND\n workspace_id = $2\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "path", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Int8", - "Text" - ] - }, - "nullable": [ - false - ] - }, - "hash": "285c136fc92ce63417e4c65e657d914a6e158d636b854248e4028ed35326f3c6" -} diff --git a/backend/.sqlx/query-28f1ecca40c8b81cc59dffb75e2913c889b374999ece04173b2e67dc74005f60.json b/backend/.sqlx/query-28f1ecca40c8b81cc59dffb75e2913c889b374999ece04173b2e67dc74005f60.json deleted file mode 100644 index a5c5e9427e..0000000000 --- a/backend/.sqlx/query-28f1ecca40c8b81cc59dffb75e2913c889b374999ece04173b2e67dc74005f60.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT draft_only FROM flow WHERE path = $1 AND workspace_id = $2 AND archived = false", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "draft_only", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - true - ] - }, - "hash": "28f1ecca40c8b81cc59dffb75e2913c889b374999ece04173b2e67dc74005f60" -} diff --git a/backend/.sqlx/query-299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4.json b/backend/.sqlx/query-299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4.json new file mode 100644 index 0000000000..642723decf --- /dev/null +++ b/backend/.sqlx/query-299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '10 minutes'\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "299b94a7972443267dd664c178a1704d195a7fc0d4e66e1014a18398e3a294f4" +} diff --git a/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json new file mode 100644 index 0000000000..f940f25b4a --- /dev/null +++ b/backend/.sqlx/query-2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb.json @@ -0,0 +1,62 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT version, columns AS \"columns: Json>\",\n snapshot_id, job_id, captured_at\n FROM materialized_asset_schema\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY version DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "columns: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "snapshot_id", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 4, + "name": "captured_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false, + true, + true, + false + ] + }, + "hash": "2a28f13c1f06a77bc9a164393cccf0d8d2a3d7e8552d8ad09512d2d6bf27c3fb" +} diff --git a/backend/.sqlx/query-2b5fc0500beb2f4c7cf5997f9aea48f77e2abe4523180c507a9a90570127be6d.json b/backend/.sqlx/query-2b5fc0500beb2f4c7cf5997f9aea48f77e2abe4523180c507a9a90570127be6d.json deleted file mode 100644 index fd205d6750..0000000000 --- a/backend/.sqlx/query-2b5fc0500beb2f4c7cf5997f9aea48f77e2abe4523180c507a9a90570127be6d.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT label, token_prefix, expiration, created_at, last_used_at, scopes, workspace_id FROM token WHERE email = $1\n ORDER BY created_at DESC LIMIT $2 OFFSET $3", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "label", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "token_prefix", - "type_info": "Varchar" - }, - { - "ordinal": 2, - "name": "expiration", - "type_info": "Timestamptz" - }, - { - "ordinal": 3, - "name": "created_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 4, - "name": "last_used_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 5, - "name": "scopes", - "type_info": "TextArray" - }, - { - "ordinal": 6, - "name": "workspace_id", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Text", - "Int8", - "Int8" - ] - }, - "nullable": [ - true, - false, - true, - false, - false, - true, - true - ] - }, - "hash": "2b5fc0500beb2f4c7cf5997f9aea48f77e2abe4523180c507a9a90570127be6d" -} diff --git a/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json b/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json new file mode 100644 index 0000000000..241778d359 --- /dev/null +++ b/backend/.sqlx/query-2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e.json @@ -0,0 +1,61 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft\n WHERE workspace_id = $1\n AND email IS NOT DISTINCT FROM (CASE WHEN $7::bool THEN NULL::text ELSE $2 END)\n AND path = $3\n AND typ = $4\n AND ($6::bool = true\n OR $5::timestamptz IS NULL\n OR created_at <= $5::timestamptz)\n RETURNING now() as \"now!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "now!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Timestamptz", + "Bool", + "Bool" + ] + }, + "nullable": [ + null + ] + }, + "hash": "2bed492ef32edf36e60e8a03268fa25bfb67dd641153d1ca23f7d0d2ae73624e" +} diff --git a/backend/.sqlx/query-311de4a5d2fb3066dc9e49693b9a1dd8e8e4a09200768a73c844810975741894.json b/backend/.sqlx/query-311de4a5d2fb3066dc9e49693b9a1dd8e8e4a09200768a73c844810975741894.json deleted file mode 100644 index 2b5e787553..0000000000 --- a/backend/.sqlx/query-311de4a5d2fb3066dc9e49693b9a1dd8e8e4a09200768a73c844810975741894.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT\n path\n FROM\n flow_version\n WHERE\n id = $1 AND\n workspace_id = $2\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "path", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Int8", - "Text" - ] - }, - "nullable": [ - false - ] - }, - "hash": "311de4a5d2fb3066dc9e49693b9a1dd8e8e4a09200768a73c844810975741894" -} diff --git a/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json b/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json new file mode 100644 index 0000000000..ded7b65a83 --- /dev/null +++ b/backend/.sqlx/query-31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "\n DELETE FROM asset\n WHERE id IN (\n SELECT id FROM (\n SELECT a.id, ROW_NUMBER() OVER (\n PARTITION BY a.workspace_id, a.path, a.kind\n ORDER BY a.created_at DESC\n ) as rn,\n limits.max_n\n FROM asset a\n INNER JOIN (\n SELECT * FROM UNNEST(\n $1::varchar[],\n $2::varchar[],\n $3::asset_kind[],\n $4::int[]\n ) AS t(workspace_id, path, kind, max_n)\n ) limits\n ON a.workspace_id = limits.workspace_id\n AND a.path = limits.path\n AND a.kind = limits.kind\n WHERE a.usage_kind = 'job'\n ) ranked\n WHERE rn > max_n\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "VarcharArray", + "VarcharArray", + { + "Custom": { + "name": "asset_kind[]", + "kind": { + "Array": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + } + } + }, + "Int4Array" + ] + }, + "nullable": [] + }, + "hash": "31bbd03932912df069cfc97fd1ca8c69a3151266e2bf475da10feb4916e436e9" +} diff --git a/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json b/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json new file mode 100644 index 0000000000..b475d068d5 --- /dev/null +++ b/backend/.sqlx/query-3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "3419f7b1ec6dad074f1688cc790a42db9eb56c32047ab28457e04776c1bff76a" +} diff --git a/backend/.sqlx/query-353c6a648720e118c0d82ef055c60033f1969fb39155e7e07bb6730505e254b7.json b/backend/.sqlx/query-353c6a648720e118c0d82ef055c60033f1969fb39155e7e07bb6730505e254b7.json new file mode 100644 index 0000000000..46fd1d2ae2 --- /dev/null +++ b/backend/.sqlx/query-353c6a648720e118c0d82ef055c60033f1969fb39155e7e07bb6730505e254b7.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) FROM skip_workspace_diff_tally WHERE workspace_id = 'wm-fork-test-workspace'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "353c6a648720e118c0d82ef055c60033f1969fb39155e7e07bb6730505e254b7" +} diff --git a/backend/.sqlx/query-355e3976f0a7e3ccbb24314f951ff37dcc4c34f800ff4f385ec9f60933f83efe.json b/backend/.sqlx/query-355e3976f0a7e3ccbb24314f951ff37dcc4c34f800ff4f385ec9f60933f83efe.json new file mode 100644 index 0000000000..66809be416 --- /dev/null +++ b/backend/.sqlx/query-355e3976f0a7e3ccbb24314f951ff37dcc4c34f800ff4f385ec9f60933f83efe.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs)\n SELECT $1, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs\n FROM flow WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "355e3976f0a7e3ccbb24314f951ff37dcc4c34f800ff4f385ec9f60933f83efe" +} diff --git a/backend/.sqlx/query-388b6af24fc77341abf7088303f1b027c022fc034dc378a071c046dc64571ac6.json b/backend/.sqlx/query-388b6af24fc77341abf7088303f1b027c022fc034dc378a071c046dc64571ac6.json new file mode 100644 index 0000000000..f1fab7b130 --- /dev/null +++ b/backend/.sqlx/query-388b6af24fc77341abf7088303f1b027c022fc034dc378a071c046dc64571ac6.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow (\n workspace_id, path, summary, description, value, edited_by, edited_at,\n archived, schema, extra_perms, dependency_job, tag,\n ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only,\n concurrency_key, versions, on_behalf_of_email, lock_error_logs\n )\n SELECT $2, path, summary, description, value, edited_by, edited_at,\n archived, schema, extra_perms, NULL, tag,\n ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only,\n concurrency_key, ARRAY[]::bigint[], on_behalf_of_email, lock_error_logs\n FROM flow\n WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "388b6af24fc77341abf7088303f1b027c022fc034dc378a071c046dc64571ac6" +} diff --git a/backend/.sqlx/query-394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db.json b/backend/.sqlx/query-394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db.json new file mode 100644 index 0000000000..3470285737 --- /dev/null +++ b/backend/.sqlx/query-394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "394e2598880aff8a7f4ee05c3fe748be58b6381f5fae5619d8376daefc3b21db" +} diff --git a/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json b/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json new file mode 100644 index 0000000000..7d46bd5a66 --- /dev/null +++ b/backend/.sqlx/query-39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Int4" + ] + }, + "nullable": [] + }, + "hash": "39870bcb46af48191794e77d9205c6fb9518738e14ca13796395df05c7ab1c91" +} diff --git a/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json b/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json new file mode 100644 index 0000000000..dd243cc2bb --- /dev/null +++ b/backend/.sqlx/query-3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job_queue SET running = true WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "3a03aa24f77e5729c54fd896281da45d4873c38ef68f9ac480b220405fe1ade1" +} diff --git a/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json b/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json new file mode 100644 index 0000000000..8a79de3aa4 --- /dev/null +++ b/backend/.sqlx/query-3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) as \"c!\" FROM v2_job_debounce_batch", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b06ecd4339e966bab32de0b85b7197b2f99174a0066e25d975c303b2e60a2e2" +} diff --git a/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json new file mode 100644 index 0000000000..eabe7f9bf6 --- /dev/null +++ b/backend/.sqlx/query-3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::INT AS \"v!\" FROM pg_stat_activity", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3b439ae7af0fcbb9df8e19faf84abf590e5e94898954711d57a602e6fd8a2f84" +} diff --git a/backend/.sqlx/query-3c5a387c2fed905838b0c1d2e0ade10b1ca1785b66a68a260d4aaf2957fbcc66.json b/backend/.sqlx/query-3c5a387c2fed905838b0c1d2e0ade10b1ca1785b66a68a260d4aaf2957fbcc66.json new file mode 100644 index 0000000000..71770359cc --- /dev/null +++ b/backend/.sqlx/query-3c5a387c2fed905838b0c1d2e0ade10b1ca1785b66a68a260d4aaf2957fbcc66.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(DISTINCT trigger_ref) AS \"n!\"\n FROM join_pending_inputs\n WHERE workspace_id = $1 AND subscriber_path = $2 AND partition = $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "n!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "3c5a387c2fed905838b0c1d2e0ade10b1ca1785b66a68a260d4aaf2957fbcc66" +} diff --git a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json index 20336885a6..3568d1723e 100644 --- a/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json +++ b/backend/.sqlx/query-3c84781704b84b8a927ecce5a3fcb3adcf0175d0a71013f2497397c1c8ccc619.json @@ -127,7 +127,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-3c8a2389c47131ed89ec9069b2ebe15b103cf1344b7c84069e615181508913e9.json b/backend/.sqlx/query-3c8a2389c47131ed89ec9069b2ebe15b103cf1344b7c84069e615181508913e9.json new file mode 100644 index 0000000000..650876cc23 --- /dev/null +++ b/backend/.sqlx/query-3c8a2389c47131ed89ec9069b2ebe15b103cf1344b7c84069e615181508913e9.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM join_pending_inputs jpi\n USING (\n SELECT workspace_id, subscriber_path, partition\n FROM join_pending_inputs\n GROUP BY workspace_id, subscriber_path, partition\n HAVING max(received_at) <= now() - ($1::bigint::text || ' s')::interval\n ) stale\n WHERE jpi.workspace_id = stale.workspace_id\n AND jpi.subscriber_path = stale.subscriber_path\n AND jpi.partition = stale.partition", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [] + }, + "hash": "3c8a2389c47131ed89ec9069b2ebe15b103cf1344b7c84069e615181508913e9" +} diff --git a/backend/.sqlx/query-3d7456e11d8686210169bfe26931f924bd4483c8f70095c05720bc4eb2d8df81.json b/backend/.sqlx/query-3d7456e11d8686210169bfe26931f924bd4483c8f70095c05720bc4eb2d8df81.json new file mode 100644 index 0000000000..335561cfef --- /dev/null +++ b/backend/.sqlx/query-3d7456e11d8686210169bfe26931f924bd4483c8f70095c05720bc4eb2d8df81.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'flow' AND (email = $3 OR email IS NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "3d7456e11d8686210169bfe26931f924bd4483c8f70095c05720bc4eb2d8df81" +} diff --git a/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json new file mode 100644 index 0000000000..8d9c34ed2b --- /dev/null +++ b/backend/.sqlx/query-3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM script WHERE archived = false AND deleted = false AND auto_kind = 'wac'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "3db1c61295c284725eef9e74a8aa2bc7822d263605a445f1f4a76e58e76a3e79" +} diff --git a/backend/.sqlx/query-66a9cb11fcc4757a3b35154840b703b87daa538cf3a72007649bbb074f56131e.json b/backend/.sqlx/query-3e51a3f545f67fb0f62956fd809324d753cb2b1fac69ceb738df4c3d2a6270a5.json similarity index 80% rename from backend/.sqlx/query-66a9cb11fcc4757a3b35154840b703b87daa538cf3a72007649bbb074f56131e.json rename to backend/.sqlx/query-3e51a3f545f67fb0f62956fd809324d753cb2b1fac69ceb738df4c3d2a6270a5.json index e9919c7da4..ebd8594a06 100644 --- a/backend/.sqlx/query-66a9cb11fcc4757a3b35154840b703b87daa538cf3a72007649bbb074f56131e.json +++ b/backend/.sqlx/query-3e51a3f545f67fb0f62956fd809324d753cb2b1fac69ceb738df4c3d2a6270a5.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT app.summary, app.policy, app_version.value\n FROM app\n JOIN app_version\n ON app_version.id = app.versions[array_upper(app.versions, 1)]\n WHERE app.workspace_id = $1 AND app.path = $2 AND COALESCE(app.draft_only, false) = false", + "query": "SELECT app.summary, app.policy, app_version.value\n FROM app\n JOIN app_version\n ON app_version.id = app.versions[array_upper(app.versions, 1)]\n WHERE app.workspace_id = $1 AND app.path = $2", "describe": { "columns": [ { @@ -31,5 +31,5 @@ false ] }, - "hash": "66a9cb11fcc4757a3b35154840b703b87daa538cf3a72007649bbb074f56131e" + "hash": "3e51a3f545f67fb0f62956fd809324d753cb2b1fac69ceb738df4c3d2a6270a5" } diff --git a/backend/.sqlx/query-3eb137e83c0aa6389b2893d59acd993e37d8a8bc67cd90682a7971760442b90a.json b/backend/.sqlx/query-3eb137e83c0aa6389b2893d59acd993e37d8a8bc67cd90682a7971760442b90a.json new file mode 100644 index 0000000000..9b65eb8a18 --- /dev/null +++ b/backend/.sqlx/query-3eb137e83c0aa6389b2893d59acd993e37d8a8bc67cd90682a7971760442b90a.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(DISTINCT trigger_ref) AS \"n!\"\n FROM script_trigger\n WHERE workspace_id = $1\n AND runnable_path = $2\n AND trigger_kind = 'asset'\n AND runnable_kind = 'script'\n AND trigger_ref LIKE '%' || $3 || '%'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "n!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "3eb137e83c0aa6389b2893d59acd993e37d8a8bc67cd90682a7971760442b90a" +} diff --git a/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json b/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json new file mode 100644 index 0000000000..6fd7d38f69 --- /dev/null +++ b/backend/.sqlx/query-40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "40a8cf5e87bb489fd172689e9a6f0f1075b878f9916145929b3cd3b1a53b777e" +} diff --git a/backend/.sqlx/query-40f0bc9a2555a7c90b3985a190bf3fce18c09693b645f2ac520de6936366f3c8.json b/backend/.sqlx/query-40f0bc9a2555a7c90b3985a190bf3fce18c09693b645f2ac520de6936366f3c8.json deleted file mode 100644 index a977f306ab..0000000000 --- a/backend/.sqlx/query-40f0bc9a2555a7c90b3985a190bf3fce18c09693b645f2ac520de6936366f3c8.json +++ /dev/null @@ -1,60 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT label, token_prefix, expiration, created_at, last_used_at, scopes, workspace_id FROM token WHERE email = $1 AND (label != 'ephemeral-script' OR label IS NULL)\n ORDER BY created_at DESC LIMIT $2 OFFSET $3", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "label", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "token_prefix", - "type_info": "Varchar" - }, - { - "ordinal": 2, - "name": "expiration", - "type_info": "Timestamptz" - }, - { - "ordinal": 3, - "name": "created_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 4, - "name": "last_used_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 5, - "name": "scopes", - "type_info": "TextArray" - }, - { - "ordinal": 6, - "name": "workspace_id", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Text", - "Int8", - "Int8" - ] - }, - "nullable": [ - true, - false, - true, - false, - false, - true, - true - ] - }, - "hash": "40f0bc9a2555a7c90b3985a190bf3fce18c09693b645f2ac520de6936366f3c8" -} diff --git a/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json new file mode 100644 index 0000000000..a6ab79cd7c --- /dev/null +++ b/backend/.sqlx/query-42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\"\n FROM workspace f\n JOIN workspace p ON p.id = f.parent_workspace_id\n WHERE f.id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "deleted!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "42322020ff9cc7dd7ebafc1cb4122ba3d670cc36bdbc6451f29b8f22f8cff688" +} diff --git a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json b/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json deleted file mode 100644 index 2f62420f87..0000000000 --- a/backend/.sqlx/query-454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_result_stream_v2\n WHERE job_id NOT IN (SELECT id FROM v2_job_queue)\n AND job_id NOT IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at > NOW() - INTERVAL '60 seconds'\n )\n RETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "454a611a5a162b2ace137c139bd5383bc7fe142c515dac3edd47991839485e51" -} diff --git a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json b/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json deleted file mode 100644 index f63afbc986..0000000000 --- a/backend/.sqlx/query-45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3)\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8", - "UuidArray" - ] - }, - "nullable": [ - false - ] - }, - "hash": "45997fcb4d9d62c7f7011966bf59bdb86e12ce1d0c8e925e738d2645121a5c1f" -} diff --git a/backend/.sqlx/query-fc9753f501974c4b570d710c7621ff1f6787a852bb27acf49593af963a6aacca.json b/backend/.sqlx/query-45b6c748090a0a6bf71a995413b6b571ae0f3355cfd5eb95a227a2a98136e02b.json similarity index 55% rename from backend/.sqlx/query-fc9753f501974c4b570d710c7621ff1f6787a852bb27acf49593af963a6aacca.json rename to backend/.sqlx/query-45b6c748090a0a6bf71a995413b6b571ae0f3355cfd5eb95a227a2a98136e02b.json index b699cc582a..0d9e4e859d 100644 --- a/backend/.sqlx/query-fc9753f501974c4b570d710c7621ff1f6787a852bb27acf49593af963a6aacca.json +++ b/backend/.sqlx/query-45b6c748090a0a6bf71a995413b6b571ae0f3355cfd5eb95a227a2a98136e02b.json @@ -1,17 +1,16 @@ { "db_name": "PostgreSQL", - "query": "SELECT path FROM flow_version WHERE id = $1 AND workspace_id = $2", + "query": "SELECT path AS \"path!\" FROM flow WHERE workspace_id = $1 AND archived = false", "describe": { "columns": [ { "ordinal": 0, - "name": "path", + "name": "path!", "type_info": "Varchar" } ], "parameters": { "Left": [ - "Int8", "Text" ] }, @@ -19,5 +18,5 @@ false ] }, - "hash": "fc9753f501974c4b570d710c7621ff1f6787a852bb27acf49593af963a6aacca" + "hash": "45b6c748090a0a6bf71a995413b6b571ae0f3355cfd5eb95a227a2a98136e02b" } diff --git a/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json b/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json deleted file mode 100644 index 3995bffd64..0000000000 --- a/backend/.sqlx/query-45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "45de6be332f4ec89482782e3b1640649ed1a6f6d4f1e1b636c71bdd36c31b618" -} diff --git a/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json b/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json new file mode 100644 index 0000000000..0c4d90b073 --- /dev/null +++ b/backend/.sqlx/query-462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM flow_conversation_message m\n USING flow_conversation c\n WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "462d2b2822b185a6f51fafcfa957cb3b31ee6b69abae79a214dddba0dee4425c" +} diff --git a/backend/.sqlx/query-464c51a8ea8c06232d33c45b8e59008b382c445476117a2311030734bc6d6d53.json b/backend/.sqlx/query-464c51a8ea8c06232d33c45b8e59008b382c445476117a2311030734bc6d6d53.json deleted file mode 100644 index df664b5b8b..0000000000 --- a/backend/.sqlx/query-464c51a8ea8c06232d33c45b8e59008b382c445476117a2311030734bc6d6d53.json +++ /dev/null @@ -1,14 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE v2_job_runtime SET ping = now() WHERE id = $1 AND ping < now()", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Uuid" - ] - }, - "nullable": [] - }, - "hash": "464c51a8ea8c06232d33c45b8e59008b382c445476117a2311030734bc6d6d53" -} diff --git a/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json b/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json new file mode 100644 index 0000000000..c269f7f340 --- /dev/null +++ b/backend/.sqlx/query-46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b.json @@ -0,0 +1,58 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH legacy AS (\n DELETE FROM draft\n WHERE workspace_id = $1 AND path = $2 AND typ = $3 AND email IS NULL\n RETURNING value\n )\n INSERT INTO draft (workspace_id, email, path, typ, value, created_at)\n SELECT $1, $4, $2, $3, value, now() FROM legacy\n ON CONFLICT (workspace_id, path, typ, email) WHERE email IS NOT NULL\n DO UPDATE SET value = EXCLUDED.value, created_at = now()\n RETURNING 1 as \"one!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "one!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Varchar" + ] + }, + "nullable": [ + null + ] + }, + "hash": "46f00a75b2e7e4ac70758a9687070f68bc0421f1aa228f80157adda63191d33b" +} diff --git a/backend/.sqlx/query-478e2ccf318be5beca35518785531e55b702d8e68031219798945e8a8e7f191b.json b/backend/.sqlx/query-478e2ccf318be5beca35518785531e55b702d8e68031219798945e8a8e7f191b.json new file mode 100644 index 0000000000..f6c2f22d39 --- /dev/null +++ b/backend/.sqlx/query-478e2ccf318be5beca35518785531e55b702d8e68031219798945e8a8e7f191b.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) AS \"n!\" FROM join_pending_inputs\n WHERE workspace_id = $1 AND subscriber_path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "n!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "478e2ccf318be5beca35518785531e55b702d8e68031219798945e8a8e7f191b" +} diff --git a/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json new file mode 100644 index 0000000000..43f527b3aa --- /dev/null +++ b/backend/.sqlx/query-48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_database_size(current_database())::BIGINT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "48a5df355a2bca557a3a541cf66c8e75790b7c3dd7845359019ff645a1f7c8bf" +} diff --git a/backend/.sqlx/query-48efd8f89df9c1dd8f8a4eb2b2640fcb380f2470c8488371b207a42cb7d4cd26.json b/backend/.sqlx/query-48efd8f89df9c1dd8f8a4eb2b2640fcb380f2470c8488371b207a42cb7d4cd26.json deleted file mode 100644 index 4621e0cc11..0000000000 --- a/backend/.sqlx/query-48efd8f89df9c1dd8f8a4eb2b2640fcb380f2470c8488371b207a42cb7d4cd26.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO variable\n (workspace_id, path, value, is_secret, description, account, is_oauth, expires_at, labels)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Varchar", - "Bool", - "Varchar", - "Int4", - "Bool", - "Timestamptz", - "TextArray" - ] - }, - "nullable": [] - }, - "hash": "48efd8f89df9c1dd8f8a4eb2b2640fcb380f2470c8488371b207a42cb7d4cd26" -} diff --git a/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json new file mode 100644 index 0000000000..e99432f11f --- /dev/null +++ b/backend/.sqlx/query-49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('max_connections')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "49e1f5663eed128ed956c9a50bc73a85256c0a3e5a701cc13c944e66f6402617" +} diff --git a/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json b/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json index 2a6930755a..7d950f6d8f 100644 --- a/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json +++ b/backend/.sqlx/query-4a43d4df6c5b2e8dda4308dcb88c23caf312ec377dd91e5307f00d3fb8ec325d.json @@ -79,7 +79,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-4c333861e736b8138162f0ff3bf2844c71b95c75bd8f173eac51336ab176ccc0.json b/backend/.sqlx/query-4c333861e736b8138162f0ff3bf2844c71b95c75bd8f173eac51336ab176ccc0.json deleted file mode 100644 index f1de82e5e6..0000000000 --- a/backend/.sqlx/query-4c333861e736b8138162f0ff3bf2844c71b95c75bd8f173eac51336ab176ccc0.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE variable SET labels = $1 WHERE path = $2 AND workspace_id = $3", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "TextArray", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "4c333861e736b8138162f0ff3bf2844c71b95c75bd8f173eac51336ab176ccc0" -} diff --git a/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json b/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json new file mode 100644 index 0000000000..c2c041e131 --- /dev/null +++ b/backend/.sqlx/query-4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext($1))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "4c93380abebe4682f280bc3cc0add2878746496a25db7ea50d857658c49a931f" +} diff --git a/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json b/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json new file mode 100644 index 0000000000..07d1753aa0 --- /dev/null +++ b/backend/.sqlx/query-4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "4cdc0932987ff69892f332229a1def48e0678c337884c3ef801278d8feb41301" +} diff --git a/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json new file mode 100644 index 0000000000..5bd3cf80a7 --- /dev/null +++ b/backend/.sqlx/query-50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM flow WHERE archived = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "50490ff42fb1f2d78864d7b374d299bf8290c3b969b576cb185c8b5b0abb0265" +} diff --git a/backend/.sqlx/query-5104cf045dc9b7b82d0028af11cfb5c2f6fd58e518085caf8e8d189951f7c4d8.json b/backend/.sqlx/query-5104cf045dc9b7b82d0028af11cfb5c2f6fd58e518085caf8e8d189951f7c4d8.json deleted file mode 100644 index 46025de086..0000000000 --- a/backend/.sqlx/query-5104cf045dc9b7b82d0028af11cfb5c2f6fd58e518085caf8e8d189951f7c4d8.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO draft\n (workspace_id, path, value, typ)\n VALUES ($1, $2, $3::text::json, $4)\n ON CONFLICT (workspace_id, path, typ)\n DO UPDATE SET value = EXCLUDED.value, created_at = now()", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Text", - { - "Custom": { - "name": "draft_type", - "kind": { - "Enum": [ - "script", - "flow", - "app" - ] - } - } - } - ] - }, - "nullable": [] - }, - "hash": "5104cf045dc9b7b82d0028af11cfb5c2f6fd58e518085caf8e8d189951f7c4d8" -} diff --git a/backend/.sqlx/query-5223412e366b22d953867425ed6156a66727e7a469abad1664ee03a3da31078a.json b/backend/.sqlx/query-5223412e366b22d953867425ed6156a66727e7a469abad1664ee03a3da31078a.json new file mode 100644 index 0000000000..be5e26ef9b --- /dev/null +++ b/backend/.sqlx/query-5223412e366b22d953867425ed6156a66727e7a469abad1664ee03a3da31078a.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path)\n path,\n value as \"value!: sqlx::types::Json>\",\n created_at\n FROM draft\n WHERE workspace_id = $1\n AND typ = 'flow'\n AND (email = $2 OR email IS NULL)\n AND NOT EXISTS (\n SELECT 1 FROM flow f\n WHERE f.workspace_id = draft.workspace_id\n AND f.path = draft.path\n )\n ORDER BY path, (email IS NULL)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value!: sqlx::types::Json>", + "type_info": "Json" + }, + { + "ordinal": 2, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "5223412e366b22d953867425ed6156a66727e7a469abad1664ee03a3da31078a" +} diff --git a/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json b/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json new file mode 100644 index 0000000000..d81d6ebd94 --- /dev/null +++ b/backend/.sqlx/query-54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f.json @@ -0,0 +1,49 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft\n WHERE workspace_id = $1 AND path = $2 AND typ = $3 AND email IS NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + } + ] + }, + "nullable": [] + }, + "hash": "54ad5cc89563fdbbacdd6bfde9be6ecfcdec3d505d28cc65b5920dcf87c0014f" +} diff --git a/backend/.sqlx/query-54c0c20fe025d4fb45f04ff3389b25915f671e7c52426fc54b2fd533b90596e2.json b/backend/.sqlx/query-54c0c20fe025d4fb45f04ff3389b25915f671e7c52426fc54b2fd533b90596e2.json deleted file mode 100644 index de1b71a1e6..0000000000 --- a/backend/.sqlx/query-54c0c20fe025d4fb45f04ff3389b25915f671e7c52426fc54b2fd533b90596e2.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO token\n (token_hash, token_prefix, token, email, label, expiration, super_admin)\n VALUES ($1, $2, $3, $4, $5, now() + ($6 || ' seconds')::interval, $7)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Varchar", - "Varchar", - "Varchar", - "Text", - "Bool" - ] - }, - "nullable": [] - }, - "hash": "54c0c20fe025d4fb45f04ff3389b25915f671e7c52426fc54b2fd533b90596e2" -} diff --git a/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json b/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json new file mode 100644 index 0000000000..8c46b8250c --- /dev/null +++ b/backend/.sqlx/query-54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT attempt FROM native_retry_attempt WHERE job_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "attempt", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false + ] + }, + "hash": "54ef5bde9d0e93d301673a5656bf40570cadd2771d94f5c19a7911554efe4d5c" +} diff --git a/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json b/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json new file mode 100644 index 0000000000..91c941593f --- /dev/null +++ b/backend/.sqlx/query-560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc.json @@ -0,0 +1,59 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT created_at FROM draft\n WHERE workspace_id = $1\n AND email IS NOT DISTINCT FROM (CASE WHEN $5::bool THEN NULL::text ELSE $2 END)\n AND path = $3 AND typ = $4", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Bool" + ] + }, + "nullable": [ + false + ] + }, + "hash": "560539adbad0ecfa57fa477c3b82d82c350857166fc27fe9eecc88bcc4b229bc" +} diff --git a/backend/.sqlx/query-567ded2a717af9370a80c00bdb50d965fba9a3422c58e67ed2ed06dd107ae139.json b/backend/.sqlx/query-567ded2a717af9370a80c00bdb50d965fba9a3422c58e67ed2ed06dd107ae139.json deleted file mode 100644 index 4bebcfa038..0000000000 --- a/backend/.sqlx/query-567ded2a717af9370a80c00bdb50d965fba9a3422c58e67ed2ed06dd107ae139.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM script WHERE hash = $1 AND workspace_id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Int8", - "Text" - ] - }, - "nullable": [] - }, - "hash": "567ded2a717af9370a80c00bdb50d965fba9a3422c58e67ed2ed06dd107ae139" -} diff --git a/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json new file mode 100644 index 0000000000..711970ff32 --- /dev/null +++ b/backend/.sqlx/query-56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT current_setting('server_version_num')::INT AS \"v!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "v!", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "56a98a07a2f6af4d694db05d57acbe0b55cc39d64f73a3c90c250ea670f9cdee" +} diff --git a/backend/.sqlx/query-57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89.json b/backend/.sqlx/query-57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89.json new file mode 100644 index 0000000000..f634fc2d4b --- /dev/null +++ b/backend/.sqlx/query-57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claim_self AS (\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE id = $1 AND consumed_at IS NULL\n RETURNING debounce_batch\n ), claim_rest AS (\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM claim_self)\n AND id <> $1 AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT debounce_batch FROM claim_self) AS claimed_batch,\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claim_rest) AS \"claimed_ids!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "had_row!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "claimed_batch", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "prev_consumed_by", + "type_info": "Uuid" + }, + { + "ordinal": 3, + "name": "claimed_ids!", + "type_info": "UuidArray" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "57f375e89d63ac118c5c6767487af401e6d800488ade04facb6af14578c30a89" +} diff --git a/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json b/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json new file mode 100644 index 0000000000..0c445c1ae5 --- /dev/null +++ b/backend/.sqlx/query-5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4.json @@ -0,0 +1,50 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft\n WHERE workspace_id = $1\n AND path = $2\n AND typ = $3\n AND (email = $4 OR email IS NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Text" + ] + }, + "nullable": [] + }, + "hash": "5884ce1906015f6b96231f311226e490d5dfcdd7a94dcbe5d05c9e5af37ac4a4" +} diff --git a/backend/.sqlx/query-58e8e13acd9f7ff951f37d555beab84dcea21e0a38906de164889cf2dacf2e43.json b/backend/.sqlx/query-58e8e13acd9f7ff951f37d555beab84dcea21e0a38906de164889cf2dacf2e43.json new file mode 100644 index 0000000000..3a09f1ee8d --- /dev/null +++ b/backend/.sqlx/query-58e8e13acd9f7ff951f37d555beab84dcea21e0a38906de164889cf2dacf2e43.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT args AS \"args!: Json>>\"\n FROM v2_job\n WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "args!: Json>>", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [ + true + ] + }, + "hash": "58e8e13acd9f7ff951f37d555beab84dcea21e0a38906de164889cf2dacf2e43" +} diff --git a/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json b/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json new file mode 100644 index 0000000000..0535031aac --- /dev/null +++ b/backend/.sqlx/query-5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dispatch_event WHERE workspace_id = $1 AND producer_job_id = ANY($2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "5a0d50d322f2ab58eb36da16407d31a8319db563b8761194b237ea166ada42a5" +} diff --git a/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json b/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json new file mode 100644 index 0000000000..c126b6371a --- /dev/null +++ b/backend/.sqlx/query-5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800.json @@ -0,0 +1,94 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n asset.kind AS \"asset_kind!: AssetKind\",\n asset.path AS \"asset_path!\",\n asset.usage_kind AS \"usage_kind!: AssetUsageKind\",\n asset.usage_path AS \"usage_path!\",\n asset.usage_access_type::text AS \"access_type\"\n FROM asset\n WHERE asset.workspace_id = $1\n AND asset.usage_kind IN ('script', 'flow')\n AND ($2::asset_kind[] IS NULL OR asset.kind = ANY($2))\n AND ($3::text IS NULL OR asset.usage_path LIKE $3)\n GROUP BY asset.kind, asset.path, asset.usage_kind, asset.usage_path, asset.usage_access_type\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_kind!: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "asset_path!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "usage_kind!: AssetUsageKind", + "type_info": { + "Custom": { + "name": "asset_usage_kind", + "kind": { + "Enum": [ + "script", + "flow", + "job" + ] + } + } + } + }, + { + "ordinal": 3, + "name": "usage_path!", + "type_info": "Varchar" + }, + { + "ordinal": 4, + "name": "access_type", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind[]", + "kind": { + "Array": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + null + ] + }, + "hash": "5ae004333c20e6f7c28025f16ad87562dbdac633f3248ac228e00b7c8b49b800" +} diff --git a/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json b/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json new file mode 100644 index 0000000000..28c09ca528 --- /dev/null +++ b/backend/.sqlx/query-5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\",\n result AS \"result: Json>\",\n started_at AS \"started_at!\"FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "success!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "result: Json>", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "started_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [ + null, + true, + true + ] + }, + "hash": "5b0847d2b95a128a5b648dd4847af44ed0992ced76286a08a33134140a454391" +} diff --git a/backend/.sqlx/query-5c2d1ae706e997bbca27dadb99cc0d55ffba53bc23ae69181a0d671c640b9ba7.json b/backend/.sqlx/query-5c2d1ae706e997bbca27dadb99cc0d55ffba53bc23ae69181a0d671c640b9ba7.json new file mode 100644 index 0000000000..c306ae1c9e --- /dev/null +++ b/backend/.sqlx/query-5c2d1ae706e997bbca27dadb99cc0d55ffba53bc23ae69181a0d671c640b9ba7.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO skip_workspace_diff_tally (workspace_id) VALUES ('wm-fork-test-workspace')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "5c2d1ae706e997bbca27dadb99cc0d55ffba53bc23ae69181a0d671c640b9ba7" +} diff --git a/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json b/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json new file mode 100644 index 0000000000..03de8472ad --- /dev/null +++ b/backend/.sqlx/query-5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389.json @@ -0,0 +1,48 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n usage_path AS \"usage_path!\",\n kind AS \"kind!: AssetKind\",\n path AS \"path!\"\n FROM asset\n WHERE workspace_id = $1\n AND usage_kind = 'script'\n AND usage_access_type IN ('w', 'rw')\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "usage_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "kind!: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 2, + "name": "path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "5c609ea0696df96ca02cba7fee359b785515a79dcda3745663e4c4f2cf328389" +} diff --git a/backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json b/backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json new file mode 100644 index 0000000000..17d372249b --- /dev/null +++ b/backend/.sqlx/query-5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19.json @@ -0,0 +1,47 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO materialized_partition\n (workspace_id, asset_kind, asset_path, partition, status,\n snapshot_id, row_count, job_id, materialized_at, error)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9)\n ON CONFLICT (workspace_id, asset_kind, asset_path, partition)\n DO UPDATE SET status = EXCLUDED.status,\n snapshot_id = EXCLUDED.snapshot_id,\n row_count = EXCLUDED.row_count,\n job_id = EXCLUDED.job_id,\n materialized_at = now(),\n error = EXCLUDED.error", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Varchar", + "Text", + { + "Custom": { + "name": "materialization_status", + "kind": { + "Enum": [ + "running", + "materialized", + "failed" + ] + } + } + }, + "Int8", + "Int8", + "Uuid", + "Text" + ] + }, + "nullable": [] + }, + "hash": "5e50ba0ae27b09a3ea1530c223e5039aa631bb5b0993ad09bc9a1381f6715f19" +} diff --git a/backend/.sqlx/query-5eaf2e0bbede9dd80a70f2b37538239273443c7d56a1f7732988208fe3c9c58f.json b/backend/.sqlx/query-5eaf2e0bbede9dd80a70f2b37538239273443c7d56a1f7732988208fe3c9c58f.json new file mode 100644 index 0000000000..3c5b4d3d15 --- /dev/null +++ b/backend/.sqlx/query-5eaf2e0bbede9dd80a70f2b37538239273443c7d56a1f7732988208fe3c9c58f.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM workspace_diff WHERE source_workspace_id = $1 OR fork_workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "5eaf2e0bbede9dd80a70f2b37538239273443c7d56a1f7732988208fe3c9c58f" +} diff --git a/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json new file mode 100644 index 0000000000..20c5c58c40 --- /dev/null +++ b/backend/.sqlx/query-60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n COUNT(*) FILTER (WHERE av.raw_app = false)::BIGINT AS \"low_code!\",\n COUNT(*) FILTER (WHERE av.raw_app = true)::BIGINT AS \"raw!\"\n FROM app a\n JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "low_code!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "raw!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "60dc0f1fa17bd2946ba7ddf0c41fe58b8a53d071cdc83eca3092194b4a9c9174" +} diff --git a/backend/.sqlx/query-651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b.json b/backend/.sqlx/query-651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b.json new file mode 100644 index 0000000000..cb20ec2ffb --- /dev/null +++ b/backend/.sqlx/query-651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job\n SET args = CASE\n WHEN args ? 'partition'\n THEN $1 || jsonb_build_object('partition', args -> 'partition')\n ELSE $1\n END,\n preprocessed = TRUE\n WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Jsonb", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "651fc12e1b971d4fd57c98a7a7efbd503d8dea799545e9cb96574d5c6020b90b" +} diff --git a/backend/.sqlx/query-65c2ecb52cc777f17ffeb77be597ea87026bb4e56c0ccfb12f2feaf1a6124c86.json b/backend/.sqlx/query-65c2ecb52cc777f17ffeb77be597ea87026bb4e56c0ccfb12f2feaf1a6124c86.json deleted file mode 100644 index 2492cbc015..0000000000 --- a/backend/.sqlx/query-65c2ecb52cc777f17ffeb77be597ea87026bb4e56c0ccfb12f2feaf1a6124c86.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs)\n SELECT $1, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs\n FROM flow WHERE workspace_id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Text" - ] - }, - "nullable": [] - }, - "hash": "65c2ecb52cc777f17ffeb77be597ea87026bb4e56c0ccfb12f2feaf1a6124c86" -} diff --git a/backend/.sqlx/query-dafc503a5f3adc5c7db7c11096775cacfffd2d3173dfdddcf37589cba356791e.json b/backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json similarity index 72% rename from backend/.sqlx/query-dafc503a5f3adc5c7db7c11096775cacfffd2d3173dfdddcf37589cba356791e.json rename to backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json index e327857d3f..80f172aa40 100644 --- a/backend/.sqlx/query-dafc503a5f3adc5c7db7c11096775cacfffd2d3173dfdddcf37589cba356791e.json +++ b/backend/.sqlx/query-66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41)", + "query": "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40)", "describe": { "columns": [], "parameters": { @@ -66,7 +66,6 @@ } }, "Varchar", - "Bool", "VarcharArray", "Int4", "Int4", @@ -96,5 +95,5 @@ }, "nullable": [] }, - "hash": "dafc503a5f3adc5c7db7c11096775cacfffd2d3173dfdddcf37589cba356791e" + "hash": "66c775b6e1120c5ed53b903d252e07f0965efb65c51580c529e61d09b8742dfd" } diff --git a/backend/.sqlx/query-67e092189a1a7be4fc3b660dc7a5fcfb1fdb296ea8893312d7d72ac80e4f506e.json b/backend/.sqlx/query-67e092189a1a7be4fc3b660dc7a5fcfb1fdb296ea8893312d7d72ac80e4f506e.json new file mode 100644 index 0000000000..dcff085b5e --- /dev/null +++ b/backend/.sqlx/query-67e092189a1a7be4fc3b660dc7a5fcfb1fdb296ea8893312d7d72ac80e4f506e.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO script\n (workspace_id, hash, path, parent_hashes, summary, description, content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels)\n\n SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels\n\n FROM script WHERE hash = $2 AND workspace_id = $3;\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Text" + ] + }, + "nullable": [] + }, + "hash": "67e092189a1a7be4fc3b660dc7a5fcfb1fdb296ea8893312d7d72ac80e4f506e" +} diff --git a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json b/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json index 50e7b53387..08ebe6bba5 100644 --- a/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json +++ b/backend/.sqlx/query-67e25a7c19ea0ffaf7ea5303fcd04af5a7eb488c76f783e690af0c2153b1d6a8.json @@ -160,7 +160,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-6814542fcdd01a178798ad7b0840d8288f1d6adbeb3e545386cf66c33f8db50f.json b/backend/.sqlx/query-6814542fcdd01a178798ad7b0840d8288f1d6adbeb3e545386cf66c33f8db50f.json new file mode 100644 index 0000000000..a74aedfdb5 --- /dev/null +++ b/backend/.sqlx/query-6814542fcdd01a178798ad7b0840d8288f1d6adbeb3e545386cf66c33f8db50f.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT count(*) AS \"n!\"\n FROM join_pending_inputs\n WHERE workspace_id = $1 AND subscriber_path = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "n!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "6814542fcdd01a178798ad7b0840d8288f1d6adbeb3e545386cf66c33f8db50f" +} diff --git a/backend/.sqlx/query-68b4a667ac15258b718e7c9c0224c4b467ff43d84618a3471eb7b1cfec13883d.json b/backend/.sqlx/query-68b4a667ac15258b718e7c9c0224c4b467ff43d84618a3471eb7b1cfec13883d.json new file mode 100644 index 0000000000..02e17b82cb --- /dev/null +++ b/backend/.sqlx/query-68b4a667ac15258b718e7c9c0224c4b467ff43d84618a3471eb7b1cfec13883d.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH inserted AS (\n INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs)\n SELECT workspace_id, REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3\n RETURNING 1\n ) SELECT COUNT(*) FROM inserted", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "68b4a667ac15258b718e7c9c0224c4b467ff43d84618a3471eb7b1cfec13883d" +} diff --git a/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json b/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json new file mode 100644 index 0000000000..144bcb13f2 --- /dev/null +++ b/backend/.sqlx/query-6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f.json @@ -0,0 +1,64 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO dispatch_event (\n workspace_id, producer_job_id, subscriber_path,\n asset_kind, asset_path, outcome,\n child_job_id, partition,\n received_inputs, required_inputs,\n debounce_s, reason\n )\n SELECT $1, $2, sp, ak, ap, oc, cj, pt, ri, rq, db, rs\n FROM unnest(\n $3::text[], $4::ASSET_KIND[], $5::text[], $6::DISPATCH_OUTCOME[],\n $7::uuid[], $8::text[], $9::int[], $10::int[], $11::int[], $12::text[]\n ) AS t(sp, ak, ap, oc, cj, pt, ri, rq, db, rs)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Uuid", + "TextArray", + { + "Custom": { + "name": "asset_kind[]", + "kind": { + "Array": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + } + } + }, + "TextArray", + { + "Custom": { + "name": "dispatch_outcome[]", + "kind": { + "Array": { + "Custom": { + "name": "dispatch_outcome", + "kind": { + "Enum": [ + "dispatched", + "join_pending", + "skipped" + ] + } + } + } + } + } + }, + "UuidArray", + "TextArray", + "Int4Array", + "Int4Array", + "Int4Array", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "6aaddd80f8c07cfafea2021c1879c3d7b2fb156a43299e9a1209d05293c4f50f" +} diff --git a/backend/.sqlx/query-6b9348e60cc1ce158314a93fc7aa55a9f8fa854b29edcea83710a9170124edf0.json b/backend/.sqlx/query-6b9348e60cc1ce158314a93fc7aa55a9f8fa854b29edcea83710a9170124edf0.json new file mode 100644 index 0000000000..43d3b6d7f6 --- /dev/null +++ b/backend/.sqlx/query-6b9348e60cc1ce158314a93fc7aa55a9f8fa854b29edcea83710a9170124edf0.json @@ -0,0 +1,64 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value as \"value!: sqlx::types::Json>\", created_at\n FROM draft\n WHERE workspace_id = $1 AND path = $2 AND typ = $3 AND email = $4", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value!: sqlx::types::Json>", + "type_info": "Json" + }, + { + "ordinal": 1, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "6b9348e60cc1ce158314a93fc7aa55a9f8fa854b29edcea83710a9170124edf0" +} diff --git a/backend/.sqlx/query-6e4afbd2931a2d96b99b88a270388006066c83a2697913e43dc3c3db4bd4bd2e.json b/backend/.sqlx/query-6e4afbd2931a2d96b99b88a270388006066c83a2697913e43dc3c3db4bd4bd2e.json new file mode 100644 index 0000000000..f4839e8e38 --- /dev/null +++ b/backend/.sqlx/query-6e4afbd2931a2d96b99b88a270388006066c83a2697913e43dc3c3db4bd4bd2e.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms)\n VALUES ('test-workspace', 'f/shared/dashboard', 'Dashboard app', '{}', ARRAY[1::bigint], '{}')", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "6e4afbd2931a2d96b99b88a270388006066c83a2697913e43dc3c3db4bd4bd2e" +} diff --git a/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json b/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json new file mode 100644 index 0000000000..1e39bfdaab --- /dev/null +++ b/backend/.sqlx/query-6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b.json @@ -0,0 +1,64 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) as \"username?\",\n d.created_at as \"draft_saved_at!\"\n FROM draft d\n LEFT JOIN usr u\n ON u.workspace_id = d.workspace_id\n AND u.email = d.email\n WHERE d.workspace_id = $1\n AND d.path = $2\n AND d.typ = $3\n AND (d.email IS NULL OR d.email <> $4)\n ORDER BY d.email NULLS LAST", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "username?", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "draft_saved_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + null, + false + ] + }, + "hash": "6fbd5826ae1fafd7bbb218324e3e2d85231b889743de637eb35e756da29ad47b" +} diff --git a/backend/.sqlx/query-70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18.json b/backend/.sqlx/query-70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18.json new file mode 100644 index 0000000000..e885c72039 --- /dev/null +++ b/backend/.sqlx/query-70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL AND consumed_at < now() - interval '1 hour'\n RETURNING 1\n ) SELECT count(*) FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "70f01b322765442de8888b6d9b79984da751f9ae10c8b15ada925bd371501e18" +} diff --git a/backend/.sqlx/query-71c945f93c0a1b561a85e8462b1687a54bd098cf6e84f57e5755eb84e1552345.json b/backend/.sqlx/query-71c945f93c0a1b561a85e8462b1687a54bd098cf6e84f57e5755eb84e1552345.json deleted file mode 100644 index da3b4cc1b9..0000000000 --- a/backend/.sqlx/query-71c945f93c0a1b561a85e8462b1687a54bd098cf6e84f57e5755eb84e1552345.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT draft_only FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "draft_only", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - true - ] - }, - "hash": "71c945f93c0a1b561a85e8462b1687a54bd098cf6e84f57e5755eb84e1552345" -} diff --git a/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json b/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json new file mode 100644 index 0000000000..d55722ed24 --- /dev/null +++ b/backend/.sqlx/query-734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by)\n VALUES ($1, $2, $3, $4, now(), $5)\n ON CONFLICT (workspace_id, name) DO UPDATE\n SET description = EXCLUDED.description,\n instructions = EXCLUDED.instructions,\n edited_at = now(),\n edited_by = EXCLUDED.edited_by", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Text", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "734781e8e55e95c55f72e094e96297aa852e20a0f0d20db4b993947792f6b0a8" +} diff --git a/backend/.sqlx/query-751f836dc8f78c330387456dd68a8803972c7b3e2b6a2b95c27f15068bed2ca5.json b/backend/.sqlx/query-751f836dc8f78c330387456dd68a8803972c7b3e2b6a2b95c27f15068bed2ca5.json new file mode 100644 index 0000000000..a784094d72 --- /dev/null +++ b/backend/.sqlx/query-751f836dc8f78c330387456dd68a8803972c7b3e2b6a2b95c27f15068bed2ca5.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtextextended($1, 0))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "751f836dc8f78c330387456dd68a8803972c7b3e2b6a2b95c27f15068bed2ca5" +} diff --git a/backend/.sqlx/query-754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc.json b/backend/.sqlx/query-754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc.json new file mode 100644 index 0000000000..c5266c66b5 --- /dev/null +++ b/backend/.sqlx/query-754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path AS \"runnable_path!\", kind::text AS \"kind!\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'asset'\n ORDER BY runnable_path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "kind!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + null + ] + }, + "hash": "754b98335e8776565d63267b395013649adacf348e3a815e991b4463b1711afc" +} diff --git a/backend/.sqlx/query-755a9c2f19d3befe68ebffca43abae28b5ba639731cb4867ab2a8e0acdcc9c32.json b/backend/.sqlx/query-755a9c2f19d3befe68ebffca43abae28b5ba639731cb4867ab2a8e0acdcc9c32.json deleted file mode 100644 index 112b8fe253..0000000000 --- a/backend/.sqlx/query-755a9c2f19d3befe68ebffca43abae28b5ba639731cb4867ab2a8e0acdcc9c32.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO flow (\n workspace_id, path, summary, description, value, edited_by, edited_at,\n archived, schema, extra_perms, dependency_job, draft_only, tag,\n ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only,\n concurrency_key, versions, on_behalf_of_email, lock_error_logs\n )\n SELECT $2, path, summary, description, value, edited_by, edited_at,\n archived, schema, extra_perms, NULL, draft_only, tag,\n ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only,\n concurrency_key, ARRAY[]::bigint[], on_behalf_of_email, lock_error_logs\n FROM flow\n WHERE workspace_id = $1", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Varchar" - ] - }, - "nullable": [] - }, - "hash": "755a9c2f19d3befe68ebffca43abae28b5ba639731cb4867ab2a8e0acdcc9c32" -} diff --git a/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json b/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json new file mode 100644 index 0000000000..694ed1887f --- /dev/null +++ b/backend/.sqlx/query-76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM debounce_key WHERE key = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "76774e6f72c8c8b7473487e4176dc17b17372b7292e39d3888a93ff4fe49e4f5" +} diff --git a/backend/.sqlx/query-77424d40104cf271e5ee5118100a988159130fc3a8cde91d419a1787b6bb8a51.json b/backend/.sqlx/query-77424d40104cf271e5ee5118100a988159130fc3a8cde91d419a1787b6bb8a51.json new file mode 100644 index 0000000000..0768563002 --- /dev/null +++ b/backend/.sqlx/query-77424d40104cf271e5ee5118100a988159130fc3a8cde91d419a1787b6bb8a51.json @@ -0,0 +1,52 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n runnable_kind AS \"runnable_kind!: AssetUsageKind\",\n runnable_path AS \"runnable_path!\",\n trigger_kind::text AS \"trigger_kind!\",\n trigger_ref AS \"trigger_ref!\"\n FROM script_trigger\n WHERE workspace_id = $1\n AND trigger_kind = 'asset'\n AND ($2::text IS NULL OR runnable_path LIKE $2)\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_kind!: AssetUsageKind", + "type_info": { + "Custom": { + "name": "asset_usage_kind", + "kind": { + "Enum": [ + "script", + "flow", + "job" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "trigger_kind!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "trigger_ref!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + null, + false + ] + }, + "hash": "77424d40104cf271e5ee5118100a988159130fc3a8cde91d419a1787b6bb8a51" +} diff --git a/backend/.sqlx/query-7753786abd0b9c7b548d0f8157023823945e84997b038d5b7915406c4ccb302d.json b/backend/.sqlx/query-7753786abd0b9c7b548d0f8157023823945e84997b038d5b7915406c4ccb302d.json new file mode 100644 index 0000000000..9dc1d0f296 --- /dev/null +++ b/backend/.sqlx/query-7753786abd0b9c7b548d0f8157023823945e84997b038d5b7915406c4ccb302d.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script (\n workspace_id, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n )\n SELECT\n $1, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n FROM script\n WHERE workspace_id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "7753786abd0b9c7b548d0f8157023823945e84997b038d5b7915406c4ccb302d" +} diff --git a/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json new file mode 100644 index 0000000000..f909a33b4c --- /dev/null +++ b/backend/.sqlx/query-7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"x\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "7b6fa8b999a0160ca349e70e1cd41b344a1710715fb230ac97aca875a9ef38a5" +} diff --git a/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json b/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json new file mode 100644 index 0000000000..82dcf60cc1 --- /dev/null +++ b/backend/.sqlx/query-80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "80618456d483c3e7607cd0587684c46f66ed686feb8e2bc846a02b8b43bb6684" +} diff --git a/backend/.sqlx/query-80f2d2f20e93b5e05ecd1fe5afeaeebc22883bf1d10dcbce41cccebb392ffd69.json b/backend/.sqlx/query-80f2d2f20e93b5e05ecd1fe5afeaeebc22883bf1d10dcbce41cccebb392ffd69.json new file mode 100644 index 0000000000..f0f7df5289 --- /dev/null +++ b/backend/.sqlx/query-80f2d2f20e93b5e05ecd1fe5afeaeebc22883bf1d10dcbce41cccebb392ffd69.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM join_pending_inputs\n WHERE workspace_id = $1 AND subscriber_path = $2 AND partition = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "80f2d2f20e93b5e05ecd1fe5afeaeebc22883bf1d10dcbce41cccebb392ffd69" +} diff --git a/backend/.sqlx/query-82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2.json b/backend/.sqlx/query-82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2.json new file mode 100644 index 0000000000..96587ed1d0 --- /dev/null +++ b/backend/.sqlx/query-82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT q.runnable_settings_handle\n FROM v2_job j JOIN v2_job_queue q ON q.id = j.id\n WHERE j.workspace_id = $1 AND j.runnable_path = $2\n AND j.trigger_kind = 'asset'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_settings_handle", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "82dcaf94ffe43da1c8c7de2a3478b4919c4f1dbf1972d04664a730cefc0594e2" +} diff --git a/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json b/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json new file mode 100644 index 0000000000..79b43245ef --- /dev/null +++ b/backend/.sqlx/query-82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "concurrency_settings", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "debouncing_settings", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "retry_settings", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [ + true, + true, + true + ] + }, + "hash": "82e2bdf46cb463a3bc0cba32d8a066b02c2793a58dde6e3804aa2151903fe496" +} diff --git a/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json b/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json new file mode 100644 index 0000000000..03fc734037 --- /dev/null +++ b/backend/.sqlx/query-851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "851fa1993bbe8a3e1f7653d9dba21ad1e81107598064689e51d13123bf8116ab" +} diff --git a/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json b/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json deleted file mode 100644 index 96e3439612..0000000000 --- a/backend/.sqlx/query-8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "8657c21ace89a9bafe4d184b30e3fc104a2c83f698f7dd657d6e6c95c4ff1f3b" -} diff --git a/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json new file mode 100644 index 0000000000..ffce4491e3 --- /dev/null +++ b/backend/.sqlx/query-87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d.json @@ -0,0 +1,67 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT ON (path) path AS \"path!\", content AS \"content!\",\n language AS \"language!: windmill_common::scripts::ScriptLang\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ORDER BY path, created_at DESC\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "content!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "language!: windmill_common::scripts::ScriptLang", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby", + "rlang" + ] + } + } + } + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "87ca1f2a34f54dade76f5a2cde5ecdac6806e1a306dca880943d97bb6d6a889d" +} diff --git a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json similarity index 68% rename from backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json rename to backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json index 56b887e8b9..f8f8511709 100644 --- a/backend/.sqlx/query-8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d.json +++ b/backend/.sqlx/query-881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829.json @@ -1,16 +1,17 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()", + "query": "INSERT INTO background_task_state\n (name, value, running, owner, started_at, finished_at, updated_at)\n VALUES ($1, $2, false, $3, now(), now(), now())\n ON CONFLICT (name) DO UPDATE SET\n value = $2, running = false, owner = $3,\n finished_at = now(), updated_at = now()\n WHERE (background_task_state.value->>'last_xmin')::bigint <= $4", "describe": { "columns": [], "parameters": { "Left": [ "Text", "Jsonb", - "Text" + "Text", + "Int8" ] }, "nullable": [] }, - "hash": "8711bb7861cb3c453519a620057e1530039c09b824065027387bbb667a49fe8d" + "hash": "881d996af5aaa1ec01693e473519b64b33c722afde7457912e8abd5743de8829" } diff --git a/backend/.sqlx/query-8bb2f6f4526231c1ce57182a779c9d7cb5d1022da6b5bf5a17c73c61773b50f4.json b/backend/.sqlx/query-8bb2f6f4526231c1ce57182a779c9d7cb5d1022da6b5bf5a17c73c61773b50f4.json new file mode 100644 index 0000000000..153b11560e --- /dev/null +++ b/backend/.sqlx/query-8bb2f6f4526231c1ce57182a779c9d7cb5d1022da6b5bf5a17c73c61773b50f4.json @@ -0,0 +1,47 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT runnable_path AS \"runnable_path!\", join_all AS \"join_all!\", debounce_s,\n retry_count, retry_delay_s\n FROM script_trigger\n WHERE workspace_id = $1\n AND trigger_kind = 'asset'\n AND trigger_ref = $2\n AND runnable_kind = 'script'\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "join_all!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "debounce_s", + "type_info": "Int4" + }, + { + "ordinal": 3, + "name": "retry_count", + "type_info": "Int2" + }, + { + "ordinal": 4, + "name": "retry_delay_s", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + true, + true, + true + ] + }, + "hash": "8bb2f6f4526231c1ce57182a779c9d7cb5d1022da6b5bf5a17c73c61773b50f4" +} diff --git a/backend/.sqlx/query-ea1637af410b48f3673f64ae783dda41f946268be4ffbaacc7fc1865c8974628.json b/backend/.sqlx/query-8bce3f969b4bbbcf3dc1b8c671ab76e947e491da84fbaf5f130f58fbc851594e.json similarity index 57% rename from backend/.sqlx/query-ea1637af410b48f3673f64ae783dda41f946268be4ffbaacc7fc1865c8974628.json rename to backend/.sqlx/query-8bce3f969b4bbbcf3dc1b8c671ab76e947e491da84fbaf5f130f58fbc851594e.json index 6dd90f0961..fe484078c2 100644 --- a/backend/.sqlx/query-ea1637af410b48f3673f64ae783dda41f946268be4ffbaacc7fc1865c8974628.json +++ b/backend/.sqlx/query-8bce3f969b4bbbcf3dc1b8c671ab76e947e491da84fbaf5f130f58fbc851594e.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "DELETE FROM draft WHERE path = $1 AND typ = $2 AND workspace_id = $3", + "query": "DELETE FROM script_trigger\n WHERE workspace_id = $1 AND runnable_kind = $2 AND runnable_path = $3", "describe": { "columns": [], "parameters": { @@ -8,12 +8,12 @@ "Text", { "Custom": { - "name": "draft_type", + "name": "asset_usage_kind", "kind": { "Enum": [ "script", "flow", - "app" + "job" ] } } @@ -23,5 +23,5 @@ }, "nullable": [] }, - "hash": "ea1637af410b48f3673f64ae783dda41f946268be4ffbaacc7fc1865c8974628" + "hash": "8bce3f969b4bbbcf3dc1b8c671ab76e947e491da84fbaf5f130f58fbc851594e" } diff --git a/backend/.sqlx/query-8be291d84471ff742a3c2a9d53cda55f57b4de71db778642ed72654bf47d26a5.json b/backend/.sqlx/query-8be291d84471ff742a3c2a9d53cda55f57b4de71db778642ed72654bf47d26a5.json new file mode 100644 index 0000000000..33e7bfe8ec --- /dev/null +++ b/backend/.sqlx/query-8be291d84471ff742a3c2a9d53cda55f57b4de71db778642ed72654bf47d26a5.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT now() as \"now!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "now!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "8be291d84471ff742a3c2a9d53cda55f57b4de71db778642ed72654bf47d26a5" +} diff --git a/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json b/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json new file mode 100644 index 0000000000..24d70efae1 --- /dev/null +++ b/backend/.sqlx/query-8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b.json @@ -0,0 +1,74 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n de.producer_job_id AS \"producer_job_id!\",\n de.child_job_id,\n de.subscriber_path AS \"subscriber_path!\",\n de.outcome::text AS \"outcome!\",\n de.asset_kind AS \"asset_kind!: windmill_common::assets::AssetKind\",\n de.asset_path AS \"asset_path!\",\n de.created_at AS \"created_at!\"\n FROM dispatch_event de\n JOIN v2_job pj ON pj.id = de.producer_job_id\n WHERE de.workspace_id = $1\n AND de.outcome IN ('dispatched', 'join_pending')\n AND de.subscriber_path LIKE $2\n AND ($3::timestamptz IS NULL OR de.created_at >= $3)\n ORDER BY de.created_at DESC, de.id DESC\n LIMIT 4000", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "producer_job_id!", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "child_job_id", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "subscriber_path!", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "outcome!", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "asset_kind!: windmill_common::assets::AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 5, + "name": "asset_path!", + "type_info": "Text" + }, + { + "ordinal": 6, + "name": "created_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Timestamptz" + ] + }, + "nullable": [ + false, + true, + false, + null, + false, + false, + false + ] + }, + "hash": "8cd02a5378bea03012e6cb937f23460b6438d0a320b5554acec0beddbdeb008b" +} diff --git a/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json new file mode 100644 index 0000000000..eac4547361 --- /dev/null +++ b/backend/.sqlx/query-8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO materialized_asset_schema\n (workspace_id, asset_kind, asset_path, version, columns,\n snapshot_id, job_id, captured_at)\n VALUES ($1, $2, $3, $4, $5, $6, $7, now())", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Varchar", + "Int8", + "Jsonb", + "Int8", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "8e829a7358ea47100c99e78067266eb7a83e312b88f0244d1c517b03a9bcdea0" +} diff --git a/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json b/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json new file mode 100644 index 0000000000..90031c14d6 --- /dev/null +++ b/backend/.sqlx/query-8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422.json @@ -0,0 +1,49 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft\n WHERE workspace_id = $1\n AND path = $2\n AND typ = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + } + ] + }, + "nullable": [] + }, + "hash": "8f163ee5adf4caaaa12a5698e68c749524f1db10a51f3de6cadd4826e6c1d422" +} diff --git a/backend/.sqlx/query-90068339285f3802a975750274d4f0707c2243f085bc3fc442065fb4391330df.json b/backend/.sqlx/query-90068339285f3802a975750274d4f0707c2243f085bc3fc442065fb4391330df.json new file mode 100644 index 0000000000..681b9d7d7d --- /dev/null +++ b/backend/.sqlx/query-90068339285f3802a975750274d4f0707c2243f085bc3fc442065fb4391330df.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at)\n SELECT workspace_id, REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "90068339285f3802a975750274d4f0707c2243f085bc3fc442065fb4391330df" +} diff --git a/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json new file mode 100644 index 0000000000..0f75ec3883 --- /dev/null +++ b/backend/.sqlx/query-9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*)::BIGINT AS \"count!\" FROM workspace WHERE deleted = false AND id NOT LIKE 'wm-fork%'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9170a350e1da0b33a421a119d4a5b86575c1be761de158ad664670e981524cbf" +} diff --git a/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json b/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json new file mode 100644 index 0000000000..794f62e49d --- /dev/null +++ b/backend/.sqlx/query-92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1 AND id = ANY($2))", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "92b011eda2a652af714b08137b447753bc03ec35f6daa8ca245b1c8ed34c405f" +} diff --git a/backend/.sqlx/query-93aa569329a85799594606a4f77fe955820f7b2761df6b38a6a6615b518188f9.json b/backend/.sqlx/query-93aa569329a85799594606a4f77fe955820f7b2761df6b38a6a6615b518188f9.json deleted file mode 100644 index c9faa982ba..0000000000 --- a/backend/.sqlx/query-93aa569329a85799594606a4f77fe955820f7b2761df6b38a6a6615b518188f9.json +++ /dev/null @@ -1,47 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE token SET last_used_at = now() WHERE\n token_hash = $1\n AND (expiration > NOW() OR expiration IS NULL)\n AND (workspace_id IS NULL OR workspace_id = $2)\n RETURNING owner, email, super_admin, scopes, label", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "owner", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "email", - "type_info": "Varchar" - }, - { - "ordinal": 2, - "name": "super_admin", - "type_info": "Bool" - }, - { - "ordinal": 3, - "name": "scopes", - "type_info": "TextArray" - }, - { - "ordinal": 4, - "name": "label", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - true, - true, - false, - true, - true - ] - }, - "hash": "93aa569329a85799594606a4f77fe955820f7b2761df6b38a6a6615b518188f9" -} diff --git a/backend/.sqlx/query-97966407e9f1fa80fd227f75686cc9ecbb767c69ee294638c1c0957f29fd440f.json b/backend/.sqlx/query-97966407e9f1fa80fd227f75686cc9ecbb767c69ee294638c1c0957f29fd440f.json deleted file mode 100644 index f14f3e5c61..0000000000 --- a/backend/.sqlx/query-97966407e9f1fa80fd227f75686cc9ecbb767c69ee294638c1c0957f29fd440f.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT draft_only FROM script WHERE path = $1 AND workspace_id = $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "draft_only", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - true - ] - }, - "hash": "97966407e9f1fa80fd227f75686cc9ecbb767c69ee294638c1c0957f29fd440f" -} diff --git a/backend/.sqlx/query-9a7f4786fc29ed2b561d9eb96c274c66da29fdfb3e862e06c10c5deb4a2b5771.json b/backend/.sqlx/query-9a7f4786fc29ed2b561d9eb96c274c66da29fdfb3e862e06c10c5deb4a2b5771.json deleted file mode 100644 index ed3cf85466..0000000000 --- a/backend/.sqlx/query-9a7f4786fc29ed2b561d9eb96c274c66da29fdfb3e862e06c10c5deb4a2b5771.json +++ /dev/null @@ -1,12 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms, draft_only)\n VALUES ('test-workspace', 'f/shared/dashboard', 'Dashboard app', '{}', ARRAY[1::bigint], '{}', false)", - "describe": { - "columns": [], - "parameters": { - "Left": [] - }, - "nullable": [] - }, - "hash": "9a7f4786fc29ed2b561d9eb96c274c66da29fdfb3e862e06c10c5deb4a2b5771" -} diff --git a/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json b/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json new file mode 100644 index 0000000000..e2132d7737 --- /dev/null +++ b/backend/.sqlx/query-9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) as \"c!\" FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "c!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9b781d92eba2f6eabe16f99cf908f566e4b42b1f9fa445f2719ac79ad535277d" +} diff --git a/backend/.sqlx/query-52b07f3ef0aa3ac4f6d6be0db60c732e396a79458489c564bbf2e4a28de06f1f.json b/backend/.sqlx/query-9b88afcbecd66e5e29658463a2b8c2aa44d714bcc95e242ef329f6fd71c345c1.json similarity index 76% rename from backend/.sqlx/query-52b07f3ef0aa3ac4f6d6be0db60c732e396a79458489c564bbf2e4a28de06f1f.json rename to backend/.sqlx/query-9b88afcbecd66e5e29658463a2b8c2aa44d714bcc95e242ef329f6fd71c345c1.json index 5de7aed2e0..08dfcfd6bd 100644 --- a/backend/.sqlx/query-52b07f3ef0aa3ac4f6d6be0db60c732e396a79458489c564bbf2e4a28de06f1f.json +++ b/backend/.sqlx/query-9b88afcbecd66e5e29658463a2b8c2aa44d714bcc95e242ef329f6fd71c345c1.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT resource.workspace_id, resource.path, resource.value, resource.description,\n resource.resource_type, resource.extra_perms, resource.created_by, resource.edited_at,\n resource.labels,\n folder_labels(resource.workspace_id, resource.path) as \"inherited_labels?\",\n (now() > account.expires_at) as is_expired, account.refresh_token != '' as is_refreshed,\n account.refresh_error,\n variable.path IS NOT NULL as is_linked,\n variable.is_oauth as \"is_oauth?\",\n variable.account,\n ws_specific.path IS NOT NULL as ws_specific\n FROM resource\n LEFT JOIN variable ON variable.path = resource.path AND variable.workspace_id = $2\n LEFT JOIN account ON variable.account = account.id AND account.workspace_id = $2\n LEFT JOIN ws_specific ON ws_specific.path = resource.path AND ws_specific.workspace_id = $2 AND ws_specific.item_kind = 'resource'\n WHERE resource.path = $1 AND resource.workspace_id = $2", + "query": "SELECT resource.workspace_id, resource.path, resource.value, resource.description,\n resource.resource_type, resource.extra_perms, resource.created_by, resource.edited_at,\n resource.labels,\n folder_labels(resource.workspace_id, resource.path) as \"inherited_labels?\",\n (now() > account.expires_at) as is_expired, account.refresh_token != '' as is_refreshed,\n account.refresh_error,\n variable.path IS NOT NULL as is_linked,\n variable.is_oauth as \"is_oauth?\",\n variable.account,\n ws_specific.path IS NOT NULL as ws_specific,\n null::bool as draft_only,\n null::bool as is_draft\n FROM resource\n LEFT JOIN variable ON variable.path = resource.path AND variable.workspace_id = $2\n LEFT JOIN account ON variable.account = account.id AND account.workspace_id = $2\n LEFT JOIN ws_specific ON ws_specific.path = resource.path AND ws_specific.workspace_id = $2 AND ws_specific.item_kind = 'resource'\n WHERE resource.path = $1 AND resource.workspace_id = $2", "describe": { "columns": [ { @@ -87,6 +87,16 @@ "ordinal": 16, "name": "ws_specific", "type_info": "Bool" + }, + { + "ordinal": 17, + "name": "draft_only", + "type_info": "Bool" + }, + { + "ordinal": 18, + "name": "is_draft", + "type_info": "Bool" } ], "parameters": { @@ -112,8 +122,10 @@ null, false, true, + null, + null, null ] }, - "hash": "52b07f3ef0aa3ac4f6d6be0db60c732e396a79458489c564bbf2e4a28de06f1f" + "hash": "9b88afcbecd66e5e29658463a2b8c2aa44d714bcc95e242ef329f6fd71c345c1" } diff --git a/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json b/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json new file mode 100644 index 0000000000..58630c5356 --- /dev/null +++ b/backend/.sqlx/query-9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "9d505f1d388f6160ccbe569204bf6cfb90400d864e409d615b0d756217727d96" +} diff --git a/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json b/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json index fd32ba2753..ad9e57801e 100644 --- a/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json +++ b/backend/.sqlx/query-9ecb404e46a4eac55f977f05a3afbafe5dc3cdecc17a3d5a7476b160c1b6e7e1.json @@ -34,7 +34,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json b/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json new file mode 100644 index 0000000000..6a39c40970 --- /dev/null +++ b/backend/.sqlx/query-9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH del AS (\n DELETE FROM v2_job_debounce_batch\n WHERE consumed_at IS NOT NULL\n AND consumed_at < now() - interval '10 minutes'\n AND id NOT IN (SELECT id FROM v2_job_queue)\n RETURNING 1\n ) SELECT count(*) FROM del", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "9f28b636e96aa3461d84de37815a04628af9dcfb6baa2c25fb7b32152227dc77" +} diff --git a/backend/.sqlx/query-9f86d16016ddbed5ff2a87c113a675a2a05eaf30237e21359c52f31bb1bddc73.json b/backend/.sqlx/query-9f86d16016ddbed5ff2a87c113a675a2a05eaf30237e21359c52f31bb1bddc73.json deleted file mode 100644 index 163dc2285b..0000000000 --- a/backend/.sqlx/query-9f86d16016ddbed5ff2a87c113a675a2a05eaf30237e21359c52f31bb1bddc73.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO token\n (token_hash, token_prefix, token, email, label, expiration, super_admin, scopes, workspace_id)\n SELECT $1, $2, $3, $4, $5, $6, $7, $8, $9\n WHERE $9::varchar IS NULL OR NOT EXISTS(\n SELECT 1 FROM workspace WHERE id = $9 AND deleted = true\n )", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Varchar", - "Varchar", - "Varchar", - "Timestamptz", - "Bool", - "TextArray", - "Varchar" - ] - }, - "nullable": [] - }, - "hash": "9f86d16016ddbed5ff2a87c113a675a2a05eaf30237e21359c52f31bb1bddc73" -} diff --git a/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json new file mode 100644 index 0000000000..24e387a783 --- /dev/null +++ b/backend/.sqlx/query-a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614" +} diff --git a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json index f88fbc8a47..9a21f228ea 100644 --- a/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json +++ b/backend/.sqlx/query-a4b6371d33206010b2f3ffd2b09e33244fe8ab9a803248fc23f334034d24aad4.json @@ -190,7 +190,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-a4e759ec0c5ff37fe5c280c2d2f816db2dad8fdf02aecbceac0979ce5a7982c8.json b/backend/.sqlx/query-a4e759ec0c5ff37fe5c280c2d2f816db2dad8fdf02aecbceac0979ce5a7982c8.json deleted file mode 100644 index 83efc5d35c..0000000000 --- a/backend/.sqlx/query-a4e759ec0c5ff37fe5c280c2d2f816db2dad8fdf02aecbceac0979ce5a7982c8.json +++ /dev/null @@ -1,24 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "WITH inserted AS (\n INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs)\n SELECT workspace_id, REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3\n RETURNING 1\n ) SELECT COUNT(*) FROM inserted", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "count", - "type_info": "Int8" - } - ], - "parameters": { - "Left": [ - "Text", - "Text", - "Text" - ] - }, - "nullable": [ - null - ] - }, - "hash": "a4e759ec0c5ff37fe5c280c2d2f816db2dad8fdf02aecbceac0979ce5a7982c8" -} diff --git a/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json b/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json new file mode 100644 index 0000000000..819928ddc1 --- /dev/null +++ b/backend/.sqlx/query-a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id)\n SELECT $1, $2, owner, false, premium,\n CASE WHEN $4 THEN parent_workspace_id ELSE NULL END\n FROM workspace WHERE id = $3", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Bool" + ] + }, + "nullable": [] + }, + "hash": "a54efa4a7466e61fd54d8fe293cb775225dcb430026cebe15ba4994ac636514d" +} diff --git a/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json b/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json new file mode 100644 index 0000000000..a8dc380f40 --- /dev/null +++ b/backend/.sqlx/query-a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH mine AS (\n SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1\n ), claimed AS (\n -- Claim the whole batch in ONE update so concurrent same-batch\n -- survivors lock rows in identical scan order (no lock-ordering\n -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual\n -- and skips rows the other already took. A claim therefore consumes\n -- every still-unclaimed row of the batch atomically.\n UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1\n WHERE debounce_batch = (SELECT debounce_batch FROM mine)\n AND consumed_at IS NULL\n RETURNING id\n )\n SELECT\n EXISTS (SELECT 1 FROM mine) AS \"had_row!\",\n (SELECT consumed_by FROM mine) AS prev_consumed_by,\n ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\",\n EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "had_row!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "prev_consumed_by", + "type_info": "Uuid" + }, + { + "ordinal": 2, + "name": "claimed_ids!", + "type_info": "UuidArray" + }, + { + "ordinal": 3, + "name": "claimed_self!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a565b2d34b3ca1d513d3fed2e23e8687718879f7e746a78675f121eb9511f6be" +} diff --git a/backend/.sqlx/query-a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10.json b/backend/.sqlx/query-a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10.json deleted file mode 100644 index dab14d9f1d..0000000000 --- a/backend/.sqlx/query-a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10.json +++ /dev/null @@ -1,89 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "WITH active_users AS (SELECT distinct username as email FROM (SELECT username, timestamp, operation FROM audit_partitioned UNION ALL SELECT username, timestamp, operation FROM audit) AS a WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n authors as (SELECT distinct email FROM usr WHERE usr.operator IS false)\n SELECT email, email NOT IN (SELECT email FROM authors) as operator_only, login_type::text, verified, super_admin, devops, name, company, username, first_time_user, role_source, disabled\n FROM password\n WHERE email IN (SELECT email FROM active_users)\n ORDER BY super_admin DESC, devops DESC\n LIMIT $1 OFFSET $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "email", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "operator_only", - "type_info": "Bool" - }, - { - "ordinal": 2, - "name": "login_type", - "type_info": "Text" - }, - { - "ordinal": 3, - "name": "verified", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "super_admin", - "type_info": "Bool" - }, - { - "ordinal": 5, - "name": "devops", - "type_info": "Bool" - }, - { - "ordinal": 6, - "name": "name", - "type_info": "Varchar" - }, - { - "ordinal": 7, - "name": "company", - "type_info": "Varchar" - }, - { - "ordinal": 8, - "name": "username", - "type_info": "Varchar" - }, - { - "ordinal": 9, - "name": "first_time_user", - "type_info": "Bool" - }, - { - "ordinal": 10, - "name": "role_source", - "type_info": "Varchar" - }, - { - "ordinal": 11, - "name": "disabled", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8" - ] - }, - "nullable": [ - false, - null, - null, - false, - false, - false, - true, - true, - true, - false, - false, - false - ] - }, - "hash": "a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10" -} diff --git a/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json b/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json new file mode 100644 index 0000000000..c3cee83397 --- /dev/null +++ b/backend/.sqlx/query-a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "a7fe8a504f418f10e4d8e84879353e007344c7be4bb040e4b50265cd497f853b" +} diff --git a/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json new file mode 100644 index 0000000000..a604c85831 --- /dev/null +++ b/backend/.sqlx/query-a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n EXISTS(SELECT 1 FROM pg_proc WHERE proname = 'aurora_version') AS \"aurora!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'rds_superuser') AS \"rds!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname = 'cloudsqlsuperuser') AS \"cloudsql!\",\n EXISTS(SELECT 1 FROM pg_roles WHERE rolname IN ('azure_pg_admin', 'azuresu')) AS \"azure!\"\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "aurora!", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "rds!", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "cloudsql!", + "type_info": "Bool" + }, + { + "ordinal": 3, + "name": "azure!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "a90e3a1d7c7c0dfb422f44b0ed599f681f5630c024ae9a437a301f149636b0db" +} diff --git a/backend/.sqlx/query-a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b.json b/backend/.sqlx/query-a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b.json new file mode 100644 index 0000000000..d12d305b92 --- /dev/null +++ b/backend/.sqlx/query-a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT path AS \"path!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND ($2::text IS NULL OR path LIKE $2)\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "a9a99880d870266f474878dd6ef541df988da527d30f663ef6f764f0c3d70d4b" +} diff --git a/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json b/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json new file mode 100644 index 0000000000..04131f8f88 --- /dev/null +++ b/backend/.sqlx/query-ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2.json @@ -0,0 +1,60 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM asset\n WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = 'script'\n RETURNING kind AS \"kind!: AssetKind\", path,\n usage_access_type AS \"usage_access_type: AssetUsageAccessType\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "kind!: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "usage_access_type: AssetUsageAccessType", + "type_info": { + "Custom": { + "name": "asset_access_type", + "kind": { + "Enum": [ + "r", + "w", + "rw" + ] + } + } + } + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "ab18d8765d6795eaa8035a8ac902790ff61549c5dd76e5b5cfb14b110a98abf2" +} diff --git a/backend/.sqlx/query-abb36bfddf707c7897b3936e982725f973eeb6ba5abbd61416060fb7d675593b.json b/backend/.sqlx/query-abb36bfddf707c7897b3936e982725f973eeb6ba5abbd61416060fb7d675593b.json new file mode 100644 index 0000000000..7795458cef --- /dev/null +++ b/backend/.sqlx/query-abb36bfddf707c7897b3936e982725f973eeb6ba5abbd61416060fb7d675593b.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n substring(path from '^f/([^/]+)/') AS \"folder!\",\n COUNT(*) AS \"script_count!\"\n FROM script\n WHERE workspace_id = $1\n AND auto_kind = 'pipeline'\n AND archived = false\n AND deleted = false\n AND path LIKE 'f/%'\n GROUP BY substring(path from '^f/([^/]+)/')\n ORDER BY substring(path from '^f/([^/]+)/')\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "folder!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "script_count!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "abb36bfddf707c7897b3936e982725f973eeb6ba5abbd61416060fb7d675593b" +} diff --git a/backend/.sqlx/query-ae7adc583cdd3f876164ed60569ed531b05eaa17fccc599306eb1a96a65ee761.json b/backend/.sqlx/query-ae7adc583cdd3f876164ed60569ed531b05eaa17fccc599306eb1a96a65ee761.json deleted file mode 100644 index 4bb689c519..0000000000 --- a/backend/.sqlx/query-ae7adc583cdd3f876164ed60569ed531b05eaa17fccc599306eb1a96a65ee761.json +++ /dev/null @@ -1,34 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT\n (elem->>'installation_id')::bigint as installation_id,\n elem->>'account_id' as account_id,\n elem->>'github_base_url' as github_base_url\n FROM workspace_settings,\n LATERAL jsonb_array_elements(git_app_installations) AS elem\n WHERE workspace_id = $1\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "installation_id", - "type_info": "Int8" - }, - { - "ordinal": 1, - "name": "account_id", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "github_base_url", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - null, - null, - null - ] - }, - "hash": "ae7adc583cdd3f876164ed60569ed531b05eaa17fccc599306eb1a96a65ee761" -} diff --git a/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json b/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json new file mode 100644 index 0000000000..3413ddc87f --- /dev/null +++ b/backend/.sqlx/query-ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "e!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + null + ] + }, + "hash": "ae8c0d0397609cf275bcffb92a5014665d47fe95b9eee0e1785441b0b4497a3c" +} diff --git a/backend/.sqlx/query-afb0762c88d9232b79090f2e5966e78437a5e4d3b5e2341ec5f7725a28870270.json b/backend/.sqlx/query-afb0762c88d9232b79090f2e5966e78437a5e4d3b5e2341ec5f7725a28870270.json index f0c91daf16..b492b38f49 100644 --- a/backend/.sqlx/query-afb0762c88d9232b79090f2e5966e78437a5e4d3b5e2341ec5f7725a28870270.json +++ b/backend/.sqlx/query-afb0762c88d9232b79090f2e5966e78437a5e4d3b5e2341ec5f7725a28870270.json @@ -13,4 +13,4 @@ "nullable": [] }, "hash": "afb0762c88d9232b79090f2e5966e78437a5e4d3b5e2341ec5f7725a28870270" -} \ No newline at end of file +} diff --git a/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json b/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json new file mode 100644 index 0000000000..406cbdb176 --- /dev/null +++ b/backend/.sqlx/query-afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings)\n VALUES ($1, $2, $3, $4)\n ON CONFLICT (hash)\n DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8", + "Int8", + "Int8", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "afc1d827992477ce921afe6cbbf847d387b44c3fafc93e9619bb4710f9645e42" +} diff --git a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json b/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json index ca95a3bca8..3efa843923 100644 --- a/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json +++ b/backend/.sqlx/query-b3771b690c5966272b1f42c9965bb6a8f961c119516e4c33dc928cd3b4f4edbc.json @@ -165,7 +165,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-b474ae4401b3d4c95add2d3353eb66512801c20ad685e8ed544fe4b86601aaa8.json b/backend/.sqlx/query-b474ae4401b3d4c95add2d3353eb66512801c20ad685e8ed544fe4b86601aaa8.json deleted file mode 100644 index 1341337cc5..0000000000 --- a/backend/.sqlx/query-b474ae4401b3d4c95add2d3353eb66512801c20ad685e8ed544fe4b86601aaa8.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO draft\n (workspace_id, path, value, typ)\n VALUES ($1, $2, $3::text::json, $4)\n ON CONFLICT (workspace_id, path, typ) DO UPDATE SET value = EXCLUDED.value", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Text", - { - "Custom": { - "name": "draft_type", - "kind": { - "Enum": [ - "script", - "flow", - "app" - ] - } - } - } - ] - }, - "nullable": [] - }, - "hash": "b474ae4401b3d4c95add2d3353eb66512801c20ad685e8ed544fe4b86601aaa8" -} diff --git a/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json new file mode 100644 index 0000000000..dc82db7591 --- /dev/null +++ b/backend/.sqlx/query-b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT audit_logs_s3_oldest_inflight_ts() AS \"cutoff?\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "cutoff?", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "b4c8f7ee9b1d065e1ab34ffe0dcfb76daed7d908be80f664a8a30e9896ab1f7e" +} diff --git a/backend/.sqlx/query-b5860f6a7672a368d740dcd367a8d5ab98fa93e0382a57a698564695db6c40ac.json b/backend/.sqlx/query-b5860f6a7672a368d740dcd367a8d5ab98fa93e0382a57a698564695db6c40ac.json deleted file mode 100644 index a08c31e743..0000000000 --- a/backend/.sqlx/query-b5860f6a7672a368d740dcd367a8d5ab98fa93e0382a57a698564695db6c40ac.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT content FROM script WHERE path = $1 AND workspace_id = $2 AND archived = false ORDER BY created_at DESC LIMIT 1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "content", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Text", - "Text" - ] - }, - "nullable": [ - false - ] - }, - "hash": "b5860f6a7672a368d740dcd367a8d5ab98fa93e0382a57a698564695db6c40ac" -} diff --git a/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json b/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json new file mode 100644 index 0000000000..da7e3d5787 --- /dev/null +++ b/backend/.sqlx/query-b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d.json @@ -0,0 +1,64 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value as \"value!: sqlx::types::Json>\", created_at\n FROM draft\n WHERE workspace_id = $1\n AND path = $2\n AND typ = $3\n AND email IS NOT DISTINCT FROM $4", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value!: sqlx::types::Json>", + "type_info": "Json" + }, + { + "ordinal": 1, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "b6cfb752675a3f36975e6cc6c454267d67f58f9cbff0d164b4df45de885b0e9d" +} diff --git a/backend/.sqlx/query-b6f95b3fd1d0431d96d0409424dbb20cb3020f93454978d078082482b86cbebe.json b/backend/.sqlx/query-b6f95b3fd1d0431d96d0409424dbb20cb3020f93454978d078082482b86cbebe.json deleted file mode 100644 index 3c45fe92ba..0000000000 --- a/backend/.sqlx/query-b6f95b3fd1d0431d96d0409424dbb20cb3020f93454978d078082482b86cbebe.json +++ /dev/null @@ -1,27 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO flow (\n workspace_id, path, summary, description,\n dependency_job, lock_error_logs, draft_only, tag,\n dedicated_worker, visible_to_runner_only, on_behalf_of_email,\n ws_error_handler_muted,\n value, schema, edited_by, edited_at, labels\n ) VALUES (\n $1, $2, $3, $4,\n NULL, '', $5, $6,\n $7, $8, $9,\n $10,\n $11, $12::text::json, $13, now(), $14\n )", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Text", - "Text", - "Bool", - "Varchar", - "Bool", - "Bool", - "Text", - "Bool", - "Jsonb", - "Text", - "Varchar", - "TextArray" - ] - }, - "nullable": [] - }, - "hash": "b6f95b3fd1d0431d96d0409424dbb20cb3020f93454978d078082482b86cbebe" -} diff --git a/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json b/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json new file mode 100644 index 0000000000..5b0f0b319c --- /dev/null +++ b/backend/.sqlx/query-b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtextextended($1, 0::int8))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b876b16b660bd1c3799abadc2c16d0f58f49a45647525658acb909a494877238" +} diff --git a/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json b/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json new file mode 100644 index 0000000000..fc002b7074 --- /dev/null +++ b/backend/.sqlx/query-b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135.json @@ -0,0 +1,25 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH target AS (\n SELECT id FROM v2_job_queue WHERE id = $1 AND NOT running FOR UPDATE\n ), completed AS (\n INSERT INTO v2_job_completed\n (workspace_id, id, started_at, duration_ms, result,\n flow_status, workflow_as_code_status, status, worker)\n SELECT\n q.workspace_id, q.id, q.started_at,\n (EXTRACT('epoch' FROM now()) - EXTRACT('epoch' FROM COALESCE(q.started_at, now()))) * 1000,\n $3::text::jsonb,\n s.flow_status,\n s.workflow_as_code_status,\n 'skipped'::job_status,\n q.worker\n FROM v2_job_queue q\n LEFT JOIN v2_job_status s ON s.id = q.id\n WHERE q.id IN (SELECT id FROM target)\n ON CONFLICT (id) DO UPDATE SET status = EXCLUDED.status, result = EXCLUDED.result\n RETURNING 1 AS x\n ), _deleted AS (\n DELETE FROM v2_job_queue WHERE id IN (SELECT id FROM target)\n ), _logged AS (\n INSERT INTO job_logs (logs, job_id, workspace_id)\n SELECT $4, $1, $2 WHERE EXISTS (SELECT 1 FROM target)\n ON CONFLICT (job_id) DO UPDATE SET logs = concat(job_logs.logs, EXCLUDED.logs)\n )\n SELECT x FROM completed\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "x", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "b9ce07cda12e2540f428d4c8e5bbfa8763c03dfe3f0c74edf958667035fb5135" +} diff --git a/backend/.sqlx/query-ba7d01509662be936c000aab4ff60f57be839ae365e5cbbb4838a7b622d1771a.json b/backend/.sqlx/query-ba7d01509662be936c000aab4ff60f57be839ae365e5cbbb4838a7b622d1771a.json new file mode 100644 index 0000000000..b7752cc7cf --- /dev/null +++ b/backend/.sqlx/query-ba7d01509662be936c000aab4ff60f57be839ae365e5cbbb4838a7b622d1771a.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff\n (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes)\n VALUES ('wm-fork-test-workspace', 'test-workspace', 'f/shared/other', 'script', 0, 1, true)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "ba7d01509662be936c000aab4ff60f57be839ae365e5cbbb4838a7b622d1771a" +} diff --git a/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json b/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json index 045d470de5..f6ff25a4bf 100644 --- a/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json +++ b/backend/.sqlx/query-bcfa34cf80abea05f0c24883b9e77429c51e6166c414bcc5ce2e97fac25bcd77.json @@ -79,7 +79,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json index a8bb03e1b3..8dc66064dc 100644 --- a/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json +++ b/backend/.sqlx/query-be6d2c92a62b7b284651c45af809746147aa9b8d0a81642a7b7cb4738a0cad66.json @@ -110,7 +110,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json b/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json new file mode 100644 index 0000000000..22618ad999 --- /dev/null +++ b/backend/.sqlx/query-bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "bf5db4bba06a2af085577f36446bcb900ac113c14584fa933d2a94318e47a354" +} diff --git a/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json b/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json new file mode 100644 index 0000000000..a051b068a5 --- /dev/null +++ b/backend/.sqlx/query-bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1.json @@ -0,0 +1,63 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n VALUES ($1, $2, $3, $4, $5, 'script', $6) ON CONFLICT DO NOTHING\n RETURNING usage_access_type AS \"usage_access_type: AssetUsageAccessType\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "usage_access_type: AssetUsageAccessType", + "type_info": { + "Custom": { + "name": "asset_access_type", + "kind": { + "Enum": [ + "r", + "w", + "rw" + ] + } + } + } + } + ], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + { + "Custom": { + "name": "asset_access_type", + "kind": { + "Enum": [ + "r", + "w", + "rw" + ] + } + } + }, + "Varchar", + "Jsonb" + ] + }, + "nullable": [ + true + ] + }, + "hash": "bfb97d2f48157a1575b7b6f3e64e0d075701d541a44dd7a0b586d1f337ced5e1" +} diff --git a/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json b/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json new file mode 100644 index 0000000000..f9fbc7a58b --- /dev/null +++ b/backend/.sqlx/query-bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM flow_conversation_message WHERE job_id = ANY($1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "UuidArray" + ] + }, + "nullable": [] + }, + "hash": "bfdd60b42e32bd81e2d20b327462893147b4e5ff078531de36147d908132d636" +} diff --git a/backend/.sqlx/query-c06796e8647cf278c6e0809562aaaa872403890eae3e606f379343671c2fff02.json b/backend/.sqlx/query-c03260b15cc3efd932312aebf4989c59bd6c6f7bce52aa3707e923b32d9be3e2.json similarity index 51% rename from backend/.sqlx/query-c06796e8647cf278c6e0809562aaaa872403890eae3e606f379343671c2fff02.json rename to backend/.sqlx/query-c03260b15cc3efd932312aebf4989c59bd6c6f7bce52aa3707e923b32d9be3e2.json index 1b1a8da18c..78014926fa 100644 --- a/backend/.sqlx/query-c06796e8647cf278c6e0809562aaaa872403890eae3e606f379343671c2fff02.json +++ b/backend/.sqlx/query-c03260b15cc3efd932312aebf4989c59bd6c6f7bce52aa3707e923b32d9be3e2.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n UPDATE\n flow\n SET\n path = $1,\n summary = $2,\n description = $3,\n dependency_job = NULL,\n lock_error_logs = '',\n draft_only = NULL,\n tag = $4,\n dedicated_worker = $5,\n visible_to_runner_only = $6,\n on_behalf_of_email = $7,\n ws_error_handler_muted = $8,\n value = $9,\n schema = $10::text::json,\n edited_by = $11,\n edited_at = now(),\n labels = COALESCE($14, labels)\n WHERE\n path = $12 AND workspace_id = $13", + "query": "\n UPDATE\n flow\n SET\n path = $1,\n summary = $2,\n description = $3,\n dependency_job = NULL,\n lock_error_logs = '',\n tag = $4,\n dedicated_worker = $5,\n visible_to_runner_only = $6,\n on_behalf_of_email = $7,\n ws_error_handler_muted = $8,\n value = $9,\n schema = $10::text::json,\n edited_by = $11,\n edited_at = now(),\n labels = COALESCE($14, labels)\n WHERE\n path = $12 AND workspace_id = $13", "describe": { "columns": [], "parameters": { @@ -23,5 +23,5 @@ }, "nullable": [] }, - "hash": "c06796e8647cf278c6e0809562aaaa872403890eae3e606f379343671c2fff02" + "hash": "c03260b15cc3efd932312aebf4989c59bd6c6f7bce52aa3707e923b32d9be3e2" } diff --git a/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json new file mode 100644 index 0000000000..b2e218e511 --- /dev/null +++ b/backend/.sqlx/query-c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75.json @@ -0,0 +1,31 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "completed_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8", + "Int8", + "UuidArray", + "Timestamptz" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75" +} diff --git a/backend/.sqlx/query-c0952a5f8a3959630f124f1bf379207ccf02da21984730391eeb497f7146ba66.json b/backend/.sqlx/query-c0952a5f8a3959630f124f1bf379207ccf02da21984730391eeb497f7146ba66.json new file mode 100644 index 0000000000..db89d7bcfa --- /dev/null +++ b/backend/.sqlx/query-c0952a5f8a3959630f124f1bf379207ccf02da21984730391eeb497f7146ba66.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT args AS \"args: sqlx::types::Json\"\n FROM v2_job\n WHERE workspace_id = $1 AND runnable_path = $2 AND trigger_kind IS NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "args: sqlx::types::Json", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "c0952a5f8a3959630f124f1bf379207ccf02da21984730391eeb497f7146ba66" +} diff --git a/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json b/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json new file mode 100644 index 0000000000..2d1e21019e --- /dev/null +++ b/backend/.sqlx/query-c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text" + ] + }, + "nullable": [ + true + ] + }, + "hash": "c167db39eeed526449dc064eaeb26e648aa9455cb81bab86fd106f76537701ba" +} diff --git a/backend/.sqlx/query-c17c39add3f70218dbae38595a909d02cfabe7e0864af577df6968428ac448ef.json b/backend/.sqlx/query-c17c39add3f70218dbae38595a909d02cfabe7e0864af577df6968428ac448ef.json deleted file mode 100644 index 36be033396..0000000000 --- a/backend/.sqlx/query-c17c39add3f70218dbae38595a909d02cfabe7e0864af577df6968428ac448ef.json +++ /dev/null @@ -1,95 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "WITH active_users AS (SELECT distinct username as email FROM (SELECT username, timestamp, operation FROM audit_partitioned UNION ALL SELECT username, timestamp, operation FROM audit) AS a WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n authors as (SELECT distinct email FROM usr WHERE usr.operator IS false)\n SELECT email as \"email!\", (email NOT IN (SELECT email FROM authors)) as operator_only, login_type::text, verified as \"verified!\", super_admin as \"super_admin!\", devops as \"devops!\", name, company, username, first_time_user as \"first_time_user!\", role_source as \"role_source!\", disabled as \"disabled!\", NULL::text as workspace_id\n FROM password\n WHERE email IN (SELECT email FROM active_users)\n UNION ALL\n SELECT email as \"email!\", true as operator_only, 'service_account'::text as login_type, true as \"verified!\", false as \"super_admin!\", false as \"devops!\", NULL::text as name, NULL::text as company, username, false as \"first_time_user!\", 'service_account'::text as \"role_source!\", disabled as \"disabled!\", workspace_id\n FROM usr\n WHERE is_service_account IS true\n ORDER BY \"super_admin!\" DESC, \"devops!\" DESC\n LIMIT $1 OFFSET $2", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "email!", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "operator_only", - "type_info": "Bool" - }, - { - "ordinal": 2, - "name": "login_type", - "type_info": "Text" - }, - { - "ordinal": 3, - "name": "verified!", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "super_admin!", - "type_info": "Bool" - }, - { - "ordinal": 5, - "name": "devops!", - "type_info": "Bool" - }, - { - "ordinal": 6, - "name": "name", - "type_info": "Varchar" - }, - { - "ordinal": 7, - "name": "company", - "type_info": "Varchar" - }, - { - "ordinal": 8, - "name": "username", - "type_info": "Varchar" - }, - { - "ordinal": 9, - "name": "first_time_user!", - "type_info": "Bool" - }, - { - "ordinal": 10, - "name": "role_source!", - "type_info": "Varchar" - }, - { - "ordinal": 11, - "name": "disabled!", - "type_info": "Bool" - }, - { - "ordinal": 12, - "name": "workspace_id", - "type_info": "Text" - } - ], - "parameters": { - "Left": [ - "Int8", - "Int8" - ] - }, - "nullable": [ - null, - null, - null, - null, - null, - null, - null, - null, - null, - null, - null, - null, - null - ] - }, - "hash": "c17c39add3f70218dbae38595a909d02cfabe7e0864af577df6968428ac448ef" -} diff --git a/backend/.sqlx/query-c241ee7efe2cbb9024792f6dc67cde48c5517ab36acd543a1f0a6119c34ce453.json b/backend/.sqlx/query-c241ee7efe2cbb9024792f6dc67cde48c5517ab36acd543a1f0a6119c34ce453.json new file mode 100644 index 0000000000..65003984ac --- /dev/null +++ b/backend/.sqlx/query-c241ee7efe2cbb9024792f6dc67cde48c5517ab36acd543a1f0a6119c34ce453.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script' AND (email = $3 OR email IS NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "c241ee7efe2cbb9024792f6dc67cde48c5517ab36acd543a1f0a6119c34ce453" +} diff --git a/backend/.sqlx/query-c269f14ae9ae4e96eff9483eb84cccbdfa316e5db051f1d52085a2d5447c81ae.json b/backend/.sqlx/query-c269f14ae9ae4e96eff9483eb84cccbdfa316e5db051f1d52085a2d5447c81ae.json deleted file mode 100644 index e566e02ac4..0000000000 --- a/backend/.sqlx/query-c269f14ae9ae4e96eff9483eb84cccbdfa316e5db051f1d52085a2d5447c81ae.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at)\n SELECT workspace_id, REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\\1'), summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at\n FROM flow\n WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Text", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "c269f14ae9ae4e96eff9483eb84cccbdfa316e5db051f1d52085a2d5447c81ae" -} diff --git a/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json b/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json new file mode 100644 index 0000000000..7981dbc983 --- /dev/null +++ b/backend/.sqlx/query-c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3.json @@ -0,0 +1,111 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT asset_kind AS \"asset_kind: AssetKind\", asset_path, partition,\n status AS \"status: MaterializationStatus\", snapshot_id,\n row_count, job_id, materialized_at, error\n FROM materialized_partition\n WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3\n ORDER BY partition DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "asset_kind: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "asset_path", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "partition", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "status: MaterializationStatus", + "type_info": { + "Custom": { + "name": "materialization_status", + "kind": { + "Enum": [ + "running", + "materialized", + "failed" + ] + } + } + } + }, + { + "ordinal": 4, + "name": "snapshot_id", + "type_info": "Int8" + }, + { + "ordinal": 5, + "name": "row_count", + "type_info": "Int8" + }, + { + "ordinal": 6, + "name": "job_id", + "type_info": "Uuid" + }, + { + "ordinal": 7, + "name": "materialized_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 8, + "name": "error", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + }, + "Text" + ] + }, + "nullable": [ + false, + false, + false, + false, + true, + true, + true, + false, + true + ] + }, + "hash": "c28e066bf24263f9e3b48a2ecabdf037178ba246092dbc1d2b3816e7a9269dc3" +} diff --git a/backend/.sqlx/query-c2c4e95ef48bf05e971b0e8d8de5f1f775a8b526382d671c95a2eb8879e1d828.json b/backend/.sqlx/query-c2c4e95ef48bf05e971b0e8d8de5f1f775a8b526382d671c95a2eb8879e1d828.json new file mode 100644 index 0000000000..44e09f64d1 --- /dev/null +++ b/backend/.sqlx/query-c2c4e95ef48bf05e971b0e8d8de5f1f775a8b526382d671c95a2eb8879e1d828.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job SET args = coalesce(args, '{}'::jsonb) || jsonb_build_object('partition', $1::text) WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "c2c4e95ef48bf05e971b0e8d8de5f1f775a8b526382d671c95a2eb8879e1d828" +} diff --git a/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json b/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json new file mode 100644 index 0000000000..876465fb8d --- /dev/null +++ b/backend/.sqlx/query-c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT j.runnable_path AS \"runnable_path!\", j.kind::text AS \"kind!\",\n q.runnable_settings_handle\n FROM v2_job j JOIN v2_job_queue q ON q.id = j.id\n WHERE j.workspace_id = $1 AND j.trigger_kind = 'asset'\n ORDER BY j.runnable_path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "kind!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "runnable_settings_handle", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + null, + true + ] + }, + "hash": "c3e4ee8cb8d5f7065d4415a57e67bb20d36921599e2bf97c0193f5853df58f59" +} diff --git a/backend/.sqlx/query-c608cdc9cd2e41992fb86431faa15edc8e637d44031b83c01ef09a96ade472ae.json b/backend/.sqlx/query-c608cdc9cd2e41992fb86431faa15edc8e637d44031b83c01ef09a96ade472ae.json new file mode 100644 index 0000000000..247d4de14a --- /dev/null +++ b/backend/.sqlx/query-c608cdc9cd2e41992fb86431faa15edc8e637d44031b83c01ef09a96ade472ae.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT created_by AS \"created_by!\"\n FROM v2_job\n WHERE id = $1 AND workspace_id = $2 AND ($3::text[] IS NULL OR tag = ANY($3))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "created_by!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Text", + "TextArray" + ] + }, + "nullable": [ + false + ] + }, + "hash": "c608cdc9cd2e41992fb86431faa15edc8e637d44031b83c01ef09a96ade472ae" +} diff --git a/backend/.sqlx/query-cf8baf59f9e87058dbf2b2335c00529ca2731a3a00110709024efc80f5b25cc5.json b/backend/.sqlx/query-c61977907414d2336030fbab114ea03e4a1d1edb4470812cd7dd57c657fa287a.json similarity index 66% rename from backend/.sqlx/query-cf8baf59f9e87058dbf2b2335c00529ca2731a3a00110709024efc80f5b25cc5.json rename to backend/.sqlx/query-c61977907414d2336030fbab114ea03e4a1d1edb4470812cd7dd57c657fa287a.json index 425b4d8502..61c60afdc4 100644 --- a/backend/.sqlx/query-cf8baf59f9e87058dbf2b2335c00529ca2731a3a00110709024efc80f5b25cc5.json +++ b/backend/.sqlx/query-c61977907414d2336030fbab114ea03e4a1d1edb4470812cd7dd57c657fa287a.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms, draft_only, custom_path)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8)\n RETURNING id", + "query": "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms, custom_path)\n VALUES ($1, $2, $3, $4, $5, $6, $7)\n RETURNING id", "describe": { "columns": [ { @@ -17,7 +17,6 @@ "Jsonb", "Int8Array", "Jsonb", - "Bool", "Text" ] }, @@ -25,5 +24,5 @@ false ] }, - "hash": "cf8baf59f9e87058dbf2b2335c00529ca2731a3a00110709024efc80f5b25cc5" + "hash": "c61977907414d2336030fbab114ea03e4a1d1edb4470812cd7dd57c657fa287a" } diff --git a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json b/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json deleted file mode 100644 index a68387f905..0000000000 --- a/backend/.sqlx/query-c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b.json +++ /dev/null @@ -1,20 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM job_perms\nWHERE job_id NOT IN (SELECT id FROM v2_job_queue)\nRETURNING job_id", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "job_id", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - false - ] - }, - "hash": "c825fa5c6e287068aeaad994c0b42b8ad59b9129f032c6b918c27426ab304f2b" -} diff --git a/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json b/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json new file mode 100644 index 0000000000..ca39442f59 --- /dev/null +++ b/backend/.sqlx/query-c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "description", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "c84087a0669d0b71829b0765c7274ca0a03fb823a781fb46d2b2b6cfc535a16b" +} diff --git a/backend/.sqlx/query-c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed.json b/backend/.sqlx/query-c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed.json new file mode 100644 index 0000000000..7361b645b5 --- /dev/null +++ b/backend/.sqlx/query-c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES\n ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'),\n ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'),\n ($3, nextval('debounce_batch_seq'), NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "c886e8af0fc8a3999a813371855c0053571e79960280f0714616d13a456d7bed" +} diff --git a/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json b/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json new file mode 100644 index 0000000000..161b9d36f9 --- /dev/null +++ b/backend/.sqlx/query-c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7.json @@ -0,0 +1,62 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO draft (workspace_id, email, path, typ, value, created_at)\n VALUES ($1, $2, $3, $4, $5::text::json, COALESCE($8::timestamptz, now()))\n ON CONFLICT (workspace_id, path, typ, email) WHERE email IS NOT NULL\n DO UPDATE SET value = EXCLUDED.value, created_at = EXCLUDED.created_at\n WHERE $7::bool = true\n OR $6::timestamptz IS NULL\n OR draft.created_at <= $6::timestamptz\n RETURNING created_at", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + }, + "Text", + "Timestamptz", + "Bool", + "Timestamptz" + ] + }, + "nullable": [ + false + ] + }, + "hash": "c8fb2a1491f90951a6d2e4e9c56d288eb60f6c6644a73cdf949de0159215a7f7" +} diff --git a/backend/.sqlx/query-ca1a39a56d36802418048ddad1301f16394c719ef55494b5f286bc402292c420.json b/backend/.sqlx/query-ca1a39a56d36802418048ddad1301f16394c719ef55494b5f286bc402292c420.json new file mode 100644 index 0000000000..5afd4065ae --- /dev/null +++ b/backend/.sqlx/query-ca1a39a56d36802418048ddad1301f16394c719ef55494b5f286bc402292c420.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO workspace_diff\n (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork)\n VALUES ('test-workspace', 'wm-fork-test-workspace', 'f/shared/leaky', 'script', 1, 0, true, true, true)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "ca1a39a56d36802418048ddad1301f16394c719ef55494b5f286bc402292c420" +} diff --git a/backend/.sqlx/query-39a72ff9bd2ab9bdf59a73ea32821645ce8b1bbcaccee13fb4f2c0eed28a6096.json b/backend/.sqlx/query-cbe8fb4935908a7eb9a0b56b1d6f330cd3c8ef1ca692147210a36e56946f7ef6.json similarity index 67% rename from backend/.sqlx/query-39a72ff9bd2ab9bdf59a73ea32821645ce8b1bbcaccee13fb4f2c0eed28a6096.json rename to backend/.sqlx/query-cbe8fb4935908a7eb9a0b56b1d6f330cd3c8ef1ca692147210a36e56946f7ef6.json index 60bb866c1c..6d682c7bb7 100644 --- a/backend/.sqlx/query-39a72ff9bd2ab9bdf59a73ea32821645ce8b1bbcaccee13fb4f2c0eed28a6096.json +++ b/backend/.sqlx/query-cbe8fb4935908a7eb9a0b56b1d6f330cd3c8ef1ca692147210a36e56946f7ef6.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", + "query": "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ IN ('app', 'raw_app')", "describe": { "columns": [], "parameters": { @@ -11,5 +11,5 @@ }, "nullable": [] }, - "hash": "39a72ff9bd2ab9bdf59a73ea32821645ce8b1bbcaccee13fb4f2c0eed28a6096" + "hash": "cbe8fb4935908a7eb9a0b56b1d6f330cd3c8ef1ca692147210a36e56946f7ef6" } diff --git a/backend/.sqlx/query-cd6a2559d76e7c6462bd18c8f0bc07c31ca9bdd96f8f92ba19d90e6060721354.json b/backend/.sqlx/query-cd6a2559d76e7c6462bd18c8f0bc07c31ca9bdd96f8f92ba19d90e6060721354.json deleted file mode 100644 index f11e9daf54..0000000000 --- a/backend/.sqlx/query-cd6a2559d76e7c6462bd18c8f0bc07c31ca9bdd96f8f92ba19d90e6060721354.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE v2_job_status\n SET flow_status = JSONB_SET(flow_status, ARRAY['modules', flow_status->>'step', 'progress'], $1)\n WHERE id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Jsonb", - "Uuid" - ] - }, - "nullable": [] - }, - "hash": "cd6a2559d76e7c6462bd18c8f0bc07c31ca9bdd96f8f92ba19d90e6060721354" -} diff --git a/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json new file mode 100644 index 0000000000..2c775af0b1 --- /dev/null +++ b/backend/.sqlx/query-cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO background_task_state (name, value)\n SELECT $1, jsonb_build_object(\n 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint,\n 'last_ts', now(),\n 'last_oldest_inflight_ts',\n COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))\n WHERE NOT EXISTS (SELECT 1 FROM global_settings WHERE name = $2)\n ON CONFLICT (name) DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "cd7c651f33629af0eb74362525395eadc68642592e3c56d4bf0b89b6440869b1" +} diff --git a/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json new file mode 100644 index 0000000000..4364da5dd0 --- /dev/null +++ b/backend/.sqlx/query-d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM job_result_stream_v2\n WHERE ctid IN (\n SELECT jrs.ctid FROM job_result_stream_v2 jrs\n WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id)\n AND NOT EXISTS (\n SELECT 1 FROM v2_job_completed c\n WHERE c.id = jrs.job_id\n AND c.completed_at > NOW() - INTERVAL '60 seconds'\n )\n LIMIT 100000\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "d059b8a3771e4ac4cd07990ecca84daaed616dc1ac609a6ca81bcd446e4dc230" +} diff --git a/backend/.sqlx/query-d3d87b9a4d62977dea5af95dd457bb759229805cff771cfa4275a46bfc80e1ab.json b/backend/.sqlx/query-d3d87b9a4d62977dea5af95dd457bb759229805cff771cfa4275a46bfc80e1ab.json new file mode 100644 index 0000000000..b5d905b9b6 --- /dev/null +++ b/backend/.sqlx/query-d3d87b9a4d62977dea5af95dd457bb759229805cff771cfa4275a46bfc80e1ab.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE v2_job\n SET args = CASE\n WHEN args ? 'partition'\n THEN $1 || jsonb_build_object('partition', args -> 'partition')\n ELSE $1\n END,\n preprocessed = TRUE\n WHERE id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Jsonb", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "d3d87b9a4d62977dea5af95dd457bb759229805cff771cfa4275a46bfc80e1ab" +} diff --git a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json index 5ed2e53367..d5365ffe94 100644 --- a/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json +++ b/backend/.sqlx/query-d41ea93fd58381b89e151c965eae1ea2fe96a1b94f5a92953fb1c1642d15c016.json @@ -110,7 +110,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json b/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json index a27bea8b2a..33a5534b42 100644 --- a/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json +++ b/backend/.sqlx/query-d4211392e174a0e8f89c7fcebdf120e5b0f629f9f04e08a2982df33ff23ac7a9.json @@ -250,7 +250,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json b/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json new file mode 100644 index 0000000000..e60f5cc187 --- /dev/null +++ b/backend/.sqlx/query-d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596.json @@ -0,0 +1,64 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value as \"value!: sqlx::types::Json>\",\n created_at\n FROM draft\n WHERE workspace_id = $1\n AND (email = $2 OR email IS NULL)\n AND path = $3\n AND typ = $4\n ORDER BY email NULLS LAST\n LIMIT 1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value!: sqlx::types::Json>", + "type_info": "Json" + }, + { + "ordinal": 1, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + { + "Custom": { + "name": "draft_kind", + "kind": { + "Enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + } + } + } + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "d4e0da9f9653d532770066310f85e59e5edda1faea72f39603814afb6a3cd596" +} diff --git a/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json b/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json new file mode 100644 index 0000000000..8c2a7716fd --- /dev/null +++ b/backend/.sqlx/query-d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT (status = 'success' OR EXISTS (\n SELECT 1 FROM native_retry_attempt nra\n JOIN v2_job jc ON jc.id = nra.job_id\n JOIN v2_job_completed cc ON cc.id = nra.job_id\n WHERE jc.parent_job = j.id AND cc.status = 'success'\n )) AS \"success!\"\n FROM v2_job j JOIN v2_job_completed USING (id)\n WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2\n AND parent_job IS NULL\n AND runnable_path = $3\n AND j.id != $4\n ORDER BY created_at DESC\n LIMIT $5", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "success!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text", + "Uuid", + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "d537182f8c8931427f2cf175bd850ce2581a33c8678212de99edb6f823f38994" +} diff --git a/backend/.sqlx/query-d6a8ab57341b8aa17d4d50ce7a710c923df7b66d1b76d36229299d2866775ffc.json b/backend/.sqlx/query-d6a8ab57341b8aa17d4d50ce7a710c923df7b66d1b76d36229299d2866775ffc.json new file mode 100644 index 0000000000..befd9903a9 --- /dev/null +++ b/backend/.sqlx/query-d6a8ab57341b8aa17d4d50ce7a710c923df7b66d1b76d36229299d2866775ffc.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT runnable_path AS \"runnable_path!\", trigger,\n args AS \"args: sqlx::types::Json\"\n FROM v2_job\n WHERE workspace_id = $1 AND trigger_kind = 'asset'\n ORDER BY runnable_path", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "runnable_path!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "trigger", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "args: sqlx::types::Json", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + true, + true + ] + }, + "hash": "d6a8ab57341b8aa17d4d50ce7a710c923df7b66d1b76d36229299d2866775ffc" +} diff --git a/backend/.sqlx/query-d8e293c0c8fd7e329b921822ffab8d305bd9ad73858e8f7b4269097b5ef4cf73.json b/backend/.sqlx/query-d8e293c0c8fd7e329b921822ffab8d305bd9ad73858e8f7b4269097b5ef4cf73.json deleted file mode 100644 index 7447facfe7..0000000000 --- a/backend/.sqlx/query-d8e293c0c8fd7e329b921822ffab8d305bd9ad73858e8f7b4269097b5ef4cf73.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "DELETE FROM resource WHERE path = ANY($1) AND workspace_id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "TextArray", - "Text" - ] - }, - "nullable": [] - }, - "hash": "d8e293c0c8fd7e329b921822ffab8d305bd9ad73858e8f7b4269097b5ef4cf73" -} diff --git a/backend/.sqlx/query-dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9.json b/backend/.sqlx/query-dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9.json new file mode 100644 index 0000000000..6ae3f60e49 --- /dev/null +++ b/backend/.sqlx/query-dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT email FROM usr WHERE workspace_id = $1 AND username = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "email", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "dadf78bae0299b24f6798e7a2cc86f0ecfcf63daa0b185994a410eb2fe41fad9" +} diff --git a/backend/.sqlx/query-e4836a1ee97e4723bddc28b94e23a54b6aa1f458f3ecca8f35432153e54b143d.json b/backend/.sqlx/query-ddcc3f07ff87b9ea47a9428ba0bb7e3239773fa3e6352aef7e98467ad07f337d.json similarity index 65% rename from backend/.sqlx/query-e4836a1ee97e4723bddc28b94e23a54b6aa1f458f3ecca8f35432153e54b143d.json rename to backend/.sqlx/query-ddcc3f07ff87b9ea47a9428ba0bb7e3239773fa3e6352aef7e98467ad07f337d.json index b101845b0e..35e384b496 100644 --- a/backend/.sqlx/query-e4836a1ee97e4723bddc28b94e23a54b6aa1f458f3ecca8f35432153e54b143d.json +++ b/backend/.sqlx/query-ddcc3f07ff87b9ea47a9428ba0bb7e3239773fa3e6352aef7e98467ad07f337d.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO app\n (workspace_id, path, summary, policy, versions, draft_only, custom_path, labels)\n VALUES ($1, $2, $3, $4, '{}', $5, $6, $7) RETURNING id", + "query": "INSERT INTO app\n (workspace_id, path, summary, policy, versions, custom_path, labels)\n VALUES ($1, $2, $3, $4, '{}', $5, $6) RETURNING id", "describe": { "columns": [ { @@ -15,7 +15,6 @@ "Varchar", "Varchar", "Jsonb", - "Bool", "Text", "TextArray" ] @@ -24,5 +23,5 @@ false ] }, - "hash": "e4836a1ee97e4723bddc28b94e23a54b6aa1f458f3ecca8f35432153e54b143d" + "hash": "ddcc3f07ff87b9ea47a9428ba0bb7e3239773fa3e6352aef7e98467ad07f337d" } diff --git a/backend/.sqlx/query-ddda19024473b4e5b1d450bf56c667056f6b3696dea7d73758e53f825bfce13e.json b/backend/.sqlx/query-ddda19024473b4e5b1d450bf56c667056f6b3696dea7d73758e53f825bfce13e.json deleted file mode 100644 index be8864a85d..0000000000 --- a/backend/.sqlx/query-ddda19024473b4e5b1d450bf56c667056f6b3696dea7d73758e53f825bfce13e.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO flow\n (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels)\n SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels\n FROM flow\n WHERE path = $2 AND workspace_id = $3", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Text", - "Text" - ] - }, - "nullable": [] - }, - "hash": "ddda19024473b4e5b1d450bf56c667056f6b3696dea7d73758e53f825bfce13e" -} diff --git a/backend/.sqlx/query-de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b.json b/backend/.sqlx/query-de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b.json new file mode 100644 index 0000000000..2ba317edc6 --- /dev/null +++ b/backend/.sqlx/query-de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b.json @@ -0,0 +1,43 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n kind AS \"kind!: AssetKind\",\n path AS \"path!\"\n FROM asset\n WHERE workspace_id = $1\n AND usage_kind = 'script'\n AND usage_path = $2\n AND usage_access_type IN ('w', 'rw')\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "kind!: AssetKind", + "type_info": { + "Custom": { + "name": "asset_kind", + "kind": { + "Enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + } + } + } + }, + { + "ordinal": 1, + "name": "path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "de06f44bad94710f14e9be4c0a6e6080e3c4faae5052500b93cc24b6fe556f2b" +} diff --git a/backend/.sqlx/query-a9e29764b5b9d94269e2b8aa755c71b61774c8ff8ae218d7a8d6ed0ac0169366.json b/backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json similarity index 64% rename from backend/.sqlx/query-a9e29764b5b9d94269e2b8aa755c71b61774c8ff8ae218d7a8d6ed0ac0169366.json rename to backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json index ef3968cdfa..037c796fc7 100644 --- a/backend/.sqlx/query-a9e29764b5b9d94269e2b8aa755c71b61774c8ff8ae218d7a8d6ed0ac0169366.json +++ b/backend/.sqlx/query-e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING", + "query": "WITH ins AS (\n INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns)\n VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING\n RETURNING usage_kind, usage_access_type\n )\n INSERT INTO notify_event (channel, payload)\n SELECT 'notify_asset_producer_change', $1\n FROM ins WHERE usage_kind = 'script' AND usage_access_type IN ('w', 'rw')", "describe": { "columns": [], "parameters": { @@ -52,5 +52,5 @@ }, "nullable": [] }, - "hash": "a9e29764b5b9d94269e2b8aa755c71b61774c8ff8ae218d7a8d6ed0ac0169366" + "hash": "e041b20c1c4166b30ced8c6a0f50bcf0691ab2554ead6b489a8441a32fc0af82" } diff --git a/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json b/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json new file mode 100644 index 0000000000..87cd891113 --- /dev/null +++ b/backend/.sqlx/query-e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = (\n SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "e050734d7642b26f8859982c55ec2c8b1fc14a8de665b15a2f2dfd6e0b5b7fdf" +} diff --git a/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json new file mode 100644 index 0000000000..20ae255eb1 --- /dev/null +++ b/backend/.sqlx/query-e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.path = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null, + null, + false + ] + }, + "hash": "e0a40d2aba02bd6c502d746471c9c14db8fcaaaf8e3c44fb5ea4ed763a1849dd" +} diff --git a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json index 1a4cc407e0..470c651020 100644 --- a/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json +++ b/backend/.sqlx/query-e4d71278fb80126a7a9da73f1889352d4d1e3cb3a8a08f1c9c03055a1cab1235.json @@ -190,7 +190,8 @@ "google", "ci_test", "github", - "azure" + "azure", + "asset" ] } } diff --git a/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json b/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json new file mode 100644 index 0000000000..121d7fe04a --- /dev/null +++ b/backend/.sqlx/query-e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "description", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "instructions", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "e50afd5156b07e550202fb9b33354dce71b37f89f68d78577b250979daa1a87d" +} diff --git a/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json b/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json new file mode 100644 index 0000000000..29ce0c6f2c --- /dev/null +++ b/backend/.sqlx/query-e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "name", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "e99fe5cd3283f1701d3a361ef31869da89fd10099b76669b9526201c85f71f61" +} diff --git a/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json b/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json new file mode 100644 index 0000000000..f236c2aae0 --- /dev/null +++ b/backend/.sqlx/query-ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2.json @@ -0,0 +1,17 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES\n ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'),\n ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'),\n ($3, nextval('debounce_batch_seq'), NULL),\n ($4, nextval('debounce_batch_seq'), now() - interval '20 minutes')", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Uuid", + "Uuid", + "Uuid", + "Uuid" + ] + }, + "nullable": [] + }, + "hash": "ea277c48e7966ab01d9b18e3c0e357033b999d8f3c23241e4c4053cc573f6ca2" +} diff --git a/backend/.sqlx/query-eb6f237cdaea4581cd839c353d8acd868f9101250b0599fc66798a7852c7cdf5.json b/backend/.sqlx/query-eb6f237cdaea4581cd839c353d8acd868f9101250b0599fc66798a7852c7cdf5.json new file mode 100644 index 0000000000..879288cc04 --- /dev/null +++ b/backend/.sqlx/query-eb6f237cdaea4581cd839c353d8acd868f9101250b0599fc66798a7852c7cdf5.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO flow (\n workspace_id, path, summary, description,\n dependency_job, lock_error_logs, tag,\n dedicated_worker, visible_to_runner_only, on_behalf_of_email,\n ws_error_handler_muted,\n value, schema, edited_by, edited_at, labels\n ) VALUES (\n $1, $2, $3, $4,\n NULL, '', $5,\n $6, $7, $8,\n $9,\n $10, $11::text::json, $12, now(), $13\n )", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Text", + "Varchar", + "Bool", + "Bool", + "Text", + "Bool", + "Jsonb", + "Text", + "Varchar", + "TextArray" + ] + }, + "nullable": [] + }, + "hash": "eb6f237cdaea4581cd839c353d8acd868f9101250b0599fc66798a7852c7cdf5" +} diff --git a/backend/.sqlx/query-ecce519d0cf0c31df4612e0ccd8d62eef3ab3b920665ca5d6f9ce0ef89e53fb3.json b/backend/.sqlx/query-ecce519d0cf0c31df4612e0ccd8d62eef3ab3b920665ca5d6f9ce0ef89e53fb3.json new file mode 100644 index 0000000000..d40882073d --- /dev/null +++ b/backend/.sqlx/query-ecce519d0cf0c31df4612e0ccd8d62eef3ab3b920665ca5d6f9ce0ef89e53fb3.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT path AS \"path!\" FROM script\n WHERE workspace_id = $1\n AND archived = false\n AND deleted = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "ecce519d0cf0c31df4612e0ccd8d62eef3ab3b920665ca5d6f9ce0ef89e53fb3" +} diff --git a/backend/.sqlx/query-9517395ac7230ab7c40c03ddd2a95fd6118b329a4421c9e8022df90ff7e775c8.json b/backend/.sqlx/query-ed8351ccac2df2a4bd327383003d86af3e3f0b9ab326e4ea89267514ecc66f71.json similarity index 73% rename from backend/.sqlx/query-9517395ac7230ab7c40c03ddd2a95fd6118b329a4421c9e8022df90ff7e775c8.json rename to backend/.sqlx/query-ed8351ccac2df2a4bd327383003d86af3e3f0b9ab326e4ea89267514ecc66f71.json index cc551bed42..7ac1fb17da 100644 --- a/backend/.sqlx/query-9517395ac7230ab7c40c03ddd2a95fd6118b329a4421c9e8022df90ff7e775c8.json +++ b/backend/.sqlx/query-ed8351ccac2df2a4bd327383003d86af3e3f0b9ab326e4ea89267514ecc66f71.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT COUNT(*) FROM script s WHERE s.workspace_id = $1 AND s.hash NOT IN (\n SELECT DISTINCT ON (path) hash FROM script\n WHERE workspace_id = $1 AND deleted = false AND draft_only IS NOT TRUE\n ORDER BY path, created_at DESC\n )", + "query": "SELECT COUNT(*) FROM script s WHERE s.workspace_id = $1 AND s.hash NOT IN (\n SELECT DISTINCT ON (path) hash FROM script\n WHERE workspace_id = $1 AND deleted = false\n ORDER BY path, created_at DESC\n )", "describe": { "columns": [ { @@ -18,5 +18,5 @@ null ] }, - "hash": "9517395ac7230ab7c40c03ddd2a95fd6118b329a4421c9e8022df90ff7e775c8" + "hash": "ed8351ccac2df2a4bd327383003d86af3e3f0b9ab326e4ea89267514ecc66f71" } diff --git a/backend/.sqlx/query-eeaad5c2284c1856cfc64ae0bc0dfc79b283c06987a6de399b6c1aaca94a2b7a.json b/backend/.sqlx/query-eeaad5c2284c1856cfc64ae0bc0dfc79b283c06987a6de399b6c1aaca94a2b7a.json new file mode 100644 index 0000000000..169fe55e62 --- /dev/null +++ b/backend/.sqlx/query-eeaad5c2284c1856cfc64ae0bc0dfc79b283c06987a6de399b6c1aaca94a2b7a.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO notify_event (channel, payload)\n VALUES ('notify_asset_producer_change', $1)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "eeaad5c2284c1856cfc64ae0bc0dfc79b283c06987a6de399b6c1aaca94a2b7a" +} diff --git a/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json new file mode 100644 index 0000000000..dd7f4da810 --- /dev/null +++ b/backend/.sqlx/query-ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db.json @@ -0,0 +1,41 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app\n FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)]\n WHERE a.id = $1 AND a.workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "policy", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "version", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "raw_app", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Int8", + "Text" + ] + }, + "nullable": [ + false, + null, + null, + false + ] + }, + "hash": "ef9caf3da759ee6059922632cdf1fdf5c006554a70a322ca8d3449cdba7840db" +} diff --git a/backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json b/backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json similarity index 55% rename from backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json rename to backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json index 4b96e7005b..84a2320a80 100644 --- a/backend/.sqlx/query-d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a.json +++ b/backend/.sqlx/query-f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3.json @@ -1,15 +1,23 @@ { "db_name": "PostgreSQL", - "query": "WITH to_delete AS (\n SELECT id FROM v2_job_queue\n JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = $1\n AND j.workspace_id = $2\n AND flow_step_id IS NULL\n AND running = false\n FOR UPDATE\n ), deleted AS (\n DELETE FROM v2_job_queue\n WHERE id IN (SELECT id FROM to_delete)\n RETURNING id\n ) DELETE FROM v2_job WHERE id IN (SELECT id FROM deleted)", + "query": "WITH to_delete AS (\n SELECT id FROM v2_job_queue\n JOIN v2_job j USING (id)\n WHERE trigger_kind = 'schedule'\n AND trigger = $1\n AND j.workspace_id = $2\n AND flow_step_id IS NULL\n AND running = false\n FOR UPDATE\n )\n DELETE FROM v2_job_queue\n WHERE id IN (SELECT id FROM to_delete)\n RETURNING id", "describe": { - "columns": [], + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + } + ], "parameters": { "Left": [ "Text", "Text" ] }, - "nullable": [] + "nullable": [ + false + ] }, - "hash": "d2a9e6a31bab0551d32093b1afe4e5d414cf439e4db3325b5bf6bbeb86c5bd2a" + "hash": "f0213dffe64fb16ee2d1f6bf5a37dc373175ae0ad54a0a49dca011f7cef4c5c3" } diff --git a/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json b/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json new file mode 100644 index 0000000000..1039dd3950 --- /dev/null +++ b/backend/.sqlx/query-f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN NULL ELSE debounce_key.job_id END,\n job_id = EXCLUDED.job_id,\n debounced_times = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN 0 ELSE debounce_key.debounced_times + 1 END,\n first_started_at = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN now() ELSE debounce_key.first_started_at END\n RETURNING debounced_times, first_started_at, previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT $1, COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq'))\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "debounced_times", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "first_started_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "job_id_to_debounce", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "f04f84ac63e98566311245a40a8dd83166a32ae4370a4f1ca622b8535c930ea2" +} diff --git a/backend/.sqlx/query-f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493.json b/backend/.sqlx/query-f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493.json deleted file mode 100644 index c1d113ee27..0000000000 --- a/backend/.sqlx/query-f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493.json +++ /dev/null @@ -1,88 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password WHERE email = $1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "email", - "type_info": "Varchar" - }, - { - "ordinal": 1, - "name": "login_type", - "type_info": "Text" - }, - { - "ordinal": 2, - "name": "super_admin", - "type_info": "Bool" - }, - { - "ordinal": 3, - "name": "devops", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "verified", - "type_info": "Bool" - }, - { - "ordinal": 5, - "name": "name", - "type_info": "Varchar" - }, - { - "ordinal": 6, - "name": "company", - "type_info": "Varchar" - }, - { - "ordinal": 7, - "name": "username", - "type_info": "Varchar" - }, - { - "ordinal": 8, - "name": "operator_only", - "type_info": "Bool" - }, - { - "ordinal": 9, - "name": "first_time_user", - "type_info": "Bool" - }, - { - "ordinal": 10, - "name": "role_source", - "type_info": "Varchar" - }, - { - "ordinal": 11, - "name": "disabled", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - false, - null, - false, - false, - false, - true, - true, - true, - null, - false, - false, - false - ] - }, - "hash": "f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493" -} diff --git a/backend/.sqlx/query-f0fcc355334f4d715b366e1b3be88b2b5e316536efae31da09217176b59db030.json b/backend/.sqlx/query-f0fcc355334f4d715b366e1b3be88b2b5e316536efae31da09217176b59db030.json deleted file mode 100644 index bddbe43ad8..0000000000 --- a/backend/.sqlx/query-f0fcc355334f4d715b366e1b3be88b2b5e316536efae31da09217176b59db030.json +++ /dev/null @@ -1,15 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO script (\n workspace_id, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n )\n SELECT\n $1, hash, path, parent_hashes, summary, description, content,\n created_by, created_at, archived, schema, deleted, is_template,\n extra_perms, lock, lock_error_logs, language, kind, tag, draft_only,\n envs, concurrent_limit, concurrency_time_window_s, cache_ttl,\n dedicated_worker, ws_error_handler_muted, priority, timeout,\n delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key,\n visible_to_runner_only, auto_kind, codebase, has_preprocessor,\n on_behalf_of_email, assets, modules\n FROM script\n WHERE workspace_id = $2", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Text" - ] - }, - "nullable": [] - }, - "hash": "f0fcc355334f4d715b366e1b3be88b2b5e316536efae31da09217176b59db030" -} diff --git a/backend/.sqlx/query-f175f0eda0dcdb26c08b743de80e73344dff5b98a33daaee144ffaccaa8a0bad.json b/backend/.sqlx/query-f175f0eda0dcdb26c08b743de80e73344dff5b98a33daaee144ffaccaa8a0bad.json deleted file mode 100644 index 2393837f07..0000000000 --- a/backend/.sqlx/query-f175f0eda0dcdb26c08b743de80e73344dff5b98a33daaee144ffaccaa8a0bad.json +++ /dev/null @@ -1,22 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n SELECT path AS \"path!\" FROM (\n (SELECT DISTINCT path FROM script WHERE workspace_id = $1 AND archived = false AND deleted = false AND draft_only IS NOT true LIMIT 5000)\n UNION\n (SELECT path FROM flow WHERE workspace_id = $1 AND archived = false AND draft_only IS NOT true LIMIT 5000)\n UNION\n (SELECT path FROM app WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM raw_app WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM variable WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM resource WHERE workspace_id = $1 LIMIT 5000)\n ) t\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "path!", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Text" - ] - }, - "nullable": [ - null - ] - }, - "hash": "f175f0eda0dcdb26c08b743de80e73344dff5b98a33daaee144ffaccaa8a0bad" -} diff --git a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json similarity index 64% rename from backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json rename to backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json index af4ada67f2..84eb4caba9 100644 --- a/backend/.sqlx/query-a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d.json +++ b/backend/.sqlx/query-f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = 'anonymous'\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", + "query": "SELECT EXISTS (\n SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id)\n WHERE j.workspace_id = $2\n AND (j.kind = 'appscript' OR j.kind = 'preview')\n AND j.created_by = $4\n AND c.started_at > now() - interval '3 hours'\n AND j.runnable_path LIKE $3 || '/%'\n AND c.result @> ('{\"s3\":\"' || $1 || '\"}')::jsonb\n )", "describe": { "columns": [ { @@ -11,6 +11,7 @@ ], "parameters": { "Left": [ + "Text", "Text", "Text", "Text" @@ -20,5 +21,5 @@ null ] }, - "hash": "a72e7fb55d7268fe1ea40015a4a84b12dd961ba732772d8f6c59d18fe05f285d" + "hash": "f2760b688a907e7679106aaa2c2063385785f7c2e97364bc04392df11cf364d7" } diff --git a/backend/.sqlx/query-f338943aa3595c2893a0d42e4a54e8dd1370fb59c0a5cfd2403c5fdaf2135cf2.json b/backend/.sqlx/query-f338943aa3595c2893a0d42e4a54e8dd1370fb59c0a5cfd2403c5fdaf2135cf2.json new file mode 100644 index 0000000000..dd1db109da --- /dev/null +++ b/backend/.sqlx/query-f338943aa3595c2893a0d42e4a54e8dd1370fb59c0a5cfd2403c5fdaf2135cf2.json @@ -0,0 +1,51 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO script_trigger\n (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all,\n debounce_s, retry_count, retry_delay_s)\n VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + { + "Custom": { + "name": "asset_usage_kind", + "kind": { + "Enum": [ + "script", + "flow", + "job" + ] + } + } + }, + "Varchar", + { + "Custom": { + "name": "script_trigger_kind", + "kind": { + "Enum": [ + "asset", + "schedule", + "webhook", + "email", + "kafka", + "mqtt", + "nats", + "postgres", + "sqs", + "gcp" + ] + } + } + }, + "Text", + "Bool", + "Int4", + "Int2", + "Int4" + ] + }, + "nullable": [] + }, + "hash": "f338943aa3595c2893a0d42e4a54e8dd1370fb59c0a5cfd2403c5fdaf2135cf2" +} diff --git a/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json b/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json new file mode 100644 index 0000000000..81b34464f4 --- /dev/null +++ b/backend/.sqlx/query-f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN NULL ELSE debounce_key.job_id END,\n job_id = EXCLUDED.job_id,\n debounced_times = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN 0 ELSE debounce_key.debounced_times + 1 END,\n first_started_at = CASE WHEN EXISTS\n (SELECT 1 FROM v2_job_queue q WHERE q.id = debounce_key.job_id AND q.running)\n THEN now() ELSE debounce_key.first_started_at END\n RETURNING debounced_times, first_started_at, previous_job_id AS job_id_to_debounce\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "debounced_times", + "type_info": "Int4" + }, + { + "ordinal": 1, + "name": "first_started_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 2, + "name": "job_id_to_debounce", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid", + "Varchar" + ] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "f44595adab2d127c83f9829b74429dcb7e5f2513bc59c2bdf7dd0be1864cd2c5" +} diff --git a/backend/.sqlx/query-f57c70d1f6756d8df09d3c26f6e4cbece8a8206a678271218c4d486a65818745.json b/backend/.sqlx/query-f57c70d1f6756d8df09d3c26f6e4cbece8a8206a678271218c4d486a65818745.json new file mode 100644 index 0000000000..394d36b747 --- /dev/null +++ b/backend/.sqlx/query-f57c70d1f6756d8df09d3c26f6e4cbece8a8206a678271218c4d486a65818745.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ IN ('app', 'raw_app') AND (email = $3 OR email IS NULL)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "f57c70d1f6756d8df09d3c26f6e4cbece8a8206a678271218c4d486a65818745" +} diff --git a/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json b/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json new file mode 100644 index 0000000000..132d1da5a1 --- /dev/null +++ b/backend/.sqlx/query-f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH ids AS (SELECT id FROM v2_job WHERE workspace_id = $1),\n _de AS (DELETE FROM dispatch_event WHERE workspace_id = $1),\n _fc AS (DELETE FROM flow_conversation_message WHERE job_id IN (SELECT id FROM ids))\n DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM ids)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "f68466d64684bf4f6542a345b9cff8553575fe3c17a060e271b381b40b4fb645" +} diff --git a/backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json b/backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json deleted file mode 100644 index 117a8bdc1b..0000000000 --- a/backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "INSERT INTO usr\n (workspace_id, email, username, is_admin, operator, is_service_account)\n VALUES ($1, $2, $3, false, true, true)", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Varchar" - ] - }, - "nullable": [] - }, - "hash": "f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853" -} diff --git a/backend/.sqlx/query-fac563138c316998d4f523edd633db97dba77d649b637c2529e4f232dce16c88.json b/backend/.sqlx/query-fac563138c316998d4f523edd633db97dba77d649b637c2529e4f232dce16c88.json new file mode 100644 index 0000000000..9520c270dc --- /dev/null +++ b/backend/.sqlx/query-fac563138c316998d4f523edd633db97dba77d649b637c2529e4f232dce16c88.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT q.id FROM v2_job_queue q JOIN v2_job j USING (id)\n WHERE j.parent_job = $1 AND q.running = true", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Uuid" + } + ], + "parameters": { + "Left": [ + "Uuid" + ] + }, + "nullable": [ + false + ] + }, + "hash": "fac563138c316998d4f523edd633db97dba77d649b637c2529e4f232dce16c88" +} diff --git a/backend/.sqlx/query-fac99e27c8396185dd21c33baf649dd75b5f8ed859e410d2322ec3f565382fa8.json b/backend/.sqlx/query-fac99e27c8396185dd21c33baf649dd75b5f8ed859e410d2322ec3f565382fa8.json new file mode 100644 index 0000000000..c624143ea4 --- /dev/null +++ b/backend/.sqlx/query-fac99e27c8396185dd21c33baf649dd75b5f8ed859e410d2322ec3f565382fa8.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path)\n path,\n value as \"value!: sqlx::types::Json>\",\n created_at\n FROM draft\n WHERE workspace_id = $1\n AND typ = 'script'\n AND (email = $2 OR email IS NULL)\n AND NOT EXISTS (\n SELECT 1 FROM script s\n WHERE s.workspace_id = draft.workspace_id\n AND s.path = draft.path\n )\n ORDER BY path, (email IS NULL)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "value!: sqlx::types::Json>", + "type_info": "Json" + }, + { + "ordinal": 2, + "name": "created_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "fac99e27c8396185dd21c33baf649dd75b5f8ed859e410d2322ec3f565382fa8" +} diff --git a/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json new file mode 100644 index 0000000000..484dd23e2a --- /dev/null +++ b/backend/.sqlx/query-fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441.json @@ -0,0 +1,29 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT req.id AS \"id!\",\n (CASE\n WHEN usr.email IS NULL THEN 'deleted'\n WHEN workspace.deleted THEN 'archived'\n ELSE 'active'\n END) AS \"status!\"\n FROM unnest($1::text[]) AS req(id)\n LEFT JOIN workspace ON workspace.id = req.id\n LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "status!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "TextArray", + "Text" + ] + }, + "nullable": [ + null, + null + ] + }, + "hash": "fc4583d1570f3a2a428bb28390ca72e61719fae68aa4b42730f9076f3bd97441" +} diff --git a/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json new file mode 100644 index 0000000000..5ac11c6a5e --- /dev/null +++ b/backend/.sqlx/query-fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS \"day!\",\n id AS \"id!\",\n timestamp AS \"ts!\",\n row_to_json(r)::text AS \"line!\"\n FROM (\n SELECT workspace_id, id, timestamp, username, operation,\n action_kind::text AS action_kind, resource, parameters, email, span\n FROM audit_partitioned\n WHERE timestamp >= $1 AND timestamp < $2\n AND (timestamp, id) > ($3, $4)\n ORDER BY timestamp, id\n LIMIT $5\n ) r\n ORDER BY timestamp, id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "day!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "id!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "ts!", + "type_info": "Timestamptz" + }, + { + "ordinal": 3, + "name": "line!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Timestamptz", + "Timestamptz", + "Timestamptz", + "Int8", + "Int8" + ] + }, + "nullable": [ + null, + false, + false, + null + ] + }, + "hash": "fd023a9365388f1f74423416bd8790770c32716ef953a2ea4af17d122d0a3b3c" +} diff --git a/backend/.sqlx/query-fea3dd1c119e859290cc875cfabc6536f9e2404002ba36046691aab955caf300.json b/backend/.sqlx/query-fea3dd1c119e859290cc875cfabc6536f9e2404002ba36046691aab955caf300.json new file mode 100644 index 0000000000..e8dedf7ab9 --- /dev/null +++ b/backend/.sqlx/query-fea3dd1c119e859290cc875cfabc6536f9e2404002ba36046691aab955caf300.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT path AS \"path!\" FROM (\n (SELECT DISTINCT path FROM script WHERE workspace_id = $1 AND archived = false AND deleted = false LIMIT 5000)\n UNION\n (SELECT path FROM flow WHERE workspace_id = $1 AND archived = false LIMIT 5000)\n UNION\n (SELECT path FROM app WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM raw_app WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM variable WHERE workspace_id = $1 LIMIT 5000)\n UNION\n (SELECT path FROM resource WHERE workspace_id = $1 LIMIT 5000)\n ) t\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "fea3dd1c119e859290cc875cfabc6536f9e2404002ba36046691aab955caf300" +} diff --git a/backend/Cargo.lock b/backend/Cargo.lock index 3979c7b016..7209b6be37 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -106,9 +106,9 @@ checksum = "cc7bb162ec39d46ab1ca8c77bf72e890535becd1751bb45f64c597edb4c8c6b3" [[package]] name = "alloc-stdlib" -version = "0.2.2" +version = "0.2.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "94fb8275041c72129eb51b7d0322c29b8387a0386127718b096429201a5d6ece" +checksum = "0e76a019e91224d279006ff972f1e984179a6e9feb050adba6ce8274aef23195" dependencies = [ "alloc-no-stdlib", ] @@ -180,9 +180,9 @@ dependencies = [ [[package]] name = "anyhow" -version = "1.0.102" +version = "1.0.103" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7f202df86484c868dbad7eaa557ef785d5c66295e41b460ef922eca0723b842c" +checksum = "2a4385e2e34eb35d6b3efe798b9eb88096925d87726c0798709bf56d9ed84af3" [[package]] name = "ar_archive_writer" @@ -237,9 +237,9 @@ checksum = "76a2e8124351fda1ef8aaaa3bbd7ebbcb486bbcd4225aca0aa0d84bb2db8fecb" [[package]] name = "arrayvec" -version = "0.7.6" +version = "0.7.7" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" +checksum = "f02882884d3e1bc524fb12c79f107f6ad0e1cfd498c536ffb494301740995dfe" [[package]] name = "arrow" @@ -481,7 +481,7 @@ checksum = "965c2d33e53cb6b267e148a4cb0760bc01f4904c1cd4bb4002a085bb016d1490" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "synstructure", ] @@ -493,7 +493,7 @@ checksum = "7b18050c2cd6fe86c3a76584ef5e0baf286d038cda203eb6223df2cc413565f7" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -504,7 +504,7 @@ checksum = "0a184645bcc6f52d69d8e7639720699c6a99efb711f886e251ed1d16db8dd90e" dependencies = [ "quote", "swc_macros_common", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -645,7 +645,7 @@ checksum = "3b43422f69d8ff38f95f1b2bb76517c91589a924d1559a0e935d7c8ce0274c11" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -667,7 +667,7 @@ checksum = "c7c24de15d275a1ecfd47a380fb4d5ec9bfe0933f309ed5e705b775596a3574d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -678,7 +678,7 @@ checksum = "9035ad2d096bed7955a320ee7e2230574d28fd3c3a0f186cbea1ff3c7eed5dbb" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1136,7 +1136,7 @@ dependencies = [ "aws-smithy-runtime-api", "aws-smithy-types", "h2 0.3.27", - "h2 0.4.14", + "h2 0.4.15", "http 0.2.12", "http 1.4.2", "http-body 0.4.6", @@ -1403,7 +1403,7 @@ checksum = "7aa268c23bfbbd2c4363b9cd302a4f504fb2a9dfe7e3451d66f35dd392e20aca" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1532,7 +1532,7 @@ dependencies = [ "regex", "rustc-hash 2.1.2", "shlex 1.3.0", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1552,7 +1552,7 @@ dependencies = [ "regex", "rustc-hash 2.1.2", "shlex 1.3.0", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1602,9 +1602,9 @@ dependencies = [ [[package]] name = "bitvec" -version = "1.0.1" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1bc2832c24239b0141d5674bb9174f9d68a8b5b3f2753311927c172ca46f7e9c" +checksum = "ddcec3d12c579d40898fe0a9a358a803c23e9c52ca3c425707f81c9436211837" dependencies = [ "funty", "radium", @@ -1716,9 +1716,9 @@ dependencies = [ [[package]] name = "bon" -version = "3.9.2" +version = "3.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b2f04f6fef12d70d42a77b1433c9e0f065238479a6cefc4f5bab105e9873a3c3" +checksum = "a602c73c7b0148ec6d12af6fd5cc7a46e2eacc8878271a999abac56eed12f561" dependencies = [ "bon-macros", "rustversion", @@ -1726,9 +1726,9 @@ dependencies = [ [[package]] name = "bon-macros" -version = "3.9.2" +version = "3.9.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7d0bd4c2f75335ad98052a37efb54f428b492f64340257143b3429c8a508fa7b" +checksum = "6dee98b0db6a962de883bf5d20362dee4d7ca0d12fe39a7c6c73c844e1cd7c1f" dependencies = [ "darling 0.23.0", "ident_case", @@ -1736,14 +1736,14 @@ dependencies = [ "proc-macro2", "quote", "rustversion", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "borsh" -version = "1.6.1" +version = "1.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "cfd1e3f8955a5d7de9fab72fc8373fade9fb8a703968cb200ae3dc6cf08e185a" +checksum = "2f3f6da4992df95bbcd9af42a6c7dcb994498fc9048230405f3b36ff7cd3f145" dependencies = [ "borsh-derive", "bytes", @@ -1752,15 +1752,15 @@ dependencies = [ [[package]] name = "borsh-derive" -version = "1.6.1" +version = "1.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bfcfdc083699101d5a7965e49925975f2f55060f94f9a05e7187be95d530ca59" +checksum = "3ae8fb4fb5740e4b2c4884ff95f5f32f5e8479db1e8fd8eb49ddbe09eb09bb7c" dependencies = [ "once_cell", "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1770,7 +1770,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "17d4f95e880cfd28c4ca5a006cf7f6af52b4bcb7b5866f573b2faa126fb7affb" dependencies = [ "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1786,13 +1786,13 @@ dependencies = [ [[package]] name = "brotli" -version = "8.0.3" +version = "8.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8119e4516436f5708bbc474a9d395bf12f1b5395e93a92a56e647ac3388c8610" +checksum = "5cc91aac060a7a1e25823bdccbfb6af1875b88f17c6daac97894eed8207166b3" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", - "brotli-decompressor 5.0.1", + "brotli-decompressor 5.0.3", ] [[package]] @@ -1807,9 +1807,9 @@ dependencies = [ [[package]] name = "brotli-decompressor" -version = "5.0.1" +version = "5.0.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5962523e1b92ce1b5e793d9169b9943eece10d39f62550bc04bb605d75b94924" +checksum = "3a32acac15fe1967bc3986b2a6347dffc965602354ea6f450ad07e8bfd253583" dependencies = [ "alloc-no-stdlib", "alloc-stdlib", @@ -1826,13 +1826,13 @@ dependencies = [ [[package]] name = "bstr" -version = "1.12.1" +version = "1.12.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63044e1ae8e69f3b5a92c736ca6269b8d12fa7efe39bf34ddb06d102cf0e2cab" +checksum = "5cee35f73844aa3014bb606320a6c1f010249dbdf43342fe54b5a4f6a8ed4b79" dependencies = [ "memchr", "regex-automata", - "serde", + "serde_core", ] [[package]] @@ -1855,9 +1855,9 @@ dependencies = [ [[package]] name = "byte-unit" -version = "5.2.0" +version = "5.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c6d47a4e2961fb8721bcfc54feae6455f2f64e7054f9bc67e875f0e77f4c58d" +checksum = "4a813de7f2bbedb7dce265b64f1cf5908ebe4d56281ece8d847e98113788b9b0" dependencies = [ "rust_decimal", "schemars 1.2.1", @@ -1904,7 +1904,7 @@ checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -1915,9 +1915,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b" [[package]] name = "bytes" -version = "1.11.1" +version = "1.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" +checksum = "8ae3f5d315924270530207e2a68396c3cc547f6dca3fbdca317cfb1a51edb593" dependencies = [ "serde", ] @@ -2042,7 +2042,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3b4a6cae9efc04cc6cbb8faf338d2c497c165c83e74509cf4dbedea948bbf6e5" dependencies = [ "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -2056,9 +2056,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.63" +version = "1.2.65" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "556e016178bb5662a08681bbe0f00f8e17631781a4dfc8c45e466e4b185ec27f" +checksum = "e228eec9be7c17ccb640b59b36a5cd805ea2a564a4c5e162c2f659fea30d3b96" dependencies = [ "find-msvc-tools", "jobserver", @@ -2101,9 +2101,9 @@ checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" [[package]] name = "chacha20" -version = "0.10.0" +version = "0.10.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +checksum = "d524456ba66e72eb8b115ff89e01e497f8e6d11d78b70b1aa13c0fbd97540a81" dependencies = [ "cfg-if", "cpufeatures 0.3.0", @@ -2205,7 +2205,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -2445,9 +2445,9 @@ dependencies = [ [[package]] name = "crc-any" -version = "2.5.0" +version = "2.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a62ec9ff5f7965e4d7280bd5482acd20aadb50d632cf6c1d74493856b011fa73" +checksum = "46db9f663dfb869b80fcf59e32d7a80fc6c464a4f6328f3f06a00f5e36d05f8c" dependencies = [ "debug-helper", ] @@ -2469,9 +2469,9 @@ dependencies = [ [[package]] name = "cron" -version = "0.16.0" +version = "0.17.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "089df96cf6a25253b4b6b6744d86f91150a3d4df546f31a95def47976b8cba97" +checksum = "a5dcd6f69605c2956916ce24e8af637b754964c9a83f4662d3a2361654cdba09" dependencies = [ "chrono", "once_cell", @@ -2638,7 +2638,7 @@ checksum = "f46882e17999c6cc590af592290432be3bce0428cb0d5f8b6715e4dc7b383eb3" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -2720,7 +2720,7 @@ dependencies = [ "proc-macro2", "quote", "strsim 0.11.1", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -2733,7 +2733,7 @@ dependencies = [ "proc-macro2", "quote", "strsim 0.11.1", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -2766,7 +2766,7 @@ checksum = "fc34b93ccb385b40dc71c6fceac4b2ad23662c7eeb248cf10d529b7e055b6ead" dependencies = [ "darling_core 0.20.11", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -2777,7 +2777,7 @@ checksum = "ac3984ec7bd6cfa798e62b4a642426a5be0e68f9401cfc2a01e3fa9ea2fcdb8d" dependencies = [ "darling_core 0.23.0", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3256,7 +3256,7 @@ checksum = "df6f88d7ee27daf8b108ba910f9015176b36fbc72902b1ca5c2a5f1d1717e1a1" dependencies = [ "datafusion-expr", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3412,9 +3412,9 @@ checksum = "c286de4e81ea2590afc24d754e0f83810c566f50a1388fa75ebd57928c0d9745" [[package]] name = "debug-helper" -version = "0.3.13" +version = "0.3.14" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f578e8e2c440e7297e008bb5486a3a8a194775224bbc23729b0dbdfaeebf162e" +checksum = "80a4af69c60438a1a82af89d362f4729fd38db7b73f305a237636fad31ceb2bf" [[package]] name = "debugid" @@ -3553,7 +3553,7 @@ checksum = "8380a4224d5d2c3f84da4d764c4326cac62e9a1e3d4960442d29136fc07be863" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3564,7 +3564,7 @@ checksum = "9b565e60a9685cdf312c888665b5f8647ac692a7da7e058a5e2268a466da8eaf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3595,7 +3595,7 @@ dependencies = [ "deno_tls", "dyn-clone", "error_reporter", - "h2 0.4.14", + "h2 0.4.15", "hickory-resolver", "http 1.4.2", "http-body-util", @@ -3733,7 +3733,7 @@ dependencies = [ "stringcase", "strum", "strum_macros", - "syn 2.0.117", + "syn 2.0.118", "thiserror 2.0.18", ] @@ -3930,7 +3930,6 @@ version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" dependencies = [ - "powerfmt", "serde_core", ] @@ -3973,7 +3972,7 @@ dependencies = [ "darling 0.20.11", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -3993,7 +3992,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ "derive_builder_core 0.20.2", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4022,7 +4021,7 @@ checksum = "cb7330aeadfbe296029522e6c40f315320aba36fc43a5b3632f3795348f3bd22" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "unicode-xid", ] @@ -4035,7 +4034,7 @@ dependencies = [ "proc-macro2", "quote", "rustc_version 0.4.1", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4161,7 +4160,7 @@ checksum = "1ac70aa55017e108007fbaf5aa0f54b021c98f92ff8af59d42eda9da96e3dd4f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4184,7 +4183,7 @@ checksum = "0fbbb781877580993a8707ec48672673ec7b81eeba04cfd2310bd28c08e47c8f" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4310,7 +4309,7 @@ dependencies = [ "enum-ordinalize", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4367,27 +4366,27 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "enum-ordinalize" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a1091a7bb1f8f2c4b28f1fe2cef4980ca2d410a3d727d67ecc3178c9b0800f0" +checksum = "07f808d588c10e464ea6f7d3eaed500049eff30aaac103460f61828c2d65b3eb" dependencies = [ "enum-ordinalize-derive", ] [[package]] name = "enum-ordinalize-derive" -version = "4.3.2" +version = "4.4.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8ca9601fb2d62598ee17836250842873a413586e5d7ed88b356e38ddbb0ec631" +checksum = "42e528e2d34ba8a67a1a650b86beae8ef69fc5fdb638016f386b973226590432" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4407,7 +4406,7 @@ checksum = "67c78a4d8fdf9953a5c9d458f9efe940fd97a0cab0941c075a813ac594733827" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4680,7 +4679,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "308530a56b099da144ebc5d8e179f343ad928fa2b3558d1eb3db9af18d6eff43" dependencies = [ "swc_macros_common", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -4806,7 +4805,7 @@ checksum = "e835b70203e41293343137df5c0664546da5745f82ec9b84d40be8336958447b" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -5067,16 +5066,14 @@ dependencies = [ [[package]] name = "getrandom" -version = "0.4.2" +version = "0.4.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0de51e6874e94e7bf76d726fc5d13ba782deca734ff60d5bb2fb2607c7406555" +checksum = "300e883d756b2e4ec94e02791f39b04b522276138852cfc41d9fb7e904106099" dependencies = [ "cfg-if", "libc", "r-efi 6.0.0", "rand_core 0.10.1", - "wasip2", - "wasip3", ] [[package]] @@ -5112,7 +5109,7 @@ checksum = "53010ccb100b96a67bc32c0175f0ed1426b31b655d562898e57325f81c023ac0" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -5236,9 +5233,9 @@ dependencies = [ [[package]] name = "gosyn" -version = "0.2.10" +version = "0.2.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c99c1502d84229dc7ddb6af755f40ebe80e7e932fa78ecef979cedcf9999ba93" +checksum = "fed1657682b1c3f63ece1fe5b60fc6c5f5923612a20d19f6af38ce79eaf361e3" dependencies = [ "anyhow", "strum", @@ -5287,9 +5284,9 @@ dependencies = [ [[package]] name = "h2" -version = "0.4.14" +version = "0.4.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "171fefbc92fe4a4de27e0698d6a5b392d6a0e333506bc49133760b3bcf948733" +checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155" dependencies = [ "atomic-waker", "bytes", @@ -5382,7 +5379,7 @@ dependencies = [ "indexmap 2.14.0", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -5701,7 +5698,7 @@ dependencies = [ "bytes", "futures-channel", "futures-core", - "h2 0.4.14", + "h2 0.4.15", "http 1.4.2", "http-body 1.0.1", "httparse", @@ -5715,9 +5712,9 @@ dependencies = [ [[package]] name = "hyper-http-proxy" -version = "1.1.0" +version = "1.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7ad4b0a1e37510028bc4ba81d0e38d239c39671b0f0ce9e02dfa93a8133f7c08" +checksum = "8021e0ae20c08eadc94d0bdafdeda66d4f0858541c146ae6e46b219bfe58497e" dependencies = [ "bytes", "futures-util", @@ -5729,7 +5726,6 @@ dependencies = [ "hyper-util", "native-tls", "pin-project-lite", - "rustls-native-certs 0.7.3", "tokio", "tokio-native-tls", "tokio-rustls 0.26.4", @@ -5801,7 +5797,7 @@ dependencies = [ "tokio", "tokio-rustls 0.26.4", "tower-service", - "webpki-roots 1.0.7", + "webpki-roots 1.0.8", ] [[package]] @@ -5994,12 +5990,6 @@ dependencies = [ "zerovec", ] -[[package]] -name = "id-arena" -version = "2.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954" - [[package]] name = "ident_case" version = "1.0.1" @@ -6141,7 +6131,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -6239,7 +6229,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" dependencies = [ "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -6462,7 +6452,7 @@ dependencies = [ "quote", "serde", "serde_json", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -6507,12 +6497,6 @@ dependencies = [ "spin 0.9.8", ] -[[package]] -name = "leb128fmt" -version = "0.1.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2" - [[package]] name = "levenshtein_automata" version = "0.2.1" @@ -6730,9 +6714,9 @@ dependencies = [ [[package]] name = "log" -version = "0.4.32" +version = "0.4.33" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "953f07c43838f8e6f9758cab68bf5bed85465e7587ebe0b823f1bcd81978ad3a" +checksum = "0ceec5bc11778974d1bcb055b18002eba7f4b3518b6a0081b3af5f21666da9ad" [[package]] name = "loom" @@ -6961,6 +6945,29 @@ dependencies = [ "malachite-nz", ] +[[package]] +name = "manyhow" +version = "0.11.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b33efb3ca6d3b07393750d4030418d594ab1139cee518f0dc88db70fec873587" +dependencies = [ + "manyhow-macros", + "proc-macro2", + "quote", + "syn 2.0.118", +] + +[[package]] +name = "manyhow-macros" +version = "0.11.4" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "46fce34d199b78b6e6073abf984c9cf5fd3e9330145a93ee0738a7443e371495" +dependencies = [ + "proc-macro-utils", + "proc-macro2", + "quote", +] + [[package]] name = "mappable-rc" version = "0.1.1" @@ -7032,15 +7039,15 @@ dependencies = [ [[package]] name = "memchr" -version = "2.8.1" +version = "2.8.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6b947ae49db0d222b1dbc6b113ce7248a3fc3a6ca21b696717bfc000ba4484d8" +checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4" [[package]] name = "memmap2" -version = "0.9.10" +version = "0.9.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "714098028fe011992e1c3962653c96b2d578c4b4bce9036e15ff220319b1e0e3" +checksum = "d1219ed1b7f229ee7104d281dd01d6802fe28bb6e95d292942c4daacdeb798c0" dependencies = [ "libc", "stable_deref_trait", @@ -7080,7 +7087,7 @@ checksum = "db5b29714e950dbb20d5e6f74f9dcec4edbcc1067bb7f8ed198c097b8c1a818b" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -7175,7 +7182,7 @@ checksum = "e4db6d5580af57bf992f59068d4ea26fd518574ff48d7639b255a36f9de6e7e9" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -7203,18 +7210,18 @@ checksum = "2195bf6aa996a481483b29d62a7663eed3fe39600c460e323f8ff41e90bdd89b" [[package]] name = "mysql-common-derive" -version = "0.32.1" +version = "0.32.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "66f62cad7623a9cb6f8f64037f0c4f69c8db8e82914334a83c9788201c2c1bfa" +checksum = "a4db8a44120571277accfaa3f3d91e7d3989d601d817c2fc01a9391b86135666" dependencies = [ - "darling 0.20.11", + "darling 0.23.0", "heck", + "manyhow", "num-bigint", "proc-macro-crate", - "proc-macro-error2", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "termcolor", "thiserror 2.0.18", ] @@ -7251,9 +7258,9 @@ dependencies = [ [[package]] name = "mysql_common" -version = "0.37.2" +version = "0.37.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4b42ced54aa8ac97226486337973f9bc3956e24f03a23e88a6e18f640959d6e2" +checksum = "0f27695f286b461da077b8c2f72f47feaa04ce3c3f9c0976257410e90e21208a" dependencies = [ "base64 0.22.1", "bitflags 2.13.0", @@ -7410,7 +7417,7 @@ dependencies = [ "proc-macro-error", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -7677,7 +7684,7 @@ dependencies = [ "proc-macro-crate", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -7855,9 +7862,9 @@ dependencies = [ [[package]] name = "openssl" -version = "0.10.80" +version = "0.10.81" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a45fa2aa886c42762255da344f0a0d313e254066c46aad76f300c3d3da62d967" +checksum = "77823a27f0babb03091cb9ed9ef80af3b39dbc82f97e8fa530374b7dafd87a45" dependencies = [ "bitflags 2.13.0", "cfg-if", @@ -7875,7 +7882,7 @@ checksum = "a948666b637a0f465e8564c73e89d4dde00d72d4d473cc972f390fc3dcee7d9c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -7901,9 +7908,9 @@ dependencies = [ [[package]] name = "openssl-sys" -version = "0.9.116" +version = "0.9.117" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f28a22dc7140cda5f096e5e7724a6962ca81a7f8bfd2979f9b18c11af56318c4" +checksum = "b47e7e6bb2c38cd930d25a23b40fa52e068c10e85f3e03a7f5ba5aaca5713695" dependencies = [ "cc", "libc", @@ -8255,7 +8262,7 @@ dependencies = [ "arrow-schema", "arrow-select", "base64 0.22.1", - "brotli 8.0.3", + "brotli 8.0.4", "bytes", "chrono", "flate2", @@ -8381,7 +8388,7 @@ dependencies = [ "pest_meta", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -8472,7 +8479,7 @@ dependencies = [ "phf_shared 0.11.3", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -8531,7 +8538,7 @@ checksum = "c96395f0a926bc13b1c17622aaddda1ecb55d49c8f1bf9777e4d877800a43f8b" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -8690,7 +8697,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "479ca8adacdd7ce8f1fb39ce9ecccbfe93a3f1344b3d0d97f20bc0196208f62b" dependencies = [ "proc-macro2", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -8734,28 +8741,6 @@ dependencies = [ "version_check", ] -[[package]] -name = "proc-macro-error-attr2" -version = "2.0.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "96de42df36bb9bba5542fe9f1a054b8cc87e172759a1868aa05c1f3acc89dfc5" -dependencies = [ - "proc-macro2", - "quote", -] - -[[package]] -name = "proc-macro-error2" -version = "2.0.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "11ec05c52be0a07b08061f7dd003e7d7092e0472bc731b4af7bb1ef876109802" -dependencies = [ - "proc-macro-error-attr2", - "proc-macro2", - "quote", - "syn 2.0.117", -] - [[package]] name = "proc-macro-rules" version = "0.4.0" @@ -8764,7 +8749,7 @@ checksum = "07c277e4e643ef00c1233393c673f655e3672cf7eb3ba08a00bdd0ea59139b5f" dependencies = [ "proc-macro-rules-macros", "proc-macro2", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -8776,7 +8761,18 @@ dependencies = [ "once_cell", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", +] + +[[package]] +name = "proc-macro-utils" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "eeaf08a13de400bc215877b5bdc088f241b12eb42f0a548d3390dc1c56bb7071" +dependencies = [ + "proc-macro2", + "quote", + "smallvec", ] [[package]] @@ -8861,7 +8857,7 @@ dependencies = [ "itertools 0.14.0", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -8917,9 +8913,9 @@ dependencies = [ [[package]] name = "pulp" -version = "0.22.2" +version = "0.22.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e205bb30d5b916c55e584c22201771bcf2bad9aabd5d4127f38387140c38632" +checksum = "046aa45b989642ec2e4717c8e72d677b13edd831a4d3b6cf37d9a3e54912496a" dependencies = [ "bytemuck", "cfg-if", @@ -8934,9 +8930,9 @@ dependencies = [ [[package]] name = "pulp-wasm-simd-flag" -version = "0.1.0" +version = "0.1.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40e24eee682d89fb193496edf918a7f407d30175b2e785fe057e4392dfd182e0" +checksum = "1d8f70e07b9c3962945a74e59ca1c511bba65b6419468acc217c457d93f3c740" [[package]] name = "pure-rust-locales" @@ -8966,21 +8962,21 @@ dependencies = [ [[package]] name = "quick_cache" -version = "0.6.23" +version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3a3db184a8b66cfe87f0263a1de147a6b554c864d1767c6f7fa4eb0e5497b565" +checksum = "403c1a912fec895cafb223201e368234842acb9220aaf08ab042ae89ba5f135c" dependencies = [ - "ahash 0.8.12", "equivalent", - "hashbrown 0.16.1", + "foldhash 0.2.0", + "hashbrown 0.17.1", "parking_lot", ] [[package]] name = "quinn" -version = "0.11.9" +version = "0.11.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b9e20a958963c291dc322d98411f541009df2ced7b5a4f2bd52337638cfccf20" +checksum = "0c1a41e437b6bbd489372cd4971de128e85c855f56c57f283d20ff016cf7c0a8" dependencies = [ "bytes", "cfg_aliases", @@ -8998,9 +8994,9 @@ dependencies = [ [[package]] name = "quinn-proto" -version = "0.11.14" +version = "0.11.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "434b42fec591c96ef50e21e886936e66d3cc3f737104fdb9b737c40ffb94c098" +checksum = "4fcb935c5bec503c2f0e306bdd3e58bb9029dcb14fa8d9ac76e3a5256ac0763e" dependencies = [ "aws-lc-rs", "bytes", @@ -9034,9 +9030,9 @@ dependencies = [ [[package]] name = "quote" -version = "1.0.45" +version = "1.0.46" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "41f2619966050689382d2b44f664f4bc593e129785a36d6ee376ddf37259b924" +checksum = "dfbc457d0c7a0759a614551b11a6409e5951f6c7537be1f1b7682b9ae9230368" dependencies = [ "proc-macro2", ] @@ -9101,7 +9097,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d2e8e8bcc7961af1fdac401278c6a831614941f6164ee3bf4ce61b7edb162207" dependencies = [ "chacha20", - "getrandom 0.4.2", + "getrandom 0.4.3", "rand_core 0.10.1", ] @@ -9298,7 +9294,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "76009fbe0614077fc1a2ce255e3a1881a2e3a3527097d5dc6d8212c585e7e38b" dependencies = [ "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -9358,7 +9354,7 @@ checksum = "b7186006dcb21920990093f30e3dea63b7d6e977bf1256be20c3563a5db070da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -9432,7 +9428,7 @@ dependencies = [ "futures-channel", "futures-core", "futures-util", - "h2 0.4.14", + "h2 0.4.15", "http 1.4.2", "http-body 1.0.1", "http-body-util", @@ -9466,7 +9462,7 @@ dependencies = [ "wasm-bindgen-futures", "wasm-streams", "web-sys", - "webpki-roots 1.0.7", + "webpki-roots 1.0.8", ] [[package]] @@ -9480,7 +9476,7 @@ dependencies = [ "encoding_rs", "futures-core", "futures-util", - "h2 0.4.14", + "h2 0.4.15", "http 1.4.2", "http-body 1.0.1", "http-body-util", @@ -9678,7 +9674,7 @@ dependencies = [ "proc-macro2", "quote", "serde_json", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -9717,7 +9713,7 @@ dependencies = [ "proc-macro2", "quote", "rquickjs-core", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -9790,7 +9786,7 @@ dependencies = [ "quote", "rust-embed-utils", "shellexpand", - "syn 2.0.117", + "syn 2.0.118", "walkdir", ] @@ -9816,9 +9812,9 @@ dependencies = [ [[package]] name = "rust_decimal" -version = "1.42.0" +version = "1.42.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c5108e3d4d903e21aac27f12ba5377b6b34f9f44b325e4894c7924169d06995" +checksum = "be2a24f50780bc85f09cc6ac299bdf1424302742d77221106859c9d8b102126a" dependencies = [ "arrayvec", "borsh", @@ -10277,7 +10273,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10289,7 +10285,7 @@ dependencies = [ "proc-macro2", "quote", "serde_derive_internals", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10466,7 +10462,7 @@ checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10477,7 +10473,7 @@ checksum = "18d26a20a969b9e3fdf2fc2d9f21eda6c40e2de84c9408bb5d3b05d499aae711" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10531,7 +10527,7 @@ checksum = "175ee3e80ae9982737ca543e96133087cbd9a485eecc3bc4de9c1a37b47ea59c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10598,7 +10594,7 @@ dependencies = [ "darling 0.23.0", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10651,7 +10647,7 @@ checksum = "94e153fc76e1c6a068703d6d29c508a0b15c061c4b7e43da59cc097bc342673c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -10976,7 +10972,7 @@ checksum = "da5fc6819faabb412da764b99d3b713bb55083c11e7e0c00144d386cd6a1939c" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11041,7 +11037,7 @@ dependencies = [ "quote", "sqlx-core", "sqlx-macros-core", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11064,7 +11060,7 @@ dependencies = [ "sqlx-mysql", "sqlx-postgres", "sqlx-sqlite", - "syn 2.0.117", + "syn 2.0.118", "tokio", "url", ] @@ -11227,7 +11223,7 @@ checksum = "ae36a4951ca7bd1cfd991c241584a9824a70f6aff1e7d4f693fb3f2465e4030e" dependencies = [ "quote", "swc_macros_common", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11286,7 +11282,7 @@ dependencies = [ "heck", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11389,7 +11385,7 @@ dependencies = [ "proc-macro2", "quote", "swc_macros_common", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11444,7 +11440,7 @@ checksum = "e276dc62c0a2625a560397827989c82a93fd545fcf6f7faec0935a82cc4ddbb8" dependencies = [ "proc-macro2", "swc_macros_common", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11545,7 +11541,7 @@ dependencies = [ "proc-macro2", "quote", "swc_macros_common", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11650,7 +11646,7 @@ checksum = "c16ce73424a6316e95e09065ba6a207eba7765496fed113702278b7711d4b632" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11661,7 +11657,7 @@ checksum = "aae1efbaa74943dc5ad2a2fb16cbd78b77d7e4d63188f3c5b4df2b4dcd2faaae" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11712,9 +11708,9 @@ dependencies = [ [[package]] name = "syn" -version = "2.0.117" +version = "2.0.118" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e665b8803e7b1d2a727f4023456bbbbe74da67099c585258af0ad9c5013b9b99" +checksum = "1b9ae57f904213ebb649ce6895b8a66c66f0203b9319718f69a5612a065b1422" dependencies = [ "proc-macro2", "quote", @@ -11738,7 +11734,7 @@ checksum = "728a70f3dbaf5bab7f0c4b1ac8d7ae5ea60a4b5549c8a5914361c99147a709d2" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -11767,7 +11763,7 @@ checksum = "181f22127402abcf8ee5c83ccd5b408933fec36a6095cf82cda545634692657e" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -12014,7 +12010,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.4.2", + "getrandom 0.4.3", "once_cell", "rustix 1.1.4", "windows-sys 0.61.2", @@ -12084,7 +12080,7 @@ checksum = "4fee6c4efc90059e10f81e6d42c60a18f76588c3d74cb83a0b242a2b6c7504c1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -12095,7 +12091,7 @@ checksum = "ebc4ee7f67670e9b64d05fa4253e753e016c6c95ff35b89b7941d6b856dec1d5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -12192,12 +12188,11 @@ dependencies = [ [[package]] name = "time" -version = "0.3.47" +version = "0.3.51" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" +checksum = "85c17d80feb7334b40c484e45ed1a5273dfd8bfda537c3be2e74a06a6686f327" dependencies = [ "deranged", - "itoa", "num-conv", "powerfmt", "serde_core", @@ -12207,15 +12202,15 @@ dependencies = [ [[package]] name = "time-core" -version = "0.1.8" +version = "0.1.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" +checksum = "9e1c906769ad99c88eaa54e728060edef082f8e358ff32030cb7c7d315e81109" [[package]] name = "time-macros" -version = "0.2.27" +version = "0.2.30" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" +checksum = "dcef1a61bdb119096e153208ec5cbec23944ce8bca13be5c7f60c634f7403935" dependencies = [ "num-conv", "time-core", @@ -12345,7 +12340,7 @@ checksum = "6e06d43f1345a3bcd39f6a56dbb7dcab2ba47e68e8ac134855e7e2bdbaf8cab8" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -12605,7 +12600,7 @@ dependencies = [ "base64 0.22.1", "bytes", "flate2", - "h2 0.4.14", + "h2 0.4.15", "http 1.4.2", "http-body 1.0.1", "http-body-util", @@ -12637,7 +12632,7 @@ dependencies = [ "axum 0.8.9", "base64 0.22.1", "bytes", - "h2 0.4.14", + "h2 0.4.15", "http 1.4.2", "http-body 1.0.1", "http-body-util", @@ -12784,7 +12779,7 @@ checksum = "7490cfa5ec963746568740651ac6781f701c9c5ea257c58e057f3ba8cf69e8da" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -13037,7 +13032,7 @@ checksum = "cf808357c6ed7e13ba0f3277ec8d8f21b2d501274895104263985330c726c1c5" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -13344,11 +13339,11 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821" [[package]] name = "uuid" -version = "1.23.3" +version = "1.23.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "144d6b123cef80b301b8f72a9e2ca4370ddec21950d0a103dd22c437006d2db7" +checksum = "bf80a72845275afea99e7f2b434723d3bc7e38470fcd1c7ed39a599c73319a53" dependencies = [ - "getrandom 0.4.2", + "getrandom 0.4.3", "js-sys", "serde_core", "wasm-bindgen", @@ -13446,20 +13441,11 @@ checksum = "ccf3ec651a847eb01de73ccad15eb7d99f80485de043efb2f370cd654f4ea44b" [[package]] name = "wasip2" -version = "1.0.3+wasi-0.2.9" +version = "1.0.4+wasi-0.2.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "20064672db26d7cdc89c7798c48a0fdfac8213434a1186e5ef29fd560ae223d6" +checksum = "b67efb37e106e55ce722a510d6b5f9c17f083e5fc79afc2badeb12cc313d9487" dependencies = [ - "wit-bindgen 0.57.1", -] - -[[package]] -name = "wasip3" -version = "0.4.0+wasi-0.3.0-rc-2026-01-06" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5" -dependencies = [ - "wit-bindgen 0.51.0", + "wit-bindgen", ] [[package]] @@ -13492,7 +13478,7 @@ dependencies = [ "log", "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "wasm-bindgen-shared", ] @@ -13527,7 +13513,7 @@ checksum = "ffc003a991398a8ee604a401e194b6b3a39677b3173d6e74495eb51b82e99a32" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "wasm-bindgen-backend", "wasm-bindgen-shared", ] @@ -13562,29 +13548,7 @@ checksum = "a369369e4360c2884c3168d22bded735c43cccae97bbc147586d4b480edd138d" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", -] - -[[package]] -name = "wasm-encoder" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319" -dependencies = [ - "leb128fmt", - "wasmparser", -] - -[[package]] -name = "wasm-metadata" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" -dependencies = [ - "anyhow", - "indexmap 2.14.0", - "wasm-encoder", - "wasmparser", + "syn 2.0.118", ] [[package]] @@ -13610,18 +13574,6 @@ dependencies = [ "thiserror 2.0.18", ] -[[package]] -name = "wasmparser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" -dependencies = [ - "bitflags 2.13.0", - "hashbrown 0.15.5", - "indexmap 2.14.0", - "semver 1.0.28", -] - [[package]] name = "wasmtimer" version = "0.4.3" @@ -13670,9 +13622,9 @@ dependencies = [ [[package]] name = "webpki-root-certs" -version = "1.0.7" +version = "1.0.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f31141ce3fc3e300ae89b78c0dd67f9708061d1d2eda54b8209346fd6be9a92c" +checksum = "0d46a5a140e6f7afeccd8eae97eff335163939eac8b929834875168b29b3d267" dependencies = [ "rustls-pki-types", ] @@ -13683,14 +13635,14 @@ version = "0.26.11" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "521bc38abb08001b01866da9f51eb7c5d647a19260e00054a8c7fd5f9e57f7a9" dependencies = [ - "webpki-roots 1.0.7", + "webpki-roots 1.0.8", ] [[package]] name = "webpki-roots" -version = "1.0.7" +version = "1.0.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "52f5ee44c96cf55f1b349600768e3ece3a8f26010c05265ab73f945bb1a2eb9d" +checksum = "bf85cb06032201fa7c6f829d7db5a7e5aa45bcc0655327713065f6f0576731bf" dependencies = [ "rustls-pki-types", ] @@ -13782,7 +13734,7 @@ dependencies = [ [[package]] name = "windmill" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -13865,7 +13817,7 @@ dependencies = [ [[package]] name = "windmill-ai" -version = "1.723.0" +version = "1.742.0" dependencies = [ "async-stream", "async-trait", @@ -13898,7 +13850,7 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -13911,7 +13863,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "argon2", @@ -14049,7 +14001,7 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14072,20 +14024,22 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", "serde", "serde_json", "sqlx", + "tracing", "windmill-api-auth", "windmill-common", + "windmill-parser-sql-asset", ] [[package]] name = "windmill-api-auth" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14111,7 +14065,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.723.0" +version = "1.742.0" dependencies = [ "reqwest 0.12.28", "serde", @@ -14121,7 +14075,7 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14138,7 +14092,7 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "base64 0.22.1", @@ -14160,7 +14114,7 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14183,7 +14137,7 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14199,7 +14153,7 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14220,7 +14174,7 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14241,7 +14195,7 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14255,7 +14209,7 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -14290,7 +14244,7 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14315,7 +14269,7 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "flate2", @@ -14333,7 +14287,7 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14355,7 +14309,7 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14375,7 +14329,7 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14399,14 +14353,20 @@ dependencies = [ "windmill-dep-map", "windmill-git-sync", "windmill-object-store", + "windmill-parser", "windmill-parser-py", + "windmill-parser-py-asset", + "windmill-parser-sql", + "windmill-parser-sql-asset", "windmill-parser-ts", + "windmill-parser-ts-asset", + "windmill-parser-yaml", "windmill-queue", ] [[package]] name = "windmill-api-settings" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14434,7 +14394,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.723.0" +version = "1.742.0" dependencies = [ "lazy_static", "serde", @@ -14446,7 +14406,7 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.723.0" +version = "1.742.0" dependencies = [ "argon2", "axum 0.8.9", @@ -14471,7 +14431,7 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14485,7 +14445,7 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.723.0" +version = "1.742.0" dependencies = [ "axum 0.8.9", "chrono", @@ -14518,7 +14478,7 @@ dependencies = [ [[package]] name = "windmill-audit" -version = "1.723.0" +version = "1.742.0" dependencies = [ "chrono", "lazy_static", @@ -14532,7 +14492,7 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "axum 0.8.9", @@ -14551,7 +14511,7 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.723.0" +version = "1.742.0" dependencies = [ "aes-gcm", "aho-corasick", @@ -14580,6 +14540,7 @@ dependencies = [ "dashmap", "datafusion", "equivalent", + "erased-serde", "futures", "futures-core", "gethostname", @@ -14652,7 +14613,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.723.0" +version = "1.742.0" dependencies = [ "chrono", "itertools 0.14.0", @@ -14671,7 +14632,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.723.0" +version = "1.742.0" dependencies = [ "regex", "serde", @@ -14686,7 +14647,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "astral-tokio-tar", @@ -14710,7 +14671,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "futures", @@ -14727,7 +14688,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.723.0" +version = "1.742.0" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -14738,12 +14699,12 @@ dependencies = [ "serde", "serde_derive", "serde_yml", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] name = "windmill-mcp" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -14764,7 +14725,7 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -14795,7 +14756,7 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "arc-swap", @@ -14820,7 +14781,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-stream", @@ -14854,7 +14815,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "futures", @@ -14872,7 +14833,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.723.0" +version = "1.742.0" dependencies = [ "convert_case 0.6.0", "serde", @@ -14881,7 +14842,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14893,7 +14854,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14905,7 +14866,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "gosyn", @@ -14917,7 +14878,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -14929,7 +14890,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -14941,7 +14902,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "nu-parser", @@ -14952,7 +14913,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14963,7 +14924,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -14975,7 +14936,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -14986,7 +14947,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -15008,7 +14969,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -15020,7 +14981,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15034,7 +14995,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -15044,14 +15005,14 @@ dependencies = [ "quote", "regex", "serde_json", - "syn 2.0.117", + "syn 2.0.118", "toml", "windmill-parser", ] [[package]] name = "windmill-parser-sql" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15064,7 +15025,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -15076,7 +15037,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -15094,7 +15055,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -15110,7 +15071,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -15126,7 +15087,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -15137,7 +15098,7 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -15171,11 +15132,12 @@ dependencies = [ "uuid", "windmill-audit", "windmill-common", + "windmill-jseval", ] [[package]] name = "windmill-runtime-nativets" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "const_format", @@ -15196,6 +15158,7 @@ dependencies = [ "futures", "itertools 0.14.0", "lazy_static", + "rcgen", "regex", "reqwest 0.13.1", "rustls 0.23.35", @@ -15213,7 +15176,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.723.0" +version = "1.742.0" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -15224,17 +15187,19 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", "axum 0.8.9", + "base64 0.22.1", "chrono", "futures", "hex", "http 1.4.2", "hyper 1.10.1", "lazy_static", + "magic-crypt", "quick_cache", "reqwest 0.13.1", "serde", @@ -15256,7 +15221,7 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15280,7 +15245,7 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15313,7 +15278,7 @@ dependencies = [ [[package]] name = "windmill-trigger-azure" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15346,7 +15311,7 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15366,7 +15331,7 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15400,7 +15365,7 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15436,7 +15401,7 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15459,7 +15424,7 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15483,7 +15448,7 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-nats", @@ -15507,7 +15472,7 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15542,7 +15507,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15570,7 +15535,7 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-trait", @@ -15595,7 +15560,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "bitflags 2.13.0", @@ -15614,7 +15579,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-once-cell", @@ -15724,7 +15689,7 @@ dependencies = [ [[package]] name = "windmill-worker-volumes" -version = "1.723.0" +version = "1.742.0" dependencies = [ "bytes", "futures", @@ -15851,7 +15816,7 @@ checksum = "f6fc35f58ecd95a9b71c4f2329b911016e6bec66b3f2e6a4aad86bd2e99e2f9b" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -15862,7 +15827,7 @@ checksum = "9107ddc059d5b6fbfbffdfa7a7fe3e22a226def0b2608f72e9d552763d3e1ad7" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -15873,7 +15838,7 @@ checksum = "053e2e040ab57b9dc951b72c264860db7eb3b0200ba345b4e4c3b14f67855ddf" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -15884,7 +15849,7 @@ checksum = "08990546bf4edef8f431fa6326e032865f27138718c587dc21bc0265bbcb57cc" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -15895,7 +15860,7 @@ checksum = "29bee4b38ea3cde66011baa44dba677c432a78593e202392d1e9070cf2a7fca7" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -15906,7 +15871,7 @@ checksum = "3f316c4a2570ba26bbec722032c4099d8c8bc095efccdc15688708623367e358" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -16337,100 +16302,12 @@ version = "0.0.19" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d135d17ab770252ad95e9a872d365cf3090e3be864a34ab46f48555993efc904" -[[package]] -name = "wit-bindgen" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5" -dependencies = [ - "wit-bindgen-rust-macro", -] - [[package]] name = "wit-bindgen" version = "0.57.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1ebf944e87a7c253233ad6766e082e3cd714b5d03812acc24c318f549614536e" -[[package]] -name = "wit-bindgen-core" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc" -dependencies = [ - "anyhow", - "heck", - "wit-parser", -] - -[[package]] -name = "wit-bindgen-rust" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" -dependencies = [ - "anyhow", - "heck", - "indexmap 2.14.0", - "prettyplease", - "syn 2.0.117", - "wasm-metadata", - "wit-bindgen-core", - "wit-component", -] - -[[package]] -name = "wit-bindgen-rust-macro" -version = "0.51.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a" -dependencies = [ - "anyhow", - "prettyplease", - "proc-macro2", - "quote", - "syn 2.0.117", - "wit-bindgen-core", - "wit-bindgen-rust", -] - -[[package]] -name = "wit-component" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" -dependencies = [ - "anyhow", - "bitflags 2.13.0", - "indexmap 2.14.0", - "log", - "serde", - "serde_derive", - "serde_json", - "wasm-encoder", - "wasm-metadata", - "wasmparser", - "wit-parser", -] - -[[package]] -name = "wit-parser" -version = "0.244.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" -dependencies = [ - "anyhow", - "id-arena", - "indexmap 2.14.0", - "log", - "semver 1.0.28", - "serde", - "serde_derive", - "serde_json", - "unicode-xid", - "wasmparser", -] - [[package]] name = "writeable" version = "0.6.3" @@ -16532,7 +16409,7 @@ checksum = "de844c262c8848816172cef550288e7dc6c7b7814b4ee56b3e1553f275f1858e" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "synstructure", ] @@ -16553,7 +16430,7 @@ checksum = "1ae7f38b72ec2a254e2b87ef277cf2cd4fb97cbebf944faa6f33354da0867930" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -16573,15 +16450,15 @@ checksum = "11532158c46691caf0f2593ea8358fed6bbf68a0315e80aae9bd41fbade684a1" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", "synstructure", ] [[package]] name = "zeroize" -version = "1.8.2" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b97154e67e32c85465826e8bcc1c59429aaaf107c1e4a9e53c8d8ccd5eff88d0" +checksum = "e13c156562582aa81c60cb29407084cdb54c4164760106ab78e6c5b0858cf64e" [[package]] name = "zerotrie" @@ -16613,7 +16490,7 @@ checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555" dependencies = [ "proc-macro2", "quote", - "syn 2.0.117", + "syn 2.0.118", ] [[package]] @@ -16630,9 +16507,9 @@ dependencies = [ [[package]] name = "zlib-rs" -version = "0.6.3" +version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" +checksum = "977347db8caa080403f6b6b7c1cda9479a8e869316f7e13a59b19076a40f94e3" [[package]] name = "zmij" diff --git a/backend/Cargo.toml b/backend/Cargo.toml index 41190428c8..827bd5d494 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.723.0" +version = "1.742.0" authors.workspace = true edition.workspace = true @@ -87,7 +87,7 @@ members = [ exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.723.0" +version = "1.742.0" authors = ["Ruben Fiszel "] edition = "2021" @@ -121,6 +121,7 @@ embedding = ["windmill-api/embedding"] parquet = ["windmill-api/parquet", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker/parquet"] prometheus = ["windmill-common/prometheus", "windmill-api/prometheus", "windmill-worker/prometheus", "windmill-queue/prometheus", "dep:prometheus"] flow_testing = ["windmill-worker/flow_testing"] +failpoints = ["windmill-worker/failpoints", "windmill-queue/failpoints"] quickjs = ["windmill-worker/quickjs", "windmill-api/quickjs"] openidconnect = ["windmill-api/openidconnect", "windmill-common/openidconnect", "windmill-object-store/openidconnect"] cloud = ["windmill-queue/cloud", "windmill-worker/cloud", "windmill-common/cloud", "windmill-api/cloud"] diff --git a/backend/THREAT_MODEL.md b/backend/THREAT_MODEL.md index 8cc71ec71c..9501b5bf0d 100644 --- a/backend/THREAT_MODEL.md +++ b/backend/THREAT_MODEL.md @@ -88,7 +88,7 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | EP12 Stored-content rendering | App builder HTML component, markdown, S3 download response headers | stored user content → admin browser (same origin) | Admin session, account takeover | | EP13 Log/file reading & export endpoints | `service_logs`, `jobs_u/getupdate` log file read (symlinks), workspace/tarball export | authed/unauth request → arbitrary file or admin-only config | Arbitrary files, global settings | | EP14 Secret-value & resource-value caches | In-memory caches in `windmill-store` keyed (historically un-keyed) by path | cache lookup crossing identity/folder boundary | Secret variables, resource creds | -| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS=false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | +| EP15 Deployment & runtime config | docker-compose defaults: dind, debugger (`REQUIRE_SIGNED_DEBUG_REQUESTS` now defaults to `true`; can still be overridden to `false`), CORS `Any`, default admin/`changeme`, exposed Postgres, `SUPERADMIN_SECRET`, `ENABLE_NSJAIL=false`, privileged containers | operator/infra default → full instance | All assets | | EP16 Supply chain | Cached hub scripts, GitHub workflow actions, vendored deps, Docker base image | build/update-time input → host & build integrity | Worker host, build integrity | | EP17 Token lifecycle | Token create/rescope/refresh, script-issued JWTs | scoped caller → broader privilege | Tokens, accounts, isolation | @@ -97,13 +97,13 @@ published advisory history (73 GHSA advisories, several rated 9.9 critical). | id | threat | actor | surface | asset | impact | likelihood | status | controls | evidence | |---|---|---|---|---|---|---|---|---|---| | T1 | SQL injection in app/internal query builders and trigger clauses compromises the metadata DB and connected databases | remote_auth | EP8 | Database, downstream connected systems | critical | almost_certain | partially_mitigated | sqlx parameterized queries elsewhere; query-builder safety reviews | GHSA-225c-j3xq-g6x6, GHSA-78p7-jc72-gv66, GHSA-hvc7-f67h-jx3g, GHSA-wrrg-f89m-f84q, GHSA-79vf-3qwm-2w64, GHSA-55p6-fxj4-v983, GHSA-5g4v-49rj-r52r, GHSA-x6cq-7xr8-53x3, 2cf4bb180b | -| T2 | Server-side request forgery via proxies/executors reaches cloud metadata, internal network, and downstream credentials | remote_auth | EP6, EP7 | Cloud metadata, internal network, downstream connected systems, resource creds | critical | almost_certain | partially_mitigated | SSRF URL validation + redirect-following disabled added piecemeal; MCP private URL access requires the instance-wide `ALLOW_PRIVATE_MCP_SERVER_URLS` opt-in; outbound network isolation (`clone_newnet`) is opt-in and off by default | GHSA-3ggp-h37f-5qfw, GHSA-98qq-g8rh-xhff, GHSA-hfw8-27mx-63jm, GHSA-3r59-qvvc-774j, GHSA-4pj9-w5jc-g8w7, GHSA-8hh3-jf25-78j5, GHSA-3pjm-4w7f-3r2w, GHSA-f44c-x9hq-h68r, GHSA-j4h4-f8fj-3m3c, 4b06881918, 96a8eb63d4, dbd3942ef3 | +| T2 | Server-side request forgery via proxies/executors reaches cloud metadata, internal network, and downstream credentials | remote_auth | EP6, EP7 | Cloud metadata, internal network, downstream connected systems, resource creds | critical | almost_certain | partially_mitigated | SSRF URL validation + redirect-following disabled added piecemeal; MCP private URL access requires the instance-wide `ALLOW_PRIVATE_MCP_SERVER_URLS` opt-in; WebSocket trigger URLs (stored, test, and runnable-resolved) are SSRF-validated at connect time behind the `ALLOW_PRIVATE_WEBSOCKET_URLS` opt-in, and the trigger test route now requires `:write` scope; outbound network isolation (`clone_newnet`) is opt-in and off by default | GHSA-3ggp-h37f-5qfw, GHSA-98qq-g8rh-xhff, GHSA-hfw8-27mx-63jm, GHSA-3r59-qvvc-774j, GHSA-4pj9-w5jc-g8w7, GHSA-8hh3-jf25-78j5, GHSA-3pjm-4w7f-3r2w, GHSA-f44c-x9hq-h68r, GHSA-j4h4-f8fj-3m3c, 4b06881918, 96a8eb63d4, dbd3942ef3 | | T3 | Broken authorization / IDOR lets a scoped token or low-privilege member read scripts, job data, and secrets across folders and workspaces | remote_auth | EP5, EP2, EP1 | Scripts, job data, secrets, isolation | critical | almost_certain | partially_mitigated | RLS, token scopes, folder ACLs, view-token HMAC (added incrementally); on managed, sensitive tenants can opt into dedicated DB/worker/namespace, but the shared tier IS the software boundary | GHSA-qfg7-x243-5hg4, GHSA-8x8x-88qc-qp4r, GHSA-2ppx-66jv-wpw5, GHSA-x3x7-g97v-mp59, GHSA-j276-g4h8-g6h5, GHSA-8mv7-hmrg-96xv, GHSA-x2wf-f962-7frq, GHSA-qc7c-gcw6-h4xp, GHSA-vxc5-w28p-m9xw, GHSA-2g34-wfvr-5qqj, GHSA-w7p6-wpxm-pp66, 7edf3f0212, 89a7a37776, ab11c7747a, 664edcdfb7 | | T4 | Remote code execution by injecting attacker-controlled identifiers into generated worker wrappers | remote_auth | EP10 | Worker host, isolation, downstream | critical | likely | partially_mitigated | entrypoint/env-var-name validation added | GHSA-wxjq-w5pj-jqhx, GHSA-5f5q-2vg2-r2x4, GHSA-8q8j-mm3g-5c2q (CVE-2026-33881), bf93657fee, bd05bcadde, 22ec4da5f0 | | T5 | Worker compromise & cross-tenant access via weak-by-default isolation (nsjail off by default → user code runs with only PID-ns `unshare`); sandbox escape where nsjail/dind/podman is enabled | remote_auth | EP9, EP15 | Worker host, isolation, downstream | critical | likely | unmitigated | nsjail off by default everywhere (`DISABLE_NSJAIL=true`); shipped compose gives PID-ns `unshare` only (`FAVOR_UNSHARE_PID=true`), bare installs get no isolation. Where nsjail enabled: read-only remounts, jail-tmp refusal, podman socket gating | GHSA-6qr8-xhg4-453q, GHSA-3vpp-vf62-wqp6, f8467f38c8, df5aec0f5d, f1b6746e0e | | T6 | Disclosure of secrets, resource credentials, and workspace encryption keys across the authorization boundary (AI proxy, MCP, caches, export); database read additionally yields plaintext instance-level `global_settings` secrets | remote_auth | EP6, EP14, EP13 | Secret variables, encryption keys, resource creds, global settings | critical | likely | partially_mitigated | RLS on `$var:`, cache scoping by caller, admin checks on export; per-workspace secret *variables* encrypted at rest, but `global_settings` is plaintext under the default DB secret backend | GHSA-jwg4-v3cj-rvfm, GHSA-8m2p-2crh-9h3w, GHSA-6635-6fch-v8px, GHSA-437f-725p-7w84, GHSA-f27g-j463-q85w (CVE-2026-26964), GHSA-j679-v6vj-jfxc, GHSA-6vrr-fq33-qpfp, 0ba128afe7, 7836a4e733, ff8e39c69b | | T7 | Full instance compromise from insecure deployment defaults (dind control, default admin/`changeme`, exposed Postgres, publicly readable SUPERADMIN_SECRET) | remote_unauth | EP15 | All assets | critical | likely | partially_mitigated | first-time-setup warning on default admin; docs recommend hardening | GHSA-3vpp-vf62-wqp6, GHSA-24fr-44f8-fqwg (CVE-2026-29059), GHSA-6q36-5p3h-766j | -| T8 | Unauthenticated RCE via the Debugger WebSocket in the default `windmill_extra` configuration | remote_unauth | EP15 | Worker host, all assets | critical | possible | unmitigated | `REQUIRE_SIGNED_DEBUG_REQUESTS` exists but defaults to false | GHSA-725h-99vx-9xr4 | +| T8 | Unauthenticated RCE via the Debugger WebSocket: `/ws_debug/*` exposed by the gateway/ingress with the debugger service as the auth boundary; signature gate was bypassable via `program`-mode launches (read+exec an arbitrary server-side file path, never signed) even with signing on, and the WS handshake had no Origin check (CSWSH) | remote_unauth | EP15 | Worker host, all assets | critical | possible | partially_mitigated | `program`-mode launches now rejected when `REQUIRE_SIGNED_DEBUG_REQUESTS` is on (signing covers every launch, not just inline `code`); shipped `docker-compose` now defaults `REQUIRE_SIGNED_DEBUG_REQUESTS=true`; opt-in `DEBUG_ALLOWED_ORIGINS` allowlist rejects cross-origin handshakes. Residual: code default is secure but operators can still set `=false`; origin allowlist is opt-in | GHSA-725h-99vx-9xr4 | | T9 | Supply-chain compromise via cached hub scripts, GitHub workflow command injection, or vulnerable base-image deps | supply_chain | EP16 | Worker host, build integrity | critical | possible | partially_mitigated | hub-script re-pin to patched versions; HUB_BASE_URL override | GHSA-w2m9-q5f7-3gpq, edf340c4d4, GHSA-8rq7-w7g6-8wvr, GHSA-vch9-39v5-4wg7 (CVE-2024-37371) | | T10 | Unauthenticated disclosure of job results, args, logs, and admin config via missing-authz public endpoints | remote_unauth | EP2, EP13 | Job results/args/logs, global settings, scripts | high | likely | partially_mitigated | anonymous-job checks, log-endpoint authz hardening | GHSA-qfg7-x243-5hg4, GHSA-v448-fmm4-52fp, 108a88a180, bb90f4ce83 | | T11 | Stored XSS leading to admin/account takeover via app HTML component, markdown, or S3 download content-type | remote_auth | EP12 | Admin session, accounts | high | likely | partially_mitigated | DOMPurify markdown sanitization, `X-Content-Type-Options: nosniff` + CSP sandbox on downloads | GHSA-9c5c-hh3c-r9mc, GHSA-qxj7-hpx3-r892, GHSA-cf2x-rg8c-v63v, bb78b1c06d, 625b67dff0 | diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index b982636197..8485cc0d14 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -5ee71322d3d21aa358bad88d3a823a661e3c781f +19d1c453d4dcbd06cd1895396ba75c174085a976 diff --git a/backend/migrations/20260423050000_script_trigger.down.sql b/backend/migrations/20260423050000_script_trigger.down.sql new file mode 100644 index 0000000000..5171e8f2d0 --- /dev/null +++ b/backend/migrations/20260423050000_script_trigger.down.sql @@ -0,0 +1,3 @@ +DROP INDEX IF EXISTS idx_script_pipeline_path; +DROP TABLE IF EXISTS script_trigger; +DROP TYPE IF EXISTS SCRIPT_TRIGGER_KIND; diff --git a/backend/migrations/20260423050000_script_trigger.up.sql b/backend/migrations/20260423050000_script_trigger.up.sql new file mode 100644 index 0000000000..cb105ba3f3 --- /dev/null +++ b/backend/migrations/20260423050000_script_trigger.up.sql @@ -0,0 +1,58 @@ +-- Execution DAG edges declared via `// on ` +-- annotations. +-- For `trigger_kind='asset'`: trigger_ref is `://` (kind from +-- parse_asset_syntax, so downstream lookups match the `asset` table). +-- The other kinds mirror the keywords the annotation parser recognises in +-- `// on ` lines (every non-integration trigger kind; their +-- trigger_ref is the trigger row path, or empty for marker-only forms). +-- +-- Per-edge columns that are in fact script-level properties (every row for +-- a given runnable carries the same value, set once at deploy) but live on +-- the edge so the dispatcher reads everything from a single query: +-- join_all `// trigger all` AND-join barrier (else OR, the default). +-- debounce_s `// on debounce=` (else script-level `// debounce`); only +-- asset-cascade edges carry one. NULL = no debounce. +-- retry_* `// retry []` cascade retry. NULL = none. +-- +-- The idempotency guards (IF NOT EXISTS / duplicate_object) are load-bearing: +-- this migration squashes several pre-release ones, so databases migrated +-- from the unsquashed history already contain the final objects and +-- re-applying must be a no-op. +DO $$ BEGIN + CREATE TYPE SCRIPT_TRIGGER_KIND AS ENUM ( + 'asset', 'schedule', 'webhook', 'email', 'kafka', 'mqtt', 'nats', + 'postgres', 'sqs', 'gcp'); +EXCEPTION WHEN duplicate_object THEN NULL; +END $$; + +CREATE TABLE IF NOT EXISTS script_trigger ( + id BIGSERIAL PRIMARY KEY, + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + runnable_kind ASSET_USAGE_KIND NOT NULL, + runnable_path VARCHAR(255) NOT NULL, + trigger_kind SCRIPT_TRIGGER_KIND NOT NULL, + trigger_ref TEXT NOT NULL, + join_all BOOLEAN NOT NULL DEFAULT FALSE, + debounce_s INTEGER, + retry_count SMALLINT, + retry_delay_s INTEGER +); + +-- Per-runnable lookup (wipe-on-deploy, list-triggers-for-script). +CREATE INDEX IF NOT EXISTS idx_script_trigger_runnable + ON script_trigger (workspace_id, runnable_kind, runnable_path); + +-- Reverse lookup: "which scripts are triggered by asset X?" (the asset → script +-- edges in the graph). trigger_ref is unbounded text so can't share the +-- asset_kind btree, but this covers the common prefix-scan use case. +CREATE INDEX IF NOT EXISTS idx_script_trigger_ref + ON script_trigger (workspace_id, trigger_kind, trigger_ref); + +-- Fast lookups for: +-- 1. "does folder F have a pipeline?" (exists check on prefix) +-- 2. "list all folders with a pipeline" (distinct folder from path) +-- The partial predicate keeps the index tiny on workspaces with few +-- pipeline scripts, and text_pattern_ops lets 'f/foo/%' LIKE scans use it. +CREATE INDEX IF NOT EXISTS idx_script_pipeline_path + ON script (workspace_id, path text_pattern_ops) + WHERE auto_kind = 'pipeline' AND archived = false AND deleted = false; diff --git a/backend/migrations/20260510174213_asset_trigger_dispatch.down.sql b/backend/migrations/20260510174213_asset_trigger_dispatch.down.sql new file mode 100644 index 0000000000..76e5b7050c --- /dev/null +++ b/backend/migrations/20260510174213_asset_trigger_dispatch.down.sql @@ -0,0 +1,3 @@ +-- Postgres has no ALTER TYPE ... DROP VALUE for enums. The 'asset' value +-- stays even on rollback; this is consistent with how other job_trigger_kind +-- values were added (see 20250323162033_add-missing-trigger-kind-...). diff --git a/backend/migrations/20260510174213_asset_trigger_dispatch.up.sql b/backend/migrations/20260510174213_asset_trigger_dispatch.up.sql new file mode 100644 index 0000000000..30cdb58ec3 --- /dev/null +++ b/backend/migrations/20260510174213_asset_trigger_dispatch.up.sql @@ -0,0 +1,5 @@ +-- Add 'asset' as a job_trigger_kind so jobs that get dispatched as a +-- consequence of an upstream pipeline script writing an asset can be +-- attributed via v2_job.trigger_kind = 'asset'. The producer's runnable +-- path goes into v2_job.trigger. +ALTER TYPE job_trigger_kind ADD VALUE IF NOT EXISTS 'asset'; diff --git a/backend/migrations/20260516194247_join_pending_inputs.down.sql b/backend/migrations/20260516194247_join_pending_inputs.down.sql new file mode 100644 index 0000000000..6e67e88e1b --- /dev/null +++ b/backend/migrations/20260516194247_join_pending_inputs.down.sql @@ -0,0 +1 @@ +DROP TABLE join_pending_inputs; diff --git a/backend/migrations/20260516194247_join_pending_inputs.up.sql b/backend/migrations/20260516194247_join_pending_inputs.up.sql new file mode 100644 index 0000000000..cbda6dd16e --- /dev/null +++ b/backend/migrations/20260516194247_join_pending_inputs.up.sql @@ -0,0 +1,19 @@ +-- AND-join barrier slot state. For a `// trigger all` subscriber, each +-- partition-bearing input arrival (an `// on` asset whose declared path +-- contains the `{partition}` token) is recorded against the +-- (subscriber, partition) slot. The subscriber is dispatched once, for a +-- given partition, only when every partition-bearing input it declares has +-- arrived for that partition — skew-immune (unlike a debounce). The slot +-- is cleared on fire so later writes re-accumulate and can re-materialize. +-- +-- trigger_ref stores the literal `{partition}`-token form (lineage is +-- partition-agnostic; the concrete value is the `partition` column), +-- matching how script_trigger / asset rows store it. +CREATE TABLE join_pending_inputs ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + subscriber_path VARCHAR(255) NOT NULL, + partition TEXT NOT NULL, + trigger_ref TEXT NOT NULL, + received_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, subscriber_path, partition, trigger_ref) +); diff --git a/backend/migrations/20260523055641_dispatch_event.down.sql b/backend/migrations/20260523055641_dispatch_event.down.sql new file mode 100644 index 0000000000..c6fdefc07c --- /dev/null +++ b/backend/migrations/20260523055641_dispatch_event.down.sql @@ -0,0 +1,2 @@ +DROP TABLE IF EXISTS dispatch_event; +DROP TYPE IF EXISTS DISPATCH_OUTCOME; diff --git a/backend/migrations/20260523055641_dispatch_event.up.sql b/backend/migrations/20260523055641_dispatch_event.up.sql new file mode 100644 index 0000000000..bf68281fbd --- /dev/null +++ b/backend/migrations/20260523055641_dispatch_event.up.sql @@ -0,0 +1,63 @@ +-- Per-decision log of what the asset-trigger dispatcher did after each +-- producer job completed. One row per (producer, subscriber, asset write) +-- decision: dispatched, debounced, join_pending (partial AND-join), or +-- skipped (with reason). Surfaced on the producer's job detail page so +-- the cascade is no longer invisible when the producer "succeeds" but +-- no child appears. +-- +-- Retention: the FK to v2_job(id) ON DELETE CASCADE means the existing +-- retention sweep (monitor.rs delete_expired_jobs_batch -> DELETE FROM +-- v2_job WHERE id = ANY(...)) reaps these rows along with their producer. +-- No separate cleanup path needed. +-- +-- Idempotency guards (duplicate_object / IF NOT EXISTS) are load-bearing: +-- re-applying this migration after a squash must be a no-op. +DO $$ BEGIN + CREATE TYPE DISPATCH_OUTCOME AS ENUM ( + 'dispatched', + 'join_pending', + 'skipped' + ); +EXCEPTION WHEN duplicate_object THEN NULL; +END $$; + +CREATE TABLE IF NOT EXISTS dispatch_event ( + id BIGSERIAL PRIMARY KEY, + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + producer_job_id UUID NOT NULL REFERENCES v2_job(id) ON DELETE CASCADE, + subscriber_path VARCHAR(255) NOT NULL, + asset_kind ASSET_KIND NOT NULL, + asset_path TEXT NOT NULL, + outcome DISPATCH_OUTCOME NOT NULL, + -- Set for 'dispatched'. Intentionally not FK'd: the subscriber job may + -- be retention-reaped independently, and we still want the row to + -- record "we dispatched " historically (UI renders a dead link). + child_job_id UUID, + partition TEXT, + -- AND-join progress at decision time. NULL for non-join subscribers. + received_inputs INTEGER, + required_inputs INTEGER, + -- Effective debounce window applied to this dispatch (NULL = none). + debounce_s INTEGER, + -- Free-text discriminator for 'skipped' outcomes (self_loop, + -- case3_non_partition_bearing, case3_missing_partition, cycle_detected, ...). + reason TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); + +-- Primary access pattern: list events for one producer (the job detail +-- panel). Ordered scans by id give chronological order for free. +CREATE INDEX IF NOT EXISTS idx_dispatch_event_producer + ON dispatch_event (producer_job_id, id); + +-- Backs the asset-graph edge listing (jobs.rs list_asset_dispatch_edges): +-- WHERE workspace_id = $1 AND subscriber_path LIKE 'prefix%' +-- AND created_at >= $3 +-- ORDER BY created_at DESC, id DESC +-- The (producer_job_id, id) index above doesn't help this access path, so +-- without this one a high-volume dispatch_event seq-scans + sorts. +-- text_pattern_ops makes the anchored LIKE prefix (built as `path_start || '%'`) +-- index-usable regardless of the column collation; created_at DESC matches +-- the ORDER BY so Postgres can satisfy ordering from the index. +CREATE INDEX IF NOT EXISTS idx_dispatch_event_subscriber + ON dispatch_event (workspace_id, subscriber_path text_pattern_ops, created_at DESC); diff --git a/backend/migrations/20260528143710_draft_user_sync_schema.down.sql b/backend/migrations/20260528143710_draft_user_sync_schema.down.sql new file mode 100644 index 0000000000..bbd036ad37 --- /dev/null +++ b/backend/migrations/20260528143710_draft_user_sync_schema.down.sql @@ -0,0 +1,25 @@ +ALTER TABLE draft DROP COLUMN id; + +-- The .up.sql secret-draft scrub is irreversible — the blanked plaintext +-- values were deliberately destroyed and cannot be recovered here. + +DROP INDEX IF EXISTS draft_workspace_path_typ_idx; +DROP INDEX IF EXISTS draft_user_listing_idx; +DROP INDEX IF EXISTS draft_pkey_legacy; +DROP INDEX IF EXISTS draft_pkey_with_user; + +-- Per-user rows can't exist under the composite PK (one row per +-- (workspace_id, path, typ)); drop them before restoring it. +DELETE FROM draft WHERE email IS NOT NULL; + +ALTER TABLE draft ADD CONSTRAINT draft_pkey PRIMARY KEY (workspace_id, path, typ); + +ALTER TABLE draft DROP CONSTRAINT IF EXISTS draft_password_fkey; +ALTER TABLE draft DROP COLUMN email; + +-- Restore the narrower DRAFT_TYPE enum; drop rows outside that set so the +-- cast doesn't fail. +CREATE TYPE DRAFT_TYPE AS ENUM ('script', 'flow', 'app'); +DELETE FROM draft WHERE typ::text NOT IN ('script', 'flow', 'app'); +ALTER TABLE draft ALTER COLUMN typ TYPE DRAFT_TYPE USING typ::text::DRAFT_TYPE; +DROP TYPE DRAFT_KIND; diff --git a/backend/migrations/20260528143710_draft_user_sync_schema.up.sql b/backend/migrations/20260528143710_draft_user_sync_schema.up.sql new file mode 100644 index 0000000000..07343c6b2c --- /dev/null +++ b/backend/migrations/20260528143710_draft_user_sync_schema.up.sql @@ -0,0 +1,82 @@ +-- Reshape `draft` for per-user bidirectional sync: add the owner `email` +-- (FK to password.email, NULL on legacy rows); replace the composite PK with +-- two partial unique indexes so per-user rows and the single legacy +-- workspace-level row coexist at the same (workspace_id, path, typ); widen +-- the DRAFT_TYPE enum to DRAFT_KIND (every UserDraftItemKind); and add a +-- synthetic BIGSERIAL `id` PK (tools like pg_dump/replication break on the +-- partial-index-only layout). + +CREATE TYPE DRAFT_KIND AS ENUM ( + 'script', + 'flow', + 'app', + 'raw_app', + 'resource', + 'variable', + 'trigger_schedule', + 'trigger_webhook', + 'trigger_default_email', + 'trigger_email', + 'trigger_http', + 'trigger_websocket', + 'trigger_postgres', + 'trigger_kafka', + 'trigger_nats', + 'trigger_mqtt', + 'trigger_sqs', + 'trigger_gcp', + 'trigger_azure', + 'trigger_poll', + 'trigger_cli', + 'trigger_nextcloud', + 'trigger_google', + 'trigger_github' +); + +ALTER TABLE draft ALTER COLUMN typ TYPE DRAFT_KIND USING typ::text::DRAFT_KIND; +DROP TYPE DRAFT_TYPE; + +ALTER TABLE draft ADD COLUMN email VARCHAR(255); + +ALTER TABLE draft + ADD CONSTRAINT draft_password_fkey + FOREIGN KEY (email) + REFERENCES password(email) + ON DELETE CASCADE + ON UPDATE CASCADE; + +ALTER TABLE draft DROP CONSTRAINT draft_pkey; + +CREATE UNIQUE INDEX draft_pkey_with_user + ON draft (workspace_id, path, typ, email) + WHERE email IS NOT NULL; + +CREATE UNIQUE INDEX draft_pkey_legacy + ON draft (workspace_id, path, typ) + WHERE email IS NULL; + +-- Serves the per-user draft listing (`WHERE workspace_id = ? AND email = ? +-- ORDER BY path`); neither partial unique index helps (both lead with +-- `path, typ`), and the trailing `path` keeps rows in output order. +CREATE INDEX draft_user_listing_idx + ON draft (workspace_id, email, path) + WHERE email IS NOT NULL; + +ALTER TABLE draft ADD COLUMN id BIGSERIAL PRIMARY KEY; + +-- Hot path: `fetch_other_drafts_users` runs on every get-by-path request +-- with `WHERE workspace_id = ? AND path = ? AND typ = ?` and no email +-- predicate. The partial unique/listing indexes can't serve it (their +-- `email IS [NOT] NULL` predicates aren't implied by the query), so a plain +-- btree is needed. Also covers `get_draft_for_user` (same three columns + +-- `email IS NOT DISTINCT FROM ?` as a filter). +CREATE INDEX draft_workspace_path_typ_idx ON draft (workspace_id, path, typ); + +-- Secret variable values must never sit in `draft.value` in plaintext. +-- `save_draft` now encrypts them at write time; this scrubs any rows +-- persisted before that guard (irreversible — see the .down.sql note). +UPDATE draft +SET value = jsonb_set(value::jsonb, '{variable,value}', '""'::jsonb)::json +WHERE typ = 'variable' + AND (value::jsonb -> 'variable' ->> 'is_secret')::boolean IS TRUE + AND value::jsonb -> 'variable' ? 'value'; diff --git a/backend/migrations/20260609165313_remove_draft_only.down.sql b/backend/migrations/20260609165313_remove_draft_only.down.sql new file mode 100644 index 0000000000..9594c93957 --- /dev/null +++ b/backend/migrations/20260609165313_remove_draft_only.down.sql @@ -0,0 +1,5 @@ +-- Restore the `draft_only` column. Irreversible data-wise: stubs deleted in +-- the up migration are gone and the column comes back NULL everywhere. +ALTER TABLE script ADD COLUMN draft_only BOOLEAN; +ALTER TABLE flow ADD COLUMN draft_only BOOLEAN; +ALTER TABLE app ADD COLUMN draft_only BOOLEAN; diff --git a/backend/migrations/20260609165313_remove_draft_only.up.sql b/backend/migrations/20260609165313_remove_draft_only.up.sql new file mode 100644 index 0000000000..9949de5540 --- /dev/null +++ b/backend/migrations/20260609165313_remove_draft_only.up.sql @@ -0,0 +1,99 @@ +-- `draft_only` items (scripts/flows/apps saved as a draft but never +-- deployed) lived as a stub row in their own table plus a `draft` row; the +-- stub is now redundant. For each stub: ensure a draft row exists at its +-- path (ON CONFLICT DO NOTHING preserves the real per-user draft most stubs +-- already have, synthesising an `email = NULL` legacy stand-in only for the +-- rare stub that lost its draft), drop the stub (version FKs cascade), then +-- drop the `draft_only` column. +-- ON CONFLICT targets `draft_pkey_legacy` — (workspace_id, path, typ) WHERE +-- email IS NULL. + +INSERT INTO draft (workspace_id, path, typ, email, value) +SELECT + s.workspace_id, + s.path, + 'script'::DRAFT_KIND, + NULL, + jsonb_strip_nulls(jsonb_build_object( + 'path', s.path, + 'summary', s.summary, + 'description', s.description, + 'content', s.content, + 'language', s.language, + 'kind', s.kind, + 'tag', s.tag, + 'schema', s.schema, + 'envs', to_jsonb(s.envs), + 'concurrent_limit', s.concurrent_limit, + 'concurrency_time_window_s', s.concurrency_time_window_s, + 'concurrency_key', s.concurrency_key, + 'cache_ttl', s.cache_ttl, + 'cache_ignore_s3_path', s.cache_ignore_s3_path, + 'dedicated_worker', s.dedicated_worker, + 'ws_error_handler_muted', s.ws_error_handler_muted, + 'priority', s.priority, + 'timeout', s.timeout, + 'delete_after_use', s.delete_after_use, + 'restart_unless_cancelled', s.restart_unless_cancelled, + 'visible_to_runner_only', s.visible_to_runner_only, + 'auto_kind', s.auto_kind, + 'has_preprocessor', s.has_preprocessor, + 'on_behalf_of_email', s.on_behalf_of_email, + 'assets', s.assets, + 'debounce_key', s.debounce_key, + 'debounce_delay_s', s.debounce_delay_s, + 'labels', to_jsonb(s.labels), + 'draft_triggers', '[]'::jsonb + ))::json +FROM script s +WHERE s.draft_only IS TRUE AND s.deleted IS FALSE AND s.archived IS FALSE +ON CONFLICT (workspace_id, path, typ) WHERE email IS NULL DO NOTHING; + +INSERT INTO draft (workspace_id, path, typ, email, value) +SELECT + f.workspace_id, + f.path, + 'flow'::DRAFT_KIND, + NULL, + jsonb_strip_nulls(jsonb_build_object( + 'path', f.path, + 'summary', f.summary, + 'description', f.description, + 'value', f.value, + 'schema', f.schema, + 'tag', f.tag, + 'dedicated_worker', f.dedicated_worker, + 'timeout', f.timeout, + 'visible_to_runner_only', f.visible_to_runner_only, + 'on_behalf_of_email', f.on_behalf_of_email, + 'ws_error_handler_muted', f.ws_error_handler_muted, + 'labels', to_jsonb(f.labels), + 'draft_triggers', '[]'::jsonb + ))::json +FROM flow f +WHERE f.draft_only IS TRUE AND f.archived IS FALSE +ON CONFLICT (workspace_id, path, typ) WHERE email IS NULL DO NOTHING; + +-- App drafts store the editor's working value directly in `draft.value` (an +-- `App` object for `app`, a `{files, runnables, data}` object for `raw_app`). +-- The deployed wrapper's summary/policy/custom_path aren't part of the App +-- type and aren't restored from a draft on reload, so they're dropped here. +INSERT INTO draft (workspace_id, path, typ, email, value) +SELECT + a.workspace_id, + a.path, + CASE WHEN av.raw_app THEN 'raw_app'::DRAFT_KIND ELSE 'app'::DRAFT_KIND END, + NULL, + av.value +FROM app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE a.draft_only IS TRUE +ON CONFLICT (workspace_id, path, typ) WHERE email IS NULL DO NOTHING; + +DELETE FROM script WHERE draft_only IS TRUE; +DELETE FROM flow WHERE draft_only IS TRUE; +DELETE FROM app WHERE draft_only IS TRUE; + +ALTER TABLE script DROP COLUMN draft_only; +ALTER TABLE flow DROP COLUMN draft_only; +ALTER TABLE app DROP COLUMN draft_only; diff --git a/backend/migrations/20260614075900_dedup_folder_labels.down.sql b/backend/migrations/20260614075900_dedup_folder_labels.down.sql new file mode 100644 index 0000000000..51650bf747 --- /dev/null +++ b/backend/migrations/20260614075900_dedup_folder_labels.down.sql @@ -0,0 +1,6 @@ +-- Restore the original passthrough definition (the one-time data cleanup is not reverted). +CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[] +LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$ + SELECT labels FROM folder + WHERE workspace_id = w_id AND item_path LIKE 'f/%' AND name = split_part(item_path, '/', 2) +$$; diff --git a/backend/migrations/20260614075900_dedup_folder_labels.up.sql b/backend/migrations/20260614075900_dedup_folder_labels.up.sql new file mode 100644 index 0000000000..fbf0be8177 --- /dev/null +++ b/backend/migrations/20260614075900_dedup_folder_labels.up.sql @@ -0,0 +1,33 @@ +-- Folder labels are exposed verbatim as `inherited_labels` (via folder_labels) and +-- rendered in keyed `{#each}` blocks in the UI, which throw `each_key_duplicate` on a +-- repeated key. The write paths now dedup, but make the read resilient regardless of +-- how a row was populated, and clean up any duplicates already persisted. + +-- Dedup while preserving first-seen order. +CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[] +LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$ + SELECT ( + SELECT array_agg(l ORDER BY first_ord) + FROM ( + SELECT u.l, min(u.ord) AS first_ord + FROM unnest(f.labels) WITH ORDINALITY AS u(l, ord) + GROUP BY u.l + ) deduped + ) + FROM folder f + WHERE f.workspace_id = w_id AND item_path LIKE 'f/%' AND f.name = split_part(item_path, '/', 2) +$$; + +-- One-time cleanup of rows that already contain duplicate labels, so direct reads of +-- folder.labels (folder list, editor) are also safe. +UPDATE folder +SET labels = ( + SELECT array_agg(l ORDER BY first_ord) + FROM ( + SELECT u.l, min(u.ord) AS first_ord + FROM unnest(labels) WITH ORDINALITY AS u(l, ord) + GROUP BY u.l + ) deduped +) +WHERE labels IS NOT NULL + AND cardinality(labels) <> (SELECT count(DISTINCT x) FROM unnest(labels) AS x); diff --git a/backend/migrations/20260615204237_fix_migrated_draft_only_created_at.down.sql b/backend/migrations/20260615204237_fix_migrated_draft_only_created_at.down.sql new file mode 100644 index 0000000000..3074d34959 --- /dev/null +++ b/backend/migrations/20260615204237_fix_migrated_draft_only_created_at.down.sql @@ -0,0 +1,4 @@ +-- Irreversible: the original `created_at` values were already lost by +-- 20260609165313 (it defaulted them to the migration's now()); this migration +-- only changed them from the migration timestamp to the epoch, so there is +-- nothing to restore. diff --git a/backend/migrations/20260615204237_fix_migrated_draft_only_created_at.up.sql b/backend/migrations/20260615204237_fix_migrated_draft_only_created_at.up.sql new file mode 100644 index 0000000000..2459dae9cf --- /dev/null +++ b/backend/migrations/20260615204237_fix_migrated_draft_only_created_at.up.sql @@ -0,0 +1,33 @@ +-- Repair `created_at` for the draft-only items migrated by +-- 20260609165313_remove_draft_only. That migration inserted the legacy +-- (email IS NULL) draft stubs without an explicit `created_at`, so every row +-- it created defaulted to the migration's `now()` — which is +-- `transaction_timestamp()`, constant for the whole transaction. They all +-- landed at the migration instant and, sorted newest-first, flooded the top +-- of the home list. +-- +-- The original per-item timestamps are unrecoverable (the source script/flow/ +-- app rows were deleted by that migration and the draft value carries no +-- timestamp), so reset them to the epoch: these never-deployed, never-resaved +-- stubs sort to the bottom instead of the top. Editing one later bumps its +-- `created_at` to now() and it floats back up naturally. +-- +-- Identification is exact and safe. sqlx applies a transactional migration and +-- inserts its `_sqlx_migrations` bookkeeping row in ONE transaction, both via +-- `DEFAULT now()`, so that row's `installed_on` is byte-identical to the +-- `created_at` of every row the migration inserted. Matching on it touches +-- only those rows and skips any edited since (their `created_at` was bumped, +-- so it no longer equals `installed_on`). If the values somehow don't match, +-- this updates nothing — it can never clobber a real draft. +-- +-- Both columns are TIMESTAMPTZ (draft.created_at since +-- 20260514233244_convert_draft_created_at_to_timestamptz), so this is an exact +-- instant comparison — no implicit timestamp/timestamptz cast or timezone +-- sensitivity. +UPDATE draft d +SET created_at = 'epoch' +FROM _sqlx_migrations m +WHERE m.version = 20260609165313 + AND d.email IS NULL + AND d.typ IN ('script', 'flow', 'app', 'raw_app') + AND d.created_at = m.installed_on; diff --git a/backend/migrations/20260616090412_add_data_pipeline_draft_kind.down.sql b/backend/migrations/20260616090412_add_data_pipeline_draft_kind.down.sql new file mode 100644 index 0000000000..fcb5ef5bbf --- /dev/null +++ b/backend/migrations/20260616090412_add_data_pipeline_draft_kind.down.sql @@ -0,0 +1,2 @@ +-- Postgres cannot drop a single enum value; leaving 'data_pipeline' in +-- DRAFT_KIND is harmless on rollback. diff --git a/backend/migrations/20260616090412_add_data_pipeline_draft_kind.up.sql b/backend/migrations/20260616090412_add_data_pipeline_draft_kind.up.sql new file mode 100644 index 0000000000..38ae68ddcb --- /dev/null +++ b/backend/migrations/20260616090412_add_data_pipeline_draft_kind.up.sql @@ -0,0 +1,6 @@ +-- A `data_pipeline` draft bundles every unsaved pipeline script of a folder +-- into a single row keyed at the folder path (typ has no deployed backing +-- table — see UserDraftItemKind::deployed_table). Lets the asset-graph view +-- store its in-flight drafts in the per-user DB draft sync instead of +-- browser-local storage. +ALTER TYPE DRAFT_KIND ADD VALUE IF NOT EXISTS 'data_pipeline'; diff --git a/backend/migrations/20260616120048_backfill_legacy_draft_emails.down.sql b/backend/migrations/20260616120048_backfill_legacy_draft_emails.down.sql new file mode 100644 index 0000000000..a0ef6d0cfe --- /dev/null +++ b/backend/migrations/20260616120048_backfill_legacy_draft_emails.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data backfill: once an email is attached, the row is +-- indistinguishable from a draft that was always per-user owned, so the +-- original NULL state cannot be reconstructed. No-op on revert. diff --git a/backend/migrations/20260616120048_backfill_legacy_draft_emails.up.sql b/backend/migrations/20260616120048_backfill_legacy_draft_emails.up.sql new file mode 100644 index 0000000000..cdf4b19c79 --- /dev/null +++ b/backend/migrations/20260616120048_backfill_legacy_draft_emails.up.sql @@ -0,0 +1,28 @@ +-- Backfill the owner `email` on legacy drafts (rows persisted before per-user +-- sync, hence `email IS NULL`). A user-owned draft path is `u//...`, +-- so resolve `` against `usr` for the same workspace and adopt that +-- user's email. +-- +-- Guards: +-- - the resolved email must exist in `password` (the `draft_password_fkey` +-- target), or the UPDATE would violate the FK; +-- - skip rows that would collide with an existing per-user draft at the same +-- (workspace_id, path, typ, email) under the `draft_pkey_with_user` partial +-- unique index — the per-user row is authoritative, so the legacy row is +-- left untouched. +UPDATE draft d +SET email = u.email +FROM usr u +WHERE d.email IS NULL + AND split_part(d.path, '/', 1) = 'u' + AND split_part(d.path, '/', 2) <> '' + AND u.workspace_id = d.workspace_id + AND u.username = split_part(d.path, '/', 2) + AND EXISTS (SELECT 1 FROM password p WHERE p.email = u.email) + AND NOT EXISTS ( + SELECT 1 FROM draft d2 + WHERE d2.workspace_id = d.workspace_id + AND d2.path = d.path + AND d2.typ = d.typ + AND d2.email = u.email + ); diff --git a/backend/migrations/20260616210529_cleanup_orphaned_workspace_diff.down.sql b/backend/migrations/20260616210529_cleanup_orphaned_workspace_diff.down.sql new file mode 100644 index 0000000000..6e1600b72a --- /dev/null +++ b/backend/migrations/20260616210529_cleanup_orphaned_workspace_diff.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data cleanup: deleted orphaned rows and reset cached verdicts +-- cannot be reconstructed. No-op on rollback. +SELECT 1; diff --git a/backend/migrations/20260616210529_cleanup_orphaned_workspace_diff.up.sql b/backend/migrations/20260616210529_cleanup_orphaned_workspace_diff.up.sql new file mode 100644 index 0000000000..2d4f82ab83 --- /dev/null +++ b/backend/migrations/20260616210529_cleanup_orphaned_workspace_diff.up.sql @@ -0,0 +1,31 @@ +-- workspace_diff / skip_workspace_diff_tally are keyed by workspace id with no FK +-- cascade, so until the matching delete_workspace cleanup landed, deleting a fork +-- left its cached diff rows behind. Workspace ids are reused (recreating a fork +-- under the same name), so those orphaned rows leaked onto the new fork and +-- produced a spurious "changes not visible" warning that hid the deploy button. + +-- Drop rows referencing workspaces that no longer exist (leftovers from past deletes). +DELETE FROM workspace_diff +WHERE source_workspace_id NOT IN (SELECT id FROM workspace) + OR fork_workspace_id NOT IN (SELECT id FROM workspace); + +DELETE FROM skip_workspace_diff_tally +WHERE workspace_id NOT IN (SELECT id FROM workspace); + +-- Reused-id victims keep a skip-tally row pointing at a LIVE workspace, so the +-- orphan cleanup above can't reach them. compare_workspaces consults +-- skip_workspace_diff_tally first and short-circuits to an empty comparison, so a +-- stale skip row would also defeat the has_changes reset below. A genuinely +-- skipped workspace is excluded from tallying and therefore never has +-- workspace_diff rows — so a skip row that coexists with workspace_diff rows for +-- that id is provably leaked from a previous occupant of a reused id. Drop those. +DELETE FROM skip_workspace_diff_tally s +WHERE EXISTS ( + SELECT 1 FROM workspace_diff d WHERE d.fork_workspace_id = s.workspace_id +); + +-- Remaining reused-id victims keep workspace_diff rows pointing at live +-- workspaces, so they can't be told apart from valid cache. Reset the cached +-- verdict to force a recompute on the next compare; compare_workspaces +-- re-evaluates NULL rows and corrects or deletes them. +UPDATE workspace_diff SET has_changes = NULL; diff --git a/backend/migrations/20260617144417_add_ai_skill.down.sql b/backend/migrations/20260617144417_add_ai_skill.down.sql new file mode 100644 index 0000000000..8fd6e1606d --- /dev/null +++ b/backend/migrations/20260617144417_add_ai_skill.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS ai_skill; diff --git a/backend/migrations/20260617144417_add_ai_skill.up.sql b/backend/migrations/20260617144417_add_ai_skill.up.sql new file mode 100644 index 0000000000..b9a5003572 --- /dev/null +++ b/backend/migrations/20260617144417_add_ai_skill.up.sql @@ -0,0 +1,15 @@ +-- Workspace-scoped AI chat skills (Claude/Codex-style SKILL.md instructions). +-- `name` is the skill folder slug; `description` is advertised in the AI chat +-- system prompt, `instructions` is the SKILL.md body fetched on demand. +CREATE TABLE ai_skill ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE, + name VARCHAR(255) NOT NULL, + description TEXT NOT NULL, + instructions TEXT NOT NULL, + edited_at TIMESTAMPTZ NOT NULL DEFAULT now(), + edited_by VARCHAR(255) NOT NULL DEFAULT '', + PRIMARY KEY (workspace_id, name) +); + +GRANT ALL ON ai_skill TO windmill_user; +GRANT ALL ON ai_skill TO windmill_admin; diff --git a/backend/migrations/20260619091631_grant_notify_event_to_windmill_roles.down.sql b/backend/migrations/20260619091631_grant_notify_event_to_windmill_roles.down.sql new file mode 100644 index 0000000000..7d29137db3 --- /dev/null +++ b/backend/migrations/20260619091631_grant_notify_event_to_windmill_roles.down.sql @@ -0,0 +1,4 @@ +REVOKE ALL ON notify_event FROM windmill_user; +REVOKE ALL ON notify_event FROM windmill_admin; +REVOKE ALL ON SEQUENCE notify_event_id_seq FROM windmill_user; +REVOKE ALL ON SEQUENCE notify_event_id_seq FROM windmill_admin; diff --git a/backend/migrations/20260619091631_grant_notify_event_to_windmill_roles.up.sql b/backend/migrations/20260619091631_grant_notify_event_to_windmill_roles.up.sql new file mode 100644 index 0000000000..0d0ac84824 --- /dev/null +++ b/backend/migrations/20260619091631_grant_notify_event_to_windmill_roles.up.sql @@ -0,0 +1,14 @@ +-- The notify_event table (migration 20260203172950_polling_based_events) was +-- created relying on ALTER DEFAULT PRIVILEGES to grant access to windmill_user +-- and windmill_admin. Those default privileges only apply to objects created by +-- the role that set them (migration 20250205131523), so deployments whose +-- migration runner is a different role leave notify_event ungranted. Trigger +-- inserts were worked around with SECURITY DEFINER, but direct application +-- inserts (clear_static_asset_usage in assets.rs, restart_worker_group in +-- settings) run as the invoking role and fail with "permission denied for table +-- notify_event". Grant explicitly to guarantee access regardless of who ran the +-- migrations. +GRANT ALL ON notify_event TO windmill_user; +GRANT ALL ON notify_event TO windmill_admin; +GRANT ALL ON SEQUENCE notify_event_id_seq TO windmill_user; +GRANT ALL ON SEQUENCE notify_event_id_seq TO windmill_admin; diff --git a/backend/migrations/20260619112847_grant_script_trigger_to_windmill_roles.down.sql b/backend/migrations/20260619112847_grant_script_trigger_to_windmill_roles.down.sql new file mode 100644 index 0000000000..c84fe703bb --- /dev/null +++ b/backend/migrations/20260619112847_grant_script_trigger_to_windmill_roles.down.sql @@ -0,0 +1,4 @@ +REVOKE ALL ON script_trigger FROM windmill_user; +REVOKE ALL ON script_trigger FROM windmill_admin; +REVOKE ALL ON SEQUENCE script_trigger_id_seq FROM windmill_user; +REVOKE ALL ON SEQUENCE script_trigger_id_seq FROM windmill_admin; diff --git a/backend/migrations/20260619112847_grant_script_trigger_to_windmill_roles.up.sql b/backend/migrations/20260619112847_grant_script_trigger_to_windmill_roles.up.sql new file mode 100644 index 0000000000..1031f22a6a --- /dev/null +++ b/backend/migrations/20260619112847_grant_script_trigger_to_windmill_roles.up.sql @@ -0,0 +1,14 @@ +-- The script_trigger table (migration 20260423050000_script_trigger) was +-- created relying on ALTER DEFAULT PRIVILEGES to grant access to windmill_user +-- and windmill_admin. Those default privileges only apply to objects created by +-- the role that set them (migration 20250205131523), so deployments whose +-- migration runner is a different role leave script_trigger ungranted. Direct +-- application writes run as the invoking role (clear_script_triggers and +-- insert_script_trigger in windmill-common/src/assets.rs, every script save) +-- and fail with "permission denied for table script_trigger". Grant explicitly +-- to guarantee access regardless of who ran the migrations (same fix as +-- notify_event in 20260619091631). +GRANT ALL ON script_trigger TO windmill_user; +GRANT ALL ON script_trigger TO windmill_admin; +GRANT ALL ON SEQUENCE script_trigger_id_seq TO windmill_user; +GRANT ALL ON SEQUENCE script_trigger_id_seq TO windmill_admin; diff --git a/backend/migrations/20260619113554_scrub_draft_value_nul.down.sql b/backend/migrations/20260619113554_scrub_draft_value_nul.down.sql new file mode 100644 index 0000000000..098a525396 --- /dev/null +++ b/backend/migrations/20260619113554_scrub_draft_value_nul.down.sql @@ -0,0 +1,2 @@ +-- Irreversible: a stripped NUL cannot be restored (and was never meaningful). +SELECT 1; diff --git a/backend/migrations/20260619113554_scrub_draft_value_nul.up.sql b/backend/migrations/20260619113554_scrub_draft_value_nul.up.sql new file mode 100644 index 0000000000..1ae927bae6 --- /dev/null +++ b/backend/migrations/20260619113554_scrub_draft_value_nul.up.sql @@ -0,0 +1,30 @@ +-- One-time cleanup of drafts whose `json` value carries a real U+0000 (NUL) +-- escape — storable only because `draft.value` is `json`, not `jsonb`. Such a +-- value makes any `->>`/`to_jsonb` extraction raise `22P05`, which 500'd +-- GET /drafts/list. New writes are sanitized in the application layer +-- (update_draft → strip_json_nul); this fixes rows written before that landed. +-- +-- Only genuinely-poisoned rows are touched: a real NUL makes `value::jsonb` +-- raise, which distinguishes it from a legitimately escaped backslash sequence +-- (which `jsonb` accepts). The text replace handles the real-world shape — a NUL +-- inside a text field. A contrived value where stripping the escape leaves +-- invalid JSON is left as-is (and can no longer be created). +DO $$ +DECLARE r RECORD; +BEGIN + FOR r IN + SELECT id, value FROM draft WHERE position(E'\\u0000' in value::text) > 0 + LOOP + BEGIN + PERFORM r.value::jsonb; -- not poisoned (legit escaped backslash): skip + EXCEPTION WHEN others THEN + BEGIN + UPDATE draft + SET value = replace(r.value::text, E'\\u0000', '')::json + WHERE id = r.id; + EXCEPTION WHEN others THEN + NULL; -- pathological shape; cannot strip in SQL, no longer creatable + END; + END; + END LOOP; +END $$; diff --git a/backend/migrations/20260619150718_native_retry_settings.down.sql b/backend/migrations/20260619150718_native_retry_settings.down.sql new file mode 100644 index 0000000000..459ebe4eaf --- /dev/null +++ b/backend/migrations/20260619150718_native_retry_settings.down.sql @@ -0,0 +1,4 @@ +ALTER TABLE runnable_settings +DROP COLUMN IF EXISTS retry_settings; + +DROP TABLE IF EXISTS retry_settings; diff --git a/backend/migrations/20260619150718_native_retry_settings.up.sql b/backend/migrations/20260619150718_native_retry_settings.up.sql new file mode 100644 index 0000000000..e03192c710 --- /dev/null +++ b/backend/migrations/20260619150718_native_retry_settings.up.sql @@ -0,0 +1,16 @@ +CREATE TABLE IF NOT EXISTS retry_settings( + hash BIGINT PRIMARY KEY, + constant_attempts INTEGER, + constant_seconds INTEGER, + exponential_attempts INTEGER, + exponential_multiplier INTEGER, + exponential_seconds INTEGER, + exponential_random_factor INTEGER, + retry_if_expr TEXT +); + +ALTER TABLE runnable_settings +ADD COLUMN IF NOT EXISTS retry_settings BIGINT DEFAULT NULL; + +GRANT ALL ON retry_settings TO windmill_admin; +GRANT ALL ON retry_settings TO windmill_user; diff --git a/backend/migrations/20260619170118_add_materialized_partition.down.sql b/backend/migrations/20260619170118_add_materialized_partition.down.sql new file mode 100644 index 0000000000..4932338956 --- /dev/null +++ b/backend/migrations/20260619170118_add_materialized_partition.down.sql @@ -0,0 +1,3 @@ +DROP INDEX IF EXISTS idx_materialized_partition_asset_status; +DROP TABLE IF EXISTS materialized_partition; +DROP TYPE IF EXISTS MATERIALIZATION_STATUS; diff --git a/backend/migrations/20260619170118_add_materialized_partition.up.sql b/backend/migrations/20260619170118_add_materialized_partition.up.sql new file mode 100644 index 0000000000..da5f806e4f --- /dev/null +++ b/backend/migrations/20260619170118_add_materialized_partition.up.sql @@ -0,0 +1,29 @@ +-- Per-partition materialization state for managed `// materialize` assets. +-- One row per (asset, partition): the latest materialization of that slice. +-- Drives: the partition-status grid (CE observability), run-stale/gap +-- detection, and the EE backfill worklist (missing/failed partitions). The +-- `partition` column uses '' as the sentinel for an unpartitioned (whole-table) +-- materialization, since partition is part of the primary key and cannot be +-- NULL. +CREATE TYPE MATERIALIZATION_STATUS AS ENUM ('running', 'materialized', 'failed'); + +CREATE TABLE IF NOT EXISTS materialized_partition ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + asset_kind ASSET_KIND NOT NULL, + asset_path VARCHAR(255) NOT NULL, + partition TEXT NOT NULL DEFAULT '', + status MATERIALIZATION_STATUS NOT NULL, + -- DuckLake snapshot id produced by the write; NULL while running / on + -- failure. The pin that makes downstream reads reproducible. + snapshot_id BIGINT, + row_count BIGINT, + job_id UUID, + materialized_at TIMESTAMPTZ NOT NULL DEFAULT now(), + error TEXT, + PRIMARY KEY (workspace_id, asset_kind, asset_path, partition) +); + +-- Backfill enumeration / grid "show only gaps": filter an asset's partitions +-- by status without scanning the whole table. +CREATE INDEX IF NOT EXISTS idx_materialized_partition_asset_status + ON materialized_partition (workspace_id, asset_kind, asset_path, status); diff --git a/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql b/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql new file mode 100644 index 0000000000..3d532de9cb --- /dev/null +++ b/backend/migrations/20260624103600_repair_folder_labels_search_path.down.sql @@ -0,0 +1,3 @@ +-- No-op: this migration only re-pins the function's search_path. Reverting would +-- mean restoring the hardcoded `SET search_path = public`, which is the very bug +-- this repairs, so there is nothing to undo. diff --git a/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql b/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql new file mode 100644 index 0000000000..f8e448891f --- /dev/null +++ b/backend/migrations/20260624103600_repair_folder_labels_search_path.up.sql @@ -0,0 +1,22 @@ +-- Repair instances that already applied the folder-labels migrations while the +-- function hardcoded `SET search_path = public`. On a non-public schema (PG_SCHEMA) +-- the function was pinned to `public`, so at runtime it read the wrong `folder` +-- table (or a stray public.folder) instead of the workspace's real one. +-- +-- `FROM CURRENT` snapshots the migration connection's search_path (the actual +-- Windmill schema) into the function, keeping the SECURITY DEFINER injection +-- hardening. On public-schema installs this re-pins to `public`, i.e. a no-op. +-- Idempotent: redefining with the same body is harmless on already-correct installs. +CREATE OR REPLACE FUNCTION folder_labels(w_id text, item_path text) RETURNS text[] +LANGUAGE sql STABLE SECURITY DEFINER SET search_path FROM CURRENT AS $$ + SELECT ( + SELECT array_agg(l ORDER BY first_ord) + FROM ( + SELECT u.l, min(u.ord) AS first_ord + FROM unnest(f.labels) WITH ORDINALITY AS u(l, ord) + GROUP BY u.l + ) deduped + ) + FROM folder f + WHERE f.workspace_id = w_id AND item_path LIKE 'f/%' AND f.name = split_part(item_path, '/', 2) +$$; diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql new file mode 100644 index 0000000000..3c69d46fc8 --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.down.sql @@ -0,0 +1,3 @@ +-- Irreversible data backfill: once a raw app's draft is retyped to 'raw_app' it +-- is indistinguishable from one saved as 'raw_app' by the per-kind code, so the +-- original typ='app' state cannot be reconstructed. No-op on revert. diff --git a/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql new file mode 100644 index 0000000000..d4b068a80c --- /dev/null +++ b/backend/migrations/20260624105229_realign_legacy_raw_app_draft_kind.up.sql @@ -0,0 +1,35 @@ +-- The pre-per-user `DRAFT_TYPE` enum had only ('script','flow','app'): a raw +-- app's draft was therefore stored as typ='app'. The new model splits app vs +-- raw_app into distinct draft kinds chosen from the deployed app's `raw_app` +-- flag, so a raw app's pre-migration draft is invisible to the per-kind lookups +-- (editor overlay, migrate-legacy, get-for-user), which all query typ='raw_app'. +-- Realign every such draft (any owner, including the legacy NULL-email row) to +-- 'raw_app' when the deployed app at that path is a raw app. + +-- Drop, don't retype, a stale 'app' row when a 'raw_app' draft already exists +-- for the same owner (the newer 'raw_app' row, saved with the per-kind code, is +-- authoritative) — retyping would collide on the draft_pkey_with_user / +-- draft_pkey_legacy partial unique indexes over (workspace_id, path, typ, email). +DELETE FROM draft d +USING app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE + AND EXISTS ( + SELECT 1 FROM draft d2 + WHERE d2.workspace_id = d.workspace_id + AND d2.path = d.path + AND d2.typ = 'raw_app' + AND d2.email IS NOT DISTINCT FROM d.email + ); + +UPDATE draft d +SET typ = 'raw_app' +FROM app a +JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] +WHERE d.typ = 'app' + AND a.workspace_id = d.workspace_id + AND a.path = d.path + AND av.raw_app IS TRUE; diff --git a/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql b/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql new file mode 100644 index 0000000000..0a09f0f42a --- /dev/null +++ b/backend/migrations/20260624221000_native_retry_attempt_marker.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS native_retry_attempt; diff --git a/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql b/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql new file mode 100644 index 0000000000..e193fb8b8f --- /dev/null +++ b/backend/migrations/20260624221000_native_retry_attempt_marker.up.sql @@ -0,0 +1,18 @@ +-- Sparse marker: one row per native retry attempt (a re-pushed Script job that +-- gained a retry). Presence = "this job is a native retry attempt"; `attempt` is +-- the chain position (1-based). Lets consumers — asset-cascade dispatch, the +-- per-occurrence schedule-handler counting, and the run-page chain — identify +-- retries explicitly instead of inferring from incidental fields (parent_job + +-- runnable + flow_innermost), which collides with schedule handlers and WAC +-- inline children. Sparse: only failed-and-retried jobs under a retry policy +-- produce rows. Lifecycle: removed with their job in the retention sweep (no FK, +-- to keep the bulk job delete cheap). +CREATE TABLE IF NOT EXISTS native_retry_attempt ( + job_id UUID PRIMARY KEY, + -- `integer` matches the retry policy's i32 attempt count; avoids any narrowing + -- on the maybe_enqueue read/write path. + attempt INTEGER NOT NULL +); + +GRANT ALL ON native_retry_attempt TO windmill_admin; +GRANT ALL ON native_retry_attempt TO windmill_user; diff --git a/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql new file mode 100644 index 0000000000..9399c18a6b --- /dev/null +++ b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.down.sql @@ -0,0 +1,16 @@ +-- Re-establish the ON DELETE CASCADE foreign keys. Once the cascades were gone the explicit +-- delete paths may have left orphan rows (or none were created); purge any orphans first so +-- the constraints can be validated. +DELETE FROM dispatch_event WHERE producer_job_id NOT IN (SELECT id FROM v2_job); +DELETE FROM flow_conversation_message WHERE job_id IS NOT NULL AND job_id NOT IN (SELECT id FROM v2_job); +DELETE FROM zombie_job_counter WHERE job_id NOT IN (SELECT id FROM v2_job); + +ALTER TABLE dispatch_event + ADD CONSTRAINT dispatch_event_producer_job_id_fkey + FOREIGN KEY (producer_job_id) REFERENCES v2_job(id) ON DELETE CASCADE; +ALTER TABLE flow_conversation_message + ADD CONSTRAINT flow_conversation_message_job_id_fkey + FOREIGN KEY (job_id) REFERENCES v2_job(id) ON DELETE CASCADE; +ALTER TABLE zombie_job_counter + ADD CONSTRAINT zombie_job_counter_job_id_fkey + FOREIGN KEY (job_id) REFERENCES v2_job(id) ON DELETE CASCADE; diff --git a/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql new file mode 100644 index 0000000000..ebaa555b0e --- /dev/null +++ b/backend/migrations/20260625092813_drop_v2_job_side_table_cascades.up.sql @@ -0,0 +1,11 @@ +-- Drop the ON DELETE CASCADE foreign keys from v2_job's sparse side tables. +-- These cascades made bulk retention deletes (DELETE FROM v2_job WHERE id = ANY(...)) +-- fire a per-row RI trigger for each FK; for flow_conversation_message, whose job_id +-- column is unindexed, that meant a sequential scan of the whole table per deleted row +-- (benchmarked at ~14x the base delete time). Deletion of these tables is now handled +-- explicitly by windmill_common::jobs::delete_jobs and the workspace/export delete paths, +-- following the existing no-FK precedent of job_logs / job_stats / native_retry_attempt. + +ALTER TABLE dispatch_event DROP CONSTRAINT IF EXISTS dispatch_event_producer_job_id_fkey; +ALTER TABLE flow_conversation_message DROP CONSTRAINT IF EXISTS flow_conversation_message_job_id_fkey; +ALTER TABLE zombie_job_counter DROP CONSTRAINT IF EXISTS zombie_job_counter_job_id_fkey; diff --git a/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql b/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql new file mode 100644 index 0000000000..0e90a4b693 --- /dev/null +++ b/backend/migrations/20260625130855_index_resume_job_flow_fk.down.sql @@ -0,0 +1 @@ +DROP INDEX IF EXISTS ix_resume_job_flow; diff --git a/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql b/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql new file mode 100644 index 0000000000..7edded95e3 --- /dev/null +++ b/backend/migrations/20260625130855_index_resume_job_flow_fk.up.sql @@ -0,0 +1,5 @@ +-- Index resume_job's FK column to v2_job_queue. Without it, every per-job-completion +-- DELETE FROM v2_job_queue (the system's hottest delete path) cascades into a sequential +-- scan of resume_job. The table is small (only currently-suspended flows), so the scan is +-- cheap today, but the index makes the cascade an index probe and removes the footgun. +CREATE INDEX IF NOT EXISTS ix_resume_job_flow ON resume_job (flow); diff --git a/backend/migrations/20260625135355_debounce_batch_consumed.down.sql b/backend/migrations/20260625135355_debounce_batch_consumed.down.sql new file mode 100644 index 0000000000..8ce64f87ac --- /dev/null +++ b/backend/migrations/20260625135355_debounce_batch_consumed.down.sql @@ -0,0 +1,4 @@ +DROP INDEX IF EXISTS idx_v2_job_debounce_batch_consumed_at; +ALTER TABLE v2_job_debounce_batch + DROP COLUMN IF EXISTS consumed_at, + DROP COLUMN IF EXISTS consumed_by; diff --git a/backend/migrations/20260625135355_debounce_batch_consumed.up.sql b/backend/migrations/20260625135355_debounce_batch_consumed.up.sql new file mode 100644 index 0000000000..2559cbb74e --- /dev/null +++ b/backend/migrations/20260625135355_debounce_batch_consumed.up.sql @@ -0,0 +1,17 @@ +-- Claim-based, exactly-once consumption of debounce batches. +-- A batch row is "claimed" by the survivor that accumulates its args. Stamping the +-- row consumed (instead of deleting it) lets a later-pulled survivor of the same +-- batch tell "my contribution was already processed" (consumed_at set -> no-op) +-- apart from "I was never batched" (no row at all; CE / legacy -> run my own args). +-- consumed_at doubles as the GC timestamp. NULL = not yet consumed. +-- consumed_by records which job claimed the row, so a job that is re-pulled (e.g. +-- crash recovery) can tell its own prior claim (keep its accumulated args) apart from +-- a sibling survivor having swept it in (run empty). +ALTER TABLE v2_job_debounce_batch + ADD COLUMN IF NOT EXISTS consumed_at TIMESTAMP WITH TIME ZONE, + ADD COLUMN IF NOT EXISTS consumed_by UUID; + +-- Keeps the GC sweep (delete consumed rows past a grace period) cheap. +CREATE INDEX IF NOT EXISTS idx_v2_job_debounce_batch_consumed_at + ON v2_job_debounce_batch (consumed_at) + WHERE consumed_at IS NOT NULL; diff --git a/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql b/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql new file mode 100644 index 0000000000..8a653a9842 --- /dev/null +++ b/backend/migrations/20260626095840_add_materialized_asset_schema.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS materialized_asset_schema; diff --git a/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql b/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql new file mode 100644 index 0000000000..9c548aded7 --- /dev/null +++ b/backend/migrations/20260626095840_add_materialized_asset_schema.up.sql @@ -0,0 +1,37 @@ +-- Captured output schema of a managed `// materialize` asset (gap #2a). +-- After a managed materialize commits, the worker DESCRIBEs the written table +-- and records its column list here as asset-level metadata. Schema is a +-- property of the asset/table, not of a partition slice, so it lives in its own +-- table keyed by (workspace, asset_kind, asset_path) rather than as a column on +-- materialized_partition (which would duplicate the identical schema across +-- every partition row). This is the producer-side capture that #2b (save-time +-- consumer-ref contract enforcement) reads back. +-- +-- Versioning across re-materializations: a new `version` row is inserted only +-- when the captured column set differs from the latest stored version for the +-- asset; an unchanged re-materialize re-affirms the latest row in place. So the +-- table is a compact schema-evolution history and MAX(version) is the current +-- contract. +CREATE TABLE IF NOT EXISTS materialized_asset_schema ( + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE ON UPDATE CASCADE, + asset_kind ASSET_KIND NOT NULL, + asset_path VARCHAR(255) NOT NULL, + -- Monotonic per (workspace, asset_kind, asset_path), starting at 1; only + -- bumped when the schema actually changes. + version BIGINT NOT NULL, + -- The captured columns, ordered as the table presents them: + -- [{"name": "...", "type": "..."}, ...]. + columns JSONB NOT NULL, + -- DuckLake snapshot the schema was captured from (NULL for non-ducklake / + -- substrates without snapshots). + snapshot_id BIGINT, + job_id UUID, + captured_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (workspace_id, asset_kind, asset_path, version) +); + +-- Default privileges (migration 20250205131523) only apply to objects created +-- by the role that set them, so grant explicitly — the API reads/writes this +-- table as the invoking role (same fix as script_trigger in 20260619112847). +GRANT ALL ON materialized_asset_schema TO windmill_user; +GRANT ALL ON materialized_asset_schema TO windmill_admin; diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql new file mode 100644 index 0000000000..1ce2c852b7 --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.down.sql @@ -0,0 +1,21 @@ +-- Restore the previous anchor: epoch sentinel + preserve-cursor (DO NOTHING). +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', '1970-01-01T00:00:00+00:00' + ) + ) + ON CONFLICT (name) DO NOTHING; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +DROP FUNCTION IF EXISTS audit_logs_s3_oldest_inflight_ts(); diff --git a/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql new file mode 100644 index 0000000000..ff2ed110c6 --- /dev/null +++ b/backend/migrations/20260626132251_audit_logs_s3_reanchor_on_enable.up.sql @@ -0,0 +1,86 @@ +-- Anchors the audit→object-store export cursor when the setting is enabled. +-- +-- `last_ts`/`last_oldest_inflight_ts` must be a *recent* floor, not epoch: the +-- export's `timestamp >= floor` predicate is the only partition-pruning bound (the +-- `age(xmin)` cursor is unindexable), so an epoch floor would scan the whole +-- `audit_partitioned` table on the first run and never finish under a +-- `statement_timeout`. The floor must be at or below the timestamp of any row whose +-- xid >= this snapshot xmin; the oldest in-flight `xact_start` is that bound when +-- stats are visible (no restricted role / prepared 2PC txn), else a bounded 7-day +-- window. +-- +-- `ON CONFLICT DO UPDATE ... WHERE last_xmin <` keeps the cursor monotonic: a +-- re-enable re-anchors it forward (so the export resumes from ~now rather than +-- rescanning the disabled gap — that gap is the backfill's job), but it never moves +-- backwards, so it is HA-safe and can't be regressed by a slower concurrent writer. +-- +-- The task name literal must match +-- `windmill_common::global_settings::AUDIT_LOGS_S3_EXPORT_TASK`. + +-- Oldest in-flight `xact_start` when this role can observe *all* sessions (so the +-- min is a true cluster-wide bound), else NULL — callers substitute a conservative +-- window. The stats-visibility check goes through `is_superuser` (a preset GUC, no +-- catalog read) first, then a best-effort `pg_has_role` probe guarded by EXCEPTION: +-- `pg_has_role` reads `pg_authid`, which some managed providers (e.g. Cloud SQL) +-- forbid even to read from an elevated context, raising "Modifying pg_authid or +-- pg_auth_members is not allowed in elevated context". The EXCEPTION block runs in +-- its own subtransaction, so a failure there returns NULL (→ conservative fallback) +-- without aborting the caller — the migration-time UPDATE below, the trigger, or the +-- export task, none of which must fail just because the optimization is unavailable. +CREATE OR REPLACE FUNCTION audit_logs_s3_oldest_inflight_ts() +RETURNS timestamptz AS $$ +DECLARE + v_can_read_all_stats boolean := current_setting('is_superuser') = 'on'; +BEGIN + IF NOT v_can_read_all_stats THEN + BEGIN + v_can_read_all_stats := pg_has_role(current_user, 'pg_read_all_stats', 'USAGE'); + EXCEPTION WHEN OTHERS THEN + v_can_read_all_stats := false; + END; + END IF; + IF v_can_read_all_stats AND NOT EXISTS (SELECT 1 FROM pg_prepared_xacts) THEN + RETURN (SELECT min(xact_start) FROM pg_stat_activity WHERE xact_start IS NOT NULL); + END IF; + RETURN NULL; +END; +$$ LANGUAGE plpgsql; + +CREATE OR REPLACE FUNCTION audit_logs_s3_anchor_on_enable() +RETURNS TRIGGER AS $$ +BEGIN + IF NEW.value = to_jsonb(true) + AND (TG_OP = 'INSERT' OR OLD.value IS DISTINCT FROM NEW.value) THEN + INSERT INTO background_task_state (name, value) + VALUES ( + 'audit_logs_s3_export', + jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', now(), + 'last_oldest_inflight_ts', + COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days') + ) + ) + ON CONFLICT (name) DO UPDATE + SET value = EXCLUDED.value + WHERE (background_task_state.value->>'last_xmin')::bigint + < (EXCLUDED.value->>'last_xmin')::bigint; + END IF; + RETURN NEW; +END; +$$ LANGUAGE plpgsql; + +-- Recovery for a legacy epoch-sentinel checkpoint (`last_ts = epoch`, never +-- drained). It cannot be safely resumed: its un-drained backlog can be arbitrarily +-- old, so stamping a recent floor over the old xmin would prune the older rows while +-- the cursor advanced past them (silent loss), and keeping the epoch floor would +-- reintroduce the full scan. Re-anchor it to now like a fresh enable; the pre-anchor +-- window is recoverable via the opt-in backfill, not silently dropped. +UPDATE background_task_state +SET value = jsonb_build_object( + 'last_xmin', txid_snapshot_xmin(txid_current_snapshot())::bigint, + 'last_ts', to_jsonb(now()), + 'last_oldest_inflight_ts', + to_jsonb(COALESCE(audit_logs_s3_oldest_inflight_ts(), now() - interval '7 days'))) +WHERE name = 'audit_logs_s3_export' + AND (value->>'last_ts')::timestamptz <= 'epoch'::timestamptz; diff --git a/backend/oauth_connect.json b/backend/oauth_connect.json index 01becd80f3..9e74822d98 100644 --- a/backend/oauth_connect.json +++ b/backend/oauth_connect.json @@ -12,6 +12,7 @@ "bitbucket": { "auth_url": "https://bitbucket.org/site/oauth2/authorize", "token_url": "https://bitbucket.org/site/oauth2/access_token", + "grant_types": ["authorization_code", "client_credentials"], "scopes": ["repository"] }, "slack": { @@ -103,6 +104,7 @@ "linkedin": { "auth_url": "https://www.linkedin.com/oauth/v2/authorization", "token_url": "https://www.linkedin.com/oauth/v2/accessToken", + "grant_types": ["authorization_code", "client_credentials"], "scopes": ["w_member_social", "r_liteprofile", "r_emailaddress"], "req_body_auth": true }, @@ -114,14 +116,46 @@ "visma": { "auth_url": "https://connect.visma.com/connect/authorize", "token_url": "https://connect.visma.com/connect/token", + "grant_types": ["authorization_code", "client_credentials"], "scopes": [ "offline_access", "vismanet_erp_interactive_api:create", "vismanet_erp_interactive_api:delete", "vismanet_erp_interactive_api:read", "vismanet_erp_interactive_api:update" + ], + "cc_scopes": [ + "vismanet_erp_service_api:create", + "vismanet_erp_service_api:delete", + "vismanet_erp_service_api:read", + "vismanet_erp_service_api:update" ] }, + "coupa": { + "grant_types": ["client_credentials"], + "cc_scopes": [ + "core.supplier.read", + "core.supplier.write", + "core.purchase_order.read", + "core.purchase_order.write", + "core.requisition.read", + "core.requisition.write", + "core.invoice.read", + "core.invoice.write", + "core.contract.read", + "core.expense.read" + ], + "connect_config_template": { + "display_name": "Coupa", + "label": "Coupa instance", + "placeholder": "your-instance", + "token_url": "https://{instance}.coupahost.com/oauth2/token", + "strip_suffix": ".coupahost.com", + "resource_mapping": { + "instance_url": "https://{instance}.coupahost.com" + } + } + }, "sage_intacct": { "auth_url": "https://api.intacct.com/ia/api/v1/oauth2/authorize", "token_url": "https://api.intacct.com/ia/api/v1/oauth2/token", @@ -130,6 +164,7 @@ "spotify": { "auth_url": "https://accounts.spotify.com/authorize", "token_url": "https://accounts.spotify.com/api/token", + "grant_types": ["authorization_code", "client_credentials"], "scopes": [ "user-read-playback-state", "user-modify-playback-state", @@ -149,12 +184,16 @@ "xero": { "auth_url": "https://login.xero.com/identity/connect/authorize", "token_url": "https://identity.xero.com/connect/token", - "scopes": ["offline_access", "accounting.transactions"] + "grant_types": ["authorization_code", "client_credentials"], + "scopes": ["offline_access", "accounting.transactions"], + "cc_scopes": ["accounting.transactions"] }, "zoho": { "auth_url": "https://accounts.zoho.com/oauth/v2/auth", "token_url": "https://accounts.zoho.com/oauth/v2/token", + "grant_types": ["authorization_code", "client_credentials"], "scopes": ["ZohoAssist.sessionapi.ALL"], + "cc_scopes": ["ZohoAssist.sessionapi.ALL"], "extra_params": { "access_type": "offline" } @@ -197,6 +236,8 @@ } }, "servicenow": { + "grant_types": ["authorization_code", "client_credentials"], + "req_body_auth": true, "connect_config_template": { "display_name": "ServiceNow", "label": "ServiceNow Instance", @@ -210,5 +251,36 @@ "instance_url": "https://{instance}.service-now.com" } } + }, + "netsuite": { + "connect_config_template": { + "display_name": "NetSuite", + "label": "NetSuite Account ID", + "placeholder": " as in your NetSuite domain, e.g. 1234567 or 1234567-sb1", + "help_url": "https://docs.oracle.com/en/cloud/saas/netsuite/ns-online-help/chapter_157769826287.html", + "auth_url": "https://{instance}.app.netsuite.com/app/login/oauth2/authorize.nl", + "token_url": "https://{instance}.suitetalk.api.netsuite.com/services/rest/auth/oauth2/v1/token", + "req_body_auth": false, + "strip_suffix": ".app.netsuite.com", + "scopes": ["rest_webservices"], + "extra_params_key": "account_id", + "resource_mapping": { + "account_id": "{instance}" + } + } + }, + "outreach": { + "auth_url": "https://api.outreach.io/oauth/authorize", + "token_url": "https://api.outreach.io/oauth/token", + "scopes": [ + "accounts.all", + "prospects.all", + "sequences.all", + "sequenceStates.all", + "tasks.all", + "mailings.read", + "mailboxes.read" + ], + "req_body_auth": true } } diff --git a/backend/parsers/windmill-parser-py-asset/src/lib.rs b/backend/parsers/windmill-parser-py-asset/src/lib.rs index 10b93a998b..e8a0a1cc40 100644 --- a/backend/parsers/windmill-parser-py-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-py-asset/src/lib.rs @@ -2,8 +2,8 @@ use rustpython_ast::{Constant, Expr, ExprConstant, Visitor}; use rustpython_parser::{ast::Suite, Parse}; use std::collections::HashMap; use windmill_parser::asset_parser::{ - asset_was_used, merge_assets, parse_asset_syntax, AssetKind, AssetUsageAccessType, - ParseAssetsOutput, ParseAssetsResult, + asset_was_used, merge_assets, parse_asset_syntax, parse_pipeline_annotations, AssetKind, + AssetUsageAccessType, ParseAssetsOutput, ParseAssetsResult, }; use AssetUsageAccessType::*; @@ -28,7 +28,12 @@ pub fn parse_assets(input: &str) -> anyhow::Result { } } - Ok(ParseAssetsOutput { assets: merge_assets(assets_finder.assets), ..Default::default() }) + let pipeline = parse_pipeline_annotations(input); + Ok(ParseAssetsOutput::new( + merge_assets(assets_finder.assets), + Vec::new(), + pipeline, + )) } type VarAssetName = String; diff --git a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs index 002ed78fc0..f7c578a274 100644 --- a/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs +++ b/backend/parsers/windmill-parser-sql-asset/src/asset_parser.rs @@ -9,7 +9,8 @@ use sqlparser::{ parser::Parser, }; use windmill_parser::asset_parser::{ - asset_was_used, merge_assets, parse_asset_syntax, AssetKind, AssetUsageAccessType, + asset_was_used, merge_assets, merge_column_lineage, parse_asset_syntax, + parse_pipeline_annotations, AssetKind, AssetUsageAccessType, ColumnLineage, ColumnRef, ParseAssetsOutput, ParseAssetsResult, }; use AssetUsageAccessType::*; @@ -33,7 +34,60 @@ pub fn parse_assets(input: &str) -> anyhow::Result { } } - Ok(ParseAssetsOutput { assets: merge_assets(collector.assets), ..Default::default() }) + let mut pipeline = parse_pipeline_annotations(input); + // Scope inferred lineage to a single output asset so columns from an + // auxiliary CTAS aren't attributed to the materialized one (the flat list + // has no per-entry output on the wire). The `// materialize` target, when + // declared, IS the output: keep entries tagged with it plus untagged + // top-level-SELECT entries (which describe that target). Without a declared + // target, keep inference only when every tagged entry shares one output + // asset — otherwise it's ambiguous which asset the flat list describes, so + // drop it rather than show false dependencies. + let target = pipeline + .materialize + .as_ref() + .map(|m| (m.target_kind, m.target_path.clone())); + let inferred: Vec = match &target { + Some(t) => collector + .column_lineage + .into_iter() + .filter(|(out, _)| out.as_ref().map_or(true, |o| o == t)) + .map(|(_, cl)| cl) + .collect(), + None => { + let mut first: Option<&(AssetKind, String)> = None; + let mut ambiguous = false; + for (out, _) in &collector.column_lineage { + if let Some(o) = out { + match first { + None => first = Some(o), + Some(f) if f != o => { + ambiguous = true; + break; + } + _ => {} + } + } + } + if ambiguous { + Vec::new() + } else { + collector + .column_lineage + .into_iter() + .map(|(_, cl)| cl) + .collect() + } + } + }; + // Body-inferred column lineage, with `// column` annotations taking + // precedence per output column (explicit declaration overrides inference). + pipeline.column_lineage = merge_column_lineage(inferred, pipeline.column_lineage); + Ok(ParseAssetsOutput::new( + merge_assets(collector.assets), + Vec::new(), + pipeline, + )) } /// Visitor that collects S3 asset literals from SQL statements @@ -49,6 +103,16 @@ struct AssetCollector { cte_name_stack: Vec>, // Locally created tables (not attached to an asset) local_table_names: HashSet, + // Inferred column-level lineage: one entry per output column of an + // output-producing query, mapping it to the upstream source columns its + // expression reads. Each is tagged with the *output asset* it belongs to — + // `Some((kind, path))` for a CTAS / CREATE VIEW into a real asset, `None` + // for a top-level managed-materialize SELECT (its output is the `// + // materialize` target, known only in `parse_assets`). `parse_assets` uses + // the tag to scope the flat list to a single output asset so columns from an + // auxiliary output don't get attributed to the materialized one. Best-effort: + // dynamic/raw SQL, INSERT…SELECT, and wildcards are left to annotations. + column_lineage: Vec<(Option<(AssetKind, String)>, ColumnLineage)>, } impl AssetCollector { @@ -60,15 +124,24 @@ impl AssetCollector { currently_used_asset: None, cte_name_stack: Vec::new(), local_table_names: HashSet::new(), + column_lineage: Vec::new(), } } - /// If the name resolves to an attached asset, record it. Otherwise, register it as a local - /// table/view so that subsequent references are not mistakenly attributed to the active asset. - fn track_table_definition(&mut self, name: &ObjectName) { - if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { - self.assets.push(asset); - } else if let Some(simple_name) = get_trivial_obj_name(name) { + /// Record a `CREATE TABLE`/`VIEW` target. A *temporary* table/view is always + /// local — even a one-part name under an active `USE dl`, which would + /// otherwise resolve to an asset (`ducklake://…/tmp`) and then leak as a + /// column source for later references. A non-temp name that resolves to an + /// attached asset is recorded as that asset; anything else is registered + /// local so subsequent references aren't attributed to the active asset. + fn track_table_definition(&mut self, name: &ObjectName, is_temporary: bool) { + if !is_temporary { + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.assets.push(asset); + return; + } + } + if let Some(simple_name) = get_trivial_obj_name(name) { self.local_table_names.insert(simple_name.to_lowercase()); } } @@ -260,6 +333,37 @@ impl AssetCollector { } } + // Collect the table-level reads (and column assets) of a query's top + // SELECT. Table-level reads are only gathered here and in the statement + // arms — the generic table-factor visitor picks up read-functions and + // string literals, not plain `FROM ` references. Called for both + // standalone SELECTs and the `AS SELECT` of CTAS / CREATE VIEW. + fn handle_query_reads(&mut self, query: &sqlparser::ast::Query) { + self.cte_name_stack.push(collect_cte_names(query)); + if let Some(select) = query.body.as_select() { + for t in &select.from { + self.handle_table_with_joins(t, Some(R)); + } + self.extract_column_assets(&select.projection, &select.from); + } + } + + // Infer the output-column lineage of a query that produces an asset, tagging + // each entry with its `output` asset. Called only for an output-producing + // query — a top-level managed-materialize SELECT (`output: None`, resolved + // to the `// materialize` target later) or a CTAS / CREATE VIEW into a real + // asset (`output: Some`). A CTAS into a local/temp staging table is never + // an output, so it's simply not passed here. + fn infer_query_output( + &mut self, + query: &sqlparser::ast::Query, + output: Option<(AssetKind, String)>, + ) { + if let Some(select) = query.body.as_select() { + self.infer_column_lineage(&select.projection, &select.from, output); + } + } + fn handle_table_with_joins( &mut self, table_with_joins: &sqlparser::ast::TableWithJoins, @@ -282,17 +386,57 @@ impl AssetCollector { } } - // Extract columns from SELECT items and create individual asset results for each column - // Only processes columns that reference known assets to avoid false positives - fn extract_column_assets( - &mut self, - projection: &[SelectItem], + // The alias-map entry (key → asset) for one FROM/JOIN table factor, or + // `None` if it isn't an asset-backed table. The key is its alias, else the + // bare table name; S3 table-functions and string-literal tables are only + // keyed when aliased (an unaliased one is ambiguous). Returns the asset with + // a single matched relation so the caller can attribute qualified columns. + fn table_alias_entry(&self, relation: &TableFactor) -> Option<(String, ParseAssetsResult)> { + let TableFactor::Table { name, alias, args, .. } = relation else { + return None; + }; + let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); + if has_args { + let alias = alias.as_ref()?; + let asset = self.get_s3_asset_from_table_function(relation)?; + return Some((alias.name.value.clone(), asset)); + } + let asset = self + .get_associated_asset_from_obj_name(name, Some(R)) + .or_else(|| self.get_s3_asset_from_str_literal_table(relation))?; + if get_str_lit_from_obj_name(name).is_some() { + // String-literal S3 table: only unambiguous when aliased. + let alias = alias.as_ref()?; + return Some((alias.name.value.clone(), asset)); + } + let key = match alias { + Some(a) => a.name.value.clone(), + None => name + .0 + .last() + .and_then(|id| id.as_ident()) + .map(|id| id.value.clone()) + .unwrap_or_default(), + }; + Some((key, asset)) + } + + // Resolve a query's FROM clause into (single-table asset, alias→asset map). + // `single_table` is `Some` only for an unambiguous one-table FROM with no + // joins (so bare column refs can be attributed); `table_to_asset` keys by + // alias/table name for qualified refs and includes every JOINed table. + // Shared by `extract_column_assets` (read columns) and `infer_column_lineage` + // (output→input edges) so both resolve identically. + fn build_from_maps( + &self, from_tables: &[sqlparser::ast::TableWithJoins], + ) -> ( + Option, + BTreeMap, ) { - // Check if this is a single-table SELECT (to avoid ambiguity). - // For S3 table functions (read_parquet/read_csv/read_json), detect the asset even - // though args are present, since we know the file path from the string literal arg. - let single_table = if from_tables.len() == 1 { + // Single unambiguous table only when there's exactly one FROM entry AND + // it has no joins — otherwise a bare column could belong to any side. + let single_table = if from_tables.len() == 1 && from_tables[0].joins.is_empty() { let relation = &from_tables[0].relation; if let TableFactor::Table { name, args, .. } = relation { let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); @@ -309,51 +453,30 @@ impl AssetCollector { None }; - // Build a map of table aliases/names to assets for multi-table queries. - // For S3 table functions, only aliased references are unambiguous - // (e.g. SELECT t.col1 FROM read_parquet('s3://...') AS t). + // Alias → asset for qualified column refs, across every FROM entry AND + // its JOINed tables (so `c.col` in `FROM a JOIN c` resolves). let mut table_to_asset: BTreeMap = BTreeMap::new(); for table_with_joins in from_tables { - if let TableFactor::Table { name, alias, args, .. } = &table_with_joins.relation { - let has_args = args.as_ref().map_or(false, |a| !a.args.is_empty()); - if has_args { - // For table functions, only add to the alias map when an alias is present - if let Some(alias) = alias { - if let Some(asset) = - self.get_s3_asset_from_table_function(&table_with_joins.relation) - { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } - } else if let Some(asset) = self - .get_associated_asset_from_obj_name(name, Some(R)) - .or_else(|| { - self.get_s3_asset_from_str_literal_table(&table_with_joins.relation) - }) - { - // For string literal S3 tables (e.g. FROM 's3:///file.parquet'), only add to - // the alias map when an alias is present (to avoid false positives). - // For regular named tables, use alias or table name as key. - let is_str_literal = get_str_lit_from_obj_name(name).is_some(); - if is_str_literal { - if let Some(alias) = alias { - table_to_asset.insert(alias.name.value.clone(), asset); - } - } else { - let table_key = if let Some(alias) = alias { - alias.name.value.clone() - } else { - name.0 - .last() - .and_then(|id| id.as_ident()) - .map(|id| id.value.clone()) - .unwrap_or_default() - }; - table_to_asset.insert(table_key, asset); - } + if let Some((k, a)) = self.table_alias_entry(&table_with_joins.relation) { + table_to_asset.insert(k, a); + } + for join in &table_with_joins.joins { + if let Some((k, a)) = self.table_alias_entry(&join.relation) { + table_to_asset.insert(k, a); } } } + (single_table, table_to_asset) + } + + // Extract columns from SELECT items and create individual asset results for each column + // Only processes columns that reference known assets to avoid false positives + fn extract_column_assets( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); // Process each SELECT item for item in projection { @@ -426,6 +549,136 @@ impl AssetCollector { } } } + + // Infer column-level lineage for an output-producing query's projection: + // each *named* output column → the upstream source columns its expression + // reads. Covers passthroughs (`amount`) and computed columns + // (`amount + tax AS total`) alike. Skipped: wildcards and unaliased + // expressions (no stable output name), and inputs that don't resolve to a + // known asset. A column with no resolved inputs is dropped. + // + // Best-effort and intentionally flat: results from every output query in the + // script accumulate into one list (the graph hangs them off the materialize + // write-edge), with no per-output-table association. This is exact for the + // common single-output member (a managed-materialize SELECT, or one CTAS), + // but a multi-statement script that stages through a TEMP table reports the + // *intermediate* column names (the final SELECT reads the temp table, whose + // columns don't resolve to an asset, so they drop out). A `// column` + // annotation overrides any output column where inference is wrong or coarse. + fn infer_column_lineage( + &mut self, + projection: &[SelectItem], + from_tables: &[sqlparser::ast::TableWithJoins], + output: Option<(AssetKind, String)>, + ) { + let (single_table, table_to_asset) = self.build_from_maps(from_tables); + for item in projection { + let (out_col, expr) = match item { + SelectItem::ExprWithAlias { expr, alias } => (alias.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::Identifier(id)) => (id.value.clone(), expr), + SelectItem::UnnamedExpr(expr @ Expr::CompoundIdentifier(parts)) => { + match parts.last() { + Some(last) => (last.value.clone(), expr), + None => continue, + } + } + _ => continue, + }; + let mut collector = ColumnIdentCollector { refs: Vec::new(), query_depth: 0 }; + let _ = expr.visit(&mut collector); + let mut inputs: Vec = Vec::new(); + for parts in &collector.refs { + if let Some(cr) = self.resolve_column_ref(parts, &single_table, &table_to_asset) { + if !inputs.contains(&cr) { + inputs.push(cr); + } + } + } + if !inputs.is_empty() { + self.column_lineage + .push((output.clone(), ColumnLineage { column: out_col, inputs })); + } + } + } + + // Resolve identifier `parts` (e.g. `["t","amount"]` or `["amount"]`) to the + // source asset column it reads, mirroring `extract_column_assets`' + // resolution: a bare ident needs an unambiguous single-table FROM; a + // qualified ident resolves its prefix via the alias map, or (≥3 parts) as a + // db/schema-qualified object name. + fn resolve_column_ref( + &self, + parts: &[String], + single_table: &Option, + table_to_asset: &BTreeMap, + ) -> Option { + let asset_to_ref = |asset: &ParseAssetsResult, col: &str| ColumnRef { + from_kind: asset.kind, + from_path: asset.path.clone(), + from_column: col.to_string(), + }; + match parts { + [col] => single_table.as_ref().map(|a| asset_to_ref(a, col)), + [.., col] => { + let prefix = parts.first()?; + if let Some(asset) = table_to_asset.get(prefix) { + Some(asset_to_ref(asset, col)) + } else if parts.len() >= 3 { + let obj_parts: Vec = parts[..parts.len() - 1] + .iter() + .map(|p| ObjectNamePart::Identifier(sqlparser::ast::Ident::new(p.clone()))) + .collect(); + let asset = + self.get_associated_asset_from_obj_name(&ObjectName(obj_parts), Some(R))?; + Some(asset_to_ref(&asset, col)) + } else { + None + } + } + [] => None, + } + } +} + +// Collects the identifier paths an expression reads, for column-lineage +// inference: `Expr::Identifier(a)` → `["a"]`, `Expr::CompoundIdentifier(t.a)` → +// `["t","a"]`. The derived `Visit` walk recurses through operators, functions, +// casts and CASE, so every leaf identifier of the outer expression is captured. +struct ColumnIdentCollector { + refs: Vec>, + // Depth of nested (sub)queries inside the expression. Identifiers are only + // captured at depth 0: a scalar/correlated subquery's columns belong to ITS + // own FROM, not the outer projection's, so descending would misattribute + // (e.g. `(SELECT x FROM other) AS c FROM orders` must NOT bind `c` to + // `orders.x`). Subquery-derived columns are simply left to annotations. + query_depth: usize, +} + +impl Visitor for ColumnIdentCollector { + type Break = (); + + fn pre_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth += 1; + std::ops::ControlFlow::Continue(()) + } + + fn post_visit_query(&mut self, _query: &sqlparser::ast::Query) -> std::ops::ControlFlow<()> { + self.query_depth = self.query_depth.saturating_sub(1); + std::ops::ControlFlow::Continue(()) + } + + fn pre_visit_expr(&mut self, expr: &Expr) -> std::ops::ControlFlow { + if self.query_depth == 0 { + match expr { + Expr::Identifier(id) => self.refs.push(vec![id.value.clone()]), + Expr::CompoundIdentifier(parts) => self + .refs + .push(parts.iter().map(|id| id.value.clone()).collect()), + _ => {} + } + } + std::ops::ControlFlow::Continue(()) + } } impl Visitor for AssetCollector { @@ -485,15 +738,11 @@ impl Visitor for AssetCollector { ) -> std::ops::ControlFlow { match statement { sqlparser::ast::Statement::Query(q) => { - self.cte_name_stack.push(collect_cte_names(q)); - if let Some(select) = q.body.as_select() { - // First, handle table references (adds table-level assets) - for t in &select.from { - self.handle_table_with_joins(t, Some(R)); - } - // Then, extract column-level assets - self.extract_column_assets(&select.projection, &select.from); - } + // A top-level SELECT is the managed-materialize output, so its + // columns ARE the materialized asset's columns (output resolved + // to the `// materialize` target in `parse_assets`). + self.handle_query_reads(q); + self.infer_query_output(q, None); } sqlparser::ast::Statement::Insert(insert) => { @@ -646,11 +895,54 @@ impl Visitor for AssetCollector { } sqlparser::ast::Statement::CreateTable(create_table) => { - self.track_table_definition(&create_table.name); + self.track_table_definition(&create_table.name, create_table.temporary); + // `CREATE TABLE x AS SELECT … FROM y` reads y. The AS-query + // isn't a `Statement::Query`, so its FROM tables are only + // caught here. Only infer output lineage when `x` is a real + // asset — a CTAS into a local/temp staging table is not the + // materialized output (its columns aren't the asset's). + if let Some(query) = &create_table.query { + self.handle_query_reads(query); + // Infer only when `x` is a real asset (its output columns + // ARE that asset's), tagged with it so `parse_assets` can + // scope lineage per output. A local/temp staging table is + // not an asset → not inferred. + if let Some(asset) = + self.get_associated_asset_from_obj_name(&create_table.name, Some(W)) + { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } + } } - sqlparser::ast::Statement::CreateView { name, .. } => { - self.track_table_definition(name); + sqlparser::ast::Statement::CreateView { name, query, temporary, .. } => { + self.track_table_definition(name, *temporary); + self.handle_query_reads(query); + if let Some(asset) = self.get_associated_asset_from_obj_name(name, Some(W)) { + self.infer_query_output(query, Some((asset.kind, asset.path))); + } + } + + // DROP TABLE/VIEW is a write to the dropped object — the + // canonical idempotent-refresh pattern (`DROP TABLE IF EXISTS x; + // CREATE TABLE x AS …`) must resolve to a table-level write. + // Without this arm, a tagged-template snippet containing only the + // DROP yields no table-level asset and the TS/Python SDK parsers + // fall back to a db-level `datatable://` reference, putting a + // stray database node on the pipeline canvas. + sqlparser::ast::Statement::Drop { object_type, names, .. } => { + if matches!( + object_type, + sqlparser::ast::ObjectType::Table + | sqlparser::ast::ObjectType::View + | sqlparser::ast::ObjectType::MaterializedView + ) { + for name in names { + // DROP is a write to the named object; resolve it as an + // asset if it is one (not a temp-creation context). + self.track_table_definition(name, false); + } + } } sqlparser::ast::Statement::Copy { target: CopyTarget::File { filename }, .. } => { @@ -702,7 +994,16 @@ impl Visitor for AssetCollector { &mut self, statement: &sqlparser::ast::Statement, ) -> std::ops::ControlFlow { - if matches!(statement, sqlparser::ast::Statement::Query(_)) { + // Balance the push done by handle_query_reads (called from the Query, + // CreateView, and CTAS arms). + let pushed = match statement { + sqlparser::ast::Statement::Query(_) | sqlparser::ast::Statement::CreateView { .. } => { + true + } + sqlparser::ast::Statement::CreateTable(ct) => ct.query.is_some(), + _ => false, + }; + if pushed { self.cte_name_stack.pop(); } std::ops::ControlFlow::Continue(()) @@ -926,6 +1227,61 @@ mod tests { ); } + #[test] + fn test_sql_asset_parser_drop_table_is_write() { + // A lone DROP (e.g. one SDK tagged-template snippet of an + // idempotent-refresh script) must resolve to a table-level write, + // not fall through to nothing. + let input = r#" + ATTACH 'datatable://main' AS dt; USE dt; + DROP TABLE IF EXISTS orders_raw; + "#; + let s = parse_assets(input).map(|s| s.assets); + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ParseAssetsResult { + kind: AssetKind::DataTable, + path: "main/orders_raw".to_string(), + access_type: Some(W), + columns: None + },]) + ); + } + + #[test] + fn test_sql_asset_parser_drop_then_create_qualified() { + // Canonical refresh pattern over an attached catalog: DROP + CREATE + // AS on the output, SELECT on the input. + let input = r#" + ATTACH 'datatable://main' AS pg; + DROP TABLE IF EXISTS pg.daily_revenue; + CREATE TABLE pg.daily_revenue AS SELECT * FROM pg.orders_clean; + "#; + let s = parse_assets(input).map(|mut s| { + s.assets.sort_by(|a, b| a.path.cmp(&b.path)); + s.assets + }); + // The DROP+CTAS combo yields a clean write of the output plus the + // read of the CTAS source (`SELECT * FROM pg.orders_clean`). + assert_eq!( + s.map_err(|e| e.to_string()), + Ok(vec![ + ParseAssetsResult { + kind: AssetKind::DataTable, + path: "main/daily_revenue".to_string(), + access_type: Some(W), + columns: None + }, + ParseAssetsResult { + kind: AssetKind::DataTable, + path: "main/orders_clean".to_string(), + access_type: Some(R), + columns: None + }, + ]) + ); + } + // Make sure a_function is not detected as main/a_function #[test] fn test_sql_asset_parser_function_table() { @@ -1796,3 +2152,329 @@ mod tests { assert_eq!(columns.get("age"), Some(&R)); } } + +#[cfg(test)] +mod ctas_read_tests { + use super::*; + + #[test] + fn test_ctas_collects_upstream_read() { + let input = r#" + ATTACH 'datatable://main' AS pg; + CREATE TABLE IF NOT EXISTS pg.exciting_809 AS + SELECT * FROM pg.fx_rates; + "#; + let assets = parse_assets(input).unwrap().assets; + assert!( + assets.iter().any(|a| a.path == "main/fx_rates" + && a.kind == AssetKind::DataTable + && a.access_type == Some(R)), + "expected read of main/fx_rates, got {:?}", + assets + ); + assert!( + assets + .iter() + .any(|a| a.path == "main/exciting_809" && a.access_type == Some(W)), + "expected write of main/exciting_809, got {:?}", + assets + ); + } + + #[test] + fn test_create_view_collects_upstream_read() { + let input = r#" + ATTACH 'datatable://main' AS pg; + CREATE VIEW pg.v AS SELECT * FROM pg.fx_rates; + "#; + let assets = parse_assets(input).unwrap().assets; + assert!( + assets + .iter() + .any(|a| a.path == "main/fx_rates" && a.access_type == Some(R)), + "expected read of main/fx_rates, got {:?}", + assets + ); + } + + fn lineage(input: &str) -> Vec { + parse_assets(input).unwrap().column_lineage + } + + #[test] + fn test_infer_lineage_computed_and_passthrough() { + // CTAS with a computed column (amount + tax) and a passthrough (id). + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_bare_column_single_table() { + // Managed-materialize form: a plain top-level SELECT, bare columns + // attributed to the single FROM table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + SELECT amount AS revenue FROM orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "revenue".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_annotation_overrides() { + // The `// column` annotation for `order_total` wins; `id` stays inferred. + let input = r#" + -- column order_total <- datatable://prod/manual.grand_total + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, dl.orders.amount + dl.orders.tax AS order_total + FROM dl.orders; + "#; + let got = lineage(input); + // Annotation entry is authoritative and listed first. + assert_eq!(got[0].column, "order_total"); + assert_eq!(got[0].inputs[0].from_path, "prod/manual"); + assert_eq!(got[0].inputs[0].from_column, "grand_total"); + // Inferred `id` survives; inferred `order_total` dropped (no dup). + assert!(got.iter().any(|c| c.column == "id")); + assert_eq!(got.iter().filter(|c| c.column == "order_total").count(), 1); + } + + #[test] + fn test_infer_lineage_skips_local_staging_ctas() { + // A CTAS into a TEMP/local table is NOT the materialized output, so its + // columns must not be reported (they'd be anchored to the script's + // `// materialize` target as if they were the final asset's columns). + // The final SELECT reads the local staging table → unresolved → empty. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TEMP TABLE tmp AS SELECT dl.orders.amount AS amt FROM dl.orders; + SELECT amt AS total FROM tmp; + "#; + assert!( + lineage(input).is_empty(), + "staging columns must not be reported as final output; got {:?}", + lineage(input) + ); + } + + #[test] + fn test_infer_lineage_ctas_into_asset_still_inferred() { + // A CTAS whose target IS an asset is the output, so it's still inferred. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT dl.orders.amount AS amt FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "amt".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn test_infer_lineage_temp_table_under_use_is_local() { + // A one-part TEMP table name under an active `USE dl` must NOT resolve to + // an asset (`warehouse/tmp`); it's local, so the final SELECT reading it + // can't invent `warehouse/tmp.amt` as a column source for the output. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + USE dl; + CREATE TEMP TABLE tmp AS SELECT amount AS amt FROM orders; + SELECT amt AS total FROM tmp; + "#; + let got = lineage(input); + assert!( + got.is_empty(), + "temp staging under USE must not leak warehouse/tmp as a source; got {:?}", + got + ); + // And no phantom `warehouse/tmp` asset is recorded. + let assets = parse_assets(input).unwrap().assets; + assert!( + !assets.iter().any(|a| a.path == "warehouse/tmp"), + "temp table must not be recorded as an asset; got {:?}", + assets + ); + } + + #[test] + fn test_infer_lineage_scopes_to_materialize_target() { + // A script with a `// materialize` target plus an AUXILIARY CTAS into a + // different asset: only the materialized target's columns are reported; + // the auxiliary output's columns must not be attributed to it. + let input = r#" + -- materialize ducklake://warehouse/final + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.audit AS SELECT dl.orders.id AS aid FROM dl.orders; + SELECT dl.orders.amount AS total FROM dl.orders; + "#; + assert_eq!( + lineage(input), + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }], + }], + "auxiliary `audit` columns must not appear on the materialized target" + ); + } + + #[test] + fn test_infer_lineage_drops_ambiguous_multi_output() { + // No `// materialize` target and two real CTAS outputs: which asset the + // flat lineage describes is ambiguous, so inference is dropped rather + // than attributed to an arbitrary one. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.a AS SELECT dl.orders.id AS x FROM dl.orders; + CREATE TABLE dl.b AS SELECT dl.orders.amount AS y FROM dl.orders; + "#; + assert!( + lineage(input).is_empty(), + "ambiguous multi-output must drop inference" + ); + } + + #[test] + fn test_infer_lineage_wildcard_yields_nothing() { + // `SELECT *` has no enumerable output columns → no inferred lineage. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS SELECT * FROM dl.orders; + "#; + assert!(lineage(input).is_empty()); + } + + #[test] + fn test_infer_lineage_resolves_joined_inputs() { + // Columns from BOTH sides of an explicit JOIN must resolve, incl. a + // computed column mixing the two. A bare column is dropped (ambiguous + // across the join) rather than misattributed to the first table. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT o.id, c.region AS cust_region, o.amount + c.discount AS net + FROM dl.orders o + JOIN dl.customers c ON c.id = o.customer_id; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ + ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }, + ColumnLineage { + column: "cust_region".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "region".to_string(), + }], + }, + ColumnLineage { + column: "net".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/customers".to_string(), + from_column: "discount".to_string(), + }, + ], + }, + ] + ); + } + + #[test] + fn test_infer_lineage_does_not_descend_into_subqueries() { + // A scalar subquery's bare column (`amount`) belongs to the subquery's + // own FROM, NOT the outer `dl.orders` — it must not be attributed to the + // outer table. The subquery column is left to annotations; the + // passthrough `id` still resolves. + let input = r#" + ATTACH 'ducklake://warehouse' AS dl; + CREATE TABLE dl.orders_daily AS + SELECT dl.orders.id, (SELECT amount FROM dl.other LIMIT 1) AS c + FROM dl.orders; + "#; + let got = lineage(input); + assert_eq!( + got, + vec![ColumnLineage { + column: "id".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "id".to_string(), + }], + }], + "subquery column `c` must be dropped, not misattributed to orders" + ); + } +} diff --git a/backend/parsers/windmill-parser-ts-asset/src/lib.rs b/backend/parsers/windmill-parser-ts-asset/src/lib.rs index 39302fb6fa..b7fe0fea5c 100644 --- a/backend/parsers/windmill-parser-ts-asset/src/lib.rs +++ b/backend/parsers/windmill-parser-ts-asset/src/lib.rs @@ -5,8 +5,8 @@ use swc_ecma_ast::{CallExpr, Expr, Lit, MemberExpr, MemberProp, ObjectLit, Prop, use swc_ecma_parser::{lexer::Lexer, Parser, StringInput, Syntax, TsSyntax}; use swc_ecma_visit::{Visit, VisitWith}; use windmill_parser::asset_parser::{ - asset_was_used, merge_assets, parse_asset_syntax, AssetKind, AssetUsageAccessType, - ParseAssetsOutput, ParseAssetsResult, SqlQueryDetails, + asset_was_used, merge_assets, parse_asset_syntax, parse_pipeline_annotations, AssetKind, + AssetUsageAccessType, ParseAssetsOutput, ParseAssetsResult, SqlQueryDetails, }; use AssetUsageAccessType::*; @@ -38,10 +38,12 @@ pub fn parse_assets(code: &str) -> anyhow::Result { let mut assets_finder = AssetsFinder { assets: vec![], sql_queries: vec![], var_identifiers: HashMap::new() }; assets_finder.visit_module_items(&ast); - Ok(ParseAssetsOutput { - assets: merge_assets(assets_finder.assets), - sql_queries: assets_finder.sql_queries, - }) + let pipeline = parse_pipeline_annotations(code); + Ok(ParseAssetsOutput::new( + merge_assets(assets_finder.assets), + assets_finder.sql_queries, + pipeline, + )) } type VarAssetName = String; diff --git a/backend/parsers/windmill-parser-wasm/Cargo.lock b/backend/parsers/windmill-parser-wasm/Cargo.lock index 5717e59c3d..096d93163b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.lock +++ b/backend/parsers/windmill-parser-wasm/Cargo.lock @@ -174,11 +174,10 @@ checksum = "7c02d123df017efcdfbd739ef81735b36c5ba83ec3c59c80a9d7ecc718f92e50" [[package]] name = "ast_node" -version = "0.9.9" +version = "3.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9184f2b369b3e8625712493c89b785881f27eedc6cde480a81883cef78868b2" +checksum = "0a184645bcc6f52d69d8e7639720699c6a99efb711f886e251ed1d16db8dd90e" dependencies = [ - "proc-macro2", "quote", "swc_macros_common", "syn 2.0.117", @@ -401,9 +400,9 @@ checksum = "2af50177e190e07a26ab74f8b1efbfe2ef87da2116221318cb1c2e82baf7de06" [[package]] name = "better_scoped_tls" -version = "0.1.2" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "297b153aa5e573b5863108a6ddc9d5c968bd0b20e75cc614ee9821d2f45679c7" +checksum = "7cd228125315b132eed175bf47619ac79b945b26e56b848ba203ae4ea8603609" dependencies = [ "scoped-tls", ] @@ -563,9 +562,6 @@ name = "bumpalo" version = "3.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "5d20789868f4b01b2f2caec9f5c4e0213b41e3e5702a50157d699ae31ced2fcb" -dependencies = [ - "allocator-api2", -] [[package]] name = "byte-unit" @@ -613,6 +609,16 @@ version = "1.11.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1e748733b7cbc798e1434b6ac524f0c1ff2ab456fe201501e6497c8417a4fc33" +[[package]] +name = "bytes-str" +version = "0.2.8" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "577d2bf5650f8554d5a372af5ac93535110a0fc75b3e702bb853369febf227c2" +dependencies = [ + "bytes", + "serde", +] + [[package]] name = "bytesize" version = "1.3.3" @@ -1372,11 +1378,10 @@ dependencies = [ [[package]] name = "from_variant" -version = "0.1.9" +version = "2.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32016f1242eb82af5474752d00fd8ebcd9004bd69b462b1c91de833972d08ed4" +checksum = "308530a56b099da144ebc5d8e179f343ad928fa2b3558d1eb3db9af18d6eff43" dependencies = [ - "proc-macro2", "swc_macros_common", "syn 2.0.117", ] @@ -1751,15 +1756,14 @@ dependencies = [ [[package]] name = "hstr" -version = "0.2.17" +version = "2.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a1a26def229ea95a8709dad32868d975d0dd40235bd2ce82920e4a8fe692b5e0" +checksum = "31f11d91d7befd2ffd9d216e9e5ea1fae6174b20a2a1b67a688138003d2f4122" dependencies = [ "hashbrown 0.14.5", "new_debug_unreachable", "once_cell", - "phf 0.11.3", - "rustc-hash 1.1.0", + "rustc-hash 2.1.1", "triomphe", ] @@ -4136,10 +4140,16 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8a7852d02fc848982e0c167ef163aaff9cd91dc640ba85e263cb1ce46fae51cd" [[package]] -name = "serde" -version = "1.0.220" +name = "seq-macro" +version = "0.3.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ceecad4c782e936ac90ecfd6b56532322e3262b14320abf30ce89a92ffdbfe22" +checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" + +[[package]] +name = "serde" +version = "1.0.228" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "9a8e94ea7f378bd32cbbd37198a4a91436180c5bb472411e48b5ec2e2124ae9e" dependencies = [ "serde_core", "serde_derive", @@ -4158,18 +4168,18 @@ dependencies = [ [[package]] name = "serde_core" -version = "1.0.220" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ddba47394f3b862d6ff6efdbd26ca4673e3566a307880a0ffb98f274bbe0ec32" +checksum = "41d385c7d4ca58e59fc732af25c3983b67ac852c1a25000afe1175de458b67ad" dependencies = [ "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.220" +version = "1.0.228" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60e1f3b1761e96def5ec6d04a6e7421c0404fa3cf5c0155f1e2848fae3d8cc08" +checksum = "d540f220d3187173da220f885ab66608367b6574e925011a9353e4badda91d79" dependencies = [ "proc-macro2", "quote", @@ -4692,11 +4702,10 @@ checksum = "a2eb9349b6444b326872e140eb1cf5e7c522154d69e7a0ffb0fb81c06b37543f" [[package]] name = "string_enum" -version = "0.4.4" +version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "05e383308aebc257e7d7920224fa055c632478d92744eca77f99be8fa1545b90" +checksum = "ae36a4951ca7bd1cfd991c241584a9824a70f6aff1e7d4f693fb3f2465e4030e" dependencies = [ - "proc-macro2", "quote", "swc_macros_common", "syn 2.0.117", @@ -4798,49 +4807,35 @@ version = "3.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b7401a30af6cb5818bb64852270bb722533397edcfc7344954a38f420819ece2" -[[package]] -name = "swc_allocator" -version = "0.1.10" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "76aa0eb65c0f39f9b6d82a7e5192c30f7ac9a78f084a21f270de1d8c600ca388" -dependencies = [ - "bumpalo", - "hashbrown 0.14.5", - "ptr_meta", - "rustc-hash 1.1.0", - "triomphe", -] - [[package]] name = "swc_atoms" -version = "0.6.7" +version = "7.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "bb6567e4e67485b3e7662b486f1565bdae54bd5b9d6b16b2ba1a9babb1e42125" +checksum = "3500dcf04c84606b38464561edc5e46f5132201cb3e23cf9613ed4033d6b1bb2" dependencies = [ "hstr", "once_cell", - "rustc-hash 1.1.0", "serde", ] [[package]] name = "swc_common" -version = "0.37.5" +version = "14.0.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "12d0a8eaaf1606c9207077d75828008cb2dfb51b095a766bd2b72ef893576e31" +checksum = "c2bb772b3a26b8b71d4e8c112ced5b5867be2266364b58517407a270328a2696" dependencies = [ + "anyhow", "ast_node", "better_scoped_tls", - "cfg-if", + "bytes-str", "either", "from_variant", "new_debug_unreachable", "num-bigint", "once_cell", - "rustc-hash 1.1.0", + "rustc-hash 2.1.1", "serde", "siphasher 0.3.11", - "swc_allocator", "swc_atoms", "swc_eq_ignore_macros", "swc_visit", @@ -4851,32 +4846,36 @@ dependencies = [ [[package]] name = "swc_ecma_ast" -version = "0.118.2" +version = "15.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a6f866d12e4d519052b92a0a86d1ac7ff17570da1272ca0c89b3d6f802cd79df" +checksum = "65c25af97d53cf8aab66a6c68f3418663313fc969ad267fc2a4d19402c329be1" dependencies = [ "bitflags", "is-macro", "num-bigint", + "once_cell", "phf 0.11.3", - "scoped-tls", + "rustc-hash 2.1.1", "string_enum", "swc_atoms", "swc_common", + "swc_visit", "unicode-id-start", ] [[package]] -name = "swc_ecma_parser" -version = "0.149.1" +name = "swc_ecma_lexer" +version = "23.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "683dada14722714588b56481399c699378b35b2ba4deb5c4db2fb627a97fb54b" +checksum = "017d06ea85008234aa9fb34d805c7dc563f2ea6e03869ed5ac5a2dc27d561e4d" dependencies = [ + "arrayvec", + "bitflags", "either", - "new_debug_unreachable", "num-bigint", - "num-traits", "phf 0.11.3", + "rustc-hash 2.1.1", + "seq-macro", "serde", "smallvec", "smartstring", @@ -4885,14 +4884,29 @@ dependencies = [ "swc_common", "swc_ecma_ast", "tracing", - "typed-arena", +] + +[[package]] +name = "swc_ecma_parser" +version = "24.0.3" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2e9011783c975ba592ffc09cd208ced92b1dfabb2e5e0ef453559e2e25286127" +dependencies = [ + "either", + "num-bigint", + "serde", + "swc_atoms", + "swc_common", + "swc_ecma_ast", + "swc_ecma_lexer", + "tracing", ] [[package]] name = "swc_ecma_visit" -version = "0.104.8" +version = "15.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b1c6802e68e51f336e8bc9644e9ff9da75d7da9c1a6247d532f2e908aa33e81" +checksum = "75a579aa8f9e212af521588df720ccead079c09fe5c8f61007cf724324aed3a0" dependencies = [ "new_debug_unreachable", "num-bigint", @@ -4905,9 +4919,9 @@ dependencies = [ [[package]] name = "swc_eq_ignore_macros" -version = "0.1.4" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "63db0adcff29d220c3d151c5b25c0eabe7e32dd936212b84cdaa1392e3130497" +checksum = "c16ce73424a6316e95e09065ba6a207eba7765496fed113702278b7711d4b632" dependencies = [ "proc-macro2", "quote", @@ -4916,9 +4930,9 @@ dependencies = [ [[package]] name = "swc_macros_common" -version = "0.3.14" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "27e18fbfe83811ffae2bb23727e45829a0d19c6870bced7c0f545cc99ad248dd" +checksum = "aae1efbaa74943dc5ad2a2fb16cbd78b77d7e4d63188f3c5b4df2b4dcd2faaae" dependencies = [ "proc-macro2", "quote", @@ -4927,9 +4941,9 @@ dependencies = [ [[package]] name = "swc_visit" -version = "0.6.2" +version = "2.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1ceb044142ba2719ef9eb3b6b454fce61ab849eb696c34d190f04651955c613d" +checksum = "62fb71484b486c185e34d2172f0eabe7f4722742aad700f426a494bb2de232a2" dependencies = [ "either", "new_debug_unreachable", @@ -5065,7 +5079,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd" dependencies = [ "fastrand", - "getrandom 0.3.4", + "getrandom 0.4.2", "once_cell", "rustix 1.1.4", "windows-sys 0.61.2", @@ -5595,12 +5609,6 @@ version = "0.2.5" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e421abadd41a4225275504ea4d6566923418b7f05506fbc9c0fe86ba7396114b" -[[package]] -name = "typed-arena" -version = "2.0.2" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6af6ae20167a9ece4bcb41af5b80f8a1f1df981f6391189ce00fd257af04126a" - [[package]] name = "typeid" version = "1.0.3" @@ -6183,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f" [[package]] name = "windmill-common" -version = "1.723.0" +version = "1.742.0" dependencies = [ "aho-corasick", "anyhow", @@ -6205,6 +6213,7 @@ dependencies = [ "croner", "dashmap", "equivalent", + "erased-serde", "futures", "futures-core", "gethostname", @@ -6263,7 +6272,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.723.0" +version = "1.742.0" dependencies = [ "proc-macro2", "quote", @@ -6275,7 +6284,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.723.0" +version = "1.742.0" dependencies = [ "convert_case", "serde", @@ -6284,7 +6293,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6296,7 +6305,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6308,7 +6317,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "gosyn", @@ -6320,7 +6329,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6332,7 +6341,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6344,7 +6353,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "nu-parser", @@ -6355,7 +6364,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6366,7 +6375,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "itertools 0.14.0", @@ -6378,7 +6387,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6389,7 +6398,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "async-recursion", @@ -6411,7 +6420,7 @@ dependencies = [ [[package]] name = "windmill-parser-r" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde_json", @@ -6423,7 +6432,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6437,7 +6446,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "convert_case", @@ -6454,7 +6463,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6467,7 +6476,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -6479,7 +6488,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "lazy_static", @@ -6497,7 +6506,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -6513,7 +6522,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "rustpython-ast", @@ -6529,7 +6538,7 @@ dependencies = [ [[package]] name = "windmill-parser-wasm" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "getrandom 0.2.17", @@ -6561,7 +6570,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "serde", @@ -6572,7 +6581,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.723.0" +version = "1.742.0" dependencies = [ "anyhow", "bitflags", diff --git a/backend/parsers/windmill-parser-wasm/Cargo.toml b/backend/parsers/windmill-parser-wasm/Cargo.toml index 3c9bf1ed61..e2afd2483b 100644 --- a/backend/parsers/windmill-parser-wasm/Cargo.toml +++ b/backend/parsers/windmill-parser-wasm/Cargo.toml @@ -12,7 +12,7 @@ resolver = "2" members = ["."] [workspace.package] -version = "1.723.0" +version = "1.742.0" edition = "2021" authors = ["Ruben Fiszel "] diff --git a/backend/parsers/windmill-parser-yaml/src/asset_parser.rs b/backend/parsers/windmill-parser-yaml/src/asset_parser.rs index 7e67d563ff..4dde12ad8e 100644 --- a/backend/parsers/windmill-parser-yaml/src/asset_parser.rs +++ b/backend/parsers/windmill-parser-yaml/src/asset_parser.rs @@ -1,5 +1,6 @@ use windmill_parser::asset_parser::{ - merge_assets, AssetKind, AssetUsageAccessType, ParseAssetsOutput, ParseAssetsResult, + merge_assets, parse_pipeline_annotations, AssetKind, AssetUsageAccessType, ParseAssetsOutput, + ParseAssetsResult, }; use crate::{parse_ansible_reqs, ResourceOrVariablePath}; @@ -39,5 +40,10 @@ pub fn parse_assets(input: &str) -> anyhow::Result { } } - Ok(ParseAssetsOutput { assets: merge_assets(assets), ..Default::default() }) + let pipeline = parse_pipeline_annotations(input); + Ok(ParseAssetsOutput::new( + merge_assets(assets), + Vec::new(), + pipeline, + )) } diff --git a/backend/parsers/windmill-parser-yaml/src/lib.rs b/backend/parsers/windmill-parser-yaml/src/lib.rs index 8a1f1bc097..5d9945cadd 100644 --- a/backend/parsers/windmill-parser-yaml/src/lib.rs +++ b/backend/parsers/windmill-parser-yaml/src/lib.rs @@ -50,7 +50,7 @@ pub fn parse_ansible_sig(inner_content: &str) -> anyhow::Result anyhow::Result anyhow::Result anyhow::Result,,...`). A newline or other config-meaningful +/// character would let a script inject arbitrary `[defaults]` directives (e.g. +/// `library`, `action_plugins`) and execute attacker-controlled code on the worker, +/// and a `,` would smuggle in an extra entry. Restrict entries to the `label@source` +/// charset so neither is possible. +pub fn validate_vault_id(value: &str) -> anyhow::Result<()> { + let is_valid = !value.is_empty() + && value + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '_' | '-' | '/' | '@')); + if !is_valid { + return Err(anyhow!( + "Invalid vault_id `{value}`: expected `label@filename` using only letters, digits and the characters `.`, `_`, `-`, `/`, `@`" + )); + } + Ok(()) +} + pub fn parse_ansible_reqs( inner_content: &str, ) -> anyhow::Result<(String, Option, String)> { @@ -528,6 +547,7 @@ pub fn parse_ansible_reqs( let Yaml::String(filename) = f else { return Err(anyhow!("The elements of the vault_id field should be strings in the format: `label@filename`")); }; + validate_vault_id(filename)?; ret.vault_id.push(filename.to_string()); } } @@ -1051,4 +1071,55 @@ delegate_to_git_repo: Some("inventories/{{ env }}") ); } + + #[test] + fn test_parse_vault_id_valid() { + let p = r#" +--- +vault_id: + - dev@vault_pass_dev.txt + - prod@./secrets/prod-pass +--- +- name: Test + hosts: all +"#; + let (_, reqs, _) = parse_ansible_reqs(p).unwrap(); + assert_eq!( + reqs.unwrap().vault_id, + vec![ + "dev@vault_pass_dev.txt".to_string(), + "prod@./secrets/prod-pass".to_string() + ] + ); + } + + #[test] + fn test_parse_vault_id_rejects_newline_injection() { + let p = "---\nvault_id:\n - \"default@/tmp/wm/x\\nlibrary = /tmp/wm/evil_modules\"\n---\n- name: Test\n hosts: all\n"; + assert!(parse_ansible_reqs(p).is_err()); + } + + #[test] + fn test_parse_vault_id_rejects_comma() { + let p = r#" +--- +vault_id: + - "a@b,c@d" +--- +- name: Test + hosts: all +"#; + assert!(parse_ansible_reqs(p).is_err()); + } + + #[test] + fn test_validate_vault_id() { + assert!(validate_vault_id("default@/tmp/wm/pass").is_ok()); + assert!(validate_vault_id("dev@pass.txt").is_ok()); + assert!(validate_vault_id("").is_err()); + assert!(validate_vault_id("a@b\nlibrary = /evil").is_err()); + assert!(validate_vault_id("a@b,c@d").is_err()); + assert!(validate_vault_id("a@b c").is_err()); + assert!(validate_vault_id("a@b=c").is_err()); + } } diff --git a/backend/parsers/windmill-parser/src/asset_parser.rs b/backend/parsers/windmill-parser/src/asset_parser.rs index 4239c1854c..2b7f6ecdbf 100644 --- a/backend/parsers/windmill-parser/src/asset_parser.rs +++ b/backend/parsers/windmill-parser/src/asset_parser.rs @@ -1,6 +1,15 @@ use serde::Serialize; use std::collections::BTreeMap; +// Token recognized inside declared asset URIs that the runtime substitutes +// with the current partition value (e.g. `s3://lake/{partition}.parquet` +// becomes `s3://lake/2024-04-29.parquet` when materialized for that day). +// Substitution is the parser's job only at lineage/graph time — at runtime +// the pipeline worker performs the actual replacement before emitting +// asset signals. Kept as a single well-known token so the parser never +// has to guess which `{...}` placeholders are partition variables. +pub const PARTITION_TOKEN: &str = "{partition}"; + #[derive(Serialize, PartialEq, Clone, Copy, Debug)] #[serde(rename_all(serialize = "lowercase"))] pub enum AssetUsageAccessType { @@ -47,6 +56,346 @@ pub struct SqlQueryDetails { pub struct ParseAssetsOutput { pub assets: Vec, pub sql_queries: Vec, + // Bare `// pipeline` (or `#` / `--`) on its own line — opt-in marker + // that sets auto_kind='pipeline' and includes the script in its + // folder's pipeline. Pipeline membership is broader than + // materialization: scripts that only assert/notify/clean up are + // members too. Outputs (when present) come from parser-detected + // `w`/`rw` usages in `assets`, not from this marker. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub in_pipeline: bool, + // Trigger annotations — execution DAG edges. Each is an independent OR + // (any fires the script). Empty = script has no automatic triggers + // (still runnable manually / via existing cron triggers). Includes both + // top-level `// schedule "..."` and `// on ...` forms. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub triggers: Vec, + // `// partitioned [opts]` — declares that this pipeline script + // produces partitioned output. The runtime resolves `{partition}` in + // declared output URIs to the current partition value before signaling + // downstream consumers. At most one per script. + #[serde(skip_serializing_if = "Option::is_none", default)] + pub partition: Option, + // `// freshness ` — SLA stating outputs must be at most + // `duration` old. Active backstop: when no other trigger has fired the + // script within the window, a watchdog re-runs it. Distinct from + // schedule (which is producer cadence); freshness is consumer SLA and + // applies regardless of which trigger last fired. + #[serde(skip_serializing_if = "Option::is_none", default)] + pub freshness: Option, + // `// trigger all` → AND join barrier; default (`any`) = OR (current + // behaviour). Threaded to the deploy path which persists it on the + // subscriber's trigger rows. + #[serde(skip_serializing_if = "JoinMode::is_any", default)] + pub join_mode: JoinMode, + // `// debounce ` — script-level default debounce window for this + // script's asset inputs. A per-`// on … debounce=` overrides it. Raw + // duration string, parsed to seconds at deploy (parser-light, like + // freshness). Absent = no debounce (fan-out, current behaviour). + #[serde(skip_serializing_if = "Option::is_none", default)] + pub debounce_default: Option, + // `// tag ` — overrides the script's worker tag at deploy. Source + // wins over any UI-set value, matching the wipe-and-reinsert convention + // of other pipeline annotations (`// schedule`, `// on`). Absent = keep + // whatever the caller (UI / CLI) supplied. + #[serde(skip_serializing_if = "Option::is_none", default)] + pub tag: Option, + // `// retry []` — re-run a pipeline-cascade triggered + // script up to `count` times on failure, waiting `delay` between + // attempts. Applies only to runs launched via the asset/schedule + // cascade (the rows in `script_trigger`); manual UI runs are unaffected. + // The delay is a raw duration string parsed at deploy (parser-light). + #[serde(skip_serializing_if = "Option::is_none", default)] + pub retry: Option, + // `// materialize [manual] [append] [key=]` — + // managed-materialization target + its strategy. At most one per script. + // Drives the worker's write-strategy + snapshot capture. + #[serde(skip_serializing_if = "Option::is_none", default)] + pub materialize: Option, + // `// data_test …` — data-quality assertions run against the + // materialized asset after the write commits. Accumulating (multiple + // lines allowed). Drives the worker's post-materialize verifier probes. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub data_tests: Vec, + // `// column <- .[, …]` — declared column-level lineage, + // one entry per output column. Accumulating. Pure metadata: drives the + // column-lineage graph view, executes nothing. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + pub column_lineage: Vec, +} + +#[derive(Serialize, Debug, PartialEq, Clone)] +#[serde(tag = "kind", rename_all = "lowercase")] +pub enum TriggerSpec { + // Refresh when `` changes. Kind comes from parse_asset_syntax so + // it matches the `asset` table. `debounce` is the optional per-input + // `// on … debounce=` override (raw duration string); it takes + // precedence over the script-level `// debounce` default, resolved at + // deploy. + Asset { + asset_kind: AssetKind, + path: String, + #[serde(skip_serializing_if = "Option::is_none", default)] + debounce: Option, + }, + // `// on ` — marker-only declaration that this script wants to be + // triggered by a native trigger of the given kind. No path: the binding + // is the trigger row's own `script_path` field (set when the user creates + // the kafka/mqtt/schedule/… trigger in its dedicated UI). The graph + // endpoint discovers attached triggers by `WHERE script_path = ` and surfaces a "missing" placeholder when an annotation has no + // matching row. + Schedule, + Webhook, + Email, + Kafka, + Mqtt, + Nats, + Postgres, + Sqs, + Gcp, + // `// on data_upload` — UI-first entry point. Unlike the other native + // kinds there is no external event source and no trigger row anywhere: + // the script declares an `S3Object` input parameter and the user uploads + // a file via the auto-generated S3 picker, which runs the pipeline. The + // graph renders it as a clickable upload source (never a "missing" + // placeholder, mirroring webhook). + #[serde(rename = "data_upload")] + DataUpload, +} + +impl TriggerSpec { + // A `// on ` whose declared path contains the `{partition}` + // token is *partition-bearing*: in an AND join its concrete partition + // value is the join key. Non-asset triggers and assets without the + // token are reference/presence-only inputs that never define the + // partition (the case-3 guard). + pub fn is_partition_bearing(&self) -> bool { + matches!(self, TriggerSpec::Asset { path, .. } if path.contains(PARTITION_TOKEN)) + } +} + +// Partitioning declaration for a pipeline script. `daily`/`hourly`/`weekly`/ +// `monthly` are time-based with the runtime supplying the current +// partition value derived from the trigger context (schedule fire time, +// freshness window, manual run arg). `dynamic` extracts the value from the +// triggering payload via JSONPath — used for per-tenant / per-shard / +// per-event-id pipelines where the partition key isn't a wall-clock value. +#[derive(Serialize, Debug, PartialEq, Clone)] +#[serde(tag = "kind", rename_all = "lowercase")] +pub enum PartitionKind { + Daily, + Hourly, + Weekly, + Monthly, + Dynamic { key: String }, +} + +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct PartitionSpec { + #[serde(flatten)] + pub kind: PartitionKind, + #[serde(skip_serializing_if = "Option::is_none")] + pub tz: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub format: Option, + // ISO-8601 (e.g. "2024-01-01") or kind-specific start anchor. Older + // partitions before this anchor are not backfilled. + #[serde(skip_serializing_if = "Option::is_none")] + pub start: Option, +} + +// Freshness SLA. The duration is kept as a raw string ("1h", "30m", "2d") +// and validated downstream — the parser deliberately doesn't bind to a +// specific duration crate so the annotation grammar stays parser-light. +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct FreshnessSpec { + pub duration: String, +} + +// Retry policy declared via `// retry []`. The delay is kept +// as a raw duration string (mirrors freshness/debounce_default) and resolved +// to seconds at deploy via `parse_duration_secs`; absent = back-to-back +// re-runs with no inter-attempt wait. +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct RetrySpec { + pub count: u32, + #[serde(skip_serializing_if = "Option::is_none")] + pub delay: Option, +} + +// `// materialize [manual] [append] [key=]` — declares that this +// script produces a *managed* materialization of `` (a `ducklake://` +// table). By default the runtime generates the write DDL around the script's +// single trailing `SELECT` and owns idempotency, partition-state and snapshot +// capture. `manual` is the escape hatch: the script writes its own DDL and the +// runtime only records state (track-only). The reconciliation strategy options +// (`append`, `key=`) apply to managed mode: none → DELETE-by-partition + +// INSERT (replace); `key=` → MERGE (dedup within slice); `append` → +// INSERT-only. `append` wins if both are given (deploy-time warning). +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct MaterializeSpec { + pub target_kind: AssetKind, + pub target_path: String, + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub manual: bool, + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + pub append: bool, + #[serde(skip_serializing_if = "Option::is_none", default)] + pub unique_key: Option, +} + +// `// data_test …` — a data-quality assertion run against the +// freshly-materialized asset (post DELETE+INSERT), failing the run on +// violation. The first extensible annotation family: the parser turns a +// `data_test` line into one of a known *vocabulary* of checks, and the +// runtime turns each check into a SQL "verifier" probe. A sibling annotation +// family (e.g. column-lineage) follows the same shape — a keyword head +// selecting a variant, the rest parsed per-variant — rather than growing a +// new closed list. See `docs/ducklake-materialization.md` §"Extensible +// annotations". Multiple `// data_test` lines accumulate (unlike the +// single-value annotations above, which are first-write-wins). +// +// Built-ins mirror dbt's generic data tests; `Custom` is the escape hatch +// (dbt's singular test): a DuckDB script path whose SELECT returns the +// violating rows. The keyword is `data_test` — NOT `test` — to stay clear +// of the unrelated `// test:` CI-test annotation (see +// `windmill_common::schema::parse_ci_test_annotation`), matching dbt 1.8's +// own `tests:` → `data_tests:` rename. +#[derive(Serialize, Debug, PartialEq, Clone)] +#[serde(tag = "type", rename_all = "snake_case")] +pub enum DataTest { + // `// data_test unique ` — no two non-NULL rows share `column`. + Unique { column: String }, + // `// data_test not_null ` — `column` is never NULL. + NotNull { column: String }, + // `// data_test accepted_values = a,b,c` — every non-NULL value of + // `column` is one of `values` (comma-separated; surrounding quotes stripped). + AcceptedValues { column: String, values: Vec }, + // `// data_test relationships -> .` — referential + // integrity: every non-NULL `column` value exists in `to_path`'s `to_column`. + Relationships { column: String, to_kind: AssetKind, to_path: String, to_column: String }, + // `// data_test ` — escape hatch: a deployed DuckDB script + // whose trailing SELECT returns the violating rows (non-empty ⇒ fail). + Custom { path: String }, +} + +// `// column <- .[, …]` — declared column-level +// lineage: one output column of this script's produced asset and the upstream +// source columns it derives from. A sibling of `DataTest` in the extensible +// annotation family (`docs/pipelines-vs-dbt.md` §3): same parse shape — a head +// token (the output column) then a per-variant tail — but accumulating, one +// line per output column. Unlike `data_test` these are pure metadata: they +// drive the column-lineage graph view, never a runtime probe. +// +// dbt derives column lineage from SQL-AST parsing; Windmill is polyglot +// (Python/TS/Bash/SQL in one DAG), so a uniform AST is not available. The +// annotation is the explicit, language-agnostic declaration — the same +// "annotations are real comments parsed strictly" stance as the rest of the +// pipeline grammar. Body-inferred per-asset column *sets* (`columns` on +// `ParseAssetsResult`) complement it but cannot express column→column edges. +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnLineage { + // The produced asset's output column this line describes. + pub column: String, + // Upstream source columns it derives from (≥1; malformed refs dropped). + pub inputs: Vec, +} + +// One `.` upstream reference inside a `// column` line. The +// asset URI accepts the default-syntax shorthands (like `// materialize` / +// `// data_test relationships`); the column is the segment after the final +// `.` (so a schema-qualified `warehouse/main.orders.amount` keeps `amount`). +#[derive(Serialize, Debug, PartialEq, Clone)] +pub struct ColumnRef { + pub from_kind: AssetKind, + pub from_path: String, + pub from_column: String, +} + +// `// trigger any` (default) vs `// trigger all`. `Any` = OR: any trigger +// firing runs the script (current behaviour). `All` = AND: the script +// runs only once every partition-bearing input has materialized at the +// same partition (plus every reference input exists) — the join barrier. +#[derive(Serialize, Debug, PartialEq, Eq, Clone, Copy, Default)] +#[serde(rename_all = "lowercase")] +pub enum JoinMode { + #[default] + Any, + All, +} + +impl JoinMode { + pub fn is_any(&self) -> bool { + matches!(self, JoinMode::Any) + } +} + +// All pipeline-level annotations parsed off a script's source. Returned by +// `parse_pipeline_annotations` and forwarded into `ParseAssetsOutput`. +#[derive(Default, Debug, PartialEq, Clone)] +pub struct PipelineAnnotations { + pub in_pipeline: bool, + pub triggers: Vec, + pub partition: Option, + pub freshness: Option, + pub join_mode: JoinMode, + pub debounce_default: Option, + pub tag: Option, + pub retry: Option, + pub materialize: Option, + pub data_tests: Vec, + pub column_lineage: Vec, +} + +impl ParseAssetsOutput { + /// Build from detected assets/queries plus the script's parsed + /// pipeline annotations, so each language asset-parser does not + /// re-list the per-annotation fields (one call site instead of six + /// lines kept in lockstep across the parser crates). + pub fn new( + assets: Vec, + sql_queries: Vec, + pipeline: PipelineAnnotations, + ) -> Self { + ParseAssetsOutput { + assets, + sql_queries, + in_pipeline: pipeline.in_pipeline, + triggers: pipeline.triggers, + partition: pipeline.partition, + freshness: pipeline.freshness, + join_mode: pipeline.join_mode, + debounce_default: pipeline.debounce_default, + tag: pipeline.tag, + retry: pipeline.retry, + materialize: pipeline.materialize, + data_tests: pipeline.data_tests, + column_lineage: pipeline.column_lineage, + } + } +} + +// Combine column lineage inferred from the body (SQL AST) with lineage declared +// via `// column` annotations. The annotation is the *override*: where both +// describe the same output column, the explicit declaration wins and the +// inferred entry is dropped. Inferred entries are also deduped by output column +// among themselves (first wins). Used by the language asset-parsers so a +// `// column` line can correct a mis-inferred edge without disabling inference +// for the rest of the columns. +pub fn merge_column_lineage( + inferred: Vec, + annotated: Vec, +) -> Vec { + let mut seen: std::collections::HashSet = + annotated.iter().map(|c| c.column.clone()).collect(); + let mut out = annotated; + for c in inferred { + if seen.insert(c.column.clone()) { + out.push(c); + } + } + out } #[derive(Debug, Clone, Serialize)] @@ -64,13 +413,12 @@ pub fn merge_assets(assets: Vec) -> Vec { .iter_mut() .find(|x| x.path == asset.path && x.kind == asset.kind) { - // merge access types + // merge access types — a None on either side means ambiguous + // usage (unknown access), which poisons the merge to None; + // otherwise delegate to the shared truth table. existing.access_type = match (asset.access_type, existing.access_type) { (None, _) | (_, None) => None, - (Some(R), Some(W)) | (Some(W), Some(R)) => Some(RW), - (Some(RW), _) | (_, Some(RW)) => Some(RW), - (Some(R), Some(R)) => Some(R), - (Some(W), Some(W)) => Some(W), + (Some(a), Some(b)) => Some(merge_access_types(a, b)), }; // merge columns: union the column sets and merge access types per column existing.columns = merge_column_maps(existing.columns.take(), asset.columns); @@ -153,3 +501,1281 @@ pub const ASSET_KINDS: &[(&str, AssetKind)] = &[ ("datatable://", AssetKind::DataTable), ("volume://", AssetKind::Volume), ]; + +// Tokenize a `key=value [key="quoted value"] ...` option string. Bare +// values run until the next whitespace; quoted values consume until the +// matching quote. Malformed pairs (missing `=` or empty key) are skipped +// rather than aborting the whole annotation. +// Split a `// on` right-hand side into the trigger ref and any trailing +// `key=value` opts. The opts section starts at the first whitespace token +// shaped like `=…` (e.g. `debounce=60s`); everything before is the +// asset/kind ref. Asset refs aren't expected to contain a space then an +// `ident=` token — the same assumption `// partitioned` already makes. +fn split_trailing_kv_opts(s: &str) -> (&str, BTreeMap) { + let mut split_at: Option = None; + for tok in s.split_whitespace() { + // `split_whitespace` yields slices borrowed from `s`, so the exact + // byte offset is the pointer delta — substring search (`find`) would + // misfire when an earlier token also appears inside a later one. + let tok_start = tok.as_ptr() as usize - s.as_ptr() as usize; + if let Some(eq) = tok.find('=') { + let key = &tok[..eq]; + if !key.is_empty() + && key.starts_with(|c: char| c.is_ascii_alphabetic() || c == '_') + && key.chars().all(|c| c.is_ascii_alphanumeric() || c == '_') + { + split_at = Some(tok_start); + break; + } + } + } + match split_at { + Some(i) => (s[..i].trim_end(), parse_kv_opts(&s[i..])), + None => (s.trim_end(), BTreeMap::new()), + } +} + +fn parse_kv_opts(s: &str) -> BTreeMap { + let mut out = BTreeMap::new(); + let mut chars = s.chars().peekable(); + loop { + while chars.peek().map_or(false, |c| c.is_whitespace()) { + chars.next(); + } + if chars.peek().is_none() { + break; + } + let mut key = String::new(); + while let Some(&c) = chars.peek() { + if c == '=' || c.is_whitespace() { + break; + } + key.push(c); + chars.next(); + } + if chars.peek() != Some(&'=') || key.is_empty() { + // Malformed — skip until next whitespace to recover. + while chars.peek().map_or(false, |c| !c.is_whitespace()) { + chars.next(); + } + continue; + } + chars.next(); // consume '=' + let value = match chars.peek().copied() { + Some(q @ ('"' | '\'')) => { + chars.next(); + let mut v = String::new(); + while let Some(&c) = chars.peek() { + chars.next(); + if c == q { + break; + } + v.push(c); + } + v + } + _ => { + let mut v = String::new(); + while let Some(&c) = chars.peek() { + if c.is_whitespace() { + break; + } + v.push(c); + chars.next(); + } + v + } + }; + out.insert(key, value); + } + out +} + +// Scan the leading comment header for pipeline annotations. Only the +// contiguous block of comment lines at the top of the file is considered +// (blank lines tolerated, scan stops at the first line of actual code) so +// that ordinary comments in the body can't false-positive as annotations. +// Language-agnostic: any header line whose first non-whitespace tokens are +// a comment prefix (`//`, `#`, or `--`) followed by one of the recognized +// keywords: +// - `pipeline` → opt-in marker (must be alone on the line) +// - `on ` → asset / native trigger edge (including +// the marker-only `on schedule` form) +// - `partitioned [opts]` → partition declaration +// - `freshness ` → SLA / active backstop +// - `tag ` → worker-tag override (annotation wins +// over UI-set value at deploy) +// - `retry []` → cascade-only retry policy +// +// `// pipeline` is intentionally strict — only whitespace allowed after the +// keyword. Without that constraint, casual prose like `// pipeline broken +// on staging` would false-positive (the word "pipeline" is far more common +// in normal comments than "materialize" was). +// +// `partition`, `freshness`, `tag`, and `retry` use first-write-wins; if +// multiple lines declare them, the first one is kept (last would be +// reasonable too, but first matches the file-top convention developers +// follow). +// Try to consume `` as a complete word from `rest`. Returns the trailing +// text after the keyword if it matched (empty or whitespace-bounded), +// `None` otherwise. Prevents `partitioned` matching `partition`, `pipelines` +// matching `pipeline`, etc. Mirrors `consumeKeyword` in +// parsePipelineAnnotations.ts. +fn consume_keyword<'a>(rest: &'a str, kw: &str) -> Option<&'a str> { + let after = rest.strip_prefix(kw)?; + if after.is_empty() || after.starts_with(|c: char| c.is_whitespace()) { + Some(after) + } else { + None + } +} + +pub fn parse_pipeline_annotations(code: &str) -> PipelineAnnotations { + let mut out = PipelineAnnotations::default(); + + for raw_line in code.lines() { + let line = raw_line.trim_start(); + if line.is_empty() { + continue; + } + let rest = if let Some(r) = line.strip_prefix("//") { + r + } else if let Some(r) = line.strip_prefix("--") { + r + } else if let Some(r) = line.strip_prefix('#') { + r + } else { + // Annotations live in the leading comment header. Stop at the first + // line of actual code so comments inside the body (e.g. a regular + // `# tag ...` prose comment) can't false-positive as annotations. + // Mirrors BashAnnotations::sandbox_image / ssh_target. + break; + }; + let rest = rest.trim_start(); + + if let Some(after_kw) = consume_keyword(rest, "pipeline") { + // Strict: keyword must be the only content on the line. Rejects + // `pipeline broken`, `pipelines`, `pipeline-related`, etc. + if after_kw.trim().is_empty() { + out.in_pipeline = true; + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "partitioned") { + if out.partition.is_none() { + if let Some(spec) = parse_partitioned_spec(after_kw.trim()) { + out.partition = Some(spec); + } + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "freshness") { + let dur = after_kw.trim(); + if !dur.is_empty() && out.freshness.is_none() { + out.freshness = Some(FreshnessSpec { duration: dur.to_string() }); + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "trigger") { + match after_kw.trim() { + "all" => out.join_mode = JoinMode::All, + "any" => out.join_mode = JoinMode::Any, + // Unknown value — leave the default rather than guess. + _ => {} + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "debounce") { + let dur = after_kw.trim(); + if !dur.is_empty() && out.debounce_default.is_none() { + out.debounce_default = Some(dur.to_string()); + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "tag") { + let name = after_kw.trim(); + // Worker tags are single-word identifiers (e.g. `heavy`, `gpu`). + // A value with whitespace or beyond the `script.tag` column width + // is almost certainly a regular comment starting with "# tag ...". + if !name.is_empty() + && !name.contains(char::is_whitespace) + && name.len() <= 50 + && out.tag.is_none() + { + out.tag = Some(name.to_string()); + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "retry") { + if out.retry.is_none() { + if let Some(spec) = parse_retry_spec(after_kw.trim()) { + out.retry = Some(spec); + } + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "materialize") { + if out.materialize.is_none() { + if let Some(spec) = parse_materialize_spec(after_kw.trim()) { + out.materialize = Some(spec); + } + } + continue; + } + + // `data_test` is checked before `on`/asset shorthands and is a complete + // word (so it never collides with the `// test:` CI annotation, which + // has no whitespace after `test`). Accumulates — every well-formed line + // adds a check; malformed lines are dropped (fail-safe, the missing + // check is then simply absent from the graph + run). + if let Some(after_kw) = consume_keyword(rest, "data_test") { + if let Some(spec) = parse_data_test_spec(after_kw.trim()) { + out.data_tests.push(spec); + } + continue; + } + + // `// column <- .[, …]` — accumulating column lineage. + // A complete word, so it never swallows a body comment that happens to + // start with `column` followed by non-lineage prose (that has no `<-` + // and is dropped fail-safe). Checked before `on`/asset shorthands. + if let Some(after_kw) = consume_keyword(rest, "column") { + if let Some(spec) = parse_column_lineage_spec(after_kw.trim()) { + out.column_lineage.push(spec); + } + continue; + } + + if let Some(after_kw) = consume_keyword(rest, "on") { + let spec_text = after_kw.trim(); + if spec_text.is_empty() { + continue; + } + // Split off trailing `key=value` opts (e.g. `debounce=60s`) + // from the asset/kind ref. The per-input debounce override is + // only meaningful for asset inputs (cascade fan-out); other + // trigger kinds ignore it. + let (ref_part, opts) = split_trailing_kv_opts(spec_text); + if let Some(mut trig) = parse_trigger_spec(ref_part) { + if let TriggerSpec::Asset { debounce, .. } = &mut trig { + *debounce = opts + .get("debounce") + .map(|s| s.trim().to_string()) + .filter(|s| !s.is_empty()); + } + if !out.triggers.contains(&trig) { + out.triggers.push(trig); + } + } + } + } + + out +} + +// Parse a `// retry []` right-hand side. `` is a +// non-negative decimal; `` is an optional raw duration string left +// for `parse_duration_secs` to validate at deploy. A bare zero count (or +// non-numeric token) is rejected — that is, the annotation must encode a +// real retry policy or be omitted, so a typo (`// retry retry 3`) fails +// safe rather than silently disabling cascade retries. +fn parse_retry_spec(s: &str) -> Option { + let mut split = s.splitn(2, char::is_whitespace); + let count_word = split.next()?.trim(); + let count: u32 = count_word.parse().ok()?; + if count == 0 { + return None; + } + let delay = split + .next() + .map(|d| d.trim().to_string()) + .filter(|d| !d.is_empty()); + Some(RetrySpec { count, delay }) +} + +// Parse a `// materialize [manual] [append] [key=]` right-hand +// side. An optional leading `manual` token (whitespace-delimited) opts out of +// managed mode (track-only). The next whitespace token is the target asset URI +// (default-syntax shorthands enabled, so `ducklake` → `ducklake://main`); the +// remainder are strategy options — bare `append` and `key=` (merge key), +// which apply to managed mode only. A missing/empty target yields `None` (the +// annotation is dropped, fail-safe). +fn parse_materialize_spec(s: &str) -> Option { + let (manual, rest) = match s.strip_prefix("manual") { + Some(after) if after.is_empty() || after.starts_with(char::is_whitespace) => { + (true, after.trim_start()) + } + _ => (false, s), + }; + let mut it = rest.trim().splitn(2, char::is_whitespace); + let asset_tok = it.next()?; + let opts_str = it.next().unwrap_or(""); + let (target_kind, path) = parse_asset_syntax(asset_tok.trim(), true)?; + if path.is_empty() { + return None; + } + let append = opts_str.split_whitespace().any(|t| t == "append"); + let unique_key = parse_kv_opts(opts_str) + .get("key") + .filter(|k| !k.is_empty()) + .cloned(); + Some(MaterializeSpec { target_kind, target_path: path.to_string(), manual, append, unique_key }) +} + +// Parse a `// data_test …` right-hand side into one `DataTest`. The +// leading token selects the variant; the remainder is parsed per-variant. +// Anything not matching a built-in keyword is the `Custom` escape hatch — a +// single script-path token. Returns `None` for malformed input so a typo +// fails safe (the check is dropped, never silently mis-parsed). +// +// This is the extension seam: a new built-in is one match arm + its parser; +// a sibling annotation family (column-lineage) reuses the same head-keyword +// dispatch shape rather than adding a parallel closed list. +fn parse_data_test_spec(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() { + return None; + } + let mut it = s.splitn(2, char::is_whitespace); + let head = it.next()?; + let rest = it.next().unwrap_or("").trim(); + match head { + "unique" => Some(DataTest::Unique { column: single_ident(rest)? }), + "not_null" => Some(DataTest::NotNull { column: single_ident(rest)? }), + "accepted_values" => parse_accepted_values(rest), + "relationships" => parse_relationships(rest), + // Custom escape hatch: the whole right-hand side must be one path token + // (`head` with no trailing content). Trailing content after a + // non-built-in head is a malformed built-in (e.g. `uniq order_id`) and + // is rejected rather than misread as a path. + _ if rest.is_empty() => Some(DataTest::Custom { path: head.to_string() }), + _ => None, + } +} + +// A single bare identifier token (column name). Rejects empty / multi-token +// input. The identifier is double-quoted + escaped at codegen, so any +// character is safe here; we only enforce "exactly one token". +fn single_ident(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() || s.split_whitespace().count() != 1 { + return None; + } + Some(s.to_string()) +} + +// Strip one layer of matching surrounding single or double quotes. +fn unquote(s: &str) -> &str { + let b = s.as_bytes(); + if b.len() >= 2 && (b[0] == b'"' || b[0] == b'\'') && b[b.len() - 1] == b[0] { + &s[1..s.len() - 1] + } else { + s + } +} + +// `= a,b,c` — column, then `=`, then a comma-separated value list. +// Surrounding quotes are stripped per value; empty values are dropped; a +// value may not itself contain a comma (v1 limitation). +fn parse_accepted_values(s: &str) -> Option { + let (col, vals) = s.split_once('=')?; + let column = single_ident(col)?; + let values: Vec = vals + .split(',') + .map(|v| unquote(v.trim()).to_string()) + .filter(|v| !v.is_empty()) + .collect(); + if values.is_empty() { + return None; + } + Some(DataTest::AcceptedValues { column, values }) +} + +// `-> .` — referential integrity. The referenced +// column is the segment after the final `.`; everything before it is the +// asset URI (default-syntax shorthands enabled, like `// materialize`). +fn parse_relationships(s: &str) -> Option { + let (col, target) = s.split_once("->")?; + let column = single_ident(col)?; + let target = target.trim(); + let (asset_uri, ref_col) = target.rsplit_once('.')?; + let to_column = single_ident(ref_col)?; + let (to_kind, to_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if to_path.is_empty() { + return None; + } + Some(DataTest::Relationships { column, to_kind, to_path: to_path.to_string(), to_column }) +} + +// Parse a `// column <- [, …]` right-hand side. The head +// (before `<-`) is the output column; the tail is a comma-separated list of +// `.` upstream references. Mirrors `parse_accepted_values`' +// "drop empties, require ≥1" stance: individually malformed refs are dropped +// and the line is kept iff at least one ref parses; a missing `<-`, a non-ident +// output column, or zero valid refs drops the whole line (fail-safe). +fn parse_column_lineage_spec(s: &str) -> Option { + let (out_col, refs) = s.split_once("<-")?; + let column = single_ident(out_col)?; + let inputs: Vec = refs + .split(',') + .filter_map(|r| parse_column_ref(r.trim())) + .collect(); + if inputs.is_empty() { + return None; + } + Some(ColumnLineage { column, inputs }) +} + +// `.` — the referenced column is the segment after the final +// `.`; everything before it is the asset URI (default-syntax shorthands +// enabled, like `// materialize`). Same shape as `parse_relationships`' target. +fn parse_column_ref(s: &str) -> Option { + let (asset_uri, ref_col) = s.rsplit_once('.')?; + let from_column = single_ident(ref_col)?; + let (from_kind, from_path) = parse_asset_syntax(asset_uri.trim(), true)?; + if from_path.is_empty() { + return None; + } + Some(ColumnRef { from_kind, from_path: from_path.to_string(), from_column }) +} + +// Parse a `// partitioned [opts]` right-hand side. Recognized kinds: +// `daily`, `hourly`, `weekly`, `monthly` (with optional tz/format/start), +// and `dynamic key=""` (plus optional format). +fn parse_partitioned_spec(s: &str) -> Option { + let mut split = s.splitn(2, char::is_whitespace); + let kind_word = split.next()?; + let opts_str = split.next().unwrap_or(""); + let opts = parse_kv_opts(opts_str); + let kind = match kind_word { + "daily" => PartitionKind::Daily, + "hourly" => PartitionKind::Hourly, + "weekly" => PartitionKind::Weekly, + "monthly" => PartitionKind::Monthly, + "dynamic" => { + let key = opts.get("key")?.clone(); + if key.is_empty() { + return None; + } + PartitionKind::Dynamic { key } + } + _ => return None, + }; + Some(PartitionSpec { + kind, + tz: opts.get("tz").cloned(), + format: opts.get("format").cloned(), + start: opts.get("start").cloned(), + }) +} + +// Parse a single `on ` right-hand side. Accepted forms: +// — where is one of +// webhook | email | kafka | mqtt | nats | postgres | sqs | gcp +// (e.g. s3://bucket/key, $res:f/foo) +// +// Native trigger keywords are *marker-only* — no trailing path. The actual +// binding lives on the native trigger row (`script_path` column). Anything +// trailing the keyword is rejected so the form stays unambiguous. +fn parse_trigger_spec(s: &str) -> Option { + // Marker-only native trigger keywords. The match table keeps the + // annotation set in lockstep with `TriggerSpec`. `schedule` is in here + // too — the cron lives on the schedule row the user creates separately; + // the annotation is just the binding declaration. + const NATIVE_KINDS: &[(&str, TriggerSpec)] = &[ + ("schedule", TriggerSpec::Schedule), + ("webhook", TriggerSpec::Webhook), + ("email", TriggerSpec::Email), + ("kafka", TriggerSpec::Kafka), + ("mqtt", TriggerSpec::Mqtt), + ("nats", TriggerSpec::Nats), + ("postgres", TriggerSpec::Postgres), + ("sqs", TriggerSpec::Sqs), + ("gcp", TriggerSpec::Gcp), + ("data_upload", TriggerSpec::DataUpload), + ]; + for (kw, spec) in NATIVE_KINDS { + if let Some(rest) = s.strip_prefix(kw) { + // Must be a complete word — `kafkalike` doesn't match `kafka`. + // Trailing whitespace alone is fine; any non-empty trailing + // content is treated as malformed (the annotation is marker-only). + if !rest.is_empty() && !rest.starts_with(|c: char| c.is_whitespace()) { + continue; + } + if !rest.trim().is_empty() { + return None; + } + return Some(spec.clone()); + } + } + + let (kind, path) = parse_asset_syntax(s, false)?; + // `debounce` is attached by the caller from the `// on` line's opts. + Some(TriggerSpec::Asset { asset_kind: kind, path: path.to_string(), debounce: None }) +} + +#[cfg(test)] +mod pipeline_annotation_tests { + use super::*; + + #[test] + fn bare_pipeline_marker() { + let out = parse_pipeline_annotations("// pipeline\nconsole.log('hi')"); + assert!(out.in_pipeline); + assert!(out.triggers.is_empty()); + } + + #[test] + fn pipeline_marker_strict_grammar_rejects_trailing_words() { + // Strict — the word "pipeline" is common in casual prose, so trailing + // content disqualifies the line. Trailing whitespace alone is fine. + let out = parse_pipeline_annotations( + "// pipeline broken on staging\n# pipeline this through grep\n-- pipeline_v2", + ); + assert!(!out.in_pipeline); + + let out = parse_pipeline_annotations("// pipeline \n"); + assert!(out.in_pipeline); + } + + #[test] + fn rejects_pipeline_keyword_variants() { + let out = parse_pipeline_annotations("// pipelines\n# pipelined\n-- pipeline-foo"); + assert!(!out.in_pipeline); + } + + #[test] + fn on_schedule_marker() { + // `// on schedule` is marker-only — the binding is the schedule row's + // own `script_path` field, just like kafka/mqtt/etc. + let out = parse_pipeline_annotations("// on schedule"); + assert_eq!(out.triggers.len(), 1); + assert_eq!(out.triggers[0], TriggerSpec::Schedule); + } + + #[test] + fn rejects_schedule_with_trailing_content() { + // Marker-only — the old `// schedule ""` form is gone, and a + // trailing path/cron on the `on schedule` form is malformed. + let out = parse_pipeline_annotations("// on schedule \"0 0 * * *\""); + assert!(out.triggers.is_empty()); + let out = parse_pipeline_annotations("// schedule \"0 0 * * *\""); + assert!(out.triggers.is_empty()); + } + + #[test] + fn on_asset_ts_py_sql() { + let code = "// on s3://a/b\n# on datatable://main\n-- on $res:f/foo"; + let out = parse_pipeline_annotations(code); + assert_eq!(out.triggers.len(), 3); + assert!(matches!( + out.triggers[0], + TriggerSpec::Asset { asset_kind: AssetKind::S3Object, .. } + )); + assert!(matches!( + out.triggers[1], + TriggerSpec::Asset { asset_kind: AssetKind::DataTable, .. } + )); + assert!(matches!( + out.triggers[2], + TriggerSpec::Asset { asset_kind: AssetKind::Resource, .. } + )); + } + + #[test] + fn on_deduplicates() { + let code = "// on s3://a/b\n# on s3://a/b"; + let out = parse_pipeline_annotations(code); + assert_eq!(out.triggers.len(), 1); + } + + #[test] + fn rejects_unknown_trigger_spec() { + let out = parse_pipeline_annotations("// on unknown://nope\n# schedule"); + assert!(out.triggers.is_empty()); + } + + #[test] + fn join_mode_defaults_to_any() { + let out = parse_pipeline_annotations("// on s3://a/b"); + assert_eq!(out.join_mode, JoinMode::Any); + assert!(out.join_mode.is_any()); + } + + #[test] + fn trigger_all_sets_and_mode() { + let out = parse_pipeline_annotations( + "// pipeline\n// on s3://lake/{partition}/x\n// trigger all", + ); + assert_eq!(out.join_mode, JoinMode::All); + assert!(!out.join_mode.is_any()); + } + + #[test] + fn trigger_any_explicit_and_other_prefixes() { + // explicit `any`, plus `#` / `--` comment prefixes are accepted. + assert_eq!( + parse_pipeline_annotations("# trigger all\n-- trigger any").join_mode, + JoinMode::Any + ); + assert_eq!( + parse_pipeline_annotations("-- trigger all").join_mode, + JoinMode::All + ); + } + + #[test] + fn trigger_unknown_or_glued_keeps_default() { + // unknown value, and `triggerall` (no whitespace) must not match. + assert_eq!( + parse_pipeline_annotations("// trigger bogus").join_mode, + JoinMode::Any + ); + assert_eq!( + parse_pipeline_annotations("// triggerall").join_mode, + JoinMode::Any + ); + } + + #[test] + fn is_partition_bearing_only_for_tokened_assets() { + let out = parse_pipeline_annotations( + "// on s3://lake/raw/{partition}/events.parquet\n\ + // on s3://lake/dim/customers.parquet\n\ + // on schedule", + ); + assert_eq!(out.triggers.len(), 3); + assert!(out.triggers[0].is_partition_bearing()); + assert!(!out.triggers[1].is_partition_bearing()); + assert!(!out.triggers[2].is_partition_bearing()); + } + + fn asset_debounce(t: &TriggerSpec) -> Option<&str> { + match t { + TriggerSpec::Asset { debounce, .. } => debounce.as_deref(), + _ => None, + } + } + + #[test] + fn debounce_none_by_default() { + let out = parse_pipeline_annotations("// on s3://a/b"); + assert_eq!(out.debounce_default, None); + assert_eq!(asset_debounce(&out.triggers[0]), None); + } + + #[test] + fn script_level_debounce_default() { + let out = parse_pipeline_annotations("// debounce 30s\n// on s3://a/b"); + assert_eq!(out.debounce_default.as_deref(), Some("30s")); + // Per-edge override absent — precedence (edge ?? default) is + // resolved at deploy, so the Asset itself stays None here. + assert_eq!(asset_debounce(&out.triggers[0]), None); + } + + #[test] + fn on_level_debounce_override() { + let out = parse_pipeline_annotations( + "// debounce 30s\n\ + // on s3://lake/{partition}/raw.parquet debounce=60s\n\ + // on $res:f/cfg", + ); + assert_eq!(out.debounce_default.as_deref(), Some("30s")); + assert_eq!(out.triggers.len(), 2); + assert_eq!(asset_debounce(&out.triggers[0]), Some("60s")); + // ref still parses correctly with the trailing opt stripped. + assert!(out.triggers[0].is_partition_bearing()); + assert_eq!(asset_debounce(&out.triggers[1]), None); + } + + #[test] + fn debounce_keyword_strictness_and_first_wins() { + // `debounced` (no space) must not match; empty ignored; first wins. + let out = parse_pipeline_annotations( + "// debounced nope\n// debounce\n// debounce 1m\n// debounce 5m", + ); + assert_eq!(out.debounce_default.as_deref(), Some("1m")); + } + + #[test] + fn native_trigger_keywords_are_marker_only() { + // Marker form: `// on kafka` parses to the unit variant. + let out = parse_pipeline_annotations("// on kafka"); + assert_eq!(out.triggers.len(), 1); + assert!(matches!(out.triggers[0], TriggerSpec::Kafka)); + + // Old path-bearing form is rejected (no path on native markers). + let out = parse_pipeline_annotations("// on webhook f/foo"); + assert!(out.triggers.is_empty()); + + // Trailing key=value opts are silently dropped by the line-level + // KV splitter before parse_trigger_spec sees them — same behaviour + // for both asset and native kinds. The opts have no meaning for a + // marker, but the marker still parses. + let out = parse_pipeline_annotations("// on mqtt debounce=30s"); + assert_eq!(out.triggers.len(), 1); + assert!(matches!(out.triggers[0], TriggerSpec::Mqtt)); + + // `kafkalike` mustn't match `kafka`. + let out = parse_pipeline_annotations("// on kafkalike"); + assert!(out.triggers.is_empty()); + } + + #[test] + fn all_native_marker_keywords_parse() { + let code = "// on webhook\n// on email\n// on kafka\n// on mqtt\n\ + // on nats\n// on postgres\n// on sqs\n// on gcp\n// on data_upload"; + let out = parse_pipeline_annotations(code); + assert_eq!(out.triggers.len(), 9); + assert!(matches!(out.triggers[0], TriggerSpec::Webhook)); + assert!(matches!(out.triggers[1], TriggerSpec::Email)); + assert!(matches!(out.triggers[2], TriggerSpec::Kafka)); + assert!(matches!(out.triggers[3], TriggerSpec::Mqtt)); + assert!(matches!(out.triggers[4], TriggerSpec::Nats)); + assert!(matches!(out.triggers[5], TriggerSpec::Postgres)); + assert!(matches!(out.triggers[6], TriggerSpec::Sqs)); + assert!(matches!(out.triggers[7], TriggerSpec::Gcp)); + assert!(matches!(out.triggers[8], TriggerSpec::DataUpload)); + } + + #[test] + fn data_upload_marker_is_marker_only() { + // `// on data_upload` parses to the unit variant — no path. + let out = parse_pipeline_annotations("// on data_upload"); + assert_eq!(out.triggers.len(), 1); + assert!(matches!(out.triggers[0], TriggerSpec::DataUpload)); + + // Trailing content makes it malformed (marker-only). + let out = parse_pipeline_annotations("// on data_upload f/foo"); + assert!(out.triggers.is_empty()); + + // `data_uploadish` mustn't match `data_upload`. + let out = parse_pipeline_annotations("// on data_uploadish"); + assert!(out.triggers.is_empty()); + } + + #[test] + fn partitioned_daily() { + let code = "// partitioned daily tz=\"UTC\" format=\"YYYY-MM-DD\" start=\"2024-01-01\""; + let out = parse_pipeline_annotations(code); + let p = out.partition.expect("partition"); + assert_eq!(p.kind, PartitionKind::Daily); + assert_eq!(p.tz.as_deref(), Some("UTC")); + assert_eq!(p.format.as_deref(), Some("YYYY-MM-DD")); + assert_eq!(p.start.as_deref(), Some("2024-01-01")); + } + + #[test] + fn partitioned_hourly_minimal() { + let out = parse_pipeline_annotations("// partitioned hourly"); + let p = out.partition.expect("partition"); + assert_eq!(p.kind, PartitionKind::Hourly); + assert!(p.tz.is_none()); + } + + #[test] + fn partitioned_dynamic_requires_key() { + let out = parse_pipeline_annotations("// partitioned dynamic"); + assert!(out.partition.is_none()); + let out = parse_pipeline_annotations("// partitioned dynamic key=\"$.tenant_id\""); + let p = out.partition.expect("partition"); + assert_eq!( + p.kind, + PartitionKind::Dynamic { key: "$.tenant_id".to_string() } + ); + } + + #[test] + fn partitioned_first_wins() { + let code = "// partitioned daily tz=\"UTC\"\n// partitioned hourly"; + let out = parse_pipeline_annotations(code); + let p = out.partition.expect("partition"); + assert_eq!(p.kind, PartitionKind::Daily); + } + + #[test] + fn partitioned_unknown_kind_is_skipped() { + let out = parse_pipeline_annotations("// partitioned bogus"); + assert!(out.partition.is_none()); + } + + #[test] + fn freshness_basic() { + let out = parse_pipeline_annotations("// freshness 1h"); + assert_eq!(out.freshness.unwrap().duration, "1h"); + } + + #[test] + fn freshness_first_wins() { + let out = parse_pipeline_annotations("// freshness 1h\n# freshness 30m"); + assert_eq!(out.freshness.unwrap().duration, "1h"); + } + + #[test] + fn tag_basic() { + let out = parse_pipeline_annotations("// tag heavy"); + assert_eq!(out.tag.as_deref(), Some("heavy")); + } + + #[test] + fn tag_first_wins() { + let out = parse_pipeline_annotations("// tag heavy\n# tag light"); + assert_eq!(out.tag.as_deref(), Some("heavy")); + } + + #[test] + fn tag_empty_is_skipped() { + let out = parse_pipeline_annotations("// tag "); + assert!(out.tag.is_none()); + } + + #[test] + fn tag_with_whitespace_is_skipped() { + // A regular English comment starting with "# tag " must not be + // mistaken for a worker-tag annotation (worker tags are single words). + let out = + parse_pipeline_annotations("# tag this function so we remember to refactor it later"); + assert!(out.tag.is_none()); + } + + #[test] + fn tag_too_long_is_skipped() { + let long = "x".repeat(51); + let out = parse_pipeline_annotations(&format!("// tag {long}")); + assert!(out.tag.is_none()); + } + + #[test] + fn annotations_in_body_are_ignored() { + // Only the leading comment header is scanned. A regular `# tag ...` + // prose comment buried in the body — the WIN-2090 false-positive that + // crashed the `script.tag` INSERT — must not be treated as an + // annotation once real code has started. + let code = concat!( + "import pandas as pd\n", + "\n", + "def main():\n", + " # tag each row with its source so downstream steps can filter\n", + " # on s3://should/not/parse\n", + " return pd.DataFrame()\n", + ); + let out = parse_pipeline_annotations(code); + assert!(out.tag.is_none()); + assert!(out.triggers.is_empty()); + } + + #[test] + fn header_allows_blank_lines_before_code() { + // Blank lines (e.g. after a shebang) don't end the header; the first + // line of real code does. + let code = concat!( + "#!/usr/bin/env python\n", + "\n", + "# tag heavy\n", + "import os\n", + "# tag light\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!(out.tag.as_deref(), Some("heavy")); + } + + #[test] + fn retry_count_only() { + let out = parse_pipeline_annotations("// retry 3"); + let r = out.retry.expect("retry"); + assert_eq!(r.count, 3); + assert_eq!(r.delay, None); + } + + #[test] + fn retry_with_delay() { + let out = parse_pipeline_annotations("// retry 3 5s"); + let r = out.retry.expect("retry"); + assert_eq!(r.count, 3); + assert_eq!(r.delay.as_deref(), Some("5s")); + } + + #[test] + fn retry_first_wins() { + let out = parse_pipeline_annotations("// retry 3 5s\n# retry 1"); + let r = out.retry.expect("retry"); + assert_eq!(r.count, 3); + assert_eq!(r.delay.as_deref(), Some("5s")); + } + + #[test] + fn retry_zero_is_skipped() { + let out = parse_pipeline_annotations("// retry 0 5s"); + assert!(out.retry.is_none()); + } + + #[test] + fn retry_non_numeric_count_is_skipped() { + let out = parse_pipeline_annotations("// retry many"); + assert!(out.retry.is_none()); + } + + #[test] + fn materialize_managed_default() { + let out = parse_pipeline_annotations("// materialize ducklake://analytics/orders_daily"); + let m = out.materialize.expect("materialize"); + assert_eq!(m.target_kind, AssetKind::Ducklake); + assert_eq!(m.target_path, "analytics/orders_daily"); + // managed by default; replace strategy (no append / key) + assert!(!m.manual); + assert!(!m.append); + assert_eq!(m.unique_key, None); + } + + #[test] + fn materialize_manual_escape_hatch() { + let out = + parse_pipeline_annotations("// materialize manual ducklake://analytics/orders_daily"); + let m = out.materialize.expect("materialize"); + assert!(m.manual); + assert_eq!(m.target_path, "analytics/orders_daily"); + } + + #[test] + fn materialize_merge_and_append_options() { + let out = + parse_pipeline_annotations("// materialize ducklake://a/orders_daily key=order_id"); + let m = out.materialize.expect("materialize"); + assert_eq!(m.unique_key.as_deref(), Some("order_id")); + assert!(!m.append); + + let out = parse_pipeline_annotations("// materialize ducklake://a/events append"); + let m = out.materialize.expect("materialize"); + assert!(m.append); + assert_eq!(m.unique_key, None); + } + + #[test] + fn materialize_default_syntax_shorthand() { + let out = parse_pipeline_annotations("// materialize ducklake"); + let m = out.materialize.expect("materialize"); + assert_eq!(m.target_kind, AssetKind::Ducklake); + assert_eq!(m.target_path, "main"); + assert!(!m.manual); + } + + #[test] + fn materialize_manual_only_is_dropped() { + // `manual` with no target is not a valid materialization. + let out = parse_pipeline_annotations("// materialize manual"); + assert!(out.materialize.is_none()); + } + + #[test] + fn materialize_first_wins() { + let out = parse_pipeline_annotations( + "// materialize ducklake://a/x\n# materialize manual ducklake://b/y", + ); + let m = out.materialize.expect("materialize"); + assert_eq!(m.target_path, "a/x"); + assert!(!m.manual); + } + + #[test] + fn combined() { + let code = concat!( + "// pipeline\n", + "// on schedule\n", + "// on s3://in.csv\n", + "// partitioned daily tz=\"UTC\"\n", + "// freshness 2h\n", + "// tag heavy\n", + "// retry 3 5s\n", + "// materialize ducklake://analytics/orders_daily key=order_id\n" + ); + let out = parse_pipeline_annotations(code); + assert!(out.in_pipeline); + assert_eq!(out.triggers.len(), 2); + assert!(out.partition.is_some()); + assert_eq!(out.freshness.unwrap().duration, "2h"); + assert_eq!(out.tag.as_deref(), Some("heavy")); + let r = out.retry.expect("retry"); + assert_eq!(r.count, 3); + assert_eq!(r.delay.as_deref(), Some("5s")); + let m = out.materialize.expect("materialize"); + assert!(!m.manual); + assert_eq!(m.target_path, "analytics/orders_daily"); + assert_eq!(m.unique_key.as_deref(), Some("order_id")); + } + + #[test] + fn split_trailing_kv_opts_uses_exact_token_offset() { + // Regression for the token-offset computation in + // `split_trailing_kv_opts`. The asset ref's path token contains the + // exact text of the trailing `debounce=60s` opt as a substring, and + // the same `debounce=60s` text also appears a second time before the + // real opt. Offsetting by the `&str` slice's pointer is exact; a + // substring scan (`find`) is what this guards against regressing to. + let (ref_part, opts) = split_trailing_kv_opts("s3://lake/debounce=60s/raw debounce=60s"); + assert_eq!(ref_part, "s3://lake/debounce=60s/raw"); + assert_eq!(opts.get("debounce").map(String::as_str), Some("60s")); + + // End-to-end through the annotation parser: the ref must parse to the + // full S3 path (not truncated at the embedded `debounce=`), and the + // per-edge debounce override must be picked up from the trailing opt. + let out = parse_pipeline_annotations("// on s3://lake/debounce=60s/raw debounce=90s"); + assert_eq!(out.triggers.len(), 1); + match &out.triggers[0] { + TriggerSpec::Asset { asset_kind, path, debounce } => { + assert_eq!(*asset_kind, AssetKind::S3Object); + assert_eq!(path, "lake/debounce=60s/raw"); + assert_eq!(debounce.as_deref(), Some("90s")); + } + other => panic!("expected asset trigger, got {other:?}"), + } + } + + #[test] + fn kv_opts_quoted_with_spaces() { + let m = parse_kv_opts("a=\"hello world\" b=plain c='single quoted'"); + assert_eq!(m.get("a").unwrap(), "hello world"); + assert_eq!(m.get("b").unwrap(), "plain"); + assert_eq!(m.get("c").unwrap(), "single quoted"); + } + + #[test] + fn kv_opts_malformed_recovers() { + let m = parse_kv_opts("garbage a=ok =alone b=fine"); + assert_eq!(m.get("a").unwrap(), "ok"); + assert_eq!(m.get("b").unwrap(), "fine"); + assert!(m.get("garbage").is_none()); + } + + #[test] + fn data_test_builtins() { + let code = concat!( + "// data_test unique order_id\n", + "// data_test not_null user_id\n", + "// data_test accepted_values status = paid,pending,refunded\n", + "// data_test relationships user_id -> datatable://prod/users.id\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.data_tests, + vec![ + DataTest::Unique { column: "order_id".to_string() }, + DataTest::NotNull { column: "user_id".to_string() }, + DataTest::AcceptedValues { + column: "status".to_string(), + values: vec![ + "paid".to_string(), + "pending".to_string(), + "refunded".to_string() + ], + }, + DataTest::Relationships { + column: "user_id".to_string(), + to_kind: AssetKind::DataTable, + to_path: "prod/users".to_string(), + to_column: "id".to_string(), + }, + ] + ); + } + + #[test] + fn data_test_accepts_quotes_and_spacing() { + let out = parse_pipeline_annotations("// data_test accepted_values kind = \"a b\", 'c' ,d"); + assert_eq!( + out.data_tests, + vec![DataTest::AcceptedValues { + column: "kind".to_string(), + values: vec!["a b".to_string(), "c".to_string(), "d".to_string()], + }] + ); + } + + #[test] + fn data_test_custom_escape_hatch() { + // A non-built-in single token is a custom script path; default-syntax + // asset shorthands are NOT triggered here (a path is just a path). + let out = parse_pipeline_annotations("// data_test f/tests/orders_amount_sane"); + assert_eq!( + out.data_tests, + vec![DataTest::Custom { path: "f/tests/orders_amount_sane".to_string() }] + ); + } + + #[test] + fn data_test_relationships_ducklake_shorthand() { + let out = parse_pipeline_annotations( + "// data_test relationships sku -> ducklake://warehouse/dim_products.sku", + ); + assert_eq!( + out.data_tests, + vec![DataTest::Relationships { + column: "sku".to_string(), + to_kind: AssetKind::Ducklake, + to_path: "warehouse/dim_products".to_string(), + to_column: "sku".to_string(), + }] + ); + } + + #[test] + fn data_test_malformed_dropped_fail_safe() { + // A misspelled built-in with trailing content is not a valid path token + // → dropped, not misread as a custom test. An empty value list, a + // missing arrow target, and a bare keyword are all dropped too. + let out = parse_pipeline_annotations(concat!( + "// data_test uniq order_id\n", // typo'd built-in + arg + "// data_test accepted_values s =\n", // no values + "// data_test relationships a -> b\n", // no `.refcol` + "// data_test unique\n", // missing column + "// data_test\n", // bare keyword + )); + assert!(out.data_tests.is_empty()); + } + + #[test] + fn data_test_not_confused_with_ci_test_annotation() { + // `// test:` is the unrelated CI-test annotation — it must NOT be + // parsed as a data test (no whitespace after `test`, and the keyword + // is `data_test` anyway). + let out = parse_pipeline_annotations("// test: f/foo/bar\n// data_test unique id"); + assert_eq!( + out.data_tests, + vec![DataTest::Unique { column: "id".to_string() }] + ); + } + + #[test] + fn column_lineage_basic() { + let code = concat!( + "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n", + "// column user_name <- datatable://prod/users.name\n", + ); + let out = parse_pipeline_annotations(code); + assert_eq!( + out.column_lineage, + vec![ + ColumnLineage { + column: "order_total".to_string(), + inputs: vec![ + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "amount".to_string(), + }, + ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/orders".to_string(), + from_column: "tax".to_string(), + }, + ], + }, + ColumnLineage { + column: "user_name".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/users".to_string(), + from_column: "name".to_string(), + }], + }, + ] + ); + } + + #[test] + fn column_lineage_schema_qualified_keeps_last_dot_as_column() { + // The column is the segment after the FINAL dot, so a schema-qualified + // ducklake table (`main.dim_products`) survives intact. + let out = parse_pipeline_annotations( + "// column sku <- ducklake://warehouse/main.dim_products.sku", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "sku".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "warehouse/main.dim_products".to_string(), + from_column: "sku".to_string(), + }], + }] + ); + } + + #[test] + fn column_lineage_drops_malformed_refs_keeps_valid() { + // `bad_no_dot` has no `.col` and is dropped; the line survives on its + // one valid ref. Mirrors accepted_values' drop-empties-keep-≥1 stance. + let out = parse_pipeline_annotations( + "// column total <- bad_no_dot, datatable://prod/orders.amount", + ); + assert_eq!( + out.column_lineage, + vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/orders".to_string(), + from_column: "amount".to_string(), + }], + }] + ); + } + + #[test] + fn merge_column_lineage_annotation_overrides_inferred() { + let inferred = vec![ + ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "amount".to_string(), + }], + }, + ColumnLineage { + column: "qty".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::Ducklake, + from_path: "w/o".to_string(), + from_column: "qty".to_string(), + }], + }, + ]; + // Annotation redefines `total` (wins) and leaves `qty` to inference. + let annotated = vec![ColumnLineage { + column: "total".to_string(), + inputs: vec![ColumnRef { + from_kind: AssetKind::DataTable, + from_path: "prod/x".to_string(), + from_column: "grand_total".to_string(), + }], + }]; + let merged = merge_column_lineage(inferred, annotated); + assert_eq!(merged.len(), 2); + // Annotation entry kept first and authoritative. + assert_eq!(merged[0].column, "total"); + assert_eq!(merged[0].inputs[0].from_column, "grand_total"); + // Inferred `qty` survives (no annotation for it); inferred `total` dropped. + assert_eq!(merged[1].column, "qty"); + } + + #[test] + fn column_lineage_malformed_lines_dropped_fail_safe() { + // No arrow, a multi-token output column, and a line whose every ref is + // malformed are all dropped entirely. + let out = parse_pipeline_annotations(concat!( + "// column no_arrow datatable://prod/x.y\n", // missing `<-` + "// column a b <- datatable://prod/x.y\n", // output not a single ident + "// column total <- bad_no_dot\n", // no valid ref + "// column\n", // bare keyword + )); + assert!(out.column_lineage.is_empty()); + } +} diff --git a/backend/parsers/windmill-parser/src/lib.rs b/backend/parsers/windmill-parser/src/lib.rs index 5a7e90bf7e..19bc5602cd 100644 --- a/backend/parsers/windmill-parser/src/lib.rs +++ b/backend/parsers/windmill-parser/src/lib.rs @@ -13,6 +13,7 @@ use serde::Serialize; use serde_json::Value; pub mod asset_parser; +pub mod sql_materialize; /// S3 output format for SQL queries (moved here to avoid pulling sqlx into WASM via windmill-types) #[derive(Clone, Copy, Debug)] diff --git a/backend/parsers/windmill-parser/src/sql_materialize.rs b/backend/parsers/windmill-parser/src/sql_materialize.rs new file mode 100644 index 0000000000..7a3c72bdce --- /dev/null +++ b/backend/parsers/windmill-parser/src/sql_materialize.rs @@ -0,0 +1,1429 @@ +//! Eligibility classifier + materialization SQL codegen for managed `// materialize`. +//! +//! Managed `// materialize` (the default) promises the script is "setup +//! statements, then one trailing SELECT" — Windmill generates the write DDL +//! around that SELECT (the `// materialize manual` escape hatch opts out and +//! writes its own DDL). This module is the single source of truth for *which +//! block is that SELECT* and *what DDL gets generated*, so save-time validation +//! (deploy path) and run-time codegen (DuckDB executor) can never disagree. +//! +//! Everything here is pure and string-level: no SQL is executed, no type +//! inference is done. The classifier is leading-keyword based and deliberately +//! conservative — anything it can't positively recognize as a read-only output +//! or a known-safe setup statement is rejected, so a script is only accepted +//! for managed mode when its shape is unambiguous. + +/// One top-level statement's role in a wrap-mode script. +#[derive(Debug, Clone, Copy, PartialEq, Eq)] +pub enum BlockClass { + /// Read-only relation the wrap writes from: `SELECT` / `WITH …SELECT` / + /// `FROM` (DuckDB from-first) / `VALUES` / `TABLE x` / `(UN)PIVOT`. + Output, + /// Known-safe preamble: `ATTACH` / `INSTALL` / `LOAD` / `SET` / `PRAGMA` / + /// `USE` / `CREATE TEMP …`. Runs verbatim before the generated write. + Setup, + /// Anything that writes or whose effect we can't vouch for: non-temp + /// `CREATE` / `INSERT` / `UPDATE` / `DELETE` / `MERGE` / `DROP` / `COPY` / + /// `ALTER` / `TRUNCATE`, or an unrecognized leading keyword. Disqualifies + /// managed mode (the user should use `// materialize manual`). + Disallowed, +} + +/// A script accepted for wrapping: zero+ setup blocks then one terminal SELECT. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct WrapPlan { + /// Setup statements in source order, verbatim, **without** trailing `;`. + pub setup: Vec, + /// The single terminal output statement, verbatim, **without** trailing `;`. + pub output: String, +} + +/// Why a script is not eligible for managed `// materialize`. Carries enough to +/// render the targeted save-time messages. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum WrapError { + /// No statements at all (empty / comments only). + Empty, + /// No terminal SELECT — nothing to wrap. + NoOutput, + /// More than one top-level SELECT. `count` is how many were found. + MultipleOutputs { count: usize }, + /// A SELECT exists but isn't the last statement (something runs after it). + OutputNotLast, + /// A write/unknown statement appears among the setup blocks. `snippet` is a + /// short prefix of the offending statement for the error message. + DisallowedBlock { snippet: String }, +} + +impl WrapError { + /// Human-facing, actionable message (matches the spec's rejection text). + pub fn message(&self) -> String { + let base = + "managed `// materialize` requires the script to be setup statements then a single trailing SELECT"; + let manual = "use `// materialize manual` to write the DDL yourself"; + match self { + WrapError::Empty => format!("{base}: the script is empty."), + WrapError::NoOutput => format!("{base}: found no SELECT — {manual}."), + WrapError::MultipleOutputs { count } => format!( + "{base}: found {count} SELECT statements; combine them with a CTE, or {manual}." + ), + WrapError::OutputNotLast => format!( + "{base}: found statements after the SELECT — move them above it, or {manual}." + ), + WrapError::DisallowedBlock { snippet } => { + format!("{base}: `{snippet}` writes or is unrecognized — {manual}.") + } + } + } +} + +/// Split SQL into top-level, `;`-separated statements, skipping line comments +/// (`-- …`), block comments (`/* … */`), single-quoted strings (`'…'` with +/// `''` escape) and double-quoted identifiers (`"…"`). Semicolons inside any of +/// those are not separators. Returns each statement trimmed, comments stripped, +/// empties dropped. Self-contained so the parser crate stays dependency-free; +/// it must stay behaviourally aligned with the executor's block splitter (both +/// route wrap through `classify_wrap`, so the split they see is this one). +pub fn split_statements(sql: &str) -> Vec { + let mut out = Vec::new(); + let mut cur = String::new(); + let bytes = sql.as_bytes(); + let mut i = 0; + let n = bytes.len(); + while i < n { + let c = bytes[i] as char; + // line comment — `--` (SQL) or `//`. The `//` form is not SQL, but it + // is how Windmill pipeline annotations (`// materialize`, `// pipeline`, + // …) are written, and they sit above the SQL in the same script; strip + // them so they don't pollute the first statement block's classification + // or the generated setup SQL. + if (c == '-' && i + 1 < n && bytes[i + 1] == b'-') + || (c == '/' && i + 1 < n && bytes[i + 1] == b'/') + { + while i < n && bytes[i] != b'\n' { + i += 1; + } + continue; + } + // block comment + if c == '/' && i + 1 < n && bytes[i + 1] == b'*' { + i += 2; + while i + 1 < n && !(bytes[i] == b'*' && bytes[i + 1] == b'/') { + i += 1; + } + i += 2; + continue; + } + // single-quoted string + if c == '\'' { + cur.push(c); + i += 1; + while i < n { + cur.push(bytes[i] as char); + if bytes[i] == b'\'' { + // doubled '' is an escaped quote, stay in string + if i + 1 < n && bytes[i + 1] == b'\'' { + cur.push('\''); + i += 2; + continue; + } + i += 1; + break; + } + i += 1; + } + continue; + } + // double-quoted identifier + if c == '"' { + cur.push(c); + i += 1; + while i < n { + cur.push(bytes[i] as char); + if bytes[i] == b'"' { + i += 1; + break; + } + i += 1; + } + continue; + } + if c == ';' { + let t = cur.trim(); + if !t.is_empty() { + out.push(t.to_string()); + } + cur.clear(); + i += 1; + continue; + } + cur.push(c); + i += 1; + } + let t = cur.trim(); + if !t.is_empty() { + out.push(t.to_string()); + } + out +} + +/// Lowercased top-level keyword tokens of a single statement (parens collapsed +/// away: tokens *inside* balanced `(...)` are skipped, so a CTE body's verbs +/// don't leak up). Strings/identifiers are already gone from the split, but we +/// re-guard quotes defensively. Used to disambiguate `WITH …` and `CREATE …`. +fn top_level_keywords(stmt: &str) -> Vec { + let mut toks = Vec::new(); + let mut cur = String::new(); + let mut depth: i32 = 0; + let bytes = stmt.as_bytes(); + let mut i = 0; + let n = bytes.len(); + let flush = |cur: &mut String, toks: &mut Vec| { + if !cur.is_empty() { + toks.push(cur.to_lowercase()); + cur.clear(); + } + }; + while i < n { + let c = bytes[i] as char; + if c == '\'' || c == '"' { + let q = bytes[i]; + i += 1; + while i < n && bytes[i] != q { + i += 1; + } + i += 1; + continue; + } + if c == '(' { + flush(&mut cur, &mut toks); + depth += 1; + i += 1; + continue; + } + if c == ')' { + if depth > 0 { + depth -= 1; + } + i += 1; + continue; + } + if depth > 0 { + i += 1; + continue; + } + if c.is_alphanumeric() || c == '_' { + cur.push(c); + } else { + flush(&mut cur, &mut toks); + } + i += 1; + } + flush(&mut cur, &mut toks); + toks +} + +const OUTPUT_KW: &[&str] = &["select", "from", "values", "table", "pivot", "unpivot"]; +const SETUP_KW: &[&str] = &["attach", "install", "load", "set", "pragma", "use"]; +const WRITE_VERBS: &[&str] = &["insert", "update", "delete", "merge"]; + +/// Classify a single statement by its leading keyword (with `WITH`/`CREATE` +/// disambiguation). See [`BlockClass`]. +pub fn classify_block(stmt: &str) -> BlockClass { + let kws = top_level_keywords(stmt); + let Some(first) = kws.first().map(String::as_str) else { + return BlockClass::Disallowed; + }; + + // CREATE TEMP … is setup (staging); any other CREATE is a write. + if first == "create" { + let temp = kws + .iter() + .skip(1) + .take(3) + .any(|k| k == "temp" || k == "temporary"); + return if temp { + BlockClass::Setup + } else { + BlockClass::Disallowed + }; + } + + // WITH … : the main statement's verb decides. CTE bodies are parenthesized, + // so their verbs are not in `kws`; the first top-level write verb or SELECT + // after the CTE list is the real one. + if first == "with" { + for k in kws.iter().skip(1) { + if k == "select" { + return BlockClass::Output; + } + if WRITE_VERBS.contains(&k.as_str()) { + return BlockClass::Disallowed; + } + } + // `WITH x AS (...) SELECT` where SELECT got collapsed is impossible + // (SELECT here is top-level), so a WITH with no top-level verb is a + // malformed/unknown statement — reject conservatively. + return BlockClass::Disallowed; + } + + if OUTPUT_KW.contains(&first) { + return BlockClass::Output; + } + if SETUP_KW.contains(&first) { + return BlockClass::Setup; + } + BlockClass::Disallowed +} + +/// Validate a script for managed `// materialize` and, on success, return the +/// setup/output split. Enforces the four conditions from the spec: +/// 1. exactly one Output block, 2. it is last, 3. all preceding blocks are +/// Setup, 4. nothing after it. +pub fn classify_wrap(sql: &str) -> Result { + let stmts = split_statements(sql); + if stmts.is_empty() { + return Err(WrapError::Empty); + } + let classes: Vec = stmts.iter().map(|s| classify_block(s)).collect(); + + let output_idxs: Vec = classes + .iter() + .enumerate() + .filter(|(_, c)| **c == BlockClass::Output) + .map(|(i, _)| i) + .collect(); + + match output_idxs.len() { + 0 => return Err(WrapError::NoOutput), + 1 => {} + count => return Err(WrapError::MultipleOutputs { count }), + } + let out_idx = output_idxs[0]; + if out_idx != stmts.len() - 1 { + return Err(WrapError::OutputNotLast); + } + // Everything before the output must be Setup (no Disallowed preamble). + for (i, c) in classes.iter().enumerate().take(out_idx) { + if *c != BlockClass::Setup { + return Err(WrapError::DisallowedBlock { snippet: snippet(&stmts[i]) }); + } + } + Ok(WrapPlan { setup: stmts[..out_idx].to_vec(), output: stmts[out_idx].clone() }) +} + +fn snippet(stmt: &str) -> String { + let one_line: String = stmt.split_whitespace().collect::>().join(" "); + if one_line.chars().count() > 40 { + let truncated: String = one_line.chars().take(40).collect(); + format!("{truncated}…") + } else { + one_line + } +} + +// --------------------------------------------------------------------------- +// Codegen +// --------------------------------------------------------------------------- + +/// How a (partition of a) materialized table is reconciled on each run. +/// Derived at deploy from `unique_key`/`append`: `append` → `Append`, else +/// `unique_key` → `Merge`, else `Replace`. +#[derive(Debug, Clone, PartialEq, Eq)] +pub enum MaterializeStrategy { + /// DELETE the current partition, then INSERT — partition becomes exactly + /// what the SELECT returned. Full-refresh of the slice. + Replace, + /// Upsert within the slice on `unique_key` (delete-by-key + insert); rows + /// absent from the SELECT are left in place. + Merge { unique_key: String }, + /// INSERT only — immutable event-log semantics. + Append, +} + +/// Inputs to materialization codegen, all resolved at run time by the worker. +/// Pure: produces SQL text; executes nothing. +#[derive(Debug, Clone)] +pub struct MaterializeCodegen<'a> { + /// Fully-qualified target, e.g. `_wm_target.orders_daily`. Always qualified + /// so a user `USE …;` in setup can't redirect the write. + pub target_qualified: &'a str, + /// The user's output SELECT (verbatim, no trailing `;`) — embedded as a + /// subquery so its own shape is irrelevant to the generated wrapper. + pub select_sql: &'a str, + /// Physical partition column added to the managed table. + pub partition_col: &'a str, + /// SQL expression for the current partition value — a literal like + /// `'2026-06-19'` or a bind placeholder. The caller is responsible for + /// safe quoting/binding. + pub partition_value_sql: &'a str, + /// Whether `// partitioned` applies. When false the table is unpartitioned + /// and the partition column / `SET PARTITIONED BY` are omitted. + pub partitioned: bool, + pub strategy: MaterializeStrategy, +} + +impl<'a> MaterializeCodegen<'a> { + /// The ordered statements that perform the materialization, to be run after + /// the setup blocks and inside the caller's execution. The first-run + /// bootstrap is idempotent (`IF NOT EXISTS`), so this is safe to run every + /// time. The DELETE/INSERT body is wrapped in one transaction so a partial + /// failure leaves the prior snapshot intact. Every strategy reduces to + /// DELETE+INSERT (no `MERGE INTO`) — see the `Merge` arm for why. + pub fn statements(&self) -> Vec { + let t = self.target_qualified; + let sel = self.select_sql; + let pcol = self.partition_col; + let pval = self.partition_value_sql; + let mut out = Vec::new(); + + // Whole-table replace: rebuild the table to match the SELECT's *current* + // schema each run with one atomic `CREATE OR REPLACE` (which DuckLake + // still snapshots). This is the only path that survives a changed SELECT + // or a pre-existing table with a different schema — the persist-and- + // mutate paths below fix the schema at first create. + if !self.partitioned && matches!(self.strategy, MaterializeStrategy::Replace) { + out.push(format!( + "CREATE OR REPLACE TABLE {t} AS SELECT * FROM ({sel});" + )); + return out; + } + + // Persist-and-mutate (partitioned, or merge/append): bootstrap the table + // if absent, then write into it. The schema is fixed at first create — + // a later SELECT-schema change needs a manual rebuild (schema evolution + // is a follow-up). + if self.partitioned { + out.push(format!( + "CREATE TABLE IF NOT EXISTS {t} AS \ + SELECT *, CAST(NULL AS VARCHAR) AS {pcol} FROM ({sel}) WHERE false;" + )); + out.push(format!("ALTER TABLE {t} SET PARTITIONED BY ({pcol});")); + } else { + out.push(format!( + "CREATE TABLE IF NOT EXISTS {t} AS SELECT * FROM ({sel}) WHERE false;" + )); + } + + out.push("BEGIN TRANSACTION;".to_string()); + // The rows to write, with the partition column appended when partitioned. + let source = if self.partitioned { + format!("SELECT *, {pval} AS {pcol} FROM ({sel})") + } else { + format!("SELECT * FROM ({sel})") + }; + match &self.strategy { + MaterializeStrategy::Replace => { + // Only reached when partitioned (whole-table replace returned above). + out.push(format!("DELETE FROM {t} WHERE {pcol} = {pval};")); + out.push(format!("INSERT INTO {t} {source};")); + } + MaterializeStrategy::Append => { + out.push(format!("INSERT INTO {t} {source};")); + } + MaterializeStrategy::Merge { unique_key } => { + // Upsert within the slice via delete-by-key + insert (dbt's + // `delete+insert`): rows whose key is in the incoming SELECT are + // replaced, others are left in place. This deliberately avoids + // `MERGE INTO` — DuckLake's MERGE fails writing the first rows of + // a fresh partition (HTTP 404 on the new parquet), and a failed + // write leaves the table needing a DROP. DELETE+INSERT is the + // same write shape as `replace`, which is reliable. The DELETE is + // scoped to the current partition when partitioned so it stays + // slice-local (a key present in another partition is untouched). + let scope = if self.partitioned { + format!("{pcol} = {pval} AND ") + } else { + String::new() + }; + out.push(format!( + "DELETE FROM {t} WHERE {scope}{unique_key} IN (SELECT {unique_key} FROM ({sel}));" + )); + out.push(format!("INSERT INTO {t} {source};")); + } + } + out.push("COMMIT;".to_string()); + out + } +} + +/// The read that captures the DuckLake snapshot id produced by the write, for +/// the given attach alias (e.g. `_wm_target`). The worker runs this last and +/// records the result into `materialized_partition`. +pub fn snapshot_capture_sql(alias: &str) -> String { + format!("SELECT max(snapshot_id) AS snapshot_id FROM ducklake_snapshots('{alias}');") +} + +/// Reserved attach alias for the materialization target, fully-qualified in all +/// generated SQL so a user `USE …;` in the setup blocks can't redirect the +/// write. The worker resolves the real `ATTACH 'ducklake:…' AS _wm_target (…)` +/// from the target ducklake's config and passes it in as `target_attach`. +pub const TARGET_ALIAS: &str = "_wm_target"; + +/// Assemble the full ordered statement list the DuckDB executor runs for a +/// managed `// materialize` script. This is the single entry point the worker +/// calls; it composes the already-tested pieces (classifier split → target +/// ATTACH → strategy codegen → snapshot capture) so their ordering lives in one +/// tested place rather than inline in the executor. +/// +/// `target_attach` is the real `ATTACH 'ducklake:…' AS _wm_target (…);` string +/// the worker built from config (it depends on resolved credentials, so it +/// can't be generated here). `target_table` is the table within that catalog +/// (e.g. `orders_daily`), referenced as `_wm_target.
`. `asset_path` is +/// the full `/
` for the result summary. The trailing statement is +/// a one-row summary read (asset / rows / snapshot_id) that is both the job's +/// result (a useful preview) and what the worker records. +pub fn build_wrap_blocks( + plan: &WrapPlan, + target_attach: &str, + target_table: &str, + asset_path: &str, + partition_col: &str, + partition_value_sql: &str, + partitioned: bool, + strategy: MaterializeStrategy, + tests: &[DataTestResolved], +) -> Result, String> { + let target_qualified = format!("{TARGET_ALIAS}.{target_table}"); + let cg = MaterializeCodegen { + target_qualified: &target_qualified, + select_sql: &plan.output, + partition_col, + partition_value_sql, + partitioned, + strategy, + }; + let ctx = DataTestCtx { + target_qualified: &target_qualified, + asset_path, + partition_col, + partition_value_sql, + partitioned, + }; + let test_sql = build_data_test_checks(tests, &ctx)?; + let mut blocks: Vec = Vec::new(); + // Setup blocks come from the splitter with their `;` stripped — re-terminate + // each so that when the executor re-joins and re-splits the assembled query, + // adjacent statements (e.g. the user ATTACH and the synthetic target ATTACH) + // don't merge into one malformed statement. + blocks.extend(plan.setup.iter().map(|s| terminate(s))); + blocks.push(target_attach.to_string()); + // Referenced-asset ATTACHes (relationships tests) — read-only, before the + // write and the summary that probes them. + blocks.extend(test_sql.attaches); + blocks.extend(cg.statements()); + // The summary read carries the per-test breakdown (when any tests apply). + blocks.push(materialize_result_sql( + &target_qualified, + asset_path, + partition_col, + partition_value_sql, + partitioned, + &test_sql.checks, + )); + Ok(blocks) +} + +/// The trailing one-row summary the materialize run returns: the asset it +/// produced, the row count of the materialized slice (the partition when +/// partitioned, else the whole table), and the DuckLake snapshot it created. +/// This is both a useful preview result and the row the worker records. +pub fn materialize_result_sql( + target_qualified: &str, + asset_path: &str, + partition_col: &str, + partition_value_sql: &str, + partitioned: bool, + checks: &[DataTestCheck], +) -> String { + let (count_expr, partition_sel) = if partitioned { + // Row count is the slice this run wrote (the partition); `partition` + // lets the UI label the count and scope the preview to it. + ( + format!( + "(SELECT count(*) FROM {target_qualified} WHERE {partition_col} = {partition_value_sql})" + ), + format!("{partition_value_sql} AS partition, "), + ) + } else { + ( + format!("(SELECT count(*) FROM {target_qualified})"), + String::new(), + ) + }; + // Capture the materialized output schema (gap #2a) in the same summary row — + // no extra round-trip. `DESCRIBE SELECT * FROM ` yields one row per + // column (`column_name`, `column_type`); fold them into a list-of-struct the + // worker reads back and persists as asset metadata. The write just + // committed, so the latest snapshot (no `AT (VERSION)` needed) is exactly the + // slice recorded in `snapshot_id`. + // + // Two correctness details: + // - `_wm_ord` (a `row_number()` over the DESCRIBE) is captured so the + // list-of-struct is ordered *explicitly* (`list(... ORDER BY _wm_ord)`). + // DESCRIBE returns columns in physical order; without the explicit ORDER + // the `list()` aggregate could reorder them and spuriously bump the schema + // version on a re-materialize. + // - For a `// partitioned` asset the physical table carries the synthetic + // `_wm_partition` column; it must be filtered out so the recorded schema is + // the producer's logical output, not Windmill's storage detail (this is the + // grain #2b contract enforcement reads back). + let partition_filter = if partitioned { + format!( + " WHERE column_name <> '{}'", + partition_col.replace('\'', "''") + ) + } else { + String::new() + }; + let schema_capture = format!( + "(SELECT list({{'name': column_name, 'type': column_type}} ORDER BY _wm_ord) \ + FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ + FROM (DESCRIBE SELECT * FROM {target_qualified}){partition_filter})) AS output_schema" + ); + let base_cols = format!( + "'ducklake://{asset_path}' AS materialized, \ + {partition_sel}{count_expr} AS rows, \ + (SELECT max(snapshot_id) FROM ducklake_snapshots('{TARGET_ALIAS}')) AS snapshot_id, \ + {schema_capture}" + ); + if checks.is_empty() { + return format!("SELECT {base_cols};"); + } + // Per-test breakdown. Each check's violating-count is computed once as a CTE + // column (`c0`, `c1`, …); the `data_tests` list-of-struct then references + // those columns — DuckDB rejects scalar subqueries *inside* a struct/list + // literal, hence the CTE. Names are single-quote-escaped. The result row + // carries the whole breakdown so the worker runs every test (no + // abort-on-first) and decides pass/fail itself. + let cte_cols = checks + .iter() + .enumerate() + .map(|(i, c)| format!("{} AS c{i}", c.violating)) + .collect::>() + .join(", "); + let list_items = checks + .iter() + .enumerate() + .map(|(i, c)| { + let name = c.name.replace('\'', "''"); + format!("{{'test': '{name}', 'violating': c{i}}}") + }) + .collect::>() + .join(", "); + format!( + "WITH _wm_tr AS (SELECT {cte_cols}) \ + SELECT {base_cols}, [{list_items}] AS data_tests FROM _wm_tr;" + ) +} + +// Ensure a statement ends with a single `;`. +fn terminate(stmt: &str) -> String { + let t = stmt.trim_end(); + if t.ends_with(';') { + t.to_string() + } else { + format!("{t};") + } +} + +// --------------------------------------------------------------------------- +// Data tests (`// data_test`) +// --------------------------------------------------------------------------- +// +// A data test is the FIRST extensible annotation: the parser yields a +// `DataTest` from a known vocabulary, and this module turns each into a +// *check* — a `(name, violating-row-count query)` pair — that runs against the +// freshly-materialized target after the write commits. The materialize summary +// query embeds every check's count in one `data_tests` column, so all tests +// run in a single pass (no abort-on-first) and the worker, not the SQL, +// decides pass/fail and reports the full per-test breakdown. +// +// The pattern is deliberately open: a verifier is just `(name, count query)`. +// Built-ins differ only in their count query; the `Custom` escape hatch +// supplies its own (a user SELECT returning the violating rows). A sibling +// annotation family (column-lineage) can emit its own checks through the same +// `push_check` shape rather than bolting on a parallel mechanism. See +// `docs/ducklake-materialization.md`. + +use crate::asset_parser::{AssetKind, DataTest}; + +/// Target context a data-test probe runs against — the materialized table and +/// the partition slice (when partitioned, tests are scoped to the slice just +/// written, so a rerun/backfill is independent of other partitions' data). +#[derive(Debug, Clone)] +pub struct DataTestCtx<'a> { + /// Fully-qualified materialized target, e.g. `_wm_target.orders`. + pub target_qualified: &'a str, + /// `/
` of the target, for human-readable probe messages. + pub asset_path: &'a str, + /// Physical partition column on the managed table. + pub partition_col: &'a str, + /// SQL literal/expression for the current partition value (already escaped). + pub partition_value_sql: &'a str, + /// Whether the target is partitioned (scopes probes to the slice). + pub partitioned: bool, +} + +/// A data test resolved enough to generate SQL. Built-ins carry only their +/// parsed `DataTest`; `Custom` additionally carries the fetched script body +/// (the parser crate can't fetch it — the worker does and passes it in). +#[derive(Debug, Clone)] +pub enum DataTestResolved { + BuiltIn(DataTest), + Custom { path: String, body: String }, +} + +/// One compiled data-test check: a human-readable `name` and a scalar SQL +/// expression (`violating`) yielding the number of rows that violate it (0 = +/// pass). The materialize summary query embeds every check's count so the +/// worker gets the whole breakdown in one result — all tests run (no +/// abort-on-first) and the worker, not the SQL, decides pass/fail. +#[derive(Debug, Clone, PartialEq, Eq)] +pub struct DataTestCheck { + pub name: String, + /// Scalar subquery yielding the violating-row count, e.g. + /// `(SELECT count(*) AS v FROM (…))`. + pub violating: String, +} + +/// The SQL a set of data tests compiles to: referenced-asset `ATTACH` +/// statements (resolved by the executor's ATTACH-transform pass) and the +/// per-test checks, both in declaration order. +#[derive(Debug, Clone, Default, PartialEq, Eq)] +pub struct DataTestChecks { + pub attaches: Vec, + pub checks: Vec, +} + +/// Alias prefix for a relationships test's referenced asset, attached +/// read-only alongside the target. `_wm_ref_` so it never collides with the +/// user's aliases or the reserved `_wm_target`. +const REF_ALIAS_PREFIX: &str = "_wm_ref_"; + +// Double-quote a SQL identifier, escaping embedded quotes — so an arbitrary +// column name from an annotation can't break out of the identifier. +fn quote_ident(id: &str) -> String { + format!("\"{}\"", id.replace('"', "\"\"")) +} + +// Quote a possibly schema-qualified table reference (`schema.table`) by quoting +// each dotted segment independently: `main.dim_products` → `"main"."dim_products"`. +// Quoting the whole thing would make DuckDB read it as one table name containing +// a literal dot, querying the wrong table. +fn quote_qualified(name: &str) -> String { + name.split('.') + .map(quote_ident) + .collect::>() + .join(".") +} + +// Single-quote a SQL string literal, escaping embedded quotes. +fn quote_lit(s: &str) -> String { + format!("'{}'", s.replace('\'', "''")) +} + +// The `WHERE`/`AND` fragment scoping a probe to the current partition, or +// empty when unpartitioned. `prefix` is `WHERE ` or `AND ` per call site. +fn partition_scope(ctx: &DataTestCtx, prefix: &str, table_alias: Option<&str>) -> String { + if !ctx.partitioned { + return String::new(); + } + let col = match table_alias { + Some(a) => format!("{a}.{}", quote_ident(ctx.partition_col)), + None => quote_ident(ctx.partition_col), + }; + format!("{prefix}{col} = {}", ctx.partition_value_sql) +} + +// Record one check: its display `name` plus `count_query` (which yields a +// single-column violating-row count) wrapped as a scalar subquery. +fn push_check(out: &mut DataTestChecks, name: String, count_query: String) { + out.checks + .push(DataTestCheck { name, violating: format!("({count_query})") }); +} + +/// Compile resolved data tests into ATTACH statements + per-test checks for +/// `ctx`'s target. Pure: returns SQL text, executes nothing. Errors carry an +/// actionable message (e.g. a relationships target that isn't an attachable +/// table). +pub fn build_data_test_checks( + tests: &[DataTestResolved], + ctx: &DataTestCtx, +) -> Result { + let t = ctx.target_qualified; + let mut out = DataTestChecks::default(); + // Dedup ref attaches by (kind, name): a database can't be attached twice, + // so multiple relationships into the same db share one alias. + let mut ref_aliases: Vec<(AssetKind, String, String)> = Vec::new(); + + for resolved in tests { + match resolved { + DataTestResolved::BuiltIn(DataTest::Unique { column }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let q = format!( + "SELECT count(*) AS v FROM (SELECT {c} FROM {t} WHERE {c} IS NOT NULL{scope} \ + GROUP BY {c} HAVING count(*) > 1)" + ); + push_check(&mut out, format!("unique({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::NotNull { column }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let q = format!("SELECT count(*) AS v FROM {t} WHERE {c} IS NULL{scope}"); + push_check(&mut out, format!("not_null({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::AcceptedValues { column, values }) => { + let c = quote_ident(column); + let scope = partition_scope(ctx, " AND ", None); + let list = values + .iter() + .map(|v| quote_lit(v)) + .collect::>() + .join(", "); + let q = format!( + "SELECT count(*) AS v FROM {t} WHERE {c} IS NOT NULL AND {c} NOT IN ({list}){scope}" + ); + push_check(&mut out, format!("accepted_values({column})"), q); + } + DataTestResolved::BuiltIn(DataTest::Relationships { + column, + to_kind, + to_path, + to_column, + }) => { + let (ref_name, ref_table) = to_path.split_once('/').ok_or_else(|| { + format!("data_test relationships: target `{to_path}` must be `/
`") + })?; + if ref_table.is_empty() { + return Err(format!( + "data_test relationships: target `{to_path}` has no table" + )); + } + let scheme = match to_kind { + AssetKind::Ducklake => "ducklake", + AssetKind::DataTable => "datatable", + other => { + return Err(format!( + "data_test relationships: target kind {other:?} is not an attachable \ + table (use ducklake:// or datatable://)" + )) + } + }; + // The materialize target's ducklake is already attached as + // `_wm_target`; a reference into that same lake must reuse it + // rather than ATTACH the same database again under a fresh alias + // (DuckDB forbids attaching one database twice). `asset_path` is + // the target's `/
`, so its lake is the part before + // the first `/`. + let target_lake = ctx.asset_path.split('/').next().unwrap_or(""); + let alias = if *to_kind == AssetKind::Ducklake && ref_name == target_lake { + TARGET_ALIAS.to_string() + } else { + // Reuse an existing alias for the same (kind, name), else mint one. + match ref_aliases + .iter() + .find(|(k, n, _)| k == to_kind && n == ref_name) + { + Some((_, _, a)) => a.clone(), + None => { + let a = format!("{REF_ALIAS_PREFIX}{}", ref_aliases.len()); + // Escape the name — it is interpolated into a + // single-quoted DuckDB literal (defense-in-depth: the + // name is deploy-time annotation content, but the parser + // places no character restriction on asset paths). + let esc_name = ref_name.replace('\'', "''"); + out.attaches + .push(format!("ATTACH '{scheme}://{esc_name}' AS {a};")); + ref_aliases.push((*to_kind, ref_name.to_string(), a.clone())); + a + } + } + }; + let c = quote_ident(column); + let rc = quote_ident(to_column); + // `ref_table` may be schema-qualified (`schema.table`); quote each + // segment so the dot stays a schema separator, not a literal. + let rt = quote_qualified(ref_table); + let scope = partition_scope(ctx, " AND ", Some("_wm_src")); + let q = format!( + "SELECT count(*) AS v FROM {t} _wm_src \ + WHERE _wm_src.{c} IS NOT NULL{scope} \ + AND NOT EXISTS (SELECT 1 FROM {alias}.{rt} _wm_ref \ + WHERE _wm_ref.{rc} = _wm_src.{c})" + ); + push_check( + &mut out, + format!("relationships({column} -> {to_path}.{to_column})"), + q, + ); + } + // A parsed Custom must be resolved (body fetched) before codegen. + DataTestResolved::BuiltIn(DataTest::Custom { path }) => { + return Err(format!( + "data_test custom `{path}`: body not resolved before codegen (internal)" + )); + } + DataTestResolved::Custom { path, body } => { + // dbt singular-test convention: the body is a *single* SELECT + // (or CTE) returning the violating rows. It is embedded as a + // subquery (`FROM ()`), so a multi-statement body would + // produce invalid SQL — validate up front with an actionable + // error. It runs in the target's connection (can read + // `_wm_target` + the user's attaches); partition substitution is + // already applied by the worker. + let stmts = split_statements(body); + if stmts.is_empty() { + return Err(format!("data_test custom `{path}`: empty test body")); + } + if stmts.len() > 1 { + return Err(format!( + "data_test custom `{path}`: must be a single SELECT returning the \ + violating rows (found {} statements)", + stmts.len() + )); + } + let q = format!("SELECT count(*) AS v FROM ({})", stmts[0]); + push_check(&mut out, format!("custom({path})"), q); + } + } + } + Ok(out) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn ok(sql: &str) -> WrapPlan { + classify_wrap(sql).expect("expected wrap-eligible") + } + fn err(sql: &str) -> WrapError { + classify_wrap(sql).expect_err("expected wrap-ineligible") + } + + #[test] + fn split_respects_strings_comments_idents() { + let sql = "SET x=1; -- a; comment\nSELECT ';' AS a, \"weird;col\" /* ; */ FROM t;"; + let s = split_statements(sql); + assert_eq!(s.len(), 2); + assert_eq!(s[0], "SET x=1"); + assert!(s[1].starts_with("SELECT")); + assert!(s[1].contains("\"weird;col\"")); + } + + #[test] + fn split_handles_escaped_quote() { + let s = split_statements("SELECT 'it''s; fine' AS a;"); + assert_eq!(s.len(), 1); + assert!(s[0].contains("it''s; fine")); + } + + #[test] + fn pipeline_annotations_are_stripped() { + // The real shape: `//` annotation lines above the SQL must not pollute + // the first block's classification (regression — they were being read + // as a leading `pipeline` keyword and rejected). + let p = ok("// pipeline\n// materialize ducklake://main/t\n// partitioned daily\nATTACH 'ducklake://main' AS dl;\nSELECT 1 AS id"); + assert_eq!(p.setup.len(), 1); + // The annotation lines are gone — the setup block starts at the real + // SQL (the `//` inside `ducklake://main` is legitimately retained). + assert!(p.setup[0].starts_with("ATTACH")); + assert!(p.output.starts_with("SELECT")); + } + + #[test] + fn bare_select_is_eligible() { + let p = ok("SELECT a, b FROM t WHERE c = '{partition}'"); + assert!(p.setup.is_empty()); + assert!(p.output.starts_with("SELECT")); + } + + #[test] + fn setup_then_select_is_eligible() { + let p = ok( + "ATTACH 'ducklake://main' AS dl;\n SET memory_limit='4GB';\n SELECT * FROM dl.orders", + ); + assert_eq!(p.setup.len(), 2); + assert!(p.output.starts_with("SELECT")); + } + + #[test] + fn create_temp_staging_is_setup() { + let p = ok("CREATE TEMP TABLE s AS SELECT 1; SELECT * FROM s"); + assert_eq!(p.setup.len(), 1); + assert_eq!( + classify_block("CREATE TEMP TABLE s AS SELECT 1"), + BlockClass::Setup + ); + assert_eq!( + classify_block("CREATE OR REPLACE TEMPORARY VIEW v AS SELECT 1"), + BlockClass::Setup + ); + } + + #[test] + fn with_cte_select_is_output_write_is_disallowed() { + assert_eq!( + classify_block("WITH x AS (SELECT 1) SELECT * FROM x"), + BlockClass::Output + ); + // CTE whose main statement inserts is a write, even though it starts WITH. + assert_eq!( + classify_block("WITH x AS (SELECT 1) INSERT INTO t SELECT * FROM x"), + BlockClass::Disallowed + ); + } + + #[test] + fn from_first_and_values_are_output() { + assert_eq!(classify_block("FROM t SELECT a"), BlockClass::Output); + assert_eq!(classify_block("VALUES (1),(2)"), BlockClass::Output); + assert_eq!(classify_block("TABLE t"), BlockClass::Output); + } + + #[test] + fn trailing_write_rejected() { + assert_eq!( + err("SELECT * FROM t; INSERT INTO u VALUES (1)"), + WrapError::OutputNotLast + ); + } + + #[test] + fn write_in_preamble_rejected() { + match err("INSERT INTO t VALUES (1); SELECT * FROM t") { + WrapError::DisallowedBlock { snippet } => assert!(snippet.starts_with("INSERT")), + e => panic!("wrong error: {e:?}"), + } + } + + #[test] + fn multiple_selects_rejected() { + assert_eq!( + err("SELECT 1; SELECT 2"), + WrapError::MultipleOutputs { count: 2 } + ); + } + + #[test] + fn no_select_and_empty_rejected() { + assert_eq!(err("CREATE TABLE t (a INT)"), WrapError::NoOutput); + assert_eq!(err(" -- just a comment\n"), WrapError::Empty); + } + + #[test] + fn use_cannot_redirect_is_classified_setup() { + // `USE` is allowed setup; generated SQL is fully qualified regardless. + assert_eq!(classify_block("USE dl"), BlockClass::Setup); + } + + #[test] + fn codegen_replace_partitioned() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.orders_daily", + select_sql: "SELECT a FROM dl.orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + strategy: MaterializeStrategy::Replace, + }; + let st = cg.statements(); + assert!(st[0].contains("CREATE TABLE IF NOT EXISTS _wm_target.orders_daily")); + assert!(st[0].contains("CAST(NULL AS VARCHAR) AS _wm_partition")); + assert!(st.iter().any( + |s| s == "ALTER TABLE _wm_target.orders_daily SET PARTITIONED BY (_wm_partition);" + )); + assert!(st.iter().any(|s| s.starts_with( + "DELETE FROM _wm_target.orders_daily WHERE _wm_partition = '2026-06-19'" + ))); + assert!(st.iter().any(|s| s.contains( + "INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19' AS _wm_partition" + ))); + assert_eq!(st.first().map(|_| &st[st.len() - 1]).unwrap(), "COMMIT;"); + } + + #[test] + fn codegen_merge_is_delete_by_key_plus_insert() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.orders_daily", + select_sql: "SELECT order_id, amount FROM dl.orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + strategy: MaterializeStrategy::Merge { unique_key: "order_id".to_string() }, + }; + let st = cg.statements(); + // upsert = delete-by-key (partition-scoped) + insert — NO `MERGE INTO` + // (DuckLake's MERGE fails on fresh partitions). + assert!(!st.iter().any(|s| s.contains("MERGE INTO"))); + let del = st + .iter() + .find(|s| s.starts_with("DELETE FROM")) + .expect("delete stmt"); + assert!(del.contains( + "WHERE _wm_partition = '2026-06-19' AND order_id IN (SELECT order_id FROM (SELECT order_id, amount FROM dl.orders))" + )); + assert!(st + .iter() + .any(|s| s.starts_with("INSERT INTO _wm_target.orders_daily SELECT *, '2026-06-19'"))); + } + + #[test] + fn codegen_append_inserts_only() { + let cg = MaterializeCodegen { + target_qualified: "_wm_target.events", + select_sql: "SELECT * FROM dl.raw", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + strategy: MaterializeStrategy::Append, + }; + let st = cg.statements(); + assert!(st + .iter() + .any(|s| s.starts_with("INSERT INTO _wm_target.events"))); + assert!(!st.iter().any(|s| s.starts_with("DELETE"))); + assert!(!st.iter().any(|s| s.starts_with("MERGE"))); + } + + #[test] + fn codegen_whole_table_replace_is_create_or_replace() { + // Unpartitioned replace must use CREATE OR REPLACE so a changed SELECT + // schema (or a pre-existing table with a different schema) doesn't break + // — and nothing else (no bootstrap / DELETE / INSERT / txn). + let cg = MaterializeCodegen { + target_qualified: "_wm_target.customer_dim", + select_sql: "SELECT a, b, c FROM dl.src", + partition_col: "_wm_partition", + partition_value_sql: "''", + partitioned: false, + strategy: MaterializeStrategy::Replace, + }; + let st = cg.statements(); + assert_eq!( + st, + vec![ + "CREATE OR REPLACE TABLE _wm_target.customer_dim AS SELECT * FROM (SELECT a, b, c FROM dl.src);" + .to_string() + ] + ); + } + + #[test] + fn snapshot_capture_targets_alias() { + assert_eq!( + snapshot_capture_sql("_wm_target"), + "SELECT max(snapshot_id) AS snapshot_id FROM ducklake_snapshots('_wm_target');" + ); + } + + #[test] + fn build_wrap_blocks_orders_setup_attach_codegen_snapshot() { + let plan = ok("ATTACH 'ducklake://main' AS dl;\n SELECT a FROM dl.orders WHERE d = '{p}'"); + let blocks = build_wrap_blocks( + &plan, + "ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');", + "orders_daily", + "main/orders_daily", + "_wm_partition", + "'2026-06-19'", + true, + MaterializeStrategy::Replace, + &[], + ) + .unwrap(); + // setup block first, then the target ATTACH, then codegen, then result. + assert!(blocks[0].starts_with("ATTACH 'ducklake://main' AS dl")); + // every setup block must be `;`-terminated so re-splitting can't merge it + // with the synthetic target ATTACH that follows. + assert!(blocks[0].ends_with(';')); + assert_eq!( + blocks[1], + "ATTACH 'ducklake:postgres:…' AS _wm_target (DATA_PATH 's3://b/p');" + ); + assert!(blocks.iter().any(|b| b.contains("_wm_target.orders_daily"))); + assert!(blocks.iter().any(|b| b.starts_with( + "DELETE FROM _wm_target.orders_daily WHERE _wm_partition = '2026-06-19'" + ))); + // the trailing block is the one-row summary (asset / rows / snapshot_id), + // partition-scoped for the row count + let last = blocks.last().unwrap(); + assert!(last.contains("'ducklake://main/orders_daily' AS materialized")); + assert!(last.contains("'2026-06-19' AS partition")); + assert!(last.contains("WHERE _wm_partition = '2026-06-19') AS rows")); + assert!(last.contains("ducklake_snapshots('_wm_target')")); + } + + // -- data tests --------------------------------------------------------- + + fn ctx_partitioned() -> DataTestCtx<'static> { + DataTestCtx { + target_qualified: "_wm_target.orders", + asset_path: "analytics/orders", + partition_col: "_wm_partition", + partition_value_sql: "'2026-06-19'", + partitioned: true, + } + } + fn ctx_unpartitioned() -> DataTestCtx<'static> { + DataTestCtx { partitioned: false, ..ctx_partitioned() } + } + + #[test] + fn data_test_unique_and_not_null_partition_scoped() { + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Unique { column: "order_id".into() }), + DataTestResolved::BuiltIn(DataTest::NotNull { column: "user_id".into() }), + ]; + let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap(); + assert!(sql.attaches.is_empty()); + assert_eq!(sql.checks.len(), 2); + // short, asset-free names (the asset is shown once by the breakdown). + assert_eq!(sql.checks[0].name, "unique(order_id)"); + assert_eq!(sql.checks[1].name, "not_null(user_id)"); + // each `violating` is a scalar count subquery. + assert!(sql.checks[0] + .violating + .starts_with("(SELECT count(*) AS v FROM")); + // unique: groups non-null keys within the slice, having count>1 + assert!(sql.checks[0] + .violating + .contains("GROUP BY \"order_id\" HAVING count(*) > 1")); + assert!(sql.checks[0] + .violating + .contains("\"order_id\" IS NOT NULL AND \"_wm_partition\" = '2026-06-19'")); + // not_null: null rows in the slice + assert!(sql.checks[1] + .violating + .contains("WHERE \"user_id\" IS NULL AND \"_wm_partition\" = '2026-06-19'")); + } + + #[test] + fn data_test_unpartitioned_has_no_partition_scope() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::NotNull { + column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0].violating.contains("WHERE \"id\" IS NULL")); + assert!(!sql.checks[0].violating.contains("_wm_partition")); + } + + #[test] + fn data_test_accepted_values_escapes_literals() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::AcceptedValues { + column: "status".into(), + values: vec!["paid".into(), "o'brien".into()], + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!(sql.checks[0] + .violating + .contains("NOT IN ('paid', 'o''brien')")); + assert!(sql.checks[0].violating.contains("\"status\" IS NOT NULL")); + } + + #[test] + fn data_test_relationships_attaches_ref_and_dedups() { + let tests = vec![ + DataTestResolved::BuiltIn(DataTest::Relationships { + column: "user_id".into(), + to_kind: AssetKind::DataTable, + to_path: "prod/users".into(), + to_column: "id".into(), + }), + // second relationship into the SAME db reuses the alias (no 2nd attach) + DataTestResolved::BuiltIn(DataTest::Relationships { + column: "buyer_id".into(), + to_kind: AssetKind::DataTable, + to_path: "prod/buyers".into(), + to_column: "id".into(), + }), + ]; + let sql = build_data_test_checks(&tests, &ctx_partitioned()).unwrap(); + assert_eq!(sql.attaches.len(), 1, "same db attached once"); + assert_eq!(sql.attaches[0], "ATTACH 'datatable://prod' AS _wm_ref_0;"); + assert!(sql.checks[0] + .violating + .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"users\"")); + assert!(sql.checks[1] + .violating + .contains("NOT EXISTS (SELECT 1 FROM _wm_ref_0.\"buyers\"")); + assert!(sql.checks[0] + .violating + .contains("_wm_src.\"_wm_partition\" = '2026-06-19'")); + assert_eq!( + sql.checks[0].name, + "relationships(user_id -> prod/users.id)" + ); + } + + #[test] + fn data_test_relationships_escapes_ref_name_in_attach() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "k".into(), + to_kind: AssetKind::DataTable, + to_path: "ev'il/users".into(), + to_column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + // single quote in the name is doubled so it can't break out of the literal + assert_eq!(sql.attaches[0], "ATTACH 'datatable://ev''il' AS _wm_ref_0;"); + } + + #[test] + fn data_test_relationships_same_lake_reuses_target() { + // A relationship into the SAME ducklake as the materialize target + // (asset_path = "analytics/orders") must NOT re-ATTACH it — _wm_target + // already holds that catalog; reuse it. + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "user_id".into(), + to_kind: AssetKind::Ducklake, + to_path: "analytics/users".into(), + to_column: "id".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert!( + sql.attaches.is_empty(), + "same-lake ref must not ATTACH again" + ); + assert!(sql.checks[0] + .violating + .contains("NOT EXISTS (SELECT 1 FROM _wm_target.\"users\"")); + } + + #[test] + fn data_test_relationships_schema_qualified_target() { + // `/.
` — the schema-qualified table must quote each + // segment so the dot stays a separator, not part of one identifier. + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "sku".into(), + to_kind: AssetKind::Ducklake, + to_path: "warehouse/main.dim_products".into(), + to_column: "sku".into(), + })]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + assert_eq!( + sql.attaches[0], + "ATTACH 'ducklake://warehouse' AS _wm_ref_0;" + ); + assert!( + sql.checks[0] + .violating + .contains("FROM _wm_ref_0.\"main\".\"dim_products\""), + "schema-qualified target should be quoted per segment: {}", + sql.checks[0].violating + ); + } + + #[test] + fn data_test_relationships_rejects_non_attachable_kind() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Relationships { + column: "k".into(), + to_kind: AssetKind::S3Object, + to_path: "bucket/file".into(), + to_column: "c".into(), + })]; + assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err()); + } + + #[test] + fn data_test_custom_wraps_body() { + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SELECT * FROM _wm_target.orders WHERE amount < 0;".into(), + }]; + let sql = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap(); + // trailing ; stripped, wrapped as a count subquery + assert!(sql.checks[0].violating.contains( + "SELECT count(*) AS v FROM (SELECT * FROM _wm_target.orders WHERE amount < 0)" + )); + assert_eq!(sql.checks[0].name, "custom(f/tests/amount)"); + } + + #[test] + fn data_test_custom_rejects_multi_statement_body() { + // The body is embedded as a subquery, so a setup-then-SELECT body would + // produce invalid SQL — reject it up front with an actionable error. + let tests = vec![DataTestResolved::Custom { + path: "f/tests/amount".into(), + body: "SET threads = 1; SELECT * FROM _wm_target.orders WHERE amount < 0".into(), + }]; + let err = build_data_test_checks(&tests, &ctx_unpartitioned()).unwrap_err(); + assert!(err.contains("single SELECT"), "unexpected error: {err}"); + } + + #[test] + fn data_test_unresolved_custom_is_internal_error() { + let tests = vec![DataTestResolved::BuiltIn(DataTest::Custom { + path: "f/x".into(), + })]; + assert!(build_data_test_checks(&tests, &ctx_unpartitioned()).is_err()); + } + + #[test] + fn materialize_result_sql_embeds_data_tests_breakdown() { + let checks = vec![ + DataTestCheck { + name: "unique(order_id)".into(), + violating: "(SELECT count(*) AS v FROM q0)".into(), + }, + DataTestCheck { + name: "custom(f/t)".into(), + violating: "(SELECT count(*) AS v FROM q1)".into(), + }, + ]; + let sql = materialize_result_sql( + "_wm_target.orders", + "analytics/orders", + "_wm_partition", + "'2026-06-19'", + false, + &checks, + ); + // counts computed once in a CTE, referenced by the list-of-struct. + assert!(sql.starts_with("WITH _wm_tr AS (SELECT (SELECT count(*) AS v FROM q0) AS c0,")); + assert!(sql.contains("[{'test': 'unique(order_id)', 'violating': c0}, ")); + assert!(sql.contains("{'test': 'custom(f/t)', 'violating': c1}] AS data_tests")); + assert!(sql.contains("FROM _wm_tr;")); + // no tests -> plain summary, no CTE / data_tests column. + let plain = materialize_result_sql( + "_wm_target.orders", + "analytics/orders", + "_wm_partition", + "'x'", + false, + &[], + ); + assert!(plain.starts_with("SELECT 'ducklake://analytics/orders' AS materialized")); + assert!(!plain.contains("data_tests")); + // Schema capture (gap #2a) is in every summary, tests or not. Unpartitioned + // → explicit ordering, no partition-column filter. + for s in [&sql, &plain] { + assert!(s.contains( + "(SELECT list({'name': column_name, 'type': column_type} ORDER BY _wm_ord) \ + FROM (SELECT column_name, column_type, row_number() OVER () AS _wm_ord \ + FROM (DESCRIBE SELECT * FROM _wm_target.orders))) AS output_schema" + )); + assert!(!s.contains("WHERE column_name <>")); + } + } + + #[test] + fn materialize_result_sql_schema_excludes_partition_column() { + // Partitioned → the synthetic `_wm_partition` column is filtered out so + // the captured schema is the producer's logical output only. + let sql = materialize_result_sql( + "_wm_target.orders_daily", + "analytics/orders_daily", + "_wm_partition", + "'2026-06-19'", + true, + &[], + ); + assert!(sql.contains( + "FROM (DESCRIBE SELECT * FROM _wm_target.orders_daily) \ + WHERE column_name <> '_wm_partition')) AS output_schema" + )); + } +} diff --git a/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json new file mode 100644 index 0000000000..4ecbc992bc --- /dev/null +++ b/backend/parsers/windmill-parser/tests/fixtures/pipeline_annotations.json @@ -0,0 +1,510 @@ +[ + { + "name": "bare pipeline marker", + "code": "// pipeline\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "pipeline keyword with trailing text is not a marker", + "code": "// pipeline for billing\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "asset triggers across kinds, duplicates deduped", + "code": "-- pipeline\n-- on datatable://main/orders\n-- on s3://bucket/raw.parquet\n-- on ducklake://lake/events\n-- on datatable://main/orders\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [ + "datatable:main/orders", + "s3object:bucket/raw.parquet", + "ducklake:lake/events" + ], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "native trigger markers", + "code": "# pipeline\n# on kafka\n# on schedule\n# on data_upload\nprint(1)", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": ["kafka", "schedule", "data_upload"], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "native keyword must be a whole word and end the line", + "code": "// on kafkalike\n// on kafka topic-extra\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "partitioned daily with options", + "code": "// pipeline\n// partitioned daily tz=Europe/Paris format=%Y-%m-%d start=2024-01-01\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": { + "kind": "daily", + "tz": "Europe/Paris", + "format": "%Y-%m-%d", + "start": "2024-01-01" + }, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "partitioned dynamic requires key, quoted values parse", + "code": "// partitioned dynamic key=\"customer id\"\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": { + "kind": "dynamic", + "key": "customer id", + "tz": null, + "format": null, + "start": null + }, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "partitioned with unknown kind is rejected", + "code": "// partitioned fortnightly\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "partitioned keyword does not match partition prefix", + "code": "// partition daily\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "freshness and tag, first value wins on duplicates", + "code": "# pipeline\n# freshness 1h\n# freshness 2h\n# tag heavy\n# tag light\nprint(1)", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": "1h", + "tag": "heavy", + "retry": null + } + }, + { + "name": "retry with count and delay", + "code": "// pipeline\n// retry 3 5s\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": { "count": 3, "delay": "5s" } + } + }, + { + "name": "retry count only", + "code": "// retry 2\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": { "count": 2, "delay": null } + } + }, + { + "name": "retry zero or malformed count is rejected", + "code": "// retry 0\n// retry 3foo\n// retry -1\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "annotations only count inside comments", + "code": "on datatable://main/raw\npipeline\nconst s = 'on s3://bucket/x'\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "partition token preserved in trigger path", + "code": "// pipeline\n// on s3://bucket/daily/{partition}/data.parquet\nexport function main() {}", + "expected": { + "in_pipeline": true, + "asset_triggers": ["s3object:bucket/daily/{partition}/data.parquet"], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "leading whitespace and mixed comment prefixes", + "code": " -- pipeline\n\t-- on datatable://main/x\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": ["datatable:main/x"], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "on with empty or unparseable spec is ignored", + "code": "// on\n// on \n// on notaprefix/foo\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "materialize managed (default) with merge key", + "code": "// pipeline\n// materialize ducklake://analytics/orders_daily key=order_id\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders_daily", + "unique_key": "order_id" + } + } + }, + { + "name": "materialize manual escape hatch, first value wins", + "code": "// materialize manual ducklake://analytics/orders_daily\n// materialize ducklake://other/x\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders_daily", + "manual": true + } + } + }, + { + "name": "materialize default-syntax shorthand with append", + "code": "// materialize ducklake append\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "main", + "append": true + } + } + }, + { + "name": "materialize manual with no target is dropped", + "code": "// materialize manual\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null + } + }, + { + "name": "data_test built-ins accumulate in order", + "code": "-- pipeline\n-- materialize ducklake://analytics/orders key=order_id\n-- data_test unique order_id\n-- data_test not_null user_id\n-- data_test accepted_values status = paid,pending,refunded\n-- data_test relationships user_id -> datatable://prod/users.id\nSELECT 1;", + "expected": { + "in_pipeline": true, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "materialize": { + "target_kind": "ducklake", + "target_path": "analytics/orders", + "unique_key": "order_id" + }, + "data_tests": [ + { "type": "unique", "column": "order_id" }, + { "type": "not_null", "column": "user_id" }, + { + "type": "accepted_values", + "column": "status", + "values": ["paid", "pending", "refunded"] + }, + { + "type": "relationships", + "column": "user_id", + "to_kind": "datatable", + "to_path": "prod/users", + "to_column": "id" + } + ] + } + }, + { + "name": "data_test accepted_values strips quotes and spacing", + "code": "# data_test accepted_values kind = \"a b\", 'c' ,d\nprint(1)", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { "type": "accepted_values", "column": "kind", "values": ["a b", "c", "d"] } + ] + } + }, + { + "name": "data_test custom escape hatch is a script path", + "code": "// data_test f/tests/orders_amount_sane\nexport function main() {}", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [{ "type": "custom", "path": "f/tests/orders_amount_sane" }] + } + }, + { + "name": "data_test relationships with ducklake shorthand target", + "code": "// data_test relationships sku -> ducklake://warehouse/dim_products.sku\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [ + { + "type": "relationships", + "column": "sku", + "to_kind": "ducklake", + "to_path": "warehouse/dim_products", + "to_column": "sku" + } + ] + } + }, + { + "name": "malformed data_test lines are dropped fail-safe", + "code": "// data_test uniq order_id\n// data_test accepted_values s =\n// data_test relationships a -> b\n// data_test unique\n// data_test\n// data_test unique id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [{ "type": "unique", "column": "id" }] + } + }, + { + "name": "ci test annotation is not a data test", + "code": "// test: f/foo/bar\n// data_test unique id\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "data_tests": [{ "type": "unique", "column": "id" }] + } + }, + { + "name": "column lineage maps output columns to upstream sources", + "code": "// column order_total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.tax\n// column user_name <- datatable://prod/users.name\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "order_total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "tax" + } + ] + }, + { + "column": "user_name", + "inputs": [ + { "from_kind": "datatable", "from_path": "prod/users", "from_column": "name" } + ] + } + ] + } + }, + { + "name": "column lineage keeps duplicate input refs (dedup is a view concern)", + "code": "// column total <- ducklake://warehouse/orders.amount, ducklake://warehouse/orders.amount\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "total", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + }, + { + "from_kind": "ducklake", + "from_path": "warehouse/orders", + "from_column": "amount" + } + ] + } + ] + } + }, + { + "name": "column lineage keeps schema-qualified table, drops malformed refs", + "code": "// column sku <- ducklake://warehouse/main.dim_products.sku, bad_no_dot\n// column no_arrow datatable://prod/x.y\n// column total <- bad_no_dot\nSELECT 1;", + "expected": { + "in_pipeline": false, + "asset_triggers": [], + "native_triggers": [], + "partition": null, + "freshness": null, + "tag": null, + "retry": null, + "column_lineage": [ + { + "column": "sku", + "inputs": [ + { + "from_kind": "ducklake", + "from_path": "warehouse/main.dim_products", + "from_column": "sku" + } + ] + } + ] + } + } +] diff --git a/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs new file mode 100644 index 0000000000..883ddebcbc --- /dev/null +++ b/backend/parsers/windmill-parser/tests/pipeline_annotations_parity.rs @@ -0,0 +1,217 @@ +//! Parser-parity guard: `parse_pipeline_annotations` (Rust, drives deploy) +//! and `parsePipelineAnnotations` (TS, drives the live graph preview — +//! frontend/src/lib/components/assets/AssetGraph/parsePipelineAnnotations.ts) +//! must stay behaviorally identical, or the graph the user previews is not +//! the graph that deploys. Both implementations run the SAME fixture corpus: +//! +//! tests/fixtures/pipeline_annotations.json +//! +//! The frontend counterpart is parsePipelineAnnotations.parity.test.ts. +//! When the annotation grammar changes, extend the corpus — a fixture that +//! passes on one side and fails on the other is exactly the drift this +//! exists to catch. Only the fields both parsers produce are compared +//! (join_mode / debounce_default are deploy-only, parsed solely in Rust). + +use serde::Deserialize; +use windmill_parser::asset_parser::{ + parse_pipeline_annotations, AssetKind, PartitionKind, TriggerSpec, +}; + +#[derive(Deserialize)] +struct Fixture { + name: String, + code: String, + expected: Expected, +} + +#[derive(Deserialize)] +struct Expected { + in_pipeline: bool, + /// `kind:path`, in declaration order, deduped. + asset_triggers: Vec, + native_triggers: Vec, + partition: Option, + freshness: Option, + tag: Option, + retry: Option, + // Default-on-absent so the pre-existing fixtures (which omit it) keep + // deserializing; only fixtures exercising materialization set it. + #[serde(default)] + materialize: Option, + // Snake_case `DataTest` serde shape (e.g. {"type":"unique","column":"x"}), + // compared against `serde_json::to_value(got.data_tests)`. Absent === []. + #[serde(default)] + data_tests: Vec, + // Snake_case `ColumnLineage` serde shape (e.g. {"column":"x","inputs": + // [{"from_kind":"datatable","from_path":"p","from_column":"c"}]}), compared + // against `to_value(got.column_lineage)`. Absent === []. + #[serde(default)] + column_lineage: Vec, +} + +#[derive(Deserialize)] +struct ExpectedMaterialize { + target_kind: String, + target_path: String, + #[serde(default)] + manual: bool, + #[serde(default)] + append: bool, + #[serde(default)] + unique_key: Option, +} + +#[derive(Deserialize)] +struct ExpectedPartition { + kind: String, + #[serde(default)] + key: Option, + tz: Option, + format: Option, + start: Option, +} + +#[derive(Deserialize)] +struct ExpectedRetry { + count: u32, + delay: Option, +} + +fn kind_str(k: AssetKind) -> &'static str { + match k { + AssetKind::S3Object => "s3object", + AssetKind::Resource => "resource", + AssetKind::Ducklake => "ducklake", + AssetKind::DataTable => "datatable", + AssetKind::Volume => "volume", + } +} + +fn native_str(t: &TriggerSpec) -> Option<&'static str> { + Some(match t { + TriggerSpec::Asset { .. } => return None, + TriggerSpec::Schedule => "schedule", + TriggerSpec::Webhook => "webhook", + TriggerSpec::Email => "email", + TriggerSpec::Kafka => "kafka", + TriggerSpec::Mqtt => "mqtt", + TriggerSpec::Nats => "nats", + TriggerSpec::Postgres => "postgres", + TriggerSpec::Sqs => "sqs", + TriggerSpec::Gcp => "gcp", + TriggerSpec::DataUpload => "data_upload", + }) +} + +#[test] +fn pipeline_annotation_fixtures_match() { + let fixtures: Vec = + serde_json::from_str(include_str!("fixtures/pipeline_annotations.json")) + .expect("fixture corpus must deserialize"); + assert!(!fixtures.is_empty()); + + for f in fixtures { + let got = parse_pipeline_annotations(&f.code); + let ctx = format!("fixture '{}'", f.name); + + assert_eq!( + got.in_pipeline, f.expected.in_pipeline, + "{ctx}: in_pipeline" + ); + + let asset_triggers: Vec = got + .triggers + .iter() + .filter_map(|t| match t { + TriggerSpec::Asset { asset_kind, path, .. } => { + Some(format!("{}:{}", kind_str(*asset_kind), path)) + } + _ => None, + }) + .collect(); + assert_eq!( + asset_triggers, f.expected.asset_triggers, + "{ctx}: asset triggers" + ); + + let native: Vec<&str> = got.triggers.iter().filter_map(native_str).collect(); + assert_eq!(native, f.expected.native_triggers, "{ctx}: native triggers"); + + match (&got.partition, &f.expected.partition) { + (None, None) => {} + (Some(p), Some(e)) => { + let (kind, key) = match &p.kind { + PartitionKind::Daily => ("daily", None), + PartitionKind::Hourly => ("hourly", None), + PartitionKind::Weekly => ("weekly", None), + PartitionKind::Monthly => ("monthly", None), + PartitionKind::Dynamic { key } => ("dynamic", Some(key.clone())), + }; + assert_eq!(kind, e.kind, "{ctx}: partition kind"); + assert_eq!(key, e.key, "{ctx}: partition key"); + assert_eq!(p.tz, e.tz, "{ctx}: partition tz"); + assert_eq!(p.format, e.format, "{ctx}: partition format"); + assert_eq!(p.start, e.start, "{ctx}: partition start"); + } + (got, want) => panic!( + "{ctx}: partition mismatch — got {:?}, want present={}", + got, + want.is_some() + ), + } + + assert_eq!( + got.freshness.as_ref().map(|fr| fr.duration.clone()), + f.expected.freshness, + "{ctx}: freshness" + ); + assert_eq!(got.tag, f.expected.tag, "{ctx}: tag"); + match (&got.retry, &f.expected.retry) { + (None, None) => {} + (Some(r), Some(e)) => { + assert_eq!(r.count, e.count, "{ctx}: retry count"); + assert_eq!(r.delay, e.delay, "{ctx}: retry delay"); + } + (got, want) => panic!( + "{ctx}: retry mismatch — got {:?}, want present={}", + got, + want.is_some() + ), + } + + match (&got.materialize, &f.expected.materialize) { + (None, None) => {} + (Some(m), Some(e)) => { + assert_eq!( + kind_str(m.target_kind), + e.target_kind, + "{ctx}: materialize kind" + ); + assert_eq!(m.target_path, e.target_path, "{ctx}: materialize path"); + assert_eq!(m.manual, e.manual, "{ctx}: materialize manual"); + assert_eq!(m.append, e.append, "{ctx}: materialize append"); + assert_eq!(m.unique_key, e.unique_key, "{ctx}: materialize key"); + } + (got, want) => panic!( + "{ctx}: materialize mismatch — got {:?}, want present={}", + got, + want.is_some() + ), + } + + let got_tests = serde_json::to_value(&got.data_tests).expect("data_tests serialize"); + assert_eq!( + got_tests, + serde_json::Value::Array(f.expected.data_tests.clone()), + "{ctx}: data tests" + ); + + let got_lineage = + serde_json::to_value(&got.column_lineage).expect("column_lineage serialize"); + assert_eq!( + got_lineage, + serde_json::Value::Array(f.expected.column_lineage.clone()), + "{ctx}: column lineage" + ); + } +} diff --git a/backend/src/main.rs b/backend/src/main.rs index a97320d5d9..94b8e2f4f6 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -1446,19 +1446,45 @@ Windmill Community Edition {GIT_VERSION} } else { None }; - monitor_db( - &conn, - &base_internal_url, - server_mode, - worker_mode, - false, - tx.clone(), - Some(MonitorIteration { - rd_shift, - iter: monitor_iteration, - }), + // Hard cap on a single monitor pass. monitor_db runs all its + // periodic tasks under one join!, so a single task stuck on a + // non-DB await (statement_timeout only bounds DB statements) + // would otherwise freeze the whole loop indefinitely — silently + // stopping critical maintenance like audit-partition creation. + // Larger than statement_timeout (5min) so a slow-but-progressing + // statement is never killed prematurely. + const MONITOR_DB_TIMEOUT: Duration = Duration::from_secs(600); + let monitor_timed_out = tokio::time::timeout( + MONITOR_DB_TIMEOUT, + monitor_db( + &conn, + &base_internal_url, + server_mode, + worker_mode, + false, + tx.clone(), + Some(MonitorIteration { + rd_shift, + iter: monitor_iteration, + }), + ), ) - .await; + .await + .is_err(); + if monitor_timed_out { + windmill_common::utils::report_critical_error( + format!( + "monitor task did not finish within {}s and was aborted; \ + a background maintenance task is likely stuck. \ + Continuing to the next iteration.", + MONITOR_DB_TIMEOUT.as_secs() + ), + db.clone(), + None, + None, + ) + .await; + } monitor_iteration += 1; if let Some(handle) = warn_handle { handle.abort(); @@ -1646,6 +1672,13 @@ async fn process_notify_event( ); windmill_common::variables::CUSTOM_ENVS_CACHE.remove(payload); } + "notify_asset_producer_change" => { + tracing::debug!( + "Asset producer change for workspace {}, invalidating producer-writes cache", + payload + ); + windmill_queue::asset_dispatch::ASSET_PRODUCER_WRITES_CACHE.remove(payload); + } "notify_workspace_key_change" => { tracing::info!( "Workspace key change detected, invalidating workspace key cache: {}", diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index e68867a89f..aed9a30be3 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -75,6 +75,7 @@ use windmill_common::{ WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING, }, indexer::load_indexer_config, + jobs::delete_jobs, jwt::JWT_SECRET, oauth2::REQUIRE_PREEXISTING_USER_FOR_OAUTH, server::load_smtp_config, @@ -1275,6 +1276,23 @@ pub async fn delete_expired_items(db: &DB) -> () { tracing::error!("Error deleting autoscaling event on CE: {:?}", e); } + // native_retry_attempt has no FK to v2_job (kept off the hot bulk delete). + // Retention sweeps markers alongside the jobs it deletes, but direct job + // deletions (workspace/job/schedule clearing) leave markers orphaned — reap + // any whose job is gone. The table is sparse, so this anti-join is cheap. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt nra WHERE NOT EXISTS (SELECT 1 FROM v2_job WHERE id = nra.job_id)" + ) + .execute(db) + .await + { + tracing::error!("Error reaping orphaned native retry markers: {:?}", e); + } + + if let Err(e) = windmill_queue::cascade::reap_stale_join_slots(db).await { + tracing::error!("Error reaping stale join_pending_inputs slots: {:?}", e); + } + match sqlx::query_scalar!( "DELETE FROM agent_token_blacklist WHERE expires_at <= now() RETURNING token", ) @@ -1320,6 +1338,9 @@ pub async fn delete_expired_items(db: &DB) -> () { let cleanup_start = Instant::now(); let mut total_deleted = 0u64; let mut batch_num = 0i32; + // Watermark carried across batches so each one resumes after the rows the previous batch + // already processed instead of re-scanning the (potentially undeletable) oldest prefix. + let mut completed_at_floor: Option> = None; // Process batches until no more expired jobs or max batches reached loop { @@ -1332,14 +1353,17 @@ pub async fn delete_expired_items(db: &DB) -> () { } // Each batch runs in its own transaction to avoid long-running locks - let batch_result = delete_expired_jobs_batch(db, job_retention_secs, batch_size).await; + let batch_result = + delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor) + .await; match batch_result { - Ok(deleted_count) => { + Ok((deleted_count, max_completed_at)) => { if deleted_count == 0 { // No more expired jobs to delete break; } + completed_at_floor = max_completed_at.or(completed_at_floor); total_deleted += deleted_count as u64; batch_num += 1; } @@ -1506,12 +1530,20 @@ pub async fn check_expiring_tokens(db: &DB) { /// Delete a batch of expired jobs with LIMIT and SKIP LOCKED for high-scale environments. /// Uses a single transaction per batch to minimize lock duration. -/// Returns the number of jobs deleted in this batch. +/// +/// `completed_at_floor` is the watermark from the previous batch in the same cleanup run (the +/// max `completed_at` it deleted); pass `None` for the first batch. It is re-applied as +/// `completed_at >= floor` so the scan resumes past the rows already processed instead of +/// re-walking them (see the inline comment on the DELETE for why this matters). +/// +/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the +/// returned watermark back in as `completed_at_floor` for the next batch. async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result { + completed_at_floor: Option>, +) -> error::Result<(usize, Option>)> { let mut tx = db.begin().await?; // Fetch active ROOT job IDs that started before the retention period. We only care about @@ -1527,26 +1559,70 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas - // ORDER BY completed_at ensures we delete oldest jobs first - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` is a watermark carried across batches within a cleanup run: it is the + // max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor` + // lets each batch resume after the rows the previous batch already processed instead of + // re-scanning them. This matters when the oldest rows are undeletable (children of a + // still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that + // same protected prefix on every batch, turning a cleanup run quadratic in prefix size. + // Floor only ever skips rows the current run already deleted, was protecting, or skip-locked — + // all correctly deferred to the next run, identical to the unbounded scan's semantics. + // + // Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at + // deletes oldest jobs first. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + // Common case: no old root flow is still running, so nothing is protected and the + // v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely. + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + // Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`: + // the subquery form lets the planner build a one-time hashed SubPlan and apply it as a + // filter on the ordered index scan, giving O(1) membership per candidate instead of a + // per-row linear array scan (which degrades sharply when many root jobs are active). The + // `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids). + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); @@ -1585,10 +1661,21 @@ async fn delete_expired_jobs_batch( Err(e) => tracing::error!("Error deleting job logs: {:?}", e), } - if let Err(e) = sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", &deleted_jobs) - .execute(&mut *tx) - .await + // Native retry markers have no FK (to keep this bulk delete cheap) — sweep + // them with their jobs here too (the periodic retention path), same as the + // other side tables. The table is created by a startup migration, so it + // always exists by the time cleanup runs. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + &deleted_jobs + ) + .execute(&mut *tx) + .await { + tracing::error!("Error deleting native retry markers: {:?}", e); + } + + if let Err(e) = delete_jobs(&mut *tx, &deleted_jobs).await { tracing::error!("Error deleting job: {:?}", e); } @@ -1606,7 +1693,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok(deleted_count) + Ok((deleted_count, max_completed_at)) } async fn delete_log_files_from_disk_and_store( @@ -1634,11 +1721,30 @@ async fn delete_log_files_from_disk_and_store( .collect(); let stream = futures::stream::iter(s3_paths).boxed(); let mut result = os.delete_stream(stream); + let mut deleted = 0u64; + let mut not_found = 0u64; + let mut failed = 0u64; while let Some(r) = result.next().await { - if let Err(e) = r { - tracing::error!("Failed to delete from object store: {e}"); + match r { + Ok(_) => deleted += 1, + // Deleting a non-existent object is a successful no-op. S3's + // DeleteObjects ignores missing keys, but GCS returns 404 per + // delete, surfacing as NotFound — count it separately rather + // than logging it as an error. + Err(windmill_object_store::object_store_reexports::ObjectStoreError::NotFound { .. }) => { + not_found += 1; + } + Err(e) => { + failed += 1; + tracing::error!("Failed to delete from object store: {e}"); + } } } + if deleted + not_found + failed > 0 { + tracing::info!( + "object store log cleanup: {deleted} deleted, {not_found} already absent (404), {failed} failed" + ); + } } } } @@ -2610,6 +2716,9 @@ pub async fn monitor_db( if let Err(e) = cleanup_debounce_orphaned_keys(&db).await { tracing::error!("Error cleaning up debounce keys: {:?}", e); } + if let Err(e) = cleanup_consumed_debounce_batches(&db).await { + tracing::error!("Error cleaning up consumed debounce batches: {:?}", e); + } } } }; @@ -4460,6 +4569,40 @@ RETURNING key,job_id Ok(()) } +/// GC for claim-based debounce batches: once a batch row has been consumed (its +/// args accumulated into some survivor's run), it only lingers to let a later-pulled +/// survivor of the same batch tell "already consumed" from "never batched". A generous +/// grace period (>> any debounce window) makes that decision safe; after it, the rows +/// are dead weight. A re-pulled survivor whose row was GC'd correctly falls back to its +/// own (already-accumulated, persisted) args, so the grace period is not correctness- +/// critical. +async fn cleanup_consumed_debounce_batches(db: &DB) -> error::Result<()> { + // Only reclaim a consumed row once its job has LEFT the queue. A consumed sibling + // (its contribution already accumulated by another survivor) can sit queued well + // past any time-based grace under a concurrency limit / worker backlog; removing its + // row while still queued would make its eventual pull treat it as never-batched and + // re-run its item (a duplicate). Keeping the row until the job is no longer queued + // guarantees that pull still sees "already consumed" and runs empty. The age floor + // is just a safety margin on top. + let deleted = sqlx::query_scalar!( + "WITH del AS ( + DELETE FROM v2_job_debounce_batch + WHERE consumed_at IS NOT NULL + AND consumed_at < now() - interval '10 minutes' + AND id NOT IN (SELECT id FROM v2_job_queue) + RETURNING 1 + ) SELECT count(*) FROM del" + ) + .fetch_one(db) + .await? + .unwrap_or(0); + + if deleted > 0 { + tracing::info!("Cleaned up {deleted} consumed debounce batch rows"); + } + Ok(()) +} + async fn cleanup_debounce_keys_for_completed_jobs(db: &DB) -> error::Result<()> { // If min version doesn't support runnable settings, clean up debounce keys for completed jobs if !windmill_common::min_version::MIN_VERSION_SUPPORTS_RUNNABLE_SETTINGS_V0 @@ -4493,39 +4636,90 @@ RETURNING key,job_id Ok(()) } -async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { - let result = sqlx::query_scalar!( - "DELETE FROM job_perms -WHERE job_id NOT IN (SELECT id FROM v2_job_queue) -RETURNING job_id" - ) - .fetch_all(db) - .await?; +// Per-statement cap keeps each delete short and lock-light; the per-cycle batch +// cap bounds total work per monitor iteration so monitor_db stays responsive. +// A large backlog drains across several iterations rather than one long delete. +// +// These sweeps anti-join the whole table to find orphans, so their cost tracks the heap's +// physical size. job_perms / job_result_stream_v2 are high-churn (one row per job, deleted +// here), so their bloat — not the query shape — is what makes the sweep slow. These sweeps run +// every monitor cycle, but the bulk vacuuming_tables() runs only ~hourly, so dead tuples pile +// up between bulk vacuums; each sweep VACUUMs its own table right after deleting (see below) to +// keep the heap near the live working set. The outer `ctid IN (SELECT ... LIMIT)` is +// deliberate: a `job_id IN (...)` rewrite adds a second scan/probe for the delete and +// benchmarks slower, so don't "simplify" it. +const ORPHAN_CLEANUP_BATCH_SIZE: u64 = 100_000; +const ORPHAN_CLEANUP_MAX_BATCHES: usize = 10; - if !result.is_empty() { - tracing::info!("Cleaned up {} orphaned job_perms rows", result.len()); +// Reclaim the dead tuples a sweep just created so the next sweep's anti-join scans a lean heap +// instead of a bloated one. Plain VACUUM (not FULL) only takes SHARE UPDATE EXCLUSIVE, so +// concurrent reads/writes (every job create touches job_perms) keep running, and the visibility +// map lets it skip unchanged pages so repeated runs are cheap. SKIP_LOCKED means HA replicas +// don't pile up: one vacuums, the rest skip rather than queue behind it. +async fn vacuum_after_sweep(db: &DB, table: &str) { + if let Err(e) = sqlx::query(&format!("VACUUM (SKIP_LOCKED) {table}")) + .execute(db) + .await + { + tracing::warn!("Error vacuuming {table} after orphan cleanup: {e:?}"); + } +} + +async fn cleanup_job_perms_orphaned(db: &DB) -> error::Result<()> { + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_perms + WHERE ctid IN ( + SELECT jp.ctid FROM job_perms jp + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jp.job_id) + LIMIT 100000 + )" + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } + + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_perms rows"); + vacuum_after_sweep(db, "job_perms").await; } Ok(()) } async fn cleanup_job_result_stream_orphaned_jobs(db: &DB) -> error::Result<()> { - let result = sqlx::query!( - "DELETE FROM job_result_stream_v2 - WHERE job_id NOT IN (SELECT id FROM v2_job_queue) - AND job_id NOT IN ( - SELECT id FROM v2_job_completed - WHERE completed_at > NOW() - INTERVAL '60 seconds' - ) - RETURNING job_id", - ) - .fetch_all(db) - .await?; + let mut total: u64 = 0; + for _ in 0..ORPHAN_CLEANUP_MAX_BATCHES { + let count = sqlx::query!( + "DELETE FROM job_result_stream_v2 + WHERE ctid IN ( + SELECT jrs.ctid FROM job_result_stream_v2 jrs + WHERE NOT EXISTS (SELECT 1 FROM v2_job_queue q WHERE q.id = jrs.job_id) + AND NOT EXISTS ( + SELECT 1 FROM v2_job_completed c + WHERE c.id = jrs.job_id + AND c.completed_at > NOW() - INTERVAL '60 seconds' + ) + LIMIT 100000 + )", + ) + .execute(db) + .await? + .rows_affected(); + total += count; + if count < ORPHAN_CLEANUP_BATCH_SIZE { + break; + } + } - if result.len() > 0 { - tracing::info!( - "Cleaned up {} orphaned job_result_stream_v2 rows", - result.len() - ); + if total > 0 { + tracing::info!("Cleaned up {total} orphaned job_result_stream_v2 rows"); + vacuum_after_sweep(db, "job_result_stream_v2").await; } Ok(()) } @@ -4561,11 +4755,19 @@ async fn audit_log_retention_days() -> i64 { } } +/// Number of days ahead (including today) for which an audit partition must +/// always exist. A missing partition in this window means audit inserts fail +/// once that date is reached — and because some callers (notably login) write +/// the audit row in the same transaction as their own work, that failure +/// poisons the whole transaction, so a missing partition is a hard outage, not +/// just a dropped audit row. +const AUDIT_PARTITION_LOOKAHEAD_DAYS: i64 = 3; + async fn manage_audit_partitions(db: &DB, retention_days: i64) { let today = chrono::Utc::now().date_naive(); - // Create partitions for today and the next 3 days - for days_ahead in 0..=3i64 { + // Create partitions for today and the next few days + for days_ahead in 0..=AUDIT_PARTITION_LOOKAHEAD_DAYS { let date = today + chrono::Duration::days(days_ahead); let next_date = date + chrono::Duration::days(1); let partition_name = format!("audit_{}", date.format("%Y%m%d")); @@ -4581,9 +4783,6 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { } } - // Drop expired partitions - let cutoff_date = today - chrono::Duration::days(retention_days); - let partitions = sqlx::query_scalar::<_, String>( "SELECT c.relname::text \ FROM pg_inherits i \ @@ -4593,28 +4792,62 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) { .fetch_all(db) .await; - match partitions { - Ok(partitions) => { - for partition_name in partitions { - if let Some(date_str) = partition_name.strip_prefix("audit_") { - if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { - if date < cutoff_date { - let quoted_name = - format!("\"{}\"", partition_name.replace('"', "\"\"")); - let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); - match sqlx::query(&sql).execute(db).await { - Ok(_) => tracing::info!( - "Dropped expired audit partition {partition_name}" - ), - Err(e) => tracing::error!( - "Error dropping audit partition {partition_name}: {e:?}" - ), - } + let partitions = match partitions { + Ok(partitions) => partitions, + Err(e) => { + tracing::error!("Error listing audit partitions: {e:?}"); + return; + } + }; + + // Verify the lookahead window is actually covered. If a create above failed + // (or this loop has not run for several days), alert loudly instead of + // letting it surface days later as failed audit inserts and broken logins. + let existing: std::collections::HashSet<&str> = partitions.iter().map(|s| s.as_str()).collect(); + let missing: Vec = (0..=AUDIT_PARTITION_LOOKAHEAD_DAYS) + .map(|days_ahead| { + format!( + "audit_{}", + (today + chrono::Duration::days(days_ahead)).format("%Y%m%d") + ) + }) + .filter(|name| !existing.contains(name.as_str())) + .collect(); + if !missing.is_empty() { + report_critical_error( + format!( + "Audit log partitions missing after maintenance run: {}. \ + Audit inserts will fail once these dates are reached, which also \ + breaks logins (the login audit row shares the login transaction). \ + Check for earlier 'Error creating audit partition' logs and verify \ + the audit-partition maintenance loop is still running.", + missing.join(", ") + ), + db.clone(), + None, + None, + ) + .await; + } + + // Drop expired partitions + let cutoff_date = today - chrono::Duration::days(retention_days); + for partition_name in &partitions { + if let Some(date_str) = partition_name.strip_prefix("audit_") { + if let Ok(date) = chrono::NaiveDate::parse_from_str(date_str, "%Y%m%d") { + if date < cutoff_date { + let quoted_name = format!("\"{}\"", partition_name.replace('"', "\"\"")); + let sql = format!("DROP TABLE IF EXISTS {quoted_name}"); + match sqlx::query(&sql).execute(db).await { + Ok(_) => { + tracing::info!("Dropped expired audit partition {partition_name}") } + Err(e) => tracing::error!( + "Error dropping audit partition {partition_name}: {e:?}" + ), } } } } - Err(e) => tracing::error!("Error listing audit partitions: {e:?}"), } } diff --git a/backend/tests/app_preview_auth.rs b/backend/tests/app_preview_auth.rs index 4653303392..2a69004a7b 100644 --- a/backend/tests/app_preview_auth.rs +++ b/backend/tests/app_preview_auth.rs @@ -17,7 +17,12 @@ //! must not over-block the legitimate editor flow), //! - preview is confined to paths the caller can read (defense-in-depth //! against scoped tokens / cross-namespace preview), and -//! - run mode (no `force_viewer_static_fields`) is unaffected by the guard. +//! - run mode (no `force_viewer_static_fields`) is unaffected by the preview +//! guard, and +//! - run mode against a deployed Viewer app rejects caller-supplied inline +//! `raw_code` whose sha is not publisher-pinned (CVE-2026-22683 residual: +//! the Viewer default-triggerable fallback let any caller / an operator run +//! arbitrary code as themselves, bypassing the content-hash pin). use serde_json::json; use sqlx::{Pool, Postgres}; @@ -254,5 +259,90 @@ async fn test_app_preview_authorization(db: Pool) -> anyhow::Result<() "rejection must be the jobs:run scope gate, got: {body}" ); + // 9. RUN-MODE REGRESSION (CVE-2026-22683 residual): in run mode (no + // `force_viewer_static_fields`) against a deployed Viewer-mode app, + // caller-supplied inline `raw_code` whose `rawscript/` is not pinned + // in the app's `triggerables_v2` must be rejected by the policy — it must + // not resolve via the Viewer default triggerable and run as the caller + // (the same preview-class execution an operator is denied in step 1). + let run_mode_raw_code = json!({ + "args": {}, + "component": "comp", + "raw_code": { + "language": "bash", + "content": "id; echo RCE_$(whoami)", + "path": "x" + } + }); + let resp = authed( + client().post(format!("{base}/u/test-user/vapp")), + "OPERATOR_TOKEN", + ) + .json(&run_mode_raw_code) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!( + status, 400, + "run-mode inline raw_code against a Viewer app must be rejected, not run as the caller (got {status}): {body}" + ); + assert!( + body.contains("forbidden by policy"), + "rejection must be the content-hash pin (unpinned rawscript), got: {body}" + ); + + // 10. The content-pin fix is not operator-specific: even a regular + // non-operator member (who could run their own code via + // `/jobs/run/preview`) must not be able to substitute unpinned code into + // someone else's deployed Viewer app — the deployed-app integrity break. + let resp = authed( + client().post(format!("{base}/u/test-user/vapp")), + "SECRET_TOKEN_2", + ) + .json(&run_mode_raw_code) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!( + status, 400, + "run-mode unpinned raw_code must be rejected for any caller, not just operators (got {status}): {body}" + ); + + // 11. The pin requirement also covers `raw_code` carrying an `app_script` + // `id`. The id resolves to `rawscript/` of any app_script row + // by number (no app scoping), so without the pin a caller could run a + // script belonging to another app against this Viewer app. Here `999777` + // belongs to `u/test-user/private`, not to the targeted `vapp`, and its + // sha is absent from `vapp`'s empty `triggerables_v2` — it must be + // rejected, not fall back to the Viewer default. + let resp = authed( + client().post(format!("{base}/u/test-user/vapp")), + "OPERATOR_TOKEN", + ) + .json(&json!({ + "args": {}, + "component": "comp", + "id": 999777, + "raw_code": { + "language": "deno", + "content": "export function main() { return 1; }", + "path": "x" + } + })) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!( + status, 400, + "run-mode raw_code with an unpinned app_script id must be rejected against a Viewer app (got {status}): {body}" + ); + assert!( + body.contains("forbidden by policy"), + "rejection must be the content-hash pin (unpinned app_script sha), got: {body}" + ); + Ok(()) } diff --git a/backend/tests/asset_trigger_dispatch.rs b/backend/tests/asset_trigger_dispatch.rs new file mode 100644 index 0000000000..198472d404 --- /dev/null +++ b/backend/tests/asset_trigger_dispatch.rs @@ -0,0 +1,1046 @@ +//! End-to-end test for asset-trigger dispatch. +//! +//! Runs a real Bash producer through a worker, lets the +//! `result_processor` hook fire `dispatch_asset_triggers`, and then makes +//! several follow-up calls into `dispatch_asset_triggers` against the same +//! seeded graph to cover the eligibility branches (self-loop, skip arg, +//! cycle guard, flow subscriber, ineligible job kinds). Direct calls share +//! the same workspace so we exercise the real query paths against real +//! `asset` / `script_trigger` rows produced by deploy-equivalent seeding. + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use uuid::Uuid; +use windmill_common::jobs::{JobKind, JobPayload}; +use windmill_common::scripts::{ScriptHash, ScriptLang}; +use windmill_queue::asset_dispatch::dispatch_asset_triggers; +use windmill_queue::cascade::reap_stale_join_slots; +use windmill_queue::MiniCompletedJob; +use windmill_test_utils::{initialize_tracing, ApiServer, RunJob}; + +const WS: &str = "test-workspace"; +const PRODUCER: &str = "u/test-user/producer"; +const SUB_S3: &str = "u/test-user/sub-s3"; +const SUB_RES: &str = "u/test-user/sub-res"; +const SUB_FLOW: &str = "u/test-user/sub-flow"; + +// ── Seeding helpers ─────────────────────────────────────────────────────── + +async fn seed_script( + db: &Pool, + path: &str, + content: &str, + language: &str, +) -> anyhow::Result { + // Hash needs to be unique per (workspace, hash). Derive from path AND + // content: the worker's script cache (`cache::script::fetch`) is keyed + // by hash alone and is process-global, so tests running in the same + // process that seed the same path with different content would poison + // each other's cache if the hash came from the path only. + let mut h = 0i64; + for b in path.bytes().chain(content.bytes()) { + h = h.wrapping_mul(31).wrapping_add(b as i64); + } + sqlx::query( + r#"INSERT INTO script (workspace_id, hash, path, summary, description, content, + created_by, language, tag, lock) + VALUES ($1, $2, $3, '', '', $4, 'test-user', $5::script_lang, 'deno', '') + ON CONFLICT DO NOTHING"#, + ) + .bind(WS) + .bind(h) + .bind(path) + .bind(content) + .bind(language) + .execute(db) + .await?; + // These tests use #[sqlx::test] isolated DBs that share one workspace id and + // reuse script paths, while the same path is seeded with different content + // (hence different hashes) across tests. The process-global deployed-script + // caches are keyed by (workspace, path)/(workspace, hash), so a concurrent + // test resolves a path to a hash that lives in another test's DB and the + // dispatch 404s. Disable them so every resolution reads the test's own DB. + windmill_common::DEPLOYED_SCRIPT_CACHE_DISABLED + .store(true, std::sync::atomic::Ordering::Relaxed); + Ok(h) +} + +async fn seed_asset_write( + db: &Pool, + producer_path: &str, + kind: &str, + asset_path: &str, +) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind) + VALUES ($1, $2, $3::asset_kind, 'w'::asset_access_type, $4, 'script'::asset_usage_kind) + ON CONFLICT DO NOTHING"#, + ) + .bind(WS) + .bind(asset_path) + .bind(kind) + .bind(producer_path) + .execute(db) + .await?; + // These tests use #[sqlx::test] isolated DBs that all share one workspace + // id, so the process-global producer cache (keyed by workspace) would + // clobber across DBs under concurrent test threads. Disable it so every + // dispatch reads the test's own DB. (Production invalidates via the + // notify_event poller instead.) + windmill_queue::asset_dispatch::ASSET_PRODUCER_CACHE_DISABLED + .store(true, std::sync::atomic::Ordering::Relaxed); + Ok(()) +} + +async fn seed_subscription( + db: &Pool, + subscriber_path: &str, + subscriber_kind: &str, + trigger_ref: &str, +) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO script_trigger + (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref) + VALUES ($1, $2::asset_usage_kind, $3, 'asset'::script_trigger_kind, $4)"#, + ) + .bind(WS) + .bind(subscriber_kind) + .bind(subscriber_path) + .bind(trigger_ref) + .execute(db) + .await?; + Ok(()) +} + +/// Insert a synthetic producer `v2_job` row used by the direct +/// `dispatch_asset_triggers` calls in the edge-case section. `args` lets the +/// test inject `_wmill_skip_asset_dispatch` or `trigger.chain` so the +/// dispatcher's arg-driven branches are exercised against real rows. +async fn seed_producer_job(db: &Pool, args: serde_json::Value) -> anyhow::Result { + let id = Uuid::new_v4(); + sqlx::query!( + r#"INSERT INTO v2_job (id, workspace_id, kind, runnable_path, args, created_by, + permissioned_as, permissioned_as_email, tag, script_lang) + VALUES ($1, $2, 'script'::job_kind, $3, $4, 'test-user', + 'u/test-user', 'test@windmill.dev', 'deno', 'bash'::script_lang)"#, + id, + WS, + PRODUCER, + args, + ) + .execute(db) + .await?; + Ok(id) +} + +/// Like `seed_producer_job` but for an arbitrary runnable path (the +/// AND-join test needs two distinct producers). +async fn seed_producer_job_path( + db: &Pool, + path: &str, + args: serde_json::Value, +) -> anyhow::Result { + let id = Uuid::new_v4(); + sqlx::query!( + r#"INSERT INTO v2_job (id, workspace_id, kind, runnable_path, args, created_by, + permissioned_as, permissioned_as_email, tag, script_lang) + VALUES ($1, $2, 'script'::job_kind, $3, $4, 'test-user', + 'u/test-user', 'test@windmill.dev', 'deno', 'bash'::script_lang)"#, + id, + WS, + path, + args, + ) + .execute(db) + .await?; + Ok(id) +} + +/// Seed an asset subscription flagged as an AND join (`// trigger all`). +async fn seed_subscription_and( + db: &Pool, + subscriber_path: &str, + trigger_ref: &str, +) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO script_trigger + (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all) + VALUES ($1, 'script'::asset_usage_kind, $2, 'asset'::script_trigger_kind, $3, TRUE)"#, + ) + .bind(WS) + .bind(subscriber_path) + .bind(trigger_ref) + .execute(db) + .await?; + Ok(()) +} + +/// Seed an asset subscription with an opt-in debounce window (seconds). +async fn seed_subscription_debounced( + db: &Pool, + subscriber_path: &str, + trigger_ref: &str, + debounce_s: i32, +) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO script_trigger + (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, debounce_s) + VALUES ($1, 'script'::asset_usage_kind, $2, 'asset'::script_trigger_kind, $3, $4)"#, + ) + .bind(WS) + .bind(subscriber_path) + .bind(trigger_ref) + .bind(debounce_s) + .execute(db) + .await?; + Ok(()) +} + +/// Seed an asset subscription with a `// retry []` policy. +async fn seed_subscription_with_retry( + db: &Pool, + subscriber_path: &str, + trigger_ref: &str, + retry_count: i16, + retry_delay_s: i32, +) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO script_trigger + (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, + retry_count, retry_delay_s) + VALUES ($1, 'script'::asset_usage_kind, $2, 'asset'::script_trigger_kind, $3, $4, $5)"#, + ) + .bind(WS) + .bind(subscriber_path) + .bind(trigger_ref) + .bind(retry_count) + .bind(retry_delay_s) + .execute(db) + .await?; + Ok(()) +} + +fn make_mini(id: Uuid, runnable_path: &str) -> MiniCompletedJob { + MiniCompletedJob { + id, + workspace_id: WS.to_string(), + runnable_id: Some(ScriptHash(1)), + scheduled_for: chrono::Utc::now(), + parent_job: None, + flow_innermost_root_job: None, + runnable_path: Some(runnable_path.to_string()), + kind: JobKind::Script, + started_at: Some(chrono::Utc::now()), + permissioned_as: "u/test-user".to_string(), + created_by: "test-user".to_string(), + script_lang: Some(ScriptLang::Bash), + permissioned_as_email: "test@windmill.dev".to_string(), + flow_step_id: None, + trigger_kind: None, + trigger: None, + priority: None, + concurrent_limit: None, + tag: "deno".to_string(), + cache_ttl: None, + cache_ignore_s3_path: None, + runnable_settings_handle: None, + } +} + +/// Read `v2_job` rows that were created by asset dispatch (filtered by +/// `trigger_kind = 'asset'` so dep-jobs / other infra rows don't leak in). +async fn fetch_dispatched( + db: &Pool, +) -> anyhow::Result, Option)>> { + let rows = sqlx::query!( + r#"SELECT runnable_path AS "runnable_path!", trigger, + args AS "args: sqlx::types::Json" + FROM v2_job + WHERE workspace_id = $1 AND trigger_kind = 'asset' + ORDER BY runnable_path"#, + WS, + ) + .fetch_all(db) + .await?; + Ok(rows + .into_iter() + .map(|r| (r.runnable_path, r.trigger, r.args.map(|j| j.0))) + .collect()) +} + +async fn clear_dispatched(db: &Pool) -> anyhow::Result<()> { + sqlx::query!( + "DELETE FROM v2_job WHERE workspace_id = $1 AND trigger_kind = 'asset'", + WS, + ) + .execute(db) + .await?; + Ok(()) +} + +// ── The test ───────────────────────────────────────────────────────────── + +/// One end-to-end test that: +/// 1. seeds a producer that writes two asset kinds (s3 + resource) with three +/// subscribers (two script subs + one flow sub that must be skipped), +/// 2. runs the producer through a real worker and asserts the +/// `result_processor` hook fired and pushed the right jobs with the +/// right trigger metadata, +/// 3. then drives `dispatch_asset_triggers` directly against the same +/// seeded graph to cover the arg-driven and eligibility branches that +/// can't be reached by varying the producer's runtime args alone: +/// - skip arg suppresses dispatch +/// - a subscriber already in the lineage is skipped (cycle guard) +/// - the lineage chain accumulates the producer path each hop +/// - self-loop subscriber is filtered +/// - producer with parent_job is ineligible +/// - producer with `Flow` kind is ineligible +#[sqlx::test(fixtures("base"))] +async fn end_to_end_asset_dispatch(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // ── Seed the graph ────────────────────────────────────────────────── + let producer_hash = seed_script(&db, PRODUCER, "echo producer", "bash").await?; + seed_script(&db, SUB_S3, "echo s3-subscriber", "bash").await?; + seed_script(&db, SUB_RES, "echo res-subscriber", "bash").await?; + // Self-loop subscriber: same path as producer → must be filtered. + seed_subscription(&db, PRODUCER, "script", "s3://f/blob").await?; + // Flow subscriber on the same asset → V1 hard-filters runnable_kind='flow'. + seed_subscription(&db, SUB_FLOW, "flow", "s3://f/blob").await?; + // Legit subscribers. + seed_subscription(&db, SUB_S3, "script", "s3://f/blob").await?; + seed_subscription(&db, SUB_RES, "script", "$res:f/cfg").await?; + // Two writes from one producer, distinct kinds. + seed_asset_write(&db, PRODUCER, "s3object", "f/blob").await?; + seed_asset_write(&db, PRODUCER, "resource", "f/cfg").await?; + + // ── 1. Real worker run: the hook must fire after producer success ─── + let job = JobPayload::ScriptHash { + path: PRODUCER.to_string(), + hash: ScriptHash(producer_hash), + cache_ttl: None, + cache_ignore_s3_path: None, + dedicated_worker: None, + language: ScriptLang::Bash, + priority: None, + apply_preprocessor: false, + concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default(), + debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(), + labels: None, + }; + let completed = RunJob::from(job).run_until_complete(&db, false, port).await; + assert!( + completed.success, + "producer must succeed for dispatch to fire" + ); + + let mut rows = fetch_dispatched(&db).await?; + rows.sort_by(|a, b| a.0.cmp(&b.0)); + assert_eq!( + rows.len(), + 2, + "expected dispatch to the two legit script subscribers (flow sub filtered, self-loop filtered)" + ); + let by_path: std::collections::HashMap<_, _> = rows.iter().map(|r| (r.0.as_str(), r)).collect(); + + let s3_row = by_path + .get(SUB_S3) + .expect("s3 subscriber should have a job"); + let s3_trig = s3_row.2.as_ref().unwrap().get("trigger").unwrap(); + assert_eq!(s3_trig["kind"], "asset"); + assert_eq!(s3_trig["asset_kind"], "s3object"); + assert_eq!(s3_trig["asset_path"], "f/blob"); + assert_eq!(s3_trig["producer_path"], PRODUCER); + assert_eq!( + s3_trig["chain"], + json!([PRODUCER]), + "lineage starts with the producer on the first hop" + ); + assert_eq!(s3_row.1.as_deref(), Some(PRODUCER)); + + let res_row = by_path + .get(SUB_RES) + .expect("resource subscriber should have a job"); + let res_trig = res_row.2.as_ref().unwrap().get("trigger").unwrap(); + assert_eq!(res_trig["asset_kind"], "resource"); + assert_eq!(res_trig["asset_path"], "f/cfg"); + + // ── 2. Direct calls to dispatch_asset_triggers for arg / eligibility + // branches that can't be reached via the runtime path ────────── + clear_dispatched(&db).await?; + + // skip arg suppresses dispatch + let id = seed_producer_job(&db, json!({ "_wmill_skip_asset_dispatch": true })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!(r.dispatched.len(), 0, "skip arg suppressed dispatch"); + + // cycle guard: a subscriber already in the lineage is skipped, but its + // siblings still dispatch (only the cyclic edge is cut). + let id = seed_producer_job(&db, json!({ "trigger": { "chain": [SUB_S3] } })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!( + r.dispatched.len(), + 1, + "cyclic subscriber (already in lineage) skipped; sibling still dispatched" + ); + + // lineage accumulates: a fresh producer extends the chain with its own path + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({ "trigger": { "chain": ["f/upstream"] } })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!(r.dispatched.len(), 2); + let rows = fetch_dispatched(&db).await?; + for row in &rows { + assert_eq!( + row.2.as_ref().unwrap()["trigger"]["chain"], + json!(["f/upstream", PRODUCER]), + "lineage accumulates the producer path" + ); + } + + // flow step (carries flow_step_id) is ineligible + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + let mut mini = make_mini(id, PRODUCER); + mini.flow_step_id = Some("a".to_string()); + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!(r.dispatched.len(), 0, "flow step producer ineligible"); + + // A native retry attempt has a native_retry_attempt marker — it stays eligible, + // so a subscriber that recovers on retry still cascades. + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + sqlx::query("INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, 1)") + .bind(id) + .execute(&db) + .await?; + let mut mini = make_mini(id, PRODUCER); + mini.parent_job = Some(Uuid::new_v4()); + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!( + r.dispatched.len(), + 2, + "native retry attempt (marked) still dispatches" + ); + + // A parented Script child WITHOUT the marker — a schedule handler or a WAC + // inline child (which re-runs the same runnable) — must NOT cascade. + clear_dispatched(&db).await?; + let id = seed_producer_job(&db, json!({})).await?; + let mut mini = make_mini(id, PRODUCER); + mini.parent_job = Some(Uuid::new_v4()); // no marker => not a retry + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!( + r.dispatched.len(), + 0, + "parented child without the marker (handler/WAC inline) does not cascade" + ); + + // producer with kind=Flow is ineligible + let id = seed_producer_job(&db, json!({})).await?; + let mut mini = make_mini(id, PRODUCER); + mini.kind = JobKind::Flow; + let r = dispatch_asset_triggers(&db, &mini).await; + assert_eq!(r.dispatched.len(), 0, "flow producer ineligible"); + + // Sanity: the eligible direct call (clean producer, no args) still fires — + // proves the assertions above are negative cases, not a broken setup. + let id = seed_producer_job(&db, json!({})).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!(r.dispatched.len(), 2, "clean direct call still dispatches"); + + Ok(()) +} + +/// Stage C: a `// partitioned dynamic` producer run through a real worker +/// must (1) resolve the partition off its triggering payload at execution +/// time, (2) persist it back into its own `v2_job.args` so the cascade +/// reads it, and (3) propagate the same value into the dispatched +/// subscriber's args + `trigger.partition`. +#[sqlx::test(fixtures("base"))] +async fn partition_dynamic_resolved_persisted_and_propagated( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Producer declares a dynamic partition keyed off the run payload. + // (Bash uses `#` comments — the annotation parser accepts `#`/`--`/`//`.) + let producer_hash = seed_script( + &db, + PRODUCER, + "# pipeline\n# partitioned dynamic key=\"$.tenant_id\"\necho producer", + "bash", + ) + .await?; + seed_script(&db, SUB_S3, "echo s3-subscriber", "bash").await?; + seed_asset_write(&db, PRODUCER, "s3object", "f/blob").await?; + seed_subscription(&db, SUB_S3, "script", "s3://f/blob").await?; + + let job = JobPayload::ScriptHash { + path: PRODUCER.to_string(), + hash: ScriptHash(producer_hash), + cache_ttl: None, + cache_ignore_s3_path: None, + dedicated_worker: None, + language: ScriptLang::Bash, + priority: None, + apply_preprocessor: false, + concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default(), + debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(), + labels: None, + }; + let completed = RunJob::from(job) + .arg("tenant_id", json!("acme")) + .run_until_complete(&db, false, port) + .await; + assert!(completed.success, "partitioned producer must succeed"); + + // (2) resolved value persisted back into the producer's own args. + let prod = sqlx::query!( + r#"SELECT args AS "args: sqlx::types::Json" + FROM v2_job + WHERE workspace_id = $1 AND runnable_path = $2 AND trigger_kind IS NULL"#, + WS, + PRODUCER, + ) + .fetch_one(&db) + .await?; + assert_eq!( + prod.args.unwrap().0["partition"], + json!("acme"), + "Stage C must persist the resolved partition into v2_job.args" + ); + + // (3) propagated into the dispatched subscriber. + let rows = fetch_dispatched(&db).await?; + let sub = rows + .iter() + .find(|r| r.0 == SUB_S3) + .expect("subscriber must be dispatched"); + let args = sub.2.as_ref().unwrap(); + assert_eq!( + args["partition"], + json!("acme"), + "subscriber gets top-level partition arg" + ); + assert_eq!( + args["trigger"]["partition"], + json!("acme"), + "subscriber gets trigger.partition" + ); + + Ok(()) +} + +/// Stage D: an AND-join subscriber (`// trigger all`) with two +/// partition-bearing inputs must NOT dispatch until both inputs have +/// arrived for the *same* partition; then it fires exactly once. Slots +/// are per-partition and cleared on fire (re-accumulate, no double-fire). +#[sqlx::test(fixtures("base"))] +async fn and_join_waits_for_all_partition_inputs(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + const PROD_A: &str = "u/test-user/prod-a"; + const PROD_B: &str = "u/test-user/prod-b"; + const SUB_J: &str = "u/test-user/sub-join"; + + seed_script(&db, SUB_J, "echo join-subscriber", "bash").await?; + // Two partition-bearing producers, one input each (literal token form). + seed_asset_write(&db, PROD_A, "s3object", "lake/{partition}/a").await?; + seed_asset_write(&db, PROD_B, "s3object", "lake/{partition}/b").await?; + seed_subscription_and(&db, SUB_J, "s3://lake/{partition}/a").await?; + seed_subscription_and(&db, SUB_J, "s3://lake/{partition}/b").await?; + + // Input A for partition "acme" → slot 1/2, must NOT dispatch. + let a_acme = seed_producer_job_path(&db, PROD_A, json!({ "partition": "acme" })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(a_acme, PROD_A)).await; + assert!( + r.dispatched.is_empty(), + "AND join must wait: only 1 of 2 inputs present" + ); + assert!( + fetch_dispatched(&db).await?.is_empty(), + "no subscriber job pushed yet" + ); + + // A different partition for input B must open its OWN slot, not + // complete acme's. + let b_globex = seed_producer_job_path(&db, PROD_B, json!({ "partition": "globex" })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(b_globex, PROD_B)).await; + assert!( + r.dispatched.is_empty(), + "different partition opens a separate slot, does not complete acme" + ); + + // Input B for "acme" → acme slot now 2/2 → dispatch exactly once. + let b_acme = seed_producer_job_path(&db, PROD_B, json!({ "partition": "acme" })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(b_acme, PROD_B)).await; + assert_eq!(r.dispatched.len(), 1, "AND join fires once both inputs in"); + + let rows = fetch_dispatched(&db).await?; + assert_eq!(rows.len(), 1); + let sub = &rows[0]; + assert_eq!(sub.0, SUB_J); + let args = sub.2.as_ref().unwrap(); + assert_eq!(args["partition"], json!("acme")); + assert_eq!(args["trigger"]["partition"], json!("acme")); + + // Slot cleared on fire: re-arrival of A/acme alone is 1/2 again, no + // double-fire. + clear_dispatched(&db).await?; + let a_acme2 = seed_producer_job_path(&db, PROD_A, json!({ "partition": "acme" })).await?; + let r = dispatch_asset_triggers(&db, &make_mini(a_acme2, PROD_A)).await; + assert!( + r.dispatched.is_empty(), + "slot was cleared on fire; single input must not re-fire" + ); + + Ok(()) +} + +/// Stage E3: a subscriber whose edge has a debounce window gets real +/// DebouncingSettings (delay + a (subscriber, partition) key) on the +/// dispatched job; an undebounced subscriber on the same asset gets none +/// (fan-out, unchanged). Asserts the wiring fetch→push→payload→handle; +/// the actual window-collapse is the queue subsystem's own concern. +#[sqlx::test(fixtures("base"))] +async fn debounce_setting_applied_to_dispatched_subscriber( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + seed_script(&db, SUB_S3, "echo debounced", "bash").await?; + seed_script(&db, SUB_RES, "echo plain", "bash").await?; + seed_asset_write(&db, PRODUCER, "s3object", "f/blob").await?; + seed_subscription_debounced(&db, SUB_S3, "s3://f/blob", 30).await?; + seed_subscription(&db, SUB_RES, "script", "s3://f/blob").await?; + + let id = seed_producer_job(&db, json!({})).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!(r.dispatched.len(), 2, "both subscribers dispatched"); + + // Resolve the persisted debounce window straight from this test's own + // (isolated) DB by walking the handle chain + // v2_job_queue.runnable_settings_handle → runnable_settings.debouncing_settings + // → debouncing_settings. Reading the rows directly rather than through + // `prefetch_cached_from_handle` keeps the assertion off the process-global + // runnable-settings cache (and its tempdir-backed file I/O), which is + // shared by every test running concurrently in this binary — a needless + // cross-test coupling for what is purely a "was the handle wired through to + // the queued job" check. An undebounced subscriber has a NULL handle, so + // the inner joins yield no row → (None, None). + async fn debounce_of( + db: &Pool, + path: &str, + ) -> anyhow::Result<(Option, Option)> { + use sqlx::Row; + let row = sqlx::query( + r#"SELECT ds.debounce_delay_s, ds.debounce_key + FROM v2_job j + JOIN v2_job_queue q ON q.id = j.id + JOIN runnable_settings rs ON rs.hash = q.runnable_settings_handle + JOIN debouncing_settings ds ON ds.hash = rs.debouncing_settings + WHERE j.workspace_id = $1 AND j.runnable_path = $2 + AND j.trigger_kind = 'asset'"#, + ) + .bind(WS) + .bind(path) + .fetch_optional(db) + .await?; + Ok(match row { + Some(r) => ( + r.try_get::, _>("debounce_delay_s")?, + r.try_get::, _>("debounce_key")?, + ), + None => (None, None), + }) + } + + let (deb_delay, deb_key) = debounce_of(&db, SUB_S3).await?; + assert_eq!(deb_delay, Some(30), "debounced edge → 30s window"); + assert!( + deb_key + .as_deref() + .is_some_and(|k| k.starts_with("asset-cascade:")), + "debounce key is scoped to the (subscriber, partition) cascade slot, got {deb_key:?}" + ); + + let (plain_delay, _) = debounce_of(&db, SUB_RES).await?; + assert_eq!( + plain_delay, None, + "undebounced edge → no debounce (fan-out)" + ); + + Ok(()) +} + +/// `// retry []` opts the subscriber into native retry: the +/// dispatcher pushes a real `JobKind::Script` (not a one-step flow) carrying +/// the policy in its `runnable_settings_handle`, so a failed subscriber re-runs +/// natively and stays eligible to trigger its own downstream. Subscribers +/// without retry are pushed as a plain `Script` with no settings handle. +#[sqlx::test(fixtures("base"))] +async fn retry_setting_dispatches_subscriber_as_native_script( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + seed_script(&db, SUB_S3, "echo retrying", "bash").await?; + seed_script(&db, SUB_RES, "echo plain", "bash").await?; + seed_asset_write(&db, PRODUCER, "s3object", "f/blob").await?; + // Retry policy on the s3 edge; res edge stays vanilla so we also assert the + // "no retry" path carries no settings handle. + seed_subscription_with_retry(&db, SUB_S3, "s3://f/blob", 3, 5).await?; + seed_subscription(&db, SUB_RES, "script", "s3://f/blob").await?; + + let id = seed_producer_job(&db, json!({})).await?; + let r = dispatch_asset_triggers(&db, &make_mini(id, PRODUCER)).await; + assert_eq!(r.dispatched.len(), 2, "both subscribers dispatched"); + + let rows: Vec<(String, String, Option)> = sqlx::query!( + r#"SELECT j.runnable_path AS "runnable_path!", j.kind::text AS "kind!", + q.runnable_settings_handle + FROM v2_job j JOIN v2_job_queue q ON q.id = j.id + WHERE j.workspace_id = $1 AND j.trigger_kind = 'asset' + ORDER BY j.runnable_path"#, + WS, + ) + .fetch_all(&db) + .await? + .into_iter() + .map(|r| (r.runnable_path, r.kind, r.runnable_settings_handle)) + .collect(); + + let s3 = rows + .iter() + .find(|(p, _, _)| p == SUB_S3) + .expect("s3 dispatched"); + let res = rows + .iter() + .find(|(p, _, _)| p == SUB_RES) + .expect("res dispatched"); + + // Native retry: a real Script (not a SingleStepFlow), with the policy in the + // runnable_settings_handle. + assert_eq!( + s3.1, "script", + "retry subscriber dispatched as a native Script" + ); + assert!( + s3.2.is_some(), + "retry subscriber carries a runnable_settings_handle (the retry policy)" + ); + assert_eq!( + res.1, "script", + "no-retry subscriber stays a plain ScriptHash push" + ); + assert!( + res.2.is_none(), + "no-retry subscriber carries no settings handle" + ); + + Ok(()) +} + +/// Regression for the AND-join check-then-act race: when a subscriber's +/// last partition-bearing inputs complete concurrently (different workers +/// finishing different upstream producers at once), the barrier must +/// still fire the subscriber exactly once for the partition. Fires all N +/// producers' dispatch simultaneously (a barrier releases them together) +/// and asserts a single dispatch and a cleared slot. This invariant holds +/// for the transactional, advisory-locked gate regardless of interleaving; +/// a regression to a non-atomic check-then-act fails it. +#[sqlx::test(fixtures("base"))] +async fn and_join_fires_once_under_concurrent_completion(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + const SUB_J: &str = "u/test-user/sub-join-conc"; + const N: usize = 5; + + seed_script(&db, SUB_J, "echo join", "bash").await?; + let mut producers = Vec::new(); + for i in 0..N { + let prod = format!("u/test-user/prod-conc-{i}"); + seed_asset_write(&db, &prod, "s3object", &format!("lake/{{partition}}/i{i}")).await?; + seed_subscription_and(&db, SUB_J, &format!("s3://lake/{{partition}}/i{i}")).await?; + let id = seed_producer_job_path(&db, &prod, json!({ "partition": "acme" })).await?; + producers.push((id, prod)); + } + + let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(N)); + let mut set = tokio::task::JoinSet::new(); + for (id, prod) in producers { + let db = db.clone(); + let barrier = barrier.clone(); + set.spawn(async move { + barrier.wait().await; + dispatch_asset_triggers(&db, &make_mini(id, &prod)) + .await + .dispatched + .len() + }); + } + let mut total = 0usize; + while let Some(r) = set.join_next().await { + total += r?; + } + + assert_eq!( + total, 1, + "AND join must dispatch the subscriber exactly once under concurrent completion" + ); + let fires = fetch_dispatched(&db) + .await? + .iter() + .filter(|r| r.0 == SUB_J) + .count(); + assert_eq!(fires, 1, "exactly one subscriber job pushed"); + + let leftover = sqlx::query_scalar!( + r#"SELECT count(*) AS "n!" + FROM join_pending_inputs + WHERE workspace_id = $1 AND subscriber_path = $2"#, + WS, + SUB_J, + ) + .fetch_one(&db) + .await?; + assert_eq!(leftover, 0, "join slot cleared after fire"); + + Ok(()) +} + +/// Fuller pipeline: a partitioned chain that fans in through an AND-join +/// and then fans out over several more hops. Asserts the resolved +/// partition propagates unchanged at every hop, the chain depth +/// increments per hop, the AND barrier fires once, and a different +/// partition opens an independent slot (no cross-partition bleed) across +/// the whole multi-hop graph. +/// +/// Shape: A,B (partitioned producers) ─┐ +/// ├─▶ J (// trigger all) ─▶ C ─▶ D +/// A,B ─────────────────────────┘ +#[sqlx::test(fixtures("base"))] +async fn fuller_partitioned_join_multihop_pipeline(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + const PA: &str = "u/test-user/p-a"; + const PB: &str = "u/test-user/p-b"; + const JN: &str = "u/test-user/p-join"; + const CN: &str = "u/test-user/p-c"; + const DN: &str = "u/test-user/p-d"; + + for p in [JN, CN, DN] { + seed_script(&db, p, "echo step", "bash").await?; + } + seed_asset_write(&db, PA, "s3object", "lake/{partition}/a").await?; + seed_asset_write(&db, PB, "s3object", "lake/{partition}/b").await?; + seed_asset_write(&db, JN, "s3object", "lake/{partition}/j").await?; + seed_asset_write(&db, CN, "s3object", "lake/{partition}/c").await?; + // J is an AND join over both partition-bearing inputs. + seed_subscription_and(&db, JN, "s3://lake/{partition}/a").await?; + seed_subscription_and(&db, JN, "s3://lake/{partition}/b").await?; + seed_subscription(&db, CN, "script", "s3://lake/{partition}/j").await?; + seed_subscription(&db, DN, "script", "s3://lake/{partition}/c").await?; + + // Helper: assert exactly one dispatch to `path` carrying partition + // `part`, with a cascade lineage of `chain_len` producer paths. + async fn assert_hop( + db: &Pool, + path: &str, + part: &str, + chain_len: usize, + ) -> anyhow::Result<()> { + let rows = fetch_dispatched(db).await?; + let hits: Vec<_> = rows.iter().filter(|r| r.0 == path).collect(); + assert_eq!(hits.len(), 1, "expected exactly one dispatch to {path}"); + let args = hits[0].2.as_ref().unwrap(); + assert_eq!(args["partition"], json!(part), "{path} top-level partition"); + assert_eq!( + args["trigger"]["partition"], + json!(part), + "{path} trigger.partition" + ); + assert_eq!( + args["trigger"]["chain"].as_array().map(|c| c.len()), + Some(chain_len), + "{path} lineage length" + ); + Ok(()) + } + + // day1: A arrives → J waits (1/2 partition-bearing inputs). + let pa = seed_producer_job_path( + &db, + PA, + json!({ "partition": "day1", "trigger": { "chain": ["s0"] } }), + ) + .await?; + let r = dispatch_asset_triggers(&db, &make_mini(pa, PA)).await; + assert!(r.dispatched.is_empty(), "J must wait: only A present"); + assert!(fetch_dispatched(&db).await?.is_empty()); + + // day1: B arrives → J fires once for day1 at depth 2. + let pb = seed_producer_job_path( + &db, + PB, + json!({ "partition": "day1", "trigger": { "chain": ["s0"] } }), + ) + .await?; + let r = dispatch_asset_triggers(&db, &make_mini(pb, PB)).await; + assert_eq!(r.dispatched.len(), 1, "J fires once when both inputs in"); + assert_hop(&db, JN, "day1", 2).await?; + clear_dispatched(&db).await?; + + // J completes for day1 → C runs for day1 at depth 3. + let jn = seed_producer_job_path( + &db, + JN, + json!({ "partition": "day1", "trigger": { "chain": ["s0", PB] } }), + ) + .await?; + let r = dispatch_asset_triggers(&db, &make_mini(jn, JN)).await; + assert_eq!(r.dispatched.len(), 1); + assert_hop(&db, CN, "day1", 3).await?; + clear_dispatched(&db).await?; + + // C completes for day1 → D (leaf) runs for day1 at depth 4. + let cn = seed_producer_job_path( + &db, + CN, + json!({ "partition": "day1", "trigger": { "chain": ["s0", PB, JN] } }), + ) + .await?; + let r = dispatch_asset_triggers(&db, &make_mini(cn, CN)).await; + assert_eq!(r.dispatched.len(), 1); + assert_hop(&db, DN, "day1", 4).await?; + clear_dispatched(&db).await?; + + // A different partition opens an independent J slot — no bleed from + // the completed day1 run. + let pa2 = seed_producer_job_path( + &db, + PA, + json!({ "partition": "day2", "trigger": { "chain": ["s0"] } }), + ) + .await?; + let r = dispatch_asset_triggers(&db, &make_mini(pa2, PA)).await; + assert!( + r.dispatched.is_empty(), + "day2 is a separate slot; J must not fire from day1's completion" + ); + + Ok(()) +} + +/// The TTL reaper deletes abandoned AND-join slots, but keyed on the +/// slot's MOST RECENT row: a slot still receiving input (newest row +/// fresh) is never reaped even if it also has rows older than the TTL. +/// This per-slot (not per-row) property is the correctness point — it +/// prevents corrupting a join whose inputs trickle in slowly. +#[sqlx::test(fixtures("base"))] +async fn reaper_clears_only_stale_join_slots(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + // Seed a join_pending_inputs row with an explicit age (days old). + async fn seed_slot_row( + db: &Pool, + sub: &str, + part: &str, + tref: &str, + age_days: i64, + ) -> anyhow::Result<()> { + sqlx::query( + r#"INSERT INTO join_pending_inputs + (workspace_id, subscriber_path, partition, trigger_ref, received_at) + VALUES ($1, $2, $3, $4, now() - ($5::bigint::text || ' d')::interval)"#, + ) + .bind(WS) + .bind(sub) + .bind(part) + .bind(tref) + .bind(age_days) + .execute(db) + .await?; + Ok(()) + } + async fn slot_count(db: &Pool, sub: &str) -> anyhow::Result { + Ok(sqlx::query_scalar!( + r#"SELECT count(*) AS "n!" FROM join_pending_inputs + WHERE workspace_id = $1 AND subscriber_path = $2"#, + WS, + sub, + ) + .fetch_one(db) + .await?) + } + + // Stale: every row older than the 60d TTL → reaped. + seed_slot_row( + &db, + "u/test-user/sub-stale", + "p1", + "s3://x/{partition}/a", + 61, + ) + .await?; + seed_slot_row( + &db, + "u/test-user/sub-stale", + "p1", + "s3://x/{partition}/b", + 90, + ) + .await?; + // Fresh: recent → kept. + seed_slot_row( + &db, + "u/test-user/sub-fresh", + "p1", + "s3://y/{partition}/a", + 0, + ) + .await?; + // Mixed: one ancient row + one fresh row in the SAME slot. max(received_at) + // is fresh, so the whole slot must be kept (the correctness property). + seed_slot_row( + &db, + "u/test-user/sub-mixed", + "p1", + "s3://z/{partition}/a", + 120, + ) + .await?; + seed_slot_row( + &db, + "u/test-user/sub-mixed", + "p1", + "s3://z/{partition}/b", + 0, + ) + .await?; + + reap_stale_join_slots(&db).await?; + + assert_eq!( + slot_count(&db, "u/test-user/sub-stale").await?, + 0, + "stale slot reaped" + ); + assert_eq!( + slot_count(&db, "u/test-user/sub-fresh").await?, + 1, + "fresh slot kept" + ); + assert_eq!( + slot_count(&db, "u/test-user/sub-mixed").await?, + 2, + "slot with a recent row must be kept entirely (per-slot, not per-row)" + ); + + Ok(()) +} diff --git a/backend/tests/batch_rerun.rs b/backend/tests/batch_rerun.rs index b750379834..b1708465f9 100644 --- a/backend/tests/batch_rerun.rs +++ b/backend/tests/batch_rerun.rs @@ -71,6 +71,7 @@ fn ssf_script_payload(hash: Option, retry: Option) -> JobPayl path: SCRIPT_PATH.to_string(), hash, flow_version: None, + language: None, args: Default::default(), retry, error_handler_path: None, @@ -92,6 +93,7 @@ fn ssf_flow_payload() -> JobPayload { path: FLOW_PATH.to_string(), hash: None, flow_version: Some(FLOW_VERSION), + language: None, args: Default::default(), retry: None, error_handler_path: None, diff --git a/backend/tests/ci_tests.rs b/backend/tests/ci_tests.rs index c8551a30a5..46bed97094 100644 --- a/backend/tests/ci_tests.rs +++ b/backend/tests/ci_tests.rs @@ -10,6 +10,7 @@ mod ci_tests { fn quick_ns(content: &str, path: &str, parent_hash: Option) -> NewScript { NewScript { + draft_only: None, content: content.into(), language: ScriptLang::Python3, lock: None, @@ -20,7 +21,6 @@ mod ci_tests { cache_ttl: None, dedicated_worker: None, description: "".to_string(), - draft_only: None, envs: vec![], is_template: None, kind: None, diff --git a/backend/tests/debounce_e2e.rs b/backend/tests/debounce_e2e.rs new file mode 100644 index 0000000000..9a2351f135 --- /dev/null +++ b/backend/tests/debounce_e2e.rs @@ -0,0 +1,112 @@ +// End-to-end debounce test: drives the FULL real path — real `push()` (EE +// `maybe_debounce` collapsing the batch), real `pull()`, real +// `maybe_apply_debouncing` (claim + accumulate), and a real worker executing the +// surviving flow — then asserts the executed result contains every accumulated item. +// Runs on --features deno_core,enterprise,private (debounce is EE/compile-gated). +#[cfg(all(feature = "deno_core", feature = "enterprise", feature = "private"))] +mod debounce_e2e { + use serde_json::json; + use sqlx::{Pool, Postgres}; + use uuid::Uuid; + use windmill_common::flows::FlowValue; + use windmill_common::jobs::JobPayload; + use windmill_common::worker::Connection; + use windmill_test_utils::*; + + async fn initialize_tracing() { + use std::sync::Once; + static ONCE: Once = Once::new(); + ONCE.call_once(|| { + let _ = windmill_common::tracing_init::initialize_tracing( + "test", + &windmill_common::utils::Mode::Standalone, + "test", + ); + }); + } + + /// A one-step flow that debounces on `items` and returns `flow_input.items`, + /// so the flow's result is exactly the accumulated batch. + fn debounce_flow() -> FlowValue { + serde_json::from_value(json!({ + "modules": [{ + "id": "a", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main(items: any[]) { return items }", + "input_transforms": { + "items": { "type": "javascript", "expr": "flow_input.items" } + } + } + }], + "debounce_delay_s": 1, + "debounce_key": "e2e_debounce_key", + "debounce_args_to_accumulate": ["items"] + })) + .expect("valid flow value") + } + + /// Fire three same-key messages; only the survivor runs, and it must execute once + /// with ALL accumulated items (none dropped, none duplicated). + #[sqlx::test(fixtures("base"))] + async fn debounce_accumulation_runs_once_with_all_items( + db: Pool, + ) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Push 3 debounced flow jobs (same key) — each collapses the previous; the last + // is the survivor. scheduled_for is ~1s out, so all 3 land before any worker pull. + let mut survivor = Uuid::nil(); + let mut superseded = Vec::new(); + for n in [1i64, 2, 3] { + if survivor != Uuid::nil() { + superseded.push(survivor); + } + survivor = RunJob::from(JobPayload::RawFlow { + value: debounce_flow(), + path: None, + restarted_from: None, + }) + .arg("items", json!([n])) + .push(&db) + .await; + } + + // Run a real worker until the survivor completes. + let listener = listen_for_completed_jobs(&db).await; + in_test_worker(Connection::Sql(db.clone()), listener.find(&survivor), port).await; + + let cj = completed_job(survivor, &db).await; + assert!(cj.success, "survivor flow must succeed"); + let mut result: Vec = + serde_json::from_value(cj.json_result().expect("result present")) + .expect("result is an array of numbers"); + result.sort(); + assert_eq!( + result, + vec![1, 2, 3], + "survivor must execute exactly once with ALL accumulated items" + ); + + // The two superseded messages must have been debounced (skipped), not run. + // (use a lightweight status query — skipped jobs have NULL started_at, which the + // full CompletedJob row decoder rejects) + for s in superseded { + let skipped: Option = + sqlx::query_scalar("SELECT status = 'skipped' FROM v2_job_completed WHERE id = $1") + .bind(s) + .fetch_optional(&db) + .await?; + assert_eq!( + skipped, + Some(true), + "superseded message {s} must be debounced (skipped), not executed" + ); + } + + Ok(()) + } +} diff --git a/backend/tests/dependency_map.rs b/backend/tests/dependency_map.rs index 48a50ef9b9..0d58615be0 100644 --- a/backend/tests/dependency_map.rs +++ b/backend/tests/dependency_map.rs @@ -16,6 +16,7 @@ mod dependency_map { parent_hash: Option, ) -> NewScript { NewScript { + draft_only: None, content: content.into(), language, lock, @@ -26,7 +27,6 @@ mod dependency_map { cache_ttl: None, dedicated_worker: None, description: "".to_string(), - draft_only: None, envs: vec![], is_template: None, kind: None, @@ -441,7 +441,6 @@ def main(): .unwrap(), ), schema: None, - draft_only: None, tag: None, dedicated_worker: None, timeout: None, diff --git a/backend/tests/drafts_nul.rs b/backend/tests/drafts_nul.rs new file mode 100644 index 0000000000..d7314b8fb6 --- /dev/null +++ b/backend/tests/drafts_nul.rs @@ -0,0 +1,80 @@ +//! Regression test for NUL bytes in draft values. +//! +//! `draft.value` is a `json` column (not `jsonb`), so a U+0000 escape can be +//! stored and then make any `->>`/`to_jsonb` extraction raise `22P05` — one +//! poisoned draft 500'd `GET /drafts/list` (silently hiding the home-page +//! "This workspace has N drafts" banner). The fix sanitizes the value on write +//! (`update_draft` -> `strip_json_nul`) so a NUL never reaches the column; this +//! drives the real endpoint and asserts the stored + listed value is NUL-free. + +use serde_json::{json, Value}; +use sqlx::{Pool, Postgres}; + +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(b: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + b.header("Authorization", "Bearer DNUL_ADMIN_TOKEN") +} + +#[sqlx::test(fixtures("drafts_nul"))] +async fn test_draft_write_strips_nul(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/dnul-ws"); + + // Save a draft whose summary and content carry a real NUL. + let resp = authed(client().post(format!( + "{base}/drafts/update/script/u/dnul-admin/poison" + ))) + .json(&json!({ + "value": { + "summary": "hi\u{0}there", + "path": "u/dnul-admin/poison", + "content": "x\u{0}y" + } + })) + .send() + .await?; + assert_eq!( + resp.status(), + 200, + "save should succeed: {}", + resp.text().await.unwrap_or_default() + ); + + // The stored value must be NUL-free (sanitized on write). + let stored: Value = authed(client().get(format!( + "{base}/drafts/get_own/script/u/dnul-admin/poison" + ))) + .send() + .await? + .json() + .await?; + let value = stored.get("value").expect("draft should exist"); + assert_eq!(value["summary"], "hithere"); + assert_eq!(value["content"], "xy"); + assert!( + !serde_json::to_string(value).unwrap().contains("\\u0000"), + "stored value still contains a NUL escape: {value}" + ); + + // The list endpoint uses raw `->>`; it works (200, no 500) because the + // stored data is clean, and the summary comes back stripped. + let items: Vec = authed(client().get(format!("{base}/drafts/list"))) + .send() + .await? + .json() + .await?; + let item = items + .iter() + .find(|d| d["path"] == "u/dnul-admin/poison") + .expect("saved draft should be listed"); + assert_eq!(item["summary"], "hithere"); + + Ok(()) +} diff --git a/backend/tests/fixtures/app_preview_auth.sql b/backend/tests/fixtures/app_preview_auth.sql index 9fcda61c51..b94e08f3fd 100644 --- a/backend/tests/fixtures/app_preview_auth.sql +++ b/backend/tests/fixtures/app_preview_auth.sql @@ -32,3 +32,11 @@ INSERT INTO app (id, workspace_id, path, summary, policy, versions) VALUES (999002, 'test-workspace', 'u/test-user-2/ownapp', 'own app', '{}'::jsonb, '{}'); INSERT INTO app_script (id, app, hash, code, code_sha256) VALUES (999778, 999002, repeat('c', 64), 'export function main(){ return "ok" }', repeat('d', 64)); + +-- A deployed empty Viewer-mode app owned by `test-user` with NO runnables pinned +-- in `triggerables_v2`. Used to assert run mode rejects caller-supplied inline +-- `raw_code` whose sha is not publisher-pinned (the CVE-2026-22683 residual: +-- the Viewer default fallback let any caller / an operator run arbitrary code). +INSERT INTO app (id, workspace_id, path, summary, policy, versions) VALUES + (999003, 'test-workspace', 'u/test-user/vapp', 'empty viewer app', + '{"execution_mode": "viewer", "triggerables_v2": {}}'::jsonb, '{}'); diff --git a/backend/tests/fixtures/drafts_nul.sql b/backend/tests/fixtures/drafts_nul.sql new file mode 100644 index 0000000000..8f457ffbc3 --- /dev/null +++ b/backend/tests/fixtures/drafts_nul.sql @@ -0,0 +1,24 @@ +-- Fixture for the draft NUL-byte write-sanitization regression test. +-- Just a workspace + admin user + token; the test itself POSTs a draft whose +-- value carries a U+0000 and asserts it is stored (and listed) NUL-free. + +INSERT INTO workspace (id, name, owner) VALUES + ('dnul-ws', 'DNUL WS', 'dnul-admin'); + +INSERT INTO workspace_key (workspace_id, kind, key) VALUES + ('dnul-ws', 'cloud', 'dnul-key'); + +INSERT INTO workspace_settings (workspace_id) VALUES + ('dnul-ws'); + +INSERT INTO group_ (workspace_id, name, summary, extra_perms) VALUES + ('dnul-ws', 'all', 'All users', '{}'); + +INSERT INTO password(email, password_hash, login_type, super_admin, verified, name, username) + VALUES ('dnul-admin@windmill.dev', 'x', 'password', true, true, 'DNUL Admin', 'dnul-admin'); + +INSERT INTO usr(workspace_id, email, username, is_admin, role) VALUES + ('dnul-ws', 'dnul-admin@windmill.dev', 'dnul-admin', true, 'Admin'); + +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin) + VALUES (encode(sha256('DNUL_ADMIN_TOKEN'::bytea), 'hex'), 'DNUL_ADMIN', 'DNUL_ADMIN_TOKEN', 'dnul-admin@windmill.dev', 't', true); diff --git a/backend/tests/fixtures/inline_preview_auth.sql b/backend/tests/fixtures/inline_preview_auth.sql new file mode 100644 index 0000000000..59fe2dc917 --- /dev/null +++ b/backend/tests/fixtures/inline_preview_auth.sql @@ -0,0 +1,14 @@ +-- Fixture for the inline preview authorization regression test (GHSA-pp5h-96x3-3wqq). +-- Layered on top of `base` (which provides test-workspace and the non-operator +-- `test-user-2`/SECRET_TOKEN_2). Adds an Operator member so we can assert that +-- Operators cannot reach the arbitrary-code inline preview path +-- (`POST /jobs/run_inline/preview`). + +INSERT INTO password(email, password_hash, login_type, super_admin, verified, name) + VALUES ('operator@windmill.dev', 'not-a-real-hash', 'password', false, true, 'Operator User'); + +INSERT INTO usr(workspace_id, email, username, is_admin, operator, role) VALUES + ('test-workspace', 'operator@windmill.dev', 'operator-user', false, true, 'Operator'); + +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin) VALUES + (encode(sha256('OPERATOR_TOKEN'::bytea), 'hex'), 'OPERATOR_T', 'OPERATOR_TOKEN', 'operator@windmill.dev', 'operator token', false); diff --git a/backend/tests/fixtures/jobs_read_auth.sql b/backend/tests/fixtures/jobs_read_auth.sql index e6b28fba0c..456ac8c801 100644 --- a/backend/tests/fixtures/jobs_read_auth.sql +++ b/backend/tests/fixtures/jobs_read_auth.sql @@ -19,6 +19,45 @@ INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, sc ARRAY['jobs:read', 'if_jobs:filter_tags:deno'] ); +-- App embed token for the admin viewer (test-user). Mirrors a minted sandboxed +-- low-code app token: carries the `app_embed` sentinel plus the embed scope set. +-- Used to assert the token is confined to jobs the viewer LAUNCHED, not every job +-- the (admin) viewer could otherwise read. +INSERT INTO token(token_hash, token_prefix, token, email, label, super_admin, scopes) VALUES ( + encode(sha256('EMBED_APP_TOKEN'::bytea), 'hex'), 'EMBED_APP_', 'EMBED_APP_TOKEN', + 'test@windmill.dev', 'app embed token', false, + ARRAY['apps:run', 'jobs:read', 'app_embed', 'resources:run', 'users:read', 'folders:read'] +); + +-- A completed app-component job LAUNCHED BY the admin viewer (created_by = +-- test-user), running as the app owner. The embed token must keep reading its own +-- launched job (the `created_by == viewer` fast path). +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner, args +) VALUES ( + '12121212-1212-1212-1212-121212121212', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false, + '{"own": "arg"}' +); +INSERT INTO public.v2_job_completed (id, workspace_id, duration_ms, status, result) VALUES + ('12121212-1212-1212-1212-121212121212', 'test-workspace', 1000, 'success'::job_status, + '{"own": "EMBED_OWN_RESULT"}'); + +-- A QUEUED job launched by the admin embed viewer (created_by = test-user). The +-- embed token may cancel its own launched job; it must NOT cancel another user's. +INSERT INTO public.v2_job ( + id, workspace_id, created_by, created_at, permissioned_as, permissioned_as_email, + kind, script_lang, runnable_path, tag, visible_to_owner +) VALUES ( + '13131313-1313-1313-1313-131313131313', 'test-workspace', 'test-user', + '2023-01-01 00:00:00', 'u/test-user-2', 'test2@windmill.dev', + 'script', 'deno', 'u/test-user-2/app_component', 'deno', false +); +INSERT INTO public.v2_job_queue (id, workspace_id, scheduled_for, running, tag) VALUES + ('13131313-1313-1313-1313-131313131313', 'test-workspace', '2023-01-01 00:00:00', false, 'deno'); + -- RUNNING job: queued (no completed row) and owned by test-user-2. Used to check -- that `completed/get_result_maybe?get_started=true` authorizes before disclosing -- running-state to a non-reader. diff --git a/backend/tests/fixtures/mcp_resource_authz.sql b/backend/tests/fixtures/mcp_resource_authz.sql new file mode 100644 index 0000000000..ecf3f01dc3 --- /dev/null +++ b/backend/tests/fixtures/mcp_resource_authz.sql @@ -0,0 +1,25 @@ +-- Fixture for the MCP resource-authorization regression test (WIN-2041, +-- GHSA-7qg3-pr4g-cq5x). +-- +-- Models a low-privileged developer (test-user-3, a plain workspace member) who +-- references, from an AI Agent flow, an MCP resource living in a private folder +-- they have NO access to. The AI agent worker must refuse to load that resource +-- because the developer's identity lacks resources:read on it. + +-- Private folder the developer cannot read (empty extra_perms, owned by admin). +INSERT INTO folder (workspace_id, name, display_name, owners, extra_perms, created_by) +VALUES ('test-workspace', 'private', 'Private Folder', '{"u/test-user"}', '{}', 'test-user'); + +-- MCP resource the developer is NOT allowed to read. The URL is a non-resolvable +-- public host so that, for an authorized caller, resolution succeeds but the +-- later connection step fails deterministically without network access. +INSERT INTO resource (workspace_id, path, value, description, resource_type, extra_perms, created_by) +VALUES ( + 'test-workspace', + 'f/private/admin_mcp', + '{"name": "admin_mcp", "url": "https://mcp.invalid.windmill.test"}', + 'Private MCP resource only admin can read', + 'mcp', + '{}', + 'test-user' +); diff --git a/backend/tests/inline_preview_auth.rs b/backend/tests/inline_preview_auth.rs new file mode 100644 index 0000000000..97b70fb545 --- /dev/null +++ b/backend/tests/inline_preview_auth.rs @@ -0,0 +1,90 @@ +//! Regression test for the inline preview authorization bypass (GHSA-pp5h-96x3-3wqq). +//! +//! `POST /api/w/:workspace/jobs/run_inline/preview` -> `run_inline_preview_script` +//! runs request-supplied code inline (in-process via DuckDB), i.e. it is an +//! arbitrary-code-execution sibling of `/jobs/run/preview`. The bug was that +//! this handler was missing the Operator guard that `run_preview_script` +//! enforces, so an authenticated Operator (a run-only user who must not be able +//! to run preview jobs) could execute arbitrary code in a single request. This +//! was the incomplete-fix residual of CVE-2026-22683, whose v1.615.0 patch only +//! covered the entity-CRUD endpoints and left this direct inline-exec sink open. +//! +//! This test pins down: +//! - an Operator is rejected by the operator guard (the core fix; pre-fix this +//! reached the inline executor instead of returning 401), and +//! - a regular non-operator passes the guard (the fix must not over-block the +//! legitimate inline preview flow): in the test harness the worker inline +//! utils are not registered, so a caller past the guard gets the distinct +//! "worker inline functions" error rather than the operator rejection. + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder { + builder.header("Authorization", format!("Bearer {}", token)) +} + +/// An inline preview request: request-supplied `content` to run via DuckDB. +fn inline_preview_body() -> serde_json::Value { + json!({ + "language": "duckdb", + "content": "SELECT content FROM read_text(['/etc/passwd']);", + "args": {} + }) +} + +const OPERATOR_GUARD_MSG: &str = "Operators cannot run preview jobs"; + +#[sqlx::test(fixtures("base", "inline_preview_auth"))] +async fn test_inline_preview_authorization(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let url = format!("http://localhost:{port}/api/w/test-workspace/jobs/run_inline/preview"); + + // 1. CORE REGRESSION: an Operator must be rejected by the operator guard. + // Pre-fix this fell through to the inline executor (arbitrary code + // execution); post-fix it returns 401 with the operator guard message. + let resp = authed(client().post(&url), "OPERATOR_TOKEN") + .json(&inline_preview_body()) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_eq!( + status, 401, + "Operator must be rejected from inline preview (got {status}): {body}" + ); + assert!( + body.contains(OPERATOR_GUARD_MSG), + "rejection must be the operator guard, got: {body}" + ); + + // 2. The fix must NOT over-block a legitimate non-operator: a regular member + // passes the operator + scope checks. The test harness does not register + // the worker inline utils, so the request proceeds past the guard and + // fails later with the distinct "worker inline functions" error — proving + // the operator guard did not reject it. + let resp = authed(client().post(&url), "SECRET_TOKEN_2") + .json(&inline_preview_body()) + .send() + .await?; + let status = resp.status(); + let body = resp.text().await?; + assert_ne!( + status, 401, + "non-operator must not be blocked by the operator guard (got {status}): {body}" + ); + assert!( + !body.contains(OPERATOR_GUARD_MSG), + "non-operator must not hit the operator guard, got: {body}" + ); + + Ok(()) +} diff --git a/backend/tests/jobs_read_auth.rs b/backend/tests/jobs_read_auth.rs index f19daf0f92..1eab0bffee 100644 --- a/backend/tests/jobs_read_auth.rs +++ b/backend/tests/jobs_read_auth.rs @@ -38,6 +38,10 @@ const TOP_SECRET_FLOW: &str = "ffffffff-ffff-ffff-ffff-ffffffffffff"; const DEEP_LEAF_JOB: &str = "88888888-8888-8888-8888-888888888888"; // A queued/running job (no completed row) owned by test-user-2. const RUNNING_JOB: &str = "77777777-7777-7777-7777-777777777777"; +// An app-component job launched BY the admin embed viewer (created_by test-user). +const EMBED_OWN_JOB: &str = "12121212-1212-1212-1212-121212121212"; +// A QUEUED job launched by the embed viewer (created_by test-user) — cancelable by it. +const EMBED_OWN_QUEUED: &str = "13131313-1313-1313-1313-131313131313"; // Secrets that must never leak to an unauthorized viewer. const RESULT_SECRET: &str = "RESULT_SECRET"; @@ -59,6 +63,19 @@ async fn get(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCod (status, body) } +async fn post(base: &str, path: &str, token: Option<&str>) -> (reqwest::StatusCode, String) { + let mut req = client() + .post(format!("{base}/{path}")) + .json(&serde_json::json!({})); + if let Some(token) = token { + req = req.header("Authorization", format!("Bearer {token}")); + } + let resp = req.send().await.expect("request"); + let status = resp.status(); + let body = resp.text().await.expect("body"); + (status, body) +} + #[sqlx::test(fixtures("base", "jobs_read_auth"))] async fn test_single_job_read_authorization(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; @@ -88,6 +105,10 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul format!("get_completed_logs_tail/{VICTIM}"), ), ("get_flow_all_logs", format!("get_flow_all_logs/{VICTIM}")), + ( + "get_flow_all_logs_structured", + format!("get_flow_all_logs_structured/{VICTIM}"), + ), ( "completed/get_timing", format!("completed/get_timing/{VICTIM}"), @@ -281,6 +302,75 @@ async fn test_single_job_read_authorization(db: Pool) -> anyhow::Resul "top flow in an unreadable folder must stay denied (got {status}): {body}" ); + // ---- APP EMBED TOKEN: confined to jobs the viewer LAUNCHED, not everything + // the (admin) viewer can otherwise read. The token carries the `app_embed` + // sentinel; an admin's normal token reads VICTIM (asserted above), but the + // embed token must stop at the `created_by == viewer` grant so user-authored + // app JS can't reuse it to read unrelated jobs by UUID. + // Its own launched component job (created_by == viewer) still reads. + let (status, body) = get( + &base, + &format!("completed/get_result/{EMBED_OWN_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must read a job it launched (got {status}): {body}" + ); + assert!( + body.contains("EMBED_OWN_RESULT"), + "embed token should get its own launched job result: {body}" + ); + // The VICTIM job — created by another user but readable by this admin viewer's + // normal token (asserted above) — is denied to the embed token across result / + // logs / live update. NotFound (not 403) so the untrusted app can't even probe + // existence, and no secret leaks. + for path in [ + format!("completed/get_result/{VICTIM}"), + format!("get_logs/{VICTIM}"), + format!("getupdate/{VICTIM}?only_result=true"), + ] { + let (status, body) = get(&base, &path, Some("EMBED_APP_TOKEN")).await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not read a job it did not launch ({path}, got {status}): {body}" + ); + for secret in [RESULT_SECRET, ARGS_SECRET, LOGS_SECRET] { + assert!( + !body.contains(secret), + "embed token response for {path} leaked `{secret}`: {body}" + ); + } + } + + // ---- APP EMBED TOKEN: cancellation confined to the app's own jobs. The token + // may cancel a job it launched (created_by == viewer), but `cancel_job_api` + // denies (NotFound) a job created by someone else, even though cancel + // otherwise has no per-job ownership check. + let (status, body) = post( + &base, + &format!("queue/cancel/{EMBED_OWN_QUEUED}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert!( + status.is_success(), + "embed token must cancel a job it launched (got {status}): {body}" + ); + let (status, body) = post( + &base, + &format!("queue/cancel/{RUNNING_JOB}"), + Some("EMBED_APP_TOKEN"), + ) + .await; + assert_eq!( + status, + reqwest::StatusCode::NOT_FOUND, + "embed token must not cancel another user's job (got {status}): {body}" + ); + // ---- UNAUTHENTICATED, unchanged: an anonymous-created job is readable // without a token (public trigger / public app result polling). let (status, body) = get(&base, &format!("completed/get_result/{ANON_JOB}"), None).await; diff --git a/backend/tests/mcp_resource_authz.rs b/backend/tests/mcp_resource_authz.rs new file mode 100644 index 0000000000..0fa35933f4 --- /dev/null +++ b/backend/tests/mcp_resource_authz.rs @@ -0,0 +1,102 @@ +//! Regression test for the MCP resource-authorization bypass +//! (WIN-2041, GHSA-7qg3-pr4g-cq5x). +//! +//! Invariant: the AI Agent worker (`load_mcp_tools`) must load an MCP resource +//! only when the job identity is allowed to read it — the same +//! `resources:read:{path}` + RLS gate the regular MCP tools API (`get_mcp_tools`) +//! enforces. It loads the resource through the job's permissioned client +//! (`AuthedClient::get_resource_value`, the `resources/get_value` path), not the +//! raw DB pool. Otherwise a low-privileged user who can edit a flow could make +//! the agent load and use an MCP resource (URL + inline headers + token) they are +//! not allowed to read: a confused-deputy bypass. +//! +//! This test pins, against the `mcp_resource_authz` fixture (an MCP resource in +//! a folder only the admin can read): +//! - a plain developer (test-user-3) is DENIED loading the private resource, +//! before any MCP connection is attempted, and the resource never leaks; +//! - an admin (test-user) clears the gate, the resource resolves, and loading +//! only fails later at the connect step — proving no over-blocking. +#![cfg(feature = "mcp")] + +use sqlx::{Pool, Postgres}; +use windmill_common::client::AuthedClient; +use windmill_test_utils::*; +use windmill_worker::{load_mcp_tools, McpResourceConfig}; + +fn config() -> Vec { + vec![McpResourceConfig { + resource_path: "$res:f/private/admin_mcp".to_string(), + include_tools: None, + exclude_tools: None, + }] +} + +// The Ok variant ((HashMap<_, Arc>, Vec)) does not implement +// Debug, so `expect_err` is unavailable — extract the error explicitly. +async fn load_err(db: &Pool, client: &AuthedClient, ctx: &str) -> String { + match load_mcp_tools(db, "test-workspace", config(), client).await { + Ok(_) => panic!("{ctx}"), + Err(e) => e.to_string(), + } +} + +#[sqlx::test(fixtures("base", "mcp_resource_authz"))] +async fn test_mcp_resource_not_loaded_without_authorization( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let base_internal_url = format!("http://localhost:{}", server.addr.port()); + + // ---- CORE REGRESSION: the developer cannot read the private MCP resource, + // so the worker must refuse to load it — before any MCP connection is + // attempted, and without the resource ever leaking. + let dev_client = AuthedClient::new( + base_internal_url.clone(), + "test-workspace".to_string(), + "SECRET_TOKEN_3".to_string(), + None, + ); + let msg = load_err( + &db, + &dev_client, + "developer must be denied loading a resource they can't read", + ) + .await; + assert!( + msg.contains("don't have access"), + "denial should come from the resource-RLS gate, not a connection error: {msg}" + ); + // An unauthorized caller must be blocked before MCP client creation, so no + // resource material (URL, headers, token) is ever loaded for them. + assert!( + !msg.contains("Failed to create MCP client"), + "developer must be blocked before the connection step (would mean the resource was loaded): {msg}" + ); + + // ---- NO OVER-BLOCKING: an admin clears the resource-RLS gate, so the + // resource resolves and loading only fails later at the connect step. + let admin_client = AuthedClient::new( + base_internal_url, + "test-workspace".to_string(), + "SECRET_TOKEN".to_string(), + None, + ); + let msg = load_err( + &db, + &admin_client, + "the fixture URL is non-resolvable, so the connect step must fail", + ) + .await; + assert!( + !msg.contains("don't have access"), + "admin must clear the resource-RLS gate, not be denied: {msg}" + ); + assert!( + msg.contains("Failed to create MCP client"), + "admin should resolve the resource and only fail at the connect step: {msg}" + ); + + Ok(()) +} diff --git a/backend/tests/python_jobs.rs b/backend/tests/python_jobs.rs index 3a1550d395..0f45d09147 100644 --- a/backend/tests/python_jobs.rs +++ b/backend/tests/python_jobs.rs @@ -1113,6 +1113,7 @@ async def main(item: str, qty: int, email: str): RunJob::from(JobPayload::Code(RawCode { language: ScriptLang::Python3, content, + tag: None, ..RawCode::default() })) .arg("item", json!("widget")) @@ -1298,6 +1299,7 @@ async def main(n: int): RunJob::from(JobPayload::Code(RawCode { language: ScriptLang::Python3, content, + tag: None, ..RawCode::default() })) .arg("n", json!(1)) diff --git a/backend/tests/relock_skip.rs b/backend/tests/relock_skip.rs index eb8dfb261d..8262a38cf4 100644 --- a/backend/tests/relock_skip.rs +++ b/backend/tests/relock_skip.rs @@ -14,6 +14,7 @@ mod relock_skip { parent_hash: Option, ) -> NewScript { NewScript { + draft_only: None, content: content.into(), language, lock, @@ -24,7 +25,6 @@ mod relock_skip { cache_ttl: None, dedicated_worker: None, description: "".to_string(), - draft_only: None, envs: vec![], is_template: None, kind: None, diff --git a/backend/tests/retry.rs b/backend/tests/retry.rs index abd7c38d8d..e804b96f0e 100644 --- a/backend/tests/retry.rs +++ b/backend/tests/retry.rs @@ -262,6 +262,65 @@ def main(last, port): Ok(()) } + /* `skip_if` evaluation goes through windmill-jseval and requires `quickjs` */ + #[cfg(all(feature = "deno_core", feature = "quickjs"))] + #[sqlx::test(fixtures("base"))] + async fn test_skip_if_sees_previous_step_result_on_retry( + db: Pool, + ) -> anyhow::Result<()> { + initialize_tracing().await; + + /* step `b` fails on its first attempt and succeeds on retry. Its `skip_if` + * references the previous step's result: when re-evaluated for the retry, + * `previous_result` must still be step `a`'s result, not the error of step + * `b`'s failed attempt (which would wrongly flip `skip_if` to true and skip + * the retry) */ + let value = serde_json::from_value(json!({ + "modules": [{ + "id": "a", + "value": { + "input_transforms": {}, + "type": "rawscript", + "language": "deno", + "content": "export function main() { return \"ok\" }", + }, + }, { + "id": "b", + "skip_if": { "expr": "results.a?.error !== undefined" }, + "value": { + "input_transforms": { + "index": { "type": "static", "value": 1 }, + "port": { "type": "javascript", "expr": "flow_input.port" }, + }, + "type": "rawscript", + "language": "deno", + "content": inner_step(), + }, + "retry": { "constant": { "attempts": 1, "seconds": 0 } }, + }], + })) + .unwrap(); + + let (attempts, responses) = [ + /* fail step `b` once, then pass on retry */ + (1, None), + (1, Some(42)), + ] + .into_iter() + .unzip::<_, _, Vec<_>, Vec<_>>(); + let server = Server::start(responses).await; + let result = RunJob::from(JobPayload::RawFlow { value, path: None, restarted_from: None }) + .arg("port", json!(server.addr.port())) + .run_until_complete(&db, false, server.addr.port()) + .await + .json_result() + .unwrap(); + + assert_eq!(server.close().await, attempts); + assert_eq!(json!(42), result); + Ok(()) + } + #[cfg(feature = "python")] #[sqlx::test(fixtures("base"))] async fn test_with_failure_module(db: Pool) -> anyhow::Result<()> { diff --git a/backend/tests/script_auto_kind_failure.rs b/backend/tests/script_auto_kind_failure.rs index 4110bd4ba5..05d22e75ec 100644 --- a/backend/tests/script_auto_kind_failure.rs +++ b/backend/tests/script_auto_kind_failure.rs @@ -6,6 +6,7 @@ use windmill_test_utils::init_client; fn quick_ns(content: &str, path: &str, kind: Option<&str>) -> NewScript { NewScript { + draft_only: None, content: content.into(), language: ScriptLang::Bun, lock: None, @@ -16,7 +17,6 @@ fn quick_ns(content: &str, path: &str, kind: Option<&str>) -> NewScript { cache_ttl: None, dedicated_worker: None, description: "".to_string(), - draft_only: None, envs: vec![], is_template: None, kind: kind.map(|s| s.to_string()), diff --git a/backend/tests/script_modules.rs b/backend/tests/script_modules.rs index 068555c7df..8755ebfb57 100644 --- a/backend/tests/script_modules.rs +++ b/backend/tests/script_modules.rs @@ -40,6 +40,7 @@ def main(name: str): path: Some("f/test/my_script".to_string()), language: ScriptLang::Python3, modules: Some(modules), + tag: None, ..RawCode::default() }); @@ -92,6 +93,7 @@ def main(a: int, b: int): path: Some("f/test/my_script".to_string()), language: ScriptLang::Python3, modules: Some(modules), + tag: None, ..RawCode::default() }); @@ -143,6 +145,7 @@ export function main(name: string) { path: Some("f/test/my_script".to_string()), language: ScriptLang::Bun, modules: Some(modules), + tag: None, ..RawCode::default() }); diff --git a/backend/tests/suspend_resume.rs b/backend/tests/suspend_resume.rs index b8468f0cbd..704fa0d256 100644 --- a/backend/tests/suspend_resume.rs +++ b/backend/tests/suspend_resume.rs @@ -628,4 +628,53 @@ mod suspend_resume { ); Ok(()) } + + /// A step that declares a `suspend` but is skipped via `skip_if` never arms + /// its approval, so it must not gate the following step. If it did, the flow + /// would park forever waiting for a resume event that is never dispatched. + #[cfg(feature = "deno_core")] + #[sqlx::test(fixtures("base"))] + async fn skipped_suspend_step_does_not_block_next_step( + db: Pool, + ) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + + let flow: FlowValue = serde_json::from_value(json!({ + "modules": [ + { + "id": "a", + "skip_if": { "type": "javascript", "expr": "true" }, + "suspend": { "required_events": 1, "timeout": 86400 }, + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main() { return 1 }", + "input_transforms": {}, + }, + }, + { + "id": "b", + "value": { + "type": "rawscript", + "language": "deno", + "content": "export async function main() { return 42 }", + "input_transforms": {}, + }, + }, + ], + })) + .unwrap(); + + let result = + RunJob::from(JobPayload::RawFlow { value: flow, path: None, restarted_from: None }) + .run_until_complete(&db, false, server.addr.port()) + .await + .json_result() + .unwrap(); + + assert_eq!(result, json!(42)); + Ok(()) + } } diff --git a/backend/tests/trigger_listener_queries.rs b/backend/tests/trigger_listener_queries.rs index d1c20da407..c3f6357d4c 100644 --- a/backend/tests/trigger_listener_queries.rs +++ b/backend/tests/trigger_listener_queries.rs @@ -200,7 +200,7 @@ async fn test_handler_queries_websocket(db: Pool) -> anyhow::Result<() assert_eq!(trigger.base.permissioned_as, "u/test-user"); let triggers = handler - .list_triggers(&mut *conn, "test-workspace", None) + .list_triggers(&mut *conn, "test-workspace", None, None) .await?; assert!(triggers.iter().any(|t| t.base.path == "f/test/handler_ws")); diff --git a/backend/tests/v2_job_delete_orphans.rs b/backend/tests/v2_job_delete_orphans.rs new file mode 100644 index 0000000000..95cd1673b6 --- /dev/null +++ b/backend/tests/v2_job_delete_orphans.rs @@ -0,0 +1,237 @@ +//! v2_job no longer cascades to its sparse side tables `dispatch_event`, +//! `flow_conversation_message`, and `zombie_job_counter` — their `ON DELETE CASCADE` +//! foreign keys were dropped (migration `drop_v2_job_side_table_cascades`) to keep bulk +//! retention deletes cheap. Every path that deletes a v2_job must therefore clean those +//! tables explicitly. These tests assert no orphan side rows survive the deletion paths: +//! direct job deletion (`delete_jobs`), schedule clearing, and workspace deletion — plus a +//! regression test that the `/jobs/delete` purge endpoint stays workspace-scoped (a workspace +//! admin must not be able to delete another workspace's side rows by passing foreign job ids). +//! +//! Uses runtime `sqlx::query` (not the compile-time macros) so no offline query cache is +//! needed, matching delete_after_secs.rs. + +use sqlx::{Pool, Postgres}; +use uuid::Uuid; +use windmill_test_utils::*; + +const WS: &str = "test-workspace"; + +/// Insert one row in each side table that used to cascade from `job_id`, in workspace `ws`. +async fn seed_side_rows(db: &Pool, ws: &str, job_id: Uuid) -> anyhow::Result<()> { + sqlx::query( + "INSERT INTO dispatch_event + (workspace_id, producer_job_id, subscriber_path, asset_kind, asset_path, outcome) + VALUES ($1, $2, 'f/sub', 'resource', 'f/res', 'dispatched')", + ) + .bind(ws) + .bind(job_id) + .execute(db) + .await?; + + let conv_id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO flow_conversation (id, workspace_id, flow_path, created_by) + VALUES ($1, $2, 'f/flow', 'test-user')", + ) + .bind(conv_id) + .bind(ws) + .execute(db) + .await?; + // created_seq is assigned by a trigger; inserting a value is rejected. + sqlx::query( + "INSERT INTO flow_conversation_message (conversation_id, message_type, content, job_id) + VALUES ($1, 'assistant', 'hi', $2)", + ) + .bind(conv_id) + .bind(job_id) + .execute(db) + .await?; + + sqlx::query("INSERT INTO zombie_job_counter (job_id, counter) VALUES ($1, 1)") + .bind(job_id) + .execute(db) + .await?; + Ok(()) +} + +async fn count(db: &Pool, sql: &str, job_id: Uuid) -> anyhow::Result { + Ok(sqlx::query_scalar::<_, i64>(sql) + .bind(job_id) + .fetch_one(db) + .await?) +} + +/// (dispatch_event, flow_conversation_message, zombie_job_counter, v2_job) row counts for `job_id`. +async fn counts(db: &Pool, job_id: Uuid) -> anyhow::Result<(i64, i64, i64, i64)> { + Ok(( + count( + db, + "SELECT count(*) FROM dispatch_event WHERE producer_job_id = $1", + job_id, + ) + .await?, + count( + db, + "SELECT count(*) FROM flow_conversation_message WHERE job_id = $1", + job_id, + ) + .await?, + count( + db, + "SELECT count(*) FROM zombie_job_counter WHERE job_id = $1", + job_id, + ) + .await?, + count(db, "SELECT count(*) FROM v2_job WHERE id = $1", job_id).await?, + )) +} + +async fn insert_job(db: &Pool, ws: &str, job_id: Uuid) -> anyhow::Result<()> { + sqlx::query("INSERT INTO v2_job (id, workspace_id, kind) VALUES ($1, $2, 'script')") + .bind(job_id) + .bind(ws) + .execute(db) + .await?; + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn test_delete_jobs_removes_side_rows(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let job_id = Uuid::new_v4(); + insert_job(&db, WS, job_id).await?; + seed_side_rows(&db, WS, job_id).await?; + assert_eq!( + counts(&db, job_id).await?, + (1, 1, 1, 1), + "seed should create one row per table" + ); + + let mut conn = db.acquire().await?; + windmill_common::jobs::delete_jobs(&mut conn, &[job_id]).await?; + drop(conn); + + let (de, fcm, zombie, job) = counts(&db, job_id).await?; + assert_eq!( + (de, fcm, zombie, job), + (0, 0, 0, 0), + "delete_jobs left orphans: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn test_clear_schedule_removes_side_rows(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + // clear_schedule deletes queued, non-running jobs whose v2_job is a schedule trigger. + let job_id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, kind, trigger_kind, trigger) + VALUES ($1, $2, 'script', 'schedule', 'f/sched')", + ) + .bind(job_id) + .bind(WS) + .execute(&db) + .await?; + sqlx::query( + "INSERT INTO v2_job_queue (id, workspace_id, scheduled_for, running) + VALUES ($1, $2, now(), false)", + ) + .bind(job_id) + .bind(WS) + .execute(&db) + .await?; + seed_side_rows(&db, WS, job_id).await?; + + let mut tx = db.begin().await?; + windmill_queue::schedule::clear_schedule(&mut tx, "f/sched", WS).await?; + tx.commit().await?; + + let (de, fcm, zombie, job) = counts(&db, job_id).await?; + assert_eq!( + (de, fcm, zombie, job), + (0, 0, 0, 0), + "clear_schedule left orphans: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} + +#[sqlx::test(fixtures("base"))] +async fn test_workspace_delete_removes_side_rows(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let job_id = Uuid::new_v4(); + insert_job(&db, WS, job_id).await?; + seed_side_rows(&db, WS, job_id).await?; + + // SECRET_TOKEN is the base fixture's instance-superadmin token for test@windmill.dev. + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let resp = reqwest::Client::new() + .delete(format!( + "http://localhost:{port}/api/workspaces/delete/{WS}" + )) + .header("Authorization", "Bearer SECRET_TOKEN") + .send() + .await?; + let status = resp.status(); + assert!( + status.is_success(), + "delete workspace failed: {status} {}", + resp.text().await? + ); + + let (de, fcm, zombie, job) = counts(&db, job_id).await?; + assert_eq!( + (de, fcm, zombie, job), + (0, 0, 0, 0), + "workspace deletion left orphans: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} + +/// The `/jobs/delete` purge endpoint must scope every side-table delete to the path +/// workspace. A `test-workspace` admin passing a job id from another workspace must not be +/// able to delete that workspace's job or side rows (the side tables no longer cascade, so +/// the scoping has to live in each explicit delete). +#[sqlx::test(fixtures("base"))] +async fn test_jobs_export_delete_is_workspace_scoped(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + // A job with side rows living in a *different* workspace. + sqlx::query( + "INSERT INTO workspace (id, name, owner) VALUES ('ws-other', 'ws-other', 'test-user')", + ) + .execute(&db) + .await?; + let other_job = Uuid::new_v4(); + insert_job(&db, "ws-other", other_job).await?; + seed_side_rows(&db, "ws-other", other_job).await?; + + // A test-workspace admin calls the purge endpoint with the FOREIGN job id. + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let resp = reqwest::Client::new() + .post(format!("http://localhost:{port}/api/w/{WS}/jobs/delete")) + .header("Authorization", "Bearer SECRET_TOKEN") + .json(&[other_job]) + .send() + .await?; + assert!( + resp.status().is_success(), + "purge request failed: {} {}", + resp.status(), + resp.text().await? + ); + + // The other workspace's job and all its side rows must survive untouched. + let (de, fcm, zombie, job) = counts(&db, other_job).await?; + assert_eq!( + (de, fcm, zombie, job), + (1, 1, 1, 1), + "jobs_export purge crossed workspaces: dispatch_event={de} flow_conversation_message={fcm} zombie_job_counter={zombie} v2_job={job}" + ); + Ok(()) +} diff --git a/backend/tests/wm_token_superadmin_guard.rs b/backend/tests/wm_token_superadmin_guard.rs new file mode 100644 index 0000000000..650cf384d4 --- /dev/null +++ b/backend/tests/wm_token_superadmin_guard.rs @@ -0,0 +1,206 @@ +//! A WM_TOKEN (job JWT) running as a superadmin must not be able to perform +//! global user/token management — promotion, password reset, user creation, +//! token creation/impersonation, offboarding, or exporting the user table. +//! A non-admin `wm_deployers` member can mint +//! such a token implicitly via an app/flow `on_behalf_of`, so trusting it would +//! let them establish *persistent* superadmin. A real superadmin who needs this +//! from a script must use a dedicated superadmin API token (which only a real +//! superadmin can create), not `$WM_TOKEN`. +//! +//! The fixture provides `test@windmill.dev` (instance superadmin, token +//! `SECRET_TOKEN`) and `test2@windmill.dev` (non-superadmin, `SECRET_TOKEN_2`). + +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_common::auth::create_jwt_token; +use windmill_common::db::Authed; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder { + builder.header("Authorization", format!("Bearer {}", token)) +} + +/// Mint a WM_TOKEN: an internally-signed job JWT (note the `job_id` claim) for +/// `email`, exactly as a running app/flow job is issued. +async fn wm_token(email: &str, is_admin: bool) -> String { + let authed = Authed { + email: email.to_string(), + username: "runner".to_string(), + is_admin, + is_operator: false, + groups: vec![], + folders: vec![], + scopes: None, + token_prefix: None, + }; + create_jwt_token( + authed, + "test-workspace", + 3600, + Some(uuid::Uuid::new_v4()), + Some("app".to_string()), + None, + None, + ) + .await + .expect("mint wm_token") +} + +#[sqlx::test(fixtures("preserve_on_behalf_of"))] +async fn test_wm_token_cannot_manage_superadmin_users(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + // The server decodes WM_TOKENs with the same in-process JWT secret, so + // setting it once lets us mint a valid one below. + set_jwt_secret().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/users"); + + // A superadmin-capable WM_TOKEN — the exact thing a deployer obtains via an + // app on_behalf_of pointed at a superadmin. + let sa_wm = wm_token("test@windmill.dev", true).await; + + // 1. Cannot mint a (superadmin) token. + let resp = authed(client().post(format!("{base}/tokens/create")), &sa_wm) + .json(&json!({})) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not create tokens: {}", + resp.text().await? + ); + + // 2. Cannot impersonate (mint a token as another user). + let resp = authed(client().post(format!("{base}/tokens/impersonate")), &sa_wm) + .json(&json!({ "impersonate_email": "test2@windmill.dev" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not impersonate: {}", + resp.text().await? + ); + + // 3. Cannot promote a user to superadmin. + let resp = authed( + client().post(format!("{base}/update/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({ "is_super_admin": true })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not promote users: {}", + resp.text().await? + ); + + // 4. Cannot reset its own (the superadmin's) password. + let resp = authed(client().post(format!("{base}/setpassword")), &sa_wm) + .json(&json!({ "password": "hunter2" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not reset passwords: {}", + resp.text().await? + ); + + // 4b. Cannot delete a user. + let resp = authed( + client().delete(format!("{base}/delete/test2@windmill.dev")), + &sa_wm, + ) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not delete users: {}", + resp.text().await? + ); + + // 4c. Cannot change a user's login type. + let resp = authed( + client().post(format!("{base}/set_login_type/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({ "login_type": "password" })) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not change login type: {}", + resp.text().await? + ); + + // 4d. Cannot offboard a global user (deletes user, tokens, password, invites, + // instance-group membership and reassigns their assets). + let resp = authed( + client().post(format!("{base}/offboard/test2@windmill.dev")), + &sa_wm, + ) + .json(&json!({})) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not offboard users: {}", + resp.text().await? + ); + + // 4e. Cannot export the global user table (leaks every user's password_hash). + let resp = authed(client().get(format!("{base}/export")), &sa_wm) + .send() + .await?; + assert_eq!( + resp.status(), + 401, + "superadmin WM_TOKEN must not export global users: {}", + resp.text().await? + ); + + // 5. Escape hatch / no false positive: a real superadmin API token + // (SECRET_TOKEN, no job_id) can still create tokens. + let resp = authed( + client().post(format!("{base}/tokens/create")), + "SECRET_TOKEN", + ) + .json(&json!({ "label": "ci" })) + .send() + .await?; + assert_eq!( + resp.status(), + 201, + "a real superadmin token must still create tokens: {}", + resp.text().await? + ); + + // 6. No collateral: a non-superadmin WM_TOKEN can still create its own + // token — the guard only fires for superadmin-capable job tokens. + let user_wm = wm_token("test2@windmill.dev", false).await; + let resp = authed(client().post(format!("{base}/tokens/create")), &user_wm) + .json(&json!({ "label": "from-script" })) + .send() + .await?; + assert_eq!( + resp.status(), + 201, + "non-superadmin WM_TOKEN must still create its own token: {}", + resp.text().await? + ); + + Ok(()) +} diff --git a/backend/tests/worker.rs b/backend/tests/worker.rs index f21862fd38..2349d476b7 100644 --- a/backend/tests/worker.rs +++ b/backend/tests/worker.rs @@ -3990,6 +3990,261 @@ async fn test_failure_module(db: Pool) -> anyhow::Result<()> { Ok(()) } +/// Push `flow`, run it on a real worker until its first step is running, then simulate +/// `monitor::handle_zombie_jobs` reaping that step unrecoverably (its worker crashed/OOM'd) +/// by calling `handle_job_error(..., unrecoverable = true, ...)` exactly as the monitor does. +/// Returns the flow's completed result. +#[cfg(feature = "deno_core")] +async fn run_flow_until_step_running_then_fail_unrecoverably( + db: &Pool, + port: u16, + flow: FlowValue, +) -> serde_json::Value { + use std::sync::atomic::AtomicU16; + use std::sync::Arc; + use tokio::sync::mpsc; + use windmill_common::auth::create_token_for_owner; + use windmill_common::client::AuthedClient; + use windmill_common::KillpillSender; + use windmill_queue::{get_queued_job_v2, MiniCompletedJob, SameWorkerPayload}; + use windmill_worker::{JobCompletedSender, SameWorkerSender}; + + let flow_id = + RunJob::from(JobPayload::RawFlow { value: flow, path: None, restarted_from: None }) + .push(db) + .await; + + let db_ = db.clone(); + in_test_worker( + db, + async move { + let db = db_; + + // Wait for the first step to be running on the worker. + let step_job = loop { + tokio::time::sleep(std::time::Duration::from_millis(200)).await; + let running = sqlx::query_scalar!( + "SELECT q.id FROM v2_job_queue q JOIN v2_job j USING (id) + WHERE j.parent_job = $1 AND q.running = true", + flow_id + ) + .fetch_optional(&db) + .await + .unwrap(); + if let Some(step_id) = running { + if let Some(job) = get_queued_job_v2(&db, &step_id).await.unwrap() { + break job; + } + } + }; + + // The dummy `same_worker_tx` mirrors the monitor, which has no live worker channel. + let (sw_tx, _sw_rx) = mpsc::channel::(1); + let sw_tx = SameWorkerSender(sw_tx, Arc::new(AtomicU16::new(0))); + let (jc_tx, _jc_rx) = JobCompletedSender::new_never_used(); + let (_kp_tx, kp_rx) = KillpillSender::new(1); + let token = create_token_for_owner( + &db, + "test-workspace", + "u/test-user", + "", + 100, + "", + &Uuid::nil(), + None, + None, + ) + .await + .unwrap(); + let client = AuthedClient::new( + format!("http://localhost:{port}"), + "test-workspace".to_string(), + token, + None, + ); + + windmill_worker::result_processor::handle_job_error( + &db, + &client, + &MiniCompletedJob::from(step_job), + 0, + None, + windmill_common::error::Error::ExecutionErr( + "simulated worker OOM crash".to_string(), + ), + true, // unrecoverable + Some(&sw_tx), + "", + "test-monitor", + jc_tx, + &kp_rx, + ) + .await; + + // The flow should now run its failure module and complete. + loop { + tokio::time::sleep(std::time::Duration::from_millis(200)).await; + let done = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM v2_job_completed WHERE id = $1)", + flow_id + ) + .fetch_one(&db) + .await + .unwrap() + .unwrap_or(false); + if done { + break; + } + } + }, + port, + ) + .await; + + completed_job(flow_id, db).await.json_result().unwrap() +} + +/// The hanging-forever first step: only ever completed by the simulated zombie handler. +#[cfg(feature = "deno_core")] +fn hanging_step_value() -> serde_json::Value { + serde_json::json!({ + "input_transforms": {}, + "type": "rawscript", + "language": "deno", + "content": "export async function main() { await new Promise((r) => setTimeout(r, 600000)); }", + }) +} + +/// The error handler module that marks itself so tests can assert it ran. +#[cfg(feature = "deno_core")] +fn marker_failure_module() -> serde_json::Value { + serde_json::json!({ + "value": { + "input_transforms": { "error": { "type": "javascript", "expr": "previous_result", } }, + "type": "rawscript", + "language": "deno", + "content": "export function main(error) { return { handled_unrecoverable: true, error } }", + } + }) +} + +/// Regression test for WIN-2070: a flow step that fails *unrecoverably* — e.g. its worker was +/// OOM-killed and the failure is surfaced by the zombie job handler — must still trigger the +/// flow's error handler (failure module). Previously, `unrecoverable` failures silently +/// completed the flow with an error and skipped the failure module entirely. +#[cfg(feature = "deno_core")] +#[sqlx::test(fixtures("base"))] +async fn test_failure_module_triggered_on_unrecoverable_failure( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let flow: FlowValue = serde_json::from_value(serde_json::json!({ + "modules": [{ "id": "a", "value": hanging_step_value() }], + "failure_module": marker_failure_module(), + })) + .unwrap(); + + let result = run_flow_until_step_running_then_fail_unrecoverably(&db, port, flow).await; + + server.close().await.unwrap(); + + assert_eq!( + result["handled_unrecoverable"], + json!(true), + "failure module (flow error handler) should run for an unrecoverable step failure, got: {result}" + ); + Ok(()) +} + +/// WIN-2070: an unrecoverable failure must NOT be retried even when the step has a retry +/// policy — the original worker is gone, so retrying is pointless. It should fall straight +/// through to the error handler (failure module) instead. +#[cfg(feature = "deno_core")] +#[sqlx::test(fixtures("base"))] +async fn test_unrecoverable_failure_skips_retry_runs_failure_module( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let flow: FlowValue = serde_json::from_value(serde_json::json!({ + "modules": [{ + "id": "a", + "value": hanging_step_value(), + "retry": { "constant": { "attempts": 5, "seconds": 0 } }, + }], + "failure_module": marker_failure_module(), + })) + .unwrap(); + + let result = run_flow_until_step_running_then_fail_unrecoverably(&db, port, flow).await; + + server.close().await.unwrap(); + + assert_eq!( + result["handled_unrecoverable"], + json!(true), + "unrecoverable failure should skip retry and run the failure module, got: {result}" + ); + Ok(()) +} + +/// WIN-2070: an unrecoverable failure on a `continue_on_error` step must still route to the +/// error handler rather than silently advancing to the next step (which would hide the worker +/// death). A normal failure on a `continue_on_error` step is tolerated and the flow continues; +/// a worker crash/OOM is not. +#[cfg(feature = "deno_core")] +#[sqlx::test(fixtures("base"))] +async fn test_unrecoverable_failure_on_continue_on_error_runs_failure_module( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Step 'a' hangs (and tolerates failures via continue_on_error); step 'b' would run next + // if the unrecoverable failure were (wrongly) tolerated. + let flow: FlowValue = serde_json::from_value(serde_json::json!({ + "modules": [ + { + "id": "a", + "value": hanging_step_value(), + "continue_on_error": true, + }, + { + "id": "b", + "value": { + "input_transforms": {}, + "type": "rawscript", + "language": "deno", + "content": "export function main() { return { ran_b: true }; }", + }, + }, + ], + "failure_module": marker_failure_module(), + })) + .unwrap(); + + let result = run_flow_until_step_running_then_fail_unrecoverably(&db, port, flow).await; + + server.close().await.unwrap(); + + assert_eq!( + result["handled_unrecoverable"], + json!(true), + "unrecoverable failure on a continue_on_error step should run the failure module, got: {result}" + ); + assert!( + result.get("ran_b").is_none(), + "the step after a continue_on_error step must NOT run on an unrecoverable failure, got: {result}" + ); + Ok(()) +} + #[cfg(feature = "deno_core")] #[sqlx::test(fixtures("base"))] async fn test_run_wait_result_early_return_with_failure_module( @@ -4178,8 +4433,8 @@ async fn test_flow_lock_all(db: Pool) -> anyhow::Result<()> { visible_to_runner_only: None, on_behalf_of_email: None, }, - draft_only: None, deployment_message: None, + draft_only: None, }, ) .await @@ -4997,6 +5252,7 @@ async fn test_workflow_as_code(db: Pool) -> anyhow::Result<()> { RunJob::from(JobPayload::Code(RawCode { language: ScriptLang::Python3, content: WORKFLOW_AS_CODE.into(), + tag: None, ..RawCode::default() })) .arg("n", json!(3)) @@ -5345,6 +5601,74 @@ async fn test_stop_after_all_iters_if_bad_expr_parallel_forloop( Ok(()) } +// Regression for the savepoint added around `evaluate_stop_after_all_iters_if`. +// The failpoint makes the in-evaluation DB read fail with a transaction-aborting +// error (SELECT 1/0). The caller swallows that error and keeps using the outer +// status-update transaction (later reads + commit). Without the savepoint the +// outer transaction would be aborted and the commit would fail, so the flow job +// would never be marked completed (the worker would error/retry). With the +// savepoint the read failure is isolated, the iteration is marked failed, and +// the flow completes — which is what this test asserts. +#[cfg(all(feature = "failpoints", feature = "quickjs", feature = "python"))] +#[sqlx::test(fixtures("base"))] +async fn test_stop_after_all_iters_if_db_error_isolated_by_savepoint( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + + let port = 123; + let flow: FlowValue = serde_json::from_value(serde_json::json!({ + "modules": [ + { + "id": "a", + "value": { + "type": "forloopflow", + "iterator": { "type": "javascript", "expr": "result.items" }, + "skip_failures": false, + "parallel": true, + "modules": [{ + "value": { + "input_transforms": { + "n": { "type": "javascript", "expr": "flow_input.iter.value" }, + }, + "type": "rawscript", + "language": "python3", + "content": "def main(n): return n", + }, + }], + }, + "stop_after_all_iters_if": { + "expr": "__wm_failpoint_abort_tx__", + "skip_if_stopped": false, + }, + }, + ], + })) + .unwrap(); + let job = JobPayload::RawFlow { value: flow, path: None, restarted_from: None }; + + // If the savepoint failed to isolate the aborted read, the status-update + // transaction would be poisoned and the job would never complete, so this + // call would hang until the worker times out rather than returning. + let cjob = RunJob::from(job) + .arg("items", json!([1, 2, 3])) + .run_until_complete(&db, false, port) + .await; + + assert!( + !cjob.success, + "iteration should be marked failed after the injected read error" + ); + let result = cjob.json_result().unwrap(); + let error_msg = result["error"]["message"].as_str().unwrap_or(""); + assert!( + error_msg.contains("stop_after_all_iters_if"), + "error should mention stop_after_all_iters_if, got: {error_msg}" + ); + + Ok(()) +} + #[cfg(all(feature = "quickjs", feature = "python"))] #[sqlx::test(fixtures("base"))] async fn test_results_length_in_input_transform(db: Pool) -> anyhow::Result<()> { diff --git a/backend/tests/workspace_export.rs b/backend/tests/workspace_export.rs index 823ff22a43..ca988562fd 100644 --- a/backend/tests/workspace_export.rs +++ b/backend/tests/workspace_export.rs @@ -40,6 +40,7 @@ async fn test_tarball_export_all_tables(db: Pool) -> anyhow::Result<() .create_script( "test-workspace", &windmill_api_client::types::NewScript { + draft_only: None, content: "export function main() { return 42; }".to_string(), language: windmill_api_client::types::ScriptLang::Bun, path: "f/test_folder/test_script".to_string(), @@ -51,7 +52,6 @@ async fn test_tarball_export_all_tables(db: Pool) -> anyhow::Result<() parent_hash: None, schema: Default::default(), is_template: None, - draft_only: None, dedicated_worker: None, ws_error_handler_muted: None, priority: None, diff --git a/backend/windmill-ai/src/ai_bedrock.rs b/backend/windmill-ai/src/ai_bedrock.rs index e549e63041..8aa9addcae 100644 --- a/backend/windmill-ai/src/ai_bedrock.rs +++ b/backend/windmill-ai/src/ai_bedrock.rs @@ -560,6 +560,26 @@ fn convert_message(msg: &OpenAIMessage) -> Result { let mut content_blocks = Vec::new(); + // Replay the Claude reasoning block first: when thinking is enabled, + // Anthropic requires the reasoning block (with its unmodified signature) to + // precede toolUse in the assistant turn it was emitted in. The proxy + // round-trips it on the tool call's extra_content (see BedrockExtraContent). + if role == ConversationRole::Assistant { + if let Some(reasoning) = msg + .tool_calls + .as_ref() + .and_then(|tcs| { + tcs.iter() + .find_map(|tc| tc.extra_content.as_ref().and_then(|ec| ec.bedrock.as_ref())) + }) + .map(bedrock_reasoning_block_from_extra) + .transpose()? + .flatten() + { + content_blocks.push(reasoning); + } + } + // Handle content (text and/or images) if let Some(content) = &msg.content { match content { @@ -597,6 +617,37 @@ fn convert_message(msg: &OpenAIMessage) -> Result { .map_err(|e| Error::internal_err(format!("Failed to build message: {}", e))) } +/// Rebuild a Bedrock reasoning content block from the round-tripped +/// [`BedrockExtraContent`](crate::ai_types::BedrockExtraContent). +fn bedrock_reasoning_block_from_extra( + extra: &crate::ai_types::BedrockExtraContent, +) -> Result, Error> { + if let Some(redacted) = extra.redacted_content.as_deref() { + let bytes = base64::Engine::decode(&base64::engine::general_purpose::STANDARD, redacted) + .map_err(|e| { + Error::internal_err(format!("Failed to decode redacted reasoning: {}", e)) + })?; + return Ok(Some(ContentBlock::ReasoningContent( + aws_sdk_bedrockruntime::types::ReasoningContentBlock::RedactedContent(bytes.into()), + ))); + } + + let Some(text) = extra.reasoning_text.as_deref() else { + return Ok(None); + }; + let mut builder = aws_sdk_bedrockruntime::types::ReasoningTextBlock::builder().text(text); + if let Some(signature) = extra.signature.as_deref() { + builder = builder.signature(signature); + } + Ok(Some(ContentBlock::ReasoningContent( + aws_sdk_bedrockruntime::types::ReasoningContentBlock::ReasoningText( + builder.build().map_err(|e| { + Error::internal_err(format!("Failed to build reasoning block: {}", e)) + })?, + ), + ))) +} + /// Convert OpenAI tool call to Bedrock ToolUse content block fn convert_tool_call_to_content(tool_call: &OpenAIToolCall) -> Result { let input = json_to_document( @@ -814,7 +865,10 @@ pub fn streaming_tool_calls_to_openai(tool_calls: Vec) -> Vec id: tc.id, function: OpenAIFunction { name: tc.name, arguments: tc.arguments }, r#type: FUNCTION_TYPE.to_string(), - extra_content: None, // Bedrock doesn't use thought signatures + // Worker agent requests never enable thinking, so there is no + // reasoning block to round-trip here (the chat proxy path does — + // see providers/bedrock.rs). + extra_content: None, }) .collect() } @@ -858,6 +912,7 @@ pub fn build_tool_config( #[cfg(test)] mod tests { use super::*; + use aws_sdk_bedrockruntime::types::ReasoningContentBlock; use serde_json::value::RawValue; fn text_message(role: &str, content: &str) -> OpenAIMessage { @@ -934,6 +989,45 @@ mod tests { )); } + #[test] + fn openai_messages_to_bedrock_replays_reasoning_before_tool_use() { + let tool_call = OpenAIToolCall { + id: "call_1".to_string(), + function: OpenAIFunction { name: "lookup".to_string(), arguments: "{}".to_string() }, + r#type: FUNCTION_TYPE.to_string(), + extra_content: Some(crate::ai_types::ExtraContent { + bedrock: Some(crate::ai_types::BedrockExtraContent { + reasoning_text: Some("let me think".to_string()), + signature: Some("sig-abc".to_string()), + redacted_content: None, + }), + ..Default::default() + }), + }; + let assistant = OpenAIMessage { + role: "assistant".to_string(), + tool_calls: Some(vec![tool_call]), + ..Default::default() + }; + let messages = vec![text_message("user", "hi"), assistant]; + + let (bedrock_messages, _) = + openai_messages_to_bedrock(&messages, false).expect("bedrock conversion succeeds"); + + let content = bedrock_messages + .last() + .expect("assistant message") + .content(); + match &content[0] { + ContentBlock::ReasoningContent(ReasoningContentBlock::ReasoningText(rt)) => { + assert_eq!(rt.text(), "let me think"); + assert_eq!(rt.signature(), Some("sig-abc")); + } + other => panic!("expected reasoning block first, got {:?}", other), + } + assert!(matches!(&content[1], ContentBlock::ToolUse(_))); + } + #[test] fn openai_messages_to_bedrock_skips_cache_points_when_disabled() { let messages = vec![ diff --git a/backend/windmill-ai/src/ai_google.rs b/backend/windmill-ai/src/ai_google.rs index 3743e6eb52..94d3ef65e0 100644 --- a/backend/windmill-ai/src/ai_google.rs +++ b/backend/windmill-ai/src/ai_google.rs @@ -139,6 +139,22 @@ pub struct GeminiGenerationConfig { pub response_mime_type: Option, #[serde(rename = "responseSchema", skip_serializing_if = "Option::is_none")] pub response_schema: Option, + #[serde(rename = "thinkingConfig", skip_serializing_if = "Option::is_none")] + pub thinking_config: Option, +} + +/// Thinking controls. Gemini 3+ models take a level token (`thinkingLevel`); +/// Gemini 2.5 models take a token budget (`thinkingBudget`, `-1` = dynamic). +/// `includeThoughts` returns thought summaries for chat display — summaries are +/// free; thinking tokens are billed whether or not they are returned. +#[derive(Serialize, Debug, PartialEq)] +pub struct GeminiThinkingConfig { + #[serde(rename = "thinkingLevel", skip_serializing_if = "Option::is_none")] + pub thinking_level: Option, + #[serde(rename = "thinkingBudget", skip_serializing_if = "Option::is_none")] + pub thinking_budget: Option, + #[serde(rename = "includeThoughts", skip_serializing_if = "Option::is_none")] + pub include_thoughts: Option, } // ============================================================================ @@ -206,6 +222,9 @@ pub struct GeminiSSEPart { pub text: Option, #[serde(rename = "functionCall")] pub function_call: Option, + /// Marks a thought-summary part (returned when `includeThoughts` is set). + #[serde(default)] + pub thought: Option, /// Thought signature for Gemini 3+ models. #[serde(rename = "thoughtSignature")] pub thought_signature: Option, @@ -293,6 +312,7 @@ impl GeminiToolCallEvent { pub fn to_extra_content(&self) -> Option { self.thought_signature.as_ref().map(|sig| ExtraContent { google: Some(GoogleExtraContent { thought_signature: Some(sig.clone()) }), + bedrock: None, }) } } @@ -301,6 +321,9 @@ impl GeminiToolCallEvent { #[derive(Debug, Default)] pub struct GeminiParsedEvent { pub text: Option, + /// Thought-summary text (parts flagged `thought: true`), kept separate from + /// the answer so it can stream as `reasoning_content`. + pub reasoning: Option, pub tool_calls: Vec, pub annotations: Vec, pub used_websearch: bool, @@ -575,6 +598,9 @@ pub fn gemini_response_to_openai(parsed: &GeminiParsedEvent, model: &str) -> ser "role": "assistant", "content": content, }); + if let Some(reasoning) = &parsed.reasoning { + message["reasoning_content"] = serde_json::json!(reasoning); + } if !tool_calls.is_empty() { message["tool_calls"] = serde_json::json!(tool_calls); } @@ -604,6 +630,20 @@ pub fn gemini_event_to_openai_sse_chunks( ) -> Vec { let mut chunks = Vec::new(); + if let Some(reasoning) = &parsed.reasoning { + let chunk = serde_json::json!({ + "id": id, + "object": "chat.completion.chunk", + "model": model, + "choices": [{ + "index": 0, + "delta": { "reasoning_content": reasoning }, + "finish_reason": null, + }] + }); + chunks.push(format!("data: {}\n\n", chunk)); + } + if let Some(text) = &parsed.text { let chunk = serde_json::json!({ "id": id, @@ -636,6 +676,29 @@ pub fn gemini_event_to_openai_sse_chunks( *tool_call_index += 1; } + // Gemini reports token counts in `usageMetadata` on its final event. Mirror + // OpenAI's `stream_options.include_usage` terminal chunk (top-level `usage`, + // empty `choices`) so the frontend's `'usage' in chunk` path records them. + if let Some(usage) = &parsed.usage { + let prompt_tokens = usage.prompt_token_count.unwrap_or(0); + let completion_tokens = usage.candidates_token_count.unwrap_or(0); + let total_tokens = usage + .total_token_count + .unwrap_or(prompt_tokens + completion_tokens); + let chunk = serde_json::json!({ + "id": id, + "object": "chat.completion.chunk", + "model": model, + "choices": [], + "usage": { + "prompt_tokens": prompt_tokens, + "completion_tokens": completion_tokens, + "total_tokens": total_tokens, + } + }); + chunks.push(format!("data: {}\n\n", chunk)); + } + chunks } @@ -718,9 +781,16 @@ fn extract_candidates_into(candidates: &[GeminiSSECandidate], parsed: &mut Gemin for part in parts { if let Some(text) = &part.text { if !text.is_empty() { - match parsed.text.as_mut() { + // Thought-summary parts go to the reasoning channel, + // not the answer. + let target = if part.thought == Some(true) { + &mut parsed.reasoning + } else { + &mut parsed.text + }; + match target.as_mut() { Some(existing) => existing.push_str(text), - None => parsed.text = Some(text.clone()), + None => *target = Some(text.clone()), } } } @@ -788,10 +858,19 @@ fn openai_tool_call_json( #[cfg(test)] mod tests { use super::{ - gemini_event_to_openai_sse_chunks, gemini_response_to_openai, sanitize_schema_for_google, - GeminiParsedEvent, GeminiToolCallEvent, + gemini_event_to_openai_sse_chunks, gemini_response_to_openai, parse_gemini_sse_event, + sanitize_schema_for_google, GeminiParsedEvent, GeminiToolCallEvent, GeminiUsageMetadata, }; + /// Strip the `data: ...\n\n` SSE framing and parse the payload as JSON. + fn parse_sse_chunk(chunk: &str) -> serde_json::Value { + let payload = chunk + .strip_prefix("data: ") + .and_then(|c| c.strip_suffix("\n\n")) + .expect("chunk should be wrapped as SSE data"); + serde_json::from_str(payload).expect("chunk should be valid JSON") + } + #[test] fn gemini_response_to_openai_preserves_thought_signature() { let parsed = GeminiParsedEvent { @@ -856,6 +935,147 @@ mod tests { assert_eq!(tool_call["index"], 0); } + #[test] + fn gemini_streaming_emits_usage_chunk() { + let parsed = GeminiParsedEvent { + text: Some("the answer".to_string()), + usage: Some(GeminiUsageMetadata { + prompt_token_count: Some(12), + candidates_token_count: Some(7), + total_token_count: Some(19), + }), + ..Default::default() + }; + + let mut tool_call_index = 0; + let chunks = gemini_event_to_openai_sse_chunks( + &parsed, + "chatcmpl-test", + "gemini-3-flash-preview", + &mut tool_call_index, + ); + + // Mirrors OpenAI's `stream_options.include_usage`: a terminal chunk with + // top-level `usage` and empty `choices`. + let usage_chunk = chunks + .iter() + .map(|c| parse_sse_chunk(c)) + .find(|v| v.get("usage").map(|u| !u.is_null()).unwrap_or(false)) + .expect("a chunk should carry top-level usage"); + + assert_eq!(usage_chunk["usage"]["prompt_tokens"], 12); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 7); + assert_eq!(usage_chunk["usage"]["total_tokens"], 19); + assert_eq!(usage_chunk["choices"], serde_json::json!([])); + } + + #[test] + fn gemini_streaming_usage_total_falls_back_to_prompt_plus_completion() { + let parsed = GeminiParsedEvent { + usage: Some(GeminiUsageMetadata { + prompt_token_count: Some(5), + candidates_token_count: Some(3), + total_token_count: None, + }), + ..Default::default() + }; + + let mut tool_call_index = 0; + let chunks = gemini_event_to_openai_sse_chunks( + &parsed, + "chatcmpl-test", + "gemini-3-flash-preview", + &mut tool_call_index, + ); + + let usage_chunk = chunks + .iter() + .map(|c| parse_sse_chunk(c)) + .find(|v| v.get("usage").map(|u| !u.is_null()).unwrap_or(false)) + .expect("a chunk should carry top-level usage"); + + assert_eq!(usage_chunk["usage"]["prompt_tokens"], 5); + assert_eq!(usage_chunk["usage"]["completion_tokens"], 3); + assert_eq!(usage_chunk["usage"]["total_tokens"], 8); + } + + #[test] + fn gemini_streaming_without_usage_emits_no_usage_chunk() { + let parsed = GeminiParsedEvent { + text: Some("the answer".to_string()), + ..Default::default() + }; + + let mut tool_call_index = 0; + let chunks = gemini_event_to_openai_sse_chunks( + &parsed, + "chatcmpl-test", + "gemini-3-flash-preview", + &mut tool_call_index, + ); + + assert!( + chunks + .iter() + .map(|c| parse_sse_chunk(c)) + .all(|v| v.get("usage").map(|u| u.is_null()).unwrap_or(true)), + "no usage chunk should be emitted when the event has no usageMetadata" + ); + } + + #[test] + fn gemini_thought_parts_route_to_reasoning() { + let event = r#"{ + "candidates": [{ + "content": { + "parts": [ + {"text": "summary of my thinking", "thought": true}, + {"text": "the answer"} + ] + } + }] + }"#; + let parsed = parse_gemini_sse_event(event) + .expect("parse succeeds") + .expect("event is recognised"); + assert_eq!(parsed.reasoning.as_deref(), Some("summary of my thinking")); + assert_eq!(parsed.text.as_deref(), Some("the answer")); + + let mut tool_call_index = 0; + let chunks = gemini_event_to_openai_sse_chunks( + &parsed, + "chatcmpl-test", + "gemini-3-flash-preview", + &mut tool_call_index, + ); + assert_eq!(chunks.len(), 2); + let reasoning_chunk: serde_json::Value = serde_json::from_str( + chunks[0] + .strip_prefix("data: ") + .and_then(|c| c.strip_suffix("\n\n")) + .unwrap(), + ) + .unwrap(); + assert_eq!( + reasoning_chunk["choices"][0]["delta"]["reasoning_content"], + "summary of my thinking" + ); + let text_chunk: serde_json::Value = serde_json::from_str( + chunks[1] + .strip_prefix("data: ") + .and_then(|c| c.strip_suffix("\n\n")) + .unwrap(), + ) + .unwrap(); + assert_eq!(text_chunk["choices"][0]["delta"]["content"], "the answer"); + + // Non-streaming conversion keeps the channels separate too. + let response = gemini_response_to_openai(&parsed, "gemini-3-flash-preview"); + let message = &response["choices"][0]["message"]; + assert_eq!(message["content"], "the answer"); + assert_eq!(message["reasoning_content"], "summary of my thinking"); + } + #[test] fn sanitize_schema_for_google_removes_property_names() { let mut schema = serde_json::json!({ diff --git a/backend/windmill-ai/src/ai_types.rs b/backend/windmill-ai/src/ai_types.rs index 47c72fd3da..c6f171daf3 100644 --- a/backend/windmill-ai/src/ai_types.rs +++ b/backend/windmill-ai/src/ai_types.rs @@ -101,11 +101,30 @@ pub struct GoogleExtraContent { pub thought_signature: Option, } +/// Bedrock-specific extra content carrying the Claude reasoning block emitted +/// in the same assistant turn as a tool call. Anthropic requires reasoning +/// blocks (text + signature, unmodified) to be replayed before `toolUse` when +/// thinking is enabled, so the proxy round-trips them through the +/// OpenAI-shaped tool call. +#[derive(Deserialize, Serialize, Clone, Debug, Default)] +pub struct BedrockExtraContent { + #[serde(skip_serializing_if = "Option::is_none")] + pub reasoning_text: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub signature: Option, + /// Base64 of a redacted (encrypted) reasoning block, when the provider + /// returned one instead of readable text. + #[serde(skip_serializing_if = "Option::is_none")] + pub redacted_content: Option, +} + /// Extra content for provider-specific metadata (e.g., Google thought signatures) #[derive(Deserialize, Serialize, Clone, Debug, Default)] pub struct ExtraContent { #[serde(skip_serializing_if = "Option::is_none")] pub google: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub bedrock: Option, } #[derive(Deserialize, Serialize, Clone, Debug)] diff --git a/backend/windmill-ai/src/credentials.rs b/backend/windmill-ai/src/credentials.rs index 75d523cf25..c9502bdbb2 100644 --- a/backend/windmill-ai/src/credentials.rs +++ b/backend/windmill-ai/src/credentials.rs @@ -20,6 +20,5 @@ pub struct ProviderCredentials { pub aws_secret_access_key: Option, pub aws_session_token: Option, pub platform: AIPlatform, - pub enable_1m_context: bool, pub custom_headers: HashMap, } diff --git a/backend/windmill-ai/src/providers/anthropic.rs b/backend/windmill-ai/src/providers/anthropic.rs index 17a27e370f..05398ce212 100644 --- a/backend/windmill-ai/src/providers/anthropic.rs +++ b/backend/windmill-ai/src/providers/anthropic.rs @@ -369,12 +369,11 @@ pub struct AnthropicQueryBuilder { #[allow(dead_code)] provider_kind: AIProvider, platform: AIPlatform, - enable_1m_context: bool, } impl AnthropicQueryBuilder { - pub fn new(provider_kind: AIProvider, platform: AIPlatform, enable_1m_context: bool) -> Self { - Self { provider_kind, platform, enable_1m_context } + pub fn new(provider_kind: AIProvider, platform: AIPlatform) -> Self { + Self { provider_kind, platform } } fn is_vertex(&self) -> bool { @@ -456,13 +455,6 @@ impl AnthropicQueryBuilder { } } - if is_anthropic_sdk && credentials.enable_1m_context { - headers.push(( - "anthropic-beta".to_string(), - "context-1m-2025-08-07".to_string(), - )); - } - if let Some(api_key) = credentials.api_key.as_ref() { headers.push(("authorization".to_string(), format!("Bearer {}", api_key))); if !is_vertex { @@ -713,14 +705,10 @@ impl QueryBuilder for AnthropicQueryBuilder { vec![("Authorization", format!("Bearer {}", api_key))] } else { // Standard Anthropic API uses x-api-key and anthropic-version header - let mut headers = vec![ + vec![ ("x-api-key", api_key.to_string()), ("anthropic-version", ANTHROPIC_VERSION_STANDARD.to_string()), - ]; - if self.enable_1m_context { - headers.push(("anthropic-beta", "context-1m-2025-08-07".to_string())); - } - headers + ] } } } @@ -747,7 +735,6 @@ mod tests { aws_secret_access_key: None, aws_session_token: None, platform, - enable_1m_context: false, custom_headers: HashMap::new(), } } @@ -762,7 +749,7 @@ mod tests { fn builds_standard_anthropic_proxy_request() { let credentials = credentials(AIPlatform::Standard); let builder = - AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard, false); + AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard); let method = Method::POST; let mut headers = HeaderMap::new(); headers.insert("anthropic-version", HeaderValue::from_static("2023-06-01")); @@ -794,41 +781,13 @@ mod tests { )); } - #[test] - fn builds_anthropic_sdk_proxy_request_with_context_beta() { - let mut credentials = credentials(AIPlatform::Standard); - credentials.enable_1m_context = true; - let builder = AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::Standard, true); - let method = Method::POST; - let mut headers = HeaderMap::new(); - headers.insert("X-Anthropic-SDK", HeaderValue::from_static("typescript")); - let body = br#"{"model":"claude-sonnet-4","messages":[]}"#; - - let request = builder - .build_proxy_request(&ProxyBuildArgs { - method: &method, - path: "v1/messages", - headers: &headers, - body, - credentials: &credentials, - }) - .unwrap(); - - assert_eq!(request.url, "https://api.anthropic.com/v1/messages"); - assert!(has_header( - &request.headers, - "anthropic-beta", - "context-1m-2025-08-07" - )); - } - #[test] fn builds_vertex_anthropic_proxy_request() { let mut credentials = credentials(AIPlatform::GoogleVertexAi); credentials.base_url = "https://us-central1-aiplatform.googleapis.com/v1/projects/p/locations/us-central1/publishers/anthropic/models".to_string(); credentials.user = Some("user-1".to_string()); let builder = - AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::GoogleVertexAi, false); + AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::GoogleVertexAi); let method = Method::POST; let mut headers = HeaderMap::new(); headers.insert("anthropic-version", HeaderValue::from_static("2023-06-01")); @@ -872,7 +831,7 @@ mod tests { fn rejects_vertex_proxy_request_without_model() { let credentials = credentials(AIPlatform::GoogleVertexAi); let builder = - AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::GoogleVertexAi, false); + AnthropicQueryBuilder::new(AIProvider::Anthropic, AIPlatform::GoogleVertexAi); let method = Method::POST; let headers = HeaderMap::new(); diff --git a/backend/windmill-ai/src/providers/bedrock.rs b/backend/windmill-ai/src/providers/bedrock.rs index 0eef359c36..ddc5f33998 100644 --- a/backend/windmill-ai/src/providers/bedrock.rs +++ b/backend/windmill-ai/src/providers/bedrock.rs @@ -12,11 +12,14 @@ use crate::{ bedrock_stream_event_to_text, bedrock_stream_event_to_tool_delta, bedrock_stream_event_to_tool_delta_with_block_index, bedrock_stream_event_to_tool_start, bedrock_stream_event_to_tool_start_with_block_index, build_tool_config, - create_inference_config, format_bedrock_error, openai_messages_to_bedrock, - streaming_tool_calls_to_openai, BearerTokenProvider, BedrockClient, StreamingToolCall, + create_inference_config, format_bedrock_error, json_to_document, + openai_messages_to_bedrock, streaming_tool_calls_to_openai, BearerTokenProvider, + BedrockClient, StreamingToolCall, }, ai_providers::USE_ENV_REGION, - ai_types::{OpenAIFunction, OpenAIToolCall, ToolDefFunction}, + ai_types::{ + BedrockExtraContent, ExtraContent, OpenAIFunction, OpenAIToolCall, ToolDefFunction, + }, image_handler::prepare_messages_for_api, proxy::ProxyBuildArgs, query_builder::{ParsedResponse, StreamEventSink}, @@ -45,6 +48,11 @@ struct OpenAIRequest { max_tokens: Option, #[serde(default)] temperature: Option, + /// Anthropic effort token from the chat reasoning setting (e.g. `low`, + /// `high`, `max`). Enables adaptive thinking via + /// `additionalModelRequestFields` — see [`bedrock_thinking_fields`]. + #[serde(default)] + reasoning_effort: Option, } #[derive(Deserialize, Debug)] @@ -349,7 +357,13 @@ async fn handle_bedrock_sdk_streaming( let enable_prompt_caching = bedrock_model_supports_prompt_caching(model); let (bedrock_messages, system_prompts) = openai_messages_to_bedrock(&openai_req.messages, enable_prompt_caching)?; - let inference_config = create_inference_config(openai_req.temperature, openai_req.max_tokens); + // Adaptive thinking rejects sampling params; drop temperature when reasoning is on. + let temperature = openai_req + .reasoning_effort + .is_none() + .then_some(openai_req.temperature) + .flatten(); + let inference_config = create_inference_config(temperature, openai_req.max_tokens); let tool_config = build_tool_config_from_request( openai_req.tools.as_deref(), openai_req.tool_choice.as_ref(), @@ -374,6 +388,11 @@ async fn handle_bedrock_sdk_streaming( request_builder = request_builder.set_tool_config(Some(config)); } + if let Some(effort) = openai_req.reasoning_effort.as_deref() { + request_builder = + request_builder.additional_model_request_fields(bedrock_thinking_fields(effort)); + } + tracing::debug!("Bedrock SDK streaming: sending converse_stream request"); let stream_output = request_builder.send().await.map_err(|e| { let error_msg = format!("Bedrock SDK streaming error: {}", format_bedrock_error(&e)); @@ -391,6 +410,17 @@ async fn handle_bedrock_sdk_streaming( }) } +/// Build the Converse `additionalModelRequestFields` enabling Claude adaptive +/// thinking at the given effort. `display: summarized` is billing-neutral on +/// Anthropic models and matches the direct-Anthropic chat path, which renders +/// summarized thinking in the UI. +fn bedrock_thinking_fields(effort: &str) -> aws_smithy_types::Document { + json_to_document(serde_json::json!({ + "thinking": { "type": "adaptive", "display": "summarized" }, + "output_config": { "effort": effort } + })) +} + pub fn sdk_stream_to_sse( stream: aws_sdk_bedrockruntime::primitives::event_stream::EventReceiver< aws_sdk_bedrockruntime::types::ConverseStreamOutput, @@ -438,6 +468,11 @@ struct BedrockSseStreamState { tool_calls: HashMap, tool_block_indexes: HashMap, next_tool_index: usize, + /// Claude reasoning block accumulated from `ReasoningContent` deltas + /// (text + signature, or redacted bytes). Attached to the first tool call + /// of the turn so the frontend round-trips it for replay. + reasoning: Option, + reasoning_attached: bool, } impl BedrockSseStreamState { @@ -449,6 +484,8 @@ impl BedrockSseStreamState { tool_calls: HashMap::new(), tool_block_indexes: HashMap::new(), next_tool_index: 0, + reasoning: None, + reasoning_attached: false, } } } @@ -459,6 +496,24 @@ fn bedrock_sse_chunks_for_event( ) -> Vec { let mut chunks = Vec::new(); + if let Some(reasoning_text) = accumulate_reasoning_delta(event, state) { + let chunk = serde_json::json!({ + "id": state.id, + "object": "chat.completion.chunk", + "created": state.created, + "model": state.model, + "choices": [{ + "index": 0, + "delta": { + "reasoning_content": reasoning_text + }, + "finish_reason": serde_json::Value::Null + }] + }); + + chunks.push(Bytes::from(format!("data: {}\n\n", chunk))); + } + if let Some((block_index, tool_call)) = bedrock_stream_event_to_tool_start_with_block_index(event) { @@ -470,6 +525,25 @@ fn bedrock_sse_chunks_for_event( (tool_call.id.clone(), tool_call.name.clone(), String::new()), ); + let mut tool_call_json = serde_json::json!({ + "index": index, + "id": tool_call.id, + "type": "function", + "function": { + "name": tool_call.name, + "arguments": "" + } + }); + // Attach the turn's reasoning block to the first tool call so the + // frontend echoes it back and the next request can replay it before + // toolUse (required by Claude when thinking is enabled). + if !state.reasoning_attached { + if let Some(reasoning) = state.reasoning.as_ref() { + tool_call_json["extra_content"] = serde_json::json!({ "bedrock": reasoning }); + state.reasoning_attached = true; + } + } + let chunk = serde_json::json!({ "id": state.id, "object": "chat.completion.chunk", @@ -478,15 +552,7 @@ fn bedrock_sse_chunks_for_event( "choices": [{ "index": 0, "delta": { - "tool_calls": [{ - "index": index, - "id": tool_call.id, - "type": "function", - "function": { - "name": tool_call.name, - "arguments": "" - } - }] + "tool_calls": [tool_call_json] }, "finish_reason": serde_json::Value::Null }] @@ -573,6 +639,61 @@ fn bedrock_sse_chunks_for_event( chunks } +/// Fold a `ReasoningContent` stream delta into the state's pending reasoning +/// block. Returns the text delta (for a `reasoning_content` SSE chunk) when the +/// event carried readable reasoning text. +fn accumulate_reasoning_delta( + event: &aws_sdk_bedrockruntime::types::ConverseStreamOutput, + state: &mut BedrockSseStreamState, +) -> Option { + let aws_sdk_bedrockruntime::types::ConverseStreamOutput::ContentBlockDelta(delta_event) = event + else { + return None; + }; + let aws_sdk_bedrockruntime::types::ContentBlockDelta::ReasoningContent(reasoning) = + delta_event.delta()? + else { + return None; + }; + + let entry = state.reasoning.get_or_insert_with(Default::default); + match reasoning { + aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Text(text) => { + entry + .reasoning_text + .get_or_insert_with(String::new) + .push_str(text); + Some(text.clone()) + } + aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::Signature(signature) => { + entry + .signature + .get_or_insert_with(String::new) + .push_str(signature); + None + } + aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta::RedactedContent(blob) => { + // Base64 of concatenated fragments != concatenated base64 fragments, + // so accumulate raw bytes and re-encode. + let mut bytes = entry + .redacted_content + .as_deref() + .and_then(|existing| { + base64::Engine::decode(&base64::engine::general_purpose::STANDARD, existing) + .ok() + }) + .unwrap_or_default(); + bytes.extend_from_slice(blob.as_ref()); + entry.redacted_content = Some(base64::Engine::encode( + &base64::engine::general_purpose::STANDARD, + bytes, + )); + None + } + _ => None, + } +} + async fn handle_bedrock_sdk_non_streaming( model: &str, body: &[u8], @@ -586,7 +707,13 @@ async fn handle_bedrock_sdk_non_streaming( let enable_prompt_caching = bedrock_model_supports_prompt_caching(model); let (bedrock_messages, system_prompts) = openai_messages_to_bedrock(&openai_req.messages, enable_prompt_caching)?; - let inference_config = create_inference_config(openai_req.temperature, openai_req.max_tokens); + // Adaptive thinking rejects sampling params; drop temperature when reasoning is on. + let temperature = openai_req + .reasoning_effort + .is_none() + .then_some(openai_req.temperature) + .flatten(); + let inference_config = create_inference_config(temperature, openai_req.max_tokens); let tool_config = build_tool_config_from_request( openai_req.tools.as_deref(), openai_req.tool_choice.as_ref(), @@ -611,6 +738,11 @@ async fn handle_bedrock_sdk_non_streaming( request_builder = request_builder.set_tool_config(Some(config)); } + if let Some(effort) = openai_req.reasoning_effort.as_deref() { + request_builder = + request_builder.additional_model_request_fields(bedrock_thinking_fields(effort)); + } + tracing::debug!("Bedrock SDK non-streaming: sending converse request"); let response = request_builder.send().await.map_err(|e| { let error_msg = format!( @@ -643,6 +775,7 @@ async fn handle_bedrock_sdk_non_streaming( let mut text_content = String::new(); let mut tool_calls: Vec = Vec::new(); + let mut reasoning: Option = None; if let Some(aws_sdk_bedrockruntime::types::ConverseOutput::Message(message)) = response.output() { @@ -651,6 +784,29 @@ async fn handle_bedrock_sdk_non_streaming( aws_sdk_bedrockruntime::types::ContentBlock::Text(text) => { text_content.push_str(text); } + aws_sdk_bedrockruntime::types::ContentBlock::ReasoningContent(rc) => { + let entry = reasoning.get_or_insert_with(Default::default); + match rc { + aws_sdk_bedrockruntime::types::ReasoningContentBlock::ReasoningText(rt) => { + entry + .reasoning_text + .get_or_insert_with(String::new) + .push_str(rt.text()); + if let Some(signature) = rt.signature() { + entry.signature = Some(signature.to_string()); + } + } + aws_sdk_bedrockruntime::types::ReasoningContentBlock::RedactedContent( + blob, + ) => { + entry.redacted_content = Some(base64::Engine::encode( + &base64::engine::general_purpose::STANDARD, + blob.as_ref(), + )); + } + _ => {} + } + } aws_sdk_bedrockruntime::types::ContentBlock::ToolUse(tool_use) => { let input_json = document_to_json(tool_use.input()); tool_calls.push(OpenAIToolCall { @@ -668,7 +824,20 @@ async fn handle_bedrock_sdk_non_streaming( } } - let message = if !tool_calls.is_empty() { + // Attach the turn's reasoning block to the first tool call so the frontend + // round-trips it for replay (required by Claude when thinking is enabled). + if let (Some(reasoning_block), Some(first_tool_call)) = + (reasoning.as_ref(), tool_calls.first_mut()) + { + first_tool_call.extra_content = + Some(ExtraContent { bedrock: Some(reasoning_block.clone()), ..Default::default() }); + } + let reasoning_content = reasoning + .as_ref() + .and_then(|r| r.reasoning_text.clone()) + .filter(|t| !t.is_empty()); + + let mut message = if !tool_calls.is_empty() { serde_json::json!({ "role": "assistant", "content": if text_content.is_empty() { serde_json::Value::Null } else { serde_json::Value::String(text_content) }, @@ -680,6 +849,9 @@ async fn handle_bedrock_sdk_non_streaming( "content": text_content }) }; + if let Some(reasoning_content) = reasoning_content { + message["reasoning_content"] = serde_json::Value::String(reasoning_content); + } let usage = if let Some(usage_data) = response.usage() { serde_json::json!({ @@ -1122,4 +1294,124 @@ mod tests { 0 ); } + + #[test] + fn bedrock_thinking_fields_carry_adaptive_thinking_and_effort() { + let fields = document_to_json(&bedrock_thinking_fields("xhigh")); + assert_eq!(fields["thinking"]["type"], "adaptive"); + assert_eq!(fields["thinking"]["display"], "summarized"); + assert_eq!(fields["output_config"]["effort"], "xhigh"); + } + + fn reasoning_delta( + block_index: i32, + delta: aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta, + ) -> ConverseStreamOutput { + ConverseStreamOutput::ContentBlockDelta( + ContentBlockDeltaEvent::builder() + .content_block_index(block_index) + .delta(ContentBlockDelta::ReasoningContent(delta)) + .build() + .unwrap(), + ) + } + + #[test] + fn bedrock_sse_streams_reasoning_and_attaches_block_to_first_tool_call() { + use aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta; + + let mut state = + BedrockSseStreamState::new("chatcmpl-test".to_string(), "model".to_string(), 1); + + // Reasoning text streams as reasoning_content deltas. + let chunks = bedrock_sse_chunks_for_event( + &reasoning_delta(0, ReasoningContentBlockDelta::Text("let me ".to_string())), + &mut state, + ); + assert_eq!( + sse_json(&chunks[0])["choices"][0]["delta"]["reasoning_content"], + "let me " + ); + bedrock_sse_chunks_for_event( + &reasoning_delta(0, ReasoningContentBlockDelta::Text("think".to_string())), + &mut state, + ); + // Signature deltas accumulate silently (no chunk emitted). + assert!(bedrock_sse_chunks_for_event( + &reasoning_delta( + 0, + ReasoningContentBlockDelta::Signature("sig-abc".to_string()) + ), + &mut state, + ) + .is_empty()); + + // The first tool call carries the full reasoning block for replay. + let tool_start = ConverseStreamOutput::ContentBlockStart( + ContentBlockStartEvent::builder() + .content_block_index(1) + .start(ContentBlockStart::ToolUse( + ToolUseBlockStart::builder() + .tool_use_id("call_1") + .name("lookup") + .build() + .unwrap(), + )) + .build() + .unwrap(), + ); + let start_json = sse_json(&bedrock_sse_chunks_for_event(&tool_start, &mut state)[0]); + let tool_call = &start_json["choices"][0]["delta"]["tool_calls"][0]; + assert_eq!( + tool_call["extra_content"]["bedrock"]["reasoning_text"], + "let me think" + ); + assert_eq!( + tool_call["extra_content"]["bedrock"]["signature"], + "sig-abc" + ); + + // Subsequent tool calls don't repeat the block. + let second_tool_start = ConverseStreamOutput::ContentBlockStart( + ContentBlockStartEvent::builder() + .content_block_index(2) + .start(ContentBlockStart::ToolUse( + ToolUseBlockStart::builder() + .tool_use_id("call_2") + .name("lookup") + .build() + .unwrap(), + )) + .build() + .unwrap(), + ); + let second_json = + sse_json(&bedrock_sse_chunks_for_event(&second_tool_start, &mut state)[0]); + assert!(second_json["choices"][0]["delta"]["tool_calls"][0] + .get("extra_content") + .is_none()); + } + + #[test] + fn bedrock_sse_accumulates_redacted_reasoning_bytes() { + use aws_sdk_bedrockruntime::types::ReasoningContentBlockDelta; + + let mut state = + BedrockSseStreamState::new("chatcmpl-test".to_string(), "model".to_string(), 1); + for fragment in [b"ab".as_slice(), b"cd".as_slice()] { + assert!(bedrock_sse_chunks_for_event( + &reasoning_delta( + 0, + ReasoningContentBlockDelta::RedactedContent(fragment.to_vec().into()), + ), + &mut state, + ) + .is_empty()); + } + + let encoded = state.reasoning.unwrap().redacted_content.unwrap(); + let decoded = + base64::Engine::decode(&base64::engine::general_purpose::STANDARD, encoded).unwrap(); + assert_eq!(decoded, b"abcd"); + } } diff --git a/backend/windmill-ai/src/providers/google_ai.rs b/backend/windmill-ai/src/providers/google_ai.rs index cf00ddc142..650e11c0c7 100644 --- a/backend/windmill-ai/src/providers/google_ai.rs +++ b/backend/windmill-ai/src/providers/google_ai.rs @@ -4,7 +4,7 @@ use crate::{ openai_tools_to_gemini, parse_gemini_response, parse_gemini_sse_event, sanitize_schema_for_google, GeminiFunctionDeclaration, GeminiGenerationConfig, GeminiImageContent, GeminiImageRequest, GeminiImageResponse, GeminiInlineData, GeminiPart, - GeminiPredictContent, GeminiTextRequest, GeminiTool, + GeminiPredictContent, GeminiTextRequest, GeminiThinkingConfig, GeminiTool, }, image_handler::{download_and_encode_s3_image, prepare_messages_for_api}, proxy::{ProxyBuildArgs, ProxyRequest}, @@ -158,6 +158,8 @@ impl GoogleAIQueryBuilder { args.max_tokens, response_mime_type, response_schema, + // Worker AI-agent requests carry no reasoning knob; keep provider defaults. + None, ) } } @@ -187,23 +189,73 @@ fn build_gemini_generation_config( max_tokens: Option, response_mime_type: Option, response_schema: Option, + thinking_config: Option, ) -> Option { if temperature.is_some() || max_tokens.is_some() || response_mime_type.is_some() || response_schema.is_some() + || thinking_config.is_some() { Some(GeminiGenerationConfig { temperature, max_output_tokens: max_tokens, response_mime_type, response_schema, + thinking_config, }) } else { None } } +/// Map an OpenAI-style `reasoning_effort` token onto Gemini's native thinking +/// controls. Gemini models think by default, so this is what makes the chat +/// effort setting (including explicit `none`) actually change model behavior. +/// +/// - Gemini 3+: `thinkingLevel` takes the token verbatim (provider-native open +/// vocabulary: `low`/`medium`/`high`, plus `minimal` on Flash). `none` maps to +/// the lowest supported level — thinking cannot be fully disabled on Pro. +/// - Gemini 2.5: `thinkingBudget` in tokens, using the same tiering as Google's +/// own OpenAI-compatibility layer. 2.5 Pro cannot disable thinking and has a +/// minimum budget of 128; Flash accepts 0 (off). Unknown tokens fall back to +/// `-1` (dynamic, the provider default). +fn gemini_thinking_config(model: &str, effort: &str) -> GeminiThinkingConfig { + let model = model.to_lowercase(); + let is_flash = model.contains("flash"); + let is_gemini_2_5 = model.contains("gemini-2.5"); + // Thought summaries are free to return (thinking tokens are billed either + // way); skip them only when the user explicitly turned reasoning off. + let include_thoughts = (effort != "none").then_some(true); + + if is_gemini_2_5 { + let budget = match effort { + "none" if is_flash => 0, + "none" => 128, + "low" => 1024, + "medium" => 8192, + "high" => 24576, + _ => -1, + }; + GeminiThinkingConfig { + thinking_level: None, + thinking_budget: Some(budget), + include_thoughts, + } + } else { + let level = match effort { + "none" if is_flash => "minimal".to_string(), + "none" => "low".to_string(), + other => other.to_string(), + }; + GeminiThinkingConfig { + thinking_level: Some(level), + thinking_budget: None, + include_thoughts, + } + } +} + #[derive(Deserialize, Debug)] struct GoogleAIProxyChatRequest { model: String, @@ -214,6 +266,11 @@ struct GoogleAIProxyChatRequest { temperature: Option, #[serde(default)] max_tokens: Option, + /// OpenAI-style effort token from the chat reasoning setting (e.g. `low`, + /// `high`, or `none` for an explicit off). Mapped to Gemini's native + /// `thinkingConfig` — see [`gemini_thinking_config`]. + #[serde(default)] + reasoning_effort: Option, #[serde(default)] tools: Option>, } @@ -344,10 +401,20 @@ fn build_google_ai_chat_proxy_request( vec![GeminiTool { function_declarations: Some(declarations), google_search: None }] }); + let thinking_config = request + .reasoning_effort + .as_deref() + .map(|effort| gemini_thinking_config(&request.model, effort)); let body = build_gemini_text_request_body( &request.messages, gemini_tools, - build_gemini_generation_config(request.temperature, request.max_tokens, None, None), + build_gemini_generation_config( + request.temperature, + request.max_tokens, + None, + None, + thinking_config, + ), )? .into_bytes(); @@ -707,7 +774,6 @@ mod tests { aws_secret_access_key: None, aws_session_token: None, platform, - enable_1m_context: false, custom_headers: HashMap::new(), } } @@ -752,9 +818,108 @@ mod tests { let body: serde_json::Value = serde_json::from_slice(&request.request.body).unwrap(); assert_eq!(body["generationConfig"]["maxOutputTokens"], 123); assert_eq!(body["generationConfig"]["temperature"], 0.2); + // No reasoning_effort in the request -> no thinkingConfig (provider defaults). + assert!(body["generationConfig"].get("thinkingConfig").is_none()); assert!(body["contents"].is_array()); } + fn proxy_body_for(model: &str, reasoning_effort: &str) -> serde_json::Value { + let credentials = credentials( + "https://generativelanguage.googleapis.com/v1beta/", + AIPlatform::Standard, + ); + let method = Method::POST; + let headers = HeaderMap::new(); + let body = format!( + r#"{{ + "model": "{model}", + "messages": [{{"role": "user", "content": "hello"}}], + "reasoning_effort": "{reasoning_effort}", + "stream": false + }}"# + ); + + let request = build_google_ai_chat_proxy_request(&ProxyBuildArgs { + method: &method, + path: "chat/completions", + headers: &headers, + body: body.as_bytes(), + credentials: &credentials, + }) + .unwrap(); + + serde_json::from_slice(&request.request.body).unwrap() + } + + #[test] + fn maps_reasoning_effort_to_thinking_level_on_gemini_3() { + let body = proxy_body_for("gemini-3-pro-preview", "low"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingLevel"], + "low" + ); + // Thought summaries are requested whenever reasoning is on... + assert_eq!( + body["generationConfig"]["thinkingConfig"]["includeThoughts"], + true + ); + // ...but not when the user explicitly turned it off. + let body = proxy_body_for("gemini-3-pro-preview", "none"); + assert!(body["generationConfig"]["thinkingConfig"] + .get("includeThoughts") + .is_none()); + assert!(body["generationConfig"]["thinkingConfig"] + .get("thinkingBudget") + .is_none()); + } + + #[test] + fn maps_reasoning_effort_none_per_gemini_3_model() { + // Pro cannot disable thinking -> lowest level. + let body = proxy_body_for("gemini-3-pro-preview", "none"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingLevel"], + "low" + ); + // Flash supports `minimal`. + let body = proxy_body_for("gemini-3-flash-preview", "none"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingLevel"], + "minimal" + ); + } + + #[test] + fn maps_reasoning_effort_to_thinking_budget_on_gemini_2_5() { + let body = proxy_body_for("gemini-2.5-flash", "medium"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingBudget"], + 8192 + ); + assert!(body["generationConfig"]["thinkingConfig"] + .get("thinkingLevel") + .is_none()); + + // Off: Flash can fully disable; Pro has a 128-token minimum. + let body = proxy_body_for("gemini-2.5-flash", "none"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingBudget"], + 0 + ); + let body = proxy_body_for("gemini-2.5-pro", "none"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingBudget"], + 128 + ); + + // Unknown token -> dynamic budget (provider default behavior). + let body = proxy_body_for("gemini-2.5-pro", "custom-level"); + assert_eq!( + body["generationConfig"]["thinkingConfig"]["thinkingBudget"], + -1 + ); + } + #[test] fn builds_standard_google_ai_endpoint_from_model_resource_name() { assert_eq!( diff --git a/backend/windmill-ai/src/providers/mod.rs b/backend/windmill-ai/src/providers/mod.rs index fb50707221..80fbc8fcca 100644 --- a/backend/windmill-ai/src/providers/mod.rs +++ b/backend/windmill-ai/src/providers/mod.rs @@ -23,7 +23,6 @@ pub fn create_query_builder(credentials: &ProviderCredentials) -> Box Box::new(AnthropicQueryBuilder::new( credentials.provider.clone(), credentials.platform.clone(), - credentials.enable_1m_context, )), AIProvider::OpenRouter => Box::new(OpenRouterQueryBuilder::new()), _ => Box::new(OtherQueryBuilder::new(credentials.provider.clone())), diff --git a/backend/windmill-ai/src/proxy.rs b/backend/windmill-ai/src/proxy.rs index 32ba35cd6d..891fa89486 100644 --- a/backend/windmill-ai/src/proxy.rs +++ b/backend/windmill-ai/src/proxy.rs @@ -165,7 +165,6 @@ mod tests { aws_secret_access_key: None, aws_session_token: None, platform: AIPlatform::Standard, - enable_1m_context: false, custom_headers: HashMap::new(), } } diff --git a/backend/windmill-ai/src/sse.rs b/backend/windmill-ai/src/sse.rs index 35a4e43acc..bb1b9d9731 100644 --- a/backend/windmill-ai/src/sse.rs +++ b/backend/windmill-ai/src/sse.rs @@ -522,6 +522,7 @@ impl SSEParser for GeminiSSEParser { let extra_content = tool_call.thought_signature.map(|sig| ExtraContent { google: Some(GoogleExtraContent { thought_signature: Some(sig) }), + bedrock: None, }); self.accumulated_tool_calls.insert( diff --git a/backend/windmill-ai/src/types.rs b/backend/windmill-ai/src/types.rs index 56a796e41d..8f864468db 100644 --- a/backend/windmill-ai/src/types.rs +++ b/backend/windmill-ai/src/types.rs @@ -193,9 +193,6 @@ pub struct ProviderResource { /// Platform (standard or google_vertex_ai) #[serde(default)] pub platform: AIPlatform, - /// Enable 1M context window for Anthropic - #[serde(alias = "enable_1M_context", default)] - pub enable_1m_context: bool, /// Custom HTTP headers to include in AI requests #[serde(default)] pub headers: HashMap, @@ -246,7 +243,6 @@ impl ProviderWithResource { aws_secret_access_key: self.resource.aws_secret_access_key.clone(), aws_session_token: self.resource.aws_session_token.clone(), platform: self.resource.platform.clone(), - enable_1m_context: self.resource.enable_1m_context, custom_headers: self.resource.headers.clone(), }) } @@ -275,10 +271,6 @@ impl ProviderWithResource { &self.resource.platform } - pub fn get_enable_1m_context(&self) -> bool { - self.resource.enable_1m_context - } - pub fn get_headers(&self) -> &HashMap { &self.resource.headers } @@ -891,6 +883,26 @@ mod tests { } } + #[test] + fn provider_resource_ignores_legacy_enable_1m_context_keys() { + // Resources created before the field was removed still carry the legacy key + // (lowercase `enable_1m_context` or the frontend alias `enable_1M_context`). + // The struct has no `deny_unknown_fields`, so both must be silently ignored. + let json = r#"{ + "base_url": "https://api.anthropic.com", + "enable_1m_context": true, + "enable_1M_context": true + }"#; + + let resource: ProviderResource = + serde_json::from_str(json).expect("legacy resource must still deserialize"); + + assert_eq!( + resource.base_url.as_deref(), + Some("https://api.anthropic.com") + ); + } + #[test] fn test_make_strict_adds_additional_properties_false() { let mut schema = object_schema(vec![("name", string_schema())]); diff --git a/backend/windmill-api-assets/Cargo.toml b/backend/windmill-api-assets/Cargo.toml index 1ea16c3f0d..c6e7676b6b 100644 --- a/backend/windmill-api-assets/Cargo.toml +++ b/backend/windmill-api-assets/Cargo.toml @@ -11,8 +11,10 @@ path = "src/lib.rs" [dependencies] windmill-api-auth.workspace = true windmill-common = { workspace = true, default-features = false } +windmill-parser-sql-asset.workspace = true axum.workspace = true chrono.workspace = true serde.workspace = true serde_json.workspace = true sqlx.workspace = true +tracing.workspace = true diff --git a/backend/windmill-api-assets/src/lib.rs b/backend/windmill-api-assets/src/lib.rs index 0c25cd243d..29122bd793 100644 --- a/backend/windmill-api-assets/src/lib.rs +++ b/backend/windmill-api-assets/src/lib.rs @@ -7,7 +7,7 @@ use serde::{Deserialize, Serialize}; use serde_json::Value; use sqlx::Row; use windmill_common::{ - assets::{AssetKind, AssetUsageKind}, + assets::{parse_asset_trigger_ref, AssetKind, AssetUsageKind}, db::UserDB, error::JsonResult, utils::escape_ilike_pattern, @@ -20,6 +20,120 @@ pub fn workspaced_service() -> Router { .route("/list", get(list_assets)) .route("/list_by_usages", post(list_assets_by_usages)) .route("/list_favorites", get(list_favorites)) + .route("/graph", get(asset_graph)) + .route("/pipelines", get(list_pipeline_folders)) + .route("/partitions", get(list_partitions)) + .route("/asset_schemas", get(list_asset_schemas)) + .route("/record_materialization", post(record_materialization)) +} + +#[derive(Deserialize)] +struct PartitionsQuery { + // The materialized asset path (`/
`). + path: String, +} + +// Per-partition materialization status for a ducklake asset — drives the +// partition-status grid and the backfill worklist. Materialization targets are +// ducklake-only in v1, so the kind is fixed. +async fn list_partitions( + authed: ApiAuthed, + Path(w_id): Path, + Extension(user_db): Extension, + Query(q): Query, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = windmill_common::materialization::list_materialized_partitions( + &mut *tx, + &w_id, + AssetKind::Ducklake, + &q.path, + ) + .await?; + tx.commit().await?; + Ok(Json(rows)) +} + +// Per-asset captured output schema versions for a ducklake asset (gap #2a) — +// the schema-evolution history persisted after each managed `// materialize`. +// Newest version first; materialization targets are ducklake-only in v1, so the +// kind is fixed. +async fn list_asset_schemas( + authed: ApiAuthed, + Path(w_id): Path, + Extension(user_db): Extension, + Query(q): Query, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = windmill_common::materialization::list_asset_schemas( + &mut *tx, + &w_id, + AssetKind::Ducklake, + &q.path, + ) + .await?; + tx.commit().await?; + Ok(Json(rows)) +} + +// Record a materialization outcome from a polyglot (Python/TS) `wmill.ducklake` +// helper running as a pipeline step. The DuckDB `// materialize` engine records +// this itself; the SDK helpers post here instead so SDK-materialized slices show +// up in the grid identically. When the helper also captured the output schema, +// that schema version is upserted too. RLS-scoped to the caller's workspace. +async fn record_materialization( + authed: ApiAuthed, + Path(w_id): Path, + Extension(user_db): Extension, + Json(req): Json, +) -> JsonResult<()> { + let mut tx = user_db.clone().begin(&authed).await?; + windmill_common::materialization::record_materialization( + &mut *tx, + &w_id, + req.asset_kind, + &req.asset_path, + &req.partition, + req.status, + req.snapshot_id, + req.row_count, + req.job_id, + req.error.as_deref(), + ) + .await?; + tx.commit().await?; + // Schema capture is independently best-effort (its own transaction for the + // per-asset advisory lock) and must never roll back the partition record + // above — mirroring the worker's `record_mat`. A lost schema version + // degrades the history, not the run. Only a successful (`Materialized`) write + // advances the recorded schema — a failed/running write must not (and a + // client shouldn't be able to bump the history by attaching a schema to one). + let is_materialized = matches!( + req.status, + windmill_common::materialization::MaterializationStatus::Materialized + ); + if let (true, Some(columns)) = (is_materialized, req.schema.as_ref()) { + let res: windmill_common::error::Result<()> = async { + let mut tx = user_db.clone().begin(&authed).await?; + windmill_common::materialization::record_asset_schema( + &mut tx, + &w_id, + req.asset_kind, + &req.asset_path, + columns, + req.snapshot_id, + req.job_id, + ) + .await?; + tx.commit().await?; + Ok(()) + } + .await; + if let Err(e) = res { + tracing::warn!("failed to record captured asset schema: {e:#}"); + } + } + Ok(Json(())) } #[derive(Deserialize)] @@ -363,3 +477,590 @@ async fn list_favorites( Ok(Json(favorites)) } + +// ------------------------------------------------------------------ +// GET /w/:workspace/assets/graph +// ------------------------------------------------------------------ +// Workspace-wide asset ↔ runnable graph. One row per unique +// (asset_kind, asset_path, usage_kind, usage_path, access_type) — the +// frontend aggregates into nodes and edges. + +#[derive(Deserialize)] +struct GraphQuery { + pub asset_kinds: Option, + pub folder: Option, +} + +#[derive(Serialize, Debug)] +struct GraphAssetNode { + kind: AssetKind, + path: String, +} + +#[derive(Serialize, Debug)] +struct GraphRunnableNode { + path: String, + usage_kind: AssetUsageKind, + // True iff the script was deployed with `// pipeline` — drives the + // pipeline-member visual state on the frontend. + #[serde(skip_serializing_if = "std::ops::Not::not", default)] + in_pipeline: bool, + // Annotation badges parsed from the deployed script body, so the canvas + // shows partition/freshness/tag/retry/data-test chips on *deployed* nodes + // (not only on live-edited drafts, which the frontend parses itself). Kept + // in lockstep with the TS `AssetGraphRunnableNode` fields the node renders. + #[serde(skip_serializing_if = "Option::is_none", default)] + partition_kind: Option, + #[serde(skip_serializing_if = "Option::is_none", default)] + freshness: Option, + #[serde(skip_serializing_if = "Option::is_none", default)] + tag: Option, + #[serde(skip_serializing_if = "Option::is_none", default)] + retry: Option, + #[serde(skip_serializing_if = "Vec::is_empty", default)] + data_tests: Vec, + // `// column <- .` declared column-level lineage, surfaced + // so the canvas can draw the column-lineage view on deployed nodes (not + // only live drafts). Lockstep with TS `AssetGraphRunnableNode.column_lineage`. + #[serde(skip_serializing_if = "Vec::is_empty", default)] + column_lineage: Vec, + // `// materialize ` target — the asset this script's `column_lineage` + // describes. Lets the column-graph anchor lineage to the exact output asset + // instead of guessing a ducklake write-edge (a multi-output script writes + // several). Absent for scripts with no `// materialize` annotation. + #[serde(skip_serializing_if = "Option::is_none", default)] + materialize_target: Option, + // Managed `// materialize` write strategy (`replace` | `append` | `merge`), + // absent for non-materializing or `manual` scripts. Surfaced so the asset + // panel can tell whether the captured schema can evolve: only whole-table + // `replace` (CREATE OR REPLACE) can change columns run-to-run; `append` / + // `merge` / any partitioned write INSERTs into a fixed-schema table. + #[serde(skip_serializing_if = "Option::is_none", default)] + materialize_strategy: Option, +} + +// The output asset a producer's column lineage belongs to (the `// materialize` +// target). Kept minimal — the column graph only needs (kind, path) to anchor. +#[derive(Serialize, Debug)] +struct MaterializeTargetNode { + kind: windmill_common::assets::AssetKind, + path: String, +} + +// The partition's kind word for the node badge (the full PartitionSpec carries +// tz/format/start, which the badge doesn't need). +fn partition_kind_word(kind: &windmill_common::assets::PartitionKind) -> &'static str { + use windmill_common::assets::PartitionKind::*; + match kind { + Daily => "daily", + Hourly => "hourly", + Weekly => "weekly", + Monthly => "monthly", + Dynamic { .. } => "dynamic", + } +} + +// Lineage edge from parsed r/w usages. One per (runnable, asset, access_type) +// tuple. Informational — not the DAG execution edges. +#[derive(Serialize, Debug)] +struct GraphEdge { + runnable_path: String, + runnable_kind: AssetUsageKind, + asset_kind: AssetKind, + asset_path: String, + access_type: Option, +} + +// Declared `// on ` trigger edge — the actual execution DAG. +// Asset edges come from `script_trigger`; the eight native variants +// (Schedule/Email/Kafka/…/Gcp) come from the per-kind trigger tables joined +// on `script_path`. Each native variant carries just the trigger row's path; +// the config (cron, broker, topic, auth, …) lives in its own UI. +// +// `webhook` is parsed as an annotation marker but has no dedicated trigger +// table — every script gets an implicit webhook endpoint — so no variant +// here. The frontend renders the marker from the source annotations alone. +#[derive(Serialize, Debug)] +#[serde(tag = "trigger_kind", rename_all = "lowercase")] +enum TriggerEdge { + Asset { + asset_kind: AssetKind, + asset_path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Schedule { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Email { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Kafka { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Mqtt { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Nats { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Postgres { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Sqs { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, + Gcp { + path: String, + runnable_kind: AssetUsageKind, + runnable_path: String, + }, +} + +#[derive(Serialize, Debug)] +struct AssetGraphResponse { + assets: Vec, + runnables: Vec, + edges: Vec, + triggers: Vec, +} + +async fn asset_graph( + authed: ApiAuthed, + Path(w_id): Path, + Extension(user_db): Extension, + Query(q): Query, +) -> JsonResult { + let mut tx = user_db.begin(&authed).await?; + + let kind_filter: Option> = q.asset_kinds.as_ref().map(|s| { + s.split(',') + .filter_map(|k| { + serde_json::from_value::(Value::String(k.trim().into())).ok() + }) + .collect() + }); + let kind_filter_ref = kind_filter.as_deref(); + + let folder_filter = q.folder.as_deref().map(|f| format!("f/{}/%", f)); + + // One row per (asset_kind, asset_path, usage_kind, usage_path, access_type). + // The `usage_kind IN ('script','flow')` clause excludes `job`-kind usage rows + // (runtime-detected, ephemeral) so the graph stays stable. + let rows = sqlx::query!( + r#" + SELECT + asset.kind AS "asset_kind!: AssetKind", + asset.path AS "asset_path!", + asset.usage_kind AS "usage_kind!: AssetUsageKind", + asset.usage_path AS "usage_path!", + asset.usage_access_type::text AS "access_type" + FROM asset + WHERE asset.workspace_id = $1 + AND asset.usage_kind IN ('script', 'flow') + AND ($2::asset_kind[] IS NULL OR asset.kind = ANY($2)) + AND ($3::text IS NULL OR asset.usage_path LIKE $3) + GROUP BY asset.kind, asset.path, asset.usage_kind, asset.usage_path, asset.usage_access_type + "#, + &w_id, + kind_filter_ref as Option<&[AssetKind]>, + folder_filter.as_deref(), + ) + .fetch_all(&mut *tx) + .await?; + + // Pipeline asset trigger edges, fetched separately so we can widen the + // runnable_set for trigger-only endpoints (e.g. an asset trigger whose + // asset has no usage in the pipeline yet). Native trigger kinds + // (schedule, kafka, mqtt, …) are *not* in `script_trigger` — they're + // discovered below by querying each native trigger table directly. + let trigger_rows = sqlx::query!( + r#" + SELECT + runnable_kind AS "runnable_kind!: AssetUsageKind", + runnable_path AS "runnable_path!", + trigger_kind::text AS "trigger_kind!", + trigger_ref AS "trigger_ref!" + FROM script_trigger + WHERE workspace_id = $1 + AND trigger_kind = 'asset' + AND ($2::text IS NULL OR runnable_path LIKE $2) + "#, + &w_id, + folder_filter.as_deref(), + ) + .fetch_all(&mut *tx) + .await?; + + // Native triggers in scope. Each native trigger table stores its + // single-destination `script_path` directly, so we resolve attachment by + // joining on that field rather than via `script_trigger`. UNION ALL keeps + // it a single round trip; the `kind` column drives the TriggerEdge ctor + // below. `schedule` lives in the `schedule` table, which has its own + // shape (no workspace_id-only filter — it shares `is_flow` like the + // others), but the columns we need line up. + let native_trigger_rows = sqlx::query!( + r#" + SELECT kind, path, script_path, is_flow FROM ( + SELECT 'schedule' AS kind, path, script_path, is_flow FROM schedule + WHERE workspace_id = $1 + AND script_path IS NOT NULL + UNION ALL + SELECT 'email', path, script_path, is_flow FROM email_trigger + WHERE workspace_id = $1 + UNION ALL + SELECT 'kafka', path, script_path, is_flow FROM kafka_trigger + WHERE workspace_id = $1 + UNION ALL + SELECT 'mqtt', path, script_path, is_flow FROM mqtt_trigger + WHERE workspace_id = $1 + UNION ALL + SELECT 'nats', path, script_path, is_flow FROM nats_trigger + WHERE workspace_id = $1 + UNION ALL + SELECT 'postgres', path, script_path, is_flow FROM postgres_trigger + WHERE workspace_id = $1 + UNION ALL + SELECT 'sqs', path, script_path, is_flow FROM sqs_trigger + WHERE workspace_id = $1 + UNION ALL + SELECT 'gcp', path, script_path, is_flow FROM gcp_trigger + WHERE workspace_id = $1 + ) t + WHERE ($2::text IS NULL OR script_path LIKE $2) + "#, + &w_id, + folder_filter.as_deref(), + ) + .fetch_all(&mut *tx) + .await?; + + // Which scripts in scope are pipeline members (have `// pipeline`). + // Pipeline members + their latest deployed body, so the graph can surface + // annotation badges (partition/freshness/tag/retry/data_test) on deployed + // nodes. `DISTINCT ON (path) … ORDER BY created_at DESC` picks the newest + // non-archived version per path (a redeploy archives the prior one, but be + // defensive against transient overlaps). + let pipeline_member_paths = sqlx::query!( + r#" + SELECT DISTINCT ON (path) path AS "path!", content AS "content!", + language AS "language!: windmill_common::scripts::ScriptLang" + FROM script + WHERE workspace_id = $1 + AND auto_kind = 'pipeline' + AND archived = false + AND deleted = false + AND ($2::text IS NULL OR path LIKE $2) + ORDER BY path, created_at DESC + "#, + &w_id, + folder_filter.as_deref(), + ) + .fetch_all(&mut *tx) + .await?; + + // Existing scripts / flows in the workspace. Used to filter out + // orphan trigger rows whose `script_path` no longer resolves — those + // would otherwise be added to `runnable_set` below and surface as + // phantom "deployed" runnables on the canvas (matching what the user + // can deploy a new trigger against: nothing). + let existing_script_paths = sqlx::query_scalar!( + r#"SELECT path AS "path!" FROM script + WHERE workspace_id = $1 + AND archived = false + AND deleted = false"#, + &w_id, + ) + .fetch_all(&mut *tx) + .await?; + let existing_flow_paths = sqlx::query_scalar!( + r#"SELECT path AS "path!" FROM flow WHERE workspace_id = $1 AND archived = false"#, + &w_id, + ) + .fetch_all(&mut *tx) + .await?; + + tx.commit().await?; + + // Parse each pipeline member's body once into its badge annotations, keyed + // by path, for the runnable-node construction below. + let annotations_by_path: std::collections::HashMap< + String, + windmill_common::assets::PipelineAnnotations, + > = pipeline_member_paths + .iter() + .map(|r| { + ( + r.path.clone(), + windmill_common::assets::parse_pipeline_annotations(&r.content), + ) + }) + .collect(); + // Column-level lineage per member. The annotation-only lineage (already + // parsed above) is the baseline. For DuckDB scripts we additionally run the + // full SQL asset parser to infer output→input column edges from the AST; it + // merges them with the `// column` annotations (annotation wins). If the SQL + // can't be parsed (DuckDB accepts grammar `sqlparser` rejects), we fall back + // to the annotation-only baseline rather than dropping explicit annotations. + let column_lineage_by_path: std::collections::HashMap< + String, + Vec, + > = pipeline_member_paths + .iter() + .map(|r| { + let annotated = || { + annotations_by_path + .get(&r.path) + .map(|a| a.column_lineage.clone()) + .unwrap_or_default() + }; + let lineage = if r.language == windmill_common::scripts::ScriptLang::DuckDb { + windmill_parser_sql_asset::parse_assets(&r.content) + .map(|o| o.column_lineage) + .unwrap_or_else(|_| annotated()) + } else { + annotated() + }; + (r.path.clone(), lineage) + }) + .collect(); + let pipeline_member_script_paths: std::collections::HashSet = + pipeline_member_paths.into_iter().map(|r| r.path).collect(); + let existing_script_paths: std::collections::HashSet = + existing_script_paths.into_iter().collect(); + let existing_flow_paths: std::collections::HashSet = + existing_flow_paths.into_iter().collect(); + let runnable_exists = |kind: AssetUsageKind, path: &str| match kind { + AssetUsageKind::Script => existing_script_paths.contains(path), + AssetUsageKind::Flow => existing_flow_paths.contains(path), + // `Job` is a runtime-detected ephemeral runnable (asset usage rows + // only), never a target of a stored trigger row. Treat as existing + // so we don't accidentally drop ephemeral lineage edges. + AssetUsageKind::Job => true, + }; + + let mut edges = Vec::with_capacity(rows.len()); + let mut asset_set: std::collections::HashSet<(AssetKind, String)> = Default::default(); + let mut runnable_set: std::collections::HashSet<(AssetUsageKind, String)> = Default::default(); + + // Every pipeline member in scope goes into the graph, even when the parser + // didn't detect any asset r/w and the script has no triggers yet. Without + // this, a freshly-saved pipeline script whose template body hasn't been + // filled in would vanish from the pipeline view on graph refetch. + for path in &pipeline_member_script_paths { + runnable_set.insert((AssetUsageKind::Script, path.clone())); + } + + for r in rows { + // Drop asset usage rows whose runnable target was archived/deleted + // but whose row in `asset` is still around — those would otherwise + // surface as a phantom "deployed" runnable on the canvas with no + // way to interact with it, since the underlying script/flow no + // longer exists. + if !runnable_exists(r.usage_kind, &r.usage_path) { + continue; + } + asset_set.insert((r.asset_kind, r.asset_path.clone())); + runnable_set.insert((r.usage_kind, r.usage_path.clone())); + edges.push(GraphEdge { + runnable_path: r.usage_path, + runnable_kind: r.usage_kind, + asset_kind: r.asset_kind, + asset_path: r.asset_path, + access_type: r.access_type, + }); + } + + let mut triggers: Vec = + Vec::with_capacity(trigger_rows.len() + native_trigger_rows.len()); + for t in trigger_rows { + // Drop orphan asset-trigger rows — their target runnable no longer + // exists (script/flow archived or deleted, or was never deployed). + // Without this, an orphan row would surface as a phantom "deployed" + // runnable on the canvas (no `unsaved` flag, can't actually be + // run / re-targeted by a new trigger). + if !runnable_exists(t.runnable_kind, &t.runnable_path) { + continue; + } + runnable_set.insert((t.runnable_kind, t.runnable_path.clone())); + if t.trigger_kind.as_str() == "asset" { + // trigger_ref is `` — parse back out so both + // endpoints match what the frontend uses for node ids. + if let Some((asset_kind, asset_path)) = parse_asset_trigger_ref(&t.trigger_ref) { + // Make sure the source asset has a node even if nothing + // reads/writes it in this folder. + asset_set.insert((asset_kind, asset_path.clone())); + triggers.push(TriggerEdge::Asset { + asset_kind, + asset_path, + runnable_kind: t.runnable_kind, + runnable_path: t.runnable_path, + }); + } + } + // Native kinds (schedule, kafka, mqtt, …) come from per-kind trigger + // tables below. + } + + // Native trigger attachments — one TriggerEdge per row, the kind chosen + // from the discriminator. Add the runnable to the set so a script with + // no asset edges but a kafka/schedule attachment still renders on the + // canvas. + for t in native_trigger_rows { + let kind = t.kind.unwrap_or_default(); + let path = t.path.unwrap_or_default(); + let script_path = t.script_path.unwrap_or_default(); + let runnable_kind = if t.is_flow.unwrap_or(false) { + AssetUsageKind::Flow + } else { + AssetUsageKind::Script + }; + // Same orphan filter as the asset-trigger loop above — drop trigger + // rows whose target script/flow no longer exists so the graph + // doesn't synthesize a phantom deployed runnable. + if !runnable_exists(runnable_kind, &script_path) { + continue; + } + runnable_set.insert((runnable_kind, script_path.clone())); + let edge = match kind.as_str() { + "schedule" => TriggerEdge::Schedule { path, runnable_kind, runnable_path: script_path }, + "email" => TriggerEdge::Email { path, runnable_kind, runnable_path: script_path }, + "kafka" => TriggerEdge::Kafka { path, runnable_kind, runnable_path: script_path }, + "mqtt" => TriggerEdge::Mqtt { path, runnable_kind, runnable_path: script_path }, + "nats" => TriggerEdge::Nats { path, runnable_kind, runnable_path: script_path }, + "postgres" => TriggerEdge::Postgres { path, runnable_kind, runnable_path: script_path }, + "sqs" => TriggerEdge::Sqs { path, runnable_kind, runnable_path: script_path }, + "gcp" => TriggerEdge::Gcp { path, runnable_kind, runnable_path: script_path }, + _ => continue, + }; + triggers.push(edge); + } + + let mut assets: Vec = asset_set + .into_iter() + .map(|(kind, path)| GraphAssetNode { kind, path }) + .collect(); + assets.sort_by(|a, b| a.path.cmp(&b.path)); + + let mut runnables: Vec = runnable_set + .into_iter() + .map(|(usage_kind, path)| { + let in_pipeline = usage_kind == AssetUsageKind::Script + && pipeline_member_script_paths.contains(&path); + // Annotation badges, only for pipeline-member scripts (the only + // bodies we parsed). Gate on the runnable kind too: a flow sharing a + // path with a pipeline script must not inherit its badges. + let ann = (usage_kind == AssetUsageKind::Script) + .then(|| annotations_by_path.get(&path)) + .flatten(); + GraphRunnableNode { + in_pipeline, + partition_kind: ann + .and_then(|a| a.partition.as_ref()) + .map(|p| partition_kind_word(&p.kind).to_string()), + freshness: ann + .and_then(|a| a.freshness.as_ref()) + .map(|f| f.duration.clone()), + tag: ann.and_then(|a| a.tag.clone()), + retry: ann.and_then(|a| a.retry.clone()), + data_tests: ann.map(|a| a.data_tests.clone()).unwrap_or_default(), + // Inferred (DuckDB AST) + annotation column lineage, gated to + // scripts like the badges above. + column_lineage: (usage_kind == AssetUsageKind::Script) + .then(|| column_lineage_by_path.get(&path)) + .flatten() + .cloned() + .unwrap_or_default(), + materialize_target: ann.and_then(|a| a.materialize.as_ref()).map(|m| { + MaterializeTargetNode { + kind: windmill_common::assets::asset_kind_from_parser(m.target_kind), + path: m.target_path.clone(), + } + }), + materialize_strategy: ann.and_then(|a| a.materialize.as_ref()).and_then(|m| { + if m.manual { + None + } else if m.append { + Some("append".to_string()) + } else if m.unique_key.is_some() { + Some("merge".to_string()) + } else { + Some("replace".to_string()) + } + }), + path, + usage_kind, + } + }) + .collect(); + runnables.sort_by(|a, b| a.path.cmp(&b.path)); + + Ok(Json(AssetGraphResponse { + assets, + runnables, + edges, + triggers, + })) +} + +// ------------------------------------------------------------------ +// GET /w/:workspace/assets/pipelines +// ------------------------------------------------------------------ +// Distinct folder names that contain at least one pipeline-member script +// (auto_kind='pipeline'). Used by the pipeline-editor folder picker and +// the "Pipeline" entry in folder views. Keyed by the partial index on +// `script (workspace_id, path) WHERE auto_kind='pipeline' ...` so this +// is effectively O(matches). + +#[derive(Serialize, Debug)] +struct PipelineFolder { + folder: String, + script_count: i64, +} + +async fn list_pipeline_folders( + authed: ApiAuthed, + Path(w_id): Path, + Extension(user_db): Extension, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = sqlx::query!( + r#" + SELECT + substring(path from '^f/([^/]+)/') AS "folder!", + COUNT(*) AS "script_count!" + FROM script + WHERE workspace_id = $1 + AND auto_kind = 'pipeline' + AND archived = false + AND deleted = false + AND path LIKE 'f/%' + GROUP BY substring(path from '^f/([^/]+)/') + ORDER BY substring(path from '^f/([^/]+)/') + "#, + &w_id, + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + + Ok(Json( + rows.into_iter() + .map(|r| PipelineFolder { folder: r.folder, script_count: r.script_count }) + .collect(), + )) +} diff --git a/backend/windmill-api-auth/src/auth.rs b/backend/windmill-api-auth/src/auth.rs index 314ba99450..a6a0fc4a8e 100644 --- a/backend/windmill-api-auth/src/auth.rs +++ b/backend/windmill-api-auth/src/auth.rs @@ -191,7 +191,11 @@ impl AuthCache { is_operator: claims.is_operator, groups: claims.groups, folders: claims.folders, - scopes: None, + // Honor the scopes embedded in the JWT (mirrors the EE + // jwt_ext_ branch). The route middleware only enforces + // scopes when Some, so a None-scoped JWT (e.g. the job + // WM_TOKEN) keeps full user privileges as before. + scopes: claims.scopes, username_override, token_prefix: claims.audit_span, read_only: false, diff --git a/backend/windmill-api-auth/src/lib.rs b/backend/windmill-api-auth/src/lib.rs index b9e6a748d4..d2347aa22e 100644 --- a/backend/windmill-api-auth/src/lib.rs +++ b/backend/windmill-api-auth/src/lib.rs @@ -205,6 +205,33 @@ pub async fn require_super_admin(db: &DB, email: &str) -> error::Result<()> { } } +/// Forbid sensitive global user/token management when authenticated as a +/// superadmin *via a job token* (`WM_TOKEN`). +/// +/// A `WM_TOKEN`'s identity is derived from an app/flow `on_behalf_of`, which a +/// non-admin `wm_deployers` member can point at a superadmin. Trusting it for +/// these operations would let them establish *persistent* superadmin (promote a +/// user, reset a superadmin's password, mint a superadmin token, ...). `job_id` +/// is set only for `WM_TOKEN`s; regular session/API tokens have it `None`, so a +/// real superadmin who needs this from a script uses a dedicated superadmin API +/// token (which only a real superadmin can create) instead of `$WM_TOKEN`. +pub async fn forbid_superadmin_job_token( + db: &DB, + email: &str, + job_id: Option, +) -> error::Result<()> { + if job_id.is_some() && is_super_admin_email(db, email).await? { + return Err(Error::NotAuthorized( + "This operation cannot be performed with a job token ($WM_TOKEN) that runs as a \ + superadmin. If a script genuinely needs to do this, create a dedicated superadmin \ + token from the User settings drawer (the 'Tokens' section), store it as a secret, \ + and use that token explicitly instead of $WM_TOKEN." + .to_owned(), + )); + } + Ok(()) +} + pub fn check_scopes(authed: &ApiAuthed, required: F) -> error::Result<()> where F: FnOnce() -> String, @@ -996,6 +1023,18 @@ pub fn require_path_read_access_for_preview( return Ok(()); }; + // Reject path traversal before any privilege-based short-circuit. A Preview's + // path is request-supplied and bypasses the DB `proper_id` CHECK that deployed + // runnables get; it then flows to the worker where it builds on-disk module + // directories. A `..` segment or an absolute path could let a write escape the + // per-job dir. + if path.starts_with('/') || path.split('/').any(|seg| seg == "..") || path.contains('\0') { + return Err(Error::BadRequest(format!( + "Invalid path for preview job: {}", + path + ))); + } + if authed.is_admin { return Ok(()); } @@ -1053,6 +1092,45 @@ mod tests { } } + // Regression tests for the Preview path traversal: a Preview's path skips the + // DB `proper_id` CHECK and reaches the worker, where it builds on-disk module + // dirs. Traversal must be rejected even for admins, who otherwise bypass the + // namespace/folder access check. + #[test] + fn preview_path_rejects_traversal() { + let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() }; + for path in [ + "u/admin/../../../../../../tmp/evil/payload", + "../../tmp/evil", + "/tmp/evil", + "u/admin/ok/../../../../etc/cron.d/x", + ] { + assert!( + require_path_read_access_for_preview(&admin, &Some(path.to_string())).is_err(), + "expected traversal path to be rejected: {path}" + ); + } + } + + #[test] + fn preview_path_allows_legitimate_paths() { + let alice = ApiAuthed { username: "alice".into(), ..Default::default() }; + assert!(require_path_read_access_for_preview(&alice, &None).is_ok()); + assert!(require_path_read_access_for_preview(&alice, &Some(String::new())).is_ok()); + assert!( + require_path_read_access_for_preview(&alice, &Some("u/alice/my_script".into())).is_ok() + ); + + let admin = ApiAuthed { is_admin: true, username: "admin".into(), ..Default::default() }; + assert!( + require_path_read_access_for_preview(&admin, &Some("hub/foo/bar/baz".into())).is_ok() + ); + // `..` only as a substring of a segment is a valid name, not traversal. + assert!( + require_path_read_access_for_preview(&admin, &Some("f/team/my..script".into())).is_ok() + ); + } + #[test] fn predicate_no_scopes_allows_all() { let authed = authed_with_scopes(None); diff --git a/backend/windmill-api-auth/src/scopes.rs b/backend/windmill-api-auth/src/scopes.rs index 87ca3a8862..041885369c 100644 --- a/backend/windmill-api-auth/src/scopes.rs +++ b/backend/windmill-api-auth/src/scopes.rs @@ -274,6 +274,7 @@ pub enum ScopeDomain { Configs, OAuth, AI, + AiSkills, Indexer, Teams, // Microsoft Teams integration @@ -294,6 +295,7 @@ pub enum ScopeDomain { RawApps, // Raw application data AgentWorkers, // Agent workers management Mcp, // MCP + Docs, // Self-hosted documentation search (read-only) } impl ScopeDomain { @@ -329,6 +331,7 @@ impl ScopeDomain { Self::Configs => "configs", Self::OAuth => "oauth", Self::AI => "ai", + Self::AiSkills => "ai_skills", Self::Capture => "capture", Self::Drafts => "drafts", Self::Favorites => "favorites", @@ -344,6 +347,7 @@ impl ScopeDomain { Self::Teams => "teams", Self::GitSync => "git_sync", Self::Mcp => "mcp", + Self::Docs => "docs", } } @@ -378,6 +382,7 @@ impl ScopeDomain { "configs" => Some(Self::Configs), "oauth" => Some(Self::OAuth), "ai" => Some(Self::AI), + "ai_skills" => Some(Self::AiSkills), "indexer" | "srch" => Some(Self::Indexer), "teams" => Some(Self::Teams), "native_triggers" => Some(Self::NativeTriggers), @@ -394,6 +399,7 @@ impl ScopeDomain { "raw_apps" => Some(Self::RawApps), "agent_workers" => Some(Self::AgentWorkers), "mcp" => Some(Self::Mcp), + "docs" => Some(Self::Docs), _ => None, } } @@ -448,6 +454,30 @@ pub fn check_route_access( // Find the domain and kind for this route let (required_domain, required_kind, route_suffix) = extract_domain_from_route(route_path)?; + // App embed tokens (sentinel) carry broad read scopes (`jobs:read`, + // `users:read`, `folders:read`) that exist only for a handful of routes. The + // whole `/users`, `/folders` and `/jobs` routers are CORS-enabled for the + // opaque app iframe, so default-deny everything in those domains except the + // intended routes — otherwise the token could enumerate/export workspace data. + if has_app_embed_sentinel(Some(token_scopes)) { + if let Some(suffix) = route_suffix.as_deref() { + if app_embed_route_denied(required_domain, suffix) { + return Err(Error::PermissionDenied( + "Access denied. App embed token cannot access this route.".to_string(), + )); + } + // The by-id job cancel is a POST (write) that the token's `jobs:read` + // wouldn't satisfy, but cancelling the app's own component runs is + // intended (most components supersede an in-flight run on re-run). Permit + // it here; `cancel_job_api` confines it to jobs the app launched + // (created_by == viewer). A read_only token is still rejected by the + // separate read-only check. + if suffix.starts_with("jobs_u/queue/cancel/") { + return Ok(()); + } + } + } + // MCP scopes (mcp:all, mcp:favorites, mcp:hub:*, etc.) use a custom format // that doesn't fit the standard domain:action model. Verify the token has at // least one mcp: scope; MCP handlers do their own fine-grained checking. @@ -534,7 +564,7 @@ const FLOW_JOBS: [&'static str; 6] = [ lazy_static::lazy_static! { static ref RUN_PATH_ACTIONS: Vec<&'static str> = { - let mut v = vec!["jobs/resume/", "jobs/run/batch_rerun_jobs", "jobs/run/workflow_as_code", "jobs/run/dependencies","jobs/run/flow_dependencies", "apps_u/execute_component"]; + let mut v = vec!["jobs/resume/", "jobs/run/batch_rerun_jobs", "jobs/run/workflow_as_code", "jobs/run/dependencies","jobs/run/flow_dependencies", "apps_u/execute_component", "apps_u/upload_s3_file"]; v.extend(SCRIPT_JOBS); v.extend(FLOW_JOBS); @@ -637,6 +667,92 @@ const RUN_WHITELISTED_GET_PATHS: [&'static str; 20] = [ "jobs/completed/get_result_maybe/", ]; +/// Sentinel scope in app embed tokens. Grants nothing itself; `check_route_access` +/// uses it to deny the workspace-wide job enumeration routes `jobs:read` would +/// otherwise reach, so an embedded app reads only jobs it launched (by id). +pub const APP_EMBED_SENTINEL: &str = "app_embed"; + +/// True if a token's scopes include the app-embed sentinel (a sandboxed app iframe +/// token). Such tokens carry the viewer's identity but represent untrusted app JS, +/// so several handlers confine them to the app's own resources/runs. +pub fn has_app_embed_sentinel(scopes: Option<&[String]>) -> bool { + scopes.is_some_and(|s| s.iter().any(|x| x == APP_EMBED_SENTINEL)) +} + +/// Routes an app embed token (sentinel) is denied. Its broad scopes (`apps:run`, +/// `jobs:read`, `users:read`, `folders:read`) exist only for a fixed set of routes a +/// running app uses, but the whole `/apps`, `/jobs`, `/users`, `/folders` routers are +/// CORS-enabled for the opaque app iframe. Default-deny those domains via an explicit +/// allowlist so the token can't reach workspace inventory, counts, exports, or +/// capability-minting routes (job signatures / resume URLs). +fn app_embed_route_denied(domain: ScopeDomain, suffix: &str) -> bool { + match domain { + ScopeDomain::Apps => !app_embed_apps_route_allowed(suffix), + ScopeDomain::Jobs => !app_embed_job_route_allowed(suffix), + ScopeDomain::Users => suffix != "users/whoami", + ScopeDomain::Folders => suffix != "folders/listnames", + _ => false, + } +} + +/// App routes a running app uses: its own definition (`apps/get/p/`, further +/// path-scoped by `apps:read:`) and the public app-serving endpoints +/// (`apps_u/*`: public_app, public_resource, get_data, and the path-taking +/// `execute_component` / `download_s3_file`, which re-check `apps:run|read:` +/// in their handlers so they stay confined to this app). Everything else in the +/// domain — workspace app inventory (`exists`, `custom_path_exists`, `list`, +/// `list_paths*`, `secret_of`, history, management) — is denied. +fn app_embed_apps_route_allowed(suffix: &str) -> bool { + // The embed-token mint endpoints live under `apps_u/` but they create + // credentials. A running app never calls them — the trusted embedder session/JWT + // mints the token and hands it to the iframe — so deny them here, otherwise an + // app embed token could renew itself indefinitely past the 12h expiry. + if suffix.starts_with("apps_u/embed_token") { + return false; + } + suffix.starts_with("apps/get/p/") || suffix.starts_with("apps_u/") +} + +/// Job routes a running app uses (the by-id poll/cancel surface driven by the +/// frontend JobLoader). Everything else in the jobs domain — enumeration, counts, +/// exports, and the `job_signature`/`resume_urls` capability-minting routes — is +/// denied. By-id reads are further confined to the app's own runs by +/// `require_job_read_access` (the `app_embed` cutoff). +fn app_embed_job_route_allowed(suffix: &str) -> bool { + // `get_root_job_id` is intentionally absent: its handler has no access check at + // all (returns any job's root id by id) and the app never calls it, so denying + // it costs nothing and avoids leaking a foreign job's flow lineage. + const ALLOWED: [&str; 15] = [ + "jobs_u/get/", + "jobs_u/getupdate/", + "jobs_u/getupdate_sse/", + "jobs_u/get_logs/", + "jobs_u/get_completed_logs_tail/", + "jobs_u/get_args/", + "jobs_u/get_flow/", + "jobs_u/get_flow_all_logs/", + "jobs_u/get_flow_debug_info/", + "jobs_u/get_log_file/", + "jobs_u/completed/get/", + "jobs_u/completed/get_result/", + "jobs_u/completed/get_result_maybe/", + "jobs_u/completed/get_timing/", + "jobs_u/queue/cancel/", + ]; + ALLOWED.iter().any(|p| suffix.starts_with(p)) +} + +/// Resource routes a metadata-only `resources:run` scope (app embed tokens) may +/// GET: pickers (`/list`) and type schemas. Excludes every value-returning route +/// (`get`, `get_value`, `get_value_interpolated`, `list_search`) so resource +/// values — which can hold credentials — are never exposed. +fn resource_metadata_route_allowed(suffix: &str) -> bool { + suffix == "resources/list" + || suffix.starts_with("resources/list_names/") + || suffix.starts_with("resources/exists/") + || suffix.starts_with("resources/type/") +} + fn scope_grants_access( scope: &ScopeDefinition, required_domain: ScopeDomain, @@ -656,6 +772,14 @@ fn scope_grants_access( let scope_action = ScopeAction::from_str(&scope.action) .ok_or_else(|| Error::BadRequest(format!("Invalid scope action: {}", scope.action)))?; + // App embed tokens carry `resources:run`: metadata-only resource access via + // default-deny + allowlist (so a new value route is never exposed by accident). + // See `resource_metadata_route_allowed`. + if scope_domain == ScopeDomain::Resources && scope_action == ScopeAction::Run { + return Ok(required_action == ScopeAction::Read + && route_path.is_some_and(resource_metadata_route_allowed)); + } + if !scope_action.includes(&required_action) && !(scope_domain == ScopeDomain::Jobs && required_action == ScopeAction::Read @@ -699,6 +823,23 @@ pub fn check_read_only_for_route(route_path: &str, http_method: &str) -> Result< } } +/// The minimal scope string that grants access to exactly `{method} {path}`, as +/// `check_route_access` would require it. Used to mint a least-privilege JWT for +/// a single proxied request (the MCP endpoint proxy), so the minted token can do +/// only that one operation rather than acting as a blank check. +/// +/// `path` is the request path (e.g. `/api/w/{workspace}/variables/get/...`). +/// Returns `None` if the route's domain can't be determined — the caller should +/// then fail closed. +pub fn scope_for_route(method: &str, path: &str) -> Option { + let action = map_http_method_to_action(method, path); + let (domain, kind, _suffix) = extract_domain_from_route(path).ok()?; + Some(match (domain, action, kind) { + (ScopeDomain::Jobs, ScopeAction::Run, Some(kind)) => format!("jobs:run:{}", kind), + (domain, action, _) => format!("{}:{}", domain.as_str(), action.as_str()), + }) +} + /// Helper function to check if scopes allow access to a route pub fn check_scopes_for_route( token_scopes: Option<&[String]>, @@ -789,6 +930,12 @@ mod tests { assert_eq!(domain, ScopeDomain::FlowConversations); assert_eq!(kind, None); assert_eq!(route_suffix, Some("flow_conversations/list".to_string())); + + let (domain, kind, route_suffix) = + extract_domain_from_route("/api/w/test_workspace/ai_skills/list").unwrap(); + assert_eq!(domain, ScopeDomain::AiSkills); + assert_eq!(kind, None); + assert_eq!(route_suffix, Some("ai_skills/list".to_string())); } #[test] @@ -845,6 +992,10 @@ mod tests { ScopeDomain::from_str("flow_conversations"), Some(ScopeDomain::FlowConversations) ); + assert_eq!( + ScopeDomain::from_str("ai_skills"), + Some(ScopeDomain::AiSkills) + ); // Test canonical string conversion assert_eq!(ScopeDomain::Acls.as_str(), "acls"); @@ -854,6 +1005,41 @@ mod tests { ScopeDomain::FlowConversations.as_str(), "flow_conversations" ); + assert_eq!(ScopeDomain::AiSkills.as_str(), "ai_skills"); + } + + #[test] + fn test_ai_skills_scope_access() { + let read_scopes = vec!["ai_skills:read".to_string()]; + assert!( + check_route_access(&read_scopes, "/api/w/test_workspace/ai_skills/list", "GET").is_ok() + ); + assert!(check_route_access( + &read_scopes, + "/api/w/test_workspace/ai_skills/get/foo", + "GET" + ) + .is_ok()); + assert!(check_route_access( + &read_scopes, + "/api/w/test_workspace/ai_skills/upload", + "POST" + ) + .is_err()); + + let write_scopes = vec!["ai_skills:write".to_string()]; + assert!(check_route_access( + &write_scopes, + "/api/w/test_workspace/ai_skills/upload", + "POST" + ) + .is_ok()); + assert!(check_route_access( + &write_scopes, + "/api/w/test_workspace/ai_skills/delete/foo", + "DELETE" + ) + .is_ok()); } #[test] @@ -1083,4 +1269,38 @@ mod tests { let scopes = vec!["jobs:read".to_string(), "mcp:all".to_string()]; assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok()); } + + #[test] + fn test_scope_for_route() { + // The minted scope must be exactly what check_route_access requires for + // the same route, so a JWT carrying it passes for that one route only. + assert_eq!( + scope_for_route("GET", "/api/w/ws/variables/get/u/x/y").as_deref(), + Some("variables:read") + ); + assert_eq!( + scope_for_route("POST", "/api/w/ws/variables/create").as_deref(), + Some("variables:write") + ); + assert_eq!( + scope_for_route("DELETE", "/api/w/ws/resources/delete/u/x/y").as_deref(), + Some("resources:write") + ); + // jobs run paths carry the runnable kind. + assert_eq!( + scope_for_route("POST", "/api/w/ws/jobs/run/p/u/x/y").as_deref(), + Some("jobs:run:scripts") + ); + assert_eq!( + scope_for_route("POST", "/api/w/ws/jobs/run/f/u/x/y").as_deref(), + Some("jobs:run:flows") + ); + + // The minted scope actually satisfies the route check it targets. + let s = scope_for_route("POST", "/api/w/ws/variables/create").unwrap(); + assert!(check_route_access(&[s], "/api/w/ws/variables/create", "POST").is_ok()); + + // Unknown route -> None so the caller fails closed. + assert!(scope_for_route("GET", "/healthz").is_none()); + } } diff --git a/backend/windmill-api-configs/src/lib.rs b/backend/windmill-api-configs/src/lib.rs index e18afc35c8..f776712063 100644 --- a/backend/windmill-api-configs/src/lib.rs +++ b/backend/windmill-api-configs/src/lib.rs @@ -217,7 +217,7 @@ async fn delete_config( let mut tx = db.begin().await?; let deleted = sqlx::query!("DELETE FROM config WHERE name = $1 RETURNING name", name) - .fetch_all(&db) + .fetch_all(&mut *tx) .await?; audit_log( diff --git a/backend/windmill-api-embeddings/src/lib.rs b/backend/windmill-api-embeddings/src/lib.rs index f0c87633d2..016e72be91 100644 --- a/backend/windmill-api-embeddings/src/lib.rs +++ b/backend/windmill-api-embeddings/src/lib.rs @@ -3,11 +3,11 @@ use anyhow::{anyhow, Error, Result}; #[cfg(feature = "embedding")] use std::{collections::HashMap, path::PathBuf, sync::Arc}; #[cfg(feature = "embedding")] +use windmill_common::utils::HTTP_CLIENT_PERMISSIVE as HTTP_CLIENT; +#[cfg(feature = "embedding")] use windmill_common::DEFAULT_HUB_BASE_URL; #[cfg(feature = "embedding")] use windmill_common::HUB_BASE_URL; -#[cfg(feature = "embedding")] -use windmill_common::utils::HTTP_CLIENT_PERMISSIVE as HTTP_CLIENT; use axum::Router; @@ -159,23 +159,52 @@ pub struct ModelInstance { #[cfg(feature = "embedding")] impl ModelInstance { + async fn get_hf_file( + repo_api: &hf_hub::api::tokio::ApiRepo, + filename: &str, + ) -> Result { + // HuggingFace downloads have no built-in retry, so a single transient + // network blip would fail the whole image build. Retry with exponential + // backoff (1s, 2s, 4s, 8s, capped at 8s) up to MAX_ATTEMPTS times. + const MAX_ATTEMPTS: u32 = 5; + let mut attempt: u32 = 0; + loop { + attempt += 1; + match repo_api.get(filename).await { + Ok(path) => return Ok(path), + Err(e) => { + if attempt >= MAX_ATTEMPTS { + return Err(anyhow!( + "Failed to get {} from hugging face after {} attempts: {}", + filename, + attempt, + e + )); + } + let delay_secs = 1u64 << (attempt - 1).min(3); + tracing::warn!( + "Failed to get {} from hugging face (attempt {}/{}): {}. Retrying in {}s...", + filename, + attempt, + MAX_ATTEMPTS, + e, + delay_secs + ); + tokio::time::sleep(std::time::Duration::from_secs(delay_secs)).await; + } + } + } + } + pub async fn load_model_files() -> Result<(PathBuf, PathBuf, PathBuf)> { let api = Api::new()?; let repo_api = api.model("thenlper/gte-small".to_string()); - let (config_filename, tokenizer_filename, weights_filename) = - ( - repo_api - .get("config.json") - .await - .map_err(|e| anyhow!("Failed to get config.json from hugging face: {}", e))?, - repo_api.get("tokenizer.json").await.map_err(|e| { - anyhow!("Failed to get tokenizer.json from hugging face: {}", e) - })?, - repo_api.get("model.safetensors").await.map_err(|e| { - anyhow!("Failed to get model.safetensors from hugging face: {}", e) - })?, - ); + let (config_filename, tokenizer_filename, weights_filename) = ( + Self::get_hf_file(&repo_api, "config.json").await?, + Self::get_hf_file(&repo_api, "tokenizer.json").await?, + Self::get_hf_file(&repo_api, "model.safetensors").await?, + ); Ok((config_filename, tokenizer_filename, weights_filename)) } diff --git a/backend/windmill-api-flows/src/flows.rs b/backend/windmill-api-flows/src/flows.rs index 4a31589baa..b5f8e2f285 100644 --- a/backend/windmill-api-flows/src/flows.rs +++ b/backend/windmill-api-flows/src/flows.rs @@ -21,7 +21,8 @@ use windmill_api_auth::{ }; use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use windmill_common::{ - utils::{WithStarredInfoQuery, HTTP_CLIENT}, + user_drafts::{overlay_or_draft_only, DraftUserRef, UserDraftItemKind, WithDraftOverlay}, + utils::HTTP_CLIENT, webhook::{WebhookMessage, WebhookShared}, DB, }; @@ -49,7 +50,6 @@ use windmill_common::{ flows::{Flow, FlowWithStarred, ListFlowQuery, ListableFlow, NewFlow}, jobs::JobPayload, schedule::Schedule, - scripts::Schema, utils::{http_get_from_hub, not_found_if_none, paginate, Pagination, RunnableKind, StripPath}, }; use windmill_dep_map::scoped_dependency_map::ScopedDependencyMap; @@ -68,7 +68,6 @@ pub fn workspaced_service() -> Router { .route("/list_tokens/{*path}", get(list_tokens)) .route("/get/{*path}", get(get_flow_by_path)) .route("/deployment_status/p/{*path}", get(get_deployment_status)) - .route("/get/draft/{*path}", get(get_flow_by_path_w_draft)) .route("/exists/{*path}", get(exists_flow_by_path)) .route("/list_paths", get(list_paths)) .route("/history/p/{*path}", get(get_flow_history)) @@ -130,6 +129,7 @@ async fn list_search_flows( async fn list_flows( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path(w_id): Path, Query(pagination): Query, Query(lq): Query, @@ -151,10 +151,15 @@ async fn list_flows( "archived", "extra_perms", "favorite.path IS NOT NULL as starred", - "draft.path IS NOT NULL as has_draft", - "draft_only", "ws_error_handler_muted", "o.labels", + "draft.email IS NOT NULL as is_draft", + // Per-path draft owners as a JSON array; see scripts.rs for the rationale + // (admins-workspace identity fallback, legacy NULL-email row). + "(SELECT json_agg(json_build_object('username', COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END)) ORDER BY COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) NULLS LAST) \ + FROM draft d \ + LEFT JOIN usr u ON u.workspace_id = d.workspace_id AND u.email = d.email \ + WHERE d.workspace_id = o.workspace_id AND d.path = o.path AND d.typ = 'flow') as draft_users", "folder_labels(o.workspace_id, o.path) as inherited_labels" ]) .left() @@ -166,7 +171,8 @@ async fn list_flows( .left() .join("draft") .on( - "draft.path = o.path AND draft.workspace_id = o.workspace_id AND draft.typ = 'flow'" + "draft.path = o.path AND draft.workspace_id = o.workspace_id AND draft.typ = 'flow' AND draft.email = ?" + .bind(&authed.email), ) .left() .join("flow_version fv") @@ -195,9 +201,6 @@ async fn list_flows( sqlb.and_where_is_not_null("favorite.path"); } - if !lq.include_draft_only.unwrap_or(false) || authed.is_operator { - sqlb.and_where("o.draft_only IS NOT TRUE"); - } if let Some(dw) = &lq.dedicated_worker { sqlb.and_where_eq("dedicated_worker", dw); } @@ -221,13 +224,92 @@ async fn list_flows( let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; let mut tx = user_db.begin(&authed).await?; let allowed = build_scope_path_predicate(&authed, "flows", "read"); - let rows = sqlx::query_as::<_, ListableFlow>(&sql) + let mut rows = sqlx::query_as::<_, ListableFlow>(&sql) .fetch_all(&mut *tx) .await? .into_iter() .filter(|r| allowed(&r.path)) .collect::>(); tx.commit().await?; + + // Append the authed user's drafts at paths with no deployed flow; see scripts.rs. + if lq.include_draft_only.unwrap_or(false) + && !authed.is_operator + && offset == 0 + && lq.path_start.is_none() + && lq.path_exact.is_none() + && lq.edited_by.is_none() + && lq.dedicated_worker.is_none() + && lq.label.is_none() + && !lq.starred_only.unwrap_or(false) + && !lq.show_archived.unwrap_or(false) + { + // `(email = $2 OR email IS NULL)` + `DISTINCT ON (path)` ordered NULL-last; see scripts.rs. + let draft_only_rows = sqlx::query!( + r#"SELECT DISTINCT ON (path) + path, + value as "value!: sqlx::types::Json>", + created_at + FROM draft + WHERE workspace_id = $1 + AND typ = 'flow' + AND (email = $2 OR email IS NULL) + AND NOT EXISTS ( + SELECT 1 FROM flow f + WHERE f.workspace_id = draft.workspace_id + AND f.path = draft.path + ) + ORDER BY path, (email IS NULL)"#, + &w_id, + &authed.email, + ) + .fetch_all(&db) + .await?; + + for row in draft_only_rows { + let v: serde_json::Value = + serde_json::from_str(row.value.0.get()).unwrap_or(serde_json::Value::Null); + // The Path widget binds `$pathStore` one-way (`flow.path → $pathStore`), + // so the editor writes a separate `draft_path` field only when the typed + // path differs from the deployed one. `None` = unchanged. + let draft_path = v + .get("draft_path") + .and_then(|s| s.as_str()) + .filter(|s| !s.is_empty() && *s != row.path.as_str()) + .map(|s| s.to_string()); + rows.push(ListableFlow { + workspace_id: w_id.clone(), + path: row.path, + summary: v + .get("summary") + .and_then(|s| s.as_str()) + .unwrap_or("") + .to_string(), + description: v + .get("description") + .and_then(|s| s.as_str()) + .map(|s| s.to_string()), + edited_by: Some(authed.email.clone()), + edited_at: Some(row.created_at), + archived: false, + extra_perms: serde_json::Value::Object(serde_json::Map::new()), + starred: false, + draft_only: Some(true), + ws_error_handler_muted: None, + deployment_msg: None, + labels: None, + // No deployed row to inherit folder labels from. + inherited_labels: None, + is_draft: true, + draft_path, + // Synthesized rows are the authed user's own draft. + draft_users: Some(sqlx::types::Json(vec![DraftUserRef { + username: Some(authed.username.clone()), + }])), + }); + } + } + Ok(Json(rows)) } @@ -517,22 +599,21 @@ async fn create_flow( sqlx::query!( r#"INSERT INTO flow ( workspace_id, path, summary, description, - dependency_job, lock_error_logs, draft_only, tag, + dependency_job, lock_error_logs, tag, dedicated_worker, visible_to_runner_only, on_behalf_of_email, ws_error_handler_muted, value, schema, edited_by, edited_at, labels ) VALUES ( $1, $2, $3, $4, - NULL, '', $5, $6, - $7, $8, $9, - $10, - $11, $12::text::json, $13, now(), $14 + NULL, '', $5, + $6, $7, $8, + $9, + $10, $11::text::json, $12, now(), $13 )"#, w_id, nf.path, nf.summary, nf.description.as_deref().unwrap_or(""), - nf.draft_only, nf.tag, nf.dedicated_worker, nf.visible_to_runner_only.unwrap_or(false), @@ -571,12 +652,15 @@ async fn create_flow( ).execute(&mut *tx).await?; // CLI / git-sync deploys ask us to preserve any existing user draft at this - // path instead of wiping it as part of the deploy. + // path instead of wiping it as part of the deploy. Only wipe the deployer's + // own draft (plus the legacy NULL-email row); see scripts.rs. if !nf.skip_draft_deletion.unwrap_or(false) { sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'flow'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'flow' \ + AND (email = $3 OR email IS NULL)", nf.path, - &w_id + &w_id, + &authed.email, ) .execute(&mut *tx) .await?; @@ -719,18 +803,6 @@ async fn check_schedule_conflict<'c>( Ok(()) } -pub async fn require_is_writer(authed: &ApiAuthed, path: &str, w_id: &str, db: DB) -> Result<()> { - return windmill_api_auth::require_is_writer( - authed, - path, - w_id, - db, - "SELECT extra_perms FROM flow WHERE path = $1 AND workspace_id = $2", - "flow", - ) - .await; -} - #[derive(Serialize)] pub struct FlowVersion { pub id: i64, @@ -799,7 +871,7 @@ async fn get_flow_version( let mut tx = user_db.begin(&authed).await?; let flow = sqlx::query_as::<_, Flow>( - "SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.draft_only, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of_email, flow.labels, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by + "SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of_email, flow.labels, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by FROM flow LEFT JOIN flow_version ON flow_version.path = flow.path AND flow_version.workspace_id = flow.workspace_id WHERE flow.path = $1 AND flow.workspace_id = $2 AND flow_version.id = $3", @@ -850,7 +922,6 @@ async fn get_flow_version_by_id( flow.description, flow.archived, flow.extra_perms, - flow.draft_only, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, @@ -987,7 +1058,6 @@ async fn update_flow( description = $3, dependency_job = NULL, lock_error_logs = '', - draft_only = NULL, tag = $4, dedicated_worker = $5, visible_to_runner_only = $6, @@ -1029,8 +1099,8 @@ async fn update_flow( // if new path, must clone flow to new path and delete old flow for flow_version foreign key constraint sqlx::query!( "INSERT INTO flow - (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels) - SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels + (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels) + SELECT workspace_id, $1, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels FROM flow WHERE path = $2 AND workspace_id = $3", nf.path, @@ -1174,12 +1244,15 @@ async fn update_flow( } // CLI / git-sync deploys ask us to preserve any existing user draft at this - // path instead of wiping it as part of the deploy. + // path instead of wiping it as part of the deploy. Only wipe the deployer's + // own draft (plus the legacy NULL-email row); see scripts.rs. if !nf.skip_draft_deletion.unwrap_or(false) { sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'flow'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'flow' \ + AND (email = $3 OR email IS NULL)", flow_path, - &w_id + &w_id, + &authed.email, ) .execute(&mut *tx) .await?; @@ -1358,10 +1431,12 @@ async fn update_flow( } async fn list_tokens( + authed: ApiAuthed, Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, ) -> JsonResult> { let path = path.to_path(); + check_scopes(&authed, || format!("flows:read:{}", path))?; list_tokens_internal(&db, &w_id, &path, true).await } @@ -1397,12 +1472,21 @@ async fn get_deployment_status( Ok(Json(deployment_status)) } +// Fields inlined rather than flattened (axum query bool quirk); see GetScriptByPathQuery in scripts.rs. +#[derive(Deserialize)] +struct GetFlowByPathQuery { + with_starred_info: Option, + #[serde(default)] + get_draft: bool, +} + async fn get_flow_by_path( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, - Query(query): Query, -) -> JsonResult { + Query(query): Query, +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || format!("flows:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; @@ -1416,9 +1500,8 @@ async fn get_flow_by_path( flow.summary, flow.description, flow.archived, - flow.extra_perms, - flow.draft_only, - flow.dedicated_worker, + flow.extra_perms, + flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, @@ -1444,7 +1527,7 @@ async fn get_flow_by_path( "#, ) .bind(path) - .bind(w_id) + .bind(&w_id) .bind(&authed.username) .fetch_optional(&mut *tx) .await? @@ -1458,9 +1541,8 @@ async fn get_flow_by_path( flow.summary, flow.description, flow.archived, - flow.extra_perms, - flow.draft_only, - flow.dedicated_worker, + flow.extra_perms, + flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, @@ -1481,90 +1563,26 @@ async fn get_flow_by_path( "#, ) .bind(path) - .bind(w_id) + .bind(&w_id) .fetch_optional(&mut *tx) .await? }; tx.commit().await?; - let flow = not_found_if_none(flow_o, "Flow", path)?; - Ok(Json(flow)) -} - -#[derive(Serialize, sqlx::FromRow)] -pub struct FlowWDraft { - pub path: String, - pub summary: String, - pub description: String, - pub schema: Option, - pub value: sqlx::types::Json>, - pub extra_perms: serde_json::Value, - #[serde(skip_serializing_if = "Option::is_none")] - pub draft: Option>>, - /// Timestamp at which the most recent DB draft was created. - #[serde(skip_serializing_if = "Option::is_none")] - pub draft_created_at: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - pub draft_only: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub tag: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub ws_error_handler_muted: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub dedicated_worker: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub visible_to_runner_only: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub on_behalf_of_email: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub labels: Option>, -} - -async fn get_flow_by_path_w_draft( - authed: ApiAuthed, - Extension(user_db): Extension, - Path((w_id, path)): Path<(String, StripPath)>, -) -> JsonResult { - let path = path.to_path(); - check_scopes(&authed, || format!("flows:read:{}", path))?; - let mut tx = user_db.begin(&authed).await?; - let flow_o = sqlx::query_as::<_, FlowWDraft>( - "SELECT - flow.path, - flow.summary, - flow.description, - flow_version.schema, - flow_version.value, - flow.extra_perms, - flow.draft_only, - flow.ws_error_handler_muted, - flow.dedicated_worker, - draft.value AS draft, - draft.created_at AS draft_created_at, - flow.tag, - flow.visible_to_runner_only, - flow.on_behalf_of_email, - flow.labels - FROM flow - LEFT JOIN draft - ON flow.path = draft.path - AND draft.workspace_id = $2 - AND draft.typ = 'flow' - LEFT JOIN flow_version - ON flow_version.id = flow.versions[array_upper(flow.versions, 1)] - WHERE flow.path = $1 - AND flow.workspace_id = $2", + // No deployed row + `get_draft`: fall back to the draft table; see scripts.rs. + let overlay = overlay_or_draft_only( + &db, + &w_id, + &authed.email, + UserDraftItemKind::Flow, + path, + query.get_draft, + flow_o, + || windmill_common::error::Error::NotFound(format!("Flow not found at path {path}")), ) - .bind(path) - .bind(w_id) - .fetch_optional(&mut *tx) .await?; - - tx.commit().await?; - - let flow = not_found_if_none(flow_o, "Flow", path)?; - Ok(Json(flow)) + Ok(Json(overlay)) } async fn exists_flow_by_path( @@ -2091,8 +2109,7 @@ mod tests { }, "stop_after_if": { "expr": "foo = 'bar'", - "skip_if_stopped": false, - "error_message": null + "skip_if_stopped": false } }, { @@ -2113,8 +2130,7 @@ mod tests { }, "stop_after_if": { "expr": "previous.isEmpty()", - "skip_if_stopped": false, - "error_message": null + "skip_if_stopped": false } } ], @@ -2127,8 +2143,7 @@ mod tests { }, "stop_after_if": { "expr": "previous.isEmpty()", - "skip_if_stopped": false, - "error_message": null + "skip_if_stopped": false } }, }); diff --git a/backend/windmill-api-groups/src/folders.rs b/backend/windmill-api-groups/src/folders.rs index 8db0130cb5..eedb43bc1a 100644 --- a/backend/windmill-api-groups/src/folders.rs +++ b/backend/windmill-api-groups/src/folders.rs @@ -86,6 +86,14 @@ pub struct UpdateFolder { pub labels: Option>, } +// Folder labels are surfaced verbatim as `inherited_labels` and rendered in keyed +// `{#each}` blocks; a repeated label is a duplicate key that crashes the list views. +// The UI dedups on entry but API/CLI/git-sync writes do not, so normalize on write. +fn dedup_labels(labels: &mut Vec) { + let mut seen = std::collections::HashSet::new(); + labels.retain(|l| seen.insert(l.clone())); +} + #[derive(Deserialize)] pub struct Owner { pub owner: String, @@ -215,7 +223,7 @@ async fn check_name_conflict<'c>( } lazy_static! { - static ref VALID_FOLDER_NAME: Regex = Regex::new(r#"^[a-zA-Z_0-9]+$"#).unwrap(); + static ref VALID_FOLDER_NAME: Regex = Regex::new(r#"^[a-zA-Z_0-9-]+$"#).unwrap(); } async fn create_folder( @@ -226,8 +234,11 @@ async fn create_folder( Extension(webhook): Extension, Extension(cache): Extension>, Path(w_id): Path, - Json(ng): Json, + Json(mut ng): Json, ) -> Result { + if let Some(labels) = ng.labels.as_mut() { + dedup_labels(labels); + } if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, AuditAuthorable::username(&authed), @@ -244,7 +255,7 @@ async fn create_folder( if !VALID_FOLDER_NAME.is_match(&ng.name) { return Err(windmill_common::error::Error::BadRequest(format!( - "Folder name can only contain alphanumeric characters, underscores" + "Folder name can only contain alphanumeric characters, underscores, and hyphens" ))); } check_name_conflict(&mut tx, &w_id, &ng.name).await?; @@ -471,7 +482,8 @@ async fn update_folder( ); } - if let Some(labels) = ng.labels.as_ref() { + if let Some(labels) = ng.labels.as_mut() { + dedup_labels(labels); if labels.is_empty() { // normalize cleared labels to NULL so the field stays out of API/tarball output sqlb.set("labels", "NULL"); @@ -967,3 +979,22 @@ pub async fn log_folder_permission_change<'c, E: sqlx::Executor<'c, Database = P .await?; Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn folder_name_allows_hyphens() { + // #8474: hyphens are valid in folder names, consistent with owner/path + // validation (which already permits them) and with folders created via + // the CLI / by deploying to an `f//...` path. + assert!(VALID_FOLDER_NAME.is_match("folder-name")); + assert!(VALID_FOLDER_NAME.is_match("foo_bar")); + assert!(VALID_FOLDER_NAME.is_match("Foo123")); + // Disallowed characters are still rejected. + assert!(!VALID_FOLDER_NAME.is_match("foo/bar")); + assert!(!VALID_FOLDER_NAME.is_match("foo bar")); + assert!(!VALID_FOLDER_NAME.is_match("")); + } +} diff --git a/backend/windmill-api-groups/src/granular_acls.rs b/backend/windmill-api-groups/src/granular_acls.rs index 1f88c33e9f..663644e8e0 100644 --- a/backend/windmill-api-groups/src/granular_acls.rs +++ b/backend/windmill-api-groups/src/granular_acls.rs @@ -105,8 +105,6 @@ async fn add_granular_acl( return Err(Error::BadRequest("Invalid kind".to_string())); } - let mut tx = user_db.begin(&authed).await?; - let identifier = if kind == "group_" || kind == "folder" || kind == "volume" { "name" } else { @@ -140,6 +138,8 @@ async fn add_granular_acl( } } + let mut tx = user_db.begin(&authed).await?; + if kind == "folder" { if let Some(obj) = sqlx::query_scalar!( "SELECT owners FROM folder WHERE name = $1 AND workspace_id = $2", diff --git a/backend/windmill-api-integration-tests/tests/apps.rs b/backend/windmill-api-integration-tests/tests/apps.rs index 63bd96345b..3d70c09ce1 100644 --- a/backend/windmill-api-integration-tests/tests/apps.rs +++ b/backend/windmill-api-integration-tests/tests/apps.rs @@ -82,12 +82,6 @@ async fn test_app_endpoints(db: Pool) -> anyhow::Result<()> { let resp = authed_get(port, "get/p", "u/test-user/nonexistent").await; assert_eq!(resp.status(), 404); - // --- get draft --- - let resp = authed_get(port, "get/draft", "u/test-user/test_app").await; - assert_eq!(resp.status(), 200); - let body = resp.json::().await?; - assert_eq!(body["path"], "u/test-user/test_app"); - // --- get lite --- let resp = authed_get(port, "get/lite", "u/test-user/test_app").await; assert_eq!(resp.status(), 200); diff --git a/backend/windmill-api-integration-tests/tests/drafts.rs b/backend/windmill-api-integration-tests/tests/drafts.rs index d86c91b6a1..b3bac8d010 100644 --- a/backend/windmill-api-integration-tests/tests/drafts.rs +++ b/backend/windmill-api-integration-tests/tests/drafts.rs @@ -1,105 +1,297 @@ use serde_json::json; use sqlx::{Pool, Postgres}; - use windmill_test_utils::*; -fn client() -> reqwest::Client { - reqwest::Client::new() +const WS: &str = "test-workspace"; + +/// A reqwest client that sends `Authorization: Bearer `. The base +/// fixture seeds: SECRET_TOKEN (test-user, admin), SECRET_TOKEN_2 +/// (test-user-2, non-admin), SECRET_TOKEN_3 (test-user-3, non-admin). +fn client_for(token: &str) -> reqwest::Client { + let mut headers = reqwest::header::HeaderMap::new(); + headers.insert( + reqwest::header::AUTHORIZATION, + reqwest::header::HeaderValue::from_str(&format!("Bearer {token}")).unwrap(), + ); + reqwest::ClientBuilder::new() + .default_headers(headers) + .build() + .unwrap() } -fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { - builder.header("Authorization", "Bearer SECRET_TOKEN") +fn save_url(port: u16, kind: &str, path: &str) -> String { + format!("http://localhost:{port}/api/w/{WS}/drafts/update/{kind}/{path}") } +async fn draft_count(db: &Pool, path: &str, kind: &str, email: &str) -> i64 { + sqlx::query_scalar::<_, i64>( + "SELECT count(*) FROM draft WHERE workspace_id = $1 AND path = $2 \ + AND typ = $3::text::DRAFT_KIND AND email = $4", + ) + .bind(WS) + .bind(path) + .bind(kind) + .bind(email) + .fetch_one(db) + .await + .unwrap() +} + +/// Upsert → conflict (stale last_sync) → force-overwrite → delete, the +/// optimistic-concurrency contract `update_draft` exists to enforce. #[sqlx::test(migrations = "../migrations", fixtures("base"))] -async fn test_draft_endpoints(db: Pool) -> anyhow::Result<()> { +async fn test_update_draft_conflict_lifecycle(db: Pool) -> anyhow::Result<()> { initialize_tracing().await; let server = ApiServer::start(db.clone()).await?; let port = server.addr.port(); - let base = format!("http://localhost:{port}/api/w/test-workspace/drafts"); + let c = client_for("SECRET_TOKEN"); + let path = "u/test-user/draft_x"; + let url = save_url(port, "script", path); - // create a script first so the draft has a valid path - let resp = authed(client().post(format!( - "http://localhost:{port}/api/w/test-workspace/scripts/create" - ))) - .json(&json!({ - "path": "u/test-user/draft_script", - "summary": "Script for draft test", - "description": "", - "content": "export async function main() { return 1; }", - "language": "deno", - "schema": { - "$schema": "https://json-schema.org/draft/2020-12/schema", - "type": "object", - "properties": {}, - "required": [] - } - })) - .send() - .await - .unwrap(); - assert_eq!(resp.status(), 201, "create script: {}", resp.text().await?); - - // --- create draft --- - let resp = authed(client().post(format!("{base}/create"))) - .json(&json!({ - "path": "u/test-user/draft_script", - "typ": "script", - "value": { - "content": "export async function main() { return 2; }", - "language": "deno" - } - })) + // First save: no last_sync ("treat as fresh") → saved. + let r = c + .post(&url) + .json(&json!({ "value": { "n": 1 } })) .send() - .await - .unwrap(); - assert_eq!(resp.status(), 201, "create draft: {}", resp.text().await?); + .await?; + assert_eq!(r.status(), 200, "first save"); + let body: serde_json::Value = r.json().await?; + assert_eq!(body["status"], "saved"); + let ts1 = body["current_timestamp"].as_str().unwrap().to_string(); + assert_eq!( + draft_count(&db, path, "script", "test@windmill.dev").await, + 1 + ); - // verify draft exists via script get/draft endpoint - let resp = authed(client().get(format!( - "http://localhost:{port}/api/w/test-workspace/scripts/get/draft/u/test-user/draft_script" - ))) - .send() - .await - .unwrap(); - assert_eq!(resp.status(), 200); - let body = resp.json::().await?; - assert!(body["draft"].is_object(), "expected draft to be present"); + // A tiny gap so the next now() is strictly greater than ts1. + tokio::time::sleep(std::time::Duration::from_millis(15)).await; - // --- update draft (create with same path overwrites) --- - let resp = authed(client().post(format!("{base}/create"))) - .json(&json!({ - "path": "u/test-user/draft_script", - "typ": "script", - "value": { - "content": "export async function main() { return 3; }", - "language": "deno" - } - })) + // Save with the matching last_sync → not stale → saved, newer ts. + let r = c + .post(&url) + .json(&json!({ "value": { "n": 2 }, "last_sync": ts1 })) .send() - .await - .unwrap(); - assert_eq!(resp.status(), 201); + .await?; + let body: serde_json::Value = r.json().await?; + assert_eq!(body["status"], "saved", "in-order save"); + let ts2 = body["current_timestamp"].as_str().unwrap().to_string(); + assert_ne!(ts1, ts2, "timestamp should advance"); - // --- delete draft --- - let resp = authed(client().delete(format!( - "{base}/delete/script/u/test-user/draft_script" - ))) - .send() - .await - .unwrap(); - assert_eq!(resp.status(), 200); + // Save with the now-stale ts1 → conflict, server reports its current ts. + let r = c + .post(&url) + .json(&json!({ "value": { "n": 3 }, "last_sync": ts1 })) + .send() + .await?; + let body: serde_json::Value = r.json().await?; + assert_eq!(body["status"], "conflict", "stale save must conflict"); + assert_eq!(body["current_timestamp"].as_str().unwrap(), ts2); - // verify draft is gone - let resp = authed(client().get(format!( - "http://localhost:{port}/api/w/test-workspace/scripts/get/draft/u/test-user/draft_script" - ))) - .send() - .await - .unwrap(); - assert_eq!(resp.status(), 200); - let body = resp.json::().await?; - assert!(body["draft"].is_null(), "expected draft to be deleted"); + // The conflicting write must NOT have landed — value is still {n:2}. + let stored: serde_json::Value = sqlx::query_scalar::<_, sqlx::types::Json>( + "SELECT value FROM draft WHERE workspace_id = $1 AND path = $2 \ + AND typ = 'script' AND email = 'test@windmill.dev'", + ) + .bind(WS) + .bind(path) + .fetch_one(&db) + .await? + .0; + assert_eq!(stored["n"], 2, "conflicting write must be rejected"); + + // force = true overrides the conflict check. + let r = c + .post(&url) + .json(&json!({ "value": { "n": 3 }, "last_sync": ts1, "force": true })) + .send() + .await?; + assert_eq!( + r.json::().await?["status"], + "saved", + "force" + ); + + // Delete (value: null) → saved, row gone. + let r = c.post(&url).json(&json!({ "value": null })).send().await?; + assert_eq!( + r.json::().await?["status"], + "saved", + "delete" + ); + assert_eq!( + draft_count(&db, path, "script", "test@windmill.dev").await, + 0, + "row removed after delete" + ); + + Ok(()) +} + +/// require_can_write_path: own namespace allowed, another user's namespace +/// rejected, operators rejected outright. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_update_draft_write_authorization(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let user2 = client_for("SECRET_TOKEN_2"); // test-user-2, non-admin + + // Own namespace → allowed. + let r = user2 + .post(save_url(port, "script", "u/test-user-2/own")) + .json(&json!({ "value": { "a": 1 } })) + .send() + .await?; + assert_eq!(r.status(), 200, "own namespace allowed"); + + // Another user's namespace, no grant → rejected. + let r = user2 + .post(save_url(port, "script", "u/test-user/theirs")) + .json(&json!({ "value": { "a": 1 } })) + .send() + .await?; + assert_eq!(r.status(), 401, "other user's namespace rejected"); + + // Operators can't save drafts at all. + sqlx::query( + "UPDATE usr SET operator = true WHERE workspace_id = $1 AND username = 'test-user-3'", + ) + .bind(WS) + .execute(&db) + .await?; + let op = client_for("SECRET_TOKEN_3"); + let r = op + .post(save_url(port, "script", "u/test-user-3/own")) + .json(&json!({ "value": { "a": 1 } })) + .send() + .await?; + assert_eq!(r.status(), 401, "operator rejected"); + + Ok(()) +} + +/// The item-level extra_perms fallback: a user granted write on a deployed +/// item (via the Share dialog) can save a draft on it even though it's +/// outside their namespace. Regression test for the authz drop. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_update_draft_extra_perms_writer(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let path = "u/test-user/shared"; + + // A deployed script owned by test-user, shared with test-user-2 (write). + sqlx::query( + "INSERT INTO script (workspace_id, hash, path, summary, description, content, \ + language, schema, extra_perms, created_by) \ + VALUES ($1, 1, $2, '', '', 'x', 'deno', '{}'::jsonb, \ + '{\"u/test-user-2\": true}'::jsonb, 'test-user')", + ) + .bind(WS) + .bind(path) + .execute(&db) + .await?; + + let user2 = client_for("SECRET_TOKEN_2"); + let r = user2 + .post(save_url(port, "script", path)) + .json(&json!({ "value": { "a": 1 } })) + .send() + .await?; + assert_eq!( + r.status(), + 200, + "extra_perms writer can save a draft: {}", + r.text().await? + ); + + // Without a grant on a different shared item → still rejected. + sqlx::query( + "INSERT INTO script (workspace_id, hash, path, summary, description, content, \ + language, schema, extra_perms, created_by) \ + VALUES ($1, 2, 'u/test-user/private', '', '', 'x', 'deno', '{}'::jsonb, \ + '{}'::jsonb, 'test-user')", + ) + .bind(WS) + .execute(&db) + .await?; + let r = user2 + .post(save_url(port, "script", "u/test-user/private")) + .json(&json!({ "value": { "a": 1 } })) + .send() + .await?; + assert_eq!(r.status(), 401, "no grant → rejected"); + + // A READ-ONLY grant (`extra_perms` value false) must not allow draft + // saves: the write check defers to RLS via `SELECT ... FOR UPDATE`, + // and locking applies the UPDATE policies — visibility under the + // SELECT policy alone isn't enough. Pins the FOR UPDATE semantics the + // probe relies on. + sqlx::query( + "INSERT INTO script (workspace_id, hash, path, summary, description, content, \ + language, schema, extra_perms, created_by) \ + VALUES ($1, 3, 'u/test-user/readonly', '', '', 'x', 'deno', '{}'::jsonb, \ + '{\"u/test-user-2\": false}'::jsonb, 'test-user')", + ) + .bind(WS) + .execute(&db) + .await?; + let r = user2 + .post(save_url(port, "script", "u/test-user/readonly")) + .json(&json!({ "value": { "a": 1 } })) + .send() + .await?; + assert_eq!(r.status(), 401, "read-only grant → rejected"); + + Ok(()) +} + +/// Cross-user draft viewing (`GET /drafts/get/{kind}/{path}`) is disabled +/// for the drawer kinds (resource/variable/triggers) so a viewer can't +/// read another user's draft; it stays available for script/flow/app. +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_cross_user_draft_privacy(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // test-user saves a variable draft and a script draft in a shared folder. + let admin = client_for("SECRET_TOKEN"); + admin + .post(save_url(port, "variable", "f/shared/v")) + .json(&json!({ "value": { "variable": { "value": "x", "is_secret": false } } })) + .send() + .await?; + + let user2 = client_for("SECRET_TOKEN_2"); + // Variable is a drawer kind → cross-user view is forbidden regardless of + // path access (the kind gate fires first). + let r = user2 + .get(format!( + "http://localhost:{port}/api/w/{WS}/drafts/get/variable/f/shared/v?username=test-user" + )) + .send() + .await?; + assert_eq!( + r.status(), + 404, + "variable drafts are private to their owner" + ); + + // Sharing kinds (script) are NOT gated by the kind check — a missing + // draft / no access yields 404 too, but the "private to their owner" + // wording is specific to the drawer kinds, so assert it's absent here. + let r = user2 + .get(format!( + "http://localhost:{port}/api/w/{WS}/drafts/get/script/f/shared/s?username=test-user" + )) + .send() + .await?; + let body = r.text().await?; + assert!( + !body.contains("private to their owner"), + "script kind must not be blocked by the cross-user privacy gate: {body}" + ); Ok(()) } diff --git a/backend/windmill-api-integration-tests/tests/flows.rs b/backend/windmill-api-integration-tests/tests/flows.rs index b6075c8e69..774c118e7b 100644 --- a/backend/windmill-api-integration-tests/tests/flows.rs +++ b/backend/windmill-api-integration-tests/tests/flows.rs @@ -82,12 +82,6 @@ async fn test_flow_endpoints(db: Pool) -> anyhow::Result<()> { let resp = authed_get(port, "get", "u/test-user/nonexistent").await; assert_eq!(resp.status(), 404); - // --- get draft --- - let resp = authed_get(port, "get/draft", "u/test-user/test_flow").await; - assert_eq!(resp.status(), 200); - let body = resp.json::().await?; - assert_eq!(body["path"], "u/test-user/test_flow"); - // --- list --- let resp = authed(client().get(format!("{base}/list"))) .send() diff --git a/backend/windmill-api-integration-tests/tests/scripts.rs b/backend/windmill-api-integration-tests/tests/scripts.rs index c374b757a4..e5f6cb8aa8 100644 --- a/backend/windmill-api-integration-tests/tests/scripts.rs +++ b/backend/windmill-api-integration-tests/tests/scripts.rs @@ -98,12 +98,6 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { let body = resp.json::().await?; assert_eq!(body["path"], "u/test-user/test_script"); - // --- get draft --- - let resp = authed_get(port, "get/draft", "u/test-user/test_script").await; - assert_eq!(resp.status(), 200); - let body = resp.json::().await?; - assert_eq!(body["path"], "u/test-user/test_script"); - // --- raw by path (requires language extension) --- let resp = authed_get(port, "raw/p", "u/test-user/test_script.ts").await; assert_eq!(resp.status(), 200); diff --git a/backend/windmill-api-integration-tests/tests/workspace_comparison.rs b/backend/windmill-api-integration-tests/tests/workspace_comparison.rs index 82447daee2..88ffe6bb26 100644 --- a/backend/windmill-api-integration-tests/tests/workspace_comparison.rs +++ b/backend/windmill-api-integration-tests/tests/workspace_comparison.rs @@ -92,8 +92,8 @@ async fn test_compare_workspaces_comprehensive(db: Pool) -> anyhow::Re // Create app sqlx::query!( - "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms, draft_only) - VALUES ('test-workspace', 'f/shared/dashboard', 'Dashboard app', '{}', ARRAY[1::bigint], '{}', false)" + "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms) + VALUES ('test-workspace', 'f/shared/dashboard', 'Dashboard app', '{}', ARRAY[1::bigint], '{}')" ) .execute(&db) .await?; @@ -1336,3 +1336,188 @@ async fn test_compare_workspaces_fork_only_folder_visibility( Ok(()) } + +/// Regression test: deleting a fork must purge its `workspace_diff` and +/// `skip_workspace_diff_tally` rows. These tables are keyed by workspace id with +/// no FK cascade, and a fork id is reused when a fork is deleted and recreated +/// under the same name. If the cached diff rows survive the delete, they leak +/// onto the next fork sharing that id and produce a spurious "changes not +/// visible" warning that hides the deploy button (WIN-2066). +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_delete_fork_purges_workspace_diff(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let client = windmill_api_client::create_client( + &format!("http://localhost:{port}"), + "SECRET_TOKEN".to_string(), + ); + let base_url = format!("http://localhost:{port}/api"); + + // Create the fork so the caller owns it (delete is authorized for fork owners). + let fork_response = client + .client() + .post(&format!( + "{base_url}/w/test-workspace/workspaces/create_fork" + )) + .json(&json!({ + "id": "wm-fork-test-workspace", + "name": "Test Fork", + "color": "#0000ff" + })) + .send() + .await?; + assert!( + fork_response.status().is_success(), + "Fork creation should succeed: {}", + fork_response.status() + ); + + // Seed cached diff state for the fork: as the fork side of a pair, as the + // source side of a pair, and a skip-tally row. + sqlx::query!( + "INSERT INTO workspace_diff + (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork) + VALUES ('test-workspace', 'wm-fork-test-workspace', 'f/shared/leaky', 'script', 1, 0, true, true, true)" + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO workspace_diff + (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes) + VALUES ('wm-fork-test-workspace', 'test-workspace', 'f/shared/other', 'script', 0, 1, true)" + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO skip_workspace_diff_tally (workspace_id) VALUES ('wm-fork-test-workspace')" + ) + .execute(&db) + .await?; + + // Delete the fork through the real handler. + let delete_response = client + .client() + .delete(&format!("{base_url}/workspaces/delete/wm-fork-test-workspace")) + .send() + .await?; + assert!( + delete_response.status().is_success(), + "Fork deletion should succeed: {}", + delete_response.status() + ); + + let leftover_diffs = sqlx::query_scalar!( + "SELECT COUNT(*) FROM workspace_diff + WHERE source_workspace_id = 'wm-fork-test-workspace' + OR fork_workspace_id = 'wm-fork-test-workspace'" + ) + .fetch_one(&db) + .await?; + assert_eq!( + leftover_diffs, + Some(0), + "workspace_diff rows referencing the deleted fork must be purged" + ); + + let leftover_skip = sqlx::query_scalar!( + "SELECT COUNT(*) FROM skip_workspace_diff_tally WHERE workspace_id = 'wm-fork-test-workspace'" + ) + .fetch_one(&db) + .await?; + assert_eq!( + leftover_skip, + Some(0), + "skip_workspace_diff_tally row for the deleted fork must be purged" + ); + + Ok(()) +} + +/// Regression test: creating a fork must start with clean diff state even when +/// the (reusable) fork id was previously occupied by a deleted fork. Stale +/// `workspace_diff` / `skip_workspace_diff_tally` rows left behind by an earlier +/// occupant would otherwise leak onto the new fork — a stale skip row suppresses +/// comparison entirely, and stale diff rows produce a spurious "changes not +/// visible" warning that hides the deploy button (WIN-2066). +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_create_fork_purges_stale_diff_state(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let client = windmill_api_client::create_client( + &format!("http://localhost:{port}"), + "SECRET_TOKEN".to_string(), + ); + let base_url = format!("http://localhost:{port}/api"); + + // Simulate leftovers from a previously deleted fork that reused this id: + // diff rows on both sides plus a skip-tally row, with no workspace yet. + sqlx::query!( + "INSERT INTO workspace_diff + (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes, exists_in_source, exists_in_fork) + VALUES ('test-workspace', 'wm-fork-test-workspace', 'f/shared/leaky', 'script', 1, 0, true, true, true)" + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO workspace_diff + (source_workspace_id, fork_workspace_id, path, kind, ahead, behind, has_changes) + VALUES ('wm-fork-test-workspace', 'test-workspace', 'f/shared/other', 'script', 0, 1, true)" + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO skip_workspace_diff_tally (workspace_id) VALUES ('wm-fork-test-workspace')" + ) + .execute(&db) + .await?; + + // Create the fork reusing that id; the conflict check passes because no + // workspace row exists for it. + let fork_response = client + .client() + .post(&format!( + "{base_url}/w/test-workspace/workspaces/create_fork" + )) + .json(&json!({ + "id": "wm-fork-test-workspace", + "name": "Test Fork", + "color": "#0000ff" + })) + .send() + .await?; + assert!( + fork_response.status().is_success(), + "Fork creation should succeed: {}", + fork_response.status() + ); + + let leftover_diffs = sqlx::query_scalar!( + "SELECT COUNT(*) FROM workspace_diff + WHERE source_workspace_id = 'wm-fork-test-workspace' + OR fork_workspace_id = 'wm-fork-test-workspace'" + ) + .fetch_one(&db) + .await?; + assert_eq!( + leftover_diffs, + Some(0), + "stale workspace_diff rows must be purged on fork creation" + ); + + let leftover_skip = sqlx::query_scalar!( + "SELECT COUNT(*) FROM skip_workspace_diff_tally WHERE workspace_id = 'wm-fork-test-workspace'" + ) + .fetch_one(&db) + .await?; + assert_eq!( + leftover_skip, + Some(0), + "stale skip_workspace_diff_tally row must be purged on fork creation" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/workspaces.rs b/backend/windmill-api-integration-tests/tests/workspaces.rs index fcbdeea8ba..92f179c39b 100644 --- a/backend/windmill-api-integration-tests/tests/workspaces.rs +++ b/backend/windmill-api-integration-tests/tests/workspaces.rs @@ -646,6 +646,20 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { .unwrap(); assert_eq!(resp.json::().await?, true); + // Regression: changing a fork's workspace id must preserve its parent + // linkage. Dropping it leaves a wm-fork- workspace with no parent — a + // "fork of nothing" that can no longer be compared or merged. + let parent: Option = + sqlx::query_scalar("SELECT parent_workspace_id FROM workspace WHERE id = $1") + .bind("wm-fork-renamed") + .fetch_one(&db) + .await?; + assert_eq!( + parent.as_deref(), + Some("new-test-ws"), + "renamed fork must keep its parent_workspace_id" + ); + // --- create_fork over an existing (active) workspace id: clear 400, not a raw SQL 500 --- let resp = authed(client().post(format!("{new_ws_base}/create_fork"))) .json(&json!({ diff --git a/backend/windmill-api-jobs/src/execution.rs b/backend/windmill-api-jobs/src/execution.rs index 0581b7acd9..8ec60a69ba 100644 --- a/backend/windmill-api-jobs/src/execution.rs +++ b/backend/windmill-api-jobs/src/execution.rs @@ -55,7 +55,7 @@ pub async fn check_tag_available_for_workspace( ) -> error::Result<()> { if let Some(tag) = tag.as_deref().filter(|t| !t.is_empty()) { let tags = get_scope_tags(authed); - check_tag_available_for_workspace_internal(&db, w_id, tag, &authed.email, tags).await + check_tag_available_for_workspace_internal(db, w_id, tag, &authed.email, tags).await } else { Ok(()) } diff --git a/backend/windmill-api-jobs/src/jobs_export.rs b/backend/windmill-api-jobs/src/jobs_export.rs index 5de2c51795..0b188c0fa0 100644 --- a/backend/windmill-api-jobs/src/jobs_export.rs +++ b/backend/windmill-api-jobs/src/jobs_export.rs @@ -16,7 +16,7 @@ use uuid::Uuid; use windmill_common::{ db::UserDB, error, - jobs::{JobKind, JobStatus, JobTriggerKind}, + jobs::{is_safe_log_file_path, JobKind, JobStatus, JobTriggerKind}, scripts::ScriptLang, utils::{paginate, paginate_without_limits, require_admin, Pagination}, }; @@ -328,6 +328,20 @@ pub async fn import_completed_jobs( ) -> error::Result { require_admin(authed.is_admin, &authed.username)?; + // log_file_index is read back by the log endpoints as paths under the windmill + // log directory; an attacker-supplied traversal here would become an arbitrary + // file read. Reject anything that could escape the log directory at ingestion. + for job in &jobs { + if let Some(file_index) = &job.log_file_index { + if file_index.iter().any(|p| !is_safe_log_file_path(p)) { + return Err(error::Error::BadRequest(format!( + "Invalid log_file_index for job {}: entries must be relative paths without '..'", + job.id + ))); + } + } + } + let mut tx = user_db.begin(&authed).await?; for job in jobs { @@ -645,8 +659,34 @@ pub async fn delete_jobs( .await? .rows_affected(); + // job_ids are request-supplied, so scope every side-table delete to the workspace exactly + // like the v2_job delete below — otherwise a workspace admin could erase another + // workspace's side rows by passing foreign job ids. zombie_job_counter and + // flow_conversation_message have no workspace_id, so scope them via v2_job / their conversation. + // (Side-table list kept in sync with windmill_common::jobs::delete_jobs.) let zombie_deleted = sqlx::query!( - "DELETE FROM zombie_job_counter WHERE job_id = ANY($1)", + "DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM v2_job WHERE workspace_id = $1 AND id = ANY($2))", + &w_id, + &job_ids + ) + .execute(&mut *tx) + .await? + .rows_affected(); + + let dispatch_event_deleted = sqlx::query!( + "DELETE FROM dispatch_event WHERE workspace_id = $1 AND producer_job_id = ANY($2)", + &w_id, + &job_ids + ) + .execute(&mut *tx) + .await? + .rows_affected(); + + let conversation_message_deleted = sqlx::query!( + "DELETE FROM flow_conversation_message m + USING flow_conversation c + WHERE m.conversation_id = c.id AND c.workspace_id = $1 AND m.job_id = ANY($2)", + &w_id, &job_ids ) .execute(&mut *tx) @@ -674,6 +714,8 @@ pub async fn delete_jobs( + queue_deleted + completed_deleted + zombie_deleted + + dispatch_event_deleted + + conversation_message_deleted + jobs_deleted; tracing::info!( diff --git a/backend/windmill-api-jobs/src/types.rs b/backend/windmill-api-jobs/src/types.rs index 6252cef9b7..c2175753e6 100644 --- a/backend/windmill-api-jobs/src/types.rs +++ b/backend/windmill-api-jobs/src/types.rs @@ -303,6 +303,7 @@ pub struct UnifiedJob { pub preprocessed: Option, pub worker: Option, pub runnable_settings_handle: Option, + pub is_retry: Option, } const CJ_FIELDS: &[&str] = &[ @@ -344,6 +345,7 @@ const CJ_FIELDS: &[&str] = &[ "v2_job.preprocessed", "v2_job_completed.worker", "null as runnable_settings_handle", + "EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", ]; const QJ_FIELDS: &[&str] = &[ @@ -385,6 +387,7 @@ const QJ_FIELDS: &[&str] = &[ "v2_job.preprocessed", "v2_job_queue.worker", "v2_job_queue.runnable_settings_handle", + "EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", ]; impl UnifiedJob { @@ -438,6 +441,7 @@ impl From for Job { priority: uj.priority, labels: uj.labels, preprocessed: uj.preprocessed, + is_retry: uj.is_retry, }, )), "QueuedJob" => Job::QueuedJob(JobExtended::new( @@ -487,6 +491,7 @@ impl From for Job { preprocessed: uj.preprocessed, runnable_settings_handle: uj.runnable_settings_handle, labels: uj.labels, + is_retry: uj.is_retry, }, )), t => panic!("job type {} not valid", t), diff --git a/backend/windmill-api-openapi/Cargo.toml b/backend/windmill-api-openapi/Cargo.toml index ee1727c059..24a6f2aed8 100644 --- a/backend/windmill-api-openapi/Cargo.toml +++ b/backend/windmill-api-openapi/Cargo.toml @@ -11,7 +11,12 @@ path = "src/lib.rs" [dependencies] windmill-api-auth.workspace = true windmill-common = { workspace = true, default-features = false } -windmill-store.workspace = true +# `try_get_resource_from_db_as` is used unconditionally below but is +# cfg-gated behind a trigger feature in windmill-store; forward +# `http_trigger` so the import resolves even when this crate's targets are +# built in isolation (e.g. `--all-targets` under resolver 2), not only via +# whole-workspace feature unification. +windmill-store = { workspace = true, features = ["http_trigger"] } windmill-trigger-http.workspace = true anyhow.workspace = true axum.workspace = true diff --git a/backend/windmill-api-schedule/src/lib.rs b/backend/windmill-api-schedule/src/lib.rs index 35f71e561e..7cc0111c28 100644 --- a/backend/windmill-api-schedule/src/lib.rs +++ b/backend/windmill-api-schedule/src/lib.rs @@ -25,6 +25,10 @@ use windmill_common::{ db::UserDB, error::{Error, JsonResult, Result}, schedule::Schedule, + user_drafts::{ + delete_all_drafts_for_path, fetch_draft_only_list_rows, overlay_or_draft_only, + UserDraftItemKind, WithDraftOverlay, WithDraftQuery, + }, utils::{ escape_ilike_pattern, not_found_if_none, paginate, Pagination, ScheduleType, StripPath, }, @@ -678,6 +682,9 @@ pub struct ListScheduleQuery { pub summary: Option, pub broad_filter: Option, pub label: Option, + /// When true, append per-user draft-only rows; picker callers leave it off + /// to stay deployed-only. See list synthesis in scripts.rs. + pub include_draft_only: Option, } #[derive(sqlx::FromRow, Serialize, Deserialize, Debug, Clone)] @@ -695,6 +702,15 @@ pub struct ScheduleLight { pub extra_perms: serde_json::Value, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + /// `Some(true)` only on synthesized draft-only rows; `None` on deployed rows. + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub draft_only: Option, + /// True when the authed user has a per-user draft at this path (drives the + /// `*` suffix on the schedules page). + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_draft: Option, /// Labels inherited from the parent folder, computed at read time. #[sqlx(default)] #[serde(skip_serializing_if = "Option::is_none")] @@ -703,6 +719,7 @@ pub struct ScheduleLight { async fn list_schedule( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path(w_id): Path, Query(lsq): Query, ) -> JsonResult> { @@ -724,13 +741,20 @@ async fn list_schedule( "labels", "folder_labels(workspace_id, path) as inherited_labels", ]) + // Scalar EXISTS flags the authed user's per-user draft; see resources.rs. + .field( + &"EXISTS(SELECT 1 FROM draft WHERE draft.workspace_id = schedule.workspace_id \ + AND draft.path = schedule.path AND draft.typ = 'trigger_schedule' \ + AND draft.email = ?) as is_draft" + .bind(&authed.email), + ) .order_by("edited_at", true) .and_where("workspace_id = ?".bind(&w_id)) .offset(offset) .limit(per_page) .clone(); - if let Some(path) = lsq.path { - sqlb.and_where_eq("script_path", "?".bind(&path)); + if let Some(path) = lsq.path.as_ref() { + sqlb.and_where_eq("script_path", "?".bind(path)); } if let Some(is_flow) = lsq.is_flow { sqlb.and_where_eq("is_flow", "?".bind(&is_flow)); @@ -773,10 +797,96 @@ async fn list_schedule( } } let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; - let rows = sqlx::query_as::<_, ScheduleLight>(&sql) + let mut rows = sqlx::query_as::<_, ScheduleLight>(&sql) .fetch_all(&mut *tx) .await?; tx.commit().await?; + + // Append the authed user's draft-only schedules; see scripts.rs. + if lsq.include_draft_only.unwrap_or(false) + && !authed.is_operator + && offset == 0 + && lsq.path.is_none() + && lsq.is_flow.is_none() + && lsq.args.is_none() + && lsq.path_start.is_none() + && lsq.schedule_path.is_none() + && lsq.description.is_none() + && lsq.summary.is_none() + && lsq.broad_filter.is_none() + && lsq.label.is_none() + { + let draft_only_rows = fetch_draft_only_list_rows( + &db, + &w_id, + &authed.email, + UserDraftItemKind::TriggerSchedule, + ) + .await?; + + for row in draft_only_rows { + let v: serde_json::Value = + serde_json::from_str(row.value.0.get()).unwrap_or(serde_json::Value::Null); + // Schedule editor's draft mirrors NewSchedule: { path, schedule, timezone, script_path, is_flow, enabled?, summary?, labels? } + let path = v + .get("path") + .and_then(|s| s.as_str()) + .unwrap_or("") + .to_string(); + if path.is_empty() { + continue; + } + let schedule = v + .get("schedule") + .and_then(|x| x.as_str()) + .unwrap_or("") + .to_string(); + let timezone = v + .get("timezone") + .and_then(|x| x.as_str()) + .unwrap_or("UTC") + .to_string(); + let script_path = v + .get("script_path") + .and_then(|x| x.as_str()) + .unwrap_or("") + .to_string(); + let is_flow = v.get("is_flow").and_then(|x| x.as_bool()).unwrap_or(false); + let enabled = v.get("enabled").and_then(|x| x.as_bool()).unwrap_or(true); + let summary = v + .get("summary") + .and_then(|x| x.as_str()) + .map(|s| s.to_string()); + let labels = v.get("labels").and_then(|x| { + x.as_array().map(|arr| { + arr.iter() + .filter_map(|s| s.as_str().map(|s| s.to_string())) + .collect::>() + }) + }); + + rows.push(ScheduleLight { + workspace_id: w_id.clone(), + path, + edited_by: String::new(), + edited_at: row.created_at, + schedule, + timezone, + enabled, + script_path, + is_flow, + summary, + extra_perms: serde_json::Value::Object(serde_json::Map::new()), + labels, + // No deployed row to inherit folder labels from. + inherited_labels: None, + draft_only: Some(true), + // Synthesized rows are the authed user's draft. + is_draft: Some(true), + }); + } + } + Ok(Json(rows)) } @@ -839,16 +949,28 @@ async fn list_schedule_with_jobs( async fn get_schedule( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, -) -> JsonResult { + Query(q): Query, +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || format!("schedules:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; let schedule_o = windmill_queue::schedule::get_schedule_opt(&mut *tx, &w_id, path).await?; - let schedule = not_found_if_none(schedule_o, "Schedule", path)?; tx.commit().await?; - Ok(Json(schedule)) + let overlay = overlay_or_draft_only( + &db, + &w_id, + &authed.email, + UserDraftItemKind::TriggerSchedule, + path, + q.get_draft, + schedule_o, + || Error::NotFound(format!("Schedule not found at path {path}")), + ) + .await?; + Ok(Json(overlay)) } async fn exists_schedule( @@ -1137,6 +1259,9 @@ async fn delete_schedule( tx.commit().await?; + // Schedule gone for everyone: wipe ALL users' drafts at this path; see scripts.rs. + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::TriggerSchedule, path).await?; + handle_deployment_metadata( &authed.email, &authed.username, diff --git a/backend/windmill-api-scripts/Cargo.toml b/backend/windmill-api-scripts/Cargo.toml index 5ea3088837..ecf0ff0b61 100644 --- a/backend/windmill-api-scripts/Cargo.toml +++ b/backend/windmill-api-scripts/Cargo.toml @@ -12,7 +12,7 @@ path = "src/lib.rs" default = [] enterprise = ["windmill-common/enterprise"] private = ["windmill-common/private", "windmill-dep-map/private"] -python = ["dep:windmill-parser-py"] +python = ["dep:windmill-parser-py", "dep:windmill-parser-py-asset"] prometheus = ["dep:prometheus", "windmill-common/prometheus"] [dependencies] windmill-common = { workspace = true, default-features = false } @@ -23,6 +23,11 @@ windmill-audit.workspace = true windmill-git-sync.workspace = true windmill-dep-map.workspace = true windmill-parser-ts.workspace = true +windmill-parser.workspace = true +windmill-parser-ts-asset.workspace = true +windmill-parser-sql-asset.workspace = true +windmill-parser-sql.workspace = true +windmill-parser-yaml.workspace = true axum.workspace = true futures.workspace = true @@ -41,3 +46,4 @@ lazy_static.workspace = true tokio.workspace = true prometheus = { workspace = true, optional = true } windmill-parser-py = { workspace = true, optional = true } +windmill-parser-py-asset = { workspace = true, optional = true } diff --git a/backend/windmill-api-scripts/src/asset_inference.rs b/backend/windmill-api-scripts/src/asset_inference.rs new file mode 100644 index 0000000000..5debb0d271 --- /dev/null +++ b/backend/windmill-api-scripts/src/asset_inference.rs @@ -0,0 +1,254 @@ +//! Server-side asset inference at script deploy. +//! +//! The `asset` rows written at deploy drive the asset-trigger cascade +//! (`fetch_producer_writes` in windmill-queue). Historically they came solely +//! from the client-supplied `NewScript.assets`, so a client with broken +//! inference (e.g. a failed wasm load) deploying `assets: null` silently +//! killed the producer side of the cascade while the subscriber side (parsed +//! server-side from `// on` annotations) kept looking wired. This module makes +//! asset *presence* server-authoritative by re-parsing the deployed content +//! with the same parsers the frontend wasm builds wrap. +//! +//! Merge semantics are a union: server-parsed assets are always present; +//! client entries are kept too (they may carry `alt_access_type` — the user's +//! manual access-type override — or come from client-side detection paths the +//! server has no parser for). For duplicate `(kind, path)` keys the server's +//! parser-derived fields win and the client's `alt_access_type` is preserved. +//! Languages without a server parser (or whose parse fails) fall back to the +//! client list unchanged, matching the previous behavior. + +use std::collections::BTreeMap; + +use windmill_common::{ + assets::{ + asset_access_type_from_parser, asset_kind_from_parser, AssetKind, AssetUsageAccessType, + AssetWithAltAccessType, + }, + scripts::ScriptLang, +}; + +/// Mirror of the frontend `inferAssets` language dispatch (infer.ts): only +/// these languages have a body-asset parser. Returns `None` for unsupported +/// languages or on parse failure — callers then keep the client-supplied list. +fn parse_assets_for_lang( + lang: &ScriptLang, + content: &str, +) -> Option> { + let parsed = match lang { + ScriptLang::DuckDb => windmill_parser_sql_asset::parse_assets(content), + ScriptLang::Bun | ScriptLang::Deno | ScriptLang::Nativets => { + windmill_parser_ts_asset::parse_assets(content) + } + #[cfg(feature = "python")] + ScriptLang::Python3 => windmill_parser_py_asset::parse_assets(content), + ScriptLang::Ansible => windmill_parser_yaml::parse_assets(content), + _ => return None, + }; + match parsed { + Ok(out) => Some(out.assets), + Err(e) => { + tracing::warn!( + "server-side asset inference failed for a {} script; falling back to \ + client-supplied assets: {e:#}", + lang.as_str() + ); + None + } + } +} + +/// Mirror of the frontend `getCommentPrefix` (infer.ts) — the languages whose +/// leading comment block is scanned for `volume: ` annotations. +fn comment_prefix(lang: &ScriptLang) -> Option<&'static str> { + match lang { + ScriptLang::Python3 + | ScriptLang::Bash + | ScriptLang::Powershell + | ScriptLang::Ansible + | ScriptLang::Ruby + | ScriptLang::Rlang => Some("#"), + ScriptLang::Deno + | ScriptLang::Bun + | ScriptLang::Bunnative + | ScriptLang::Nativets + | ScriptLang::Go => Some("//"), + _ => None, + } +} + +/// Mirror of the frontend `parseVolumeAnnotations` (infer.ts): ` +/// volume: ` lines in the leading comment block, each an `rw` volume +/// asset. Scanning stops at the first non-comment line (blank lines are +/// skipped), exactly like the frontend. +fn parse_volume_annotations(content: &str, prefix: &str) -> Vec { + let mut volumes = Vec::new(); + for line in content.lines() { + let trimmed = line.trim(); + if trimmed.is_empty() { + continue; + } + let Some(after) = trimmed.strip_prefix(prefix) else { + break; + }; + let after = after.trim(); + if let Some(rest) = after.strip_prefix("volume:") { + if let Some(path) = rest.trim().split_whitespace().next() { + volumes.push(AssetWithAltAccessType { + path: path.to_string(), + kind: AssetKind::Volume, + access_type: Some(AssetUsageAccessType::RW), + alt_access_type: None, + columns: None, + }); + } + } + } + volumes +} + +/// Parse the deployed content into asset usages, mirroring the frontend's +/// `inferAssets` (body parser per language + volume annotations). `None` +/// means "no server parser produced anything for this language" — distinct +/// from `Some(vec![])`, which is an authoritative "this script uses no +/// assets". +fn infer_script_assets(lang: &ScriptLang, content: &str) -> Option> { + let body_assets = parse_assets_for_lang(lang, content); + let volume_assets = comment_prefix(lang) + .map(|p| parse_volume_annotations(content, p)) + .unwrap_or_default(); + if body_assets.is_none() && volume_assets.is_empty() { + return None; + } + let mut out: Vec = body_assets + .unwrap_or_default() + .into_iter() + .map(|a| AssetWithAltAccessType { + path: a.path, + kind: asset_kind_from_parser(a.kind), + access_type: a.access_type.map(asset_access_type_from_parser), + alt_access_type: None, + columns: a.columns.map(|cols| { + cols.into_iter() + .map(|(k, v)| (k, asset_access_type_from_parser(v))) + .collect::>() + }), + }) + .collect(); + out.extend(volume_assets); + Some(out) +} + +/// The asset list to persist at deploy: server-parsed assets unioned with the +/// client-supplied ones. See the module docs for the exact semantics. +pub fn effective_script_assets( + lang: &ScriptLang, + content: &str, + client_assets: Option>, +) -> Option> { + let Some(inferred) = infer_script_assets(lang, content) else { + return client_assets; + }; + let mut merged: Vec = inferred; + for client in client_assets.into_iter().flatten() { + if let Some(existing) = merged + .iter_mut() + .find(|a| a.kind == client.kind && a.path == client.path) + { + // Server parse wins for parser-derived fields; the client's + // alt_access_type is the user's manual override — keep it. + if existing.alt_access_type.is_none() { + existing.alt_access_type = client.alt_access_type; + } + } else { + merged.push(client); + } + } + Some(merged) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn asset( + kind: AssetKind, + path: &str, + access: Option, + ) -> AssetWithAltAccessType { + AssetWithAltAccessType { + path: path.to_string(), + kind, + access_type: access, + alt_access_type: None, + columns: None, + } + } + + // The bug class this module exists for: client deploys with no assets, + // but the content demonstrably writes one — the server parse must + // produce the producer row anyway. + #[test] + fn duckdb_write_survives_empty_client_assets() { + let content = "ATTACH 'datatable://main' AS pg;\n\ + CREATE TABLE IF NOT EXISTS pg.out_table AS\n\ + SELECT * FROM (SELECT 1 AS placeholder);"; + let got = effective_script_assets(&ScriptLang::DuckDb, content, None).unwrap(); + assert_eq!(got.len(), 1); + assert_eq!(got[0].kind, AssetKind::DataTable); + assert_eq!(got[0].path, "main/out_table"); + assert_eq!(got[0].access_type, Some(AssetUsageAccessType::W)); + } + + #[test] + fn client_alt_access_type_is_preserved_on_match() { + let content = "ATTACH 'datatable://main' AS pg;\n\ + CREATE TABLE IF NOT EXISTS pg.out_table AS SELECT 1;"; + let mut client = asset(AssetKind::DataTable, "main/out_table", None); + client.alt_access_type = Some(AssetUsageAccessType::RW); + let got = + effective_script_assets(&ScriptLang::DuckDb, content, Some(vec![client])).unwrap(); + assert_eq!(got.len(), 1); + // Parser-derived access wins; the user's alt override rides along. + assert_eq!(got[0].access_type, Some(AssetUsageAccessType::W)); + assert_eq!(got[0].alt_access_type, Some(AssetUsageAccessType::RW)); + } + + #[test] + fn client_only_entries_are_kept() { + let content = "ATTACH 'datatable://main' AS pg;\n\ + CREATE TABLE IF NOT EXISTS pg.out_table AS SELECT 1;"; + let extra = asset( + AssetKind::S3Object, + "bucket/file.parquet", + Some(AssetUsageAccessType::R), + ); + let got = effective_script_assets(&ScriptLang::DuckDb, content, Some(vec![extra])).unwrap(); + assert_eq!(got.len(), 2); + assert!(got.iter().any(|a| a.kind == AssetKind::S3Object)); + } + + #[test] + fn unsupported_language_falls_back_to_client() { + let client = vec![asset( + AssetKind::S3Object, + "b/f.json", + Some(AssetUsageAccessType::W), + )]; + let got = effective_script_assets(&ScriptLang::Go, "package main", Some(client.clone())); + assert_eq!(got.map(|v| v.len()), Some(1)); + assert_eq!( + effective_script_assets(&ScriptLang::Go, "package main", None).is_none(), + true + ); + } + + #[test] + fn volume_annotations_parsed_from_leading_comment_block() { + let content = "// volume: my_vol\n// some other comment\nconsole.log(1)\n// volume: ignored_after_code\n"; + let got = effective_script_assets(&ScriptLang::Bun, content, None).unwrap(); + let vols: Vec<_> = got.iter().filter(|a| a.kind == AssetKind::Volume).collect(); + assert_eq!(vols.len(), 1); + assert_eq!(vols[0].path, "my_vol"); + assert_eq!(vols[0].access_type, Some(AssetUsageAccessType::RW)); + } +} diff --git a/backend/windmill-api-scripts/src/lib.rs b/backend/windmill-api-scripts/src/lib.rs index 9bc8318fc4..3ac3764e96 100644 --- a/backend/windmill-api-scripts/src/lib.rs +++ b/backend/windmill-api-scripts/src/lib.rs @@ -6,4 +6,5 @@ * LICENSE-AGPL for a copy of the license. */ +pub mod asset_inference; pub mod scripts; diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index ce375fa64f..9aecd66947 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -13,6 +13,7 @@ use windmill_api_auth::{ ApiAuthed, }; use windmill_common::{ + user_drafts::{overlay_or_draft_only, DraftUserRef, UserDraftItemKind, WithDraftOverlay}, utils::{BulkDeleteRequest, WithStarredInfoQuery, HTTP_CLIENT}, webhook::{WebhookMessage, WebhookShared}, workspaces::{check_deploy_rules, RuleCheckResult}, @@ -33,7 +34,6 @@ use itertools::Itertools; use quick_cache::sync::Cache; use serde::{Deserialize, Serialize}; use serde_json::json; -use serde_json::value::RawValue; use sql_builder::prelude::*; use sqlx::{FromRow, Postgres, Transaction}; use std::{collections::HashMap, sync::Arc}; @@ -44,8 +44,9 @@ use windmill_dep_map::scoped_dependency_map::ScopedDependencyMap; use windmill_common::{ assets::{ - clear_static_asset_usage, clear_static_asset_usage_by_script_hash, - insert_static_asset_usage, AssetUsageKind, AssetWithAltAccessType, + clear_script_triggers, clear_static_asset_usage, clear_static_asset_usage_by_script_hash, + insert_script_trigger, parse_duration_secs, parse_pipeline_annotations, + replace_static_asset_usage, trigger_spec_to_row, AssetUsageKind, TriggerSpec, }, error::{self, to_anyhow}, min_version::{MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2}, @@ -82,122 +83,6 @@ use windmill_queue::{ const MAX_HASH_HISTORY_LENGTH_STORED: usize = 20; -#[derive(Serialize, sqlx::FromRow)] -pub struct ScriptWDraft { - pub hash: ScriptHash, - pub path: String, - pub summary: String, - pub description: String, - pub content: String, - pub language: ScriptLang, - pub kind: ScriptKind, - pub tag: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub draft: Option>>, - /// Timestamp at which the most recent DB draft was created. - #[serde(skip_serializing_if = "Option::is_none")] - pub draft_created_at: Option>, - pub schema: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub draft_only: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub envs: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_ttl: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub cache_ignore_s3_path: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub dedicated_worker: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub ws_error_handler_muted: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub priority: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub restart_unless_cancelled: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub delete_after_use: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub delete_after_secs: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub timeout: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub visible_to_runner_only: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub auto_kind: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub has_preprocessor: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub on_behalf_of_email: Option, - #[serde(skip_serializing_if = "Option::is_none")] - #[sqlx(json(nullable))] - pub assets: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - #[sqlx(json(nullable))] - pub modules: Option>, - #[serde(skip_serializing_if = "Option::is_none")] - pub labels: Option>, - #[serde(flatten)] - #[sqlx(flatten)] - pub runnable_settings: SR, -} - -impl ScriptWDraft { - pub async fn prefetch_cached<'a>( - self, - db: &DB, - ) -> error::Result> { - let (debouncing_settings, concurrency_settings) = - windmill_common::runnable_settings::prefetch_cached_from_handle( - self.runnable_settings.runnable_settings_handle, - db, - ) - .await?; - - Ok(ScriptWDraft { - runnable_settings: ScriptRunnableSettingsInline { - concurrency_settings: concurrency_settings.maybe_fallback( - self.runnable_settings.concurrency_key, - self.runnable_settings.concurrent_limit, - self.runnable_settings.concurrency_time_window_s, - ), - debouncing_settings: debouncing_settings.maybe_fallback( - self.runnable_settings.debounce_key, - self.runnable_settings.debounce_delay_s, - ), - }, - hash: self.hash, - path: self.path, - summary: self.summary, - description: self.description, - content: self.content, - language: self.language, - kind: self.kind, - tag: self.tag, - draft: self.draft, - draft_created_at: self.draft_created_at, - schema: self.schema, - draft_only: self.draft_only, - envs: self.envs, - cache_ttl: self.cache_ttl, - cache_ignore_s3_path: self.cache_ignore_s3_path, - dedicated_worker: self.dedicated_worker, - ws_error_handler_muted: self.ws_error_handler_muted, - priority: self.priority, - restart_unless_cancelled: self.restart_unless_cancelled, - delete_after_use: self.delete_after_use, - delete_after_secs: self.delete_after_secs, - timeout: self.timeout, - visible_to_runner_only: self.visible_to_runner_only, - auto_kind: self.auto_kind, - has_preprocessor: self.has_preprocessor, - on_behalf_of_email: self.on_behalf_of_email, - assets: self.assets, - modules: self.modules, - labels: self.labels, - }) - } -} - pub fn global_service() -> Router { Router::new() .route("/hub/top", get(get_top_hub_scripts)) @@ -222,7 +107,6 @@ pub fn workspaced_service() -> Router { .route("/create", post(create_script)) .route("/create_snapshot", post(create_snapshot_script)) .route("/archive/p/{*path}", post(archive_script_by_path)) - .route("/get/draft/{*path}", get(get_script_by_path_w_draft)) .route("/get/p/{*path}", get(get_script_by_path)) .route("/list_tokens/{*path}", get(list_tokens)) .route("/raw/p/{*path}", get(raw_script_by_path)) @@ -295,6 +179,7 @@ async fn list_search_scripts( async fn list_scripts( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path(w_id): Path, Query(pagination): Query, Query(lq): Query, @@ -305,7 +190,7 @@ async fn list_scripts( "hash", "o.path", "summary", - "COALESCE(draft.created_at, o.created_at) as created_at", + "o.created_at as created_at", "archived", "extra_perms", if !lq.without_description.unwrap_or(false) { @@ -317,13 +202,22 @@ async fn list_scripts( "language", "favorite.path IS NOT NULL as starred", "tag", - "draft.path IS NOT NULL as has_draft", - "draft_only", "ws_error_handler_muted", "auto_kind", "codebase IS NOT NULL as use_codebase", "kind", "o.labels", + "draft.email IS NOT NULL as is_draft", + // Canonical reference for the draft-feature comments; flows/apps point here. + // Per-path draft owners as a JSON array (`Json>`); NULL -> None, + // never an empty array. LEFT JOIN `usr` keeps orphaned drafts (user left workspace) + // visible with `username = None`. In the `admins` workspace username IS the email, + // so fall back to `d.email` there or the authed user's own draft resolves to a phantom + // "Legacy workspace draft"; the genuine NULL-email legacy row stays None. + "(SELECT json_agg(json_build_object('username', COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END)) ORDER BY COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) NULLS LAST) \ + FROM draft d \ + LEFT JOIN usr u ON u.workspace_id = d.workspace_id AND u.email = d.email \ + WHERE d.workspace_id = o.workspace_id AND d.path = o.path AND d.typ = 'script') as draft_users", "folder_labels(o.workspace_id, o.path) as inherited_labels" ]) .left() @@ -335,7 +229,8 @@ async fn list_scripts( .left() .join("draft") .on( - "draft.path = o.path AND draft.workspace_id = o.workspace_id AND draft.typ = 'script'" + "draft.path = o.path AND draft.workspace_id = o.workspace_id AND draft.typ = 'script' AND draft.email = ?" + .bind(&authed.email), ) .order_desc("favorite.path IS NOT NULL") .order_by("created_at", lq.order_desc.unwrap_or(true)) @@ -361,10 +256,6 @@ async fn list_scripts( sqlb.and_where("(o.auto_kind IS NULL OR o.auto_kind <> 'lib')"); } - if !lq.include_draft_only.unwrap_or(false) || authed.is_operator { - sqlb.and_where("draft_only IS NOT TRUE"); - } - if lq.show_archived.unwrap_or(false) { sqlb.and_where_eq( "o.ctid", @@ -434,7 +325,7 @@ async fn list_scripts( .fields(&["dm.deployment_msg"]); } - if let Some(languages) = lq.languages { + if let Some(languages) = &lq.languages { sqlb.and_where_in( "language", &languages @@ -447,13 +338,114 @@ async fn list_scripts( let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; let mut tx = user_db.begin(&authed).await?; let allowed = build_scope_path_predicate(&authed, "scripts", "read"); - let rows = sqlx::query_as::<_, ListableScript>(&sql) + let mut rows = sqlx::query_as::<_, ListableScript>(&sql) .fetch_all(&mut *tx) .await? .into_iter() .filter(|r| allowed(&r.path)) .collect::>(); tx.commit().await?; + + // Canonical reference for draft-only synthesis; the other kinds point here. + // Append the authed user's drafts at paths with no deployed script. Gated on + // `include_draft_only` so picker callers stay deployed-only (home page opts in); + // skipped past page 0 or under any narrowing filter to keep pagination clean. + if lq.include_draft_only.unwrap_or(false) + && !authed.is_operator + && offset == 0 + && lq.path_start.is_none() + && lq.path_exact.is_none() + && lq.created_by.is_none() + && lq.first_parent_hash.is_none() + && lq.last_parent_hash.is_none() + && lq.parent_hash.is_none() + && lq.is_template.is_none() + && lq.dedicated_worker.is_none() + && lq.label.is_none() + && lq.languages.is_none() + && !lq.starred_only.unwrap_or(false) + && !lq.show_archived.unwrap_or(false) + { + // `(email = $2 OR email IS NULL)` surfaces the user's own draft-only rows plus + // legacy NULL-email workspace rows; `DISTINCT ON (path)` ordered `email IS NULL` + // last collapses a path holding both to the owned row. + let draft_only_rows = sqlx::query!( + r#"SELECT DISTINCT ON (path) + path, + value as "value!: sqlx::types::Json>", + created_at + FROM draft + WHERE workspace_id = $1 + AND typ = 'script' + AND (email = $2 OR email IS NULL) + AND NOT EXISTS ( + SELECT 1 FROM script s + WHERE s.workspace_id = draft.workspace_id + AND s.path = draft.path + ) + ORDER BY path, (email IS NULL)"#, + &w_id, + &authed.email, + ) + .fetch_all(&db) + .await?; + + for row in draft_only_rows { + let v: serde_json::Value = + serde_json::from_str(row.value.0.get()).unwrap_or(serde_json::Value::Null); + let language: ScriptLang = v + .get("language") + .and_then(|x| serde_json::from_value(x.clone()).ok()) + .unwrap_or_default(); + let kind: ScriptKind = v + .get("kind") + .and_then(|x| serde_json::from_value(x.clone()).ok()) + .unwrap_or(ScriptKind::Script); + // Scripts bind the Path widget to `script.path`, so the typed path + // round-trips through the draft JSON's own `path` (no `draft_path` field). + let draft_path = v + .get("path") + .and_then(|s| s.as_str()) + .filter(|s| !s.is_empty() && *s != row.path.as_str()) + .map(|s| s.to_string()); + rows.push(ListableScript { + hash: ScriptHash(0), + path: row.path, + summary: v + .get("summary") + .and_then(|s| s.as_str()) + .unwrap_or("") + .to_string(), + created_at: row.created_at, + archived: false, + extra_perms: serde_json::Value::Object(serde_json::Map::new()), + language, + starred: false, + tag: v.get("tag").and_then(|s| s.as_str()).map(|s| s.to_string()), + description: v + .get("description") + .and_then(|s| s.as_str()) + .map(|s| s.to_string()), + draft_only: Some(true), + has_deploy_errors: false, + ws_error_handler_muted: None, + auto_kind: None, + use_codebase: false, + deployment_msg: None, + kind, + labels: None, + // Synthesized rows have no deployed row to inherit folder labels from. + inherited_labels: None, + is_draft: true, + draft_path, + // Synthesized rows are the authed user's own draft (single-user case). + draft_users: Some(sqlx::types::Json(vec![DraftUserRef { + username: Some(authed.username.clone()), + }])), + }); + } + } + Ok(Json(rows)) } @@ -721,7 +713,6 @@ async fn is_noop_deploy_against_parent( language, kind, tag, - draft_only, envs, concurrency_settings, debouncing_settings, @@ -798,7 +789,6 @@ async fn is_noop_deploy_against_parent( || visible_to_runner_only != &parent.visible_to_runner_only || has_preprocessor != &parent.has_preprocessor || is_template.unwrap_or(false) != parent.is_template.unwrap_or(false) - || draft_only.unwrap_or(false) != parent.draft_only.unwrap_or(false) { return Ok(false); } @@ -1026,20 +1016,10 @@ async fn create_script_internal<'c>( let parent_hashes_and_perms: Option = match (&ns.parent_hash, clashing_script) { (None, None) => Ok(None), - (None, Some(s)) if !s.draft_only.unwrap_or(false) => Err(Error::BadRequest(format!( + (None, Some(s)) => Err(Error::BadRequest(format!( "Path conflict for {} with non-archived hash {}", &ns.path, &s.hash ))), - (None, Some(s)) => { - sqlx::query!( - "DELETE FROM script WHERE hash = $1 AND workspace_id = $2", - s.hash.0, - &w_id - ) - .execute(&mut *tx) - .await?; - Ok(None) - } (Some(p_hash), o) => { // Lock the parent row to prevent concurrent updates with the same parent_hash // This ensures linear lineage - only one script can have a given parent at a time @@ -1256,7 +1236,121 @@ async fn create_script_internal<'c>( let ci_test_refs = windmill_common::schema::parse_ci_test_annotation(&ns.content, &lang.as_comment_lit()); - let auto_kind = if ci_test_refs.is_some() { + // `pipeline` wins over `test` and any client-supplied auto_kind. The + // bare `// pipeline` marker is the opt-in signal for pipeline + // membership; parsed writes tell us what is produced (we don't record + // them in auto_kind itself). + let pipeline_annotations = parse_pipeline_annotations(&ns.content); + // `// freshness` is parsed but enforcement is a not-yet-implemented + // enterprise feature (skeleton in windmill_common::pipeline_advanced). + // Surface a clear TODO at deploy rather than silently accepting an + // annotation that does nothing. + if pipeline_annotations.freshness.is_some() { + tracing::warn!( + "{}", + windmill_common::pipeline_advanced::freshness_enforcement_todo() + ); + } + // `// materialize` materializes a `ducklake:///
` target from a + // DuckDB script. These two constraints hold for *both* modes: a non-DuckLake + // target would otherwise deploy, register a producer in the asset graph, then + // silently no-op at run time (`build_materialized_query` returns `Ok(None)`), + // and a non-DuckDB script never reaches the executor that records state. The + // managed-only checks (single trailing SELECT, no SQL args) come after — a + // `manual` script owns its DDL and skips them. + if let Some(m) = pipeline_annotations.materialize.as_ref() { + if ns.language != ScriptLang::DuckDb { + return Err(Error::BadRequest(format!( + "`// materialize` is only supported for DuckDB scripts, not {}. Use the \ + wmll.ducklake helpers to materialize from other languages.", + ns.language.as_str() + ))); + } + if m.target_kind != windmill_parser::asset_parser::AssetKind::Ducklake { + return Err(Error::BadRequest( + "`// materialize` only supports a DuckLake target \ + (`ducklake:///
`); other asset kinds aren't materializable." + .to_string(), + )); + } + if !m.target_path.contains('/') { + return Err(Error::BadRequest(format!( + "`// materialize` needs a table in the target: \ + `ducklake://{0}/
` (got `ducklake://{0}`).", + m.target_path + ))); + } + if !m.manual { + if let Err(e) = windmill_parser::sql_materialize::classify_wrap(&ns.content) { + return Err(Error::BadRequest(e.message())); + } + // SQL args are supported: managed materialize strips line comments + // (including `-- $name (type)` declarations) when it wraps the SELECT, + // but the executor parses the signature from the un-wrapped script, so + // `$name` references in the SELECT stay bound at run time. + } + // `key=` (merge) and `append` are mutually exclusive reconciliation + // strategies; append (INSERT-only) wins. Surface the conflict rather + // than silently dropping the dedup the author may have intended. + if m.unique_key.is_some() && m.append { + tracing::warn!( + "script {}: both `key=` and `append` set on // materialize; append wins (INSERT-only, no dedup)", + ns.path + ); + } + } + let in_pipeline = pipeline_annotations.in_pipeline; + // `// trigger all` → AND join barrier (else OR, the default). + let pipeline_join_all = !pipeline_annotations.join_mode.is_any(); + // Script-level `// debounce ` default; a per-`// on debounce=` + // overrides it (precedence resolved per edge below). + let pipeline_debounce_default = pipeline_annotations.debounce_default; + let pipeline_triggers = pipeline_annotations.triggers; + // `// tag ` overrides the caller-supplied tag at deploy. Source + // wins, matching the wipe-and-reinsert convention of other pipeline + // annotations. Applied before the dep-job tag selection below (which + // special-cases dedicated_worker / bunnative / `$args[`) so that path + // sees the annotation-overridden value. + if let Some(t) = pipeline_annotations.tag.clone() { + ns.tag = Some(t); + } + // `// retry []` is persisted to `script_trigger` below (asset + // edges only) and drives native subscriber retry in the cascade: a failed + // subscriber re-runs as a `Script` job (not a flow step), so it stays + // eligible for asset dispatch and can trigger its own downstream on success. + // Asset presence is server-authoritative: re-parse the deployed content + // (same parsers the frontend wasm wraps) and union with the client list. + // The `asset` rows written below drive the asset-trigger cascade, so a + // client deploying `assets: null` (e.g. broken wasm inference) must not + // silently kill the producer side while `// on` subscribers stay wired. + let effective_assets = crate::asset_inference::effective_script_assets( + &ns.language, + &ns.content, + ns.assets.take(), + ); + // Register the `// materialize` target as a write asset so the deployed + // asset graph shows this script as the producer of the managed table — the + // body's `SELECT` doesn't express the write (the runtime generates it), so + // server-side inference wouldn't otherwise link it. + let effective_assets = if let Some(m) = pipeline_annotations.materialize.as_ref() { + let kind = windmill_common::assets::asset_kind_from_parser(m.target_kind); + let mut a = effective_assets.unwrap_or_default(); + if !a.iter().any(|x| x.kind == kind && x.path == m.target_path) { + a.push(windmill_common::assets::AssetWithAltAccessType { + path: m.target_path.clone(), + kind, + access_type: Some(windmill_common::assets::AssetUsageAccessType::W), + alt_access_type: None, + columns: None, + }); + } + Some(a) + } else { + effective_assets + }; + let auto_kind = if in_pipeline { + Some("pipeline".to_string()) + } else if ci_test_refs.is_some() { Some("test".to_string()) } else { auto_kind @@ -1266,6 +1360,7 @@ async fn create_script_internal<'c>( RunnableSettings { debouncing_settings: ns.debouncing_settings.insert_cached(&db).await?, concurrency_settings: ns.concurrency_settings.insert_cached(&db).await?, + retry_settings: None, }, &db, ) @@ -1292,10 +1387,10 @@ async fn create_script_internal<'c>( sqlx::query!( "INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, \ content, created_by, schema, is_template, extra_perms, lock, language, kind, tag, \ - draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, \ + envs, concurrent_limit, concurrency_time_window_s, cache_ttl, \ dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \ delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, cache_ignore_s3_path, runnable_settings_handle, modules, labels) \ - VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40, $41)", + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9::text::json, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, $39, $40)", &w_id, &hash.0, ns.path, @@ -1311,7 +1406,6 @@ async fn create_script_internal<'c>( lang as ScriptLang, ns.kind.unwrap_or(ScriptKind::Script) as ScriptKind, ns.tag, - ns.draft_only, envs, guarded_concurrent_limit, guarded_concurrency_time_window_s, @@ -1334,7 +1428,9 @@ async fn create_script_internal<'c>( &authed, ), validate_schema, - ns.assets.as_ref().and_then(|a| serde_json::to_value(a).ok()), + effective_assets + .as_ref() + .and_then(|a| serde_json::to_value(a).ok()), guarded_debounce_key, guarded_debounce_delay_s, ns.cache_ignore_s3_path, @@ -1376,10 +1472,15 @@ async fn create_script_internal<'c>( let p_path_opt = parent_hashes_and_perms.as_ref().map(|x| x.p_path.clone()); if let Some(ref p_path) = p_path_opt { if !skip_draft_deletion { + // Canonical: on deploy only wipe the deployer's own draft (plus the legacy + // NULL-email row). Teammates' drafts are independent — they stay and fire the + // StaleDraftModal on the teammate's next reload rather than vanishing silently. sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script' \ + AND (email = $3 OR email IS NULL)", p_path, - &w_id + &w_id, + &authed.email, ) .execute(&mut *tx) .await?; @@ -1463,10 +1564,14 @@ async fn create_script_internal<'c>( } } } else if !skip_draft_deletion { + // See the matching branch above — only wipe the deployer's own + // draft (plus the legacy NULL-email row). sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script' \ + AND (email = $3 OR email IS NULL)", ns.path, - &w_id + &w_id, + &authed.email, ) .execute(&mut *tx) .await?; @@ -1563,11 +1668,86 @@ async fn create_script_internal<'c>( ); } - clear_static_asset_usage(&mut *tx, &w_id, &script_path, AssetUsageKind::Script).await?; - for asset in ns.assets.as_ref().into_iter().flatten() { - insert_static_asset_usage(&mut *tx, &w_id, &asset, &ns.path, AssetUsageKind::Script) - .await?; + // Clear + reinsert this script's producer rows at script_path (== ns.path), + // invalidating the producer-writes cache once iff the write-producer set + // changed (see replace_static_asset_usage). + replace_static_asset_usage( + &mut tx, + &w_id, + &script_path, + effective_assets.as_deref().unwrap_or(&[]), + ) + .await?; + + // Pipeline trigger edges: wipe-and-reinsert per deploy so removing an + // `// on ...` annotation drops the edge. Only Asset / Schedule produce + // a row — native trigger marker annotations (`// on kafka`, etc.) are + // discovered by the graph endpoint directly from the per-kind trigger + // tables, so `trigger_spec_to_row` returns None for those. + clear_script_triggers(&mut *tx, &w_id, &ns.path, AssetUsageKind::Script).await?; + // On rename, also drop the OLD path's trigger rows. clear is keyed by + // path (no by-hash variant), and only `ns.path` is wiped above — without + // this, stale `// on` edges for the old path keep matching producers and + // would trigger a script later recreated at that path even if it has no + // annotation (P1). (Producer/asset rows for the old path are already + // cleared via clear_static_asset_usage_by_script_hash on the parent.) + if let Some(ref old) = p_path_opt { + if old != &ns.path { + clear_script_triggers(&mut *tx, &w_id, old, AssetUsageKind::Script).await?; + } } + for spec in &pipeline_triggers { + let Some((trigger_kind, trigger_ref)) = trigger_spec_to_row(spec) else { + continue; + }; + // Effective debounce for this edge: per-`// on debounce=` wins, + // else the script-level `// debounce` default. Debounce only + // applies to asset-cascade edges; other trigger kinds get none. + let debounce_s = match spec { + TriggerSpec::Asset { debounce: Some(d), .. } => parse_duration_secs(d), + TriggerSpec::Asset { .. } => pipeline_debounce_default + .as_deref() + .and_then(parse_duration_secs), + _ => None, + }; + // `// retry` applies only to the asset cascade: a failed subscriber is + // re-run natively (a `Script` job, not a flow step), so it can still + // trigger its own downstream on success — see asset_dispatch. + let (retry_count, retry_delay_s) = match spec { + TriggerSpec::Asset { .. } => ( + pipeline_annotations + .retry + .as_ref() + // `// retry ` count is u32; saturate the narrowing to i16. + .map(|r| r.count.min(i16::MAX as u32) as i16), + pipeline_annotations + .retry + .as_ref() + .and_then(|r| r.delay.as_deref()) + .and_then(parse_duration_secs), + ), + _ => (None, None), + }; + insert_script_trigger( + &mut *tx, + &w_id, + AssetUsageKind::Script, + &ns.path, + trigger_kind, + &trigger_ref, + pipeline_join_all, + debounce_s, + retry_count, + retry_delay_s, + ) + .await?; + } + + // Schedule annotations (`// on schedule`) are marker-only — the binding + // lives on the schedule row's own `script_path` field, which the user + // creates separately via the schedule editor. No script-create-time + // reconciliation is needed (and there are no "managed" schedules to + // upsert/delete anymore). let permissioned_as = username_to_permissioned_as(&authed.username); if let Some(parent_hash) = ns.parent_hash { @@ -1773,14 +1953,25 @@ pub async fn pick_hub_script_by_path( Ok::<_, Error>((status_code, headers, response)) } +// Canonical: fields inlined rather than `#[serde(flatten)]` from +// `WithStarredInfoQuery` / `WithDraftQuery`. axum's `serde_urlencoded` extractor +// drops type info through flatten, so `?get_draft=true` arrives as a String and +// fails the inner bool deserializer. Inlining lets the bool adapter see it directly. +#[derive(Deserialize)] +struct GetScriptByPathQuery { + with_starred_info: Option, + #[serde(default)] + get_draft: bool, +} + #[axum::debug_handler] async fn get_script_by_path( authed: ApiAuthed, Extension(user_db): Extension, Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, - Query(query): Query, -) -> JsonResult> { + Query(query): Query, +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || format!("scripts:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; @@ -1800,7 +1991,7 @@ async fn get_script_by_path( ORDER BY s.created_at DESC LIMIT 1", ) .bind(path) - .bind(w_id) + .bind(&w_id) .bind(&authed.username) .fetch_optional(&mut *tx) .await? @@ -1812,53 +2003,43 @@ async fn get_script_by_path( ), ) .bind(path) - .bind(w_id) + .bind(&w_id) .fetch_optional(&mut *tx) .await? }; tx.commit().await?; - let script = windmill_common::scripts::prefetch_cached_script_with_starred( - not_found_if_none(script_o, "Script", path)?, + // Canonical: with no deployed row and `get_draft` set, fall back to the draft + // table so editing a never-deployed draft works like a deployed reload. + let deployed = match script_o { + Some(script_o) => Some( + windmill_common::scripts::prefetch_cached_script_with_starred(script_o, &db).await?, + ), + None => None, + }; + let overlay = overlay_or_draft_only( &db, + &w_id, + &authed.email, + UserDraftItemKind::Script, + path, + query.get_draft, + deployed, + || windmill_common::error::Error::NotFound(format!("Script not found at path {path}")), ) .await?; - Ok(Json(script)) + Ok(Json(overlay)) } async fn list_tokens( + authed: ApiAuthed, Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, ) -> JsonResult> { - let path = path.to_path(); - list_tokens_internal(&db, &w_id, &path, false).await -} - -async fn get_script_by_path_w_draft( - authed: ApiAuthed, - Extension(db): Extension, - Extension(user_db): Extension, - Path((w_id, path)): Path<(String, StripPath)>, -) -> JsonResult> { let path = path.to_path(); check_scopes(&authed, || format!("scripts:read:{}", path))?; - let mut tx = user_db.begin(&authed).await?; - - let script_o = sqlx::query_as::<_, ScriptWDraft>( - "SELECT hash, script.path, summary, description, content, language, kind, tag, schema, draft_only, envs, runnable_settings_handle, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, ws_error_handler_muted, draft.value as draft, draft.created_at as draft_created_at, dedicated_worker, priority, restart_unless_cancelled, delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, has_preprocessor, on_behalf_of_email, assets, modules, debounce_key, debounce_delay_s, labels FROM script LEFT JOIN draft ON - script.path = draft.path AND script.workspace_id = draft.workspace_id AND draft.typ = 'script' - WHERE script.path = $1 AND script.workspace_id = $2 - ORDER BY script.created_at DESC LIMIT 1", - ) - .bind(path) - .bind(w_id) - .fetch_optional(&mut *tx) - .await?; - tx.commit().await?; - - let script = not_found_if_none(script_o, "Script", path)?; - Ok(Json(script.prefetch_cached(&db).await?)) + list_tokens_internal(&db, &w_id, &path, false).await } async fn get_script_history( @@ -2521,18 +2702,6 @@ async fn get_deployment_status( Ok(Json(deployment_status)) } -pub async fn require_is_writer(authed: &ApiAuthed, path: &str, w_id: &str, db: DB) -> Result<()> { - return windmill_api_auth::require_is_writer( - authed, - path, - w_id, - db, - "SELECT extra_perms FROM script WHERE path = $1 AND workspace_id = $2 ORDER BY created_at DESC LIMIT 1", - "script", - ) - .await; -} - async fn archive_script_by_path( authed: ApiAuthed, Extension(webhook): Extension, @@ -2572,6 +2741,10 @@ async fn archive_script_by_path( .map_err(|e| Error::internal_err(format!("archiving script in {w_id}: {e:#}")))?; clear_static_asset_usage(&mut *tx, &w_id, path, AssetUsageKind::Script).await?; + // Pipeline event hygiene: an archived script must not be triggered by + // anything. Wipe declared `// on ...` edges (asset-event subscribers + // look these up). + clear_script_triggers(&mut *tx, &w_id, path, AssetUsageKind::Script).await?; audit_log( &mut *tx, @@ -2649,6 +2822,9 @@ async fn archive_script_by_hash( check_scopes(&authed, || format!("scripts:write:{}", &script.path))?; clear_static_asset_usage_by_script_hash(&mut *tx, &w_id, hash).await?; + // Pipeline event hygiene: archived scripts must not be triggered by + // anything. Wipe declared `// on ...` edges. + clear_script_triggers(&mut *tx, &w_id, &script.path, AssetUsageKind::Script).await?; audit_log( &mut *tx, @@ -2709,6 +2885,10 @@ async fn delete_script_by_hash( check_scopes(&authed, || format!("scripts:write:{}", &script.path))?; clear_static_asset_usage_by_script_hash(&mut *tx, &w_id, hash).await?; + // Pipeline event hygiene: a deleted script must not be triggered by + // anything. Wipe declared `// on ...` edges. Idempotent — safe even if + // the script was never a pipeline member. + clear_script_triggers(&mut *tx, &w_id, &script.path, AssetUsageKind::Script).await?; audit_log( &mut *tx, @@ -2769,18 +2949,7 @@ async fn delete_script_by_path( let mut tx = user_db.begin(&authed).await?; - let draft_only = sqlx::query_scalar!( - "SELECT draft_only FROM script WHERE path = $1 AND workspace_id = $2", - path, - w_id - ) - .fetch_one(&db) - .await? - .unwrap_or(false); - - if !draft_only { - require_admin(authed.is_admin, &authed.username)?; - } + require_admin(authed.is_admin, &authed.username)?; // Capture all script versions and drafts for trashbin before deleting let trash_scripts: Vec = sqlx::query_scalar( @@ -2832,6 +3001,11 @@ async fn delete_script_by_path( .execute(&mut *tx) .await?; + // Pipeline event hygiene: a deleted script must not be triggered by + // anything. Wipe declared `// on ...` edges. Idempotent — safe even if + // the script was never a pipeline member. + clear_script_triggers(&mut *tx, &w_id, path, AssetUsageKind::Script).await?; + if !query.keep_captures.unwrap_or(false) { sqlx::query!( "DELETE FROM capture_config WHERE path = $1 AND workspace_id = $2 AND is_flow IS FALSE", diff --git a/backend/windmill-api-settings/src/audit_logs_s3.rs b/backend/windmill-api-settings/src/audit_logs_s3.rs index 9fa4574a1e..d7d749cb58 100644 --- a/backend/windmill-api-settings/src/audit_logs_s3.rs +++ b/backend/windmill-api-settings/src/audit_logs_s3.rs @@ -20,9 +20,11 @@ use windmill_common::DB; pub struct AuditLogsS3ExportStatus { /// xid cursor: rows of transactions below this have been exported. pub last_xmin: i64, - /// Partition-pruning floor (the epoch sentinel while still bootstrapping). + /// Partition-pruning floor: the latest audit-row timestamp the cursor has + /// reached (also the read side's 7-day-fallback anchor). pub last_ts: Option>, - /// True until the initial post-enable backlog has been fully drained. + /// True while the exporter is draining a backlog — the last run was capped + /// at `MAX_XID_INTERVAL` xids and has not yet caught up to the live snapshot. pub bootstrapping: bool, /// The latest audit-row timestamp actually written to object storage so /// far (monotonic) — the "how current is the mirror" figure. @@ -50,8 +52,7 @@ pub async fn get_status(db: &DB) -> error::Result::from_timestamp(0, 0).unwrap(); - let bootstrapping = last_ts.map(|t| t <= epoch).unwrap_or(true); + let bootstrapping = v.get("draining").and_then(|x| x.as_bool()).unwrap_or(false); Ok(Some(AuditLogsS3ExportStatus { last_xmin: v.get("last_xmin").and_then(|x| x.as_i64()).unwrap_or(0), last_ts, diff --git a/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs new file mode 100644 index 0000000000..0c5c55f7ed --- /dev/null +++ b/backend/windmill-api-settings/src/audit_logs_s3_backfill.rs @@ -0,0 +1,708 @@ +#![cfg(feature = "parquet")] +//! Opt-in historical backfill of audit logs to the instance object store. +//! +//! The steady-state exporter (the EE `export_audit_logs_to_object_store`) cursors +//! on transaction xmin and, by design, only exports rows committed *after* the +//! feature was enabled — it never rescans history (an `age(xmin)` predicate is +//! unindexable, so scanning the whole partitioned table can't survive a +//! `statement_timeout`). This module covers the complementary need: exporting a +//! chosen historical `[from, to)` window (e.g. the gap left while the export was +//! disabled) on demand. +//! +//! It is safe to run on a large table because it scans strictly by `timestamp` +//! (the partition key — pruned and indexed) in bounded keyset pages, so every +//! query touches at most one page worth of rows and survives a statement timeout. +//! It does not touch the xmin cursor / checkpoint at all. +//! +//! Objects are written next to the steady-state ones under `logs/audit/dt=/` +//! as `audit_backfill__.ndjson`, with the exact same row shape, so +//! a consumer reads them uniformly. The key includes the requested window so two +//! different backfill ranges never overwrite each other (a per-page `min_id` alone +//! is not unique across windows). Re-running the *same* window is deterministic +//! (audit history is append-only), so it overwrites the same objects rather than +//! duplicating. A window that overlaps already-exported steady-state rows simply +//! re-emits them under a different key; consumers dedupe by `id`. +//! +//! Scope: like the steady-state export, this reads only `audit_partitioned`. The +//! pre-partitioning `audit` table is intentionally not exported; a window that +//! overlaps any legacy `audit` row is rejected (see [`try_start`]) so a backfill +//! never silently reports success while omitting them. +//! +//! Progress is persisted in `background_task_state` (name [`TASK_NAME`]) so any +//! API replica can serve the status endpoint, mirroring `log_cleanup`. + +use std::sync::Arc; + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use tokio::sync::RwLock; + +use crate::background_task; +use windmill_common::error::{self}; +use windmill_common::tracing_init::LOGS_AUDIT; +use windmill_common::{DB, INSTANCE_NAME}; + +use windmill_object_store::object_store_reexports::{ObjectStore, Path as ObjectPath}; + +pub const TASK_NAME: &str = "audit_logs_s3_backfill"; + +/// Rows fetched per keyset page. Bounds each query so it stays well under any +/// `statement_timeout` even on a busy partition, and bounds peak memory (one +/// page of ndjson is buffered before the day-grouped PUTs). +const PAGE_ROWS: i64 = 10_000; + +/// Test-only override for [`PAGE_ROWS`] (0 = use the default), so a test can force +/// multi-page / page-spanning-day keyset behaviour with only a handful of rows. +#[cfg(test)] +static PAGE_ROWS_OVERRIDE: std::sync::atomic::AtomicI64 = std::sync::atomic::AtomicI64::new(0); + +fn page_rows() -> i64 { + #[cfg(test)] + { + match PAGE_ROWS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed) { + 0 => PAGE_ROWS, + n => n, + } + } + #[cfg(not(test))] + { + PAGE_ROWS + } +} + +#[derive(Clone, Serialize, Deserialize)] +pub struct AuditBackfillProgress { + pub running: bool, + pub started_at: DateTime, + pub finished_at: Option>, + /// Human-readable description of the current phase. + pub phase: String, + /// Inclusive lower / exclusive upper bound of the window being exported. + pub from: DateTime, + pub to: DateTime, + /// Audit rows written to object storage so far. + pub rows_written: u64, + /// Object PUTs issued so far (one per day per page). + pub objects_written: u64, + /// Keyset cursor: the timestamp of the last row exported (how far the + /// backfill has progressed through the window). + pub last_ts: Option>, + pub errors: u64, + pub last_error: Option, +} + +impl AuditBackfillProgress { + fn new_running(from: DateTime, to: DateTime) -> Self { + Self { + running: true, + started_at: Utc::now(), + finished_at: None, + phase: "starting".to_string(), + from, + to, + rows_written: 0, + objects_written: 0, + last_ts: None, + errors: 0, + last_error: None, + } + } +} + +struct Session { + db: DB, + owner: String, + progress: RwLock, +} + +impl Session { + async fn update(&self, f: F) { + let snapshot = { + let mut p = self.progress.write().await; + f(&mut p); + p.clone() + }; + if let Err(e) = + background_task::update_state(&self.db, TASK_NAME, &self.owner, &snapshot).await + { + tracing::warn!("audit backfill: failed to persist progress: {e:#}"); + } + } + + async fn record_error(&self, msg: String) { + tracing::error!("audit backfill: {msg}"); + self.update(|p| { + p.errors = p.errors.saturating_add(1); + p.last_error = Some(msg); + }) + .await; + } + + async fn release(&self) { + let snapshot = { + let mut p = self.progress.write().await; + p.running = false; + p.finished_at = Some(Utc::now()); + p.phase = "done".to_string(); + p.clone() + }; + tracing::info!( + "audit backfill finished: {} row(s) in {} object(s) for [{}, {}), {} error(s)", + snapshot.rows_written, + snapshot.objects_written, + snapshot.from, + snapshot.to, + snapshot.errors + ); + if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await + { + tracing::warn!("audit backfill: failed to release lease: {e:#}"); + } + } +} + +#[derive(Deserialize)] +pub struct BackfillRequest { + pub from: DateTime, + pub to: DateTime, +} + +/// Atomically claim the backfill lease, or error if one is already running. +pub async fn try_start(db: &DB, from: DateTime, to: DateTime) -> error::Result<()> { + if from >= to { + return Err(error::Error::BadRequest( + "audit backfill: `from` must be strictly before `to`".to_string(), + )); + } + // The backfill keyset-pages by `(timestamp, id)` over rows visible at scan time and + // declares the window fully exported once the scan runs dry. But a row's `timestamp` + // is its inserting transaction's `xact_start`, so a transaction that started inside + // `[from, to)` yet commits after the scan has passed that timestamp — or any row + // committed when `to` is in the future — would be silently omitted. Require `to` to + // be at or before the oldest in-flight `xact_start`: everything strictly older than + // the oldest running transaction is already committed and stable. + // + // That bound is only sound when we can see every xmin-holding transaction. A role + // without pg_read_all_stats/superuser sees only its own sessions, and a prepared + // (2PC) transaction is invisible to pg_stat_activity — in either case an old + // transaction could still commit rows inside an accepted window after our scan ends. + // Since a backfill asserts completeness, we REJECT in those cases (the function + // returns NULL). (The continuous exporter, which only claims bounded lag, keeps the + // 7-day fallback instead.) The probe lives in the `audit_logs_s3_oldest_inflight_ts()` + // SQL function (migration 20260626132251) so its `pg_has_role`/`pg_authid` read is + // wrapped in a subtransaction EXCEPTION: managed providers (e.g. Cloud SQL) forbid + // reading pg_authid from an elevated context, which would otherwise surface here as + // an opaque error instead of NULL → the actionable rejection below. + let settled_cutoff: Option> = + sqlx::query_scalar!(r#"SELECT audit_logs_s3_oldest_inflight_ts() AS "cutoff?""#) + .fetch_one(db) + .await?; + let Some(settled_cutoff) = settled_cutoff else { + return Err(error::Error::BadRequest( + "audit backfill: cannot determine a trustworthy settled-time boundary, so completeness \ + can't be guaranteed. The windmill DB role needs pg_read_all_stats (or superuser) and \ + there must be no prepared (2PC) transactions in progress — otherwise an old or \ + invisible transaction could later commit audit rows inside the requested window and \ + the backfill would miss them. Grant the privilege / resolve prepared transactions and \ + retry." + .to_string(), + )); + }; + if to > settled_cutoff { + return Err(error::Error::BadRequest(format!( + "audit backfill: `to` ({to}) must be at or before {settled_cutoff}, the newest point \ + guaranteed settled (the oldest in-flight transaction's start); choose an earlier \ + upper bound." + ))); + } + // The backfill (like the steady-state export) reads only `audit_partitioned`. Audit + // history from before partitioning was introduced lives in the legacy `audit` table + // and is intentionally not exported. If the requested window overlaps any legacy row, + // reject — otherwise a "completed" backfill would silently omit them. Checking the + // legacy table directly (rather than min(audit_partitioned)) also covers an upgraded + // instance whose `audit_partitioned` is still empty, where a min() guard would no-op. + // Non-macro query: no compile-time-checked entry needed. + let overlaps_legacy: bool = sqlx::query_scalar::<_, bool>( + "SELECT EXISTS (SELECT 1 FROM audit WHERE timestamp >= $1 AND timestamp < $2)", + ) + .bind(from) + .bind(to) + .fetch_one(db) + .await?; + if overlaps_legacy { + return Err(error::Error::BadRequest( + "audit backfill: the requested window overlaps rows in the legacy (pre-partitioning) \ + `audit` table, which is not exported to object storage. Restrict the window to the \ + partitioned era (after audit-log partitioning was introduced)." + .to_string(), + )); + } + let claimed = background_task::try_claim( + db, + TASK_NAME, + &*INSTANCE_NAME, + &AuditBackfillProgress::new_running(from, to), + ) + .await?; + if !claimed { + return Err(error::Error::BadRequest( + "An audit log backfill is already running".to_string(), + )); + } + Ok(()) +} + +/// Fetch the current backfill status. Any API server can call this. +pub async fn get_status(db: &DB) -> error::Result> { + let Some(r) = background_task::get(db, TASK_NAME).await? else { + return Ok(None); + }; + match serde_json::from_value::(r.value) { + Ok(mut p) => { + // get() collapses `running` to false when the heartbeat is stale. + p.running = r.running; + Ok(Some(p)) + } + Err(e) => Err(error::Error::internal_err(format!( + "deserialize audit backfill progress: {e:#}" + ))), + } +} + +pub fn spawn_backfill(db: DB, from: DateTime, to: DateTime) { + use futures::FutureExt; + use std::panic::AssertUnwindSafe; + + tokio::spawn(async move { + let session = Arc::new(Session { + db: db.clone(), + owner: INSTANCE_NAME.clone(), + progress: RwLock::new(AuditBackfillProgress::new_running(from, to)), + }); + + let s = session.clone(); + let task = async move { + let store = match windmill_object_store::get_object_store().await { + Some(st) => st, + None => { + s.record_error("Object storage is not configured".to_string()) + .await; + return; + } + }; + if let Err(e) = run_backfill(&s, &db, &store, from, to).await { + s.record_error(format!("backfill failed: {e:#}")).await; + } + }; + + // catch_unwind so a panic can't leave the lease held forever. + if let Err(panic) = AssertUnwindSafe(task).catch_unwind().await { + let msg = panic + .downcast_ref::<&str>() + .map(|s| s.to_string()) + .or_else(|| panic.downcast_ref::().cloned()) + .unwrap_or_else(|| "unknown panic".to_string()); + session + .record_error(format!("backfill task panicked: {msg}")) + .await; + } + + session.release().await; + }); +} + +/// Export `[from, to)` in keyset pages ordered by `(timestamp, id)`. Each page is +/// a bounded, partition-pruned scan; rows are grouped by UTC day and written one +/// object per day per page. +async fn run_backfill( + session: &Session, + db: &DB, + store: &Arc, + from: DateTime, + to: DateTime, +) -> error::Result<()> { + session.update(|p| p.phase = "exporting".to_string()).await; + + // Keyset cursor over (timestamp, id). `id` starts below any real value so the + // first page includes rows at exactly `from`. + let mut cursor_ts = from; + let mut cursor_id: i64 = -1; + let page_rows = page_rows(); + // Namespace object keys by the requested window. The per-page `min_id` alone is not + // unique across runs: a narrower, overlapping backfill can start a day's page at the + // same first row (same `min_id`) but contain fewer rows, and `put` would overwrite a + // broader run's object — silently dropping the rows only that object held. Including + // the window makes different ranges write disjoint keys (same window re-runs stay + // idempotent); consumers already dedupe overlapping rows by `id`. + let window_key = format!("{}_{}", from.timestamp_millis(), to.timestamp_millis()); + + loop { + let rows = sqlx::query!( + r#"SELECT to_char(timestamp AT TIME ZONE 'UTC', 'YYYY-MM-DD') AS "day!", + id AS "id!", + timestamp AS "ts!", + row_to_json(r)::text AS "line!" + FROM ( + SELECT workspace_id, id, timestamp, username, operation, + action_kind::text AS action_kind, resource, parameters, email, span + FROM audit_partitioned + WHERE timestamp >= $1 AND timestamp < $2 + AND (timestamp, id) > ($3, $4) + ORDER BY timestamp, id + LIMIT $5 + ) r + ORDER BY timestamp, id"#, + from, + to, + cursor_ts, + cursor_id, + page_rows + ) + .fetch_all(db) + .await?; + + if rows.is_empty() { + break; + } + + // Group this page's ndjson lines by day, preserving (timestamp, id) order, + // and track the min id per day for a deterministic, collision-free key. + let mut by_day: Vec<(String, i64, String)> = Vec::new(); // (day, min_id, ndjson) + for row in &rows { + match by_day.last_mut() { + Some((day, _min_id, acc)) if *day == row.day => { + acc.push('\n'); + acc.push_str(&row.line); + } + _ => by_day.push((row.day.clone(), row.id, row.line.clone())), + } + } + + for (day, min_id, ndjson) in &by_day { + let object_path = ObjectPath::from(format!( + "{LOGS_AUDIT}dt={day}/audit_backfill_{window_key}_{min_id}.ndjson" + )); + store + .put(&object_path, ndjson.clone().into_bytes().into()) + .await + .map_err(|e| error::Error::internal_err(format!("upload {object_path}: {e:#}")))?; + let n = ndjson.lines().count() as u64; + // Persist progress (and refresh the lease heartbeat) after every object, + // not just once the page completes: a stale heartbeat lets another replica + // re-claim the lease and run a concurrent backfill, so the gap between + // heartbeats must stay well under STALE_HEARTBEAT_SECS even if a page's + // uploads are slow. + session + .update(|p| { + p.rows_written = p.rows_written.saturating_add(n); + p.objects_written = p.objects_written.saturating_add(1); + }) + .await; + } + + // Advance the keyset cursor past the last row of this page. + let last = rows.last().expect("page is non-empty"); + cursor_ts = last.ts; + cursor_id = last.id; + + let new_last_ts = last.ts; + session.update(|p| p.last_ts = Some(new_last_ts)).await; + + // A short page means the window is exhausted. + if (rows.len() as i64) < page_rows { + break; + } + } + + Ok(()) +} + +#[cfg(all(test, feature = "parquet"))] +mod tests { + use super::*; + use futures::stream::StreamExt; + use std::sync::atomic::Ordering; + use std::sync::Arc; + use windmill_object_store::object_store_reexports::{InMemory, ObjectStore, Path as OsPath}; + + /// A private, per-test object store. `run_backfill` takes the store as a parameter, + /// so tests use a local one and never touch the process-global `OBJECT_STORE_SETTINGS` + /// (which would otherwise race across the parallel test runner). + fn local_store() -> (Arc, Arc) { + let store = Arc::new(InMemory::new()); + let dynstore: Arc = store.clone(); + (store, dynstore) + } + + /// Serializes the tests that touch the `PAGE_ROWS_OVERRIDE` process global (read + /// inside `run_backfill`) so they can't observe each other's value under the parallel + /// runner. + static PAGE_OVERRIDE_LOCK: tokio::sync::Mutex<()> = tokio::sync::Mutex::const_new(()); + + /// Resets `PAGE_ROWS_OVERRIDE` on drop so a failing assertion can't leak a non-default + /// page size into another test. + struct ResetPageOverride; + impl Drop for ResetPageOverride { + fn drop(&mut self) { + PAGE_ROWS_OVERRIDE.store(0, Ordering::Relaxed); + } + } + + /// Insert an audit row `days` days in the past (creating the daily partition if + /// needed). The row's `timestamp` defaults to that point, landing it in the + /// matching partition. + async fn insert_audit_days_ago(db: &DB, operation: &str, days: i64) -> i64 { + sqlx::query(&format!( + "DO $$ DECLARE d date := current_date - {days}; BEGIN \ + EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_partitioned \ + FOR VALUES FROM (%L) TO (%L)', 'audit_'||to_char(d,'YYYYMMDD'), d, d + 1); END $$;" + )) + .execute(db) + .await + .ok(); + sqlx::query_scalar::<_, i64>(&format!( + "INSERT INTO audit_partitioned + (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{{}}'::jsonb, + now() - interval '{days} days') + RETURNING id" + )) + .bind(operation) + .fetch_one(db) + .await + .expect("insert audit row") + } + + /// Insert an audit row at an exact timestamp (creating the daily partition if + /// needed), for tests that need distinct in-day timestamps. + async fn insert_audit_at(db: &DB, operation: &str, ts: DateTime) -> i64 { + sqlx::query(&format!( + "DO $$ DECLARE d date := '{}'; BEGIN \ + EXECUTE format('CREATE TABLE IF NOT EXISTS %I PARTITION OF audit_partitioned \ + FOR VALUES FROM (%L) TO (%L)', 'audit_'||to_char(d,'YYYYMMDD'), d, d + 1); END $$;", + ts.format("%Y-%m-%d") + )) + .execute(db) + .await + .ok(); + sqlx::query_scalar::<_, i64>( + "INSERT INTO audit_partitioned + (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{}'::jsonb,$2) + RETURNING id", + ) + .bind(operation) + .bind(ts) + .fetch_one(db) + .await + .expect("insert audit row") + } + + /// All ids across every `audit_backfill_*.ndjson` object, and the set of object + /// paths (to assert pagination/day keying). + async fn backfilled(store: &InMemory) -> (Vec, Vec) { + let prefix = OsPath::from("logs/audit"); + let metas = store + .list(Some(&prefix)) + .collect::>() + .await + .into_iter() + .map(|m| m.expect("list object")) + .collect::>(); + let mut ids = Vec::new(); + let mut paths = Vec::new(); + for meta in metas { + paths.push(meta.location.to_string()); + let bytes = store + .get(&meta.location) + .await + .expect("get object") + .bytes() + .await + .expect("read bytes"); + for line in std::str::from_utf8(&bytes).unwrap().lines() { + if line.is_empty() { + continue; + } + let v: serde_json::Value = serde_json::from_str(line).expect("valid ndjson"); + ids.push(v.get("id").and_then(|x| x.as_i64()).expect("row has id")); + } + } + ids.sort(); + paths.sort(); + (ids, paths) + } + + fn session(db: &DB, from: DateTime, to: DateTime) -> Session { + Session { + db: db.clone(), + owner: INSTANCE_NAME.clone(), + progress: RwLock::new(AuditBackfillProgress::new_running(from, to)), + } + } + + // End-to-end backfill: a settled multi-day window is exported in bounded keyset + // pages (forced to 2 rows/page) — every in-window row lands exactly once, rows + // outside [from,to) are excluded, a day that spans a page boundary produces more + // than one object, and re-running is idempotent (same keys overwritten, no dupes). + #[sqlx::test(migrations = "../migrations")] + async fn backfill_exports_window_in_pages(db: DB) -> anyhow::Result<()> { + let _serial = PAGE_OVERRIDE_LOCK.lock().await; + let _reset = ResetPageOverride; // restores the page override even on panic + let (store, dyn_store) = local_store(); + // Force multi-page / page-spanning-day keyset behaviour with a handful of rows. + PAGE_ROWS_OVERRIDE.store(2, Ordering::Relaxed); + + // In window [now-6d, now-2d): days 5, 4, 3 ago. + let mut want = Vec::new(); + for i in 0..3 { + want.push(insert_audit_days_ago(&db, &format!("bf.d5.{i}"), 5).await); + } + for i in 0..2 { + want.push(insert_audit_days_ago(&db, &format!("bf.d4.{i}"), 4).await); + } + for i in 0..2 { + want.push(insert_audit_days_ago(&db, &format!("bf.d3.{i}"), 3).await); + } + want.sort(); + // Out of window: before `from` and at/after `to`. + let before = insert_audit_days_ago(&db, "bf.before", 7).await; + let after = insert_audit_days_ago(&db, "bf.after", 1).await; + + let from = Utc::now() - chrono::Duration::days(6); + let to = Utc::now() - chrono::Duration::days(2); + + let s = session(&db, from, to); + run_backfill(&s, &db, &dyn_store, from, to).await?; + + let (ids, paths) = backfilled(&store).await; + assert_eq!(ids, want, "exactly the in-window rows, each once: {ids:?}"); + assert!( + !ids.contains(&before) && !ids.contains(&after), + "rows outside [from,to) must not be exported" + ); + // 3 rows on the day-5 partition at a 2-row page size => that day spans pages, + // so it yields >1 object — proving keyset paging across a day boundary. + let day5_objects = paths + .iter() + .filter(|p| p.contains("audit_backfill_")) + .count(); + assert!( + day5_objects >= 4, + "expected multiple paged objects (incl. a split day), got {paths:?}" + ); + { + let p = s.progress.read().await; + assert_eq!(p.rows_written, want.len() as u64, "progress row count"); + } + + // Idempotent re-run: deterministic keys are overwritten, never duplicated. + let s2 = session(&db, from, to); + run_backfill(&s2, &db, &dyn_store, from, to).await?; + let (ids2, _) = backfilled(&store).await; + assert_eq!(ids2, want, "re-run stays exactly once per row: {ids2:?}"); + + Ok(()) + } + + // A narrower backfill overlapping a broader one must not overwrite (and drop rows + // from) the broader run's object: the object key includes the window. The two + // windows share a day and the same first row (so the same `min_id`), but the + // narrower one holds fewer rows. + #[sqlx::test(migrations = "../migrations")] + async fn backfill_window_in_key_prevents_overwrite(db: DB) -> anyhow::Result<()> { + // Hold the lock so no concurrent test's PAGE_ROWS_OVERRIDE is observed; this test + // wants the default (large) page size so each day is one object per window. + let _serial = PAGE_OVERRIDE_LOCK.lock().await; + let (store, dyn_store) = local_store(); + + // Four rows on the same day at distinct times. + let base = Utc::now() - chrono::Duration::days(5); + let r0 = insert_audit_at(&db, "ov.0", base).await; + let r1 = insert_audit_at(&db, "ov.1", base + chrono::Duration::seconds(10)).await; + let r2 = insert_audit_at(&db, "ov.2", base + chrono::Duration::seconds(20)).await; + let r3 = insert_audit_at(&db, "ov.3", base + chrono::Duration::seconds(30)).await; + + // Broad run covers all four (one object for the day, keyed by r0). + let a_from = base - chrono::Duration::seconds(1); + let a_to = base + chrono::Duration::seconds(31); + run_backfill(&session(&db, a_from, a_to), &db, &dyn_store, a_from, a_to).await?; + + // Narrow run starts at the same first row (same min_id) but holds only r0, r1. + let b_from = base - chrono::Duration::seconds(1); + let b_to = base + chrono::Duration::seconds(15); + run_backfill(&session(&db, b_from, b_to), &db, &dyn_store, b_from, b_to).await?; + + let (ids, _) = backfilled(&store).await; + for id in [r0, r1, r2, r3] { + assert!( + ids.contains(&id), + "row {id} lost — a narrower overlapping window overwrote the broader run's \ + object: {ids:?}" + ); + } + Ok(()) + } + + // The endpoint rejects a window whose upper bound is not yet settled (a row's + // timestamp is its txn's xact_start, so a future/live `to` could miss late + // commits), but accepts a window safely in the past. + #[sqlx::test(migrations = "../migrations")] + async fn backfill_rejects_unstable_window(db: DB) -> anyhow::Result<()> { + let future = Utc::now() + chrono::Duration::days(1); + let past_from = Utc::now() - chrono::Duration::days(2); + let err = try_start(&db, past_from, future).await.unwrap_err(); + assert!( + matches!(err, error::Error::BadRequest(_)), + "a future `to` must be rejected as unstable, got {err:?}" + ); + + // A window fully in the settled past is accepted. + let from = Utc::now() - chrono::Duration::days(3); + let to = Utc::now() - chrono::Duration::days(2); + try_start(&db, from, to) + .await + .expect("a settled past window is accepted"); + Ok(()) + } + + /// Insert a row into the legacy (non-partitioned) `audit` table at an exact time. + async fn insert_legacy_audit_at(db: &DB, operation: &str, ts: DateTime) { + sqlx::query( + "INSERT INTO audit (workspace_id, username, operation, action_kind, parameters, timestamp) + VALUES ('test-ws','tester',$1,'create'::action_kind,'{}'::jsonb,$2)", + ) + .bind(operation) + .bind(ts) + .execute(db) + .await + .expect("insert legacy audit row"); + } + + // A window overlapping rows in the legacy (non-partitioned) `audit` table is rejected: + // those rows are not exported, so the backfill must not report success while silently + // omitting them. Covers the empty-`audit_partitioned` case (a min(partitioned) guard + // would no-op there). + #[sqlx::test(migrations = "../migrations")] + async fn backfill_rejects_window_overlapping_legacy(db: DB) -> anyhow::Result<()> { + // A legacy row ~5 days ago, and no partitioned rows at all. + insert_legacy_audit_at(&db, "legacy.row", Utc::now() - chrono::Duration::days(5)).await; + + // A window covering it is rejected. + let from = Utc::now() - chrono::Duration::days(6); + let to = Utc::now() - chrono::Duration::days(2); + let err = try_start(&db, from, to).await.unwrap_err(); + assert!( + matches!(err, error::Error::BadRequest(_)), + "a window overlapping legacy audit rows must be rejected, got {err:?}" + ); + + // A window clear of any legacy row is accepted. + let from_ok = Utc::now() - chrono::Duration::days(2); + let to_ok = Utc::now() - chrono::Duration::days(1); + try_start(&db, from_ok, to_ok) + .await + .expect("a window with no legacy overlap is accepted"); + Ok(()) + } +} diff --git a/backend/windmill-api-settings/src/lib.rs b/backend/windmill-api-settings/src/lib.rs index 9fd1091d66..a7e1322ad9 100644 --- a/backend/windmill-api-settings/src/lib.rs +++ b/backend/windmill-api-settings/src/lib.rs @@ -14,6 +14,8 @@ use std::{ #[cfg(feature = "parquet")] mod audit_logs_s3; #[cfg(feature = "parquet")] +mod audit_logs_s3_backfill; +#[cfg(feature = "parquet")] mod background_task; #[cfg(feature = "private")] mod ee; @@ -204,7 +206,12 @@ pub fn global_service() -> Router { ) .route("/run_log_cleanup", post(run_log_cleanup)) .route("/log_cleanup_status", get(log_cleanup_status)) - .route("/audit_logs_s3_status", get(audit_logs_s3_status)); + .route("/audit_logs_s3_status", get(audit_logs_s3_status)) + .route("/audit_logs_s3_backfill", post(run_audit_logs_s3_backfill)) + .route( + "/audit_logs_s3_backfill_status", + get(audit_logs_s3_backfill_status), + ); } #[cfg(not(feature = "parquet"))] @@ -249,57 +256,307 @@ use windmill_object_store::build_object_store_from_settings; #[cfg(feature = "parquet")] pub async fn test_s3_bucket( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Json(test_s3_bucket): Json, ) -> error::Result { use bytes::Bytes; use futures::StreamExt; + // The probe executes on the API server itself. On multi-tenant Cloud that is a shared control + // plane, so we constrain untrusted callers to remove the SSRF / credential-exfiltration / + // local-filesystem surface (see validate_object_storage_test). On self-hosted instances the + // object store usually lives on the local/private network and all authenticated users are + // trusted, so testing there stays unrestricted. Super admins keep the unrestricted path too. + let is_super_admin = is_super_admin_email(&db, &authed.email).await?; + let restrict = !is_super_admin && *CLOUD_HOSTED; + if restrict { + validate_object_storage_test(&test_s3_bucket).await?; + } + let client = build_object_store_from_settings(test_s3_bucket, Some(&db)) .await? .store; - let mut list = client.list(Some( - &windmill_object_store::object_store_reexports::Path::from("".to_string()), - )); - let first_file = list.next().await; - if first_file.is_some() { - if let Err(e) = first_file.as_ref().unwrap() { - tracing::error!("error listing bucket: {e:#}"); - error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}")); + let run = async { + let mut list = client.list(Some( + &windmill_object_store::object_store_reexports::Path::from("".to_string()), + )); + let first_file = list.next().await; + if first_file.is_some() { + if let Err(e) = first_file.as_ref().unwrap() { + tracing::error!("error listing bucket: {e:#}"); + error::Error::internal_err(format!("Failed to list files in blob storage: {e:#}")); + } + tracing::info!("Listed files: {:?}", first_file.unwrap()); + } else { + tracing::info!("No files in blob storage"); } - tracing::info!("Listed files: {:?}", first_file.unwrap()); + + let path = windmill_object_store::object_store_reexports::Path::from(format!( + "/test-s3-bucket-{uuid}", + uuid = uuid::Uuid::new_v4() + )); + tracing::info!("Testing blob storage at path: {path}"); + client + .put( + &path, + windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"), + ) + .await + .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?; + let content = client + .get(&path) + .await + .map_err(to_anyhow)? + .bytes() + .await + .map_err(to_anyhow)?; + if content != Bytes::from_static(b"hello") { + return Err(error::Error::internal_err( + "Failed to read back from blob storage".to_string(), + )); + } + client.delete(&path).await.map_err(to_anyhow)?; + Ok::("Tested blob storage successfully".to_string()) + }; + + if restrict { + // The object-store client is built with timeouts disabled, so a malicious endpoint could + // otherwise hold the API server connection open indefinitely. + tokio::time::timeout(Duration::from_secs(15), run) + .await + .map_err(|_| { + error::Error::internal_err("Object storage connectivity test timed out".to_string()) + })? } else { - tracing::info!("No files in blob storage"); + run.await + } +} + +// Hardening for the object-storage connectivity test by an untrusted (non-super-admin) caller on +// Cloud. The probe runs on the shared API server, so without these constraints an authenticated +// user could coerce the server into connecting to arbitrary internal endpoints (SSRF), signing +// requests with the instance role (credential exfiltration), or reading/writing the server's local +// disk (filesystem object store). +#[cfg(feature = "parquet")] +async fn validate_object_storage_test(settings: &ObjectSettings) -> error::Result<()> { + fn non_empty(opt: &Option) -> bool { + opt.as_ref().is_some_and(|s| !s.is_empty()) } - let path = windmill_object_store::object_store_reexports::Path::from(format!( - "/test-s3-bucket-{uuid}", - uuid = uuid::Uuid::new_v4() - )); - tracing::info!("Testing blob storage at path: {path}"); - client - .put( - &path, - windmill_object_store::object_store_reexports::PutPayload::from_static(b"hello"), - ) + // Reject backends that rely on the server's identity or local filesystem, require explicit + // credentials for the rest (so the server never falls back to its own ambient credentials), and + // resolve the host the client will actually connect to. We derive the *effective* endpoint here + // — mirroring build_*_from_settings: the region/account-derived default and the virtual-hosted + // bucket prefix — rather than only validating a caller-supplied `endpoint`, so caller-controlled + // `region`/`account_name`/`bucket` cannot smuggle an internal host past the check (e.g. an empty + // endpoint with region = "@169.254.169.254/" otherwise resolves to the cloud metadata service). + let effective_endpoint: Option = match settings { + ObjectSettings::Filesystem(_) => { + return Err(error::Error::NotAuthorized( + "Testing a local filesystem object store requires a super admin".to_string(), + )); + } + ObjectSettings::AwsOidc(_) => { + return Err(error::Error::NotAuthorized( + "Testing OIDC-based object storage requires a super admin".to_string(), + )); + } + ObjectSettings::S3(s3) => { + if !(non_empty(&s3.access_key) && non_empty(&s3.secret_key)) { + return Err(error::Error::NotAuthorized( + "Testing S3 storage without explicit credentials requires a super admin" + .to_string(), + )); + } + let region = s3 + .region + .clone() + .filter(|r| !r.is_empty()) + .or_else(|| std::env::var("AWS_REGION").ok().filter(|r| !r.is_empty())) + .unwrap_or_else(|| "us-east-1".to_string()); + let raw_endpoint = s3 + .endpoint + .clone() + .filter(|e| !e.is_empty()) + .or_else(|| std::env::var("S3_ENDPOINT").ok().filter(|e| !e.is_empty())) + .unwrap_or_else(|| format!("s3.{region}.amazonaws.com")); + Some(windmill_object_store::render_endpoint( + raw_endpoint, + !s3.allow_http.unwrap_or(true), + s3.port, + s3.path_style, + s3.bucket.clone().unwrap_or_default(), + )) + } + ObjectSettings::Azure(azure) => { + if !non_empty(&azure.access_key) { + return Err(error::Error::NotAuthorized( + "Testing Azure storage without an explicit access key requires a super admin" + .to_string(), + )); + } + Some( + azure + .endpoint + .clone() + .filter(|e| !e.is_empty()) + .unwrap_or_else(|| format!("{}.blob.core.windows.net", azure.account_name)), + ) + } + ObjectSettings::Gcs(gcs) => { + if gcs.service_account_key.is_empty() { + return Err(error::Error::NotAuthorized( + "Testing GCS storage without a service account key requires a super admin" + .to_string(), + )); + } + // The service-account-key JSON can override the data-plane URL (`gcs_base_url`) and the + // OAuth token endpoint (`token_uri`); the GCS client connects to whatever they point at. + // Validate every http(s) URL embedded in the key. When none override it, the host stays + // the public storage.googleapis.com, so no further check is needed. + if let Ok(serde_json::Value::Object(map)) = + serde_json::from_str::(&gcs.service_account_key) + { + for value in map.values() { + if let Some(url) = value.as_str() { + // Match how the URL parser reads the value: leading whitespace/control is + // ignored and the scheme is case-insensitive. + let url = + url.trim_start_matches(|c: char| c.is_whitespace() || c.is_control()); + if strip_http_scheme(url).is_some() { + validate_public_endpoint(url).await?; + } + } + } + } + None + } + }; + + // Block non-public network targets (internal services, cloud metadata, loopback, ...). + if let Some(endpoint) = effective_endpoint { + validate_public_endpoint(&endpoint).await?; + } + Ok(()) +} + +#[cfg(feature = "parquet")] +async fn validate_public_endpoint(endpoint: &str) -> error::Result<()> { + let host = extract_host(endpoint).ok_or_else(|| { + error::Error::BadRequest(format!("Invalid object storage endpoint: {endpoint}")) + })?; + + let addrs: Vec = tokio::net::lookup_host((host.as_str(), 443u16)) .await - .map_err(|e| anyhow::anyhow!("error writing file to {path}: {e:#}"))?; - let content = client - .get(&path) - .await - .map_err(to_anyhow)? - .bytes() - .await - .map_err(to_anyhow)?; - if content != Bytes::from_static(b"hello") { - return Err(error::Error::internal_err( - "Failed to read back from blob storage".to_string(), - )); + .map_err(|e| { + error::Error::BadRequest(format!( + "Could not resolve object storage endpoint '{host}': {e}" + )) + })? + .collect(); + + if addrs.is_empty() { + return Err(error::Error::BadRequest(format!( + "Could not resolve object storage endpoint '{host}'" + ))); + } + + // Reject if any resolved address is non-public, which also defeats the simplest DNS-rebinding + // attempts (a name resolving to both a public and a private address). + for addr in addrs { + if is_forbidden_ip(addr.ip()) { + return Err(error::Error::NotAuthorized( + "Testing object storage at a private, loopback, or link-local endpoint requires a super admin" + .to_string(), + )); + } + } + Ok(()) +} + +// Strip a leading `http://`/`https://` scheme case-insensitively (URL schemes are +// case-insensitive), returning the remainder when one was present. +#[cfg(feature = "parquet")] +fn strip_http_scheme(s: &str) -> Option<&str> { + for scheme in ["https://", "http://"] { + let b = scheme.as_bytes(); + if s.len() >= b.len() && s.as_bytes()[..b.len()].eq_ignore_ascii_case(b) { + return Some(&s[b.len()..]); + } + } + None +} + +#[cfg(feature = "parquet")] +fn extract_host(endpoint: &str) -> Option { + let mut s = endpoint.trim(); + if let Some(rest) = strip_http_scheme(s) { + s = rest; + } + s = s.split(['/', '?', '#', '\\']).next().unwrap_or(s); + if let Some((_, rest)) = s.rsplit_once('@') { + s = rest; + } + let host = if let Some(rest) = s.strip_prefix('[') { + // IPv6 literal, e.g. [::1]:9000 + rest.split(']').next().unwrap_or(rest) + } else { + // host or host:port + s.split(':').next().unwrap_or(s) + } + .trim(); + if host.is_empty() { + None + } else { + Some(host.to_string()) + } +} + +#[cfg(feature = "parquet")] +fn is_forbidden_ip(ip: std::net::IpAddr) -> bool { + use std::net::{IpAddr, Ipv4Addr}; + match ip { + IpAddr::V4(v4) => { + v4.is_loopback() + || v4.is_private() + || v4.is_link_local() // 169.254.0.0/16, incl. the cloud metadata endpoint + || v4.is_unspecified() + || v4.is_broadcast() + || v4.is_documentation() + || v4.is_multicast() + || v4.octets()[0] == 0 // 0.0.0.0/8 + || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xc0) == 64) // 100.64.0.0/10 CGNAT + } + IpAddr::V6(v6) => { + // Any IPv4 embedded in an IPv6 address (IPv4-mapped ::ffff:0:0/96, IPv4-compatible + // ::/96, or NAT64 64:ff9b::/96) is re-checked against the IPv4 rules, so e.g. + // 64:ff9b::169.254.169.254 cannot route to the metadata endpoint in a NAT64 network. + let seg = v6.segments(); + let is_v4_compatible = seg[0..6] == [0, 0, 0, 0, 0, 0]; + let is_nat64 = seg[0] == 0x0064 && seg[1] == 0xff9b && seg[2..6] == [0, 0, 0, 0]; + if let Some(v4) = v6.to_ipv4_mapped() { + return is_forbidden_ip(IpAddr::V4(v4)); + } + if is_v4_compatible || is_nat64 { + let embedded = Ipv4Addr::new( + (seg[6] >> 8) as u8, + (seg[6] & 0xff) as u8, + (seg[7] >> 8) as u8, + (seg[7] & 0xff) as u8, + ); + if is_forbidden_ip(IpAddr::V4(embedded)) { + return true; + } + } + v6.is_loopback() + || v6.is_unspecified() + || v6.is_multicast() + || (seg[0] & 0xfe00) == 0xfc00 // fc00::/7 unique local + || (seg[0] & 0xffc0) == 0xfe80 // fe80::/10 link-local + } } - client.delete(&path).await.map_err(to_anyhow)?; - Ok("Tested blob storage successfully".to_string()) } #[cfg(feature = "parquet")] @@ -351,6 +608,32 @@ async fn audit_logs_s3_status( Ok(Json(audit_logs_s3::get_status(&db).await?)) } +#[cfg(feature = "parquet")] +async fn run_audit_logs_s3_backfill( + Extension(db): Extension, + authed: ApiAuthed, + Json(req): Json, +) -> error::Result { + require_super_admin(&db, &authed.email).await?; + if !matches!(get_license_plan().await, LicensePlan::Enterprise) { + return Err(error::Error::BadRequest( + "Audit log export to object storage is an Enterprise feature".to_string(), + )); + } + audit_logs_s3_backfill::try_start(&db, req.from, req.to).await?; + audit_logs_s3_backfill::spawn_backfill(db.clone(), req.from, req.to); + Ok(axum::http::StatusCode::ACCEPTED) +} + +#[cfg(feature = "parquet")] +async fn audit_logs_s3_backfill_status( + Extension(db): Extension, + authed: ApiAuthed, +) -> error::JsonResult> { + require_super_admin(&db, &authed.email).await?; + Ok(Json(audit_logs_s3_backfill::get_status(&db).await?)) +} + #[derive(Deserialize)] pub struct TestKey { pub license_key: String, @@ -1273,8 +1556,8 @@ async fn setup_custom_instance_pg_database_inner( // Validate name to ensure it only contains alphanumeric characters // Prevents SQL injection on the instance database lazy_static::lazy_static! { - // Must start with a letter, then alphanumeric/underscore - static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_]*$").unwrap(); + // Must start with a letter, then alphanumeric/underscore/hyphen + static ref VALID_NAME: regex::Regex = regex::Regex::new(r"^[a-zA-Z][a-zA-Z0-9_-]*$").unwrap(); } let dbname = dbname.trim(); if dbname.is_empty() { @@ -1290,7 +1573,7 @@ async fn setup_custom_instance_pg_database_inner( } if !VALID_NAME.is_match(dbname) { return Err(error::Error::BadRequest( - "Database name must start with a letter and contain only alphanumeric characters or underscores".to_string(), + "Database name must start with a letter and contain only alphanumeric characters, underscores, or hyphens".to_string(), )); } // Additional check: block PostgreSQL reserved/special names @@ -1859,3 +2142,111 @@ mod tests { ); } } + +#[cfg(all(test, feature = "parquet"))] +mod object_storage_test_hardening { + use super::{extract_host, is_forbidden_ip, validate_object_storage_test}; + use std::net::IpAddr; + use windmill_object_store::ObjectSettings; + + // IP literals (not hostnames) keep validate_public_endpoint deterministic — `lookup_host` + // parses them without any network round-trip. + fn gcs_settings(gcs_base_url: &str) -> ObjectSettings { + serde_json::from_value(serde_json::json!({ + "type": "Gcs", + "bucket": "b", + "serviceAccountKey": { "gcs_base_url": gcs_base_url, "client_email": "x@y.z" } + })) + .unwrap() + } + + #[tokio::test] + async fn rejects_gcs_internal_base_url() { + // gcs_base_url in the service-account key must not smuggle an internal host past the check, + // including via a mixed-case scheme (URL schemes are case-insensitive). + for url in [ + "http://169.254.169.254", + "HTTP://169.254.169.254", + "Https://10.0.0.5", + ] { + assert!( + validate_object_storage_test(&gcs_settings(url)) + .await + .is_err(), + "{url} should be rejected" + ); + } + } + + #[tokio::test] + async fn allows_gcs_public_base_url() { + assert!( + validate_object_storage_test(&gcs_settings("https://8.8.8.8")) + .await + .is_ok() + ); + } + + fn ip(s: &str) -> IpAddr { + s.parse().unwrap() + } + + #[test] + fn forbids_internal_ips() { + for s in [ + "127.0.0.1", // loopback + "169.254.169.254", // cloud metadata (link-local) + "10.0.0.5", // private + "172.16.3.4", // private + "192.168.1.10", // private + "0.0.0.0", // unspecified + "100.64.0.1", // CGNAT + "::1", // IPv6 loopback + "fe80::1", // IPv6 link-local + "fc00::1", // IPv6 unique local + "::ffff:127.0.0.1", // IPv4-mapped loopback + "::ffff:169.254.169.254", // IPv4-mapped metadata + "::169.254.169.254", // IPv4-compatible metadata + "64:ff9b::169.254.169.254", // NAT64-embedded metadata + "64:ff9b::a9fe:a9fe", // NAT64-embedded metadata (hex form) + ] { + assert!(is_forbidden_ip(ip(s)), "{s} should be forbidden"); + } + } + + #[test] + fn allows_public_ips() { + for s in ["8.8.8.8", "1.1.1.1", "52.95.110.1", "2606:4700:4700::1111"] { + assert!(!is_forbidden_ip(ip(s)), "{s} should be allowed"); + } + } + + #[test] + fn extracts_host_from_endpoint() { + let cases = [ + ("s3.amazonaws.com", Some("s3.amazonaws.com")), + ("https://minio.internal:9000", Some("minio.internal")), + ("http://10.0.0.5:9000/bucket", Some("10.0.0.5")), + ("user:pass@host.example:443", Some("host.example")), + ("[::1]:9000", Some("::1")), + ("https://[fe80::1]/x", Some("fe80::1")), + ("", None), + // Injection via region/bucket interpolation into the default endpoint string: the + // userinfo `@` and the path `/` must not hide the real authority from the host check. + ( + "https://s3.@169.254.169.254/.amazonaws.com", + Some("169.254.169.254"), + ), + ( + "https://@169.254.169.254/mybucket.s3.amazonaws.com", + Some("169.254.169.254"), + ), + ("s3.#@169.254.169.254/x.amazonaws.com", Some("s3.")), + // Scheme is case-insensitive. + ("HTTP://169.254.169.254", Some("169.254.169.254")), + ]; + for (input, expected) in cases { + assert_eq!(extract_host(input).as_deref(), expected, "input: {input}"); + } + } +} diff --git a/backend/windmill-api-settings/src/log_cleanup.rs b/backend/windmill-api-settings/src/log_cleanup.rs index 890b46ba9e..f5e5eab680 100644 --- a/backend/windmill-api-settings/src/log_cleanup.rs +++ b/backend/windmill-api-settings/src/log_cleanup.rs @@ -27,11 +27,14 @@ use uuid::Uuid; use crate::background_task; use windmill_common::error::{self}; +use windmill_common::jobs::delete_jobs; use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE}; use windmill_common::worker::WINDMILL_DIR; use windmill_common::{DB, INSTANCE_NAME, JOB_RETENTION_SECS, SERVICE_LOG_RETENTION_SECS}; -use windmill_object_store::object_store_reexports::{ObjectStore, Path as ObjectPath}; +use windmill_object_store::object_store_reexports::{ + ObjectStore, ObjectStoreError, Path as ObjectPath, +}; pub const TASK_NAME: &str = "log_cleanup"; @@ -61,6 +64,10 @@ pub struct LogCleanupProgress { pub total_jobs: u64, pub processed_jobs: u64, pub s3_deleted: u64, + /// Number of delete calls that returned 404 (object already absent — a no-op + /// success). GCS returns 404 per missing key where S3's DeleteObjects stays silent. + #[serde(default)] + pub s3_not_found: u64, /// Number of S3 objects inspected during the orphan scan phase. pub orphans_scanned: u64, /// Number of orphan S3 objects deleted (no corresponding DB row). @@ -81,6 +88,7 @@ impl LogCleanupProgress { total_jobs: 0, processed_jobs: 0, s3_deleted: 0, + s3_not_found: 0, orphans_scanned: 0, orphans_deleted: 0, errors: 0, @@ -132,6 +140,13 @@ impl Session { p.phase = "done".to_string(); p.clone() }; + tracing::info!( + "log cleanup finished: {} object(s) deleted from object store, {} already absent (404), {} orphans deleted, {} error(s)", + snapshot.s3_deleted, + snapshot.s3_not_found, + snapshot.orphans_deleted, + snapshot.errors + ); if let Err(e) = background_task::release(&self.db, TASK_NAME, &self.owner, &snapshot).await { tracing::warn!("log cleanup: failed to release lease: {e:#}"); @@ -179,21 +194,32 @@ pub async fn get_status(db: &DB) -> error::Result> { async fn s3_bulk_delete( store: &Arc, paths: Vec, -) -> (u64 /* deleted */, u64 /* errors */) { +) -> ( + u64, /* deleted */ + u64, /* not_found */ + u64, /* errors */ +) { let stream = futures::stream::iter(paths.into_iter().map(Ok)).boxed(); let mut deleted = 0u64; + let mut not_found = 0u64; let mut errors = 0u64; let mut res = store.delete_stream(stream); while let Some(r) = res.next().await { match r { Ok(_) => deleted += 1, + // Deleting a non-existent object is a successful no-op. S3's DeleteObjects + // ignores missing keys, but GCS returns 404 per delete, surfacing as + // NotFound — track it separately so it isn't reported as an error. + Err(ObjectStoreError::NotFound { .. }) => { + not_found += 1; + } Err(e) => { errors += 1; tracing::warn!("log cleanup: failed to delete object: {e:#}"); } } } - (deleted, errors) + (deleted, not_found, errors) } /// Delete the given relative paths from the local filesystem under `base_dir`. @@ -265,7 +291,7 @@ async fn cleanup_service_logs( .iter() .map(|p| ObjectPath::from(format!("{}{}", LOGS_SERVICE, p))) .collect(); - let (deleted, errors) = s3_bulk_delete(store, s3_paths).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, s3_paths).await; disk_bulk_delete(&*TMP_WINDMILL_LOGS_SERVICE, &rel_paths).await; session @@ -275,6 +301,7 @@ async fn cleanup_service_logs( p.total_service = p.processed_service; } p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; @@ -313,19 +340,21 @@ async fn cleanup_job_logs( return Ok(()); } + let mut completed_at_floor: Option> = None; loop { - let (deleted_count, rel_paths) = - delete_expired_jobs_batch(db, retention_secs, JOB_BATCH).await?; + let (deleted_count, rel_paths, max_completed_at) = + delete_expired_jobs_batch(db, retention_secs, JOB_BATCH, completed_at_floor).await?; if deleted_count == 0 { break; } + completed_at_floor = max_completed_at.or(completed_at_floor); let s3_paths: Vec = rel_paths .iter() .map(|p| ObjectPath::from(p.clone())) .collect(); - let (deleted, errors) = s3_bulk_delete(store, s3_paths).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, s3_paths).await; disk_bulk_delete(&*WINDMILL_DIR, &rel_paths).await; session @@ -335,6 +364,7 @@ async fn cleanup_job_logs( p.total_jobs = p.processed_jobs; } p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; @@ -355,7 +385,8 @@ async fn delete_expired_jobs_batch( db: &DB, job_retention_secs: i64, batch_size: i64, -) -> error::Result<(usize, Vec)> { + completed_at_floor: Option>, +) -> error::Result<(usize, Vec, Option>)> { let mut tx = db.begin().await?; let active_root_job_ids: Vec = sqlx::query_scalar!( @@ -368,29 +399,61 @@ async fn delete_expired_jobs_batch( .fetch_all(&mut *tx) .await?; - let deleted_jobs: Vec = sqlx::query_scalar!( - "DELETE FROM v2_job_completed - WHERE id IN ( - SELECT jc.id FROM v2_job_completed jc - LEFT JOIN v2_job j ON j.id = jc.id - WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval - AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) != ALL($3) - ORDER BY jc.completed_at ASC - LIMIT $2 - FOR UPDATE OF jc SKIP LOCKED - ) - RETURNING id", - job_retention_secs, - batch_size, - &active_root_job_ids - ) - .fetch_all(&mut *tx) - .await?; + // `completed_at_floor` carries a watermark across batches so each one resumes after the rows + // the previous batch processed instead of re-scanning the (potentially undeletable) oldest + // prefix; the empty-active-roots branch skips the v2_job join entirely. See + // backend/src/monitor.rs::delete_expired_jobs_batch for the full rationale. + let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() { + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT id FROM v2_job_completed + WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($3::timestamptz IS NULL OR completed_at >= $3) + ORDER BY completed_at ASC + LIMIT $2 + FOR UPDATE SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + } else { + let rows = sqlx::query!( + "DELETE FROM v2_job_completed + WHERE id IN ( + SELECT jc.id FROM v2_job_completed jc + LEFT JOIN v2_job j ON j.id = jc.id + WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval + AND ($4::timestamptz IS NULL OR jc.completed_at >= $4) + AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN ( + SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL + ) + ORDER BY jc.completed_at ASC + LIMIT $2 + FOR UPDATE OF jc SKIP LOCKED + ) + RETURNING id, completed_at", + job_retention_secs, + batch_size, + &active_root_job_ids, + completed_at_floor, + ) + .fetch_all(&mut *tx) + .await?; + let max = rows.iter().map(|r| r.completed_at).max(); + (rows.into_iter().map(|r| r.id).collect::>(), max) + }; let deleted_count = deleted_jobs.len(); if deleted_count == 0 { tx.commit().await?; - return Ok((0, Vec::new())); + return Ok((0, Vec::new(), max_completed_at)); } if let Err(e) = sqlx::query!( @@ -421,10 +484,21 @@ async fn delete_expired_jobs_batch( } }; - if let Err(e) = sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", &deleted_jobs) - .execute(&mut *tx) - .await + // Native retry markers have no FK (to keep this bulk delete cheap) — sweep + // them with their jobs here, same as the other side tables above. The table + // is created by a startup migration, so it always exists by the time cleanup + // runs. + if let Err(e) = sqlx::query!( + "DELETE FROM native_retry_attempt WHERE job_id = ANY($1)", + &deleted_jobs + ) + .execute(&mut *tx) + .await { + tracing::error!("log cleanup: error deleting native retry markers: {e:?}"); + } + + if let Err(e) = delete_jobs(&mut *tx, &deleted_jobs).await { tracing::error!("log cleanup: error deleting job: {e:?}"); } @@ -440,7 +514,7 @@ async fn delete_expired_jobs_batch( tx.commit().await?; - Ok((deleted_count, log_paths)) + Ok((deleted_count, log_paths, max_completed_at)) } /// Scan S3 under the `logs/` prefix for orphan log files and delete them. @@ -561,11 +635,12 @@ async fn flush_service_orphans( batch: &mut Vec, ) { let paths = std::mem::take(batch); - let (deleted, errors) = s3_bulk_delete(store, paths).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, paths).await; session .update(|p| { p.orphans_deleted = p.orphans_deleted.saturating_add(deleted); p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; @@ -616,11 +691,12 @@ async fn flush_job_orphans( return; } - let (deleted, errors) = s3_bulk_delete(store, to_delete).await; + let (deleted, not_found, errors) = s3_bulk_delete(store, to_delete).await; session .update(|p| { p.orphans_deleted = p.orphans_deleted.saturating_add(deleted); p.s3_deleted = p.s3_deleted.saturating_add(deleted); + p.s3_not_found = p.s3_not_found.saturating_add(not_found); p.errors = p.errors.saturating_add(errors); }) .await; diff --git a/backend/windmill-api-users/src/users.rs b/backend/windmill-api-users/src/users.rs index db9ecb118d..2963b91a50 100644 --- a/backend/windmill-api-users/src/users.rs +++ b/backend/windmill-api-users/src/users.rs @@ -27,7 +27,7 @@ use axum::{ Json, Router, }; use hyper::{header::LOCATION, StatusCode}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin, OptJobAuthed}; use windmill_common::usernames::{ generate_instance_wide_unique_username, get_instance_username_or_create_pending, }; @@ -1415,11 +1415,13 @@ async fn convert_user_to_group( async fn update_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(email_to_update): Path, Extension(db): Extension, Json(eu): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; let mut new_super_admin: Option = None; @@ -1581,10 +1583,12 @@ async fn update_user( async fn delete_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(email_to_delete): Path, Extension(db): Extension, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!("DELETE FROM token WHERE email = $1", &email_to_delete) @@ -1840,7 +1844,7 @@ async fn delete_workspace_user( username_to_delete, &w_id, ) - .fetch_optional(&db) + .fetch_optional(&mut *tx) .await?; let email_to_delete = not_found_if_none(email_to_delete_o, "User", &username_to_delete)?; @@ -1877,9 +1881,11 @@ async fn set_login_type( Extension(db): Extension, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(et): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!( @@ -2019,8 +2025,6 @@ async fn refresh_token( authed: ApiAuthed, cookies: Cookies, ) -> Result { - let mut tx = db.begin().await?; - if let Some(thresh_s) = query.if_expiring_in_less_than_s { let t_hash = windmill_common::auth::hash_token(&token); let not_expired = sqlx::query_scalar!("SELECT true FROM token WHERE token_hash = $1 and expiration IS NOT NULL and expiration > now() + $2::int * '1 sec'::interval", &t_hash, thresh_s) @@ -2033,6 +2037,8 @@ async fn refresh_token( } } + let mut tx = db.begin().await?; + let super_admin = sqlx::query_scalar!( "SELECT super_admin FROM password WHERE email = $1 AND disabled = false", &authed.email @@ -2159,8 +2165,10 @@ pub async fn create_session_token<'c>( async fn create_token( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(token_config): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; check_token_create_rate_limit(&authed.username)?; windmill_api_auth::ensure_scopes_within_caller(&authed, token_config.scopes.as_deref())?; @@ -2176,6 +2184,7 @@ async fn create_token( async fn impersonate( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(new_token): Json, ) -> Result<(StatusCode, String)> { use windmill_common::min_version::MIN_VERSION_SUPPORTS_TOKEN_HASH; @@ -2189,6 +2198,7 @@ async fn impersonate( Some(&token) }; require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; if new_token.impersonate_email.is_none() { return Err(Error::BadRequest( @@ -2707,8 +2717,10 @@ struct ExportedGlobalUser { async fn export_global_users( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, ) -> JsonResult> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; let users = sqlx::query_as!( ExportedGlobalUser, @@ -2744,9 +2756,11 @@ async fn export_global_users() -> JsonResult { async fn overwrite_global_users( Extension(db): Extension, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(users): Json>, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; sqlx::query!("DELETE FROM password") .execute(&mut *tx) diff --git a/backend/windmill-api-workspaces/src/deployment_requests.rs b/backend/windmill-api-workspaces/src/deployment_requests.rs index 7782382073..a279f82d6c 100644 --- a/backend/windmill-api-workspaces/src/deployment_requests.rs +++ b/backend/windmill-api-workspaces/src/deployment_requests.rs @@ -717,16 +717,27 @@ async fn create_deployment_request_comment( // ---- helpers ------------------------------------------------------------ async fn parent_of_fork(db: &DB, w_id: &str) -> Result { - sqlx::query_scalar!( - "SELECT parent_workspace_id FROM workspace WHERE id = $1", + // Resolve the fork's parent and require it to still exist and be active. A + // parent that is archived (soft-deleted) can no longer be accessed, so a + // diff or deployment request against it targets an unreachable workspace. + let parent = sqlx::query!( + "SELECT p.id AS \"id!\", p.deleted AS \"deleted!\" + FROM workspace f + JOIN workspace p ON p.id = f.parent_workspace_id + WHERE f.id = $1", w_id, ) .fetch_optional(db) - .await? - .flatten() - .ok_or_else(|| { - Error::BadRequest(format!( + .await?; + + match parent { + None => Err(Error::BadRequest(format!( "workspace {w_id} is not a fork (no parent_workspace_id)" - )) - }) + ))), + Some(p) if p.deleted => Err(Error::BadRequest(format!( + "parent workspace {} of fork {w_id} is archived", + p.id + ))), + Some(p) => Ok(p.id), + } } diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 895c3a50ad..d2c5718ce1 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -48,7 +48,9 @@ use windmill_common::workspaces::{Ducklake, DucklakeCatalogResourceType}; use windmill_common::PgDatabase; use windmill_common::{ error::{Error, JsonResult, Result}, - global_settings::AUTOMATE_USERNAME_CREATION_SETTING, + global_settings::{ + AUTOMATE_USERNAME_CREATION_SETTING, DISABLE_WORKSPACE_INVITE_EMAILS_SETTING, + }, oauth2::WORKSPACE_SLACK_BOT_TOKEN_PATH, utils::{paginate, rd_string, require_admin, Pagination}, }; @@ -195,6 +197,7 @@ pub fn global_service() -> Router { .route("/list_as_superadmin", get(list_workspaces_as_super_admin)) .route("/list", get(list_workspaces)) .route("/users", get(user_workspaces)) + .route("/session_workspace_status", post(session_workspace_status)) .route("/create", post(create_workspace)) .route("/create_fork", post(deprecated_create_workspace_fork)) .route("/exists", post(exists_workspace)) @@ -2263,7 +2266,7 @@ async fn edit_ducklake_config( "#, &w_id ) - .fetch_one(&db) + .fetch_one(&mut *tx) .await? .unwrap_or(serde_json::Value::Null); let old_ducklakes: HashMap = @@ -2335,7 +2338,7 @@ async fn edit_datatable_config( "#, &w_id ) - .fetch_one(&db) + .fetch_one(&mut *tx) .await? .unwrap_or(serde_json::Value::Null); let old_datatables: HashMap = @@ -3654,6 +3657,47 @@ async fn user_workspaces( Ok(Json(WorkspaceList { email, workspaces })) } +#[derive(Deserialize)] +struct SessionWorkspaceStatusRequest { + workspace_ids: Vec, +} + +/// Reconciliation support for client-side AI sessions, which the backend cannot touch +/// directly. The client posts the workspace ids its sessions reference and uses the +/// per-id status to keep sessions in sync with workspace lifecycle: `deleted` (no row / +/// no access → unresolvable) drops the sessions, `archived` (soft-deleted, still a +/// member) archives them, `active` restores ones previously archived-by-workspace. +/// Archived and hard-deleted workspaces are absent from `user_workspaces`, so this is the +/// only way the client learns about a change made while it was away or on another device. +async fn session_workspace_status( + Extension(db): Extension, + ApiAuthed { email, .. }: ApiAuthed, + Json(req): Json, +) -> JsonResult> { + if req.workspace_ids.len() > 1000 { + return Err(Error::BadRequest( + "Too many workspace ids (max 1000)".to_string(), + )); + } + let rows = sqlx::query!( + "SELECT req.id AS \"id!\", + (CASE + WHEN usr.email IS NULL THEN 'deleted' + WHEN workspace.deleted THEN 'archived' + ELSE 'active' + END) AS \"status!\" + FROM unnest($1::text[]) AS req(id) + LEFT JOIN workspace ON workspace.id = req.id + LEFT JOIN usr ON usr.workspace_id = workspace.id AND usr.email = $2", + &req.workspace_ids[..], + email, + ) + .fetch_all(&db) + .await?; + let statuses = rows.into_iter().map(|r| (r.id, r.status)).collect(); + Ok(Json(statuses)) +} + pub async fn check_w_id_conflict<'c>(tx: &mut Transaction<'c, Postgres>, w_id: &str) -> Result<()> { if w_id == "global" { return Err(windmill_common::error::Error::BadRequest( @@ -3694,6 +3738,30 @@ async fn check_fork_w_id_conflict(db: &DB, w_id: &str) -> Result<()> { } } +/// A fork id is reusable: it is freed when a fork is deleted and can be claimed +/// again under the same name. `workspace_diff` and `skip_workspace_diff_tally` +/// are keyed by workspace id with no FK cascade, so a freshly created fork could +/// inherit cached diff state from a previous occupant of its id — a stale skip +/// row suppresses comparison entirely, and stale workspace_diff rows produce a +/// spurious "changes not visible" warning that hides the deploy button. Clear +/// both so a new fork always starts with clean diff state, regardless of how the +/// id was freed. +async fn purge_stale_fork_diff_state(db: &DB, fork_id: &str) -> Result<()> { + sqlx::query!( + "DELETE FROM workspace_diff WHERE source_workspace_id = $1 OR fork_workspace_id = $1", + fork_id + ) + .execute(db) + .await?; + sqlx::query!( + "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1", + fork_id + ) + .execute(db) + .await?; + Ok(()) +} + lazy_static::lazy_static! { pub static ref CREATE_WORKSPACE_REQUIRE_SUPERADMIN: bool = { @@ -3877,10 +3945,15 @@ async fn create_workspace( Ok(format!("Created workspace {}", &nw.id)) } +// `authed_email` is the forker's email — `clone_drafts` only carries this +// user's per-user drafts (and the legacy NULL-email workspace draft, if any) +// across, since other users aren't added to the fork's `usr` table and +// their drafts would dangle as orphans. async fn clone_workspace_data( tx: &mut Transaction<'_, Postgres>, source_workspace_id: &str, target_workspace_id: &str, + authed_email: &str, ) -> Result<()> { // Clone workspace settings (merge with existing basic settings) update_workspace_settings(tx, source_workspace_id, target_workspace_id).await?; @@ -3921,6 +3994,14 @@ async fn clone_workspace_data( // Clone raw apps clone_raw_apps(tx, source_workspace_id, target_workspace_id).await?; + // Clone the forker's own per-user drafts (plus the legacy NULL-email + // workspace draft, if any) so they keep their pending edits in the + // fork. Other users' drafts are intentionally NOT cloned — they don't + // own a `usr` row in the fork (see `clone_workspace_full`) so their + // drafts would dangle and the home-page `draft_users` aggregate would + // surface them as duplicate legacy entries. + clone_drafts(tx, source_workspace_id, target_workspace_id, authed_email).await?; + // Clone workspace runnable dependencies and dependency map clone_workspace_runnable_dependencies(tx, source_workspace_id, target_workspace_id).await?; @@ -4397,7 +4478,7 @@ async fn clone_scripts( r#"INSERT INTO script ( workspace_id, hash, path, parent_hashes, summary, description, content, created_by, created_at, archived, schema, deleted, is_template, - extra_perms, lock, lock_error_logs, language, kind, tag, draft_only, + extra_perms, lock, lock_error_logs, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, timeout, delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key, @@ -4407,7 +4488,7 @@ async fn clone_scripts( SELECT $1, hash, path, parent_hashes, summary, description, content, created_by, created_at, archived, schema, deleted, is_template, - extra_perms, lock, lock_error_logs, language, kind, tag, draft_only, + extra_perms, lock, lock_error_logs, language, kind, tag, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, dedicated_worker, ws_error_handler_muted, priority, timeout, delete_after_use, delete_after_secs, restart_unless_cancelled, concurrency_key, @@ -4450,12 +4531,12 @@ async fn clone_flows( sqlx::query!( "INSERT INTO flow ( workspace_id, path, summary, description, value, edited_by, edited_at, - archived, schema, extra_perms, dependency_job, draft_only, tag, + archived, schema, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, concurrency_key, versions, on_behalf_of_email, lock_error_logs ) SELECT $2, path, summary, description, value, edited_by, edited_at, - archived, schema, extra_perms, NULL, draft_only, tag, + archived, schema, extra_perms, NULL, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, concurrency_key, ARRAY[]::bigint[], on_behalf_of_email, lock_error_logs FROM flow @@ -4536,7 +4617,7 @@ async fn clone_apps( ) -> Result> { // Get all apps from source workspace let apps = sqlx::query!( - "SELECT id, workspace_id, path, summary, policy, versions, extra_perms, draft_only, custom_path + "SELECT id, workspace_id, path, summary, policy, versions, extra_perms, custom_path FROM app WHERE workspace_id = $1", source_workspace_id @@ -4549,8 +4630,8 @@ async fn clone_apps( // Clone apps with new IDs for app in apps { let new_app_id = sqlx::query_scalar!( - "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms, draft_only, custom_path) - VALUES ($1, $2, $3, $4, $5, $6, $7, $8) + "INSERT INTO app (workspace_id, path, summary, policy, versions, extra_perms, custom_path) + VALUES ($1, $2, $3, $4, $5, $6, $7) RETURNING id", target_workspace_id, app.path, @@ -4558,7 +4639,6 @@ async fn clone_apps( app.policy, &Vec::::new(), // Start with empty versions array app.extra_perms, - app.draft_only, app.custom_path, ) .fetch_one(&mut **tx) @@ -4762,6 +4842,39 @@ async fn clone_raw_apps( Ok(()) } +/// Clone every per-user draft (and the legacy NULL-email workspace draft, +/// if present) from the parent. The fork target is empty at create time so +/// a plain INSERT is safe — no need to UPSERT against the partial unique +/// indexes (`draft_pkey_with_user` / `draft_pkey_legacy`). `id` is the +/// BIGSERIAL synthetic PK and is regenerated by the default; we don't list +/// it in the column set. `created_at` is preserved so the per-tab +/// `last_sync` baseline the editor reads (`?get_draft=true` → overlay's +/// `draft_saved_at`) lines up with the parent's timeline — otherwise the +/// fork's first POST from any open editor would race a stale `last_sync` +/// and trip the conflict modal on every cloned draft. +// Only `email = authed_email` and the legacy NULL row are cloned — see +// `clone_workspace_data` for the rationale. +async fn clone_drafts( + tx: &mut Transaction<'_, Postgres>, + source_workspace_id: &str, + target_workspace_id: &str, + authed_email: &str, +) -> Result<()> { + sqlx::query!( + "INSERT INTO draft (workspace_id, path, typ, value, created_at, email) + SELECT $2, path, typ, value, created_at, email + FROM draft + WHERE workspace_id = $1 AND (email = $3 OR email IS NULL)", + source_workspace_id, + target_workspace_id, + authed_email, + ) + .execute(&mut **tx) + .await?; + + Ok(()) +} + async fn clone_workspace_runnable_dependencies( tx: &mut Transaction<'_, Postgres>, source_workspace_id: &str, @@ -4852,6 +4965,7 @@ async fn create_workspace_fork_branch( // Fail before creating any git branch so a name conflict doesn't leave a // dangling branch on the synced repos. check_fork_w_id_conflict(&db, &nw.id).await?; + purge_stale_fork_diff_state(&db, &nw.id).await?; Ok(Json( handle_fork_branch_creation(&authed.email, &authed.username, &db, &w_id, &nw.id).await?, @@ -4996,6 +5110,7 @@ async fn create_workspace_fork( // re-using a taken (possibly archived) fork id reports the actual // conflict instead of a misleading "maximum number of workspaces" error. check_fork_w_id_conflict(&db, &nw.id).await?; + purge_stale_fork_diff_state(&db, &nw.id).await?; #[cfg(not(feature = "enterprise"))] _check_nb_of_workspaces(&db).await?; @@ -5056,7 +5171,7 @@ async fn create_workspace_fork( .await?; // Clone all data from the parent workspace using Rust implementation - clone_workspace_data(&mut tx, &parent_workspace_id, &forked_id).await?; + clone_workspace_data(&mut tx, &parent_workspace_id, &forked_id, &authed.email).await?; // Clone triggers and schedules unconditionally, always with mode='disabled' / // enabled=false. Disabled rows have no side effects (no listener @@ -5226,6 +5341,18 @@ async fn archive_workspace( ActionKind::Update, &w_id, Some(&authed.email), + Some(audit_params_refs.clone()), + ) + .await?; + // Also record under the instance-level "admins" workspace so superadmins can + // discover who archived a workspace after it becomes hidden from the UI. + audit_log( + &mut *tx, + &authed, + "workspaces.archive", + ActionKind::Update, + "admins", + Some(&w_id), Some(audit_params_refs), ) .await?; @@ -5287,11 +5414,37 @@ async fn unarchive_workspace( None, ) .await?; + // Also record under the instance-level "admins" workspace so superadmins keep + // a durable trail of who unarchived a workspace. + audit_log( + &mut *tx, + &authed, + "workspaces.unarchive", + ActionKind::Update, + "admins", + Some(&w_id), + None, + ) + .await?; tx.commit().await?; Ok(format!("Unarchived workspace {}", &w_id)) } +/// Whether the instance is configured to suppress the email notifications sent +/// when a user is invited or added to a workspace. Defaults to false (emails on). +async fn workspace_invite_emails_disabled(db: &DB) -> Result { + Ok( + windmill_common::global_settings::load_value_from_global_settings( + db, + DISABLE_WORKSPACE_INVITE_EMAILS_SETTING, + ) + .await? + .and_then(|v| v.as_bool()) + .unwrap_or(false), + ) +} + async fn invite_user( ApiAuthed { username, is_admin, .. }: ApiAuthed, Extension(db): Extension, @@ -5350,16 +5503,18 @@ async fn invite_user( tx.commit().await?; - send_email_if_possible( - &format!("Invited to Windmill's workspace: {w_id}"), - &format!( - "You have been granted access to Windmill's workspace {w_id} + if !workspace_invite_emails_disabled(&db).await? { + send_email_if_possible( + &format!("Invited to Windmill's workspace: {w_id}"), + &format!( + "You have been granted access to Windmill's workspace {w_id} If you do not have an account on {}, login with SSO or ask an admin to create an account for you.", - (**BASE_URL.load()).clone() - ), - &nu.email, - ); + (**BASE_URL.load()).clone() + ), + &nu.email, + ); + } webhook.send_instance_event(InstanceEvent::UserInvitedWorkspace { email: nu.email.clone(), @@ -5502,17 +5657,19 @@ async fn add_user( ) .await?; - send_email_if_possible( - &format!("Added to Windmill's workspace: {w_id}"), - &format!( - "You have been granted access to Windmill's workspace {w_id} by {} + if !workspace_invite_emails_disabled(&db).await? { + send_email_if_possible( + &format!("Added to Windmill's workspace: {w_id}"), + &format!( + "You have been granted access to Windmill's workspace {w_id} by {} If you do not have an account on {}, login with SSO or ask an admin to create an account for you.", - authed.email, - (**BASE_URL.load()).clone() - ), - &nu.email, - ); + authed.email, + (**BASE_URL.load()).clone() + ), + &nu.email, + ); + } webhook.send_instance_event(InstanceEvent::UserAddedWorkspace { workspace: w_id.clone(), @@ -7005,7 +7162,7 @@ async fn compare_two_apps( FROM app JOIN app_version ON app_version.id = app.versions[array_upper(app.versions, 1)] - WHERE app.workspace_id = $1 AND app.path = $2 AND COALESCE(app.draft_only, false) = false", + WHERE app.workspace_id = $1 AND app.path = $2", source_workspace_id, path ) @@ -7017,7 +7174,7 @@ async fn compare_two_apps( FROM app JOIN app_version ON app_version.id = app.versions[array_upper(app.versions, 1)] - WHERE app.workspace_id = $1 AND app.path = $2 AND COALESCE(app.draft_only, false) = false", + WHERE app.workspace_id = $1 AND app.path = $2", fork_workspace_id, path ) @@ -7463,7 +7620,7 @@ async fn get_cloud_quotas( let scripts_prunable = sqlx::query_scalar!( "SELECT COUNT(*) FROM script s WHERE s.workspace_id = $1 AND s.hash NOT IN ( SELECT DISTINCT ON (path) hash FROM script - WHERE workspace_id = $1 AND deleted = false AND draft_only IS NOT TRUE + WHERE workspace_id = $1 AND deleted = false ORDER BY path, created_at DESC )", &w_id @@ -7558,7 +7715,7 @@ async fn prune_versions( "DELETE FROM script WHERE workspace_id = $1 AND hash NOT IN ( SELECT DISTINCT ON (path) hash FROM script - WHERE workspace_id = $1 AND deleted = false AND draft_only IS NOT TRUE + WHERE workspace_id = $1 AND deleted = false ORDER BY path, created_at DESC )", ) diff --git a/backend/windmill-api-workspaces/src/workspaces_extra.rs b/backend/windmill-api-workspaces/src/workspaces_extra.rs index 55473a8861..3cb2b29f79 100644 --- a/backend/windmill-api-workspaces/src/workspaces_extra.rs +++ b/backend/windmill-api-workspaces/src/workspaces_extra.rs @@ -65,13 +65,22 @@ pub(crate) async fn change_workspace_id( old_id, rw.new_id ); - // Create new workspace with new id and name + // Create new workspace with new id and name. A fork that keeps a wm-fork- + // id must carry its parent_workspace_id over, otherwise it becomes a + // parentless "fork of nothing" with no source to compare or merge against. + // A non-fork target id means the workspace is being promoted out of a fork, + // so the parent pointer is intentionally cleared. info!("Creating new workspace row"); + let new_is_fork = rw.new_id.starts_with(WM_FORK_PREFIX); sqlx::query!( - "INSERT INTO workspace SELECT $1, $2, owner, false, premium FROM workspace WHERE id = $3", + "INSERT INTO workspace (id, name, owner, deleted, premium, parent_workspace_id) + SELECT $1, $2, owner, false, premium, + CASE WHEN $4 THEN parent_workspace_id ELSE NULL END + FROM workspace WHERE id = $3", &rw.new_id, &rw.new_name, - &old_id + &old_id, + new_is_fork ) .execute(&mut *tx) .await?; @@ -279,8 +288,8 @@ pub(crate) async fn change_workspace_id( info!("Duplicating flow table rows"); sqlx::query!( "INSERT INTO flow - (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs) - SELECT $1, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs + (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs) + SELECT $1, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, lock_error_logs FROM flow WHERE workspace_id = $2", &rw.new_id, &old_id @@ -347,6 +356,18 @@ pub(crate) async fn change_workspace_id( .execute(&mut *tx) .await?; + // Re-parent child forks: any fork whose parent_workspace_id was the old id + // must follow the renamed parent to the new id, otherwise it is left + // pointing at the soft-deleted old shell (whose data has moved here). + info!("Re-parenting child forks to the new workspace id"); + sqlx::query!( + "UPDATE workspace SET parent_workspace_id = $1 WHERE parent_workspace_id = $2", + &rw.new_id, + &old_id + ) + .execute(&mut *tx) + .await?; + info!("Updating workspace_protection_rule table"); sqlx::query!( "UPDATE workspace_protection_rule SET workspace_id = $1 WHERE workspace_id = $2", @@ -745,6 +766,18 @@ pub(crate) async fn delete_workspace( sqlx::query!("DELETE FROM v2_job_queue WHERE workspace_id = $1", &w_id) .execute(&mut *tx) .await?; + // dispatch_event / flow_conversation_message / zombie_job_counter no longer cascade from + // v2_job (see migration drop_v2_job_side_table_cascades); delete them before v2_job so the + // workspace's jobs leave no orphan side rows. One round-trip, scanning v2_job once. + sqlx::query!( + "WITH ids AS (SELECT id FROM v2_job WHERE workspace_id = $1), + _de AS (DELETE FROM dispatch_event WHERE workspace_id = $1), + _fc AS (DELETE FROM flow_conversation_message WHERE job_id IN (SELECT id FROM ids)) + DELETE FROM zombie_job_counter WHERE job_id IN (SELECT id FROM ids)", + &w_id + ) + .execute(&mut *tx) + .await?; sqlx::query!("DELETE FROM v2_job WHERE workspace_id = $1", &w_id) .execute(&mut *tx) .await?; @@ -868,17 +901,39 @@ pub(crate) async fn delete_workspace( .execute(&mut *tx) .await?; + // workspace_diff and skip_workspace_diff_tally are keyed by workspace id with no + // FK cascade. A fork id is reused when a fork is deleted and recreated under the + // same name, so leaving these rows behind leaks the previous fork's cached diff + // verdicts onto the new fork — causing a spurious "changes not visible" warning + // that hides the deploy button. + sqlx::query!( + "DELETE FROM workspace_diff WHERE source_workspace_id = $1 OR fork_workspace_id = $1", + &w_id + ) + .execute(&mut *tx) + .await?; + + sqlx::query!( + "DELETE FROM skip_workspace_diff_tally WHERE workspace_id = $1", + &w_id + ) + .execute(&mut *tx) + .await?; + sqlx::query!("DELETE FROM workspace WHERE id = $1", &w_id) .execute(&mut *tx) .await?; + // Record under the instance-level "admins" workspace. The per-workspace audit + // rows are deleted along with the workspace, so this instance-level entry is the + // only durable, superadmin-discoverable record of who deleted the workspace. audit_log( &mut *tx, &authed, "workspaces.delete", ActionKind::Delete, - &w_id, - Some(&authed.email), + "admins", + Some(&w_id), None, ) .await?; diff --git a/backend/windmill-api/Cargo.toml b/backend/windmill-api/Cargo.toml index 94e72e7c4a..6ce463a5aa 100644 --- a/backend/windmill-api/Cargo.toml +++ b/backend/windmill-api/Cargo.toml @@ -43,7 +43,7 @@ mcp = ["dep:windmill-mcp", "windmill-mcp/server", "windmill-mcp/auth", "windmill bedrock = ["windmill-ai/bedrock"] python = ["windmill-dep-map/python", "dep:windmill-parser-py", "dep:windmill-parser-py-imports", "windmill-api-scripts/python", "windmill-api-configs/python", "windmill-api-agent-workers?/python", "windmill-trigger/python", "windmill-common/python"] no_auth = ["windmill-api-auth/no_auth", "windmill-store/no_auth", "windmill-api-users/no_auth"] -quickjs = ["windmill-jseval/quickjs"] +quickjs = ["windmill-jseval/quickjs", "windmill-queue/quickjs"] [dependencies] windmill-ai = { workspace = true, default-features = false } diff --git a/backend/windmill-api/docs_snapshot/README.md b/backend/windmill-api/docs_snapshot/README.md new file mode 100644 index 0000000000..09a1ac6615 --- /dev/null +++ b/backend/windmill-api/docs_snapshot/README.md @@ -0,0 +1,19 @@ +# Vendored docs snapshot + +`llms.txt.gz` (curated page index) and `llms-full.txt.gz` (full corpus, every docs +page concatenated and delimited by `Source:` lines) are a gzipped snapshot of +`https://www.windmill.dev/llms.txt` and `/llms-full.txt`. + +They are embedded into the binary by `../src/docs/corpus.rs` (`include_bytes!`) and +decompressed/parsed once at first use. This lets in-product docs search +(`GET /api/docs/search`, `GET /api/docs/page`) — used by the AI chat, the MCP +`searchDocs`/`readDocsPage` tools, and the `wmill docs` CLI — work with **no +runtime network egress**, including on air-gapped instances. + +The tradeoff is staleness: the snapshot is pinned to whatever was published when +`fetch.sh` was last run. Refresh on each release: + +```bash +./fetch.sh # re-downloads and re-gzips both files +git add llms.txt.gz llms-full.txt.gz +``` diff --git a/backend/windmill-api/docs_snapshot/fetch.sh b/backend/windmill-api/docs_snapshot/fetch.sh new file mode 100755 index 0000000000..bf24d42f7b --- /dev/null +++ b/backend/windmill-api/docs_snapshot/fetch.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +# Refresh the vendored documentation snapshot embedded into windmill-api. +# +# The backend self-hosts the docs corpus (see ../src/docs/) so docs search works +# with no runtime egress. This snapshot is pinned to whatever was published on +# windmill.dev when this script was last run — re-run it on each release to keep +# the in-product docs search reasonably fresh, then commit the updated *.gz. +set -euo pipefail + +DOCS_ORIGIN="${DOCS_ORIGIN:-https://www.windmill.dev}" +DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +fetch() { + local name="$1" + echo "Fetching ${DOCS_ORIGIN}/${name} ..." + curl -fSL "${DOCS_ORIGIN}/${name}" -o "${DIR}/${name}" + # -9 max compression; -n omit the original name/timestamp so the artifact is + # reproducible and diffs only when the docs actually change. + gzip -9 -n -c "${DIR}/${name}" > "${DIR}/${name}.gz" + rm -f "${DIR}/${name}" + echo " wrote ${name}.gz ($(wc -c < "${DIR}/${name}.gz") bytes)" +} + +fetch "llms.txt" +fetch "llms-full.txt" +echo "Done. Commit the updated *.gz files." diff --git a/backend/windmill-api/docs_snapshot/llms-full.txt.gz b/backend/windmill-api/docs_snapshot/llms-full.txt.gz new file mode 100644 index 0000000000..f5420b1cb5 Binary files /dev/null and b/backend/windmill-api/docs_snapshot/llms-full.txt.gz differ diff --git a/backend/windmill-api/docs_snapshot/llms.txt.gz b/backend/windmill-api/docs_snapshot/llms.txt.gz new file mode 100644 index 0000000000..abdd2b0b27 Binary files /dev/null and b/backend/windmill-api/docs_snapshot/llms.txt.gz differ diff --git a/backend/windmill-api/openapi-deref.json b/backend/windmill-api/openapi-deref.json index 00727e626f..e511578821 100644 --- a/backend/windmill-api/openapi-deref.json +++ b/backend/windmill-api/openapi-deref.json @@ -1,7 +1,7 @@ { "openapi": "3.0.3", "info": { - "version": "1.713.1", + "version": "1.740.0", "title": "Windmill API", "contact": { "name": "Windmill Team", @@ -174,52 +174,124 @@ } } }, - "/inkeep": { - "post": { - "summary": "query Windmill AI documentation assistant (EE only)", - "operationId": "queryDocumentation", + "/docs/search": { + "get": { + "summary": "Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more.", + "operationId": "searchDocs", "x-mcp-tool": true, "tags": [ "documentation" ], - "requestBody": { - "description": "query to send to the AI documentation assistant", - "required": true, - "content": { - "application/json": { - "schema": { - "type": "object", - "properties": { - "query": { - "type": "string", - "description": "The documentation query to send to the AI assistant" - } - }, - "required": [ - "query" - ] - } + "parameters": [ + { + "name": "query", + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better.", + "in": "query", + "required": true, + "schema": { + "type": "string" } } - }, + ], "responses": { "200": { - "description": "AI documentation assistant response", + "description": "matching documentation pages", "content": { "application/json": { "schema": { "type": "object", - "description": "Response from Inkeep service" + "properties": { + "text": { + "type": "string", + "description": "Model-ready rendering of the results" + }, + "results": { + "type": "array", + "items": { + "type": "object", + "properties": { + "url": { + "type": "string" + }, + "title": { + "type": "string" + }, + "score": { + "type": "integer" + }, + "snippets": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "url", + "title", + "score", + "snippets" + ] + } + } + }, + "required": [ + "text", + "results" + ] } } } + } + } + } + }, + "/docs/page": { + "get": { + "summary": "Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section.", + "operationId": "readDocsPage", + "x-mcp-tool": true, + "tags": [ + "documentation" + ], + "parameters": [ + { + "name": "url", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted.", + "in": "query", + "required": true, + "schema": { + "type": "string" + } }, - "403": { - "description": "Enterprise Edition required", + { + "name": "section", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page.", + "in": "query", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "documentation page content", "content": { - "text/plain": { + "application/json": { "schema": { - "type": "string" + "type": "object", + "properties": { + "text": { + "type": "string" + }, + "source_url": { + "type": "string" + } + }, + "required": [ + "text", + "source_url" + ] } } } @@ -1531,6 +1603,56 @@ } } }, + "/workspaces/session_workspace_status": { + "post": { + "summary": "get the lifecycle status of workspaces referenced by client-side sessions", + "operationId": "getSessionWorkspaceStatus", + "tags": [ + "workspace" + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "workspace_ids": { + "type": "array", + "items": { + "type": "string" + } + } + }, + "required": [ + "workspace_ids" + ] + } + } + } + }, + "responses": { + "200": { + "description": "map of workspace id to status (active, archived, or deleted)", + "content": { + "application/json": { + "schema": { + "type": "object", + "additionalProperties": { + "type": "string", + "enum": [ + "active", + "archived", + "deleted" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/workspaces/get_as_superadmin": { "get": { "summary": "get workspace as super admin (require to be super admin)", @@ -2492,6 +2614,10 @@ "type": "integer", "format": "int64" }, + "s3_not_found": { + "type": "integer", + "format": "int64" + }, "orphans_scanned": { "type": "integer", "format": "int64" @@ -2593,6 +2719,124 @@ } } }, + "/settings/audit_logs_s3_backfill": { + "post": { + "summary": "start an opt-in historical backfill of audit logs to object storage", + "operationId": "runAuditLogsS3Backfill", + "tags": [ + "setting" + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "from": { + "type": "string", + "format": "date-time", + "description": "inclusive lower bound of the window to export" + }, + "to": { + "type": "string", + "format": "date-time", + "description": "exclusive upper bound of the window to export" + } + }, + "required": [ + "from", + "to" + ] + } + } + } + }, + "responses": { + "202": { + "description": "backfill started" + } + } + } + }, + "/settings/audit_logs_s3_backfill_status": { + "get": { + "summary": "get status of the audit-log object-store historical backfill", + "operationId": "getAuditLogsS3BackfillStatus", + "tags": [ + "setting" + ], + "responses": { + "200": { + "description": "current backfill status (null if never run)", + "content": { + "application/json": { + "schema": { + "nullable": true, + "type": "object", + "properties": { + "running": { + "type": "boolean" + }, + "started_at": { + "type": "string", + "format": "date-time" + }, + "finished_at": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "phase": { + "type": "string" + }, + "from": { + "type": "string", + "format": "date-time" + }, + "to": { + "type": "string", + "format": "date-time" + }, + "rows_written": { + "type": "integer", + "format": "int64" + }, + "objects_written": { + "type": "integer", + "format": "int64" + }, + "last_ts": { + "type": "string", + "format": "date-time", + "nullable": true + }, + "errors": { + "type": "integer", + "format": "int64" + }, + "last_error": { + "type": "string", + "nullable": true + } + }, + "required": [ + "running", + "started_at", + "phase", + "from", + "to", + "rows_written", + "objects_written", + "errors" + ] + } + } + } + } + } + } + }, "/settings/send_stats": { "post": { "summary": "send stats", @@ -7879,6 +8123,54 @@ } } }, + "/users/tokens/update_label/{token_prefix}": { + "post": { + "summary": "update label of an existing token (owner only)", + "operationId": "updateTokenLabel", + "tags": [ + "user" + ], + "parameters": [ + { + "name": "token_prefix", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "requestBody": { + "description": "new label (null or omitted = no label)", + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "label": { + "type": "string", + "nullable": true + } + } + } + } + } + }, + "responses": { + "200": { + "description": "label updated", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, "/users/tokens/list": { "get": { "summary": "list token", @@ -8202,6 +8494,9 @@ "schema": { "type": "boolean" } + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -8210,7 +8505,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/ListableVariable" + "allOf": [ + { + "$ref": "#/components/schemas/ListableVariable" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -8350,6 +8652,15 @@ "schema": { "type": "string" } + }, + { + "name": "include_draft_only", + "description": "When true, append per-user draft variables whose path has no\ndeployed variable. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge.\n", + "in": "query", + "required": false, + "schema": { + "type": "boolean" + } } ], "responses": { @@ -9633,9 +9944,13 @@ "type": "string", "description": "OAuth client secret for resource-level credentials (client_credentials flow only)" }, + "cc_instance": { + "type": "string", + "description": "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side (client_credentials flow only). The token URL is never caller-supplied." + }, "cc_token_url": { "type": "string", - "description": "OAuth token URL override for resource-level authentication (client_credentials flow only)" + "description": "Bring-your-own token endpoint override (client_credentials flow only). Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; ignored/rejected on the shared-instance path." }, "mcp_server_url": { "type": "string", @@ -9672,7 +9987,7 @@ } } }, - "/oauth/connect_client_credentials/{client}": { + "/w/{workspace}/oauth/connect_client_credentials/{client}": { "post": { "summary": "connect OAuth using client credentials", "operationId": "connectClientCredentials", @@ -9680,6 +9995,9 @@ "oauth" ], "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, { "name": "client", "in": "path", @@ -9706,21 +10024,21 @@ }, "cc_client_id": { "type": "string", - "description": "OAuth client ID for resource-level authentication" + "description": "OAuth client ID. Omit to use the credentials configured on the provider's instance OAuth entry." }, "cc_client_secret": { "type": "string", - "description": "OAuth client secret for resource-level authentication" + "description": "OAuth client secret. Omit to use the credentials configured on the provider's instance OAuth entry." + }, + "cc_instance": { + "type": "string", + "description": "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side. The token URL is never caller-supplied." }, "cc_token_url": { "type": "string", - "description": "OAuth token URL override for resource-level authentication" + "description": "Bring-your-own token endpoint override. Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; rejected on the shared-instance path." } - }, - "required": [ - "cc_client_id", - "cc_client_secret" - ] + } } } } @@ -9933,7 +10251,23 @@ "schema": { "type": "array", "items": { - "type": "string" + "type": "object", + "properties": { + "name": { + "type": "string" + }, + "supports_client_credentials": { + "type": "boolean" + }, + "has_shared_credentials": { + "type": "boolean" + } + }, + "required": [ + "name", + "supports_client_credentials", + "has_shared_credentials" + ] } } } @@ -9982,6 +10316,10 @@ "items": { "type": "string" } + }, + "client_credentials_configured": { + "type": "boolean", + "description": "The instance OAuth entry carries shared client-credentials, so the connect dialog can skip the bring-your-own form and run the exchange server-side" } } } @@ -10264,6 +10602,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -10272,7 +10613,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/Resource" + "allOf": [ + { + "$ref": "#/components/schemas/ListableResource" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -10507,6 +10855,15 @@ "schema": { "type": "string" } + }, + { + "name": "include_draft_only", + "description": "When true, append per-user draft resources whose path has\nno deployed resource. Synthesized rows carry\n`draft_only: true`.\n", + "in": "query", + "required": false, + "schema": { + "type": "boolean" + } } ], "responses": { @@ -11509,6 +11866,32 @@ } } }, + "/apps_u/embed_token_by_custom_path/{custom_path}": { + "get": { + "summary": "get app embed token by custom path", + "operationId": "getAppEmbedTokenByCustomPath", + "tags": [ + "app" + ], + "parameters": [ + { + "$ref": "#/components/parameters/CustomPath" + } + ], + "responses": { + "200": { + "description": "embed token", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/EmbedTokenResponse" + } + } + } + } + } + } + }, "/scripts/hub/get/{path}": { "get": { "summary": "get hub script content by path", @@ -12005,7 +12388,37 @@ "schema": { "type": "array", "items": { - "$ref": "#/components/schemas/Script" + "allOf": [ + { + "$ref": "#/components/schemas/Script" + }, + { + "type": "object", + "properties": { + "is_draft": { + "type": "boolean", + "description": "True when the authed user has a draft for this\nscript — either no deployed row exists at this\npath (draft-only) or the user saved a per-user\ndraft on top of the deployed row.\n" + }, + "draft_path": { + "type": "string", + "description": "User-typed path the editor has staged but not\nyet deployed. Surfaced for draft-only rows so\nthe home list can render the meaningful name\ninstead of the autogenerated\n`u/{user}/draft_{uuid}` URL path. Omitted\nwhen unchanged.\n" + }, + "draft_users": { + "description": "Workspace users (including the authed user, and\nthe legacy NULL-email row if any) who have a\nper-user draft at this path. Drives the home\npage's user-avatar circles inside the Draft\nbadge. Omitted when no drafts exist.\n", + "type": "array", + "items": { + "type": "object", + "properties": { + "username": { + "type": "string", + "nullable": true + } + } + } + } + } + } + ] } } } @@ -12043,54 +12456,94 @@ } } }, - "/w/{workspace}/drafts/create": { - "post": { - "summary": "create draft", - "operationId": "createDraft", + "/w/{workspace}/drafts/list": { + "get": { + "summary": "list every draft the current user has in this workspace, across all kinds", + "operationId": "listDrafts", "tags": [ "draft" ], "parameters": [ { "$ref": "#/components/parameters/WorkspaceId" - } - ], - "requestBody": { - "required": true, - "content": { - "application/json": { - "schema": { - "type": "object", - "properties": { - "path": { - "type": "string" - }, - "typ": { - "type": "string", - "enum": [ - "flow", - "script", - "app" - ] - }, - "value": {} - }, - "required": [ - "path", - "typ", - "enum" - ] - } + }, + { + "name": "all_users", + "in": "query", + "description": "List every draft in the workspace (all users), not just the current user's own + legacy rows. Other users' rows come back with `mine=false` (view-only).", + "schema": { + "type": "boolean" } } - }, + ], "responses": { - "201": { - "description": "draft created", + "200": { + "description": "the user's drafts", "content": { - "text/plain": { + "application/json": { "schema": { - "type": "string" + "type": "array", + "items": { + "type": "object", + "properties": { + "kind": { + "$ref": "#/components/schemas/UserDraftItemKind" + }, + "path": { + "type": "string" + }, + "summary": { + "type": "string", + "description": "Best-effort, read from the draft JSON's `summary` field when the editor shape carries one." + }, + "draft_path": { + "type": "string", + "description": "User-typed friendly path from the draft JSON's `draft_path`, when set and different from the storage path (e.g. a never-deployed item parked at `u/{user}/draft_{uuid}`)." + }, + "draft_only": { + "type": "boolean", + "description": "No deployed counterpart exists at this path — the draft is the whole item." + }, + "legacy_draft": { + "type": "boolean", + "description": "The listed draft is a legacy workspace-level row (email NULL) predating the per-user drafts migration. Only true when no per-user draft exists at this path." + }, + "created_at": { + "type": "string", + "format": "date-time" + }, + "can_write": { + "type": "boolean", + "description": "Whether the current user may deploy/discard this draft (same check the deploy/discard endpoints enforce)." + }, + "mine": { + "type": "boolean", + "description": "The row belongs to the current user (own draft or the legacy no-owner row) and is therefore actionable. Always true in the default listing; with `all_users=true`, other users' rows are false (view-only)." + }, + "draft_users": { + "description": "Draft authors at this (path, kind) — the legacy NULL-email row surfaced as a null username.\nPopulated only for the shared full-page-editor kinds (script/flow/app/raw_app); omitted for\ndrawer kinds, which keep their drafts private. Feeds the Draft badge's owner-avatar circles.\n", + "type": "array", + "items": { + "type": "object", + "properties": { + "username": { + "type": "string", + "nullable": true + } + } + } + } + }, + "required": [ + "kind", + "path", + "draft_only", + "legacy_draft", + "created_at", + "can_write", + "mine" + ] + } } } } @@ -12098,10 +12551,10 @@ } } }, - "/w/{workspace}/drafts/delete/{kind}/{path}": { - "delete": { - "summary": "delete draft", - "operationId": "deleteDraft", + "/w/{workspace}/drafts/get/{kind}/{path}": { + "get": { + "summary": "fetch a single draft's content by workspace username (or the legacy workspace-level row)", + "operationId": "getDraftForUser", "tags": [ "draft" ], @@ -12114,12 +12567,67 @@ "in": "path", "required": true, "schema": { - "type": "string", - "enum": [ - "script", - "flow", - "app" - ] + "$ref": "#/components/schemas/UserDraftItemKind" + } + }, + { + "$ref": "#/components/parameters/ScriptPath" + }, + { + "name": "username", + "in": "query", + "required": false, + "description": "Workspace username of the draft owner. Omit to fetch the legacy workspace-level (NULL email) row.", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "draft content", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "value": {}, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "value", + "created_at" + ] + } + } + } + }, + "404": { + "description": "no draft for that owner at that path" + } + } + } + }, + "/w/{workspace}/drafts/get_own/{kind}/{path}": { + "get": { + "summary": "fetch the current user's own draft content at a path (any kind)", + "operationId": "getOwnDraft", + "tags": [ + "draft" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "kind", + "in": "path", + "required": true, + "schema": { + "$ref": "#/components/schemas/UserDraftItemKind" } }, { @@ -12128,7 +12636,168 @@ ], "responses": { "200": { - "description": "draft deleted", + "description": "the user's draft content, or null when none exists", + "content": { + "application/json": { + "schema": { + "nullable": true, + "type": "object", + "properties": { + "value": {}, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "value", + "created_at" + ] + } + } + } + } + } + } + }, + "/w/{workspace}/drafts/update/{kind}/{path}": { + "post": { + "summary": "upsert (or clear) the current user's draft at a path", + "operationId": "updateDraft", + "tags": [ + "draft" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "kind", + "in": "path", + "required": true, + "schema": { + "$ref": "#/components/schemas/UserDraftItemKind" + } + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "value": { + "nullable": true, + "description": "Draft content to save. `null` (or omitted) signals a delete — the row is removed under the same conflict rules." + }, + "last_sync": { + "type": "string", + "format": "date-time", + "description": "Server timestamp of the client's last known sync for this draft. Omit on first save." + }, + "force": { + "type": "boolean", + "description": "Skip the conflict check and overwrite the server copy." + }, + "legacy": { + "type": "boolean", + "description": "Delete-only. Target the legacy workspace-level row (email NULL) instead of the current user's row. Used to discard a legacy draft from the review page." + }, + "created_at": { + "type": "string", + "format": "date-time", + "description": "Upsert-only override for the stored creation timestamp. Normal saves omit it (stamped server-side); the localStorage→DB migration passes the draft's original write time so migrated drafts keep their age." + } + } + } + } + } + }, + "responses": { + "200": { + "description": "save result", + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "status": { + "type": "string", + "enum": [ + "saved", + "conflict" + ] + }, + "current_timestamp": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "status", + "current_timestamp" + ] + } + } + } + } + } + } + }, + "/w/{workspace}/drafts/migrate_legacy/{kind}/{path}": { + "post": { + "summary": "resolve a legacy (workspace-level) draft (admin only)", + "description": "Delete a legacy draft (email NULL) or assign it to the authed admin as a per-user draft. Workspace admins / superadmins only.", + "operationId": "migrateLegacyDraft", + "tags": [ + "draft" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "kind", + "in": "path", + "required": true, + "schema": { + "$ref": "#/components/schemas/UserDraftItemKind" + } + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "properties": { + "action": { + "type": "string", + "enum": [ + "delete", + "assign_to_self" + ], + "description": "delete the legacy draft, or take ownership of it." + } + }, + "required": [ + "action" + ] + } + } + } + }, + "responses": { + "200": { + "description": "migration result", "content": { "text/plain": { "schema": { @@ -12146,9 +12815,10 @@ "description": "Creates a new script when the path does not already exist.\nCreates a new version of an existing script when called with the same path and the current `parent_hash`.\n", "operationId": "createScript", "x-mcp-tool": true, - "x-mcp-instructions": "To create a script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed.", + "x-mcp-instructions": "To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history.", "x-mcp-tool-include-fields": [ "path", + "parent_hash", "content", "language", "summary", @@ -12729,6 +13399,9 @@ "schema": { "type": "boolean" } + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -12737,7 +13410,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/Script" + "allOf": [ + { + "$ref": "#/components/schemas/Script" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -12806,35 +13486,6 @@ } } }, - "/w/{workspace}/scripts/get/draft/{path}": { - "get": { - "summary": "get script by path with draft", - "operationId": "getScriptByPathWithDraft", - "tags": [ - "script" - ], - "parameters": [ - { - "$ref": "#/components/parameters/WorkspaceId" - }, - { - "$ref": "#/components/parameters/ScriptPath" - } - ], - "responses": { - "200": { - "description": "script details", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/NewScriptWithDraft" - } - } - } - } - } - } - }, "/w/{workspace}/scripts/history/p/{path}": { "get": { "summary": "get history of a script by path", @@ -14541,6 +15192,42 @@ } } }, + "/w/{workspace}/jobs/job_view_token/{id}": { + "get": { + "summary": "mint a read-only share token for a job", + "description": "Returns a stateless `{job_id}.{hmac}` token that grants an authenticated workspace member read access to this job (and its flow subtree) via a `view_token` query param or `X-View-Token` header. Only callable by a user who can already read the job.\n", + "operationId": "getJobViewToken", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "id", + "in": "path", + "required": true, + "schema": { + "type": "string", + "format": "uuid" + } + } + ], + "responses": { + "200": { + "description": "the share read token", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, "/w/{workspace}/flows/list_paths": { "get": { "summary": "list all flow paths", @@ -14722,11 +15409,29 @@ { "type": "object", "properties": { - "has_draft": { - "type": "boolean" - }, "draft_only": { "type": "boolean" + }, + "is_draft": { + "type": "boolean", + "description": "True when the authed user has a draft for this\nflow — either no deployed row exists at this\npath (draft-only) or the user saved a per-user\ndraft on top of the deployed row.\n" + }, + "draft_path": { + "type": "string", + "description": "User-typed path the editor has staged but not\nyet deployed. Sourced from the draft JSON's\n`draft_path` field (the editor only writes it\nwhen the typed path differs from the deployed\none). Lets the home list render the meaningful\nname instead of the autogenerated\n`u/{user}/draft_{uuid}` URL path. Omitted when\nunchanged.\n" + }, + "draft_users": { + "description": "Workspace users (including the authed user, and\nthe legacy NULL-email row if any) who have a\nper-user draft at this path. Drives the home\npage's user-avatar circles inside the Draft\nbadge. Omitted when no drafts exist.\n", + "type": "array", + "items": { + "type": "object", + "properties": { + "username": { + "type": "string", + "nullable": true + } + } + } } } } @@ -14950,6 +15655,9 @@ "schema": { "type": "boolean" } + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -14958,7 +15666,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/Flow" + "allOf": [ + { + "$ref": "#/components/schemas/Flow" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -15110,52 +15825,6 @@ } } }, - "/w/{workspace}/flows/get/draft/{path}": { - "get": { - "summary": "get flow by path with draft", - "operationId": "getFlowByPathWithDraft", - "tags": [ - "flow" - ], - "parameters": [ - { - "$ref": "#/components/parameters/WorkspaceId" - }, - { - "$ref": "#/components/parameters/ScriptPath" - } - ], - "responses": { - "200": { - "description": "flow details with draft", - "content": { - "application/json": { - "schema": { - "allOf": [ - { - "$ref": "#/components/schemas/Flow" - }, - { - "type": "object", - "properties": { - "draft": { - "$ref": "#/components/schemas/Flow" - }, - "draft_created_at": { - "type": "string", - "format": "date-time", - "description": "Timestamp at which the most recent DB draft was created. Used by the frontend's UserDraft staleness check." - } - } - } - ] - } - } - } - } - } - } - }, "/w/{workspace}/flows/exists/{path}": { "get": { "summary": "exists flow by path", @@ -15224,9 +15893,6 @@ { "type": "object", "properties": { - "draft_only": { - "type": "boolean" - }, "deployment_message": { "type": "string" }, @@ -15643,6 +16309,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -15854,6 +16523,202 @@ } } }, + "/w/{workspace}/ai_skills/list": { + "get": { + "summary": "list the workspace AI chat skills (name + description only)", + "operationId": "listAiSkills", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + } + ], + "responses": { + "200": { + "description": "skill listing", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "required": [ + "name", + "description" + ], + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + } + } + } + } + } + } + } + } + } + }, + "/w/{workspace}/ai_skills/get/{name}": { + "get": { + "summary": "get a workspace AI chat skill including its instructions", + "operationId": "getAiSkill", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "skill", + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "name", + "description", + "instructions" + ], + "properties": { + "name": { + "type": "string" + }, + "description": { + "type": "string" + }, + "instructions": { + "type": "string" + } + } + } + } + } + } + } + } + }, + "/w/{workspace}/ai_skills/upload": { + "post": { + "summary": "upsert workspace AI chat skills (admin only)", + "operationId": "uploadAiSkills", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + } + ], + "requestBody": { + "required": true, + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "skills" + ], + "properties": { + "skills": { + "type": "array", + "maxItems": 50, + "items": { + "type": "object", + "required": [ + "name", + "description", + "instructions" + ], + "properties": { + "name": { + "type": "string", + "minLength": 1, + "maxLength": 64, + "pattern": "^[a-z0-9-]+$" + }, + "description": { + "type": "string", + "minLength": 1, + "maxLength": 1024 + }, + "instructions": { + "type": "string", + "minLength": 1, + "maxLength": 65536 + } + } + } + } + } + } + } + } + }, + "responses": { + "200": { + "description": "uploaded", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, + "/w/{workspace}/ai_skills/delete/{name}": { + "delete": { + "summary": "delete a workspace AI chat skill (admin only)", + "operationId": "deleteAiSkill", + "tags": [ + "workspace" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "name", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "deleted", + "content": { + "text/plain": { + "schema": { + "type": "string" + } + } + } + } + } + } + }, "/w/{workspace}/apps/get_data/v/{secretWithExtension}": { "get": { "summary": "get raw app data by", @@ -15869,6 +16734,7 @@ "name": "secretWithExtension", "in": "path", "required": true, + "description": "App version secret suffixed with the requested file type extension. Supported extensions are `.js` (JavaScript bundle), `.css` (stylesheet), and `.html` (sandboxed wrapper document).", "schema": { "type": "string" } @@ -15882,6 +16748,16 @@ "schema": { "type": "string" } + }, + "text/css": { + "schema": { + "type": "string" + } + }, + "text/html": { + "schema": { + "type": "string" + } } } } @@ -16059,9 +16935,6 @@ "policy": { "$ref": "#/components/schemas/Policy" }, - "draft_only": { - "type": "boolean" - }, "deployment_message": { "type": "string" }, @@ -16140,9 +17013,6 @@ "policy": { "$ref": "#/components/schemas/Policy" }, - "draft_only": { - "type": "boolean" - }, "deployment_message": { "type": "string" }, @@ -16245,6 +17115,17 @@ "schema": { "type": "boolean" } + }, + { + "$ref": "#/components/parameters/GetDraft" + }, + { + "name": "raw_app", + "in": "query", + "description": "When no deployed app exists at this path and `get_draft` is set,\ndisambiguates which draft kind (`raw_app` or `app`) to look up.\nIgnored when a deployed row exists.\n", + "schema": { + "type": "boolean" + } } ], "responses": { @@ -16253,7 +17134,43 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/AppWithLastVersion" + "allOf": [ + { + "$ref": "#/components/schemas/AppWithLastVersion" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] + } + } + } + } + } + } + }, + "/w/{workspace}/apps/embed_token/p/{path}": { + "get": { + "summary": "get app embed token by path", + "operationId": "getAppEmbedTokenByPath", + "tags": [ + "app" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/ScriptPath" + } + ], + "responses": { + "200": { + "description": "embed token", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/EmbedTokenResponse" } } } @@ -16290,35 +17207,6 @@ } } }, - "/w/{workspace}/apps/get/draft/{path}": { - "get": { - "summary": "get app by path with draft", - "operationId": "getAppByPathWithDraft", - "tags": [ - "app" - ], - "parameters": [ - { - "$ref": "#/components/parameters/WorkspaceId" - }, - { - "$ref": "#/components/parameters/ScriptPath" - } - ], - "responses": { - "200": { - "description": "app details with draft", - "content": { - "application/json": { - "schema": { - "$ref": "#/components/schemas/AppWithLastVersionWDraft" - } - } - } - } - } - } - }, "/w/{workspace}/apps/history/p/{path}": { "get": { "summary": "get app history by path", @@ -16492,6 +17380,40 @@ } } }, + "/w/{workspace}/apps_u/embed_token/{secret}": { + "get": { + "summary": "get app embed token by secret", + "operationId": "getAppEmbedTokenBySecret", + "tags": [ + "app" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "secret", + "in": "path", + "required": true, + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "embed token", + "content": { + "application/json": { + "schema": { + "$ref": "#/components/schemas/EmbedTokenResponse" + } + } + } + } + } + } + }, "/w/{workspace}/apps_u/public_resource/{path}": { "get": { "summary": "get public resource", @@ -17636,6 +18558,14 @@ "type": "boolean" } }, + { + "name": "timeout", + "description": "custom timeout in seconds for this preview run", + "in": "query", + "schema": { + "type": "integer" + } + }, { "$ref": "#/components/parameters/NewJobId" } @@ -18545,6 +19475,20 @@ "type": "boolean" } }, + { + "name": "status", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.", + "in": "query", + "schema": { + "type": "string", + "enum": [ + "success", + "failure", + "canceled", + "skipped" + ] + } + }, { "name": "all_workspaces", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)", @@ -18962,6 +19906,20 @@ { "$ref": "#/components/parameters/Success" }, + { + "name": "status", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.", + "in": "query", + "schema": { + "type": "string", + "enum": [ + "success", + "failure", + "canceled", + "skipped" + ] + } + }, { "$ref": "#/components/parameters/JobKinds" }, @@ -19345,6 +20303,20 @@ "type": "boolean" } }, + { + "name": "status", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.", + "in": "query", + "schema": { + "type": "string", + "enum": [ + "success", + "failure", + "canceled", + "skipped" + ] + } + }, { "name": "all_workspaces", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)", @@ -19619,6 +20591,89 @@ } } }, + "/w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}": { + "get": { + "summary": "get all logs for a flow job in a structured format", + "operationId": "getFlowAllLogsStructured", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/JobId" + } + ], + "responses": { + "200": { + "description": "structured logs of all flow steps, one entry per job", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "job_id": { + "type": "string" + }, + "label": { + "type": "string", + "description": "human-readable label describing the job's position in the flow tree" + }, + "kind": { + "type": "string", + "description": "job kind (script, flow, forloopflow, ...)" + }, + "flow_step_id": { + "type": "string", + "nullable": true + }, + "step_path": { + "type": "string", + "nullable": true, + "description": "materialized step path (e.g. \"a/b\")" + }, + "depth": { + "type": "integer", + "description": "depth in the flow tree (0 for the root flow job)" + }, + "parent_module_type": { + "type": "string", + "nullable": true, + "description": "parent module type (forloopflow, branchall, ...)" + }, + "sibling_index": { + "type": "integer", + "description": "1-based index of this job among siblings sharing the same step" + }, + "sibling_count": { + "type": "integer", + "description": "total number of siblings sharing the same step" + }, + "logs": { + "type": "string" + } + }, + "required": [ + "job_id", + "label", + "kind", + "depth", + "sibling_index", + "sibling_count", + "logs" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/jobs_u/get_completed_logs_tail/{id}": { "get": { "summary": "get completed job logs tail", @@ -20131,6 +21186,192 @@ } } }, + "/w/{workspace}/jobs_u/dispatch_events/{id}": { + "get": { + "summary": "list asset-trigger dispatch events for a producer job", + "description": "Returns the chronological log of decisions the asset-trigger dispatcher made after this producer job completed. Each row is one (subscriber, asset write) decision: `dispatched` (with `child_job_id`), `join_pending` (with `received_inputs` / `required_inputs` / `partition`), or `skipped` (with `reason`). Rows are reaped automatically when the producer's `v2_job` row is deleted by the retention sweep.\n", + "operationId": "listDispatchEvents", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "$ref": "#/components/parameters/JobId" + } + ], + "responses": { + "200": { + "description": "dispatch events for this producer job", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "subscriber_path": { + "type": "string" + }, + "asset_kind": { + "type": "string", + "enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + }, + "asset_path": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "dispatched", + "join_pending", + "skipped" + ] + }, + "child_job_id": { + "type": "string", + "format": "uuid" + }, + "partition": { + "type": "string" + }, + "received_inputs": { + "type": "integer" + }, + "required_inputs": { + "type": "integer" + }, + "debounce_s": { + "type": "integer" + }, + "reason": { + "type": "string" + }, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "subscriber_path", + "asset_kind", + "asset_path", + "outcome", + "created_at" + ] + } + } + } + } + } + } + } + }, + "/w/{workspace}/jobs/asset_dispatch_edges": { + "get": { + "summary": "list asset-cascade producer→child job edges for a folder", + "description": "Returns the `dispatched` asset-trigger edges (producer job → child job) whose subscriber lives under `path_start`. Lets a pipeline view reconstruct the cascade tree of a folder by job id and group connected runs. Visibility follows the producer job's RLS.\n", + "operationId": "listAssetDispatchEdges", + "tags": [ + "job" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "path_start", + "in": "query", + "required": true, + "description": "Folder path prefix the children live under, e.g. `f/orders/`.", + "schema": { + "type": "string" + } + }, + { + "name": "created_after", + "in": "query", + "required": false, + "description": "Only edges dispatched at/after this instant.", + "schema": { + "type": "string", + "format": "date-time" + } + } + ], + "responses": { + "200": { + "description": "asset-cascade edges for the folder", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "properties": { + "producer_job_id": { + "type": "string", + "format": "uuid" + }, + "child_job_id": { + "type": "string", + "format": "uuid", + "description": "Set for `dispatched`; absent for `join_pending` inputs." + }, + "subscriber_path": { + "type": "string" + }, + "outcome": { + "type": "string", + "enum": [ + "dispatched", + "join_pending" + ] + }, + "asset_kind": { + "type": "string", + "enum": [ + "s3object", + "resource", + "variable", + "ducklake", + "datatable", + "volume" + ] + }, + "asset_path": { + "type": "string" + }, + "created_at": { + "type": "string", + "format": "date-time" + } + }, + "required": [ + "producer_job_id", + "subscriber_path", + "outcome", + "asset_kind", + "asset_path", + "created_at" + ] + } + } + } + } + } + } + } + }, "/w/{workspace}/jobs/completed/delete/{id}": { "post": { "summary": "delete completed job (erase content but keep run id)", @@ -20827,6 +22068,10 @@ } } } + }, + "view_token": { + "type": "string", + "description": "Share-read-link token for the flow. An authenticated workspace member can append it as a `view_token` query param on the run page to read a flow they don't otherwise have access to." } } } @@ -21247,6 +22492,10 @@ "approver" ] } + }, + "view_token": { + "type": "string", + "description": "Share-read-link token for the parent flow. An authenticated workspace member can append it as a `view_token` query param on the run page to read a flow they don't otherwise have access to." } }, "required": [ @@ -21488,6 +22737,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -21496,7 +22748,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/Schedule" + "allOf": [ + { + "$ref": "#/components/schemas/Schedule" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -21618,6 +22877,15 @@ "type": "string" }, "description": "Filter by label" + }, + { + "name": "include_draft_only", + "description": "When true, append per-user draft schedules whose path has\nno deployed schedule. Synthesized rows carry\n`draft_only: true`.\n", + "in": "query", + "required": false, + "schema": { + "type": "boolean" + } } ], "responses": { @@ -21966,6 +23234,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -21974,7 +23245,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/HttpTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/HttpTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -22030,6 +23308,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -22303,6 +23584,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -22311,7 +23595,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/WebsocketTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/WebsocketTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -22367,6 +23658,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -22638,6 +23932,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -22646,7 +23943,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/KafkaTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/KafkaTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -22702,6 +24006,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -23039,6 +24346,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -23047,7 +24357,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/NatsTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/NatsTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -23103,6 +24420,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -23367,6 +24687,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -23375,7 +24698,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/SqsTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/SqsTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -23431,6 +24761,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -24145,6 +25478,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -24620,6 +25956,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -24628,7 +25967,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/MqttTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/MqttTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -24684,6 +26030,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -24948,6 +26297,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -24956,7 +26308,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/GcpTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/GcpTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -25012,6 +26371,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -25389,6 +26751,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -25397,7 +26762,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/AzureTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/AzureTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -25443,6 +26815,9 @@ "schema": { "type": "string" } + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -26302,6 +27677,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -26310,7 +27688,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/PostgresTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/PostgresTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -26366,6 +27751,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -26630,6 +28018,9 @@ }, { "$ref": "#/components/parameters/Path" + }, + { + "$ref": "#/components/parameters/GetDraft" } ], "responses": { @@ -26638,7 +28029,14 @@ "content": { "application/json": { "schema": { - "$ref": "#/components/schemas/EmailTrigger" + "allOf": [ + { + "$ref": "#/components/schemas/EmailTrigger" + }, + { + "$ref": "#/components/schemas/UserDraftOverlay" + } + ] } } } @@ -26694,6 +28092,9 @@ "type": "string" }, "description": "Filter by label" + }, + { + "$ref": "#/components/parameters/IncludeDraftOnly" } ], "responses": { @@ -27667,6 +29068,12 @@ }, "default_permissioned_as": { "$ref": "#/components/schemas/FolderDefaultPermissionedAs" + }, + "labels": { + "type": "array", + "items": { + "type": "string" + } } }, "required": [ @@ -27729,6 +29136,12 @@ }, "default_permissioned_as": { "$ref": "#/components/schemas/FolderDefaultPermissionedAs" + }, + "labels": { + "type": "array", + "items": { + "type": "string" + } } } } @@ -31494,6 +32907,20 @@ "type": "boolean" } }, + { + "name": "status", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.", + "in": "query", + "schema": { + "type": "string", + "enum": [ + "success", + "failure", + "canceled", + "skipped" + ] + } + }, { "name": "all_workspaces", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)", @@ -32289,6 +33716,241 @@ } } }, + "/w/{workspace}/assets/graph": { + "get": { + "summary": "Get the workspace-wide asset <-> runnable graph", + "operationId": "getAssetsGraph", + "tags": [ + "asset" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + }, + { + "name": "asset_kinds", + "in": "query", + "description": "Filter by asset kinds (comma-separated list)", + "schema": { + "type": "string" + } + }, + { + "name": "folder", + "in": "query", + "description": "Scope the graph to runnables in a single folder", + "schema": { + "type": "string" + } + } + ], + "responses": { + "200": { + "description": "asset graph nodes, lineage edges and trigger edges", + "content": { + "application/json": { + "schema": { + "type": "object", + "required": [ + "assets", + "runnables", + "edges", + "triggers" + ], + "properties": { + "assets": { + "type": "array", + "items": { + "type": "object", + "required": [ + "kind", + "path" + ], + "properties": { + "kind": { + "$ref": "#/components/schemas/AssetKind" + }, + "path": { + "type": "string" + } + } + } + }, + "runnables": { + "type": "array", + "items": { + "type": "object", + "required": [ + "path", + "usage_kind" + ], + "properties": { + "path": { + "type": "string" + }, + "usage_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "in_pipeline": { + "type": "boolean", + "description": "True iff the script is a pipeline member (deployed with `// pipeline`). Omitted when false." + } + } + } + }, + "edges": { + "type": "array", + "items": { + "type": "object", + "required": [ + "runnable_path", + "runnable_kind", + "asset_kind", + "asset_path" + ], + "properties": { + "runnable_path": { + "type": "string" + }, + "runnable_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "asset_kind": { + "$ref": "#/components/schemas/AssetKind" + }, + "asset_path": { + "type": "string" + }, + "access_type": { + "$ref": "#/components/schemas/AssetUsageAccessType" + } + } + } + }, + "triggers": { + "type": "array", + "items": { + "oneOf": [ + { + "type": "object", + "description": "Asset trigger edge (`// on `)", + "required": [ + "trigger_kind", + "asset_kind", + "asset_path", + "runnable_kind", + "runnable_path" + ], + "properties": { + "trigger_kind": { + "type": "string", + "enum": [ + "asset" + ] + }, + "asset_kind": { + "$ref": "#/components/schemas/AssetKind" + }, + "asset_path": { + "type": "string" + }, + "runnable_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "runnable_path": { + "type": "string" + } + } + }, + { + "type": "object", + "description": "Native trigger edge (schedule, email, kafka, ...). `path` is the trigger row's path.", + "required": [ + "trigger_kind", + "path", + "runnable_kind", + "runnable_path" + ], + "properties": { + "trigger_kind": { + "type": "string", + "enum": [ + "schedule", + "email", + "kafka", + "mqtt", + "nats", + "postgres", + "sqs", + "gcp" + ] + }, + "path": { + "type": "string" + }, + "runnable_kind": { + "$ref": "#/components/schemas/AssetUsageKind" + }, + "runnable_path": { + "type": "string" + } + } + } + ] + } + } + } + } + } + } + } + } + } + }, + "/w/{workspace}/assets/pipelines": { + "get": { + "summary": "List folders that contain at least one pipeline-member script", + "operationId": "listPipelineFolders", + "tags": [ + "asset" + ], + "parameters": [ + { + "$ref": "#/components/parameters/WorkspaceId" + } + ], + "responses": { + "200": { + "description": "folders containing pipeline scripts, with their script counts", + "content": { + "application/json": { + "schema": { + "type": "array", + "items": { + "type": "object", + "required": [ + "folder", + "script_count" + ], + "properties": { + "folder": { + "type": "string", + "description": "The folder name (without the `f/` prefix)" + }, + "script_count": { + "type": "integer", + "format": "int64", + "description": "Number of pipeline-member scripts in the folder" + } + } + } + } + } + } + } + } + } + }, "/w/{workspace}/volumes/list": { "get": { "summary": "List all volumes in the workspace", @@ -32604,6 +34266,24 @@ } }, "parameters": { + "GetDraft": { + "name": "get_draft", + "in": "query", + "required": false, + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload.", + "schema": { + "type": "boolean" + } + }, + "IncludeDraftOnly": { + "name": "include_draft_only", + "in": "query", + "required": false, + "description": "When true, append per-user draft rows whose path has no\ndeployed counterpart. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge. Gated to\nnon-operators + page 0 + no narrowing filters on the backend so\npicker callers stay deployed-only and pagination stays clean.\n", + "schema": { + "type": "boolean" + } + }, "Id": { "name": "id", "in": "path", @@ -33148,6 +34828,82 @@ } }, "schemas": { + "UserDraftOverlay": { + "type": "object", + "description": "Overlay fields added to every \"get by path\" response that accepts\nthe `get_draft` query parameter. The deployed payload is sent\nuntouched in the response body; the authed user's saved draft\nfor this path — whatever shape the editor wrote — is attached\nas the sibling `draft` field when `get_draft=true` and a draft\nexists. The frontend pairs the two to present diff / reset /\ndiscard UI; the server never merges them.\n\nWhen `no_deployed=true` there is no deployed row at this path —\nthe response body is a best-effort stand-in synthesized from\nthe draft, and only `draft` is canonical. Callers should disable\n\"diff vs deployed\" UI in that case.\n", + "properties": { + "is_draft": { + "type": "boolean" + }, + "draft_saved_at": { + "type": "string", + "format": "date-time" + }, + "no_deployed": { + "type": "boolean" + }, + "draft": { + "type": "object", + "additionalProperties": true + }, + "other_drafts_users": { + "description": "Other workspace users (and the legacy NULL-email row, if any)\nwith a saved draft at the same path. Populated only on the\nauthed user's \"get by path\" responses for kinds the editor\nsurfaces a fork banner for (script, flow, app, raw_app).\nEmpty / omitted for kinds without that UI.\n", + "type": "array", + "items": { + "type": "object", + "properties": { + "username": { + "type": "string", + "nullable": true, + "description": "Workspace username of the draft owner. `null` represents\nthe legacy workspace-level (NULL-email) row. Emails never\nleave the server.\n" + }, + "draft_saved_at": { + "type": "string", + "format": "date-time", + "description": "When this user's draft was last saved (`draft.created_at`),\nsurfaced in the fork modal as \"Last updated\".\n" + } + }, + "required": [ + "draft_saved_at" + ] + } + } + }, + "required": [ + "is_draft" + ] + }, + "UserDraftItemKind": { + "type": "string", + "description": "Closed set of item kinds a user can autosave as a draft. Mirrors the\nPostgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`.\n", + "enum": [ + "script", + "flow", + "app", + "raw_app", + "resource", + "variable", + "trigger_schedule", + "trigger_webhook", + "trigger_default_email", + "trigger_email", + "trigger_http", + "trigger_websocket", + "trigger_postgres", + "trigger_kafka", + "trigger_nats", + "trigger_mqtt", + "trigger_sqs", + "trigger_gcp", + "trigger_azure", + "trigger_poll", + "trigger_cli", + "trigger_nextcloud", + "trigger_google", + "trigger_github", + "data_pipeline" + ] + }, "OpenFlow": { "type": "object", "description": "Top-level flow definition containing metadata, configuration, and the flow structure", @@ -33354,6 +35110,10 @@ "type": "string", "nullable": true, "description": "Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised." + }, + "error_include_result": { + "type": "boolean", + "description": "When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false." } }, "required": [ @@ -34174,6 +35934,10 @@ "aiagent" ] }, + "tag": { + "type": "string", + "description": "Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)" + }, "omit_output_from_conversation": { "type": "boolean", "default": false, @@ -35367,6 +37131,9 @@ "items": { "type": "string" } + }, + "web_search_enabled": { + "type": "boolean" } }, "required": [ @@ -35633,9 +37400,6 @@ "tag": { "type": "string" }, - "has_draft": { - "type": "boolean" - }, "draft_only": { "type": "boolean" }, @@ -35723,6 +37487,13 @@ "type": "string" }, "default": [] + }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" } }, "required": [ @@ -35752,6 +37523,10 @@ "parent_hash": { "type": "string" }, + "auto_parent": { + "type": "boolean", + "description": "When true, the backend resolves the parent to the current deployed head for this path within the transaction (ignoring parent_hash), instead of failing with a \"lineage must be linear\" error when the supplied parent_hash is stale." + }, "summary": { "type": "string" }, @@ -35787,9 +37562,6 @@ "tag": { "type": "string" }, - "draft_only": { - "type": "boolean" - }, "envs": { "type": "array", "items": { @@ -35930,32 +37702,6 @@ "language" ] }, - "NewScriptWithDraft": { - "allOf": [ - { - "$ref": "#/components/schemas/NewScript" - }, - { - "type": "object", - "properties": { - "draft": { - "$ref": "#/components/schemas/NewScript" - }, - "draft_created_at": { - "type": "string", - "format": "date-time", - "description": "Timestamp at which the most recent DB draft was created. Used by the frontend's UserDraft staleness check." - }, - "hash": { - "type": "string" - } - }, - "required": [ - "hash" - ] - } - ] - }, "ScriptHistory": { "type": "object", "properties": { @@ -36182,6 +37928,9 @@ "preprocessed": { "type": "boolean" }, + "is_retry": { + "type": "boolean" + }, "worker": { "type": "string" } @@ -36339,6 +38088,9 @@ "preprocessed": { "type": "boolean" }, + "is_retry": { + "type": "boolean" + }, "worker": { "type": "string" } @@ -36846,6 +38598,12 @@ "type": "string" } }, + "folders_read": { + "type": "array", + "items": { + "type": "string" + } + }, "folders_owners": { "type": "array", "items": { @@ -36873,6 +38631,7 @@ "operator", "disabled", "folders", + "folders_read", "folders_owners" ] }, @@ -37391,6 +39150,13 @@ "type": "string" } }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" + }, "ws_specific": { "type": "boolean" }, @@ -37400,6 +39166,14 @@ }, "edited_by": { "type": "string" + }, + "draft_only": { + "description": "True when this row is a per-user draft with no deployed\nvariable at the same path. Frontend renders a \"Draft\" badge.\n", + "type": "boolean" + }, + "is_draft": { + "description": "True when the authed user has a per-user draft at this path\n(over a deployed row or a synthesized draft-only row).\nFrontend appends a `*` to the displayed name.\n", + "type": "boolean" } }, "required": [ @@ -38115,6 +39889,13 @@ "type": "string" } }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" + }, "ws_specific": { "type": "boolean" } @@ -38178,8 +39959,23 @@ "type": "string" } }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" + }, "ws_specific": { "type": "boolean" + }, + "draft_only": { + "description": "True when this row is a per-user draft with no deployed\nresource at the same path. Frontend renders a \"Draft\" badge.\n", + "type": "boolean" + }, + "is_draft": { + "description": "True when the authed user has a per-user draft at this path\n(over a deployed row or a synthesized draft-only row).\nFrontend appends a `*` to the displayed name.\n", + "type": "boolean" } }, "required": [ @@ -38384,6 +40180,21 @@ "type": "string" }, "default": [] + }, + "draft_only": { + "description": "True when this row is a per-user draft with no deployed\nschedule at the same path. Frontend renders a \"Draft\" badge.\n", + "type": "boolean" + }, + "is_draft": { + "description": "True when the authed user has a per-user draft at this path\n(over a deployed row or a synthesized draft-only row).\nFrontend appends a `*` to the displayed name.\n", + "type": "boolean" + }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" } }, "required": [ @@ -38724,7 +40535,8 @@ "gcp", "azure", "google", - "github" + "github", + "asset" ] }, "TriggerMode": { @@ -38785,6 +40597,14 @@ "type": "string" }, "default": [] + }, + "draft_only": { + "description": "True when this row is a per-user draft with no deployed\ntrigger at the same path. Set by list endpoints when\n`include_draft_only=true` synthesizes the row from the\ndraft. Frontend renders a \"Draft\" badge.\n", + "type": "boolean" + }, + "is_draft": { + "description": "True when the authed user has a per-user draft at this path\n(over a deployed row or a synthesized draft-only row).\nFrontend appends a `*` to the displayed name.\n", + "type": "boolean" } }, "required": [ @@ -41871,6 +43691,13 @@ }, "default_permissioned_as": { "$ref": "#/components/schemas/FolderDefaultPermissionedAs" + }, + "labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels set on the folder. Items inside the folder inherit them, exposed as `inherited_labels` on scripts and flows and stamped into job labels at run time.\n" } }, "required": [ @@ -42369,6 +44196,13 @@ "type": "string" }, "default": [] + }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" } }, "required": [ @@ -42548,6 +44382,10 @@ }, "on_behalf_of_email": { "type": "string" + }, + "sandbox": { + "type": "boolean", + "description": "Publisher opt-in to app sandbox isolation (alpha). When true the app is isolated from each viewer's Windmill session. When false/absent the app runs same-origin with the viewer's full session (the default, pre-isolation behavior).\n" } } }, @@ -42599,6 +44437,34 @@ "type": "string" }, "default": [] + }, + "is_draft": { + "type": "boolean", + "description": "True when the authed user has a draft for this app — either no\ndeployed row exists at this path (draft-only) or the user has\nsaved a per-user draft on top of the deployed row.\n" + }, + "draft_path": { + "type": "string", + "description": "User-typed path the editor has staged but not yet deployed.\nSourced from the draft JSON's `draft_path` field (the editor\nonly writes it when the typed path differs from the deployed\none). Lets the home list render the meaningful name instead of\nthe autogenerated `u/{user}/draft_{uuid}` URL path. Omitted\nwhen unchanged.\n" + }, + "draft_users": { + "description": "Workspace users (including the authed user, and the legacy\nNULL-email row if any) who have a per-user draft at this\npath. Drives the home page's user-avatar circles inside the\nDraft badge. Omitted when no drafts exist.\n", + "type": "array", + "items": { + "type": "object", + "properties": { + "username": { + "type": "string", + "nullable": true + } + } + } + }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" } }, "required": [ @@ -42691,6 +44557,13 @@ "type": "string" }, "default": [] + }, + "inherited_labels": { + "type": "array", + "items": { + "type": "string" + }, + "description": "Labels inherited from the parent folder, computed at read time. Read-only — edit them on the folder.\n" } }, "required": [ @@ -42780,27 +44653,6 @@ "raw_app" ] }, - "AppWithLastVersionWDraft": { - "allOf": [ - { - "$ref": "#/components/schemas/AppWithLastVersion" - }, - { - "type": "object", - "properties": { - "draft_only": { - "type": "boolean" - }, - "draft": {}, - "draft_created_at": { - "type": "string", - "format": "date-time", - "description": "Timestamp at which the most recent DB draft was created. Used by the frontend's UserDraft staleness check." - } - } - } - ] - }, "AppHistory": { "type": "object", "properties": { @@ -42815,6 +44667,44 @@ "version" ] }, + "EmbedTokenResponse": { + "type": "object", + "properties": { + "token": { + "type": "string", + "nullable": true, + "description": "Narrowly-scoped embed token for the iframe. Absent for fully anonymous or raw apps, which load without a scoped token." + }, + "expiration": { + "type": "string", + "format": "date-time", + "nullable": true, + "description": "Expiration of the embed token." + }, + "raw_app": { + "type": "boolean", + "description": "Raw apps render single-iframe and skip the opaque-viewer indirection and the embed token entirely." + }, + "sandbox": { + "type": "boolean", + "description": "Publisher opted this app into sandbox isolation. When false the viewer runs the app same-origin with its full session." + }, + "app_path": { + "type": "string", + "nullable": true, + "description": "The resolved app path; the embedder uses it to scope the app's backing localStorage per app." + }, + "workspace_id": { + "type": "string", + "nullable": true, + "description": "The resolved workspace; pairs with app_path so apps at the same path in different workspaces don't share a localStorage store." + } + }, + "required": [ + "raw_app", + "sandbox" + ] + }, "FlowVersion": { "type": "object", "properties": { @@ -44350,7 +46240,8 @@ "enum": [ "DisableDirectDeployment", "DisableWorkspaceForking", - "RestrictDeployToDeployers" + "RestrictDeployToDeployers", + "RestrictAnonymousAppDeployment" ] }, "RuleBypasserGroups": { @@ -46030,6 +47921,10 @@ "aiagent" ] }, + "tag": { + "type": "string", + "description": "Worker group tag for execution routing. If not set, the AI agent step runs on the flow's tag (default `flow`)" + }, "omit_output_from_conversation": { "type": "boolean", "default": false, @@ -46062,6 +47957,10 @@ "type": "string", "nullable": true, "description": "Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised." + }, + "error_include_result": { + "type": "boolean", + "description": "When stopping with an error (error_message set), embed the stopping step's own result inside the raised error object (as error.result) instead of discarding it. The top-level result stays { error }. Defaults to false." } }, "required": [ diff --git a/backend/windmill-api/openapi-deref.yaml b/backend/windmill-api/openapi-deref.yaml index 4ced5ef265..e17c38efaf 100644 --- a/backend/windmill-api/openapi-deref.yaml +++ b/backend/windmill-api/openapi-deref.yaml @@ -1,6 +1,6 @@ openapi: 3.0.3 info: - version: 1.713.1 + version: 1.740.0 title: Windmill API contact: name: Windmill Team @@ -146,18 +146,18 @@ paths: checks: type: object description: Detailed health checks - required: &ref_347 + required: &ref_354 - database - readiness - properties: &ref_348 + properties: &ref_355 database: type: object description: Database health status - required: &ref_349 + required: &ref_356 - healthy - latency_ms - pool - properties: &ref_350 + properties: &ref_357 healthy: type: boolean description: Whether the database is reachable @@ -168,11 +168,11 @@ paths: pool: type: object description: Database connection pool statistics - required: &ref_351 + required: &ref_358 - size - idle - max_connections - properties: &ref_352 + properties: &ref_359 size: type: integer description: Current number of connections in the pool @@ -186,13 +186,13 @@ paths: description: Workers health status nullable: true type: object - required: &ref_353 + required: &ref_360 - healthy - active_count - worker_groups - min_version - versions - properties: &ref_354 + properties: &ref_361 healthy: type: boolean description: Whether any workers are active @@ -219,10 +219,10 @@ paths: description: Job queue status nullable: true type: object - required: &ref_355 + required: &ref_362 - pending_jobs - running_jobs - properties: &ref_356 + properties: &ref_363 pending_jobs: type: integer format: int64 @@ -234,9 +234,9 @@ paths: readiness: type: object description: Server readiness status - required: &ref_357 + required: &ref_364 - healthy - properties: &ref_358 + properties: &ref_365 healthy: type: boolean description: Whether the server is ready to accept requests @@ -275,40 +275,107 @@ paths: text/plain: schema: type: string - /inkeep: - post: - summary: query Windmill AI documentation assistant (EE only) - operationId: queryDocumentation + /docs/search: + get: + summary: >- + Full-text search across the entire Windmill documentation. Provide one + or more keywords; returns the most relevant docs pages, each with its + Source URL and short matching snippets. Use this FIRST to find relevant + pages by their content (a flag, function, error message, config key or + concept). If the snippets answer the question, answer directly; + otherwise call readDocsPage with a returned Source URL to read more. + operationId: searchDocs x-mcp-tool: true tags: - documentation - requestBody: - description: query to send to the AI documentation assistant - required: true - content: - application/json: - schema: - type: object - properties: - query: - type: string - description: The documentation query to send to the AI assistant - required: - - query + parameters: + - name: query + description: >- + Keywords to search for in the documentation body, e.g. "chromium + worker tag" or "retry exponential backoff". Fewer, more distinctive + words match better. + in: query + required: true + schema: + type: string responses: '200': - description: AI documentation assistant response + description: matching documentation pages content: application/json: schema: type: object - description: Response from Inkeep service - '403': - description: Enterprise Edition required + properties: + text: + type: string + description: Model-ready rendering of the results + results: + type: array + items: + type: object + properties: + url: + type: string + title: + type: string + score: + type: integer + snippets: + type: array + items: + type: string + required: + - url + - title + - score + - snippets + required: + - text + - results + /docs/page: + get: + summary: >- + Fetch the markdown of a single Windmill documentation page. Provide the + `url` of a page found via searchDocs (its Source URL). If the page is + large, this returns its list of section headings instead of the full + content; call again with the `section` argument set to one of those + headings to read that section. + operationId: readDocsPage + x-mcp-tool: true + tags: + - documentation + parameters: + - name: url + description: >- + The docs page to read, as a Source URL returned by searchDocs (e.g. + https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. + /docs/core_concepts/jobs) is also accepted. + in: query + required: true + schema: + type: string + - name: section + description: >- + Optional. A heading title from the page outline to read just that + section instead of the full page. + in: query + schema: + type: string + responses: + '200': + description: documentation page content content: - text/plain: + application/json: schema: - type: string + type: object + properties: + text: + type: string + source_url: + type: string + required: + - text + - source_url /openapi.yaml: get: summary: get openapi yaml spec @@ -337,7 +404,7 @@ paths: - name: id in: path required: true - schema: &ref_79 + schema: &ref_84 type: integer responses: '200': @@ -488,24 +555,24 @@ paths: - name: before description: filter on started before (inclusive) timestamp in: query - schema: &ref_299 + schema: &ref_306 type: string format: date-time - name: after description: filter on created after (exclusive) timestamp in: query - schema: &ref_300 + schema: &ref_307 type: string format: date-time - name: username description: filter on exact username of user in: query - schema: &ref_308 + schema: &ref_315 type: string - name: operation description: filter on exact or prefix name of operation in: query - schema: &ref_309 + schema: &ref_316 type: string - name: operations in: query @@ -520,12 +587,12 @@ paths: - name: resource description: filter on exact or prefix name of resource in: query - schema: &ref_310 + schema: &ref_317 type: string - name: action_kind description: filter on type of operation in: query - schema: &ref_311 + schema: &ref_318 type: string enum: - Create @@ -562,12 +629,12 @@ paths: application/json: schema: type: object - properties: &ref_397 + properties: &ref_405 email: type: string password: type: string - required: &ref_398 + required: &ref_406 - email - password responses: @@ -749,6 +816,10 @@ paths: type: array items: type: string + folders_read: + type: array + items: + type: string folders_owners: type: array items: @@ -757,7 +828,7 @@ paths: nullable: true allOf: - type: object - properties: &ref_394 + properties: &ref_402 source: type: string enum: @@ -775,7 +846,7 @@ paths: description: >- The instance group name (when source is 'instance_group') - required: &ref_395 + required: &ref_403 - source is_service_account: type: boolean @@ -788,6 +859,7 @@ paths: - operator - disabled - folders + - folders_read - folders_owners /w/{workspace}/users/update/{username}: post: @@ -813,7 +885,7 @@ paths: application/json: schema: type: object - properties: &ref_399 + properties: &ref_407 is_admin: type: boolean operator: @@ -1187,7 +1259,7 @@ paths: type: array items: type: object - properties: &ref_413 + properties: &ref_421 jwt_hash: type: integer format: int64 @@ -1210,7 +1282,7 @@ paths: last_used_at: type: string format: date-time - required: &ref_414 + required: &ref_422 - jwt_hash - email - username @@ -1342,7 +1414,7 @@ paths: type: array items: type: object - properties: &ref_400 + properties: &ref_408 label: type: string scopes: @@ -1351,7 +1423,7 @@ paths: type: string expiration: type: string - required: &ref_401 + required: &ref_409 - label - scopes description: Tokens owned by this user (will be deleted) @@ -1395,7 +1467,7 @@ paths: application/json: schema: type: object - properties: &ref_402 + properties: &ref_410 reassign_to: type: string description: 'Target for reassignment: ''u/{username}'' or ''f/{folder}''' @@ -1409,7 +1481,7 @@ paths: type: boolean default: true description: Whether to also remove the user from the workspace - required: &ref_403 + required: &ref_411 - reassign_to responses: '200': @@ -1428,7 +1500,7 @@ paths: on success. summary: type: object - properties: &ref_404 + properties: &ref_412 scripts_reassigned: type: integer flows_reassigned: @@ -1445,7 +1517,7 @@ paths: type: integer drafts_deleted: type: integer - required: &ref_405 + required: &ref_413 - scripts_reassigned - flows_reassigned - apps_reassigned @@ -1475,12 +1547,12 @@ paths: application/json: schema: type: object - properties: &ref_406 + properties: &ref_414 workspaces: type: array items: type: object - properties: &ref_408 + properties: &ref_416 workspace_id: type: string username: @@ -1489,11 +1561,11 @@ paths: type: object properties: *ref_12 required: *ref_13 - required: &ref_409 + required: &ref_417 - workspace_id - username - preview - required: &ref_407 + required: &ref_415 - workspaces /users/offboard/{email}: post: @@ -1515,12 +1587,12 @@ paths: application/json: schema: type: object - properties: &ref_410 + properties: &ref_418 reassignments: type: object additionalProperties: type: object - properties: &ref_411 + properties: &ref_419 reassign_to: type: string description: 'Target: ''u/{username}'' or ''f/{folder}''' @@ -1529,7 +1601,7 @@ paths: description: >- Required when reassign_to is a folder. Username to use as permissioned_as. - required: &ref_412 + required: &ref_420 - reassign_to description: Map of workspace_id to reassignment config delete_user: @@ -1589,7 +1661,7 @@ paths: application/json: schema: type: array - items: &ref_562 + items: &ref_565 type: object properties: workspace_id: @@ -1699,7 +1771,7 @@ paths: application/json: schema: type: object - properties: &ref_502 + properties: &ref_506 email: type: string workspaces: @@ -1774,9 +1846,43 @@ paths: - username - color - disabled - required: &ref_503 + required: &ref_507 - email - workspaces + /workspaces/session_workspace_status: + post: + summary: >- + get the lifecycle status of workspaces referenced by client-side + sessions + operationId: getSessionWorkspaceStatus + tags: + - workspace + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + workspace_ids: + type: array + items: + type: string + required: + - workspace_ids + responses: + '200': + description: map of workspace id to status (active, archived, or deleted) + content: + application/json: + schema: + type: object + additionalProperties: + type: string + enum: + - active + - archived + - deleted /w/{workspace}/workspaces/get_as_superadmin: get: summary: get workspace as super admin (require to be super admin) @@ -1836,7 +1942,7 @@ paths: application/json: schema: type: object - properties: &ref_504 + properties: &ref_508 id: type: string name: @@ -1845,7 +1951,7 @@ paths: type: string color: type: string - required: &ref_505 + required: &ref_509 - id - name responses: @@ -2021,7 +2127,7 @@ paths: properties: &ref_24 logs: type: object - properties: &ref_470 + properties: &ref_474 super_admin: type: string enum: &ref_21 @@ -2548,6 +2654,9 @@ paths: s3_deleted: type: integer format: int64 + s3_not_found: + type: integer + format: int64 orphans_scanned: type: integer format: int64 @@ -2618,6 +2727,90 @@ paths: - bootstrapping - last_run_exported - updated_at + /settings/audit_logs_s3_backfill: + post: + summary: start an opt-in historical backfill of audit logs to object storage + operationId: runAuditLogsS3Backfill + tags: + - setting + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + from: + type: string + format: date-time + description: inclusive lower bound of the window to export + to: + type: string + format: date-time + description: exclusive upper bound of the window to export + required: + - from + - to + responses: + '202': + description: backfill started + /settings/audit_logs_s3_backfill_status: + get: + summary: get status of the audit-log object-store historical backfill + operationId: getAuditLogsS3BackfillStatus + tags: + - setting + responses: + '200': + description: current backfill status (null if never run) + content: + application/json: + schema: + nullable: true + type: object + properties: + running: + type: boolean + started_at: + type: string + format: date-time + finished_at: + type: string + format: date-time + nullable: true + phase: + type: string + from: + type: string + format: date-time + to: + type: string + format: date-time + rows_written: + type: integer + format: int64 + objects_written: + type: integer + format: int64 + last_ts: + type: string + format: date-time + nullable: true + errors: + type: integer + format: int64 + last_error: + type: string + nullable: true + required: + - running + - started_at + - phase + - from + - to + - rows_written + - objects_written + - errors /settings/send_stats: post: summary: send stats @@ -2828,11 +3021,11 @@ paths: type: array items: type: object - properties: &ref_545 + properties: &ref_548 name: type: string value: {} - required: &ref_546 + required: &ref_549 - name - value /settings/instance_config: @@ -2930,9 +3123,9 @@ paths: application/json: schema: type: object - required: &ref_370 + required: &ref_377 - keys - properties: &ref_371 + properties: &ref_378 keys: type: array items: @@ -3049,11 +3242,11 @@ paths: type: array items: type: object - required: &ref_368 + required: &ref_375 - workspace_id - path - error - properties: &ref_369 + properties: &ref_376 workspace_id: type: string description: Workspace ID where the secret is located @@ -4348,13 +4541,13 @@ paths: application/json: schema: type: object - required: &ref_554 + required: &ref_557 - all_ahead_items_visible - all_behind_items_visible - skipped_comparison - diffs - summary - properties: &ref_555 + properties: &ref_558 all_ahead_items_visible: type: boolean description: >- @@ -4375,7 +4568,7 @@ paths: description: List of differences found between workspaces items: type: object - required: &ref_556 + required: &ref_559 - kind - path - ahead @@ -4383,7 +4576,7 @@ paths: - has_changes - exists_in_source - exists_in_fork - properties: &ref_557 + properties: &ref_560 kind: type: string enum: @@ -4428,7 +4621,7 @@ paths: summary: description: Summary statistics of the comparison type: object - required: &ref_558 + required: &ref_561 - total_diffs - total_ahead - total_behind @@ -4442,7 +4635,7 @@ paths: - schedules_changed - triggers_changed - conflicts - properties: &ref_559 + properties: &ref_562 total_diffs: type: integer description: Total number of items with differences @@ -4639,12 +4832,12 @@ paths: type: array items: type: object - properties: &ref_531 + properties: &ref_534 pattern: type: string allow: type: string - required: &ref_532 + required: &ref_535 - pattern - allow secondary_storage: @@ -4777,7 +4970,7 @@ paths: auto_invite: type: object description: Configuration for auto-inviting users to the workspace - properties: &ref_359 + properties: &ref_366 enabled: type: boolean default: false @@ -4818,14 +5011,16 @@ paths: type: object additionalProperties: type: object - properties: &ref_378 + properties: &ref_385 resource_path: type: string models: type: array items: type: string - required: &ref_379 + web_search_enabled: + type: boolean + required: &ref_386 - resource_path - models default_model: @@ -4871,7 +5066,7 @@ paths: error_handler: type: object description: Configuration for the workspace error handler - properties: &ref_360 + properties: &ref_367 path: type: string description: Path to the error handler script or flow @@ -4888,7 +5083,7 @@ paths: success_handler: type: object description: Configuration for the workspace success handler - properties: &ref_361 + properties: &ref_368 path: type: string description: Path to the success handler script or flow @@ -5192,7 +5387,7 @@ paths: type: array items: type: object - properties: &ref_507 + properties: &ref_511 importer_path: type: string importer_kind: @@ -5206,7 +5401,7 @@ paths: items: type: string nullable: true - required: &ref_508 + required: &ref_512 - importer_path - importer_kind /w/{workspace}/workspaces/get_imports/{importer_path}: @@ -5264,13 +5459,13 @@ paths: type: array items: type: object - properties: &ref_509 + properties: &ref_513 imported_path: type: string count: type: integer format: int64 - required: &ref_510 + required: &ref_514 - imported_path - count /w/{workspace}/workspaces/get_dependency_map: @@ -5293,7 +5488,7 @@ paths: type: array items: type: object - properties: &ref_506 + properties: &ref_510 workspace_id: type: string nullable: true @@ -5825,7 +6020,7 @@ paths: type: array items: type: object - properties: &ref_380 + properties: &ref_387 provider: type: string enum: *ref_51 @@ -5833,7 +6028,7 @@ paths: type: array items: type: string - required: &ref_381 + required: &ref_388 - provider - models default_model: @@ -5924,10 +6119,10 @@ paths: Request body for editing the workspace error handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: &ref_362 + oneOf: &ref_369 - type: object description: New grouped format for editing error handler - properties: &ref_363 + properties: &ref_370 path: type: string description: Path to the error handler script or flow @@ -5945,7 +6140,7 @@ paths: description: >- Legacy flat format for editing error handler (deprecated, use new format) - properties: &ref_364 + properties: &ref_371 error_handler: type: string description: Path to the error handler script or flow @@ -5984,10 +6179,10 @@ paths: Request body for editing the workspace success handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: &ref_365 + oneOf: &ref_372 - type: object description: New grouped format for editing success handler - properties: &ref_366 + properties: &ref_373 path: type: string description: Path to the success handler script or flow @@ -5999,7 +6194,7 @@ paths: description: >- Legacy flat format for editing success handler (deprecated, use new format) - properties: &ref_367 + properties: &ref_374 success_handler: type: string description: Path to the success handler script or flow @@ -6116,10 +6311,10 @@ paths: type: array items: type: object - required: &ref_523 + required: &ref_526 - datatable_name - schemas - properties: &ref_524 + properties: &ref_527 datatable_name: type: string schemas: @@ -6161,10 +6356,10 @@ paths: type: array items: type: object - required: &ref_525 + required: &ref_528 - datatable_name - schemas - properties: &ref_526 + properties: &ref_529 datatable_name: type: string schemas: @@ -6209,12 +6404,12 @@ paths: application/json: schema: type: object - required: &ref_527 + required: &ref_530 - datatable_name - schema_name - table_name - columns - properties: &ref_528 + properties: &ref_531 datatable_name: type: string schema_name: @@ -6975,7 +7170,7 @@ paths: type: array items: type: object - properties: &ref_396 + properties: &ref_404 email: type: string executions: @@ -7038,7 +7233,7 @@ paths: type: array items: type: object - properties: &ref_518 + properties: &ref_522 name: type: string description: @@ -7048,7 +7243,7 @@ paths: type: array items: type: object - properties: &ref_516 + properties: &ref_520 value: type: string label: @@ -7058,11 +7253,11 @@ paths: nullable: true requires_resource_path: type: boolean - required: &ref_517 + required: &ref_521 - value - label - requires_resource_path - required: &ref_519 + required: &ref_523 - name - scopes /users/tokens/create: @@ -7078,7 +7273,7 @@ paths: application/json: schema: type: object - properties: &ref_415 + properties: &ref_423 label: type: string expiration: @@ -7119,7 +7314,7 @@ paths: application/json: schema: type: object - properties: &ref_416 + properties: &ref_424 label: type: string expiration: @@ -7129,7 +7324,7 @@ paths: type: string workspace_id: type: string - required: &ref_417 + required: &ref_425 - impersonate_email responses: '201': @@ -7189,6 +7384,36 @@ paths: text/plain: schema: type: string + /users/tokens/update_label/{token_prefix}: + post: + summary: update label of an existing token (owner only) + operationId: updateTokenLabel + tags: + - user + parameters: + - name: token_prefix + in: path + required: true + schema: + type: string + requestBody: + description: new label (null or omitted = no label) + required: true + content: + application/json: + schema: + type: object + properties: + label: + type: string + nullable: true + responses: + '200': + description: label updated + content: + text/plain: + schema: + type: string /users/tokens/list: get: summary: list token @@ -7217,7 +7442,7 @@ paths: type: array items: type: object - properties: &ref_102 + properties: &ref_109 label: type: string expiration: @@ -7241,7 +7466,7 @@ paths: type: string read_only: type: boolean - required: &ref_103 + required: &ref_110 - token_prefix - created_at - last_used_at @@ -7297,7 +7522,7 @@ paths: application/json: schema: type: object - properties: &ref_420 + properties: &ref_428 path: type: string description: The path to the variable @@ -7326,7 +7551,7 @@ paths: type: string ws_specific: type: boolean - required: &ref_421 + required: &ref_429 - path - value - is_secret @@ -7448,7 +7673,7 @@ paths: application/json: schema: type: object - properties: &ref_422 + properties: &ref_430 path: type: string description: The path to the variable @@ -7504,59 +7729,178 @@ paths: in: query schema: type: boolean + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: &ref_77 + type: boolean responses: '200': description: variable content: application/json: schema: - type: object - properties: &ref_61 - workspace_id: - type: string - path: - type: string - value: - type: string - is_secret: - type: boolean - description: - type: string - account: - type: integer - is_oauth: - type: boolean - extra_perms: - type: object - additionalProperties: - type: boolean - is_expired: - type: boolean - refresh_error: - type: string - is_linked: - type: boolean - is_refreshed: - type: boolean - expires_at: - type: string - format: date-time - labels: - type: array - items: - type: string - ws_specific: - type: boolean - edited_at: - type: string - format: date-time - edited_by: - type: string - required: &ref_62 - - workspace_id - - path - - is_secret - - extra_perms + allOf: + - type: object + properties: &ref_61 + workspace_id: + type: string + path: + type: string + value: + type: string + is_secret: + type: boolean + description: + type: string + account: + type: integer + is_oauth: + type: boolean + extra_perms: + type: object + additionalProperties: + type: boolean + is_expired: + type: boolean + refresh_error: + type: string + is_linked: + type: boolean + is_refreshed: + type: boolean + expires_at: + type: string + format: date-time + labels: + type: array + items: + type: string + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at + read time. Read-only — edit them on the folder. + ws_specific: + type: boolean + edited_at: + type: string + format: date-time + edited_by: + type: string + draft_only: + description: > + True when this row is a per-user draft with no + deployed + + variable at the same path. Frontend renders a "Draft" + badge. + type: boolean + is_draft: + description: > + True when the authed user has a per-user draft at this + path + + (over a deployed row or a synthesized draft-only row). + + Frontend appends a `*` to the displayed name. + type: boolean + required: &ref_62 + - workspace_id + - path + - is_secret + - extra_perms + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: &ref_78 + is_draft: + type: boolean + draft_saved_at: + type: string + format: date-time + no_deployed: + type: boolean + draft: + type: object + additionalProperties: true + other_drafts_users: + description: > + Other workspace users (and the legacy NULL-email row, + if any) + + with a saved draft at the same path. Populated only on + the + + authed user's "get by path" responses for kinds the + editor + + surfaces a fork banner for (script, flow, app, + raw_app). + + Empty / omitted for kinds without that UI. + type: array + items: + type: object + properties: + username: + type: string + nullable: true + description: > + Workspace username of the draft owner. `null` + represents + + the legacy workspace-level (NULL-email) row. + Emails never + + leave the server. + draft_saved_at: + type: string + format: date-time + description: > + When this user's draft was last saved + (`draft.created_at`), + + surfaced in the fork modal as "Last updated". + required: + - draft_saved_at + required: &ref_79 + - is_draft /w/{workspace}/variables/get_value/{path}: get: summary: get variable value @@ -7662,6 +8006,15 @@ paths: required: false schema: type: string + - name: include_draft_only + description: | + When true, append per-user draft variables whose path has no + deployed variable. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. + in: query + required: false + schema: + type: boolean responses: '200': description: variable list @@ -7693,7 +8046,7 @@ paths: type: array items: type: object - properties: &ref_418 + properties: &ref_426 name: type: string value: @@ -7702,7 +8055,7 @@ paths: type: string is_custom: type: boolean - required: &ref_419 + required: &ref_427 - name - value - description @@ -7889,12 +8242,12 @@ paths: description: >- A workspace protection rule defining restrictions and bypass permissions - required: &ref_565 + required: &ref_568 - name - rules - bypass_groups - bypass_users - properties: &ref_566 + properties: &ref_569 name: type: string description: Unique name for the protection rule @@ -7906,10 +8259,11 @@ paths: description: Configuration of protection restrictions items: &ref_64 type: string - enum: &ref_567 + enum: &ref_570 - DisableDirectDeployment - DisableWorkspaceForking - RestrictDeployToDeployers + - RestrictAnonymousAppDeployment bypass_groups: type: array description: Groups that can bypass this ruleset @@ -8066,11 +8420,11 @@ paths: type: array items: type: object - required: &ref_568 + required: &ref_571 - username - email - is_admin - properties: &ref_569 + properties: &ref_572 username: type: string email: @@ -8125,10 +8479,10 @@ paths: type: array items: type: object - required: &ref_570 + required: &ref_573 - username - email - properties: &ref_571 + properties: &ref_574 username: type: string email: @@ -8749,11 +9103,21 @@ paths: description: >- OAuth client secret for resource-level credentials (client_credentials flow only) + cc_instance: + type: string + description: >- + Instance name for built-in providers whose + client-credentials token URL is instance-templated; + substituted into the fixed-host registry template + server-side (client_credentials flow only). The token URL is + never caller-supplied. cc_token_url: type: string description: >- - OAuth token URL override for resource-level authentication - (client_credentials flow only) + Bring-your-own token endpoint override (client_credentials + flow only). Only honored together with + cc_client_id/cc_client_secret and mutually exclusive with + cc_instance; ignored/rejected on the shared-instance path. mcp_server_url: type: string description: MCP server URL for MCP OAuth token refresh @@ -8775,13 +9139,17 @@ paths: text/plain: schema: type: string - /oauth/connect_client_credentials/{client}: + /w/{workspace}/oauth/connect_client_credentials/{client}: post: summary: connect OAuth using client credentials operationId: connectClientCredentials tags: - oauth parameters: + - name: workspace + in: path + required: true + schema: *ref_4 - name: client in: path description: OAuth client name @@ -8802,16 +9170,28 @@ paths: type: string cc_client_id: type: string - description: OAuth client ID for resource-level authentication + description: >- + OAuth client ID. Omit to use the credentials configured on + the provider's instance OAuth entry. cc_client_secret: type: string - description: OAuth client secret for resource-level authentication + description: >- + OAuth client secret. Omit to use the credentials configured + on the provider's instance OAuth entry. + cc_instance: + type: string + description: >- + Instance name for built-in providers whose + client-credentials token URL is instance-templated; + substituted into the fixed-host registry template + server-side. The token URL is never caller-supplied. cc_token_url: type: string - description: OAuth token URL override for resource-level authentication - required: - - cc_client_id - - cc_client_secret + description: >- + Bring-your-own token endpoint override. Only honored + together with cc_client_id/cc_client_secret and mutually + exclusive with cc_instance; rejected on the shared-instance + path. responses: '200': description: OAuth token response @@ -8962,7 +9342,18 @@ paths: schema: type: array items: - type: string + type: object + properties: + name: + type: string + supports_client_credentials: + type: boolean + has_shared_credentials: + type: boolean + required: + - name + - supports_client_credentials + - has_shared_credentials /oauth/get_connect/{client}: get: summary: get oauth connect @@ -8994,6 +9385,12 @@ paths: type: array items: type: string + client_credentials_configured: + type: boolean + description: >- + The instance OAuth entry carries shared + client-credentials, so the connect dialog can skip the + bring-your-own form and run the exchange server-side /teams/activities: post: summary: send update to Microsoft Teams activity @@ -9051,7 +9448,7 @@ paths: application/json: schema: type: object - properties: &ref_427 + properties: &ref_435 path: type: string description: The path to the resource @@ -9068,7 +9465,7 @@ paths: type: string ws_specific: type: boolean - required: &ref_428 + required: &ref_436 - path - value - resource_type @@ -9159,7 +9556,7 @@ paths: application/json: schema: type: object - properties: &ref_429 + properties: &ref_437 path: type: string description: The path to the resource @@ -9230,44 +9627,123 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: resource content: application/json: schema: - type: object - properties: &ref_430 - workspace_id: - type: string - path: - type: string - description: - type: string - resource_type: - type: string - value: {} - is_oauth: - type: boolean - extra_perms: - type: object - additionalProperties: - type: boolean - created_by: - type: string - edited_at: - type: string - format: date-time - labels: - type: array - items: - type: string - ws_specific: - type: boolean - required: &ref_431 - - path - - resource_type - - is_oauth + allOf: + - type: object + properties: &ref_80 + workspace_id: + type: string + path: + type: string + description: + type: string + resource_type: + type: string + value: {} + is_oauth: + type: boolean + extra_perms: + type: object + additionalProperties: + type: boolean + is_expired: + type: boolean + refresh_error: + type: string + is_linked: + type: boolean + is_refreshed: + type: boolean + account: + type: number + created_by: + type: string + edited_at: + type: string + format: date-time + labels: + type: array + items: + type: string + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at + read time. Read-only — edit them on the folder. + ws_specific: + type: boolean + draft_only: + description: > + True when this row is a per-user draft with no + deployed + + resource at the same path. Frontend renders a "Draft" + badge. + type: boolean + is_draft: + description: > + True when the authed user has a per-user draft at this + path + + (over a deployed row or a synthesized draft-only row). + + Frontend appends a `*` to the displayed name. + type: boolean + required: &ref_81 + - path + - resource_type + - is_oauth + - is_linked + - is_refreshed + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/resources/get_value_interpolated/{path}: get: summary: get resource interpolated (variables and resources are fully unrolled) @@ -9438,6 +9914,15 @@ paths: required: false schema: type: string + - name: include_draft_only + description: | + When true, append per-user draft resources whose path has + no deployed resource. Synthesized rows carry + `draft_only: true`. + in: query + required: false + schema: + type: boolean responses: '200': description: resource list @@ -9447,49 +9932,8 @@ paths: type: array items: type: object - properties: &ref_432 - workspace_id: - type: string - path: - type: string - description: - type: string - resource_type: - type: string - value: {} - is_oauth: - type: boolean - extra_perms: - type: object - additionalProperties: - type: boolean - is_expired: - type: boolean - refresh_error: - type: string - is_linked: - type: boolean - is_refreshed: - type: boolean - account: - type: number - created_by: - type: string - edited_at: - type: string - format: date-time - labels: - type: array - items: - type: string - ws_specific: - type: boolean - required: &ref_433 - - path - - resource_type - - is_oauth - - is_linked - - is_refreshed + properties: *ref_80 + required: *ref_81 /w/{workspace}/resources/list_search: get: summary: list resources for search @@ -9565,7 +10009,7 @@ paths: - name: name in: path required: true - schema: &ref_273 + schema: &ref_280 type: string responses: '200': @@ -9602,7 +10046,7 @@ paths: application/json: schema: type: object - properties: &ref_77 + properties: &ref_82 workspace_id: type: string name: @@ -9619,7 +10063,7 @@ paths: type: string is_fileset: type: boolean - required: &ref_78 + required: &ref_83 - name responses: '201': @@ -9698,7 +10142,7 @@ paths: application/json: schema: type: object - properties: &ref_434 + properties: &ref_438 schema: {} description: type: string @@ -9733,8 +10177,8 @@ paths: application/json: schema: type: object - properties: *ref_77 - required: *ref_78 + properties: *ref_82 + required: *ref_83 /w/{workspace}/resources/type/exists/{path}: get: summary: does resource_type exists @@ -9778,8 +10222,8 @@ paths: type: array items: type: object - properties: *ref_77 - required: *ref_78 + properties: *ref_82 + required: *ref_83 /w/{workspace}/resources/type/listnames: get: summary: list resource_types names @@ -10061,7 +10505,7 @@ paths: - name: id in: path required: true - schema: *ref_79 + schema: *ref_84 responses: '200': description: flow @@ -10075,7 +10519,7 @@ paths: description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: &ref_120 + properties: &ref_125 summary: type: string description: Short description of what this flow does @@ -10087,7 +10531,7 @@ paths: description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: &ref_610 + properties: &ref_613 modules: type: array description: >- @@ -10098,7 +10542,7 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: &ref_82 + properties: &ref_87 id: type: string description: >- @@ -10112,14 +10556,14 @@ paths: Can be a script (inline or referenced), subflow, loop, branch, or special module type - oneOf: &ref_88 + oneOf: &ref_93 - type: object description: >- Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms - properties: &ref_323 + properties: &ref_330 input_transforms: type: object description: >- @@ -10133,14 +10577,14 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: &ref_80 + oneOf: &ref_85 - type: object description: >- Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource' - properties: &ref_136 + properties: &ref_142 value: description: >- The static value. For resources, use @@ -10149,7 +10593,7 @@ paths: type: string enum: - static - required: &ref_137 + required: &ref_143 - type - type: object description: >- @@ -10159,7 +10603,7 @@ paths: inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value - properties: &ref_84 + properties: &ref_89 expr: type: string description: >- @@ -10172,7 +10616,7 @@ paths: type: string enum: - javascript - required: &ref_85 + required: &ref_90 - expr - type - type: object @@ -10180,14 +10624,14 @@ paths: Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter. - properties: &ref_86 + properties: &ref_91 type: type: string enum: - ai - required: &ref_87 + required: &ref_92 - type - discriminator: &ref_81 + discriminator: &ref_86 propertyName: type mapping: static: >- @@ -10297,7 +10741,7 @@ paths: - r - w - rw - required: &ref_324 + required: &ref_331 - type - content - language @@ -10307,7 +10751,7 @@ paths: Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code - properties: &ref_325 + properties: &ref_332 input_transforms: type: object description: >- @@ -10321,8 +10765,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 path: type: string description: >- @@ -10347,7 +10791,7 @@ paths: description: >- If true, this script is a trigger that can start the flow - required: &ref_326 + required: &ref_333 - type - path - input_transforms @@ -10356,7 +10800,7 @@ paths: Reference to an existing flow by path. Use this to call another flow as a subflow - properties: &ref_327 + properties: &ref_334 input_transforms: type: object description: >- @@ -10370,8 +10814,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 path: type: string description: >- @@ -10381,7 +10825,7 @@ paths: type: string enum: - flow - required: &ref_328 + required: &ref_335 - type - path - input_transforms @@ -10394,7 +10838,7 @@ paths: 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations - properties: &ref_329 + properties: &ref_336 modules: type: array description: >- @@ -10406,8 +10850,8 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: &ref_83 + properties: *ref_87 + required: &ref_88 - value - id iterator: @@ -10416,8 +10860,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 skip_failures: type: boolean description: >- @@ -10439,11 +10883,11 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 squash: type: boolean - required: &ref_330 + required: &ref_337 - modules - iterator - skip_failures @@ -10455,7 +10899,7 @@ paths: condition after each iteration. Use stop_after_if on modules to control loop termination - properties: &ref_331 + properties: &ref_338 modules: type: array description: >- @@ -10467,8 +10911,8 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 skip_failures: type: boolean description: >- @@ -10489,11 +10933,11 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 squash: type: boolean - required: &ref_332 + required: &ref_339 - modules - skip_failures - type @@ -10505,7 +10949,7 @@ paths: one with a true expression runs. If no branches match, the default branch executes - properties: &ref_333 + properties: &ref_340 branches: type: array description: >- @@ -10536,8 +10980,8 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 required: - modules - expr @@ -10551,13 +10995,13 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 type: type: string enum: - branchone - required: &ref_334 + required: &ref_341 - branches - default - type @@ -10568,7 +11012,7 @@ paths: BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently - properties: &ref_335 + properties: &ref_342 branches: type: array description: >- @@ -10595,8 +11039,8 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 required: - modules type: @@ -10609,7 +11053,7 @@ paths: If true, all branches execute concurrently. If false, they execute sequentially - required: &ref_336 + required: &ref_343 - branches - type - type: object @@ -10617,7 +11061,7 @@ paths: Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder - properties: &ref_337 + properties: &ref_344 type: type: string enum: @@ -10627,7 +11071,7 @@ paths: description: >- If true, marks this as a flow identity (special handling) - required: &ref_338 + required: &ref_345 - type - type: object description: >- @@ -10635,7 +11079,7 @@ paths: accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task - properties: &ref_339 + properties: &ref_346 input_transforms: type: object description: >- @@ -10647,22 +11091,22 @@ paths: Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined - oneOf: &ref_341 + oneOf: &ref_348 - type: object description: >- Static provider configuration passed directly to the AI agent - properties: &ref_595 + properties: &ref_598 value: type: object description: >- Complete AI provider configuration with resource reference and model selection - properties: &ref_593 + properties: &ref_596 kind: type: string description: Supported AI provider types - enum: &ref_316 + enum: &ref_323 - openai - azure_openai - anthropic @@ -10685,7 +11129,7 @@ paths: description: >- Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro') - required: &ref_594 + required: &ref_597 - kind - resource - model @@ -10693,7 +11137,7 @@ paths: type: string enum: - static - required: &ref_596 + required: &ref_599 - type - value - type: object @@ -10704,16 +11148,16 @@ paths: inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value - properties: *ref_84 - required: *ref_85 + properties: *ref_89 + required: *ref_90 - type: object description: >- Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter. - properties: *ref_86 - required: *ref_87 - discriminator: &ref_342 + properties: *ref_91 + required: *ref_92 + discriminator: &ref_349 propertyName: type mapping: static: >- @@ -10728,8 +11172,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Output format type. @@ -10743,8 +11187,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: >- The user's prompt/message to the AI agent. Supports variable interpolation @@ -10756,8 +11200,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: >- System instructions that guide the AI's behavior, persona, and response style. @@ -10769,8 +11213,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Boolean. If true, stream the AI response incrementally. @@ -10783,27 +11227,27 @@ paths: Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined - oneOf: &ref_343 + oneOf: &ref_350 - type: object description: >- Static memory configuration passed directly to the AI agent - properties: &ref_601 + properties: &ref_604 value: description: Conversation memory configuration - oneOf: &ref_599 + oneOf: &ref_602 - type: object description: No conversation memory/context - properties: &ref_317 + properties: &ref_324 kind: type: string enum: - 'off' - required: &ref_318 + required: &ref_325 - kind - type: object description: Automatic context management - properties: &ref_319 + properties: &ref_326 kind: type: string enum: @@ -10818,11 +11262,11 @@ paths: description: >- Identifier for persistent memory across agent invocations - required: &ref_320 + required: &ref_327 - kind - type: object description: Explicit message history - properties: &ref_321 + properties: &ref_328 kind: type: string enum: @@ -10832,7 +11276,7 @@ paths: items: type: object description: A single message in conversation history - properties: &ref_597 + properties: &ref_600 role: type: string enum: @@ -10841,13 +11285,13 @@ paths: - system content: type: string - required: &ref_598 + required: &ref_601 - role - content - required: &ref_322 + required: &ref_329 - kind - messages - discriminator: &ref_600 + discriminator: &ref_603 propertyName: kind mapping: 'off': '#/components/schemas/MemoryOff' @@ -10857,7 +11301,7 @@ paths: type: string enum: - static - required: &ref_602 + required: &ref_605 - type - value - type: object @@ -10868,16 +11312,16 @@ paths: inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value - properties: *ref_84 - required: *ref_85 + properties: *ref_89 + required: *ref_90 - type: object description: >- Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter. - properties: *ref_86 - required: *ref_87 - discriminator: &ref_344 + properties: *ref_91 + required: *ref_92 + discriminator: &ref_351 propertyName: type mapping: static: >- @@ -10892,8 +11336,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > JSON Schema object defining structured output format. Used when you need the AI @@ -10914,8 +11358,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Array of file references (images or PDFs) for the AI agent. @@ -10932,8 +11376,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Integer. Maximum number of tokens the AI will generate in its response. @@ -10947,8 +11391,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Float. Controls randomness/creativity of responses. @@ -10967,8 +11411,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Number. Limits how many times the agent can loop through reasoning and tool use. @@ -10990,7 +11434,7 @@ paths: A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool - properties: &ref_345 + properties: &ref_352 id: type: string description: >- @@ -11008,12 +11452,12 @@ paths: The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference - oneOf: &ref_608 + oneOf: &ref_611 - description: >- A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module - allOf: &ref_603 + allOf: &ref_606 - type: object properties: tool_type: @@ -11027,8 +11471,8 @@ paths: step. Can be a script (inline or referenced), subflow, loop, branch, or special module type - oneOf: *ref_88 - discriminator: &ref_89 + oneOf: *ref_93 + discriminator: &ref_94 propertyName: type mapping: rawscript: '#/components/schemas/schemas-RawScript' @@ -11046,7 +11490,7 @@ paths: Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers - properties: &ref_604 + properties: &ref_607 tool_type: type: string enum: @@ -11070,7 +11514,7 @@ paths: MCP server items: type: string - required: &ref_605 + required: &ref_608 - tool_type - resource_path - type: object @@ -11078,26 +11522,32 @@ paths: A tool implemented as a websearch tool. The AI can call this like any other websearch tool - properties: &ref_606 + properties: &ref_609 tool_type: type: string enum: - websearch - required: &ref_607 + required: &ref_610 - tool_type - discriminator: &ref_609 + discriminator: &ref_612 propertyName: tool_type mapping: flowmodule: '#/components/schemas/FlowModuleTool' mcp: '#/components/schemas/McpToolValue' websearch: '#/components/schemas/WebsearchToolValue' - required: &ref_346 + required: &ref_353 - id - value type: type: string enum: - aiagent + tag: + type: string + description: >- + Worker group tag for execution routing. + If not set, the AI agent step runs on + the flow's tag (default `flow`) omit_output_from_conversation: type: boolean default: false @@ -11111,15 +11561,15 @@ paths: description: >- If true, the agent can execute multiple tool calls in parallel - required: &ref_340 + required: &ref_347 - tools - type - input_transforms - discriminator: *ref_89 + discriminator: *ref_94 stop_after_if: description: Early termination condition for a module type: object - properties: &ref_90 + properties: &ref_95 skip_if_stopped: type: boolean description: >- @@ -11143,13 +11593,22 @@ paths: If empty string, a default error message is used. If null or omitted, no error is raised. - required: &ref_91 + error_include_result: + type: boolean + description: >- + When stopping with an error + (error_message set), embed the stopping + step's own result inside the raised + error object (as error.result) instead + of discarding it. The top-level result + stays { error }. Defaults to false. + required: &ref_96 - expr stop_after_all_iters_if: description: Early termination condition for a module type: object - properties: *ref_90 - required: *ref_91 + properties: *ref_95 + required: *ref_96 skip_if: type: object description: >- @@ -11169,8 +11628,8 @@ paths: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 cache_ttl: type: number description: >- @@ -11183,8 +11642,8 @@ paths: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 delete_after_secs: type: integer description: >- @@ -11243,8 +11702,8 @@ paths: a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 self_approval_disabled: type: boolean description: >- @@ -11275,7 +11734,7 @@ paths: Retry configuration for failed module executions type: object - properties: &ref_315 + properties: &ref_322 constant: type: object description: >- @@ -11316,14 +11775,14 @@ paths: description: >- Conditional retry based on error or result - properties: &ref_195 + properties: &ref_201 expr: type: string description: >- JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables - required: &ref_196 + required: &ref_202 - expr debouncing: description: >- @@ -11360,21 +11819,21 @@ paths: description: >- Maximum number of debounces before forced execution - required: *ref_83 + required: *ref_88 failure_module: description: >- A single step in a flow. Can be a script, subflow, loop, or branch type: object - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 preprocessor_module: description: >- A single step in a flow. Can be a script, subflow, loop, or branch type: object - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 same_worker: type: boolean description: >- @@ -11464,7 +11923,7 @@ paths: description: >- A sticky note attached to a flow for documentation and annotation - properties: &ref_145 + properties: &ref_151 id: type: string description: Unique identifier for the note @@ -11523,7 +11982,7 @@ paths: description: >- For group notes, the IDs of nodes contained within this group - required: &ref_146 + required: &ref_152 - id - text - color @@ -11543,7 +12002,7 @@ paths: collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id. - properties: &ref_147 + properties: &ref_153 summary: type: string description: Display name for this group @@ -11569,10 +12028,10 @@ paths: color: type: string description: Color for the group in the flow editor - required: &ref_148 + required: &ref_154 - start_id - end_id - required: &ref_611 + required: &ref_614 - modules schema: type: object @@ -11586,7 +12045,7 @@ paths: description: >- The flow will be run with the permissions of the user with this email. - required: &ref_121 + required: &ref_126 - summary - value /apps/hub/list: @@ -11639,7 +12098,7 @@ paths: - name: id in: path required: true - schema: *ref_79 + schema: *ref_84 responses: '200': description: app @@ -11669,7 +12128,7 @@ paths: - name: id in: path required: true - schema: *ref_79 + schema: *ref_84 responses: '200': description: raw app @@ -11699,7 +12158,7 @@ paths: - name: custom_path in: path required: true - schema: &ref_133 + schema: &ref_97 type: string responses: '200': @@ -11709,7 +12168,7 @@ paths: schema: allOf: - type: object - properties: &ref_128 + properties: &ref_133 id: type: integer workspace_id: @@ -11730,7 +12189,7 @@ paths: value: {} policy: type: object - properties: &ref_127 + properties: &ref_132 triggerables: type: object additionalProperties: @@ -11762,6 +12221,14 @@ paths: type: string on_behalf_of_email: type: string + sandbox: + type: boolean + description: > + Publisher opt-in to app sandbox isolation (alpha). + When true the app is isolated from each viewer's + Windmill session. When false/absent the app runs + same-origin with the viewer's full session (the + default, pre-isolation behavior). execution_mode: type: string enum: @@ -11783,7 +12250,7 @@ paths: items: type: string default: [] - required: &ref_129 + required: &ref_134 - id - workspace_id - path @@ -11800,6 +12267,64 @@ paths: properties: workspace_id: type: string + /apps_u/embed_token_by_custom_path/{custom_path}: + get: + summary: get app embed token by custom path + operationId: getAppEmbedTokenByCustomPath + tags: + - app + parameters: + - name: custom_path + in: path + required: true + schema: *ref_97 + responses: + '200': + description: embed token + content: + application/json: + schema: + type: object + properties: &ref_135 + token: + type: string + nullable: true + description: >- + Narrowly-scoped embed token for the iframe. Absent for + fully anonymous or raw apps, which load without a scoped + token. + expiration: + type: string + format: date-time + nullable: true + description: Expiration of the embed token. + raw_app: + type: boolean + description: >- + Raw apps render single-iframe and skip the opaque-viewer + indirection and the embed token entirely. + sandbox: + type: boolean + description: >- + Publisher opted this app into sandbox isolation. When + false the viewer runs the app same-origin with its full + session. + app_path: + type: string + nullable: true + description: >- + The resolved app path; the embedder uses it to scope the + app's backing localStorage per app. + workspace_id: + type: string + nullable: true + description: >- + The resolved workspace; pairs with app_path so apps at the + same path in different workspaces don't share a + localStorage store. + required: &ref_136 + - raw_app + - sandbox /scripts/hub/get/{path}: get: summary: get hub script content by path @@ -11810,7 +12335,7 @@ paths: - name: path in: path required: true - schema: &ref_92 + schema: &ref_98 type: string responses: '200': @@ -11829,7 +12354,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script details @@ -11860,7 +12385,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script pick recorded @@ -11923,7 +12448,7 @@ paths: type: number kind: type: string - enum: &ref_93 + enum: &ref_99 - script - failure - trigger @@ -11994,7 +12519,7 @@ paths: type: string kind: type: string - enum: *ref_93 + enum: *ref_99 score: type: number required: @@ -12056,7 +12581,7 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: &ref_118 + schema: &ref_123 type: boolean - name: created_by description: >- @@ -12064,7 +12589,7 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: &ref_119 + schema: &ref_124 type: string - name: path_start description: mask to filter matching starting path @@ -12204,183 +12729,225 @@ paths: schema: type: array items: - type: object - properties: &ref_99 - workspace_id: - type: string - hash: - type: string - path: - type: string - parent_hashes: - type: array - description: > - The first element is the direct parent of the script, - the second is the parent of the first, etc - items: - type: string - summary: - type: string - description: - type: string - content: - type: string - created_by: - type: string - created_at: - type: string - format: date-time - archived: - type: boolean - schema: - type: object - deleted: - type: boolean - is_template: - type: boolean - extra_perms: - type: object - additionalProperties: - type: boolean - lock: - type: string - lock_error_logs: - type: string - language: - type: string - enum: &ref_94 - - python3 - - deno - - go - - bash - - powershell - - postgresql - - mysql - - bigquery - - snowflake - - mssql - - oracledb - - graphql - - nativets - - bun - - php - - rust - - ansible - - csharp - - nu - - java - - ruby - - rlang - - duckdb - - bunnative - kind: - type: string - enum: - - script - - failure - - trigger - - command - - approval - - preprocessor - starred: - type: boolean - tag: - type: string - has_draft: - type: boolean - draft_only: - type: boolean - envs: - type: array - items: - type: string - concurrent_limit: - type: integer - concurrency_time_window_s: - type: integer - concurrency_key: - type: string - debounce_key: - type: string - debounce_delay_s: - type: integer - debounce_args_to_accumulate: - type: array - items: - type: string - max_total_debouncing_time: - type: integer - max_total_debounces_amount: - type: integer - cache_ttl: - type: number - dedicated_worker: - type: boolean - ws_error_handler_muted: - type: boolean - priority: - type: integer - restart_unless_cancelled: - type: boolean - timeout: - type: integer - delete_after_secs: - type: integer - description: >- - If set, delete the job's args, result and logs after - this many seconds following job completion - visible_to_runner_only: - type: boolean - auto_kind: - type: string - codebase: - type: string - has_preprocessor: - type: boolean - on_behalf_of_email: - type: string - modules: - type: object - nullable: true - description: Additional script modules keyed by relative file path - additionalProperties: - type: object - description: An additional module file associated with a script - properties: &ref_95 - content: + allOf: + - type: object + properties: &ref_106 + workspace_id: + type: string + hash: + type: string + path: + type: string + parent_hashes: + type: array + description: > + The first element is the direct parent of the + script, the second is the parent of the first, etc + items: type: string - description: The source code content of this module - language: + summary: + type: string + description: + type: string + content: + type: string + created_by: + type: string + created_at: + type: string + format: date-time + archived: + type: boolean + schema: + type: object + deleted: + type: boolean + is_template: + type: boolean + extra_perms: + type: object + additionalProperties: + type: boolean + lock: + type: string + lock_error_logs: + type: string + language: + type: string + enum: &ref_100 + - python3 + - deno + - go + - bash + - powershell + - postgresql + - mysql + - bigquery + - snowflake + - mssql + - oracledb + - graphql + - nativets + - bun + - php + - rust + - ansible + - csharp + - nu + - java + - ruby + - rlang + - duckdb + - bunnative + kind: + type: string + enum: + - script + - failure + - trigger + - command + - approval + - preprocessor + starred: + type: boolean + tag: + type: string + draft_only: + type: boolean + envs: + type: array + items: type: string - enum: *ref_94 - lock: + concurrent_limit: + type: integer + concurrency_time_window_s: + type: integer + concurrency_key: + type: string + debounce_key: + type: string + debounce_delay_s: + type: integer + debounce_args_to_accumulate: + type: array + items: type: string - nullable: true - description: Lock file content for this module's dependencies - required: &ref_96 - - content - - language - labels: - type: array - items: - type: string - default: [] - required: &ref_100 - - hash - - path - - summary - - description - - content - - created_by - - created_at - - archived - - deleted - - is_template - - extra_perms - - language - - kind - - starred - - has_preprocessor + max_total_debouncing_time: + type: integer + max_total_debounces_amount: + type: integer + cache_ttl: + type: number + dedicated_worker: + type: boolean + ws_error_handler_muted: + type: boolean + priority: + type: integer + restart_unless_cancelled: + type: boolean + timeout: + type: integer + delete_after_secs: + type: integer + description: >- + If set, delete the job's args, result and logs after + this many seconds following job completion + visible_to_runner_only: + type: boolean + auto_kind: + type: string + codebase: + type: string + has_preprocessor: + type: boolean + on_behalf_of_email: + type: string + modules: + type: object + nullable: true + description: >- + Additional script modules keyed by relative file + path + additionalProperties: + type: object + description: An additional module file associated with a script + properties: &ref_102 + content: + type: string + description: The source code content of this module + language: + type: string + enum: *ref_100 + lock: + type: string + nullable: true + description: >- + Lock file content for this module's + dependencies + required: &ref_103 + - content + - language + labels: + type: array + items: + type: string + default: [] + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at + read time. Read-only — edit them on the folder. + required: &ref_107 + - hash + - path + - summary + - description + - content + - created_by + - created_at + - archived + - deleted + - is_template + - extra_perms + - language + - kind + - starred + - has_preprocessor + - type: object + properties: + is_draft: + type: boolean + description: | + True when the authed user has a draft for this + script — either no deployed row exists at this + path (draft-only) or the user saved a per-user + draft on top of the deployed row. + draft_path: + type: string + description: | + User-typed path the editor has staged but not + yet deployed. Surfaced for draft-only rows so + the home list can render the meaningful name + instead of the autogenerated + `u/{user}/draft_{uuid}` URL path. Omitted + when unchanged. + draft_users: + description: | + Workspace users (including the authed user, and + the legacy NULL-email row if any) who have a + per-user draft at this path. Drives the home + page's user-avatar circles inside the Draft + badge. Omitted when no drafts exist. + type: array + items: + type: object + properties: + username: + type: string + nullable: true /w/{workspace}/scripts/list_paths: get: summary: list all scripts paths @@ -12401,10 +12968,12 @@ paths: type: array items: type: string - /w/{workspace}/drafts/create: - post: - summary: create draft - operationId: createDraft + /w/{workspace}/drafts/list: + get: + summary: >- + list every draft the current user has in this workspace, across all + kinds + operationId: listDrafts tags: - draft parameters: @@ -12412,37 +12981,129 @@ paths: in: path required: true schema: *ref_4 - requestBody: - required: true - content: - application/json: - schema: - type: object - properties: - path: - type: string - typ: - type: string - enum: - - flow - - script - - app - value: {} - required: - - path - - typ - - enum + - name: all_users + in: query + description: >- + List every draft in the workspace (all users), not just the current + user's own + legacy rows. Other users' rows come back with + `mine=false` (view-only). + schema: + type: boolean responses: - '201': - description: draft created + '200': + description: the user's drafts content: - text/plain: + application/json: schema: - type: string - /w/{workspace}/drafts/delete/{kind}/{path}: - delete: - summary: delete draft - operationId: deleteDraft + type: array + items: + type: object + properties: + kind: + type: string + description: > + Closed set of item kinds a user can autosave as a draft. + Mirrors the + + Postgres `DRAFT_KIND` enum and the backend + `UserDraftItemKind`. + enum: &ref_101 + - script + - flow + - app + - raw_app + - resource + - variable + - trigger_schedule + - trigger_webhook + - trigger_default_email + - trigger_email + - trigger_http + - trigger_websocket + - trigger_postgres + - trigger_kafka + - trigger_nats + - trigger_mqtt + - trigger_sqs + - trigger_gcp + - trigger_azure + - trigger_poll + - trigger_cli + - trigger_nextcloud + - trigger_google + - trigger_github + - data_pipeline + path: + type: string + summary: + type: string + description: >- + Best-effort, read from the draft JSON's `summary` field + when the editor shape carries one. + draft_path: + type: string + description: >- + User-typed friendly path from the draft JSON's + `draft_path`, when set and different from the storage + path (e.g. a never-deployed item parked at + `u/{user}/draft_{uuid}`). + draft_only: + type: boolean + description: >- + No deployed counterpart exists at this path — the draft + is the whole item. + legacy_draft: + type: boolean + description: >- + The listed draft is a legacy workspace-level row (email + NULL) predating the per-user drafts migration. Only true + when no per-user draft exists at this path. + created_at: + type: string + format: date-time + can_write: + type: boolean + description: >- + Whether the current user may deploy/discard this draft + (same check the deploy/discard endpoints enforce). + mine: + type: boolean + description: >- + The row belongs to the current user (own draft or the + legacy no-owner row) and is therefore actionable. Always + true in the default listing; with `all_users=true`, + other users' rows are false (view-only). + draft_users: + description: > + Draft authors at this (path, kind) — the legacy + NULL-email row surfaced as a null username. + + Populated only for the shared full-page-editor kinds + (script/flow/app/raw_app); omitted for + + drawer kinds, which keep their drafts private. Feeds the + Draft badge's owner-avatar circles. + type: array + items: + type: object + properties: + username: + type: string + nullable: true + required: + - kind + - path + - draft_only + - legacy_draft + - created_at + - can_write + - mine + /w/{workspace}/drafts/get/{kind}/{path}: + get: + summary: >- + fetch a single draft's content by workspace username (or the legacy + workspace-level row) + operationId: getDraftForUser tags: - draft parameters: @@ -12455,17 +13116,210 @@ paths: required: true schema: type: string - enum: - - script - - flow - - app + description: > + Closed set of item kinds a user can autosave as a draft. Mirrors + the + + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 + - name: username + in: query + required: false + description: >- + Workspace username of the draft owner. Omit to fetch the legacy + workspace-level (NULL email) row. + schema: + type: string responses: '200': - description: draft deleted + description: draft content + content: + application/json: + schema: + type: object + properties: + value: {} + created_at: + type: string + format: date-time + required: + - value + - created_at + '404': + description: no draft for that owner at that path + /w/{workspace}/drafts/get_own/{kind}/{path}: + get: + summary: fetch the current user's own draft content at a path (any kind) + operationId: getOwnDraft + tags: + - draft + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: kind + in: path + required: true + schema: + type: string + description: > + Closed set of item kinds a user can autosave as a draft. Mirrors + the + + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 + - name: path + in: path + required: true + schema: *ref_98 + responses: + '200': + description: the user's draft content, or null when none exists + content: + application/json: + schema: + nullable: true + type: object + properties: + value: {} + created_at: + type: string + format: date-time + required: + - value + - created_at + /w/{workspace}/drafts/update/{kind}/{path}: + post: + summary: upsert (or clear) the current user's draft at a path + operationId: updateDraft + tags: + - draft + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: kind + in: path + required: true + schema: + type: string + description: > + Closed set of item kinds a user can autosave as a draft. Mirrors + the + + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 + - name: path + in: path + required: true + schema: *ref_98 + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + value: + nullable: true + description: >- + Draft content to save. `null` (or omitted) signals a delete + — the row is removed under the same conflict rules. + last_sync: + type: string + format: date-time + description: >- + Server timestamp of the client's last known sync for this + draft. Omit on first save. + force: + type: boolean + description: Skip the conflict check and overwrite the server copy. + legacy: + type: boolean + description: >- + Delete-only. Target the legacy workspace-level row (email + NULL) instead of the current user's row. Used to discard a + legacy draft from the review page. + created_at: + type: string + format: date-time + description: >- + Upsert-only override for the stored creation timestamp. + Normal saves omit it (stamped server-side); the + localStorage→DB migration passes the draft's original write + time so migrated drafts keep their age. + responses: + '200': + description: save result + content: + application/json: + schema: + type: object + properties: + status: + type: string + enum: + - saved + - conflict + current_timestamp: + type: string + format: date-time + required: + - status + - current_timestamp + /w/{workspace}/drafts/migrate_legacy/{kind}/{path}: + post: + summary: resolve a legacy (workspace-level) draft (admin only) + description: >- + Delete a legacy draft (email NULL) or assign it to the authed admin as a + per-user draft. Workspace admins / superadmins only. + operationId: migrateLegacyDraft + tags: + - draft + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: kind + in: path + required: true + schema: + type: string + description: > + Closed set of item kinds a user can autosave as a draft. Mirrors + the + + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 + - name: path + in: path + required: true + schema: *ref_98 + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + action: + type: string + enum: + - delete + - assign_to_self + description: delete the legacy draft, or take ownership of it. + required: + - action + responses: + '200': + description: migration result content: text/plain: schema: @@ -12481,11 +13335,17 @@ paths: operationId: createScript x-mcp-tool: true x-mcp-instructions: >- - To create a script, specify the path (e.g., 'f/my_folder/my_script'), - the content (source code), and the language. For TypeScript, use 'bun' - unless deno-specific APIs are needed. + To create a NEW script, specify the path (e.g., + 'f/my_folder/my_script'), the content (source code), and the language, + and leave parent_hash unset. For TypeScript, use 'bun' unless + deno-specific APIs are needed. To UPDATE an existing script, do NOT + delete and recreate it: call this tool with the same path and set + parent_hash to the script's current hash, which you can read from the + `hash` field returned by getScriptByPath. This creates a new version + while preserving the script's history. x-mcp-tool-include-fields: - path + - parent_hash - content - language - summary @@ -12507,11 +13367,18 @@ paths: application/json: schema: type: object - properties: &ref_104 + properties: &ref_394 path: type: string parent_hash: type: string + auto_parent: + type: boolean + description: >- + When true, the backend resolves the parent to the current + deployed head for this path within the transaction (ignoring + parent_hash), instead of failing with a "lineage must be + linear" error when the supplied parent_hash is stale. summary: type: string description: @@ -12526,7 +13393,7 @@ paths: type: string language: type: string - enum: *ref_94 + enum: *ref_100 kind: type: string enum: @@ -12538,8 +13405,6 @@ paths: - preprocessor tag: type: string - draft_only: - type: boolean envs: type: array items: @@ -12611,7 +13476,7 @@ paths: type: string kind: type: string - enum: &ref_301 + enum: &ref_308 - s3object - resource - ducklake @@ -12636,8 +13501,8 @@ paths: additionalProperties: type: object description: An additional module file associated with a script - properties: *ref_95 - required: *ref_96 + properties: *ref_102 + required: *ref_103 labels: type: array items: @@ -12647,7 +13512,7 @@ paths: description: >- When true (set by the CLI / git sync), deploying this script does not delete an existing user draft at the same path. - required: &ref_105 + required: &ref_395 - path - summary - content @@ -12673,7 +13538,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: Workspace error handler enabled required: true @@ -12780,19 +13645,19 @@ paths: application/json: schema: type: object - properties: &ref_385 + properties: &ref_392 workspace_id: type: string language: type: string - enum: *ref_94 + enum: *ref_100 name: type: string description: type: string content: type: string - required: &ref_386 + required: &ref_393 - workspace_id - language - content @@ -12819,7 +13684,7 @@ paths: required: true schema: type: string - enum: *ref_94 + enum: *ref_100 - name: name in: query required: false @@ -12847,7 +13712,7 @@ paths: required: true schema: type: string - enum: *ref_94 + enum: *ref_100 - name: name in: query required: false @@ -12879,7 +13744,7 @@ paths: type: array items: type: object - properties: &ref_97 + properties: &ref_104 id: type: integer archived: @@ -12892,13 +13757,13 @@ paths: type: string language: type: string - enum: *ref_94 + enum: *ref_100 workspace_id: type: string created_at: type: string format: date-time - required: &ref_98 + required: &ref_105 - workspace_id - language - created_at @@ -12921,7 +13786,7 @@ paths: required: true schema: type: string - enum: *ref_94 + enum: *ref_100 - name: name in: query required: false @@ -12934,8 +13799,8 @@ paths: application/json: schema: type: object - properties: *ref_97 - required: *ref_98 + properties: *ref_104 + required: *ref_105 /w/{workspace}/scripts/archive/p/{path}: post: summary: archive script by path @@ -12950,7 +13815,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script archived @@ -12972,7 +13837,7 @@ paths: - name: hash in: path required: true - schema: &ref_101 + schema: &ref_108 type: string responses: '200': @@ -12981,8 +13846,8 @@ paths: application/json: schema: type: object - properties: *ref_99 - required: *ref_100 + properties: *ref_106 + required: *ref_107 /w/{workspace}/scripts/delete/h/{hash}: post: summary: delete script by hash (erase content but keep hash, require admin) @@ -12998,7 +13863,7 @@ paths: - name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 responses: '200': description: script details @@ -13006,8 +13871,8 @@ paths: application/json: schema: type: object - properties: *ref_99 - required: *ref_100 + properties: *ref_106 + required: *ref_107 /w/{workspace}/scripts/delete/p/{path}: post: summary: delete script at a given path (require admin) @@ -13023,7 +13888,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: keep_captures description: keep captures in: query @@ -13085,20 +13950,63 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: with_starred_info in: query schema: type: boolean + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: script details content: application/json: schema: - type: object - properties: *ref_99 - required: *ref_100 + allOf: + - type: object + properties: *ref_106 + required: *ref_107 + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/scripts/get_triggers_count/{path}: get: summary: get triggers count of script @@ -13113,7 +14021,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: triggers count @@ -13121,7 +14029,7 @@ paths: application/json: schema: type: object - properties: &ref_125 + properties: &ref_130 primary_schedule: type: object properties: @@ -13173,7 +14081,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: tokens list @@ -13183,50 +14091,8 @@ paths: type: array items: type: object - properties: *ref_102 - required: *ref_103 - /w/{workspace}/scripts/get/draft/{path}: - get: - summary: get script by path with draft - operationId: getScriptByPathWithDraft - tags: - - script - parameters: - - name: workspace - in: path - required: true - schema: *ref_4 - - name: path - in: path - required: true - schema: *ref_92 - responses: - '200': - description: script details - content: - application/json: - schema: - allOf: &ref_387 - - type: object - properties: *ref_104 - required: *ref_105 - - type: object - properties: - draft: - type: object - properties: *ref_104 - required: *ref_105 - draft_created_at: - type: string - format: date-time - description: >- - Timestamp at which the most recent DB draft was - created. Used by the frontend's UserDraft staleness - check. - hash: - type: string - required: - - hash + properties: *ref_109 + required: *ref_110 /w/{workspace}/scripts/history/p/{path}: get: summary: get history of a script by path @@ -13241,7 +14107,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script history @@ -13251,12 +14117,12 @@ paths: type: array items: type: object - properties: &ref_106 + properties: &ref_111 script_hash: type: string deployment_msg: type: string - required: &ref_107 + required: &ref_112 - script_hash /w/{workspace}/scripts/list_paths_from_workspace_runnable/{path}: get: @@ -13272,7 +14138,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: list of script paths @@ -13294,7 +14160,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 tags: - script responses: @@ -13304,8 +14170,8 @@ paths: application/json: schema: type: object - properties: *ref_106 - required: *ref_107 + properties: *ref_111 + required: *ref_112 /w/{workspace}/scripts/history_update/h/{hash}/p/{path}: post: summary: update history of a script @@ -13320,11 +14186,11 @@ paths: - name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: Script deployment message required: true @@ -13412,7 +14278,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script content @@ -13436,12 +14302,12 @@ paths: - name: token in: path required: true - schema: &ref_305 + schema: &ref_312 type: string - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script content @@ -13463,7 +14329,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: does it exists @@ -13485,7 +14351,7 @@ paths: - name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 - name: with_starred_info in: query schema: @@ -13501,8 +14367,8 @@ paths: application/json: schema: type: object - properties: *ref_99 - required: *ref_100 + properties: *ref_106 + required: *ref_107 /w/{workspace}/scripts/raw/h/{path}: get: summary: raw script by hash @@ -13517,7 +14383,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: script content @@ -13539,7 +14405,7 @@ paths: - name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 responses: '200': description: script details @@ -13589,7 +14455,7 @@ paths: type: array items: type: object - properties: &ref_108 + properties: &ref_113 test_script_path: type: string job_id: @@ -13603,7 +14469,7 @@ paths: type: string format: date-time nullable: true - required: &ref_109 + required: &ref_114 - test_script_path /w/{workspace}/scripts/ci_test_results_batch: post: @@ -13652,8 +14518,8 @@ paths: type: array items: type: object - properties: *ref_108 - required: *ref_109 + properties: *ref_113 + required: *ref_114 /w/{workspace}/scripts/raw_temp/store: post: summary: store raw script content temporarily for CLI lock generation @@ -13720,7 +14586,7 @@ paths: type: string language: type: string - enum: *ref_94 + enum: *ref_100 name: description: named workspace dependency (null for default) type: string @@ -13817,7 +14683,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: scheduled_for description: when to schedule this job (leave empty for immediate run) in: query @@ -13839,20 +14705,20 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: &ref_110 + schema: &ref_115 type: string format: uuid - name: tag description: Override the tag to use in: query - schema: &ref_111 + schema: &ref_116 type: string - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: &ref_112 + schema: &ref_117 type: string - name: job_id description: >- @@ -13861,7 +14727,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: &ref_113 + schema: &ref_118 type: string format: uuid - name: invisible_to_owner @@ -13900,23 +14766,23 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -13924,7 +14790,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -13933,19 +14799,19 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: &ref_114 + schema: &ref_119 type: string - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: &ref_115 + schema: &ref_120 type: string - name: skip_preprocessor description: skip the preprocessor in: query - schema: &ref_116 + schema: &ref_121 type: boolean requestBody: description: script args @@ -13975,23 +14841,23 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -13999,7 +14865,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14008,13 +14874,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14022,12 +14888,12 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: &ref_117 + schema: &ref_122 type: string - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 responses: '200': description: job result @@ -14048,7 +14914,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14057,13 +14923,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14071,11 +14937,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14123,13 +14989,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14137,11 +15003,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14188,13 +15054,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14202,7 +15068,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14210,11 +15076,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14241,7 +15107,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14250,13 +15116,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14264,11 +15130,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14310,7 +15176,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14319,13 +15185,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14333,7 +15199,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14341,11 +15207,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14391,13 +15257,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14405,11 +15271,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14463,13 +15329,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14477,7 +15343,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14485,11 +15351,11 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -14523,23 +15389,23 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14547,7 +15413,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14556,17 +15422,17 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -14602,23 +15468,23 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14626,7 +15492,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14635,13 +15501,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14649,11 +15515,11 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -14688,17 +15554,17 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14706,7 +15572,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14715,17 +15581,17 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -14768,17 +15634,17 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -14786,7 +15652,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -14795,13 +15661,13 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -14809,11 +15675,11 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: poll_delay_ms description: delay between polling for job updates in milliseconds in: query @@ -14854,6 +15720,35 @@ paths: content: application/json: schema: {} + /w/{workspace}/jobs/job_view_token/{id}: + get: + summary: mint a read-only share token for a job + description: > + Returns a stateless `{job_id}.{hmac}` token that grants an authenticated + workspace member read access to this job (and its flow subtree) via a + `view_token` query param or `X-View-Token` header. Only callable by a + user who can already read the job. + operationId: getJobViewToken + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: id + in: path + required: true + schema: + type: string + format: uuid + responses: + '200': + description: the share read token + content: + text/plain: + schema: + type: string /w/{workspace}/flows/list_paths: get: summary: list all flow paths @@ -14924,14 +15819,14 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: path_start description: mask to filter matching starting path in: query @@ -15006,15 +15901,15 @@ paths: type: array items: allOf: - - allOf: &ref_124 + - allOf: &ref_129 - type: object description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: *ref_120 - required: *ref_121 + properties: *ref_125 + required: *ref_126 - type: object - properties: &ref_512 + properties: &ref_516 workspace_id: type: string path: @@ -15028,7 +15923,7 @@ paths: type: boolean extra_perms: type: object - additionalProperties: &ref_511 + additionalProperties: &ref_515 type: boolean starred: type: boolean @@ -15053,7 +15948,15 @@ paths: items: type: string default: [] - required: &ref_513 + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, + computed at read time. Read-only — edit them on + the folder. + required: &ref_517 - path - edited_by - edited_at @@ -15067,10 +15970,40 @@ paths: type: number - type: object properties: - has_draft: - type: boolean draft_only: type: boolean + is_draft: + type: boolean + description: | + True when the authed user has a draft for this + flow — either no deployed row exists at this + path (draft-only) or the user saved a per-user + draft on top of the deployed row. + draft_path: + type: string + description: | + User-typed path the editor has staged but not + yet deployed. Sourced from the draft JSON's + `draft_path` field (the editor only writes it + when the typed path differs from the deployed + one). Lets the home list render the meaningful + name instead of the autogenerated + `u/{user}/draft_{uuid}` URL path. Omitted when + unchanged. + draft_users: + description: | + Workspace users (including the authed user, and + the legacy NULL-email row if any) who have a + per-user draft at this path. Drives the home + page's user-avatar circles inside the Draft + badge. Omitted when no drafts exist. + type: array + items: + type: object + properties: + username: + type: string + nullable: true /w/{workspace}/flows/history/p/{path}: get: summary: get flow history by path @@ -15083,7 +16016,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 tags: - flow responses: @@ -15095,7 +16028,7 @@ paths: type: array items: type: object - properties: &ref_122 + properties: &ref_127 id: type: integer created_at: @@ -15103,7 +16036,7 @@ paths: format: date-time deployment_msg: type: string - required: &ref_123 + required: &ref_128 - id - created_at /w/{workspace}/flows/get_latest_version/{path}: @@ -15118,7 +16051,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 tags: - flow responses: @@ -15128,8 +16061,8 @@ paths: application/json: schema: type: object - properties: *ref_122 - required: *ref_123 + properties: *ref_127 + required: *ref_128 /w/{workspace}/flows/list_paths_from_workspace_runnable/{runnable_kind}/{path}: get: summary: list flow paths from workspace runnable @@ -15144,7 +16077,7 @@ paths: - name: runnable_kind in: path required: true - schema: &ref_132 + schema: &ref_139 type: string enum: - script @@ -15152,7 +16085,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: match_path_start in: query schema: @@ -15188,7 +16121,7 @@ paths: content: application/json: schema: - allOf: *ref_124 + allOf: *ref_129 /w/{workspace}/flows/history_update/v/{version}: post: summary: update flow history @@ -15239,18 +16172,61 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: with_starred_info in: query schema: type: boolean + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: flow details content: application/json: schema: - allOf: *ref_124 + allOf: + - allOf: *ref_129 + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/flows/deployment_status/p/{path}: get: summary: get flow deployment status @@ -15265,7 +16241,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: flow status @@ -15293,7 +16269,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: triggers count @@ -15301,7 +16277,7 @@ paths: application/json: schema: type: object - properties: *ref_125 + properties: *ref_130 /w/{workspace}/flows/list_tokens/{path}: get: summary: get tokens with flow scope @@ -15316,7 +16292,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: tokens list @@ -15326,8 +16302,8 @@ paths: type: array items: type: object - properties: *ref_102 - required: *ref_103 + properties: *ref_109 + required: *ref_110 /w/{workspace}/flows/toggle_workspace_error_handler/{path}: post: summary: Toggle ON and OFF the workspace error handler for a given flow @@ -15342,7 +16318,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: Workspace error handler enabled required: true @@ -15360,40 +16336,6 @@ paths: text/plain: schema: type: string - /w/{workspace}/flows/get/draft/{path}: - get: - summary: get flow by path with draft - operationId: getFlowByPathWithDraft - tags: - - flow - parameters: - - name: workspace - in: path - required: true - schema: *ref_4 - - name: path - in: path - required: true - schema: *ref_92 - responses: - '200': - description: flow details with draft - content: - application/json: - schema: - allOf: - - allOf: *ref_124 - - type: object - properties: - draft: - allOf: *ref_124 - draft_created_at: - type: string - format: date-time - description: >- - Timestamp at which the most recent DB draft was - created. Used by the frontend's UserDraft staleness - check. /w/{workspace}/flows/exists/{path}: get: summary: exists flow by path @@ -15408,7 +16350,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: flow details @@ -15446,13 +16388,13 @@ paths: application/json: schema: allOf: - - allOf: &ref_126 + - allOf: &ref_131 - type: object description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: *ref_120 - required: *ref_121 + properties: *ref_125 + required: *ref_126 - type: object properties: path: @@ -15485,8 +16427,6 @@ paths: - path - type: object properties: - draft_only: - type: boolean deployment_message: type: string skip_draft_deletion: @@ -15528,7 +16468,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: Partially filled flow required: true @@ -15536,7 +16476,7 @@ paths: application/json: schema: allOf: - - allOf: *ref_126 + - allOf: *ref_131 - type: object properties: deployment_message: @@ -15568,7 +16508,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: archiveFlow required: true @@ -15601,7 +16541,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: keep_captures description: keep captures in: query @@ -15647,14 +16587,14 @@ paths: type: array items: type: object - required: &ref_372 + required: &ref_379 - id - workspace_id - flow_path - created_at - updated_at - created_by - properties: &ref_373 + properties: &ref_380 id: type: string format: uuid @@ -15747,14 +16687,14 @@ paths: type: array items: type: object - required: &ref_374 + required: &ref_381 - id - conversation_id - message_type - content - created_at - created_seq - properties: &ref_375 + properties: &ref_382 id: type: string format: uuid @@ -15860,14 +16800,14 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: path_start description: mask to filter matching starting path in: query @@ -15891,6 +16831,17 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: &ref_214 + type: boolean responses: '200': description: All raw apps @@ -15900,7 +16851,7 @@ paths: type: array items: type: object - properties: &ref_520 + properties: &ref_524 workspace_id: type: string path: @@ -15923,7 +16874,14 @@ paths: items: type: string default: [] - required: &ref_521 + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at + read time. Read-only — edit them on the folder. + required: &ref_525 - workspace_id - path - summary @@ -16063,6 +17021,141 @@ paths: text/plain: schema: type: string + /w/{workspace}/ai_skills/list: + get: + summary: list the workspace AI chat skills (name + description only) + operationId: listAiSkills + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + responses: + '200': + description: skill listing + content: + application/json: + schema: + type: array + items: + type: object + required: + - name + - description + properties: + name: + type: string + description: + type: string + /w/{workspace}/ai_skills/get/{name}: + get: + summary: get a workspace AI chat skill including its instructions + operationId: getAiSkill + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: skill + content: + application/json: + schema: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + description: + type: string + instructions: + type: string + /w/{workspace}/ai_skills/upload: + post: + summary: upsert workspace AI chat skills (admin only) + operationId: uploadAiSkills + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - skills + properties: + skills: + type: array + maxItems: 50 + items: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + minLength: 1 + maxLength: 64 + pattern: ^[a-z0-9-]+$ + description: + type: string + minLength: 1 + maxLength: 1024 + instructions: + type: string + minLength: 1 + maxLength: 65536 + responses: + '200': + description: uploaded + content: + text/plain: + schema: + type: string + /w/{workspace}/ai_skills/delete/{name}: + delete: + summary: delete a workspace AI chat skill (admin only) + operationId: deleteAiSkill + tags: + - workspace + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: name + in: path + required: true + schema: + type: string + responses: + '200': + description: deleted + content: + text/plain: + schema: + type: string /w/{workspace}/apps/get_data/v/{secretWithExtension}: get: summary: get raw app data by @@ -16077,6 +17170,10 @@ paths: - name: secretWithExtension in: path required: true + description: >- + App version secret suffixed with the requested file type extension. + Supported extensions are `.js` (JavaScript bundle), `.css` + (stylesheet), and `.html` (sandboxed wrapper document). schema: type: string responses: @@ -16086,6 +17183,12 @@ paths: text/javascript: schema: type: string + text/css: + schema: + type: string + text/html: + schema: + type: string /w/{workspace}/apps/list_search: get: summary: list apps for search @@ -16135,14 +17238,14 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: path_start description: mask to filter matching starting path in: query @@ -16189,7 +17292,7 @@ paths: type: array items: type: object - properties: &ref_514 + properties: &ref_518 id: type: integer workspace_id: @@ -16222,7 +17325,61 @@ paths: items: type: string default: [] - required: &ref_515 + is_draft: + type: boolean + description: > + True when the authed user has a draft for this app — + either no + + deployed row exists at this path (draft-only) or the + user has + + saved a per-user draft on top of the deployed row. + draft_path: + type: string + description: > + User-typed path the editor has staged but not yet + deployed. + + Sourced from the draft JSON's `draft_path` field (the + editor + + only writes it when the typed path differs from the + deployed + + one). Lets the home list render the meaningful name + instead of + + the autogenerated `u/{user}/draft_{uuid}` URL path. + Omitted + + when unchanged. + draft_users: + description: > + Workspace users (including the authed user, and the + legacy + + NULL-email row if any) who have a per-user draft at this + + path. Drives the home page's user-avatar circles inside + the + + Draft badge. Omitted when no drafts exist. + type: array + items: + type: object + properties: + username: + type: string + nullable: true + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at + read time. Read-only — edit them on the folder. + required: &ref_519 - id - workspace_id - path @@ -16267,9 +17424,7 @@ paths: type: string policy: type: object - properties: *ref_127 - draft_only: - type: boolean + properties: *ref_132 deployment_message: type: string custom_path: @@ -16330,9 +17485,7 @@ paths: type: string policy: type: object - properties: *ref_127 - draft_only: - type: boolean + properties: *ref_132 deployment_message: type: string custom_path: @@ -16406,20 +17559,95 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: with_starred_info in: query schema: type: boolean + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 + - name: raw_app + in: query + description: | + When no deployed app exists at this path and `get_draft` is set, + disambiguates which draft kind (`raw_app` or `app`) to look up. + Ignored when a deployed row exists. + schema: + type: boolean responses: '200': description: app details + content: + application/json: + schema: + allOf: + - type: object + properties: *ref_133 + required: *ref_134 + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 + /w/{workspace}/apps/embed_token/p/{path}: + get: + summary: get app embed token by path + operationId: getAppEmbedTokenByPath + tags: + - app + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: path + in: path + required: true + schema: *ref_98 + responses: + '200': + description: embed token content: application/json: schema: type: object - properties: *ref_128 - required: *ref_129 + properties: *ref_135 + required: *ref_136 /w/{workspace}/apps/get/lite/{path}: get: summary: get app lite by path @@ -16434,7 +17662,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: app lite details @@ -16442,45 +17670,8 @@ paths: application/json: schema: type: object - properties: *ref_128 - required: *ref_129 - /w/{workspace}/apps/get/draft/{path}: - get: - summary: get app by path with draft - operationId: getAppByPathWithDraft - tags: - - app - parameters: - - name: workspace - in: path - required: true - schema: *ref_4 - - name: path - in: path - required: true - schema: *ref_92 - responses: - '200': - description: app details with draft - content: - application/json: - schema: - allOf: &ref_522 - - type: object - properties: *ref_128 - required: *ref_129 - - type: object - properties: - draft_only: - type: boolean - draft: {} - draft_created_at: - type: string - format: date-time - description: >- - Timestamp at which the most recent DB draft was - created. Used by the frontend's UserDraft staleness - check. + properties: *ref_133 + required: *ref_134 /w/{workspace}/apps/history/p/{path}: get: summary: get app history by path @@ -16495,7 +17686,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: app history @@ -16505,12 +17696,12 @@ paths: type: array items: type: object - properties: &ref_130 + properties: &ref_137 version: type: integer deployment_msg: type: string - required: &ref_131 + required: &ref_138 - version /w/{workspace}/apps/get_latest_version/{path}: get: @@ -16524,7 +17715,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 tags: - app responses: @@ -16534,8 +17725,8 @@ paths: application/json: schema: type: object - properties: *ref_130 - required: *ref_131 + properties: *ref_137 + required: *ref_138 /w/{workspace}/apps/list_paths_from_workspace_runnable/{runnable_kind}/{path}: get: summary: list app paths from workspace runnable @@ -16550,11 +17741,11 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_132 + schema: *ref_139 - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 responses: '200': description: list of app paths @@ -16578,11 +17769,11 @@ paths: - name: id in: path required: true - schema: *ref_79 + schema: *ref_84 - name: version in: path required: true - schema: &ref_306 + schema: &ref_313 type: integer requestBody: description: App deployment message @@ -16623,8 +17814,33 @@ paths: application/json: schema: type: object - properties: *ref_128 - required: *ref_129 + properties: *ref_133 + required: *ref_134 + /w/{workspace}/apps_u/embed_token/{secret}: + get: + summary: get app embed token by secret + operationId: getAppEmbedTokenBySecret + tags: + - app + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: secret + in: path + required: true + schema: + type: string + responses: + '200': + description: embed token + content: + application/json: + schema: + type: object + properties: *ref_135 + required: *ref_136 /w/{workspace}/apps_u/public_resource/{path}: get: summary: get public resource @@ -16704,7 +17920,7 @@ paths: - name: id in: path required: true - schema: *ref_79 + schema: *ref_84 responses: '200': description: app details @@ -16712,8 +17928,8 @@ paths: application/json: schema: type: object - properties: *ref_128 - required: *ref_129 + properties: *ref_133 + required: *ref_134 /w/{workspace}/apps/delete/{path}: delete: summary: delete app @@ -16760,7 +17976,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: update app required: true @@ -16776,7 +17992,7 @@ paths: value: {} policy: type: object - properties: *ref_127 + properties: *ref_132 deployment_message: type: string custom_path: @@ -16817,7 +18033,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: update app required: true @@ -16836,7 +18052,7 @@ paths: value: {} policy: type: object - properties: *ref_127 + properties: *ref_132 deployment_message: type: string custom_path: @@ -16883,7 +18099,7 @@ paths: - name: custom_path in: path required: true - schema: *ref_133 + schema: *ref_97 responses: '200': description: custom path exists @@ -16914,7 +18130,7 @@ paths: type: array items: type: object - properties: &ref_134 + properties: &ref_140 s3: type: string filename: @@ -16923,7 +18139,7 @@ paths: type: string presigned: type: string - required: &ref_135 + required: &ref_141 - s3 required: - s3_objects @@ -16936,8 +18152,8 @@ paths: type: array items: type: object - properties: *ref_134 - required: *ref_135 + properties: *ref_140 + required: *ref_141 /w/{workspace}/apps_u/execute_component/{path}: post: summary: executeComponent @@ -16952,7 +18168,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: update app required: true @@ -17137,7 +18353,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 - name: scheduled_for description: when to schedule this job (leave empty for immediate run) in: query @@ -17152,17 +18368,17 @@ paths: - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17170,7 +18386,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -17179,7 +18395,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the flow owner (default false) in: query @@ -17240,17 +18456,17 @@ paths: - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17258,7 +18474,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -17267,7 +18483,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the flow owner (default false) in: query @@ -17335,14 +18551,14 @@ paths: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: &ref_138 + oneOf: &ref_144 - type: object description: >- Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource' - properties: *ref_136 - required: *ref_137 + properties: *ref_142 + required: *ref_143 - type: object description: >- JavaScript expression evaluated at runtime. Can @@ -17351,16 +18567,16 @@ paths: 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value - properties: *ref_84 - required: *ref_85 + properties: *ref_89 + required: *ref_90 - type: object description: >- Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter. - properties: *ref_86 - required: *ref_87 - discriminator: &ref_139 + properties: *ref_91 + required: *ref_92 + discriminator: &ref_145 propertyName: type mapping: static: '#/components/schemas/schemas-StaticTransform' @@ -17380,8 +18596,8 @@ paths: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_138 - discriminator: *ref_139 + oneOf: *ref_144 + discriminator: *ref_145 use_latest_version: type: boolean responses: @@ -17407,7 +18623,7 @@ paths: - name: id in: path required: true - schema: &ref_172 + schema: &ref_178 type: string format: uuid - name: scheduled_for @@ -17426,11 +18642,11 @@ paths: The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17438,7 +18654,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -17447,7 +18663,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the flow owner (default false) in: query @@ -17521,7 +18737,7 @@ paths: - name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 - name: scheduled_for description: when to schedule this job (leave empty for immediate run) in: query @@ -17536,23 +18752,23 @@ paths: - name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 - name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17560,7 +18776,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: include_header description: > List of headers's keys (separated with ',') whove value are added to @@ -17569,7 +18785,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the script owner (default false) in: query @@ -17609,12 +18825,17 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the script owner (default false) in: query schema: type: boolean + - name: timeout + description: custom timeout in seconds for this preview run + in: query + schema: + type: integer - name: job_id description: >- The job id to assign to the created job. if missing, job is chosen @@ -17622,7 +18843,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 requestBody: description: preview required: true @@ -17630,7 +18851,7 @@ paths: application/json: schema: type: object - properties: &ref_141 + properties: &ref_147 content: type: string description: The code to run @@ -17646,7 +18867,7 @@ paths: additionalProperties: true language: type: string - enum: *ref_94 + enum: *ref_100 tag: type: string kind: @@ -17668,8 +18889,8 @@ paths: additionalProperties: type: object description: An additional module file associated with a script - properties: *ref_95 - required: *ref_96 + properties: *ref_102 + required: *ref_103 temp_script_refs: type: object nullable: true @@ -17679,7 +18900,7 @@ paths: local content instead of the deployed script additionalProperties: type: string - required: &ref_142 + required: &ref_148 - args responses: '201': @@ -17707,7 +18928,7 @@ paths: application/json: schema: type: object - properties: &ref_423 + properties: &ref_431 content: type: string description: The code to run @@ -17717,8 +18938,8 @@ paths: additionalProperties: true language: type: string - enum: *ref_94 - required: &ref_424 + enum: *ref_100 + required: &ref_432 - content - args - language @@ -17742,7 +18963,7 @@ paths: - name: path in: path required: true - schema: *ref_92 + schema: *ref_98 requestBody: description: script args required: true @@ -17750,7 +18971,7 @@ paths: application/json: schema: type: object - properties: &ref_140 + properties: &ref_146 args: type: object description: The arguments to pass to the script or flow @@ -17775,7 +18996,7 @@ paths: - name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 requestBody: description: script args required: true @@ -17783,7 +19004,7 @@ paths: application/json: schema: type: object - properties: *ref_140 + properties: *ref_146 responses: '200': description: script result @@ -17817,8 +19038,8 @@ paths: application/json: schema: type: object - properties: *ref_141 - required: *ref_142 + properties: *ref_147 + required: *ref_148 responses: '200': description: job result @@ -17853,12 +19074,12 @@ paths: application/json: schema: type: object - properties: &ref_425 + properties: &ref_433 args: type: object description: The arguments to pass to the script or flow additionalProperties: true - required: &ref_426 + required: &ref_434 - args responses: '201': @@ -17891,15 +19112,15 @@ paths: type: array items: type: object - properties: &ref_143 + properties: &ref_149 raw_code: type: string path: type: string language: type: string - enum: *ref_94 - required: &ref_144 + enum: *ref_100 + required: &ref_150 - raw_code - path - language @@ -17943,8 +19164,8 @@ paths: type: array items: type: object - properties: *ref_143 - required: *ref_144 + properties: *ref_149 + required: *ref_150 entrypoint: type: string required: @@ -17984,7 +19205,7 @@ paths: description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: &ref_149 + properties: &ref_155 modules: type: array description: >- @@ -17995,22 +19216,22 @@ paths: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 failure_module: description: >- A single step in a flow. Can be a script, subflow, loop, or branch type: object - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 preprocessor_module: description: >- A single step in a flow. Can be a script, subflow, loop, or branch type: object - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 same_worker: type: boolean description: >- @@ -18093,8 +19314,8 @@ paths: description: >- A sticky note attached to a flow for documentation and annotation - properties: *ref_145 - required: *ref_146 + properties: *ref_151 + required: *ref_152 groups: type: array description: Semantic groups of modules for organizational purposes @@ -18107,9 +19328,9 @@ paths: naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id. - properties: *ref_147 - required: *ref_148 - required: &ref_150 + properties: *ref_153 + required: *ref_154 + required: &ref_156 - modules required: - path @@ -18141,7 +19362,7 @@ paths: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 - name: invisible_to_owner description: make the run invisible to the the script owner (default false) in: query @@ -18154,7 +19375,7 @@ paths: queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 - name: memory_id description: memory ID for chat-enabled flows in: query @@ -18168,14 +19389,14 @@ paths: application/json: schema: type: object - properties: &ref_152 + properties: &ref_158 value: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_149 - required: *ref_150 + properties: *ref_155 + required: *ref_156 path: type: string args: @@ -18186,7 +19407,7 @@ paths: type: string restarted_from: type: object - properties: &ref_151 + properties: &ref_157 flow_job_id: type: string format: uuid @@ -18219,7 +19440,7 @@ paths: `RestartedFlow` against `nested.flow_job_id` instead of fresh-launching it. type: object - properties: *ref_151 + properties: *ref_157 temp_script_refs: type: object nullable: true @@ -18230,7 +19451,7 @@ paths: of the deployed script additionalProperties: type: string - required: &ref_153 + required: &ref_159 - value - content - args @@ -18266,8 +19487,8 @@ paths: application/json: schema: type: object - properties: *ref_152 - required: *ref_153 + properties: *ref_158 + required: *ref_159 responses: '200': description: job result @@ -18292,7 +19513,7 @@ paths: application/json: schema: type: object - properties: &ref_529 + properties: &ref_532 entrypoint_function: type: string description: Name of the function to execute for dynamic select @@ -18313,7 +19534,7 @@ paths: description: Path to the deployed script or flow runnable_kind: type: string - enum: &ref_200 + enum: &ref_206 - script - flow required: @@ -18331,11 +19552,11 @@ paths: description: Code content for inline execution language: type: string - enum: *ref_94 + enum: *ref_100 required: - source - code - required: &ref_530 + required: &ref_533 - entrypoint_function - runnable_ref responses: @@ -18361,27 +19582,27 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: worker description: >- filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: &ref_156 + schema: &ref_162 type: string - name: script_path_exact description: >- @@ -18389,7 +19610,7 @@ paths: (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: &ref_157 + schema: &ref_163 type: string - name: script_path_start description: >- @@ -18397,12 +19618,12 @@ paths: 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: &ref_158 + schema: &ref_164 type: string - name: schedule_path description: mask to filter by schedule path in: query - schema: &ref_159 + schema: &ref_165 type: string - name: trigger_path description: >- @@ -18410,7 +19631,7 @@ paths: 'f/trigger1,f/trigger2') and negation by prefixing all values with '!' (e.g. '!f/trigger1,!f/trigger2') in: query - schema: &ref_307 + schema: &ref_314 type: string - name: trigger_kind description: >- @@ -18419,34 +19640,34 @@ paths: (e.g. '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: &ref_188 + schema: &ref_194 type: string - name: script_hash description: mask to filter exact matching path in: query - schema: &ref_160 + schema: &ref_166 type: string - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: &ref_161 + schema: &ref_167 type: string format: date-time - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: &ref_162 + schema: &ref_168 type: string format: date-time - name: success description: filter on successful jobs in: query - schema: &ref_170 + schema: &ref_176 type: boolean - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: &ref_164 + schema: &ref_170 type: boolean - name: job_kinds description: >- @@ -18454,36 +19675,36 @@ paths: ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: &ref_165 + schema: &ref_171 type: string - name: suspended description: filter on suspended jobs in: query - schema: &ref_166 + schema: &ref_172 type: boolean - name: running description: filter on running jobs in: query - schema: &ref_163 + schema: &ref_169 type: boolean - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: &ref_167 + schema: &ref_173 type: string - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: &ref_169 + schema: &ref_175 type: string - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: &ref_171 + schema: &ref_177 type: boolean - name: tag description: >- @@ -18491,7 +19712,7 @@ paths: 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: &ref_168 + schema: &ref_174 type: string - name: page description: which page to return (start at 1, default 1) @@ -18522,7 +19743,7 @@ paths: type: array items: type: object - properties: &ref_191 + properties: &ref_197 workspace_id: type: string id: @@ -18597,14 +19818,14 @@ paths: by extension its DT_TOKEN. flow_status: type: object - properties: &ref_175 + properties: &ref_181 step: type: integer modules: type: array items: type: object - properties: &ref_154 + properties: &ref_160 type: type: string enum: @@ -18758,20 +19979,20 @@ paths: type: array items: type: boolean - required: &ref_155 + required: &ref_161 - type user_states: additionalProperties: true preprocessor_module: allOf: - type: object - properties: *ref_154 - required: *ref_155 + properties: *ref_160 + required: *ref_161 failure_module: allOf: - type: object - properties: *ref_154 - required: *ref_155 + properties: *ref_160 + required: *ref_161 - type: object properties: parent_module: @@ -18786,13 +20007,13 @@ paths: items: type: string format: uuid - required: &ref_176 + required: &ref_182 - step - modules - failure_module workflow_as_code_status: type: object - properties: &ref_177 + properties: &ref_183 scheduled_for: type: string format: date-time @@ -18808,13 +20029,13 @@ paths: description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_149 - required: *ref_150 + properties: *ref_155 + required: *ref_156 is_flow_step: type: boolean language: type: string - enum: *ref_94 + enum: *ref_100 email: type: string visible_to_owner: @@ -18833,9 +20054,11 @@ paths: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string - required: &ref_192 + required: &ref_198 - id - running - canceled @@ -18951,14 +20174,14 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: &ref_174 + schema: &ref_180 type: string - name: worker description: >- @@ -18966,117 +20189,117 @@ paths: (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_156 + schema: *ref_162 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_157 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_158 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_159 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_160 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_161 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_162 + schema: *ref_168 - name: created_before description: filter on created before (inclusive) timestamp in: query - schema: &ref_182 + schema: &ref_188 type: string format: date-time - name: created_after description: filter on created after (exclusive) timestamp in: query - schema: &ref_183 + schema: &ref_189 type: string format: date-time - name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: &ref_184 + schema: &ref_190 type: string format: date-time - name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: &ref_185 + schema: &ref_191 type: string format: date-time - name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: &ref_186 + schema: &ref_192 type: string format: date-time - name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: &ref_187 + schema: &ref_193 type: string format: date-time - name: running description: filter on running jobs in: query - schema: *ref_163 + schema: *ref_169 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_164 + schema: *ref_170 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_165 + schema: *ref_171 - name: suspended description: filter on suspended jobs in: query - schema: *ref_166 + schema: *ref_172 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_168 + schema: *ref_174 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_169 + schema: *ref_175 - name: page description: which page to return (start at 1, default 1) in: query @@ -19105,6 +20328,19 @@ paths: in: query schema: type: boolean + - name: status + description: >- + filter on the exact completed job status. Unlike `success=true` + (which also matches `skipped`), `status=success` matches only + `success`. + in: query + schema: + type: string + enum: + - success + - failure + - canceled + - skipped - name: all_workspaces description: >- get jobs from all workspaces (only valid if request come from the @@ -19140,96 +20376,96 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_157 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_158 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_159 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_160 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_161 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_162 + schema: *ref_168 - name: success description: filter on successful jobs in: query - schema: *ref_170 + schema: *ref_176 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_164 + schema: *ref_170 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_165 + schema: *ref_171 - name: suspended description: filter on suspended jobs in: query - schema: *ref_166 + schema: *ref_172 - name: running description: filter on running jobs in: query - schema: *ref_163 + schema: *ref_169 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_169 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_171 + schema: *ref_177 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_168 + schema: *ref_174 - name: page description: which page to return (start at 1, default 1) in: query @@ -19315,7 +20551,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: list of OTEL Span objects (compatible with OpenTelemetry Span proto) @@ -19343,7 +20579,7 @@ paths: schema: description: job trigger kind (schedule, http, websocket...) type: string - enum: &ref_173 + enum: &ref_179 - webhook - default_email - email @@ -19359,6 +20595,7 @@ paths: - azure - google - github + - asset - name: trigger_path description: The path of the trigger (can contain forward slashes) in: path @@ -19409,7 +20646,7 @@ paths: schema: description: job trigger kind (schedule, http, websocket...) type: string - enum: *ref_173 + enum: *ref_179 - name: trigger_path description: The path of the trigger (can contain forward slashes) in: path @@ -19456,98 +20693,111 @@ paths: - name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 - name: created_by description: >- filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_174 + schema: *ref_180 - name: worker description: >- filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_156 + schema: *ref_162 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_157 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_158 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_159 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_160 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_161 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_162 + schema: *ref_168 - name: success description: filter on successful jobs in: query - schema: *ref_170 + schema: *ref_176 + - name: status + description: >- + filter on the exact completed job status. Unlike `success=true` + (which also matches `skipped`), `status=success` matches only + `success`. + in: query + schema: + type: string + enum: + - success + - failure + - canceled + - skipped - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_165 + schema: *ref_171 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_169 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_171 + schema: *ref_177 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_168 + schema: *ref_174 - name: page description: which page to return (start at 1, default 1) in: query @@ -19585,7 +20835,7 @@ paths: type: array items: type: object - properties: &ref_189 + properties: &ref_195 workspace_id: type: string id: @@ -19662,23 +20912,23 @@ paths: by extension its DT_TOKEN. flow_status: type: object - properties: *ref_175 - required: *ref_176 + properties: *ref_181 + required: *ref_182 workflow_as_code_status: type: object - properties: *ref_177 + properties: *ref_183 raw_flow: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_149 - required: *ref_150 + properties: *ref_155 + required: *ref_156 is_flow_step: type: boolean language: type: string - enum: *ref_94 + enum: *ref_100 is_skipped: type: boolean email: @@ -19701,9 +20951,11 @@ paths: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string - required: &ref_190 + required: &ref_196 - id - created_by - duration_ms @@ -19747,7 +20999,7 @@ paths: items: type: object description: Completed job with full data for export/import operations - properties: &ref_178 + properties: &ref_184 id: type: string format: uuid @@ -19842,7 +21094,7 @@ paths: type: boolean language: type: string - enum: *ref_94 + enum: *ref_100 is_skipped: type: boolean email: @@ -19885,7 +21137,7 @@ paths: status: type: string description: Actual job status from database - required: &ref_179 + required: &ref_185 - id - created_by - created_at @@ -19913,8 +21165,8 @@ paths: items: type: object description: Completed job with full data for export/import operations - properties: *ref_178 - required: *ref_179 + properties: *ref_184 + required: *ref_185 responses: '200': description: Successfully imported completed jobs @@ -19951,7 +21203,7 @@ paths: items: type: object description: Queued job with full data for export/import operations - properties: &ref_180 + properties: &ref_186 id: type: string format: uuid @@ -20041,7 +21293,7 @@ paths: type: boolean language: type: string - enum: *ref_94 + enum: *ref_100 email: type: string visible_to_owner: @@ -20082,7 +21334,7 @@ paths: suspend_until: type: string format: date-time - required: &ref_181 + required: &ref_187 - id - created_by - created_at @@ -20110,8 +21362,8 @@ paths: items: type: object description: Queued job with full data for export/import operations - properties: *ref_180 - required: *ref_181 + properties: *ref_186 + required: *ref_187 responses: '200': description: Successfully imported queued jobs @@ -20166,123 +21418,123 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_174 + schema: *ref_180 - name: worker description: >- filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_156 + schema: *ref_162 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_157 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_158 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_159 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_160 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_161 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_162 + schema: *ref_168 - name: created_before description: filter on created before (inclusive) timestamp in: query - schema: *ref_182 + schema: *ref_188 - name: created_after description: filter on created after (exclusive) timestamp in: query - schema: *ref_183 + schema: *ref_189 - name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_184 + schema: *ref_190 - name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_185 + schema: *ref_191 - name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: *ref_186 + schema: *ref_192 - name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: *ref_187 + schema: *ref_193 - name: running description: filter on running jobs in: query - schema: *ref_163 + schema: *ref_169 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_164 + schema: *ref_170 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_165 + schema: *ref_171 - name: suspended description: filter on suspended jobs in: query - schema: *ref_166 + schema: *ref_172 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_168 + schema: *ref_174 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_169 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_171 + schema: *ref_177 - name: per_page description: number of items to return for a given page (default 30, max 100) in: query @@ -20294,7 +21546,7 @@ paths: (e.g. '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: *ref_188 + schema: *ref_194 - name: is_skipped description: is the job skipped in: query @@ -20315,6 +21567,19 @@ paths: in: query schema: type: boolean + - name: status + description: >- + filter on the exact completed job status. Unlike `success=true` + (which also matches `skipped`), `status=success` matches only + `success`. + in: query + schema: + type: string + enum: + - success + - failure + - canceled + - skipped - name: all_workspaces description: >- get jobs from all workspaces (only valid if request come from the @@ -20349,11 +21614,11 @@ paths: schema: type: array items: - oneOf: &ref_193 + oneOf: &ref_199 - allOf: - type: object - properties: *ref_189 - required: *ref_190 + properties: *ref_195 + required: *ref_196 - type: object properties: type: @@ -20362,15 +21627,15 @@ paths: - CompletedJob - allOf: - type: object - properties: *ref_191 - required: *ref_192 + properties: *ref_197 + required: *ref_198 - type: object properties: type: type: string enum: - QueuedJob - discriminator: &ref_194 + discriminator: &ref_200 propertyName: type /jobs/db_clock: get: @@ -20438,7 +21703,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: no_logs in: query schema: @@ -20461,8 +21726,8 @@ paths: content: application/json: schema: - oneOf: *ref_193 - discriminator: *ref_194 + oneOf: *ref_199 + discriminator: *ref_200 /w/{workspace}/jobs_u/get_root_job_id/{id}: get: summary: get root job id @@ -20477,7 +21742,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: get root job id @@ -20500,7 +21765,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: remove_ansi_warnings in: query schema: @@ -20526,7 +21791,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: concatenated logs of all flow steps @@ -20534,6 +21799,73 @@ paths: text/plain: schema: type: string + /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}: + get: + summary: get all logs for a flow job in a structured format + operationId: getFlowAllLogsStructured + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: id + in: path + required: true + schema: *ref_178 + responses: + '200': + description: structured logs of all flow steps, one entry per job + content: + application/json: + schema: + type: array + items: + type: object + properties: + job_id: + type: string + label: + type: string + description: >- + human-readable label describing the job's position in + the flow tree + kind: + type: string + description: job kind (script, flow, forloopflow, ...) + flow_step_id: + type: string + nullable: true + step_path: + type: string + nullable: true + description: materialized step path (e.g. "a/b") + depth: + type: integer + description: depth in the flow tree (0 for the root flow job) + parent_module_type: + type: string + nullable: true + description: parent module type (forloopflow, branchall, ...) + sibling_index: + type: integer + description: >- + 1-based index of this job among siblings sharing the + same step + sibling_count: + type: integer + description: total number of siblings sharing the same step + logs: + type: string + required: + - job_id + - label + - kind + - depth + - sibling_index + - sibling_count + - logs /w/{workspace}/jobs_u/get_completed_logs_tail/{id}: get: summary: get completed job logs tail @@ -20548,7 +21880,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: completed job logs tail @@ -20570,7 +21902,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: job args @@ -20621,7 +21953,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: running in: query schema: @@ -20668,11 +22000,11 @@ paths: type: string flow_status: type: object - properties: *ref_175 - required: *ref_176 + properties: *ref_181 + required: *ref_182 workflow_as_code_status: type: object - properties: *ref_177 + properties: *ref_183 /w/{workspace}/jobs_u/getupdate_sse/{id}: get: summary: get job updates via server-sent events @@ -20687,7 +22019,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: running in: query schema: @@ -20760,7 +22092,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: flow debug info details @@ -20781,7 +22113,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: job details @@ -20789,8 +22121,8 @@ paths: application/json: schema: type: object - properties: *ref_189 - required: *ref_190 + properties: *ref_195 + required: *ref_196 /w/{workspace}/jobs_u/completed/get_result/{id}: get: summary: get completed job result @@ -20805,7 +22137,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: suspended_job in: query schema: @@ -20842,10 +22174,10 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: get_started in: query - schema: &ref_313 + schema: &ref_320 type: boolean responses: '200': @@ -20879,7 +22211,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: job timing details @@ -20898,6 +22230,154 @@ paths: type: integer required: - created_at + /w/{workspace}/jobs_u/dispatch_events/{id}: + get: + summary: list asset-trigger dispatch events for a producer job + description: > + Returns the chronological log of decisions the asset-trigger dispatcher + made after this producer job completed. Each row is one (subscriber, + asset write) decision: `dispatched` (with `child_job_id`), + `join_pending` (with `received_inputs` / `required_inputs` / + `partition`), or `skipped` (with `reason`). Rows are reaped + automatically when the producer's `v2_job` row is deleted by the + retention sweep. + operationId: listDispatchEvents + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: id + in: path + required: true + schema: *ref_178 + responses: + '200': + description: dispatch events for this producer job + content: + application/json: + schema: + type: array + items: + type: object + properties: + subscriber_path: + type: string + asset_kind: + type: string + enum: + - s3object + - resource + - variable + - ducklake + - datatable + - volume + asset_path: + type: string + outcome: + type: string + enum: + - dispatched + - join_pending + - skipped + child_job_id: + type: string + format: uuid + partition: + type: string + received_inputs: + type: integer + required_inputs: + type: integer + debounce_s: + type: integer + reason: + type: string + created_at: + type: string + format: date-time + required: + - subscriber_path + - asset_kind + - asset_path + - outcome + - created_at + /w/{workspace}/jobs/asset_dispatch_edges: + get: + summary: list asset-cascade producer→child job edges for a folder + description: > + Returns the `dispatched` asset-trigger edges (producer job → child job) + whose subscriber lives under `path_start`. Lets a pipeline view + reconstruct the cascade tree of a folder by job id and group connected + runs. Visibility follows the producer job's RLS. + operationId: listAssetDispatchEdges + tags: + - job + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: path_start + in: query + required: true + description: Folder path prefix the children live under, e.g. `f/orders/`. + schema: + type: string + - name: created_after + in: query + required: false + description: Only edges dispatched at/after this instant. + schema: + type: string + format: date-time + responses: + '200': + description: asset-cascade edges for the folder + content: + application/json: + schema: + type: array + items: + type: object + properties: + producer_job_id: + type: string + format: uuid + child_job_id: + type: string + format: uuid + description: Set for `dispatched`; absent for `join_pending` inputs. + subscriber_path: + type: string + outcome: + type: string + enum: + - dispatched + - join_pending + asset_kind: + type: string + enum: + - s3object + - resource + - variable + - ducklake + - datatable + - volume + asset_path: + type: string + created_at: + type: string + format: date-time + required: + - producer_job_id + - subscriber_path + - outcome + - asset_kind + - asset_path + - created_at /w/{workspace}/jobs/completed/delete/{id}: post: summary: delete completed job (erase content but keep run id) @@ -20912,7 +22392,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: job details @@ -20920,8 +22400,8 @@ paths: application/json: schema: type: object - properties: *ref_189 - required: *ref_190 + properties: *ref_195 + required: *ref_196 /w/{workspace}/jobs_u/queue/cancel/{id}: post: summary: cancel queued or running job @@ -20936,7 +22416,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 requestBody: description: reason required: true @@ -21000,7 +22480,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 requestBody: description: reason required: true @@ -21062,7 +22542,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: scheduled for timestamp @@ -21084,7 +22564,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: resume_id in: path required: true @@ -21115,7 +22595,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: resume_id in: path required: true @@ -21165,7 +22645,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: approver in: query schema: @@ -21226,7 +22706,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: approver in: query schema: @@ -21400,6 +22880,13 @@ paths: type: integer approver: type: string + view_token: + type: string + description: >- + Share-read-link token for the flow. An authenticated + workspace member can append it as a `view_token` query + param on the run page to read a flow they don't otherwise + have access to. /w/{workspace}/jobs_u/resume/{id}/{resume_id}/{signature}: get: summary: resume a job for a suspended flow @@ -21414,7 +22901,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: payload description: > The base64 encoded payload that has been encoded as a JSON. e.g how @@ -21422,7 +22909,7 @@ paths: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 - name: resume_id in: path required: true @@ -21457,7 +22944,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: resume_id in: path required: true @@ -21499,7 +22986,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: key in: path required: true @@ -21531,7 +23018,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: key in: path required: true @@ -21557,7 +23044,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 requestBody: required: true content: @@ -21585,7 +23072,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: resume_id in: path required: true @@ -21620,7 +23107,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: resume_id in: path required: true @@ -21662,7 +23149,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 - name: resume_id in: path required: true @@ -21686,8 +23173,8 @@ paths: type: object properties: job: - oneOf: *ref_193 - discriminator: *ref_194 + oneOf: *ref_199 + discriminator: *ref_200 approvers: type: array items: @@ -21700,6 +23187,13 @@ paths: required: - resume_id - approver + view_token: + type: string + description: >- + Share-read-link token for the parent flow. An + authenticated workspace member can append it as a + `view_token` query param on the run page to read a flow + they don't otherwise have access to. required: - job - approvers @@ -21762,7 +23256,7 @@ paths: application/json: schema: type: object - properties: &ref_436 + properties: &ref_440 path: type: string description: >- @@ -21855,7 +23349,7 @@ paths: nullable: true type: object description: Retry configuration for failed module executions - properties: &ref_197 + properties: &ref_203 constant: type: object description: Retry with constant delay between attempts @@ -21890,8 +23384,8 @@ paths: retry_if: type: object description: Conditional retry based on error or result - properties: *ref_195 - required: *ref_196 + properties: *ref_201 + required: *ref_202 no_flow_overlap: type: boolean description: >- @@ -21944,7 +23438,7 @@ paths: type: array items: type: string - required: &ref_437 + required: &ref_441 - path - schedule - timezone @@ -21988,7 +23482,7 @@ paths: application/json: schema: type: object - properties: &ref_438 + properties: &ref_442 schedule: type: string description: >- @@ -22063,7 +23557,7 @@ paths: nullable: true type: object description: Retry configuration for failed module executions - properties: *ref_197 + properties: *ref_203 no_flow_overlap: type: boolean description: >- @@ -22119,7 +23613,7 @@ paths: type: array items: type: string - required: &ref_439 + required: &ref_443 - schedule - timezone - args @@ -22208,189 +23702,257 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: schedule deleted content: application/json: schema: - type: object - properties: &ref_198 - path: - type: string - description: >- - The unique Windmill path for this schedule. Must be of the - form `u//` or `f//`. - edited_by: - type: string - description: Username of the last person who edited this schedule - edited_at: - type: string - format: date-time - description: Timestamp of the last edit - schedule: - type: string - description: >- - Cron expression with 6 fields (seconds, minutes, hours, - day of month, month, day of week). Example '0 0 12 * * *' - for daily at noon - timezone: - type: string - description: >- - IANA timezone for the schedule (e.g., 'UTC', - 'Europe/Paris', 'America/New_York') - enabled: - type: boolean - description: >- - Whether the schedule is currently active and will trigger - jobs - script_path: - type: string - description: Path to the script or flow to execute when triggered - is_flow: - type: boolean - description: >- - True if script_path points to a flow, false if it points - to a script - args: - nullable: true - type: object - description: The arguments to pass to the script or flow - additionalProperties: true - extra_perms: - type: object - additionalProperties: - type: boolean - description: Additional permissions for this schedule - email: - type: string - description: >- - Email of the user who owns this schedule, used for - permissioned_as - permissioned_as: - type: string - description: >- - The user or group this schedule runs as (e.g., 'u/admin' - or 'g/mygroup') - error: - type: string - nullable: true - description: Last error message if the schedule failed to trigger - on_failure: - type: string - nullable: true - description: >- - Path to a script or flow to run when the scheduled job - fails - on_failure_times: - type: number - nullable: true - description: >- - Number of consecutive failures before the on_failure - handler is triggered (default 1) - on_failure_exact: - type: boolean - nullable: true - description: >- - If true, trigger on_failure handler only on exactly N - failures, not on every failure after N - on_failure_extra_args: - nullable: true - type: object - description: The arguments to pass to the script or flow - additionalProperties: true - on_recovery: - type: string - nullable: true - description: >- - Path to a script or flow to run when the schedule recovers - after failures - on_recovery_times: - type: number - nullable: true - description: >- - Number of consecutive successes before the on_recovery - handler is triggered (default 1) - on_recovery_extra_args: - nullable: true - type: object - description: The arguments to pass to the script or flow - additionalProperties: true - on_success: - type: string - nullable: true - description: >- - Path to a script or flow to run after each successful - execution - on_success_extra_args: - nullable: true - type: object - description: The arguments to pass to the script or flow - additionalProperties: true - ws_error_handler_muted: - type: boolean - description: >- - If true, the workspace-level error handler will not be - triggered for this schedule's failures - retry: - nullable: true - type: object - description: Retry configuration for failed module executions - properties: *ref_197 - summary: - type: string - nullable: true - description: Short summary describing the purpose of this schedule - description: - type: string - nullable: true - description: Detailed description of what this schedule does - no_flow_overlap: - type: boolean - description: >- - If true, skip this schedule's execution if the previous - run is still in progress (prevents concurrent runs) - tag: - type: string - nullable: true - description: Worker tag to route jobs to specific worker groups - paused_until: - type: string - format: date-time - nullable: true - description: >- - ISO 8601 datetime until which the schedule is paused. - Schedule resumes automatically after this time - cron_version: - type: string - nullable: true - description: >- - Cron parser version. Use 'v2' for extended syntax with - additional features - dynamic_skip: - type: string - nullable: true - description: >- - Path to a script that validates scheduled datetimes. - Receives scheduled_for datetime and returns boolean to - skip (true) or run (false) - labels: - type: array - items: - type: string - default: [] - required: &ref_199 - - path - - edited_by - - edited_at - - schedule - - script_path - - timezone - - extra_perms - - is_flow - - enabled - - email - - permissioned_as + allOf: + - type: object + properties: &ref_204 + path: + type: string + description: >- + The unique Windmill path for this schedule. Must be of + the form `u//` or `f//`. + edited_by: + type: string + description: Username of the last person who edited this schedule + edited_at: + type: string + format: date-time + description: Timestamp of the last edit + schedule: + type: string + description: >- + Cron expression with 6 fields (seconds, minutes, + hours, day of month, month, day of week). Example '0 0 + 12 * * *' for daily at noon + timezone: + type: string + description: >- + IANA timezone for the schedule (e.g., 'UTC', + 'Europe/Paris', 'America/New_York') + enabled: + type: boolean + description: >- + Whether the schedule is currently active and will + trigger jobs + script_path: + type: string + description: Path to the script or flow to execute when triggered + is_flow: + type: boolean + description: >- + True if script_path points to a flow, false if it + points to a script + args: + nullable: true + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + extra_perms: + type: object + additionalProperties: + type: boolean + description: Additional permissions for this schedule + email: + type: string + description: >- + Email of the user who owns this schedule, used for + permissioned_as + permissioned_as: + type: string + description: >- + The user or group this schedule runs as (e.g., + 'u/admin' or 'g/mygroup') + error: + type: string + nullable: true + description: Last error message if the schedule failed to trigger + on_failure: + type: string + nullable: true + description: >- + Path to a script or flow to run when the scheduled job + fails + on_failure_times: + type: number + nullable: true + description: >- + Number of consecutive failures before the on_failure + handler is triggered (default 1) + on_failure_exact: + type: boolean + nullable: true + description: >- + If true, trigger on_failure handler only on exactly N + failures, not on every failure after N + on_failure_extra_args: + nullable: true + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + on_recovery: + type: string + nullable: true + description: >- + Path to a script or flow to run when the schedule + recovers after failures + on_recovery_times: + type: number + nullable: true + description: >- + Number of consecutive successes before the on_recovery + handler is triggered (default 1) + on_recovery_extra_args: + nullable: true + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + on_success: + type: string + nullable: true + description: >- + Path to a script or flow to run after each successful + execution + on_success_extra_args: + nullable: true + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + ws_error_handler_muted: + type: boolean + description: >- + If true, the workspace-level error handler will not be + triggered for this schedule's failures + retry: + nullable: true + type: object + description: Retry configuration for failed module executions + properties: *ref_203 + summary: + type: string + nullable: true + description: Short summary describing the purpose of this schedule + description: + type: string + nullable: true + description: Detailed description of what this schedule does + no_flow_overlap: + type: boolean + description: >- + If true, skip this schedule's execution if the + previous run is still in progress (prevents concurrent + runs) + tag: + type: string + nullable: true + description: Worker tag to route jobs to specific worker groups + paused_until: + type: string + format: date-time + nullable: true + description: >- + ISO 8601 datetime until which the schedule is paused. + Schedule resumes automatically after this time + cron_version: + type: string + nullable: true + description: >- + Cron parser version. Use 'v2' for extended syntax with + additional features + dynamic_skip: + type: string + nullable: true + description: >- + Path to a script that validates scheduled datetimes. + Receives scheduled_for datetime and returns boolean to + skip (true) or run (false) + labels: + type: array + items: + type: string + default: [] + draft_only: + description: > + True when this row is a per-user draft with no + deployed + + schedule at the same path. Frontend renders a "Draft" + badge. + type: boolean + is_draft: + description: > + True when the authed user has a per-user draft at this + path + + (over a deployed row or a synthesized draft-only row). + + Frontend appends a `*` to the displayed name. + type: boolean + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at + read time. Read-only — edit them on the folder. + required: &ref_205 + - path + - edited_by + - edited_at + - schedule + - script_path + - timezone + - extra_perms + - is_flow + - enabled + - email + - permissioned_as + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/schedules/exists/{path}: get: summary: does schedule exists @@ -22438,7 +24000,7 @@ paths: filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: path description: filter by path (script path) in: query @@ -22482,6 +24044,15 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + description: | + When true, append per-user draft schedules whose path has + no deployed schedule. Synthesized rows carry + `draft_only: true`. + in: query + required: false + schema: + type: boolean responses: '200': description: schedule list @@ -22491,8 +24062,8 @@ paths: type: array items: type: object - properties: *ref_198 - required: *ref_199 + properties: *ref_204 + required: *ref_205 /w/{workspace}/schedules/list_with_jobs: get: summary: list schedules with last 20 jobs @@ -22520,10 +24091,10 @@ paths: schema: type: array items: - allOf: &ref_435 + allOf: &ref_439 - type: object - properties: *ref_198 - required: *ref_199 + properties: *ref_204 + required: *ref_205 - type: object properties: jobs: @@ -22601,10 +24172,10 @@ paths: application/json: schema: type: object - properties: &ref_201 + properties: &ref_207 info: type: object - properties: &ref_445 + properties: &ref_449 title: type: string version: @@ -22633,28 +24204,28 @@ paths: type: string required: - name - required: &ref_446 + required: &ref_450 - title - version url: type: string openapi_spec_format: type: string - enum: &ref_440 + enum: &ref_444 - yaml - json http_route_filters: type: array items: type: object - properties: &ref_441 + properties: &ref_445 folder_regex: type: string path_regex: type: string route_path_regex: type: string - required: &ref_442 + required: &ref_446 - folder_regex - path_regex - route_path_regex @@ -22662,7 +24233,7 @@ paths: type: array items: type: object - properties: &ref_443 + properties: &ref_447 user_or_folder_regex: type: string enum: @@ -22675,8 +24246,8 @@ paths: type: string runnable_kind: type: string - enum: *ref_200 - required: &ref_444 + enum: *ref_206 + required: &ref_448 - user_or_folder_regex - user_or_folder_regex_value - path @@ -22705,7 +24276,7 @@ paths: application/json: schema: type: object - properties: *ref_201 + properties: *ref_207 responses: '200': description: Downloaded OpenAPI spec @@ -22734,7 +24305,7 @@ paths: type: array items: type: object - properties: &ref_202 + properties: &ref_208 path: type: string description: >- @@ -22788,7 +24359,7 @@ paths: HTTP method (get, post, put, delete, patch) that triggers this endpoint type: string - enum: &ref_204 + enum: &ref_210 - get - post - put @@ -22810,7 +24381,7 @@ paths: 'async' returns job ID immediately, 'sync_sse' streams results via Server-Sent Events type: string - enum: &ref_205 + enum: &ref_211 - sync - async - sync_sse @@ -22820,7 +24391,7 @@ paths: 'windmill' (Windmill token), 'api_key', 'basic_http', 'custom_script', 'signature' type: string - enum: &ref_206 + enum: &ref_212 - none - windmill - api_key @@ -22838,7 +24409,7 @@ paths: mode: description: job trigger mode type: string - enum: &ref_207 + enum: &ref_213 - enabled - disabled - suspended @@ -22859,7 +24430,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -22875,7 +24446,7 @@ paths: type: array items: type: string - required: &ref_203 + required: &ref_209 - path - script_path - route_path @@ -22908,8 +24479,8 @@ paths: application/json: schema: type: object - properties: *ref_202 - required: *ref_203 + properties: *ref_208 + required: *ref_209 responses: '201': description: http trigger created @@ -22939,7 +24510,7 @@ paths: application/json: schema: type: object - properties: &ref_447 + properties: &ref_451 path: type: string description: >- @@ -22999,7 +24570,7 @@ paths: HTTP method (get, post, put, delete, patch) that triggers this endpoint type: string - enum: *ref_204 + enum: *ref_210 is_async: type: boolean description: Deprecated, use request_type instead @@ -23009,14 +24580,14 @@ paths: 'async' returns job ID immediately, 'sync_sse' streams results via Server-Sent Events type: string - enum: *ref_205 + enum: *ref_211 authentication_method: description: >- How requests are authenticated - 'none' (public), 'windmill' (Windmill token), 'api_key', 'basic_http', 'custom_script', 'signature' type: string - enum: *ref_206 + enum: *ref_212 is_static_website: type: boolean description: >- @@ -23040,7 +24611,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -23056,7 +24627,7 @@ paths: type: array items: type: string - required: &ref_448 + required: &ref_452 - path - script_path - is_flow @@ -23108,167 +24679,239 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: http trigger deleted content: application/json: schema: - allOf: &ref_208 - - type: object - properties: &ref_214 - path: + allOf: + - allOf: &ref_215 + - type: object + properties: &ref_221 + path: + type: string + description: >- + The unique Windmill path for this trigger. Must be + of the form `u//` or + `f//`. This is the trigger object + path, not the HTTP route path. + script_path: + type: string + description: >- + Path to the script or flow to execute when + triggered + permissioned_as: + type: string + description: >- + The user or group this trigger runs as + (permissioned_as) + extra_perms: + type: object + description: Additional permissions for this trigger + additionalProperties: + type: boolean + workspace_id: + type: string + description: The workspace this trigger belongs to + edited_by: + type: string + description: >- + Username of the last person who edited this + trigger + edited_at: + type: string + format: date-time + description: Timestamp of the last edit + is_flow: + type: boolean + description: >- + True if script_path points to a flow, false if it + points to a script + mode: + description: job trigger mode + type: string + enum: *ref_213 + labels: + type: array + items: + type: string + default: [] + draft_only: + description: > + True when this row is a per-user draft with no + deployed + + trigger at the same path. Set by list endpoints + when + + `include_draft_only=true` synthesizes the row from + the + + draft. Frontend renders a "Draft" badge. + type: boolean + is_draft: + description: > + True when the authed user has a per-user draft at + this path + + (over a deployed row or a synthesized draft-only + row). + + Frontend appends a `*` to the displayed name. + type: boolean + required: &ref_222 + - path + - script_path + - permissioned_as + - extra_perms + - workspace_id + - edited_by + - edited_at + - is_flow + - mode + type: object + properties: &ref_216 + route_path: type: string description: >- - The unique Windmill path for this trigger. Must be of - the form `u//` or `f//`. - This is the trigger object path, not the HTTP route - path. - script_path: - type: string - description: Path to the script or flow to execute when triggered - permissioned_as: - type: string - description: >- - The user or group this trigger runs as - (permissioned_as) - extra_perms: + The URL route path that will trigger this endpoint + (e.g., 'api/myendpoint'). Must NOT start with a /. + static_asset_config: type: object - description: Additional permissions for this trigger - additionalProperties: - type: boolean - workspace_id: + nullable: true + description: >- + Configuration for serving static assets (s3 bucket, + storage path, filename) + properties: + s3: + type: string + description: S3 bucket path for static assets + storage: + type: string + description: Storage path for static assets + filename: + type: string + description: Filename for the static asset + required: + - s3 + http_method: + description: >- + HTTP method (get, post, put, delete, patch) that + triggers this endpoint type: string - description: The workspace this trigger belongs to - edited_by: + enum: *ref_210 + authentication_resource_path: type: string - description: Username of the last person who edited this trigger - edited_at: + nullable: true + description: >- + Path to the resource containing authentication + configuration (for api_key, basic_http, custom_script, + signature methods) + summary: type: string - format: date-time - description: Timestamp of the last edit - is_flow: + nullable: true + description: Short summary describing the purpose of this trigger + description: + type: string + nullable: true + description: Detailed description of what this trigger does + request_type: + description: >- + How the request is handled - 'sync' waits for result, + 'async' returns job ID immediately, 'sync_sse' streams + results via Server-Sent Events + type: string + enum: *ref_211 + authentication_method: + description: >- + How requests are authenticated - 'none' (public), + 'windmill' (Windmill token), 'api_key', 'basic_http', + 'custom_script', 'signature' + type: string + enum: *ref_212 + is_static_website: type: boolean description: >- - True if script_path points to a flow, false if it - points to a script - mode: - description: job trigger mode + If true, serves static files from S3/storage instead + of running a script + workspaced_route: + type: boolean + description: >- + If true, the route includes the workspace ID in the + path + wrap_body: + type: boolean + description: If true, wraps the request body in a 'body' parameter + raw_string: + type: boolean + description: >- + If true, passes the request body as a raw string + instead of parsing as JSON + error_handler_path: type: string - enum: *ref_207 - labels: - type: array - items: - type: string - default: [] - required: &ref_215 - - path - - script_path - - permissioned_as - - extra_perms - - workspace_id - - edited_by - - edited_at - - is_flow - - mode - type: object - properties: &ref_209 - route_path: - type: string - description: >- - The URL route path that will trigger this endpoint (e.g., - 'api/myendpoint'). Must NOT start with a /. - static_asset_config: - type: object - nullable: true - description: >- - Configuration for serving static assets (s3 bucket, - storage path, filename) - properties: - s3: - type: string - description: S3 bucket path for static assets - storage: - type: string - description: Storage path for static assets - filename: - type: string - description: Filename for the static asset - required: - - s3 - http_method: - description: >- - HTTP method (get, post, put, delete, patch) that triggers - this endpoint - type: string - enum: *ref_204 - authentication_resource_path: - type: string - nullable: true - description: >- - Path to the resource containing authentication - configuration (for api_key, basic_http, custom_script, - signature methods) - summary: - type: string - nullable: true - description: Short summary describing the purpose of this trigger - description: - type: string - nullable: true - description: Detailed description of what this trigger does - request_type: - description: >- - How the request is handled - 'sync' waits for result, - 'async' returns job ID immediately, 'sync_sse' streams - results via Server-Sent Events - type: string - enum: *ref_205 - authentication_method: - description: >- - How requests are authenticated - 'none' (public), - 'windmill' (Windmill token), 'api_key', 'basic_http', - 'custom_script', 'signature' - type: string - enum: *ref_206 - is_static_website: - type: boolean - description: >- - If true, serves static files from S3/storage instead of - running a script - workspaced_route: - type: boolean - description: If true, the route includes the workspace ID in the path - wrap_body: - type: boolean - description: If true, wraps the request body in a 'body' parameter - raw_string: - type: boolean - description: >- - If true, passes the request body as a raw string instead - of parsing as JSON - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_210 - - route_path - - request_type - - authentication_method - - http_method - - is_static_website - - workspaced_route - - wrap_body - - raw_string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_217 + - route_path + - request_type + - authentication_method + - http_method + - is_static_website + - workspaced_route + - wrap_body + - raw_string + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/http_triggers/list: get: summary: list http triggers @@ -23307,6 +24950,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: http trigger list @@ -23315,10 +24968,10 @@ paths: schema: type: array items: - allOf: *ref_208 + allOf: *ref_215 type: object - properties: *ref_209 - required: *ref_210 + properties: *ref_216 + required: *ref_217 /w/{workspace}/http_triggers/exists/{path}: get: summary: does http trigger exists @@ -23364,7 +25017,7 @@ paths: type: string http_method: type: string - enum: *ref_204 + enum: *ref_210 trigger_path: type: string workspaced_route: @@ -23404,7 +25057,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -23437,7 +25090,7 @@ paths: application/json: schema: type: object - properties: &ref_449 + properties: &ref_453 path: type: string description: >- @@ -23462,7 +25115,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 filters: type: array description: >- @@ -23493,7 +25146,7 @@ paths: Messages to send immediately after connecting (can be raw strings or computed by runnables) items: - anyOf: &ref_211 + anyOf: &ref_218 - type: object properties: raw_message: @@ -23536,7 +25189,7 @@ paths: nullable: true description: Optional periodic heartbeat message configuration type: object - properties: &ref_212 + properties: &ref_219 interval_secs: type: integer minimum: 1 @@ -23553,7 +25206,7 @@ paths: Optional. Top-level JSON field to extract from incoming messages. The extracted value replaces {{state}} in the heartbeat message. - required: &ref_213 + required: &ref_220 - interval_secs - message error_handler_path: @@ -23566,7 +25219,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -23582,7 +25235,7 @@ paths: type: array items: type: string - required: &ref_450 + required: &ref_454 - path - script_path - url @@ -23619,7 +25272,7 @@ paths: application/json: schema: type: object - properties: &ref_451 + properties: &ref_455 url: type: string description: >- @@ -23671,7 +25324,7 @@ paths: Messages to send immediately after connecting (can be raw strings or computed by runnables) items: - anyOf: *ref_211 + anyOf: *ref_218 url_runnable_args: description: The arguments to pass to the script or flow nullable: true @@ -23689,8 +25342,8 @@ paths: nullable: true description: Optional periodic heartbeat message configuration type: object - properties: *ref_212 - required: *ref_213 + properties: *ref_219 + required: *ref_220 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -23701,7 +25354,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -23717,7 +25370,7 @@ paths: type: array items: type: string - required: &ref_452 + required: &ref_456 - path - script_path - url @@ -23769,103 +25422,149 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: websocket trigger deleted content: application/json: schema: - allOf: &ref_216 + allOf: + - allOf: &ref_223 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_224 + url: + type: string + description: >- + The WebSocket URL to connect to (can be a static URL + or computed by a runnable) + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal) + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal) + error: + type: string + description: Last error message if the trigger failed + filters: + type: array + description: >- + Array of key-value filters to match incoming messages + (only matching messages trigger the script) + items: + type: object + properties: + key: + type: string + value: {} + required: + - key + - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: >- + Logic to apply when evaluating filters. 'and' requires + all filters to match, 'or' requires any filter to + match. + initial_messages: + type: array + nullable: true + description: >- + Messages to send immediately after connecting (can be + raw strings or computed by runnables) + items: + anyOf: *ref_218 + url_runnable_args: + description: The arguments to pass to the script or flow + nullable: true + type: object + additionalProperties: true + can_return_message: + type: boolean + description: >- + If true, the script can return a message to send back + through the WebSocket + can_return_error_result: + type: boolean + description: >- + If true, error results are sent back through the + WebSocket + heartbeat: + nullable: true + description: Optional periodic heartbeat message configuration + type: object + properties: *ref_219 + required: *ref_220 + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_225 + - url + - filters + - can_return_message + - can_return_error_result - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_217 - url: - type: string - description: >- - The WebSocket URL to connect to (can be a static URL or - computed by a runnable) - server_id: - type: string - description: >- - ID of the server currently handling this trigger - (internal) - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal) - error: - type: string - description: Last error message if the trigger failed - filters: - type: array - description: >- - Array of key-value filters to match incoming messages - (only matching messages trigger the script) - items: - type: object - properties: - key: - type: string - value: {} - required: - - key - - value - filter_logic: - type: string - enum: - - and - - or - default: and - description: >- - Logic to apply when evaluating filters. 'and' requires all - filters to match, 'or' requires any filter to match. - initial_messages: - type: array - nullable: true - description: >- - Messages to send immediately after connecting (can be raw - strings or computed by runnables) - items: - anyOf: *ref_211 - url_runnable_args: - description: The arguments to pass to the script or flow - nullable: true - type: object - additionalProperties: true - can_return_message: - type: boolean - description: >- - If true, the script can return a message to send back - through the WebSocket - can_return_error_result: - type: boolean - description: If true, error results are sent back through the WebSocket - heartbeat: - nullable: true - description: Optional periodic heartbeat message configuration - type: object - properties: *ref_212 - required: *ref_213 - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_218 - - url - - filters - - can_return_message - - can_return_error_result + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/websocket_triggers/list: get: summary: list websocket triggers @@ -23904,6 +25603,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: websocket trigger list @@ -23912,10 +25621,10 @@ paths: schema: type: array items: - allOf: *ref_216 + allOf: *ref_223 type: object - properties: *ref_217 - required: *ref_218 + properties: *ref_224 + required: *ref_225 /w/{workspace}/websocket_triggers/exists/{path}: get: summary: does websocket trigger exists @@ -23964,7 +25673,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -24034,7 +25743,7 @@ paths: application/json: schema: type: object - properties: &ref_484 + properties: &ref_488 path: type: string description: >- @@ -24103,7 +25812,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -24114,7 +25823,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24130,7 +25839,7 @@ paths: type: array items: type: string - required: &ref_485 + required: &ref_489 - path - script_path - is_flow @@ -24167,7 +25876,7 @@ paths: application/json: schema: type: object - properties: &ref_486 + properties: &ref_490 kafka_resource_path: type: string description: >- @@ -24243,7 +25952,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24259,7 +25968,7 @@ paths: type: array items: type: string - required: &ref_487 + required: &ref_491 - path - script_path - kafka_resource_path @@ -24311,98 +26020,143 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: kafka trigger deleted content: application/json: schema: - allOf: &ref_219 - - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_220 - kafka_resource_path: - type: string - description: >- - Path to the Kafka resource containing connection - configuration - group_id: - type: string - description: Kafka consumer group ID for this trigger - topics: - type: array - items: - type: string - description: Array of Kafka topic names to subscribe to - filters: - type: array - items: - type: object - properties: - key: + allOf: + - allOf: &ref_226 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_227 + kafka_resource_path: + type: string + description: >- + Path to the Kafka resource containing connection + configuration + group_id: + type: string + description: Kafka consumer group ID for this trigger + topics: + type: array + items: type: string - value: {} - required: - - key - - value - filter_logic: - type: string - enum: - - and - - or - default: and - description: >- - Logic to apply when evaluating filters. 'and' requires all - filters to match, 'or' requires any filter to match. - auto_offset_reset: - type: string - enum: - - latest - - earliest - default: latest - description: >- - Initial offset behavior when consumer group has no - committed offset. 'latest' starts from new messages only, - 'earliest' starts from the beginning. - auto_commit: - type: boolean - default: true - description: >- - When true (default), offsets are committed automatically - after receiving each message. When false, you must - manually commit offsets using the commit_offsets endpoint. - server_id: - type: string - description: >- - ID of the server currently handling this trigger - (internal) - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal) - error: - type: string - description: Last error message if the trigger failed - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_221 - - kafka_resource_path - - group_id - - topics - - filters + description: Array of Kafka topic names to subscribe to + filters: + type: array + items: + type: object + properties: + key: + type: string + value: {} + required: + - key + - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: >- + Logic to apply when evaluating filters. 'and' requires + all filters to match, 'or' requires any filter to + match. + auto_offset_reset: + type: string + enum: + - latest + - earliest + default: latest + description: >- + Initial offset behavior when consumer group has no + committed offset. 'latest' starts from new messages + only, 'earliest' starts from the beginning. + auto_commit: + type: boolean + default: true + description: >- + When true (default), offsets are committed + automatically after receiving each message. When + false, you must manually commit offsets using the + commit_offsets endpoint. + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal) + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal) + error: + type: string + description: Last error message if the trigger failed + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_228 + - kafka_resource_path + - group_id + - topics + - filters + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/kafka_triggers/list: get: summary: list kafka triggers @@ -24441,6 +26195,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: kafka trigger list @@ -24449,10 +26213,10 @@ paths: schema: type: array items: - allOf: *ref_219 + allOf: *ref_226 type: object - properties: *ref_220 - required: *ref_221 + properties: *ref_227 + required: *ref_228 /w/{workspace}/kafka_triggers/exists/{path}: get: summary: does kafka trigger exists @@ -24501,7 +26265,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -24620,7 +26384,7 @@ paths: application/json: schema: type: object - properties: &ref_488 + properties: &ref_492 path: type: string description: >- @@ -24663,7 +26427,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -24674,7 +26438,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24690,7 +26454,7 @@ paths: type: array items: type: string - required: &ref_489 + required: &ref_493 - path - script_path - is_flow @@ -24726,7 +26490,7 @@ paths: application/json: schema: type: object - properties: &ref_490 + properties: &ref_494 nats_resource_path: type: string description: >- @@ -24776,7 +26540,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -24792,7 +26556,7 @@ paths: type: array items: type: string - required: &ref_491 + required: &ref_495 - path - script_path - nats_resource_path @@ -24843,72 +26607,117 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: nats trigger deleted content: application/json: schema: - allOf: &ref_222 + allOf: + - allOf: &ref_229 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_230 + nats_resource_path: + type: string + description: >- + Path to the NATS resource containing connection + configuration + use_jetstream: + type: boolean + description: >- + If true, uses NATS JetStream for durable message + delivery + stream_name: + type: string + nullable: true + description: >- + JetStream stream name (required when use_jetstream is + true) + consumer_name: + type: string + nullable: true + description: >- + JetStream consumer name (required when use_jetstream + is true) + subjects: + type: array + items: + type: string + description: Array of NATS subjects to subscribe to + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal) + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal) + error: + type: string + description: Last error message if the trigger failed + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_231 + - nats_resource_path + - use_jetstream + - subjects - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_223 - nats_resource_path: - type: string - description: >- - Path to the NATS resource containing connection - configuration - use_jetstream: - type: boolean - description: If true, uses NATS JetStream for durable message delivery - stream_name: - type: string - nullable: true - description: >- - JetStream stream name (required when use_jetstream is - true) - consumer_name: - type: string - nullable: true - description: >- - JetStream consumer name (required when use_jetstream is - true) - subjects: - type: array - items: - type: string - description: Array of NATS subjects to subscribe to - server_id: - type: string - description: >- - ID of the server currently handling this trigger - (internal) - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal) - error: - type: string - description: Last error message if the trigger failed - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_224 - - nats_resource_path - - use_jetstream - - subjects + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/nats_triggers/list: get: summary: list nats triggers @@ -24947,6 +26756,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: nats trigger list @@ -24955,10 +26774,10 @@ paths: schema: type: array items: - allOf: *ref_222 + allOf: *ref_229 type: object - properties: *ref_223 - required: *ref_224 + properties: *ref_230 + required: *ref_231 /w/{workspace}/nats_triggers/exists/{path}: get: summary: does nats trigger exists @@ -25007,7 +26826,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -25070,7 +26889,7 @@ paths: application/json: schema: type: object - properties: &ref_471 + properties: &ref_475 queue_url: type: string description: The full URL of the AWS SQS queue to poll for messages @@ -25079,7 +26898,7 @@ paths: Authentication type - 'credentials' for access key/secret, 'oidc' for OpenID Connect type: string - enum: &ref_225 + enum: &ref_232 - oidc - credentials aws_resource_path: @@ -25114,7 +26933,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -25125,7 +26944,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -25141,7 +26960,7 @@ paths: type: array items: type: string - required: &ref_472 + required: &ref_476 - queue_url - aws_resource_path - path @@ -25177,7 +26996,7 @@ paths: application/json: schema: type: object - properties: &ref_473 + properties: &ref_477 queue_url: type: string description: The full URL of the AWS SQS queue to poll for messages @@ -25186,7 +27005,7 @@ paths: Authentication type - 'credentials' for access key/secret, 'oidc' for OpenID Connect type: string - enum: *ref_225 + enum: *ref_232 aws_resource_path: type: string description: >- @@ -25219,7 +27038,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -25230,7 +27049,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -25246,7 +27065,7 @@ paths: type: array items: type: string - required: &ref_474 + required: &ref_478 - queue_url - aws_resource_path - path @@ -25298,69 +27117,112 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: sqs trigger deleted content: application/json: schema: - allOf: &ref_226 + allOf: + - allOf: &ref_233 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_234 + queue_url: + type: string + description: The full URL of the AWS SQS queue to poll for messages + aws_auth_resource_type: + description: >- + Authentication type - 'credentials' for access + key/secret, 'oidc' for OpenID Connect + type: string + enum: *ref_232 + aws_resource_path: + type: string + description: >- + Path to the AWS resource containing credentials or + OIDC configuration + message_attributes: + type: array + nullable: true + items: + type: string + description: >- + Array of SQS message attribute names to include with + each message + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal) + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal) + error: + type: string + description: Last error message if the trigger failed + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_235 + - queue_url + - aws_resource_path + - aws_auth_resource_type - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_227 - queue_url: - type: string - description: The full URL of the AWS SQS queue to poll for messages - aws_auth_resource_type: - description: >- - Authentication type - 'credentials' for access key/secret, - 'oidc' for OpenID Connect - type: string - enum: *ref_225 - aws_resource_path: - type: string - description: >- - Path to the AWS resource containing credentials or OIDC - configuration - message_attributes: - type: array - nullable: true - items: - type: string - description: >- - Array of SQS message attribute names to include with each - message - server_id: - type: string - description: >- - ID of the server currently handling this trigger - (internal) - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal) - error: - type: string - description: Last error message if the trigger failed - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_228 - - queue_url - - aws_resource_path - - aws_auth_resource_type + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/sqs_triggers/list: get: summary: list sqs triggers @@ -25399,6 +27261,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: sqs trigger list @@ -25407,10 +27279,10 @@ paths: schema: type: array items: - allOf: *ref_226 + allOf: *ref_233 type: object - properties: *ref_227 - required: *ref_228 + properties: *ref_234 + required: *ref_235 /w/{workspace}/sqs_triggers/exists/{path}: get: summary: does sqs trigger exists @@ -25459,7 +27331,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -25524,17 +27396,17 @@ paths: type: array items: type: object - properties: &ref_576 + properties: &ref_579 service_name: type: string - enum: &ref_229 + enum: &ref_236 - nextcloud - google - github oauth_data: nullable: true type: object - properties: &ref_230 + properties: &ref_237 client_id: type: string description: The OAuth client ID for the workspace @@ -25549,7 +27421,7 @@ paths: type: string format: uri description: The OAuth redirect URI - required: &ref_231 + required: &ref_238 - client_id - client_secret - base_url @@ -25558,7 +27430,7 @@ paths: type: string nullable: true description: Path to the resource storing the OAuth token - required: &ref_577 + required: &ref_580 - service_name /w/{workspace}/native_triggers/integrations/{service_name}/exists: get: @@ -25576,7 +27448,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 responses: '200': description: integration exists @@ -25600,7 +27472,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 requestBody: description: new native trigger service required: true @@ -25608,8 +27480,8 @@ paths: application/json: schema: type: object - properties: *ref_230 - required: *ref_231 + properties: *ref_237 + required: *ref_238 responses: '201': description: native trigger service created @@ -25633,7 +27505,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 requestBody: description: redirect_uri required: true @@ -25641,10 +27513,10 @@ paths: application/json: schema: type: object - properties: &ref_232 + properties: &ref_239 redirect_uri: type: string - required: &ref_233 + required: &ref_240 - redirect_uri responses: '200': @@ -25669,7 +27541,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 responses: '200': description: whether instance sharing is available @@ -25693,7 +27565,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 requestBody: description: redirect_uri required: true @@ -25701,8 +27573,8 @@ paths: application/json: schema: type: object - properties: *ref_232 - required: *ref_233 + properties: *ref_239 + required: *ref_240 responses: '200': description: authorization URL using instance credentials @@ -25726,7 +27598,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 responses: '200': description: native trigger service deleted @@ -25750,7 +27622,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 requestBody: description: OAuth callback data required: true @@ -25799,7 +27671,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 requestBody: description: new native trigger configuration required: true @@ -25808,7 +27680,7 @@ paths: schema: type: object description: Data for creating or updating a native trigger - properties: &ref_234 + properties: &ref_241 script_path: type: string description: The path to the script or flow that will be triggered @@ -25825,7 +27697,7 @@ paths: type: string nullable: true description: Short summary to be displayed when listed - required: &ref_235 + required: &ref_242 - script_path - is_flow - service_config @@ -25837,13 +27709,13 @@ paths: schema: type: object description: Response returned when a native trigger is created - properties: &ref_579 + properties: &ref_582 external_id: type: string description: >- The external ID of the created trigger from the external service - required: &ref_580 + required: &ref_583 - external_id /w/{workspace}/native_triggers/{service_name}/update/{external_id}: post: @@ -25866,7 +27738,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 - name: external_id in: path required: true @@ -25881,8 +27753,8 @@ paths: schema: type: object description: Data for creating or updating a native trigger - properties: *ref_234 - required: *ref_235 + properties: *ref_241 + required: *ref_242 responses: '200': description: native trigger updated @@ -25911,7 +27783,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 - name: external_id in: path required: true @@ -25928,7 +27800,7 @@ paths: description: >- Full trigger response containing both Windmill data and external service data - properties: &ref_574 + properties: &ref_577 external_id: type: string description: The unique identifier from the external service @@ -25937,7 +27809,7 @@ paths: description: The workspace this trigger belongs to service_name: type: string - enum: *ref_229 + enum: *ref_236 script_path: type: string description: The path to the script or flow that will be triggered @@ -25964,7 +27836,7 @@ paths: type: object description: Configuration data from the external service additionalProperties: true - required: &ref_575 + required: &ref_578 - external_id - workspace_id - service_name @@ -25993,7 +27865,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 - name: external_id in: path required: true @@ -26024,7 +27896,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 - name: page description: which page to return (start at 1, default 1) in: query @@ -26049,6 +27921,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: native triggers list @@ -26059,7 +27941,7 @@ paths: items: type: object description: A native trigger stored in Windmill - properties: &ref_572 + properties: &ref_575 external_id: type: string description: The unique identifier from the external service @@ -26068,7 +27950,7 @@ paths: description: The workspace this trigger belongs to service_name: type: string - enum: *ref_229 + enum: *ref_236 script_path: type: string description: The path to the script or flow that will be triggered @@ -26091,7 +27973,7 @@ paths: type: string nullable: true description: Short summary to be displayed when listed - required: &ref_573 + required: &ref_576 - external_id - workspace_id - service_name @@ -26115,7 +27997,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 - name: external_id in: path required: true @@ -26145,7 +28027,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 responses: '200': description: sync completed successfully @@ -26170,7 +28052,7 @@ paths: type: array items: type: object - properties: &ref_581 + properties: &ref_584 id: type: string name: @@ -26181,7 +28063,7 @@ paths: type: string path: type: string - required: &ref_582 + required: &ref_585 - id - name - path @@ -26206,7 +28088,7 @@ paths: type: array items: type: object - properties: &ref_583 + properties: &ref_586 id: type: string summary: @@ -26214,7 +28096,7 @@ paths: primary: type: boolean default: false - required: &ref_584 + required: &ref_587 - id - summary /w/{workspace}/native_triggers/google/drive/files: @@ -26257,12 +28139,12 @@ paths: application/json: schema: type: object - properties: &ref_587 + properties: &ref_590 files: type: array items: type: object - properties: &ref_585 + properties: &ref_588 id: type: string name: @@ -26272,13 +28154,13 @@ paths: is_folder: type: boolean default: false - required: &ref_586 + required: &ref_589 - id - name - mime_type next_page_token: type: string - required: &ref_588 + required: &ref_591 - files /w/{workspace}/native_triggers/google/drive/shared_drives: get: @@ -26301,12 +28183,12 @@ paths: type: array items: type: object - properties: &ref_589 + properties: &ref_592 id: type: string name: type: string - required: &ref_590 + required: &ref_593 - id - name /w/{workspace}/native_triggers/github/repos: @@ -26330,7 +28212,7 @@ paths: type: array items: type: object - properties: &ref_591 + properties: &ref_594 full_name: type: string name: @@ -26339,7 +28221,7 @@ paths: type: string private: type: boolean - required: &ref_592 + required: &ref_595 - full_name - name - owner @@ -26356,7 +28238,7 @@ paths: required: true schema: type: string - enum: *ref_229 + enum: *ref_236 - name: workspace_id in: path required: true @@ -26405,7 +28287,7 @@ paths: application/json: schema: type: object - properties: &ref_454 + properties: &ref_458 mqtt_resource_path: type: string description: >- @@ -26415,16 +28297,16 @@ paths: type: array items: type: object - properties: &ref_236 + properties: &ref_243 qos: type: string - enum: &ref_453 + enum: &ref_457 - qos0 - qos1 - qos2 topic: type: string - required: &ref_237 + required: &ref_244 - qos - topic description: >- @@ -26438,7 +28320,7 @@ paths: nullable: true description: MQTT v3 specific configuration (clean_session) type: object - properties: &ref_238 + properties: &ref_245 clean_session: type: boolean v5_config: @@ -26447,7 +28329,7 @@ paths: MQTT v5 specific configuration (clean_start, topic_alias_maximum, session_expiry_interval) type: object - properties: &ref_239 + properties: &ref_246 clean_start: type: boolean topic_alias_maximum: @@ -26458,7 +28340,7 @@ paths: nullable: true description: MQTT protocol version ('v3' or 'v5') type: string - enum: &ref_240 + enum: &ref_247 - v3 - v5 path: @@ -26480,7 +28362,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -26491,7 +28373,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -26507,7 +28389,7 @@ paths: type: array items: type: string - required: &ref_455 + required: &ref_459 - path - script_path - is_flow @@ -26542,7 +28424,7 @@ paths: application/json: schema: type: object - properties: &ref_456 + properties: &ref_460 mqtt_resource_path: type: string description: >- @@ -26552,8 +28434,8 @@ paths: type: array items: type: object - properties: *ref_236 - required: *ref_237 + properties: *ref_243 + required: *ref_244 description: >- Array of MQTT topics to subscribe to, each with topic name and QoS level @@ -26565,19 +28447,19 @@ paths: nullable: true description: MQTT v3 specific configuration (clean_session) type: object - properties: *ref_238 + properties: *ref_245 v5_config: nullable: true description: >- MQTT v5 specific configuration (clean_start, topic_alias_maximum, session_expiry_interval) type: object - properties: *ref_239 + properties: *ref_246 client_version: nullable: true description: MQTT protocol version ('v3' or 'v5') type: string - enum: *ref_240 + enum: *ref_247 path: type: string description: >- @@ -26597,7 +28479,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -26608,7 +28490,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -26624,7 +28506,7 @@ paths: type: array items: type: string - required: &ref_457 + required: &ref_461 - path - script_path - is_flow @@ -26675,81 +28557,124 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: mqtt trigger deleted content: application/json: schema: - allOf: &ref_241 + allOf: + - allOf: &ref_248 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_249 + mqtt_resource_path: + type: string + description: >- + Path to the MQTT resource containing broker connection + configuration + subscribe_topics: + type: array + items: + type: object + properties: *ref_243 + required: *ref_244 + description: >- + Array of MQTT topics to subscribe to, each with topic + name and QoS level + v3_config: + nullable: true + description: MQTT v3 specific configuration (clean_session) + type: object + properties: *ref_245 + v5_config: + nullable: true + description: >- + MQTT v5 specific configuration (clean_start, + topic_alias_maximum, session_expiry_interval) + type: object + properties: *ref_246 + client_id: + type: string + nullable: true + description: MQTT client ID for this connection + client_version: + nullable: true + description: MQTT protocol version ('v3' or 'v5') + type: string + enum: *ref_247 + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal) + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal) + error: + type: string + description: Last error message if the trigger failed + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_250 + - subscribe_topics + - mqtt_resource_path - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_242 - mqtt_resource_path: - type: string - description: >- - Path to the MQTT resource containing broker connection - configuration - subscribe_topics: - type: array - items: - type: object - properties: *ref_236 - required: *ref_237 - description: >- - Array of MQTT topics to subscribe to, each with topic name - and QoS level - v3_config: - nullable: true - description: MQTT v3 specific configuration (clean_session) - type: object - properties: *ref_238 - v5_config: - nullable: true - description: >- - MQTT v5 specific configuration (clean_start, - topic_alias_maximum, session_expiry_interval) - type: object - properties: *ref_239 - client_id: - type: string - nullable: true - description: MQTT client ID for this connection - client_version: - nullable: true - description: MQTT protocol version ('v3' or 'v5') - type: string - enum: *ref_240 - server_id: - type: string - description: >- - ID of the server currently handling this trigger - (internal) - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal) - error: - type: string - description: Last error message if the trigger failed - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_243 - - subscribe_topics - - mqtt_resource_path + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/mqtt_triggers/list: get: summary: list mqtt triggers @@ -26788,6 +28713,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: mqtt trigger list @@ -26796,10 +28731,10 @@ paths: schema: type: array items: - allOf: *ref_241 + allOf: *ref_248 type: object - properties: *ref_242 - required: *ref_243 + properties: *ref_249 + required: *ref_250 /w/{workspace}/mqtt_triggers/exists/{path}: get: summary: does mqtt trigger exists @@ -26848,7 +28783,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -26912,7 +28847,7 @@ paths: schema: type: object description: Data for creating or updating a Google Cloud Pub/Sub trigger. - properties: &ref_244 + properties: &ref_251 gcp_resource_path: type: string description: >- @@ -26920,7 +28855,7 @@ paths: credentials for authentication. subscription_mode: type: string - enum: &ref_249 + enum: &ref_256 - existing - create_update description: >- @@ -26938,7 +28873,7 @@ paths: description: Base URL for push delivery endpoint. delivery_type: type: string - enum: &ref_246 + enum: &ref_253 - push - pull description: >- @@ -26949,7 +28884,7 @@ paths: nullable: true type: object description: Configuration for push delivery mode. - properties: &ref_247 + properties: &ref_254 audience: type: string description: >- @@ -26960,7 +28895,7 @@ paths: description: >- If true, push messages will include OIDC authentication tokens. - required: &ref_248 + required: &ref_255 - authenticate - base_endpoint path: @@ -26982,7 +28917,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 auto_acknowledge_msg: type: boolean description: >- @@ -27009,7 +28944,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -27025,7 +28960,7 @@ paths: type: array items: type: string - required: &ref_245 + required: &ref_252 - path - script_path - is_flow @@ -27062,8 +28997,8 @@ paths: schema: type: object description: Data for creating or updating a Google Cloud Pub/Sub trigger. - properties: *ref_244 - required: *ref_245 + properties: *ref_251 + required: *ref_252 responses: '200': description: gcp trigger updated @@ -27108,83 +29043,127 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: gcp trigger deleted content: application/json: schema: - allOf: &ref_250 + allOf: + - allOf: &ref_257 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + description: >- + A Google Cloud Pub/Sub trigger that executes a script or + flow when messages are received. + properties: &ref_258 + gcp_resource_path: + type: string + description: >- + Path to the GCP resource containing service account + credentials for authentication. + topic_id: + type: string + description: Google Cloud Pub/Sub topic ID to subscribe to. + subscription_id: + type: string + description: Google Cloud Pub/Sub subscription ID. + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal use). + delivery_type: + type: string + enum: *ref_253 + description: >- + Delivery mode for messages. 'push' for HTTP push + delivery where messages are sent to a webhook + endpoint, 'pull' for polling where the trigger + actively fetches messages. + delivery_config: + nullable: true + type: object + description: Configuration for push delivery mode. + properties: *ref_254 + required: *ref_255 + subscription_mode: + type: string + enum: *ref_256 + description: >- + The mode of subscription. 'existing' means using an + existing GCP subscription, while 'create_update' + involves creating or updating a new subscription. + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal use). + error: + type: string + description: Last error message if the trigger failed. + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails. + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_259 + - gcp_resource_path + - topic_id + - subscription_id + - delivery_type + - subscription_mode - type: object - properties: *ref_214 - required: *ref_215 - type: object - description: >- - A Google Cloud Pub/Sub trigger that executes a script or flow - when messages are received. - properties: &ref_251 - gcp_resource_path: - type: string - description: >- - Path to the GCP resource containing service account - credentials for authentication. - topic_id: - type: string - description: Google Cloud Pub/Sub topic ID to subscribe to. - subscription_id: - type: string - description: Google Cloud Pub/Sub subscription ID. - server_id: - type: string - description: >- - ID of the server currently handling this trigger (internal - use). - delivery_type: - type: string - enum: *ref_246 - description: >- - Delivery mode for messages. 'push' for HTTP push delivery - where messages are sent to a webhook endpoint, 'pull' for - polling where the trigger actively fetches messages. - delivery_config: - nullable: true - type: object - description: Configuration for push delivery mode. - properties: *ref_247 - required: *ref_248 - subscription_mode: - type: string - enum: *ref_249 - description: >- - The mode of subscription. 'existing' means using an - existing GCP subscription, while 'create_update' involves - creating or updating a new subscription. - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal use). - error: - type: string - description: Last error message if the trigger failed. - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails. - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_252 - - gcp_resource_path - - topic_id - - subscription_id - - delivery_type - - subscription_mode + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/gcp_triggers/list: get: summary: list gcp triggers @@ -27223,6 +29202,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: gcp trigger list @@ -27231,13 +29220,13 @@ paths: schema: type: array items: - allOf: *ref_250 + allOf: *ref_257 type: object description: >- A Google Cloud Pub/Sub trigger that executes a script or flow when messages are received. - properties: *ref_251 - required: *ref_252 + properties: *ref_258 + required: *ref_259 /w/{workspace}/gcp_triggers/exists/{path}: get: summary: does gcp trigger exists @@ -27286,7 +29275,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -27353,10 +29342,10 @@ paths: application/json: schema: type: object - properties: &ref_460 + properties: &ref_464 subscription_id: type: string - required: &ref_461 + required: &ref_465 - subscription_id responses: '200': @@ -27411,10 +29400,10 @@ paths: application/json: schema: type: object - properties: &ref_458 + properties: &ref_462 topic_id: type: string - required: &ref_459 + required: &ref_463 - topic_id responses: '200': @@ -27443,12 +29432,12 @@ paths: schema: type: object description: Data for creating or updating an Azure Event Grid trigger. - properties: &ref_253 + properties: &ref_260 azure_resource_path: type: string azure_mode: type: string - enum: &ref_255 + enum: &ref_262 - basic_push - namespace_push - namespace_pull @@ -27480,7 +29469,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 error_handler_path: type: string error_handler_args: @@ -27490,7 +29479,7 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string preserve_permissioned_as: @@ -27499,7 +29488,7 @@ paths: type: array items: type: string - required: &ref_254 + required: &ref_261 - path - script_path - is_flow @@ -27536,8 +29525,8 @@ paths: schema: type: object description: Data for creating or updating an Azure Event Grid trigger. - properties: *ref_253 - required: *ref_254 + properties: *ref_260 + required: *ref_261 responses: '200': description: azure trigger updated @@ -27582,65 +29571,110 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: azure trigger content: application/json: schema: - allOf: &ref_256 - - type: object - properties: *ref_214 - required: *ref_215 - type: object - description: >- - An Azure Event Grid trigger that executes a script or flow - when events arrive. - properties: &ref_257 - azure_resource_path: - type: string - azure_mode: - type: string - enum: *ref_255 - description: Azure Event Grid trigger mode. - scope_resource_id: - type: string + allOf: + - allOf: &ref_263 + - type: object + properties: *ref_221 + required: *ref_222 + type: object description: >- - ARM resource ID of the topic (basic) or namespace - (namespace modes). - topic_name: - type: string - nullable: true - description: Topic name within the namespace (namespace modes only). - subscription_name: - type: string - event_type_filters: - type: array - items: - type: string - nullable: true - server_id: - type: string - last_server_ping: - type: string - format: date-time - error: - type: string - error_handler_path: - type: string - error_handler_args: - type: object - description: The arguments to pass to the script or flow - additionalProperties: true - retry: - type: object - description: Retry configuration for failed module executions - properties: *ref_197 - required: &ref_258 - - azure_resource_path - - azure_mode - - scope_resource_id - - subscription_name + An Azure Event Grid trigger that executes a script or flow + when events arrive. + properties: &ref_264 + azure_resource_path: + type: string + azure_mode: + type: string + enum: *ref_262 + description: Azure Event Grid trigger mode. + scope_resource_id: + type: string + description: >- + ARM resource ID of the topic (basic) or namespace + (namespace modes). + topic_name: + type: string + nullable: true + description: >- + Topic name within the namespace (namespace modes + only). + subscription_name: + type: string + event_type_filters: + type: array + items: + type: string + nullable: true + server_id: + type: string + last_server_ping: + type: string + format: date-time + error: + type: string + error_handler_path: + type: string + error_handler_args: + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + retry: + type: object + description: Retry configuration for failed module executions + properties: *ref_203 + required: &ref_265 + - azure_resource_path + - azure_mode + - scope_resource_id + - subscription_name + - type: object + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/azure_triggers/list: get: summary: list azure triggers @@ -27673,6 +29707,16 @@ paths: in: query schema: type: string + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: azure trigger list @@ -27681,13 +29725,13 @@ paths: schema: type: array items: - allOf: *ref_256 + allOf: *ref_263 type: object description: >- An Azure Event Grid trigger that executes a script or flow when events arrive. - properties: *ref_257 - required: *ref_258 + properties: *ref_264 + required: *ref_265 /w/{workspace}/azure_triggers/exists/{path}: get: summary: check whether an azure trigger exists @@ -27735,7 +29779,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -27767,10 +29811,10 @@ paths: application/json: schema: type: object - properties: &ref_464 + properties: &ref_468 azure_resource_path: type: string - required: &ref_465 + required: &ref_469 - azure_resource_path responses: '200': @@ -27800,10 +29844,10 @@ paths: application/json: schema: type: object - properties: &ref_466 + properties: &ref_470 scope_resource_id: type: string - required: &ref_467 + required: &ref_471 - scope_resource_id responses: '200': @@ -27835,12 +29879,12 @@ paths: application/json: schema: type: object - properties: &ref_468 + properties: &ref_472 scope_resource_id: type: string topic_name: type: string - required: &ref_469 + required: &ref_473 - scope_resource_id - topic_name responses: @@ -27873,10 +29917,10 @@ paths: application/json: schema: type: object - properties: &ref_462 + properties: &ref_466 azure_mode: type: string - enum: *ref_255 + enum: *ref_262 description: Azure Event Grid trigger mode. scope_resource_id: type: string @@ -27885,7 +29929,7 @@ paths: nullable: true subscription_name: type: string - required: &ref_463 + required: &ref_467 - azure_mode - scope_resource_id - subscription_name @@ -27921,7 +29965,7 @@ paths: items: type: object description: An ARM resource the service principal can see. - properties: &ref_259 + properties: &ref_266 id: type: string name: @@ -27930,7 +29974,7 @@ paths: type: string type: type: string - required: &ref_260 + required: &ref_267 - id - name - type @@ -27961,8 +30005,8 @@ paths: items: type: object description: An ARM resource the service principal can see. - properties: *ref_259 - required: *ref_260 + properties: *ref_266 + required: *ref_267 /w/{workspace}/postgres_triggers/postgres/version/{path}: get: summary: get postgres version @@ -28025,19 +30069,19 @@ paths: application/json: schema: type: object - properties: &ref_478 + properties: &ref_482 postgres_resource_path: type: string relations: type: array items: type: object - properties: &ref_262 + properties: &ref_269 schema_name: type: string table_to_track: type: array - items: &ref_476 + items: &ref_480 type: object properties: table_name: @@ -28050,14 +30094,14 @@ paths: type: string required: - table_name - required: &ref_263 + required: &ref_270 - schema_name - table_to_track language: type: string - enum: &ref_477 + enum: &ref_481 - Typescript - required: &ref_479 + required: &ref_483 - postgres_resource_path - relations - language @@ -28082,7 +30126,7 @@ paths: - name: id in: path required: true - schema: &ref_304 + schema: &ref_311 type: string responses: '200': @@ -28115,7 +30159,7 @@ paths: type: array items: type: object - properties: &ref_475 + properties: &ref_479 slot_name: type: string active: @@ -28142,7 +30186,7 @@ paths: application/json: schema: type: object - properties: &ref_261 + properties: &ref_268 name: type: string responses: @@ -28174,7 +30218,7 @@ paths: application/json: schema: type: object - properties: *ref_261 + properties: *ref_268 responses: '200': description: postgres replication slot deleted @@ -28225,7 +30269,7 @@ paths: in: path required: true description: The name of the publication - schema: &ref_264 + schema: &ref_271 type: string responses: '200': @@ -28234,18 +30278,18 @@ paths: application/json: schema: type: object - properties: &ref_265 + properties: &ref_272 table_to_track: type: array items: type: object - properties: *ref_262 - required: *ref_263 + properties: *ref_269 + required: *ref_270 transaction_to_track: type: array items: type: string - required: &ref_266 + required: &ref_273 - transaction_to_track /w/{workspace}/postgres_triggers/publication/create/{publication}/{path}: post: @@ -28266,7 +30310,7 @@ paths: in: path required: true description: The name of the publication - schema: *ref_264 + schema: *ref_271 requestBody: description: new publication for postgres required: true @@ -28274,8 +30318,8 @@ paths: application/json: schema: type: object - properties: *ref_265 - required: *ref_266 + properties: *ref_272 + required: *ref_273 responses: '201': description: publication created @@ -28302,7 +30346,7 @@ paths: in: path required: true description: The name of the publication - schema: *ref_264 + schema: *ref_271 requestBody: description: update publication for postgres required: true @@ -28310,8 +30354,8 @@ paths: application/json: schema: type: object - properties: *ref_265 - required: *ref_266 + properties: *ref_272 + required: *ref_273 responses: '201': description: publication updated @@ -28338,7 +30382,7 @@ paths: in: path required: true description: The name of the publication - schema: *ref_264 + schema: *ref_271 responses: '200': description: postgres publication deleted @@ -28364,7 +30408,7 @@ paths: application/json: schema: type: object - properties: &ref_480 + properties: &ref_484 replication_slot_name: type: string description: Name of the PostgreSQL logical replication slot to use @@ -28392,7 +30436,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 postgres_resource_path: type: string description: >- @@ -28403,8 +30447,8 @@ paths: Configuration for creating/managing the publication (tables, operations) type: object - properties: *ref_265 - required: *ref_266 + properties: *ref_272 + required: *ref_273 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -28415,7 +30459,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -28431,7 +30475,7 @@ paths: type: array items: type: string - required: &ref_481 + required: &ref_485 - path - script_path - is_flow @@ -28466,7 +30510,7 @@ paths: application/json: schema: type: object - properties: &ref_482 + properties: &ref_486 replication_slot_name: type: string description: Name of the PostgreSQL logical replication slot to use @@ -28494,7 +30538,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 postgres_resource_path: type: string description: >- @@ -28505,8 +30549,8 @@ paths: Configuration for creating/managing the publication (tables, operations) type: object - properties: *ref_265 - required: *ref_266 + properties: *ref_272 + required: *ref_273 error_handler_path: type: string description: Path to a script or flow to run when the triggered job fails @@ -28517,7 +30561,7 @@ paths: retry: description: Retry configuration for failed module executions type: object - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -28533,7 +30577,7 @@ paths: type: array items: type: string - required: &ref_483 + required: &ref_487 - path - script_path - is_flow @@ -28585,60 +30629,103 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: get postgres trigger content: application/json: schema: - allOf: &ref_267 + allOf: + - allOf: &ref_274 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_275 + postgres_resource_path: + type: string + description: >- + Path to the PostgreSQL resource containing connection + configuration + publication_name: + type: string + description: >- + Name of the PostgreSQL publication to subscribe to for + change data capture + server_id: + type: string + description: >- + ID of the server currently handling this trigger + (internal) + replication_slot_name: + type: string + description: Name of the PostgreSQL logical replication slot to use + error: + type: string + description: Last error message if the trigger failed + last_server_ping: + type: string + format: date-time + description: Timestamp of last server heartbeat (internal) + error_handler_path: + type: string + description: >- + Path to a script or flow to run when the triggered job + fails + error_handler_args: + description: The arguments to pass to the script or flow + type: object + additionalProperties: true + retry: + description: Retry configuration for failed module executions + type: object + properties: *ref_203 + required: &ref_276 + - postgres_resource_path + - replication_slot_name + - publication_name - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_268 - postgres_resource_path: - type: string - description: >- - Path to the PostgreSQL resource containing connection - configuration - publication_name: - type: string - description: >- - Name of the PostgreSQL publication to subscribe to for - change data capture - server_id: - type: string - description: >- - ID of the server currently handling this trigger - (internal) - replication_slot_name: - type: string - description: Name of the PostgreSQL logical replication slot to use - error: - type: string - description: Last error message if the trigger failed - last_server_ping: - type: string - format: date-time - description: Timestamp of last server heartbeat (internal) - error_handler_path: - type: string - description: >- - Path to a script or flow to run when the triggered job - fails - error_handler_args: - description: The arguments to pass to the script or flow - type: object - additionalProperties: true - retry: - description: Retry configuration for failed module executions - type: object - properties: *ref_197 - required: &ref_269 - - postgres_resource_path - - replication_slot_name - - publication_name + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/postgres_triggers/list: get: summary: list postgres triggers @@ -28677,6 +30764,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: postgres trigger list @@ -28685,10 +30782,10 @@ paths: schema: type: array items: - allOf: *ref_267 + allOf: *ref_274 type: object - properties: *ref_268 - required: *ref_269 + properties: *ref_275 + required: *ref_276 /w/{workspace}/postgres_triggers/exists/{path}: get: summary: does postgres trigger exists @@ -28737,7 +30834,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -28800,7 +30897,7 @@ paths: application/json: schema: type: object - properties: &ref_492 + properties: &ref_496 path: type: string script_path: @@ -28820,11 +30917,11 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_197 + properties: *ref_203 mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 permissioned_as: type: string description: >- @@ -28840,7 +30937,7 @@ paths: type: array items: type: string - required: &ref_493 + required: &ref_497 - path - script_path - local_part @@ -28874,7 +30971,7 @@ paths: application/json: schema: type: object - properties: &ref_494 + properties: &ref_498 path: type: string script_path: @@ -28894,7 +30991,7 @@ paths: retry: type: object description: Retry configuration for failed module executions - properties: *ref_197 + properties: *ref_203 permissioned_as: type: string description: >- @@ -28910,7 +31007,7 @@ paths: type: array items: type: string - required: &ref_495 + required: &ref_499 - path - script_path - is_flow @@ -28958,34 +31055,77 @@ paths: in: path required: true schema: *ref_60 + - name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the + deployed payload. + schema: *ref_77 responses: '200': description: email trigger retrieved content: application/json: schema: - allOf: &ref_270 + allOf: + - allOf: &ref_277 + - type: object + properties: *ref_221 + required: *ref_222 + type: object + properties: &ref_278 + local_part: + type: string + workspaced_local_part: + type: boolean + error_handler_path: + type: string + error_handler_args: + type: object + description: The arguments to pass to the script or flow + additionalProperties: true + retry: + type: object + description: Retry configuration for failed module executions + properties: *ref_203 + required: &ref_279 + - local_part - type: object - properties: *ref_214 - required: *ref_215 - type: object - properties: &ref_271 - local_part: - type: string - workspaced_local_part: - type: boolean - error_handler_path: - type: string - error_handler_args: - type: object - description: The arguments to pass to the script or flow - additionalProperties: true - retry: - type: object - description: Retry configuration for failed module executions - properties: *ref_197 - required: &ref_272 - - local_part + description: > + Overlay fields added to every "get by path" response that + accepts + + the `get_draft` query parameter. The deployed payload is + sent + + untouched in the response body; the authed user's saved + draft + + for this path — whatever shape the editor wrote — is + attached + + as the sibling `draft` field when `get_draft=true` and a + draft + + exists. The frontend pairs the two to present diff / reset + / + + discard UI; the server never merges them. + + + When `no_deployed=true` there is no deployed row at this + path — + + the response body is a best-effort stand-in synthesized + from + + the draft, and only `draft` is canonical. Callers should + disable + + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 /w/{workspace}/email_triggers/list: get: summary: list email triggers @@ -29024,6 +31164,16 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 responses: '200': description: email trigger list @@ -29032,10 +31182,10 @@ paths: schema: type: array items: - allOf: *ref_270 + allOf: *ref_277 type: object - properties: *ref_271 - required: *ref_272 + properties: *ref_278 + required: *ref_279 /w/{workspace}/email_triggers/exists/{path}: get: summary: does email trigger exists @@ -29117,7 +31267,7 @@ paths: mode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 force: type: boolean description: > @@ -29147,9 +31297,9 @@ paths: type: array items: type: object - required: &ref_496 + required: &ref_500 - name - properties: &ref_497 + properties: &ref_501 name: type: string summary: @@ -29179,9 +31329,9 @@ paths: type: array items: type: object - required: &ref_274 + required: &ref_281 - name - properties: &ref_275 + properties: &ref_282 name: type: string summary: @@ -29200,14 +31350,14 @@ paths: type: array items: type: object - properties: &ref_498 + properties: &ref_502 workspace_id: type: string workspace_name: type: string role: type: string - required: &ref_499 + required: &ref_503 - name /groups/get/{name}: get: @@ -29219,7 +31369,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: instance group @@ -29227,8 +31377,8 @@ paths: application/json: schema: type: object - required: *ref_274 - properties: *ref_275 + required: *ref_281 + properties: *ref_282 /groups/create: post: summary: create instance group @@ -29266,7 +31416,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: update instance group required: true @@ -29302,7 +31452,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: instance group deleted @@ -29320,7 +31470,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: user to add to instance group required: true @@ -29350,7 +31500,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: user to remove from instance group required: true @@ -29385,7 +31535,7 @@ paths: type: array items: type: object - properties: &ref_276 + properties: &ref_283 name: type: string summary: @@ -29406,7 +31556,7 @@ paths: enum: - superadmin - devops - required: &ref_277 + required: &ref_284 - name /groups/overwrite: post: @@ -29423,8 +31573,8 @@ paths: type: array items: type: object - properties: *ref_276 - required: *ref_277 + properties: *ref_283 + required: *ref_284 responses: '200': description: success message @@ -29460,7 +31610,7 @@ paths: type: array items: type: object - properties: &ref_278 + properties: &ref_285 name: type: string summary: @@ -29473,7 +31623,7 @@ paths: type: object additionalProperties: type: boolean - required: &ref_279 + required: &ref_286 - name /w/{workspace}/groups/listnames: get: @@ -29546,7 +31696,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: updated group required: true @@ -29578,7 +31728,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: group deleted @@ -29600,7 +31750,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: group @@ -29608,8 +31758,8 @@ paths: application/json: schema: type: object - properties: *ref_278 - required: *ref_279 + properties: *ref_285 + required: *ref_286 /w/{workspace}/groups/adduser/{name}: post: summary: add user to group @@ -29624,7 +31774,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: added user to group required: true @@ -29656,7 +31806,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: added user to group required: true @@ -29688,7 +31838,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 - name: page description: which page to return (start at 1, default 1) in: query @@ -29747,7 +31897,7 @@ paths: type: array items: type: object - properties: &ref_281 + properties: &ref_288 name: type: string owners: @@ -29773,7 +31923,7 @@ paths: (relative to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: &ref_280 + items: &ref_287 type: object required: - path_glob @@ -29794,7 +31944,15 @@ paths: permissioned as. Must be `u/`, `g/`, or an email that exists in this workspace. - required: &ref_282 + labels: + type: array + items: + type: string + description: > + Labels set on the folder. Items inside the folder + inherit them, exposed as `inherited_labels` on scripts + and flows and stamped into job labels at run time. + required: &ref_289 - name - owners - extra_perms @@ -29861,7 +32019,11 @@ paths: to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: *ref_280 + items: *ref_287 + labels: + type: array + items: + type: string required: - name responses: @@ -29885,7 +32047,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: update folder required: true @@ -29911,7 +32073,11 @@ paths: to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: *ref_280 + items: *ref_287 + labels: + type: array + items: + type: string responses: '200': description: folder updated @@ -29933,7 +32099,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: folder deleted @@ -29955,7 +32121,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: folder @@ -29963,8 +32129,8 @@ paths: application/json: schema: type: object - properties: *ref_281 - required: *ref_282 + properties: *ref_288 + required: *ref_289 /w/{workspace}/folders/exists/{name}: get: summary: exists folder @@ -29979,7 +32145,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: folder exists @@ -30001,7 +32167,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: folder @@ -30043,7 +32209,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: owner user to folder required: true @@ -30077,7 +32243,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: added owner to folder required: true @@ -30113,7 +32279,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 - name: page description: which page to return (start at 1, default 1) in: query @@ -30177,7 +32343,7 @@ paths: type: array items: type: object - properties: &ref_500 + properties: &ref_504 worker: type: string worker_instance: @@ -30223,7 +32389,7 @@ paths: type: string native_mode: type: boolean - required: &ref_501 + required: &ref_505 - worker - worker_instance - ping_at @@ -30388,7 +32554,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: a config @@ -30397,12 +32563,12 @@ paths: schema: type: object nullable: true - properties: &ref_384 + properties: &ref_391 alerts: type: array items: type: object - properties: &ref_382 + properties: &ref_389 name: type: string tags_to_monitor: @@ -30415,7 +32581,7 @@ paths: type: integer alert_time_threshold_seconds: type: integer - required: &ref_383 + required: &ref_390 - name - tags_to_monitor - jobs_num_threshold @@ -30431,7 +32597,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 requestBody: description: worker group required: true @@ -30454,7 +32620,7 @@ paths: - name: name in: path required: true - schema: *ref_273 + schema: *ref_280 responses: '200': description: Delete config @@ -30477,12 +32643,12 @@ paths: type: array items: type: object - properties: &ref_547 + properties: &ref_550 name: type: string config: type: object - required: &ref_548 + required: &ref_551 - name /configs/list_autoscaling_events/{worker_group}: get: @@ -30513,7 +32679,7 @@ paths: type: array items: type: object - properties: &ref_551 + properties: &ref_554 id: type: integer format: int64 @@ -30580,7 +32746,7 @@ paths: type: string language: type: string - enum: *ref_94 + enum: *ref_100 required: - workspace_id - language @@ -30606,7 +32772,7 @@ paths: type: string language: type: string - enum: *ref_94 + enum: *ref_100 workspace_dep_names: type: array items: @@ -30946,7 +33112,7 @@ paths: properties: trigger_kind: type: string - enum: &ref_283 + enum: &ref_290 - webhook - http - websocket @@ -30992,11 +33158,11 @@ paths: required: true schema: type: string - enum: *ref_283 + enum: *ref_290 - name: runnable_kind in: path required: true - schema: *ref_132 + schema: *ref_139 - name: path in: path required: true @@ -31018,7 +33184,7 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_132 + schema: *ref_139 - name: path in: path required: true @@ -31032,17 +33198,17 @@ paths: type: array items: type: object - properties: &ref_552 + properties: &ref_555 trigger_config: {} trigger_kind: type: string - enum: *ref_283 + enum: *ref_290 error: type: string last_server_ping: type: string format: date-time - required: &ref_553 + required: &ref_556 - trigger_kind /w/{workspace}/capture/list/{runnable_kind}/{path}: get: @@ -31058,7 +33224,7 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_132 + schema: *ref_139 - name: path in: path required: true @@ -31067,7 +33233,7 @@ paths: in: query schema: type: string - enum: *ref_283 + enum: *ref_290 - name: page description: which page to return (start at 1, default 1) in: query @@ -31085,10 +33251,10 @@ paths: type: array items: type: object - properties: &ref_284 + properties: &ref_291 trigger_kind: type: string - enum: *ref_283 + enum: *ref_290 main_args: {} preprocessor_args: {} id: @@ -31096,7 +33262,7 @@ paths: created_at: type: string format: date-time - required: &ref_285 + required: &ref_292 - trigger_kind - main_args - preprocessor_args @@ -31116,7 +33282,7 @@ paths: - name: runnable_kind in: path required: true - schema: *ref_132 + schema: *ref_139 - name: path in: path required: true @@ -31161,8 +33327,8 @@ paths: application/json: schema: type: object - properties: *ref_284 - required: *ref_285 + properties: *ref_291 + required: *ref_292 delete: summary: delete a capture operationId: deleteCapture @@ -31254,13 +33420,13 @@ paths: schema: *ref_4 - name: runnable_id in: query - schema: &ref_286 + schema: &ref_293 type: string - name: runnable_type in: query - schema: &ref_287 + schema: &ref_294 type: string - enum: &ref_392 + enum: &ref_400 - ScriptHash - ScriptPath - FlowPath @@ -31277,7 +33443,7 @@ paths: filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: include_preview in: query schema: @@ -31291,7 +33457,7 @@ paths: type: array items: type: object - properties: &ref_288 + properties: &ref_295 id: type: string name: @@ -31305,7 +33471,7 @@ paths: type: boolean success: type: boolean - required: &ref_289 + required: &ref_296 - id - name - args @@ -31355,10 +33521,10 @@ paths: schema: *ref_4 - name: runnable_id in: query - schema: *ref_286 + schema: *ref_293 - name: runnable_type in: query - schema: *ref_287 + schema: *ref_294 - name: page description: which page to return (start at 1, default 1) in: query @@ -31376,8 +33542,8 @@ paths: type: array items: type: object - properties: *ref_288 - required: *ref_289 + properties: *ref_295 + required: *ref_296 /w/{workspace}/inputs/create: post: summary: Create an Input for future use in a script or flow @@ -31391,10 +33557,10 @@ paths: schema: *ref_4 - name: runnable_id in: query - schema: *ref_286 + schema: *ref_293 - name: runnable_type in: query - schema: *ref_287 + schema: *ref_294 requestBody: description: Input required: true @@ -31402,12 +33568,12 @@ paths: application/json: schema: type: object - properties: &ref_388 + properties: &ref_396 name: type: string args: type: object - required: &ref_389 + required: &ref_397 - name - args - created_by @@ -31437,14 +33603,14 @@ paths: application/json: schema: type: object - properties: &ref_390 + properties: &ref_398 id: type: string name: type: string is_public: type: boolean - required: &ref_391 + required: &ref_399 - id - name - is_public @@ -31470,7 +33636,7 @@ paths: - name: input in: path required: true - schema: &ref_312 + schema: &ref_319 type: string responses: '200': @@ -31503,7 +33669,7 @@ paths: properties: s3_resource: type: object - properties: &ref_290 + properties: &ref_297 bucket: type: string region: @@ -31518,7 +33684,7 @@ paths: type: string pathStyle: type: boolean - required: &ref_291 + required: &ref_298 - bucket - region - endPoint @@ -31596,8 +33762,8 @@ paths: properties: s3_resource: type: object - properties: *ref_290 - required: *ref_291 + properties: *ref_297 + required: *ref_298 responses: '200': description: Connection settings @@ -31618,10 +33784,10 @@ paths: type: boolean client_kwargs: type: object - properties: &ref_292 + properties: &ref_299 region_name: type: string - required: &ref_293 + required: &ref_300 - region_name required: - endpoint_url @@ -31676,8 +33842,8 @@ paths: type: boolean client_kwargs: type: object - properties: *ref_292 - required: *ref_293 + properties: *ref_299 + required: *ref_300 required: - endpoint_url - use_ssl @@ -31735,8 +33901,8 @@ paths: application/json: schema: type: object - properties: *ref_290 - required: *ref_291 + properties: *ref_297 + required: *ref_298 /w/{workspace}/job_helpers/test_connection: get: summary: Test connection to the workspace object storage @@ -31800,10 +33966,10 @@ paths: type: array items: type: object - properties: &ref_294 + properties: &ref_301 s3: type: string - required: &ref_295 + required: &ref_302 - s3 restricted_access: type: boolean @@ -31836,7 +34002,7 @@ paths: application/json: schema: type: object - properties: &ref_298 + properties: &ref_305 mime_type: type: string size_in_bytes: @@ -31900,7 +34066,7 @@ paths: application/json: schema: type: object - properties: &ref_296 + properties: &ref_303 msg: type: string content: @@ -31912,7 +34078,7 @@ paths: - Csv - Parquet - Unknown - required: &ref_297 + required: &ref_304 - content_type /w/{workspace}/job_helpers/list_git_repo_files: get: @@ -31960,8 +34126,8 @@ paths: type: array items: type: object - properties: *ref_294 - required: *ref_295 + properties: *ref_301 + required: *ref_302 restricted_access: type: boolean required: @@ -32020,8 +34186,8 @@ paths: application/json: schema: type: object - properties: *ref_296 - required: *ref_297 + properties: *ref_303 + required: *ref_304 /w/{workspace}/job_helpers/load_git_repo_file_metadata: get: summary: >- @@ -32052,7 +34218,7 @@ paths: application/json: schema: type: object - properties: *ref_298 + properties: *ref_305 /w/{workspace}/job_helpers/check_s3_folder_exists: get: summary: Check if S3 path exists and is a folder @@ -32503,7 +34669,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 requestBody: description: parameters for statistics retrieval required: true @@ -32532,46 +34698,46 @@ paths: type: array items: type: object - properties: &ref_533 + properties: &ref_536 id: type: string name: type: string - required: &ref_534 + required: &ref_537 - id scalar_metrics: type: array items: type: object - properties: &ref_535 + properties: &ref_538 metric_id: type: string value: type: number - required: &ref_536 + required: &ref_539 - id - value timeseries_metrics: type: array items: type: object - properties: &ref_537 + properties: &ref_540 metric_id: type: string values: type: array items: type: object - properties: &ref_539 + properties: &ref_542 timestamp: type: string format: date-time value: type: number - required: &ref_540 + required: &ref_543 - timestamp - value - required: &ref_538 + required: &ref_541 - id - values /w/{workspace}/job_metrics/set_progress/{id}: @@ -32588,7 +34754,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 requestBody: description: parameters for statistics retrieval required: true @@ -32622,7 +34788,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: job progress between 0 and 99 @@ -32640,11 +34806,11 @@ paths: - name: before description: filter on started before (inclusive) timestamp in: query - schema: *ref_299 + schema: *ref_306 - name: after description: filter on created after (exclusive) timestamp in: query - schema: *ref_300 + schema: *ref_307 - name: with_error in: query required: false @@ -32716,12 +34882,12 @@ paths: type: array items: type: object - properties: &ref_541 + properties: &ref_544 concurrency_key: type: string total_running: type: number - required: &ref_542 + required: &ref_545 - concurrency_key - total_running /concurrency_groups/prune/{concurrency_id}: @@ -32734,7 +34900,7 @@ paths: - name: concurrency_id in: path required: true - schema: &ref_314 + schema: &ref_321 type: string responses: '200': @@ -32754,7 +34920,7 @@ paths: - name: id in: path required: true - schema: *ref_172 + schema: *ref_178 responses: '200': description: concurrency key for given job @@ -32790,104 +34956,104 @@ paths: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 - name: label description: >- filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_174 + schema: *ref_180 - name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 - name: script_path_exact description: >- filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_157 + schema: *ref_163 - name: script_path_start description: >- filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_158 + schema: *ref_164 - name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_159 + schema: *ref_165 - name: script_hash description: mask to filter exact matching path in: query - schema: *ref_160 + schema: *ref_166 - name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_161 + schema: *ref_167 - name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_162 + schema: *ref_168 - name: running description: filter on running jobs in: query - schema: *ref_163 + schema: *ref_169 - name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_164 + schema: *ref_170 - name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_184 + schema: *ref_190 - name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_185 + schema: *ref_191 - name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: *ref_186 + schema: *ref_192 - name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: *ref_187 + schema: *ref_193 - name: job_kinds description: >- filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_165 + schema: *ref_171 - name: args description: >- filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 - name: tag description: >- filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_168 + schema: *ref_174 - name: result description: >- filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_169 + schema: *ref_175 - name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_171 + schema: *ref_177 - name: page description: which page to return (start at 1, default 1) in: query @@ -32903,7 +35069,7 @@ paths: (e.g. '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: *ref_188 + schema: *ref_194 - name: is_skipped description: is the job skipped in: query @@ -32924,6 +35090,19 @@ paths: in: query schema: type: boolean + - name: status + description: >- + filter on the exact completed job status. Unlike `success=true` + (which also matches `skipped`), `status=success` matches only + `success`. + in: query + schema: + type: string + enum: + - success + - failure + - canceled + - skipped - name: all_workspaces description: >- get jobs from all workspaces (only valid if request come from the @@ -32943,17 +35122,17 @@ paths: application/json: schema: type: object - properties: &ref_543 + properties: &ref_546 jobs: type: array items: - oneOf: *ref_193 - discriminator: *ref_194 + oneOf: *ref_199 + discriminator: *ref_200 obscured_jobs: type: array items: type: object - properties: &ref_393 + properties: &ref_401 typ: type: string started_at: @@ -32966,7 +35145,7 @@ paths: Obscured jobs omitted for security because of too specific filtering type: boolean - required: &ref_544 + required: &ref_547 - jobs - obscured_jobs /srch/w/{workspace}/index/search/job: @@ -33010,7 +35189,7 @@ paths: type: array items: type: object - properties: &ref_549 + properties: &ref_552 dancer: type: string hit_count: @@ -33089,7 +35268,7 @@ paths: type: array items: type: object - properties: &ref_550 + properties: &ref_553 dancer: type: string /srch/index/search/count_service_logs: @@ -33364,7 +35543,7 @@ paths: type: string kind: type: string - enum: *ref_301 + enum: *ref_308 usages: type: array items: @@ -33377,13 +35556,13 @@ paths: type: string kind: type: string - enum: &ref_303 + enum: &ref_310 - script - flow - job access_type: type: string - enum: &ref_302 + enum: &ref_309 - r - w - rw @@ -33393,7 +35572,7 @@ paths: description: The columns used (for tables) additionalProperties: type: string - enum: *ref_302 + enum: *ref_309 nullable: true created_at: type: string @@ -33466,7 +35645,7 @@ paths: type: string kind: type: string - enum: *ref_303 + enum: *ref_310 responses: '200': description: all assets used by the given usage paths, in the same order @@ -33486,10 +35665,10 @@ paths: type: string kind: type: string - enum: *ref_301 + enum: *ref_308 access_type: type: string - enum: *ref_302 + enum: *ref_309 nullable: true /w/{workspace}/assets/list_favorites: get: @@ -33517,6 +35696,181 @@ paths: path: type: string description: The asset path + /w/{workspace}/assets/graph: + get: + summary: Get the workspace-wide asset <-> runnable graph + operationId: getAssetsGraph + tags: + - asset + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + - name: asset_kinds + in: query + description: Filter by asset kinds (comma-separated list) + schema: + type: string + - name: folder + in: query + description: Scope the graph to runnables in a single folder + schema: + type: string + responses: + '200': + description: asset graph nodes, lineage edges and trigger edges + content: + application/json: + schema: + type: object + required: + - assets + - runnables + - edges + - triggers + properties: + assets: + type: array + items: + type: object + required: + - kind + - path + properties: + kind: + type: string + enum: *ref_308 + path: + type: string + runnables: + type: array + items: + type: object + required: + - path + - usage_kind + properties: + path: + type: string + usage_kind: + type: string + enum: *ref_310 + in_pipeline: + type: boolean + description: >- + True iff the script is a pipeline member (deployed + with `// pipeline`). Omitted when false. + edges: + type: array + items: + type: object + required: + - runnable_path + - runnable_kind + - asset_kind + - asset_path + properties: + runnable_path: + type: string + runnable_kind: + type: string + enum: *ref_310 + asset_kind: + type: string + enum: *ref_308 + asset_path: + type: string + access_type: + type: string + enum: *ref_309 + nullable: true + triggers: + type: array + items: + oneOf: + - type: object + description: Asset trigger edge (`// on `) + required: + - trigger_kind + - asset_kind + - asset_path + - runnable_kind + - runnable_path + properties: + trigger_kind: + type: string + enum: + - asset + asset_kind: + type: string + enum: *ref_308 + asset_path: + type: string + runnable_kind: + type: string + enum: *ref_310 + runnable_path: + type: string + - type: object + description: >- + Native trigger edge (schedule, email, kafka, ...). + `path` is the trigger row's path. + required: + - trigger_kind + - path + - runnable_kind + - runnable_path + properties: + trigger_kind: + type: string + enum: + - schedule + - email + - kafka + - mqtt + - nats + - postgres + - sqs + - gcp + path: + type: string + runnable_kind: + type: string + enum: *ref_310 + runnable_path: + type: string + /w/{workspace}/assets/pipelines: + get: + summary: List folders that contain at least one pipeline-member script + operationId: listPipelineFolders + tags: + - asset + parameters: + - name: workspace + in: path + required: true + schema: *ref_4 + responses: + '200': + description: folders containing pipeline scripts, with their script counts + content: + application/json: + schema: + type: array + items: + type: object + required: + - folder + - script_count + properties: + folder: + type: string + description: The folder name (without the `f/` prefix) + script_count: + type: integer + format: int64 + description: Number of pipeline-member scripts in the folder /w/{workspace}/volumes/list: get: summary: List all volumes in the workspace @@ -33537,13 +35891,13 @@ paths: type: array items: type: object - required: &ref_563 + required: &ref_566 - name - size_bytes - file_count - created_at - created_by - properties: &ref_564 + properties: &ref_567 name: type: string size_bytes: @@ -33658,13 +36012,13 @@ paths: type: array items: type: object - required: &ref_376 + required: &ref_383 - name - description - instructions - path - method - properties: &ref_377 + properties: &ref_384 name: type: string description: The tool name/operation ID @@ -33797,11 +36151,30 @@ components: in: cookie name: token parameters: + GetDraft: + name: get_draft + in: query + required: false + description: >- + When true, overlay the authed user's draft (if any) onto the deployed + payload. + schema: *ref_77 + IncludeDraftOnly: + name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: *ref_214 Id: name: id in: path required: true - schema: *ref_304 + schema: *ref_311 Key: name: key in: path @@ -33817,7 +36190,7 @@ components: in: path required: true description: The name of the publication - schema: *ref_264 + schema: *ref_271 VersionId: name: version in: path @@ -33828,7 +36201,7 @@ components: name: token in: path required: true - schema: *ref_305 + schema: *ref_312 AccountId: name: id in: path @@ -33843,17 +36216,17 @@ components: name: path in: path required: true - schema: *ref_92 + schema: *ref_98 ScriptHash: name: hash in: path required: true - schema: *ref_101 + schema: *ref_108 JobId: name: id in: path required: true - schema: *ref_172 + schema: *ref_178 Path: name: path in: path @@ -33863,22 +36236,22 @@ components: name: custom_path in: path required: true - schema: *ref_133 + schema: *ref_97 PathId: name: id in: path required: true - schema: *ref_79 + schema: *ref_84 PathVersion: name: version in: path required: true - schema: *ref_306 + schema: *ref_313 Name: name: name in: path required: true - schema: *ref_273 + schema: *ref_280 Page: name: page description: which page to return (start at 1, default 1) @@ -33897,12 +36270,12 @@ components: '!schedule,!webhook') in: query x-go-name: JobTriggerKindParam - schema: *ref_188 + schema: *ref_194 OrderDesc: name: order_desc description: order by desc order (default true) in: query - schema: *ref_118 + schema: *ref_123 CreatedBy: name: created_by description: >- @@ -33910,7 +36283,7 @@ components: (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob') in: query - schema: *ref_119 + schema: *ref_124 Label: name: label description: >- @@ -33918,7 +36291,7 @@ components: 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release') in: query - schema: *ref_174 + schema: *ref_180 Worker: name: worker description: >- @@ -33926,26 +36299,26 @@ components: 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2') in: query - schema: *ref_156 + schema: *ref_162 ParentJob: name: parent_job description: >- The parent job that is at the origin and responsible for the execution of this script if any in: query - schema: *ref_110 + schema: *ref_115 WorkerTag: name: tag description: Override the tag to use in: query - schema: *ref_111 + schema: *ref_116 CacheTtl: name: cache_ttl description: >- Override the cache time to live (in seconds). Can not be used to disable caching, only override with a new cache ttl in: query - schema: *ref_112 + schema: *ref_117 NewJobId: name: job_id description: >- @@ -33953,7 +36326,7 @@ components: randomly using the ULID scheme. If a job id already exists in the queue or as a completed job, the request to create one will fail (Bad Request) in: query - schema: *ref_113 + schema: *ref_118 IncludeHeader: name: include_header description: > @@ -33963,19 +36336,19 @@ components: Header's key lowercased and '-'' replaced to '_' such that 'Content-Type' becomes the 'content_type' arg key in: query - schema: *ref_114 + schema: *ref_119 QueueLimit: name: queue_limit description: > The maximum size of the queue for which the request would get rejected if that job would push it above that limit in: query - schema: *ref_115 + schema: *ref_120 SkipPreprocessor: name: skip_preprocessor description: skip the preprocessor in: query - schema: *ref_116 + schema: *ref_121 Payload: name: payload description: > @@ -33984,7 +36357,7 @@ components: `encodeURIComponent(btoa(JSON.stringify({a: 2})))` in: query - schema: *ref_117 + schema: *ref_122 ScriptStartPath: name: script_path_start description: >- @@ -33992,12 +36365,12 @@ components: 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2') in: query - schema: *ref_158 + schema: *ref_164 SchedulePath: name: schedule_path description: mask to filter by schedule path in: query - schema: *ref_159 + schema: *ref_165 TriggerPath: name: trigger_path description: >- @@ -34005,7 +36378,7 @@ components: 'f/trigger1,f/trigger2') and negation by prefixing all values with '!' (e.g. '!f/trigger1,!f/trigger2') in: query - schema: *ref_307 + schema: *ref_314 ScriptExactPath: name: script_path_exact description: >- @@ -34013,87 +36386,87 @@ components: (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2') in: query - schema: *ref_157 + schema: *ref_163 ScriptExactHash: name: script_hash description: mask to filter exact matching path in: query - schema: *ref_160 + schema: *ref_166 CreatedBefore: name: created_before description: filter on created before (inclusive) timestamp in: query - schema: *ref_182 + schema: *ref_188 CreatedAfter: name: created_after description: filter on created after (exclusive) timestamp in: query - schema: *ref_183 + schema: *ref_189 StartedBefore: name: started_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_161 + schema: *ref_167 StartedAfter: name: started_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_162 + schema: *ref_168 Before: name: before description: filter on started before (inclusive) timestamp in: query - schema: *ref_299 + schema: *ref_306 CompletedBefore: name: completed_before description: filter on started before (inclusive) timestamp in: query - schema: *ref_184 + schema: *ref_190 CompletedAfter: name: completed_after description: filter on started after (exclusive) timestamp in: query - schema: *ref_185 + schema: *ref_191 CreatedAfterQueue: name: created_after_queue description: filter on jobs created after X for jobs in the queue only in: query - schema: *ref_187 + schema: *ref_193 CreatedBeforeQueue: name: created_before_queue description: filter on jobs created before X for jobs in the queue only in: query - schema: *ref_186 + schema: *ref_192 Success: name: success description: filter on successful jobs in: query - schema: *ref_170 + schema: *ref_176 ScheduledForBeforeNow: name: scheduled_for_before_now description: filter on jobs scheduled_for before now (hence waitinf for a worker) in: query - schema: *ref_164 + schema: *ref_170 Suspended: name: suspended description: filter on suspended jobs in: query - schema: *ref_166 + schema: *ref_172 Running: name: running description: filter on running jobs in: query - schema: *ref_163 + schema: *ref_169 AllowWildcards: name: allow_wildcards description: allow wildcards (*) in the filter of label, tag, worker in: query - schema: *ref_171 + schema: *ref_177 ArgsFilter: name: args description: filter on jobs containing those args as a json subset (@> in postgres) in: query - schema: *ref_167 + schema: *ref_173 Tag: name: tag description: >- @@ -34101,37 +36474,37 @@ components: 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem') in: query - schema: *ref_168 + schema: *ref_174 ResultFilter: name: result description: filter on jobs containing those result as a json subset (@> in postgres) in: query - schema: *ref_169 + schema: *ref_175 After: name: after description: filter on created after (exclusive) timestamp in: query - schema: *ref_300 + schema: *ref_307 Username: name: username description: filter on exact username of user in: query - schema: *ref_308 + schema: *ref_315 Operation: name: operation description: filter on exact or prefix name of operation in: query - schema: *ref_309 + schema: *ref_316 ResourceName: name: resource description: filter on exact or prefix name of resource in: query - schema: *ref_310 + schema: *ref_317 ActionKind: name: action_kind description: filter on type of operation in: query - schema: *ref_311 + schema: *ref_318 JobKinds: name: job_kinds description: >- @@ -34139,53 +36512,76 @@ components: 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies') in: query - schema: *ref_165 + schema: *ref_171 RunnableId: name: runnable_id in: query - schema: *ref_286 + schema: *ref_293 RunnableTypeQuery: name: runnable_type in: query - schema: *ref_287 + schema: *ref_294 InputId: name: input in: path required: true - schema: *ref_312 + schema: *ref_319 GetStarted: name: get_started in: query - schema: *ref_313 + schema: *ref_320 ConcurrencyId: name: concurrency_id in: path required: true - schema: *ref_314 + schema: *ref_321 RunnableKind: name: runnable_kind in: path required: true - schema: *ref_132 + schema: *ref_139 schemas: + UserDraftOverlay: + type: object + description: | + Overlay fields added to every "get by path" response that accepts + the `get_draft` query parameter. The deployed payload is sent + untouched in the response body; the authed user's saved draft + for this path — whatever shape the editor wrote — is attached + as the sibling `draft` field when `get_draft=true` and a draft + exists. The frontend pairs the two to present diff / reset / + discard UI; the server never merges them. + + When `no_deployed=true` there is no deployed row at this path — + the response body is a best-effort stand-in synthesized from + the draft, and only `draft` is canonical. Callers should disable + "diff vs deployed" UI in that case. + properties: *ref_78 + required: *ref_79 + UserDraftItemKind: + type: string + description: | + Closed set of item kinds a user can autosave as a draft. Mirrors the + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: *ref_101 OpenFlow: type: object description: >- Top-level flow definition containing metadata, configuration, and the flow structure - properties: *ref_120 - required: *ref_121 + properties: *ref_125 + required: *ref_126 FlowValue: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_149 - required: *ref_150 + properties: *ref_155 + required: *ref_156 Retry: type: object description: Retry configuration for failed module executions - properties: *ref_197 + properties: *ref_203 StopAfterIf: type: object description: Early termination condition for a module @@ -34206,6 +36602,13 @@ components: with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised. + error_include_result: + type: boolean + description: >- + When stopping with an error (error_message set), embed the stopping + step's own result inside the raised error object (as error.result) + instead of discarding it. The top-level result stays { error }. + Defaults to false. required: - expr FlowModule: @@ -34222,18 +36625,18 @@ components: description: >- The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type - oneOf: *ref_88 - discriminator: *ref_89 + oneOf: *ref_93 + discriminator: *ref_94 stop_after_if: description: Early termination condition for a module type: object - properties: *ref_90 - required: *ref_91 + properties: *ref_95 + required: *ref_96 stop_after_all_iters_if: description: Early termination condition for a module type: object - properties: *ref_90 - required: *ref_91 + properties: *ref_95 + required: *ref_96 skip_if: type: object description: >- @@ -34252,8 +36655,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 cache_ttl: type: number description: Cache duration in seconds for this step's results @@ -34264,8 +36667,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 delete_after_secs: type: integer description: >- @@ -34308,8 +36711,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 self_approval_disabled: type: boolean description: If true, the user who started the flow cannot approve @@ -34328,7 +36731,7 @@ components: retry: description: Retry configuration for failed module executions type: object - properties: *ref_315 + properties: *ref_322 debouncing: description: Debounce configuration for this step (EE only) type: object @@ -34361,8 +36764,8 @@ components: description: >- Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_138 - discriminator: *ref_139 + oneOf: *ref_144 + discriminator: *ref_145 StaticTransform: type: object description: >- @@ -34419,7 +36822,7 @@ components: kind: type: string description: Supported AI provider types - enum: *ref_316 + enum: *ref_323 resource: type: string description: >- @@ -34439,16 +36842,16 @@ components: oneOf: - type: object description: No conversation memory/context - properties: *ref_317 - required: *ref_318 + properties: *ref_324 + required: *ref_325 - type: object description: Automatic context management - properties: *ref_319 - required: *ref_320 + properties: *ref_326 + required: *ref_327 - type: object description: Explicit message history - properties: *ref_321 - required: *ref_322 + properties: *ref_328 + required: *ref_329 discriminator: propertyName: kind mapping: @@ -34465,62 +36868,62 @@ components: Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms - properties: *ref_323 - required: *ref_324 + properties: *ref_330 + required: *ref_331 - type: object description: >- Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code - properties: *ref_325 - required: *ref_326 + properties: *ref_332 + required: *ref_333 - type: object description: >- Reference to an existing flow by path. Use this to call another flow as a subflow - properties: *ref_327 - required: *ref_328 + properties: *ref_334 + required: *ref_335 - type: object description: >- Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations - properties: *ref_329 - required: *ref_330 + properties: *ref_336 + required: *ref_337 - type: object description: >- Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination - properties: *ref_331 - required: *ref_332 + properties: *ref_338 + required: *ref_339 - type: object description: >- Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes - properties: *ref_333 - required: *ref_334 + properties: *ref_340 + required: *ref_341 - type: object description: >- Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently - properties: *ref_335 - required: *ref_336 + properties: *ref_342 + required: *ref_343 - type: object description: >- Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder - properties: *ref_337 - required: *ref_338 + properties: *ref_344 + required: *ref_345 - type: object description: >- AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task - properties: *ref_339 - required: *ref_340 + properties: *ref_346 + required: *ref_347 discriminator: propertyName: type mapping: @@ -34550,8 +36953,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 content: type: string description: The script source code. Should export a 'main' function @@ -34667,8 +37070,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 path: type: string description: Path to the script in the workspace (e.g., 'f/scripts/send_email') @@ -34705,8 +37108,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 path: type: string description: Path to the flow in the workspace (e.g., 'f/flows/process_user') @@ -34734,15 +37137,15 @@ components: items: type: object description: A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 iterator: description: >- Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 skip_failures: type: boolean description: >- @@ -34762,8 +37165,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 squash: type: boolean required: @@ -34786,8 +37189,8 @@ components: items: type: object description: A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 skip_failures: type: boolean description: >- @@ -34807,8 +37210,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 squash: type: boolean required: @@ -34846,8 +37249,8 @@ components: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 required: - modules - expr @@ -34857,8 +37260,8 @@ components: items: type: object description: A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 type: type: string enum: @@ -34896,8 +37299,8 @@ components: description: >- A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 required: - modules type: @@ -34926,16 +37329,16 @@ components: description: >- Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined - oneOf: *ref_341 - discriminator: *ref_342 + oneOf: *ref_348 + discriminator: *ref_349 output_type: allOf: - description: >- Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Output format type. @@ -34947,8 +37350,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: >- The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax. @@ -34958,8 +37361,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: >- System instructions that guide the AI's behavior, persona, and response style. Optional. @@ -34969,8 +37372,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Boolean. If true, stream the AI response incrementally. @@ -34980,16 +37383,16 @@ components: description: >- Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined - oneOf: *ref_343 - discriminator: *ref_344 + oneOf: *ref_350 + discriminator: *ref_351 output_schema: allOf: - description: >- Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape. @@ -35006,8 +37409,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Array of file references (images or PDFs) for the AI agent. @@ -35021,8 +37424,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Integer. Maximum number of tokens the AI will generate in its response. @@ -35035,8 +37438,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: | Float. Controls randomness/creativity of responses. Range: 0.0 to 2.0 (provider-dependent) @@ -35049,8 +37452,8 @@ components: Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 description: > Number. Limits how many times the agent can loop through reasoning and tool use. @@ -35070,12 +37473,17 @@ components: description: >- A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool - properties: *ref_345 - required: *ref_346 + properties: *ref_352 + required: *ref_353 type: type: string enum: - aiagent + tag: + type: string + description: >- + Worker group tag for execution routing. If not set, the AI agent + step runs on the flow's tag (default `flow`) omit_output_from_conversation: type: boolean default: false @@ -35106,8 +37514,8 @@ components: - type FlowStatus: type: object - properties: *ref_175 - required: *ref_176 + properties: *ref_181 + required: *ref_182 FlowStatusModule: type: object properties: @@ -35329,8 +37737,8 @@ components: - type CiTestResult: type: object - properties: *ref_108 - required: *ref_109 + properties: *ref_113 + required: *ref_114 HealthStatusResponse: type: object description: Health status response (cached with 5s TTL) @@ -35344,75 +37752,75 @@ components: HealthChecks: type: object description: Detailed health checks - required: *ref_347 - properties: *ref_348 + required: *ref_354 + properties: *ref_355 DatabaseHealth: type: object description: Database health status - required: *ref_349 - properties: *ref_350 + required: *ref_356 + properties: *ref_357 PoolStats: type: object description: Database connection pool statistics - required: *ref_351 - properties: *ref_352 + required: *ref_358 + properties: *ref_359 WorkersHealth: type: object description: Workers health status - required: *ref_353 - properties: *ref_354 + required: *ref_360 + properties: *ref_361 QueueHealth: type: object description: Job queue status - required: *ref_355 - properties: *ref_356 + required: *ref_362 + properties: *ref_363 ReadinessHealth: type: object description: Server readiness status - required: *ref_357 - properties: *ref_358 + required: *ref_364 + properties: *ref_365 AutoInviteConfig: type: object description: Configuration for auto-inviting users to the workspace - properties: *ref_359 + properties: *ref_366 ErrorHandlerConfig: type: object description: Configuration for the workspace error handler - properties: *ref_360 + properties: *ref_367 SuccessHandlerConfig: type: object description: Configuration for the workspace success handler - properties: *ref_361 + properties: *ref_368 EditErrorHandler: description: >- Request body for editing the workspace error handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: *ref_362 + oneOf: *ref_369 EditErrorHandlerNew: type: object description: New grouped format for editing error handler - properties: *ref_363 + properties: *ref_370 EditErrorHandlerLegacy: type: object description: >- Legacy flat format for editing error handler (deprecated, use new format) - properties: *ref_364 + properties: *ref_371 EditSuccessHandler: description: >- Request body for editing the workspace success handler. Accepts both new grouped format and legacy flat format for backward compatibility. - oneOf: *ref_365 + oneOf: *ref_372 EditSuccessHandlerNew: type: object description: New grouped format for editing success handler - properties: *ref_366 + properties: *ref_373 EditSuccessHandlerLegacy: type: object description: >- Legacy flat format for editing success handler (deprecated, use new format) - properties: *ref_367 + properties: *ref_374 VaultSettings: type: object required: *ref_27 @@ -35427,28 +37835,28 @@ components: properties: *ref_34 SecretMigrationFailure: type: object - required: *ref_368 - properties: *ref_369 + required: *ref_375 + properties: *ref_376 SecretMigrationReport: type: object required: *ref_29 properties: *ref_30 JwksResponse: type: object - required: *ref_370 - properties: *ref_371 + required: *ref_377 + properties: *ref_378 FlowConversation: type: object - required: *ref_372 - properties: *ref_373 + required: *ref_379 + properties: *ref_380 FlowConversationMessage: type: object - required: *ref_374 - properties: *ref_375 + required: *ref_381 + properties: *ref_382 EndpointTool: type: object - required: *ref_376 - properties: *ref_377 + required: *ref_383 + properties: *ref_384 AIProvider: type: string enum: *ref_51 @@ -35461,114 +37869,112 @@ components: required: *ref_44 AIProviderConfig: type: object - properties: *ref_378 - required: *ref_379 + properties: *ref_385 + required: *ref_386 AIConfig: type: object properties: *ref_50 InstanceAIProviderSummary: type: object - properties: *ref_380 - required: *ref_381 + properties: *ref_387 + required: *ref_388 InstanceAISummary: type: object properties: *ref_52 required: *ref_53 Alert: type: object - properties: *ref_382 - required: *ref_383 + properties: *ref_389 + required: *ref_390 Configs: type: object nullable: true - properties: *ref_384 + properties: *ref_391 WorkspaceDependencies: - type: object - properties: *ref_97 - required: *ref_98 - NewWorkspaceDependencies: - type: object - properties: *ref_385 - required: *ref_386 - Script: - type: object - properties: *ref_99 - required: *ref_100 - NewScript: type: object properties: *ref_104 required: *ref_105 - NewScriptWithDraft: - allOf: *ref_387 - ScriptHistory: + NewWorkspaceDependencies: + type: object + properties: *ref_392 + required: *ref_393 + Script: type: object properties: *ref_106 required: *ref_107 + NewScript: + type: object + properties: *ref_394 + required: *ref_395 + ScriptHistory: + type: object + properties: *ref_111 + required: *ref_112 ScriptArgs: type: object description: The arguments to pass to the script or flow additionalProperties: true Input: type: object - properties: *ref_288 - required: *ref_289 + properties: *ref_295 + required: *ref_296 CreateInput: type: object - properties: *ref_388 - required: *ref_389 + properties: *ref_396 + required: *ref_397 UpdateInput: type: object - properties: *ref_390 - required: *ref_391 + properties: *ref_398 + required: *ref_399 RunnableType: type: string - enum: *ref_392 + enum: *ref_400 QueuedJob: type: object - properties: *ref_191 - required: *ref_192 + properties: *ref_197 + required: *ref_198 CompletedJob: type: object - properties: *ref_189 - required: *ref_190 + properties: *ref_195 + required: *ref_196 ExportableCompletedJob: type: object description: Completed job with full data for export/import operations - properties: *ref_178 - required: *ref_179 + properties: *ref_184 + required: *ref_185 ExportableQueuedJob: type: object description: Queued job with full data for export/import operations - properties: *ref_180 - required: *ref_181 + properties: *ref_186 + required: *ref_187 ObscuredJob: type: object - properties: *ref_393 + properties: *ref_401 Job: - oneOf: *ref_193 - discriminator: *ref_194 + oneOf: *ref_199 + discriminator: *ref_200 User: type: object properties: *ref_35 required: *ref_36 UserSource: type: object - properties: *ref_394 - required: *ref_395 + properties: *ref_402 + required: *ref_403 UserUsage: type: object - properties: *ref_396 + properties: *ref_404 Login: type: object - properties: *ref_397 - required: *ref_398 + properties: *ref_405 + required: *ref_406 PasswordResetResponse: type: object properties: *ref_7 required: *ref_8 EditWorkspaceUser: type: object - properties: *ref_399 + properties: *ref_407 OffboardAffectedPaths: type: object properties: *ref_11 @@ -35578,64 +37984,64 @@ components: required: *ref_13 OffboardTokenInfo: type: object - properties: *ref_400 - required: *ref_401 + properties: *ref_408 + required: *ref_409 OffboardRequest: type: object - properties: *ref_402 - required: *ref_403 + properties: *ref_410 + required: *ref_411 OffboardResponse: type: object properties: *ref_14 OffboardSummary: type: object - properties: *ref_404 - required: *ref_405 + properties: *ref_412 + required: *ref_413 GlobalOffboardPreview: type: object - properties: *ref_406 - required: *ref_407 + properties: *ref_414 + required: *ref_415 WorkspaceOffboardPreview: - type: object - properties: *ref_408 - required: *ref_409 - GlobalOffboardRequest: - type: object - properties: *ref_410 - WorkspaceReassignment: - type: object - properties: *ref_411 - required: *ref_412 - TruncatedToken: - type: object - properties: *ref_102 - required: *ref_103 - ExternalJwtToken: - type: object - properties: *ref_413 - required: *ref_414 - NewToken: - type: object - properties: *ref_415 - NewTokenImpersonate: type: object properties: *ref_416 required: *ref_417 + GlobalOffboardRequest: + type: object + properties: *ref_418 + WorkspaceReassignment: + type: object + properties: *ref_419 + required: *ref_420 + TruncatedToken: + type: object + properties: *ref_109 + required: *ref_110 + ExternalJwtToken: + type: object + properties: *ref_421 + required: *ref_422 + NewToken: + type: object + properties: *ref_423 + NewTokenImpersonate: + type: object + properties: *ref_424 + required: *ref_425 ListableVariable: type: object properties: *ref_61 required: *ref_62 ContextualVariable: type: object - properties: *ref_418 - required: *ref_419 + properties: *ref_426 + required: *ref_427 CreateVariable: type: object - properties: *ref_420 - required: *ref_421 + properties: *ref_428 + required: *ref_429 EditVariable: type: object - properties: *ref_422 + properties: *ref_430 AuditLog: type: object properties: *ref_5 @@ -35772,63 +38178,99 @@ components: - has_preprocessor ScriptLang: type: string - enum: *ref_94 + enum: *ref_100 ScriptModule: type: object description: An additional module file associated with a script - properties: *ref_95 - required: *ref_96 + properties: *ref_102 + required: *ref_103 Preview: type: object - properties: *ref_141 - required: *ref_142 + properties: *ref_147 + required: *ref_148 PreviewInline: type: object - properties: *ref_423 - required: *ref_424 + properties: *ref_431 + required: *ref_432 InlineScriptArgs: type: object - properties: *ref_140 + properties: *ref_146 WorkflowTask: type: object - properties: *ref_425 - required: *ref_426 + properties: *ref_433 + required: *ref_434 WorkflowStatusRecord: type: object additionalProperties: type: object - properties: *ref_177 + properties: *ref_183 WorkflowStatus: type: object - properties: *ref_177 + properties: *ref_183 CreateResource: type: object - properties: *ref_427 - required: *ref_428 + properties: *ref_435 + required: *ref_436 EditResource: type: object - properties: *ref_429 + properties: *ref_437 Resource: type: object - properties: *ref_430 - required: *ref_431 + properties: + workspace_id: + type: string + path: + type: string + description: + type: string + resource_type: + type: string + value: {} + is_oauth: + type: boolean + extra_perms: + type: object + additionalProperties: + type: boolean + created_by: + type: string + edited_at: + type: string + format: date-time + labels: + type: array + items: + type: string + inherited_labels: + type: array + items: + type: string + description: > + Labels inherited from the parent folder, computed at read time. + Read-only — edit them on the folder. + ws_specific: + type: boolean + required: + - path + - resource_type + - is_oauth ListableResource: type: object - properties: *ref_432 - required: *ref_433 + properties: *ref_80 + required: *ref_81 ResourceType: type: object - properties: *ref_77 - required: *ref_78 + properties: *ref_82 + required: *ref_83 EditResourceType: type: object - properties: *ref_434 + properties: *ref_438 Schedule: type: object - properties: *ref_198 - required: *ref_199 + properties: *ref_204 + required: *ref_205 ScheduleWJobs: - allOf: *ref_435 + allOf: *ref_439 ErrorHandler: type: string enum: @@ -35838,121 +38280,121 @@ components: - email NewSchedule: type: object - properties: *ref_436 - required: *ref_437 + properties: *ref_440 + required: *ref_441 EditSchedule: type: object - properties: *ref_438 - required: *ref_439 + properties: *ref_442 + required: *ref_443 JobTriggerKind: description: job trigger kind (schedule, http, websocket...) type: string - enum: *ref_173 + enum: *ref_179 TriggerMode: description: job trigger mode type: string - enum: *ref_207 + enum: *ref_213 TriggerExtraProperty: type: object - properties: *ref_214 - required: *ref_215 + properties: *ref_221 + required: *ref_222 AuthenticationMethod: type: string - enum: *ref_206 + enum: *ref_212 RunnableKind: type: string - enum: *ref_200 + enum: *ref_206 OpenapiSpecFormat: type: string - enum: *ref_440 + enum: *ref_444 OpenapiHttpRouteFilters: - type: object - properties: *ref_441 - required: *ref_442 - WebhookFilters: - type: object - properties: *ref_443 - required: *ref_444 - OpenapiV3Info: type: object properties: *ref_445 required: *ref_446 - GenerateOpenapiSpec: - type: object - properties: *ref_201 - HttpMethod: - type: string - enum: *ref_204 - HttpRequestType: - type: string - enum: *ref_205 - HttpTrigger: - allOf: *ref_208 - type: object - properties: *ref_209 - required: *ref_210 - NewHttpTrigger: - type: object - properties: *ref_202 - required: *ref_203 - EditHttpTrigger: + WebhookFilters: type: object properties: *ref_447 required: *ref_448 - TriggersCount: - type: object - properties: *ref_125 - WebsocketHeartbeat: - type: object - properties: *ref_212 - required: *ref_213 - WebsocketTrigger: - allOf: *ref_216 - type: object - properties: *ref_217 - required: *ref_218 - NewWebsocketTrigger: + OpenapiV3Info: type: object properties: *ref_449 required: *ref_450 - EditWebsocketTrigger: + GenerateOpenapiSpec: + type: object + properties: *ref_207 + HttpMethod: + type: string + enum: *ref_210 + HttpRequestType: + type: string + enum: *ref_211 + HttpTrigger: + allOf: *ref_215 + type: object + properties: *ref_216 + required: *ref_217 + NewHttpTrigger: + type: object + properties: *ref_208 + required: *ref_209 + EditHttpTrigger: type: object properties: *ref_451 required: *ref_452 + TriggersCount: + type: object + properties: *ref_130 + WebsocketHeartbeat: + type: object + properties: *ref_219 + required: *ref_220 + WebsocketTrigger: + allOf: *ref_223 + type: object + properties: *ref_224 + required: *ref_225 + NewWebsocketTrigger: + type: object + properties: *ref_453 + required: *ref_454 + EditWebsocketTrigger: + type: object + properties: *ref_455 + required: *ref_456 WebsocketTriggerInitialMessage: - anyOf: *ref_211 + anyOf: *ref_218 MqttQoS: type: string - enum: *ref_453 + enum: *ref_457 MqttV3Config: type: object - properties: *ref_238 + properties: *ref_245 MqttV5Config: type: object - properties: *ref_239 + properties: *ref_246 MqttSubscribeTopic: type: object - properties: *ref_236 - required: *ref_237 + properties: *ref_243 + required: *ref_244 MqttClientVersion: type: string - enum: *ref_240 + enum: *ref_247 MqttTrigger: - allOf: *ref_241 + allOf: *ref_248 type: object - properties: *ref_242 - required: *ref_243 + properties: *ref_249 + required: *ref_250 NewMqttTrigger: type: object - properties: *ref_454 - required: *ref_455 + properties: *ref_458 + required: *ref_459 EditMqttTrigger: type: object - properties: *ref_456 - required: *ref_457 + properties: *ref_460 + required: *ref_461 DeliveryType: type: string - enum: *ref_246 + enum: *ref_253 description: >- Delivery mode for messages. 'push' for HTTP push delivery where messages are sent to a webhook endpoint, 'pull' for polling where the trigger @@ -35960,19 +38402,19 @@ components: PushConfig: type: object description: Configuration for push delivery mode. - properties: *ref_247 - required: *ref_248 + properties: *ref_254 + required: *ref_255 GcpTrigger: - allOf: *ref_250 + allOf: *ref_257 type: object description: >- A Google Cloud Pub/Sub trigger that executes a script or flow when messages are received. - properties: *ref_251 - required: *ref_252 + properties: *ref_258 + required: *ref_259 SubscriptionMode: type: string - enum: *ref_249 + enum: *ref_256 description: >- The mode of subscription. 'existing' means using an existing GCP subscription, while 'create_update' involves creating or updating a new @@ -35980,68 +38422,68 @@ components: GcpTriggerData: type: object description: Data for creating or updating a Google Cloud Pub/Sub trigger. - properties: *ref_244 - required: *ref_245 + properties: *ref_251 + required: *ref_252 GetAllTopicSubscription: type: object - properties: *ref_458 - required: *ref_459 + properties: *ref_462 + required: *ref_463 DeleteGcpSubscription: type: object - properties: *ref_460 - required: *ref_461 + properties: *ref_464 + required: *ref_465 AzureMode: type: string - enum: *ref_255 + enum: *ref_262 description: Azure Event Grid trigger mode. AzureArmResource: type: object description: An ARM resource the service principal can see. - properties: *ref_259 - required: *ref_260 + properties: *ref_266 + required: *ref_267 AzureDeleteSubscription: type: object - properties: *ref_462 - required: *ref_463 + properties: *ref_466 + required: *ref_467 AzureTrigger: - allOf: *ref_256 + allOf: *ref_263 type: object description: >- An Azure Event Grid trigger that executes a script or flow when events arrive. - properties: *ref_257 - required: *ref_258 + properties: *ref_264 + required: *ref_265 AzureTriggerData: type: object description: Data for creating or updating an Azure Event Grid trigger. - properties: *ref_253 - required: *ref_254 + properties: *ref_260 + required: *ref_261 TestAzureConnection: - type: object - properties: *ref_464 - required: *ref_465 - AzureListTopics: - type: object - properties: *ref_466 - required: *ref_467 - AzureListSubscriptions: type: object properties: *ref_468 required: *ref_469 + AzureListTopics: + type: object + properties: *ref_470 + required: *ref_471 + AzureListSubscriptions: + type: object + properties: *ref_472 + required: *ref_473 AwsAuthResourceType: type: string - enum: *ref_225 + enum: *ref_232 SqsTrigger: - allOf: *ref_226 + allOf: *ref_233 type: object - properties: *ref_227 - required: *ref_228 + properties: *ref_234 + required: *ref_235 LoggedWizardStatus: type: string enum: *ref_21 CustomInstanceDbLogs: type: object - properties: *ref_470 + properties: *ref_474 CustomInstanceDbTag: type: string enum: *ref_22 @@ -36051,108 +38493,108 @@ components: properties: *ref_24 NewSqsTrigger: type: object - properties: *ref_471 - required: *ref_472 + properties: *ref_475 + required: *ref_476 EditSqsTrigger: type: object - properties: *ref_473 - required: *ref_474 + properties: *ref_477 + required: *ref_478 Slot: type: object - properties: *ref_261 + properties: *ref_268 SlotList: type: object - properties: *ref_475 + properties: *ref_479 PublicationData: type: object - properties: *ref_265 - required: *ref_266 + properties: *ref_272 + required: *ref_273 TableToTrack: type: array - items: *ref_476 + items: *ref_480 Relations: type: object - properties: *ref_262 - required: *ref_263 + properties: *ref_269 + required: *ref_270 Language: type: string - enum: *ref_477 + enum: *ref_481 TemplateScript: - type: object - properties: *ref_478 - required: *ref_479 - PostgresTrigger: - allOf: *ref_267 - type: object - properties: *ref_268 - required: *ref_269 - NewPostgresTrigger: - type: object - properties: *ref_480 - required: *ref_481 - EditPostgresTrigger: type: object properties: *ref_482 required: *ref_483 - KafkaTrigger: - allOf: *ref_219 + PostgresTrigger: + allOf: *ref_274 type: object - properties: *ref_220 - required: *ref_221 - NewKafkaTrigger: + properties: *ref_275 + required: *ref_276 + NewPostgresTrigger: type: object properties: *ref_484 required: *ref_485 - EditKafkaTrigger: + EditPostgresTrigger: type: object properties: *ref_486 required: *ref_487 - NatsTrigger: - allOf: *ref_222 + KafkaTrigger: + allOf: *ref_226 type: object - properties: *ref_223 - required: *ref_224 - NewNatsTrigger: + properties: *ref_227 + required: *ref_228 + NewKafkaTrigger: type: object properties: *ref_488 required: *ref_489 - EditNatsTrigger: + EditKafkaTrigger: type: object properties: *ref_490 required: *ref_491 - EmailTrigger: - allOf: *ref_270 + NatsTrigger: + allOf: *ref_229 type: object - properties: *ref_271 - required: *ref_272 - NewEmailTrigger: + properties: *ref_230 + required: *ref_231 + NewNatsTrigger: type: object properties: *ref_492 required: *ref_493 - EditEmailTrigger: + EditNatsTrigger: type: object properties: *ref_494 required: *ref_495 - Group: + EmailTrigger: + allOf: *ref_277 type: object properties: *ref_278 required: *ref_279 - InstanceGroup: + NewEmailTrigger: type: object - required: *ref_496 - properties: *ref_497 - InstanceGroupWithWorkspaces: - type: object - required: *ref_274 - properties: *ref_275 - WorkspaceInfo: + properties: *ref_496 + required: *ref_497 + EditEmailTrigger: type: object properties: *ref_498 required: *ref_499 + Group: + type: object + properties: *ref_285 + required: *ref_286 + InstanceGroup: + type: object + required: *ref_500 + properties: *ref_501 + InstanceGroupWithWorkspaces: + type: object + required: *ref_281 + properties: *ref_282 + WorkspaceInfo: + type: object + properties: *ref_502 + required: *ref_503 Folder: type: object - properties: *ref_281 - required: *ref_282 + properties: *ref_288 + required: *ref_289 FolderDefaultPermissionedAs: description: > Ordered list of rules applied at create-time when admins or @@ -36160,19 +38602,19 @@ components: `path_glob` matches the item path (relative to the folder root) wins, and its `permissioned_as` is used as the default. type: array - items: *ref_280 + items: *ref_287 WorkerPing: - type: object - properties: *ref_500 - required: *ref_501 - UserWorkspaceList: - type: object - properties: *ref_502 - required: *ref_503 - CreateWorkspace: type: object properties: *ref_504 required: *ref_505 + UserWorkspaceList: + type: object + properties: *ref_506 + required: *ref_507 + CreateWorkspace: + type: object + properties: *ref_508 + required: *ref_509 CreateWorkspaceFork: type: object properties: *ref_19 @@ -36183,15 +38625,15 @@ components: required: *ref_16 DependencyMap: type: object - properties: *ref_506 + properties: *ref_510 DependencyDependent: type: object - properties: *ref_507 - required: *ref_508 + properties: *ref_511 + required: *ref_512 DependentsAmount: type: object - properties: *ref_509 - required: *ref_510 + properties: *ref_513 + required: *ref_514 WorkspaceInvite: type: object properties: *ref_41 @@ -36201,56 +38643,58 @@ components: properties: *ref_39 required: *ref_40 Flow: - allOf: *ref_124 + allOf: *ref_129 ExtraPerms: type: object - additionalProperties: *ref_511 + additionalProperties: *ref_515 FlowMetadata: - type: object - properties: *ref_512 - required: *ref_513 - OpenFlowWPath: - allOf: *ref_126 - FlowPreview: - type: object - properties: *ref_152 - required: *ref_153 - RestartedFrom: - type: object - properties: *ref_151 - Policy: - type: object - properties: *ref_127 - ListableApp: - type: object - properties: *ref_514 - required: *ref_515 - ScopeDefinition: type: object properties: *ref_516 required: *ref_517 - ScopeDomain: + OpenFlowWPath: + allOf: *ref_131 + FlowPreview: + type: object + properties: *ref_158 + required: *ref_159 + RestartedFrom: + type: object + properties: *ref_157 + Policy: + type: object + properties: *ref_132 + ListableApp: type: object properties: *ref_518 required: *ref_519 - ListableRawApp: + ScopeDefinition: type: object properties: *ref_520 required: *ref_521 + ScopeDomain: + type: object + properties: *ref_522 + required: *ref_523 + ListableRawApp: + type: object + properties: *ref_524 + required: *ref_525 AppWithLastVersion: type: object - properties: *ref_128 - required: *ref_129 - AppWithLastVersionWDraft: - allOf: *ref_522 + properties: *ref_133 + required: *ref_134 AppHistory: type: object - properties: *ref_130 - required: *ref_131 + properties: *ref_137 + required: *ref_138 + EmbedTokenResponse: + type: object + properties: *ref_135 + required: *ref_136 FlowVersion: type: object - properties: *ref_122 - required: *ref_123 + properties: *ref_127 + required: *ref_128 SlackToken: type: object properties: @@ -36274,11 +38718,11 @@ components: required: *ref_75 HubScriptKind: type: string - enum: *ref_93 + enum: *ref_99 PolarsClientKwargs: type: object - properties: *ref_292 - required: *ref_293 + properties: *ref_299 + required: *ref_300 LargeFileStorage: type: object properties: *ref_45 @@ -36292,35 +38736,35 @@ components: properties: *ref_47 DataTableSchema: type: object - required: *ref_523 - properties: *ref_524 + required: *ref_526 + properties: *ref_527 DataTableTables: type: object - required: *ref_525 - properties: *ref_526 + required: *ref_528 + properties: *ref_529 DataTableTableSchema: type: object - required: *ref_527 - properties: *ref_528 + required: *ref_530 + properties: *ref_531 DynamicInputData: type: object - properties: *ref_529 - required: *ref_530 + properties: *ref_532 + required: *ref_533 WindmillLargeFile: type: object - properties: *ref_294 - required: *ref_295 + properties: *ref_301 + required: *ref_302 WindmillFileMetadata: type: object - properties: *ref_298 + properties: *ref_305 WindmillFilePreview: type: object - properties: *ref_296 - required: *ref_297 + properties: *ref_303 + required: *ref_304 S3Resource: type: object - properties: *ref_290 - required: *ref_291 + properties: *ref_297 + required: *ref_298 WorkspaceGitSyncSettings: type: object properties: *ref_56 @@ -36332,48 +38776,48 @@ components: properties: *ref_59 S3PermissionRule: type: object - properties: *ref_531 - required: *ref_532 + properties: *ref_534 + required: *ref_535 GitRepositorySettings: type: object properties: *ref_57 required: *ref_58 MetricMetadata: type: object - properties: *ref_533 - required: *ref_534 + properties: *ref_536 + required: *ref_537 ScalarMetric: type: object - properties: *ref_535 - required: *ref_536 + properties: *ref_538 + required: *ref_539 TimeseriesMetric: type: object - properties: *ref_537 - required: *ref_538 + properties: *ref_540 + required: *ref_541 MetricDataPoint: type: object - properties: *ref_539 - required: *ref_540 + properties: *ref_542 + required: *ref_543 RawScriptForDependencies: type: object - properties: *ref_143 - required: *ref_144 + properties: *ref_149 + required: *ref_150 ConcurrencyGroup: type: object - properties: *ref_541 - required: *ref_542 + properties: *ref_544 + required: *ref_545 ExtendedJobs: type: object - properties: *ref_543 - required: *ref_544 + properties: *ref_546 + required: *ref_547 ExportedUser: type: object properties: *ref_9 required: *ref_10 GlobalSetting: type: object - properties: *ref_545 - required: *ref_546 + properties: *ref_548 + required: *ref_549 InstanceConfig: type: object description: >- @@ -36382,35 +38826,35 @@ components: properties: *ref_26 Config: type: object - properties: *ref_547 - required: *ref_548 + properties: *ref_550 + required: *ref_551 ExportedInstanceGroup: type: object - properties: *ref_276 - required: *ref_277 + properties: *ref_283 + required: *ref_284 JobSearchHit: type: object - properties: *ref_549 + properties: *ref_552 LogSearchHit: type: object - properties: *ref_550 + properties: *ref_553 AutoscalingEvent: type: object - properties: *ref_551 + properties: *ref_554 CriticalAlert: type: object properties: *ref_63 CaptureTriggerKind: type: string - enum: *ref_283 + enum: *ref_290 Capture: type: object - properties: *ref_284 - required: *ref_285 + properties: *ref_291 + required: *ref_292 CaptureConfig: type: object - properties: *ref_552 - required: *ref_553 + properties: *ref_555 + required: *ref_556 OperatorSettings: nullable: true type: object @@ -36418,16 +38862,16 @@ components: properties: *ref_38 WorkspaceComparison: type: object - required: *ref_554 - properties: *ref_555 + required: *ref_557 + properties: *ref_558 WorkspaceItemDiff: type: object - required: *ref_556 - properties: *ref_557 + required: *ref_559 + properties: *ref_560 CompareSummary: type: object - required: *ref_558 - properties: *ref_559 + required: *ref_561 + properties: *ref_562 TeamInfo: type: object required: @@ -36448,12 +38892,12 @@ components: description: List of channels within the team items: type: object - required: &ref_560 + required: &ref_563 - channel_id - channel_name - tenant_id - service_url - properties: &ref_561 + properties: &ref_564 channel_id: type: string description: The unique identifier of the channel @@ -36473,11 +38917,11 @@ components: https://smba.trafficmanager.net/amer/12345678-1234-1234-1234-123456789012/ ChannelInfo: type: object - required: *ref_560 - properties: *ref_561 + required: *ref_563 + properties: *ref_564 GithubInstallations: type: array - items: *ref_562 + items: *ref_565 WorkspaceGithubInstallation: type: object properties: @@ -36490,8 +38934,8 @@ components: - installation_id S3Object: type: object - properties: *ref_134 - required: *ref_135 + properties: *ref_140 + required: *ref_141 TeamsChannel: type: object required: @@ -36518,14 +38962,14 @@ components: minLength: 1 AssetUsageKind: type: string - enum: *ref_303 + enum: *ref_310 AssetUsageAccessType: type: string - enum: *ref_302 + enum: *ref_309 nullable: true AssetKind: type: string - enum: *ref_301 + enum: *ref_308 Asset: type: object properties: @@ -36533,26 +38977,26 @@ components: type: string kind: type: string - enum: *ref_301 + enum: *ref_308 required: - path - kind Volume: type: object - required: *ref_563 - properties: *ref_564 + required: *ref_566 + properties: *ref_567 ProtectionRuleset: type: object description: A workspace protection rule defining restrictions and bypass permissions - required: *ref_565 - properties: *ref_566 + required: *ref_568 + properties: *ref_569 ProtectionRules: type: array description: Configuration of protection restrictions items: *ref_64 ProtectionRuleKind: type: string - enum: *ref_567 + enum: *ref_570 RuleBypasserGroups: type: array description: Groups that can bypass this ruleset @@ -36563,12 +39007,12 @@ components: items: *ref_66 DeploymentRequestEligibleDeployer: type: object - required: *ref_568 - properties: *ref_569 + required: *ref_571 + properties: *ref_572 DeploymentRequestAssignee: type: object - required: *ref_570 - properties: *ref_571 + required: *ref_573 + properties: *ref_574 DeploymentRequestComment: type: object required: *ref_69 @@ -36583,27 +39027,27 @@ components: required: *ref_72 NativeServiceName: type: string - enum: *ref_229 + enum: *ref_236 NativeTrigger: type: object description: A native trigger stored in Windmill - properties: *ref_572 - required: *ref_573 + properties: *ref_575 + required: *ref_576 NativeTriggerWithExternal: type: object description: >- Full trigger response containing both Windmill data and external service data - properties: *ref_574 - required: *ref_575 + properties: *ref_577 + required: *ref_578 WorkspaceIntegrations: type: object - properties: *ref_576 - required: *ref_577 + properties: *ref_579 + required: *ref_580 WorkspaceOAuthConfig: type: object - properties: *ref_230 - required: *ref_231 + properties: *ref_237 + required: *ref_238 WebhookEvent: type: object properties: @@ -36614,7 +39058,7 @@ components: request_type: type: string description: The type of webhook request (define possible values here) - enum: &ref_578 + enum: &ref_581 - async - sync required: @@ -36623,21 +39067,21 @@ components: WebhookRequestType: type: string description: The type of webhook request (define possible values here) - enum: *ref_578 + enum: *ref_581 RedirectUri: type: object - properties: *ref_232 - required: *ref_233 + properties: *ref_239 + required: *ref_240 NativeTriggerData: type: object description: Data for creating or updating a native trigger - properties: *ref_234 - required: *ref_235 + properties: *ref_241 + required: *ref_242 CreateTriggerResponse: type: object description: Response returned when a native trigger is created - properties: *ref_579 - required: *ref_580 + properties: *ref_582 + required: *ref_583 SyncResult: type: object properties: @@ -36661,35 +39105,35 @@ components: - total_windmill NextCloudEventType: type: object - properties: *ref_581 - required: *ref_582 + properties: *ref_584 + required: *ref_585 GoogleCalendarEntry: type: object - properties: *ref_583 - required: *ref_584 + properties: *ref_586 + required: *ref_587 GoogleDriveFile: type: object - properties: *ref_585 - required: *ref_586 + properties: *ref_588 + required: *ref_589 GoogleDriveFilesResponse: type: object - properties: *ref_587 - required: *ref_588 + properties: *ref_590 + required: *ref_591 SharedDriveEntry: type: object - properties: *ref_589 - required: *ref_590 + properties: *ref_592 + required: *ref_593 GithubRepoEntry: type: object - properties: *ref_591 - required: *ref_592 + properties: *ref_594 + required: *ref_595 schemas-StaticTransform: type: object description: >- Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource' - properties: *ref_136 - required: *ref_137 + properties: *ref_142 + required: *ref_143 schemas-JavascriptTransform: type: object description: >- @@ -36697,48 +39141,48 @@ components: results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value - properties: *ref_84 - required: *ref_85 + properties: *ref_89 + required: *ref_90 schemas-AiTransform: type: object description: >- Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter. - properties: *ref_86 - required: *ref_87 + properties: *ref_91 + required: *ref_92 schemas-InputTransform: description: >- Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs - oneOf: *ref_80 - discriminator: *ref_81 + oneOf: *ref_85 + discriminator: *ref_86 schemas-RawScript: type: object description: >- Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms - properties: *ref_323 - required: *ref_324 + properties: *ref_330 + required: *ref_331 schemas-PathScript: type: object description: >- Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code - properties: *ref_325 - required: *ref_326 + properties: *ref_332 + required: *ref_333 schemas-PathFlow: type: object description: >- Reference to an existing flow by path. Use this to call another flow as a subflow - properties: *ref_327 - required: *ref_328 + properties: *ref_334 + required: *ref_335 schemas-FlowModule: type: object description: A single step in a flow. Can be a script, subflow, loop, or branch - properties: *ref_82 - required: *ref_83 + properties: *ref_87 + required: *ref_88 schemas-ForloopFlow: type: object description: >- @@ -36746,160 +39190,160 @@ components: 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations - properties: *ref_329 - required: *ref_330 + properties: *ref_336 + required: *ref_337 schemas-WhileloopFlow: type: object description: >- Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination - properties: *ref_331 - required: *ref_332 + properties: *ref_338 + required: *ref_339 schemas-BranchOne: type: object description: >- Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes - properties: *ref_333 - required: *ref_334 + properties: *ref_340 + required: *ref_341 schemas-BranchAll: type: object description: >- Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently - properties: *ref_335 - required: *ref_336 + properties: *ref_342 + required: *ref_343 schemas-Identity: type: object description: >- Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder - properties: *ref_337 - required: *ref_338 + properties: *ref_344 + required: *ref_345 AIProviderKind: type: string description: Supported AI provider types - enum: *ref_316 + enum: *ref_323 schemas-ProviderConfig: type: object description: >- Complete AI provider configuration with resource reference and model selection - properties: *ref_593 - required: *ref_594 + properties: *ref_596 + required: *ref_597 StaticProviderTransform: type: object description: Static provider configuration passed directly to the AI agent - properties: *ref_595 - required: *ref_596 + properties: *ref_598 + required: *ref_599 ProviderTransform: description: >- Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined - oneOf: *ref_341 - discriminator: *ref_342 + oneOf: *ref_348 + discriminator: *ref_349 MemoryOff: type: object description: No conversation memory/context - properties: *ref_317 - required: *ref_318 + properties: *ref_324 + required: *ref_325 MemoryAuto: type: object description: Automatic context management - properties: *ref_319 - required: *ref_320 + properties: *ref_326 + required: *ref_327 MemoryMessage: type: object description: A single message in conversation history - properties: *ref_597 - required: *ref_598 + properties: *ref_600 + required: *ref_601 MemoryManual: type: object description: Explicit message history - properties: *ref_321 - required: *ref_322 + properties: *ref_328 + required: *ref_329 schemas-MemoryConfig: description: Conversation memory configuration - oneOf: *ref_599 - discriminator: *ref_600 + oneOf: *ref_602 + discriminator: *ref_603 StaticMemoryTransform: type: object description: Static memory configuration passed directly to the AI agent - properties: *ref_601 - required: *ref_602 + properties: *ref_604 + required: *ref_605 MemoryTransform: description: >- Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined - oneOf: *ref_343 - discriminator: *ref_344 + oneOf: *ref_350 + discriminator: *ref_351 schemas-FlowModuleValue: description: >- The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type - oneOf: *ref_88 - discriminator: *ref_89 + oneOf: *ref_93 + discriminator: *ref_94 FlowModuleTool: description: >- A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module - allOf: *ref_603 + allOf: *ref_606 McpToolValue: type: object description: >- Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers - properties: *ref_604 - required: *ref_605 + properties: *ref_607 + required: *ref_608 WebsearchToolValue: type: object description: >- A tool implemented as a websearch tool. The AI can call this like any other websearch tool - properties: *ref_606 - required: *ref_607 + properties: *ref_609 + required: *ref_610 ToolValue: description: >- The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference - oneOf: *ref_608 - discriminator: *ref_609 + oneOf: *ref_611 + discriminator: *ref_612 AgentTool: type: object description: >- A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool - properties: *ref_345 - required: *ref_346 + properties: *ref_352 + required: *ref_353 schemas-AiAgent: type: object description: >- AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task - properties: *ref_339 - required: *ref_340 + properties: *ref_346 + required: *ref_347 schemas-StopAfterIf: type: object description: Early termination condition for a module - properties: *ref_90 - required: *ref_91 + properties: *ref_95 + required: *ref_96 RetryIf: type: object description: Conditional retry based on error or result - properties: *ref_195 - required: *ref_196 + properties: *ref_201 + required: *ref_202 schemas-Retry: type: object description: Retry configuration for failed module executions - properties: *ref_315 + properties: *ref_322 schemas-FlowNote: type: object description: A sticky note attached to a flow for documentation and annotation - properties: *ref_145 - required: *ref_146 + properties: *ref_151 + required: *ref_152 FlowGroup: type: object description: >- @@ -36908,16 +39352,16 @@ components: flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id. - properties: *ref_147 - required: *ref_148 + properties: *ref_153 + required: *ref_154 schemas-FlowValue: type: object description: >- The flow structure containing modules and optional preprocessor/failure handlers - properties: *ref_610 - required: *ref_611 + properties: *ref_613 + required: *ref_614 schemas-FlowStatusModule: type: object - properties: *ref_154 - required: *ref_155 + properties: *ref_160 + required: *ref_161 diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index 066f4a7f72..70fb08650a 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.723.0 + version: 1.742.0 title: Windmill API contact: @@ -127,40 +127,89 @@ paths: schema: type: string - /inkeep: - post: - summary: query Windmill AI documentation assistant (EE only) - operationId: queryDocumentation + /docs/search: + get: + summary: "Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more." + operationId: searchDocs x-mcp-tool: true tags: - documentation - requestBody: - description: query to send to the AI documentation assistant - required: true - content: - application/json: - schema: - type: object - properties: - query: - type: string - description: The documentation query to send to the AI assistant - required: - - query + parameters: + - name: query + description: 'Keywords to search for in the documentation body, e.g. "chromium worker tag" or "retry exponential backoff". Fewer, more distinctive words match better.' + in: query + required: true + schema: + type: string responses: "200": - description: AI documentation assistant response + description: matching documentation pages content: application/json: schema: type: object - description: Response from Inkeep service - "403": - description: Enterprise Edition required + properties: + text: + type: string + description: Model-ready rendering of the results + results: + type: array + items: + type: object + properties: + url: + type: string + title: + type: string + score: + type: integer + snippets: + type: array + items: + type: string + required: + - url + - title + - score + - snippets + required: + - text + - results + + /docs/page: + get: + summary: "Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section." + operationId: readDocsPage + x-mcp-tool: true + tags: + - documentation + parameters: + - name: url + description: "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + in: query + required: true + schema: + type: string + - name: section + description: "Optional. A heading title from the page outline to read just that section instead of the full page." + in: query + schema: + type: string + responses: + "200": + description: documentation page content content: - text/plain: + application/json: schema: - type: string + type: object + properties: + text: + type: string + source_url: + type: string + required: + - text + - source_url /openapi.yaml: get: @@ -997,6 +1046,39 @@ paths: schema: $ref: "#/components/schemas/UserWorkspaceList" + /workspaces/session_workspace_status: + post: + summary: get the lifecycle status of workspaces referenced by client-side sessions + operationId: getSessionWorkspaceStatus + tags: + - workspace + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + workspace_ids: + type: array + items: + type: string + required: + - workspace_ids + responses: + "200": + description: map of workspace id to status (active, archived, or deleted) + content: + application/json: + schema: + type: object + additionalProperties: + type: string + enum: + - active + - archived + - deleted + /w/{workspace}/workspaces/get_as_superadmin: get: summary: get workspace as super admin (require to be super admin) @@ -1655,6 +1737,9 @@ paths: s3_deleted: type: integer format: int64 + s3_not_found: + type: integer + format: int64 orphans_scanned: type: integer format: int64 @@ -1727,6 +1812,92 @@ paths: - last_run_exported - updated_at + /settings/audit_logs_s3_backfill: + post: + summary: start an opt-in historical backfill of audit logs to object storage + operationId: runAuditLogsS3Backfill + tags: + - setting + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + from: + type: string + format: date-time + description: inclusive lower bound of the window to export + to: + type: string + format: date-time + description: exclusive upper bound of the window to export + required: + - from + - to + responses: + "202": + description: backfill started + + /settings/audit_logs_s3_backfill_status: + get: + summary: get status of the audit-log object-store historical backfill + operationId: getAuditLogsS3BackfillStatus + tags: + - setting + responses: + "200": + description: current backfill status (null if never run) + content: + application/json: + schema: + nullable: true + type: object + properties: + running: + type: boolean + started_at: + type: string + format: date-time + finished_at: + type: string + format: date-time + nullable: true + phase: + type: string + from: + type: string + format: date-time + to: + type: string + format: date-time + rows_written: + type: integer + format: int64 + objects_written: + type: integer + format: int64 + last_ts: + type: string + format: date-time + nullable: true + errors: + type: integer + format: int64 + last_error: + type: string + nullable: true + required: + - running + - started_at + - phase + - from + - to + - rows_written + - objects_written + - errors + /settings/send_stats: post: summary: send stats @@ -5355,13 +5526,16 @@ paths: in: query schema: type: boolean + - $ref: "#/components/parameters/GetDraft" responses: "200": description: variable content: application/json: schema: - $ref: "#/components/schemas/ListableVariable" + allOf: + - $ref: "#/components/schemas/ListableVariable" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/variables/get_value/{path}: get: @@ -5445,6 +5619,15 @@ paths: required: false schema: type: string + - name: include_draft_only + description: | + When true, append per-user draft variables whose path has no + deployed variable. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. + in: query + required: false + schema: + type: boolean responses: "200": description: variable list @@ -6276,9 +6459,12 @@ paths: cc_client_secret: type: string description: "OAuth client secret for resource-level credentials (client_credentials flow only)" + cc_instance: + type: string + description: "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side (client_credentials flow only). The token URL is never caller-supplied." cc_token_url: type: string - description: "OAuth token URL override for resource-level authentication (client_credentials flow only)" + description: "Bring-your-own token endpoint override (client_credentials flow only). Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; ignored/rejected on the shared-instance path." mcp_server_url: type: string description: "MCP server URL for MCP OAuth token refresh" @@ -6299,13 +6485,14 @@ paths: schema: type: string - /oauth/connect_client_credentials/{client}: + /w/{workspace}/oauth/connect_client_credentials/{client}: post: summary: connect OAuth using client credentials operationId: connectClientCredentials tags: - oauth parameters: + - $ref: "#/components/parameters/WorkspaceId" - name: client in: path description: OAuth client name @@ -6326,16 +6513,16 @@ paths: type: string cc_client_id: type: string - description: "OAuth client ID for resource-level authentication" + description: "OAuth client ID. Omit to use the credentials configured on the provider's instance OAuth entry." cc_client_secret: type: string - description: "OAuth client secret for resource-level authentication" + description: "OAuth client secret. Omit to use the credentials configured on the provider's instance OAuth entry." + cc_instance: + type: string + description: "Instance name for built-in providers whose client-credentials token URL is instance-templated; substituted into the fixed-host registry template server-side. The token URL is never caller-supplied." cc_token_url: type: string - description: "OAuth token URL override for resource-level authentication" - required: - - cc_client_id - - cc_client_secret + description: "Bring-your-own token endpoint override. Only honored together with cc_client_id/cc_client_secret and mutually exclusive with cc_instance; rejected on the shared-instance path." responses: "200": description: OAuth token response @@ -6469,7 +6656,18 @@ paths: schema: type: array items: - type: string + type: object + properties: + name: + type: string + supports_client_credentials: + type: boolean + has_shared_credentials: + type: boolean + required: + - name + - supports_client_credentials + - has_shared_credentials /oauth/get_connect/{client}: get: @@ -6502,6 +6700,9 @@ paths: type: array items: type: string + client_credentials_configured: + type: boolean + description: "The instance OAuth entry carries shared client-credentials, so the connect dialog can skip the bring-your-own form and run the exchange server-side" /teams/activities: post: @@ -6678,13 +6879,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: resource content: application/json: schema: - $ref: "#/components/schemas/Resource" + allOf: + - $ref: "#/components/schemas/ListableResource" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/resources/get_value_interpolated/{path}: get: @@ -6825,6 +7029,15 @@ paths: required: false schema: type: string + - name: include_draft_only + description: | + When true, append per-user draft resources whose path has + no deployed resource. Synthesized rows carry + `draft_only: true`. + in: query + required: false + schema: + type: boolean responses: "200": description: resource list @@ -7456,6 +7669,22 @@ paths: workspace_id: type: string + /apps_u/embed_token_by_custom_path/{custom_path}: + get: + summary: get app embed token by custom path + operationId: getAppEmbedTokenByCustomPath + tags: + - app + parameters: + - $ref: "#/components/parameters/CustomPath" + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /scripts/hub/get/{path}: get: summary: get hub script content by path @@ -7813,7 +8042,40 @@ paths: schema: type: array items: - $ref: "#/components/schemas/Script" + allOf: + - $ref: "#/components/schemas/Script" + - type: object + properties: + is_draft: + type: boolean + description: | + True when the authed user has a draft for this + script — either no deployed row exists at this + path (draft-only) or the user saved a per-user + draft on top of the deployed row. + draft_path: + type: string + description: | + User-typed path the editor has staged but not + yet deployed. Surfaced for draft-only rows so + the home list can render the meaningful name + instead of the autogenerated + `u/{user}/draft_{uuid}` URL path. Omitted + when unchanged. + draft_users: + description: | + Workspace users (including the authed user, and + the legacy NULL-email row if any) who have a + per-user draft at this path. Drives the home + page's user-avatar circles inside the Draft + badge. Omitted when no drafts exist. + type: array + items: + type: object + properties: + username: + type: string + nullable: true /w/{workspace}/scripts/list_paths: get: @@ -7833,43 +8095,72 @@ paths: items: type: string - /w/{workspace}/drafts/create: - post: - summary: create draft - operationId: createDraft + /w/{workspace}/drafts/list: + get: + summary: list every draft the current user has in this workspace, across all kinds + operationId: listDrafts tags: - draft parameters: - $ref: "#/components/parameters/WorkspaceId" - requestBody: - required: true - content: - application/json: - schema: - type: object - properties: - path: - type: string - typ: - type: string - enum: ["flow", "script", "app"] - value: {} - required: - - path - - typ - - enum + - name: all_users + in: query + description: List every draft in the workspace (all users), not just the current user's own + legacy rows. Other users' rows come back with `mine=false` (view-only). + schema: + type: boolean responses: - "201": - description: draft created + "200": + description: the user's drafts content: - text/plain: + application/json: schema: - type: string + type: array + items: + type: object + properties: + kind: + $ref: "#/components/schemas/UserDraftItemKind" + path: + type: string + summary: + type: string + description: Best-effort, read from the draft JSON's `summary` field when the editor shape carries one. + draft_path: + type: string + description: User-typed friendly path from the draft JSON's `draft_path`, when set and different from the storage path (e.g. a never-deployed item parked at `u/{user}/draft_{uuid}`). + draft_only: + type: boolean + description: No deployed counterpart exists at this path — the draft is the whole item. + legacy_draft: + type: boolean + description: The listed draft is a legacy workspace-level row (email NULL) predating the per-user drafts migration. Only true when no per-user draft exists at this path. + created_at: + type: string + format: date-time + can_write: + type: boolean + description: Whether the current user may deploy/discard this draft (same check the deploy/discard endpoints enforce). + mine: + type: boolean + description: The row belongs to the current user (own draft or the legacy no-owner row) and is therefore actionable. Always true in the default listing; with `all_users=true`, other users' rows are false (view-only). + draft_users: + description: | + Draft authors at this (path, kind) — the legacy NULL-email row surfaced as a null username. + Populated only for the shared full-page-editor kinds (script/flow/app/raw_app); omitted for + drawer kinds, which keep their drafts private. Feeds the Draft badge's owner-avatar circles. + type: array + items: + type: object + properties: + username: + type: string + nullable: true + required: [kind, path, draft_only, legacy_draft, created_at, can_write, mine] - /w/{workspace}/drafts/delete/{kind}/{path}: - delete: - summary: delete draft - operationId: deleteDraft + /w/{workspace}/drafts/get/{kind}/{path}: + get: + summary: fetch a single draft's content by workspace username (or the legacy workspace-level row) + operationId: getDraftForUser tags: - draft parameters: @@ -7877,16 +8168,144 @@ paths: - name: kind in: path required: true + schema: + $ref: "#/components/schemas/UserDraftItemKind" + - $ref: "#/components/parameters/ScriptPath" + - name: username + in: query + required: false + description: Workspace username of the draft owner. Omit to fetch the legacy workspace-level (NULL email) row. schema: type: string - enum: - - script - - flow - - app + responses: + "200": + description: draft content + content: + application/json: + schema: + type: object + properties: + value: {} + created_at: + type: string + format: date-time + required: [value, created_at] + "404": + description: no draft for that owner at that path + + /w/{workspace}/drafts/get_own/{kind}/{path}: + get: + summary: fetch the current user's own draft content at a path (any kind) + operationId: getOwnDraft + tags: + - draft + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: kind + in: path + required: true + schema: + $ref: "#/components/schemas/UserDraftItemKind" - $ref: "#/components/parameters/ScriptPath" responses: "200": - description: draft deleted + description: the user's draft content, or null when none exists + content: + application/json: + schema: + nullable: true + type: object + properties: + value: {} + created_at: + type: string + format: date-time + required: [value, created_at] + + /w/{workspace}/drafts/update/{kind}/{path}: + post: + summary: upsert (or clear) the current user's draft at a path + operationId: updateDraft + tags: + - draft + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: kind + in: path + required: true + schema: + $ref: "#/components/schemas/UserDraftItemKind" + - $ref: "#/components/parameters/ScriptPath" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + value: + nullable: true + description: Draft content to save. `null` (or omitted) signals a delete — the row is removed under the same conflict rules. + last_sync: + type: string + format: date-time + description: Server timestamp of the client's last known sync for this draft. Omit on first save. + force: + type: boolean + description: Skip the conflict check and overwrite the server copy. + legacy: + type: boolean + description: Delete-only. Target the legacy workspace-level row (email NULL) instead of the current user's row. Used to discard a legacy draft from the review page. + created_at: + type: string + format: date-time + description: Upsert-only override for the stored creation timestamp. Normal saves omit it (stamped server-side); the localStorage→DB migration passes the draft's original write time so migrated drafts keep their age. + responses: + "200": + description: save result + content: + application/json: + schema: + type: object + properties: + status: + type: string + enum: [saved, conflict] + current_timestamp: + type: string + format: date-time + required: [status, current_timestamp] + + /w/{workspace}/drafts/migrate_legacy/{kind}/{path}: + post: + summary: resolve a legacy (workspace-level) draft (admin only) + description: Delete a legacy draft (email NULL) or assign it to the authed admin as a per-user draft. Workspace admins / superadmins only. + operationId: migrateLegacyDraft + tags: + - draft + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: kind + in: path + required: true + schema: + $ref: "#/components/schemas/UserDraftItemKind" + - $ref: "#/components/parameters/ScriptPath" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + action: + type: string + enum: [delete, assign_to_self] + description: delete the legacy draft, or take ownership of it. + required: [action] + responses: + "200": + description: migration result content: text/plain: schema: @@ -7900,9 +8319,10 @@ paths: Creates a new version of an existing script when called with the same path and the current `parent_hash`. operationId: createScript x-mcp-tool: true - x-mcp-instructions: "To create a script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed." + x-mcp-instructions: "To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history." x-mcp-tool-include-fields: - path + - parent_hash - content - language - summary @@ -8269,13 +8689,16 @@ paths: in: query schema: type: boolean + - $ref: "#/components/parameters/GetDraft" responses: "200": description: script details content: application/json: schema: - $ref: "#/components/schemas/Script" + allOf: + - $ref: "#/components/schemas/Script" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/scripts/get_triggers_count/{path}: get: @@ -8313,23 +8736,6 @@ paths: items: $ref: "#/components/schemas/TruncatedToken" - /w/{workspace}/scripts/get/draft/{path}: - get: - summary: get script by path with draft - operationId: getScriptByPathWithDraft - tags: - - script - parameters: - - $ref: "#/components/parameters/WorkspaceId" - - $ref: "#/components/parameters/ScriptPath" - responses: - "200": - description: script details - content: - application/json: - schema: - $ref: "#/components/schemas/NewScriptWithDraft" - /w/{workspace}/scripts/history/p/{path}: get: summary: get history of a script by path @@ -9505,10 +9911,40 @@ paths: - $ref: "#/components/schemas/Flow" - type: object properties: - has_draft: - type: boolean draft_only: type: boolean + is_draft: + type: boolean + description: | + True when the authed user has a draft for this + flow — either no deployed row exists at this + path (draft-only) or the user saved a per-user + draft on top of the deployed row. + draft_path: + type: string + description: | + User-typed path the editor has staged but not + yet deployed. Sourced from the draft JSON's + `draft_path` field (the editor only writes it + when the typed path differs from the deployed + one). Lets the home list render the meaningful + name instead of the autogenerated + `u/{user}/draft_{uuid}` URL path. Omitted when + unchanged. + draft_users: + description: | + Workspace users (including the authed user, and + the legacy NULL-email row if any) who have a + per-user draft at this path. Drives the home + page's user-avatar circles inside the Draft + badge. Omitted when no drafts exist. + type: array + items: + type: object + properties: + username: + type: string + nullable: true /w/{workspace}/flows/history/p/{path}: get: @@ -9638,13 +10074,16 @@ paths: in: query schema: type: boolean + - $ref: "#/components/parameters/GetDraft" responses: "200": description: flow details content: application/json: schema: - $ref: "#/components/schemas/Flow" + allOf: + - $ref: "#/components/schemas/Flow" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/flows/deployment_status/p/{path}: get: @@ -9732,32 +10171,6 @@ paths: schema: type: string - /w/{workspace}/flows/get/draft/{path}: - get: - summary: get flow by path with draft - operationId: getFlowByPathWithDraft - tags: - - flow - parameters: - - $ref: "#/components/parameters/WorkspaceId" - - $ref: "#/components/parameters/ScriptPath" - responses: - "200": - description: flow details with draft - content: - application/json: - schema: - allOf: - - $ref: "#/components/schemas/Flow" - - type: object - properties: - draft: - $ref: "#/components/schemas/Flow" - draft_created_at: - type: string - format: date-time - description: Timestamp at which the most recent DB draft was created. Used by the frontend's UserDraft staleness check. - /w/{workspace}/flows/exists/{path}: get: summary: exists flow by path @@ -9805,8 +10218,6 @@ paths: - $ref: "#/components/schemas/OpenFlowWPath" - type: object properties: - draft_only: - type: boolean deployment_message: type: string skip_draft_deletion: @@ -10068,6 +10479,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: All raw apps @@ -10203,6 +10615,133 @@ paths: schema: type: string + /w/{workspace}/ai_skills/list: + get: + summary: list the workspace AI chat skills (name + description only) + operationId: listAiSkills + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: skill listing + content: + application/json: + schema: + type: array + items: + type: object + required: + - name + - description + properties: + name: + type: string + description: + type: string + + /w/{workspace}/ai_skills/get/{name}: + get: + summary: get a workspace AI chat skill including its instructions + operationId: getAiSkill + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: name + in: path + required: true + schema: + type: string + responses: + "200": + description: skill + content: + application/json: + schema: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + description: + type: string + instructions: + type: string + + /w/{workspace}/ai_skills/upload: + post: + summary: upsert workspace AI chat skills (admin only) + operationId: uploadAiSkills + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + required: + - skills + properties: + skills: + type: array + maxItems: 50 + items: + type: object + required: + - name + - description + - instructions + properties: + name: + type: string + minLength: 1 + maxLength: 64 + pattern: "^[a-z0-9-]+$" + description: + type: string + minLength: 1 + maxLength: 1024 + instructions: + type: string + minLength: 1 + maxLength: 65536 + responses: + "200": + description: uploaded + content: + text/plain: + schema: + type: string + + /w/{workspace}/ai_skills/delete/{name}: + delete: + summary: delete a workspace AI chat skill (admin only) + operationId: deleteAiSkill + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: name + in: path + required: true + schema: + type: string + responses: + "200": + description: deleted + content: + text/plain: + schema: + type: string + /w/{workspace}/apps/get_data/v/{secretWithExtension}: get: summary: get raw app data by @@ -10214,6 +10753,10 @@ paths: - name: secretWithExtension in: path required: true + description: >- + App version secret suffixed with the requested file type extension. + Supported extensions are `.js` (JavaScript bundle), `.css` + (stylesheet), and `.html` (sandboxed wrapper document). schema: type: string responses: @@ -10223,6 +10766,12 @@ paths: text/javascript: schema: type: string + text/css: + schema: + type: string + text/html: + schema: + type: string /w/{workspace}/apps/list_search: get: @@ -10341,8 +10890,6 @@ paths: type: string policy: $ref: "#/components/schemas/Policy" - draft_only: - type: boolean deployment_message: type: string custom_path: @@ -10396,8 +10943,6 @@ paths: type: string policy: $ref: "#/components/schemas/Policy" - draft_only: - type: boolean deployment_message: type: string custom_path: @@ -10459,13 +11004,41 @@ paths: in: query schema: type: boolean + - $ref: "#/components/parameters/GetDraft" + - name: raw_app + in: query + description: | + When no deployed app exists at this path and `get_draft` is set, + disambiguates which draft kind (`raw_app` or `app`) to look up. + Ignored when a deployed row exists. + schema: + type: boolean responses: "200": description: app details content: application/json: schema: - $ref: "#/components/schemas/AppWithLastVersion" + allOf: + - $ref: "#/components/schemas/AppWithLastVersion" + - $ref: "#/components/schemas/UserDraftOverlay" + + /w/{workspace}/apps/embed_token/p/{path}: + get: + summary: get app embed token by path + operationId: getAppEmbedTokenByPath + tags: + - app + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/ScriptPath" + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" /w/{workspace}/apps/get/lite/{path}: get: @@ -10484,23 +11057,6 @@ paths: schema: $ref: "#/components/schemas/AppWithLastVersion" - /w/{workspace}/apps/get/draft/{path}: - get: - summary: get app by path with draft - operationId: getAppByPathWithDraft - tags: - - app - parameters: - - $ref: "#/components/parameters/WorkspaceId" - - $ref: "#/components/parameters/ScriptPath" - responses: - "200": - description: app details with draft - content: - application/json: - schema: - $ref: "#/components/schemas/AppWithLastVersionWDraft" - /w/{workspace}/apps/history/p/{path}: get: summary: get app history by path @@ -10601,6 +11157,27 @@ paths: schema: $ref: "#/components/schemas/AppWithLastVersion" + /w/{workspace}/apps_u/embed_token/{secret}: + get: + summary: get app embed token by secret + operationId: getAppEmbedTokenBySecret + tags: + - app + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: secret + in: path + required: true + schema: + type: string + responses: + "200": + description: embed token + content: + application/json: + schema: + $ref: "#/components/schemas/EmbedTokenResponse" + /w/{workspace}/apps_u/public_resource/{path}: get: summary: get public resource @@ -11309,6 +11886,11 @@ paths: in: query schema: type: boolean + - name: timeout + description: custom timeout in seconds for this preview run + in: query + schema: + type: integer - $ref: "#/components/parameters/NewJobId" requestBody: @@ -12492,6 +13074,64 @@ paths: schema: type: string + /w/{workspace}/jobs_u/get_flow_all_logs_structured/{id}: + get: + summary: get all logs for a flow job in a structured format + operationId: getFlowAllLogsStructured + tags: + - job + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/JobId" + responses: + "200": + description: structured logs of all flow steps, one entry per job + content: + application/json: + schema: + type: array + items: + type: object + properties: + job_id: + type: string + label: + type: string + description: human-readable label describing the job's position in the flow tree + kind: + type: string + description: job kind (script, flow, forloopflow, ...) + flow_step_id: + type: string + nullable: true + step_path: + type: string + nullable: true + description: materialized step path (e.g. "a/b") + depth: + type: integer + description: depth in the flow tree (0 for the root flow job) + parent_module_type: + type: string + nullable: true + description: parent module type (forloopflow, branchall, ...) + sibling_index: + type: integer + description: 1-based index of this job among siblings sharing the same step + sibling_count: + type: integer + description: total number of siblings sharing the same step + logs: + type: string + required: + - job_id + - label + - kind + - depth + - sibling_index + - sibling_count + - logs + /w/{workspace}/jobs_u/get_completed_logs_tail/{id}: get: summary: get completed job logs tail @@ -12802,6 +13442,147 @@ paths: required: - created_at + /w/{workspace}/jobs_u/dispatch_events/{id}: + get: + summary: list asset-trigger dispatch events for a producer job + description: > + Returns the chronological log of decisions the asset-trigger + dispatcher made after this producer job completed. Each row is one + (subscriber, asset write) decision: `dispatched` (with + `child_job_id`), `join_pending` (with `received_inputs` / + `required_inputs` / `partition`), or `skipped` (with `reason`). + Rows are reaped automatically when the producer's `v2_job` row is + deleted by the retention sweep. + operationId: listDispatchEvents + tags: + - job + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - $ref: "#/components/parameters/JobId" + responses: + "200": + description: dispatch events for this producer job + content: + application/json: + schema: + type: array + items: + type: object + properties: + subscriber_path: + type: string + asset_kind: + type: string + enum: + - s3object + - resource + - variable + - ducklake + - datatable + - volume + asset_path: + type: string + outcome: + type: string + enum: + - dispatched + - join_pending + - skipped + child_job_id: + type: string + format: uuid + partition: + type: string + received_inputs: + type: integer + required_inputs: + type: integer + debounce_s: + type: integer + reason: + type: string + created_at: + type: string + format: date-time + required: + - subscriber_path + - asset_kind + - asset_path + - outcome + - created_at + + /w/{workspace}/jobs/asset_dispatch_edges: + get: + summary: list asset-cascade producer→child job edges for a folder + description: > + Returns the `dispatched` asset-trigger edges (producer job → child job) + whose subscriber lives under `path_start`. Lets a pipeline view + reconstruct the cascade tree of a folder by job id and group connected + runs. Visibility follows the producer job's RLS. + operationId: listAssetDispatchEdges + tags: + - job + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: path_start + in: query + required: true + description: Folder path prefix the children live under, e.g. `f/orders/`. + schema: + type: string + - name: created_after + in: query + required: false + description: Only edges dispatched at/after this instant. + schema: + type: string + format: date-time + responses: + "200": + description: asset-cascade edges for the folder + content: + application/json: + schema: + type: array + items: + type: object + properties: + producer_job_id: + type: string + format: uuid + child_job_id: + type: string + format: uuid + description: Set for `dispatched`; absent for `join_pending` inputs. + subscriber_path: + type: string + outcome: + type: string + enum: + - dispatched + - join_pending + asset_kind: + type: string + enum: + - s3object + - resource + - variable + - ducklake + - datatable + - volume + asset_path: + type: string + created_at: + type: string + format: date-time + required: + - producer_job_id + - subscriber_path + - outcome + - asset_kind + - asset_path + - created_at + /w/{workspace}/jobs/completed/delete/{id}: post: summary: delete completed job (erase content but keep run id) @@ -13248,6 +14029,12 @@ paths: type: integer approver: type: string + view_token: + type: string + description: >- + Share-read-link token for the flow. An authenticated workspace + member can append it as a `view_token` query param on the run + page to read a flow they don't otherwise have access to. /w/{workspace}/jobs_u/resume/{id}/{resume_id}/{signature}: get: @@ -13499,6 +14286,13 @@ paths: required: - resume_id - approver + view_token: + type: string + description: >- + Share-read-link token for the parent flow. An authenticated + workspace member can append it as a `view_token` query param + on the run page to read a flow they don't otherwise have + access to. required: - job - approvers @@ -13657,13 +14451,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: schedule deleted content: application/json: schema: - $ref: "#/components/schemas/Schedule" + allOf: + - $ref: "#/components/schemas/Schedule" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/schedules/exists/{path}: get: @@ -13735,6 +14532,15 @@ paths: schema: type: string description: Filter by label + - name: include_draft_only + description: | + When true, append per-user draft schedules whose path has + no deployed schedule. Synthesized rows carry + `draft_only: true`. + in: query + required: false + schema: + type: boolean responses: "200": description: schedule list @@ -13946,13 +14752,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: http trigger deleted content: application/json: schema: - $ref: "#/components/schemas/HttpTrigger" + allOf: + - $ref: "#/components/schemas/HttpTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/http_triggers/list: get: @@ -13984,6 +14793,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: http trigger list @@ -14152,13 +14962,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: websocket trigger deleted content: application/json: schema: - $ref: "#/components/schemas/WebsocketTrigger" + allOf: + - $ref: "#/components/schemas/WebsocketTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/websocket_triggers/list: get: @@ -14190,6 +15003,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: websocket trigger list @@ -14357,13 +15171,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: kafka trigger deleted content: application/json: schema: - $ref: "#/components/schemas/KafkaTrigger" + allOf: + - $ref: "#/components/schemas/KafkaTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/kafka_triggers/list: get: @@ -14395,6 +15212,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: kafka trigger list @@ -14603,13 +15421,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: nats trigger deleted content: application/json: schema: - $ref: "#/components/schemas/NatsTrigger" + allOf: + - $ref: "#/components/schemas/NatsTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/nats_triggers/list: get: @@ -14641,6 +15462,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: nats trigger list @@ -14803,13 +15625,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: sqs trigger deleted content: application/json: schema: - $ref: "#/components/schemas/SqsTrigger" + allOf: + - $ref: "#/components/schemas/SqsTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/sqs_triggers/list: get: @@ -14841,6 +15666,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: sqs trigger list @@ -15296,6 +16122,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: native triggers list @@ -15596,13 +16423,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: mqtt trigger deleted content: application/json: schema: - $ref: "#/components/schemas/MqttTrigger" + allOf: + - $ref: "#/components/schemas/MqttTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/mqtt_triggers/list: get: @@ -15634,6 +16464,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: mqtt trigger list @@ -15796,13 +16627,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: gcp trigger deleted content: application/json: schema: - $ref: "#/components/schemas/GcpTrigger" + allOf: + - $ref: "#/components/schemas/GcpTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/gcp_triggers/list: get: @@ -15834,6 +16668,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: gcp trigger list @@ -16063,13 +16898,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: azure trigger content: application/json: schema: - $ref: "#/components/schemas/AzureTrigger" + allOf: + - $ref: "#/components/schemas/AzureTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/azure_triggers/list: get: @@ -16094,6 +16932,7 @@ paths: in: query schema: type: string + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: azure trigger list @@ -16606,13 +17445,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: get postgres trigger content: application/json: schema: - $ref: "#/components/schemas/PostgresTrigger" + allOf: + - $ref: "#/components/schemas/PostgresTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/postgres_triggers/list: get: @@ -16644,6 +17486,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: postgres trigger list @@ -16806,13 +17649,16 @@ paths: parameters: - $ref: "#/components/parameters/WorkspaceId" - $ref: "#/components/parameters/Path" + - $ref: "#/components/parameters/GetDraft" responses: "200": description: email trigger retrieved content: application/json: schema: - $ref: "#/components/schemas/EmailTrigger" + allOf: + - $ref: "#/components/schemas/EmailTrigger" + - $ref: "#/components/schemas/UserDraftOverlay" /w/{workspace}/email_triggers/list: get: @@ -16844,6 +17690,7 @@ paths: schema: type: string description: Filter by label + - $ref: "#/components/parameters/IncludeDraftOnly" responses: "200": description: email trigger list @@ -20365,6 +21212,149 @@ paths: type: string description: The asset path + /w/{workspace}/assets/graph: + get: + summary: Get the workspace-wide asset <-> runnable graph + operationId: getAssetsGraph + tags: + - asset + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: asset_kinds + in: query + description: Filter by asset kinds (comma-separated list) + schema: + type: string + - name: folder + in: query + description: Scope the graph to runnables in a single folder + schema: + type: string + responses: + "200": + description: asset graph nodes, lineage edges and trigger edges + content: + application/json: + schema: + type: object + required: [assets, runnables, edges, triggers] + properties: + assets: + type: array + items: + type: object + required: [kind, path] + properties: + kind: + $ref: "#/components/schemas/AssetKind" + path: + type: string + runnables: + type: array + items: + type: object + required: [path, usage_kind] + properties: + path: + type: string + usage_kind: + $ref: "#/components/schemas/AssetUsageKind" + in_pipeline: + type: boolean + description: True iff the script is a pipeline member (deployed with `// pipeline`). Omitted when false. + edges: + type: array + items: + type: object + required: + [runnable_path, runnable_kind, asset_kind, asset_path] + properties: + runnable_path: + type: string + runnable_kind: + $ref: "#/components/schemas/AssetUsageKind" + asset_kind: + $ref: "#/components/schemas/AssetKind" + asset_path: + type: string + access_type: + $ref: "#/components/schemas/AssetUsageAccessType" + triggers: + type: array + items: + oneOf: + - type: object + description: Asset trigger edge (`// on `) + required: + [ + trigger_kind, + asset_kind, + asset_path, + runnable_kind, + runnable_path, + ] + properties: + trigger_kind: + type: string + enum: [asset] + asset_kind: + $ref: "#/components/schemas/AssetKind" + asset_path: + type: string + runnable_kind: + $ref: "#/components/schemas/AssetUsageKind" + runnable_path: + type: string + - type: object + description: Native trigger edge (schedule, email, kafka, ...). `path` is the trigger row's path. + required: + [trigger_kind, path, runnable_kind, runnable_path] + properties: + trigger_kind: + type: string + enum: + - schedule + - email + - kafka + - mqtt + - nats + - postgres + - sqs + - gcp + path: + type: string + runnable_kind: + $ref: "#/components/schemas/AssetUsageKind" + runnable_path: + type: string + + /w/{workspace}/assets/pipelines: + get: + summary: List folders that contain at least one pipeline-member script + operationId: listPipelineFolders + tags: + - asset + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: folders containing pipeline scripts, with their script counts + content: + application/json: + schema: + type: array + items: + type: object + required: [folder, script_count] + properties: + folder: + type: string + description: The folder name (without the `f/` prefix) + script_count: + type: integer + format: int64 + description: Number of pipeline-member scripts in the folder + /w/{workspace}/volumes/list: get: summary: List all volumes in the workspace @@ -20569,6 +21559,25 @@ components: name: token parameters: + GetDraft: + name: get_draft + in: query + required: false + description: When true, overlay the authed user's draft (if any) onto the deployed payload. + schema: + type: boolean + IncludeDraftOnly: + name: include_draft_only + in: query + required: false + description: | + When true, append per-user draft rows whose path has no + deployed counterpart. Synthesized rows carry `draft_only: true` + so the home page can render a "Draft" badge. Gated to + non-operators + page 0 + no narrowing filters on the backend so + picker callers stay deployed-only and pagination stays clean. + schema: + type: boolean Id: name: id in: path @@ -20998,6 +22007,89 @@ components: # NOTE: Not so many generators and validators support this format: # $ref: "../../openflow.openapi.yaml#/components/schemas" # This is why it is better to inline each of schemas for better compat + UserDraftOverlay: + type: object + description: | + Overlay fields added to every "get by path" response that accepts + the `get_draft` query parameter. The deployed payload is sent + untouched in the response body; the authed user's saved draft + for this path — whatever shape the editor wrote — is attached + as the sibling `draft` field when `get_draft=true` and a draft + exists. The frontend pairs the two to present diff / reset / + discard UI; the server never merges them. + + When `no_deployed=true` there is no deployed row at this path — + the response body is a best-effort stand-in synthesized from + the draft, and only `draft` is canonical. Callers should disable + "diff vs deployed" UI in that case. + properties: + is_draft: + type: boolean + draft_saved_at: + type: string + format: date-time + no_deployed: + type: boolean + draft: + type: object + additionalProperties: true + other_drafts_users: + description: | + Other workspace users (and the legacy NULL-email row, if any) + with a saved draft at the same path. Populated only on the + authed user's "get by path" responses for kinds the editor + surfaces a fork banner for (script, flow, app, raw_app). + Empty / omitted for kinds without that UI. + type: array + items: + type: object + properties: + username: + type: string + nullable: true + description: | + Workspace username of the draft owner. `null` represents + the legacy workspace-level (NULL-email) row. Emails never + leave the server. + draft_saved_at: + type: string + format: date-time + description: | + When this user's draft was last saved (`draft.created_at`), + surfaced in the fork modal as "Last updated". + required: [draft_saved_at] + required: [is_draft] + UserDraftItemKind: + type: string + description: | + Closed set of item kinds a user can autosave as a draft. Mirrors the + Postgres `DRAFT_KIND` enum and the backend `UserDraftItemKind`. + enum: + - script + - flow + - app + - raw_app + - resource + - variable + - trigger_schedule + - trigger_webhook + - trigger_default_email + - trigger_email + - trigger_http + - trigger_websocket + - trigger_postgres + - trigger_kafka + - trigger_nats + - trigger_mqtt + - trigger_sqs + - trigger_gcp + - trigger_azure + - trigger_poll + - trigger_cli + - trigger_nextcloud + - trigger_google + - trigger_github + - data_pipeline # Do not change next line. It is used by python-client for pre-processing # -- INLINE START -- OpenFlow: @@ -21822,8 +22914,6 @@ components: type: boolean tag: type: string - has_draft: - type: boolean draft_only: type: boolean envs: @@ -21917,6 +23007,13 @@ components: type: string parent_hash: type: string + auto_parent: + type: boolean + description: >- + When true, the backend resolves the parent to the current deployed + head for this path within the transaction (ignoring parent_hash), + instead of failing with a "lineage must be linear" error when the + supplied parent_hash is stale. summary: type: string description: @@ -21936,8 +23033,6 @@ components: enum: [script, failure, trigger, command, approval, preprocessor] tag: type: string - draft_only: - type: boolean envs: type: array items: @@ -22030,22 +23125,6 @@ components: - content - language - NewScriptWithDraft: - allOf: - - $ref: "#/components/schemas/NewScript" - - type: object - properties: - draft: - $ref: "#/components/schemas/NewScript" - draft_created_at: - type: string - format: date-time - description: Timestamp at which the most recent DB draft was created. Used by the frontend's UserDraft staleness check. - hash: - type: string - required: - - hash - ScriptHistory: type: object properties: @@ -22216,6 +23295,8 @@ components: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string required: @@ -22336,6 +23417,8 @@ components: type: number preprocessed: type: boolean + is_retry: + type: boolean worker: type: string required: @@ -22710,6 +23793,10 @@ components: type: array items: type: string + folders_read: + type: array + items: + type: string folders_owners: type: array items: @@ -22729,6 +23816,7 @@ components: - operator - disabled - folders + - folders_read - folders_owners UserSource: @@ -23120,6 +24208,17 @@ components: format: date-time edited_by: type: string + draft_only: + description: | + True when this row is a per-user draft with no deployed + variable at the same path. Frontend renders a "Draft" badge. + type: boolean + is_draft: + description: | + True when the authed user has a per-user draft at this path + (over a deployed row or a synthesized draft-only row). + Frontend appends a `*` to the displayed name. + type: boolean required: - workspace_id - path @@ -23717,6 +24816,17 @@ components: Read-only — edit them on the folder. ws_specific: type: boolean + draft_only: + description: | + True when this row is a per-user draft with no deployed + resource at the same path. Frontend renders a "Draft" badge. + type: boolean + is_draft: + description: | + True when the authed user has a per-user draft at this path + (over a deployed row or a synthesized draft-only row). + Frontend appends a `*` to the displayed name. + type: boolean required: - path - resource_type @@ -23873,6 +24983,17 @@ components: items: type: string default: [] + draft_only: + description: | + True when this row is a per-user draft with no deployed + schedule at the same path. Frontend renders a "Draft" badge. + type: boolean + is_draft: + description: | + True when the authed user has a per-user draft at this path + (over a deployed row or a synthesized draft-only row). + Frontend appends a `*` to the displayed name. + type: boolean inherited_labels: type: array items: @@ -24147,6 +25268,7 @@ components: - azure - google - github + - asset TriggerMode: description: job trigger mode @@ -24194,6 +25316,19 @@ components: items: type: string default: [] + draft_only: + description: | + True when this row is a per-user draft with no deployed + trigger at the same path. Set by list endpoints when + `include_draft_only=true` synthesizes the row from the + draft. Frontend renders a "Draft" badge. + type: boolean + is_draft: + description: | + True when the authed user has a per-user draft at this path + (over a deployed row or a synthesized draft-only row). + Frontend appends a `*` to the displayed name. + type: boolean required: - path - script_path @@ -26997,6 +28132,13 @@ components: type: string on_behalf_of_email: type: string + sandbox: + type: boolean + description: > + Publisher opt-in to app sandbox isolation (alpha). When true the app + is isolated from each viewer's Windmill session. When false/absent + the app runs same-origin with the viewer's full session (the + default, pre-isolation behavior). ListableApp: type: object @@ -27030,6 +28172,34 @@ components: items: type: string default: [] + is_draft: + type: boolean + description: | + True when the authed user has a draft for this app — either no + deployed row exists at this path (draft-only) or the user has + saved a per-user draft on top of the deployed row. + draft_path: + type: string + description: | + User-typed path the editor has staged but not yet deployed. + Sourced from the draft JSON's `draft_path` field (the editor + only writes it when the typed path differs from the deployed + one). Lets the home list render the meaningful name instead of + the autogenerated `u/{user}/draft_{uuid}` URL path. Omitted + when unchanged. + draft_users: + description: | + Workspace users (including the authed user, and the legacy + NULL-email row if any) who have a per-user draft at this + path. Drives the home page's user-avatar circles inside the + Draft badge. Omitted when no drafts exist. + type: array + items: + type: object + properties: + username: + type: string + nullable: true inherited_labels: type: array items: @@ -27178,19 +28348,6 @@ components: - raw_app - AppWithLastVersionWDraft: - allOf: - - $ref: "#/components/schemas/AppWithLastVersion" - - type: object - properties: - draft_only: - type: boolean - draft: {} - draft_created_at: - type: string - format: date-time - description: Timestamp at which the most recent DB draft was created. Used by the frontend's UserDraft staleness check. - AppHistory: type: object properties: @@ -27201,6 +28358,36 @@ components: required: - version + EmbedTokenResponse: + type: object + properties: + token: + type: string + nullable: true + description: Narrowly-scoped embed token for the iframe. Absent for fully anonymous or raw apps, which load without a scoped token. + expiration: + type: string + format: date-time + nullable: true + description: Expiration of the embed token. + raw_app: + type: boolean + description: Raw apps render single-iframe and skip the opaque-viewer indirection and the embed token entirely. + sandbox: + type: boolean + description: Publisher opted this app into sandbox isolation. When false the viewer runs the app same-origin with its full session. + app_path: + type: string + nullable: true + description: The resolved app path; the embedder uses it to scope the app's backing localStorage per app. + workspace_id: + type: string + nullable: true + description: The resolved workspace; pairs with app_path so apps at the same path in different workspaces don't share a localStorage store. + required: + - raw_app + - sandbox + FlowVersion: type: object properties: diff --git a/backend/windmill-api/src/ai.rs b/backend/windmill-api/src/ai.rs index fb92d28cb3..6195fe5644 100644 --- a/backend/windmill-api/src/ai.rs +++ b/backend/windmill-api/src/ai.rs @@ -168,9 +168,6 @@ struct AIStandardResource { /// Platform (standard or google_vertex_ai) #[serde(default)] platform: AIPlatform, - /// Enable 1M context window for Anthropic - #[serde(alias = "enable_1M_context", default)] - enable_1m_context: bool, /// Custom HTTP headers to include in AI requests #[serde(default)] headers: HashMap, @@ -263,7 +260,6 @@ async fn resolve_provider_credentials( aws_secret_access_key, aws_session_token, platform: resource.platform, - enable_1m_context: resource.enable_1m_context, custom_headers: resource.headers, }) } @@ -288,7 +284,6 @@ async fn resolve_provider_credentials( aws_secret_access_key: None, aws_session_token: None, platform: AIPlatform::Standard, - enable_1m_context: false, custom_headers: HashMap::new(), }) } @@ -548,7 +543,6 @@ async fn global_proxy( aws_secret_access_key: None, aws_session_token: None, platform: AIPlatform::Standard, - enable_1m_context: false, custom_headers: HashMap::new(), }; @@ -958,11 +952,30 @@ mod tests { aws_secret_access_key: None, aws_session_token: None, platform: AIPlatform::Standard, - enable_1m_context: false, custom_headers: HashMap::new(), } } + #[test] + fn ai_standard_resource_ignores_legacy_enable_1m_context_keys() { + // Resources created before the field was removed still carry the legacy key + // (lowercase `enable_1m_context` or the frontend alias `enable_1M_context`). + // The struct has no `deny_unknown_fields`, so both must be silently ignored. + let json = r#"{ + "base_url": "https://api.anthropic.com", + "enable_1m_context": true, + "enable_1M_context": true + }"#; + + let resource: AIStandardResource = + serde_json::from_str(json).expect("legacy resource must still deserialize"); + + assert_eq!( + resource.base_url.as_deref(), + Some("https://api.anthropic.com") + ); + } + #[test] fn invalidates_all_cached_providers_for_workspace() { let _guard = TEST_LOCK.lock().unwrap(); diff --git a/backend/windmill-api/src/ai_skills.rs b/backend/windmill-api/src/ai_skills.rs new file mode 100644 index 0000000000..6deb8dc58f --- /dev/null +++ b/backend/windmill-api/src/ai_skills.rs @@ -0,0 +1,394 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2026 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +use crate::db::{ApiAuthed, DB}; +use std::collections::HashSet; +use axum::{ + extract::{Extension, Json, Path}, + routing::{delete, get, post}, + Router, +}; +use serde::{Deserialize, Serialize}; +use windmill_audit::audit_oss::audit_log; +use windmill_audit::ActionKind; +use windmill_common::{ + db::UserDB, + error::{Error, JsonResult, Result}, + utils::require_admin, +}; + +pub fn workspaced_service() -> Router { + Router::new() + .route("/list", get(list_skills)) + .route("/get/{name}", get(get_skill)) + .route("/upload", post(upload_skills)) + .route("/delete/{name}", delete(delete_skill)) +} + +/// Cheap listing surfaced in the AI chat system prompt — no `instructions` body. +#[derive(Serialize)] +pub struct SkillListItem { + pub name: String, + pub description: String, +} + +/// Full skill, including the SKILL.md body, fetched on demand by `read_skill`. +#[derive(Serialize)] +pub struct Skill { + pub name: String, + pub description: String, + pub instructions: String, +} + +#[derive(Deserialize)] +pub struct UploadSkills { + pub skills: Vec, +} + +#[derive(Deserialize)] +pub struct SkillUpload { + pub name: String, + pub description: String, + pub instructions: String, +} + +const MAX_SKILLS_PER_UPLOAD: usize = 50; +// Every stored skill's name + description is advertised in the global AI chat +// system prompt, so bound the total a workspace can accumulate across uploads. +const MAX_SKILLS_PER_WORKSPACE: usize = 100; +// `name` and `description` follow the Claude SKILL.md spec +// (https://platform.claude.com/docs/en/agents-and-tools/agent-skills): both are +// loaded into the AI chat system prompt and `name` is the model-facing skill id, +// so matching the upstream limits keeps skills portable with Claude Code. +const MAX_SKILL_NAME_CHARS: usize = 64; +const MAX_SKILL_DESCRIPTION_CHARS: usize = 1_024; +// Not a spec field — a payload bound on the SKILL.md body, so measured in bytes. +const MAX_SKILL_INSTRUCTIONS_BYTES: usize = 64 * 1024; + +fn validate_skill(skill: &SkillUpload) -> Result<()> { + let name = skill.name.trim(); + if name.is_empty() || name.chars().count() > MAX_SKILL_NAME_CHARS { + return Err(Error::BadRequest(format!( + "skill name must be between 1 and {MAX_SKILL_NAME_CHARS} characters, got {:?}", + skill.name + ))); + } + if !name + .chars() + .all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-') + { + return Err(Error::BadRequest(format!( + "skill name {name:?} must only contain lowercase letters, digits or '-'" + ))); + } + if skill.description.trim().is_empty() { + return Err(Error::BadRequest(format!( + "skill {name:?} is missing a description (the SKILL.md frontmatter `description`)" + ))); + } + if skill.description.chars().count() > MAX_SKILL_DESCRIPTION_CHARS { + return Err(Error::BadRequest(format!( + "skill {name:?} description must be at most {MAX_SKILL_DESCRIPTION_CHARS} characters" + ))); + } + if skill.instructions.trim().is_empty() { + return Err(Error::BadRequest(format!( + "skill {name:?} has an empty SKILL.md body" + ))); + } + if skill.instructions.len() > MAX_SKILL_INSTRUCTIONS_BYTES { + return Err(Error::BadRequest(format!( + "skill {name:?} instructions must be at most {MAX_SKILL_INSTRUCTIONS_BYTES} bytes" + ))); + } + Ok(()) +} + +/// Collect the trimmed skill names, rejecting duplicates within a single upload. +/// The insert upserts by name, so a duplicate would silently keep only the last +/// and make the reported/audited count wrong. +fn collect_upload_names(skills: &[SkillUpload]) -> Result> { + let mut names = Vec::with_capacity(skills.len()); + let mut seen = HashSet::with_capacity(skills.len()); + for skill in skills { + let name = skill.name.trim().to_string(); + if !seen.insert(name.clone()) { + return Err(Error::BadRequest(format!( + "duplicate skill name {name:?} in upload" + ))); + } + names.push(name); + } + Ok(names) +} + +/// Reject an upload that would push the workspace past `MAX_SKILLS_PER_WORKSPACE`. +/// Uploads upsert, so names already present (`replacing`) don't count as new. +fn check_workspace_skill_capacity( + existing_total: i64, + replacing: i64, + upload_count: usize, +) -> Result<()> { + let new_count = upload_count as i64 - replacing; + if existing_total + new_count > MAX_SKILLS_PER_WORKSPACE as i64 { + return Err(Error::BadRequest(format!( + "workspace cannot store more than {MAX_SKILLS_PER_WORKSPACE} skills" + ))); + } + Ok(()) +} + +async fn list_skills( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + let rows = sqlx::query!( + "SELECT name, description FROM ai_skill WHERE workspace_id = $1 ORDER BY name", + &w_id + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + + Ok(Json( + rows.into_iter() + .map(|r| SkillListItem { name: r.name, description: r.description }) + .collect(), + )) +} + +async fn get_skill( + authed: ApiAuthed, + Extension(user_db): Extension, + Path((w_id, name)): Path<(String, String)>, +) -> JsonResult { + let mut tx = user_db.begin(&authed).await?; + let row = sqlx::query!( + "SELECT name, description, instructions FROM ai_skill WHERE workspace_id = $1 AND name = $2", + &w_id, + &name + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + + row.map(|r| { + Json(Skill { name: r.name, description: r.description, instructions: r.instructions }) + }) + .ok_or_else(|| Error::NotFound(format!("no skill named {name:?} in workspace {w_id}"))) +} + +/// Bulk upsert the uploaded skills by name. Existing skills not in the payload +/// are left untouched — removal goes through `delete_skill`. +async fn upload_skills( + authed: ApiAuthed, + Extension(db): Extension, + Path(w_id): Path, + Json(payload): Json, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + + if payload.skills.is_empty() { + return Err(Error::BadRequest("no skills to upload".to_string())); + } + if payload.skills.len() > MAX_SKILLS_PER_UPLOAD { + return Err(Error::BadRequest(format!( + "cannot upload more than {MAX_SKILLS_PER_UPLOAD} skills at a time" + ))); + } + for skill in &payload.skills { + validate_skill(skill)?; + } + let names = collect_upload_names(&payload.skills)?; + + let mut tx = db.begin().await?; + let counts = sqlx::query!( + r#"SELECT + COUNT(*)::bigint AS "total!", + COUNT(*) FILTER (WHERE name = ANY($2::text[]))::bigint AS "replacing!" + FROM ai_skill + WHERE workspace_id = $1"#, + &w_id, + &names + ) + .fetch_one(&mut *tx) + .await?; + check_workspace_skill_capacity(counts.total, counts.replacing, names.len())?; + + for (skill, name) in payload.skills.iter().zip(names.iter()) { + sqlx::query!( + r#"INSERT INTO ai_skill (workspace_id, name, description, instructions, edited_at, edited_by) + VALUES ($1, $2, $3, $4, now(), $5) + ON CONFLICT (workspace_id, name) DO UPDATE + SET description = EXCLUDED.description, + instructions = EXCLUDED.instructions, + edited_at = now(), + edited_by = EXCLUDED.edited_by"#, + &w_id, + name, + skill.description, + skill.instructions, + &authed.username, + ) + .execute(&mut *tx) + .await?; + } + + let audit_resource = names.join(","); + audit_log( + &mut *tx, + &authed, + "ai_skills.upload", + ActionKind::Update, + &w_id, + Some(&audit_resource), + Some([("skill_count", &names.len().to_string()[..])].into()), + ) + .await?; + tx.commit().await?; + + Ok(format!( + "Uploaded {} skill(s) to workspace {}", + payload.skills.len(), + &w_id + )) +} + +async fn delete_skill( + authed: ApiAuthed, + Extension(db): Extension, + Path((w_id, name)): Path<(String, String)>, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + + let mut tx = db.begin().await?; + let deleted = sqlx::query_scalar!( + "DELETE FROM ai_skill WHERE workspace_id = $1 AND name = $2 RETURNING name", + &w_id, + &name + ) + .fetch_optional(&mut *tx) + .await?; + + if deleted.is_none() { + tx.commit().await?; + return Err(Error::NotFound(format!( + "no skill named {name:?} in workspace {w_id}" + ))); + } + + audit_log( + &mut *tx, + &authed, + "ai_skills.delete", + ActionKind::Delete, + &w_id, + Some(&name), + None, + ) + .await?; + tx.commit().await?; + + Ok(format!("Deleted skill {name} from workspace {w_id}")) +} + +#[cfg(test)] +mod tests { + use super::*; + + fn skill() -> SkillUpload { + SkillUpload { + name: "test-skill".to_string(), + description: "Useful for tests".to_string(), + instructions: "# Test\n\nDo the thing.".to_string(), + } + } + + #[test] + fn validate_skill_rejects_oversized_description() { + let mut skill = skill(); + skill.description = "x".repeat(MAX_SKILL_DESCRIPTION_CHARS + 1); + + assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_)))); + } + + #[test] + fn validate_skill_rejects_oversized_instructions() { + let mut skill = skill(); + skill.instructions = "x".repeat(MAX_SKILL_INSTRUCTIONS_BYTES + 1); + + assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_)))); + } + + #[test] + fn validate_skill_rejects_oversized_name() { + let mut skill = skill(); + skill.name = "a".repeat(MAX_SKILL_NAME_CHARS + 1); + + assert!(matches!(validate_skill(&skill), Err(Error::BadRequest(_)))); + } + + #[test] + fn validate_skill_rejects_non_slug_name() { + // Uppercase, underscore, space and punctuation are all outside the + // Claude SKILL.md `[a-z0-9-]` name charset. + for bad in ["My-Skill", "my_skill", "my skill", "skill!"] { + let mut skill = skill(); + skill.name = bad.to_string(); + + assert!( + matches!(validate_skill(&skill), Err(Error::BadRequest(_))), + "{bad:?} should be rejected" + ); + } + } + + #[test] + fn validate_skill_counts_description_in_characters() { + // 1024 two-byte chars exceed the byte limit but sit exactly on the + // character limit, so they must be accepted. + let mut skill = skill(); + skill.description = "é".repeat(MAX_SKILL_DESCRIPTION_CHARS); + + assert!(validate_skill(&skill).is_ok()); + } + + #[test] + fn workspace_capacity_allows_replacement_at_cap() { + // Already at the cap, but the upload only replaces an existing skill. + let at_cap = MAX_SKILLS_PER_WORKSPACE as i64; + assert!(check_workspace_skill_capacity(at_cap, 1, 1).is_ok()); + } + + #[test] + fn workspace_capacity_rejects_new_skill_over_cap() { + let at_cap = MAX_SKILLS_PER_WORKSPACE as i64; + assert!(matches!( + check_workspace_skill_capacity(at_cap, 0, 1), + Err(Error::BadRequest(_)) + )); + } + + #[test] + fn collect_upload_names_trims_and_collects() { + let names = collect_upload_names(&[skill()]).unwrap(); + assert_eq!(names, vec!["test-skill".to_string()]); + } + + #[test] + fn collect_upload_names_rejects_duplicates() { + // Names are compared after trimming, so whitespace can't smuggle a dup in. + let dup = SkillUpload { name: " test-skill ".to_string(), ..skill() }; + assert!(matches!( + collect_upload_names(&[skill(), dup]), + Err(Error::BadRequest(_)) + )); + } +} diff --git a/backend/windmill-api/src/apps.rs b/backend/windmill-api/src/apps.rs index ed75c25c6f..08a41e2d1f 100644 --- a/backend/windmill-api/src/apps.rs +++ b/backend/windmill-api/src/apps.rs @@ -12,7 +12,7 @@ use crate::{ db::{ApiAuthed, DB}, jobs::RunJobQuery, users::{require_owner_of_path, require_path_read_access_for_preview, OptAuthed}, - utils::{check_scopes, WithStarredInfoQuery}, + utils::{build_scope_path_predicate, check_scopes}, webhook_util::{WebhookMessage, WebhookShared}, HTTP_CLIENT, }; @@ -58,6 +58,7 @@ use windmill_common::{ get_payload_tag_from_prefixed_path, resolve_delete_after_secs, schedule_job_deletion, JobPayload, RawCode, }, + user_drafts::{overlay_or_draft_only, DraftUserRef, UserDraftItemKind, WithDraftOverlay}, users::username_to_permissioned_as, utils::{ http_get_from_hub, not_found_if_none, paginate, query_elems_from_hub, require_admin, @@ -74,6 +75,7 @@ use windmill_common::{ use windmill_object_store::object_store_reexports::{Attribute, Attributes}; use windmill_store::resources::get_resource_value_interpolated_internal; +use windmill_api_auth::{create_token_internal, ensure_scopes_within_caller, NewToken}; use windmill_git_sync::{handle_deployment_metadata, DeployedObject}; use windmill_queue::{push, PushArgs, PushArgsOwned, PushIsolationLevel}; @@ -89,8 +91,8 @@ pub fn workspaced_service(raw_app_body_limit: usize) -> Router { .route("/list", get(list_apps)) .route("/list_search", get(list_search_apps)) .route("/get/p/{*path}", get(get_app)) + .route("/embed_token/p/{*path}", get(get_app_embed_token_for_path)) .route("/get/lite/{*path}", get(get_app_lite)) - .route("/get/draft/{*path}", get(get_app_w_draft)) .route("/secret_of/{*path}", get(get_secret_id)) .route( "/secret_of_latest_version/{*path}", @@ -134,6 +136,7 @@ pub fn unauthed_service() -> Router { .route("/delete_s3_file", delete(delete_s3_file_from_app)) .route("/download_s3_file/{*path}", get(download_s3_file_from_app)) .route("/public_app/{secret}", get(get_public_app_by_secret)) + .route("/embed_token/{secret}", get(get_app_embed_token)) .route("/public_resource/{*path}", get(get_public_resource)) .route("/get_data/v/{*id}", get(get_raw_app_data)) } @@ -155,7 +158,9 @@ pub struct ListableApp { pub execution_mode: String, pub starred: bool, pub edited_at: Option>, - pub has_draft: bool, + /// `Some(true)` only on rows synthesised from the `draft` table; `None` for + /// deployed rows. See ListableScript in windmill-types/src/scripts.rs. + #[sqlx(default)] #[serde(skip_serializing_if = "Option::is_none")] pub draft_only: Option, #[sqlx(default)] @@ -165,6 +170,20 @@ pub struct ListableApp { pub raw_app: bool, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + /// True when the authed user has a draft for this app (draft-only or layered + /// over the deployed row). See ListableScript in windmill-types/src/scripts.rs. + #[serde(default, skip_serializing_if = "is_false")] + pub is_draft: bool, + /// User-typed staged path from the draft JSON's `draft_path`; `None` = unchanged. + /// See ListableScript in windmill-types/src/scripts.rs. + #[sqlx(default)] + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_path: Option, + /// Per-path draft owners driving the home-page avatar circles. + /// See ListableScript in windmill-types/src/scripts.rs. + #[sqlx(default)] + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_users: Option>>, /// Labels inherited from the parent folder, computed at read time. #[sqlx(default)] #[serde(default, skip_serializing_if = "Option::is_none")] @@ -214,20 +233,6 @@ pub struct AppWithLastVersionAndStarred { pub starred: Option, } -#[derive(Serialize, Deserialize, FromRow)] -pub struct AppWithLastVersionAndDraft { - #[sqlx(flatten)] - #[serde(flatten)] - pub app: AppWithLastVersion, - #[serde(skip_serializing_if = "Option::is_none")] - pub draft: Option>>, - #[serde(skip_serializing_if = "Option::is_none")] - pub draft_only: Option, - /// Timestamp at which the most recent DB draft was created. - #[serde(skip_serializing_if = "Option::is_none")] - pub draft_created_at: Option>, -} - #[derive(Serialize)] pub struct AppHistory { pub app_id: i64, @@ -298,6 +303,13 @@ pub struct Policy { pub execution_mode: ExecutionMode, pub s3_inputs: Option>, pub allowed_s3_keys: Option>, + // WIN-2006: publisher opt-in to iframe sandbox isolation (alpha). When true the + // app is isolated from each viewer's Windmill session: low-code renders in an + // opaque-origin iframe with a scoped embed token, raw renders its bundle in an + // opaque iframe. Default/absent means unsandboxed — the app runs same-origin + // with the viewer's full session, the pre-isolation behavior. + #[serde(skip_serializing_if = "Option::is_none")] + pub sandbox: Option, } #[derive(Deserialize)] @@ -306,7 +318,6 @@ pub struct CreateApp { pub summary: String, pub value: sqlx::types::Json>, pub policy: Policy, - pub draft_only: Option, pub deployment_message: Option, pub custom_path: Option, pub preserve_on_behalf_of: Option, @@ -347,6 +358,12 @@ async fn list_search_apps( Path(w_id): Path, Extension(user_db): Extension, ) -> JsonResult> { + // Require domain-level read: this returns every visible app's full value (code). + // The route layer treats `apps:run` as satisfying read, so without this handler + // check a scoped embed token (apps:run + apps:read:) could read all + // apps' definitions. `check_scopes` uses ScopeDefinition::includes, where run + // does NOT include read, so it correctly denies such tokens. + check_scopes(&authed, || "apps:read".to_string())?; #[cfg(feature = "enterprise")] let n = 1000; @@ -354,6 +371,8 @@ async fn list_search_apps( let n = 3; let mut tx = user_db.begin(&authed).await?; + let allowed = build_scope_path_predicate(&authed, "apps", "read"); + let rows = sqlx::query_as::<_, SearchApp>( "SELECT path, app_version.value from app LEFT JOIN app_version ON app_version.id = versions[array_upper(versions, 1)] WHERE workspace_id = $1 LIMIT $2", ) @@ -362,6 +381,7 @@ async fn list_search_apps( .fetch_all(&mut *tx) .await? .into_iter() + .filter(|r| allowed(&r.path)) .collect::>(); tx.commit().await?; Ok(Json(rows)) @@ -370,10 +390,14 @@ async fn list_search_apps( async fn list_apps( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path(w_id): Path, Query(pagination): Query, Query(lq): Query, ) -> JsonResult> { + // Domain-level read (see list_search_apps): keeps a scoped embed token, whose + // `apps:run` only satisfies read at the route layer, from listing all apps. + check_scopes(&authed, || "apps:read".to_string())?; let (per_page, offset) = paginate(pagination); let mut sqlb = SqlBuilder::select_from("app") @@ -387,10 +411,18 @@ async fn list_apps( "app_version.created_at as edited_at", "app.extra_perms", "favorite.path IS NOT NULL as starred", - "draft.path IS NOT NULL as has_draft", - "draft_only", "app_version.raw_app", "app.labels", + "draft.path IS NOT NULL as is_draft", + // Per-path draft owners as a JSON array; see scripts.rs for the rationale + // (admins-workspace identity fallback, legacy NULL-email row). + // `app`/`raw_app` are separate draft kinds over one `app` table — match + // either (like the `is_draft` join below), else a deployed raw app's draft + // owners are dropped and the row shows "Draft" with no user badge. + "(SELECT json_agg(json_build_object('username', COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END)) ORDER BY COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) NULLS LAST) \ + FROM draft d \ + LEFT JOIN usr u ON u.workspace_id = d.workspace_id AND u.email = d.email \ + WHERE d.workspace_id = app.workspace_id AND d.path = app.path AND d.typ IN ('app', 'raw_app')) as draft_users", "folder_labels(app.workspace_id, app.path) as inherited_labels", ]) .left() @@ -400,15 +432,19 @@ async fn list_apps( .bind(&authed.username), ) .left() + // `app`/`raw_app` are separate draft kinds over one `app` table — match either + // for `is_draft`. DISTINCT in the subquery: a path with both kinds for the same + // user would otherwise fan the deployed row into two identical entries. + .join( + "(SELECT DISTINCT path, workspace_id FROM draft WHERE typ IN ('app', 'raw_app') AND email = ?) draft" + .bind(&authed.email), + ) + .on("draft.path = app.path AND draft.workspace_id = app.workspace_id") + .left() .join("app_version") .on( "app_version.id = versions[array_upper(versions, 1)]" ) - .left() - .join("draft") - .on( - "draft.path = app.path AND draft.workspace_id = app.workspace_id AND draft.typ = 'app'" - ) .order_desc("favorite.path IS NOT NULL") .order_by("app_version.created_at", true) .and_where("app.workspace_id = ?".bind(&w_id)) @@ -428,10 +464,6 @@ async fn list_apps( sqlb.and_where_eq("app.path", "?".bind(path_exact)); } - if !lq.include_draft_only.unwrap_or(false) || authed.is_operator { - sqlb.and_where("app.draft_only IS NOT TRUE"); - } - if let Some(label) = &lq.label { for l in label.split(',') { sqlb.and_where( @@ -451,17 +483,96 @@ async fn list_apps( let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; let mut tx = user_db.begin(&authed).await?; - let rows = sqlx::query_as::<_, ListableApp>(&sql) + let mut rows = sqlx::query_as::<_, ListableApp>(&sql) .fetch_all(&mut *tx) .await?; tx.commit().await?; + // Append the authed user's `app`/`raw_app` drafts at paths with no deployed app; + // see scripts.rs. + if lq.include_draft_only.unwrap_or(false) + && !authed.is_operator + && offset == 0 + && lq.path_start.is_none() + && lq.path_exact.is_none() + && lq.label.is_none() + && !lq.starred_only.unwrap_or(false) + { + // DISTINCT ON (path), newest first: collapse a path holding both `app` and + // `raw_app` drafts to one row (the home list keyed by `type/path` would crash + // on duplicates). `(email IS NULL)` last keeps the owned row over the legacy one. + let draft_only_rows = sqlx::query!( + r#"SELECT DISTINCT ON (path) + path, + value as "value!: sqlx::types::Json>", + created_at, + typ::text as "typ!" + FROM draft + WHERE workspace_id = $1 + AND typ IN ('app', 'raw_app') + AND (email = $2 OR email IS NULL) + AND NOT EXISTS ( + SELECT 1 FROM app a + WHERE a.workspace_id = draft.workspace_id + AND a.path = draft.path + ) + ORDER BY path, (email IS NULL), created_at DESC"#, + &w_id, + &authed.email, + ) + .fetch_all(&db) + .await?; + + for row in draft_only_rows { + let v: serde_json::Value = + serde_json::from_str(row.value.0.get()).unwrap_or(serde_json::Value::Null); + // App/raw-app drafts are the bare editor value with no `path`, so the editor + // writes a separate `draft_path` only when it differs from deployed; see flows.rs. + let draft_path = v + .get("draft_path") + .and_then(|s| s.as_str()) + .filter(|s| !s.is_empty() && *s != row.path.as_str()) + .map(|s| s.to_string()); + rows.push(ListableApp { + id: 0, + workspace_id: w_id.clone(), + path: row.path, + summary: v + .get("summary") + .and_then(|s| s.as_str()) + .unwrap_or("") + .to_string(), + version: 0, + extra_perms: serde_json::Value::Object(serde_json::Map::new()), + execution_mode: String::new(), + starred: false, + edited_at: Some(row.created_at), + draft_only: Some(true), + deployment_msg: None, + raw_app: row.typ == "raw_app", + labels: None, + // No deployed row to inherit folder labels from. + inherited_labels: None, + is_draft: true, + draft_path, + // Synthesized rows are the authed user's own draft. + draft_users: Some(sqlx::types::Json(vec![DraftUserRef { + username: Some(authed.username.clone()), + }])), + }); + } + } + + let allowed = build_scope_path_predicate(&authed, "apps", "read"); + rows.retain(|r| allowed(&r.path)); + Ok(Json(rows)) } async fn get_raw_app_data( Path((w_id, secret_with_ext)): Path<(String, String)>, + Query(query): Query>, Extension(db): Extension, ) -> Result { #[cfg(all(feature = "enterprise", feature = "parquet"))] @@ -484,13 +595,52 @@ async fn get_raw_app_data( .await?; let file_type = splitted.next().unwrap_or(""); + + // Sandboxed wrapper document that hosts the bundle. Served from a real URL + // (not blob:/srcdoc) so we can attach `CSP: sandbox` as a response header, + // which forces an opaque origin even on direct navigation — a raw-app + // bundle can then never reach the authenticated Windmill origin (WIN-2006). + // The `.js`/`.css` are loaded as same-path subresources by this document. + if file_type == "html" { + // ALWAYS served with `CSP: sandbox`, which forces an opaque origin even on + // direct top-level navigation — so this real-origin URL can never be used + // to run a raw-app bundle with the viewer's session (WIN-2006). The + // unsandboxed (default) render is NOT applied here: it is handled entirely + // on the viewer side, which builds its own same-origin wrapper. Relaxing + // this header from a policy flag would let anyone with the share secret + // hand a logged-in victim a same-origin URL that runs the bundle with + // their session — so the standalone document stays sandboxed no matter how + // it is reached. + let html = raw_app_wrapper_html(secret_id); + let mut builder = Response::builder() + .header(http::header::CONTENT_TYPE, "text/html; charset=utf-8") + .header("X-Content-Type-Options", "nosniff") + .header("Cross-Origin-Resource-Policy", "cross-origin") + .header( + http::header::CONTENT_SECURITY_POLICY, + "sandbox allow-scripts allow-forms allow-popups \ + allow-popups-to-escape-sandbox allow-downloads allow-modals \ + allow-top-navigation", + ); + // When the public app page is embedded in a cross-origin-isolated page + // (`wm_coep` opt-in, COEP `require-corp`), this nested wrapper document + // must itself assert COEP to be allowed to load. Opt-in only — COEP + // restricts the bundle's own subresources to CORP'd/same-origin ones + // (e.g. external images would break), so it must not be always-on. The + // viewer propagates the flag from the page URL (see RawAppPreview). + if query.contains_key("wm_coep") { + builder = builder.header("Cross-Origin-Embedder-Policy", "require-corp"); + } + return Ok(builder.body(Body::from(html)).unwrap()); + } + let file_type = if file_type == "css" { "css" } else if file_type == "js" { "js" } else { return Err(Error::BadRequest( - "Invalid file type, only .css and .js are supported".to_string(), + "Invalid file type, only .css, .js and .html are supported".to_string(), )); }; // tracing::info!("file_type: {}", file_type); @@ -541,20 +691,128 @@ async fn get_raw_app_data( if let Some(body) = body { // let stream = tokio_util::io::ReaderStream::new(file); - let res = Response::builder().header( - http::header::CONTENT_TYPE, - if file_type == "css" { - "text/css" - } else { - "text/javascript" - }, - ); + let res = Response::builder() + .header( + http::header::CONTENT_TYPE, + if file_type == "css" { + "text/css" + } else { + "text/javascript" + }, + ) + // nosniff + CORP so the bundle loads correctly as a subresource of + // the opaque, sandboxed wrapper (incl. under a cross-origin-isolated + // / COEP `require-corp` embedder). + .header("X-Content-Type-Options", "nosniff") + .header("Cross-Origin-Resource-Policy", "cross-origin"); Ok(res.body(body).unwrap()) } else { return Err(Error::NotFound("File not found".to_string())); } } +/// HTML wrapper that hosts a raw-app bundle inside a sandboxed, opaque-origin +/// iframe. Served by [`get_raw_app_data`] for the `.html` "file type". It loads +/// the bundle `.js`/`.css` as same-path subresources, shims web storage (which +/// an opaque origin disallows), and waits for the embedder to hand it the user +/// context via `postMessage` before evaluating the bundle — so the bundle never +/// receives a credential and `window.ctx` is set synchronously when it runs. +fn raw_app_wrapper_html(secret: &str) -> String { + const TEMPLATE: &str = r##" + + + +App + + + + +
+ + +"##; + TEMPLATE.replace("__SECRET__", secret) +} + // async fn get_app_version( // authed: ApiAuthed, // Extension(user_db): Extension, @@ -578,12 +836,25 @@ async fn get_raw_app_data( // Ok(Json(version)) // } +// Fields inlined rather than flattened (axum query bool quirk); see GetScriptByPathQuery in scripts.rs. +#[derive(Deserialize)] +struct GetAppQuery { + with_starred_info: Option, + #[serde(default)] + get_draft: bool, + /// Picks the draft kind for a draft-only lookup (`/apps_raw/...` → true). + /// Ignored when a deployed row exists — its own `raw_app` column wins. + #[serde(default)] + raw_app: Option, +} + async fn get_app( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, - Query(query): Query, -) -> JsonResult { + Query(query): Query, +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || format!("apps:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; @@ -597,9 +868,9 @@ async fn get_app( JOIN app_version ON app_version.id = app.versions[array_upper(app.versions, 1)] LEFT JOIN favorite - ON favorite.favorite_kind = 'app' - AND favorite.workspace_id = app.workspace_id - AND favorite.path = app.path + ON favorite.favorite_kind = 'app' + AND favorite.workspace_id = app.workspace_id + AND favorite.path = app.path AND favorite.usr = $3 WHERE app.path = $1 AND app.workspace_id = $2", ) @@ -623,8 +894,27 @@ async fn get_app( }; tx.commit().await?; - let app = not_found_if_none(app_o, "App", path)?; - Ok(Json(app)) + // No deployed row + `get_draft`: fall back to the draft table; see scripts.rs. + // Draft kind comes from the deployed row's `raw_app` flag, or for a draft-only + // path from the caller's `raw_app` query param. + let kind = match &app_o { + Some(app) if app.app.raw_app => UserDraftItemKind::RawApp, + Some(_) => UserDraftItemKind::App, + None if query.raw_app.unwrap_or(false) => UserDraftItemKind::RawApp, + None => UserDraftItemKind::App, + }; + let overlay = overlay_or_draft_only( + &db, + &w_id, + &authed.email, + kind, + path, + query.get_draft, + app_o, + || windmill_common::error::Error::NotFound(format!("App not found at path {path}")), + ) + .await?; + Ok(Json(overlay)) } async fn get_app_lite( @@ -655,55 +945,6 @@ async fn get_app_lite( Ok(Json(app)) } -async fn get_app_w_draft( - authed: ApiAuthed, - Extension(user_db): Extension, - Path((w_id, path)): Path<(String, StripPath)>, -) -> JsonResult { - let path = path.to_path(); - check_scopes(&authed, || format!("apps:read:{}", path))?; - let mut tx = user_db.begin(&authed).await?; - - let app_o = sqlx::query_as::<_, AppWithLastVersionAndDraft>( - r#" - SELECT - app.id, - app.path, - app.summary, - app.versions, - app.policy, - app.custom_path, - app.extra_perms, - app_version.value, - app_version.created_at, - app_version.created_by, - app.draft_only, - draft.value AS "draft", - draft.created_at AS "draft_created_at", - app_version.raw_app, - app.labels - FROM app - INNER JOIN app_version - ON app_version.id = app.versions[array_upper(app.versions, 1)] - LEFT JOIN draft - ON app.path = draft.path - AND draft.workspace_id = $2 - AND draft.typ = 'app' - WHERE app.path = $1 - AND app.workspace_id = $2 - "#, - ) - .bind(path.to_owned()) - .bind(&w_id) - .fetch_optional(&mut *tx) - .await?; - - tx.commit().await?; - - let app = not_found_if_none(app_o, "App", path)?; - Ok(Json(app)) -} - async fn get_app_history( authed: ApiAuthed, Extension(user_db): Extension, @@ -863,6 +1104,17 @@ async fn get_public_app_by_secret( let mut app = not_found_if_none(app_o, "App", id.to_string())?; + // Confine the app embed token (the only credential handed to untrusted app JS, + // carrying the viewer's identity + `apps:read:`) to the app the secret + // resolves to: without this, app JS could reuse the viewer's identity to read any + // app it can see by secret via the RLS check below. Scoped to embed tokens only — + // other callers (anonymous, cookie, plain external JWT) keep their existing access. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:read:{}", app.path))?; + } + } + let policy = serde_json::from_str::(app.policy.0.get()).map_err(to_anyhow)?; if !matches!(policy.execution_mode, ExecutionMode::Anonymous) { @@ -897,6 +1149,300 @@ async fn get_public_app_by_secret( Ok(Json(app)) } +/// Scopes granted to a short-lived "app embed token". This is the token the +/// app-embedder page hands the (opaque-origin) app iframe at startup so the app +/// never receives the viewer's session cookie. Instead of restricting which +/// routes a *domain* may hit, we restrict which routes the *token* may hit, so +/// that even a malicious or compromised app document can only reach the +/// endpoints an app legitimately needs. The `app_embed` sentinel turns each of +/// these into a strict route allowlist (`app_embed_route_denied`): +/// - `jobs:read` → by-id job poll/cancel only; enumeration, counts, exports, +/// and `job_signature`/`resume_urls` are denied, and by-id +/// reads are confined to the app's own runs. +/// - `app_embed` → sentinel tagging this as an app embed token (grants nothing). +/// - `resources:run` → resource metadata only (pickers, type schemas), never values. +/// - `users:read` → `users/whoami` only. +/// - `folders:read` → `folders/listnames` only. +/// Plus two path-scoped scopes minted per app (see `mint_app_embed_token`): +/// - `apps:read:` → the app's own definition (`apps/get/p/`); no +/// `apps:write`, so management routes are unreachable. +/// - `apps:run:` → run THIS app's components (`execute_component`, which +/// re-checks the path); `apps_u/*` public-serving routes. +pub const APP_EMBED_SCOPES: [&str; 5] = [ + "jobs:read", + windmill_api_auth::scopes::APP_EMBED_SENTINEL, + "resources:run", + "users:read", + "folders:read", +]; + +/// How long an app embed token stays valid. The embedder re-mints on demand +/// (e.g. after a `401` from the iframe) so this can stay short. +const APP_EMBED_TOKEN_VALIDITY_HOURS: i64 = 12; + +#[derive(Serialize)] +pub struct EmbedTokenResponse { + /// Narrowly-scoped token for the iframe. `None` for fully anonymous access + /// (the iframe then calls the public endpoints anonymously). + pub token: Option, + pub expiration: Option>, + /// WIN-2006: raw apps render single-iframe (the bundle is already isolated in + /// its own opaque iframe), so the viewer skips the opaque-viewer indirection + /// and the embed token entirely — it loads the app with the page credential. + #[serde(default)] + pub raw_app: bool, + /// WIN-2006: publisher opted this app into sandbox isolation. When false the + /// viewer runs the app same-origin with its full session (the default, + /// pre-isolation behavior). + #[serde(default)] + pub sandbox: bool, + /// WIN-2006: the resolved app path. The embedder uses it (together with + /// `workspace_id`) to scope the app's backing `localStorage` per app (so + /// sandboxed apps don't share one store). Not a new disclosure — the viewer + /// already receives `path` when it loads the app (e.g. `get_public_app_by_secret`). + #[serde(default, skip_serializing_if = "Option::is_none")] + pub app_path: Option, + /// WIN-2006: the resolved workspace. Pairs with `app_path` for the per-app + /// `localStorage` key so two apps at the same path in different workspaces don't + /// share a store. For custom-path apps the viewer can't derive this itself. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub workspace_id: Option, +} + +/// Mint a short-lived, narrowly-scoped embed token for `app_path` when a caller +/// is authenticated. When `opt_authed` is `None` (anonymous access to an +/// anonymous app) no token is minted and the iframe relies on the public +/// endpoints. +/// +/// The CALLER MUST verify the viewer's access to `app_path` before calling: this +/// mints a token on behalf of `opt_authed` unconditionally (DB access remains +/// gated by the viewer's own RLS, but the token's existence is not access-checked +/// here). All current call sites (`get_app_embed_token`, +/// `get_app_embed_token_for_path`, and the EE custom-path variant) do this. +/// +/// Scope confinement IS enforced here: the minted scopes must be within the +/// caller's own (`ensure_scopes_within_caller`), so a scope-restricted bearer +/// token cannot bootstrap a broader-scoped embed token. For the normal caller — +/// an unscoped browser session — this is a no-op and the mint is purely +/// narrowing. +pub async fn mint_app_embed_token( + db: &DB, + w_id: &str, + app_path: &str, + opt_authed: Option<&ApiAuthed>, +) -> Result { + let token_and_exp = if let Some(authed) = opt_authed { + // An app embed token represents untrusted app JS in the sandboxed iframe; it + // must never reach this mint path to renew itself. The 12h expiry is the + // blast-radius cap on a leaked embed token, and `ensure_scopes_within_caller` + // below would pass a same-scoped renewal (the requested scopes equal the + // caller's own), making the credential indefinitely self-renewable. Refresh + // minting is the trusted embedder session/JWT's job. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotAuthorized( + "App embed tokens cannot mint or renew embed tokens".to_string(), + )); + } + let expiration = + chrono::Utc::now() + chrono::Duration::hours(APP_EMBED_TOKEN_VALIDITY_HOURS); + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Path-scoped read so the app can fetch its OWN definition (apps/get/p, + // which the in-workspace sandboxed viewer uses) — but no other app's. The + // public viewer fetches via apps_u/public_app and doesn't rely on this. + scopes.push(format!("apps:read:{app_path}")); + // Path-scoped run (NOT unqualified `apps:run`) so the token can only execute + // THIS app's components: `execute_component` re-checks `apps:run:` for + // the requested app, so the token can't drive another app's runnables. + scopes.push(format!("apps:run:{app_path}")); + // A scope-restricted caller token must not bootstrap a broader-scoped + // embed token (`create_token_internal` deliberately does not check this + // itself). No-op for unscoped sessions — the normal embed flow. + ensure_scopes_within_caller(authed, Some(&scopes))?; + let token_config = NewToken::new( + Some(format!("embed_app:{app_path}")), + Some(expiration), + None, + Some(scopes), + Some(w_id.to_string()), + // Never let an embed token gain write capability the caller's own + // session lacks. + Some(authed.read_only), + ); + let mut tx = db.begin().await?; + let token = create_token_internal(&mut *tx, db, authed, token_config).await?; + tx.commit().await?; + Some((token, expiration)) + } else { + None + }; + + Ok(EmbedTokenResponse { + token: token_and_exp.as_ref().map(|(t, _)| t.clone()), + expiration: token_and_exp.map(|(_, e)| e), + raw_app: false, + sandbox: false, + app_path: Some(app_path.to_string()), + workspace_id: Some(w_id.to_string()), + }) +} + +/// Issue an embed token for a public app addressed by its (secret) share id. +/// Mirrors the access check in [`get_public_app_by_secret`]: anonymous apps are +/// reachable without auth, otherwise the caller must be logged in and have read +/// access to the app. +async fn get_app_embed_token( + OptAuthed(opt_authed): OptAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, secret)): Path<(String, String)>, +) -> JsonResult { + let id = get_id_from_secret(&db, &w_id, secret, None).await?; + + let app = sqlx::query!( + "SELECT a.path, a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.id = $1 AND a.workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + let app = not_found_if_none(app, "App", id.to_string())?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient field-level read instead of a strict `Policy` parse: a legacy app + // whose stored policy predates newer required fields must still resolve to + // its (unsandboxed) render here rather than erroring out of the viewer. + let policy = parse_embed_policy(&policy_str)?; + + let authed_for_token = if policy.anonymous_execution { + // Anonymous app: still mint a scoped token if the viewer happens to be + // logged in (so the app sees their identity), otherwise stay anonymous. + opt_authed + } else { + let authed = opt_authed.ok_or_else(|| { + Error::NotAuthorized( + "App visibility does not allow public access and you are not logged in".to_string(), + ) + })?; + let mut tx = user_db.begin(&authed).await?; + let is_visible = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM app WHERE id = $1 AND workspace_id = $2)", + id, + &w_id + ) + .fetch_one(&mut *tx) + .await?; + tx.commit().await?; + if !is_visible.unwrap_or(false) { + return Err(Error::NotAuthorized( + "App visibility does not allow public access and you are logged in but you have no read-access to that app".to_string(), + )); + } + Some(authed) + }; + + // The token is only consumed by the sandboxed low-code render. Raw apps + // render single-iframe with the page credential (WIN-2006 Variant A), and + // unsandboxed apps render same-origin with the viewer's own session — minting + // for those would write a useless token row per view and, worse, could fail + // the whole render for a scope-restricted caller (`ensure_scopes_within_caller`) + // even though no token is needed. The access check above still gates + // visibility in every case. + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, &app.path, authed_for_token.as_ref()).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(app.path); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + +/// Minimal, lenient view of an app policy for the embed-token endpoints +/// (WIN-2006). Reads only the fields the sandbox decision needs, via +/// `serde_json::Value`, so a legacy policy that no longer satisfies the strict +/// [`Policy`] struct (e.g. `triggerables_v2` entries predating now-required +/// fields) still renders instead of failing the viewer with "Not found". +/// A missing/unknown `execution_mode` is treated as NOT anonymous — the +/// strictest access interpretation. +pub struct EmbedPolicyView { + pub anonymous_execution: bool, + pub sandbox: bool, +} + +pub fn parse_embed_policy(policy_str: &str) -> Result { + let v: serde_json::Value = serde_json::from_str(policy_str).map_err(to_anyhow)?; + Ok(EmbedPolicyView { + anonymous_execution: v.get("execution_mode").and_then(|m| m.as_str()) == Some("anonymous"), + sandbox: v.get("sandbox").and_then(|b| b.as_bool()).unwrap_or(false), + }) +} + +/// Authenticated, path-based embed token for the in-workspace app viewer +/// (WIN-2006). Mirrors [`get_app_embed_token`] but keyed by app path and gated by +/// the caller's read access (RLS), so the logged-in `/apps/get` viewer can render +/// the app sandboxed — isolated from the member's full session — using the same +/// scoped token. Raw apps get no token (single-iframe with the page credential). +async fn get_app_embed_token_for_path( + authed: ApiAuthed, + Extension(user_db): Extension, + Extension(db): Extension, + Path((w_id, path)): Path<(String, StripPath)>, +) -> JsonResult { + let path = path.to_path(); + check_scopes(&authed, || format!("apps:read:{}", path))?; + // RLS: the caller must have read access to this app, otherwise it's not found. + let mut tx = user_db.begin(&authed).await?; + let app = sqlx::query!( + "SELECT a.policy::text as policy, a.versions[array_upper(a.versions, 1)] as version, av.raw_app as raw_app + FROM app a JOIN app_version av ON av.id = a.versions[array_upper(a.versions, 1)] + WHERE a.path = $1 AND a.workspace_id = $2", + path, + &w_id + ) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + let app = not_found_if_none(app, "App", path)?; + let raw_app = app.raw_app; + let policy_str = app + .policy + .ok_or_else(|| Error::internal_err("App policy missing".to_string()))?; + // Lenient parse + mint only for the sandboxed low-code render — see + // [`get_app_embed_token`] for the rationale (identical here). + let policy = parse_embed_policy(&policy_str)?; + + let mut resp = if raw_app || !policy.sandbox { + EmbedTokenResponse { + token: None, + expiration: None, + raw_app, + sandbox: policy.sandbox, + app_path: None, + workspace_id: None, + } + } else { + mint_app_embed_token(&db, &w_id, path, Some(&authed)).await? + }; + resp.raw_app = raw_app; + resp.sandbox = policy.sandbox; + resp.app_path = Some(path.to_string()); + resp.workspace_id = Some(w_id.to_string()); + Ok(Json(resp)) +} + async fn get_id_from_secret( db: &DB, w_id: &str, @@ -1171,8 +1717,6 @@ async fn create_app_raw<'a>( ) .await?; - check_scopes(&authed, || format!("apps:write:{}", path))?; - webhook.send_message( w_id.clone(), WebhookMessage::CreateApp { workspace: w_id, path: path.clone() }, @@ -1216,7 +1760,6 @@ async fn create_app( )); } let path = app.path.clone(); - check_scopes(&authed, || format!("apps:write:{}", &path))?; if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, @@ -1230,6 +1773,7 @@ async fn create_app( return Err(Error::PermissionDenied(msg)); } + // scope is enforced inside create_app_internal, before any persistence. let (new_tx, _path, _id) = create_app_internal(authed, db, user_db, &w_id, false, app).await?; new_tx.commit().await?; @@ -1276,6 +1820,10 @@ async fn create_app_internal<'a>( raw_app: bool, mut app: CreateApp, ) -> Result<(sqlx::Transaction<'a, sqlx::Postgres>, String, i64)> { + // Enforce scope before any persistence: the raw-app create path commits + // inside process_app_multipart!, so checking after this call would leave a + // denied app committed in the DB. + check_scopes(&authed, || format!("apps:write:{}", &app.path))?; if *CLOUD_HOSTED { let nb_apps = sqlx::query_scalar!("SELECT COUNT(*) FROM app WHERE workspace_id = $1", &w_id) @@ -1293,7 +1841,9 @@ async fn create_app_internal<'a>( )); } } - let mut tx = user_db.clone().begin(&authed).await?; + // Resolve the on-behalf-of defaults on the (non-RLS) pool *before* opening + // the RLS transaction below: doing these lookups mid-transaction would hold + // a second simultaneous connection while `tx` is still checked out. let should_preserve = app.preserve_on_behalf_of.unwrap_or(false) && windmill_common::can_preserve_on_behalf_of(&authed) && app.policy.on_behalf_of.is_some(); @@ -1319,6 +1869,8 @@ async fn create_app_internal<'a>( app.policy.on_behalf_of_email = Some(authed.email.clone()); } } + + let mut tx = user_db.clone().begin(&authed).await?; let path = app.path.clone(); if &app.path == "" { return Err(Error::BadRequest("App path cannot be empty".to_string())); @@ -1374,25 +1926,27 @@ async fn create_app_internal<'a>( } } // CLI / git-sync deploys ask us to preserve any existing user draft at this - // path instead of wiping it as part of the deploy. + // path instead of wiping it as part of the deploy. Only wipe the deployer's + // own draft (plus the legacy NULL-email row); see scripts.rs. if !app.skip_draft_deletion.unwrap_or(false) { sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ IN ('app', 'raw_app') \ + AND (email = $3 OR email IS NULL)", &app.path, - &w_id + &w_id, + &authed.email, ) .execute(&mut *tx) .await?; } let id = sqlx::query_scalar!( "INSERT INTO app - (workspace_id, path, summary, policy, versions, draft_only, custom_path, labels) - VALUES ($1, $2, $3, $4, '{}', $5, $6, $7) RETURNING id", + (workspace_id, path, summary, policy, versions, custom_path, labels) + VALUES ($1, $2, $3, $4, '{}', $5, $6) RETURNING id", w_id, app.path, app.summary, json!(app.policy), - app.draft_only, app.custom_path .as_ref() .map(|s| if s.is_empty() { None } else { Some(s) }) @@ -1598,15 +2152,16 @@ async fn delete_app( .await?; let trash_drafts: Vec = sqlx::query_scalar( - "SELECT to_jsonb(t) FROM draft t WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", + "SELECT to_jsonb(t) FROM draft t WHERE path = $1 AND workspace_id = $2 AND typ IN ('app', 'raw_app')", ) .bind(path) .bind(&w_id) .fetch_all(&mut *tx) .await?; + // Both `app` and `raw_app` draft kinds back the same `app` table. sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ IN ('app', 'raw_app')", path, &w_id ) @@ -1814,6 +2369,13 @@ async fn update_app_internal<'a>( ns: EditApp, ) -> Result<(sqlx::Transaction<'a, sqlx::Postgres>, String, i64)> { use sql_builder::prelude::*; + + // A rename moves the app to ns.path, so the destination must also be within + // the token's write scope, not just the source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("apps:write:{}", npath))?; + } + let mut tx = user_db.clone().begin(&authed).await?; let mut preserved_on_behalf_of: Option = None; @@ -1827,7 +2389,6 @@ async fn update_app_internal<'a>( sqlb.and_where_eq("path", "?".bind(&path)); sqlb.and_where_eq("workspace_id", "?".bind(&w_id)); - sqlb.set("draft_only", "NULL"); if let Some(npath) = &ns.path { if npath != path { require_owner_of_path(&authed, path)?; @@ -2014,12 +2575,15 @@ async fn update_app_internal<'a>( } }; // CLI / git-sync deploys ask us to preserve any existing user draft at this - // path instead of wiping it as part of the deploy. + // path instead of wiping it as part of the deploy. Only wipe the deployer's + // own draft (plus the legacy NULL-email row) — see create_app_internal. if !ns.skip_draft_deletion.unwrap_or(false) { sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", + "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ IN ('app', 'raw_app') \ + AND (email = $3 OR email IS NULL)", path, - &w_id + &w_id, + &authed.email, ) .execute(&mut *tx) .await?; @@ -2197,6 +2761,17 @@ async fn execute_component( Path((w_id, path)): Path<(String, StripPath)>, Json(mut payload): Json, ) -> Result { + let path = path.to_path(); + // Authorize FIRST, before touching the payload: confine the app embed token (the + // only credential handed to untrusted app JS, carrying `apps:run:`) to + // the app it was minted for. The route layer can't path-check the apps domain, so + // enforce it here. Scoped to embed tokens only — other callers (anonymous, cookie, + // plain external JWT) keep their existing access; the run is still policy-gated. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path))?; + } + } // Only honor temp_script_refs for the inline-script preview path: // preview/editor mode (force_viewer_static_fields set, == `is_preview`), // raw_code present, and no deployed app_script id — i.e. `wmill app dev`. @@ -2219,7 +2794,6 @@ async fn execute_component( _ => {} }; - let path = path.to_path(); let (arc_policy, policy): (Arc, Policy); let policy_triggerables_default = Default::default(); // Preview mode means the request was issued from the editor; the editing @@ -2333,6 +2907,15 @@ async fn execute_component( &policy }; + // Caller-supplied inline code (`raw_code`), with or without an + // `app_script` id. Its resolved `rawscript/` key must be present + // in the policy triggerables below — it must never resolve via the + // Viewer default fallback. Without `id` the caller supplies the code + // verbatim; with `id` it selects any `app_script` row by number (no + // app/workspace scoping), so both let a caller run code the publisher + // never pinned for this app. + let is_inline_raw_code = payload.raw_code.is_some(); + // Compute the path for the triggerables map: // - flow: `flow/` // - script: `script/` @@ -2370,7 +2953,15 @@ async fn execute_component( .get(path) // start with `path` in case we can avoid the next` format!`. .or_else(|| triggerables_v2.get(&format!("{}:{}", payload.component, &path))) .or(match policy.execution_mode { - ExecutionMode::Viewer => Some(&policy_triggerables_default), + // A Viewer app may invoke any deployed `script`/`flow` it + // references (resolved as the caller), but caller-supplied + // inline `raw_code` must match a publisher-pinned + // `rawscript/` entry — otherwise an unauthorized caller + // (e.g. an operator, barred from `/jobs/run/preview`) could + // run code the publisher never pinned for this app. + ExecutionMode::Viewer if !is_inline_raw_code => { + Some(&policy_triggerables_default) + } _ => None, }) .ok_or_else(|| Error::BadRequest(format!("Path {path} forbidden by policy")))?; @@ -2666,6 +3257,15 @@ async fn upload_s3_file_from_app( Query(query): Query, request: axum::extract::Request, ) -> JsonResult { + // Confine an app embed token (untrusted app JS) to uploading for its OWN app. + // The route is reachable with `apps:run` (RUN_PATH_ACTIONS), so without this a + // token minted for app A could drive app B's upload policy. Mirrors + // execute_component / download_s3_file; other callers are unaffected. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + check_scopes(authed, || format!("apps:run:{}", path.to_path()))?; + } + } let policy = if let Some(file_key_regex) = query.force_viewer_file_key_regex { // `force_viewer_*` lets the caller supply a synthetic upload policy that // bypasses the deployed app's file_key_regex / resource restrictions. @@ -2700,6 +3300,7 @@ async fn upload_s3_file_from_app( .unwrap_or_default(), }]), allowed_s3_keys: None, + sandbox: None, }) } else { let policy_o = sqlx::query_scalar!( @@ -3059,6 +3660,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: Some(force_allowed_s3_keys), + sandbox: None, } } else { // TODO: improve db query to not return uneeded fields @@ -3081,6 +3683,7 @@ async fn get_on_behalf_authed_from_app( on_behalf_of_email: None, s3_inputs: None, allowed_s3_keys: None, + sandbox: None, }) }; @@ -3124,34 +3727,46 @@ async fn check_if_allowed_to_access_s3_file_from_app( return Err(Error::InternalErr( "Internal error: signature validation is not supported in open source mode".to_string(), )); - } else if opt_authed.is_some() { + } else if opt_authed.as_ref().is_some_and(|authed| { + !windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) + }) { + // A normal logged-in caller (editor / full session) may fetch any file they + // can reach. An app embed token also carries an identity but represents + // untrusted app JS, so it falls through to the allowlist below instead of + // this bypass — otherwise the app could read arbitrary S3 keys the + // viewer/on-behalf identity can see, beyond its own declared keys/outputs. Ok(()) } else { - let allowed = policy - .allowed_s3_keys + // Anonymous viewer, or an app embed token: confine to the app's declared S3 + // keys, or files produced by THIS app's own component runs. The producing + // identity is the embed viewer for a token, else `anonymous`. + let creator = opt_authed .as_ref() - .unwrap() - .iter() - .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) - || { - sqlx::query_scalar!( - r#"SELECT EXISTS ( + .map(|authed| authed.username.clone()) + .unwrap_or_else(|| "anonymous".to_string()); + let allowed = policy.allowed_s3_keys.as_ref().is_some_and(|keys| { + keys.iter() + .any(|key| key.s3_path == file_query.s3 && key.storage == file_query.storage) + }) || { + sqlx::query_scalar!( + r#"SELECT EXISTS ( SELECT 1 FROM v2_job_completed c JOIN v2_job j USING (id) WHERE j.workspace_id = $2 AND (j.kind = 'appscript' OR j.kind = 'preview') - AND j.created_by = 'anonymous' + AND j.created_by = $4 AND c.started_at > now() - interval '3 hours' AND j.runnable_path LIKE $3 || '/%' AND c.result @> ('{"s3":"' || $1 || '"}')::jsonb )"#, - file_query.s3, - w_id, - path, - ) - .fetch_one(db) - .await? - .unwrap_or(false) - }; + file_query.s3, + w_id, + path, + creator, + ) + .fetch_one(db) + .await? + .unwrap_or(false) + }; if !allowed { Err(Error::BadRequest("File restricted".to_string())) @@ -3190,6 +3805,15 @@ async fn download_s3_file_from_app( let path = path.to_path(); + // Authorize the app path first: a scoped caller (notably an app embed token, + // which carries `apps:read:`) may only download files for the app it + // was minted for — otherwise it could read another app's S3 files via that app's + // on-behalf policy. Unscoped sessions / anonymous callers pass through (the + // latter still gated by the policy allowlist in `check_if_allowed_...`). + if let Some(authed) = opt_authed.as_ref() { + check_scopes(authed, || format!("apps:read:{}", path))?; + } + let force_viewer_allowed_s3_keys = if let Some(force_viewer_allowed_s3_keys) = query.force_viewer_allowed_s3_keys.clone() { @@ -3264,18 +3888,6 @@ fn get_on_behalf_of(policy: &Policy) -> Result<(String, String)> { Ok((permissioned_as, email)) } -pub async fn require_is_writer(authed: &ApiAuthed, path: &str, w_id: &str, db: DB) -> Result<()> { - return crate::users::require_is_writer( - authed, - path, - w_id, - db, - "SELECT extra_perms FROM app WHERE path = $1 AND workspace_id = $2", - "app", - ) - .await; -} - async fn exists_app( Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, @@ -3478,3 +4090,253 @@ async fn build_args( job_id, )) } + +#[cfg(test)] +mod embed_token_tests { + use super::APP_EMBED_SCOPES; + use windmill_api_auth::scopes::check_scopes_for_route; + + /// The embed token must reach exactly the endpoints an app needs and nothing + /// else. This locks the allow/deny matrix that confines a malicious or + /// compromised app to app-only routes (WIN-2006). + #[test] + fn embed_scopes_allow_app_routes_and_deny_the_rest() { + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // Mirror mint_app_embed_token: the per-app path-scoped read + run. + scopes.push("apps:read:u/admin/app".to_string()); + scopes.push("apps:run:u/admin/app".to_string()); + let scopes = Some(scopes.as_slice()); + + // Allowed: the routes a running app legitimately calls. + let allowed = [ + // Own definition + the public app-serving / execution endpoints. + ("/api/w/test/apps/get/p/u/admin/app", "GET"), + ("/api/w/test/apps_u/public_app/secret", "GET"), + ("/api/w/test/apps_u/get_data/v/secret.js", "GET"), + ("/api/w/test/apps_u/public_resource/f/app_themes/t", "GET"), + ("/api/w/test/apps_u/execute_component/u/admin/app", "POST"), + // S3 file upload from the app's S3 File Input component: a `run` action + // (RUN_PATH_ACTIONS) so the embed token reaches it; the handler re-checks + // `apps:run:` to confine it to this app, like execute_component. + ("/api/w/test/apps_u/upload_s3_file/u/admin/app", "POST"), + // By-id job poll routes (the JobLoader surface) stay allowed. + ("/api/w/test/jobs_u/get/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate/some-uuid", "GET"), + ("/api/w/test/jobs_u/getupdate_sse/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_result/some-uuid", "GET"), + ("/api/w/test/jobs_u/completed/get_timing/some-uuid", "GET"), + // By-id cancel (POST): permitted at the route layer; the handler confines + // it to the app's own jobs (created_by == viewer). + ("/api/w/test/jobs_u/queue/cancel/some-uuid", "POST"), + ("/api/w/test/users/whoami", "GET"), + // Resource METADATA only (picker list + type schemas) — never values. + ("/api/w/test/resources/list", "GET"), + ("/api/w/test/resources/exists/u/admin/r", "GET"), + ("/api/w/test/resources/type/list", "GET"), + ("/api/w/test/folders/listnames", "GET"), + ]; + for (path, method) in allowed { + assert!( + check_scopes_for_route(scopes, path, method).is_ok(), + "embed token should allow {method} {path}" + ); + } + + // Denied: anything outside what an app needs, including app management + // (apps:write is intentionally withheld), resource VALUE reads (which can + // hold credentials), and other workspace domains. + let denied = [ + ("/api/w/test/apps/update/u/admin/app", "POST"), + ("/api/w/test/apps/delete/u/admin/app", "DELETE"), + // Workspace app inventory must NOT be reachable (Apps domain is + // default-denied for the embed sentinel; only own-def + apps_u/* allowed). + ("/api/w/test/apps/exists/u/admin/app", "GET"), + ("/api/w/test/apps/custom_path_exists/foo", "GET"), + ( + "/api/w/test/apps/list_paths_from_workspace_runnable/script/u/admin/x", + "GET", + ), + ("/api/w/test/apps/list", "GET"), + // The embed-token MINT endpoints are public app routes (`apps_u/`) but + // create credentials — denied so a captured embed token can't renew + // itself indefinitely past the 12h expiry (refresh is the embedder's job). + ("/api/w/test/apps_u/embed_token/secret", "GET"), + ("/api/w/test/apps_u/embed_token_by_custom_path/foo", "GET"), + ("/api/w/test/scripts/list", "GET"), + ("/api/w/test/variables/list", "GET"), + ("/api/w/test/resources/update/u/admin/r", "POST"), + // Resource value reads must NOT be reachable with the embed token. + ("/api/w/test/resources/get/u/admin/r", "GET"), + ("/api/w/test/resources/get_value/u/admin/r", "GET"), + ( + "/api/w/test/resources/get_value_interpolated/u/admin/r", + "GET", + ), + ("/api/w/test/resources/list_search", "GET"), + // Workspace-wide job enumeration/export must NOT be reachable — an app + // reads only jobs it launched, by id (blocked via the app_embed sentinel). + ("/api/w/test/jobs/list", "GET"), + ("/api/w/test/jobs/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/completed/list", "GET"), + ("/api/w/test/jobs/completed/export", "GET"), + ("/api/w/test/jobs/queue/list", "GET"), + ("/api/w/test/jobs/queue/list_filtered_uuids", "GET"), + ("/api/w/test/jobs/queue/export", "GET"), + // Job counts (workspace-wide aggregates) and the capability-minting + // routes (signed resume/approval URLs) are NOT by-id polling — denied. + ("/api/w/test/jobs/completed/count", "GET"), + ("/api/w/test/jobs/completed/count_jobs", "GET"), + ("/api/w/test/jobs/queue/count", "GET"), + ("/api/w/test/jobs/job_signature/some-uuid/some-rid", "GET"), + ("/api/w/test/jobs/resume_urls/some-uuid/some-rid", "GET"), + // get_root_job_id has no access check in its handler and the app never + // calls it — denied so the token can't probe foreign jobs' flow lineage. + ("/api/w/test/jobs_u/get_root_job_id/some-uuid", "GET"), + // `users:read`/`folders:read` exist only for whoami/listnames — every + // other route in those domains is denied via the app_embed sentinel + // (the whole /users and /folders routers are CORS-enabled for the iframe). + ("/api/w/test/users/list", "GET"), + ("/api/w/test/users/list_usage", "GET"), + ("/api/w/test/users/username_to_email/admin", "GET"), + ("/api/w/test/folders/list", "GET"), + ("/api/w/test/folders/get/myfolder", "GET"), + ("/api/w/test/folders/getusage/myfolder", "GET"), + ]; + for (path, method) in denied { + assert!( + check_scopes_for_route(scopes, path, method).is_err(), + "embed token should deny {method} {path}" + ); + } + } + + /// `apps:run` satisfies read at the route layer, so `apps/list` / `apps/list_search` + /// pass the route check — that's why those handlers ALSO call + /// `check_scopes(apps:read)`, which uses `ScopeDefinition::includes` (where run + /// does NOT include read). Lock that: no embed scope, including the + /// dynamically-minted path-scoped read, satisfies a domain-level `apps:read`, so + /// the token cannot list all apps' definitions (their full `value`/code). + #[test] + fn embed_scopes_cannot_satisfy_domain_app_read() { + use windmill_api_auth::scopes::ScopeDefinition; + let mut scopes: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + // mint_app_embed_token also grants read scoped to the single app path: + scopes.push("apps:read:u/admin/app".to_string()); + let required = ScopeDefinition::from_scope_string("apps:read").unwrap(); + for s in &scopes { + // The `app_embed` sentinel intentionally doesn't parse as a domain:action + // scope (it grants nothing; it only drives the job-enumeration deny). + let Ok(def) = ScopeDefinition::from_scope_string(s) else { + continue; + }; + assert!( + !def.includes(&required), + "embed scope {s} must not satisfy domain-level apps:read (would leak apps/list[_search])" + ); + } + // Sanity: a genuine domain-level apps:read token does satisfy it. + assert!(ScopeDefinition::from_scope_string("apps:read") + .unwrap() + .includes(&required)); + } + + /// The token carries path-scoped `apps:run:` and `apps:read:` + /// (NOT unqualified `apps:run`). Every handler that resolves an app and acts on + /// its behalf re-checks the requested path via `ScopeDefinition::includes`, so the + /// token is confined to its OWN app: + /// - `apps:run:` — `execute_component`. + /// - `apps:read:` — `get_app` (apps/get/p), `get_public_app_by_secret`, + /// the EE custom-path `get_public_app_by_custom_path`, and + /// `download_s3_file_from_app`. + /// This blocks cross-app execution, definition reads (by secret / custom path), + /// and S3 file reads through another app's on-behalf policy. + #[test] + fn embed_run_scope_is_path_scoped_to_its_app() { + use windmill_api_auth::scopes::ScopeDefinition; + // The mint must not grant unqualified run (which would include any path). + assert!( + !APP_EMBED_SCOPES.contains(&"apps:run"), + "embed scopes must not include unqualified apps:run" + ); + for action in ["run", "read"] { + let own = + ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")).unwrap(); + assert!( + own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/app")) + .unwrap() + ), + "apps:{action} must grant its own app" + ); + assert!( + !own.includes( + &ScopeDefinition::from_scope_string(&format!("apps:{action}:u/admin/other")) + .unwrap() + ), + "apps:{action} must NOT grant another app (cross-app)" + ); + } + } + + /// `mint_app_embed_token` guards its `create_token_internal` call with + /// `ensure_scopes_within_caller`, so a scope-restricted bearer token cannot + /// bootstrap the broader embed-scope set. Lock that boundary on the exact + /// scope vec the mint builds: rejected for a path-scoped caller, no-op for + /// the unscoped browser session that is the normal embed flow. + #[test] + fn embed_token_mint_is_scope_bounded() { + use windmill_api_auth::{ensure_scopes_within_caller, ApiAuthed}; + + // Same scope set mint_app_embed_token assembles for an app. + let mut minted: Vec = APP_EMBED_SCOPES.iter().map(|s| s.to_string()).collect(); + minted.push("apps:read:u/admin/app".to_string()); + + // A caller restricted to a single app read must not widen to the full + // embed set (apps:run, jobs:read, resources:read, ...). + let restricted = ApiAuthed { + scopes: Some(vec!["apps:read:u/admin/app".to_string()]), + ..Default::default() + }; + assert!( + ensure_scopes_within_caller(&restricted, Some(&minted)).is_err(), + "a path-scoped caller must not mint the broader embed-scope set" + ); + + // An unscoped session (the normal embed flow) passes — the mint only + // narrows. + let unscoped = ApiAuthed { scopes: None, ..Default::default() }; + assert!(ensure_scopes_within_caller(&unscoped, Some(&minted)).is_ok()); + } + + /// The embed-token endpoints must keep working for legacy apps whose stored + /// policy no longer satisfies the strict `Policy` struct (pre-dating + /// now-required fields): `parse_embed_policy` reads only the sandbox-decision + /// fields, leniently, and treats a missing/unknown `execution_mode` as NOT + /// anonymous (the strictest access interpretation). + #[test] + fn embed_policy_parse_is_lenient() { + use super::parse_embed_policy; + + // Quirky legacy policy: triggerables_v2 entry missing required fields, + // no execution_mode at all — must still parse, and absent `sandbox` + // resolves to the unsandboxed default. + let p = parse_embed_policy(r#"{"triggerables_v2": {"x": {}}}"#).unwrap(); + assert!(!p.sandbox); + assert!( + !p.anonymous_execution, + "missing execution_mode must not grant anonymous access" + ); + + // Normal policies map field-for-field. + let p = parse_embed_policy(r#"{"execution_mode": "anonymous", "sandbox": true}"#).unwrap(); + assert!(p.anonymous_execution); + assert!(p.sandbox); + + // Unknown execution_mode value: lenient parse, but not anonymous. + let p = parse_embed_policy(r#"{"execution_mode": "weird"}"#).unwrap(); + assert!(!p.anonymous_execution); + + // Invalid JSON still errors. + assert!(parse_embed_policy("not json").is_err()); + } +} diff --git a/backend/windmill-api/src/db.rs b/backend/windmill-api/src/db.rs index c8ed841e19..b81a9dbb84 100644 --- a/backend/windmill-api/src/db.rs +++ b/backend/windmill-api/src/db.rs @@ -84,6 +84,12 @@ lazy_static::lazy_static! { (20260228000000, include_str!( "../../migrations/20260228000000_v2_job_completed_failure_index.up.sql" ).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")), + (20260610151334, include_str!( + "../../migrations/20260610151334_folder_labels.up.sql" + ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()), + (20260614075900, include_str!( + "../../migrations/20260614075900_dedup_folder_labels.up.sql" + ).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()), ].into_iter().collect(); } @@ -282,6 +288,18 @@ pub async fn migrate( 20260207000002, 20260207000003, 20260207000004, + // Squashed pre-release pipeline migrations: the per-column ALTERs on + // script_trigger and the dispatch_event subscriber index were folded + // back into these two CREATEs, changing their checksum. Both are + // idempotent, so re-applying on an already-migrated DB is a no-op. + 20260423050000, + 20260523055641, + // Reworked to stop reading pg_authid (via pg_has_role) from an elevated + // context, which managed providers (e.g. Cloud SQL) forbid — the original + // aborted startup. The new file is idempotent (CREATE OR REPLACE + an + // epoch-guarded UPDATE that no-ops once anchored), so re-applying on an + // already-migrated DB is safe. + 20260626132251, ]; for m in migrator.migrations.iter() { if m.migration_type.is_down_migration() { diff --git a/backend/windmill-api/src/docs/corpus.rs b/backend/windmill-api/src/docs/corpus.rs new file mode 100644 index 0000000000..a797879cfe --- /dev/null +++ b/backend/windmill-api/src/docs/corpus.rs @@ -0,0 +1,86 @@ +//! The vendored documentation snapshot, embedded into the binary and parsed once. +//! +//! `docs_snapshot/*.gz` are refreshed by `docs_snapshot/fetch.sh`. Embedding them +//! lets docs search work with no runtime egress (including air-gapped instances). + +use std::io::Read; +use std::sync::OnceLock; + +use flate2::read::GzDecoder; + +use super::search::{ + canonical_docs_page_url, canonical_search_url, parse_docs_full_text, parse_docs_index, + DocsFullPage, DocsIndexEntry, +}; + +const LLMS_FULL_GZ: &[u8] = + include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs_snapshot/llms-full.txt.gz")); +const LLMS_INDEX_GZ: &[u8] = + include_bytes!(concat!(env!("CARGO_MANIFEST_DIR"), "/docs_snapshot/llms.txt.gz")); + +pub struct DocsCorpus { + /// Every docs page, keyed by its `Source:` URL (from llms-full.txt). + pub pages: Vec, + /// The curated page index with one-line descriptions (from llms.txt). + pub index: Vec, +} + +impl DocsCorpus { + /// Finds the page whose `Source:` URL matches a model/CLI-supplied path or URL + /// after canonicalization (origin re-anchored, `.md` and ordering prefixes + /// stripped). + pub fn find_page(&self, path: &str) -> Option<&DocsFullPage> { + let key = canonical_search_url(&canonical_docs_page_url(path)); + self.pages.iter().find(|p| canonical_search_url(&p.url) == key) + } +} + +static CORPUS: OnceLock = OnceLock::new(); + +fn decompress(bytes: &[u8]) -> String { + let mut out = String::new(); + if let Err(e) = GzDecoder::new(bytes).read_to_string(&mut out) { + // The embedded snapshot is valid gzip text; a decode failure is a + // build-time packaging error, so failing closed to an empty corpus + // (docs search returns "no matches") is acceptable. + tracing::error!("failed to decompress embedded docs snapshot: {e}"); + return String::new(); + } + out +} + +/// Returns the parsed docs corpus, decompressing and parsing the embedded +/// snapshot once on first access. +pub fn corpus() -> &'static DocsCorpus { + CORPUS.get_or_init(|| { + let pages = parse_docs_full_text(&decompress(LLMS_FULL_GZ)); + let index = parse_docs_index(&decompress(LLMS_INDEX_GZ)); + DocsCorpus { pages, index } + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn embedded_corpus_parses_to_non_empty() { + let corpus = corpus(); + assert!(corpus.pages.len() > 50, "expected many pages, got {}", corpus.pages.len()); + assert!(corpus.index.len() > 50, "expected many index entries, got {}", corpus.index.len()); + assert!(corpus + .pages + .iter() + .all(|p| p.url.starts_with("https://www.windmill.dev/docs/") && !p.body.is_empty())); + } + + #[test] + fn find_page_matches_by_canonical_url() { + let corpus = corpus(); + // A page that is expected to exist in the published docs. + let by_path = corpus.find_page("/docs/core_concepts/worker_groups"); + let by_url = corpus.find_page("https://www.windmill.dev/docs/core_concepts/worker_groups.md"); + assert!(by_path.is_some()); + assert_eq!(by_path.map(|p| &p.url), by_url.map(|p| &p.url)); + } +} diff --git a/backend/windmill-api/src/docs/mod.rs b/backend/windmill-api/src/docs/mod.rs new file mode 100644 index 0000000000..cd903e593f --- /dev/null +++ b/backend/windmill-api/src/docs/mod.rs @@ -0,0 +1,124 @@ +//! Self-hosted documentation search. +//! +//! The backend embeds a vendored docs snapshot (see [`corpus`]) and exposes two +//! read-only endpoints over it, so docs search works with no runtime egress: +//! - `GET /api/docs/search?query=...` — full-text + index search +//! - `GET /api/docs/page?url=...§ion=...` — read one page (or a section) +//! +//! These back the AI chat `search_docs`/`read_docs_page` tools, the MCP +//! `searchDocs`/`readDocsPage` tools, and the `wmill docs` CLI. The routes are +//! nested behind the global authed service in `lib.rs`, so a valid token is +//! required but no workspace. + +mod corpus; +mod search; + +use axum::{extract::Query, routing::get, Json, Router}; +use serde::{Deserialize, Serialize}; +use windmill_common::error::JsonResult; + +use search::DocsSearchResult; + +pub fn global_service() -> Router { + Router::new() + .route("/search", get(search_docs)) + .route("/page", get(read_docs_page)) +} + +#[derive(Deserialize)] +struct SearchQuery { + query: String, +} + +#[derive(Serialize)] +struct SearchResponse { + /// Model-ready rendering of the results (the exact string the AI/MCP tool + /// returns). Built once here so every consumer is identical. + text: String, + /// Structured results for non-AI consumers (e.g. the CLI's pretty/`--json`). + results: Vec, +} + +#[derive(Deserialize)] +struct PageQuery { + /// A page's `Source` URL (as returned by the docs search tool); a bare `/docs/...` + /// path is also accepted and canonicalized before lookup. + url: String, + section: Option, +} + +#[derive(Serialize)] +struct PageResponse { + text: String, + source_url: String, +} + +async fn search_docs(Query(q): Query) -> JsonResult { + let query = q.query.trim().to_string(); + if query.is_empty() { + return Ok(Json(SearchResponse { + text: "No search query was provided. Provide a `query` of one or more keywords." + .to_string(), + results: Vec::new(), + })); + } + + // Lazy corpus init (gzip decompress + parse) and the per-query full-corpus scan + // are CPU-bound; keep them off the async runtime. + let (text, results) = tokio::task::spawn_blocking(move || { + let corpus = corpus::corpus(); + // Body grep first (concrete content hits), then index titles/descriptions to + // surface named features body grep misses; merge dedupes by canonical URL. + let body = search::search_docs_pages(&corpus.pages, &query, 5); + let index = search::search_docs_index(&corpus.index, &query, 4); + let results = search::merge_docs_search_results(body, index, search::SEARCH_MAX_PAGES); + let text = search::format_docs_search_results(&query, &results); + (text, results) + }) + .await + .map_err(|e| windmill_common::error::Error::InternalErr(format!("docs search task: {e}")))?; + + Ok(Json(SearchResponse { text, results })) +} + +async fn read_docs_page(Query(q): Query) -> JsonResult { + let url = q.url.trim(); + if url.is_empty() { + return Ok(Json(PageResponse { + text: "No documentation page URL was provided. Provide a `url` — e.g. a `Source` URL returned by the docs search tool.".to_string(), + source_url: String::new(), + })); + } + + let url = url.to_string(); + let section = q.section.filter(|s| !s.trim().is_empty()); + + // Corpus init + page sanitize/render are CPU-bound; keep them off the runtime. + let resp = tokio::task::spawn_blocking(move || { + let corpus = corpus::corpus(); + match corpus.find_page(&url) { + Some(page) => { + // Rewrite docusaurus source-file links to canonical published URLs + // so the model never echoes a broken `.mdx` path. + let sanitized = search::sanitize_docs_markdown_links(&page.body, &page.url); + let rendered = search::render_docs_page_result(&sanitized, section.as_deref()); + let text = format!( + "Source page — cite this URL when referencing this page: {}\n\n{}", + page.url, rendered + ); + PageResponse { text, source_url: page.url.clone() } + } + None => PageResponse { + text: format!( + "No documentation page found for \"{}\". Use the docs search tool to find the correct Source URL first.", + url + ), + source_url: search::canonical_docs_page_url(&url), + }, + } + }) + .await + .map_err(|e| windmill_common::error::Error::InternalErr(format!("docs page task: {e}")))?; + + Ok(Json(resp)) +} diff --git a/backend/windmill-api/src/docs/search.rs b/backend/windmill-api/src/docs/search.rs new file mode 100644 index 0000000000..733558ceb4 --- /dev/null +++ b/backend/windmill-api/src/docs/search.rs @@ -0,0 +1,729 @@ +//! Documentation search & page rendering. +//! +//! Direct port of the pure functions in the frontend's +//! `copilot/chat/docs/core.ts`, so the AI chat, the MCP `searchDocs`/`readDocsPage` +//! tools and the `wmill docs` CLI all return identical results from one place. +//! Operates over the vendored corpus parsed in [`super::corpus`]. + +use lazy_static::lazy_static; +use regex::Regex; +use serde::Serialize; +use std::collections::HashSet; + +pub const DOCS_ORIGIN: &str = "https://www.windmill.dev"; + +// Above this size, return an outline of the page's headings instead of the full +// content, prompting the model to request a specific section. +const FULL_PAGE_CHAR_LIMIT: usize = 20_000; + +// search result caps — keep the returned payload small (the whole point of search +// vs. dumping the index or full pages is token economy). +pub const SEARCH_MAX_PAGES: usize = 8; +const SEARCH_MAX_SNIPPETS_PER_PAGE: usize = 3; +const SEARCH_MAX_SNIPPET_CHARS: usize = 200; +// Each distinct query term triggers a full-corpus scan; cap it so a long, +// caller-controlled query can't multiply the scan cost without bound. Real +// queries are a handful of keywords, so this never truncates a useful search. +const MAX_QUERY_TERMS: usize = 24; + +// --------------------------------------------------------------------------- +// Corpus record types +// --------------------------------------------------------------------------- + +/// A single page extracted from llms-full.txt, keyed by its `Source:` URL. +pub struct DocsFullPage { + pub url: String, + pub title: String, + pub body: String, + /// `body` lowercased once at parse time, so the per-query full-corpus scan + /// doesn't re-allocate a lowercase copy of every page on each request. + pub body_lower: String, +} + +/// A line of the llms.txt index: title, URL and one-line description. +pub struct DocsIndexEntry { + pub title: String, + pub url: String, + pub description: String, + /// `title`/`description` lowercased once at parse time (same rationale as + /// `DocsFullPage::body_lower`). + pub title_lower: String, + pub description_lower: String, +} + +#[derive(Serialize, Clone)] +pub struct DocsSearchResult { + pub url: String, + pub title: String, + /// Higher = more relevant. Distinct query terms matched dominate raw occurrences. + pub score: i64, + pub snippets: Vec, +} + +// --------------------------------------------------------------------------- +// Corpus parsing +// --------------------------------------------------------------------------- + +lazy_static! { + // In llms-full.txt every page's `Source:` line is preceded by a category-header + // lead-in: `...page body...\n\n---\n\n## \n\nSource: `. Splitting + // on `Source:` lines leaves that lead-in on the *previous* page, so strip a + // trailing `---` + level-2-heading block to avoid mis-attributing the next + // page's category title to the previous page. + static ref TRAILING_LEAD_IN_RE: Regex = + Regex::new(r"\n+-{3,}[ \t]*\n+#{2}[ \t]+.*[ \t]*\n*$").unwrap(); + // A line in llms.txt: `- [Title](https://.../page.md): question-phrased description`. + static ref INDEX_ENTRY_RE: Regex = + Regex::new(r"^\s*-\s*\[([^\]]+)\]\(([^)\s]+)\)\s*:?\s*(.*)$").unwrap(); + static ref ORDERING_PREFIX_RE: Regex = Regex::new(r"^\d+[_-]").unwrap(); + // Markdown inline link `](target "optional title")`. + static ref MD_LINK_RE: Regex = Regex::new(r#"\]\(([^)\s]+?)(\s+"[^"]*")?\)"#).unwrap(); +} + +/// `^Source:\s*(\S+)\s*$` — returns the single non-whitespace URL token. +fn parse_source_line(line: &str) -> Option<&str> { + let rest = line.strip_prefix("Source:")?; + let trimmed = rest.trim(); + if trimmed.is_empty() || trimmed.contains(char::is_whitespace) { + return None; + } + Some(trimmed) +} + +/// Splits the llms-full.txt corpus into per-page records keyed by the `Source:` +/// URL. Content before the first `Source:` line (the corpus preamble) is dropped. +pub fn parse_docs_full_text(full_text: &str) -> Vec { + let mut pages = Vec::new(); + let mut url: Option = None; + let mut buffer: Vec<&str> = Vec::new(); + + for line in full_text.split('\n') { + if let Some(u) = parse_source_line(line) { + flush_page(&mut pages, &url, &buffer); + url = Some(u.to_string()); + buffer.clear(); + continue; + } + if url.is_some() { + buffer.push(line); + } + } + flush_page(&mut pages, &url, &buffer); + pages +} + +fn flush_page(pages: &mut Vec, url: &Option, buffer: &[&str]) { + let Some(url) = url else { + return; + }; + let joined = buffer.join("\n"); + let body = TRAILING_LEAD_IN_RE.replace(&joined, ""); + let body = body.trim(); + if !body.is_empty() { + let title = first_heading(body).unwrap_or_else(|| url.clone()); + let body = body.to_string(); + let body_lower = body.to_lowercase(); + pages.push(DocsFullPage { url: url.clone(), title, body, body_lower }); + } +} + +fn first_heading(body: &str) -> Option { + for line in body.split('\n') { + if let Some((_, title)) = parse_heading_line(line, 6) { + return Some(title); + } + } + None +} + +/// Parses the llms.txt index into per-page entries (title, URL, description). +pub fn parse_docs_index(index_text: &str) -> Vec { + let mut entries = Vec::new(); + for line in index_text.split('\n') { + if let Some(caps) = INDEX_ENTRY_RE.captures(line) { + let url = caps[2].trim(); + if !url.contains("/docs/") { + continue; + } + let title = caps[1].trim().to_string(); + let description = caps[3].trim().to_string(); + entries.push(DocsIndexEntry { + title_lower: title.to_lowercase(), + description_lower: description.to_lowercase(), + title, + url: url.to_string(), + description, + }); + } + } + entries +} + +// --------------------------------------------------------------------------- +// Headings, outline, section extraction +// --------------------------------------------------------------------------- + +pub struct DocsHeading { + pub level: usize, + pub title: String, + /// Byte offset of the start of the heading line within the document. + pub start_index: usize, +} + +/// `^(#{1,max_level})\s+(.*\S)\s*$` — heading level and trimmed title. +fn parse_heading_line(line: &str, max_level: usize) -> Option<(usize, String)> { + let hashes = line.bytes().take_while(|&b| b == b'#').count(); + if hashes < 1 || hashes > max_level { + return None; + } + let rest = &line[hashes..]; + // require at least one whitespace after the hashes (\s+) + if !rest.starts_with(|c: char| c.is_whitespace()) { + return None; + } + let title = rest.trim(); + if title.is_empty() { + return None; + } + Some((hashes, title.to_string())) +} + +fn match_fence(line: &str) -> Option { + let trimmed = line.trim_start(); + let first = trimmed.chars().next()?; + if first != '`' && first != '~' { + return None; + } + let count = trimmed.chars().take_while(|&c| c == first).count(); + if count >= 3 { + Some(std::iter::repeat(first).take(count).collect()) + } else { + None + } +} + +/// Parses the markdown headings (`#`–`####`) of a docs page, ignoring any +/// heading-like lines inside fenced code blocks (which are common in samples). +pub fn parse_docs_headings(content: &str) -> Vec { + let mut headings = Vec::new(); + let mut offset = 0usize; + let mut fence_marker: Option = None; + + for line in content.split('\n') { + if let Some(fence) = match_fence(line) { + match &fence_marker { + None => fence_marker = Some(fence), + Some(marker) if line.trim_start().starts_with(marker.as_str()) => { + fence_marker = None; + } + _ => {} + } + offset += line.len() + 1; + continue; + } + + if fence_marker.is_none() { + if let Some((level, title)) = parse_heading_line(line, 4) { + headings.push(DocsHeading { level, title, start_index: offset }); + } + } + offset += line.len() + 1; + } + headings +} + +fn section_end_index(content: &str, headings: &[DocsHeading], index: usize) -> usize { + let level = headings[index].level; + // A section ends at the next heading of the same or higher (shallower) level. + for h in &headings[index + 1..] { + if h.level <= level { + return h.start_index; + } + } + content.len() +} + +/// Builds a human-readable outline of a page's headings, with an approximate +/// size for each section. Used when a page is too large to return whole. +pub fn build_docs_outline(content: &str) -> String { + let headings = parse_docs_headings(content); + if headings.is_empty() { + return "(no markdown headings found on this page)".to_string(); + } + headings + .iter() + .enumerate() + .map(|(i, h)| { + let end = section_end_index(content, &headings, i); + let approx = end.saturating_sub(h.start_index); + let indent = " ".repeat(h.level.saturating_sub(1)); + format!("{}- {} (~{} chars)", indent, h.title, approx) + }) + .collect::>() + .join("\n") +} + +/// Normalizes a heading title for tolerant, case/punctuation-insensitive matching. +fn normalize_heading_title(title: &str) -> String { + // toLowerCase, replace /[^a-z0-9]+/g with ' ', trim + let mut out = String::new(); + let mut prev_space = false; + for ch in title.to_lowercase().chars() { + if ch.is_ascii_alphanumeric() { + out.push(ch); + prev_space = false; + } else if !prev_space { + out.push(' '); + prev_space = true; + } + } + out.trim().to_string() +} + +/// Extracts the content of the section whose heading matches `section` (from the +/// heading up to the next heading of the same/higher level). Case-insensitive and +/// tolerant of minor punctuation differences. `None` when no heading matches. +pub fn extract_docs_section(content: &str, section: &str) -> Option { + let headings = parse_docs_headings(content); + let target = normalize_heading_title(section); + if target.is_empty() { + return None; + } + let match_index = headings + .iter() + .position(|h| normalize_heading_title(&h.title) == target) + .or_else(|| { + // Fall back to a contains match so "Result streaming" matches "Result". + headings + .iter() + .position(|h| normalize_heading_title(&h.title).contains(&target)) + })?; + + let start = headings[match_index].start_index; + let end = section_end_index(content, &headings, match_index); + Some(content[start..end].trim().to_string()) +} + +/// Decides what to return for read_docs_page: a requested section, the full page, +/// or an outline asking the model to pick a section. +pub fn render_docs_page_result(content: &str, section: Option<&str>) -> String { + if let Some(section) = section { + if let Some(extracted) = extract_docs_section(content, section) { + return extracted; + } + return format!( + "No section matching \"{}\" was found on this page. Available sections:\n\n{}", + section, + build_docs_outline(content) + ); + } + + if content.len() <= FULL_PAGE_CHAR_LIMIT { + return content.to_string(); + } + + format!( + "This documentation page is large. Below is its list of sections with approximate sizes.\n\ + Call the docs page-reading tool again with the same `url` argument and a `section` set to one of these headings to read that section.\n\n{}", + build_docs_outline(content) + ) +} + +// --------------------------------------------------------------------------- +// Ranking +// --------------------------------------------------------------------------- + +/// Splits a query into distinct, lowercased, non-empty terms (insertion order), +/// capped at `MAX_QUERY_TERMS`. +fn tokenize_query(query: &str) -> Vec { + let mut seen = HashSet::new(); + let mut out = Vec::new(); + for term in query.to_lowercase().split_whitespace() { + if seen.insert(term.to_string()) { + out.push(term.to_string()); + if out.len() >= MAX_QUERY_TERMS { + break; + } + } + } + out +} + +fn count_occurrences(haystack: &str, needle: &str) -> usize { + if needle.is_empty() { + return 0; + } + haystack.matches(needle).count() +} + +struct Scored { + res: DocsSearchResult, + distinct_terms: usize, + order: usize, +} + +/// Prefer pages that cover every query term; sort by score desc then input order; +/// take the top `max_pages`. +fn finalize_pool(mut scored: Vec, term_count: usize, max_pages: usize) -> Vec { + let has_full = scored.iter().any(|s| s.distinct_terms == term_count); + if has_full { + scored.retain(|s| s.distinct_terms == term_count); + } + scored.sort_by(|a, b| b.res.score.cmp(&a.res.score).then(a.order.cmp(&b.order))); + scored.into_iter().take(max_pages).map(|s| s.res).collect() +} + +/// Ranks docs pages for a keyword query. Score is `distinctTermsMatched` +/// (dominant) then total occurrences. Each result carries up to +/// `SEARCH_MAX_SNIPPETS_PER_PAGE` of its most term-dense lines. +pub fn search_docs_pages(pages: &[DocsFullPage], query: &str, max_pages: usize) -> Vec { + let terms = tokenize_query(query); + if terms.is_empty() { + return Vec::new(); + } + + let mut scored = Vec::new(); + for (order, page) in pages.iter().enumerate() { + let mut distinct = 0usize; + let mut occurrences = 0usize; + for term in &terms { + let count = count_occurrences(&page.body_lower, term); + if count > 0 { + distinct += 1; + occurrences += count; + } + } + if distinct == 0 { + continue; + } + scored.push(Scored { + res: DocsSearchResult { + url: page.url.clone(), + title: page.title.clone(), + // distinctTerms dominates so a page matching all terms always + // outranks one matching fewer, regardless of raw occurrences. + score: distinct as i64 * 1_000_000 + occurrences as i64, + snippets: select_snippets( + &page.body, + &terms, + SEARCH_MAX_SNIPPETS_PER_PAGE, + SEARCH_MAX_SNIPPET_CHARS, + ), + }, + distinct_terms: distinct, + order, + }); + } + finalize_pool(scored, terms.len(), max_pages) +} + +/// Ranks index entries by matching query terms against each entry's title and +/// description (title matches weigh more). The description becomes the result's +/// single snippet. Recovers "named feature" discovery that full-text grep misses. +pub fn search_docs_index(entries: &[DocsIndexEntry], query: &str, max_pages: usize) -> Vec { + let terms = tokenize_query(query); + if terms.is_empty() { + return Vec::new(); + } + + let mut scored = Vec::new(); + for (order, entry) in entries.iter().enumerate() { + let mut distinct = 0usize; + let mut score = 0i64; + for term in &terms { + let in_title = entry.title_lower.contains(term.as_str()); + let in_desc = entry.description_lower.contains(term.as_str()); + if in_title || in_desc { + distinct += 1; + score += if in_title { 5 } else { 0 } + if in_desc { 1 } else { 0 }; + } + } + if distinct == 0 { + continue; + } + scored.push(Scored { + res: DocsSearchResult { + url: entry.url.clone(), + title: entry.title.clone(), + score: distinct as i64 * 1_000_000 + score, + snippets: if entry.description.is_empty() { + Vec::new() + } else { + vec![entry.description.clone()] + }, + }, + distinct_terms: distinct, + order, + }); + } + finalize_pool(scored, terms.len(), max_pages) +} + +/// Picks the most term-dense lines of a page body as snippets, in document order, +/// deduped, each trimmed to `max_chars` around the first matched term. +fn select_snippets(body: &str, terms: &[String], max_snippets: usize, max_chars: usize) -> Vec { + struct LineHit { + text: String, + distinct: usize, + order: usize, + } + let mut hits = Vec::new(); + for (order, line) in body.split('\n').enumerate() { + let lower = line.to_lowercase(); + let distinct = terms.iter().filter(|t| lower.contains(t.as_str())).count(); + if distinct == 0 { + continue; + } + let text = make_snippet(line, terms, max_chars); + if !text.is_empty() { + hits.push(LineHit { text, distinct, order }); + } + } + hits.sort_by(|a, b| b.distinct.cmp(&a.distinct).then(a.order.cmp(&b.order))); + + let mut seen = HashSet::new(); + let mut result = Vec::new(); + for hit in hits { + if !seen.insert(hit.text.clone()) { + continue; + } + result.push(hit.text); + if result.len() >= max_snippets { + break; + } + } + result +} + +/// Collapses a matched line to a single-line snippet of at most `max_chars`, +/// windowed around the first matched term (with ellipses) when the line is long. +/// Operates on `char`s so multibyte content can't split mid-codepoint. +fn make_snippet(line: &str, terms: &[String], max_chars: usize) -> String { + let collapsed = line.split_whitespace().collect::>().join(" "); + let chars: Vec = collapsed.chars().collect(); + if chars.len() <= max_chars { + return collapsed; + } + + let lower = collapsed.to_lowercase(); + let mut first_index: Option = None; + for term in terms { + if let Some(byte_idx) = lower.find(term.as_str()) { + let char_idx = lower[..byte_idx].chars().count(); + first_index = Some(first_index.map_or(char_idx, |f| f.min(char_idx))); + } + } + + let total = chars.len(); + match first_index { + None => { + let slice: String = chars[..max_chars].iter().collect(); + format!("{}…", slice.trim_end()) + } + Some(fi) => { + let start = fi.saturating_sub(max_chars / 3); + let end = (start + max_chars).min(total); + let prefix = if start > 0 { "…" } else { "" }; + let suffix = if end < total { "…" } else { "" }; + let slice: String = chars[start..end].iter().collect(); + format!("{}{}{}", prefix, slice.trim(), suffix) + } + } +} + +/// Strips the `.md` suffix and trailing slash so index/body URLs dedupe. +pub fn canonical_search_url(url: &str) -> String { + let stripped = strip_md_suffix(url); + stripped.strip_suffix('/').unwrap_or(stripped).to_string() +} + +fn strip_md_suffix(s: &str) -> &str { + if s.len() >= 3 && s[s.len() - 3..].eq_ignore_ascii_case(".md") { + &s[..s.len() - 3] + } else { + s + } +} + +/// Merges full-text (body) results with index-description results. Body matches +/// come first; index-only matches fill remaining slots — so a named feature +/// surfaced only by its index entry still appears even when body grep missed it. +pub fn merge_docs_search_results( + body_results: Vec, + index_results: Vec, + max_pages: usize, +) -> Vec { + let mut seen: HashSet = + body_results.iter().map(|r| canonical_search_url(&r.url)).collect(); + let mut merged = body_results; + for entry in index_results { + let key = canonical_search_url(&entry.url); + if seen.insert(key) { + merged.push(entry); + } + } + merged.truncate(max_pages); + merged +} + +/// Renders search results as the string returned to the model. +pub fn format_docs_search_results(query: &str, results: &[DocsSearchResult]) -> String { + if results.is_empty() { + return format!( + "No documentation pages matched \"{}\". Try fewer or more general keywords (a single distinctive term often works best).", + query + ); + } + + let blocks = results + .iter() + .map(|r| { + let mut lines = vec![format!("## {}", r.title), format!("Source: {}", r.url)]; + for snippet in &r.snippets { + lines.push(format!(" - {}", snippet)); + } + lines.join("\n") + }) + .collect::>() + .join("\n\n"); + + format!( + "Found {} documentation page(s) matching \"{}\", most relevant first:\n\n{}\n\n\ + Cite the exact \"Source\" URL when referencing a page. If these snippets are not enough, call the docs page-reading tool with a Source URL as its `url` argument to read the full page or a section.", + results.len(), + query, + blocks + ) +} + +// --------------------------------------------------------------------------- +// URL normalization & link sanitization +// --------------------------------------------------------------------------- + +/// Strips docusaurus numeric ordering prefixes (`13_`, `8-`) from each path +/// segment so it matches the published route. +fn strip_docs_path_prefixes(path: &str) -> String { + path.split('/') + .map(|seg| ORDERING_PREFIX_RE.replace(seg, "").into_owned()) + .collect::>() + .join("/") +} + +/// Normalizes a user/model-supplied docs reference to a fully-qualified `.md` URL +/// on the docs origin. Accepts a full URL, `/docs/...`, or `docs/...`. +pub fn normalize_docs_url(input: &str) -> String { + let mut value = input.trim().to_string(); + + if is_http_url(&value) { + // Strip the origin so we re-anchor to DOCS_ORIGIN and normalize the path. + if let Ok(parsed) = url::Url::parse(&value) { + value = parsed.path().to_string(); + } + } + + // Drop any query string or hash fragment. + value = value.split('#').next().unwrap().split('?').next().unwrap().to_string(); + + if !value.starts_with('/') { + value = format!("/{}", value); + } + // Strip a trailing slash (but keep the leading one). + if value.len() > 1 && value.ends_with('/') { + value.pop(); + } + + value = strip_docs_path_prefixes(&value); + if let Some(stripped) = value.strip_suffix(".mdx") { + value = format!("{}.md", stripped); + } + if !value.ends_with(".md") { + value = format!("{}.md", value); + } + + format!("{}{}", DOCS_ORIGIN, value) +} + +/// The canonical published URL a model should cite for a docs page (the `.md` +/// fetch URL without the suffix). +pub fn canonical_docs_page_url(path: &str) -> String { + strip_md_suffix(&normalize_docs_url(path)).to_string() +} + +fn is_http_url(s: &str) -> bool { + let lower = s.to_ascii_lowercase(); + lower.starts_with("http://") || lower.starts_with("https://") +} + +/// Rewrites relative/source-file doc links inside raw page markdown to canonical +/// published URLs, so the model never echoes a docusaurus source path into its +/// answer as a broken link. Non-doc links (external, images, anchors) and `../` +/// cross-directory links are left untouched. +pub fn sanitize_docs_markdown_links(content: &str, page_url: &str) -> String { + let base = url::Url::parse(page_url).ok(); + MD_LINK_RE + .replace_all(content, |caps: ®ex::Captures| { + let whole = caps.get(0).unwrap().as_str(); + let target = &caps[1]; + let title = caps.get(2).map(|m| m.as_str()).unwrap_or(""); + + // Only rewrite links to docusaurus source files (.md/.mdx); leave + // images, external URLs and bare anchors untouched. + if !is_md_target(target) { + return whole.to_string(); + } + // `../` cross-directory links are authored against the docusaurus + // source tree, whose depth differs from the published URL, so strict + // resolution is unreliable. Leave them for the canonical-URL header. + if has_parent_traversal(target) { + return whole.to_string(); + } + let Some(base) = &base else { + return whole.to_string(); + }; + let Ok(resolved) = base.join(target) else { + return whole.to_string(); + }; + if resolved.scheme() != "https" + || resolved.host_str() != Some("www.windmill.dev") + || !resolved.path().starts_with("/docs/") + { + return whole.to_string(); + } + let pathname = strip_docs_path_prefixes(resolved.path()); + let pathname = strip_md_or_mdx(&pathname); + let hash = resolved.fragment().map(|f| format!("#{}", f)).unwrap_or_default(); + format!("]({}{}{}{})", DOCS_ORIGIN, pathname, hash, title) + }) + .into_owned() +} + +/// `\.mdx?($|[#?])` — the target points at a markdown source file. +fn is_md_target(target: &str) -> bool { + for marker in [".md", ".mdx"] { + if let Some(idx) = target.to_ascii_lowercase().find(marker) { + let after = &target[idx + marker.len()..]; + // `.md` must not be a prefix of `.mdx` here: only accept when the + // extension is followed by end / `#` / `?`. + if after.is_empty() || after.starts_with('#') || after.starts_with('?') { + return true; + } + } + } + false +} + +/// `(^|/)\.\./` — the target contains a parent-directory traversal segment. +fn has_parent_traversal(target: &str) -> bool { + target == ".." || target.starts_with("../") || target.contains("/../") +} + +fn strip_md_or_mdx(path: &str) -> &str { + if let Some(stripped) = path.strip_suffix(".mdx") { + stripped + } else { + strip_md_suffix(path) + } +} + +#[cfg(test)] +mod tests; diff --git a/backend/windmill-api/src/docs/search/tests.rs b/backend/windmill-api/src/docs/search/tests.rs new file mode 100644 index 0000000000..84c422aa73 --- /dev/null +++ b/backend/windmill-api/src/docs/search/tests.rs @@ -0,0 +1,267 @@ +//! Parity tests ported from the frontend `copilot/chat/docs/core.test.ts`. + +use super::*; + +const SAMPLE: &str = "# Jobs\n\nIntro text about jobs.\n\n## Job kinds\n\nSome kinds.\n\n## Result\n\n### Result of jobs that failed\n\n```\n{ \"error\": \"boom\" }\n```\n\n### Result streaming\n\n#### Returning a stream directly\n\n```python\n# Returning a stream directly is a comment heading that must be ignored\ndef main():\n pass\n```\n\n## Retention policy\n\nFinal section.\n"; + +// Mirrors the llms-full.txt layout: a corpus preamble, then per-page blocks each +// introduced by a `---` + `## ` lead-in followed by a `Source:` line. +const SAMPLE_FULL: &str = "# Windmill\n\n> Preamble blurb that precedes the first Source line and must be ignored.\n\n## Browser automation\n\nSource: https://www.windmill.dev/docs/advanced/browser_automation\n\n# Browser automation\n\nBy default, a worker group named `reports` handles jobs with the `chromium` tag.\nThe chromium binary will be available on these workers at /usr/bin/chromium.\nYou can disable the sandbox by passing the --no-sandbox flag.\n\n---\n\n## Worker groups\n\nSource: https://www.windmill.dev/docs/core_concepts/worker_groups\n\n# Worker groups\n\nWorker groups let you assign tags to workers.\nSet the chromium tag on a worker so it can run browser jobs.\n\n---\n\n## Scheduling\n\nSource: https://www.windmill.dev/docs/core_concepts/scheduling\n\n# Scheduling\n\nUse cron expressions to schedule scripts and flows.\n"; + +#[test] +fn parses_headings_and_ignores_fenced_blocks() { + let titles: Vec = parse_docs_headings(SAMPLE) + .iter() + .map(|h| format!("{}:{}", h.level, h.title)) + .collect(); + assert_eq!( + titles, + vec![ + "1:Jobs", + "2:Job kinds", + "2:Result", + "3:Result of jobs that failed", + "3:Result streaming", + "4:Returning a stream directly", + "2:Retention policy", + ] + ); +} + +#[test] +fn heading_start_index_points_at_the_heading_line() { + for h in parse_docs_headings(SAMPLE) { + let at = &SAMPLE[h.start_index..]; + assert!(at.starts_with(&"#".repeat(h.level))); + assert!(at[h.level..].trim_start().starts_with(&h.title)); + } +} + +#[test] +fn handles_tilde_fences() { + let content = "# Title\n\n~~~\n# not a heading\n~~~\n\n## Real\n"; + let titles: Vec = parse_docs_headings(content).iter().map(|h| h.title.clone()).collect(); + assert_eq!(titles, vec!["Title", "Real"]); +} + +#[test] +fn extracts_section_up_to_next_same_or_higher_heading() { + let section = extract_docs_section(SAMPLE, "Result").unwrap(); + assert!(section.contains("## Result")); + assert!(section.contains("### Result of jobs that failed")); + assert!(section.contains("### Result streaming")); + assert!(!section.contains("## Retention policy")); +} + +#[test] +fn extract_section_is_case_and_punctuation_tolerant() { + let section = extract_docs_section(SAMPLE, "retention-policy!").unwrap(); + assert!(section.contains("## Retention policy")); + assert!(section.contains("Final section.")); +} + +#[test] +fn extract_section_returns_none_when_missing() { + assert!(extract_docs_section(SAMPLE, "Nonexistent section").is_none()); +} + +#[test] +fn build_outline_lists_headings_with_indent() { + let outline = build_docs_outline(SAMPLE); + assert!(outline.contains("- Jobs (~")); + assert!(outline.contains(" - Job kinds (~")); + assert!(outline.contains(" - Result of jobs that failed (~")); +} + +#[test] +fn build_outline_handles_no_headings() { + assert_eq!( + build_docs_outline("just some text\nwith no headings"), + "(no markdown headings found on this page)" + ); +} + +#[test] +fn render_page_returns_whole_small_page() { + assert_eq!(render_docs_page_result(SAMPLE, None), SAMPLE); +} + +#[test] +fn render_page_returns_outline_for_large_page() { + let large = format!("# Big\n\n{}\n\n## Tail\n\nmore", "x".repeat(25_000)); + let result = render_docs_page_result(&large, None); + assert!(result.contains("This documentation page is large")); + assert!(result.contains("same `url` argument")); + assert!(!result.contains("same path")); + assert!(!result.contains("read_docs_page")); + assert!(result.contains("- Big (~")); + assert!(result.contains("- Tail (~")); +} + +#[test] +fn render_page_returns_requested_section() { + let result = render_docs_page_result(SAMPLE, Some("Job kinds")); + assert!(result.contains("## Job kinds")); + assert!(result.contains("Some kinds.")); +} + +#[test] +fn render_page_missing_section_returns_outline_note() { + let result = render_docs_page_result(SAMPLE, Some("Does not exist")); + assert!(result.contains("No section matching \"Does not exist\" was found")); + assert!(result.contains("- Jobs (~")); +} + +#[test] +fn normalize_docs_url_cases() { + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("docs/core_concepts/jobs"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!( + normalize_docs_url("https://www.windmill.dev/docs/core_concepts/jobs#result?foo=bar"), + "https://www.windmill.dev/docs/core_concepts/jobs.md" + ); + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs.md"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("/docs/core_concepts/jobs/"), "https://www.windmill.dev/docs/core_concepts/jobs.md"); + assert_eq!(normalize_docs_url("/docs/flows/13_flow_branches"), "https://www.windmill.dev/docs/flows/flow_branches.md"); + assert_eq!(normalize_docs_url("/docs/flows/13_flow_branches.mdx"), "https://www.windmill.dev/docs/flows/flow_branches.md"); +} + +#[test] +fn canonical_docs_page_url_cases() { + assert_eq!(canonical_docs_page_url("/docs/flows/flow_editor"), "https://www.windmill.dev/docs/flows/flow_editor"); + assert_eq!(canonical_docs_page_url("/docs/flows/14_retries.md"), "https://www.windmill.dev/docs/flows/retries"); +} + +#[test] +fn sanitize_markdown_links_cases() { + let page = "https://www.windmill.dev/docs/flows/flow_editor.md"; + assert_eq!( + sanitize_docs_markdown_links("See [retries](./14_retries.mdx) for more.", page), + "See [retries](https://www.windmill.dev/docs/flows/retries) for more." + ); + assert_eq!( + sanitize_docs_markdown_links("[handling](./8_error_handling.mdx)", page), + "[handling](https://www.windmill.dev/docs/flows/error_handling)" + ); + assert_eq!( + sanitize_docs_markdown_links("[branch all](./13_flow_branches.mdx#branch-all)", page), + "[branch all](https://www.windmill.dev/docs/flows/flow_branches#branch-all)" + ); + // images & external links untouched + let external = "![diagram](./assets/flow_example.png) and [site](https://example.com/page.md)"; + assert_eq!(sanitize_docs_markdown_links(external, page), external); + // bare anchor untouched + assert_eq!(sanitize_docs_markdown_links("[top](#introduction)", page), "[top](#introduction)"); + // ../ cross-directory links untouched + let parent = "[handling](../core_concepts/8_error_handling.mdx)"; + assert_eq!(sanitize_docs_markdown_links(parent, page), parent); + let parent2 = "[retries](../../flows/14_retries.md)"; + assert_eq!(sanitize_docs_markdown_links(parent2, page), parent2); +} + +#[test] +fn parse_full_text_splits_pages_and_drops_preamble() { + let pages = parse_docs_full_text(SAMPLE_FULL); + assert_eq!( + pages.iter().map(|p| p.url.clone()).collect::>(), + vec![ + "https://www.windmill.dev/docs/advanced/browser_automation", + "https://www.windmill.dev/docs/core_concepts/worker_groups", + "https://www.windmill.dev/docs/core_concepts/scheduling", + ] + ); + assert_eq!( + pages.iter().map(|p| p.title.clone()).collect::>(), + vec!["Browser automation", "Worker groups", "Scheduling"] + ); + // The next page's "## Worker groups" lead-in must not leak into this body. + let browser = pages.iter().find(|p| p.url.ends_with("/browser_automation")).unwrap(); + assert!(!browser.body.contains("Worker groups")); + assert!(!browser.body.contains("---")); +} + +#[test] +fn search_pages_ranks_more_occurrences_first() { + let pages = parse_docs_full_text(SAMPLE_FULL); + let results = search_docs_pages(&pages, "chromium", 5); + assert_eq!( + results.iter().map(|r| r.url.clone()).collect::>(), + vec![ + "https://www.windmill.dev/docs/advanced/browser_automation", + "https://www.windmill.dev/docs/core_concepts/worker_groups", + ] + ); + assert!(!results[0].snippets.is_empty()); + assert!(results[0].snippets.join("\n").contains("chromium")); +} + +#[test] +fn search_prefers_pages_covering_all_terms() { + let pages = parse_docs_full_text(SAMPLE_FULL); + // Only browser_automation contains "sandbox"; worker_groups has "chromium" but + // not "sandbox". A full-coverage page exists, so partial matches are dropped. + let results = search_docs_pages(&pages, "chromium sandbox", 5); + assert_eq!( + results.iter().map(|r| r.url.clone()).collect::>(), + vec!["https://www.windmill.dev/docs/advanced/browser_automation"] + ); +} + +#[test] +fn merge_dedupes_index_against_body_by_canonical_url() { + let body = vec![DocsSearchResult { + url: "https://www.windmill.dev/docs/a".to_string(), + title: "A".to_string(), + score: 10, + snippets: vec![], + }]; + let index = vec![ + DocsSearchResult { + url: "https://www.windmill.dev/docs/a.md".to_string(), + title: "A".to_string(), + score: 5, + snippets: vec![], + }, + DocsSearchResult { + url: "https://www.windmill.dev/docs/b.md".to_string(), + title: "B".to_string(), + score: 5, + snippets: vec![], + }, + ]; + let merged = merge_docs_search_results(body, index, SEARCH_MAX_PAGES); + assert_eq!( + merged.iter().map(|r| r.url.clone()).collect::>(), + vec!["https://www.windmill.dev/docs/a", "https://www.windmill.dev/docs/b.md"] + ); +} + +#[test] +fn empty_query_returns_no_results() { + let pages = parse_docs_full_text(SAMPLE_FULL); + assert!(search_docs_pages(&pages, " ", 5).is_empty()); +} + +#[test] +fn format_search_results_no_matches() { + let out = format_docs_search_results("zzz", &[]); + assert!(out.contains("No documentation pages matched \"zzz\"")); +} + +#[test] +fn format_search_results_uses_caller_neutral_followup_guidance() { + let out = format_docs_search_results( + "jobs", + &[DocsSearchResult { + url: "https://www.windmill.dev/docs/core_concepts/jobs".to_string(), + title: "Jobs".to_string(), + score: 1, + snippets: vec!["Jobs run scripts and flows.".to_string()], + }], + ); + + assert!(out.contains("Source: https://www.windmill.dev/docs/core_concepts/jobs")); + assert!(out.contains("Source URL as its `url` argument")); + assert!(!out.contains("read_docs_page")); + assert!(!out.contains("readDocsPage")); +} diff --git a/backend/windmill-api/src/drafts.rs b/backend/windmill-api/src/drafts.rs index b0cd5f61b0..9d4976489b 100644 --- a/backend/windmill-api/src/drafts.rs +++ b/backend/windmill-api/src/drafts.rs @@ -6,133 +6,862 @@ * LICENSE-AGPL for a copy of the license. */ -use crate::{ - db::{ApiAuthed, DB}, - users::{maybe_refresh_folders, require_owner_of_path}, -}; +use crate::db::{ApiAuthed, DB}; use axum::{ - extract::{Extension, Path}, - routing::{delete, post}, + extract::{Extension, Path, Query}, + routing::{get, post}, Json, Router, }; -use hyper::StatusCode; use serde::{Deserialize, Serialize}; -use windmill_common::{db::UserDB, error::Result, utils::StripPath}; +use windmill_common::{ + db::UserDB, + error::{Error, Result}, + user_drafts::{DraftUserRef, UserDraftItemKind, ENCRYPTED_DRAFT_PREFIX}, + variables::{build_crypt, encrypt}, +}; pub fn workspaced_service() -> Router { Router::new() - .route("/create", post(create_draft)) - .route("/delete/{kind}/{*path}", delete(delete_draft)) + .route("/list", get(list_drafts)) + .route("/get/{kind}/{*path}", get(get_draft_for_user)) + .route("/get_own/{kind}/{*path}", get(get_own_draft)) + .route("/update/{kind}/{*path}", post(update_draft)) + .route("/migrate_legacy/{kind}/{*path}", post(migrate_legacy_draft)) } -#[derive(sqlx::Type, Serialize, Deserialize, Debug, PartialEq, Clone)] -#[sqlx(type_name = "DRAFT_TYPE", rename_all = "lowercase")] -#[serde(rename_all(serialize = "lowercase", deserialize = "lowercase"))] -pub enum DraftType { - Script, - Flow, - App, -} - -#[derive(Deserialize, Serialize, Debug)] -pub struct Draft { +#[derive(Serialize, sqlx::FromRow)] +pub struct DraftListItem { + pub kind: UserDraftItemKind, pub path: String, - pub value: sqlx::types::Json>, - pub typ: DraftType, + /// Best-effort, read from the draft JSON's `summary` field when present. + #[serde(skip_serializing_if = "Option::is_none")] + pub summary: Option, + /// User-typed friendly path read from the draft JSON's `draft_path` (set by + /// the editors when it differs from the storage path, e.g. a never-deployed + /// item parked at `u/{user}/draft_{uuid}`). `None` when absent. Lets the + /// review page show the friendly name instead of the storage path, like the + /// home-page list endpoints. + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_path: Option, + /// No deployed counterpart exists at this path — the draft is the whole + /// item. Kinds without a per-path backing table report `true`. + pub draft_only: bool, + /// The listed row is a legacy workspace-level draft (`email IS NULL`), + /// predating the per-user drafts migration. Only `true` when no per-user + /// row exists at this (path, kind) — the DISTINCT ON prefers an owned row. + pub legacy_draft: bool, + pub created_at: chrono::DateTime, + /// All draft authors at this `(path, kind)`, for the shared full-page-editor + /// kinds (script/flow/app/raw_app) only — feeds the home-page-style owner + /// circles on the review page. `None` for drawer kinds, which keep their + /// drafts private. + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_users: Option>>, + /// Whether the authed user may deploy/discard this draft — the same check + /// the deploy/discard endpoints enforce. Computed per row after the query, + /// so it defaults to `false` when read from the row. + #[sqlx(default)] + pub can_write: bool, + /// The listed row belongs to the authed user (own draft or the legacy + /// no-owner row) and is therefore actionable by them. Always `true` in the + /// default (own-drafts) listing; only meaningful with `all_users=true`, + /// where other users' rows surface as `false` (view-only — you can't deploy + /// someone else's draft). + pub mine: bool, } -pub async fn require_writer_of_path( - authed: &ApiAuthed, - path: &str, - w_id: &str, - db: DB, - kind: &DraftType, -) -> Result<()> { - if authed.is_admin { - return Ok(()); - } else if require_owner_of_path(authed, path).is_ok() { - return Ok(()); - } else { - match kind { - DraftType::Script => crate::scripts::require_is_writer(authed, path, w_id, db).await, - DraftType::Flow => crate::flows::require_is_writer(authed, path, w_id, db).await, - DraftType::App => crate::apps::require_is_writer(authed, path, w_id, db).await, - } - } +#[derive(Deserialize)] +pub struct ListDraftsQuery { + /// List every draft in the workspace (all users), not just the authed + /// user's own + legacy rows. Other users' rows come back with `mine=false`. + pub all_users: Option, } -async fn create_draft( +/// Every draft the authed user has in this workspace, across all kinds — the +/// single source for the "Review & deploy drafts" page and the home-page +/// draft-count banner. One query over `draft`; `draft_only` is computed per +/// kind against the deployed table. +async fn list_drafts( authed: ApiAuthed, Extension(db): Extension, Extension(user_db): Extension, Path(w_id): Path, - Json(draft): Json, -) -> Result<(StatusCode, String)> { - let authed = maybe_refresh_folders(&draft.path, &w_id, authed, &db).await; - - let mut tx = user_db.begin(&authed).await?; - - require_writer_of_path(&authed, &draft.path, &w_id, db, &draft.typ).await?; - - sqlx::query!( - "INSERT INTO draft - (workspace_id, path, value, typ) - VALUES ($1, $2, $3::text::json, $4) - ON CONFLICT (workspace_id, path, typ) - DO UPDATE SET value = EXCLUDED.value, created_at = now()", - &w_id, - draft.path, - //to preserve key orders - serde_json::to_string(&draft.value).unwrap(), - draft.typ as DraftType, - ) - .execute(&mut *tx) - .await?; - - tx.commit().await?; - - Ok((StatusCode::CREATED, format!("draft {} created", draft.path))) + Query(query): Query, +) -> Result>> { + // Operators have no drafts of their own (they can't write any, see + // `require_can_write_path`), so this list is always empty for them. They + // can still READ some collaborators' drafts via `/drafts/get`. + if authed.is_operator { + return Ok(Json(vec![])); + } + let all_users = query.all_users.unwrap_or(false); + let rows = sqlx::query_as::<_, DraftListItem>(&list_drafts_query(all_users)) + .bind(&w_id) + .bind(&authed.email) + .fetch_all(&db) + .await?; + // Per-row permission gating: + // - own drafts (incl. legacy no-owner rows, `mine = true`): the actionable + // gate is write permission — run the exact check deploy/discard enforce so + // the UI never offers an action that would 403. + // - other users' drafts (only present with `all_users`, `mine = false`): the + // UI never lets you act on them (`isSelectable` requires `mine`), so skip + // the write probe (`can_write = false`) and instead require READ access — + // otherwise the broadened listing would disclose the path/summary/authors + // of items the caller can't see. Unreadable rows are dropped, mirroring the + // `require_can_read_path` gate on `/drafts/get`. + let mut out = Vec::with_capacity(rows.len()); + for mut row in rows { + if row.mine { + row.can_write = + match require_can_write_path(&authed, &db, &user_db, &w_id, row.kind, &row.path) + .await + { + Ok(()) => true, + Err(Error::NotAuthorized(_)) => false, + Err(e) => return Err(e), + }; + out.push(row); + } else { + // `require_can_read_path` denies with `NotFound` (it hides existence) + // and, for some paths, `NotAuthorized` — both mean "not visible to the + // caller", so drop the row. Any other error is a real failure. + match require_can_read_path(&authed, &user_db, &w_id, row.kind, &row.path).await { + Ok(()) => { + row.can_write = false; + out.push(row); + } + Err(Error::NotFound(_)) | Err(Error::NotAuthorized(_)) => {} + Err(e) => return Err(e), + } + } + } + Ok(Json(out)) } -async fn delete_draft( +/// Build the `list_drafts` SQL, generating the `draft_only` CASE from +/// `deployed_table()` (shared single source — can't drift from the access +/// check). Table names come from the closed enum, never user input. Kinds +/// with no path-keyed table get no arm and fall to `ELSE true`. +/// `$1` = workspace_id, `$2` = email. With `all_users` the owner filter is +/// dropped so every workspace draft is listed (others' rows get `mine=false`). +fn list_drafts_query(all_users: bool) -> String { + let mut case = String::from("CASE d.typ::text\n"); + for kind in UserDraftItemKind::ALL { + let Some(table) = kind.deployed_table() else { + continue; + }; + // `script` rows are soft-deleted — a deleted script counts as "not + // deployed". No other backing table has a `deleted` flag. + let extra = if matches!(kind, UserDraftItemKind::Script) { + " AND t.deleted = false" + } else { + "" + }; + case.push_str(&format!( + " WHEN '{}' THEN NOT EXISTS(SELECT 1 FROM {} t WHERE t.workspace_id = d.workspace_id AND t.path = d.path{})\n", + kind.as_str(), + table, + extra + )); + } + case.push_str(" ELSE true\nEND"); + // Owner circles, mirroring the home-page list subquery (see apps.rs): every + // draft author at this (path, kind), legacy NULL-email row surfaced as a + // null username. Restricted to the shared full-page-editor kinds — drawer + // kinds keep their drafts private, so we never reveal their authors. + let draft_users = r#"CASE WHEN d.typ::text IN ('script', 'flow', 'app', 'raw_app') THEN ( + SELECT json_agg(json_build_object('username', COALESCE(u.username, CASE WHEN du.workspace_id = 'admins' THEN du.email END)) + ORDER BY COALESCE(u.username, CASE WHEN du.workspace_id = 'admins' THEN du.email END) NULLS LAST) + FROM draft du + LEFT JOIN usr u ON u.workspace_id = du.workspace_id AND u.email = du.email + WHERE du.workspace_id = d.workspace_id AND du.path = d.path AND du.typ = d.typ + ) ELSE NULL END"#; + // Default lists the user's own drafts AND the legacy NULL-email rows; with + // `all_users` the filter is dropped to list every workspace draft. + let owner_filter = if all_users { + "" + } else { + " AND (d.email = $2 OR d.email IS NULL)" + }; + // `DISTINCT ON (d.path, d.typ)` keeps one row per item; the ORDER BY + // priority below picks the user's own row first, then the legacy NULL row, + // then (only with `all_users`) another user's row. `mine`/`legacy_draft` + // describe that kept row. + format!( + r#"SELECT DISTINCT ON (d.path, d.typ) + d.path, + d.typ AS kind, + d.created_at, + d.value ->> 'summary' AS summary, + {draft_users} AS draft_users, + -- Friendly typed path, by kind (mirrors the home-page list + -- endpoints): scripts bind the Path widget to `script.path`, + -- so it round-trips through the draft JSON's own `path`; + -- flows/apps/raw-apps carry a separate `draft_path`. NULLIF + -- drops it when empty or equal to the storage path. + NULLIF( + NULLIF( + CASE WHEN d.typ::text = 'script' + THEN d.value ->> 'path' + ELSE d.value ->> 'draft_path' END, + ''), + d.path + ) AS draft_path, + (d.email IS NULL) AS legacy_draft, + (d.email = $2 OR d.email IS NULL) AS mine, + {case} AS draft_only + FROM draft d + WHERE d.workspace_id = $1{owner_filter} + ORDER BY d.path, d.typ, + CASE WHEN d.email = $2 THEN 0 WHEN d.email IS NULL THEN 1 ELSE 2 END"# + ) +} + +#[derive(Deserialize, Debug)] +pub struct SaveDraftRequest { + /// Draft content to save. `null` (or omitted) signals a delete — the + /// row is removed under the same conflict rules as an upsert. + #[serde(default)] + pub value: Option>>, + /// Client's last known sync timestamp. When present and `force` is false, + /// the save is rejected if the server's `created_at` is more recent + /// (another writer moved the row forward). Omit on a first save. + #[serde(default)] + pub last_sync: Option>, + /// Skip the conflict check and unconditionally overwrite the server + /// copy. Use after the client has resolved the conflict locally. + #[serde(default)] + pub force: bool, + /// Delete-only: target the legacy workspace-level row (`email IS NULL`) + /// rather than the authed user's row. An upsert ignores it (always writes + /// the user's own row). Lets the review page discard a legacy draft, which + /// the email-scoped delete otherwise can't reach. + #[serde(default)] + pub legacy: bool, + /// Upsert-only override for the stored `created_at`. Normal saves omit it + /// and the row is stamped `now()`; the localStorage→DB migration passes the + /// draft's original write time (or epoch 0 when unknown) so migrated drafts + /// keep their age instead of all resurfacing to the top as freshly created. + #[serde(default)] + pub created_at: Option>, +} + +#[derive(Serialize, Debug)] +#[serde(rename_all = "lowercase")] +pub enum SaveDraftStatus { + Saved, + Conflict, +} + +#[derive(Serialize, Debug)] +pub struct SaveDraftResponse { + pub status: SaveDraftStatus, + /// On `saved`: when the change was applied (client remembers it as the + /// next `last_sync`). On `conflict`: the existing row's `created_at`. + pub current_timestamp: chrono::DateTime, +} + +/// Apply the current user's draft at (workspace, kind, path): non-null `value` +/// upserts, `null` (or omitted) deletes. Either way, when the existing row is +/// newer than `last_sync` (and `force` is false) the op is skipped and the +/// response is `status = conflict` + the server's current timestamp. +async fn update_draft( authed: ApiAuthed, + Extension(db): Extension, Extension(user_db): Extension, - Path((w_id, kind, path)): Path<(String, DraftType, StripPath)>, -) -> Result { - let mut tx = user_db.begin(&authed).await?; + Path((w_id, kind, path)): Path<(String, UserDraftItemKind, windmill_common::utils::StripPath)>, + Json(req): Json, +) -> Result> { + let email = &authed.email; + let path = path.to_path(); + // Saving a draft requires write permission on the underlying path. Deleting + // (discarding) one's OWN draft does not: the email-scoped row belongs to the + // authed user, so they can always discard it even after losing write access + // to the underlying item (e.g. a draft-only item whose folder perms changed). + // The DELETE below is scoped to `email = authed.email`, so it can only ever + // touch the caller's own row. Legacy (NULL-email) rows aren't owned by anyone + // — they keep the write gate. + let is_own_discard = req.value.is_none() && !req.legacy; + if !is_own_discard { + require_can_write_path(&authed, &db, &user_db, &w_id, kind, path).await?; + } - sqlx::query!( - "DELETE FROM draft WHERE path = $1 AND typ = $2 AND workspace_id = $3", - path.to_path(), - kind as DraftType, - w_id + let applied_at = if let Some(value) = &req.value { + // Secret variable values must never sit in `draft.value` in plaintext + // (see `encrypt_secret_variable_value`). + let serialized = if kind == UserDraftItemKind::Variable { + encrypt_secret_variable_value(&db, &w_id, value.0.get()).await? + } else { + serde_json::to_string(value).unwrap() + }; + // `draft.value` is a `json` column, so a U+0000 (NUL) would persist as an + // escape and later make any `->>`/`to_jsonb` extraction raise `22P05`. + // Strip it here so a NUL never reaches the column. + let serialized = strip_json_nul(serialized); + // Upsert. The conflict check rides on the DO UPDATE WHERE clause — + // when the row is newer than `last_sync`, RETURNING yields nothing. + // `created_at` defaults to `now()` but the migration overrides it ($8) + // so a migrated draft keeps its original age instead of jumping to top. + sqlx::query_scalar!( + r#"INSERT INTO draft (workspace_id, email, path, typ, value, created_at) + VALUES ($1, $2, $3, $4, $5::text::json, COALESCE($8::timestamptz, now())) + ON CONFLICT (workspace_id, path, typ, email) WHERE email IS NOT NULL + DO UPDATE SET value = EXCLUDED.value, created_at = EXCLUDED.created_at + WHERE $7::bool = true + OR $6::timestamptz IS NULL + OR draft.created_at <= $6::timestamptz + RETURNING created_at"#, + &w_id, + email, + path, + kind as UserDraftItemKind, + serialized, + req.last_sync, + req.force, + req.created_at, + ) + .fetch_optional(&db) + .await? + } else { + // Delete, same conflict rule in the WHERE clause. Returns NULL when + // the row was too new (conflict) OR already absent (idempotent) — + // disambiguated below. `legacy` ($7) retargets to the NULL-email row. + sqlx::query_scalar!( + r#"DELETE FROM draft + WHERE workspace_id = $1 + AND email IS NOT DISTINCT FROM (CASE WHEN $7::bool THEN NULL::text ELSE $2 END) + AND path = $3 + AND typ = $4 + AND ($6::bool = true + OR $5::timestamptz IS NULL + OR created_at <= $5::timestamptz) + RETURNING now() as "now!""#, + &w_id, + email, + path, + kind as UserDraftItemKind, + req.last_sync, + req.force, + req.legacy, + ) + .fetch_optional(&db) + .await? + }; + + if let Some(ts) = applied_at { + return Ok(Json(SaveDraftResponse { + status: SaveDraftStatus::Saved, + current_timestamp: ts, + })); + } + + // No row affected: either the row was newer than `last_sync` (conflict), + // or it was a delete with no row present (idempotent ok). Distinguished + // by re-reading. + let existing = sqlx::query_scalar!( + r#"SELECT created_at FROM draft + WHERE workspace_id = $1 + AND email IS NOT DISTINCT FROM (CASE WHEN $5::bool THEN NULL::text ELSE $2 END) + AND path = $3 AND typ = $4"#, + &w_id, + email, + path, + kind as UserDraftItemKind, + req.legacy, ) - .execute(&mut *tx) + .fetch_optional(&db) .await?; - tx.commit().await?; - Ok(format!("deleted draft")) + match existing { + Some(ts) => Ok(Json(SaveDraftResponse { + status: SaveDraftStatus::Conflict, + current_timestamp: ts, + })), + // Delete + nothing-was-there ⇒ report success with server's NOW(). + None => { + let now = sqlx::query_scalar!(r#"SELECT now() as "now!""#) + .fetch_one(&db) + .await?; + Ok(Json(SaveDraftResponse { + status: SaveDraftStatus::Saved, + current_timestamp: now, + })) + } + } } -// async fn get_draft( -// authed: ApiAuthed, -// Extension(user_db): Extension, -// Path((w_id, path)): Path<(String, StripPath)>, -// ) -> JsonResult { -// let path = path.to_path(); -// let mut tx = user_db.begin(&authed).await?; +#[derive(Deserialize, Debug)] +#[serde(rename_all = "snake_case")] +pub enum MigrateLegacyDraftAction { + /// Discard the legacy row entirely. + Delete, + /// Move the legacy row's content onto the authed admin's own row, then + /// drop the legacy row — so it becomes a normal per-user draft. + AssignToSelf, +} -// let script_o = sqlx::query_as!( -// Draft, -// r#"SELECT path, value, typ as "typ: DraftType" FROM draft WHERE path = $1 AND workspace_id = $2"#, -// path, -// w_id -// ) -// .fetch_optional(&mut *tx) -// .await?; -// tx.commit().await?; +#[derive(Deserialize, Debug)] +pub struct MigrateLegacyDraftRequest { + pub action: MigrateLegacyDraftAction, +} -// let draft = not_found_if_none(script_o, "draft", path)?; -// Ok(Json(draft)) -// } +/// Resolve a LEGACY (workspace-level, `email IS NULL`) draft. These predate the +/// per-user drafts migration and have no owner, so only workspace admins (and +/// superadmins, which carry `is_admin` in a workspace) may delete one or claim +/// it as their own. +async fn migrate_legacy_draft( + authed: ApiAuthed, + Extension(db): Extension, + Path((w_id, kind, path)): Path<(String, UserDraftItemKind, windmill_common::utils::StripPath)>, + Json(req): Json, +) -> Result { + if !authed.is_admin { + return Err(Error::NotAuthorized( + "only workspace admins can migrate legacy drafts".to_string(), + )); + } + let path = path.to_path(); + match req.action { + MigrateLegacyDraftAction::Delete => { + sqlx::query!( + r#"DELETE FROM draft + WHERE workspace_id = $1 AND path = $2 AND typ = $3 AND email IS NULL"#, + &w_id, + path, + kind as UserDraftItemKind, + ) + .execute(&db) + .await?; + Ok(format!("Deleted legacy draft at {path}")) + } + MigrateLegacyDraftAction::AssignToSelf => { + // Take ownership: move the legacy value onto the admin's own row + // (replacing any existing own draft) and drop the legacy row, in one + // statement. `ON CONFLICT` matches the partial unique index that + // covers `email IS NOT NULL`. + let moved = sqlx::query_scalar!( + r#"WITH legacy AS ( + DELETE FROM draft + WHERE workspace_id = $1 AND path = $2 AND typ = $3 AND email IS NULL + RETURNING value + ) + INSERT INTO draft (workspace_id, email, path, typ, value, created_at) + SELECT $1, $4, $2, $3, value, now() FROM legacy + ON CONFLICT (workspace_id, path, typ, email) WHERE email IS NOT NULL + DO UPDATE SET value = EXCLUDED.value, created_at = now() + RETURNING 1 as "one!""#, + &w_id, + path, + kind as UserDraftItemKind, + &authed.email, + ) + .fetch_optional(&db) + .await?; + if moved.is_none() { + return Err(Error::NotFound(format!("no legacy draft at {path}"))); + } + Ok(format!("Assigned legacy draft at {path} to you")) + } + } +} + +/// Remove every U+0000 (NUL) from a serialized JSON document so it is safe to +/// store in the `json`-typed `draft.value` (a NUL there would later make any +/// `->>`/`to_jsonb` extraction raise `22P05`). +/// +/// A NUL can only appear in JSON text as a backslash-u0000 escape, and a +/// backslash only ever occurs inside a string, so one backslash-parity-aware +/// pass removes every real NUL escape — covering values and keys alike — while +/// leaving a legitimate `\\u0000` (an escaped backslash followed by the literal +/// text `u0000`) intact. O(n) over the bytes with no `serde_json::Value` tree to +/// allocate, and the fast path (no such substring at all) returns the input +/// untouched. The slow path is reached not only by genuinely poisoned values but +/// by any value that legitimately contains `u0000` after a backslash (e.g. script +/// source), so it must stay allocation-light for potentially large drafts. +fn strip_json_nul(serialized: String) -> String { + if !serialized.contains("\\u0000") { + return serialized; + } + let bytes = serialized.as_bytes(); + let mut out: Vec = Vec::with_capacity(bytes.len()); + let mut i = 0; + while i < bytes.len() { + if bytes[i] != b'\\' { + out.push(bytes[i]); + i += 1; + continue; + } + // Consume the whole run of backslashes. An even run is N/2 escaped + // backslashes and leaves the next char unescaped; an odd run ends in an + // escaping backslash, so a following `u0000` is a real NUL escape. + let run_start = i; + while i < bytes.len() && bytes[i] == b'\\' { + i += 1; + } + let run = i - run_start; + if run % 2 == 1 && bytes[i..].starts_with(b"u0000") { + // Drop the escaping backslash + `u0000`; keep the leading literal pairs. + out.extend(std::iter::repeat(b'\\').take(run - 1)); + i += 5; + } else { + out.extend(std::iter::repeat(b'\\').take(run)); + } + } + // Only whole ASCII backslash-u0000 escapes were removed, so the bytes remain + // valid UTF-8 (and valid JSON). + String::from_utf8(out).expect("removing a NUL escape preserves valid UTF-8") +} + +/// For variable-kind drafts with `variable.is_secret == true`, encrypt +/// `variable.value` with the workspace crypt key and mark it +/// `$encrypted:` so the secret never persists in plaintext at rest. +/// Already-marked values pass through untouched. Unexpected/malformed shapes +/// pass through unchanged — the draft store is schema-less by design. +async fn encrypt_secret_variable_value(db: &DB, w_id: &str, raw: &str) -> Result { + let Ok(mut v) = serde_json::from_str::(raw) else { + return Ok(raw.to_string()); + }; + let is_secret = v + .get("variable") + .and_then(|x| x.get("is_secret")) + .and_then(|x| x.as_bool()) + .unwrap_or(false); + if is_secret { + if let Some(serde_json::Value::String(s)) = + v.get_mut("variable").and_then(|x| x.get_mut("value")) + { + if !s.is_empty() && !s.starts_with(ENCRYPTED_DRAFT_PREFIX) { + let mc = build_crypt(db, w_id).await?; + *s = format!("{ENCRYPTED_DRAFT_PREFIX}{}", encrypt(&mc, s)); + } + } + } + Ok(v.to_string()) +} + +#[derive(Deserialize, Debug)] +pub struct GetDraftQuery { + /// Workspace username of the draft owner. Omit to fetch the legacy + /// NULL-email row, if any. Resolved to an email server-side — emails are + /// not part of the public draft API. + pub username: Option, +} + +#[derive(Serialize, Debug)] +pub struct DraftForUser { + pub value: sqlx::types::Json>, + pub created_at: chrono::DateTime, +} + +/// Fetch a specific user's (or the legacy NULL row's) draft content at a path. +/// Backs the "other users' drafts" banner in editors. The owner is identified +/// by workspace username so emails never reach the client. +async fn get_draft_for_user( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, kind, path)): Path<(String, UserDraftItemKind, windmill_common::utils::StripPath)>, + axum::extract::Query(query): axum::extract::Query, +) -> Result> { + let path = path.to_path(); + // Drawer kinds keep drafts private to their owner (see + // `shares_drafts_across_users`) — also what blocks reading another user's + // secret-variable `$encrypted:` ciphertext. + if !kind.shares_drafts_across_users() { + return Err(Error::NotFound( + "drafts for this item kind are private to their owner".to_string(), + )); + } + require_can_read_path(&authed, &user_db, &w_id, kind, path).await?; + + // Username -> email, scoped to the workspace. None signals "fetch the + // legacy NULL-email row" (distinct from a username with no draft, which + // 404s below). + let owner_email: Option = if let Some(username) = &query.username { + let email = sqlx::query_scalar!( + r#"SELECT email FROM usr WHERE workspace_id = $1 AND username = $2"#, + &w_id, + username, + ) + .fetch_optional(&db) + .await?; + match email { + Some(e) => Some(e), + // The `admins` workspace has no `usr` rows (username IS the email + // there), so accept it as the owner email directly. + None if w_id == "admins" => Some(username.clone()), + None => { + return Err(Error::NotFound(format!( + "no user with username {username} in workspace" + ))) + } + } + } else { + None + }; + + let row = sqlx::query_as!( + DraftForUser, + r#"SELECT value as "value!: sqlx::types::Json>", created_at + FROM draft + WHERE workspace_id = $1 + AND path = $2 + AND typ = $3 + AND email IS NOT DISTINCT FROM $4"#, + &w_id, + path, + kind as UserDraftItemKind, + owner_email, + ) + .fetch_optional(&db) + .await?; + + row.map(Json).ok_or_else(|| { + Error::NotFound(format!( + "no draft for {} at {path}", + query.username.as_deref().unwrap_or("") + )) + }) +} + +/// Fetch the AUTHED user's OWN draft at a path, for any kind — including +/// private kinds (`shares_drafts_across_users() == false`). Backs editors with +/// no deployed-item GET to overlay a draft onto: the `data_pipeline` bundle is +/// keyed at a folder path with no runnable to hang `get_draft` on, so it loads +/// its in-flight state from here. Returns `null` (200) when the user has no +/// draft there, so a fresh pipeline isn't a 404. Secret-variable values come +/// back `$encrypted:`-prefixed, same as `get_draft_for_user` — variable editors +/// use their own overlay GET, not this route. +async fn get_own_draft( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, kind, path)): Path<(String, UserDraftItemKind, windmill_common::utils::StripPath)>, +) -> Result>> { + let path = path.to_path(); + require_can_read_path(&authed, &user_db, &w_id, kind, path).await?; + let row = sqlx::query_as!( + DraftForUser, + r#"SELECT value as "value!: sqlx::types::Json>", created_at + FROM draft + WHERE workspace_id = $1 AND path = $2 AND typ = $3 AND email = $4"#, + &w_id, + path, + kind as UserDraftItemKind, + &authed.email, + ) + .fetch_optional(&db) + .await?; + Ok(Json(row)) +} + +/// The deployed table RLS resolves item-level `extra_perms` against. +/// Delegates to `UserDraftItemKind::deployed_table()` (the shared single +/// source); `None` kinds fall through to the path-only access check. +fn table_for_kind(kind: UserDraftItemKind) -> Option<&'static str> { + kind.deployed_table() +} + +/// Resolves to `Ok(())` if `authed` may SAVE a draft at `path`. Operators are +/// rejected outright. Two layers: +/// 1. Claim-based namespace rules (admin, own `u/`, member `g/`, writable +/// `f/`) — mirror what RLS reads from the same JWT claims, and are the +/// ENTIRE check for draft-only paths (no deployed row for RLS to use). +/// 2. An RLS write-probe on the deployed row (`SELECT ... FOR UPDATE`) for +/// what the path can't answer, above all item-level extra_perms grants. +async fn require_can_write_path( + authed: &ApiAuthed, + db: &DB, + user_db: &UserDB, + w_id: &str, + kind: UserDraftItemKind, + path: &str, +) -> Result<()> { + if authed.is_admin { + return Ok(()); + } + // Operators are read-only and never WRITE drafts. Read access is + // deliberately asymmetric: `require_can_read_path` has no operator block, + // so an operator can still READ a draft they can read via `/drafts/get`, + // mirroring their read access to deployed content. Intended. + if authed.is_operator { + return Err(Error::NotAuthorized( + "operators cannot save drafts".to_string(), + )); + } + // Cheap claim-based namespace checks first: they evaluate the same JWT + // claims RLS reads, so the outcome matches the policies while sparing the + // autosave hot path a DB round-trip. They are also the ENTIRE check for + // draft-only paths (no deployed row for RLS) — without them any member + // could plant a draft in another user's `u/` namespace, surfaced to every + // reader of the path. `require_owner_of_path` covers admin / `u/{own}` / + // folder owner; group membership and the folder WRITE bit are layered on. + if windmill_api_auth::require_owner_of_path(authed, path).is_ok() { + return Ok(()); + } + let parts: Vec<&str> = path.splitn(3, '/').collect(); + if parts.len() >= 3 { + match parts[0] { + "g" if authed.groups.iter().any(|g| g == parts[1]) => return Ok(()), + "f" => { + let folder = parts[1]; + let has_write = |a: &ApiAuthed| { + a.folders + .iter() + .any(|(name, write, owner)| name == folder && (*write || *owner)) + }; + if has_write(authed) { + return Ok(()); + } + let refreshed = + windmill_api_auth::maybe_refresh_folders(path, w_id, authed.clone(), db).await; + if has_write(&refreshed) { + return Ok(()); + } + } + _ => {} + } + } + // Defer to RLS for what the path can't answer (item-level extra_perms + // grants). Postgres applies UPDATE policies to rows locked via `SELECT + // ... FOR UPDATE`, so a returned row means the canonical write policies + // would let this user UPDATE it — no rule re-implemented here. Draft-only + // paths have no row, so the namespace rules above were the whole check. + if let Some(table) = kind.deployed_table() { + // `table` is from the closed enum, never user input. LIMIT 1 keeps the + // probe to one row lock — `script` has a row per version at the path, + // and locking the whole history would serialize against deploys. + let query = format!( + "SELECT 1 FROM {table} WHERE path = $1 AND workspace_id = $2 LIMIT 1 FOR UPDATE" + ); + let mut tx = user_db.clone().begin(authed).await?; + let row = sqlx::query_scalar::<_, i32>(&query) + .bind(path) + .bind(w_id) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + if row.is_some() { + return Ok(()); + } + } + Err(Error::NotAuthorized(format!( + "you don't have write permission on {path}" + ))) +} + +/// Resolves to `Ok(())` if `authed` can read at `path`. Three layers: +/// 1. admin → always. +/// 2. Path-prefix match against own `u/{username}` or any folder in +/// `authed.folders` (the precomputed read set, with groups + direct +/// grants already factored in). +/// 3. RLS-aware `SELECT 1` against the backing table — covers item-level +/// extra_perms grants that bypass folder/owner checks. +/// Both "not readable" and "doesn't exist" return 404 — don't leak existence. +/// +/// Operators are deliberately NOT rejected here (unlike +/// `require_can_write_path`): read-only users keep their read access to +/// deployed content, so an operator can view a collaborator's draft for the +/// cross-user kinds they can already read, while never writing one. Drawer +/// kinds never reach this (`get_draft_for_user` rejects them up front). +async fn require_can_read_path( + authed: &ApiAuthed, + user_db: &UserDB, + w_id: &str, + kind: UserDraftItemKind, + path: &str, +) -> Result<()> { + if authed.is_admin { + return Ok(()); + } + let parts: Vec<&str> = path.splitn(3, '/').collect(); + if parts.len() >= 2 { + match parts[0] { + "u" if parts[1] == authed.username => return Ok(()), + "f" => { + let folder = parts[1]; + if authed.folders.iter().any(|(name, _, _)| name == folder) { + return Ok(()); + } + } + _ => {} + } + } + if let Some(table) = table_for_kind(kind) { + let mut tx = user_db.clone().begin(authed).await?; + let query = format!("SELECT 1 FROM {table} WHERE path = $1 AND workspace_id = $2 LIMIT 1"); + let row = sqlx::query_scalar::<_, i32>(&query) + .bind(path) + .bind(w_id) + .fetch_optional(&mut *tx) + .await?; + tx.commit().await?; + if row.is_some() { + return Ok(()); + } + } + Err(Error::NotFound(format!("no draft visible at {path}"))) +} + + +#[cfg(test)] +mod tests { + use super::strip_json_nul; + + // Parse the (NUL-free) result so assertions read clearly. + fn parsed(s: String) -> serde_json::Value { + serde_json::from_str(&s).expect("strip_json_nul must return valid JSON") + } + + #[test] + fn clean_value_is_returned_byte_for_byte() { + let s = r#"{"summary":"all good","n":1}"#.to_string(); + assert_eq!(strip_json_nul(s.clone()), s); + } + + #[test] + fn real_nul_in_value_is_stripped() { + let out = strip_json_nul(r#"{"summary":"hi\u0000there"}"#.to_string()); + assert!(!out.contains(r"\u0000")); + assert_eq!(parsed(out)["summary"], "hithere"); + } + + #[test] + fn legit_escaped_backslash_is_a_noop() { + // JSON "a\\u0000b" decodes to the 8-char string a,backslash,u,0,0,0,0,b + // — not a NUL — so the value is already clean and round-trips byte-for-byte. + let s = r#"{"summary":"a\\u0000b"}"#.to_string(); + assert_eq!(strip_json_nul(s.clone()), s); + } + + #[test] + fn collision_real_and_literal_both_handled() { + // "a" carries a real NUL; "b" carries the literal text backslash-u0000. + // The value walk strips the former and leaves the latter intact — the + // pathological case that needed a fallback in SQL is trivial in Rust. + let v = parsed(strip_json_nul(r#"{"a":"x\u0000y","b":"p\\u0000q"}"#.to_string())); + assert_eq!(v["a"], "xy"); + assert_eq!(v["b"], "p\\u0000q"); + } + + #[test] + fn nested_values_and_keys_are_cleaned() { + let out = strip_json_nul( + r#"{"o":{"k\u0000":["a\u0000b",{"deep\u0000":"v\u0000"}]}}"#.to_string(), + ); + assert!(!out.contains(r"\u0000")); + let v = parsed(out); + assert_eq!(v["o"]["k"][0], "ab"); + assert_eq!(v["o"]["k"][1]["deep"], "v"); + } + + #[test] + fn odd_backslash_run_keeps_literal_drops_nul() { + // JSON "a\\\u0000b" is an escaped backslash (kept) immediately followed by + // a real NUL escape (dropped) -> decodes to a,backslash,b. + let v = parsed(strip_json_nul(r#"{"x":"a\\\u0000b"}"#.to_string())); + assert_eq!(v["x"], "a\\b"); + } +} diff --git a/backend/windmill-api/src/health.rs b/backend/windmill-api/src/health.rs index fab9979c12..4dc8947de6 100644 --- a/backend/windmill-api/src/health.rs +++ b/backend/windmill-api/src/health.rs @@ -219,12 +219,16 @@ struct DatabaseCheckResult { async fn check_database_with_latency(db: &DB) -> DatabaseCheckResult { let start = std::time::Instant::now(); + // `pg_is_in_recovery()` is true on standbys/read-only replicas, so a primary + // returns true here. A read-only replica (e.g. after a failover where the + // primary became a secondary) reports unhealthy, letting liveness probes + // restart the pod instead of silently failing all writes. let healthy = tokio::time::timeout( HEALTH_CHECK_TIMEOUT, - sqlx::query_scalar!("SELECT 1").fetch_one(db), + sqlx::query_scalar!("SELECT NOT pg_is_in_recovery()").fetch_one(db), ) .await - .map(|r| r.is_ok()) + .map(|r| matches!(r, Ok(Some(true)))) .unwrap_or(false); let latency_ms = start.elapsed().as_millis() as i64; diff --git a/backend/windmill-api/src/inkeep_oss.rs b/backend/windmill-api/src/inkeep_oss.rs deleted file mode 100644 index a34ea91435..0000000000 --- a/backend/windmill-api/src/inkeep_oss.rs +++ /dev/null @@ -1,23 +0,0 @@ -#[cfg(feature = "private")] -#[allow(unused)] -pub use crate::inkeep_ee::*; - -#[cfg(not(feature = "private"))] -use axum::{routing::post, Router}; - -#[cfg(not(feature = "private"))] -use windmill_common::error::Error; - -#[cfg(not(feature = "private"))] -pub fn global_service() -> Router { - Router::new().route("/", post(inkeep_not_available)) -} - -#[cfg(not(feature = "private"))] -async fn inkeep_not_available() -> windmill_common::error::Result<()> { - Err(Error::Generic( - http::StatusCode::FORBIDDEN, - "Inkeep AI documentation assistant is only available in Windmill Enterprise Edition" - .to_string(), - )) -} diff --git a/backend/windmill-api/src/jobs.rs b/backend/windmill-api/src/jobs.rs index 895dbed886..5192d27724 100644 --- a/backend/windmill-api/src/jobs.rs +++ b/backend/windmill-api/src/jobs.rs @@ -34,8 +34,8 @@ use windmill_common::db::UserDbWithAuthed; use windmill_common::error::JsonResult; use windmill_common::flow_status::{JobResult, RestartedFrom}; use windmill_common::jobs::{ - format_completed_job_result, format_result, is_valid_entrypoint_name, DynamicInput, - ENTRYPOINT_OVERRIDE, + format_completed_job_result, format_result, is_safe_log_file_path, is_valid_entrypoint_name, + DynamicInput, ENTRYPOINT_OVERRIDE, }; #[cfg(feature = "run_inline")] use windmill_common::jobs::{ @@ -270,6 +270,7 @@ pub fn workspaced_service() -> Router { ) .route("/run/dynamic_select", post(run_dynamic_select)) .route("/list", get(list_jobs)) + .route("/asset_dispatch_edges", get(list_asset_dispatch_edges)) .route( "/list_selected_job_groups", // We use post because sending a huge array as a query param can produce @@ -389,6 +390,10 @@ pub fn workspace_unauthed_service() -> Router { .route("/get/{id}", get(get_job)) .route("/get_logs/{id}", get(get_job_logs)) .route("/get_flow_all_logs/{id}", get(get_flow_all_logs)) + .route( + "/get_flow_all_logs_structured/{id}", + get(get_flow_all_logs_structured), + ) .route( "/get_completed_logs_tail/{id}", get(get_completed_job_logs_tail), @@ -403,6 +408,7 @@ pub fn workspace_unauthed_service() -> Router { get(get_completed_job_result_maybe), ) .route("/completed/get_timing/{id}", get(get_completed_job_timing)) + .route("/dispatch_events/{id}", get(get_dispatch_events)) .route("/getupdate/{id}", get(get_job_update)) .route("/getupdate_sse/{id}", get(get_job_update_sse)) .route("/get_log_file/{*file_path}", get(get_log_file)) @@ -511,6 +517,26 @@ async fn cancel_job_api( Path((w_id, id)): Path<(String, Uuid)>, Json(CancelJob { reason }): Json, ) -> error::Result { + // App embed tokens (the sandboxed app iframe) may cancel ONLY jobs they launched + // — their app's component runs, stamped created_by == viewer. cancel_job_api has + // no other per-job ownership check, so without this an embed token (which carries + // the viewer's identity) could cancel any job by id. NotFound (not 403) so the + // untrusted app can't probe job existence. + if let Some(authed) = opt_authed.as_ref() { + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + let created_by = sqlx::query_scalar!( + "SELECT created_by FROM v2_job WHERE id = $1 AND workspace_id = $2", + id, + &w_id + ) + .fetch_optional(&db) + .await?; + if created_by.as_deref() != Some(authed.username.as_str()) { + return Err(Error::NotFound(format!("Job {id} not found"))); + } + } + } + let tx = db.begin().await?; let audit_author: AuditAuthor = match opt_authed.as_ref() { @@ -1005,6 +1031,17 @@ async fn require_job_read_access( return Ok(()); } + // App embed tokens (the sandboxed app iframe) carry the viewer's identity so the + // app can read its own component runs — which are stamped `created_by == viewer` + // and so already returned above. They must NOT inherit the viewer's *broader* + // job access (share links, folder ACLs, admin RLS): user-authored app JS holds + // this token, and letting it reach any job merely visible to the viewer would + // expose unrelated runs' results/logs. Stop at the launched-by-viewer grant. + // NotFound (not PermissionDenied) so the untrusted app can't probe job existence. + if windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref()) { + return Err(Error::NotFound(format!("Job {job_id} not found"))); + } + // `username_override` is derived from the token *label* (`username_override_from_label`), // which is fully user-controlled with no uniqueness/ownership check (webhook-/http-/ // email-/ws- trigger tokens, `ephemeral-script-end-user-*`, and the generic `label-*` @@ -1386,6 +1423,7 @@ macro_rules! get_job_query { @impl "v2_job_completed", ($($opts)*), "v2_job_completed.duration_ms, v2_job_completed.completed_at, CASE WHEN status = 'success' OR status = 'skipped' THEN true ELSE false END as success, result_columns, deleted, status = 'skipped' as is_skipped, \ v2_job.labels, \ + EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry, \ CASE WHEN result is null or pg_column_size(result) < 90000 THEN result ELSE '\"WINDMILL_TOO_BIG\"'::jsonb END as result", "", ) @@ -1395,7 +1433,8 @@ macro_rules! get_job_query { @impl "v2_job_queue", ($($opts)*), "scheduled_for, running, ping as last_ping, suspend, suspend_until, same_worker, pre_run_error, visible_to_owner, \ flow_innermost_root_job AS root_job, flow_leaf_jobs AS leaf_jobs, concurrent_limit, concurrency_time_window_s, timeout, flow_step_id, cache_ttl, cache_ignore_s3_path, runnable_settings_handle, \ - script_entrypoint_override, v2_job.labels", + script_entrypoint_override, v2_job.labels, \ + EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = v2_job.id) as is_retry", "LEFT JOIN v2_job_runtime ON v2_job_runtime.id = v2_job_queue.id LEFT JOIN v2_job_status ON v2_job_status.id = v2_job_queue.id", ) }; @@ -1912,12 +1951,17 @@ async fn get_logs_from_disk( if log_offset > 0 { if let Some(file_index) = log_file_index.clone() { for file_p in &file_index { - if !tokio::fs::metadata(format!("{}/{file_p}", *WINDMILL_DIR)) - .await - .is_ok() - { + if !is_safe_log_file_path(file_p) { return None; } + let local_file = format!("{}/{file_p}", *WINDMILL_DIR); + // Defense in depth: refuse to read through a symlink so a planted + // symlink under the log directory cannot exfiltrate arbitrary files. + match tokio::fs::symlink_metadata(&local_file).await { + Ok(meta) if meta.file_type().is_symlink() => return None, + Ok(_) => {} + Err(_) => return None, + } } let logs = logs.to_string(); @@ -2170,14 +2214,40 @@ async fn resolve_logs_to_string( logs.to_string() } -async fn get_flow_all_logs( - OptViewToken(view_token): OptViewToken, - OptAuthed(opt_authed): OptAuthed, +/// A single job in a flow's execution tree, with its resolved logs and a +/// human-readable label describing its position (iteration, branch, subflow…). +#[derive(Serialize)] +struct FlowLogEntry { + job_id: String, + /// Human-readable label, e.g. "Step a (iteration 2/3)" or "Flow". + label: String, + /// Job kind (script, flow, forloopflow, …). + kind: String, + /// The flow step id this job corresponds to, if any. + flow_step_id: Option, + /// Materialized step path (e.g. "a/b") used to locate the step in the flow. + step_path: Option, + /// Depth in the flow tree (0 for the root flow job). + depth: i32, + /// The parent module type (forloopflow, branchall, …), if any. + parent_module_type: Option, + /// 1-based index of this job among its siblings sharing the same step. + sibling_index: i32, + /// Total number of siblings sharing the same step. + sibling_count: i32, + /// Resolved logs for this job (pulled from disk/object store as needed). + logs: String, +} + +async fn collect_flow_log_entries( + view_token: Option, + opt_authed: Option, opt_tokened: OptTokened, - Extension(db): Extension, - Extension(user_db): Extension, - Path((w_id, id)): Path<(String, Uuid)>, -) -> error::Result { + db: &DB, + user_db: &UserDB, + w_id: &str, + id: Uuid, +) -> error::Result> { let tags = opt_authed .as_ref() .map(|authed| get_scope_tags(authed).map(|v| v.iter().map(|s| s.to_string()).collect_vec())) @@ -2190,17 +2260,17 @@ async fn get_flow_all_logs( w_id, tags.as_ref().map(|v| v.as_slice()) ) - .fetch_optional(&db) + .fetch_optional(db) .await?; let root_job = not_found_if_none(root_job, "Job", id.to_string())?; if let Some(authed) = opt_authed.as_ref() { require_job_read_access( - &db, - &user_db, + db, + user_db, authed, - &w_id, + w_id, &id, &root_job.created_by, view_token.as_deref(), @@ -2213,10 +2283,10 @@ async fn get_flow_all_logs( } log_job_view( - &db, + db, opt_authed.as_ref(), opt_tokened.token.as_deref(), - &w_id, + w_id, &id, ) .await?; @@ -2283,10 +2353,10 @@ async fn get_flow_all_logs( w_id, id, ) - .fetch_all(&db) + .fetch_all(db) .await?; - let mut all_logs = String::new(); + let mut entries = Vec::with_capacity(records.len()); for record in &records { let kind = record.kind.as_deref().unwrap_or(""); @@ -2349,18 +2419,89 @@ async fn get_flow_all_logs( }; let job_id = record.id.map(|u| u.to_string()).unwrap_or_default(); - all_logs.push_str(&format!("\n=== {} (Job: {}) ===\n", label, job_id)); let logs = record.logs.as_deref().unwrap_or(""); let resolved = resolve_logs_to_string(record.log_offset.unwrap_or(0), logs, &record.log_file_index) .await; - all_logs.push_str(&resolved); + + entries.push(FlowLogEntry { + job_id, + label, + kind: kind.to_string(), + flow_step_id: record.flow_step_id.clone(), + step_path: record.path_label.clone(), + depth, + parent_module_type: if parent_module_type.is_empty() { + None + } else { + Some(parent_module_type.to_string()) + }, + sibling_index, + sibling_count, + logs: resolved, + }); + } + + Ok(entries) +} + +async fn get_flow_all_logs( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + opt_tokened: OptTokened, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> error::Result { + let entries = collect_flow_log_entries( + view_token, + opt_authed, + opt_tokened, + &db, + &user_db, + &w_id, + id, + ) + .await?; + + let mut all_logs = String::new(); + for entry in &entries { + all_logs.push_str(&format!( + "\n=== {} (Job: {}) ===\n", + entry.label, entry.job_id + )); + all_logs.push_str(&entry.logs); all_logs.push('\n'); } Ok(content_plain(Body::from(all_logs))) } +/// Structured alternative to `get_flow_all_logs`: returns the same flow log +/// tree as a JSON array of entries (one per job) instead of a flat text blob, +/// so callers can render or process logs per-step without parsing delimiters. +async fn get_flow_all_logs_structured( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + opt_tokened: OptTokened, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> JsonResult> { + let entries = collect_flow_log_entries( + view_token, + opt_authed, + opt_tokened, + &db, + &user_db, + &w_id, + id, + ) + .await?; + + Ok(Json(entries)) +} + async fn get_args( OptViewToken(view_token): OptViewToken, OptAuthed(opt_authed): OptAuthed, @@ -3197,6 +3338,12 @@ struct ApprovalInfo { #[serde(skip_serializing_if = "Option::is_none")] hide_cancel: Option, approvers: Vec, + /// Share-read-link token for the flow, minted only for callers allowed to view this + /// approval. Lets an authenticated workspace-member approver open the run details of + /// a flow they don't otherwise have read access to (the run page reads it as a + /// `view_token` query param). + #[serde(skip_serializing_if = "Option::is_none")] + view_token: Option, } /// Whether `opt_authed` is allowed to approve — and therefore view — this approval step. @@ -3419,6 +3566,7 @@ async fn get_approval_info( user_auth_required, hide_cancel: None, approvers: vec![], + view_token: None, })); } @@ -3436,6 +3584,12 @@ async fn get_approval_info( }) .collect(); + // Possession of view rights over this approval is sufficient to mint a + // share-read-link token for the flow: it only grants read (no resume), and only to + // an authenticated workspace member, so it never widens what the approver can do. + let hmac = generate_view_token(&w_id, row.id, &db).await?; + let view_token = Some(format!("{}.{hmac}", row.id)); + Ok(Json(ApprovalInfo { flow_id: row.id, form_schema, @@ -3447,6 +3601,7 @@ async fn get_approval_info( user_auth_required, hide_cancel, approvers, + view_token, })) } @@ -3841,6 +3996,12 @@ pub async fn cancel_suspended_job( pub struct SuspendedJobFlow { pub job: Job, pub approvers: Vec, + /// Share-read-link token for the parent flow, minted because the caller proved + /// possession of the approval secret. Lets an authenticated workspace-member + /// approver open the run details of a flow they don't otherwise have read access + /// to (the run page reads it as a `view_token` query param). + #[serde(skip_serializing_if = "Option::is_none")] + pub view_token: Option, } pub async fn get_suspended_job_flow( @@ -3925,7 +4086,13 @@ pub async fn get_suspended_job_flow( ) .await?; - Ok(Json(SuspendedJobFlow { job: flow, approvers }).into_response()) + // Possession of a valid approval secret is sufficient to mint a share-read-link + // token for the parent flow: it only grants read (no resume), and only to an + // authenticated workspace member, so it never widens what the approver can do. + let hmac = generate_view_token(&w_id, flow_id, &db).await?; + let view_token = Some(format!("{flow_id}.{hmac}")); + + Ok(Json(SuspendedJobFlow { job: flow, approvers, view_token }).into_response()) } fn conditionally_require_authed_user( @@ -3989,11 +4156,17 @@ fn conditionally_require_authed_user( } pub async fn create_job_signature( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>, Query(approver): Query, ) -> error::Result { + // The HMAC is treated as full authority by the resume endpoints, so minting + // it requires run scope on the suspended job's flow — not merely any + // jobs:run scope. No-op for unscoped tokens (incl. the in-flow substep token + // used by wmill.get_resume_urls()). + let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?; + check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?; let key = get_workspace_key(&w_id, &db).await?; create_signature(key, job_id, resume_id, approver.approver) } @@ -4076,11 +4249,17 @@ fn build_resume_url( } pub async fn get_resume_urls( - _authed: ApiAuthed, + authed: ApiAuthed, Extension(db): Extension, Path((w_id, job_id, resume_id)): Path<(String, Uuid, u32)>, Query(approver): Query, ) -> error::JsonResult { + // These URLs embed a resume signature (full resume capability), so a scoped + // token must hold run scope on the suspended job's flow. No-op for unscoped + // tokens (incl. the in-flow substep token). Trusted internal callers use + // get_resume_urls_internal directly and are unaffected. + let flow_path = resume_target_flow_path(&db, &w_id, job_id).await?; + check_scopes(&authed, || format!("jobs:run:flows:{}", flow_path))?; get_resume_urls_internal( Extension(db), Path((w_id, job_id, resume_id)), @@ -4147,6 +4326,46 @@ pub async fn get_resume_urls_internal( Ok(Json(res)) } +/// Resolve the runnable path of the flow a (possibly step) job belongs to, used +/// to scope-check resume-signature minting against `jobs:run:flows:`. +/// Returns an empty string when the path can't be resolved (e.g. previews or an +/// unknown job); an empty path only matters for path-restricted tokens, which +/// would not be running such a flow. Never hard-fails, so it can't break resume +/// for unscoped tokens (the in-flow `get_resume_urls()` path). +async fn resume_target_flow_path(db: &DB, w_id: &str, job_id: Uuid) -> error::Result { + let job = sqlx::query!( + r#"SELECT kind::text as "kind!", parent_job, runnable_path + FROM v2_job WHERE id = $1 AND workspace_id = $2"#, + job_id, + w_id + ) + .fetch_optional(db) + .await?; + let Some(job) = job else { + return Ok(String::new()); + }; + // All flow kinds: the job itself is the flow whose path scopes the resume. + if matches!( + job.kind.as_str(), + "flow" | "flowpreview" | "flownode" | "singlestepflow" + ) { + return Ok(job.runnable_path.unwrap_or_default()); + } + // Otherwise it's a step; its parent is the flow. + if let Some(parent) = job.parent_job { + return Ok(sqlx::query_scalar!( + "SELECT runnable_path FROM v2_job WHERE id = $1 AND workspace_id = $2", + parent, + w_id + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or_default()); + } + Ok(job.runnable_path.unwrap_or_default()) +} + /// Get the flow ID for a job. If the job is a flow, returns the job_id. /// If the job is a step in a flow, returns the parent flow ID. async fn get_flow_id_for_job(db: &DB, job_id: Uuid) -> error::Result { @@ -6315,8 +6534,14 @@ async fn run_inline_preview_script( Path(w_id): Path, Json(preview): Json, ) -> error::Result { - // Same arbitrary-code class as run_preview_script: a narrowly-scoped token - // must not be able to run request-supplied code through inline preview. + // Same arbitrary-code class as run_preview_script: operators are blocked from + // running request-supplied code, and a narrowly-scoped token must not escape + // its scope through inline preview. + if authed.is_operator { + return Err(error::Error::NotAuthorized( + "Operators cannot run preview jobs for security reasons".to_string(), + )); + } check_scopes(&authed, || format!("jobs:run"))?; if let Some(job_id) = job_id { register_potential_assets_on_inline_execution(job_id, &w_id, &preview); @@ -9021,6 +9246,198 @@ struct JobTiming { duration_ms: Option, } +/// One row of the producer's "Dispatch" panel — what the asset-trigger +/// dispatcher decided for a single (subscriber, asset write) pair. See +/// `windmill_queue::asset_dispatch` for the writer and the discriminants +/// of the `outcome` / `reason` fields. +#[derive(Serialize)] +struct DispatchEvent { + subscriber_path: String, + asset_kind: windmill_common::assets::AssetKind, + asset_path: String, + outcome: String, + #[serde(skip_serializing_if = "Option::is_none")] + child_job_id: Option, + #[serde(skip_serializing_if = "Option::is_none")] + partition: Option, + #[serde(skip_serializing_if = "Option::is_none")] + received_inputs: Option, + #[serde(skip_serializing_if = "Option::is_none")] + required_inputs: Option, + #[serde(skip_serializing_if = "Option::is_none")] + debounce_s: Option, + #[serde(skip_serializing_if = "Option::is_none")] + reason: Option, + created_at: chrono::DateTime, +} + +async fn get_dispatch_events( + OptViewToken(view_token): OptViewToken, + OptAuthed(opt_authed): OptAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, Uuid)>, +) -> error::JsonResult> { + let tags = opt_authed + .as_ref() + .map(|authed| get_scope_tags(authed)) + .flatten(); + + // Gate on the producer job's visibility, exactly like + // get_completed_job_timing on the same unauthed router: scope tags + // first, then per-job read access for authed users, anonymous-only + // jobs otherwise. The dispatch_event FK to v2_job(id) guarantees the + // producer row exists for any extant event. + let producer = sqlx::query!( + r#"SELECT created_by AS "created_by!" + FROM v2_job + WHERE id = $1 AND workspace_id = $2 AND ($3::text[] IS NULL OR tag = ANY($3))"#, + id, + &w_id, + tags.as_ref().map(|v| v.as_slice()) as Option<&[&str]>, + ) + .fetch_optional(&db) + .await?; + let producer = not_found_if_none(producer, "Job", id.to_string())?; + + if let Some(authed) = opt_authed.as_ref() { + require_job_read_access( + &db, + &user_db, + authed, + &w_id, + &id, + &producer.created_by, + view_token.as_deref(), + ) + .await?; + } else if producer.created_by != "anonymous" { + return Err(Error::BadRequest( + "As a non logged in user, you can only see jobs ran by anonymous users".to_string(), + )); + } + + let rows = sqlx::query!( + r#"SELECT + subscriber_path AS "subscriber_path!", + asset_kind AS "asset_kind!: windmill_common::assets::AssetKind", + asset_path AS "asset_path!", + outcome::text AS "outcome!", + child_job_id, + partition, + received_inputs, + required_inputs, + debounce_s, + reason, + created_at AS "created_at!" + FROM dispatch_event + WHERE producer_job_id = $1 AND workspace_id = $2 + ORDER BY id"#, + id, + &w_id, + ) + .fetch_all(&db) + .await?; + + Ok(Json( + rows.into_iter() + .map(|r| DispatchEvent { + subscriber_path: r.subscriber_path, + asset_kind: r.asset_kind, + asset_path: r.asset_path, + outcome: r.outcome, + child_job_id: r.child_job_id, + partition: r.partition, + received_inputs: r.received_inputs, + required_inputs: r.required_inputs, + debounce_s: r.debounce_s, + reason: r.reason, + created_at: r.created_at, + }) + .collect(), + )) +} + +/// One asset-cascade dispatch record, for reconstructing the cascade graph of a +/// pipeline folder in the Activity panel. `dispatched` rows carry the resolved +/// `child_job_id` (a real producer→child job edge); `join_pending` rows are the +/// pre-completion inputs of an AND-join (no child yet) — the client links them +/// to the eventual child of the same `subscriber_path` so a join's separate +/// trigger chains merge into one group. `skipped` is omitted. +#[derive(Serialize)] +struct AssetDispatchEdge { + producer_job_id: Uuid, + #[serde(skip_serializing_if = "Option::is_none")] + child_job_id: Option, + subscriber_path: String, + outcome: String, + asset_kind: windmill_common::assets::AssetKind, + asset_path: String, + created_at: chrono::DateTime, +} + +#[derive(Deserialize)] +struct AssetDispatchEdgesQuery { + /// Folder path prefix the children live under, e.g. `f/orders/`. Matched + /// against `subscriber_path` — every intra-pipeline cascade edge has its + /// child in the folder, so this captures the whole folder's cascades. + path_start: String, + /// Only edges dispatched at/after this instant (align with the activity + /// window the client already loaded). Omit for the default cap. + created_after: Option>, +} + +/// Asset-cascade edges for a pipeline folder. RLS on the joined `v2_job` +/// producer row limits this to cascades whose producer the caller can already +/// see (same visibility as the folder's job list). +async fn list_asset_dispatch_edges( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> error::JsonResult> { + let like = format!("{}%", query.path_start); + let mut tx = user_db.begin(&authed).await?; + let rows = sqlx::query!( + r#"SELECT + de.producer_job_id AS "producer_job_id!", + de.child_job_id, + de.subscriber_path AS "subscriber_path!", + de.outcome::text AS "outcome!", + de.asset_kind AS "asset_kind!: windmill_common::assets::AssetKind", + de.asset_path AS "asset_path!", + de.created_at AS "created_at!" + FROM dispatch_event de + JOIN v2_job pj ON pj.id = de.producer_job_id + WHERE de.workspace_id = $1 + AND de.outcome IN ('dispatched', 'join_pending') + AND de.subscriber_path LIKE $2 + AND ($3::timestamptz IS NULL OR de.created_at >= $3) + ORDER BY de.created_at DESC, de.id DESC + LIMIT 4000"#, + &w_id, + like, + query.created_after, + ) + .fetch_all(&mut *tx) + .await?; + tx.commit().await?; + + Ok(Json( + rows.into_iter() + .map(|r| AssetDispatchEdge { + producer_job_id: r.producer_job_id, + child_job_id: r.child_job_id, + subscriber_path: r.subscriber_path, + outcome: r.outcome, + asset_kind: r.asset_kind, + asset_path: r.asset_path, + created_at: r.created_at, + }) + .collect(), + )) +} + async fn get_completed_job_timing( OptViewToken(view_token): OptViewToken, OptAuthed(opt_authed): OptAuthed, @@ -9257,16 +9674,31 @@ mod approval_view_gate_tests { fn anonymous_cannot_view_when_auth_required() { // The regression: an unauthenticated holder of the approval token must see nothing. let c = Some(conds(true, vec![])); - assert!(!can_view(&None, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(!can_view( + &None, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); } #[test] fn anonymous_can_view_when_no_auth_required() { // Unchanged behaviour: token alone is sufficient when auth isn't required. let c = Some(conds(false, vec![])); - assert!(can_view(&None, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &None, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); // No approval conditions at all also allows token-only view. - assert!(can_view(&None, &None, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &None, + &None, + Some("f/team/flow"), + "trigger@example.com" + )); } #[test] @@ -9291,7 +9723,17 @@ mod approval_view_gate_tests { let member = Some(authed("carol", false, vec!["approvers".to_string()])); let outsider = Some(authed("dave", false, vec!["other".to_string()])); // Use a non-owned folder path so ownership doesn't short-circuit the check. - assert!(can_view(&member, &c, Some("f/team/flow"), "trigger@example.com")); - assert!(!can_view(&outsider, &c, Some("f/team/flow"), "trigger@example.com")); + assert!(can_view( + &member, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); + assert!(!can_view( + &outsider, + &c, + Some("f/team/flow"), + "trigger@example.com" + )); } } diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index 787e0ceb93..cbbb406db4 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -66,6 +66,7 @@ use crate::scim_oss::has_scim_token; use windmill_common::error::AppError; mod ai; +mod ai_skills; mod apps; pub mod args; mod audit; @@ -77,8 +78,9 @@ mod capture; mod concurrency_groups; mod db; mod db_health; - +mod docs; mod drafts; + #[cfg(feature = "private")] pub mod ee; pub mod ee_oss; @@ -94,9 +96,6 @@ mod health; #[cfg(feature = "private")] pub mod indexer_ee; mod indexer_oss; -#[cfg(feature = "private")] -mod inkeep_ee; -mod inkeep_oss; mod integration; mod internal_db; mod live_migrations; @@ -545,7 +544,15 @@ pub async fn run_server( Router::new() // Reordered alphabetically .nest("/acls", granular_acls::workspaced_service()) - .nest("/apps", apps::workspaced_service(request_size_limit * 5)) + // CORS so the opaque-origin in-workspace app viewer (WIN-2006, + // sandboxed /apps/get) can read the app definition by path + // (apps/get/p, apps/embed_token/p) with a scoped embed token. + // Bearer-token-only (no cookies), consistent with the other + // workspaced services the iframe calls. + .nest( + "/apps", + apps::workspaced_service(request_size_limit * 5).layer(cors.clone()), + ) .nest("/assets", windmill_api_assets::workspaced_service()) .nest("/audit", audit::workspaced_service()) .nest("/capture", capture::workspaced_service()) @@ -553,8 +560,8 @@ pub async fn run_server( "/concurrency_groups", concurrency_groups::workspaced_service(), ) - .nest("/embeddings", embeddings::workspaced_service()) .nest("/drafts", drafts::workspaced_service()) + .nest("/embeddings", embeddings::workspaced_service()) .nest("/favorites", favorite::workspaced_service()) .nest("/flows", flows::workspaced_service()) .nest( @@ -565,7 +572,13 @@ pub async fn run_server( "/flow_conversations", windmill_api_flow_conversations::workspaced_service(), ) - .nest("/folders", folders::workspaced_service()) + // CORS so an opaque-origin app iframe (WIN-2006 embed, + // no separate domain) can read folders/listnames with a + // scoped embed token. Consistent with apps_u/jobs_u cors. + .nest( + "/folders", + folders::workspaced_service().layer(cors.clone()), + ) .nest("/folders_history", folder_history::workspaced_service()) .nest("/groups", groups::workspaced_service()) .nest("/groups_history", group_history::workspaced_service()) @@ -608,20 +621,30 @@ pub async fn run_server( Router::new() }) .nest("/ai", ai::workspaced_service()) + .nest("/ai_skills", ai_skills::workspaced_service()) .nest("/npm_proxy", windmill_api_npm_proxy::workspaced_service()) .nest( "/path_autocomplete", path_autocomplete::workspaced_service(), ) .nest("/raw_apps", raw_apps::workspaced_service()) - .nest("/resources", resources::workspaced_service()) + // CORS so the opaque-origin app iframe can read + // resources/list, resources/type/* with a scoped token. + .nest( + "/resources", + resources::workspaced_service().layer(cors.clone()), + ) .nest("/shared_ui", workspace_shared_ui::workspaced_service()) .nest("/schedules", windmill_api_schedule::workspaced_service()) .nest("/scripts", scripts::workspaced_service()) .nest("/trash", trash::workspaced_service()) .nest( "/users", - users::workspaced_service().layer(Extension(argon2.clone())), + // CORS so the opaque-origin app iframe can read + // users/whoami with a scoped embed token. + users::workspaced_service() + .layer(Extension(argon2.clone())) + .layer(cors.clone()), ) .nest("/variables", variables::workspaced_service()) .nest("/volumes", volumes_oss::workspaced_service()) @@ -662,7 +685,7 @@ pub async fn run_server( .nest("/schedules", windmill_api_schedule::global_service()) .nest("/embeddings", embeddings::global_service()) .nest("/ai", ai::global_service()) - .nest("/inkeep", inkeep_oss::global_service()) + .nest("/docs", docs::global_service()) .nest("/indexer", indexer_oss::management_service()) .nest("/mcp/w/{workspace_id}/list_tools", mcp_list_tools_service) .nest("/db_health", db_health::global_service()) @@ -727,7 +750,11 @@ pub async fn run_server( .nest("/apps_u", { #[cfg(feature = "enterprise")] { - apps_oss::global_unauthed_service() + // CORS so the opaque-origin app viewer (WIN-2006 embed, no + // separate domain) can load a custom-path public app via + // public_app_by_custom_path cross-origin. Consistent with + // the workspaced /w/{workspace_id}/apps_u mount below. + apps_oss::global_unauthed_service().layer(cors.clone()) } #[cfg(not(feature = "enterprise"))] diff --git a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs index fc3b1164f5..d44484f9db 100644 --- a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs +++ b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs @@ -7,25 +7,53 @@ use windmill_mcp::server::EndpointTool; pub fn all_tools() -> Vec { vec![ EndpointTool { - name: Cow::Borrowed("queryDocumentation"), - description: Cow::Borrowed("query Windmill AI documentation assistant (EE only)"), + name: Cow::Borrowed("searchDocs"), + description: Cow::Borrowed("Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more."), instructions: Cow::Borrowed(""), - path: Cow::Borrowed("/inkeep"), - method: Cow::Borrowed("POST"), + path: Cow::Borrowed("/docs/search"), + method: Cow::Borrowed("GET"), path_params_schema: None, - query_params_schema: None, - body_schema: Some(serde_json::json!({ + query_params_schema: Some(serde_json::json!({ "type": "object", "properties": { "query": { "type": "string", - "description": "The documentation query to send to the AI assistant" + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better." } }, "required": [ "query" ] })), + body_schema: None, + path_field_renames: None, + query_field_renames: None, + body_field_renames: None, + }, + EndpointTool { + name: Cow::Borrowed("readDocsPage"), + description: Cow::Borrowed("Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section."), + instructions: Cow::Borrowed(""), + path: Cow::Borrowed("/docs/page"), + method: Cow::Borrowed("GET"), + path_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + }, + "section": { + "type": "string", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page." + } + }, + "required": [ + "url" + ] +})), + body_schema: None, path_field_renames: None, query_field_renames: None, body_field_renames: None, @@ -84,6 +112,9 @@ pub fn all_tools() -> Vec { "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -169,6 +200,9 @@ pub fn all_tools() -> Vec { "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the variable (body parameter)" @@ -210,6 +244,10 @@ pub fn all_tools() -> Vec { "include_encrypted": { "type": "boolean", "description": "ask to include the encrypted value if secret and decrypt secret is not true (default: false)\n" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -260,6 +298,10 @@ pub fn all_tools() -> Vec { "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft variables whose path has no\ndeployed variable. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge.\n" } }, "required": [] @@ -309,6 +351,9 @@ pub fn all_tools() -> Vec { "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -383,6 +428,9 @@ pub fn all_tools() -> Vec { "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the resource (body parameter)" @@ -414,7 +462,16 @@ pub fn all_tools() -> Vec { "path" ] })), - query_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +})), body_schema: None, path_field_renames: None, query_field_renames: None, @@ -469,6 +526,10 @@ pub fn all_tools() -> Vec { "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft resources whose path has\nno deployed resource. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] @@ -593,7 +654,7 @@ pub fn all_tools() -> Vec { name: Cow::Borrowed("createScript"), description: Cow::Borrowed("create script: Creates a new script when the path does not already exist. Creates a new version of an existing script when called with the same path and the current `parent_hash`"), - instructions: Cow::Borrowed("To create a script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed."), + instructions: Cow::Borrowed("To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history."), path: Cow::Borrowed("/w/{workspace}/scripts/create"), method: Cow::Borrowed("POST"), path_params_schema: None, @@ -604,6 +665,9 @@ Creates a new version of an existing script when called with the same path and t "path": { "type": "string" }, + "parent_hash": { + "type": "string" + }, "summary": { "type": "string" }, @@ -716,6 +780,10 @@ Creates a new version of an existing script when called with the same path and t "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -844,6 +912,10 @@ Creates a new version of an existing script when called with the same path and t "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -1181,6 +1253,14 @@ Creates a new version of an existing script when called with the same path and t "language" ] } + }, + "temp_script_refs": { + "type": "object", + "nullable": true, + "description": "Map of relative-import script path -> temp storage hash so the preview job resolves those imports from not-yet-deployed local content instead of the deployed script", + "additionalProperties": { + "type": "string" + } } }, "required": [ @@ -1453,6 +1533,10 @@ Creates a new version of an existing script when called with the same path and t "type": "boolean", "description": "filter on successful jobs" }, + "status": { + "type": "string", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.. Possible values: success, failure, canceled, skipped" + }, "all_workspaces": { "type": "boolean", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)" @@ -1461,6 +1545,10 @@ Creates a new version of an existing script when called with the same path and t "type": "boolean", "description": "is not a scheduled job" }, + "excludes_entrypoint_override": { + "type": "boolean", + "description": "exclude jobs that were started with a `_ENTRYPOINT_OVERRIDE` arg (e.g. dynamic-select helper runs and preprocessor previews)" + }, "broad_filter": { "type": "string", "description": "broad search across multiple fields (case-insensitive substring match on path, tag, schedule path, trigger kind, label)" @@ -1499,6 +1587,10 @@ Creates a new version of an existing script when called with the same path and t }, "no_code": { "type": "boolean" + }, + "approval_token": { + "type": "string", + "description": "Approval token granting read access to the job when not logged in. The token must be the one issued for this job's flow (i.e. the flow id used when generating the approval URL)." } }, "required": [] @@ -1556,7 +1648,7 @@ You should get the schema of the script or flow before creating the schedule to "properties": { "path": { "type": "string", - "description": "The unique path identifier for this schedule" + "description": "The unique Windmill path for this schedule. Must be of the form `u//` or `f//`." }, "schedule": { "type": "string", @@ -1999,7 +2091,16 @@ You should get the schema of the script or flow before updating the schedule to "path" ] })), - query_params_schema: None, + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +})), body_schema: None, path_field_renames: None, query_field_renames: None, @@ -2058,6 +2159,10 @@ You should get the schema of the script or flow before updating the schedule to "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft schedules whose path has\nno deployed schedule. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] diff --git a/backend/windmill-api/src/mcp/oauth_server.rs b/backend/windmill-api/src/mcp/oauth_server.rs index 7cfabd2319..c5142c3031 100644 --- a/backend/windmill-api/src/mcp/oauth_server.rs +++ b/backend/windmill-api/src/mcp/oauth_server.rs @@ -18,6 +18,7 @@ use windmill_common::{ }; use crate::db::ApiAuthed; +use windmill_mcp::parse_mcp_scopes; /// Token expiration for MCP OAuth tokens (1 week in seconds) const MCP_OAUTH_TOKEN_EXPIRATION_SECS: u64 = 7 * 24 * 60 * 60; @@ -585,6 +586,8 @@ async fn handle_refresh_token_grant( Some(&new_access_token) }; let new_refresh_token = rd_string(32); + // Re-issues the already-approved (hence already-contained) scopes verbatim; + // containment is enforced once at approval time, so no re-check here. let scopes = token_row.scopes; // Create new access token (rejects archived workspaces inline) @@ -820,6 +823,40 @@ async fn oauth_approve_inner( .map(|s| s.to_string()) .collect(); + // The approver's own token bounds what it may grant: a scope-restricted MCP + // token must not approve a broader one (e.g. mcp:scripts:f/x -> mcp:all). An + // unrestricted approver (interactive session, scopes None) grants freely, + // which is the normal consent flow. This is the legitimate MCP-narrowing + // path, so it uses MCP-pattern containment rather than the byte-identical + // rule ensure_scopes_within_caller applies on the generic token endpoints. + let caller_restricted = authed + .scopes + .as_deref() + .is_some_and(|s| s.iter().any(|x| !x.starts_with("if_jobs:filter_tags:"))); + if caller_restricted { + // An empty grant would mint a token the auth layer treats as unscoped + // (full privileges), so a restricted approver must not produce one. + if scopes.is_empty() { + return Err(Error::NotAuthorized( + "A scope-restricted token cannot approve an empty scope grant".to_string(), + )); + } + if scopes.iter().any(|s| !s.starts_with("mcp:")) { + return Err(Error::NotAuthorized( + "A scope-restricted token can only approve MCP (mcp:*) scopes".to_string(), + )); + } + let caller_config = parse_mcp_scopes(authed.scopes.as_deref().unwrap_or(&[])) + .map_err(|e| Error::InternalErr(format!("Failed to parse caller MCP scopes: {e}")))?; + let requested_config = parse_mcp_scopes(&scopes) + .map_err(|e| Error::BadRequest(format!("Failed to parse requested MCP scopes: {e}")))?; + if !caller_config.contains(&requested_config) { + return Err(Error::NotAuthorized( + "Requested scopes exceed the approving token's own MCP scopes".to_string(), + )); + } + } + sqlx::query!( "INSERT INTO mcp_oauth_server_code (code, client_id, user_email, workspace_id, scopes, redirect_uri, code_challenge, code_challenge_method) diff --git a/backend/windmill-api/src/mcp/utils.rs b/backend/windmill-api/src/mcp/utils.rs index c7a1b291dd..2a0c34ae8d 100644 --- a/backend/windmill-api/src/mcp/utils.rs +++ b/backend/windmill-api/src/mcp/utils.rs @@ -41,7 +41,6 @@ pub async fn get_item_schema( sqlb.and_where("o.path = ?".bind(&path)); sqlb.and_where("o.workspace_id = ?".bind(&workspace_id)); sqlb.and_where("o.archived = false"); - sqlb.and_where("o.draft_only IS NOT TRUE"); let sql = sqlb.sql().map_err(|e| { tracing::error!("failed to build sql: {}", e); ErrorData::internal_error(format!("failed to build sql: {}", e), None) @@ -147,8 +146,7 @@ pub async fn get_items sqlx::FromRow<'a, sqlx::postgres::PgRow> + Sen .bind(&authed.username)); } sqlb.and_where("o.workspace_id = ?".bind(&workspace_id)) - .and_where("o.archived = false") - .and_where("o.draft_only IS NOT TRUE"); + .and_where("o.archived = false"); if item_type == "script" { sqlb.and_where("o.auto_kind IS NULL"); @@ -457,9 +455,35 @@ pub async fn create_http_request( } }; + // Bound the minted JWT to exactly this proxied route so a scope-restricted + // MCP token can't be widened into a full-privilege blank check. The + // endpoint-name gate (in the MCP runner) already authorized *which* endpoint + // may be called; this constrains what the resulting request can do. Unscoped + // callers (cookie / full-privilege tokens) keep an unscoped JWT to preserve + // existing behavior. A scope-restricted caller whose route can't be resolved + // fails closed. + let caller_restricted = api_authed + .scopes + .as_deref() + .is_some_and(|s| s.iter().any(|x| !x.starts_with("if_jobs:filter_tags:"))); + let scopes = if caller_restricted { + let parsed = reqwest::Url::parse(url) + .map_err(|e| ErrorData::internal_error(format!("Invalid proxied URL: {}", e), None))?; + let scope = + windmill_api_auth::scopes::scope_for_route(method, parsed.path()).ok_or_else(|| { + ErrorData::internal_error( + "Could not derive route scope for proxied MCP endpoint".to_string(), + None, + ) + })?; + Some(vec![scope]) + } else { + None + }; + // Add authorization header let authed = Authed::from(api_authed.clone()); - let token = create_jwt_token(authed, workspace_id, 3600, None, None, None, None) + let token = create_jwt_token(authed, workspace_id, 3600, None, None, None, scopes) .await .map_err(|e| ErrorData::internal_error(e.to_string(), None))?; request_builder = request_builder.header("Authorization", format!("Bearer {}", token)); diff --git a/backend/windmill-api/src/offboarding.rs b/backend/windmill-api/src/offboarding.rs index c929a85a84..24b6d3f10d 100644 --- a/backend/windmill-api/src/offboarding.rs +++ b/backend/windmill-api/src/offboarding.rs @@ -1,13 +1,13 @@ use std::collections::HashMap; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use axum::{ extract::{Extension, Path}, Json, }; use serde::{Deserialize, Serialize}; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_api_users::users::delete_workspace_user_internal; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; @@ -483,11 +483,13 @@ pub(crate) async fn global_offboard_preview( pub(crate) async fn offboard_global_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Path(email): Path, Json(req): Json, ) -> Result> { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let workspaces = sqlx::query!( "SELECT workspace_id, username FROM usr WHERE email = $1", @@ -847,8 +849,8 @@ async fn offboard_user_from_workspace<'c>( let flows_reassigned = sqlx::query_scalar!( r#"WITH inserted AS ( INSERT INTO flow - (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs) - SELECT workspace_id, REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\1'), summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs + (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs) + SELECT workspace_id, REGEXP_REPLACE(path, 'u/' || $2 || '/(.*)', $1 || '/\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at, labels, lock_error_logs FROM flow WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3 RETURNING 1 diff --git a/backend/windmill-api/src/path_autocomplete.rs b/backend/windmill-api/src/path_autocomplete.rs index 56bfd62ce5..271e9497fe 100644 --- a/backend/windmill-api/src/path_autocomplete.rs +++ b/backend/windmill-api/src/path_autocomplete.rs @@ -70,9 +70,9 @@ async fn list_paths( let mut paths: Vec = sqlx::query_scalar!( r#" SELECT path AS "path!" FROM ( - (SELECT DISTINCT path FROM script WHERE workspace_id = $1 AND archived = false AND deleted = false AND draft_only IS NOT true LIMIT 5000) + (SELECT DISTINCT path FROM script WHERE workspace_id = $1 AND archived = false AND deleted = false LIMIT 5000) UNION - (SELECT path FROM flow WHERE workspace_id = $1 AND archived = false AND draft_only IS NOT true LIMIT 5000) + (SELECT path FROM flow WHERE workspace_id = $1 AND archived = false LIMIT 5000) UNION (SELECT path FROM app WHERE workspace_id = $1 LIMIT 5000) UNION diff --git a/backend/windmill-api/src/secret_backend_ext.rs b/backend/windmill-api/src/secret_backend_ext.rs index f76ba0ba64..fd17b8271e 100644 --- a/backend/windmill-api/src/secret_backend_ext.rs +++ b/backend/windmill-api/src/secret_backend_ext.rs @@ -8,245 +8,24 @@ //! Secret backend extension for the API layer //! -//! This module provides helper functions for integrating the SecretBackend -//! trait with variable operations in the API. +//! Backend resolution and read helpers live in +//! `windmill_common::secret_backend`; this module keeps the API-specific bulk +//! rename helper used when renaming users. //! //! Note: HashiCorp Vault integration requires Enterprise Edition. //! The OSS version only supports the database backend. -#[cfg(all(feature = "private", feature = "enterprise"))] -use std::sync::Arc; - use windmill_common::{db::DB, error::Result}; -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::error::Error; - -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::secret_backend::{database::DatabaseBackend, SecretBackend}; - #[cfg(all(feature = "private", feature = "enterprise"))] use windmill_common::{ - global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, + error::Error, secret_backend::{ - AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, - AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, + get_secret_backend, is_aws_sm_stored_value, is_azure_kv_stored_value, + is_external_stored_value, is_vault_backend_configured, }, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use tokio::sync::RwLock; - -// Cached Vault backend to avoid recreating it for every request -// This enables connection pooling and avoids repeated setup overhead -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedVaultBackend { - backend: Arc, - settings: VaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached Azure Key Vault backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAzureKvBackend { - backend: Arc, - settings: AzureKeyVaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached AWS Secrets Manager backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAwsSmBackend { - backend: Arc, - settings: AwsSecretsManagerSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -/// Get the current secret backend based on global settings (EE only) -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_secret_backend(db: &DB) -> Result> { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - match config { - SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), - SecretBackendConfig::HashiCorpVault(settings) => { - get_or_create_vault_backend(db, settings).await - } - SecretBackendConfig::AzureKeyVault(settings) => { - get_or_create_azure_kv_backend(db, settings).await - } - SecretBackendConfig::AwsSecretsManager(settings) => { - get_or_create_aws_sm_backend(db, settings).await - } - } -} - -/// Get a cached Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_vault_backend( - _db: &DB, - settings: VaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = VAULT_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = VAULT_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = { - #[cfg(feature = "openidconnect")] - if settings.token.is_none() { - Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) - } else { - Arc::new(VaultBackend::new(settings.clone())) - } - - #[cfg(not(feature = "openidconnect"))] - Arc::new(VaultBackend::new(settings.clone())) - }; - - // Cache it - *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached Azure Key Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_azure_kv_backend( - _db: &DB, - settings: AzureKeyVaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = AZURE_KV_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = AZURE_KV_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); - - // Cache it - *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached AWS SM backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_aws_sm_backend( - _db: &DB, - settings: AwsSecretsManagerSettings, -) -> Result> { - { - let cache = AWS_SM_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - let mut cache = AWS_SM_BACKEND_CACHE.write().await; - - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - let backend: Arc = - Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); - - *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Check if an external secret backend is currently configured (EE only) -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn is_vault_backend_configured(db: &DB) -> Result { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - Ok(matches!( - config, - SecretBackendConfig::HashiCorpVault(_) - | SecretBackendConfig::AzureKeyVault(_) - | SecretBackendConfig::AwsSecretsManager(_) - )) -} - -/// Check if a value is stored in Vault (indicated by the $vault: prefix) -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_vault_stored_value(value: &str) -> bool { - value.starts_with("$vault:") -} - -/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_azure_kv_stored_value(value: &str) -> bool { - value.starts_with("$azure_kv:") -} - -/// Check if a value is stored in AWS Secrets Manager -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_aws_sm_stored_value(value: &str) -> bool { - value.starts_with("$aws_sm:") -} - -/// Check if a value is stored in any external secret backend -#[cfg(all(feature = "private", feature = "enterprise"))] -fn is_external_stored_value(value: &str) -> bool { - is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) -} - /// Bulk rename secrets in Vault when a path prefix changes (e.g., user rename) /// EE only feature. /// diff --git a/backend/windmill-api/src/token.rs b/backend/windmill-api/src/token.rs index ac4da6497e..7a4a4887f8 100644 --- a/backend/windmill-api/src/token.rs +++ b/backend/windmill-api/src/token.rs @@ -98,6 +98,7 @@ fn build_standard_scope_domains() -> Vec { ("configs", "Configs", "Configuration management", false), ("oauth", "OAuth", "OAuth management", false), ("ai", "AI", "AI feature management", false), + ("ai_skills", "AI Skills", "AI skill management", false), ( "agent_workers", "Agent Workers", @@ -193,6 +194,19 @@ lazy_static! { ], }]; + // Read-only: `/api/docs/*` exposes only GET routes, so there is no + // `docs:write`. Kept out of build_standard_scope_domains (which mints a + // read+write pair) for that reason. + groups.push(ScopeDomain { + name: "Documentation".to_string(), + description: Some("Read-only documentation search".to_string()), + scopes: vec![ScopeOption { + value: "docs:read".to_string(), + label: "Read".to_string(), + requires_resource_path: false, + }], + }); + groups.extend(build_standard_scope_domains()); groups.extend(build_trigger_scope_domains()); @@ -207,3 +221,21 @@ pub fn global_service() -> Router { async fn get_all_available_scopes() -> JsonResult> { Ok(Json(ALL_SCOPES.clone())) } + +#[cfg(test)] +mod tests { + use super::*; + + /// The token-scope picker is driven by this catalog, so a scope that is + /// enforced but absent here can't be granted through the supported UI. + #[test] + fn docs_read_scope_is_exposed_read_only() { + let values: Vec<&str> = ALL_SCOPES + .iter() + .flat_map(|d| d.scopes.iter()) + .map(|s| s.value.as_str()) + .collect(); + assert!(values.contains(&"docs:read"), "docs:read must be selectable"); + assert!(!values.contains(&"docs:write"), "docs has no write surface"); + } +} diff --git a/backend/windmill-api/src/triggers/http/handler.rs b/backend/windmill-api/src/triggers/http/handler.rs index 0fd7717582..0508586182 100644 --- a/backend/windmill-api/src/triggers/http/handler.rs +++ b/backend/windmill-api/src/triggers/http/handler.rs @@ -216,7 +216,7 @@ async fn get_http_route_trigger( let email = windmill_common::users::get_email_from_permissioned_as( &trigger.permissioned_as, &trigger.workspace_id, - &db, + db, ) .await?; let authed = windmill_api_auth::fetch_api_authed_from_permissioned_as( diff --git a/backend/windmill-api/src/users.rs b/backend/windmill-api/src/users.rs index d53998355b..36b0d6c734 100644 --- a/backend/windmill-api/src/users.rs +++ b/backend/windmill-api/src/users.rs @@ -11,7 +11,7 @@ pub use windmill_api_users::users::*; use std::sync::Arc; -use crate::db::ApiAuthed; +use crate::db::{ApiAuthed, OptJobAuthed}; use crate::secret_backend_ext::rename_vault_secrets_with_prefix; use argon2::Argon2; use axum::{ @@ -21,7 +21,7 @@ use axum::{ }; use hyper::StatusCode; use serde::Deserialize; -use windmill_api_auth::require_super_admin; +use windmill_api_auth::{forbid_superadmin_job_token, require_super_admin}; use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; use windmill_common::audit::AuditAuthor; @@ -71,11 +71,13 @@ pub fn make_unauthed_service() -> Router { async fn create_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Extension(db): Extension, Extension(webhook): Extension, Extension(argon2): Extension>>, Json(nu): Json, ) -> Result<(StatusCode, String)> { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::create_user(authed, db, webhook, argon2, nu).await } @@ -141,8 +143,10 @@ async fn set_password( Extension(db): Extension, Extension(argon2): Extension>>, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let email = authed.email.clone(); crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -152,9 +156,11 @@ async fn set_password_of_user( Extension(argon2): Extension>>, Path(email): Path, authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Json(ep): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; crate::users_oss::set_password(db, argon2, authed, &email, ep).await } @@ -165,11 +171,13 @@ struct RenameUser { async fn rename_user( authed: ApiAuthed, + OptJobAuthed { job_id, .. }: OptJobAuthed, Path(user_email): Path, Extension(db): Extension, Json(ru): Json, ) -> Result { require_super_admin(&db, &authed.email).await?; + forbid_superadmin_job_token(&db, &authed.email, job_id).await?; let mut tx = db.begin().await?; @@ -457,8 +465,8 @@ async fn update_username_in_workpsace<'c>( // ---- flows ---- sqlx::query!( r#"INSERT INTO flow - (workspace_id, path, summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at) - SELECT workspace_id, REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\1'), summary, description, archived, extra_perms, dependency_job, draft_only, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at + (workspace_id, path, summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at) + SELECT workspace_id, REGEXP_REPLACE(path,'u/' || $2 || '/(.*)','u/' || $1 || '/\1'), summary, description, archived, extra_perms, dependency_job, tag, ws_error_handler_muted, dedicated_worker, timeout, visible_to_runner_only, on_behalf_of_email, concurrency_key, versions, value, schema, edited_by, edited_at FROM flow WHERE path LIKE ('u/' || $2 || '/%') AND workspace_id = $3"#, new_username, diff --git a/backend/windmill-api/src/utils.rs b/backend/windmill-api/src/utils.rs index d63720d9a9..5f472c49d3 100644 --- a/backend/windmill-api/src/utils.rs +++ b/backend/windmill-api/src/utils.rs @@ -9,11 +9,12 @@ use axum::{body::Body, response::Response}; use serde::{Deserialize, Deserializer}; -pub use windmill_api_auth::{check_scopes, require_devops_role, require_super_admin}; +pub use windmill_api_auth::{ + build_scope_path_predicate, check_scopes, require_devops_role, require_super_admin, +}; #[cfg(feature = "private")] pub use windmill_common::usernames::generate_instance_wide_unique_username; -pub use windmill_common::utils::WithStarredInfoQuery; #[cfg(feature = "enterprise")] pub use windmill_alerting::{ diff --git a/backend/windmill-api/src/workspaces_export.rs b/backend/windmill-api/src/workspaces_export.rs index afb39e2cca..66bda77bac 100644 --- a/backend/windmill-api/src/workspaces_export.rs +++ b/backend/windmill-api/src/workspaces_export.rs @@ -12,6 +12,8 @@ use crate::db::ApiAuthed; use crate::{apps::AppWithLastVersion, db::DB, folders::Folder}; +use windmill_api_auth::check_scopes; + #[cfg(any( feature = "http_trigger", feature = "websocket", @@ -367,7 +369,6 @@ where "edited_by", "permissioned_as", "archived", - "has_draft", "error", "last_server_ping", "server_id", @@ -583,6 +584,18 @@ pub(crate) async fn tarball_workspace( skip_resources ); + // The route is gated by workspaces:read, but exporting DECRYPTED secrets is a + // variable-read capability beyond workspace metadata. Require variables:read + // only on the plaintext-secret path: ordinary tarball pulls (structure and + // encrypted-only values) keep working with workspaces:read, and the workspace + // key itself stays admin-only (include_key). No-op for unscoped tokens. + if plain_secret.or(plain_secrets).unwrap_or(false) + && !skip_secrets.unwrap_or(false) + && !skip_variables.unwrap_or(false) + { + check_scopes(&authed, || "variables:read".to_string())?; + } + // Opt-in behavior for surfacing per-resource ACLs on flow/app rows. // Folder and group rows have always carried `extra_perms` in source and // continue to do so unconditionally (`KeepEvenEmpty`) so existing @@ -593,8 +606,37 @@ pub(crate) async fn tarball_workspace( ExtraPermsBehavior::Drop }; + // Resolve workspace dependencies on the pool *before* opening the RLS + // transaction: fetching them mid-transaction would hold a second + // simultaneous connection while `tx` is still checked out. + let workspace_dependencies = if include_workspace_dependencies.unwrap_or(false) + && require_admin(authed.is_admin, &authed.username).is_ok() + { + Some(WorkspaceDependencies::list(&w_id, &db).await?) + } else { + None + }; + let mut tx = user_db.begin(&authed).await?; + // Exporting decrypted secrets in bulk is the same capability as a per-item + // secret read, so record it for parity with variables.decrypt_secret. + if plain_secret.or(plain_secrets).unwrap_or(false) + && !skip_variables.unwrap_or(false) + && !skip_secrets.unwrap_or(false) + { + windmill_audit::audit_oss::audit_log( + &mut *tx, + &authed, + "variables.decrypt_secret", + windmill_audit::ActionKind::Execute, + &w_id, + Some("workspace_tarball_export"), + None, + ) + .await?; + } + // Source-of-truth for fork-ness: the workspace's parent_workspace_id column. // The wm-fork-* prefix is a creation-time naming convention that could in // principle drift (rename, manual SQL); the column is the contract that @@ -650,7 +692,6 @@ pub(crate) async fn tarball_workspace( { let scripts = sqlx::query_as::<_, Script>(&format!( "SELECT {} FROM script as o WHERE workspace_id = $1 AND archived = false - AND (draft_only IS NULL OR draft_only = false) AND created_at = (select max(created_at) from script where path = o.path AND \ workspace_id = $1)", windmill_common::scripts::SCRIPT_COLUMNS, @@ -786,10 +827,10 @@ pub(crate) async fn tarball_workspace( { let flows = sqlx::query_as::<_, Flow>( - "SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.draft_only, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of_email, flow.labels, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by + "SELECT flow.workspace_id, flow.path, flow.summary, flow.description, flow.archived, flow.extra_perms, flow.dedicated_worker, flow.tag, flow.ws_error_handler_muted, flow.timeout, flow.visible_to_runner_only, flow.on_behalf_of_email, flow.labels, flow_version.schema, flow_version.value, flow_version.created_at as edited_at, flow_version.created_by as edited_by FROM flow LEFT JOIN flow_version ON flow_version.id = flow.versions[array_upper(flow.versions, 1)] - WHERE flow.workspace_id = $1 AND flow.archived = false AND (flow.draft_only IS NULL OR flow.draft_only = false)", + WHERE flow.workspace_id = $1 AND flow.archived = false", ) .bind(&w_id) .fetch_all(&mut *tx) @@ -838,8 +879,7 @@ pub(crate) async fn tarball_workspace( "SELECT app.id, app.path, app.summary, app.versions, app.policy, app.custom_path, app.extra_perms, app_version.value, app_version.created_at, app_version.created_by, app_version.raw_app, app.labels from app, app_version - WHERE app.workspace_id = $1 AND app_version.id = app.versions[array_upper(app.versions, 1)] - AND (app.draft_only IS NULL OR app.draft_only = false)", + WHERE app.workspace_id = $1 AND app_version.id = app.versions[array_upper(app.versions, 1)]", ) .bind(&w_id) .fetch_all(&mut *tx) @@ -854,11 +894,8 @@ pub(crate) async fn tarball_workspace( } } - if include_workspace_dependencies.unwrap_or(false) - && require_admin(authed.is_admin, &authed.username).is_ok() - { + if let Some(workspace_dependencies) = workspace_dependencies { tracing::info!("Including workspace dependencies in tarball export"); - let workspace_dependencies = WorkspaceDependencies::list(&w_id, &db).await?; tracing::info!( "Found {} workspace dependencies", workspace_dependencies.len() @@ -918,7 +955,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::http::HttpTrigger; let handler = HttpTrigger; - let http_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let http_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -948,7 +985,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::websocket::WebsocketTrigger; let handler = WebsocketTrigger; - let websocket_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let websocket_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -978,7 +1015,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::kafka::KafkaTrigger; let handler = KafkaTrigger; - let kafka_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let kafka_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1008,7 +1045,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::sqs::SqsTrigger; let handler = SqsTrigger; - let sqs_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let sqs_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1038,7 +1075,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::gcp::GcpTrigger; let handler = GcpTrigger; - let gcp_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let gcp_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1068,7 +1105,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::azure::AzureTrigger; let handler = AzureTrigger; - let azure_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let azure_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1098,7 +1135,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::nats::NatsTrigger; let handler = NatsTrigger; - let nats_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let nats_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1128,7 +1165,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::postgres::PostgresTrigger; let handler = PostgresTrigger; - let postgres_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let postgres_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1158,7 +1195,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::mqtt::MqttTrigger; let handler = MqttTrigger; - let mqtt_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let mqtt_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, @@ -1188,7 +1225,7 @@ pub(crate) async fn tarball_workspace( { use crate::triggers::email::EmailTrigger; let handler = EmailTrigger; - let email_triggers = handler.list_triggers(&mut *tx, &w_id, None).await?; + let email_triggers = handler.list_triggers(&mut *tx, &w_id, None, None).await?; let parent_modes = fork_parent_trigger_modes( &db, ::TABLE_NAME, diff --git a/backend/windmill-common/Cargo.toml b/backend/windmill-common/Cargo.toml index 88a7706b88..7e30547d9b 100644 --- a/backend/windmill-common/Cargo.toml +++ b/backend/windmill-common/Cargo.toml @@ -38,6 +38,7 @@ anyhow.workspace = true serde.workspace = true serde_json.workspace = true serde_yml.workspace = true +erased-serde = "0.4" chrono.workspace = true chrono-tz.workspace = true hex.workspace = true diff --git a/backend/windmill-common/src/assets.rs b/backend/windmill-common/src/assets.rs index 2968493ecc..c186f59141 100644 --- a/backend/windmill-common/src/assets.rs +++ b/backend/windmill-common/src/assets.rs @@ -1,9 +1,30 @@ -use sqlx::PgExecutor; +use std::collections::HashSet; + +use sqlx::{PgExecutor, Postgres, Transaction}; use crate::{error, scripts::ScriptHash}; +pub use windmill_parser::asset_parser::{ + merge_column_lineage, parse_pipeline_annotations, ColumnLineage, ColumnRef, DataTest, + PartitionKind, PipelineAnnotations, RetrySpec, TriggerSpec, PARTITION_TOKEN, +}; pub use windmill_types::assets::*; +#[derive(sqlx::Type, Debug, Clone, Copy, PartialEq)] +#[sqlx(type_name = "SCRIPT_TRIGGER_KIND", rename_all = "lowercase")] +pub enum ScriptTriggerKind { + Asset, + Schedule, + Webhook, + Email, + Kafka, + Mqtt, + Nats, + Postgres, + Sqs, + Gcp, +} + pub async fn insert_static_asset_usage<'e>( executor: impl PgExecutor<'e>, workspace_id: &str, @@ -17,9 +38,22 @@ pub async fn insert_static_asset_usage<'e>( .as_ref() .map(|cols| serde_json::to_value(cols).unwrap_or(serde_json::Value::Null)); + // Invalidate the per-workspace producer-writes cache only when this insert + // actually adds a write producer: the cache (asset_dispatch:: + // ASSET_PRODUCER_WRITES_CACHE) tracks script rows with 'w'/'rw' access, so + // a row that was a no-op (ON CONFLICT skipped), a flow usage, or read-only + // can't change it. Emitting in the same statement keeps the notify atomic + // with the insert and visible to pollers only on commit. See the matching + // delete-side guard in clear_static_asset_usage. sqlx::query!( - r#"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns) - VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING"#, + r#"WITH ins AS ( + INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns) + VALUES ($1, $2, $3, $4, $5, $6, $7) ON CONFLICT DO NOTHING + RETURNING usage_kind, usage_access_type + ) + INSERT INTO notify_event (channel, payload) + SELECT 'notify_asset_producer_change', $1 + FROM ins WHERE usage_kind = 'script' AND usage_access_type IN ('w', 'rw')"#, workspace_id, asset.path, asset.kind as AssetKind, @@ -40,8 +74,25 @@ pub async fn clear_static_asset_usage<'e>( usage_path: &str, usage_kind: AssetUsageKind, ) -> error::Result<()> { + // Invalidate the per-workspace producer-writes cache that gates the + // asset-trigger dispatch hook (windmill-queue + // asset_dispatch::ASSET_PRODUCER_WRITES_CACHE). That cache only tracks + // script rows with 'w'/'rw' access, so emit the notify only when the + // delete actually removed such a write producer: flow usage, read-only + // usage, and deletes that matched no producer row leave the cache + // unchanged (the common case — most deploys touch no write asset). The + // matching add side lives in insert_static_asset_usage. Emitting in the + // same statement keeps the notify atomic with the delete and visible to + // pollers only on commit. sqlx::query!( - r#"DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3"#, + r#"WITH del AS ( + DELETE FROM asset WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = $3 + RETURNING usage_access_type + ) + INSERT INTO notify_event (channel, payload) + SELECT 'notify_asset_producer_change', $1 + WHERE $3 = 'script' + AND EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))"#, workspace_id, usage_path, usage_kind as AssetUsageKind @@ -56,8 +107,18 @@ pub async fn clear_static_asset_usage_by_script_hash<'e>( workspace_id: &str, script_hash: ScriptHash, ) -> error::Result<()> { + // Always script usage → invalidate the producer-writes cache for this + // workspace, but only when the delete actually removed a write producer + // ('w'/'rw'); see clear_static_asset_usage. Atomic with the delete. sqlx::query!( - "DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script' AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1)", + r#"WITH del AS ( + DELETE FROM asset WHERE workspace_id = $1 AND usage_kind = 'script' + AND usage_path = (SELECT path FROM script WHERE hash = $2 AND workspace_id = $1) + RETURNING usage_access_type + ) + INSERT INTO notify_event (channel, payload) + SELECT 'notify_asset_producer_change', $1 + WHERE EXISTS (SELECT 1 FROM del WHERE usage_access_type IN ('w', 'rw'))"#, workspace_id, script_hash.0 ) @@ -66,6 +127,253 @@ pub async fn clear_static_asset_usage_by_script_hash<'e>( Ok(()) } +fn is_write_access(access: Option) -> bool { + matches!( + access, + Some(AssetUsageAccessType::W) | Some(AssetUsageAccessType::RW) + ) +} + +/// Clear and reinsert the full static-asset usage set of a script in one tx, +/// invalidating the producer-writes cache at most once and only on a real +/// change. The cache (asset_dispatch::ASSET_PRODUCER_WRITES_CACHE) keys a +/// workspace by the set of (kind, path) script rows with 'w'/'rw' access, so a +/// redeploy that keeps the same write producers must leave it untouched. We +/// diff the old write set (captured from the clearing DELETE's RETURNING) +/// against the new one and emit a single notify only when they differ — +/// emitting per statement, as clear/insert_static_asset_usage do, would fire +/// twice on every write-asset redeploy (the clear removes the row, the reinsert +/// adds it back). The notify rides the deploy tx, so it stays atomic with the +/// DML and visible to pollers only on commit. +/// +/// DELETE and INSERT of the same primary key cannot share a single statement +/// (both would read the pre-statement snapshot, so the reinsert's ON CONFLICT +/// would silently drop the row), which is why this clears and reinserts as +/// separate statements rather than one CTE. +/// +/// Performs no authorization itself: `tx` must already be scoped to a caller +/// authorized for `workspace_id`/`usage_path` (e.g. `user_db.begin(&authed)`, +/// which applies RLS), exactly like the sibling clear/insert helpers. +pub async fn replace_static_asset_usage( + tx: &mut Transaction<'_, Postgres>, + workspace_id: &str, + usage_path: &str, + assets: &[AssetWithAltAccessType], +) -> error::Result<()> { + let cleared = sqlx::query!( + r#"DELETE FROM asset + WHERE workspace_id = $1 AND usage_path = $2 AND usage_kind = 'script' + RETURNING kind AS "kind!: AssetKind", path, + usage_access_type AS "usage_access_type: AssetUsageAccessType""#, + workspace_id, + usage_path, + ) + .fetch_all(&mut **tx) + .await?; + let old_writes: HashSet<(AssetKind, String)> = cleared + .into_iter() + .filter(|r| is_write_access(r.usage_access_type)) + .map(|r| (r.kind, r.path)) + .collect(); + + // Build new_writes from rows actually inserted (RETURNING), not the + // requested slice: ON CONFLICT DO NOTHING is first-writer-wins, so a payload + // with duplicate (kind, path) entries at conflicting access types persists + // only the first. Since the DELETE above emptied this usage_path, every + // non-conflicting insert lands, so the inserted rows are exactly the new + // persisted set. + let mut new_writes: HashSet<(AssetKind, String)> = HashSet::new(); + for asset in assets { + let access = asset.access_type.or(asset.alt_access_type); + let columns_json = asset + .columns + .as_ref() + .map(|cols| serde_json::to_value(cols).unwrap_or(serde_json::Value::Null)); + let inserted = sqlx::query!( + r#"INSERT INTO asset (workspace_id, path, kind, usage_access_type, usage_path, usage_kind, columns) + VALUES ($1, $2, $3, $4, $5, 'script', $6) ON CONFLICT DO NOTHING + RETURNING usage_access_type AS "usage_access_type: AssetUsageAccessType""#, + workspace_id, + asset.path, + asset.kind as AssetKind, + access as Option, + usage_path, + columns_json as Option, + ) + .fetch_optional(&mut **tx) + .await?; + if let Some(row) = inserted { + if is_write_access(row.usage_access_type) { + new_writes.insert((asset.kind, asset.path.clone())); + } + } + } + + if old_writes != new_writes { + sqlx::query!( + r#"INSERT INTO notify_event (channel, payload) + VALUES ('notify_asset_producer_change', $1)"#, + workspace_id, + ) + .execute(&mut **tx) + .await?; + } + Ok(()) +} + +// Wipe all pipeline trigger declarations held by the given runnable. Used at +// deploy time: redeploying a script wipes its prior `// on` annotations so +// removing them implicitly un-declares those edges. +pub async fn clear_script_triggers<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + runnable_path: &str, + runnable_kind: AssetUsageKind, +) -> error::Result<()> { + sqlx::query!( + r#"DELETE FROM script_trigger + WHERE workspace_id = $1 AND runnable_kind = $2 AND runnable_path = $3"#, + workspace_id, + runnable_kind as AssetUsageKind, + runnable_path, + ) + .execute(executor) + .await?; + Ok(()) +} + +// Insert a single trigger declaration. Caller is expected to wipe first. +// `join_all` is the script-level `// trigger all` flag (AND join barrier); +// `retry_count` / `retry_delay_s` are the `// retry []` policy. +// All three are script-level — the same value for every row of a given +// runnable — but stored per-row to keep the wipe-and-reinsert pattern and a +// single-query subscriber lookup. +pub async fn insert_script_trigger<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + runnable_kind: AssetUsageKind, + runnable_path: &str, + trigger_kind: ScriptTriggerKind, + trigger_ref: &str, + join_all: bool, + debounce_s: Option, + retry_count: Option, + retry_delay_s: Option, +) -> error::Result<()> { + sqlx::query!( + r#"INSERT INTO script_trigger + (workspace_id, runnable_kind, runnable_path, trigger_kind, trigger_ref, join_all, + debounce_s, retry_count, retry_delay_s) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)"#, + workspace_id, + runnable_kind as AssetUsageKind, + runnable_path, + trigger_kind as ScriptTriggerKind, + trigger_ref, + join_all, + debounce_s, + retry_count, + retry_delay_s, + ) + .execute(executor) + .await?; + Ok(()) +} + +/// Parse a debounce duration into whole seconds. Accepts a bare integer +/// (seconds) or an `` with an `s`/`m`/`h`/`d` suffix (e.g. `30s`, +/// `5m`, `2h`, `1d`). Returns `None` for empty / malformed / non-positive +/// input — the caller treats `None` as "no debounce" (fan-out), so a typo +/// fails safe rather than silently debouncing. +pub fn parse_duration_secs(s: &str) -> Option { + let s = s.trim(); + if s.is_empty() { + return None; + } + let (num, mult): (&str, i64) = match s.as_bytes().last() { + Some(b's') => (&s[..s.len() - 1], 1), + Some(b'm') => (&s[..s.len() - 1], 60), + Some(b'h') => (&s[..s.len() - 1], 3600), + Some(b'd') => (&s[..s.len() - 1], 86400), + Some(c) if c.is_ascii_digit() => (s, 1), + _ => return None, + }; + let n: i64 = num.trim().parse().ok()?; + let secs = n.checked_mul(mult)?; + if secs <= 0 || secs > i32::MAX as i64 { + return None; + } + Some(secs as i32) +} + +#[cfg(test)] +mod debounce_duration_tests { + use super::parse_duration_secs; + + #[test] + fn parses_units_and_bare_seconds() { + assert_eq!(parse_duration_secs("60"), Some(60)); + assert_eq!(parse_duration_secs("30s"), Some(30)); + assert_eq!(parse_duration_secs("5m"), Some(300)); + assert_eq!(parse_duration_secs("2h"), Some(7200)); + assert_eq!(parse_duration_secs(" 1d "), Some(86400)); + } + + #[test] + fn rejects_garbage_and_nonpositive() { + assert_eq!(parse_duration_secs(""), None); + assert_eq!(parse_duration_secs("abc"), None); + assert_eq!(parse_duration_secs("0"), None); + assert_eq!(parse_duration_secs("-5"), None); + assert_eq!(parse_duration_secs("10x"), None); + assert_eq!(parse_duration_secs("s"), None); + } +} + +// Inverse of trigger_spec_to_row for the Asset variant: parses a stored +// trigger_ref (e.g. `s3://foo`, `$res:bar`) back into the (kind, path) pair +// used as a graph node id. Returns None for refs that don't match any known +// asset prefix — callers should skip those edges. +pub fn parse_asset_trigger_ref(s: &str) -> Option<(AssetKind, String)> { + let (kind, path) = windmill_parser::asset_parser::parse_asset_syntax(s, false)?; + Some((asset_kind_from_parser(kind), path.to_string())) +} + +// Convert a parser TriggerSpec into the `(kind, ref)` pair stored in +// script_trigger. Asset refs get their canonical prefix back so the +// trigger_ref matches what downstream lookups expect. +// +// Returns `None` for native trigger kinds (Kafka, Mqtt, Postgres, …) — +// those annotations are marker-only and don't produce a `script_trigger` +// row. The actual binding lives on the trigger row's own `script_path` +// column; the graph endpoint looks it up directly per kind. +pub fn trigger_spec_to_row(spec: &TriggerSpec) -> Option<(ScriptTriggerKind, String)> { + match spec { + TriggerSpec::Asset { asset_kind, path, .. } => { + // Single source of truth for the canonical prefix lives on the + // common AssetKind; map the parser kind across first. The parser + // enum has no Variable variant, so canonical_prefix is always Some. + let prefix = asset_kind_from_parser(*asset_kind).canonical_prefix()?; + Some((ScriptTriggerKind::Asset, format!("{}{}", prefix, path))) + } + // Schedule joins the native-trigger family — no script_trigger row + // is inserted for the annotation. The binding lives on the schedule + // row's own `script_path` field, same as kafka/mqtt/etc. + TriggerSpec::Schedule + | TriggerSpec::Webhook + | TriggerSpec::Email + | TriggerSpec::Kafka + | TriggerSpec::Mqtt + | TriggerSpec::Nats + | TriggerSpec::Postgres + | TriggerSpec::Sqs + | TriggerSpec::Gcp + // data_upload is a UI-first marker — no event source, no trigger row. + // The script's S3Object input + auto-generated S3 picker drive it. + | TriggerSpec::DataUpload => None, + } +} + pub fn asset_kind_from_parser(parser_kind: windmill_parser::asset_parser::AssetKind) -> AssetKind { match parser_kind { windmill_parser::asset_parser::AssetKind::S3Object => AssetKind::S3Object, diff --git a/backend/windmill-common/src/auth.rs b/backend/windmill-common/src/auth.rs index 7fc6a0825a..5c1cedc6d8 100644 --- a/backend/windmill-common/src/auth.rs +++ b/backend/windmill-common/src/auth.rs @@ -287,7 +287,7 @@ impl From for Authed { } } -pub async fn is_super_admin_email(db: &DB, email: &str) -> Result { +pub async fn is_super_admin_email<'c>(db: impl sqlx::PgExecutor<'c>, email: &str) -> Result { if email == SUPERADMIN_SECRET_EMAIL || email == SUPERADMIN_NOTIFICATION_EMAIL { return Ok(true); } diff --git a/backend/windmill-common/src/bench.rs b/backend/windmill-common/src/bench.rs index 1ed2b9aa56..68c3cb5a31 100644 --- a/backend/windmill-common/src/bench.rs +++ b/backend/windmill-common/src/bench.rs @@ -258,6 +258,8 @@ pub async fn benchmark_init(benchmark_jobs: i32, db: &DB) { .execute(db) .await .unwrap_or_else(|e| panic!("failed to clean up concurrency_counter: {e:#}")); + // Benchmark jobs never produce dispatch_event / flow_conversation_message / + // zombie_job_counter rows, so this cleanup needs no side-table deletes (cf. delete_jobs). sqlx::query!("DELETE FROM v2_job WHERE workspace_id = 'admins'") .execute(db) .await diff --git a/backend/windmill-common/src/cache.rs b/backend/windmill-common/src/cache.rs index e39fd6ea43..6c34db8169 100644 --- a/backend/windmill-common/src/cache.rs +++ b/backend/windmill-common/src/cache.rs @@ -1189,11 +1189,25 @@ const _: () = { use std::fs::OpenOptions; use std::io::Write; + // Atomic write: truncate+write a uniquely-named temp file (UUID, not pid — pids + // collide across container PID namespaces on a shared cache volume), fsync, then + // rename(2) over the target. Without this a shorter overwrite leaves a stale tail + // and concurrent writers tear the file — a corrupt entry a reader would import. + let final_path = item.path(self); + let tmp_path = final_path.with_extension(format!("tmp.{}", Uuid::new_v4())); OpenOptions::new() .write(true) .create(true) - .open(item.path(self)) - .and_then(|mut file| file.write_all(data.as_ref())) + .truncate(true) + .open(&tmp_path) + .and_then(|mut file| { + file.write_all(data.as_ref())?; + file.sync_all() + }) + .and_then(|()| std::fs::rename(&tmp_path, &final_path)) + .inspect_err(|_| { + let _ = std::fs::remove_file(&tmp_path); + }) } } @@ -1237,6 +1251,33 @@ mod tests { use super::*; use serde_json::json; + #[test] + fn fs_cache_put_overwrites_without_stale_tail() { + // Regression for the non-truncating, non-atomic `put`: overwriting a value with a + // shorter one must not leave stale trailing bytes (which imported as corrupt/wrong + // content — the #9751 worker-cache hazard). + let dir = tempfile::tempdir().unwrap(); + let root = dir.path(); + + root.put("k", b"a-long-cached-value-0123456789").unwrap(); + assert_eq!(root.get("k").unwrap(), b"a-long-cached-value-0123456789"); + + root.put("k", b"short").unwrap(); + assert_eq!( + root.get("k").unwrap(), + b"short", + "shorter overwrite must fully replace, no stale tail" + ); + + // No temp files left behind after a successful write. + let leftover: Vec<_> = std::fs::read_dir(root) + .unwrap() + .filter_map(|e| e.ok()) + .filter(|e| e.file_name().to_string_lossy().contains(".tmp.")) + .collect(); + assert!(leftover.is_empty(), "temp files must be renamed/cleaned up"); + } + #[test] fn flow_data_extras_preserves_notes_and_groups() { let raw = serde_json::value::to_raw_value(&json!({ diff --git a/backend/windmill-common/src/git_sync_oss.rs b/backend/windmill-common/src/git_sync_oss.rs index 4cd7a7c3d1..1e17532ec7 100644 --- a/backend/windmill-common/src/git_sync_oss.rs +++ b/backend/windmill-common/src/git_sync_oss.rs @@ -15,6 +15,29 @@ pub async fn get_github_app_token_internal( )); } +lazy_static::lazy_static! { + /// Matches a `user:password@` (or `user@`) userinfo component right after the URL scheme. + static ref GIT_URL_USERINFO_RE: regex::Regex = + regex::Regex::new(r"://[^/@]+@").unwrap(); +} + +/// Strip embedded credentials (the `user:password@` userinfo component) from a git URL so it can be +/// safely included in error messages and logs. Falls back to a regex when the URL does not parse. +pub fn sanitize_git_url(url: &str) -> String { + if let Ok(mut parsed) = Url::parse(url) { + if !parsed.username().is_empty() || parsed.password().is_some() { + // These setters only fail for cannot-be-a-base URLs, in which case we keep the parsed + // string as-is and let the regex fallback below handle stripping. + let _ = parsed.set_username(""); + let _ = parsed.set_password(None); + } + return GIT_URL_USERINFO_RE + .replace(parsed.as_str(), "://***@") + .into_owned(); + } + GIT_URL_USERINFO_RE.replace(url, "://***@").into_owned() +} + pub fn prepend_token_to_github_url( github_url: &str, installation_token: &str, @@ -32,3 +55,41 @@ pub fn prepend_token_to_github_url( url.path() )) } + +#[cfg(test)] +mod tests { + use super::sanitize_git_url; + + #[test] + fn strips_username_and_password() { + assert_eq!( + sanitize_git_url("https://user:p4ssw0rd@github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn strips_token_only_userinfo() { + assert_eq!( + sanitize_git_url("https://ghp_secrettoken@github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn leaves_credential_free_url_untouched() { + assert_eq!( + sanitize_git_url("https://github.com/org/repo.git"), + "https://github.com/org/repo.git" + ); + } + + #[test] + fn strips_credentials_from_unparseable_url() { + // scp-like syntax that `url::Url` cannot parse + assert_eq!( + sanitize_git_url("not a url://user:secret@host/repo"), + "not a url://***@host/repo" + ); + } +} diff --git a/backend/windmill-common/src/global_settings.rs b/backend/windmill-common/src/global_settings.rs index 732b9cdc30..4d1d48a53d 100644 --- a/backend/windmill-common/src/global_settings.rs +++ b/backend/windmill-common/src/global_settings.rs @@ -1,3 +1,8 @@ +// Adding a global setting? Decide whether agent workers may read it. Agent +// workers (remote workers connected over HTTP) fetch settings through an +// endpoint that is deny-by-exception: every key is served except those in +// AGENT_WORKER_BLOCKED_SETTINGS (defined below). If a new setting holds an +// instance secret the server should keep to itself, add its key there. pub const CUSTOM_TAGS_SETTING: &str = "custom_tags"; pub const DEFAULT_TAGS_PER_WORKSPACE_SETTING: &str = "default_tags_per_workspace"; pub const DEFAULT_TAGS_WORKSPACES_SETTING: &str = "default_tags_workspaces"; @@ -71,6 +76,7 @@ pub const OBJECT_STORE_CONFIG_SETTING: &str = "object_store_cache_config"; pub const HUB_API_SECRET_SETTING: &str = "hub_api_secret"; pub const AUTOMATE_USERNAME_CREATION_SETTING: &str = "automate_username_creation"; +pub const DISABLE_WORKSPACE_INVITE_EMAILS_SETTING: &str = "disable_workspace_invite_emails"; pub const DISABLE_PASSWORD_LOGIN_SETTING: &str = "disable_password_login"; pub const AUTO_LOGIN_PROVIDER_SETTING: &str = "auto_login_provider"; pub const HUB_BASE_URL_SETTING: &str = "hub_base_url"; @@ -104,6 +110,53 @@ pub const WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING: &str = "workspace_fairness_max pub const WORKSPACE_FAIRNESS_DURATION_SECS_SETTING: &str = "workspace_fairness_duration_secs"; pub const WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING: &str = "workspace_fairness_min_total_jobs"; +/// Global settings an agent worker (a remote worker connected over HTTP instead +/// of to the database) must NEVER read through +/// `GET /api/agent_workers/get_global_setting/{key}`. Every other key is served. +/// +/// SECURITY: that endpoint is authenticated only by an agent-worker JWT and +/// returns the raw `global_settings` value for the requested key. Because the +/// policy is deny-by-exception (anything not listed here is readable), every +/// setting that holds an instance secret or credential an agent worker does not +/// need MUST be listed below. Missing one discloses it to every agent worker — +/// `jwt_secret` is the worst case (a token holder could forge a superadmin JWT), +/// but `oauths`, `smtp_settings`, `secret_backend`, object-store credentials, +/// etc. are instance-wide secrets too. +/// +/// NOT blocked, on purpose: the operational credentials an agent worker loads to +/// run jobs (`license_key`, `hub_api_secret`, `sandbox_registry_auth`, +/// `powershell_repo_pat`, `npmrc`, ...). Those are already within an agent +/// worker's trust boundary, and blocking them breaks worker startup or +/// dependency installation. When adding a new setting that stores a secret the +/// server keeps to itself, add it here. +pub const AGENT_WORKER_BLOCKED_SETTINGS: &[&str] = &[ + // Instance identity / auth secrets — disclosure enables privilege escalation + // or impersonation. + JWT_SECRET_SETTING, + OAUTH_SETTING, + SMTP_SETTING, + SCIM_TOKEN_SETTING, + SAML_METADATA_SETTING, + SECRET_BACKEND_SETTING, + GITHUB_ENTERPRISE_APP_SETTING, + OBJECT_STORE_CONFIG_SETTING, + AI_CONFIG_SETTING, + TEAMS_SETTING, + INDEXER_SETTING, + // Server-only configs that may embed credentials, webhook URLs or tokens and + // are never loaded by an agent worker. + CRITICAL_ERROR_CHANNELS_SETTING, + INSTANCE_EVENTS_WEBHOOK_SETTING, + OTEL_SETTING, + OTEL_TRACING_PROXY_SETTING, +]; + +/// Whether an agent worker may read the given global setting over HTTP. +/// Deny-by-exception: everything is readable except [`AGENT_WORKER_BLOCKED_SETTINGS`]. +pub fn is_setting_readable_by_agent_worker(name: &str) -> bool { + !AGENT_WORKER_BLOCKED_SETTINGS.contains(&name) +} + use std::sync::atomic::AtomicBool; lazy_static::lazy_static! { @@ -265,6 +318,62 @@ pub fn workspace_integration_auth_endpoint(client_name: &str, base_url: &str) -> } } +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn agent_workers_can_read_operational_settings() { + // Operational knobs and the credentials a worker needs to run jobs are + // intentionally NOT blocked. Deny-by-exception also means an arbitrary + // unlisted key is readable. + for key in [ + NPMRC_SETTING, + PIP_INDEX_URL_SETTING, + JOB_ISOLATION_SETTING, + LICENSE_KEY_SETTING, + HUB_API_SECRET_SETTING, + SANDBOX_REGISTRY_AUTH_SETTING, + POWERSHELL_REPO_PAT_SETTING, + "some_future_operational_setting", + ] { + assert!( + is_setting_readable_by_agent_worker(key), + "'{key}' must remain readable by agent workers" + ); + } + } + + #[test] + fn agent_workers_cannot_read_instance_secrets() { + // Disclosing any of these to a remote worker enables privilege + // escalation (jwt_secret -> forged superadmin JWT) or leaks instance + // secrets. They must never be served by the agent-worker endpoint. + for key in [ + JWT_SECRET_SETTING, + OAUTH_SETTING, + SMTP_SETTING, + SCIM_TOKEN_SETTING, + SAML_METADATA_SETTING, + SECRET_BACKEND_SETTING, + GITHUB_ENTERPRISE_APP_SETTING, + OBJECT_STORE_CONFIG_SETTING, + AI_CONFIG_SETTING, + TEAMS_SETTING, + INDEXER_SETTING, + CRITICAL_ERROR_CHANNELS_SETTING, + INSTANCE_EVENTS_WEBHOOK_SETTING, + OTEL_SETTING, + OTEL_TRACING_PROXY_SETTING, + ] { + assert!( + !is_setting_readable_by_agent_worker(key), + "'{key}' is an instance secret and must not be readable by agent workers" + ); + } + } +} + pub async fn set_value_in_global_settings( db: &Pool, setting_name: &str, diff --git a/backend/windmill-common/src/jobs.rs b/backend/windmill-common/src/jobs.rs index fd05c03ec0..2178d9a023 100644 --- a/backend/windmill-common/src/jobs.rs +++ b/backend/windmill-common/src/jobs.rs @@ -283,6 +283,19 @@ pub fn format_completed_job_result(mut cj: CompletedJob) -> CompletedJob { cj } +/// `log_file_index` is normally written by the worker as job-id-scoped relative +/// paths under the windmill log directory. Any code path that lets a request +/// control this value (e.g. job import) must reject entries that could escape +/// that directory, otherwise the log-reading endpoints become an arbitrary file +/// read primitive. Rejects path traversal (`..`) and absolute paths; on-disk +/// readers additionally refuse symlinks (see `get_logs_from_disk`). +pub fn is_safe_log_file_path(file_p: &str) -> bool { + !file_p.is_empty() + && !file_p.starts_with('/') + && !file_p.starts_with('\\') + && !file_p.split(['/', '\\']).any(|c| c == "..") +} + pub async fn get_logs_from_disk( log_offset: i32, logs: &str, @@ -291,12 +304,17 @@ pub async fn get_logs_from_disk( if log_offset > 0 { if let Some(file_index) = log_file_index.clone() { for file_p in &file_index { - if !tokio::fs::metadata(format!("{}/{file_p}", *WINDMILL_DIR)) - .await - .is_ok() - { + if !is_safe_log_file_path(file_p) { return None; } + let local_file = format!("{}/{file_p}", *WINDMILL_DIR); + // Defense in depth: refuse to read through a symlink so a planted + // symlink under the log directory cannot exfiltrate arbitrary files. + match tokio::fs::symlink_metadata(&local_file).await { + Ok(meta) if meta.file_type().is_symlink() => return None, + Ok(_) => {} + Err(_) => return None, + } } let logs = logs.to_string(); @@ -325,8 +343,8 @@ lazy_static::lazy_static! { ).unwrap_or(false); } -pub async fn check_tag_available_for_workspace_internal( - db: &DB, +pub async fn check_tag_available_for_workspace_internal<'c>( + db: impl sqlx::PgExecutor<'c>, w_id: &str, tag: &str, email: &str, @@ -439,3 +457,71 @@ pub struct WorkerInternalServerInlineUtils { // The server cannot call the worker functions directly because they are independent crates pub static WORKER_INTERNAL_SERVER_INLINE_UTILS: OnceCell = OnceCell::new(); + +/// Deletes the given jobs from `v2_job` together with the side tables that reference it +/// without an `ON DELETE CASCADE` foreign key. +/// +/// **Authorization contract:** this helper does NO authorization and NO workspace scoping — +/// it deletes exactly the `ids` passed, regardless of which workspace they belong to. Callers +/// MUST ensure `ids` only contains jobs the caller is allowed to delete (either a trusted +/// internal id set, e.g. a retention batch, or ids already filtered by `workspace_id`). +/// Passing user-supplied, unvalidated ids would reintroduce the cross-workspace side-row +/// deletion this centralizes. It is deliberately not workspace-scoped at the signature level +/// because its primary caller — retention — deletes expired jobs across every workspace at +/// once; a `workspace_id` parameter cannot express that. (This is the same trust model as the +/// `ON DELETE CASCADE` FK it replaces: given a job id, the row and its side rows go.) +/// +/// Those FKs were removed (migration `drop_v2_job_side_table_cascades`) because they turned +/// every bulk retention delete into a per-row RI trigger; for the unindexed +/// `flow_conversation_message.job_id` that was a sequential scan per deleted row. The +/// set-based deletes below cost one scan per table per call instead. Because the cascade no +/// longer fires, every code path that deletes from `v2_job` by id must go through this helper +/// (or delete these tables itself) or it will leave orphan rows behind. +pub async fn delete_jobs(conn: &mut sqlx::PgConnection, ids: &[uuid::Uuid]) -> error::Result<()> { + sqlx::query!( + "DELETE FROM dispatch_event WHERE producer_job_id = ANY($1)", + ids + ) + .execute(&mut *conn) + .await?; + sqlx::query!( + "DELETE FROM flow_conversation_message WHERE job_id = ANY($1)", + ids + ) + .execute(&mut *conn) + .await?; + sqlx::query!("DELETE FROM zombie_job_counter WHERE job_id = ANY($1)", ids) + .execute(&mut *conn) + .await?; + sqlx::query!("DELETE FROM v2_job WHERE id = ANY($1)", ids) + .execute(&mut *conn) + .await?; + Ok(()) +} + +#[cfg(test)] +mod tests { + use super::is_safe_log_file_path; + + #[test] + fn safe_log_file_paths_are_accepted() { + // Legit worker-written entries are job-id-scoped relative paths. + assert!(is_safe_log_file_path( + "0190d3e2-0000-7000-8000-000000000000/0.txt" + )); + assert!(is_safe_log_file_path("logs/abc/chunk1.log")); + assert!(is_safe_log_file_path("file..with..dots.txt")); + } + + #[test] + fn traversal_and_absolute_paths_are_rejected() { + assert!(!is_safe_log_file_path("")); + assert!(!is_safe_log_file_path("../../../../etc/passwd")); + assert!(!is_safe_log_file_path("a/../../etc/passwd")); + assert!(!is_safe_log_file_path("..")); + assert!(!is_safe_log_file_path("/etc/passwd")); + assert!(!is_safe_log_file_path("/proc/self/environ")); + assert!(!is_safe_log_file_path("\\windows\\path")); + assert!(!is_safe_log_file_path("a\\..\\..\\b")); + } +} diff --git a/backend/windmill-common/src/lib.rs b/backend/windmill-common/src/lib.rs index c12d242737..be336a84e1 100644 --- a/backend/windmill-common/src/lib.rs +++ b/backend/windmill-common/src/lib.rs @@ -61,6 +61,7 @@ pub mod indexer; pub mod instance_config; pub mod job_metrics; pub mod log_context; +pub mod materialization; pub mod min_version; pub mod notify_events; pub mod runtime_assets; @@ -81,6 +82,20 @@ pub mod oidc_oss; #[cfg(feature = "private")] pub mod otel_ee; pub mod otel_oss; +#[cfg(feature = "private")] +pub mod partition_ee; +pub mod partition_oss; +#[cfg(feature = "private")] +pub use partition_ee as partition; +#[cfg(not(feature = "private"))] +pub use partition_oss as partition; +#[cfg(feature = "private")] +pub mod pipeline_advanced_ee; +pub mod pipeline_advanced_oss; +#[cfg(feature = "private")] +pub use pipeline_advanced_ee as pipeline_advanced; +#[cfg(not(feature = "private"))] +pub use pipeline_advanced_oss as pipeline_advanced; pub mod query_builders; pub mod queue; pub mod result_stream; @@ -102,6 +117,7 @@ pub mod teams_oss; pub mod tracing_init; pub mod trashbin; pub mod triggers; +pub mod user_drafts; pub mod usernames; pub mod users; pub mod utils; @@ -269,6 +285,16 @@ lazy_static::lazy_static! { const LATEST_VERSION_ID_CACHE_TTL: std::time::Duration = std::time::Duration::from_secs(60); +/// Test hook: disables the process-global deployed-script hash/info caches so +/// every resolution reads the current DB. Integration tests use `#[sqlx::test]` +/// isolated DBs that share one workspace id and reuse script paths, so a cache +/// keyed by `(workspace, path)`/`(workspace, hash)` resolves a path to a hash +/// that lives in a *different* test's DB — and when the info cache misses for +/// that foreign hash the lookup 404s in the wrong DB. Always `false` in +/// production (the caches are TTL/LRU-bounded against real deploys). +pub static DEPLOYED_SCRIPT_CACHE_DISABLED: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + pub async fn shutdown_signal( tx: KillpillSender, mut rx: tokio::sync::broadcast::Receiver<()>, @@ -513,6 +539,31 @@ mod classify_python_logging_line_tests { } } +#[cfg(test)] +mod validate_dbname_tests { + use super::validate_dbname; + + #[test] + fn accepts_letters_digits_underscores_and_hyphens() { + assert!(validate_dbname("mydb").is_ok()); + assert!(validate_dbname("my_db").is_ok()); + assert!(validate_dbname("my-database").is_ok()); + assert!(validate_dbname("My-Db_1").is_ok()); + } + + #[test] + fn rejects_invalid_names() { + // Must start with a letter (hyphen/digit/underscore leads are rejected). + assert!(validate_dbname("-db").is_err()); + assert!(validate_dbname("1db").is_err()); + assert!(validate_dbname("_db").is_err()); + // No other special characters or whitespace. + assert!(validate_dbname("my db").is_err()); + assert!(validate_dbname("my;db").is_err()); + assert!(validate_dbname("").is_err()); + } +} + #[derive(Serialize, Debug)] pub struct PrepareQueryColumnInfo { pub name: String, @@ -578,13 +629,22 @@ impl PgDatabase { Some(s) => s.to_string(), None => "prefer".to_string(), }; + // Encode host/dbname too: an unencoded '@', '/', '?' or '&' would otherwise + // reshape the parsed URI (inject libpq params / alter host). Bracketed IPv6 + // literals ([::1]) are passed through unencoded — percent-encoding their + // '['/']'/':' would stop them parsing as a host. + let host = if self.host.starts_with('[') && self.host.ends_with(']') { + self.host.clone() + } else { + urlencoding::encode(&self.host).into_owned() + }; format!( "postgres://{user}:{password}@{host}:{port}/{dbname}?sslmode={sslmode}", user = urlencoding::encode(&self.user.as_deref().unwrap_or("postgres")), password = urlencoding::encode(&self.password.as_deref().unwrap_or("")), - host = &self.host, + host = host, port = self.port.unwrap_or(5432), - dbname = self.dbname, + dbname = urlencoding::encode(&self.dbname), sslmode = sslmode ) } @@ -787,7 +847,7 @@ impl PgDatabase { } /// Validate a database name to prevent SQL injection. -/// Must start with a letter, contain only alphanumeric characters or underscores, and be <= 63 chars. +/// Must start with a letter, contain only alphanumeric characters, underscores, or hyphens, and be <= 63 chars. pub fn validate_dbname(dbname: &str) -> error::Result<()> { let dbname = dbname.trim(); if dbname.is_empty() { @@ -811,10 +871,11 @@ pub fn validate_dbname(dbname: &str) -> error::Result<()> { } if !dbname .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '_') + .all(|c| c.is_ascii_alphanumeric() || c == '_' || c == '-') { return Err(error::Error::BadRequest( - "Database name must contain only alphanumeric characters or underscores".to_string(), + "Database name must contain only alphanumeric characters, underscores, or hyphens" + .to_string(), )); } Ok(()) @@ -1254,8 +1315,12 @@ pub fn get_latest_deployed_hash_for_path<'e>( ) -> impl Future>> + Send + 'e { async move { let cache_key = (w_id.to_string(), script_path.to_string()); + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); let mut computed_hash = None; - let hash = match DEPLOYED_SCRIPT_HASH_CACHE.get(&cache_key) { + let hash = match DEPLOYED_SCRIPT_HASH_CACHE + .get(&cache_key) + .filter(|_| use_cache) + { Some(cached_hash) if cached_hash.expires_at > std::time::Instant::now() && db.as_ref().is_none_or(|x| { @@ -1298,13 +1363,15 @@ pub fn get_latest_deployed_hash_for_path<'e>( }; let hash = utils::not_found_if_none(hash, "script", script_path)?; - DEPLOYED_SCRIPT_HASH_CACHE.insert( - cache_key, - ExpiringLatestVersionId { - id: hash, - expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, - }, - ); + if use_cache { + DEPLOYED_SCRIPT_HASH_CACHE.insert( + cache_key, + ExpiringLatestVersionId { + id: hash, + expires_at: std::time::Instant::now() + LATEST_VERSION_ID_CACHE_TTL, + }, + ); + } hash } @@ -1336,9 +1403,10 @@ pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( hash: i64, ) -> error::Result> { let key = (w_id.to_string(), hash); + let use_cache = !DEPLOYED_SCRIPT_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); let mut computed_hash = None; - match DEPLOYED_SCRIPT_INFO_CACHE.get(&key) { + match DEPLOYED_SCRIPT_INFO_CACHE.get(&key).filter(|_| use_cache) { Some(info) if db_authed.as_ref().is_none_or(|x| { let r = HASH_PERMS_CACHE.check_perms_in_cache(x.authed, scripts::ScriptHash(hash)); @@ -1367,7 +1435,9 @@ pub async fn get_script_info_for_hash<'e, E: sqlx::PgExecutor<'e>>( let info = utils::not_found_if_none(info, "script", &hash.to_string())?; - DEPLOYED_SCRIPT_INFO_CACHE.insert(key, info.clone()); + if use_cache { + DEPLOYED_SCRIPT_INFO_CACHE.insert(key, info.clone()); + } Ok(info) } diff --git a/backend/windmill-common/src/materialization.rs b/backend/windmill-common/src/materialization.rs new file mode 100644 index 0000000000..8d9e31899f --- /dev/null +++ b/backend/windmill-common/src/materialization.rs @@ -0,0 +1,284 @@ +//! CE materialization state — the per-partition status recorded by the managed +//! `// materialize` write (in windmill-worker), read by the partition-status +//! grid and by the EE backfill worklist. +//! +//! The write engine and this state are CE; only automatic partition +//! *resolution* (`partition_ee`) and *backfill* orchestration +//! (`pipeline_advanced_ee`) are enterprise. This module is the shared seam: +//! the EE backfill enumerates the partitions in a range, diffs them against +//! these rows to find the missing/failed set, and pushes one CE materialization +//! job per gap (with an explicit `partition` arg — which runs idempotently and +//! upserts the row here). Nothing about that orchestration lives in this file; +//! it only needs the rows to exist, which is why recording is CE. + +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; +use sqlx::types::Json; +use sqlx::{PgExecutor, Postgres, Transaction}; +use uuid::Uuid; + +use crate::assets::AssetKind; +use crate::error::Result; + +/// Sentinel `partition` value for an unpartitioned (whole-table) +/// materialization — partition is part of the primary key and cannot be NULL. +pub const UNPARTITIONED: &str = ""; + +/// Mirrors the `MATERIALIZATION_STATUS` pg enum (see migration +/// `20260619170118_add_materialized_partition`). +#[derive(sqlx::Type, Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)] +#[sqlx(type_name = "MATERIALIZATION_STATUS", rename_all = "lowercase")] +#[serde(rename_all = "lowercase")] +pub enum MaterializationStatus { + Running, + Materialized, + Failed, +} + +/// One column of a captured asset output schema: its name and substrate type +/// (e.g. `{"name": "order_id", "type": "BIGINT"}`). `type` is the substrate's +/// own type spelling (DuckDB for ducklake) — kept verbatim so #2b can compare +/// declared vs. captured without a lossy normalization step. +#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)] +pub struct SchemaColumn { + pub name: String, + #[serde(rename = "type")] + pub data_type: String, +} + +/// The materialization outcome an agent worker (`Connection::Http`, no direct +/// DB) sends to the API to be recorded. Mirrors the `record_materialization` +/// args; the API handler unpacks it and calls that function with its own DB. +#[derive(Debug, Clone, Serialize, Deserialize)] +pub struct RecordMaterializationRequest { + pub asset_kind: AssetKind, + pub asset_path: String, + pub partition: String, + pub status: MaterializationStatus, + pub snapshot_id: Option, + pub row_count: Option, + pub job_id: Option, + pub error: Option, + /// Captured output schema of the materialized asset (`None` when the + /// substrate/run produced no schema, e.g. a failed run or a polyglot helper + /// that doesn't DESCRIBE). When present, the recorder also upserts a + /// `materialized_asset_schema` version. Defaults to `None` so older agents + /// stay wire-compatible. + #[serde(default)] + pub schema: Option>, +} + +/// Upsert the latest materialization state for one (asset, partition) slice. +/// The worker records the terminal outcome once the write finishes: +/// `Materialized` (with the DuckLake `snapshot_id` + `row_count`) or `Failed` +/// (with `error`). `Running` mirrors the pg enum but has no writer in this flow. +/// Idempotent: re-running the same partition overwrites the row — exactly the +/// backfill / failure-recovery contract. +#[allow(clippy::too_many_arguments)] +pub async fn record_materialization<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, + partition: &str, + status: MaterializationStatus, + snapshot_id: Option, + row_count: Option, + job_id: Option, + error: Option<&str>, +) -> Result<()> { + sqlx::query!( + "INSERT INTO materialized_partition + (workspace_id, asset_kind, asset_path, partition, status, + snapshot_id, row_count, job_id, materialized_at, error) + VALUES ($1, $2, $3, $4, $5, $6, $7, $8, now(), $9) + ON CONFLICT (workspace_id, asset_kind, asset_path, partition) + DO UPDATE SET status = EXCLUDED.status, + snapshot_id = EXCLUDED.snapshot_id, + row_count = EXCLUDED.row_count, + job_id = EXCLUDED.job_id, + materialized_at = now(), + error = EXCLUDED.error", + workspace_id, + asset_kind as AssetKind, + asset_path, + partition, + status as MaterializationStatus, + snapshot_id, + row_count, + job_id, + error, + ) + .execute(executor) + .await?; + Ok(()) +} + +/// One materialized-partition row, for the status grid / backfill diff. +#[derive(sqlx::FromRow, Debug, Clone, Serialize)] +pub struct MaterializedPartition { + pub asset_kind: AssetKind, + pub asset_path: String, + pub partition: String, + pub status: MaterializationStatus, + pub snapshot_id: Option, + pub row_count: Option, + pub job_id: Option, + pub materialized_at: DateTime, + pub error: Option, +} + +/// All recorded partitions for one asset, newest first — the grid's data and +/// the backfill worklist's "what already exists" set. +pub async fn list_materialized_partitions<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, +) -> Result> { + let rows = sqlx::query_as!( + MaterializedPartition, + r#"SELECT asset_kind AS "asset_kind: AssetKind", asset_path, partition, + status AS "status: MaterializationStatus", snapshot_id, + row_count, job_id, materialized_at, error + FROM materialized_partition + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY partition DESC"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_all(executor) + .await?; + Ok(rows) +} + +/// One captured schema version of an asset, newest first — the schema-evolution +/// history surfaced on the asset node and read by #2b contract enforcement. +#[derive(sqlx::FromRow, Debug, Clone, Serialize)] +pub struct AssetSchemaVersion { + pub version: i64, + pub columns: Json>, + pub snapshot_id: Option, + pub job_id: Option, + pub captured_at: DateTime, +} + +/// Record the captured output schema of a freshly-materialized asset. +/// +/// **Authorization:** like the sibling `record_materialization`, this performs +/// no access control of its own — it writes the row for whatever `workspace_id` +/// it is given. Callers MUST pass a workspace-authorized executor and a +/// `workspace_id` the caller is allowed to write: an RLS-scoped `user_db` +/// transaction for API / agent-worker entry points, or the trusted worker DB +/// pool for the in-worker recorder. Do not expose it to an unauthenticated path. +/// +/// Versioning across re-materializations: a new `version` row is inserted only +/// when `columns` differs from the latest stored version; an unchanged +/// re-materialize re-affirms the latest row in place (updates its +/// `snapshot_id`/`job_id`/`captured_at`). The result is a compact +/// schema-evolution history where `MAX(version)` is the current contract. +/// +/// Runs in a transaction guarded by a per-asset advisory lock so two concurrent +/// materializations of the same asset can't both insert the same next version +/// or interleave a stale comparison. Returns `true` if a new version was +/// inserted (the schema changed), `false` if the latest was re-affirmed. +pub async fn record_asset_schema( + tx: &mut Transaction<'_, Postgres>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, + columns: &[SchemaColumn], + snapshot_id: Option, + job_id: Option, +) -> Result { + // Serialize concurrent captures of the *same* asset; the lock auto-releases + // at tx end. Hash the identity into the bigint advisory-lock key space. + sqlx::query!( + "SELECT pg_advisory_xact_lock(hashtextextended($1, 0::int8))", + format!("materialized_asset_schema:{workspace_id}:{asset_kind:?}:{asset_path}"), + ) + .fetch_one(&mut **tx) + .await?; + + let latest = sqlx::query!( + r#"SELECT version, columns AS "columns: Json>" + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY version DESC + LIMIT 1"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_optional(&mut **tx) + .await?; + + let columns_json = Json(columns.to_vec()); + let next_version = match latest { + Some(latest) if latest.columns.0.as_slice() == columns => { + // Unchanged schema — re-affirm the latest version in place. + sqlx::query!( + "UPDATE materialized_asset_schema + SET snapshot_id = $5, job_id = $6, captured_at = now() + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + AND version = $4", + workspace_id, + asset_kind as AssetKind, + asset_path, + latest.version, + snapshot_id, + job_id, + ) + .execute(&mut **tx) + .await?; + return Ok(false); + } + Some(latest) => latest.version + 1, + None => 1, + }; + sqlx::query!( + "INSERT INTO materialized_asset_schema + (workspace_id, asset_kind, asset_path, version, columns, + snapshot_id, job_id, captured_at) + VALUES ($1, $2, $3, $4, $5, $6, $7, now())", + workspace_id, + asset_kind as AssetKind, + asset_path, + next_version, + columns_json as Json>, + snapshot_id, + job_id, + ) + .execute(&mut **tx) + .await?; + Ok(true) +} + +/// All captured schema versions for one asset, newest version first. +/// +/// **Authorization:** performs no access control (mirrors +/// `list_materialized_partitions`); the caller must pass a workspace-authorized +/// executor (an RLS-scoped `user_db` transaction on the API read path) and a +/// `workspace_id` it is allowed to read. +pub async fn list_asset_schemas<'e>( + executor: impl PgExecutor<'e>, + workspace_id: &str, + asset_kind: AssetKind, + asset_path: &str, +) -> Result> { + let rows = sqlx::query_as!( + AssetSchemaVersion, + r#"SELECT version, columns AS "columns: Json>", + snapshot_id, job_id, captured_at + FROM materialized_asset_schema + WHERE workspace_id = $1 AND asset_kind = $2 AND asset_path = $3 + ORDER BY version DESC"#, + workspace_id, + asset_kind as AssetKind, + asset_path, + ) + .fetch_all(executor) + .await?; + Ok(rows) +} diff --git a/backend/windmill-common/src/partition_oss.rs b/backend/windmill-common/src/partition_oss.rs new file mode 100644 index 0000000000..0e9c05ac36 --- /dev/null +++ b/backend/windmill-common/src/partition_oss.rs @@ -0,0 +1,55 @@ +//! OSS fallback for pipeline partition resolution. +//! +//! Partition resolution is a `private` feature (see `partition_ee`). In the +//! public build it is absent: `resolve_partition` yields no partition, so the +//! cascade runs partition-agnostically, and the args helpers degrade to a +//! plain (non-partition-preserving) write. The real implementation lives in +//! `windmill-ee-private`. + +use std::collections::HashMap; + +use chrono::{DateTime, Utc}; +use serde_json::value::RawValue; +use sqlx::types::Json; +use sqlx::PgExecutor; +use uuid::Uuid; +use windmill_parser::asset_parser::PartitionSpec; + +use crate::error::Result; + +/// Well-known arg key the resolved partition value is injected under. +pub const PARTITION_ARG: &str = "partition"; + +/// No-op: OSS never resolves a partition, so there is nothing to persist. +pub async fn set_resolved_partition<'e>( + _executor: impl PgExecutor<'e>, + _job_id: Uuid, + _value: &str, +) -> Result<()> { + Ok(()) +} + +/// Plain args replace — OSS has no resolved partition to carry forward. +pub async fn merge_args_preserving_partition<'e>( + executor: impl PgExecutor<'e>, + job_id: Uuid, + new_args: HashMap>, +) -> Result<()> { + sqlx::query!( + "UPDATE v2_job SET args = $1, preprocessed = TRUE WHERE id = $2", + Json(new_args) as Json>>, + job_id, + ) + .execute(executor) + .await?; + Ok(()) +} + +/// No partition in the OSS build. +pub fn resolve_partition( + _spec: &PartitionSpec, + _at: DateTime, + _payload: Option<&serde_json::Value>, +) -> Result> { + Ok(None) +} diff --git a/backend/windmill-common/src/pipeline_advanced_oss.rs b/backend/windmill-common/src/pipeline_advanced_oss.rs new file mode 100644 index 0000000000..009e783bd5 --- /dev/null +++ b/backend/windmill-common/src/pipeline_advanced_oss.rs @@ -0,0 +1,16 @@ +//! OSS fallback: pipeline freshness/SLA enforcement and partition backfills +//! are enterprise features; their implementations live in windmill-ee-private +//! (see `pipeline_advanced_ee`). In the public build the entry points report +//! that the enterprise edition is required. + +use crate::error::Error; + +pub fn freshness_enforcement_todo() -> Error { + Error::internal_err( + "Pipeline freshness/SLA enforcement requires the enterprise edition".to_string(), + ) +} + +pub fn backfill_todo() -> Error { + Error::internal_err("Pipeline partition backfill requires the enterprise edition".to_string()) +} diff --git a/backend/windmill-common/src/query_builders.rs b/backend/windmill-common/src/query_builders.rs index f077687d6f..ce7dbc2533 100644 --- a/backend/windmill-common/src/query_builders.rs +++ b/backend/windmill-common/src/query_builders.rs @@ -172,6 +172,19 @@ pub struct SimpleColumn { pub struct SelectOptions { pub limit: Option, pub offset: Option, + /// DuckLake time-travel: when set (DuckDB only), the read is pinned to this + /// catalog snapshot via `AT (VERSION => n)`. Ignored for other db types. + pub version: Option, +} + +/// DuckLake time-travel suffix appended after a table name in a FROM clause. +/// `n` is a server-controlled `i64` (a snapshot id), so inlining it is +/// injection-safe. Empty string when unpinned (reads the latest snapshot). +fn duckdb_version_suffix(version: Option) -> String { + match version { + Some(v) => format!(" AT (VERSION => {})", v), + None => String::new(), + } } // --------------------------------------------------------------------------- @@ -190,6 +203,8 @@ struct SelectPayload { #[serde(rename = "fixPgIntTypes")] fix_pg_int_types: Option, ducklake: Option, + /// DuckLake snapshot to time-travel the read to (DuckDB only). + version: Option, } #[derive(Deserialize)] @@ -200,6 +215,21 @@ struct CountPayload { #[serde(rename = "whereClause")] where_clause: Option, ducklake: Option, + /// DuckLake snapshot to time-travel the count to (DuckDB only). + version: Option, +} + +/// `WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS` payload — lists the time-travel history +/// of a ducklake table. DuckLake snapshots are catalog-wide commits, so without +/// a `table` this lists every commit; with one it is scoped to snapshots where +/// that table exists (see `expand_ducklake_snapshots`). +#[derive(Deserialize)] +struct DucklakeSnapshotsPayload { + ducklake: String, + /// Schema-qualified table name (e.g. `main.events_daily`) to scope the + /// history to. Snapshots predating the table's creation are excluded — a + /// time-travel read can't target a version where the table didn't exist. + table: Option, } #[derive(Deserialize)] @@ -304,6 +334,10 @@ pub fn try_expand_internal_db_query( expand_primary_key_constraint(json_str, db_type).map(ExpandedQuery::sql) } "SNOWFLAKE_PRIMARY_KEYS" => expand_snowflake_primary_keys(json_str).map(ExpandedQuery::sql), + // DuckLake time-travel: list a ducklake's snapshot history + "DUCKLAKE_SNAPSHOTS" => { + expand_ducklake_snapshots(json_str, db_type).map(ExpandedQuery::sql) + } _ => Err(format!("Unknown WM_INTERNAL_DB operation: {}", op)), }; @@ -324,7 +358,8 @@ fn expand_select(json_str: &str, db_type: DbType) -> Result { let payload: SelectPayload = serde_json::from_str(json_str).map_err(|e| format!("Invalid SELECT payload: {}", e))?; - let options = SelectOptions { limit: payload.limit, offset: payload.offset }; + let options = + SelectOptions { limit: payload.limit, offset: payload.offset, version: payload.version }; let breaking = payload .fix_pg_int_types .map(|v| BreakingFeatures { fix_pg_int_types: v }); @@ -350,11 +385,47 @@ fn expand_count(json_str: &str, db_type: DbType) -> Result { &payload.table, payload.where_clause.as_deref(), &payload.column_defs, + payload.version, )?; Ok(maybe_wrap_ducklake(query, payload.ducklake.as_deref())) } +/// Expand `DUCKLAKE_SNAPSHOTS` into the catalog's time-travel history. DuckLake +/// snapshots are catalog-wide commits, so `ducklake_snapshots('dl')` (the alias +/// `maybe_wrap_ducklake` attaches) lists every version any `AT (VERSION => n)` +/// read can target, newest first. +fn expand_ducklake_snapshots(json_str: &str, db_type: DbType) -> Result { + if db_type != DbType::Duckdb { + return Err("DUCKLAKE_SNAPSHOTS is only supported for DuckDB".to_string()); + } + let payload: DucklakeSnapshotsPayload = serde_json::from_str(json_str) + .map_err(|e| format!("Invalid DUCKLAKE_SNAPSHOTS payload: {}", e))?; + // `dl` is the alias `wrap_ducklake_query` attaches and `USE`s below. + let query = match &payload.table { + // Scope to snapshots from the table's first creation onward. A DuckLake + // table created at snapshot N can't be read before N (the catalog-wide + // list would otherwise offer impossible versions). The creation snapshot + // is the earliest whose `changes.tables_created` names the table; + // COALESCE to 0 (show all) if it is never found. + Some(table) => { + let table = escape_sql_literal(table); + format!( + "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') \ + WHERE snapshot_id >= COALESCE((\ + SELECT min(snapshot_id) FROM ducklake_snapshots('dl') \ + WHERE list_contains(changes.tables_created, '{table}')), 0) \ + ORDER BY snapshot_id DESC" + ) + } + None => { + "SELECT snapshot_id, snapshot_time FROM ducklake_snapshots('dl') ORDER BY snapshot_id DESC" + .to_string() + } + }; + Ok(maybe_wrap_ducklake(query, Some(&payload.ducklake))) +} + /// Filter columns to primary keys only; fall back to all columns if none are marked. fn pk_columns_or_all(columns: &[ColumnDef]) -> Vec { let pks: Vec = columns.iter().filter(|c| c.isprimarykey).cloned().collect(); @@ -950,9 +1021,10 @@ pub fn make_select_query( ); query.push_str(&format!( - "SELECT {} FROM {}\n", + "SELECT {} FROM {}{}\n", filtered_columns.join(", "), - quote_table_name(table, db_type) + quote_table_name(table, db_type), + duckdb_version_suffix(options.and_then(|o| o.version)) )); query.push_str(&format!( " WHERE {} {}\n", @@ -977,6 +1049,8 @@ pub fn make_count_query( table: &str, where_clause: Option<&str>, column_defs: &[ColumnDef], + // DuckLake time-travel snapshot (DuckDB only); `None` counts the latest. + version: Option, ) -> Result { let where_prefix = " WHERE "; let and_condition = " AND "; @@ -1118,8 +1192,9 @@ pub fn make_count_query( quicksearch_condition.push_str(" ($quicksearch = '' OR 1 = 1)"); } query.push_str(&format!( - "SELECT COUNT(*) as count FROM {}", - quote_table_name(table, db_type) + "SELECT COUNT(*) as count FROM {}{}", + quote_table_name(table, db_type), + duckdb_version_suffix(version) )); } } @@ -1739,7 +1814,7 @@ struct PrimaryKeyConstraintPayload { fn db_supports_schemas(db_type: DbType) -> bool { matches!( db_type, - DbType::Postgresql | DbType::Snowflake | DbType::Bigquery + DbType::Postgresql | DbType::Snowflake | DbType::Bigquery | DbType::Duckdb ) } @@ -2410,8 +2485,15 @@ fn make_load_table_metadata_query( ) -> Result { match db_type { DbType::Duckdb => { - // For ducklake, the ducklake ATTACH is handled by the ducklake wrapper. - let mut q = String::from( + // For ducklake, the ducklake ATTACH is handled by the ducklake wrapper, so the + // ducklake catalog is the current database. information_schema spans every attached + // catalog, so we always scope to current_database() to stay within the ducklake. + let extra_col = if table.is_none() { + ",\n TABLE_SCHEMA as schema_name" + } else { + "" + }; + let mut q = format!( "SELECT COLUMN_NAME as field, DATA_TYPE as DataType, @@ -2420,12 +2502,20 @@ fn make_load_table_metadata_query( false as IsIdentity, CASE WHEN IS_NULLABLE = true THEN 'YES' ELSE 'NO' END as IsNullable, false as IsEnum, - TABLE_NAME as table_name + TABLE_NAME as table_name{} FROM information_schema.columns c -WHERE table_schema = current_schema()", +WHERE table_catalog = current_database()", + extra_col ); if let Some(t) = table { - q.push_str(&format!(" AND TABLE_NAME = '{}'", escape_sql_literal(t))); + let parts: Vec<&str> = t.split('.').collect(); + let tname = parts[parts.len() - 1]; + let schema = if parts.len() > 1 { parts[0] } else { "main" }; + q.push_str(&format!( + " AND TABLE_NAME = '{}' AND TABLE_SCHEMA = '{}'", + escape_sql_literal(tname), + escape_sql_literal(schema) + )); } Ok(q) } @@ -2983,7 +3073,7 @@ mod tests { #[test] fn test_select_snowflake_custom_limit() { let cols = vec![col("id", "int")]; - let opts = SelectOptions { limit: Some(50), offset: Some(10) }; + let opts = SelectOptions { limit: Some(50), offset: Some(10), version: None }; let result = make_select_query( "my_table", &cols, @@ -3057,7 +3147,7 @@ mod tests { #[test] fn test_count_postgresql_basic() { let cols = vec![col("id", "int4"), col("name", "text")]; - let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- $1 quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\"")); @@ -3075,6 +3165,7 @@ mod tests { "my_table", Some("status = 'active'"), &cols, + None, ) .unwrap(); @@ -3090,7 +3181,7 @@ mod tests { c.ignored = Some(true); c }]; - let result = make_count_query(DbType::Postgresql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Postgresql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("($1 = '' OR 1 = 1)")); } @@ -3101,7 +3192,7 @@ mod tests { #[test] fn test_count_mysql_basic() { let cols = vec![col("id", "int"), col("name", "varchar")]; - let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- :quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`")); @@ -3115,7 +3206,7 @@ mod tests { #[test] fn test_count_mssql_basic() { let cols = vec![col("id", "int"), col("name", "nvarchar")]; - let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap(); assert!(result.contains("SELECT COUNT(*) as count FROM [my_table]")); assert!(result.contains("(@p1 = '' OR CONCAT([id], [name]) LIKE '%' + @p1 + '%')")); @@ -3128,7 +3219,7 @@ mod tests { #[test] fn test_count_snowflake_basic() { let cols = vec![col("id", "int"), col("name", "text")]; - let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap(); // Two quicksearch params for snowflake with visible columns assert!(result.contains("-- ? quicksearch (text)\n-- ? quicksearch (text)")); @@ -3143,7 +3234,7 @@ mod tests { c.ignored = Some(true); c }]; - let result = make_count_query(DbType::Snowflake, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Snowflake, "my_table", None, &cols, None).unwrap(); // One quicksearch param let param_lines: Vec<&str> = result.lines().filter(|l| l.starts_with("-- ?")).collect(); assert_eq!(param_lines.len(), 1); @@ -3157,7 +3248,7 @@ mod tests { #[test] fn test_count_bigquery_basic() { let cols = vec![col("id", "INTEGER"), col("name", "STRING")]; - let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- @quicksearch (string)")); assert!(result.contains("SELECT COUNT(*) as count FROM `my_table`")); @@ -3167,7 +3258,7 @@ mod tests { #[test] fn test_count_bigquery_json_type() { let cols = vec![col("id", "INTEGER"), col("data", "JSON")]; - let result = make_count_query(DbType::Bigquery, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Bigquery, "my_table", None, &cols, None).unwrap(); assert!(result.contains("TO_JSON_STRING(`data`)")); } @@ -3178,7 +3269,7 @@ mod tests { #[test] fn test_count_duckdb_basic() { let cols = vec![col("id", "int"), col("name", "text")]; - let result = make_count_query(DbType::Duckdb, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Duckdb, "my_table", None, &cols, None).unwrap(); assert!(result.contains("-- $quicksearch (text)")); assert!(result.contains("SELECT COUNT(*) as count FROM \"my_table\"")); @@ -3187,6 +3278,74 @@ mod tests { ); } + // ----------------------------------------------------------------------- + // DuckLake time-travel (AT VERSION) + snapshot history + // ----------------------------------------------------------------------- + + #[test] + fn test_select_duckdb_time_travel() { + let cols = vec![col("id", "int"), col("name", "text")]; + let opts = SelectOptions { limit: None, offset: None, version: Some(42) }; + let result = + make_select_query("orders", &cols, None, DbType::Duckdb, Some(&opts), None).unwrap(); + // Read is pinned to the catalog snapshot via AT (VERSION => n). + assert!(result.contains("FROM \"orders\" AT (VERSION => 42)\n")); + } + + #[test] + fn test_select_duckdb_no_version_unpinned() { + let cols = vec![col("id", "int")]; + let result = make_select_query("orders", &cols, None, DbType::Duckdb, None, None).unwrap(); + // Without a version the read targets the latest snapshot — no AT clause. + assert!(result.contains("FROM \"orders\"\n")); + assert!(!result.contains("AT (VERSION")); + } + + #[test] + fn test_count_duckdb_time_travel() { + let cols = vec![col("id", "int")]; + let result = make_count_query(DbType::Duckdb, "orders", None, &cols, Some(7)).unwrap(); + assert!(result.contains("FROM \"orders\" AT (VERSION => 7)")); + } + + #[test] + fn test_version_ignored_for_non_duckdb() { + // AT (VERSION) is DuckLake-only; other dialects must never emit it even + // if a version is somehow passed through. + let cols = vec![col("id", "int4")]; + let opts = SelectOptions { limit: None, offset: None, version: Some(5) }; + let result = + make_select_query("orders", &cols, None, DbType::Postgresql, Some(&opts), None) + .unwrap(); + assert!(!result.contains("AT (VERSION")); + } + + #[test] + fn test_expand_ducklake_snapshots() { + let json = r#"{"ducklake": "analytics"}"#; + let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap(); + assert!(result.contains("ATTACH 'ducklake://analytics' AS dl;USE dl;")); + assert!(result.contains("ducklake_snapshots('dl')")); + assert!(result.contains("ORDER BY snapshot_id DESC")); + // Unscoped: no per-table existence filter. + assert!(!result.contains("tables_created")); + } + + #[test] + fn test_expand_ducklake_snapshots_scoped_to_table() { + let json = r#"{"ducklake": "analytics", "table": "main.events_daily"}"#; + let result = expand_ducklake_snapshots(json, DbType::Duckdb).unwrap(); + // Scoped to snapshots from the table's first creation onward. + assert!(result.contains("list_contains(changes.tables_created, 'main.events_daily')")); + assert!(result.contains("snapshot_id >= COALESCE")); + } + + #[test] + fn test_expand_ducklake_snapshots_non_duckdb_errors() { + let json = r#"{"ducklake": "analytics"}"#; + assert!(expand_ducklake_snapshots(json, DbType::Postgresql).is_err()); + } + // ----------------------------------------------------------------------- // DELETE - all DB types // ----------------------------------------------------------------------- @@ -3485,7 +3644,7 @@ mod tests { #[test] fn test_count_mssql_no_where() { let cols = vec![col("id", "int")]; - let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::MsSqlServer, "my_table", None, &cols, None).unwrap(); // MSSQL uses WHERE directly (no AND replacement) assert!(result.contains("SELECT COUNT(*) as count FROM [my_table] WHERE ")); } @@ -3493,7 +3652,7 @@ mod tests { #[test] fn test_count_mysql_no_where_uses_where_keyword() { let cols = vec![col("id", "int")]; - let result = make_count_query(DbType::Mysql, "my_table", None, &cols).unwrap(); + let result = make_count_query(DbType::Mysql, "my_table", None, &cols, None).unwrap(); // The AND should be replaced with WHERE assert!(result.contains("FROM `my_table` WHERE ")); assert!(!result.contains("FROM `my_table` AND ")); @@ -3722,9 +3881,10 @@ mod tests { table_ref("users", Some("myschema"), DbType::Mysql), "`users`" ); + // DuckDB (ducklake) supports schemas assert_eq!( table_ref("users", Some("myschema"), DbType::Duckdb), - r#""users""# + r#""myschema"."users""# ); } @@ -3856,6 +4016,13 @@ mod tests { assert!(sql.contains("DROP TABLE \"users\";")); } + #[test] + fn test_expand_drop_table_ducklake_with_schema() { + let marker = r#"-- WM_INTERNAL_DB_DROP_TABLE {"table":"events","schema":"analytics","ducklake":"my_lake"}"#; + let sql = expand_code(marker, &ScriptLang::DuckDb); + assert!(sql.contains("DROP TABLE \"analytics\".\"events\";")); + } + // ----------------------------------------------------------------------- // CREATE SCHEMA / DROP SCHEMA // ----------------------------------------------------------------------- @@ -4355,7 +4522,27 @@ mod tests { let marker = r#"-- WM_INTERNAL_DB_LOAD_TABLE_METADATA {"table":"users","ducklake":"lake"}"#; let sql = expand_code(marker, &ScriptLang::DuckDb); assert!(sql.starts_with("ATTACH 'ducklake://lake' AS dl;USE dl;\n")); - assert!(sql.contains("TABLE_NAME = 'users'")); + assert!(sql.contains("table_catalog = current_database()")); + // Unqualified table defaults to the "main" schema. + assert!(sql.contains("TABLE_NAME = 'users' AND TABLE_SCHEMA = 'main'")); + } + + #[test] + fn test_expand_load_table_metadata_ducklake_qualified_schema() { + let marker = r#"-- WM_INTERNAL_DB_LOAD_TABLE_METADATA {"table":"analytics.events","ducklake":"lake"}"#; + let sql = expand_code(marker, &ScriptLang::DuckDb); + assert!(sql.contains("TABLE_NAME = 'events' AND TABLE_SCHEMA = 'analytics'")); + } + + #[test] + fn test_expand_load_table_metadata_ducklake_all_tables() { + let marker = r#"-- WM_INTERNAL_DB_LOAD_TABLE_METADATA {"ducklake":"lake"}"#; + let sql = expand_code(marker, &ScriptLang::DuckDb); + assert!(sql.starts_with("ATTACH 'ducklake://lake' AS dl;USE dl;\n")); + // All-tables listing scopes to the ducklake catalog and exposes the schema per table. + assert!(sql.contains("table_catalog = current_database()")); + assert!(sql.contains("TABLE_SCHEMA as schema_name")); + assert!(!sql.contains("TABLE_NAME = '")); } // ----------------------------------------------------------------------- diff --git a/backend/windmill-common/src/runnable_settings/mod.rs b/backend/windmill-common/src/runnable_settings/mod.rs index c5692d40bd..33d4c1a039 100644 --- a/backend/windmill-common/src/runnable_settings/mod.rs +++ b/backend/windmill-common/src/runnable_settings/mod.rs @@ -52,13 +52,10 @@ pub trait RunnableSettingsTrait: /// get [[Self]] from cache or fetch from db /// if not found, returns Error - fn get<'a>( + fn get<'e>( hash: i64, - db: &'a Pool, - ) -> impl Future> - where - Self: 'a, - { + db: impl sqlx::PgExecutor<'e>, + ) -> impl Future> { async move { let v = RUNNABLE_INDIVIDUAL_SETTINGS .get_or_insert_async(hash, async { diff --git a/backend/windmill-common/src/runnable_settings/settings.rs b/backend/windmill-common/src/runnable_settings/settings.rs index 04b62fbaab..b814646553 100644 --- a/backend/windmill-common/src/runnable_settings/settings.rs +++ b/backend/windmill-common/src/runnable_settings/settings.rs @@ -1,5 +1,3 @@ -use std::future::Future; - use sqlx::{Pool, Postgres}; use crate::{ @@ -38,29 +36,63 @@ pub async fn prefetch_cached_from_handle( prefetch_cached(&rs, db).await } +/// Like [`prefetch_cached`], but reuses a held transaction's connection instead +/// of checking out a second one from the pool. Use this on paths that already +/// hold an open `tx` to avoid dual-connection pool contention. +pub async fn prefetch_cached_tx( + rs: &RunnableSettings, + tx: &mut sqlx::Transaction<'_, Postgres>, +) -> error::Result<(DebouncingSettings, ConcurrencySettings)> { + Ok(( + if let Some(hash) = rs.debouncing_settings { + DebouncingSettings::get(hash, &mut **tx).await? + } else { + Default::default() + }, + if let Some(hash) = rs.concurrency_settings { + ConcurrencySettings::get(hash, &mut **tx).await? + } else { + Default::default() + }, + )) +} + +/// Resolve the retry policy (if any) for a job from its `runnable_settings_handle`. +/// Returns `None` when the job carries no retry policy. Read lazily on the +/// failure path only — never on the hot job-pull path. +pub async fn prefetch_retry_from_handle( + hash: Option, + db: &DB, +) -> error::Result> { + let rs = from_handle(hash, db).await?; + Ok(if let Some(hash) = rs.retry_settings { + Some(RetrySettings::get(hash, db).await?) + } else { + None + }) +} + /// Returns error if provided `hash` has no corresponding entry in db /// If `hash` is None, returns Default -pub fn from_handle<'a>( +pub async fn from_handle<'e>( hash: Option, - db: &'a DB, -) -> impl Future> + 'a { - async move { - if let Some(hash) = hash { - super::RUNNABLE_SETTINGS_REFERENCES - .get_or_insert_async(hash, async { - sqlx::query_as!( - RunnableSettings, - r#"SELECT concurrency_settings, debouncing_settings FROM runnable_settings WHERE hash = $1"#, - hash - ) - .fetch_one(db) - .await - .map_err(error::Error::from) - }) + db: impl sqlx::PgExecutor<'e>, +) -> error::Result { + if let Some(hash) = hash { + super::RUNNABLE_SETTINGS_REFERENCES + .get_or_insert_async(hash, async { + sqlx::query_as!( + RunnableSettings, + r#"SELECT concurrency_settings, debouncing_settings, retry_settings FROM runnable_settings WHERE hash = $1"#, + hash + ) + .fetch_one(db) .await - } else { - Ok(RunnableSettings::default()) - } + .map_err(error::Error::from) + }) + .await + } else { + Ok(RunnableSettings::default()) } } @@ -68,7 +100,9 @@ pub async fn insert_rs(rs: RunnableSettings, db: &Pool) -> error::Resu use std::hash::{Hash, Hasher}; if !min_version_supports_runnable_settings_v0().await - || (rs.debouncing_settings.is_none() && rs.concurrency_settings.is_none()) + || (rs.debouncing_settings.is_none() + && rs.concurrency_settings.is_none() + && rs.retry_settings.is_none()) { return Ok(None); } @@ -82,13 +116,14 @@ pub async fn insert_rs(rs: RunnableSettings, db: &Pool) -> error::Resu super::RUNNABLE_SETTINGS_REFERENCES .get_or_insert_async(hash, async { sqlx::query!( - "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings) - VALUES ($1, $2, $3) + "INSERT INTO runnable_settings (hash, debouncing_settings, concurrency_settings, retry_settings) + VALUES ($1, $2, $3, $4) ON CONFLICT (hash) DO NOTHING", hash, rs.debouncing_settings, - rs.concurrency_settings + rs.concurrency_settings, + rs.retry_settings ) .execute(db) .await?; @@ -132,3 +167,25 @@ impl super::private_mod::RunnableSettingsTraitInternal for ConcurrencySettings { } } impl super::RunnableSettingsTrait for ConcurrencySettings {} +impl super::private_mod::RunnableSettingsTraitInternal for RetrySettings { + const SETTINGS_NAME: &str = "retry_settings"; + const INCLUDE_FIELDS: &[&str] = &[ + "constant_attempts", + "constant_seconds", + "exponential_attempts", + "exponential_multiplier", + "exponential_seconds", + "exponential_random_factor", + "retry_if_expr", + ]; + fn bind_arguments<'a>(&'a self, q: Q<'a>) -> Q<'a> { + q.bind(&self.constant_attempts) + .bind(&self.constant_seconds) + .bind(&self.exponential_attempts) + .bind(&self.exponential_multiplier) + .bind(&self.exponential_seconds) + .bind(&self.exponential_random_factor) + .bind(&self.retry_if_expr) + } +} +impl super::RunnableSettingsTrait for RetrySettings {} diff --git a/backend/windmill-common/src/runtime_assets.rs b/backend/windmill-common/src/runtime_assets.rs index ecd863f482..70185ca48c 100644 --- a/backend/windmill-common/src/runtime_assets.rs +++ b/backend/windmill-common/src/runtime_assets.rs @@ -127,12 +127,19 @@ async fn prune_runtime_assets( .map(|((w, p, k), v)| (w.clone(), p.clone(), k.clone(), (max_n - v.len()) as i32)) .multiunzip(); + // Delete the surplus job rows by `id`, NOT by `(workspace_id, path, + // kind)`. Deleting by tuple would also wipe the static + // `usage_kind='script'|'flow'` rows that share the same path+kind (a + // producer's persisted write/read lineage), which silently breaks the + // asset-trigger cascade (`fetch_producer_writes` finds no writes). The + // inner query already scopes to `usage_kind='job'`; keep the delete + // scoped to exactly those over-cap rows. let delete_result = sqlx::query!( r#" DELETE FROM asset - WHERE (workspace_id, path, kind) IN ( - SELECT workspace_id, path, kind FROM ( - SELECT a.workspace_id, a.path, a.kind, a.usage_kind, ROW_NUMBER() OVER ( + WHERE id IN ( + SELECT id FROM ( + SELECT a.id, ROW_NUMBER() OVER ( PARTITION BY a.workspace_id, a.path, a.kind ORDER BY a.created_at DESC ) as rn, @@ -140,14 +147,14 @@ async fn prune_runtime_assets( FROM asset a INNER JOIN ( SELECT * FROM UNNEST( - $1::varchar[], - $2::varchar[], + $1::varchar[], + $2::varchar[], $3::asset_kind[], $4::int[] ) AS t(workspace_id, path, kind, max_n) ) limits - ON a.workspace_id = limits.workspace_id - AND a.path = limits.path + ON a.workspace_id = limits.workspace_id + AND a.path = limits.path AND a.kind = limits.kind WHERE a.usage_kind = 'job' ) ranked diff --git a/backend/windmill-common/src/scripts.rs b/backend/windmill-common/src/scripts.rs index 366ed75e8e..f3abf14e1c 100644 --- a/backend/windmill-common/src/scripts.rs +++ b/backend/windmill-common/src/scripts.rs @@ -97,7 +97,6 @@ pub async fn prefetch_cached_script( language: script.language, kind: script.kind, tag: script.tag, - draft_only: script.draft_only, envs: script.envs, dedicated_worker: script.dedicated_worker, ws_error_handler_muted: script.ws_error_handler_muted, @@ -346,10 +345,10 @@ pub async fn clone_script<'c>( ))); }; - let rs = - runnable_settings::from_handle(s.runnable_settings.runnable_settings_handle, db).await?; + let rs = runnable_settings::from_handle(s.runnable_settings.runnable_settings_handle, &mut *tx) + .await?; let (debouncing_settings, concurrency_settings) = - runnable_settings::prefetch_cached(&rs, db).await?; + runnable_settings::prefetch_cached_tx(&rs, &mut tx).await?; let ns = NewScript { path: s.path.clone(), @@ -363,7 +362,6 @@ pub async fn clone_script<'c>( language: s.language, kind: Some(s.kind), tag: s.tag, - draft_only: s.draft_only, envs: s.envs, concurrency_settings: concurrency_settings.maybe_fallback( s.runnable_settings.concurrency_key, @@ -410,14 +408,14 @@ pub async fn clone_script<'c>( INSERT INTO script (workspace_id, hash, path, parent_hashes, summary, description, content, \ created_by, schema, is_template, extra_perms, lock, language, kind, tag, \ - draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, \ + envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, \ dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \ delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, \ codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels) SELECT workspace_id, $1, path, array_prepend($2::bigint, COALESCE(parent_hashes, '{}'::bigint[])), summary, description, \ content, created_by, schema, is_template, extra_perms, NULL, language, kind, tag, \ - draft_only, envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, \ + envs, concurrent_limit, concurrency_time_window_s, cache_ttl, cache_ignore_s3_path, \ dedicated_worker, ws_error_handler_muted, priority, restart_unless_cancelled, \ delete_after_use, delete_after_secs, timeout, concurrency_key, visible_to_runner_only, auto_kind, \ codebase, has_preprocessor, on_behalf_of_email, schema_validation, assets, debounce_key, debounce_delay_s, runnable_settings_handle, modules, labels diff --git a/backend/windmill-common/src/secret_backend/mod.rs b/backend/windmill-common/src/secret_backend/mod.rs index 71fa2ea999..596539cd08 100644 --- a/backend/windmill-common/src/secret_backend/mod.rs +++ b/backend/windmill-common/src/secret_backend/mod.rs @@ -13,6 +13,9 @@ //! vaults like HashiCorp Vault (Enterprise Edition). pub mod database; +pub mod resolver; + +pub use resolver::*; #[cfg(feature = "private")] pub mod vault_ee; diff --git a/backend/windmill-common/src/secret_backend/resolver.rs b/backend/windmill-common/src/secret_backend/resolver.rs new file mode 100644 index 0000000000..949300a3d6 --- /dev/null +++ b/backend/windmill-common/src/secret_backend/resolver.rs @@ -0,0 +1,324 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2024 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +//! Resolution of the configured secret backend. +//! +//! Lives in `windmill-common` (rather than the API/store crates) so that +//! lower-level helpers such as [`crate::variables::get_variable_or_self`] can +//! route secret reads through the configured backend. With an external backend +//! (Vault / Azure Key Vault / AWS Secrets Manager), the `variable.value` column +//! holds a `$vault:`/`$azure_kv:`/`$aws_sm:` marker rather than base64 +//! ciphertext, so decrypting it directly fails — reads must go through the +//! backend instead. +//! +//! Note: external backends require Enterprise Edition. The OSS version only +//! supports the database backend. + +use std::sync::Arc; + +use crate::{ + db::DB, + error::{Error, Result}, + secret_backend::{database::DatabaseBackend, SecretBackend}, + variables::{build_crypt, decrypt}, +}; + +#[cfg(all(feature = "private", feature = "enterprise"))] +use crate::{ + global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, + secret_backend::{ + AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, + AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, + }, +}; + +#[cfg(all(feature = "private", feature = "enterprise"))] +use tokio::sync::RwLock; + +// Cached Vault backend to avoid recreating it for every request +// This enables connection pooling and avoids repeated setup overhead +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedVaultBackend { + backend: Arc, + settings: VaultSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedAzureKvBackend { + backend: Arc, + settings: AzureKeyVaultSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +// Cached AWS Secrets Manager backend +#[cfg(all(feature = "private", feature = "enterprise"))] +struct CachedAwsSmBackend { + backend: Arc, + settings: AwsSecretsManagerSettings, +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +lazy_static::lazy_static! { + static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); +} + +/// Get the current secret backend based on global settings +/// +/// OSS: Always returns DatabaseBackend +/// EE: Returns configured backend (Database or Vault) +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn get_secret_backend(db: &DB) -> Result> { + Ok(Arc::new(DatabaseBackend::new(db.clone()))) +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +pub async fn get_secret_backend(db: &DB) -> Result> { + let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { + Some(value) => serde_json::from_value::(value).unwrap_or_default(), + None => SecretBackendConfig::default(), + }; + + match config { + SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), + SecretBackendConfig::HashiCorpVault(settings) => { + get_or_create_vault_backend(db, settings).await + } + SecretBackendConfig::AzureKeyVault(settings) => { + get_or_create_azure_kv_backend(db, settings).await + } + SecretBackendConfig::AwsSecretsManager(settings) => { + get_or_create_aws_sm_backend(db, settings).await + } + } +} + +/// Get a cached Vault backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_vault_backend( + _db: &DB, + settings: VaultSettings, +) -> Result> { + // Check if we have a cached backend with matching settings (read lock) + { + let cache = VAULT_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + // Need to create a new backend - acquire write lock + let mut cache = VAULT_BACKEND_CACHE.write().await; + + // Double-check (another task may have created it while we waited) + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + // Create new backend + let backend: Arc = { + #[cfg(feature = "openidconnect")] + if settings.token.is_none() { + Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) + } else { + Arc::new(VaultBackend::new(settings.clone())) + } + + #[cfg(not(feature = "openidconnect"))] + Arc::new(VaultBackend::new(settings.clone())) + }; + + // Cache it + *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Get a cached Azure Key Vault backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_azure_kv_backend( + _db: &DB, + settings: AzureKeyVaultSettings, +) -> Result> { + // Check if we have a cached backend with matching settings (read lock) + { + let cache = AZURE_KV_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + // Need to create a new backend - acquire write lock + let mut cache = AZURE_KV_BACKEND_CACHE.write().await; + + // Double-check (another task may have created it while we waited) + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + // Create new backend + let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); + + // Cache it + *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Get a cached AWS SM backend or create a new one if settings changed +#[cfg(all(feature = "private", feature = "enterprise"))] +async fn get_or_create_aws_sm_backend( + _db: &DB, + settings: AwsSecretsManagerSettings, +) -> Result> { + { + let cache = AWS_SM_BACKEND_CACHE.read().await; + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + } + + let mut cache = AWS_SM_BACKEND_CACHE.write().await; + + if let Some(ref cached) = *cache { + if cached.settings == settings { + return Ok(cached.backend.clone()); + } + } + + let backend: Arc = + Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); + + *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); + + Ok(backend) +} + +/// Check if a Vault backend is currently configured +/// +/// OSS: Always returns false +/// EE: Checks global settings +#[cfg(not(all(feature = "private", feature = "enterprise")))] +pub async fn is_vault_backend_configured(_db: &DB) -> Result { + Ok(false) +} + +#[cfg(all(feature = "private", feature = "enterprise"))] +pub async fn is_vault_backend_configured(db: &DB) -> Result { + let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { + Some(value) => serde_json::from_value::(value).unwrap_or_default(), + None => SecretBackendConfig::default(), + }; + + Ok(matches!( + config, + SecretBackendConfig::HashiCorpVault(_) + | SecretBackendConfig::AzureKeyVault(_) + | SecretBackendConfig::AwsSecretsManager(_) + )) +} + +/// Get a secret value using the configured backend +/// +/// For database backend: decrypts `encrypted_value` using the workspace key +/// For external backends (EE only): fetches from the backend at `path`, +/// ignoring `encrypted_value` (which holds only a `$...:` marker) +pub async fn get_secret_value( + db: &DB, + workspace_id: &str, + path: &str, + encrypted_value: &str, +) -> Result { + let backend = get_secret_backend(db).await?; + + match backend.backend_name() { + "database" => { + // Use existing database decryption + let mc = build_crypt(db, workspace_id).await?; + decrypt(&mc, encrypted_value.to_string()).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) + }) + } + "hashicorp_vault" => { + // Fetch from Vault directly + backend.get_secret(workspace_id, path).await + } + "azure_key_vault" => backend.get_secret(workspace_id, path).await, + "aws_secrets_manager" => backend.get_secret(workspace_id, path).await, + _ => Err(Error::internal_err(format!( + "Unknown backend: {}", + backend.backend_name() + ))), + } +} + +/// Check if a value is stored in Vault (indicated by the $vault: prefix) +pub fn is_vault_stored_value(value: &str) -> bool { + value.starts_with("$vault:") +} + +/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) +pub fn is_azure_kv_stored_value(value: &str) -> bool { + value.starts_with("$azure_kv:") +} + +/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix) +pub fn is_aws_sm_stored_value(value: &str) -> bool { + value.starts_with("$aws_sm:") +} + +/// Check if a value is stored in any external secret backend +pub fn is_external_stored_value(value: &str) -> bool { + is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn external_markers_are_detected() { + assert!(is_external_stored_value("$vault:u/admin/secret")); + assert!(is_external_stored_value("$azure_kv:u/admin/secret")); + assert!(is_external_stored_value("$aws_sm:u/admin/secret")); + } + + #[test] + fn base64_ciphertext_is_not_treated_as_external() { + // A base64 magic_crypt blob must route through `decrypt`, never the + // external backend. The leading `$` is what distinguishes a marker from + // ciphertext; decrypting a marker fails with "Invalid byte 36" (`$`), + // which is the bug this gate prevents. + for v in [ + "bm90LWEtbWFya2Vy", + "AAAA1234+/abcd==", + "", + "$something_else", + ] { + assert!(!is_external_stored_value(v), "unexpected external: {v:?}"); + } + } +} diff --git a/backend/windmill-common/src/user_drafts.rs b/backend/windmill-common/src/user_drafts.rs new file mode 100644 index 0000000000..620cbe2a5f --- /dev/null +++ b/backend/windmill-common/src/user_drafts.rs @@ -0,0 +1,549 @@ +/* + * Author: Diego Imbert + * Copyright: Windmill Labs, Inc 2026 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +//! Shared types and helpers for the per-user `draft` table. Lives in +//! `windmill-common` so entity crates can use it without depending on the +//! top-level `windmill-api` crate. Keep it free of HTTP/axum concerns. + +// `DraftUserRef` lives in `windmill-types` (where the list-endpoint row +// structs `ListableScript`/`ListableFlow` declare `Vec` and +// can't reach `windmill-common` without a cycle). Re-exported here so draft +// handlers keep a single import path. +pub use windmill_types::user_drafts::DraftUserRef; + +use crate::db::DB; +use crate::error::Result; +use chrono::{DateTime, Utc}; +use serde::{Deserialize, Serialize}; + +/// Item kinds a user can have an autosaved draft on. Must stay in lockstep +/// with the frontend `USER_DRAFT_ITEM_KINDS` and the Postgres `DRAFT_KIND` +/// enum (adding a kind also needs an `ALTER TYPE ... ADD VALUE` migration). +/// `snake_case` is the shared wire/DB encoding (HTTP params, JSON, `draft.typ`). +#[derive(sqlx::Type, Debug, Clone, Copy, PartialEq, Eq, Hash, Serialize, Deserialize)] +#[sqlx(type_name = "DRAFT_KIND", rename_all = "snake_case")] +#[serde(rename_all = "snake_case")] +pub enum UserDraftItemKind { + Script, + Flow, + App, + RawApp, + Resource, + Variable, + TriggerSchedule, + TriggerWebhook, + TriggerDefaultEmail, + TriggerEmail, + TriggerHttp, + TriggerWebsocket, + TriggerPostgres, + TriggerKafka, + TriggerNats, + TriggerMqtt, + TriggerSqs, + TriggerGcp, + TriggerAzure, + TriggerPoll, + TriggerCli, + TriggerNextcloud, + TriggerGoogle, + TriggerGithub, + /// All unsaved scripts of one data pipeline, bundled into a single draft + /// keyed at the pipeline's folder path. Not a runnable: it has no deployed + /// backing table and is private to its owner. + DataPipeline, +} + +impl UserDraftItemKind { + /// The snake_case wire/DB string, for interpolating into dynamically-built + /// SQL (`?::DRAFT_KIND` binds want a string). + pub fn as_str(&self) -> &'static str { + match self { + UserDraftItemKind::Script => "script", + UserDraftItemKind::Flow => "flow", + UserDraftItemKind::App => "app", + UserDraftItemKind::RawApp => "raw_app", + UserDraftItemKind::Resource => "resource", + UserDraftItemKind::Variable => "variable", + UserDraftItemKind::TriggerSchedule => "trigger_schedule", + UserDraftItemKind::TriggerWebhook => "trigger_webhook", + UserDraftItemKind::TriggerDefaultEmail => "trigger_default_email", + UserDraftItemKind::TriggerEmail => "trigger_email", + UserDraftItemKind::TriggerHttp => "trigger_http", + UserDraftItemKind::TriggerWebsocket => "trigger_websocket", + UserDraftItemKind::TriggerPostgres => "trigger_postgres", + UserDraftItemKind::TriggerKafka => "trigger_kafka", + UserDraftItemKind::TriggerNats => "trigger_nats", + UserDraftItemKind::TriggerMqtt => "trigger_mqtt", + UserDraftItemKind::TriggerSqs => "trigger_sqs", + UserDraftItemKind::TriggerGcp => "trigger_gcp", + UserDraftItemKind::TriggerAzure => "trigger_azure", + UserDraftItemKind::TriggerPoll => "trigger_poll", + UserDraftItemKind::TriggerCli => "trigger_cli", + UserDraftItemKind::TriggerNextcloud => "trigger_nextcloud", + UserDraftItemKind::TriggerGoogle => "trigger_google", + UserDraftItemKind::TriggerGithub => "trigger_github", + UserDraftItemKind::DataPipeline => "data_pipeline", + } + } + + /// Every variant, for code that must enumerate kinds (e.g. generating + /// the `draft_only` existence SQL). + pub const ALL: [UserDraftItemKind; 25] = [ + UserDraftItemKind::Script, + UserDraftItemKind::Flow, + UserDraftItemKind::App, + UserDraftItemKind::RawApp, + UserDraftItemKind::Resource, + UserDraftItemKind::Variable, + UserDraftItemKind::TriggerSchedule, + UserDraftItemKind::TriggerWebhook, + UserDraftItemKind::TriggerDefaultEmail, + UserDraftItemKind::TriggerEmail, + UserDraftItemKind::TriggerHttp, + UserDraftItemKind::TriggerWebsocket, + UserDraftItemKind::TriggerPostgres, + UserDraftItemKind::TriggerKafka, + UserDraftItemKind::TriggerNats, + UserDraftItemKind::TriggerMqtt, + UserDraftItemKind::TriggerSqs, + UserDraftItemKind::TriggerGcp, + UserDraftItemKind::TriggerAzure, + UserDraftItemKind::TriggerPoll, + UserDraftItemKind::TriggerCli, + UserDraftItemKind::TriggerNextcloud, + UserDraftItemKind::TriggerGoogle, + UserDraftItemKind::TriggerGithub, + UserDraftItemKind::DataPipeline, + ]; + + /// The deployed table backing this kind, keyed by `(workspace_id, path)`. + /// SINGLE SOURCE for both the draft access check (which table RLS resolves + /// item-level `extra_perms` against) and the `draft_only` existence check. + /// `None` for kinds with no per-path backing table (webhook is a property + /// of a script/flow row; native triggers are keyed by external_id, not + /// path) — callers treat that as "no deployed counterpart": `draft_only = + /// true` and a path-only access check. + pub fn deployed_table(&self) -> Option<&'static str> { + use UserDraftItemKind::*; + match self { + Script => Some("script"), + Flow => Some("flow"), + App | RawApp => Some("app"), + Resource => Some("resource"), + Variable => Some("variable"), + TriggerSchedule => Some("schedule"), + TriggerHttp => Some("http_trigger"), + TriggerWebsocket => Some("websocket_trigger"), + TriggerPostgres => Some("postgres_trigger"), + TriggerKafka => Some("kafka_trigger"), + TriggerNats => Some("nats_trigger"), + TriggerMqtt => Some("mqtt_trigger"), + TriggerSqs => Some("sqs_trigger"), + TriggerGcp => Some("gcp_trigger"), + TriggerAzure => Some("azure_trigger"), + TriggerEmail | TriggerDefaultEmail => Some("email_trigger"), + TriggerWebhook | TriggerPoll | TriggerCli | TriggerNextcloud | TriggerGoogle + | TriggerGithub => None, + // Keyed at a folder path, not a runnable; access falls back to the + // path-only (folder write) check. + DataPipeline => None, + } + } + + /// Whether OTHER users' drafts at a path are visible to a viewer (the + /// "others are editing" list, owner circles, and the `get_draft_for_user` + /// View JSON / Fork endpoint). Enabled only for the full-page editor items + /// which have the cross-user draft UI. Drawer items keep drafts private to + /// their owner: they have no such UI, and exposing a secret variable draft + /// would hand out the `$encrypted:` ciphertext, which a viewer could + /// launder into plaintext via a deploy. + pub fn shares_drafts_across_users(&self) -> bool { + use UserDraftItemKind::*; + matches!(self, Script | Flow | App | RawApp) + } +} + +/// Query-string flag accepted by every "get by path" route that supports +/// the draft overlay. `#[serde(flatten)]` into a route-specific query struct +/// when the route has other query fields. +#[derive(Debug, Deserialize, Default)] +pub struct WithDraftQuery { + /// When true, attach the authed user's draft (if any) as a separate + /// `draft` field. Defaults to false so non-editor callers see the + /// deployed shape unchanged. + #[serde(default)] + pub get_draft: bool, +} + +/// One row of `other_drafts_users`: a draft on the same path owned by +/// someone other than the authed user. `username` is `None` for the legacy +/// NULL-email row, surfaced in the frontend as a "Legacy draft" entry. +#[derive(Debug, Serialize)] +pub struct OtherDraftUser { + /// `None` represents a legacy workspace-level draft (no owner). + pub username: Option, + /// When this user's draft was last saved (the `draft.created_at` upsert + /// timestamp), surfaced in the fork modal as "Last updated". + pub draft_saved_at: DateTime, +} + +/// Response wrapper: the deployed entity untouched plus the authed user's +/// draft (if any) as a sibling `draft` field, which the frontend pairs to +/// diff/restore/discard. The deployed and the draft are NEVER merged on the +/// server — the editor's saved shape can diverge arbitrarily, so any per-kind +/// translation lives in the frontend loader. `inner` is boxed-erased so a +/// possibly MB-scale deployed payload serializes in ONE pass (no +/// `serde_json::Value` round-trip) while keeping the struct non-generic. +#[derive(Serialize)] +pub struct WithDraftOverlay { + /// Deployed payload, flattened to the top level. + #[serde(flatten)] + pub inner: Box, + pub is_draft: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_saved_at: Option>, + /// True when no deployed row exists at this path: `inner` is only a + /// best-effort stand-in synthesized from the draft and only `draft` is + /// canonical. Frontend uses this to disable "diff/reset vs deployed" and + /// skip its deployed-shape parsing of `inner`. Omitted when false. + #[serde(skip_serializing_if = "std::ops::Not::not")] + pub no_deployed: bool, + /// The user's saved draft payload (whatever shape the editor wrote). + /// Present when `get_draft=true` and a draft exists. + #[serde(skip_serializing_if = "Option::is_none")] + pub draft: Option, + /// Other users with a draft on the same path (excludes the authed user). + /// Empty list is omitted to keep the common-case response lean. + #[serde(skip_serializing_if = "Vec::is_empty")] + pub other_drafts_users: Vec, +} + +/// List every other user (and the legacy NULL-email row, if any) with a +/// draft at `(workspace, kind, path)`. Returns usernames only — emails never +/// leave the server. LEFT JOIN against `usr` so an orphaned draft (user +/// removed from the workspace) still surfaces with `username = None`. The +/// authed user is excluded via `email <> authed_email`; the legacy row +/// matches because `email IS NULL` fails that comparison. +async fn fetch_other_drafts_users( + db: &DB, + w_id: &str, + authed_email: &str, + kind: UserDraftItemKind, + path: &str, +) -> Result> { + // The `admins` workspace has no `usr` rows (username IS the email there), + // so fall back to `d.email` when the join misses, else a real teammate's + // draft renders as a phantom "Legacy draft". The genuine NULL-email legacy + // row keeps `username = None` (its `d.email` is NULL, so the CASE yields NULL). + let rows = sqlx::query_as!( + OtherDraftUser, + r#"SELECT COALESCE(u.username, CASE WHEN d.workspace_id = 'admins' THEN d.email END) as "username?", + d.created_at as "draft_saved_at!" + FROM draft d + LEFT JOIN usr u + ON u.workspace_id = d.workspace_id + AND u.email = d.email + WHERE d.workspace_id = $1 + AND d.path = $2 + AND d.typ = $3 + AND (d.email IS NULL OR d.email <> $4) + ORDER BY d.email NULLS LAST"#, + w_id, + path, + kind as UserDraftItemKind, + authed_email, + ) + .fetch_all(db) + .await?; + Ok(rows) +} + +/// If `get_draft` is true AND the authed user has a draft for +/// `(workspace, kind, path)`, attach it as `draft`. `deployed` is always +/// serialized into `inner` untouched. +pub async fn maybe_overlay_draft( + db: &DB, + w_id: &str, + email: &str, + kind: UserDraftItemKind, + path: &str, + get_draft: bool, + deployed: T, +) -> Result +where + T: serde::Serialize + Send + 'static, +{ + // Non-editor callers (worker/CLI reads of possibly MB-scale flows/apps) + // pass `get_draft = false` and render no overlay, so skip the `usr` join. + if !get_draft { + return Ok(WithDraftOverlay { + inner: Box::new(deployed), + is_draft: false, + draft_saved_at: None, + no_deployed: false, + draft: None, + other_drafts_users: Vec::new(), + }); + } + + // Independent of the authed user's OWN draft: reset-to-deployed reloads + // still need to know who else is editing this path. Only the cross-user + // kinds surface it (see `shares_drafts_across_users`). + let other_drafts_users = if kind.shares_drafts_across_users() { + fetch_other_drafts_users(db, w_id, email, kind, path).await? + } else { + Vec::new() + }; + + // Prefer the user's OWN per-user draft, falling back to the legacy + // NULL-email workspace draft. `NULLS LAST` + `LIMIT 1` drops the legacy + // row when an owned one exists. + let row = sqlx::query!( + r#"SELECT value as "value!: sqlx::types::Json>", + created_at + FROM draft + WHERE workspace_id = $1 + AND (email = $2 OR email IS NULL) + AND path = $3 + AND typ = $4 + ORDER BY email NULLS LAST + LIMIT 1"#, + w_id, + email, + path, + kind as UserDraftItemKind, + ) + .fetch_optional(db) + .await?; + + let Some(row) = row else { + return Ok(WithDraftOverlay { + inner: Box::new(deployed), + is_draft: false, + draft_saved_at: None, + no_deployed: false, + draft: None, + other_drafts_users, + }); + }; + + let draft_json: serde_json::Value = serde_json::from_str(row.value.0.get())?; + + Ok(WithDraftOverlay { + inner: Box::new(deployed), + is_draft: true, + draft_saved_at: Some(row.created_at), + no_deployed: false, + draft: Some(draft_json), + other_drafts_users, + }) +} + +/// One row of a "draft-only" list synthesis: a draft at `path` with no +/// deployed counterpart. `value` is the editor's saved JSON (each handler +/// maps it into its own `Listable*` shape). +#[derive(sqlx::FromRow)] +pub struct DraftOnlyListRow { + pub path: String, + pub value: sqlx::types::Json>, + pub created_at: DateTime, +} + +/// Fetch the authed user's draft rows at paths with NO deployed counterpart, +/// for synthesizing draft-only entries into a list response. Absence is +/// checked against `kind.deployed_table()` (the shared single source). +/// Returns empty for kinds with no path-keyed table. Callers keep their own +/// gating (`include_draft_only`, page 0, no filters) and row mapping. +pub async fn fetch_draft_only_list_rows( + db: &DB, + w_id: &str, + email: &str, + kind: UserDraftItemKind, +) -> Result> { + let Some(table) = kind.deployed_table() else { + return Ok(Vec::new()); + }; + // `table` is from the closed `deployed_table()` enum, never user input. + // `(email = $3 OR email IS NULL)` surfaces the user's own draft-only rows + // AND the legacy NULL-email rows; `DISTINCT ON (path)` with `email IS NULL` + // last collapses a path that has both to the owned row. + let sql = format!( + "SELECT DISTINCT ON (path) path, value, created_at FROM draft \ + WHERE workspace_id = $1 AND typ = $2::text::DRAFT_KIND \ + AND (email = $3 OR email IS NULL) \ + AND NOT EXISTS (SELECT 1 FROM {table} t \ + WHERE t.workspace_id = draft.workspace_id AND t.path = draft.path) \ + ORDER BY path, (email IS NULL)" + ); + let rows = sqlx::query_as::<_, DraftOnlyListRow>(&sql) + .bind(w_id) + .bind(kind.as_str()) + .bind(email) + .fetch_all(db) + .await?; + Ok(rows) +} + +/// The get-by-path draft choreography, shared by every entity's "get by path" +/// route. Given the deployed entity as an `Option` (caller maps its own "not +/// found" to `None`): +/// - `Some(deployed)` → overlay the authed user's draft (if `get_draft`). +/// - `None` + `get_draft` → draft-only response (`no_deployed = true`) when +/// a draft exists, else the caller's 404 via `not_found`. +/// - `None` without `get_draft` → the caller's 404. +pub async fn overlay_or_draft_only( + db: &DB, + w_id: &str, + email: &str, + kind: UserDraftItemKind, + path: &str, + get_draft: bool, + deployed: Option, + not_found: impl FnOnce() -> crate::error::Error, +) -> Result { + match deployed { + Some(deployed) => { + maybe_overlay_draft(db, w_id, email, kind, path, get_draft, deployed).await + } + None if get_draft => fetch_draft_only(db, w_id, email, kind, path) + .await? + .ok_or_else(not_found), + None => Err(not_found()), + } +} + +/// Delete EVERY user's draft (and the legacy NULL-email row) at a path+kind. +/// Use when the item is DELETED outright: it's gone for everyone, so leaving +/// teammates' drafts behind would orphan them forever. Discarding one's OWN +/// draft while the item lives on goes through `update_draft` with `value: null`. +/// Idempotent on the no-draft case. +pub async fn delete_all_drafts_for_path( + db: &DB, + w_id: &str, + kind: UserDraftItemKind, + path: &str, +) -> Result<()> { + sqlx::query!( + r#"DELETE FROM draft + WHERE workspace_id = $1 + AND path = $2 + AND typ = $3"#, + w_id, + path, + kind as UserDraftItemKind, + ) + .execute(db) + .await?; + Ok(()) +} + +/// Discard the deploying user's OWN draft (plus the legacy NULL-email row) +/// for a path+kind, leaving teammates' drafts intact. Use on RENAME: the +/// item moved, so the draft at the old path is orphaned (no FK to cascade). +/// Teammates keep theirs and get the StaleDraftModal on their next reload. +/// Idempotent on the no-draft case. +pub async fn delete_own_draft_for_path( + db: &DB, + w_id: &str, + kind: UserDraftItemKind, + path: &str, + email: &str, +) -> Result<()> { + sqlx::query!( + r#"DELETE FROM draft + WHERE workspace_id = $1 + AND path = $2 + AND typ = $3 + AND (email = $4 OR email IS NULL)"#, + w_id, + path, + kind as UserDraftItemKind, + email, + ) + .execute(db) + .await?; + Ok(()) +} + +/// Fetch the authed user's draft as a standalone payload, for "get by path" +/// routes when no deployed row exists but a draft might. Returns it as a +/// `WithDraftOverlay` with `inner` and `draft` both set to the same JSON and +/// `no_deployed = true`. Callers must have established no deployed row exists; +/// `Ok(None)` when there's also no draft (caller should 404). +/// +/// The draft JSON is expected to be an object (so `serde(flatten)` on `inner` +/// works); a non-object draft renders with no fields flattened. +pub async fn fetch_draft_only( + db: &DB, + w_id: &str, + email: &str, + kind: UserDraftItemKind, + path: &str, +) -> Result> { + // Own draft first, legacy NULL-email row as fallback (see `maybe_overlay_draft`). + let row = sqlx::query!( + r#"SELECT value as "value!: sqlx::types::Json>", + created_at + FROM draft + WHERE workspace_id = $1 + AND (email = $2 OR email IS NULL) + AND path = $3 + AND typ = $4 + ORDER BY email NULLS LAST + LIMIT 1"#, + w_id, + email, + path, + kind as UserDraftItemKind, + ) + .fetch_optional(db) + .await?; + + let Some(row) = row else { + return Ok(None); + }; + + let draft_json: serde_json::Value = serde_json::from_str(row.value.0.get())?; + let other_drafts_users = if kind.shares_drafts_across_users() { + fetch_other_drafts_users(db, w_id, email, kind, path).await? + } else { + Vec::new() + }; + Ok(Some(WithDraftOverlay { + // Best-effort stand-in for the missing deployed — same JSON as `draft`. + inner: Box::new(draft_json.clone()), + is_draft: true, + draft_saved_at: Some(row.created_at), + no_deployed: true, + draft: Some(draft_json), + other_drafts_users, + })) +} + +/// Marker prefix for draft secret values encrypted at rest with the workspace +/// crypt key (`build_crypt`). Written by `update_draft` for secret variables; +/// resolved back to plaintext by the variable deploy endpoints. +pub const ENCRYPTED_DRAFT_PREFIX: &str = "$encrypted:"; + +fn draft_decrypt_error() -> crate::error::Error { + crate::error::Error::BadRequest( + "An encrypted draft secret could not be decrypted (the workspace encryption key may \ + have changed since the draft was saved). Reset the field and re-enter the secret." + .to_string(), + ) +} + +/// Decrypt a `$encrypted:`-marked draft value back to plaintext with the +/// workspace crypt key. Fails with a user-facing 400 when it doesn't decrypt +/// (e.g. the workspace key was rotated after the draft save). +pub async fn decrypt_draft_secret_value(db: &DB, w_id: &str, value: &str) -> Result { + let encrypted = value.strip_prefix(ENCRYPTED_DRAFT_PREFIX).unwrap_or(value); + let mc = crate::variables::build_crypt(db, w_id).await?; + crate::variables::decrypt(&mc, encrypted.to_string()).map_err(|_| draft_decrypt_error()) +} diff --git a/backend/windmill-common/src/users.rs b/backend/windmill-common/src/users.rs index f91c21701a..d40b5cd19c 100644 --- a/backend/windmill-common/src/users.rs +++ b/backend/windmill-common/src/users.rs @@ -50,10 +50,10 @@ const EMAIL_CACHE_TTL_SECS: u64 = 60; /// - "u/{username}" → lookup email from usr table (cached) /// - "g/{group}" → "group-{group}@windmill.dev" /// - raw email → return as-is -pub async fn get_email_from_permissioned_as( +pub async fn get_email_from_permissioned_as<'c>( permissioned_as: &str, workspace_id: &str, - db: &sqlx::Pool, + db: impl sqlx::PgExecutor<'c>, ) -> crate::error::Result { if let Some(username) = permissioned_as.strip_prefix(PERMISSIONED_AS_USER_PREFIX) { let lookup = EmailCacheKey(workspace_id, username); diff --git a/backend/windmill-common/src/variables.rs b/backend/windmill-common/src/variables.rs index faabd27142..b43eb07e9c 100644 --- a/backend/windmill-common/src/variables.rs +++ b/backend/windmill-common/src/variables.rs @@ -9,6 +9,7 @@ use crate::db::{Authable, UserDB}; use crate::error::{self, Error}; use crate::scripts::ScriptHash; +use crate::secret_backend::{get_secret_value, is_external_stored_value}; use crate::utils::WarnAfterExt; use crate::worker::Connection; use crate::{worker::WORKER_GROUP, BASE_URL, DB}; @@ -59,6 +60,19 @@ pub struct ListableVariable { pub edited_at: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub edited_by: Option, + /// True when this row is a per-user draft with no deployed variable + /// at the same path. Surfaced by `include_draft_only` so the frontend + /// can render a "Draft" badge and the editor can open from the draft + /// alone. `None`/omitted on rows fetched from the `variable` table. + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub draft_only: Option, + /// True when the authed user has a per-user draft at this path — + /// layered over a deployed variable or a synthesized draft-only row. + /// Drives the `*` suffix on the variables page. + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_draft: Option, } #[derive(Serialize, Deserialize, sqlx::FromRow)] @@ -221,13 +235,17 @@ pub async fn get_secret_value_as_admin( let r = if variable.is_secret { let value = variable.value; if !value.is_empty() { - let mc = build_crypt(db, w_id).await?; - decrypt(&mc, value).map_err(|e| { - crate::error::Error::internal_err(format!( - "Error decrypting variable {}: {}", - variable.path, e - )) - })? + if is_external_stored_value(&value) { + get_secret_value(db, w_id, &variable.path, &value).await? + } else { + let mc = build_crypt(db, w_id).await?; + decrypt(&mc, value).map_err(|e| { + crate::error::Error::internal_err(format!( + "Error decrypting variable {}: {}", + variable.path, e + )) + })? + } } else { "".to_string() } @@ -533,10 +551,14 @@ pub async fn get_variable_or_self( if let Some(record) = record { let mut value = record.value; if record.is_secret { - let mc = build_crypt(db, w_id).await?; - value = decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) - })?; + if is_external_stored_value(&value) { + value = get_secret_value(db, w_id, &path, &value).await?; + } else { + let mc = build_crypt(db, w_id).await?; + value = decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) + })?; + } } Ok(value) @@ -578,10 +600,14 @@ pub async fn get_variable_or_self_as( if let Some(record) = record { let mut value = record.value; if record.is_secret { - let mc = build_crypt(db, w_id).await?; - value = decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e)) - })?; + if is_external_stored_value(&value) { + value = get_secret_value(db, w_id, &var_path, &value).await?; + } else { + let mc = build_crypt(db, w_id).await?; + value = decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", var_path, e)) + })?; + } } Ok(value) diff --git a/backend/windmill-common/src/worker.rs b/backend/windmill-common/src/worker.rs index e3bd7ef7ef..3a1469d5e3 100644 --- a/backend/windmill-common/src/worker.rs +++ b/backend/windmill-common/src/worker.rs @@ -693,8 +693,6 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error let full_path = job_dir.join(&user_path); - // let normalized_job_dir = std::fs::canonicalize(job_dir)?; - // let normalized_full_path = std::fs::canonicalize(&full_path)?; let normalized_job_dir = normalize_path(job_dir); let normalized_full_path = normalize_path(&full_path); @@ -706,6 +704,36 @@ pub fn is_allowed_file_location(job_dir: &str, user_defined_path: &str) -> error .into()); } + // The lexical check above cannot see symlinks: a symlink planted inside the + // job dir - e.g. by an earlier Ansible `git_repos` clone whose tracked + // content includes one - would let a later `git clone` or file write follow + // it out of the job dir while still passing the textual `starts_with` check. + // Walk the *normalized* relative path (`..`/`.` already collapsed) so each + // step matches the real on-disk resolution, and reject any existing component + // that is a symlink. Walking the raw user path would drift on an in-bounds + // `..` (e.g. `foo/../link`, which normalizes back inside the job dir) and miss + // the real symlinked component. Not-yet-existing components are safe: a path + // that does not exist cannot itself be a symlink. + let relative = normalized_full_path + .strip_prefix(&normalized_job_dir) + .unwrap_or(&normalized_full_path); + let mut current = normalized_job_dir.clone(); + for component in relative.components() { + if let Component::Normal(c) = component { + current.push(c); + if std::fs::symlink_metadata(¤t) + .map(|m| m.file_type().is_symlink()) + .unwrap_or(false) + { + return Err(std::io::Error::new( + std::io::ErrorKind::PermissionDenied, + "Path traverses a symlink, which is not allowed.", + ) + .into()); + } + } + } + Ok(normalized_full_path) } @@ -2828,4 +2856,71 @@ mod tests { let _ = std::fs::remove_dir_all(&base); } + + #[test] + fn test_is_allowed_file_location_allows_plain_relative() { + let base = std::env::temp_dir().join(format!("wm_allowed_loc_ok_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + let out = is_allowed_file_location(job_dir_str, "repo/sub/playbook.yml").unwrap(); + assert_eq!(out, normalize_path(&job_dir.join("repo/sub/playbook.yml"))); + + let _ = std::fs::remove_dir_all(&base); + } + + #[test] + fn test_is_allowed_file_location_rejects_parent_and_absolute() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_esc_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + assert!(is_allowed_file_location(job_dir_str, "../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "a/../../escape").is_err()); + assert!(is_allowed_file_location(job_dir_str, "/etc/passwd").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } + + // Regression for GHSA-v934-cvpf-6fjw: a symlink planted inside the job dir + // (e.g. by an earlier `git_repos` clone) must not let a later target traverse + // it out of the job dir, even though the lexical path stays "inside". + #[cfg(unix)] + #[test] + fn test_is_allowed_file_location_rejects_symlink_traversal() { + let base = + std::env::temp_dir().join(format!("wm_allowed_loc_symlink_{}", uuid::Uuid::new_v4())); + let job_dir = base.join("job"); + std::fs::create_dir_all(&job_dir).unwrap(); + // Stand-in for the shared cache dir living outside the job dir. + let outside = base.join("outside"); + std::fs::create_dir_all(&outside).unwrap(); + let job_dir_str = job_dir.to_str().unwrap(); + + // Plant `job/repo` -> `../outside`, as a malicious first clone would. + let planted = job_dir.join("repo"); + std::os::unix::fs::symlink(&outside, &planted).unwrap(); + + // Both the symlink itself and any path traversing it are rejected. + assert!(is_allowed_file_location(job_dir_str, "repo").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/payload").is_err()); + assert!(is_allowed_file_location(job_dir_str, "repo/sub/payload").is_err()); + + // An in-bounds `..` must not bypass the check: `foo/../repo/payload` + // normalizes back to `repo/payload` and still traverses the symlink. + assert!(is_allowed_file_location(job_dir_str, "foo/../repo/payload").is_err()); + std::fs::create_dir(job_dir.join("real")).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "real/../repo/payload").is_err()); + + // A dangling symlink (target does not exist yet) is still caught: + // `symlink_metadata` does not follow the link. + let dangling = job_dir.join("dangling"); + std::os::unix::fs::symlink(base.join("nonexistent"), &dangling).unwrap(); + assert!(is_allowed_file_location(job_dir_str, "dangling/payload").is_err()); + + let _ = std::fs::remove_dir_all(&base); + } } diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index 1870144ad7..7b040035c8 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -8,6 +8,7 @@ use strum::AsRefStr; use crate::{ error::{self, to_anyhow, Error, Result}, get_database_url, + secret_backend::{get_secret_value, is_external_stored_value}, utils::get_custom_pg_instance_password, variables::{build_crypt, decrypt}, PgDatabase, DB, @@ -165,7 +166,7 @@ pub enum ObjectType { WorkspaceDependencies, } -pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28261/sync-script-to-git-repo-windmill"; +pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28719/sync-script-to-git-repo-windmill"; /// Hub script that applies a repository's state back into a workspace /// (the repo → Windmill / "pull" direction). Same script the UI runs from @@ -822,10 +823,14 @@ async fn transform_json_unchecked( .await .map_err(to_anyhow)?; let value = if is_secret { - let mc = build_crypt(&db, &w_id).await?; - decrypt(&mc, value).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", &s, e)) - })? + if is_external_stored_value(&value) { + get_secret_value(db, w_id, &s[5..], &value).await? + } else { + let mc = build_crypt(&db, &w_id).await?; + decrypt(&mc, value).map_err(|e| { + Error::internal_err(format!("Error decrypting variable {}: {}", &s, e)) + })? + } } else { value }; diff --git a/backend/windmill-common/tests/asset_producer_notify.rs b/backend/windmill-common/tests/asset_producer_notify.rs new file mode 100644 index 0000000000..d09b0114eb --- /dev/null +++ b/backend/windmill-common/tests/asset_producer_notify.rs @@ -0,0 +1,474 @@ +/*! + * Tests that the asset producer-writes cache invalidation + * (`notify_asset_producer_change`) is emitted only when a deploy actually + * changes the set of script write-producers ('w'/'rw' asset usage), not on + * every deploy. The cache (asset_dispatch::ASSET_PRODUCER_WRITES_CACHE) only + * tracks script rows with write access, so read-only usage, flow usage, and + * deploys touching no write asset must NOT emit an event. + */ + +use sqlx::{Pool, Postgres}; +use windmill_common::assets::{ + clear_static_asset_usage, clear_static_asset_usage_by_script_hash, insert_static_asset_usage, + replace_static_asset_usage, AssetKind, AssetUsageAccessType, AssetUsageKind, + AssetWithAltAccessType, +}; +use windmill_common::scripts::ScriptHash; + +const WS: &str = "test-workspace"; + +/// Run the deploy-time clear+reinsert against `usage_path` in its own tx, +/// mirroring how create_script_internal calls it. +async fn replace(db: &Pool, usage_path: &str, assets: &[AssetWithAltAccessType]) { + let mut tx = db.begin().await.expect("begin"); + replace_static_asset_usage(&mut tx, WS, usage_path, assets) + .await + .expect("replace static asset usage"); + tx.commit().await.expect("commit"); +} + +fn asset_at( + path: &str, + kind: AssetKind, + access: Option, +) -> AssetWithAltAccessType { + AssetWithAltAccessType { + path: path.to_string(), + kind, + access_type: access, + alt_access_type: None, + columns: None, + } +} + +async fn producer_notify_count(db: &Pool) -> i64 { + sqlx::query_scalar::<_, i64>( + "SELECT COUNT(*) FROM notify_event WHERE channel = 'notify_asset_producer_change'", + ) + .fetch_one(db) + .await + .expect("count notify events") +} + +async fn reset_notify(db: &Pool) { + sqlx::query("DELETE FROM notify_event WHERE channel = 'notify_asset_producer_change'") + .execute(db) + .await + .expect("reset notify events"); +} + +fn asset(access: Option) -> AssetWithAltAccessType { + AssetWithAltAccessType { + path: "u/test-user/res".to_string(), + kind: AssetKind::Resource, + access_type: access, + alt_access_type: None, + columns: None, + } +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn insert_write_script_asset_emits(db: Pool) { + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::W)), + "u/test-user/script", + AssetUsageKind::Script, + ) + .await + .unwrap(); + assert_eq!(producer_notify_count(&db).await, 1, "write asset must emit"); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn insert_readonly_script_asset_does_not_emit(db: Pool) { + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::R)), + "u/test-user/script", + AssetUsageKind::Script, + ) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 0, + "read-only asset is not a write producer" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn insert_write_flow_asset_does_not_emit(db: Pool) { + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::W)), + "u/test-user/flow", + AssetUsageKind::Flow, + ) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 0, + "flow usage does not affect the script producer cache" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn clear_plain_script_does_not_emit(db: Pool) { + // No asset rows for this path: a plain script deploy must not emit. + clear_static_asset_usage(&db, WS, "u/test-user/plain", AssetUsageKind::Script) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 0, + "plain script deploy must not emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn clear_removes_write_producer_emits(db: Pool) { + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::RW)), + "u/test-user/script", + AssetUsageKind::Script, + ) + .await + .unwrap(); + reset_notify(&db).await; // ignore the insert-side event; isolate the clear + + clear_static_asset_usage(&db, WS, "u/test-user/script", AssetUsageKind::Script) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 1, + "removing a write producer must emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn clear_removes_only_readonly_does_not_emit(db: Pool) { + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::R)), + "u/test-user/script", + AssetUsageKind::Script, + ) + .await + .unwrap(); + reset_notify(&db).await; + + clear_static_asset_usage(&db, WS, "u/test-user/script", AssetUsageKind::Script) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 0, + "removing only read-only usage must not emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn clear_flow_usage_does_not_emit(db: Pool) { + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::W)), + "u/test-user/flow", + AssetUsageKind::Flow, + ) + .await + .unwrap(); + reset_notify(&db).await; + + clear_static_asset_usage(&db, WS, "u/test-user/flow", AssetUsageKind::Flow) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 0, + "clearing flow usage must not emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn clear_by_script_hash_gated_on_write(db: Pool) { + let hash = 123456789_i64; + sqlx::query( + r#"INSERT INTO script (workspace_id, hash, path, summary, description, content, + created_by, language, lock) + VALUES ($1, $2, 'u/test-user/byhash', '', '', '', 'test-user', 'deno', '')"#, + ) + .bind(WS) + .bind(hash) + .execute(&db) + .await + .unwrap(); + + // Read-only usage on that script path → clear must not emit. + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::R)), + "u/test-user/byhash", + AssetUsageKind::Script, + ) + .await + .unwrap(); + reset_notify(&db).await; + + clear_static_asset_usage_by_script_hash(&db, WS, ScriptHash(hash)) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 0, + "by-hash clear of read-only usage must not emit" + ); + + // Now a write usage → by-hash clear must emit. + insert_static_asset_usage( + &db, + WS, + &asset(Some(AssetUsageAccessType::W)), + "u/test-user/byhash", + AssetUsageKind::Script, + ) + .await + .unwrap(); + reset_notify(&db).await; + + clear_static_asset_usage_by_script_hash(&db, WS, ScriptHash(hash)) + .await + .unwrap(); + assert_eq!( + producer_notify_count(&db).await, + 1, + "by-hash clear of write usage must emit" + ); +} + +const SP: &str = "u/test-user/script"; + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_plain_deploy_does_not_emit(db: Pool) { + replace(&db, SP, &[]).await; + assert_eq!( + producer_notify_count(&db).await, + 0, + "deploy with no assets must not emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_gaining_write_emits_once(db: Pool) { + replace( + &db, + SP, + &[asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + )], + ) + .await; + assert_eq!( + producer_notify_count(&db).await, + 1, + "gaining a write producer must emit exactly once" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_unchanged_write_set_does_not_emit(db: Pool) { + let assets = [asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + )]; + replace(&db, SP, &assets).await; + reset_notify(&db).await; // isolate the redeploy + + // Redeploy with the identical write-producer set: clear removes the row and + // the reinsert adds it back, but the cache value is unchanged → no emit. + replace(&db, SP, &assets).await; + assert_eq!( + producer_notify_count(&db).await, + 0, + "redeploy keeping the same write producers must not emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_dropping_write_emits(db: Pool) { + let assets = [asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::RW), + )]; + replace(&db, SP, &assets).await; + reset_notify(&db).await; + + // Redeploy with no assets: the write producer disappears → emit. + replace(&db, SP, &[]).await; + assert_eq!( + producer_notify_count(&db).await, + 1, + "dropping the last write producer must emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_readonly_to_write_emits(db: Pool) { + replace( + &db, + SP, + &[asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::R), + )], + ) + .await; + reset_notify(&db).await; + + // Same asset path, access flips read-only → write: cache gains a row → emit. + replace( + &db, + SP, + &[asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + )], + ) + .await; + assert_eq!( + producer_notify_count(&db).await, + 1, + "read-only → write transition must emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_duplicate_entries_use_persisted_state(db: Pool) { + replace( + &db, + SP, + &[asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + )], + ) + .await; + reset_notify(&db).await; + + // Redeploy with conflicting duplicates for the same (kind, path), read-only + // first: ON CONFLICT DO NOTHING persists the read-only row and drops the + // write one, so the write producer is really gone → must emit (the diff must + // follow the persisted state, not the requested slice). + replace( + &db, + SP, + &[ + asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::R), + ), + asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + ), + ], + ) + .await; + assert_eq!( + producer_notify_count(&db).await, + 1, + "duplicate entries losing the persisted write producer must emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_duplicate_entries_keeping_write_does_not_emit(db: Pool) { + replace( + &db, + SP, + &[asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + )], + ) + .await; + reset_notify(&db).await; + + // Same duplicates but write first: the write row persists, so the write set + // is unchanged → no emit. + replace( + &db, + SP, + &[ + asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::W), + ), + asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::R), + ), + ], + ) + .await; + assert_eq!( + producer_notify_count(&db).await, + 0, + "duplicate entries keeping the persisted write producer must not emit" + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn replace_changing_only_readonly_does_not_emit(db: Pool) { + replace( + &db, + SP, + &[asset_at( + "u/test-user/res", + AssetKind::Resource, + Some(AssetUsageAccessType::R), + )], + ) + .await; + reset_notify(&db).await; + + // Swap one read-only producer for another: no write producer either side, so + // the write-set cache is untouched → no emit. + replace( + &db, + SP, + &[asset_at( + "u/test-user/other", + AssetKind::Resource, + Some(AssetUsageAccessType::R), + )], + ) + .await; + assert_eq!( + producer_notify_count(&db).await, + 0, + "changes confined to read-only producers must not emit" + ); +} diff --git a/backend/windmill-common/tests/asset_schema_capture.rs b/backend/windmill-common/tests/asset_schema_capture.rs new file mode 100644 index 0000000000..f12ff3cc14 --- /dev/null +++ b/backend/windmill-common/tests/asset_schema_capture.rs @@ -0,0 +1,105 @@ +/*! + * Tests the schema-capture versioning contract (gap #2a): + * `record_asset_schema` inserts a new `materialized_asset_schema` version only + * when the captured column set changes, re-affirms the latest row in place when + * it doesn't, and `list_asset_schemas` returns the evolution history newest + * first. + */ + +use sqlx::{Pool, Postgres}; +use windmill_common::assets::AssetKind; +use windmill_common::materialization::{list_asset_schemas, record_asset_schema, SchemaColumn}; + +const WS: &str = "test-workspace"; +const PATH: &str = "analytics/orders"; + +fn col(name: &str, ty: &str) -> SchemaColumn { + SchemaColumn { name: name.to_string(), data_type: ty.to_string() } +} + +async fn record(db: &Pool, cols: &[SchemaColumn], snapshot_id: i64) -> bool { + let mut tx = db.begin().await.expect("begin"); + let inserted = record_asset_schema( + &mut tx, + WS, + AssetKind::Ducklake, + PATH, + cols, + Some(snapshot_id), + None, + ) + .await + .expect("record asset schema"); + tx.commit().await.expect("commit"); + inserted +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn first_capture_inserts_version_one(db: Pool) { + let cols = [col("order_id", "BIGINT"), col("status", "VARCHAR")]; + assert!( + record(&db, &cols, 10).await, + "first capture inserts a version" + ); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 1); + assert_eq!(versions[0].version, 1); + assert_eq!(versions[0].snapshot_id, Some(10)); + assert_eq!(versions[0].columns.0, cols); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn unchanged_schema_reaffirms_without_new_version(db: Pool) { + let cols = [col("order_id", "BIGINT")]; + record(&db, &cols, 10).await; + // Identical column set on a later snapshot: no new version, but the latest + // row's snapshot_id advances. + assert!( + !record(&db, &cols, 20).await, + "unchanged schema must not insert a new version" + ); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 1, "still a single version"); + assert_eq!(versions[0].version, 1); + assert_eq!(versions[0].snapshot_id, Some(20), "snapshot re-affirmed"); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn changed_schema_bumps_version_newest_first(db: Pool) { + record(&db, &[col("order_id", "BIGINT")], 10).await; + // A column added → schema changed → new version. + let evolved = [col("order_id", "BIGINT"), col("amount", "DOUBLE")]; + assert!(record(&db, &evolved, 20).await, "changed schema inserts v2"); + + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 2); + // Newest first. + assert_eq!(versions[0].version, 2); + assert_eq!(versions[0].columns.0, evolved); + assert_eq!(versions[1].version, 1); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn column_order_change_is_a_new_version(db: Pool) { + let a = [col("a", "BIGINT"), col("b", "VARCHAR")]; + let reordered = [col("b", "VARCHAR"), col("a", "BIGINT")]; + record(&db, &a, 10).await; + // Same columns, different physical order: the captured list is ordered, so a + // reorder is a real schema change (downstream `SELECT *` consumers see it). + assert!( + record(&db, &reordered, 20).await, + "column reorder is a distinct schema version" + ); + let versions = list_asset_schemas(&db, WS, AssetKind::Ducklake, PATH) + .await + .unwrap(); + assert_eq!(versions.len(), 2); +} diff --git a/backend/windmill-duckdb-ffi-internal/src/lib.rs b/backend/windmill-duckdb-ffi-internal/src/lib.rs index 594b0f6df4..f3ee59cab7 100644 --- a/backend/windmill-duckdb-ffi-internal/src/lib.rs +++ b/backend/windmill-duckdb-ffi-internal/src/lib.rs @@ -266,6 +266,12 @@ fn setup_duckdb_connection( .unwrap_or(("http", &base_internal_url)); let s3_endpoint_ssl = s3_endpoint_ssl == "https"; + // Escape values interpolated into single-quoted SQL literals, consistent with + // configure_duckdb_resource_limits above (a stray quote would break the statement). + let s3_access_key = sql_single_quote(s3_access_key); + let s3_secret_key = sql_single_quote(s3_secret_key); + let endpoint = sql_single_quote(&format!("{s3_endpoint}/api/w/{w_id}/s3_proxy")); + conn.execute_batch(&format!( "INSTALL httpfs; LOAD httpfs; INSTALL azure; LOAD azure; @@ -274,7 +280,7 @@ fn setup_duckdb_connection( PROVIDER config, KEY_ID '{s3_access_key}', SECRET '{s3_secret_key}', - ENDPOINT '{s3_endpoint}/api/w/{w_id}/s3_proxy', + ENDPOINT '{endpoint}', URL_STYLE path, USE_SSL {s3_endpoint_ssl} ); @@ -282,7 +288,7 @@ fn setup_duckdb_connection( TYPE gcs, KEY_ID '{s3_access_key}', SECRET '{s3_secret_key}', - ENDPOINT '{s3_endpoint}/api/w/{w_id}/s3_proxy', + ENDPOINT '{endpoint}', USE_SSL {s3_endpoint_ssl} ); ", diff --git a/backend/windmill-mcp/src/common/schema.rs b/backend/windmill-mcp/src/common/schema.rs index de90248689..96dcd023fc 100644 --- a/backend/windmill-mcp/src/common/schema.rs +++ b/backend/windmill-mcp/src/common/schema.rs @@ -193,6 +193,14 @@ pub fn enrich_resource_schemas( } } +/// The seven type names permitted by the JSON Schema draft 2020-12 `type` keyword. +fn is_valid_json_schema_type(t: &str) -> bool { + matches!( + t, + "null" | "boolean" | "object" | "array" | "number" | "string" | "integer" + ) +} + /// Transform a JSON schema for maximum MCP client compatibility. /// /// Ensures schemas conform to JSON Schema draft 2020-12 by: @@ -200,9 +208,12 @@ pub fn enrich_resource_schemas( /// - Removing invalid non-array `enum` values /// - Stripping non-standard keywords (`originalType`, `format` with `resource-*` prefix) /// - Rewriting the Windmill pseudo-type `type: "resource"` to `type: "string"` -/// - Fixing contradictory schemas (`type: "string"` with `properties` → `type: "object"`) +/// - Fixing contradictory schemas (`type: "string"` with `properties` → `type: "object"`, +/// or with `items` → `type: "array"`) +/// - Dropping invalid `type` values (e.g. the empty string `""` Windmill emits for +/// untyped fields) so the node validates as "any type" /// - Removing `default: null` when the type doesn't include `null` -/// - Adding `type: "object"` to empty schemas that have no type +/// - Adding `type: "object"` to property-bearing schemas that have no type pub fn make_schema_compatible(schema: &mut Value) { let Value::Object(obj) = schema else { return }; @@ -239,6 +250,55 @@ pub fn make_schema_compatible(schema: &mut Value) { } } + // 3b. Fix contradictory type: if `items` is present (and the node isn't an + // object), type must be "array". Windmill serializes a `list[...]` field as + // `type: "string"` with an `items` subschema; "string" + items is nonsense + // and leaves the element schema unreachable to the client. + if obj.contains_key("items") && !obj.contains_key("properties") { + match obj.get("type").and_then(|v| v.as_str()) { + Some("array") => {} + _ => { + obj.insert("type".to_string(), Value::String("array".to_string())); + } + } + } + + // 3c. Drop invalid `type` values. Windmill emits `type: ""` for fields + // declared without an explicit type; the empty string (and any other name + // outside the draft 2020-12 type enum) makes a strict validator reject the + // whole schema (e.g. Anthropic's tool registration). Removing it leaves the + // node untyped, which accepts any value -- the meaning of an untyped field. + match obj.get("type") { + None => {} + Some(Value::String(s)) => { + if !is_valid_json_schema_type(s) { + obj.remove("type"); + } + } + Some(Value::Array(arr)) => { + let filtered: Vec = arr + .iter() + .filter(|v| v.as_str().is_some_and(is_valid_json_schema_type)) + .cloned() + .collect(); + match filtered.len() { + 0 => { + obj.remove("type"); + } + 1 => { + obj.insert("type".to_string(), filtered.into_iter().next().unwrap()); + } + _ => { + obj.insert("type".to_string(), Value::Array(filtered)); + } + } + } + // `type` as null/number/bool/object is not a valid keyword value at all. + Some(_) => { + obj.remove("type"); + } + } + // 4. Convert integer to number if let Some(type_val) = obj.get_mut("type") { match type_val { @@ -273,11 +333,6 @@ pub fn make_schema_compatible(schema: &mut Value) { obj.remove("enum"); } - // 7. Ensure schemas with no type but with properties get type: "object" - if !obj.contains_key("type") && !obj.is_empty() { - obj.insert("type".to_string(), Value::String("object".to_string())); - } - // Recursively process nested schemas if let Some(Value::Object(props)) = obj.get_mut("properties") { for value in props.values_mut() { @@ -782,4 +837,137 @@ mod tests { json!("string") ); } + + /// Recursively assert no `type` keyword anywhere in the schema carries an + /// empty string or a name outside the draft 2020-12 type enum -- the exact + /// shape Anthropic rejects with "input_schema: JSON Schema is invalid". + fn assert_all_types_valid(node: &Value) { + if let Some(obj) = node.as_object() { + match obj.get("type") { + Some(Value::String(s)) => { + assert!(is_valid_json_schema_type(s), "invalid type string: {s:?}") + } + Some(Value::Array(arr)) => { + for t in arr { + let s = t.as_str().expect("type array entries must be strings"); + assert!(is_valid_json_schema_type(s), "invalid type entry: {s:?}"); + } + } + _ => {} + } + for v in obj.values() { + assert_all_types_valid(v); + } + } else if let Some(arr) = node.as_array() { + for v in arr { + assert_all_types_valid(v); + } + } + } + + #[test] + fn drops_empty_type_string() { + let mut schema = json!({ + "type": "object", + "properties": { + "value": { "type": "", "description": "" } + } + }); + + make_schema_compatible(&mut schema); + + // Empty type is removed entirely (untyped == accepts any value); it is + // NOT re-typed to object, so a scalar value still validates. + assert!(schema["properties"]["value"].get("type").is_none()); + assert_all_types_valid(&schema); + } + + #[test] + fn infers_array_type_from_items() { + let mut schema = json!({ + "type": "object", + "properties": { + "services": { + "type": "string", + "description": "An object parameter.", + "items": { "type": "object" } + } + } + }); + + make_schema_compatible(&mut schema); + + assert_eq!(schema["properties"]["services"]["type"], json!("array")); + assert_all_types_valid(&schema); + } + + #[test] + fn items_does_not_override_object_with_properties() { + // A node carrying both `properties` and a stray `items` is an object, + // not an array -- the object signal wins. + let mut schema = json!({ + "type": "object", + "properties": { + "name": { "type": "string" } + }, + "items": { "type": "string" } + }); + + make_schema_compatible(&mut schema); + + assert_eq!(schema["type"], json!("object")); + } + + #[test] + fn filters_invalid_type_array_entries() { + let mut schema = json!({ "type": ["string", ""] }); + + make_schema_compatible(&mut schema); + + // Sole surviving entry collapses to a bare string. + assert_eq!(schema["type"], json!("string")); + } + + #[test] + fn drops_type_array_when_all_entries_invalid() { + let mut schema = json!({ "type": ["", "bogus"], "description": "x" }); + + make_schema_compatible(&mut schema); + + assert!(schema.get("type").is_none()); + } + + #[test] + fn customer_services_field_repro() { + // Repro of the reported failure: a `list[object]` script param that + // Windmill serialized as `type: "string"` + `items`, whose element + // object carried an untyped `value` field (`type: ""`). Anthropic + // rejected the whole tool list with + // "tools..custom.input_schema: JSON Schema is invalid". + let mut schema = json!({ + "type": "object", + "properties": { + "services": { + "items": { + "type": "object", + "properties": { + "serviceTypeId": { "description": "", "type": "string" }, + "value": { "description": "", "type": "" } + } + }, + "description": "An object parameter.", + "type": "string" + } + }, + "required": ["services"] + }); + + make_schema_compatible(&mut schema); + + assert_eq!(schema["properties"]["services"]["type"], json!("array")); + assert!(schema["properties"]["services"]["items"]["properties"]["value"] + .get("type") + .is_none()); + assert_all_types_valid(&schema); + } } diff --git a/backend/windmill-mcp/src/common/scope.rs b/backend/windmill-mcp/src/common/scope.rs index 7da4e2cebc..f43095d740 100644 --- a/backend/windmill-mcp/src/common/scope.rs +++ b/backend/windmill-mcp/src/common/scope.rs @@ -38,6 +38,71 @@ impl McpScopeConfig { is_resource_allowed(path, patterns) } + + /// Directional subset check: does this config grant at least everything + /// `requested` grants? Used to enforce monotonic containment when an MCP + /// OAuth approval mints a token (the granted scopes must be within the + /// approving token's own scopes). + /// + /// Unlike `is_allowed` (which tests a single concrete path with OR + /// semantics), this requires every requested pattern to be covered by some + /// caller pattern — so `mcp:scripts:f/x` cannot widen into `mcp:scripts:*`. + pub fn contains(&self, requested: &McpScopeConfig) -> bool { + if self.all { + return true; + } + if requested.all { + return false; + } + if requested.favorites && !self.favorites { + return false; + } + if let Some(req_hub) = requested.hub_apps.as_ref() { + match self.hub_apps.as_ref() { + Some(caller_hub) => { + let caller_apps: std::collections::HashSet<&str> = + caller_hub.split(',').map(|s| s.trim()).collect(); + if !req_hub + .split(',') + .map(|s| s.trim()) + .all(|a| caller_apps.contains(a)) + { + return false; + } + } + None => return false, + } + } + resource_list_covers(&self.scripts, &requested.scripts) + && resource_list_covers(&self.flows, &requested.flows) + && resource_list_covers(&self.endpoints, &requested.endpoints) + } +} + +/// Every requested pattern must be covered by some caller pattern. +fn resource_list_covers(caller: &[String], requested: &[String]) -> bool { + requested + .iter() + .all(|req| caller.iter().any(|c| pattern_covers(c, req))) +} + +/// Directional: does the single caller pattern cover `requested`? `caller` may +/// be `*`, an exact path/name, or a `/*` subtree; `requested` may itself +/// be a subtree wildcard, in which case the whole requested subtree must fall +/// within the caller's. Mirrors the route-scope containment in windmill-api-auth. +fn pattern_covers(caller: &str, requested: &str) -> bool { + if caller == "*" || caller == requested { + return true; + } + // An exact caller pattern only covers itself (handled above); a wildcard + // requested can never be covered by a non-`*` exact caller. + let Some(prefix) = caller.strip_suffix("/*") else { + return false; + }; + let requested_base = requested.strip_suffix("/*").unwrap_or(requested); + requested_base == prefix + || (requested_base.starts_with(prefix) + && requested_base.as_bytes().get(prefix.len()) == Some(&b'/')) } /// Parse MCP scopes from token scope strings @@ -254,4 +319,51 @@ mod tests { assert!(config.is_allowed("flow", "f/automation/test")); assert!(!config.is_allowed("flow", "f/other/test")); } + + fn cfg(scopes: &[&str]) -> McpScopeConfig { + parse_mcp_scopes(&scopes.iter().map(|s| s.to_string()).collect::>()).unwrap() + } + + #[test] + fn test_contains_subset_and_widening() { + // mcp:all contains anything. + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:all"]))); + + // A wildcard caller covers narrower requests, but not other domains/all. + let star = cfg(&["mcp:scripts:*"]); + assert!(star.contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(star.contains(&cfg(&["mcp:scripts:*"]))); + assert!(!star.contains(&cfg(&["mcp:all"]))); + assert!(!star.contains(&cfg(&["mcp:flows:f/x"]))); + + // The core regression: a single-path caller must NOT widen into `*` or + // into another path. + let narrow = cfg(&["mcp:scripts:f/x"]); + assert!(narrow.contains(&cfg(&["mcp:scripts:f/x"]))); + assert!(!narrow.contains(&cfg(&["mcp:scripts:*"]))); + assert!(!narrow.contains(&cfg(&["mcp:scripts:f/y"]))); + assert!(!narrow.contains(&cfg(&["mcp:all"]))); + + // Subtree wildcard covers paths within it but not a sibling subtree. + let subtree = cfg(&["mcp:scripts:f/team/*"]); + assert!(subtree.contains(&cfg(&["mcp:scripts:f/team/sub"]))); + assert!(subtree.contains(&cfg(&["mcp:scripts:f/team/sub/*"]))); + assert!(!subtree.contains(&cfg(&["mcp:scripts:f/other/x"]))); + } + + #[test] + fn test_contains_favorites_and_endpoints() { + assert!(cfg(&["mcp:favorites"]).contains(&cfg(&["mcp:favorites"]))); + // A caller without favorites cannot grant favorites. + assert!(!cfg(&["mcp:scripts:*"]).contains(&cfg(&["mcp:favorites"]))); + + // Endpoint names match exactly (or via `*`). + let ep = cfg(&["mcp:endpoints:getVariable"]); + assert!(ep.contains(&cfg(&["mcp:endpoints:getVariable"]))); + assert!(!ep.contains(&cfg(&["mcp:endpoints:getResource"]))); + assert!(!ep.contains(&cfg(&["mcp:all"]))); + // mcp:all grants all endpoints. + assert!(cfg(&["mcp:all"]).contains(&cfg(&["mcp:endpoints:getResource"]))); + } } diff --git a/backend/windmill-oauth/src/lib.rs b/backend/windmill-oauth/src/lib.rs index 252f7ace28..e314f13d75 100644 --- a/backend/windmill-oauth/src/lib.rs +++ b/backend/windmill-oauth/src/lib.rs @@ -67,6 +67,13 @@ pub struct ClientWithScopes { pub allowed_domains: Option>, pub userinfo_url: Option, pub grant_types: Vec, + /// Resolved token endpoint, exposed so the connect dialog can prefill and + /// persist it on client-credentials accounts. + pub token_url: String, + /// Whether the instance entry carries shared credentials (non-empty id + + /// secret). Providers without them are bring-your-own only — the connect + /// dialog lists them under "Others", not "Instance-configured". + pub has_shared_credentials: bool, } /// Map of OAuth client names to their configurations @@ -81,6 +88,13 @@ pub struct OAuthConfig { pub token_url: String, pub userinfo_url: Option, pub scopes: Option>, + /// Default scopes for the client-credentials (2-legged) flow. These differ + /// from the authorization-code `scopes` for most providers (member/consent + /// scopes are invalid in a 2-legged token request), so CC never defaults to + /// `scopes`. Absent means no default scope — the caller supplies any + /// provider-specific scopes themselves. + #[serde(skip_serializing_if = "Option::is_none")] + pub cc_scopes: Option>, pub extra_params: Option>, pub extra_params_callback: Option>, pub req_body_auth: Option, @@ -91,10 +105,12 @@ pub struct OAuthConfig { /// entry, `build_oauth_clients` registers a second client under that key. #[serde(skip_serializing_if = "Option::is_none")] pub sandbox: Option, - /// Frontend-only metadata for per-instance OAuth providers (Snowflake, - /// ServiceNow, …) whose authorize/token URLs are derived from an - /// admin-entered instance name. Ignored by the backend, which only ever - /// sees the resulting concrete `connect_config`. + /// Metadata for per-instance OAuth providers (Snowflake, ServiceNow, Coupa, + /// …) whose authorize/token URLs carry an `{instance}` placeholder filled + /// from an instance name. The instance-settings UI uses it to build the + /// per-client `connect_config` for the authorization-code flow; the + /// client-credentials flow reads its `token_url`/`strip_suffix`/`label` + /// directly to host-pin the exchange. #[serde(skip_serializing_if = "Option::is_none")] pub connect_config_template: Option, } @@ -111,11 +127,13 @@ pub struct OAuthSandboxOverride { pub userinfo_url: Option, } -/// Frontend metadata for a per-instance OAuth provider. The instance-settings -/// UI renders one generic instance-name input and substitutes `{instance}` into +/// Metadata for a per-instance OAuth provider. The instance-settings UI renders +/// one generic instance-name input and substitutes `{instance}` into /// `auth_url`/`token_url` to build the per-client `connect_config`. Adding a new /// per-instance provider needs only a registry entry carrying this template — -/// no frontend code change. The backend never reads it. +/// no frontend code change. The client-credentials flow additionally reads +/// `token_url`, `strip_suffix`, and `label` from it server-side (see +/// `resolve_cc_token_url_input`) to host-pin the token exchange. #[derive(Clone, Debug, Serialize, Deserialize)] pub struct ConnectConfigTemplate { /// Properly-cased provider name for the settings dropdown (e.g. "ServiceNow"); @@ -126,10 +144,17 @@ pub struct ConnectConfigTemplate { pub placeholder: String, #[serde(skip_serializing_if = "Option::is_none")] pub help_url: Option, - pub auth_url: String, + /// Authorize endpoint (with `{instance}`). Absent for client-credentials-only + /// providers (e.g. Coupa) that have no browser sign-in flow. + #[serde(default, skip_serializing_if = "Option::is_none")] + pub auth_url: Option, pub token_url: String, #[serde(skip_serializing_if = "Option::is_none")] pub req_body_auth: Option, + /// Scopes copied into the built `connect_config` (e.g. NetSuite's + /// `rest_webservices`). Templated providers default to no scopes. + #[serde(skip_serializing_if = "Option::is_none")] + pub scopes: Option>, /// Key under `connect_config.extra_params` where the instance name is /// stored (defaults to `instance`). Snowflake uses `account_identifier` for /// backward compatibility with previously-saved configs. @@ -237,8 +262,12 @@ fn empty_string() -> String { "".to_string() } +/// Placeholder authorize URL for providers that only support the +/// client-credentials grant (the authorize endpoint is never used by it). +pub const MISSING_AUTH_URL: &str = "https://missing-auth-url"; + fn empty_auth() -> String { - "https://missing-auth-url".to_string() + MISSING_AUTH_URL.to_string() } fn default_grant_types() -> Vec { @@ -348,77 +377,371 @@ pub async fn build_slack_client( Ok(client) } -/// Build OAuth client for client credentials flow with resource-level credentials +/// Build OAuth client for client credentials flow with resource-level credentials. +/// +/// No instance-level entry is required: the provider endpoint config resolves +/// from the instance `oauths` entry when one exists, else from the static +/// registry, else is synthesized from the token URL override alone. Returns the +/// built client together with the resolved [`OAuthConfig`] so callers can reuse +/// its scopes / `extra_params_callback`. pub async fn build_client_credentials_oauth_client( db: &DB, client_name: &str, client_id: &str, client_secret: &str, - cc_token_url_override: Option<&str>, + resolved_token_url: Option<&str>, connect_configs_json: &str, -) -> error::Result<(OClient, OAuthClient)> { +) -> error::Result<(OClient, OAuthConfig)> { use windmill_common::global_settings::{load_value_from_global_settings, OAUTH_SETTING}; let oauths = load_value_from_global_settings(db, OAUTH_SETTING).await?; - let oauths = oauths.unwrap_or_default(); - let oauth_config = oauths - .get(client_name) - .ok_or_else(|| error::Error::BadRequest("OAuth configuration not found".to_string()))?; + let instance_entry: Option = oauths + .as_ref() + .and_then(|o| o.get(client_name)) + .and_then(|v| match serde_json::from_value(v.clone()) { + Ok(entry) => Some(entry), + Err(e) => { + tracing::warn!( + client = %client_name, + "Invalid instance OAuth entry, falling back to static registry: {e}" + ); + None + } + }); - let oauth_client_config: OAuthClient = serde_json::from_value(oauth_config.clone()) - .map_err(|e| error::Error::BadRequest(format!("Invalid OAuth config: {}", e)))?; - - let parse_static_configs = || { - serde_json::from_str::>(connect_configs_json).map_err(|e| { - error::Error::InternalErr(format!("Failed to parse oauth_connect.json: {}", e)) - }) + let resolve_from_registry = |client_name: &str| -> error::Result> { + let static_configs = + serde_json::from_str::>(connect_configs_json).map_err( + |e| error::Error::InternalErr(format!("Failed to parse oauth_connect.json: {}", e)), + )?; + Ok(resolve_registry_config(&static_configs, client_name)) }; - let resolve_from_registry = |client_name: &str| -> error::Result { - let static_configs = parse_static_configs()?; - resolve_registry_config(&static_configs, client_name).ok_or_else(|| { + + // A token URL alone is enough for client credentials: providers that only + // support this grant have no authorize endpoint to configure. + let instance_connect_config = instance_entry + .as_ref() + .and_then(|e| e.connect_config.clone()) + .filter(|c| !c.token_url.is_empty()) + .map(|mut c| { + if c.auth_url.is_empty() { + c.auth_url = empty_auth(); + } + c + }); + + let from_instance = instance_connect_config.is_some(); + let mut connect_config = match instance_connect_config { + Some(config) => config, + None => resolve_from_registry(client_name)?.ok_or_else(|| { error::Error::BadRequest(format!( - "OAuth configuration not found for '{}' in either global settings or static config", + "No token URL available for '{}': not found in instance OAuth settings or static \ + config", client_name )) - }) + })?, }; - let mut connect_config = if let Some(ref config) = oauth_client_config.connect_config { - if !config.auth_url.is_empty() && !config.token_url.is_empty() { - config.clone() - } else { - resolve_from_registry(client_name)? - } - } else { - resolve_from_registry(client_name)? - }; - - if let Some(override_url) = cc_token_url_override { - connect_config.token_url = override_url.to_string(); + // Registry providers default their client-credentials scopes from `cc_scopes`, + // never the authorization-code `scopes` (which several providers reject for a + // 2-legged token request). Instance-configured entries keep their admin-set + // scopes untouched. + if !from_instance { + connect_config.scopes = connect_config.cc_scopes.clone(); } + let caller_supplied_creds = !client_id.is_empty() && !client_secret.is_empty(); + + // Apply the resolved concrete token URL. Instance-templated providers (e.g. + // Coupa) carry an empty or `{instance}`-templated token URL in their registry + // config; the resolved value (host-pinned for instance-name connections, + // persisted on the row for refresh) is what completes it. For bring-your-own + // connections this value may instead be a caller-supplied override — safe + // because only the caller's own credentials are ever sent to it. + if let Some(url) = resolved_token_url { + connect_config.token_url = url.to_string(); + } + if connect_config.token_url.is_empty() { + return Err(error::Error::BadRequest(format!( + "No token URL configured for '{}'", + client_name + ))); + } + + // Fall back to the instance entry's own credentials when the caller supplies + // none: the shared instance-level client-credentials setup, where an admin + // configures one service-account client for everyone and the secret never + // leaves the server. Only entries that explicitly enable the + // client_credentials grant qualify, so an authorization-code-only client's + // secret is never reused for this flow. + let instance_cc_creds = instance_entry.as_ref().filter(|e| { + e.grant_types.iter().any(|g| g == "client_credentials") + && !e.id.is_empty() + && !e.secret.is_empty() + }); + // All-or-nothing: use the caller's credentials only when both id and secret + // are present, otherwise fall back entirely to the instance entry. Never mix + // a caller-supplied id with the admin secret (or vice versa). + let (resolved_client_id, resolved_client_secret) = if caller_supplied_creds { + (client_id.to_string(), client_secret.to_string()) + } else { + instance_cc_creds + .map(|e| (e.id.clone(), e.secret.clone())) + .unwrap_or_default() + }; + let resource_oauth_client = OAuthClient { - id: client_id.to_string(), - secret: client_secret.to_string(), - allowed_domains: oauth_client_config.allowed_domains.clone(), + id: resolved_client_id, + secret: resolved_client_secret, + allowed_domains: instance_entry + .as_ref() + .and_then(|e| e.allowed_domains.clone()), connect_config: Some(connect_config.clone()), - login_config: oauth_client_config.login_config.clone(), - display_name: oauth_client_config.display_name.clone(), - grant_types: oauth_client_config.grant_types.clone(), - tenant: oauth_client_config.tenant.clone(), + login_config: instance_entry.as_ref().and_then(|e| e.login_config.clone()), + display_name: instance_entry.as_ref().and_then(|e| e.display_name.clone()), + grant_types: instance_entry + .as_ref() + .map(|e| e.grant_types.clone()) + .unwrap_or_else(default_grant_types), + tenant: instance_entry.as_ref().and_then(|e| e.tenant.clone()), }; let base_url = (**BASE_URL.load()).clone(); let (_, client) = build_basic_client( client_name.to_string(), - connect_config, + connect_config.clone(), resource_oauth_client, false, &base_url, None, )?; - Ok((client, oauth_client_config)) + Ok((client, connect_config)) +} + +/// Shared instance-level client-credentials for `client_name`: the `(id, secret, +/// token_url)` from its instance `oauths` entry, but only when that entry both +/// declares the `client_credentials` grant and carries non-empty credentials. +/// Lets the connect flow use one admin-configured service-account client instead +/// of asking each user for their own. +/// +/// # Authorization +/// Returns the admin's shared service-account secret, so callers MUST first +/// verify the caller's authorization to use it (workspace membership plus +/// read-write access — operators and read-only tokens are excluded). This helper +/// performs no authorization itself. +pub async fn resolve_instance_cc_credentials( + db: &DB, + client_name: &str, +) -> error::Result)>> { + use windmill_common::global_settings::{load_value_from_global_settings, OAUTH_SETTING}; + + let oauths = load_value_from_global_settings(db, OAUTH_SETTING).await?; + let entry: Option = oauths + .as_ref() + .and_then(|o| o.get(client_name)) + .and_then(|v| serde_json::from_value(v.clone()).ok()); + + Ok(entry.and_then(|e| { + let cc_grant = e.grant_types.iter().any(|g| g == "client_credentials"); + if cc_grant && !e.id.is_empty() && !e.secret.is_empty() { + // Token URL from the entry's connect_config (built by instance settings + // from the connect_config_template), so the account row is + // self-contained for refresh. + let token_url = e + .connect_config + .as_ref() + .map(|c| c.token_url.clone()) + .filter(|u| !u.is_empty()); + Some((e.id, e.secret, token_url)) + } else { + None + } + })) +} + +/// Resolve the concrete client-credentials token URL for a bring-your-own +/// connection. The caller never supplies a token URL: it always comes from the +/// built-in registry, so the exchange host can never be redirected. +/// +/// Supported only for registry providers. For one whose CC token URL carries an +/// `{instance}` placeholder (Coupa, ServiceNow, …) — declared in its +/// `connect_config_template` — the caller supplies only an instance name, +/// validated as a bare hostname label and substituted into the fixed-host +/// template. A fixed-host registry provider uses its registry token URL directly. +/// A custom resource type (no registry entry) is rejected: there is no known host +/// to send credentials to. +pub fn resolve_cc_token_url_input( + connect_configs_json: &str, + client_name: &str, + caller_instance: Option<&str>, +) -> error::Result { + let Some(cfg) = serde_json::from_str::>(connect_configs_json) + .ok() + .and_then(|m| resolve_registry_config(&m, client_name)) + else { + return Err(error::Error::BadRequest(format!( + "Client credentials with your own credentials are only supported for built-in OAuth \ + providers, not '{client_name}'. Configure shared credentials on the instance OAuth \ + entry instead." + ))); + }; + + // Instance-templated providers carry the `{instance}` token URL (and its + // label/strip_suffix) in `connect_config_template`; fixed-host providers use + // the plain `token_url`. + let tmpl = cfg.connect_config_template.as_ref(); + let template = tmpl + .map(|t| t.token_url.clone()) + .filter(|u| !u.is_empty()) + .or_else(|| Some(cfg.token_url.clone()).filter(|u| !u.is_empty())) + .ok_or_else(|| { + error::Error::BadRequest(format!("No token URL is configured for '{client_name}'")) + })?; + + if !template.contains("{instance}") { + // Fixed-host registry provider: its registry token URL is authoritative. + return Ok(template); + } + + // Structural host-pinning guard: only substitute when `{instance}` is the + // leftmost host label of a fixed-host template (`scheme://{instance}.fixed-host/…`). + // The hostname-label validation below keeps the value clean, but only this + // check guarantees the substituted value can never change the registrable + // domain — so a malformed template (e.g. `https://{instance}/token`) can't turn + // the caller's instance name into a full attacker-controlled host (SSRF / + // credential exfiltration). The template is a code-reviewed registry file, so a + // violation is a programming error. + let placeholder = "{instance}"; + let idx = template.find(placeholder).unwrap(); + let after = &template[idx + placeholder.len()..]; + if !template[..idx].ends_with("://") || !after.starts_with('.') { + return Err(error::Error::InternalErr(format!( + "Invalid instance-templated token URL for '{client_name}': {{instance}} must be the \ + leftmost host label (scheme://{{instance}}.fixed-host/…)" + ))); + } + + let raw = caller_instance + .map(str::trim) + .filter(|s| !s.is_empty()) + .ok_or_else(|| { + error::Error::BadRequest(format!( + "{} is required for {client_name}", + tmpl.map(|t| t.label.as_str()).unwrap_or("An instance name") + )) + })?; + // Strip an optional known host suffix so the user can paste a full host or a + // bare name, then accept only a hostname label — never any character that + // could move the host out of the template's domain. + let value = tmpl + .and_then(|t| t.strip_suffix.as_deref()) + .and_then(|sfx| raw.strip_suffix(sfx)) + .unwrap_or(raw) + .trim_end_matches('.'); + let valid = !value.is_empty() + && !value.starts_with(['-', '.']) + && value + .bytes() + .all(|b| b.is_ascii_alphanumeric() || b == b'-' || b == b'.'); + if !valid { + return Err(error::Error::BadRequest(format!( + "invalid instance name '{raw}' for {client_name}" + ))); + } + Ok(template.replace("{instance}", value)) +} + +/// Whether a built-in provider's client-credentials token URL is host-pinned via +/// an `{instance}` template (e.g. servicenow, snowflake, coupa). Such providers +/// only accept an instance name substituted into a fixed-host template, so a +/// free-form caller token URL override must be rejected for them — otherwise the +/// exchange host could be redirected, which is exactly what the template pins. +/// Fixed-host registry providers and custom (non-registry) providers return +/// `false`: an override is allowed there. +pub fn is_instance_templated_cc(connect_configs_json: &str, client_name: &str) -> bool { + serde_json::from_str::>(connect_configs_json) + .ok() + .and_then(|m| resolve_registry_config(&m, client_name)) + .map(|cfg| { + cfg.connect_config_template + .as_ref() + .map(|t| t.token_url.clone()) + .filter(|u| !u.is_empty()) + .unwrap_or(cfg.token_url) + .contains("{instance}") + }) + .unwrap_or(false) +} + +/// Resolve the concrete bring-your-own client-credentials token URL for any +/// provider, never from a caller-supplied URL: +/// - **Built-in registry providers** resolve from the registry via +/// [`resolve_cc_token_url_input`] (host-pinned from the caller's instance name +/// for instance-templated ones). +/// - **Custom providers configured at the instance level** use the admin's +/// `connect_config.token_url`. The caller has no instance template to fill, so +/// an instance name is rejected. +/// +/// This is the single entry point the connect/account-creation handlers should +/// use so both resolve identically. +pub async fn resolve_cc_token_url( + db: &DB, + client_name: &str, + caller_instance: Option<&str>, + connect_configs_json: &str, +) -> error::Result { + use windmill_common::global_settings::{load_value_from_global_settings, OAUTH_SETTING}; + + let supports_cc = + |grant_types: &[String]| grant_types.iter().any(|g| g == "client_credentials"); + + let registry_cfg = serde_json::from_str::>(connect_configs_json) + .ok() + .and_then(|m| resolve_registry_config(&m, client_name)); + if let Some(cfg) = registry_cfg { + // Built-in provider: only honor it for client credentials if it actually + // declares that grant, so an authorization-code-only provider can't be + // driven through the CC API. + if !supports_cc(&cfg.grant_types) { + return Err(error::Error::BadRequest(format!( + "'{client_name}' is not enabled for the client_credentials grant" + ))); + } + return resolve_cc_token_url_input(connect_configs_json, client_name, caller_instance); + } + + // Custom (non-registry) provider: the token URL comes from the admin's + // instance connect_config (an admin-configured, trusted host), never the + // caller. The instance entry must also enable the client-credentials grant. + let entry: Option = load_value_from_global_settings(db, OAUTH_SETTING) + .await? + .as_ref() + .and_then(|o| o.get(client_name)) + .and_then(|v| serde_json::from_value(v.clone()).ok()); + let instance_token_url = entry + .as_ref() + .filter(|e| supports_cc(&e.grant_types)) + .and_then(|e| e.connect_config.clone()) + .map(|c| c.token_url) + .filter(|u| !u.is_empty()); + match instance_token_url { + Some(_) + if caller_instance + .map(|s| !s.trim().is_empty()) + .unwrap_or(false) => + { + Err(error::Error::BadRequest(format!( + "An instance name only applies to built-in instance-templated providers, not \ + '{client_name}'" + ))) + } + Some(url) => Ok(url), + None => Err(error::Error::BadRequest(format!( + "Client credentials with your own credentials require '{client_name}' to be a built-in \ + OAuth provider or an instance entry that enables the client_credentials grant" + ))), + } } /// Exchange authorization code for tokens @@ -462,7 +785,7 @@ pub async fn exchange_token( client: OClient, refresh_token: &str, grant_type: &str, - oauth_client_info: Option<&ClientWithScopes>, + extra_params_callback: Option<&HashMap>, http_client: &reqwest::Client, scopes: Option<&[String]>, ) -> Result { @@ -483,11 +806,9 @@ pub async fn exchange_token( "client_credentials" => { let mut token_request = client.exchange_client_credentials(); - if let Some(oauth_info) = oauth_client_info { - if let Some(extra_params) = oauth_info.extra_params_callback.as_ref() { - for (key, value) in extra_params.iter() { - token_request = token_request.param(key.clone(), value.clone()); - } + if let Some(extra_params) = extra_params_callback { + for (key, value) in extra_params.iter() { + token_request = token_request.param(key.clone(), value.clone()); } } @@ -579,49 +900,78 @@ pub async fn refresh_token_for_account<'c>( http_client: &reqwest::Client, connect_configs_json: &str, ) -> error::Result { - let oauth_client_info = oauth_clients - .connects - .get(&account.client) - .ok_or_else(|| error::Error::BadRequest("invalid client".to_string()))? - .clone(); + // Instance-configured client: required for authorization_code (the refresh + // token exchange uses the instance app's credentials). For client_credentials + // it is resolved inside `build_client_credentials_oauth_client` instead. + let oauth_client_info = oauth_clients.connects.get(&account.client).cloned(); - let mut client = if account.grant_type == "client_credentials" { - match (&account.cc_client_id, &account.cc_client_secret) { - (Some(client_id), Some(client_secret)) => { - let (client, _) = build_client_credentials_oauth_client( - db, - &account.client, - client_id, - client_secret, - account.cc_token_url.as_deref(), - connect_configs_json, - ) - .await?; - client - } - _ => { - return Err(error::Error::BadRequest( - "client_credentials flow requires cc_client_id and cc_client_secret to be stored in account".to_string() - )); - } - } + let is_client_credentials = account.grant_type == "client_credentials"; + + let (mut client, cc_config) = if is_client_credentials { + // Bring-your-own accounts store their own credentials (and resolved token + // URL) on the row. Shared instance accounts store none: passing empty + // credentials makes the builder re-resolve the admin's service-account + // credentials and token URL from the instance entry on every refresh, so a + // rotated or removed shared secret takes effect immediately (mirrors the + // authorization-code model, where the row never holds the app secret). + let (client_id, client_secret) = match (&account.cc_client_id, &account.cc_client_secret) { + (Some(id), Some(secret)) => (id.as_str(), secret.as_str()), + _ => ("", ""), + }; + let (client, config) = build_client_credentials_oauth_client( + db, + &account.client, + client_id, + client_secret, + account.cc_token_url.as_deref(), + connect_configs_json, + ) + .await?; + (client, Some(config)) } else { - oauth_client_info.client.to_owned() + let info = oauth_client_info + .as_ref() + .ok_or_else(|| error::Error::BadRequest("invalid client".to_string()))?; + (info.client.to_owned(), None) }; - // Account-level scopes override instance-level scopes + // Account-level scopes (when stored) override these defaults. Client-credentials + // accounts default to the resolved CC config's scopes (`cc_scopes` for registry + // providers, the admin's instance scopes for custom ones) — never the instance + // client's authorization-code scopes, which are invalid in a 2-legged request. + // Authorization-code accounts default to the instance client's scopes. + let fallback_scopes = if is_client_credentials { + cc_config + .as_ref() + .and_then(|c| c.scopes.clone()) + .unwrap_or_default() + } else { + oauth_client_info + .as_ref() + .map(|i| i.scopes.clone()) + .unwrap_or_default() + }; let effective_scopes = account .scopes .as_deref() .filter(|s| !s.is_empty()) - .unwrap_or(&oauth_client_info.scopes); + .unwrap_or(&fallback_scopes); - if account.grant_type == "client_credentials" { + if is_client_credentials { for scope in effective_scopes.iter() { client.add_scope(scope); } } + let extra_params_callback = oauth_client_info + .as_ref() + .and_then(|i| i.extra_params_callback.clone()) + .or_else(|| { + cc_config + .as_ref() + .and_then(|c| c.extra_params_callback.clone()) + }); + tracing::info!( grant_type = %account.grant_type, client = %account.client, @@ -634,7 +984,7 @@ pub async fn refresh_token_for_account<'c>( client, &account.refresh_token, &account.grant_type, - Some(&oauth_client_info), + extra_params_callback.as_ref(), http_client, Some(effective_scopes), ) @@ -846,6 +1196,7 @@ mod tests { token_url: "https://account.example.com/oauth/token".to_string(), userinfo_url: Some("https://account.example.com/userinfo".to_string()), scopes: Some(vec!["signature".to_string()]), + cc_scopes: None, extra_params: None, extra_params_callback: None, req_body_auth: None, @@ -927,4 +1278,114 @@ mod tests { registry.insert("docusign".to_string(), sample_oauth_config(false)); assert!(resolve_registry_config(®istry, "docusign_sandbox").is_none()); } + + const CC_REGISTRY: &str = r#"{ + "coupa": { + "grant_types": ["client_credentials"], + "connect_config_template": { + "label": "Coupa instance", + "placeholder": "x", + "token_url": "https://{instance}.coupahost.com/oauth2/token", + "strip_suffix": ".coupahost.com" + } + }, + "servicenow": { + "grant_types": ["authorization_code", "client_credentials"], + "connect_config_template": { + "label": "ServiceNow instance", + "placeholder": "dev12345", + "auth_url": "https://{instance}.service-now.com/oauth_auth.do", + "token_url": "https://{instance}.service-now.com/oauth_token.do", + "strip_suffix": ".service-now.com" + } + }, + "visma": { + "auth_url": "https://connect.visma.com/connect/authorize", + "token_url": "https://connect.visma.com/connect/token", + "grant_types": ["authorization_code", "client_credentials"] + }, + "bad_host_tpl": { + "grant_types": ["client_credentials"], + "connect_config_template": { + "label": "x", "placeholder": "x", + "token_url": "https://{instance}/token" + } + }, + "bad_mid_tpl": { + "grant_types": ["client_credentials"], + "connect_config_template": { + "label": "x", "placeholder": "x", + "token_url": "https://api.{instance}.evil.com/token" + } + } + }"#; + + #[test] + fn cc_token_url_templated_substitutes_instance() { + let url = resolve_cc_token_url_input(CC_REGISTRY, "coupa", Some("acme")).unwrap(); + assert_eq!(url, "https://acme.coupahost.com/oauth2/token"); + } + + #[test] + fn cc_token_url_templated_from_connect_config_template() { + // ServiceNow's CC token URL comes from its connect_config_template. + let url = resolve_cc_token_url_input(CC_REGISTRY, "servicenow", Some("dev99")).unwrap(); + assert_eq!(url, "https://dev99.service-now.com/oauth_token.do"); + } + + #[test] + fn cc_token_url_strips_known_host_suffix() { + let url = + resolve_cc_token_url_input(CC_REGISTRY, "coupa", Some("acme.coupahost.com")).unwrap(); + assert_eq!(url, "https://acme.coupahost.com/oauth2/token"); + } + + #[test] + fn cc_token_url_rejects_instance_that_escapes_the_host() { + // A '/' (or any non-hostname char) must not let the caller move the host + // out of the template's domain. + assert!(resolve_cc_token_url_input(CC_REGISTRY, "coupa", Some("evil.com/oauth")).is_err()); + assert!(resolve_cc_token_url_input(CC_REGISTRY, "coupa", Some("a@b")).is_err()); + } + + #[test] + fn cc_token_url_requires_instance_when_templated() { + assert!(resolve_cc_token_url_input(CC_REGISTRY, "coupa", None).is_err()); + } + + #[test] + fn cc_token_url_fixed_host_uses_registry_url() { + let url = resolve_cc_token_url_input(CC_REGISTRY, "visma", None).unwrap(); + assert_eq!(url, "https://connect.visma.com/connect/token"); + } + + #[test] + fn cc_token_url_rejects_custom_provider() { + // No registry entry: bring-your-own client credentials are not allowed. + assert!(resolve_cc_token_url_input(CC_REGISTRY, "my_custom_thing", Some("acme")).is_err()); + } + + #[test] + fn cc_token_url_rejects_template_not_in_subdomain_position() { + // `{instance}` must be the leftmost host label of a fixed-host template, so + // a malformed template can't let the instance value control the host. + assert!(resolve_cc_token_url_input(CC_REGISTRY, "bad_host_tpl", Some("evil.com")).is_err()); + assert!(resolve_cc_token_url_input(CC_REGISTRY, "bad_mid_tpl", Some("evil")).is_err()); + } + + #[test] + fn instance_templated_cc_true_for_templated_providers() { + // Host-pinned via `{instance}`: a bring-your-own token URL override must be + // refused for these (only the instance-name path may set their URL). + assert!(is_instance_templated_cc(CC_REGISTRY, "coupa")); + assert!(is_instance_templated_cc(CC_REGISTRY, "servicenow")); + } + + #[test] + fn instance_templated_cc_false_for_fixed_host_and_unknown() { + // Fixed-host registry provider and custom (non-registry) provider both allow + // an override, so neither is reported as instance-templated. + assert!(!is_instance_templated_cc(CC_REGISTRY, "visma")); + assert!(!is_instance_templated_cc(CC_REGISTRY, "my_custom_thing")); + } } diff --git a/backend/windmill-object-store/src/lib.rs b/backend/windmill-object-store/src/lib.rs index aea5a0e4cd..cf30efe116 100644 --- a/backend/windmill-object-store/src/lib.rs +++ b/backend/windmill-object-store/src/lib.rs @@ -52,6 +52,8 @@ use tokio::task; #[cfg(feature = "parquet")] use windmill_common::error::to_anyhow; #[cfg(feature = "parquet")] +use windmill_common::jobs::is_safe_log_file_path; +#[cfg(feature = "parquet")] use windmill_common::utils::rd_string; #[cfg(all(feature = "parquet", feature = "private"))] pub mod job_s3_helpers_ee; @@ -533,7 +535,134 @@ pub fn build_filesystem_client(root_path: &str) -> error::Result) -> std::fmt::Result { + self.0.fmt(f) + } +} + +#[cfg(feature = "parquet")] +#[async_trait] +impl ObjectStore for FilesystemStoreIgnoringAttributes { + async fn put_opts( + &self, + location: &object_store::path::Path, + payload: object_store::PutPayload, + mut opts: object_store::PutOptions, + ) -> object_store::Result { + opts.attributes = Default::default(); + self.0.put_opts(location, payload, opts).await + } + + async fn put_multipart_opts( + &self, + location: &object_store::path::Path, + mut opts: object_store::PutMultipartOpts, + ) -> object_store::Result> { + opts.attributes = Default::default(); + self.0.put_multipart_opts(location, opts).await + } + + async fn get_opts( + &self, + location: &object_store::path::Path, + options: object_store::GetOptions, + ) -> object_store::Result { + self.0.get_opts(location, options).await + } + + async fn get_range( + &self, + location: &object_store::path::Path, + range: std::ops::Range, + ) -> object_store::Result { + self.0.get_range(location, range).await + } + + async fn get_ranges( + &self, + location: &object_store::path::Path, + ranges: &[std::ops::Range], + ) -> object_store::Result> { + self.0.get_ranges(location, ranges).await + } + + async fn head( + &self, + location: &object_store::path::Path, + ) -> object_store::Result { + self.0.head(location).await + } + + async fn delete(&self, location: &object_store::path::Path) -> object_store::Result<()> { + self.0.delete(location).await + } + + fn list( + &self, + prefix: Option<&object_store::path::Path>, + ) -> futures::stream::BoxStream<'static, object_store::Result> { + self.0.list(prefix) + } + + fn list_with_offset( + &self, + prefix: Option<&object_store::path::Path>, + offset: &object_store::path::Path, + ) -> futures::stream::BoxStream<'static, object_store::Result> { + self.0.list_with_offset(prefix, offset) + } + + async fn list_with_delimiter( + &self, + prefix: Option<&object_store::path::Path>, + ) -> object_store::Result { + self.0.list_with_delimiter(prefix).await + } + + async fn copy( + &self, + from: &object_store::path::Path, + to: &object_store::path::Path, + ) -> object_store::Result<()> { + self.0.copy(from, to).await + } + + async fn rename( + &self, + from: &object_store::path::Path, + to: &object_store::path::Path, + ) -> object_store::Result<()> { + self.0.rename(from, to).await + } + + async fn copy_if_not_exists( + &self, + from: &object_store::path::Path, + to: &object_store::path::Path, + ) -> object_store::Result<()> { + self.0.copy_if_not_exists(from, to).await + } + + async fn rename_if_not_exists( + &self, + from: &object_store::path::Path, + to: &object_store::path::Path, + ) -> object_store::Result<()> { + self.0.rename_if_not_exists(from, to).await + } } #[cfg(feature = "parquet")] @@ -1296,6 +1425,9 @@ pub async fn get_logs_from_store( ) -> Option>> { if log_offset > 0 { if let Some(file_index) = log_file_index.clone() { + if file_index.iter().any(|p| !is_safe_log_file_path(p)) { + return None; + } if let Some(os) = get_object_store().await { let logs = logs.to_string(); let stream = async_stream::stream! { diff --git a/backend/windmill-queue/Cargo.toml b/backend/windmill-queue/Cargo.toml index ac956cb709..b9bd6fd28e 100644 --- a/backend/windmill-queue/Cargo.toml +++ b/backend/windmill-queue/Cargo.toml @@ -17,10 +17,14 @@ benchmark = ["windmill-common/benchmark"] failpoints = [] prometheus = ["dep:prometheus"] smtp = [] +# Enables native evaluation of `retry_if` expressions on the job-failure path. +# Without it, a `retry_if` gate cannot be evaluated and the job does not retry. +quickjs = ["dep:windmill-jseval", "windmill-jseval/quickjs"] [dependencies] windmill-audit.workspace = true windmill-common = { workspace = true, default-features = false } +windmill-jseval = { workspace = true, optional = true } anyhow.workspace = true hmac.workspace = true sql-builder.workspace = true diff --git a/backend/windmill-queue/src/asset_dispatch.rs b/backend/windmill-queue/src/asset_dispatch.rs new file mode 100644 index 0000000000..753614d07b --- /dev/null +++ b/backend/windmill-queue/src/asset_dispatch.rs @@ -0,0 +1,763 @@ +/* + * Author: Ruben Fiszel + * Copyright: Windmill Labs, Inc 2022 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +//! Runtime fan-out for asset-triggered scripts. +//! +//! When a script writes an asset and a downstream script subscribes to +//! that asset via `// on s3://...`, this module pushes a job for each +//! subscriber after the producer's job completes successfully. Any +//! asset-writing top-level script cascades — there is no `// pipeline` +//! gate on the producer side; subscriptions alone define the graph. +//! +//! Eligibility (V1, narrow on purpose): +//! - Producer kind is `Script` or `Preview`. Flows defer. +//! - Producer is top-level (no `parent_job`, no `flow_step_id`). +//! - Producer succeeded. +//! - The producer's args do not contain `_wmill_skip_asset_dispatch: true`. +//! +//! Subscribers (V1): +//! - Only `script` runnables. Flow subscribers defer. +//! - The subscriber must have at least one non-archived script row. +//! - A subscriber is skipped if its path equals the producer's path +//! (self-loop) or already appears in the cascade lineage +//! (`trigger.chain`) — cycle detection, which bounds the cascade +//! without capping legitimate depth. +//! +//! Args sent to subscribers: +//! ```json +//! { +//! "trigger": { +//! "kind": "asset", +//! "asset_kind": "s3object", +//! "asset_path": "...", +//! "producer_path": "...", +//! "producer_job_id": "...", +//! "chain": ["f/a/producer0", "f/a/producer1"] +//! } +//! } +//! ``` +//! +//! Errors are logged but never bubble up to fail the producer's job. + +use crate::{push, MiniCompletedJob, PushArgs, PushIsolationLevel}; +use serde_json::value::RawValue; +use sqlx::types::Json; +use sqlx::{Pool, Postgres}; +use std::collections::HashMap; +use std::sync::Arc; +use uuid::Uuid; +use windmill_common::assets::AssetKind; +use windmill_common::error::{self, Result}; +use windmill_common::get_latest_deployed_hash_for_path; +use windmill_common::jobs::{JobKind, JobPayload, JobTriggerKind}; +use windmill_common::partition::PARTITION_ARG; +use windmill_common::scripts::ScriptHash; +use windmill_common::triggers::TriggerMetadata; +use windmill_common::users::{get_email_from_permissioned_as, username_to_permissioned_as}; +use windmill_common::worker::to_raw_value; +use windmill_common::DB; + +/// Reserved arg key that suppresses asset-trigger dispatch for a single run. +/// Set by the test panel when the user opts out of the cascade. +pub const SKIP_ASSET_DISPATCH_ARG: &str = "_wmill_skip_asset_dispatch"; + +/// Arg key holding the cascade trigger object (carries `chain`, `partition`, +/// producer metadata) injected into every dispatched subscriber. +const TRIGGER_ARG: &str = "trigger"; + +/// Arg key (under `trigger.chain`) carrying the cascade lineage: the ordered +/// list of producer paths already run in this chain. Used to detect cycles +/// (a producer re-appearing) and stop only the cyclic edge — so deep but +/// *acyclic* pipelines are never truncated. +const CHAIN_KEY: &str = "chain"; + +/// Safety backstop on lineage length. Cycle detection already bounds an +/// acyclic cascade (a path can't repeat), so this only guards against a +/// runaway from a bug. Set far above any real pipeline depth. +const MAX_CHAIN_LEN: usize = 1000; + +/// Returned to the caller (the worker's completed-job hook) so logs can +/// reference the dispatched ids. +#[derive(Debug, Default)] +pub struct DispatchResult { + pub dispatched: Vec, +} + +/// Per-decision outcome persisted to `dispatch_event` so the producer's +/// job detail page can show what happened to each subscriber. Mirrors +/// the `DISPATCH_OUTCOME` Postgres enum exactly. +#[derive(Debug, Clone, Copy, sqlx::Type)] +#[sqlx(type_name = "DISPATCH_OUTCOME", rename_all = "snake_case")] +enum DispatchOutcome { + Dispatched, + JoinPending, + Skipped, +} + +/// Outcome-specific fields. Event constructors take the four "always-present" +/// columns positionally and bundle the rest here so each call site only +/// names what it actually carries. +#[derive(Debug, Default)] +struct EventOptions<'a> { + child_job_id: Option, + partition: Option<&'a str>, + received_inputs: Option, + required_inputs: Option, + debounce_s: Option, + reason: Option<&'a str>, +} + +/// One accumulated `dispatch_event` row. Owned (not borrowed) so the whole +/// dispatch pass can collect rows and flush them in a single batched INSERT +/// at the end, avoiding an N+1 (one INSERT per subscriber × asset write). +#[derive(Debug)] +struct EventRow { + subscriber_path: String, + asset_kind: AssetKind, + asset_path: String, + outcome: DispatchOutcome, + child_job_id: Option, + partition: Option, + received_inputs: Option, + required_inputs: Option, + debounce_s: Option, + reason: Option, +} + +impl EventRow { + fn new( + subscriber_path: &str, + asset_kind: AssetKind, + asset_path: &str, + outcome: DispatchOutcome, + opts: EventOptions<'_>, + ) -> Self { + EventRow { + subscriber_path: subscriber_path.to_string(), + asset_kind, + asset_path: asset_path.to_string(), + outcome, + child_job_id: opts.child_job_id, + partition: opts.partition.map(str::to_string), + received_inputs: opts.received_inputs, + required_inputs: opts.required_inputs, + debounce_s: opts.debounce_s, + reason: opts.reason.map(str::to_string), + } + } +} + +/// Best-effort batched insert into `dispatch_event`. Never propagates — the +/// dispatch contract is "logging failures must not retroactively fail the +/// producer's job." All rows accumulated over a dispatch pass go in one +/// INSERT (UNNEST) to avoid an N+1 across (subscriber × asset write). +async fn flush_events(db: &DB, workspace_id: &str, producer_job_id: Uuid, events: &[EventRow]) { + if events.is_empty() { + return; + } + // Column-oriented arrays for UNNEST. Each Vec is one column across all rows. + let subscriber_paths: Vec = events.iter().map(|e| e.subscriber_path.clone()).collect(); + let asset_kinds: Vec = events.iter().map(|e| e.asset_kind).collect(); + let asset_paths: Vec = events.iter().map(|e| e.asset_path.clone()).collect(); + let outcomes: Vec = events.iter().map(|e| e.outcome).collect(); + let child_job_ids: Vec> = events.iter().map(|e| e.child_job_id).collect(); + let partitions: Vec> = events.iter().map(|e| e.partition.clone()).collect(); + let received_inputs: Vec> = events.iter().map(|e| e.received_inputs).collect(); + let required_inputs: Vec> = events.iter().map(|e| e.required_inputs).collect(); + let debounce_s: Vec> = events.iter().map(|e| e.debounce_s).collect(); + let reasons: Vec> = events.iter().map(|e| e.reason.clone()).collect(); + + let res = sqlx::query!( + r#"INSERT INTO dispatch_event ( + workspace_id, producer_job_id, subscriber_path, + asset_kind, asset_path, outcome, + child_job_id, partition, + received_inputs, required_inputs, + debounce_s, reason + ) + SELECT $1, $2, sp, ak, ap, oc, cj, pt, ri, rq, db, rs + FROM unnest( + $3::text[], $4::ASSET_KIND[], $5::text[], $6::DISPATCH_OUTCOME[], + $7::uuid[], $8::text[], $9::int[], $10::int[], $11::int[], $12::text[] + ) AS t(sp, ak, ap, oc, cj, pt, ri, rq, db, rs)"#, + workspace_id, + producer_job_id, + &subscriber_paths, + asset_kinds as Vec, + &asset_paths, + outcomes as Vec, + &child_job_ids as &[Option], + &partitions as &[Option], + &received_inputs as &[Option], + &required_inputs as &[Option], + &debounce_s as &[Option], + &reasons as &[Option], + ) + .execute(db) + .await; + if let Err(e) = res { + tracing::error!( + "failed to record {} dispatch_event row(s) for producer {}: {e:#}", + events.len(), + producer_job_id + ); + } +} + +/// Top-level entry. Returns `Ok(default)` and logs on any internal failure +/// rather than propagating, because dispatch is best-effort and must not +/// retroactively fail the producer. +pub async fn dispatch_asset_triggers(db: &DB, job: &MiniCompletedJob) -> DispatchResult { + match try_dispatch(db, job).await { + Ok(r) => r, + Err(e) => { + tracing::error!("asset-trigger dispatch failed for job {}: {e:#}", job.id); + DispatchResult::default() + } + } +} + +async fn try_dispatch(db: &DB, job: &MiniCompletedJob) -> Result { + if !is_eligible_kind(job) { + return Ok(DispatchResult::default()); + } + // A parented script is dispatch-eligible only as a native retry attempt — a + // re-run of the SAME runnable as its chain parent. Schedule/error/recovery + // handlers are also parented `Script` children but run a DIFFERENT script; + // excluding them stops a handler that happens to declare assets from + // triggering a cascade (the pre-native-retry `parent_job IS NULL` guard + // excluded every parented child). + if job.parent_job.is_some() && !is_native_retry_attempt(db, job).await? { + return Ok(DispatchResult::default()); + } + let runnable_path = match job.runnable_path.as_deref() { + Some(p) if !p.is_empty() => p, + _ => return Ok(DispatchResult::default()), + }; + + // Producer gate (cached): this hook fires on every top-level + // script/preview completion, and the overwhelmingly common case is a + // script that writes no asset. The per-workspace producer→writes map is + // cached and invalidated by a trigger on `asset`, so a non-producer + // completion costs one in-memory lookup and zero queries. The map is + // keyed on the deploy-time `asset` table by path, so an undeployed/new + // preview (no asset rows for its path) is a non-producer and never + // cascades — same as the previous per-completion lookup. + let producers = workspace_producer_writes(db, &job.workspace_id).await?; + let Some(writes) = producers.get(runnable_path).cloned() else { + return Ok(DispatchResult::default()); + }; + + let args = fetch_args(db, &job.workspace_id, job.id).await?; + if read_skip_arg(args.as_ref()) { + return Ok(DispatchResult::default()); + } + // Parse the cascade `trigger` object once; both the lineage chain and + // the propagated partition are read from it. + let trigger_map = args + .as_ref() + .and_then(|a| a.get(TRIGGER_ARG)) + .and_then(|t| serde_json::from_str::>>(t.get()).ok()); + let chain = read_chain(trigger_map.as_ref()); + let partition = read_partition(args.as_ref(), trigger_map.as_ref()); + if chain.len() >= MAX_CHAIN_LEN { + tracing::warn!( + "asset-trigger dispatch skipped: cascade lineage length {} >= backstop {} (job {}, path {})", + chain.len(), + MAX_CHAIN_LEN, + job.id, + runnable_path + ); + return Ok(DispatchResult::default()); + } + // Lineage propagated to any subscriber pushed from this producer: the + // ancestors that already ran, plus this producer. + let mut next_chain = chain.clone(); + next_chain.push(runnable_path.to_string()); + + let mut dispatched = Vec::new(); + // Best-effort dispatch_event rows accumulated over the whole pass and + // flushed in one batched INSERT at the end (avoids an N+1 over + // subscriber × asset write). The mid-pass join-slot writes + // (record_and_check_join_slot) are a separate table and unaffected. + let mut events: Vec = Vec::new(); + for (asset_kind, asset_path) in writes { + let Some(prefix) = asset_kind.canonical_prefix() else { + continue; + }; + let trigger_ref = format!("{}{}", prefix, asset_path); + let subs = fetch_subscribers(db, &job.workspace_id, &trigger_ref).await?; + for sub in subs { + let Subscriber { path: sub_path, join_all, debounce_s, retry_count, retry_delay_s } = + sub; + if sub_path == runnable_path { + events.push(EventRow::new( + &sub_path, + asset_kind, + &asset_path, + DispatchOutcome::Skipped, + EventOptions { reason: Some("self_loop"), ..Default::default() }, + )); + continue; + } + // Cycle guard: a subscriber already in this producer's lineage + // would re-enter the chain (A→…→A), looping forever. Stop only + // this edge — sibling branches still dispatch, and acyclic chains + // of any depth are unaffected. + if chain.iter().any(|p| p == &sub_path) { + events.push(EventRow::new( + &sub_path, + asset_kind, + &asset_path, + DispatchOutcome::Skipped, + EventOptions { reason: Some("cycle_detected"), ..Default::default() }, + )); + continue; + } + if join_all { + match crate::cascade::handle_join( + db, + &job.workspace_id, + &sub_path, + &trigger_ref, + partition.as_deref(), + ) + .await + { + Ok(crate::cascade::JoinDecision::Skip(reason)) => { + events.push(EventRow::new( + &sub_path, + asset_kind, + &asset_path, + DispatchOutcome::Skipped, + EventOptions { reason: Some(reason), ..Default::default() }, + )); + continue; + } + Ok(crate::cascade::JoinDecision::Pending { received, required }) => { + events.push(EventRow::new( + &sub_path, + asset_kind, + &asset_path, + DispatchOutcome::JoinPending, + EventOptions { + partition: partition.as_deref(), + received_inputs: Some(received), + required_inputs: Some(required), + ..Default::default() + }, + )); + continue; // slot incomplete — wait for the rest + } + Ok(crate::cascade::JoinDecision::Fire) => {} // fall through to push + Err(e) => { + tracing::error!("join-slot check failed for {}: {e:#}", sub_path); + continue; + } + } + } + match push_subscriber( + db, + job, + &sub_path, + asset_kind, + &asset_path, + runnable_path, + &next_chain, + partition.as_deref(), + debounce_s, + retry_count, + retry_delay_s, + ) + .await + { + Ok(id) => { + events.push(EventRow::new( + &sub_path, + asset_kind, + &asset_path, + DispatchOutcome::Dispatched, + EventOptions { + child_job_id: Some(id), + partition: partition.as_deref(), + debounce_s, + ..Default::default() + }, + )); + dispatched.push(id); + } + Err(e) => { + tracing::error!("failed to push asset-triggered job for {}: {e:#}", sub_path) + } + } + } + } + + flush_events(db, &job.workspace_id, job.id, &events).await; + + if !dispatched.is_empty() { + tracing::info!( + "asset-trigger dispatch from job {} ({}): pushed {} downstream jobs", + job.id, + runnable_path, + dispatched.len() + ); + } + Ok(DispatchResult { dispatched }) +} + +fn is_eligible_kind(job: &MiniCompletedJob) -> bool { + if !matches!(job.kind, JobKind::Script | JobKind::Preview) { + return false; + } + // Flow steps (and sub-flow jobs) carry `flow_step_id` and are ineligible. + // Native script-retry attempts carry `parent_job` (the chain root) but no + // `flow_step_id`; whether a parented job is actually a retry attempt (vs a + // schedule/error handler child) is decided in `try_dispatch`. + if job.flow_step_id.is_some() { + return false; + } + true +} + +// Native retry attempts carry an explicit `native_retry_attempt` marker; no +// other parented `Script` child (schedule handlers, WAC inline children, flow +// steps) does. One indexed point lookup, only for parented jobs. +async fn is_native_retry_attempt(db: &DB, job: &MiniCompletedJob) -> Result { + Ok(sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM native_retry_attempt WHERE job_id = $1) AS \"exists!\"", + job.id, + ) + .fetch_one(db) + .await?) +} + +async fn fetch_args( + db: &Pool, + workspace_id: &str, + job_id: Uuid, +) -> Result>>> { + // Read from v2_job because args live there permanently — v2_job_completed + // is the *result* row and doesn't carry args. The producer's v2_job row + // is still present at dispatch time (deletion happens later in the + // completion pipeline, after this hook). + let row = sqlx::query!( + r#"SELECT args AS "args!: Json>>" + FROM v2_job + WHERE workspace_id = $1 AND id = $2"#, + workspace_id, + job_id, + ) + .fetch_optional(db) + .await?; + Ok(row.map(|r| r.args.0)) +} + +fn read_skip_arg(args: Option<&HashMap>>) -> bool { + args.and_then(|a| a.get(SKIP_ASSET_DISPATCH_ARG)) + .and_then(|v| serde_json::from_str::(v.get()).ok()) + .unwrap_or(false) +} + +fn read_chain(trigger_map: Option<&HashMap>>) -> Vec { + trigger_map + .and_then(|m| m.get(CHAIN_KEY)) + .and_then(|v| serde_json::from_str::>(v.get()).ok()) + .unwrap_or_default() +} + +/// The partition value the producer ran with, if any. Resolved once at the +/// top of a chain (run-start) and threaded down here so every cascaded job +/// materializes the same partition without re-resolving. Top-level +/// `partition` arg (run-start injection) takes precedence over the +/// `trigger.partition` carried from an upstream cascade hop. +fn read_partition( + args: Option<&HashMap>>, + trigger_map: Option<&HashMap>>, +) -> Option { + if let Some(v) = args.and_then(|a| a.get(PARTITION_ARG)) { + if let Ok(s) = serde_json::from_str::(v.get()) { + return Some(s); + } + } + serde_json::from_str::(trigger_map?.get(PARTITION_ARG)?.get()).ok() +} + +lazy_static::lazy_static! { + /// Per-workspace map of producer script path → the assets it writes + /// (`usage_access_type IN ('w','rw')`). Serves both the producer gate + /// (is this path a producer?) and the writes themselves, so a completion + /// that isn't a producer costs a single in-memory lookup and zero + /// queries — the dispatch hook fires on every top-level script/preview + /// completion instance-wide, the overwhelming majority of which write no + /// asset. An empty map means the workspace has no asset producers (no + /// pipelines). Invalidated per workspace by `notify_asset_producer_change` + /// (a trigger on `asset`) through the polling notify system; until the + /// next poll a freshly-deployed producer may not cascade (sub-poll lag, + /// acceptable for a data pipeline). + pub static ref ASSET_PRODUCER_WRITES_CACHE: + quick_cache::sync::Cache>>> = + quick_cache::sync::Cache::new(1000); +} + +/// Test hook: disables the producer-writes cache so every dispatch reads the +/// current DB. Integration tests use `#[sqlx::test]` isolated DBs that all +/// share one workspace id, so a process-global cache keyed by workspace would +/// clobber across DBs under concurrent test threads. Always `false` in +/// production (the cache is invalidated via the notify_event poller instead). +pub static ASSET_PRODUCER_CACHE_DISABLED: std::sync::atomic::AtomicBool = + std::sync::atomic::AtomicBool::new(false); + +/// Load (cached) the producer→writes map for a workspace. The single load +/// query replaces the per-completion producer lookup; once cached, every +/// completion in the workspace is served from memory until invalidation. +async fn workspace_producer_writes( + db: &Pool, + workspace_id: &str, +) -> Result>>> { + let use_cache = !ASSET_PRODUCER_CACHE_DISABLED.load(std::sync::atomic::Ordering::Relaxed); + if use_cache { + if let Some(map) = ASSET_PRODUCER_WRITES_CACHE.get(workspace_id) { + return Ok(map); + } + } + let rows = sqlx::query!( + r#" + SELECT + usage_path AS "usage_path!", + kind AS "kind!: AssetKind", + path AS "path!" + FROM asset + WHERE workspace_id = $1 + AND usage_kind = 'script' + AND usage_access_type IN ('w', 'rw') + "#, + workspace_id, + ) + .fetch_all(db) + .await?; + let mut map: HashMap> = HashMap::new(); + for r in rows { + map.entry(r.usage_path).or_default().push((r.kind, r.path)); + } + let map = Arc::new(map); + if use_cache { + ASSET_PRODUCER_WRITES_CACHE.insert(workspace_id.to_string(), map.clone()); + } + Ok(map) +} + +/// A subscriber row resolved from `script_trigger`. Bundles the per-edge +/// options (debounce) and the script-level policy fields (`join_all`, +/// retry) that travel together to dispatch. +struct Subscriber { + path: String, + join_all: bool, + debounce_s: Option, + retry_count: Option, + retry_delay_s: Option, +} + +async fn fetch_subscribers( + db: &Pool, + workspace_id: &str, + trigger_ref: &str, +) -> Result> { + // V1: script subscribers only. Flow subscribers (`runnable_kind = 'flow'`) + // are intentionally excluded — wiring them is straightforward but the + // payload shape and permissioning need their own pass. + // `join_all` = `// trigger all` (AND join); `debounce_s` = the opt-in + // debounce window resolved at deploy (NULL = fan-out, the default). + // `retry_count` / `retry_delay_s` = the `// retry []` policy + // (NULL = no retry). + let rows = sqlx::query!( + r#" + SELECT runnable_path AS "runnable_path!", join_all AS "join_all!", debounce_s, + retry_count, retry_delay_s + FROM script_trigger + WHERE workspace_id = $1 + AND trigger_kind = 'asset' + AND trigger_ref = $2 + AND runnable_kind = 'script' + "#, + workspace_id, + trigger_ref, + ) + .fetch_all(db) + .await?; + Ok(rows + .into_iter() + .map(|r| Subscriber { + path: r.runnable_path, + join_all: r.join_all, + debounce_s: r.debounce_s, + retry_count: r.retry_count, + retry_delay_s: r.retry_delay_s, + }) + .collect()) +} + +async fn push_subscriber( + db: &DB, + producer: &MiniCompletedJob, + subscriber_path: &str, + asset_kind: AssetKind, + asset_path: &str, + producer_path: &str, + chain: &[String], + partition: Option<&str>, + debounce_s: Option, + retry_count: Option, + retry_delay_s: Option, +) -> Result { + // Same resolution as every other trigger path (`script_path_to_payload`): + // latest deployed hash plus the script's own runnable settings + // (concurrency, debounce, timeout), resolved through the + // runnable-settings handle. The cascade must not bypass a subscriber's + // concurrency limit just because it was triggered by an asset write. + let script = get_latest_deployed_hash_for_path( + None, + db.clone(), + &producer.workspace_id, + subscriber_path, + ) + .await? + .prefetch_cached(db) + .await?; + let hash = ScriptHash(script.hash); + let tag = script.tag; + let concurrency_settings = script.runnable_settings.concurrency_settings; + + // Debounce / retry semantics are a `private` feature (see `cascade`). + // OSS degrades both: debounce falls back to the subscriber's own + // script-level settings, retry is never applied. + let debouncing_settings = crate::cascade::cascade_debouncing_settings( + subscriber_path, + partition, + debounce_s, + script.runnable_settings.debouncing_settings, + ); + + // When the cascade declares a retry, hand `push` a one-step-flow request + // carrying the policy + `language`; `push` materializes it into a native + // retryable `Script` (not a flow), so a failed/recovered subscriber stays + // eligible to trigger its own downstream. No retry = plain `ScriptHash`. + let payload = if let Some(retry) = crate::cascade::cascade_retry(retry_count, retry_delay_s) { + JobPayload::SingleStepFlow { + path: subscriber_path.to_string(), + hash: Some(hash), + flow_version: None, + language: Some(script.language), + args: HashMap::new(), + retry: Some(retry), + error_handler_path: None, + error_handler_args: None, + skip_handler: None, + cache_ttl: script.cache_ttl, + cache_ignore_s3_path: script.cache_ignore_s3_path, + priority: script.priority, + tag_override: tag.clone(), + trigger_path: None, + apply_preprocessor: false, + concurrency_settings, + debouncing_settings, + } + } else { + JobPayload::ScriptHash { + hash, + path: subscriber_path.to_string(), + cache_ttl: script.cache_ttl, + cache_ignore_s3_path: script.cache_ignore_s3_path, + dedicated_worker: script.dedicated_worker, + language: script.language, + priority: script.priority, + apply_preprocessor: false, + debouncing_settings, + concurrency_settings, + labels: script.labels, + } + }; + + // Run the subscriber under its deployer's identity — never the + // producer's. Subscriptions are workspace-wide, so attributing the run + // to the producer would let anyone who can deploy a `// on` script + // execute code with the permissions of whoever happens to write the + // asset (e.g. an admin's scheduled job). `on_behalf_of_email` (an + // explicit service-account opt-in at deploy) takes precedence for the + // email; otherwise the deployer's email is resolved from their + // username. + let permissioned_as = username_to_permissioned_as(&script.created_by); + let email = match script.on_behalf_of_email { + Some(obo) => obo, + None => { + get_email_from_permissioned_as(&permissioned_as, &producer.workspace_id, db).await? + } + }; + + let mut args: HashMap> = HashMap::new(); + let trigger_payload = serde_json::json!({ + "kind": "asset", + "asset_kind": serde_json::to_value(&asset_kind).expect("AssetKind serializes"), + "asset_path": asset_path, + "producer_path": producer_path, + "producer_job_id": producer.id.to_string(), + CHAIN_KEY: chain, + PARTITION_ARG: partition, + }); + args.insert(TRIGGER_ARG.to_string(), to_raw_value(&trigger_payload)); + // Carry the producer's resolved partition forward as a top-level arg so + // the subscriber's body can read it and the next cascade hop's + // `read_partition` picks it up — keeps the whole chain on one partition, + // resolved once at the top. Omitted entirely for non-partitioned chains. + if let Some(p) = partition { + args.insert(PARTITION_ARG.to_string(), to_raw_value(&p)); + } + + // Attribute the dispatched run to a synthetic user so audit logs reflect + // it came from the asset cascade, not the original human runner. + let pseudo_user = format!("asset-{producer_path}"); + + let tx = PushIsolationLevel::IsolatedRoot(db.clone()); + let (id, tx) = push( + db, + tx, + &producer.workspace_id, + payload, + PushArgs { args: &args, extra: None }, + &pseudo_user, + &email, + permissioned_as, + Some(producer_path), + None, + Some(producer_path.to_string()), + None, + None, + None, + None, + false, + false, + None, + true, + tag, + script.timeout, + None, + None, + None, + false, + None, + Some(TriggerMetadata::new( + Some(producer_path.to_string()), + JobTriggerKind::Asset, + )), + None, + ) + .await + .map_err(|e| error::Error::internal_err(format!("push asset-triggered job: {e:#}")))?; + tx.commit().await?; + Ok(id) +} diff --git a/backend/windmill-queue/src/cascade_oss.rs b/backend/windmill-queue/src/cascade_oss.rs new file mode 100644 index 0000000000..be6f93d732 --- /dev/null +++ b/backend/windmill-queue/src/cascade_oss.rs @@ -0,0 +1,60 @@ +//! OSS fallback for the asset-trigger cascade's AND-join / debounce / retry. +//! +//! The richer cascade semantics are a `private` feature (see `cascade_ee`). +//! In the public build they are absent and the cascade degrades to a plain +//! OR fan-out: every join always fires immediately, no slots are recorded or +//! reaped, debounce falls back to the subscriber's own script-level settings, +//! and retry is never applied (the subscriber pushes as a bare `ScriptHash`). +//! The real implementation lives in `windmill-ee-private`. + +use windmill_common::error::Result; +use windmill_common::flows::Retry; +use windmill_common::runnable_settings::DebouncingSettings; +use windmill_common::DB; + +/// Outcome of evaluating an AND-join barrier for one (subscriber, input). +/// Kept identical to the EE definition so the core matcher in +/// `asset_dispatch` compiles against either build. +pub enum JoinDecision { + /// Input does not advance the join (recorded as Skipped with `reason`). + Skip(&'static str), + /// Join advanced but is not yet complete (recorded as JoinPending). + Pending { received: i32, required: i32 }, + /// All required inputs are present — push the subscriber. + Fire, +} + +/// OSS has no AND-join: every input fires immediately (plain OR fan-out). +pub async fn handle_join( + _db: &DB, + _workspace_id: &str, + _sub_path: &str, + _trigger_ref: &str, + _partition: Option<&str>, +) -> Result { + Ok(JoinDecision::Fire) +} + +/// Unused in OSS (no slots are ever recorded), kept for API parity. +pub const JOIN_SLOT_TTL_SECS: i64 = 60 * 24 * 60 * 60; // 60 days + +/// No-op: OSS never records join slots, so there is nothing to reap. +pub async fn reap_stale_join_slots(_db: &DB) -> Result<()> { + Ok(()) +} + +/// No per-edge debounce in OSS — always defer to the subscriber's own +/// script-level debounce settings. +pub fn cascade_debouncing_settings( + _subscriber_path: &str, + _partition: Option<&str>, + _debounce_s: Option, + fallback: DebouncingSettings, +) -> DebouncingSettings { + fallback +} + +/// No per-edge retry in OSS — the subscriber pushes as a bare `ScriptHash`. +pub fn cascade_retry(_retry_count: Option, _retry_delay_s: Option) -> Option { + None +} diff --git a/backend/windmill-queue/src/jobs.rs b/backend/windmill-queue/src/jobs.rs index 5cb8911c7e..c92cea9ef3 100644 --- a/backend/windmill-queue/src/jobs.rs +++ b/backend/windmill-queue/src/jobs.rs @@ -45,8 +45,8 @@ use windmill_common::min_version::{ MIN_VERSION_SUPPORTS_DEBOUNCING, MIN_VERSION_SUPPORTS_DEBOUNCING_V2, }; use windmill_common::runnable_settings::{ - ConcurrencySettings, ConcurrencySettingsWithCustom, DebouncingSettings, RunnableSettings, - RunnableSettingsTrait, + ConcurrencySettings, ConcurrencySettingsWithCustom, DebouncingSettings, RetrySettings, + RunnableSettings, RunnableSettingsTrait, }; use windmill_common::triggers::TriggerMetadata; use windmill_common::utils::{calculate_hash, configure_client, now_from_db}; @@ -68,7 +68,7 @@ use windmill_common::{ }, flows::{ add_virtual_items_if_necessary, FlowModule, FlowModuleValue, FlowValue, InputTransform, - StopAfterIf, + Retry, StopAfterIf, }, jobs::{get_payload_tag_from_prefixed_path, JobKind, JobPayload, QueuedJob, RawCode}, min_version::{MIN_VERSION_IS_AT_LEAST_1_432, MIN_VERSION_IS_AT_LEAST_1_440}, @@ -958,6 +958,28 @@ pub async fn add_completed_job( )); } + // Native script retry: a failed `Script` job that carries a retry policy and + // has attempts left gets its next attempt enqueued here — before the queue + // row (which holds the attempt counter) is removed by commit. The failed + // attempt is still recorded as a completed job below. `maybe_enqueue_…` + // self-guards on kind/cancellation/policy, so the success path is unaffected. + let retry_pending = if !success && !skipped && !from_cache { + // Serialized lazily, and only when a `retry_if` policy actually needs it. + let result_fn = || serde_json::value::to_raw_value(&result).ok(); + match maybe_enqueue_native_script_retry(db, completed_job, &canceled_by, &result_fn).await { + Ok(enqueued) => enqueued, + Err(e) => { + tracing::error!( + "native retry enqueue failed for {}: {e:#}", + completed_job.id + ); + false + } + } + } else { + false + }; + let result_columns = result_columns.as_ref(); let (opt_uuid, duration, _skip_downstream_error_handlers, wac_job_ids) = (|| { commit_completed_job( @@ -972,6 +994,7 @@ pub async fn add_completed_job( flow_is_done, duration, from_cache, + retry_pending, ) .warn_after_seconds(10) }) @@ -1031,6 +1054,9 @@ async fn commit_completed_job( flow_is_done: bool, duration: Option, from_cache: bool, + // True when a native script retry was enqueued for this failed attempt, i.e. + // this is not the terminal attempt — schedule completion handlers must wait. + retry_pending: bool, ) -> windmill_common::error::Result<(Option, i64, bool, Option)> { // let start = std::time::Instant::now(); @@ -1050,6 +1076,19 @@ async fn commit_completed_job( return value; } + // Resolve the concurrency-limit settings on the pool *before* opening the + // completion transaction: doing it inside the tx would hold a second + // simultaneous connection from the small per-worker pool. + let has_concurrent_limit = completed_job.concurrent_limit.is_some() + || windmill_common::runnable_settings::prefetch_cached_from_handle( + completed_job.runnable_settings_handle, + db, + ) + .await? + .1 + .concurrent_limit + .is_some(); + let mut tx = db.begin().warn_after_seconds(10).await?; let duration = sqlx::query_scalar!( @@ -1258,11 +1297,17 @@ async fn commit_completed_job( // for flows, only try to schedule next tick here if flow failed and because first handle_flow failed (step = 0, modules[0] = {type: 'Failure', 'job': uuid::nil()}) // or job was cancelled before first handle_flow was called (step = 0, modules = [] OR modules[0].type == 'WaitingForPriorSteps') // otherwise flow rescheduling is done inside handle_flow - let schedule_next_tick = !completed_job.is_flow() - || from_cache - || !success - && sqlx::query_scalar!( - "SELECT + // Native retry attempts carry the schedule trigger (so the + // terminal attempt can drive handlers) but `parent_job` is set — + // they must not each push the next cron tick (the root already + // did), so gate next-tick on a top-level (`parent_job IS NULL`) + // occurrence. + let schedule_next_tick = completed_job.parent_job.is_none() + && (!completed_job.is_flow() + || from_cache + || !success + && sqlx::query_scalar!( + "SELECT flow_status->>'step' = '0' AND ( jsonb_array_length(flow_status->'modules') = 0 @@ -1273,15 +1318,15 @@ async fn commit_completed_job( ) ) FROM v2_job_completed WHERE id = $2 AND workspace_id = $3", - Uuid::nil().to_string(), - &completed_job.id, - &completed_job.workspace_id - ) - .fetch_optional(&mut *tx) - .warn_after_seconds(10) - .await? - .flatten() - .unwrap_or(false); + Uuid::nil().to_string(), + &completed_job.id, + &completed_job.workspace_id + ) + .fetch_optional(&mut *tx) + .warn_after_seconds(10) + .await? + .flatten() + .unwrap_or(false)); if schedule_next_tick { let (returned_tx, schedule_push_err) = @@ -1292,42 +1337,55 @@ async fn commit_completed_job( } } + // Defer schedule completion handlers (on_failure/on_success/ + // on_recovery) while a native retry is pending: only the terminal + // attempt should drive them. apply_schedule_handlers resolves + // per-occurrence failure/recovery status across the whole retry + // chain, so multi-count/exact handler policies work even though + // each attempt is its own completed job. #[cfg(all(feature = "enterprise", feature = "private"))] - if let Err(err) = crate::jobs_ee::apply_schedule_handlers( - db, - &schedule, - &script_path, - &completed_job.workspace_id, - success, - result, - job_id, - completed_job.started_at.unwrap_or(chrono::Utc::now()), - completed_job.priority, - ) - .warn_after_seconds(10) - .await - { - if !success { - tracing::error!("Could not apply schedule error handler: {}", err); - let base_url = windmill_common::BASE_URL.load(); - let w_id: &String = &completed_job.workspace_id; - if !matches!(err, Error::QuotaExceeded(_)) { - report_error_to_workspace_handler_or_critical_side_channel( - &completed_job, - db, - format!( - "Failed to push schedule error handler job to handle failed job ({base_url}/run/{}?workspace={w_id}): {}", - completed_job.id, - err - ), - ) - .warn_after_seconds(10) - .await; + if !retry_pending { + if let Err(err) = crate::jobs_ee::apply_schedule_handlers( + db, + &schedule, + &script_path, + &completed_job.workspace_id, + success, + result, + job_id, + // Current occurrence's root: the terminal native-retry + // attempt's parent, else the job itself. + completed_job.parent_job.unwrap_or(job_id), + completed_job.started_at.unwrap_or(chrono::Utc::now()), + completed_job.priority, + ) + .warn_after_seconds(10) + .await + { + if !success { + tracing::error!("Could not apply schedule error handler: {}", err); + let base_url = windmill_common::BASE_URL.load(); + let w_id: &String = &completed_job.workspace_id; + if !matches!(err, Error::QuotaExceeded(_)) { + report_error_to_workspace_handler_or_critical_side_channel( + &completed_job, + db, + format!( + "Failed to push schedule error handler job to handle failed job ({base_url}/run/{}?workspace={w_id}): {}", + completed_job.id, + err + ), + ) + .warn_after_seconds(10) + .await; + } + } else { + tracing::error!("Could not apply schedule recovery handler: {}", err); } - } else { - tracing::error!("Could not apply schedule recovery handler: {}", err); - } - }; + }; + } + #[cfg(not(all(feature = "enterprise", feature = "private")))] + let _ = retry_pending; } else { tracing::error!( "Schedule {schedule_path} in {} not found. Impossible to schedule again and apply schedule handlers", @@ -1337,16 +1395,7 @@ async fn commit_completed_job( } } - if completed_job.concurrent_limit.is_some() - || windmill_common::runnable_settings::prefetch_cached_from_handle( - completed_job.runnable_settings_handle, - db, - ) - .await? - .1 - .concurrent_limit - .is_some() - { + if has_concurrent_limit { let concurrency_key = sqlx::query_scalar!( "SELECT key FROM concurrency_key WHERE job_id = $1", &completed_job.id @@ -1589,6 +1638,267 @@ async fn restart_job_if_perpetual_inner( Ok(()) } +/// Evaluate a `retry_if` JS expression. `result`/`previous_result` are the +/// failure output and `flow_input` the job args. Defaults to retrying on eval +/// error (an unevaluable gate shouldn't silently swallow retries). +#[cfg(feature = "quickjs")] +async fn eval_retry_if( + expr: &str, + result: Option<&serde_json::value::RawValue>, + args: &HashMap>, +) -> bool { + let result_val = result + .and_then(|r| serde_json::from_str::(r.get()).ok()) + .unwrap_or(serde_json::Value::Null); + let mut globals = HashMap::new(); + globals.insert("result".to_string(), result_val.clone()); + globals.insert("previous_result".to_string(), result_val); + globals.insert( + "flow_input".to_string(), + serde_json::to_value(args).unwrap_or(serde_json::Value::Null), + ); + match windmill_jseval::eval_simple_js(format!("Boolean({expr})"), globals).await { + Ok(v) => v.get() == "true", + Err(e) => { + tracing::warn!("Failed to evaluate retry_if expression, retrying anyway: {e:#}"); + true + } + } +} + +/// `retry_if` is unsupported on a worker built without the `quickjs` feature +/// (the expression cannot be evaluated). Such a worker can't run JS jobs either, +/// so this is not reached in practice; we fail closed and do not retry. +#[cfg(not(feature = "quickjs"))] +async fn eval_retry_if( + _expr: &str, + _result: Option<&serde_json::value::RawValue>, + _args: &HashMap>, +) -> bool { + tracing::warn!("retry_if is unsupported without the quickjs feature; not retrying"); + false +} + +/// Native script retry. When a failed `Script` job carries a retry policy (via +/// `runnable_settings_handle`) and has attempts left, enqueue a fresh attempt of +/// the same script after the policy's backoff delay — instead of having wrapped +/// it in a one-step flow. Each attempt is a real `Script` job; the attempt +/// counter lives in the `native_retry_attempt` marker, written here and read only +/// on the next failure (never on the hot job-pull path). +/// +/// Returns `true` if a retry was enqueued. +/// +/// Authorization: this performs no auth check by design. It is `pub` only so the +/// integration test can reach it; the sole production caller is the worker +/// job-completion path (`add_completed_job`), which passes a `MiniCompletedJob` +/// built from a real, already-persisted completed job — its workspace/identity +/// fields come from the DB, not from request input. Callers MUST uphold this: +/// never invoke it with caller-supplied or unauthorized job identity. +pub async fn maybe_enqueue_native_script_retry( + db: &Pool, + job: &MiniCompletedJob, + canceled_by: &Option, + // Lazily serialize the failure result: only `retry_if` policies need it, so + // the common (no-retry_if) failure never pays the serialization cost. + result_fn: &(dyn Fn() -> Option> + Sync), +) -> Result { + // Only plain top-level scripts retry natively; cancellation always wins. + if canceled_by.is_some() || !matches!(job.kind, JobKind::Script) || job.is_flow_step() { + return Ok(false); + } + + let Some(retry_settings) = windmill_common::runnable_settings::prefetch_retry_from_handle( + job.runnable_settings_handle, + db, + ) + .await? + else { + return Ok(false); + }; + let policy: Retry = retry_settings.into(); + if !policy.has_attempts() { + return Ok(false); + } + + // Attempt counter for this job: its `native_retry_attempt` marker, or 0 for + // the first (un-marked) attempt. The marker is persistent (unlike the queue + // row), so it doubles as the explicit "this job is a retry attempt" signal + // consumers key off of. + let prev_attempts = sqlx::query_scalar!( + "SELECT attempt FROM native_retry_attempt WHERE job_id = $1", + job.id, + ) + .fetch_optional(db) + .await? + .unwrap_or(0) as u32; + let root = job.parent_job.unwrap_or(job.id); + // Scheduled chains keep the schedule trigger so the terminal attempt drives + // the schedule completion handlers (on_failure/on_success); `parent_job` + // keeps every retry out of the per-occurrence handler counting queries. + let trigger = job + .schedule_path() + .map(|sp| TriggerMetadata::new(Some(sp), JobTriggerKind::Schedule)); + + let Some(delay) = policy.interval(prev_attempts, false) else { + // Attempts exhausted — let the failure finalize normally. + return Ok(false); + }; + // Cap the backoff to match the flow-runtime retry path (evaluate_retry). + let delay = std::cmp::min(delay, MAX_RETRY_INTERVAL); + let scheduled_for = chrono::Utc::now() + + chrono::Duration::from_std(delay).unwrap_or_else(|_| chrono::Duration::zero()); + + let args = sqlx::query_scalar!( + "SELECT args as \"args: sqlx::types::Json>>\" FROM v2_job WHERE id = $1 AND workspace_id = $2", + job.id, + job.workspace_id, + ) + .fetch_optional(db) + .await? + .flatten() + .unwrap_or_default(); + + // Optional `retry_if`: gate the retry on a JS expression over the failure + // `result` and `flow_input` (the job args). Evaluated by `eval_retry_if`, + // which on a worker built without the `quickjs` feature cannot evaluate the + // expression and fails closed (no retry). + if let Some(retry_if) = policy.retry_if.as_ref() { + let result = result_fn(); + if !eval_retry_if(&retry_if.expr, result.as_deref(), &args.0).await { + return Ok(false); + } + } + + // Re-push as a one-step-flow request; `push` materializes it back into a + // native retryable `Script` carrying the policy again. `parent_job = root` + // links the chain (and excludes retries from schedule-handler counting); the + // schedule trigger, when present, lets the terminal attempt fire handlers. + // + // Idempotent retry id: deterministic per (root, next attempt). If a worker + // dies between this push and the current attempt's finalization, the reaper + // re-handles the un-finalized attempt and we land here again with the same + // id, so the retry is enqueued exactly once (no double-retry). + let retry_job_id = { + use std::hash::{Hash, Hasher}; + let next_attempt = prev_attempts + 1; + let mut high = std::hash::DefaultHasher::new(); + root.hash(&mut high); + next_attempt.hash(&mut high); + let mut low = std::hash::DefaultHasher::new(); + "native-retry".hash(&mut low); + next_attempt.hash(&mut low); + root.hash(&mut low); + Uuid::from_u64_pair(high.finish(), low.finish()) + }; + // If that retry already exists (the crash-and-reaper-replay case above), + // report it as pending WITHOUT re-pushing. Deriving `retry_pending` from the + // push *result* would otherwise flip to false on the duplicate-id error and + // let the schedule completion handlers fire for this non-terminal attempt. + if sqlx::query_scalar!("SELECT 1 FROM v2_job WHERE id = $1", retry_job_id) + .fetch_optional(db) + .await? + .is_some() + { + return Ok(true); + } + // Carry forward the failed attempt's concurrency/debouncing settings (same + // runnable_settings_handle as the retry policy) so a retry of a concurrency- + // limited script still inserts its concurrency_key and respects the limit, + // rather than running unbounded with only the retry policy. + let (debouncing_settings, concurrency_settings) = + windmill_common::runnable_settings::prefetch_cached_from_handle( + job.runnable_settings_handle, + db, + ) + .await?; + let tx = PushIsolationLevel::IsolatedRoot(db.clone()); + let (new_id, mut tx) = match push( + db, + tx, + &job.workspace_id, + JobPayload::SingleStepFlow { + path: job.runnable_path.clone().unwrap_or_default(), + hash: job.runnable_id, + flow_version: None, + language: job.script_lang.clone(), + args: HashMap::new(), + retry: Some(policy), + error_handler_path: None, + error_handler_args: None, + skip_handler: None, + cache_ttl: job.cache_ttl, + cache_ignore_s3_path: job.cache_ignore_s3_path, + priority: job.priority, + tag_override: Some(job.tag.clone()), + trigger_path: None, + apply_preprocessor: false, + concurrency_settings, + debouncing_settings, + }, + PushArgs::from(&args.0), + &job.created_by, + &job.permissioned_as_email, + job.permissioned_as.clone(), + Some(&format!("retry.{}", job.id)), + Some(scheduled_for), + None, + Some(root), + None, + None, + Some(retry_job_id), + false, + false, + None, + true, + Some(job.tag.clone()), + None, + None, + job.priority, + None, + false, + None, + trigger, + None, + ) + .await + { + Ok(v) => v, + Err(e) => { + // Race with a concurrent completion of the same attempt: it may have + // inserted the deterministic retry id between our pre-check and this + // push, so the push fails on the duplicate id. The retry IS pending — + // re-check and report it as such instead of propagating the error + // (which would flip `retry_pending` to false and fire the schedule + // completion handlers for this non-terminal attempt). + if sqlx::query_scalar!("SELECT 1 FROM v2_job WHERE id = $1", retry_job_id) + .fetch_optional(db) + .await? + .is_some() + { + return Ok(true); + } + return Err(e); + } + }; + + sqlx::query!( + "INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, $2)", + new_id, + (prev_attempts + 1) as i32, + ) + .execute(&mut *tx) + .await?; + + tx.commit().await?; + tracing::info!( + "Native retry: enqueued attempt {} of script {:?} (root {root}) in {}s as {new_id}", + prev_attempts + 1, + job.runnable_path, + delay.as_secs(), + ); + Ok(true) +} + #[cfg(feature = "cloud")] fn apply_completed_job_cloud_usage( db: &Pool, @@ -1995,7 +2305,7 @@ pub async fn try_schedule_next_job<'c>( let email = match windmill_common::users::get_email_from_permissioned_as( &permissioned_as, &job.workspace_id, - db, + &mut *tx, ) .await { @@ -3172,108 +3482,189 @@ impl PulledJobResult { if let Some(args) = &mut j.args { args.remove(field_name); } + + // No accumulation on this path: just clean up the batch rows. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j_id, + ) + .execute(db) + .await?; } else if let Some(arg_name_to_accumulate) = // TODO: Maybe support multiple arguments in future debounce_args_to_accumulate.as_ref().and_then(|v| v.get(0)) { - tracing::debug!( - job_id = %j_id, - job_kind = ?kind, - arg_name = arg_name_to_accumulate, - "Accumulating debounced arguments from batch" - ); - let mut accumulated_arg: Vec> = vec![]; - for str_o in sqlx::query_scalar!( - "WITH ids AS ( - SELECT id as job_id FROM v2_job_debounce_batch WHERE debounce_batch = ( - SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 - ) - ) SELECT args->>$2 FROM ids LEFT JOIN v2_job ON v2_job.id = ids.job_id + // Claim this job's contribution to its debounce batch exactly once. + // Instead of deleting the batch rows, mark them consumed (stamping + // consumed_by = this job). A batch normally has a single survivor that + // sweeps every row; only a narrow push/pull race can leave two survivors + // on one batch. The claim lets the second survivor tell apart: + // - already swept in by the other survivor -> run empty (no duplicate), + // - its own earlier claim on re-pull -> keep its accumulated args, + // - never batched (CE / workers behind v2) -> keep its own args. + // Consumed rows are GC'd by the monitor. + // Claim + accumulate + persist atomically: a crash between stamping the + // batch rows consumed_by=self and persisting the merged args would + // otherwise let a zombie re-pull see its own prior claim and keep only + // its own args (dropping the siblings it had claimed). One transaction + // makes the claim and the merged-args write commit together (or neither). + let mut tx = db.begin().await?; + // Emitted AFTER the transaction commits — writing logs via a second pool + // connection while the claim tx + row locks are held risks pool-exhaustion + // stalls under concurrent debounced pulls. + let mut accumulation_log: Option = None; + let claim = sqlx::query!( + "WITH mine AS ( + SELECT debounce_batch, consumed_by FROM v2_job_debounce_batch WHERE id = $1 + ), claimed AS ( + -- Claim the whole batch in ONE update so concurrent same-batch + -- survivors lock rows in identical scan order (no lock-ordering + -- deadlock); each re-evaluates `consumed_at IS NULL` under EvalPlanQual + -- and skips rows the other already took. A claim therefore consumes + -- every still-unclaimed row of the batch atomically. + UPDATE v2_job_debounce_batch SET consumed_at = now(), consumed_by = $1 + WHERE debounce_batch = (SELECT debounce_batch FROM mine) + AND consumed_at IS NULL + RETURNING id + ) + SELECT + EXISTS (SELECT 1 FROM mine) AS \"had_row!\", + (SELECT consumed_by FROM mine) AS prev_consumed_by, + ARRAY(SELECT id FROM claimed) AS \"claimed_ids!\", + EXISTS (SELECT 1 FROM claimed WHERE id = $1) AS \"claimed_self!\" ", j_id, - arg_name_to_accumulate, ) - .fetch_all(db) - .await? - .into_iter() - { - if let Some(s) = str_o.as_ref() { - match serde_json::from_str::>>(s) { - Ok(ref mut vec) => accumulated_arg.append(vec), - Err(_) => { - // Value is not an array — wrap the scalar into a - // single-element array. This supports union types - // like T | T[] where the caller may pass a bare T. - match RawValue::from_string(s.to_string()) { - Ok(raw) => accumulated_arg.push(raw), - Err(e) => { - return Err(error::Error::ArgumentErr(format!("cannot consolidate argument `{arg_name_to_accumulate}`: value is neither a valid list nor a valid JSON value\nUnwrapped Error: {e}"))); + .fetch_one(&mut *tx) + .await?; + + if !claim.had_row { + // Never batched (CE / workers behind v2): keep the job's own args. + tracing::debug!( + job_id = %j_id, + "Debounce: no batch row, keeping original args" + ); + } else if claim.claimed_self { + // We claimed our own row; since a claim takes the whole batch, this also + // swept any not-yet-claimed siblings. Accumulate exactly the rows we own. + let ids = claim.claimed_ids; + + tracing::debug!( + job_id = %j_id, + job_kind = ?kind, + arg_name = arg_name_to_accumulate, + claimed = ids.len(), + "Accumulating debounced arguments from claimed batch rows" + ); + + let mut accumulated_arg: Vec> = vec![]; + for str_o in sqlx::query_scalar!( + "SELECT args->>$2 FROM v2_job WHERE id = ANY($1)", + &ids, + arg_name_to_accumulate, + ) + .fetch_all(&mut *tx) + .await? + .into_iter() + { + if let Some(s) = str_o.as_ref() { + match serde_json::from_str::>>(s) { + Ok(ref mut vec) => accumulated_arg.append(vec), + Err(_) => { + // Value is not an array — wrap the scalar into a + // single-element array. This supports union types + // like T | T[] where the caller may pass a bare T. + match RawValue::from_string(s.to_string()) { + Ok(raw) => accumulated_arg.push(raw), + Err(e) => { + return Err(error::Error::ArgumentErr(format!("cannot consolidate argument `{arg_name_to_accumulate}`: value is neither a valid list nor a valid JSON value\nUnwrapped Error: {e}"))); + } } } } } } - } - tracing::debug!( - job_id = %j_id, - arg_name = arg_name_to_accumulate, - accumulated_count = accumulated_arg.len(), - "Accumulated arguments from debounced jobs in batch" - ); + if !accumulated_arg.is_empty() { + let new_value = to_raw_value(&accumulated_arg); - // If the batch query returned no entries (e.g. CE where - // v2_job_debounce_batch is never populated), keep the - // original value unchanged instead of replacing it with []. - if !accumulated_arg.is_empty() { - let new_value = to_raw_value(&accumulated_arg); + let original_value = j + .args + .as_ref() + .and_then(|a| a.get(arg_name_to_accumulate)) + .map(|v| v.get().to_string()) + .unwrap_or_else(|| "null".to_string()); - let original_value = j - .args - .as_ref() - .and_then(|a| a.get(arg_name_to_accumulate)) - .map(|v| v.get().to_string()) - .unwrap_or_else(|| "null".to_string()); - - append_logs( - &j_id, - &j.workspace_id, - format!( + accumulation_log = Some(format!( "Accumulating debounced argument `{arg_name_to_accumulate}`:\n original: {original_value}\n accumulated: {}\n\n", &new_value - ), - &(db.into()), - ) - .await; + )); + j.args + .get_or_insert(Json(Default::default())) + .as_mut() + .insert(arg_name_to_accumulate.to_owned(), new_value); + + // Persist accumulated args to v2_job so that flow steps + // re-reading from the DB (via get_mini_pulled_job) see them + if let Some(ref args) = j.args { + sqlx::query!( + "UPDATE v2_job SET args = $2 WHERE id = $1", + j_id, + args as &Json>>, + ) + .execute(&mut *tx) + .await?; + } + } + } else if claim.prev_consumed_by == Some(j_id) { + // Our own prior claim seen again on a re-pull (e.g. crash recovery): + // keep the args we already persisted on the first pull. + } else { + // Another survivor already accumulated this job's contribution + // (consumed_by a different job); run as a no-op so its items are not + // reprocessed. + tracing::info!( + job_id = %j_id, + arg_name = arg_name_to_accumulate, + "Debounce: contribution already consumed by a concurrent survivor, running empty" + ); j.args .get_or_insert(Json(Default::default())) .as_mut() - .insert(arg_name_to_accumulate.to_owned(), new_value); - - // Persist accumulated args to v2_job so that flow steps - // re-reading from the DB (via get_mini_pulled_job) see them + .insert( + arg_name_to_accumulate.to_owned(), + to_raw_value(&Vec::>::new()), + ); if let Some(ref args) = j.args { sqlx::query!( "UPDATE v2_job SET args = $2 WHERE id = $1", j_id, args as &Json>>, ) - .execute(db) + .execute(&mut *tx) .await?; } } - } + tx.commit().await?; - // Clean up the debounce batch entries now that the job has been pulled - sqlx::query!( - "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( - SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 - )", - j_id, - ) - .execute(db) - .await?; + if let Some(msg) = accumulation_log { + append_logs(&j_id, &j.workspace_id, msg, &(db.into())).await; + } + } else { + // Debounced but no args to accumulate (plain debounce / dependency job): + // consume the batch by removing this job's rows. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j_id, + ) + .execute(db) + .await?; + } // Handle dependency job debouncing cleanup when a job is pulled for execution if is_djob_to_debounce { @@ -4926,6 +5317,7 @@ async fn push_inner<'c, 'd>( _low_level_priority: Option, concurrency_settings: ConcurrencySettings, debouncing_settings: DebouncingSettings, + retry_settings: RetrySettings, labels: Option>, } let mut preprocessed = None; @@ -4944,6 +5336,7 @@ async fn push_inner<'c, 'd>( _low_level_priority, mut concurrency_settings, debouncing_settings, + retry_settings, labels, } = match job_payload { JobPayload::ScriptHash { @@ -5250,6 +5643,7 @@ async fn push_inner<'c, 'd>( path, hash, flow_version, + language, retry, error_handler_path, error_handler_args, @@ -5263,10 +5657,71 @@ async fn push_inner<'c, 'd>( apply_preprocessor, debouncing_settings, concurrency_settings, - } => { + } => 'ssf: { // Determine if this is a flow or a script let is_flow = flow_version.is_some(); + // Native retry: a bare script wrapped only to gain a retry (no + // skip/error-handler modules) is pushed as a real `Script` job that + // carries the policy in `runnable_settings`. This avoids spawning a + // one-step flow — and its extra job rows, flow_status, and UI + // projection — for the common schedule/pipeline retry case. The flow + // path below is kept only for handler-bearing or flow-wrapping cases. + // + // Gated on the runnable-settings min version: on a mixed-version + // fleet the policy can't be persisted (`insert_rs` would drop it), so + // we fall back to the flow wrapper to preserve retry semantics. + // `retry_if` is always materialized natively and evaluated on the + // failure path (see `eval_retry_if`). On a worker built without the + // `quickjs` feature it cannot be evaluated and fails closed (no retry); + // the flow path is not a fallback, since the flow runtime needs quickjs + // too. + let native_retry = !is_flow + && skip_handler.is_none() + && error_handler_path.is_none() + && hash.is_some() + && language.is_some() + && windmill_common::runnable_settings::min_version_supports_runnable_settings_v0() + .await; + if native_retry { + if apply_preprocessor { + preprocessed = Some(false); + } + // Preserve worker affinity: normal ScriptHash pushes carry the + // script's `dedicated_worker` (it drives the dedicated tag below), + // but the SingleStepFlow payload doesn't — resolve it from the + // script row so a dedicated-worker script keeps its dedicated pool. + let dedicated_worker = if let Some(h) = &hash { + // Read on the non-RLS pool: push_inner is also entered with RLS + // isolation variants under which the script row may be invisible, + // which would mis-resolve dedicated_worker routing. + sqlx::query_scalar::<_, Option>( + "SELECT dedicated_worker FROM script WHERE hash = $1 AND workspace_id = $2", + ) + .bind(h.0) + .bind(workspace_id) + .fetch_optional(db) + .await? + .flatten() + } else { + None + }; + break 'ssf JobPayloadUntagged { + runnable_id: hash.map(|h| h.0), + runnable_path: Some(path), + job_kind: JobKind::Script, + language, + dedicated_worker, + concurrency_settings, + debouncing_settings, + retry_settings: retry.as_ref().map(RetrySettings::from).unwrap_or_default(), + cache_ttl, + cache_ignore_s3_path, + _low_level_priority: priority, + ..Default::default() + }; + } + // Build modules list let mut modules = vec![]; @@ -5903,6 +6358,7 @@ async fn push_inner<'c, 'd>( RunnableSettings { debouncing_settings: debouncing_settings.insert_cached(db).await?, concurrency_settings: concurrency_settings.insert_cached(db).await?, + retry_settings: retry_settings.insert_cached(db).await?, }, db, ) diff --git a/backend/windmill-queue/src/lib.rs b/backend/windmill-queue/src/lib.rs index be0d1590b6..6f689c7026 100644 --- a/backend/windmill-queue/src/lib.rs +++ b/backend/windmill-queue/src/lib.rs @@ -6,6 +6,14 @@ * LICENSE-AGPL for a copy of the license. */ +pub mod asset_dispatch; +#[cfg(feature = "private")] +pub mod cascade_ee; +pub mod cascade_oss; +#[cfg(feature = "private")] +pub use cascade_ee as cascade; +#[cfg(not(feature = "private"))] +pub use cascade_oss as cascade; pub mod jobs; #[cfg(feature = "private")] pub mod jobs_ee; diff --git a/backend/windmill-queue/src/schedule.rs b/backend/windmill-queue/src/schedule.rs index 3f6c587be2..8816eeae25 100644 --- a/backend/windmill-queue/src/schedule.rs +++ b/backend/windmill-queue/src/schedule.rs @@ -255,6 +255,7 @@ pub async fn push_scheduled_job<'c>( path: schedule.script_path.clone(), hash, flow_version, + language: None, args: args.clone(), retry, error_handler_path: None, @@ -352,6 +353,11 @@ pub async fn push_scheduled_job<'c>( .warn_after_seconds_with_sql(1, "get_latest_hash_for_path".to_string()) .await?; + // NB: read on the non-RLS pool (`db`), not `tx`. push_scheduled_job is + // also invoked with an RLS user_db transaction (api-schedule/api-flows), + // under which these lookups would resolve against the caller's row + // visibility rather than the full table. The dual-connection here is + // intentional and required for correctness. let (debouncing_settings, concurrency_settings) = windmill_common::runnable_settings::prefetch_cached_from_handle( runnable_settings_handle, @@ -371,12 +377,20 @@ pub async fn push_scheduled_job<'c>( for (arg_name, arg_value) in args.clone() { static_args.insert(arg_name, arg_value); } - // if retry is set, we wrap the script into a one step flow with a retry on the module + // A retry on a scheduled script is materialized into a native retry + // (see `push`): `Some(language)` opts in. Completion handlers are + // driven from the terminal attempt, and the per-occurrence + // failure/recovery counting queries (apply_schedule_handlers) resolve + // terminal status across the retry chain — so on_failure/on_recovery + // (incl. multi-count/exact) are all handled. A `retry_if` gate is + // evaluated at failure time; on a worker built without quickjs it + // cannot be evaluated and fails closed (no retry). ( JobPayload::SingleStepFlow { path: schedule.script_path.clone(), hash: Some(hash), flow_version: None, + language: Some(language), retry: Some(parsed_retry), error_handler_path: None, error_handler_args: None, @@ -388,8 +402,12 @@ pub async fn push_scheduled_job<'c>( tag_override: schedule.tag.clone(), trigger_path: None, apply_preprocessor: false, - concurrency_settings: ConcurrencySettings::default(), - debouncing_settings: DebouncingSettings::default(), + // Carry the script's concurrency/debounce settings (fetched + // above) into the native retry materialization, so a retrying + // concurrency-limited scheduled script still inserts its + // concurrency_key instead of running unbounded. + concurrency_settings, + debouncing_settings, }, if schedule.tag.as_ref().is_some_and(|x| x != "") { schedule.tag.clone() @@ -497,7 +515,7 @@ pub async fn push_scheduled_job<'c>( if let Some(tag) = tag.as_deref().filter(|t| !t.is_empty()) { check_tag_available_for_workspace_internal( - &db, + db, &schedule.workspace_id, &tag, &email, @@ -600,7 +618,9 @@ pub async fn clear_schedule<'c>( w_id: &str, ) -> Result<()> { tracing::info!("Clearing schedule {}", path); - sqlx::query!( + // Delete the queued jobs (cascading their v2_job_queue-keyed side tables), then route the + // freed ids through delete_jobs so v2_job and its no-longer-cascading side tables go too. + let deleted_ids: Vec = sqlx::query_scalar!( "WITH to_delete AS ( SELECT id FROM v2_job_queue JOIN v2_job j USING (id) @@ -610,15 +630,16 @@ pub async fn clear_schedule<'c>( AND flow_step_id IS NULL AND running = false FOR UPDATE - ), deleted AS ( - DELETE FROM v2_job_queue - WHERE id IN (SELECT id FROM to_delete) - RETURNING id - ) DELETE FROM v2_job WHERE id IN (SELECT id FROM deleted)", + ) + DELETE FROM v2_job_queue + WHERE id IN (SELECT id FROM to_delete) + RETURNING id", path, w_id ) - .execute(&mut **tx) + .fetch_all(&mut **tx) .await?; + + windmill_common::jobs::delete_jobs(&mut **tx, &deleted_ids).await?; Ok(()) } diff --git a/backend/windmill-queue/tests/debounce_test.rs b/backend/windmill-queue/tests/debounce_test.rs index a2c6971c1c..d74f20ff3f 100644 --- a/backend/windmill-queue/tests/debounce_test.rs +++ b/backend/windmill-queue/tests/debounce_test.rs @@ -3364,6 +3364,7 @@ mod debounce { let rs = RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: concurrency_hash, + retry_settings: None, }; let rs_handle = insert_rs(rs, &db).await?; @@ -3741,6 +3742,7 @@ mod debounce { RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: concurrency_hash, + retry_settings: None, }, db, ) @@ -3878,6 +3880,1577 @@ mod debounce { Ok(()) } + /// Helper: push a script job through push-time `maybe_debounce` with the given key. + /// Returns the args JSON it was pushed with. + async fn push_debounced_script( + db: &Pool, + id: Uuid, + items: Vec, + settings: &DebouncingSettings, + rs_handle: Option, + ) -> serde_json::Value { + let args_val = serde_json::json!({ "items": items }); + insert_script_job_with_args(db, id, "test-workspace", "f/test/script", &args_val).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(db) + .await + .unwrap(); + let args_hm: HashMap> = + serde_json::from_value(args_val.clone()).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &push_args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + args_val + } + + /// Regression test for the "running survivor" data-loss bug. + /// + /// When a debounce survivor has already been pulled and is executing (running), it + /// has committed its batch and can no longer accumulate later arrivals. The old + /// behavior superseded the running survivor anyway: it was completed/skipped + /// ("Debounced Running by ...") and deleted from the queue, silently dropping its + /// accumulated work, while the late arrival could not merge into it. + /// + /// Fix: a late arrival that finds the current survivor already running starts a + /// FRESH debounce window. The running survivor is left to finish with its own + /// accumulated batch; the late arrival accumulates only its own batch. No job is + /// killed and no item is dropped or double-run. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J1, J2 share a window; J2 is the survivor with batch {J1, J2}. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + assert!(is_completed(&db, &j1).await, "J1 should be debounced by J2"); + assert!(is_queued(&db, &j2).await, "J2 should be the survivor"); + + // Worker pulls J2 and marks it running: the window where the key still points + // to J2 and its batch is intact, but J2 can no longer accumulate new arrivals. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3 while J2 is running. + let j3 = Uuid::new_v4(); + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + + // The running survivor J2 must NOT be superseded: still queued, not completed. + assert!( + is_queued(&db, &j2).await, + "running survivor J2 must stay in the queue" + ); + assert!( + !is_completed(&db, &j2).await, + "running survivor J2 must not be completed/skipped" + ); + + // J3 must own the debounce key as the head of a FRESH window (no previous job). + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3, "J3 should hold the debounce key"); + assert!( + dk_prev.is_none(), + "J3 should start a fresh window with no previous job (got {dk_prev:?})" + ); + assert_eq!( + dk_times, 0, + "fresh window should reset debounced_times to 0" + ); + + // The running survivor J2 accumulates only its own committed batch: [1, 2]. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &serde_json::json!({"items": [2]}), + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert!( + j2_res.job.is_some(), + "running survivor J2 must still execute (not nulled out)" + ); + assert_accumulated_items(&j2_res, &[1, 2], "items"); + + // The late arrival J3 accumulates only its own batch: [3]. No overlap with J2. + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert!(j3_res.job.is_some(), "J3 must execute"); + assert_accumulated_items(&j3_res, &[3], "items"); + + Ok(()) + } + + /// A second arrival debouncing a NON-running survivor must keep accumulating into + /// the same batch (the normal debounce behavior must be unchanged by the + /// running-survivor guard). + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_queued_survivor_still_accumulates( + db: Pool, + ) -> anyhow::Result<()> { + let key = "queued_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Three arrivals, none running: classic debounce, all accumulate into J3. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + let j3 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + + assert!(is_completed(&db, &j1).await, "J1 debounced"); + assert!(is_completed(&db, &j2).await, "J2 debounced"); + assert!(is_queued(&db, &j3).await, "J3 is the survivor"); + + // Window keeps growing: J3 is the third arrival in the same batch. + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3); + assert_eq!(dk_prev, Some(j2), "previous job should be J2"); + assert_eq!(dk_times, 2, "debounced_times should keep incrementing"); + + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j3_res, &[1, 2, 3], "items"); + + Ok(()) + } + + /// The running-survivor guard must also apply to plain debounce (delay only, no + /// argument accumulation): a running survivor must never be completed/skipped. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_no_accumulation( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_no_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // No debounce_args_to_accumulate. + ..Default::default() + }; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + insert_noop_job(&db, j1, "test-workspace").await; + insert_noop_job(&db, j2, "test-workspace").await; + + let push = |id: Uuid| { + let settings = settings.clone(); + let db = db.clone(); + async move { + let args_hm = empty_args(); + let args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Noop, + id, + &args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + }; + + push(j1).await; + push(j2).await; + assert!(is_completed(&db, &j1).await, "J1 debounced by J2"); + + // J2 starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3. + let j3 = Uuid::new_v4(); + insert_noop_job(&db, j3, "test-workspace").await; + push(j3).await; + + // Running survivor J2 is preserved; J3 takes over a fresh window. + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key) + .await + .expect("debounce key exists"); + assert_eq!(dk_job, j3); + assert!(dk_prev.is_none(), "fresh window: no previous job"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + + Ok(()) + } + + /// Once a survivor has fully been pulled (batch + key consumed by + /// maybe_apply_debouncing) and is running, a later arrival naturally starts a new + /// window. This locks in that the committed-running case stays correct alongside + /// the in-flight-running guard. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_committed_running_survivor_independent( + db: Pool, + ) -> anyhow::Result<()> { + let key = "committed_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // J2 is pulled: accumulate its batch and consume key + batch. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j2_res, &[1, 2], "items"); + assert!( + get_debounce_key(&db, key).await.is_none(), + "key consumed when survivor pulled" + ); + + // J2 now running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Late arrival J3: fresh window, independent batch, J2 untouched. + let j3 = Uuid::new_v4(); + let j3_args = push_debounced_script(&db, j3, vec![3], &settings, rs_handle).await; + assert!(is_queued(&db, &j2).await, "running J2 untouched"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, _, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, j3); + assert_eq!(dk_times, 0); + + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&j3_res, &[3], "items"); + + Ok(()) + } + + /// Flow post-preprocessing debounce must apply the same running-survivor guard: + /// a running flow survivor must not be completed/skipped by a late flow arrival. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_post_preprocessing_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "pp_running_survivor_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + ..Default::default() + }; + let args_hm = empty_args(); + + let flow1 = Uuid::new_v4(); + let flow2 = Uuid::new_v4(); + insert_flow_job(&db, flow1, "test-workspace", "f/test/flow").await; + insert_flow_job(&db, flow2, "test-workspace", "f/test/flow").await; + + let pp = |id: Uuid| { + let settings = settings.clone(); + let db = db.clone(); + let args_hm = args_hm.clone(); + async move { + let args = PushArgs::from(&args_hm); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow".to_string()), + "test-workspace", + id, + &args, + &db, + ) + .await + .unwrap() + } + }; + + pp(flow1).await; + pp(flow2).await; + assert!(is_completed(&db, &flow1).await, "flow1 debounced by flow2"); + + // flow2 (the survivor) starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + flow2 + ) + .execute(&db) + .await?; + + // Late flow3 arrival while flow2 is running. + let flow3 = Uuid::new_v4(); + insert_flow_job(&db, flow3, "test-workspace", "f/test/flow").await; + let sched = pp(flow3).await; + assert!(sched.is_some(), "flow3 should be debounced (fresh window)"); + + // Running flow2 must be preserved; flow3 owns a fresh window. + assert!(is_queued(&db, &flow2).await, "running flow2 stays queued"); + assert!( + !is_completed(&db, &flow2).await, + "running flow2 must not be completed" + ); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, flow3, "flow3 holds the key"); + assert!(dk_prev.is_none(), "fresh window: no previous job"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + + Ok(()) + } + + /// A running survivor resets the debounce window for the late arrival, so an + /// inherited high `debounced_times` cannot push the new arrival over + /// max_total_debounces_amount and force an immediate (un-debounced) run. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_running_survivor_resets_limit_window( + db: Pool, + ) -> anyhow::Result<()> { + let key = "running_limit_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // Limit of 3: J1, J2 stay debounced; a third arrival in the SAME window + // would trip the limit (current_amount + 1 >= 3) and fire immediately. + max_total_debounces_amount: Some(3), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Build up the window close to the limit: J1, J2 (debounced_times = 1 on J2). + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let (_, _, times_before) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(times_before, 1); + + // J2 starts running. + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // J3 arrives. Without the reset it would inherit debounced_times and could trip + // the max-count limit and fire immediately, killing running J2. With the guard + // it starts a fresh window (debounced_times = 0) and is debounced normally. + let j3 = Uuid::new_v4(); + let mut scheduled_for = None; + { + let args_val = serde_json::json!({ "items": [3] }); + insert_script_job_with_args(&db, j3, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j3, + ) + .execute(&db) + .await?; + let args_hm: HashMap> = serde_json::from_value(args_val).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + j3, + &push_args, + &mut tx, + ) + .await?; + tx.commit().await?; + } + + // J3 is debounced (scheduled_for set, not fired immediately) and J2 survives. + assert!( + scheduled_for.is_some(), + "J3 should be debounced, not fired immediately" + ); + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, j3); + assert!(dk_prev.is_none()); + assert_eq!(dk_times, 0, "fresh window resets the limit counter"); + + Ok(()) + } + + /// Concurrency regression: two late arrivals racing AFTER the survivor started + /// running must not both spawn independent windows. The running check reads the + /// post-conflict-lock holder (`debounce_key.job_id`), so the row lock serializes the + /// two upserts: the first observes the running survivor and opens a fresh window; + /// the second observes that fresh-window head (queued, not running) and debounces + /// into it. Exactly one late arrival survives, the other is debounced — never both. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_concurrent_arrivals_after_running_survivor( + db: Pool, + ) -> anyhow::Result<()> { + let key = "concurrent_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J2 is the survivor and starts running. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + sqlx::query!( + "UPDATE v2_job_queue SET running = true, started_at = now() WHERE id = $1", + j2 + ) + .execute(&db) + .await?; + + // Insert the two late arrivals up front, then race only their maybe_debounce calls. + let j3 = Uuid::new_v4(); + let j4 = Uuid::new_v4(); + for (id, items) in [(j3, 3i64), (j4, 4i64)] { + let args_val = serde_json::json!({ "items": [items] }); + insert_script_job_with_args(&db, id, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(&db) + .await?; + } + + let race = |id: Uuid, items: i64| { + let db = db.clone(); + let settings = settings.clone(); + async move { + let args_hm: HashMap> = + serde_json::from_value(serde_json::json!({ "items": [items] })).unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &push_args, + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + }; + tokio::join!(race(j3, 3), race(j4, 4)); + + // The running survivor is untouched. + assert!(is_queued(&db, &j2).await, "running J2 stays queued"); + assert!(!is_completed(&db, &j2).await, "running J2 not completed"); + + // Exactly one late arrival survives; the other is debounced into the same window. + let j3q = is_queued(&db, &j3).await; + let j4q = is_queued(&db, &j4).await; + let j3c = is_completed(&db, &j3).await; + let j4c = is_completed(&db, &j4).await; + assert!( + (j3q && j4c && !j4q && !j3c) || (j4q && j3c && !j3q && !j4c), + "exactly one late arrival must survive and the other be debounced \ + (not two independent windows); got j3 queued={j3q} completed={j3c}, \ + j4 queued={j4q} completed={j4c}" + ); + + // The surviving holder chained the debounced arrival into one window. + let (holder, prev, times) = get_debounce_key(&db, key).await.expect("key exists"); + let (survivor, debounced) = if j3q { (j3, j4) } else { (j4, j3) }; + assert_eq!(holder, survivor, "key points to the surviving late arrival"); + assert_eq!( + prev, + Some(debounced), + "the surviving window debounced the other late arrival" + ); + assert_eq!(times, 1, "single fresh window with one debounce"); + + // Both late arrivals must share a batch: when the survivor is pulled, its + // accumulation must include the debounced arrival's items, not just its own. + let survivor_args = serde_json::json!({ "items": [if j3q { 3 } else { 4 }] }); + let mut survivor_res = make_pulled_job_result( + survivor, + "test-workspace", + "f/test/script", + &survivor_args, + JobKind::Script, + "deno", + rs_handle, + ); + survivor_res.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&survivor_res, &[3, 4], "items"); + + Ok(()) + } + + /// Regression: a push that would chain onto a queued holder must not error if the + /// worker pull path concurrently deletes that holder's debounce_key + /// (`DELETE ... WHERE job_id = ...`, which does NOT take the push advisory lock). + /// The upsert is a single atomic `INSERT ... ON CONFLICT`, so a deleted holder simply + /// yields a fresh window rather than a "no row updated" failure. Races the two and + /// asserts the push always succeeds and leaves a consistent key. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_push_races_key_deletion_by_pull( + db: Pool, + ) -> anyhow::Result<()> { + let key = "push_vs_pull_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // 50 rounds to give the interleaving a chance to land in the read/write window + // that the old split read+UPDATE path would have failed on. + for round in 0..50 { + sqlx::query!("DELETE FROM debounce_key WHERE key = $1", key) + .execute(&db) + .await?; + let holder = Uuid::new_v4(); + push_debounced_script(&db, holder, vec![round], &settings, rs_handle).await; + + let late = Uuid::new_v4(); + let late_args = serde_json::json!({ "items": [round * 1000] }); + insert_script_job_with_args(&db, late, "test-workspace", "f/test/script", &late_args) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + late, + ) + .execute(&db) + .await?; + + // Race: push the late arrival (chains onto `holder`) against the worker pull + // cleanup deleting `holder`'s key. + let push = { + let db = db.clone(); + let settings = settings.clone(); + async move { + let args_hm: HashMap> = + serde_json::from_value(serde_json::json!({ "items": [round * 1000] })) + .unwrap(); + let push_args = PushArgs::from(&args_hm); + let mut scheduled_for = None; + let mut tx = db.begin().await.unwrap(); + let res = windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut scheduled_for, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + late, + &push_args, + &mut tx, + ) + .await; + if res.is_ok() { + tx.commit().await.unwrap(); + } + res + } + }; + let delete = { + let db = db.clone(); + async move { + sqlx::query!("DELETE FROM debounce_key WHERE job_id = $1", holder) + .execute(&db) + .await + } + }; + let (push_res, _) = tokio::join!(push, delete); + assert!( + push_res.is_ok(), + "round {round}: push must not error when the holder key is concurrently deleted: {push_res:?}" + ); + } + + Ok(()) + } + + /// Claim-based exactly-once: if two survivors end up on the same batch (only + /// possible in a narrow push/pull race), the args of each member are accumulated + /// into exactly ONE run. The survivor that claims the batch first accumulates + /// everyone; the second survivor finds its contribution already consumed and runs + /// empty — no item is dropped and none is processed twice. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_batch_consumed_exactly_once(db: Pool) -> anyhow::Result<()> { + let key = "exactly_once_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // J1 superseded, J2 the (first) survivor of batch B = {J1, J2}. + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // Simulate the race outcome: a second survivor J3 ended up on the SAME batch B. + let j3 = Uuid::new_v4(); + let j3_args = serde_json::json!({ "items": [3] }); + insert_script_job_with_args(&db, j3, "test-workspace", "f/test/script", &j3_args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j3, + ) + .execute(&db) + .await?; + sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch) + SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + j3, + j2, + ) + .execute(&db) + .await?; + + // J2 pulled first: claims the whole batch, accumulates everyone's items. + let mut j2_res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + j2_res.maybe_apply_debouncing(&db).await?; + assert!(j2_res.job.is_some(), "J2 runs"); + assert_accumulated_items(&j2_res, &[1, 2, 3], "items"); + + // J3 pulled next: its contribution was already consumed by J2 -> runs empty, + // so [3] is not processed a second time. + let mut j3_res = make_pulled_job_result( + j3, + "test-workspace", + "f/test/script", + &j3_args, + JobKind::Script, + "deno", + rs_handle, + ); + j3_res.maybe_apply_debouncing(&db).await?; + let job = j3_res.job.as_ref().expect("J3 still runs (empty)"); + let items: Vec = + serde_json::from_str(job.job.args.as_ref().unwrap().get("items").unwrap().get())?; + assert!( + items.is_empty(), + "J3's items must be empty (already consumed by J2), got {items:?}" + ); + + Ok(()) + } + + /// A survivor re-pulled (e.g. crash recovery) must NOT mistake its own earlier + /// claim for a sibling's and wipe its accumulated args. consumed_by = self is + /// distinguished from consumed_by = another job. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_repull_keeps_accumulated(db: Pool) -> anyhow::Result<()> { + let key = "repull_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + // First pull: J2 claims its batch and accumulates [1, 2]. + let mut first = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + first.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&first, &[1, 2], "items"); + + // Re-pull with the args persisted by the first pull: J2 sees its OWN prior claim + // (consumed_by = j2), so it keeps the accumulated args rather than running empty. + let persisted = first.job.as_ref().unwrap().job.args.as_ref().unwrap(); + let persisted_json = serde_json::to_value(persisted).unwrap(); + let mut second = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &persisted_json, + JobKind::Script, + "deno", + rs_handle, + ); + second.maybe_apply_debouncing(&db).await?; + assert!(second.job.is_some(), "re-pulled J2 still runs"); + assert_accumulated_items(&second, &[1, 2], "items"); + + Ok(()) + } + + /// Helper: insert a script job and put it on the SAME debounce batch as `of_job` + /// (simulating a chained survivor). Returns its args JSON. + async fn add_survivor_to_batch_of( + db: &Pool, + id: Uuid, + items: Vec, + of_job: Uuid, + rs_handle: Option, + ) -> serde_json::Value { + let args = serde_json::json!({ "items": items }); + insert_script_job_with_args(db, id, "test-workspace", "f/test/script", &args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(db) + .await + .unwrap(); + let inserted = sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch) + SELECT $1, debounce_batch FROM v2_job_debounce_batch WHERE id = $2", + id, + of_job, + ) + .execute(db) + .await + .unwrap(); + // `of_job` must already have a batch row, else this no-ops and the test would + // pass vacuously (the job would end up never-batched, keeping its own args). + assert_eq!( + inserted.rows_affected(), + 1, + "add_survivor_to_batch_of: {of_job} has no batch row to share" + ); + args + } + + /// Helper: read the accumulated `items` of a pulled job as a sorted Vec. + fn items_of(result: &windmill_queue::PulledJobResult) -> Vec { + let job = result.job.as_ref().expect("job present"); + let raw = job.job.args.as_ref().unwrap().get("items").unwrap(); + let mut v: Vec = serde_json::from_str::>(raw.get()) + .unwrap() + .iter() + .map(|x| x.as_i64().unwrap()) + .collect(); + v.sort(); + v + } + + /// Edge: an accumulate-debounced job that was NEVER batched (CE / workers behind v2: + /// no v2_job_debounce_batch row) must keep its own args, not be emptied. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_never_batched_keeps_own_args(db: Pool) -> anyhow::Result<()> { + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("never_batched_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Insert a job with the debounce handle but DO NOT push through maybe_debounce, + // so it has no batch row at all. + let j = Uuid::new_v4(); + let args = serde_json::json!({ "items": [7, 8] }); + insert_script_job_with_args(&db, j, "test-workspace", "f/test/script", &args).await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + j, + ) + .execute(&db) + .await?; + + let mut res = make_pulled_job_result( + j, + "test-workspace", + "f/test/script", + &args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + assert!(res.job.is_some(), "never-batched job still runs"); + assert_accumulated_items(&res, &[7, 8], "items"); + Ok(()) + } + + /// Edge: two survivors of one batch pulled CONCURRENTLY. The atomic claim must + /// partition the batch disjointly — the union of what they each accumulate is the + /// full set, with NO item processed by both. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_concurrent_claim_disjoint(db: Pool) -> anyhow::Result<()> { + let key = "concurrent_claim_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; // superseded + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; // survivor 1 + let j3 = Uuid::new_v4(); + let j3_args = add_survivor_to_batch_of(&db, j3, vec![3], j2, rs_handle).await; // survivor 2 + + let pull = |id: Uuid, args: serde_json::Value| { + let db = db.clone(); + async move { + let mut res = make_pulled_job_result( + id, + "test-workspace", + "f/test/script", + &args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await.unwrap(); + res + } + }; + let (r2, r3) = tokio::join!(pull(j2, j2_args), pull(j3, j3_args)); + + let mut union = items_of(&r2); + union.extend(items_of(&r3)); + union.sort(); + assert_eq!( + union, + vec![1, 2, 3], + "every item accumulated exactly once across the two concurrent survivors" + ); + // disjoint: no overlap between the two survivors' items + let i2 = items_of(&r2); + let i3 = items_of(&r3); + assert!( + !i2.iter().any(|x| i3.contains(x)), + "no item processed by both survivors; got j2={i2:?} j3={i3:?}" + ); + Ok(()) + } + + /// Edge: three survivors on one batch pulled in sequence. The first claims the whole + /// batch; the rest find themselves consumed and run empty. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_three_survivors_first_takes_all( + db: Pool, + ) -> anyhow::Result<()> { + let key = "three_survivors_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + let j3 = Uuid::new_v4(); + let j3_args = add_survivor_to_batch_of(&db, j3, vec![3], j2, rs_handle).await; + let j4 = Uuid::new_v4(); + let j4_args = add_survivor_to_batch_of(&db, j4, vec![4], j2, rs_handle).await; + + let mk = |id, args: &serde_json::Value| { + make_pulled_job_result( + id, + "test-workspace", + "f/test/script", + args, + JobKind::Script, + "deno", + rs_handle, + ) + }; + let (mut r2, mut r3, mut r4) = (mk(j2, &j2_args), mk(j3, &j3_args), mk(j4, &j4_args)); + r2.maybe_apply_debouncing(&db).await?; + r3.maybe_apply_debouncing(&db).await?; + r4.maybe_apply_debouncing(&db).await?; + + assert_eq!( + items_of(&r2), + vec![1, 2, 3, 4], + "first survivor takes the whole batch" + ); + assert!(items_of(&r3).is_empty(), "second survivor runs empty"); + assert!(items_of(&r4).is_empty(), "third survivor runs empty"); + Ok(()) + } + + /// Edge: plain debounce (no accumulate args) must HARD-DELETE its batch rows on pull + /// (not leave consumed rows lingering), so the non-accumulate path doesn't leak. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_non_accumulate_deletes_batch(db: Pool) -> anyhow::Result<()> { + let key = "non_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + // no debounce_args_to_accumulate + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + // push_debounced_script sends {items:[...]} but with no accumulate arg configured, + // the batch is created yet never accumulated. + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + let batch_rows_before: i64 = + sqlx::query_scalar!("SELECT count(*) as \"c!\" FROM v2_job_debounce_batch") + .fetch_one(&db) + .await?; + assert!(batch_rows_before >= 2, "batch rows exist before pull"); + + let mut res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + + let remaining: i64 = + sqlx::query_scalar!("SELECT count(*) as \"c!\" FROM v2_job_debounce_batch") + .fetch_one(&db) + .await?; + assert_eq!( + remaining, 0, + "non-accumulate pull hard-deletes the batch rows" + ); + Ok(()) + } + + /// Edge: GC sweep deletes consumed rows past the grace period but keeps recently + /// consumed and not-yet-consumed rows. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_gc_consumed_batches(db: Pool) -> anyhow::Result<()> { + let old = Uuid::new_v4(); + let recent = Uuid::new_v4(); + let unconsumed = Uuid::new_v4(); + // A consumed-long-ago sibling that is STILL QUEUED (e.g. stuck behind a + // concurrency limit): its marker must survive GC so its eventual pull still sees + // "already consumed" and runs empty (no duplicate). + let queued_old = Uuid::new_v4(); + insert_script_job_with_args( + &db, + queued_old, + "test-workspace", + "f/test/script", + &serde_json::json!({ "items": [9] }), + ) + .await; + sqlx::query!( + "INSERT INTO v2_job_debounce_batch (id, debounce_batch, consumed_at) VALUES + ($1, nextval('debounce_batch_seq'), now() - interval '20 minutes'), + ($2, nextval('debounce_batch_seq'), now() - interval '1 minute'), + ($3, nextval('debounce_batch_seq'), NULL), + ($4, nextval('debounce_batch_seq'), now() - interval '20 minutes')", + old, + recent, + unconsumed, + queued_old, + ) + .execute(&db) + .await?; + + // Mirror the monitor GC sweep (age floor + only-if-no-longer-queued). + let deleted = sqlx::query_scalar!( + "WITH del AS ( + DELETE FROM v2_job_debounce_batch + WHERE consumed_at IS NOT NULL + AND consumed_at < now() - interval '10 minutes' + AND id NOT IN (SELECT id FROM v2_job_queue) + RETURNING 1 + ) SELECT count(*) as \"c!\" FROM del" + ) + .fetch_one(&db) + .await?; + assert_eq!( + deleted, 1, + "only the old, no-longer-queued consumed row is GC'd" + ); + + let exists = |id: Uuid, db: Pool| async move { + sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM v2_job_debounce_batch WHERE id = $1) as \"e!\"", + id + ) + .fetch_one(&db) + .await + .unwrap() + }; + assert!(!exists(old, db.clone()).await, "old consumed row gone"); + assert!( + exists(recent, db.clone()).await, + "recently consumed row kept" + ); + assert!(exists(unconsumed, db.clone()).await, "unconsumed row kept"); + assert!( + exists(queued_old, db.clone()).await, + "old consumed row whose job is still queued must be kept" + ); + Ok(()) + } + + /// Helper: mark a queued job as running (simulates a survivor that the + /// concurrency limiter has just started executing). + async fn set_running(db: &Pool, job_id: &Uuid) { + sqlx::query!( + "UPDATE v2_job_queue SET running = true WHERE id = $1", + job_id + ) + .execute(db) + .await + .expect("set running"); + } + + /// Regression (ref #9781): post-preprocessing debounce with + /// `debounce_args_to_accumulate` under a concurrency limit. A survivor accumulates + /// its own element and starts running; a later same-key message must start a NEW + /// batch (survive) rather than be folded into the running survivor and silently + /// dropped. Exercises the full EE path via `jobs_ee::maybe_debounce_post_preprocessing`. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_post_preprocessing_debounce_into_running_survivor_loses_message( + db: Pool, + ) -> anyhow::Result<()> { + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("ported_running_survivor_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // --- Wave 1: a single message becomes the survivor and starts running. --- + let survivor = Uuid::new_v4(); + let survivor_args = serde_json::json!({ "items": [1] }); + insert_flow_job_with_preprocessor( + &db, + survivor, + "test-workspace", + "f/test/flow_run", + true, + 0, + ) + .await; + sqlx::query!( + "UPDATE v2_job SET args = $2 WHERE id = $1", + survivor, + survivor_args + ) + .execute(&db) + .await?; + + let survivor_args_hm: HashMap> = + serde_json::from_value(survivor_args.clone()).unwrap(); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow_run".to_string()), + "test-workspace", + survivor, + &PushArgs::from(&survivor_args_hm), + &db, + ) + .await?; + assert!(is_queued(&db, &survivor).await, "survivor should be queued"); + + // Worker pulls the survivor: accumulate its own [1], consume the batch, run. + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + survivor, + ) + .execute(&db) + .await?; + let mut pulled = make_pulled_job_result( + survivor, + "test-workspace", + "f/test/flow_run", + &survivor_args, + JobKind::Flow, + "flow", + rs_handle, + ); + pulled.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&pulled, &[1], "items"); + set_running(&db, &survivor).await; // survivor is now RUNNING + + // --- Wave 2: a new message arrives while the survivor is running. --- + let late = Uuid::new_v4(); + let late_args = serde_json::json!({ "items": [2] }); + insert_flow_job_with_preprocessor(&db, late, "test-workspace", "f/test/flow_run", true, 0) + .await; + sqlx::query!("UPDATE v2_job SET args = $2 WHERE id = $1", late, late_args) + .execute(&db) + .await?; + + let late_args_hm: HashMap> = + serde_json::from_value(late_args.clone()).unwrap(); + windmill_queue::jobs_ee::maybe_debounce_post_preprocessing( + &settings, + &Some("f/test/flow_run".to_string()), + "test-workspace", + late, + &PushArgs::from(&late_args_hm), + &db, + ) + .await?; + + // The late message must survive (new batch): still queued, and the debounce_key + // moved off the already-running survivor. + let late_survived = is_queued(&db, &late).await && !is_completed(&db, &late).await; + let dk = get_debounce_key(&db, "ported_running_survivor_key").await; + let key_moved_off_running_survivor = + dk.map(|(job_id, _, _)| job_id != survivor).unwrap_or(true); + assert!( + late_survived && key_moved_off_running_survivor, + "message arriving while the survivor is running must start a new batch \ + (survive), not be folded into the running survivor and dropped. \ + late_survived={late_survived}, key_moved_off={key_moved_off_running_survivor}" + ); + Ok(()) + } + + /// Flow-node debounce (third EE entry point, `jobs_ee::maybe_debounce_flow_node`): + /// a running survivor child must not be superseded by a later same-key child; the + /// late child starts a fresh window and the running child is left to finish. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_flow_node_debounce_running_survivor_not_superseded( + db: Pool, + ) -> anyhow::Result<()> { + let key = "flow_node_running_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + ..Default::default() + }; + let args_hm = empty_args(); + + let flow1 = Uuid::new_v4(); + let child1 = Uuid::new_v4(); + insert_flow_job(&db, flow1, "test-workspace", "f/test/my_flow").await; + insert_child_job_with_parent(&db, child1, flow1, "test-workspace").await; + + // child1 becomes the survivor, then starts running. + { + let args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce_flow_node( + &settings, + child1, + flow1, + "f/test/my_flow", + "step_a", + "test-workspace", + &args, + &mut tx, + &db, + ) + .await?; + tx.commit().await?; + } + set_running(&db, &child1).await; + + // child2 (later same-key child) arrives while child1 is running. + let flow2 = Uuid::new_v4(); + let child2 = Uuid::new_v4(); + insert_flow_job(&db, flow2, "test-workspace", "f/test/my_flow").await; + insert_child_job_with_parent(&db, child2, flow2, "test-workspace").await; + { + let args = PushArgs::from(&args_hm); + let mut tx = db.begin().await?; + windmill_queue::jobs_ee::maybe_debounce_flow_node( + &settings, + child2, + flow2, + "f/test/my_flow", + "step_a", + "test-workspace", + &args, + &mut tx, + &db, + ) + .await?; + tx.commit().await?; + } + + // The running child1 (and its parent flow1) must be left alone; child2 owns a + // fresh window. + assert!(is_queued(&db, &child1).await, "running child1 stays queued"); + assert!( + !is_completed(&db, &child1).await, + "running child1 not completed" + ); + assert!( + !is_completed(&db, &flow1).await, + "flow1 of running child not completed" + ); + let (dk_job, dk_prev, dk_times) = get_debounce_key(&db, key).await.expect("key exists"); + assert_eq!(dk_job, child2, "child2 holds the key"); + assert!(dk_prev.is_none(), "fresh window: no previous child"); + assert_eq!(dk_times, 0, "fresh window resets debounced_times"); + Ok(()) + } + + /// Edge: accumulate values that are bare scalars (not arrays) — the `T | T[]` union + /// case. Each scalar contribution must be wrapped into a single-element list so the + /// survivor accumulates them all. Exercises the non-array fallback in the claim path. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_accumulate_scalar_values(db: Pool) -> anyhow::Result<()> { + let key = "scalar_accum_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + // Push two jobs whose `items` is a BARE SCALAR, not an array. + let push_scalar = |id: Uuid, v: i64, db: Pool, settings: DebouncingSettings| async move { + let args_val = serde_json::json!({ "items": v }); + insert_script_job_with_args(&db, id, "test-workspace", "f/test/script", &args_val) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, + id, + ) + .execute(&db) + .await + .unwrap(); + let hm: HashMap> = serde_json::from_value(args_val).unwrap(); + let mut sf = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, + &mut sf, + &Some("f/test/script".to_string()), + "test-workspace", + JobKind::Script, + id, + &PushArgs::from(&hm), + &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + }; + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_scalar(j1, 1, db.clone(), settings.clone()).await; + push_scalar(j2, 2, db.clone(), settings.clone()).await; + + let mut res = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &serde_json::json!({ "items": 2 }), + JobKind::Script, + "deno", + rs_handle, + ); + res.maybe_apply_debouncing(&db).await?; + // Both bare scalars are wrapped and accumulated into a list. + assert_accumulated_items(&res, &[1, 2], "items"); + Ok(()) + } + + /// Edge: GC reclaiming a survivor's consumed row before a re-pull must NOT lose data — + /// the re-pull finds no row (had_row=false) and keeps its already-persisted accumulated + /// args, rather than running empty. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + async fn test_debounce_repull_after_gc_keeps_accumulated( + db: Pool, + ) -> anyhow::Result<()> { + let key = "repull_gc_key"; + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some(key.to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let j1 = Uuid::new_v4(); + let j2 = Uuid::new_v4(); + push_debounced_script(&db, j1, vec![1], &settings, rs_handle).await; + let j2_args = push_debounced_script(&db, j2, vec![2], &settings, rs_handle).await; + + let mut first = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &j2_args, + JobKind::Script, + "deno", + rs_handle, + ); + first.maybe_apply_debouncing(&db).await?; + assert_accumulated_items(&first, &[1, 2], "items"); + + // Simulate the GC sweep reclaiming the (now consumed) batch rows for this batch. + sqlx::query!( + "DELETE FROM v2_job_debounce_batch WHERE debounce_batch = ( + SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = $1 + )", + j2, + ) + .execute(&db) + .await + .ok(); + // (and any that were already consumed elsewhere) + sqlx::query!("DELETE FROM v2_job_debounce_batch WHERE consumed_at IS NOT NULL") + .execute(&db) + .await?; + + // Re-pull with the args persisted on the first pull: no batch row now, so it must + // fall back to its own (already-accumulated) args — no loss. + let persisted = + serde_json::to_value(first.job.as_ref().unwrap().job.args.as_ref().unwrap()).unwrap(); + let mut second = make_pulled_job_result( + j2, + "test-workspace", + "f/test/script", + &persisted, + JobKind::Script, + "deno", + rs_handle, + ); + second.maybe_apply_debouncing(&db).await?; + assert!(second.job.is_some(), "re-pulled survivor still runs"); + assert_accumulated_items(&second, &[1, 2], "items"); + Ok(()) + } + + /// Throughput benchmark for the FULL debounce path (EE push + + /// `jobs_ee::maybe_debounce`/`complete_debounced_job`/`upsert_debounce_key`, then OSS + /// `maybe_apply_debouncing` claim/accumulate/consume). #[ignore]d — run manually: + /// cargo test -p windmill-queue --test debounce_test --features private,enterprise \ + /// bench_debounce_full_path -- --ignored --nocapture --test-threads=1 + /// Each cycle = a burst of BURST pushes to one key (debounced) + one survivor pull + /// (accumulate+consume), run across CONCURRENCY tasks. Compare before/after by running + /// it on each code revision. + #[sqlx::test(migrations = "../migrations", fixtures("base"))] + #[ignore] + async fn bench_debounce_full_path(db: Pool) -> anyhow::Result<()> { + const CONCURRENCY: usize = 4; + const CYCLES_PER_TASK: usize = 300; + const BURST: usize = 3; + + let settings = DebouncingSettings { + debounce_delay_s: Some(5), + debounce_key: Some("bench_key".to_string()), + debounce_args_to_accumulate: Some(vec!["items".to_string()]), + ..Default::default() + }; + let rs_handle = setup_debouncing_settings(&db, &settings).await; + + let start = std::time::Instant::now(); + let tasks: Vec<_> = (0..CONCURRENCY) + .map(|t| { + let db = db.clone(); + let settings = DebouncingSettings { + // distinct key space per task so bursts collapse independently + debounce_key: Some(format!("bench_key_{t}")), + ..settings.clone() + }; + tokio::spawn(async move { + for c in 0..CYCLES_PER_TASK { + // Fresh key per cycle so each cycle is one full collapse+pull. + let key = format!("bench_{t}_{c}"); + let settings = DebouncingSettings { + debounce_key: Some(key.clone()), + ..settings.clone() + }; + let mut survivor = Uuid::new_v4(); + for b in 0..BURST { + let id = Uuid::new_v4(); + survivor = id; + let args_val = serde_json::json!({ "items": [b as i64] }); + insert_script_job_with_args( + &db, id, "test-workspace", "f/test/script", &args_val, + ) + .await; + sqlx::query!( + "UPDATE v2_job_queue SET runnable_settings_handle = $1 WHERE id = $2", + rs_handle, id, + ) + .execute(&db) + .await + .unwrap(); + let hm: HashMap> = + serde_json::from_value(args_val).unwrap(); + let mut sf = None; + let mut tx = db.begin().await.unwrap(); + windmill_queue::jobs_ee::maybe_debounce( + &settings, &mut sf, &Some("f/test/script".to_string()), + "test-workspace", JobKind::Script, id, &PushArgs::from(&hm), &mut tx, + ) + .await + .unwrap(); + tx.commit().await.unwrap(); + } + // Survivor pull: claim + accumulate + consume. + let mut res = make_pulled_job_result( + survivor, "test-workspace", "f/test/script", + &serde_json::json!({ "items": [] }), + JobKind::Script, "deno", rs_handle, + ); + res.maybe_apply_debouncing(&db).await.unwrap(); + } + }) + }) + .collect(); + for t in tasks { + t.await.unwrap(); + } + let elapsed = start.elapsed(); + let cycles = CONCURRENCY * CYCLES_PER_TASK; + let jobs = cycles * BURST; + eprintln!( + "BENCH full debounce path: {cycles} cycles ({jobs} pushed jobs + {cycles} pulls) in {:.2?} | {:.0} pushes/s | {:.0} pulls/s", + elapsed, + jobs as f64 / elapsed.as_secs_f64(), + cycles as f64 / elapsed.as_secs_f64(), + ); + Ok(()) + } + /// Test: Push-time (script) debounce with max_total_debounces_amount=2. /// 5 calls, each sending {x: [i]}. Expected: /// Call 1: debounced (scheduled_for set) diff --git a/backend/windmill-queue/tests/native_retry_test.rs b/backend/windmill-queue/tests/native_retry_test.rs new file mode 100644 index 0000000000..fb8046149b --- /dev/null +++ b/backend/windmill-queue/tests/native_retry_test.rs @@ -0,0 +1,376 @@ +// Integration tests for native single-script retry (no one-step-flow wrapping). +// +// These use the *runtime* sqlx API (`sqlx::query`/`query_as`, not the `!` macros) +// like schedule_push.rs, so they need no `.sqlx` offline cache entry. +mod native_retry { + use sqlx::{Pool, Postgres}; + use uuid::Uuid; + + use windmill_common::flows::{ConstantDelay, Retry}; + use windmill_common::jobs::{JobKind, JobTriggerKind}; + use windmill_common::runnable_settings::{ + from_handle, insert_rs, ConcurrencySettings, RetrySettings, RunnableSettings, + RunnableSettingsTrait, + }; + use windmill_common::scripts::{ScriptHash, ScriptLang}; + use windmill_common::users::username_to_permissioned_as; + use windmill_queue::jobs::{maybe_enqueue_native_script_retry, MiniCompletedJob}; + + const WS: &str = "test-workspace"; + const SCHED: &str = "f/system/test_schedule"; + const SCRIPT: &str = "f/system/test_script"; + + fn mini(id: Uuid, parent_job: Option, handle: Option) -> MiniCompletedJob { + MiniCompletedJob { + id, + workspace_id: WS.to_string(), + runnable_id: Some(ScriptHash(100001)), + scheduled_for: chrono::Utc::now(), + parent_job, + flow_innermost_root_job: None, + runnable_path: Some(SCRIPT.to_string()), + kind: JobKind::Script, + started_at: Some(chrono::Utc::now()), + permissioned_as: username_to_permissioned_as("test-user"), + created_by: "test-user".to_string(), + script_lang: Some(ScriptLang::Deno), + permissioned_as_email: "test@windmill.dev".to_string(), + flow_step_id: None, + trigger_kind: Some(JobTriggerKind::Schedule), + trigger: Some(SCHED.to_string()), + priority: None, + concurrent_limit: None, + tag: "deno".to_string(), + cache_ttl: None, + cache_ignore_s3_path: None, + runnable_settings_handle: handle, + } + } + + async fn count_retries(db: &Pool, root: Uuid) -> i64 { + sqlx::query_scalar::<_, i64>("SELECT count(*) FROM v2_job WHERE parent_job = $1") + .bind(root) + .fetch_one(db) + .await + .unwrap() + } + + /// The queued retry with the given attempt number, if any: (id, kind, parent_job, backoff_s, handle). + async fn retry_by_attempt( + db: &Pool, + root: Uuid, + attempt: i64, + ) -> Option<(Uuid, String, Option, f64, Option)> { + sqlx::query_as::<_, (Uuid, String, Option, Option, Option)>( + "SELECT j.id, j.kind::text, j.parent_job, + EXTRACT(EPOCH FROM (q.scheduled_for - now()))::float8, + q.runnable_settings_handle + FROM v2_job j + JOIN v2_job_queue q ON q.id = j.id + JOIN native_retry_attempt nra ON nra.job_id = j.id + WHERE j.parent_job = $1 AND nra.attempt = $2", + ) + .bind(root) + .bind(attempt) + .fetch_optional(db) + .await + .unwrap() + .map(|(id, kind, parent, backoff, handle)| { + (id, kind, parent, backoff.unwrap_or(0.0), handle) + }) + } + + fn no_result() -> Option> { + None + } + + // attempt0 -> retry1 -> retry2 -> exhausted, with crash-replay idempotency. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn chains_attempts_and_is_idempotent(db: Pool) -> anyhow::Result<()> { + // Policy: 2 constant attempts, 1s apart. + let retry = Retry { + constant: ConstantDelay { attempts: 2, seconds: 1 }, + exponential: Default::default(), + retry_if: None, + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: None, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + assert!(handle.is_some(), "a retry policy must produce a handle"); + + // attempt 0 (the schedule root); maybe_enqueue tolerates the absent queue row (counter -> 0). + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + + // First failure -> retry 1. + assert!( + maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?, + "first failure enqueues a retry" + ); + let (r1_id, kind, parent, backoff, r1_handle) = retry_by_attempt(&db, root_id, 1) + .await + .expect("retry attempt 1 exists"); + assert_eq!( + kind, "script", + "retry is a native Script, not a singlestepflow" + ); + assert_eq!(parent, Some(root_id), "retry links to the chain root"); + assert!( + r1_handle.is_some(), + "retry carries the policy for further chaining" + ); + assert!( + backoff > 0.0 && backoff <= 3.0, + "constant 1s backoff, got {backoff}s" + ); + + // Crash-replay: the SAME completion again must not double-enqueue, and must + // still report pending (so schedule handlers stay deferred). Regression for P1. + assert!( + maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?, + "replay still reports the retry as pending" + ); + assert_eq!( + count_retries(&db, root_id).await, + 1, + "no double retry on crash replay" + ); + + // retry 1 fails -> retry 2 (still within attempts = 2). + let r1 = mini(r1_id, Some(root_id), r1_handle); + assert!(maybe_enqueue_native_script_retry(&db, &r1, &None, &no_result).await?); + assert_eq!(count_retries(&db, root_id).await, 2); + + // retry 2 fails -> attempts exhausted, no retry 3. + let (r2_id, _, _, _, r2_handle) = retry_by_attempt(&db, root_id, 2) + .await + .expect("retry attempt 2 exists"); + let r2 = mini(r2_id, Some(root_id), r2_handle); + assert!( + !maybe_enqueue_native_script_retry(&db, &r2, &None, &no_result).await?, + "exhausted policy does not enqueue" + ); + assert_eq!( + count_retries(&db, root_id).await, + 2, + "no retry past max attempts" + ); + Ok(()) + } + + // Cancellation always wins over a pending retry. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn canceled_job_does_not_retry(db: Pool) -> anyhow::Result<()> { + let retry = Retry { + constant: ConstantDelay { attempts: 3, seconds: 1 }, + exponential: Default::default(), + retry_if: None, + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: None, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + let canceled = Some(windmill_queue::jobs::CanceledBy { + username: Some("test-user".to_string()), + reason: Some("manual".to_string()), + }); + assert!(!maybe_enqueue_native_script_retry(&db, &root, &canceled, &no_result).await?); + assert_eq!( + count_retries(&db, root_id).await, + 0, + "canceled job must not retry" + ); + Ok(()) + } + + // A concurrency-limited script that also retries must carry its concurrency + // settings into each retry, otherwise the retry runs unbounded. Regression: P1. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn retry_preserves_concurrency_settings(db: Pool) -> anyhow::Result<()> { + let retry = Retry { + constant: ConstantDelay { attempts: 1, seconds: 0 }, + exponential: Default::default(), + retry_if: None, + }; + let concurrency = ConcurrencySettings { + concurrency_key: Some("f/system/test_script".to_string()), + concurrent_limit: Some(1), + concurrency_time_window_s: Some(60), + }; + let handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: concurrency.insert_cached(&db).await?, + retry_settings: RetrySettings::from(&retry).insert_cached(&db).await?, + }, + &db, + ) + .await?; + + let root_id = Uuid::new_v4(); + let root = mini(root_id, None, handle); + assert!(maybe_enqueue_native_script_retry(&db, &root, &None, &no_result).await?); + + let (_id, _kind, _parent, _backoff, r1_handle) = retry_by_attempt(&db, root_id, 1) + .await + .expect("retry attempt 1 exists"); + // The retry's own handle must resolve to the same concurrency settings, not + // just the retry policy — otherwise it would run with no concurrency_key. + let rs = from_handle(r1_handle, &db).await?; + let resolved = ConcurrencySettings::get( + rs.concurrency_settings + .expect("retry must carry concurrency settings forward"), + &db, + ) + .await?; + assert_eq!( + resolved.concurrency_key.as_deref(), + Some("f/system/test_script") + ); + assert_eq!(resolved.concurrent_limit, Some(1)); + assert!( + rs.retry_settings.is_some(), + "retry policy is still carried for further chaining" + ); + Ok(()) + } + + // ------------------------------------------------------------------ + // Per-occurrence terminal status (drives on_failure_times / on_recovery). + // Mirrors the exact query in windmill-ee-private jobs_ee::apply_schedule_handlers. + // ------------------------------------------------------------------ + // `is_retry` marks the seeded job as a native retry attempt (the explicit + // native_retry_attempt marker), the same signal `apply_schedule_handlers` + // keys off. Handlers / WAC inline children are seeded without it. + async fn seed_job( + db: &Pool, + id: Uuid, + parent: Option, + is_retry: bool, + status: &str, + ) { + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, kind, runnable_path, trigger_kind, trigger, parent_job) + VALUES ($1, $2, 'script', $3, 'schedule', $4, $5)", + ) + .bind(id) + .bind(WS) + .bind(SCRIPT) + .bind(SCHED) + .bind(parent) + .execute(db) + .await + .unwrap(); + sqlx::query( + "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, status) + VALUES ($1, $2, 1, $3::job_status)", + ) + .bind(id) + .bind(WS) + .bind(status) + .execute(db) + .await + .unwrap(); + if is_retry { + sqlx::query("INSERT INTO native_retry_attempt (job_id, attempt) VALUES ($1, 1)") + .bind(id) + .execute(db) + .await + .unwrap(); + } + } + + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn per_occurrence_status_counts_recovered_as_success(db: Pool) { + // A native retry attempt is marked (native_retry_attempt); handler and WAC + // inline children are parented Script children WITHOUT the marker. + // a: root fails, a marked retry succeeds -> RECOVERED -> success + // b: root fails, retry also fails -> failure + // c: root succeeds directly -> success + // e: root fails, only its on_failure HANDLER (unmarked) succeeds + // -> failure: handler child must NOT count as a recovery + // f: root fails, only a WAC inline child (unmarked) succeeds + // -> failure: WAC inline child must NOT count as a recovery + // d: the current occurrence (excluded by `j.id != $4`) + let (a, b, c, e, f, d) = ( + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + Uuid::new_v4(), + ); + seed_job(&db, a, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(a), true, "success").await; // a's retry attempt (marked) + seed_job(&db, b, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(b), true, "failure").await; // b's failed retry + seed_job(&db, c, None, false, "success").await; + seed_job(&db, e, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(e), false, "success").await; // e's handler child (unmarked) + seed_job(&db, f, None, false, "failure").await; + seed_job(&db, Uuid::new_v4(), Some(f), false, "success").await; // f's WAC inline child (unmarked) + seed_job(&db, d, None, false, "failure").await; // current occurrence + + // Exact expression from jobs_ee::apply_schedule_handlers: the EXISTS counts + // only marked native retry children, so neither handler nor WAC inline + // children count as a recovery. + let rows = sqlx::query_as::<_, (Uuid, bool)>( + "SELECT j.id, (status = 'success' OR EXISTS ( + SELECT 1 FROM native_retry_attempt nra + JOIN v2_job jc ON jc.id = nra.job_id + JOIN v2_job_completed cc ON cc.id = nra.job_id + WHERE jc.parent_job = j.id AND cc.status = 'success' + )) + FROM v2_job j JOIN v2_job_completed USING (id) + WHERE j.workspace_id = $1 AND trigger_kind = 'schedule' AND trigger = $2 + AND parent_job IS NULL AND runnable_path = $3 AND j.id != $4 + ORDER BY created_at DESC", + ) + .bind(WS) + .bind(SCHED) + .bind(SCRIPT) + .bind(d) + .fetch_all(&db) + .await + .unwrap(); + + let status: std::collections::HashMap = rows.into_iter().collect(); + // Retries/handlers/WAC children (parent_job set) are NOT occurrences, and the + // current one is excluded: exactly the five roots a, b, c, e, f remain. + assert_eq!( + status.len(), + 5, + "child jobs excluded from occurrence counting; current excluded" + ); + assert_eq!( + status[&a], true, + "recovered occurrence (same-runnable retry succeeded) = success" + ); + assert_eq!( + status[&b], false, + "all-attempts-failed occurrence = failure" + ); + assert_eq!(status[&c], true, "direct success"); + assert_eq!( + status[&e], false, + "on_failure handler success must NOT count as a recovery" + ); + assert_eq!( + status[&f], false, + "WAC inline child success must NOT count as a recovery" + ); + } +} diff --git a/backend/windmill-queue/tests/schedule_push.rs b/backend/windmill-queue/tests/schedule_push.rs index c0e09003f0..70a53763e0 100644 --- a/backend/windmill-queue/tests/schedule_push.rs +++ b/backend/windmill-queue/tests/schedule_push.rs @@ -3,6 +3,9 @@ mod schedule_push { use sqlx::{Pool, Postgres}; use windmill_common::db::Authed; use windmill_common::jobs::{JobKind, JobTriggerKind}; + use windmill_common::runnable_settings::{ + from_handle, insert_rs, ConcurrencySettings, RunnableSettings, RunnableSettingsTrait, + }; use windmill_common::schedule::Schedule; use windmill_common::scripts::ScriptHash; use windmill_common::users::username_to_permissioned_as; @@ -233,13 +236,78 @@ mod schedule_push { assert_eq!(count_queued_jobs(&db).await, 1); - // When retry is set, the job kind is singlescriptflow (SingleStepFlow wraps it) - let kind = sqlx::query_scalar::<_, String>( - "SELECT kind::text FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + // Native retry: a scheduled script with a retry policy is pushed as a plain + // Script carrying the policy via runnable_settings_handle — no SingleStepFlow. + let (kind, handle) = sqlx::query_as::<_, (String, Option)>( + "SELECT kind::text, q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", ) .fetch_one(&db) .await?; - assert_eq!(kind, "singlestepflow"); + assert_eq!(kind, "script"); + assert!( + handle.is_some(), + "retry policy carried via runnable_settings_handle" + ); + Ok(()) + } + + // A scheduled, concurrency-limited script with a retry policy: the materialized + // root attempt's handle must resolve to BOTH the retry policy and the script's + // concurrency settings — otherwise the retry chain runs unbounded. Regression: P1. + #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] + async fn test_push_script_with_retry_keeps_concurrency( + db: Pool, + ) -> anyhow::Result<()> { + let concurrency = ConcurrencySettings { + concurrency_key: Some("f/system/test_script".to_string()), + concurrent_limit: Some(1), + concurrency_time_window_s: Some(60), + }; + let script_handle = insert_rs( + RunnableSettings { + debouncing_settings: None, + concurrency_settings: concurrency.insert_cached(&db).await?, + retry_settings: None, + }, + &db, + ) + .await?; + sqlx::query( + "UPDATE script SET runnable_settings_handle = $1 WHERE workspace_id = 'test-workspace' AND path = 'f/system/test_script'", + ) + .bind(script_handle) + .execute(&db) + .await?; + + let schedule = make_schedule(|s| { + s.retry = Some(serde_json::json!({ "constant": { "attempts": 3, "seconds": 10 } })); + }); + let authed = make_authed(); + let tx = db.begin().await?; + let tx = push_scheduled_job(&db, tx, &schedule, Some(&authed), None).await?; + tx.commit().await?; + + let handle = sqlx::query_scalar::<_, Option>( + "SELECT q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + ) + .fetch_one(&db) + .await?; + let rs = from_handle(handle, &db).await?; + assert!( + rs.retry_settings.is_some(), + "root attempt carries the retry policy" + ); + let resolved = ConcurrencySettings::get( + rs.concurrency_settings + .expect("root attempt must carry concurrency settings, not just retry"), + &db, + ) + .await?; + assert_eq!(resolved.concurrent_limit, Some(1)); + assert_eq!( + resolved.concurrency_key.as_deref(), + Some("f/system/test_script") + ); Ok(()) } @@ -779,7 +847,7 @@ mod schedule_push { } // ----------------------------------------------------------------------- - // try_schedule_next_job: script with retry wraps in SingleStepFlow + // try_schedule_next_job: script with retry is a native Script (no wrapping) // ----------------------------------------------------------------------- #[sqlx::test(migrations = "../migrations", fixtures("base", "schedule_push"))] @@ -798,12 +866,16 @@ mod schedule_push { assert!(err.is_none()); assert_eq!(count_queued_jobs(&db).await, 1); - let kind = sqlx::query_scalar::<_, String>( - "SELECT kind::text FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", + let (kind, handle) = sqlx::query_as::<_, (String, Option)>( + "SELECT kind::text, q.runnable_settings_handle FROM v2_job j JOIN v2_job_queue q ON j.id = q.id LIMIT 1", ) .fetch_one(&db) .await?; - assert_eq!(kind, "singlestepflow"); + assert_eq!(kind, "script"); + assert!( + handle.is_some(), + "retry policy carried via runnable_settings_handle" + ); Ok(()) } diff --git a/backend/windmill-runtime-nativets/Cargo.toml b/backend/windmill-runtime-nativets/Cargo.toml index edad615ae5..d3686d2264 100644 --- a/backend/windmill-runtime-nativets/Cargo.toml +++ b/backend/windmill-runtime-nativets/Cargo.toml @@ -48,6 +48,9 @@ futures.workspace = true sqlx.workspace = true rustls.workspace = true +[dev-dependencies] +rcgen = "0.13.2" + [build-dependencies] deno_fetch.workspace = true deno_webidl.workspace = true diff --git a/backend/windmill-runtime-nativets/src/cert_tests.rs b/backend/windmill-runtime-nativets/src/cert_tests.rs new file mode 100644 index 0000000000..677dc1be9b --- /dev/null +++ b/backend/windmill-runtime-nativets/src/cert_tests.rs @@ -0,0 +1,184 @@ +//! Regression tests for custom CA support in the in-process nativets fetch +//! runtime (WIN-2055). +//! +//! `deno_fetch` with `root_cert_store_provider: None` trusts only the Mozilla +//! webpki roots, so scripts calling internal APIs fronted by a corporate CA +//! failed with `invalid peer certificate: UnknownIssuer`. The provider built by +//! `build_native_root_cert_store_provider` merges CAs from `DENO_CERT` / +//! `SSL_CERT_FILE` / `NODE_EXTRA_CA_CERTS` / `DENO_TLS_CA_STORE=system` into the +//! default store. These tests pin that behaviour. + +use crate::{build_native_root_cert_store_provider, load_pem_certs_from_path}; + +/// The CA-related env vars the provider inspects. Cleared around each test so a +/// CI runner that happens to set one of them can't perturb the result. +const CA_ENV_VARS: &[&str] = &[ + "DENO_CERT", + "SSL_CERT_FILE", + "NODE_EXTRA_CA_CERTS", + "DENO_TLS_CA_STORE", +]; + +fn write_test_ca(suffix: &str) -> std::path::PathBuf { + let cert = rcgen::generate_simple_self_signed(vec!["windmill-test-ca".to_string()]) + .expect("generate self-signed cert"); + let pem = cert.cert.pem(); + let path = std::env::temp_dir().join(format!( + "windmill-nativets-ca-{}-{}.pem", + std::process::id(), + suffix + )); + std::fs::write(&path, pem).expect("write cert"); + path +} + +/// Serializes the env-mutating tests against each other — env is process-global, +/// so concurrent `set_var`/`remove_var` would otherwise interleave. +static ENV_LOCK: std::sync::Mutex<()> = std::sync::Mutex::new(()); + +fn with_cleared_ca_env(f: impl FnOnce() -> T) -> T { + let _guard = ENV_LOCK.lock().unwrap_or_else(|e| e.into_inner()); + let saved: Vec<(&str, Option)> = CA_ENV_VARS + .iter() + .map(|k| (*k, std::env::var(k).ok())) + .collect(); + for k in CA_ENV_VARS { + std::env::remove_var(k); + } + let out = f(); + for (k, v) in saved { + match v { + Some(v) => std::env::set_var(k, v), + None => std::env::remove_var(k), + } + } + out +} + +#[test] +fn load_pem_certs_parses_self_signed_cert() { + let path = write_test_ca("load"); + let certs = load_pem_certs_from_path(path.to_str().unwrap()).expect("load certs"); + assert_eq!(certs.len(), 1, "expected exactly one cert in the bundle"); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn load_pem_certs_errors_on_missing_file() { + let missing = std::env::temp_dir().join("windmill-nativets-does-not-exist.pem"); + assert!(load_pem_certs_from_path(missing.to_str().unwrap()).is_err()); +} + +#[test] +fn no_ca_env_yields_no_provider() { + // serialize against the env-mutating tests via the shared guard + with_cleared_ca_env(|| { + assert!( + build_native_root_cert_store_provider().is_none(), + "without any CA env var the provider must stay None (default-only behaviour)" + ); + }); +} + +#[test] +fn ssl_cert_file_adds_custom_root() { + let path = write_test_ca("ssl"); + with_cleared_ca_env(|| { + std::env::set_var("SSL_CERT_FILE", &path); + let provider = build_native_root_cert_store_provider() + .expect("a custom CA was configured, provider must be Some"); + let store = provider.get_or_try_init().expect("store init"); + let default_len = deno_tls::create_default_root_cert_store().len(); + assert_eq!( + store.len(), + default_len + 1, + "custom CA should be added on top of the Mozilla defaults" + ); + }); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn node_extra_ca_certs_adds_custom_root() { + let path = write_test_ca("node"); + with_cleared_ca_env(|| { + std::env::set_var("NODE_EXTRA_CA_CERTS", &path); + let provider = build_native_root_cert_store_provider() + .expect("provider must be Some for NODE_EXTRA_CA_CERTS"); + let store = provider.get_or_try_init().expect("store init"); + assert_eq!( + store.len(), + deno_tls::create_default_root_cert_store().len() + 1 + ); + }); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn multiple_ca_env_vars_pointing_at_same_file_dedupe_to_one_root() { + // The tracing proxy points SSL_CERT_FILE, NODE_EXTRA_CA_CERTS and DENO_CERT at + // the same bundle; the path dedupe in build_native_root_cert_store_provider + // loads it once, so the store grows by exactly one (rustls' add does not dedupe). + let path = write_test_ca("dupe"); + with_cleared_ca_env(|| { + std::env::set_var("SSL_CERT_FILE", &path); + std::env::set_var("NODE_EXTRA_CA_CERTS", &path); + std::env::set_var("DENO_CERT", &path); + let provider = build_native_root_cert_store_provider().expect("provider must be Some"); + let store = provider.get_or_try_init().expect("store init"); + assert_eq!( + store.len(), + deno_tls::create_default_root_cert_store().len() + 1 + ); + }); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn worker_config_env_var_adds_custom_root() { + // A CA configured only through the worker-group config (DB `env_vars_static` + // / allowlisted forwarded vars) lands in `WORKER_CONFIG.env_vars`, not the + // worker's own process env. Child Deno/Bun jobs receive it via `.envs(...)`; + // nativets must pick it up from the same place. Regression for the in-process + // path missing that source. + use windmill_common::worker::WORKER_CONFIG; + + let path = write_test_ca("workercfg"); + with_cleared_ca_env(|| { + let prev = WORKER_CONFIG.load_full(); + let mut cfg = (*prev).clone(); + cfg.env_vars.insert( + "SSL_CERT_FILE".to_string(), + path.to_string_lossy().into_owned(), + ); + WORKER_CONFIG.store(std::sync::Arc::new(cfg)); + + let provider = build_native_root_cert_store_provider(); + // restore before asserting so a failure can't leak the mutated global + WORKER_CONFIG.store(prev); + + let provider = provider.expect("worker-config CA must produce a provider"); + let store = provider.get_or_try_init().expect("store init"); + assert_eq!( + store.len(), + deno_tls::create_default_root_cert_store().len() + 1 + ); + }); + let _ = std::fs::remove_file(&path); +} + +#[test] +fn deno_cert_adds_custom_root() { + let path = write_test_ca("deno"); + with_cleared_ca_env(|| { + std::env::set_var("DENO_CERT", &path); + let provider = + build_native_root_cert_store_provider().expect("provider must be Some for DENO_CERT"); + let store = provider.get_or_try_init().expect("store init"); + assert_eq!( + store.len(), + deno_tls::create_default_root_cert_store().len() + 1 + ); + }); + let _ = std::fs::remove_file(&path); +} diff --git a/backend/windmill-runtime-nativets/src/lib.rs b/backend/windmill-runtime-nativets/src/lib.rs index 52e23f11da..2058268256 100644 --- a/backend/windmill-runtime-nativets/src/lib.rs +++ b/backend/windmill-runtime-nativets/src/lib.rs @@ -18,6 +18,9 @@ pub use dedicated::{ExecutingIsolate, PrewarmedIsolate, PrewarmedResult}; #[cfg(test)] mod smoke_tests; +#[cfg(test)] +mod cert_tests; + use std::{ borrow::Cow, cell::RefCell, @@ -35,8 +38,10 @@ use deno_core::{ v8::{self, IsolateHandle}, Extension, JsRuntime, OpState, PollEventLoopOptions, RuntimeOptions, }; +use deno_error::JsErrorBox; use deno_fetch::FetchPermissions; use deno_net::NetPermissions; +use deno_tls::{rustls::pki_types::CertificateDer, rustls::RootCertStore, RootCertStoreProvider}; use deno_web::{BlobStore, TimersPermission}; use itertools::Itertools; use lazy_static::lazy_static; @@ -214,6 +219,123 @@ lazy_static! { Regex::new(r"^(https?)://(([^:@\s]+):([^:@\s]+)@)?([^:@\s]+)(:(\d+))?$").unwrap(); } +lazy_static! { + /// Root cert store for the in-process nativets fetch runtime. + /// + /// Unlike the Deno/Bun executors, nativets never spawns a child process, so + /// the CA env vars those executors forward (`DENO_CERT`, `DENO_TLS_CA_STORE`, + /// `SSL_CERT_FILE`/`NODE_EXTRA_CA_CERTS`) are never consumed by deno's CLI + /// layer. `deno_fetch` with `root_cert_store_provider: None` falls back to + /// the Mozilla webpki roots only, so corporate CAs fail with `UnknownIssuer`. + /// We read those env vars here and merge the certs into the default store. + /// + /// Snapshotted once for the process lifetime, like the Deno executor's + /// `DENO_CERT`/`DENO_TLS_CA_STORE` lazy statics (`deno_executor.rs`): the + /// fetch root store is shared across all (potentially prewarmed) isolates, so + /// per-job CA reconfiguration is out of scope. A later `WORKER_CONFIG` reload + /// is not picked up until the process restarts. + static ref NATIVE_ROOT_CERT_STORE_PROVIDER: Option> = + build_native_root_cert_store_provider(); +} + +struct NativeRootCertStoreProvider { + store: RootCertStore, +} + +impl RootCertStoreProvider for NativeRootCertStoreProvider { + fn get_or_try_init(&self) -> Result<&RootCertStore, JsErrorBox> { + Ok(&self.store) + } +} + +/// Resolve a CA-related env var the same way the child executors see it: the +/// worker's own process env, then the worker-group config (`env_vars_allowlist` +/// forwarded values + DB `env_vars_static` literals, resolved into +/// `WORKER_CONFIG.env_vars`). Child Deno/Bun jobs receive that config map via +/// `.envs(...)`, so nativets must consult it too or a CA set only through worker +/// config would silently not apply in-process. +fn resolve_ca_env_var(name: &str) -> Option { + if let Ok(v) = std::env::var(name) { + if !v.is_empty() { + return Some(v); + } + } + windmill_common::worker::WORKER_CONFIG + .load() + .env_vars + .get(name) + .filter(|v| !v.is_empty()) + .cloned() +} + +/// Build a root cert store seeded with the Mozilla webpki roots plus any custom +/// CAs configured via env. Returns `None` when no custom CA is configured, which +/// preserves the previous default-only behaviour. +fn build_native_root_cert_store_provider() -> Option> { + let mut store = deno_tls::create_default_root_cert_store(); + let mut added = 0usize; + + // File-path env vars, each pointing at a PEM bundle of one or more certs. + // `DENO_CERT` mirrors the Deno CLI; `SSL_CERT_FILE` is the OpenSSL standard + // also honoured by Bun/Node (via NODE_EXTRA_CA_CERTS). Dedupe by path because + // the tracing proxy points several of these at the same bundle, and rustls' + // RootCertStore::add does not dedupe — we'd otherwise trust the same root N times. + let mut seen_paths = std::collections::HashSet::new(); + for var in ["DENO_CERT", "SSL_CERT_FILE", "NODE_EXTRA_CA_CERTS"] { + let Some(path) = resolve_ca_env_var(var).filter(|p| !p.is_empty()) else { + continue; + }; + if !seen_paths.insert(path.clone()) { + continue; + } + match load_pem_certs_from_path(&path) { + Ok(certs) => { + for cert in certs { + if let Err(e) = store.add(cert) { + tracing::warn!("nativets: failed to add cert from {var}={path}: {e}"); + } else { + added += 1; + } + } + } + Err(e) => tracing::warn!("nativets: failed to read CA file {var}={path}: {e}"), + } + } + + // `DENO_TLS_CA_STORE=system` (comma-separated, may also contain `mozilla`) + // pulls in the OS trust store. Unlike the Deno CLI — where the list selects + // and orders the stores — this is purely additive: the Mozilla defaults are + // always seeded above, and `system` augments them. That is a strict superset + // of the public roots, which is what the corporate-CA use case needs. + if resolve_ca_env_var("DENO_TLS_CA_STORE") + .map(|v| v.split(',').any(|s| s.trim() == "system")) + .unwrap_or(false) + { + match deno_tls::deno_native_certs::load_native_certs() { + Ok(certs) => { + for cert in certs { + if store.add(CertificateDer::from(cert.0)).is_ok() { + added += 1; + } + } + } + Err(e) => tracing::warn!("nativets: failed to load system CA store: {e}"), + } + } + + if added == 0 { + return None; + } + tracing::info!("nativets: loaded {added} custom CA cert(s) into fetch root store"); + Some(Arc::new(NativeRootCertStoreProvider { store })) +} + +fn load_pem_certs_from_path(path: &str) -> anyhow::Result>> { + let file = std::fs::File::open(path)?; + let mut reader = std::io::BufReader::new(file); + deno_tls::load_certs(&mut reader).map_err(|e| anyhow::anyhow!(e)) +} + // ── Public interface ───────────────────────────────────────────────── /// Set up the deno_core/V8 runtime. Idempotent — safe to call multiple times. @@ -433,7 +555,7 @@ pub(crate) fn create_nativets_runtime( let ext = Extension { name: "windmill", ops: ops.into(), ..Default::default() }; let fetch_options = deno_fetch::Options { - root_cert_store_provider: None, + root_cert_store_provider: NATIVE_ROOT_CERT_STORE_PROVIDER.clone(), user_agent: ann.useragent.unwrap_or_else(|| "windmill/beta".to_string()), proxy: ann.proxy.map(|x| deno_tls::Proxy::Http { url: x.0, diff --git a/backend/windmill-store/Cargo.toml b/backend/windmill-store/Cargo.toml index b3aca5e669..fdd82a7e5d 100644 --- a/backend/windmill-store/Cargo.toml +++ b/backend/windmill-store/Cargo.toml @@ -53,3 +53,7 @@ futures.workspace = true chrono.workspace = true reqwest.workspace = true anyhow.workspace = true +base64.workspace = true + +[dev-dependencies] +magic-crypt.workspace = true diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index a52ffbcf27..b9cea536f3 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -43,6 +43,11 @@ use windmill_common::{ db::{DbWithOptAuthed, UserDB}, error::{self, Error, JsonResult, Result}, get_database_url, + user_drafts::{ + delete_all_drafts_for_path, delete_own_draft_for_path, fetch_draft_only, + fetch_draft_only_list_rows, maybe_overlay_draft, UserDraftItemKind, WithDraftOverlay, + WithDraftQuery, + }, utils::{not_found_if_none, paginate, require_admin, Pagination, StripPath}, variables, worker::{CLOUD_HOSTED, WINDMILL_DIR}, @@ -153,6 +158,15 @@ pub struct ListableResource { pub inherited_labels: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub ws_specific: Option, + /// `Some(true)` only on synthesized draft-only rows; `None` on deployed rows. + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub draft_only: Option, + /// True when the authed user has a per-user draft at this path (drives the + /// `*` suffix on the resources page). + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_draft: Option, } #[derive(Deserialize)] @@ -185,6 +199,9 @@ pub struct ListResourceQuery { pub value: Option, pub broad_filter: Option, pub label: Option, + /// When true, append per-user draft-only rows; picker callers leave it off + /// to stay deployed-only. See list synthesis in scripts.rs. + pub include_draft_only: Option, } #[derive(Serialize, FromRow)] @@ -252,6 +269,7 @@ async fn list_resources( Query(lq): Query, Query(pagination): Query, Extension(user_db): Extension, + Extension(db): Extension, Path(w_id): Path, ) -> JsonResult> { let (per_page, offset) = paginate(pagination); @@ -276,6 +294,13 @@ async fn list_resources( "folder_labels(resource.workspace_id, resource.path) as inherited_labels", "ws_specific.path IS NOT NULL as ws_specific", ]) + // Scalar EXISTS flags the authed user's per-user draft without fanning rows out. + .field( + &"EXISTS(SELECT 1 FROM draft WHERE draft.workspace_id = resource.workspace_id \ + AND draft.path = resource.path AND draft.typ = 'resource' \ + AND draft.email = ?) as is_draft" + .bind(&authed.email), + ) .left() .join("variable") .on("variable.path = resource.path AND variable.workspace_id = resource.workspace_id") @@ -352,7 +377,7 @@ async fn list_resources( let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; let mut tx = user_db.begin(&authed).await?; let allowed = build_scope_path_predicate(&authed, "resources", "read"); - let rows = sqlx::query_as::<_, ListableResource>(&sql) + let mut rows = sqlx::query_as::<_, ListableResource>(&sql) .fetch_all(&mut *tx) .await? .into_iter() @@ -361,6 +386,101 @@ async fn list_resources( tx.commit().await?; + // Append the authed user's draft-only resources; see scripts.rs. + // `resource_type` / `resource_type_exclude` are deliberately NOT in the bail-out + // list (the resources page always passes `resource_type_exclude`); they're applied + // per-row below against the draft JSON's `resource_type` instead. + if lq.include_draft_only.unwrap_or(false) + && !authed.is_operator + && offset == 0 + && lq.path_start.is_none() + && lq.path.is_none() + && lq.description.is_none() + && lq.value.is_none() + && lq.broad_filter.is_none() + && lq.label.is_none() + { + let rt_filter: Option> = lq + .resource_type + .as_deref() + .map(|s| s.split(',').map(str::trim).collect()); + let rt_exclude: Option> = lq + .resource_type_exclude + .as_deref() + .map(|s| s.split(',').map(str::trim).collect()); + let draft_only_rows = + fetch_draft_only_list_rows(&db, &w_id, &authed.email, UserDraftItemKind::Resource) + .await?; + + for row in draft_only_rows { + let v: serde_json::Value = + serde_json::from_str(row.value.0.get()).unwrap_or(serde_json::Value::Null); + // ResourceEditor's `ResourceState`: { path, description, args, labels?, wsSpecific, resource_type? } + let path = v + .get("path") + .and_then(|s| s.as_str()) + .unwrap_or("") + .to_string(); + if path.is_empty() || !allowed(&path) { + continue; + } + let description = v + .get("description") + .and_then(|x| x.as_str()) + .map(|s| s.to_string()); + let value = v.get("args").cloned(); + let resource_type = v + .get("resource_type") + .and_then(|x| x.as_str()) + .unwrap_or("") + .to_string(); + // Mirror the deployed query's resource_type narrowing for the + // synthesized rows (see the gate comment above). + if let Some(ref rts) = rt_filter { + if !rts.contains(&resource_type.as_str()) { + continue; + } + } + if let Some(ref excl) = rt_exclude { + if excl.contains(&resource_type.as_str()) { + continue; + } + } + let labels = v.get("labels").and_then(|x| { + x.as_array().map(|arr| { + arr.iter() + .filter_map(|s| s.as_str().map(|s| s.to_string())) + .collect::>() + }) + }); + let ws_specific = v.get("wsSpecific").and_then(|x| x.as_bool()); + + rows.push(ListableResource { + workspace_id: w_id.clone(), + path, + value, + description, + resource_type, + extra_perms: serde_json::Value::Object(serde_json::Map::new()), + created_by: None, + edited_at: Some(row.created_at), + is_linked: None, + is_refreshed: None, + is_oauth: None, + is_expired: None, + refresh_error: None, + account: None, + labels, + // No deployed row to inherit folder labels from. + inherited_labels: None, + ws_specific, + draft_only: Some(true), + // Synthesized rows are the authed user's draft. + is_draft: Some(true), + }); + } + } + Ok(Json(rows)) } @@ -369,13 +489,15 @@ async fn get_resource( Extension(user_db): Extension, Extension(db): Extension, Path((w_id, path)): Path<(String, StripPath)>, -) -> JsonResult { + Query(q): Query, +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || format!("resources:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; let resource_o = sqlx::query_as!( ListableResource, + // `null::bool` columns align with the struct fields; deployed rows are never draft-only. "SELECT resource.workspace_id, resource.path, resource.value, resource.description, resource.resource_type, resource.extra_perms, resource.created_by, resource.edited_at, resource.labels, @@ -385,7 +507,9 @@ async fn get_resource( variable.path IS NOT NULL as is_linked, variable.is_oauth as \"is_oauth?\", variable.account, - ws_specific.path IS NOT NULL as ws_specific + ws_specific.path IS NOT NULL as ws_specific, + null::bool as draft_only, + null::bool as is_draft FROM resource LEFT JOIN variable ON variable.path = resource.path AND variable.workspace_id = $2 LEFT JOIN account ON variable.account = account.id AND account.workspace_id = $2 @@ -397,11 +521,30 @@ async fn get_resource( .fetch_optional(&mut *tx) .await?; tx.commit().await?; + if resource_o.is_none() && q.get_draft { + // No deployed row + `get_draft`: synthesize the response from the draft + // alone (`no_deployed = true`); see scripts.rs. + if let Some(overlay) = + fetch_draft_only(&db, &w_id, &authed.email, UserDraftItemKind::Resource, path).await? + { + return Ok(Json(overlay)); + } + } if resource_o.is_none() { explain_resource_perm_error(&path, &w_id, &db, &authed).await?; } let resource = not_found_if_none(resource_o, "Resource", path)?; - Ok(Json(resource)) + let overlay = maybe_overlay_draft( + &db, + &w_id, + &authed.email, + UserDraftItemKind::Resource, + path, + q.get_draft, + resource, + ) + .await?; + Ok(Json(overlay)) } async fn exists_resource( @@ -1168,6 +1311,13 @@ async fn delete_resource( .await?; tx.commit().await?; + // Resource gone for everyone: wipe ALL users' drafts at this path (and any linked + // variables cascaded into) so teammates' drafts don't orphan. Idempotent on no-draft. + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Resource, path).await?; + for var_path in &deleted_linked_variables { + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Variable, var_path).await?; + } + handle_deployment_metadata( &authed.email, &authed.username, @@ -1437,6 +1587,14 @@ async fn delete_resources_bulk( tx.commit().await?; + // Wipe ALL users' drafts at these paths (and linked variables); see delete_resource. + for path in &deleted_paths { + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Resource, path).await?; + } + for var_path in &linked_var_paths { + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Variable, var_path).await?; + } + try_join_all(deleted_paths.iter().map(|path| { handle_deployment_metadata( &authed.email, @@ -1476,6 +1634,12 @@ async fn update_resource( let path = path.to_path(); check_scopes(&authed, || format!("resources:write:{}", path))?; + // A rename moves the resource (and its linked variable) to ns.path, so the + // destination must also be within the token's write scope, not just the + // source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("resources:write:{}", npath))?; + } if let RuleCheckResult::Blocked(msg) = check_deploy_rules( &w_id, AuditAuthorable::username(&authed), @@ -1664,6 +1828,28 @@ async fn update_resource( // Detect if this was a rename operation let old_path_if_renamed = if npath != path { Some(path) } else { None }; + // On rename the draft at the OLD path orphans (no SQL FK); clear the deployer's + // own (+ legacy NULL) there, teammates keep theirs (StaleDraftModal). The linked + // variable renames alongside the resource, so its old-path draft orphans too. + if let Some(old_path) = old_path_if_renamed { + delete_own_draft_for_path( + &db, + &w_id, + UserDraftItemKind::Resource, + old_path, + &authed.email, + ) + .await?; + delete_own_draft_for_path( + &db, + &w_id, + UserDraftItemKind::Variable, + old_path, + &authed.email, + ) + .await?; + } + handle_deployment_metadata( &authed.email, &authed.username, diff --git a/backend/windmill-store/src/secret_backend_ext.rs b/backend/windmill-store/src/secret_backend_ext.rs index 1ae32c80ce..d5272384c0 100644 --- a/backend/windmill-store/src/secret_backend_ext.rs +++ b/backend/windmill-store/src/secret_backend_ext.rs @@ -6,269 +6,27 @@ * LICENSE-AGPL for a copy of the license. */ -//! Secret backend extension for the API layer +//! Secret backend extension for the store layer //! -//! This module provides helper functions for integrating the SecretBackend -//! trait with variable operations in the API. +//! Write-side helpers for integrating the SecretBackend trait with variable +//! operations. Backend resolution and read helpers live in +//! `windmill_common::secret_backend` (so lower-level crates can resolve secrets +//! too) and are re-exported here for existing callers. //! //! Note: HashiCorp Vault integration requires Enterprise Edition. //! The OSS version only supports the database backend. -use std::sync::Arc; - use windmill_common::{ db::DB, error::{Error, Result}, - secret_backend::{database::DatabaseBackend, SecretBackend}, - variables::{build_crypt, decrypt, encrypt}, + variables::{build_crypt, encrypt}, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use windmill_common::{ - global_settings::{load_value_from_global_settings, SECRET_BACKEND_SETTING}, - secret_backend::{ - AwsSecretsManagerBackend, AwsSecretsManagerSettings, AzureKeyVaultBackend, - AzureKeyVaultSettings, SecretBackendConfig, VaultBackend, VaultSettings, - }, +pub use windmill_common::secret_backend::{ + get_secret_backend, get_secret_value, is_aws_sm_stored_value, is_azure_kv_stored_value, + is_external_stored_value, is_vault_backend_configured, is_vault_stored_value, }; -#[cfg(all(feature = "private", feature = "enterprise"))] -use tokio::sync::RwLock; - -// Cached Vault backend to avoid recreating it for every request -// This enables connection pooling and avoids repeated setup overhead -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedVaultBackend { - backend: Arc, - settings: VaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref VAULT_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAzureKvBackend { - backend: Arc, - settings: AzureKeyVaultSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AZURE_KV_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -// Cached AWS Secrets Manager backend -#[cfg(all(feature = "private", feature = "enterprise"))] -struct CachedAwsSmBackend { - backend: Arc, - settings: AwsSecretsManagerSettings, -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -lazy_static::lazy_static! { - static ref AWS_SM_BACKEND_CACHE: RwLock> = RwLock::new(None); -} - -/// Get the current secret backend based on global settings -/// -/// OSS: Always returns DatabaseBackend -/// EE: Returns configured backend (Database or Vault) -#[cfg(not(all(feature = "private", feature = "enterprise")))] -pub async fn get_secret_backend(db: &DB) -> Result> { - Ok(Arc::new(DatabaseBackend::new(db.clone()))) -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -pub async fn get_secret_backend(db: &DB) -> Result> { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - match config { - SecretBackendConfig::Database => Ok(Arc::new(DatabaseBackend::new(db.clone()))), - SecretBackendConfig::HashiCorpVault(settings) => { - get_or_create_vault_backend(db, settings).await - } - SecretBackendConfig::AzureKeyVault(settings) => { - get_or_create_azure_kv_backend(db, settings).await - } - SecretBackendConfig::AwsSecretsManager(settings) => { - get_or_create_aws_sm_backend(db, settings).await - } - } -} - -/// Get a cached Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_vault_backend( - _db: &DB, - settings: VaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = VAULT_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = VAULT_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = { - #[cfg(feature = "openidconnect")] - if settings.token.is_none() { - Arc::new(VaultBackend::new_with_db(settings.clone(), _db.clone())) - } else { - Arc::new(VaultBackend::new(settings.clone())) - } - - #[cfg(not(feature = "openidconnect"))] - Arc::new(VaultBackend::new(settings.clone())) - }; - - // Cache it - *cache = Some(CachedVaultBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached Azure Key Vault backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_azure_kv_backend( - _db: &DB, - settings: AzureKeyVaultSettings, -) -> Result> { - // Check if we have a cached backend with matching settings (read lock) - { - let cache = AZURE_KV_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - // Need to create a new backend - acquire write lock - let mut cache = AZURE_KV_BACKEND_CACHE.write().await; - - // Double-check (another task may have created it while we waited) - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - // Create new backend - let backend: Arc = Arc::new(AzureKeyVaultBackend::new(settings.clone())); - - // Cache it - *cache = Some(CachedAzureKvBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Get a cached AWS SM backend or create a new one if settings changed -#[cfg(all(feature = "private", feature = "enterprise"))] -async fn get_or_create_aws_sm_backend( - _db: &DB, - settings: AwsSecretsManagerSettings, -) -> Result> { - { - let cache = AWS_SM_BACKEND_CACHE.read().await; - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - } - - let mut cache = AWS_SM_BACKEND_CACHE.write().await; - - if let Some(ref cached) = *cache { - if cached.settings == settings { - return Ok(cached.backend.clone()); - } - } - - let backend: Arc = - Arc::new(AwsSecretsManagerBackend::new_with_client(settings.clone()).await?); - - *cache = Some(CachedAwsSmBackend { backend: backend.clone(), settings }); - - Ok(backend) -} - -/// Check if a Vault backend is currently configured -/// -/// OSS: Always returns false -/// EE: Checks global settings -#[cfg(not(all(feature = "private", feature = "enterprise")))] -pub async fn is_vault_backend_configured(_db: &DB) -> Result { - Ok(false) -} - -#[cfg(all(feature = "private", feature = "enterprise"))] -pub async fn is_vault_backend_configured(db: &DB) -> Result { - let config = match load_value_from_global_settings(db, SECRET_BACKEND_SETTING).await? { - Some(value) => serde_json::from_value::(value).unwrap_or_default(), - None => SecretBackendConfig::default(), - }; - - Ok(matches!( - config, - SecretBackendConfig::HashiCorpVault(_) - | SecretBackendConfig::AzureKeyVault(_) - | SecretBackendConfig::AwsSecretsManager(_) - )) -} - -/// Get a secret value using the configured backend -/// -/// For database backend: decrypts using workspace key -/// For vault backend (EE only): fetches from Vault directly -pub async fn get_secret_value( - db: &DB, - workspace_id: &str, - path: &str, - encrypted_value: &str, -) -> Result { - let backend = get_secret_backend(db).await?; - - match backend.backend_name() { - "database" => { - // Use existing database decryption - let mc = build_crypt(db, workspace_id).await?; - decrypt(&mc, encrypted_value.to_string()).map_err(|e| { - Error::internal_err(format!("Error decrypting variable {}: {}", path, e)) - }) - } - "hashicorp_vault" => { - // Fetch from Vault directly - backend.get_secret(workspace_id, path).await - } - "azure_key_vault" => backend.get_secret(workspace_id, path).await, - "aws_secrets_manager" => backend.get_secret(workspace_id, path).await, - _ => Err(Error::internal_err(format!( - "Unknown backend: {}", - backend.backend_name() - ))), - } -} - /// Store a secret value using the configured backend /// /// For database backend: encrypts using workspace key and returns encrypted value @@ -412,26 +170,6 @@ pub async fn delete_secret_from_backend(db: &DB, workspace_id: &str, path: &str) } } -/// Check if a value is stored in Vault (indicated by the $vault: prefix) -pub fn is_vault_stored_value(value: &str) -> bool { - value.starts_with("$vault:") -} - -/// Check if a value is stored in Azure Key Vault (indicated by the $azure_kv: prefix) -pub fn is_azure_kv_stored_value(value: &str) -> bool { - value.starts_with("$azure_kv:") -} - -/// Check if a value is stored in AWS Secrets Manager (indicated by the $aws_sm: prefix) -pub fn is_aws_sm_stored_value(value: &str) -> bool { - value.starts_with("$aws_sm:") -} - -/// Check if a value is stored in any external secret backend -pub fn is_external_stored_value(value: &str) -> bool { - is_vault_stored_value(value) || is_azure_kv_stored_value(value) || is_aws_sm_stored_value(value) -} - /// Rename a secret in Vault when a variable path changes (EE only) #[cfg(not(all(feature = "private", feature = "enterprise")))] pub async fn rename_vault_secret( diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index 347415f27e..55047ec0c2 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -14,8 +14,8 @@ use windmill_common::db::DB; use windmill_common::workspaces::{check_deploy_rules, RuleCheckResult}; use crate::secret_backend_ext::{ - delete_secret_from_backend, get_secret_value, is_vault_stored_value, rename_vault_secret, - store_secret_value, + delete_secret_from_backend, get_secret_value, is_external_stored_value, is_vault_stored_value, + rename_vault_secret, store_secret_value, }; use windmill_common::utils::{escape_ilike_pattern, BulkDeleteRequest}; use windmill_common::webhook::{WebhookMessage, WebhookShared}; @@ -25,6 +25,7 @@ use axum::{ routing::{delete, get, post}, Json, Router, }; +use base64::{engine::general_purpose::STANDARD, Engine as _}; use futures::future::try_join_all; use hyper::StatusCode; use serde_json::Value; @@ -35,6 +36,11 @@ use windmill_common::{ db::{DbWithOptAuthed, UserDB}, error::{Error, JsonResult, Result}, scripts::ScriptHash, + user_drafts::{ + decrypt_draft_secret_value, delete_all_drafts_for_path, delete_own_draft_for_path, + fetch_draft_only, fetch_draft_only_list_rows, maybe_overlay_draft, UserDraftItemKind, + WithDraftOverlay, ENCRYPTED_DRAFT_PREFIX, + }, utils::{not_found_if_none, paginate, Pagination, StripPath, WarnAfterExt}, variables::{ build_crypt, get_reserved_variables, ContextualVariable, CreateVariable, ListableVariable, @@ -109,11 +115,15 @@ struct ListVariableQuery { pub value: Option, pub broad_filter: Option, pub label: Option, + /// When true, append per-user draft-only rows; picker callers leave it off + /// to stay deployed-only. See list synthesis in scripts.rs. + pub include_draft_only: Option, } async fn list_variables( authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path(w_id): Path, Query(lq): Query, Query(pagination): Query, @@ -143,6 +153,13 @@ async fn list_variables( "variable.edited_at", "variable.edited_by", ]) + // Scalar EXISTS flags the authed user's per-user draft; see resources.rs. + .field( + &"EXISTS(SELECT 1 FROM draft WHERE draft.workspace_id = variable.workspace_id \ + AND draft.path = variable.path AND draft.typ = 'variable' \ + AND draft.email = ?) as is_draft" + .bind(&authed.email), + ) .left() .join("account") .on("variable.account = account.id AND account.workspace_id = ?".bind(&w_id)) @@ -199,7 +216,7 @@ async fn list_variables( let sql = sqlb.sql().map_err(|e| Error::internal_err(e.to_string()))?; let mut tx = user_db.begin(&authed).await?; let allowed = build_scope_path_predicate(&authed, "variables", "read"); - let rows = sqlx::query_as::<_, ListableVariable>(&sql) + let mut rows = sqlx::query_as::<_, ListableVariable>(&sql) .fetch_all(&mut *tx) .await? .into_iter() @@ -207,13 +224,102 @@ async fn list_variables( .collect::>(); tx.commit().await?; + + // Append the authed user's draft-only variables; see scripts.rs. + if lq.include_draft_only.unwrap_or(false) + && !authed.is_operator + && offset == 0 + && lq.path_start.is_none() + && lq.path.is_none() + && lq.description.is_none() + && lq.value.is_none() + && lq.broad_filter.is_none() + && lq.label.is_none() + { + let draft_only_rows = + fetch_draft_only_list_rows(&db, &w_id, &authed.email, UserDraftItemKind::Variable) + .await?; + + for row in draft_only_rows { + let v: serde_json::Value = + serde_json::from_str(row.value.0.get()).unwrap_or(serde_json::Value::Null); + // VariableEditor's `VariableState`: { path, variable: { value, is_secret, description }, labels?, wsSpecific } + let path = v + .get("path") + .and_then(|s| s.as_str()) + .unwrap_or("") + .to_string(); + if path.is_empty() || !allowed(&path) { + continue; + } + let variable = v + .get("variable") + .cloned() + .unwrap_or(serde_json::Value::Null); + let is_secret = variable + .get("is_secret") + .and_then(|x| x.as_bool()) + .unwrap_or(false); + let description = variable + .get("description") + .and_then(|x| x.as_str()) + .unwrap_or("") + .to_string(); + // Secret variables never expose their value in the list response, even from a draft. + let value = if is_secret { + None + } else { + variable + .get("value") + .and_then(|x| x.as_str()) + .map(|s| s.to_string()) + }; + let labels = v.get("labels").and_then(|x| { + x.as_array().map(|arr| { + arr.iter() + .filter_map(|s| s.as_str().map(|s| s.to_string())) + .collect::>() + }) + }); + let ws_specific = v.get("wsSpecific").and_then(|x| x.as_bool()); + + rows.push(ListableVariable { + workspace_id: w_id.clone(), + path, + value, + is_secret, + description, + extra_perms: serde_json::Value::Object(serde_json::Map::new()), + account: None, + is_oauth: None, + is_expired: None, + is_refreshed: None, + refresh_error: None, + is_linked: None, + expires_at: None, + labels, + // No deployed row to inherit folder labels from. + inherited_labels: None, + ws_specific, + edited_at: Some(row.created_at), + edited_by: None, + draft_only: Some(true), + // Synthesized rows are the authed user's draft. + is_draft: Some(true), + }); + } + } + Ok(Json(rows)) } +// `get_draft` inlined rather than flattened (axum query bool quirk); see GetScriptByPathQuery in scripts.rs. #[derive(Deserialize)] struct GetVariableQuery { decrypt_secret: Option, include_encrypted: Option, + #[serde(default)] + get_draft: bool, } async fn get_variable( @@ -222,7 +328,7 @@ async fn get_variable( Extension(db): Extension, Query(q): Query, Path((w_id, path)): Path<(String, StripPath)>, -) -> JsonResult { +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || format!("variables:read:{}", path))?; @@ -252,6 +358,17 @@ async fn get_variable( let variable = if let Some(variable) = variable_o { variable + } else if q.get_draft { + // No deployed row + `get_draft`: fall back to the draft (see scripts.rs). + // Drop the user_db tx first since `fetch_draft_only` runs on `db`. + tx.commit().await?; + if let Some(overlay) = + fetch_draft_only(&db, &w_id, &authed.email, UserDraftItemKind::Variable, path).await? + { + return Ok(Json(overlay)); + } + explain_variable_perm_error(&path, &w_id, &db).await?; + unreachable!() } else { explain_variable_perm_error(&path, &w_id, &db).await?; unreachable!() @@ -310,7 +427,17 @@ async fn get_variable( variable }; - Ok(Json(r)) + let overlay = maybe_overlay_draft( + &db, + &w_id, + &authed.email, + UserDraftItemKind::Variable, + path, + q.get_draft, + r, + ) + .await?; + Ok(Json(overlay)) } #[derive(Deserialize)] @@ -409,6 +536,35 @@ async fn check_path_conflict(db: &DB, w_id: &str, path: &str) -> Result<()> { return Ok(()); } +/// Reject a secret value flagged as already-encrypted (`already_encrypted=true`) +/// that is not actually workspace-key ciphertext — e.g. plaintext mistakenly +/// pushed as encrypted. Storing plaintext in the encrypted `value` column +/// silently bricks the variable: every later read fails to decrypt it. +/// +/// The check is purely structural and never decrypts, so it cannot act as a +/// decryption/padding oracle for a caller who can write but not read secrets. +/// `encrypt` (AES-256-CBC) always yields standard base64 decoding to a non-zero +/// multiple of the 16-byte block size; anything else cannot be our ciphertext. +/// Values stored by an external backend ($vault:/$aws_sm:/$azure_kv: markers) +/// are not workspace ciphertext and are passed through untouched. +fn validate_already_encrypted_secret(path: &str, value: &str) -> Result<()> { + if is_external_stored_value(value) { + return Ok(()); + } + let looks_like_ciphertext = STANDARD + .decode(value) + .map(|bytes| !bytes.is_empty() && bytes.len() % 16 == 0) + .unwrap_or(false); + if !looks_like_ciphertext { + return Err(Error::BadRequest(format!( + "Variable {path} was sent as already-encrypted (already_encrypted=true) but its \ + value is not valid workspace-encrypted ciphertext. To push a plaintext secret, \ + send it without already_encrypted (CLI: use --plain-secrets) so it gets encrypted." + ))); + } + Ok(()) +} + async fn create_variable( authed: ApiAuthed, Extension(db): Extension, @@ -449,9 +605,21 @@ async fn create_variable( check_path_conflict(&db, &w_id, &variable.path).await?; let value = if variable.is_secret && !already_encrypted.unwrap_or(false) { + // A restored draft sends the `$encrypted:` marker as-is; decrypt it back + // (validating against the workspace key) before the secret backend re-stores it. + let plain = if variable.value.starts_with(ENCRYPTED_DRAFT_PREFIX) { + decrypt_draft_secret_value(&db, &w_id, &variable.value).await? + } else { + variable.value.clone() + }; // Use secret backend for encryption (supports both DB and Vault) - store_secret_value(&db, &w_id, &variable.path, &variable.value).await? + store_secret_value(&db, &w_id, &variable.path, &plain).await? } else { + if variable.is_secret { + // already_encrypted == true: value is stored verbatim, so it must be + // ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(&variable.path, &variable.value)?; + } variable.value }; @@ -647,6 +815,13 @@ async fn delete_variable( tx.commit().await?; + // Variable gone for everyone: wipe ALL users' drafts at this path (see resources.rs). + // Resource included because variables cascade-delete the linked resource at the same path. + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Variable, path).await?; + if deleted_linked_resource.is_some() { + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Resource, path).await?; + } + // If variable was a secret, also delete from Vault backend (if configured) if is_secret { delete_secret_from_backend(&db, &w_id, path).await?; @@ -785,12 +960,12 @@ async fn delete_variables_bulk( ) .execute(&mut *tx) .await?; - sqlx::query!( - "DELETE FROM resource WHERE path = ANY($1) AND workspace_id = $2", + let deleted_resource_paths = sqlx::query_scalar!( + "DELETE FROM resource WHERE path = ANY($1) AND workspace_id = $2 RETURNING path", &deleted_paths, w_id ) - .execute(&mut *tx) + .fetch_all(&mut *tx) .await?; sqlx::query!( @@ -814,6 +989,14 @@ async fn delete_variables_bulk( tx.commit().await?; + // Wipe ALL users' drafts at these paths (and linked resources); see delete_variable. + for path in &deleted_paths { + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Variable, path).await?; + } + for path in &deleted_resource_paths { + delete_all_drafts_for_path(&db, &w_id, UserDraftItemKind::Resource, path).await?; + } + // Delete secrets from Vault backend (if configured) for path in &secret_paths { if deleted_paths.contains(path) { @@ -889,6 +1072,12 @@ async fn update_variable( let path = path.to_path(); check_scopes(&authed, || format!("variables:write:{}", path))?; + // A rename moves the (possibly secret) variable to ns.path, so the + // destination must also be within the token's write scope, not just the + // source path. + if let Some(npath) = ns.path.as_deref() { + check_scopes(&authed, || format!("variables:write:{}", npath))?; + } let authed = maybe_refresh_folders(&path, &w_id, authed, &db).await; let mut sqlb = SqlBuilder::update_table("variable"); @@ -918,10 +1107,21 @@ async fn update_variable( }; let value = if is_secret && !already_encrypted.unwrap_or(false) { + // Decrypt a restored draft's `$encrypted:` marker before re-storing; see create_variable. + let plain = if nvalue.starts_with(ENCRYPTED_DRAFT_PREFIX) { + decrypt_draft_secret_value(&db, &w_id, &nvalue).await? + } else { + nvalue + }; // Use secret backend for encryption (supports both DB and Vault) // Store at target_path (new path if renaming, otherwise current path) - store_secret_value(&db, &w_id, target_path, &nvalue).await? + store_secret_value(&db, &w_id, target_path, &plain).await? } else { + if is_secret { + // already_encrypted == true: value is stored verbatim, so it must + // be ciphertext and not plaintext mislabeled as encrypted. + validate_already_encrypted_secret(target_path, &nvalue)?; + } nvalue }; sqlb.set_str("value", &value); @@ -1171,6 +1371,27 @@ async fn update_variable( // Detect if this was a rename operation let old_path_if_renamed = if npath != path { Some(path) } else { None }; + // On rename the old-path draft orphans (see resources.rs); the linked resource + // renames alongside the variable, so its old-path draft orphans too. + if let Some(old_path) = old_path_if_renamed { + delete_own_draft_for_path( + &db, + &w_id, + UserDraftItemKind::Variable, + old_path, + &authed.email, + ) + .await?; + delete_own_draft_for_path( + &db, + &w_id, + UserDraftItemKind::Resource, + old_path, + &authed.email, + ) + .await?; + } + handle_deployment_metadata( &authed.email, &authed.username, @@ -1332,3 +1553,61 @@ pub async fn get_value_internal<'a>( Ok(r) } + +#[cfg(test)] +mod tests { + use super::*; + use magic_crypt::MagicCryptTrait; + + #[test] + fn accepts_real_workspace_ciphertext() { + // The exact shape produced by `encrypt` (AES-256-CBC, base64). + let mc = magic_crypt::new_magic_crypt!("a-test-workspace-key", 256); + for plain in [ + "", + "original-secret", + "some: plaintext\n", + "a".repeat(500).as_str(), + ] { + let ciphertext = mc.encrypt_str_to_base64(plain); + assert!( + validate_already_encrypted_secret("f/x/cfg", &ciphertext).is_ok(), + "should accept genuine ciphertext for plaintext {plain:?}: {ciphertext}" + ); + } + } + + #[test] + fn rejects_plaintext_mislabeled_as_encrypted() { + // Plaintext mislabeled as encrypted: storing it verbatim would make the + // variable undecryptable on every read, so it must be rejected. + for plaintext in [ + "some: plaintext\n", + "original-secret", + "hunter2", + "{\"a\": 1}", + "not base64!!", + " leading-space", + ] { + assert!( + validate_already_encrypted_secret("f/x/cfg", plaintext).is_err(), + "should reject plaintext mislabeled as encrypted: {plaintext:?}" + ); + } + } + + #[test] + fn rejects_empty_and_non_block_aligned() { + // Valid base64 but not a whole number of AES blocks -> cannot be our ciphertext. + assert!(validate_already_encrypted_secret("p", "").is_err()); + assert!(validate_already_encrypted_secret("p", "dGVzdA==").is_err()); // "test" -> 4 bytes + } + + #[test] + fn passes_through_external_backend_markers() { + // External secret backends store $-prefixed markers, not workspace ciphertext. + for marker in ["$vault:f/x/cfg", "$aws_sm:f/x/cfg", "$azure_kv:f/x/cfg"] { + assert!(validate_already_encrypted_secret("f/x/cfg", marker).is_ok()); + } + } +} diff --git a/backend/windmill-test-utils/src/lib.rs b/backend/windmill-test-utils/src/lib.rs index ae2de880a4..c9c019c6e8 100644 --- a/backend/windmill-test-utils/src/lib.rs +++ b/backend/windmill-test-utils/src/lib.rs @@ -773,7 +773,6 @@ pub async fn assert_lockfile( cache_ttl: None, dedicated_worker: None, description: "".to_string(), - draft_only: None, envs: vec![], is_template: None, kind: None, @@ -796,6 +795,7 @@ pub async fn assert_lockfile( on_behalf_of_email: None, assets: vec![], modules: None, + draft_only: None, }, ) .await @@ -871,7 +871,6 @@ pub async fn run_deployed_relative_imports( cache_ttl: None, dedicated_worker: None, description: "".to_string(), - draft_only: None, envs: vec![], is_template: None, kind: None, @@ -894,6 +893,7 @@ pub async fn run_deployed_relative_imports( on_behalf_of_email: None, assets: vec![], modules: None, + draft_only: None, }, ) .await diff --git a/backend/windmill-trigger-azure/src/handler_oss.rs b/backend/windmill-trigger-azure/src/handler_oss.rs index e56c76a2fd..30257203b6 100644 --- a/backend/windmill-trigger-azure/src/handler_oss.rs +++ b/backend/windmill-trigger-azure/src/handler_oss.rs @@ -26,6 +26,7 @@ impl TriggerCrud for AzureTrigger { const TABLE_NAME: &'static str = ""; const TRIGGER_TYPE: &'static str = ""; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerAzure; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/azure_triggers"; diff --git a/backend/windmill-trigger-email/src/handler_oss.rs b/backend/windmill-trigger-email/src/handler_oss.rs index b6cac94cc2..fccbde4b96 100644 --- a/backend/windmill-trigger-email/src/handler_oss.rs +++ b/backend/windmill-trigger-email/src/handler_oss.rs @@ -29,6 +29,7 @@ impl TriggerCrud for EmailTrigger { const TABLE_NAME: &'static str = ""; const TRIGGER_TYPE: &'static str = ""; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerEmail; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/email_triggers"; diff --git a/backend/windmill-trigger-gcp/src/handler_oss.rs b/backend/windmill-trigger-gcp/src/handler_oss.rs index 1cacf7f594..c7e444f5ac 100644 --- a/backend/windmill-trigger-gcp/src/handler_oss.rs +++ b/backend/windmill-trigger-gcp/src/handler_oss.rs @@ -26,6 +26,7 @@ impl TriggerCrud for GcpTrigger { const TABLE_NAME: &'static str = ""; const TRIGGER_TYPE: &'static str = ""; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerGcp; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/gcp_triggers"; diff --git a/backend/windmill-trigger-http/src/handler.rs b/backend/windmill-trigger-http/src/handler.rs index 74fd748f55..17321b7775 100644 --- a/backend/windmill-trigger-http/src/handler.rs +++ b/backend/windmill-trigger-http/src/handler.rs @@ -7,7 +7,7 @@ use axum::{extract::Path, routing::post, Extension, Json, Router}; use http::StatusCode; use sqlx::PgConnection; use std::collections::HashSet; -use windmill_api_auth::ApiAuthed; +use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_audit::{audit_oss::audit_log, ActionKind}; use windmill_common::global_settings::HTTP_ROUTE_WORKSPACED_ROUTE; use windmill_common::{ @@ -262,6 +262,12 @@ pub async fn create_many_http_triggers( let mut route_path_keys = Vec::with_capacity(new_http_triggers.len()); for new_http_trigger in new_http_triggers.iter() { + // Per-item write scope, matching the single-create handler. The bulk + // endpoint must not let a path-scoped token create triggers outside it. + check_scopes(&authed, || { + format!("http_triggers:write:{}", &new_http_trigger.base.path) + })?; + handler .validate_new(&db, &w_id, &new_http_trigger.config) .await @@ -373,6 +379,8 @@ impl TriggerCrud for HttpTrigger { const TABLE_NAME: &'static str = "http_trigger"; const TRIGGER_TYPE: &'static str = "http"; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = + windmill_common::user_drafts::UserDraftItemKind::TriggerHttp; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/http_triggers"; diff --git a/backend/windmill-trigger-kafka/src/handler_oss.rs b/backend/windmill-trigger-kafka/src/handler_oss.rs index 57e786b0ea..0445664ddb 100644 --- a/backend/windmill-trigger-kafka/src/handler_oss.rs +++ b/backend/windmill-trigger-kafka/src/handler_oss.rs @@ -29,6 +29,7 @@ impl TriggerCrud for KafkaTrigger { const TABLE_NAME: &'static str = ""; const TRIGGER_TYPE: &'static str = ""; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerKafka; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/kafka_triggers"; diff --git a/backend/windmill-trigger-mqtt/src/handler.rs b/backend/windmill-trigger-mqtt/src/handler.rs index 49fb701241..f1a6b6ff9d 100644 --- a/backend/windmill-trigger-mqtt/src/handler.rs +++ b/backend/windmill-trigger-mqtt/src/handler.rs @@ -25,6 +25,7 @@ impl TriggerCrud for MqttTrigger { const TABLE_NAME: &'static str = "mqtt_trigger"; const TRIGGER_TYPE: &'static str = "mqtt"; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerMqtt; const SUPPORTS_SERVER_STATE: bool = true; const SUPPORTS_TEST_CONNECTION: bool = true; const ROUTE_PREFIX: &'static str = "/mqtt_triggers"; diff --git a/backend/windmill-trigger-nats/src/handler_oss.rs b/backend/windmill-trigger-nats/src/handler_oss.rs index 226bd653fd..b377774ee3 100644 --- a/backend/windmill-trigger-nats/src/handler_oss.rs +++ b/backend/windmill-trigger-nats/src/handler_oss.rs @@ -29,6 +29,7 @@ impl TriggerCrud for NatsTrigger { const TABLE_NAME: &'static str = ""; const TRIGGER_TYPE: &'static str = ""; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerNats; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/nats_triggers"; diff --git a/backend/windmill-trigger-postgres/src/handler.rs b/backend/windmill-trigger-postgres/src/handler.rs index dc2f4776fd..10a26e0066 100644 --- a/backend/windmill-trigger-postgres/src/handler.rs +++ b/backend/windmill-trigger-postgres/src/handler.rs @@ -46,6 +46,7 @@ impl TriggerCrud for PostgresTrigger { const TABLE_NAME: &'static str = "postgres_trigger"; const TRIGGER_TYPE: &'static str = "postgres"; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerPostgres; const SUPPORTS_SERVER_STATE: bool = true; const SUPPORTS_TEST_CONNECTION: bool = true; const ROUTE_PREFIX: &'static str = "/postgres_triggers"; diff --git a/backend/windmill-trigger-postgres/src/listener.rs b/backend/windmill-trigger-postgres/src/listener.rs index 22612db8d1..14d348b5f2 100644 --- a/backend/windmill-trigger-postgres/src/listener.rs +++ b/backend/windmill-trigger-postgres/src/listener.rs @@ -7,10 +7,12 @@ use pg_escape::{quote_identifier, quote_literal}; use rust_postgres::{Client, CopyBothDuplex, SimpleQueryMessage}; use tokio::sync::RwLock; use tokio_stream::StreamExt; +use windmill_api_auth::ApiAuthed; use windmill_common::{ db::UserDB, error::{to_anyhow, Error, Result}, jobs::JobTriggerKind, + utils::{report_critical_error, report_recovered_critical_error}, worker::to_raw_value, DB, }; @@ -31,6 +33,10 @@ use super::{ const ERROR_REPLICATION_SLOT_NOT_EXISTS: &str = r#"The replication slot associated with this trigger no longer exists. Recreate a new replication slot or select an existing one in the advanced tab, or delete and recreate a new trigger"#; +// Wait this long between reconnection attempts after a connection failure or a +// dropped replication stream. Matches the Kafka trigger listener's backoff. +const RECONNECT_DELAY_SECS: u64 = 30; + pub struct LogicalReplicationSettings { pub streaming: bool, } @@ -126,9 +132,70 @@ impl PostgresSimpleClient { } } +/// Resolves the Postgres resource, validates that the configured publication and +/// replication slot still exist, and opens a fresh logical replication stream. +/// +/// Returns `Error::BadConfig` when the publication or slot is missing (an +/// unrecoverable misconfiguration). Any other error is treated as transient +/// (connection refused, network interruption, ...) and is retried by the caller. +/// The resource is re-resolved on every call so credential rotations are picked +/// up across reconnections. +async fn connect_logical_replication_stream( + authed: &ApiAuthed, + db: &DB, + listening_trigger: &ListeningTrigger, +) -> Result<(CopyBothDuplex, LogicalReplicationSettings)> { + let ListeningTrigger { workspace_id, trigger_config, .. } = listening_trigger; + let PostgresConfig { postgres_resource_path, publication_name, replication_slot_name, .. } = + trigger_config; + + let database = resolve_postgres_resource( + authed, + Some(UserDB::new(db.clone())), + db, + postgres_resource_path, + workspace_id, + ) + .await?; + + let client = PostgresSimpleClient::new(&database).await?; + + let publication = client + .execute_query(&format!( + "SELECT pubname FROM pg_publication WHERE pubname = {}", + quote_literal(publication_name) + )) + .await + .map_err(to_anyhow)?; + + if !publication.row_exist() { + return Err(Error::BadConfig( + ERROR_PUBLICATION_NAME_NOT_EXISTS.to_string(), + )); + } + + let replication_slot = client + .execute_query(&format!( + "SELECT slot_name FROM pg_replication_slots WHERE slot_name = {}", + quote_literal(replication_slot_name) + )) + .await + .map_err(to_anyhow)?; + + if !replication_slot.row_exist() { + return Err(Error::BadConfig( + ERROR_REPLICATION_SLOT_NOT_EXISTS.to_string(), + )); + } + + client + .get_logical_replication_stream(publication_name, replication_slot_name) + .await +} + #[async_trait::async_trait] impl Listener for PostgresTrigger { - type Consumer = (CopyBothDuplex, LogicalReplicationSettings); + type Consumer = ApiAuthed; type Extra = (); type ExtraState = (); const JOB_TRIGGER_KIND: JobTriggerKind = JobTriggerKind::Postgres; @@ -140,65 +207,13 @@ impl Listener for PostgresTrigger { _err_message: Arc>>, _killpill_rx: tokio::sync::broadcast::Receiver<()>, ) -> Result> { - let ListeningTrigger:: { workspace_id, trigger_config, .. } = - listening_trigger; - - let PostgresConfig { - postgres_resource_path, publication_name, replication_slot_name, .. - } = trigger_config; - + // The actual replication connection is established (and retried) inside + // `consume`. Here we only resolve the auth context that connection needs. let authed = listening_trigger .authed(db, &Self::TRIGGER_KIND.to_string()) .await?; - let database = resolve_postgres_resource( - &authed, - Some(UserDB::new(db.clone())), - &db, - postgres_resource_path, - workspace_id, - ) - .await?; - - let client = PostgresSimpleClient::new(&database).await?; - - let publication = client - .execute_query(&format!( - "SELECT pubname FROM pg_publication WHERE pubname = {}", - quote_literal(&publication_name) - )) - .await - .map_err(to_anyhow)?; - - if !publication.row_exist() { - return Err(Error::BadConfig( - ERROR_PUBLICATION_NAME_NOT_EXISTS.to_string(), - )); - } - - let replication_slot = client - .execute_query(&format!( - "SELECT slot_name FROM pg_replication_slots WHERE slot_name = {}", - quote_literal(&replication_slot_name) - )) - .await - .map_err(to_anyhow)?; - - if !replication_slot.row_exist() { - return Err(Error::BadConfig( - ERROR_REPLICATION_SLOT_NOT_EXISTS.to_string(), - )); - } - - let (logical_replication_stream, logical_replication_settings) = client - .get_logical_replication_stream(&publication_name, &replication_slot_name) - .await - .map_err(to_anyhow)?; - - Ok(Some(( - logical_replication_stream, - logical_replication_settings, - ))) + Ok(Some(authed)) } async fn consume( &self, @@ -209,218 +224,376 @@ impl Listener for PostgresTrigger { _killpill_rx: tokio::sync::broadcast::Receiver<()>, _extra_state: Option<&Self::ExtraState>, ) { - let (logical_replication_stream, logical_replication_settings) = consumer; - pin_mut!(logical_replication_stream); - let mut relations = RelationConverter::new(); - tracing::info!( - "Starting to listen for postgres trigger {}", - &listening_trigger.path - ); - - // Highest WAL position received (and processed) so far. Reported back - // to Postgres via standby status updates so the replication slot can - // advance and retained WAL gets released. Without periodic updates the - // slot's LSNs stay frozen and WAL grows unbounded. - let mut last_lsn: u64 = 0; - let mut status_interval = tokio::time::interval(Duration::from_secs(10)); - status_interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); - // First tick resolves immediately; consume it so the periodic cadence - // starts one full interval from now. - status_interval.tick().await; + let authed = consumer; + // Consecutive failed connection attempts. Reset to 0 once the replication + // stream is (re)established. + let mut tries = 0_usize; loop { - let next = tokio::select! { - _ = status_interval.tick() => None, - message = logical_replication_stream.next() => Some(message), - }; - - let message = match next { - None => { - PostgresSimpleClient::send_status_update( - last_lsn, - &mut logical_replication_stream, - ) - .await; - continue; - } - Some(message) => message, - }; - let message = match message { - Some(message) => message, - None => { - tracing::error!( - "Stream for postgres trigger {} closed", - &listening_trigger.path - ); - if let None = self - .update_ping_and_loop_ping_status( - db, - listening_trigger, - err_message.clone(), - Some("Stream closed".to_string()), - ) - .await - { + let (logical_replication_stream, logical_replication_settings) = + match connect_logical_replication_stream(&authed, db, listening_trigger).await { + Ok(stream) => stream, + // Publication or replication slot missing: retrying cannot fix + // this, so disable the trigger as before. + Err(Error::BadConfig(err)) => { + self.disable_with_error(db, listening_trigger, err).await; return; } - return; - } - }; + // Transient failure (connection refused, network drop, ...): + // back off and retry instead of permanently disabling. + Err(err) => { + let status = format!( + "Failed to connect (attempt {}), retrying in {} seconds: {}", + tries + 1, + RECONNECT_DELAY_SECS, + err + ); + if let None = self + .update_ping_and_loop_ping_status( + db, + listening_trigger, + err_message.clone(), + Some(status), + ) + .await + { + return; + } - let message = match message { - Ok(message) => message, - Err(err) => { - let err = format!( - "Postgres trigger named {} had an error while receiving a message : {}", - &listening_trigger.path, - err.to_string() - ); - self.disable_with_error(db, listening_trigger, err).await; - return; - } - }; + tracing::error!( + "Failed to connect postgres trigger {} (attempt {}), retrying in {} seconds: {}", + &listening_trigger.path, + tries + 1, + RECONNECT_DELAY_SECS, + err + ); - let logical_message = match ReplicationMessage::parse(message) { - Ok(logical_message) => logical_message, - Err(err) => { - let err = format!( - "Postgres trigger named: {} had an error while parsing message: {}", - &listening_trigger.path, - err.to_string() - ); - self.disable_with_error(db, listening_trigger, err).await; - return; - } - }; + if tries % 10 == 0 && listening_trigger.trigger_mode { + report_critical_error( + format!( + "Failed to connect postgres trigger {} (attempt {}), retrying in {} seconds. This alert will repeat every 10 failed attempts. Error: {}", + &listening_trigger.path, + tries + 1, + RECONNECT_DELAY_SECS, + err + ), + db.clone(), + Some(&listening_trigger.workspace_id), + Some(&format!("postgres_trigger:{}", &listening_trigger.path)), + ) + .await; + } - match logical_message { - ReplicationMessage::PrimaryKeepAlive(primary_keep_alive) => { - last_lsn = last_lsn.max(primary_keep_alive.wal_end); - if primary_keep_alive.reply { + tries += 1; + tokio::time::sleep(Duration::from_secs(RECONNECT_DELAY_SECS)).await; + continue; + } + }; + + // The retry counter is reset (and recovery reported, ping cleared) only + // once the stream actually delivers a message in the inner loop. A + // connection that drops before making any progress therefore keeps + // counting toward the reconnection alert instead of ping-ponging + // silently. + pin_mut!(logical_replication_stream); + let mut relations = RelationConverter::new(); + tracing::info!( + "Starting to listen for postgres trigger {}", + &listening_trigger.path + ); + + // Highest WAL position received (and processed) so far. Reported back + // to Postgres via standby status updates so the replication slot can + // advance and retained WAL gets released. Without periodic updates the + // slot's LSNs stay frozen and WAL grows unbounded. + let mut last_lsn: u64 = 0; + let mut status_interval = tokio::time::interval(Duration::from_secs(10)); + status_interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Skip); + // First tick resolves immediately; consume it so the periodic cadence + // starts one full interval from now. + status_interval.tick().await; + + 'stream: loop { + let next = tokio::select! { + _ = status_interval.tick() => None, + message = logical_replication_stream.next() => Some(message), + }; + + let message = match next { + None => { PostgresSimpleClient::send_status_update( last_lsn, &mut logical_replication_stream, ) .await; + continue; } - } - ReplicationMessage::XLogData(x_log_data) => { - last_lsn = last_lsn.max(x_log_data.wal_end); - let logical_replication_message = match x_log_data - .parse(&logical_replication_settings) - { - Ok(logical_replication_message) => logical_replication_message, - Err(err) => { - tracing::error!("Postgres trigger named: {} had an error while trying to parse incomming stream message: {}", &listening_trigger.path, err.to_string()); - continue; + Some(message) => message, + }; + let message = match message { + Some(message) => message, + None => { + if let None = self + .update_ping_and_loop_ping_status( + db, + listening_trigger, + err_message.clone(), + Some(format!( + "Stream closed (attempt {}), reconnecting in {} seconds", + tries + 1, + RECONNECT_DELAY_SECS + )), + ) + .await + { + return; } - }; - - let json = match logical_replication_message { - Relation(relation_body) => { - relations.add_relation(relation_body); - None - } - Begin | Type | Commit => None, - Insert(insert) => Some(( - insert.o_id, - Ok(None), - relations.row_to_json((insert.o_id, insert.tuple)), - "insert", - )), - Update(update) => { - let old_row = update - .old_tuple - .map(|old_tuple| relations.row_to_json((update.o_id, old_tuple))) - .transpose(); - let row = relations.row_to_json((update.o_id, update.new_tuple)); - Some((update.o_id, old_row, row, "update")) - } - Delete(delete) => { - let row = delete - .old_tuple - .unwrap_or_else(|| delete.key_tuple.unwrap()); - Some(( - delete.o_id, - Ok(None), - relations.row_to_json((delete.o_id, row)), - "delete", - )) - } - }; - match json { - Some((o_id, Ok(old_row), Ok(row), transaction_type)) => { - let relation = match relations.get_relation(o_id) { - Ok(relation) => relation, - Err(err) => { - tracing::error!( - "Postgres trigger named: {}, error: {}", - &listening_trigger.path, - err.to_string() - ); - continue; - } - }; - let database_info = HashMap::from([ - ("schema_name".to_string(), to_raw_value(&relation.namespace)), - ("table_name".to_string(), to_raw_value(&relation.name)), - ( - "transaction_type".to_string(), - to_raw_value(&transaction_type), + tracing::error!( + "Stream for postgres trigger {} closed (attempt {}), reconnecting in {} seconds", + &listening_trigger.path, + tries + 1, + RECONNECT_DELAY_SECS + ); + if tries % 10 == 0 && listening_trigger.trigger_mode { + report_critical_error( + format!( + "Postgres trigger {} stream closed (attempt {}), reconnecting in {} seconds. This alert will repeat every 10 failed attempts.", + &listening_trigger.path, + tries + 1, + RECONNECT_DELAY_SECS ), - ("old_row".to_string(), to_raw_value(&old_row)), - ("row".to_string(), to_raw_value(&row)), - ]); - let _ = self - .handle_event( - db, - listening_trigger, - database_info, - HashMap::new(), - None, - ) - .await; + db.clone(), + Some(&listening_trigger.workspace_id), + Some(&format!("postgres_trigger:{}", &listening_trigger.path)), + ) + .await; } - Some((o_id, old_row, row, transaction_type)) => { - let relation = match relations.get_relation(o_id) { - Ok(relation) => relation, - Err(err) => { + tries += 1; + tokio::time::sleep(Duration::from_secs(RECONNECT_DELAY_SECS)).await; + break 'stream; + } + }; + + let message = match message { + Ok(message) => message, + Err(err) => { + if let None = self + .update_ping_and_loop_ping_status( + db, + listening_trigger, + err_message.clone(), + Some(format!( + "Error receiving message (attempt {}), reconnecting in {} seconds: {}", + tries + 1, + RECONNECT_DELAY_SECS, + err + )), + ) + .await + { + return; + } + tracing::error!( + "Postgres trigger {} had an error while receiving a message (attempt {}), reconnecting in {} seconds: {}", + &listening_trigger.path, + tries + 1, + RECONNECT_DELAY_SECS, + err.to_string() + ); + if tries % 10 == 0 && listening_trigger.trigger_mode { + report_critical_error( + format!( + "Postgres trigger {} error while receiving a message (attempt {}), reconnecting in {} seconds. This alert will repeat every 10 failed attempts. Error: {}", + &listening_trigger.path, + tries + 1, + RECONNECT_DELAY_SECS, + err + ), + db.clone(), + Some(&listening_trigger.workspace_id), + Some(&format!("postgres_trigger:{}", &listening_trigger.path)), + ) + .await; + } + tries += 1; + tokio::time::sleep(Duration::from_secs(RECONNECT_DELAY_SECS)).await; + break 'stream; + } + }; + + // First successful read after a (re)connection means the stream is + // making progress: clear the error status, report recovery, and + // reset the retry counter. Deferred to here (rather than on connect) + // so a stream that drops before delivering anything keeps counting + // toward the reconnection alert. + if tries > 0 { + if let None = self + .update_ping_and_loop_ping_status( + db, + listening_trigger, + err_message.clone(), + None, + ) + .await + { + return; + } + if listening_trigger.trigger_mode { + report_recovered_critical_error( + format!("Postgres trigger {} reconnected", &listening_trigger.path), + db.clone(), + Some(&listening_trigger.workspace_id), + Some(&format!("postgres_trigger:{}", &listening_trigger.path)), + ) + .await; + } + tries = 0; + } + + let logical_message = match ReplicationMessage::parse(message) { + Ok(logical_message) => logical_message, + Err(err) => { + let err = format!( + "Postgres trigger named: {} had an error while parsing message: {}", + &listening_trigger.path, + err.to_string() + ); + self.disable_with_error(db, listening_trigger, err).await; + return; + } + }; + + match logical_message { + ReplicationMessage::PrimaryKeepAlive(primary_keep_alive) => { + last_lsn = last_lsn.max(primary_keep_alive.wal_end); + if primary_keep_alive.reply { + PostgresSimpleClient::send_status_update( + last_lsn, + &mut logical_replication_stream, + ) + .await; + } + } + ReplicationMessage::XLogData(x_log_data) => { + last_lsn = last_lsn.max(x_log_data.wal_end); + let logical_replication_message = match x_log_data + .parse(&logical_replication_settings) + { + Ok(logical_replication_message) => logical_replication_message, + Err(err) => { + tracing::error!("Postgres trigger named: {} had an error while trying to parse incomming stream message: {}", &listening_trigger.path, err.to_string()); + continue; + } + }; + + let json = match logical_replication_message { + Relation(relation_body) => { + relations.add_relation(relation_body); + None + } + Begin | Type | Commit => None, + Insert(insert) => Some(( + insert.o_id, + Ok(None), + relations.row_to_json((insert.o_id, insert.tuple)), + "insert", + )), + Update(update) => { + let old_row = update + .old_tuple + .map(|old_tuple| { + relations.row_to_json((update.o_id, old_tuple)) + }) + .transpose(); + let row = relations.row_to_json((update.o_id, update.new_tuple)); + Some((update.o_id, old_row, row, "update")) + } + Delete(delete) => { + let row = delete + .old_tuple + .unwrap_or_else(|| delete.key_tuple.unwrap()); + Some(( + delete.o_id, + Ok(None), + relations.row_to_json((delete.o_id, row)), + "delete", + )) + } + }; + match json { + Some((o_id, Ok(old_row), Ok(row), transaction_type)) => { + let relation = match relations.get_relation(o_id) { + Ok(relation) => relation, + Err(err) => { + tracing::error!( + "Postgres trigger named: {}, error: {}", + &listening_trigger.path, + err.to_string() + ); + continue; + } + }; + let database_info = HashMap::from([ + ("schema_name".to_string(), to_raw_value(&relation.namespace)), + ("table_name".to_string(), to_raw_value(&relation.name)), + ( + "transaction_type".to_string(), + to_raw_value(&transaction_type), + ), + ("old_row".to_string(), to_raw_value(&old_row)), + ("row".to_string(), to_raw_value(&row)), + ]); + let _ = self + .handle_event( + db, + listening_trigger, + database_info, + HashMap::new(), + None, + ) + .await; + } + Some((o_id, old_row, row, transaction_type)) => { + let relation = match relations.get_relation(o_id) { + Ok(relation) => relation, + Err(err) => { + tracing::error!( + "Postgres trigger named: {}, error: {}", + &listening_trigger.path, + err.to_string() + ); + continue; + } + }; + + if let Err(err) = old_row { tracing::error!( - "Postgres trigger named: {}, error: {}", - &listening_trigger.path, - err.to_string() + transaction_type = ?transaction_type, + schema = %relation.namespace, + table = %relation.name, + error = %err, + "Failed to decode OLD row for {} transaction on {}.{}", + transaction_type, + relation.namespace, + relation.name, ); - continue; } - }; - if let Err(err) = old_row { - tracing::error!( - transaction_type = ?transaction_type, - schema = %relation.namespace, - table = %relation.name, - error = %err, - "Failed to decode OLD row for {} transaction on {}.{}", - transaction_type, - relation.namespace, - relation.name, - ); - } - - if let Err(err) = row { - tracing::error!( - transaction_type = ?transaction_type, - schema = %relation.namespace, - table = %relation.name, - error = %err, - "Failed to decode NEW row for {} transaction on {}.{}", - transaction_type, - relation.namespace, - relation.name, - ); + if let Err(err) = row { + tracing::error!( + transaction_type = ?transaction_type, + schema = %relation.namespace, + table = %relation.name, + error = %err, + "Failed to decode NEW row for {} transaction on {}.{}", + transaction_type, + relation.namespace, + relation.name, + ); + } } + _ => {} } - _ => {} } } } diff --git a/backend/windmill-trigger-sqs/src/handler_oss.rs b/backend/windmill-trigger-sqs/src/handler_oss.rs index fc72159e24..0f54a548fe 100644 --- a/backend/windmill-trigger-sqs/src/handler_oss.rs +++ b/backend/windmill-trigger-sqs/src/handler_oss.rs @@ -26,6 +26,7 @@ impl TriggerCrud for SqsTrigger { const TABLE_NAME: &'static str = ""; const TRIGGER_TYPE: &'static str = ""; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = windmill_common::user_drafts::UserDraftItemKind::TriggerSqs; const SUPPORTS_SERVER_STATE: bool = false; const SUPPORTS_TEST_CONNECTION: bool = false; const ROUTE_PREFIX: &'static str = "/sqs_triggers"; diff --git a/backend/windmill-trigger-websocket/src/handler.rs b/backend/windmill-trigger-websocket/src/handler.rs index df8bccaaa6..529e5902cc 100644 --- a/backend/windmill-trigger-websocket/src/handler.rs +++ b/backend/windmill-trigger-websocket/src/handler.rs @@ -4,7 +4,7 @@ use async_trait::async_trait; use itertools::Itertools; use serde_json::value::RawValue; use sqlx::{types::Json as SqlxJson, PgConnection}; -use windmill_api_auth::ApiAuthed; +use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_common::DB; use windmill_common::{ db::UserDB, @@ -15,10 +15,39 @@ use windmill_git_sync::DeployedObject; use windmill_trigger::{Trigger, TriggerCrud, TriggerData}; use super::{ - get_url_from_runnable_value, proxy::connect_async_with_proxy, TestWebsocketConfig, - WebsocketConfig, WebsocketConfigRequest, WebsocketTrigger, + get_url_from_runnable_value, listener::InitialMessage, proxy::connect_async_with_proxy, + validate_websocket_url_for_ssrf, TestWebsocketConfig, WebsocketConfig, WebsocketConfigRequest, + WebsocketTrigger, }; +/// A websocket_triggers:write token can configure secondary runnables that the +/// listener later executes under the trigger owner's identity: a `$flow:`/ +/// `$script:` URL resolver and `initial_messages` of kind `runnable_result`. +/// That execution happens in a background task where the reconstructed authed is +/// scopeless (so its check_scopes is a no-op), so enforce run scope here, at +/// create/update time, against the API caller's token. +fn check_secondary_runnable_scopes( + authed: &ApiAuthed, + config: &WebsocketConfigRequest, +) -> Result<()> { + if let Some(rest) = config.url.strip_prefix("$flow:") { + check_scopes(authed, || format!("jobs:run:flows:{}", rest))?; + } else if let Some(rest) = config.url.strip_prefix("$script:") { + check_scopes(authed, || format!("jobs:run:scripts:{}", rest))?; + } + if let Some(messages) = config.initial_messages.as_ref() { + for msg in messages { + if let Ok(InitialMessage::RunnableResult { path, is_flow, .. }) = + serde_json::from_value::(msg.clone()) + { + let kind = if is_flow { "flows" } else { "scripts" }; + check_scopes(authed, || format!("jobs:run:{}:{}", kind, path))?; + } + } + } + Ok(()) +} + #[async_trait] impl TriggerCrud for WebsocketTrigger { type TriggerConfig = WebsocketConfig; @@ -28,6 +57,8 @@ impl TriggerCrud for WebsocketTrigger { const TABLE_NAME: &'static str = "websocket_trigger"; const TRIGGER_TYPE: &'static str = "websocket"; + const DRAFT_KIND: windmill_common::user_drafts::UserDraftItemKind = + windmill_common::user_drafts::UserDraftItemKind::TriggerWebsocket; const SUPPORTS_SERVER_STATE: bool = true; const SUPPORTS_TEST_CONNECTION: bool = true; const ROUTE_PREFIX: &'static str = "/websocket_triggers"; @@ -60,6 +91,13 @@ impl TriggerCrud for WebsocketTrigger { )); } + // Reject SSRF targets at save time for static URLs. A `$flow:`/`$script:` + // URL is only known at runtime, so it is validated at connect time + // instead (in the listener and test handler). + if !config.url.starts_with('$') { + validate_websocket_url_for_ssrf(&config.url).await?; + } + if let Some(args) = &config.url_runnable_args { if !args.is_object() { return Err(Error::BadRequest( @@ -92,6 +130,7 @@ impl TriggerCrud for WebsocketTrigger { w_id: &str, trigger: TriggerData, ) -> Result<()> { + check_secondary_runnable_scopes(authed, &trigger.config)?; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); let filters = trigger @@ -169,6 +208,7 @@ impl TriggerCrud for WebsocketTrigger { path: &str, trigger: TriggerData, ) -> Result<()> { + check_secondary_runnable_scopes(authed, &trigger.config)?; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); let filters = trigger @@ -276,6 +316,8 @@ impl TriggerCrud for WebsocketTrigger { Cow::Borrowed(&url) }; + validate_websocket_url_for_ssrf(&connect_url).await?; + connect_async_with_proxy(&*connect_url) .await .map_err(|err| { diff --git a/backend/windmill-trigger-websocket/src/lib.rs b/backend/windmill-trigger-websocket/src/lib.rs index e61c067479..4111606b2b 100644 --- a/backend/windmill-trigger-websocket/src/lib.rs +++ b/backend/windmill-trigger-websocket/src/lib.rs @@ -104,6 +104,58 @@ pub fn value_to_args_hashmap( Ok(args) } +/// Env var that opts a deployment out of SSRF validation for WebSocket trigger +/// URLs, permitting connections to private/internal addresses. Off by default. +pub const ALLOW_PRIVATE_WEBSOCKET_URLS_ENV: &str = "ALLOW_PRIVATE_WEBSOCKET_URLS"; + +/// Reject WebSocket URLs that target (or resolve to) a private/internal address, +/// blocking SSRF probes of the host's internal network and cloud metadata +/// endpoints. +/// +/// `ws://`/`wss://` are mapped to `http`/`https` so the shared +/// `validate_url_for_ssrf` host + DNS-resolution checks apply. The +/// security-critical call sites are the outbound connects (the test handler and +/// every listener (re)connect): validating the *resolved* URL there means a +/// `$flow:`/`$script:` URL is checked on its returned value and re-checked on +/// each reconnect (DNS rebinding). `validate_config` also calls this at save +/// time to reject static URLs early. +pub async fn validate_websocket_url_for_ssrf(url: &str) -> Result<()> { + if std::env::var(ALLOW_PRIVATE_WEBSOCKET_URLS_ENV) + .ok() + .is_some_and(|v| v == "true" || v == "1") + { + return Ok(()); + } + + // `ws`/`wss` aren't recognised by `validate_url_for_ssrf`'s scheme check, so + // map them to the http(s) equivalent the same connection would tunnel over. + // The prefixes are ASCII, so byte-slicing at their length stays on a char + // boundary. + let lower = url.to_ascii_lowercase(); + let http_url = if lower.starts_with("wss://") { + format!("https://{}", &url["wss://".len()..]) + } else if lower.starts_with("ws://") { + format!("http://{}", &url["ws://".len()..]) + } else { + url.to_string() + }; + + windmill_common::ssrf::validate_url_for_ssrf(&http_url) + .await + .map_err(|e| match e { + // The env-var hint is only actionable for a well-formed URL blocked + // for targeting a private address; a malformed URL or bad scheme + // surfaces its real error so the user fixes the URL (see #9171). + e @ windmill_common::ssrf::SsrfValidationError::Private { .. } => { + Error::BadRequest(format!( + "{e}. If you need to connect to private/internal WebSocket endpoints, \ + set the {ALLOW_PRIVATE_WEBSOCKET_URLS_ENV}=true environment variable" + )) + } + e => Error::from(e), + }) +} + pub async fn get_url_from_runnable_value( path: &str, is_flow: bool, @@ -144,3 +196,40 @@ pub async fn get_url_from_runnable_value( )) }) } + +#[cfg(test)] +mod tests { + use super::*; + + #[tokio::test] + async fn ssrf_blocks_private_and_metadata_ws_urls() { + // ws:// → http:// mapping must still reach the IP-literal block. + let err = validate_websocket_url_for_ssrf("ws://127.0.0.1:6379/") + .await + .unwrap_err(); + assert!(matches!(err, Error::BadRequest(_))); + // Private errors carry the opt-out hint so operators can allow internal + // targets deliberately. + assert!(err.to_string().contains(ALLOW_PRIVATE_WEBSOCKET_URLS_ENV)); + + // wss:// → https:// mapping blocks the cloud metadata endpoint. + assert!( + validate_websocket_url_for_ssrf("wss://169.254.169.254/latest/meta-data") + .await + .is_err() + ); + assert!(validate_websocket_url_for_ssrf("ws://10.0.0.5:6379/") + .await + .is_err()); + } + + #[tokio::test] + async fn ssrf_rejects_non_ws_scheme_without_private_hint() { + // A non-ws scheme isn't mapped and fails the scheme check; it must not + // get the "set ALLOW_PRIVATE_WEBSOCKET_URLS" hint (issue #9171). + let err = validate_websocket_url_for_ssrf("file:///etc/passwd") + .await + .unwrap_err(); + assert!(!err.to_string().contains(ALLOW_PRIVATE_WEBSOCKET_URLS_ENV)); + } +} diff --git a/backend/windmill-trigger-websocket/src/listener.rs b/backend/windmill-trigger-websocket/src/listener.rs index 54dd4ddf60..d46faf1b1a 100644 --- a/backend/windmill-trigger-websocket/src/listener.rs +++ b/backend/windmill-trigger-websocket/src/listener.rs @@ -1,5 +1,6 @@ use super::{ - get_url_from_runnable_value, proxy::connect_async_with_proxy, WebsocketConfig, WebsocketTrigger, + get_url_from_runnable_value, proxy::connect_async_with_proxy, validate_websocket_url_for_ssrf, + WebsocketConfig, WebsocketTrigger, }; use anyhow::Context; use async_trait::async_trait; @@ -173,6 +174,8 @@ impl Listener for WebsocketTrigger { Cow::Borrowed(&url) }; + validate_websocket_url_for_ssrf(&connect_url).await?; + let connection = connect_async_with_proxy(&*connect_url) .await .map(|conn| Some(conn)) @@ -506,7 +509,7 @@ impl Clone for ReturnMessageChannels { } #[derive(Debug, Deserialize)] -enum InitialMessage { +pub(crate) enum InitialMessage { #[serde(rename = "raw_message")] RawMessage(String), #[serde(rename = "runnable_result")] diff --git a/backend/windmill-trigger/src/global_handler.rs b/backend/windmill-trigger/src/global_handler.rs index 3fbbcddd0e..98b689dda9 100644 --- a/backend/windmill-trigger/src/global_handler.rs +++ b/backend/windmill-trigger/src/global_handler.rs @@ -14,7 +14,7 @@ use windmill_api_jobs::execution::cancel_jobs; use windmill_common::{ db::{UserDB, DB}, error::{self, Error, Result}, - jobs::JobTriggerKind, + jobs::{delete_jobs, JobTriggerKind}, triggers::TriggerMetadata, }; @@ -262,9 +262,7 @@ pub async fn resume_suspended_trigger_jobs( .execute(&mut *tx) .await?; - sqlx::query!("DELETE FROM v2_job WHERE id = $1", job.id) - .execute(&mut *tx) - .await?; + delete_jobs(&mut *tx, &[job.id]).await?; } } diff --git a/backend/windmill-trigger/src/handler.rs b/backend/windmill-trigger/src/handler.rs index 3591c88931..65072bc9ab 100644 --- a/backend/windmill-trigger/src/handler.rs +++ b/backend/windmill-trigger/src/handler.rs @@ -16,6 +16,10 @@ use windmill_api_auth::{check_scopes, ApiAuthed}; use windmill_common::{ db::UserDB, error::{Error, JsonResult, Result}, + user_drafts::{ + delete_all_drafts_for_path, delete_own_draft_for_path, fetch_draft_only_list_rows, + overlay_or_draft_only, UserDraftItemKind, WithDraftOverlay, WithDraftQuery, + }, utils::{paginate, Pagination, StripPath}, worker::CLOUD_HOSTED, DB, @@ -60,7 +64,10 @@ pub trait TriggerCrud: Send + Sync + 'static { + for<'r> FromRow<'r, sqlx::postgres::PgRow> + Send + Sync - + Unpin; + + Unpin + // `'static` so the deployed trigger can be boxed into + // `WithDraftOverlay`'s erased-serde inner (it's an owned row). + + 'static; type TriggerConfig: Debug + DeserializeOwned @@ -77,6 +84,9 @@ pub trait TriggerCrud: Send + Sync + 'static { /// constant set by each trigger impl — it is never user-controllable. const TABLE_NAME: &'static str; const TRIGGER_TYPE: &'static str; + /// `UserDraftItemKind` for this trigger's per-user `draft` rows. Required (no + /// default) so a trigger that forgets it is a compile error, not a runtime panic. + const DRAFT_KIND: UserDraftItemKind; const SUPPORTS_SERVER_STATE: bool; const SUPPORTS_TEST_CONNECTION: bool; const ROUTE_PREFIX: &'static str; @@ -127,6 +137,11 @@ pub trait TriggerCrud: Send + Sync + 'static { &Self::ROUTE_PREFIX[1..] } + /// Accessor for `DRAFT_KIND` used at the draft-lookup call sites. + fn user_draft_item_kind() -> UserDraftItemKind { + Self::DRAFT_KIND + } + async fn create_trigger( &self, db: &DB, @@ -349,11 +364,14 @@ pub trait TriggerCrud: Send + Sync + 'static { count } + /// `authed_email = Some` adds the per-user `is_draft` flag (scalar EXISTS); + /// `None` (e.g. workspace export) leaves it omitted. async fn list_triggers( &self, tx: &mut PgConnection, workspace_id: &str, query: Option<&StandardTriggerQuery>, + authed_email: Option<&str>, ) -> Result> { let mut fields = vec![ "workspace_id", @@ -381,6 +399,19 @@ pub trait TriggerCrud: Send + Sync + 'static { .order_by("edited_at", true) .and_where("workspace_id = ?".bind(&workspace_id)); + if let Some(email) = authed_email { + // SAFETY: interpolated TABLE_NAME and draft kind are compile-time constants; email is bound. + sqlb.field( + &format!( + "EXISTS(SELECT 1 FROM draft WHERE draft.workspace_id = {t}.workspace_id \ + AND draft.path = {t}.path AND draft.typ = '{k}' AND draft.email = ?) as is_draft", + t = Self::TABLE_NAME, + k = Self::user_draft_item_kind().as_str(), + ) + .bind(&email), + ); + } + if let Some(query) = query { let (per_page, offset) = paginate(Pagination { per_page: query.per_page, page: query.page }); @@ -563,15 +594,74 @@ async fn list_triggers( Extension(handler): Extension>, authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path(workspace_id): Path, Query(query): Query, ) -> JsonResult> { let mut tx = user_db.begin(&authed).await?; - let triggers = handler - .list_triggers(&mut *tx, &workspace_id, Some(&query)) + let mut triggers = handler + .list_triggers(&mut *tx, &workspace_id, Some(&query), Some(&authed.email)) .await?; tx.commit().await?; + // Append the authed user's draft-only triggers of this kind; see scripts.rs. + // Best-effort: the editor's TriggerData shape overlaps T::Trigger but a per-kind + // config can deviate, so drop a row on deserialize failure rather than fail the list. + if query.include_draft_only.unwrap_or(false) + && !authed.is_operator + && query.page.unwrap_or(0) == 0 + && query.path.is_none() + && query.is_flow.is_none() + && query.path_start.is_none() + && query.label.is_none() + { + let draft_only_rows = fetch_draft_only_list_rows( + &db, + &workspace_id, + &authed.email, + T::user_draft_item_kind(), + ) + .await?; + + for row in draft_only_rows { + let created_at = row.created_at; + let v: serde_json::Value = match serde_json::from_str(row.value.0.get()) { + Ok(v) => v, + Err(_) => continue, + }; + let serde_json::Value::Object(mut map) = v else { + continue; + }; + // Fill operational fields the editor draft omits so the merged JSON matches + // `Trigger`'s flattened shape (mode derived from `enabled`). + map.insert( + "workspace_id".into(), + serde_json::Value::String(workspace_id.clone()), + ); + map.insert("edited_by".into(), serde_json::Value::String(String::new())); + if let Ok(at) = serde_json::to_value(&created_at) { + map.insert("edited_at".into(), at); + } + map.entry("permissioned_as") + .or_insert(serde_json::Value::String(String::new())); + map.entry("extra_perms").or_insert(serde_json::Value::Null); + if !map.contains_key("mode") { + let enabled = map.get("enabled").and_then(|x| x.as_bool()).unwrap_or(true); + map.insert( + "mode".into(), + serde_json::Value::String(if enabled { "enabled" } else { "disabled" }.into()), + ); + } + map.insert("draft_only".into(), serde_json::Value::Bool(true)); + // Synthesized rows are the authed user's draft. + map.insert("is_draft".into(), serde_json::Value::Bool(true)); + match serde_json::from_value::(serde_json::Value::Object(map)) { + Ok(t) => triggers.push(t), + Err(_) => continue, + } + } + } + Ok(Json(triggers)) } @@ -579,21 +669,41 @@ async fn get_trigger( Extension(handler): Extension>, authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path((workspace_id, path)): Path<(String, StripPath)>, -) -> JsonResult { + Query(q): Query, +) -> JsonResult { let path = path.to_path(); check_scopes(&authed, || { format!("{}:read:{}", T::scope_domain_name(), &path) })?; let mut tx = user_db.begin(&authed).await?; - let trigger = handler + let trigger_res = handler .get_trigger_by_path(&mut *tx, &workspace_id, path) - .await?; - + .await; tx.commit().await?; - Ok(Json(trigger)) + // Map "no deployed trigger" to `None` and let the shared choreography + // handle the draft overlay / draft-only fallback / 404. + let deployed = match trigger_res { + Ok(t) => Some(t), + Err(Error::NotFound(_)) => None, + Err(e) => return Err(e), + }; + + let overlay = overlay_or_draft_only( + &db, + &workspace_id, + &authed.email, + T::user_draft_item_kind(), + path, + q.get_draft, + deployed, + || Error::NotFound(format!("Trigger not found at path: {}", path)), + ) + .await?; + Ok(Json(overlay)) } async fn update_trigger( @@ -717,6 +827,19 @@ async fn update_trigger( tx.commit().await?; + // On rename the old-path draft orphans (no SQL FK); clear the deployer's own + // (+ legacy NULL) there, teammates keep theirs (StaleDraftModal). See scripts.rs. + if path != new_path { + delete_own_draft_for_path( + &db, + &workspace_id, + T::user_draft_item_kind(), + path, + &authed.email, + ) + .await?; + } + Ok(format!("Trigger '{}' updated", path)) } @@ -724,6 +847,7 @@ async fn delete_trigger( Extension(handler): Extension>, authed: ApiAuthed, Extension(user_db): Extension, + Extension(db): Extension, Path((workspace_id, path)): Path<(String, StripPath)>, ) -> Result { let path = path.to_path(); @@ -781,6 +905,9 @@ async fn delete_trigger( tx.commit().await?; + // Trigger gone for everyone: wipe ALL users' drafts at this path; see scripts.rs. + delete_all_drafts_for_path(&db, &workspace_id, T::user_draft_item_kind(), path).await?; + Ok(format!("Trigger '{}' deleted", path)) } @@ -930,6 +1057,10 @@ async fn test_connection( Path(workspace_id): Path, Json(config): Json, ) -> Result<()> { + // Test connection opens an outbound connection to a caller-supplied target, + // so gate it behind write access like the other mutating trigger routes. + check_scopes(&authed, || format!("{}:write", T::scope_domain_name()))?; + let connect_f = async move { handler .test_connection(&db, &authed, &user_db, &workspace_id, config) diff --git a/backend/windmill-trigger/src/trigger_helpers.rs b/backend/windmill-trigger/src/trigger_helpers.rs index 44e5884420..1427f7f102 100644 --- a/backend/windmill-trigger/src/trigger_helpers.rs +++ b/backend/windmill-trigger/src/trigger_helpers.rs @@ -924,6 +924,8 @@ async fn trigger_script_with_retry_and_error_handler<'c>( path, hash: Some(hash), flow_version: None, + // Keep the flow path until native retry covers handler semantics. + language: None, args: HashMap::from(&push_args), retry, error_handler_path, diff --git a/backend/windmill-trigger/src/types.rs b/backend/windmill-trigger/src/types.rs index 8b42c5b1f9..77d90bea20 100644 --- a/backend/windmill-trigger/src/types.rs +++ b/backend/windmill-trigger/src/types.rs @@ -27,6 +27,10 @@ pub struct StandardTriggerQuery { pub is_flow: Option, pub path_start: Option, pub label: Option, + /// When true, append per-user draft rows whose path has no + /// deployed trigger of this kind. Same gate as scripts/flows/apps: + /// non-operators, offset 0, no narrowing filters. + pub include_draft_only: Option, } #[derive(Debug, FromRow, Clone, Serialize, Deserialize)] @@ -42,6 +46,22 @@ pub struct BaseTrigger { pub extra_perms: Option, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + /// True when this row is a per-user draft with no deployed trigger + /// at the same path. Set by `list_triggers` when the response + /// includes synthesized draft-only rows (gated on + /// `include_draft_only`). Always `None`/omitted on deployed rows + /// fetched from the trigger table. + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub draft_only: Option, + /// True when the authed user has a per-user draft at this path — + /// either layered over a deployed trigger (EXISTS subquery in the + /// list SQL) or a synthesized draft-only row. Drives the `*` suffix + /// on the trigger list pages. `None`/omitted for callers that list + /// without an authed context (e.g. workspace export). + #[serde(skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_draft: Option, } #[derive(Debug, FromRow, Clone, Serialize, Deserialize)] @@ -192,6 +212,7 @@ impl Default for StandardTriggerQuery { path_start: None, is_flow: None, label: None, + include_draft_only: None, } } } @@ -260,6 +281,7 @@ mod tests { is_flow: None, path_start: None, label: None, + include_draft_only: None, }; assert_eq!(q.offset(), 100); assert_eq!(q.limit(), 50); @@ -274,6 +296,7 @@ mod tests { is_flow: None, path_start: None, label: None, + include_draft_only: None, }; assert_eq!(q.offset(), 0); assert_eq!(q.limit(), 100); diff --git a/backend/windmill-types/src/assets.rs b/backend/windmill-types/src/assets.rs index be20cfae3e..fb624382c4 100644 --- a/backend/windmill-types/src/assets.rs +++ b/backend/windmill-types/src/assets.rs @@ -16,6 +16,23 @@ pub enum AssetKind { Volume, } +impl AssetKind { + /// The canonical URI prefix used in asset trigger refs (e.g. `s3://`, + /// `$res:`). Single source of truth for both trigger-ref construction + /// and runtime cascade dispatch. `Variable` is deprecated and has no + /// canonical ref, so it returns `None`. + pub fn canonical_prefix(&self) -> Option<&'static str> { + match self { + AssetKind::S3Object => Some("s3://"), + AssetKind::Resource => Some("$res:"), + AssetKind::Ducklake => Some("ducklake://"), + AssetKind::DataTable => Some("datatable://"), + AssetKind::Volume => Some("volume://"), + AssetKind::Variable => None, + } + } +} + #[derive( Serialize, Deserialize, Debug, PartialEq, Copy, Clone, Hash, Eq, sqlx::Type, PartialOrd, Ord, )] diff --git a/backend/windmill-types/src/flows.rs b/backend/windmill-types/src/flows.rs index 14e2a14fe6..006185dbd9 100644 --- a/backend/windmill-types/src/flows.rs +++ b/backend/windmill-types/src/flows.rs @@ -30,8 +30,6 @@ pub struct Flow { pub schema: Option, pub extra_perms: serde_json::Value, #[serde(skip_serializing_if = "Option::is_none")] - pub draft_only: Option, - #[serde(skip_serializing_if = "Option::is_none")] pub dedicated_worker: Option, #[serde(skip_serializing_if = "Option::is_none")] pub tag: Option, @@ -83,7 +81,9 @@ pub struct ListableFlow { pub archived: bool, pub extra_perms: serde_json::Value, pub starred: bool, - pub has_draft: bool, + /// `Some(true)` only on synthesised draft-only rows; `None` on deployed rows. + /// See ListableScript in scripts.rs. + #[sqlx(default)] #[serde(skip_serializing_if = "Option::is_none")] pub draft_only: Option, #[serde(skip_serializing_if = "Option::is_none")] @@ -93,6 +93,20 @@ pub struct ListableFlow { pub deployment_msg: Option, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + /// True when the authed user has a draft for this flow (draft-only or layered + /// over the deployed row). See ListableScript in scripts.rs. + #[serde(default)] + pub is_draft: bool, + /// User-typed staged path from the draft JSON's `draft_path`; `None` = unchanged. + /// See ListableScript in scripts.rs. + #[sqlx(default)] + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_path: Option, + /// Per-path draft owners driving the home-page avatar circles. + /// See ListableScript in scripts.rs. + #[sqlx(default)] + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_users: Option>>, /// Labels inherited from the parent folder, computed at read time. #[sqlx(default)] #[serde(skip_serializing_if = "Option::is_none")] @@ -107,7 +121,6 @@ pub struct NewFlow { #[serde(deserialize_with = "validate_flow_value")] pub value: Box, pub schema: Option, - pub draft_only: Option, pub tag: Option, pub dedicated_worker: Option, pub timeout: Option, @@ -142,6 +155,19 @@ fn validate_retry(retry: &Retry, module_id: &str) -> anyhow::Result<()> { Ok(()) } +/// Script/sub-flow step references must be workspace paths (`u/`, `f/`, `g/`) or a hub +/// reference (`hub/`). Empty is tolerated for intermediate/incomplete steps. This blocks +/// absolute or local filesystem paths (e.g. `/tmp/.../ops/scripts/...` baked in by a +/// `wmill sync push` from a feature-branch checkout) from being persisted into a flow, +/// where they silently mis-resolve to an unrelated script at runtime (#9751). +fn is_workspace_runnable_path(path: &str) -> bool { + path.is_empty() + || path.starts_with("u/") + || path.starts_with("f/") + || path.starts_with("g/") + || path.starts_with("hub/") +} + fn validate_flow_value<'de, D>(deserializer: D) -> Result, D::Error> where D: Deserializer<'de>, @@ -151,21 +177,38 @@ where let flow_value: FlowValue = serde_json::from_str(raw_value.get()) .map_err(|e| serde::de::Error::custom(format!("Invalid flow value: {}", e)))?; - FlowModule::traverse_modules(&flow_value.modules, &mut |module| { + let mut validate_module = |module: &FlowModule| -> anyhow::Result<()> { if let Some(ref retry) = module.retry { validate_retry(retry, &module.id)?; } - return Ok(()); - }) - .map_err(|e| serde::de::Error::custom(e.to_string()))?; + if let Ok(FlowModuleValue::Script { path, .. } | FlowModuleValue::Flow { path, .. }) = + module.get_value() + { + if !is_workspace_runnable_path(&path) { + return Err(anyhow::anyhow!( + "step '{}' references '{}', which is not a workspace path (expected u/, \ + f/, g/ or hub/). Absolute or local filesystem paths are not allowed in \ + flow steps.", + module.id, + path + )); + } + } + Ok(()) + }; - if let Some(ref _failure_module) = flow_value.failure_module { - //add validation logic here for failure module - } - - if let Some(ref _preprocessor_module) = flow_value.preprocessor_module { - //add validation logic here for preprocessor module - } + // The API is the authoritative guard (it can be called directly, bypassing the CLI), so + // it must cover every step that resolves a path: the main modules AND the failure / + // preprocessor modules (which can themselves be sub-flows/loops/branches). + let extra_modules: Vec = flow_value + .failure_module + .iter() + .chain(flow_value.preprocessor_module.iter()) + .map(|m| (**m).clone()) + .collect(); + FlowModule::traverse_modules(&flow_value.modules, &mut validate_module) + .and_then(|()| FlowModule::traverse_modules(&extra_modules, &mut validate_module)) + .map_err(|e| serde::de::Error::custom(e.to_string()))?; Ok(raw_value) } @@ -322,6 +365,7 @@ impl Step { pub struct StopAfterIf { pub expr: String, pub skip_if_stopped: bool, + #[serde(default, skip_serializing_if = "Option::is_none")] pub error_message: Option, /// When stopping with an error (`error_message` set), embed the stopping /// step's own result inside the raised error object (as `error.result`) @@ -338,7 +382,9 @@ pub struct RetryIf { #[derive(Deserialize, Serialize, Debug, Clone, Default, PartialEq)] #[serde(default)] pub struct Retry { + #[serde(skip_serializing_if = "is_default")] pub constant: ConstantDelay, + #[serde(skip_serializing_if = "is_default")] pub exponential: ExponentialDelay, #[serde(skip_serializing_if = "Option::is_none")] pub retry_if: Option, @@ -1212,6 +1258,108 @@ mod tests { assert_eq!(val.modules.len(), 1); } + #[test] + fn flow_rejects_absolute_step_path() { + // #9751: an absolute local path baked into a step must be rejected on deploy. + let bad = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "validate_onboard_target", + "value": { + "type": "script", + "path": "/tmp/tmp.X/f/ops/scripts/clean_device/pre_clean", + "input_transforms": {} + } + }]} + }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "unexpected error: {err}" + ); + assert!( + err.contains("validate_onboard_target"), + "error should name the step: {err}" + ); + } + + #[test] + fn flow_rejects_absolute_step_path_in_nested_module() { + let bad = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "loop", + "value": { + "type": "forloopflow", + "iterator": {"type": "javascript", "expr": "[1]"}, + "modules": [{ + "id": "inner", + "value": {"type": "script", "path": "/abs/path", "input_transforms": {}} + }] + } + }]} + }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "unexpected error: {err}" + ); + } + + #[test] + fn flow_rejects_absolute_path_in_failure_and_preprocessor_modules() { + for slot in ["failure_module", "preprocessor_module"] { + // Build the value with the slot as an explicit (interpolated) key. + let mut value = serde_json::Map::new(); + value.insert("modules".to_string(), json!([])); + value.insert( + slot.to_string(), + json!({ + "id": slot, + "value": {"type": "script", "path": "/abs/path", "input_transforms": {}} + }), + ); + let bad = json!({ "path": "f/test/flow", "summary": "", "value": value }); + let err = serde_json::from_value::(bad) + .unwrap_err() + .to_string(); + assert!( + err.contains("not a workspace path"), + "{slot} should be validated, got: {err}" + ); + } + } + + #[test] + fn flow_accepts_workspace_step_paths() { + for p in [ + "f/ops/scripts/x", + "u/me/y", + "g/grp/z", + "hub/123/foo", + "", // tolerated for incomplete steps + ] { + let ok = json!({ + "path": "f/test/flow", + "summary": "", + "value": { "modules": [{ + "id": "a", + "value": {"type": "script", "path": p, "input_transforms": {}} + }]} + }); + assert!( + serde_json::from_value::(ok).is_ok(), + "path {p:?} should be accepted" + ); + } + } + #[test] fn ai_agent_omit_output_from_conversation_defaults_to_false() { let input = json!({ @@ -1281,4 +1429,47 @@ mod tests { let output = serde_json::to_string(&val).unwrap(); assert!(!output.contains("tag")); } + + #[test] + fn retry_omits_default_constant_and_exponential() { + // A constant-only retry must not materialize a default exponential block + // on serialization (and vice-versa). Round-trips through Retry used to + // emit seconds:0 / random_factor:null, which the CLI linter rejected. + let input = json!({ "constant": { "attempts": 1, "seconds": 60 } }); + let retry: Retry = serde_json::from_value(input).unwrap(); + + // Deserialization still fills in defaults in memory. + assert_eq!(retry.exponential, ExponentialDelay::default()); + + let output = serde_json::to_value(&retry).unwrap(); + assert!(output.get("constant").is_some()); + assert!(output.get("exponential").is_none()); + + // A fully-default retry serializes to an empty object. + let empty = serde_json::to_value(&Retry::default()).unwrap(); + assert_eq!(empty, json!({})); + } + + #[test] + fn stop_after_if_omits_null_error_message() { + let input = json!({ "expr": "result == 404", "skip_if_stopped": true }); + let stop: StopAfterIf = serde_json::from_value(input).unwrap(); + assert!(stop.error_message.is_none()); + + let output = serde_json::to_value(&stop).unwrap(); + assert!(output.get("error_message").is_none()); + + // A set error message still round-trips. + let with_msg = StopAfterIf { + expr: "true".to_string(), + skip_if_stopped: false, + error_message: Some("boom".to_string()), + error_include_result: false, + }; + let output = serde_json::to_value(&with_msg).unwrap(); + assert_eq!( + output.get("error_message").and_then(|v| v.as_str()), + Some("boom") + ); + } } diff --git a/backend/windmill-types/src/jobs.rs b/backend/windmill-types/src/jobs.rs index 75cde586e9..448e6f4d01 100644 --- a/backend/windmill-types/src/jobs.rs +++ b/backend/windmill-types/src/jobs.rs @@ -43,6 +43,9 @@ pub enum JobTriggerKind { #[serde(rename = "ci_test")] #[sqlx(rename = "ci_test")] CiTest, + // A run dispatched because an upstream pipeline script wrote an asset + // this runnable subscribes to via `// on s3://...` annotations. + Asset, } impl std::fmt::Display for JobTriggerKind { @@ -64,6 +67,7 @@ impl std::fmt::Display for JobTriggerKind { JobTriggerKind::Google => "google", JobTriggerKind::Github => "github", JobTriggerKind::CiTest => "ci_test", + JobTriggerKind::Asset => "asset", }; write!(f, "{}", kind) } @@ -228,6 +232,14 @@ pub struct QueuedJob { pub runnable_settings_handle: Option, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + // True when this job is a native retry attempt (has a native_retry_attempt + // marker). Lets the run-page chain distinguish real retries from other + // same-script children (e.g. WAC inline children). The list and single-job + // GET endpoints select it; `#[sqlx(default)]` lets any other query omit the + // column and default to None. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_retry: Option, } impl QueuedJob { @@ -303,6 +315,7 @@ impl Default for QueuedJob { preprocessed: None, runnable_settings_handle: None, labels: None, + is_retry: None, } } } @@ -358,6 +371,14 @@ pub struct CompletedJob { pub labels: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub preprocessed: Option, + // True when this job is a native retry attempt (has a native_retry_attempt + // marker). Lets the run-page chain distinguish real retries from other + // same-script children (e.g. WAC inline children). The list and single-job + // GET endpoints select it; `#[sqlx(default)]` lets any other query omit the + // column and default to None. + #[serde(default, skip_serializing_if = "Option::is_none")] + #[sqlx(default)] + pub is_retry: Option, } impl CompletedJob { @@ -469,6 +490,10 @@ pub enum JobPayload { path: String, hash: Option, flow_version: Option, + // Set when wrapping a script (not a flow). Lets `push` materialize a + // bare-script-with-retry as a native retryable `Script` job instead of + // spawning a one-step flow. + language: Option, args: HashMap>, retry: Option, error_handler_path: Option, diff --git a/backend/windmill-types/src/lib.rs b/backend/windmill-types/src/lib.rs index 4d9b96c2af..9144d61f89 100644 --- a/backend/windmill-types/src/lib.rs +++ b/backend/windmill-types/src/lib.rs @@ -19,6 +19,8 @@ pub mod schedule; pub mod scripts; #[cfg(not(target_arch = "wasm32"))] pub mod triggers; +#[cfg(not(target_arch = "wasm32"))] +pub mod user_drafts; /// Duplicated from windmill-common::worker::to_raw_value. /// windmill-types cannot depend on windmill-common (it would be circular). diff --git a/backend/windmill-types/src/runnable_settings.rs b/backend/windmill-types/src/runnable_settings.rs index dfff0df79e..ea0fd30dc0 100644 --- a/backend/windmill-types/src/runnable_settings.rs +++ b/backend/windmill-types/src/runnable_settings.rs @@ -1,9 +1,75 @@ use serde::{Deserialize, Serialize}; +use crate::flows::{ConstantDelay, ExponentialDelay, Retry, RetryIf}; + #[derive(Deserialize, Clone, Copy, Serialize, Default, Hash)] pub struct RunnableSettings { pub debouncing_settings: Option, pub concurrency_settings: Option, + pub retry_settings: Option, +} + +/// Flattened, dedup-friendly representation of a [`Retry`] policy. Native script +/// retry stores the policy here (via `runnable_settings_handle`) instead of +/// wrapping the script in a one-step flow. +#[derive( + Debug, Default, Clone, Serialize, Deserialize, Hash, PartialEq, sqlx::FromRow, sqlx::Decode, +)] +pub struct RetrySettings { + #[serde(skip_serializing_if = "Option::is_none")] + pub constant_attempts: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub constant_seconds: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_attempts: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_multiplier: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_seconds: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub exponential_random_factor: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub retry_if_expr: Option, +} + +impl From<&Retry> for RetrySettings { + fn from(r: &Retry) -> Self { + Self { + // attempts are u32; saturate the narrowing to i32 (the seconds/ + // multiplier/random_factor fields are u16/i8 and can't overflow i32). + constant_attempts: Some(r.constant.attempts.min(i32::MAX as u32) as i32), + constant_seconds: Some(r.constant.seconds as i32), + exponential_attempts: Some(r.exponential.attempts.min(i32::MAX as u32) as i32), + exponential_multiplier: Some(r.exponential.multiplier as i32), + exponential_seconds: Some(r.exponential.seconds as i32), + exponential_random_factor: r.exponential.random_factor.map(|x| x as i32), + retry_if_expr: r.retry_if.as_ref().map(|x| x.expr.clone()), + } + } +} + +impl From for Retry { + fn from(s: RetrySettings) -> Self { + Retry { + constant: ConstantDelay { + attempts: s.constant_attempts.unwrap_or(0).max(0) as u32, + seconds: s.constant_seconds.unwrap_or(0).clamp(0, u16::MAX as i32) as u16, + }, + exponential: ExponentialDelay { + attempts: s.exponential_attempts.unwrap_or(0).max(0) as u32, + // Mirror ExponentialDelay::default().multiplier (1) when absent. + multiplier: s + .exponential_multiplier + .unwrap_or(1) + .clamp(0, u16::MAX as i32) as u16, + seconds: s.exponential_seconds.unwrap_or(0).clamp(0, u16::MAX as i32) as u16, + random_factor: s + .exponential_random_factor + .map(|x| x.clamp(i8::MIN as i32, i8::MAX as i32) as i8), + }, + retry_if: s.retry_if_expr.map(|expr| RetryIf { expr }), + } + } } // TODO: Add validation logic. @@ -111,3 +177,56 @@ impl From for ConcurrencySettings { } } } + +#[cfg(test)] +mod tests { + use super::*; + use crate::flows::{ConstantDelay, ExponentialDelay, RetryIf}; + + #[test] + fn retry_settings_roundtrips_retry() { + let cases = [ + // constant only + Retry { + constant: ConstantDelay { attempts: 3, seconds: 5 }, + exponential: ExponentialDelay::default(), + retry_if: None, + }, + // exponential with jitter + Retry { + constant: ConstantDelay::default(), + exponential: ExponentialDelay { + attempts: 4, + multiplier: 2, + seconds: 3, + random_factor: Some(20), + }, + retry_if: None, + }, + // mixed + retry_if + max-ish narrowings + Retry { + constant: ConstantDelay { attempts: 1, seconds: u16::MAX }, + exponential: ExponentialDelay { + attempts: 2, + multiplier: u16::MAX, + seconds: 7, + random_factor: Some(i8::MIN), + }, + retry_if: Some(RetryIf { expr: "result.error.code != 'fatal'".to_string() }), + }, + ]; + for r in cases { + let back: Retry = RetrySettings::from(&r).into(); + assert_eq!(back, r, "RetrySettings round-trip must preserve {r:?}"); + } + } + + #[test] + fn retry_settings_default_maps_to_default_exponential() { + // All-None settings must mirror ExponentialDelay::default() (multiplier 1), + // so a row with no exponential values doesn't decode to a 0 multiplier. + let r: Retry = RetrySettings::default().into(); + assert_eq!(r, Retry::default()); + assert_eq!(r.exponential.multiplier, 1); + } +} diff --git a/backend/windmill-types/src/s3.rs b/backend/windmill-types/src/s3.rs index 0ab1794d7e..5a7634043c 100644 --- a/backend/windmill-types/src/s3.rs +++ b/backend/windmill-types/src/s3.rs @@ -364,13 +364,11 @@ mod tests { assert_eq!(deserialized, S3Permission::READ | S3Permission::WRITE); // Unknown permissions are silently ignored - let deserialized: S3Permission = - serde_json::from_str("\"read,unknown,delete\"").unwrap(); + let deserialized: S3Permission = serde_json::from_str("\"read,unknown,delete\"").unwrap(); assert_eq!(deserialized, S3Permission::READ | S3Permission::DELETE); // All four permissions - let all: S3Permission = - serde_json::from_str("\"read,write,delete,list\"").unwrap(); + let all: S3Permission = serde_json::from_str("\"read,write,delete,list\"").unwrap(); assert_eq!( all, S3Permission::READ | S3Permission::WRITE | S3Permission::DELETE | S3Permission::LIST @@ -409,20 +407,15 @@ mod tests { ); // Region set, endpoint empty → use region - let with_region = S3Resource { - region: "ap-southeast-1".to_string(), - ..resource.clone() - }; + let with_region = S3Resource { region: "ap-southeast-1".to_string(), ..resource.clone() }; assert_eq!( with_region.endpoint_with_region_fallback(Some("ignored".to_string())), "s3.ap-southeast-1.amazonaws.com" ); // Endpoint set → return as-is - let with_endpoint = S3Resource { - endpoint: "custom.s3.endpoint.com".to_string(), - ..resource.clone() - }; + let with_endpoint = + S3Resource { endpoint: "custom.s3.endpoint.com".to_string(), ..resource.clone() }; assert_eq!( with_endpoint.endpoint_with_region_fallback(Some("ignored".to_string())), "custom.s3.endpoint.com" @@ -431,10 +424,8 @@ mod tests { #[test] fn test_lfs_methods_filesystem() { - let rules = vec![S3PermissionRule { - pattern: "**/*.csv".to_string(), - allow: S3Permission::READ, - }]; + let rules = + vec![S3PermissionRule { pattern: "**/*.csv".to_string(), allow: S3Permission::READ }]; let lfs = LargeFileStorage::FilesystemStorage(FilesystemStorage { root_path: "/data/workspace".to_string(), public_resource: Some(true), diff --git a/backend/windmill-types/src/scripts.rs b/backend/windmill-types/src/scripts.rs index a777d69790..5e5c7f313e 100644 --- a/backend/windmill-types/src/scripts.rs +++ b/backend/windmill-types/src/scripts.rs @@ -323,7 +323,7 @@ pub fn id_to_codebase_info(id: &str) -> CodebaseInfo { pub const SCRIPT_COLUMNS: &str = concat!( "workspace_id, hash, path, parent_hashes, summary, description, content, ", "created_by, created_at, archived, schema, deleted, is_template, extra_perms, ", - "lock, lock_error_logs, language, kind, tag, draft_only, envs, ", + "lock, lock_error_logs, language, kind, tag, envs, ", "dedicated_worker, ws_error_handler_muted, priority, cache_ttl, cache_ignore_s3_path, ", "timeout, delete_after_use, delete_after_secs, restart_unless_cancelled, ", "visible_to_runner_only, auto_kind, codebase, has_preprocessor, on_behalf_of_email, ", @@ -355,8 +355,6 @@ pub struct Script { pub kind: ScriptKind, pub tag: Option, #[serde(skip_serializing_if = "Option::is_none")] - pub draft_only: Option, - #[serde(skip_serializing_if = "Option::is_none")] pub envs: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub dedicated_worker: Option, @@ -449,8 +447,10 @@ pub struct ListableScript { pub tag: Option, #[serde(skip_serializing_if = "Option::is_none")] pub description: Option, - #[serde(skip_serializing_if = "Option::is_none")] - pub has_draft: Option, + /// `Some(true)` only on rows synthesised from the `draft` table (never-deployed + /// items the user owns a draft for); `None` on deployed rows. Kept on the public + /// response so consumers checking `draft_only === true` keep working. + #[sqlx(default)] #[serde(skip_serializing_if = "Option::is_none")] pub draft_only: Option, pub has_deploy_errors: bool, @@ -465,6 +465,22 @@ pub struct ListableScript { pub kind: ScriptKind, #[serde(skip_serializing_if = "Option::is_none")] pub labels: Option>, + /// `true` when this entry is the authed user's draft — draft-only, or a deployed + /// row the user has saved a draft on top of. Distinguishes user state from team state. + #[serde(skip_serializing_if = "is_false")] + pub is_draft: bool, + /// User-typed staged path, so the home list shows a meaningful name over the + /// autogenerated `u/{user}/draft_{uuid}`. Sourced from the draft JSON: scripts use + /// `value.path` (the Path widget binds `script.path`); flows/apps/raw apps use an + /// explicit `value.draft_path` written only when it differs from deployed. `None` = unchanged. + #[sqlx(default)] + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_path: Option, + /// Per-path draft owners (`{ username }`, `None` for the legacy NULL-email row), + /// driving the home-page avatar circles. `None` when no drafts; never an empty array. + #[sqlx(default)] + #[serde(skip_serializing_if = "Option::is_none")] + pub draft_users: Option>>, /// Labels inherited from the parent folder, computed at read time. #[sqlx(default)] #[serde(skip_serializing_if = "Option::is_none")] @@ -516,7 +532,6 @@ pub struct NewScript { pub language: ScriptLang, pub kind: Option, pub tag: Option, - pub draft_only: Option, pub envs: Option>, #[serde(flatten)] pub concurrency_settings: ConcurrencySettings, @@ -574,7 +589,6 @@ impl Hash for NewScript { self.language.hash(state); self.kind.hash(state); self.tag.hash(state); - self.draft_only.hash(state); self.envs.hash(state); self.concurrency_settings.hash(state); self.debouncing_settings.hash(state); diff --git a/backend/windmill-types/src/user_drafts.rs b/backend/windmill-types/src/user_drafts.rs new file mode 100644 index 0000000000..30740fc561 --- /dev/null +++ b/backend/windmill-types/src/user_drafts.rs @@ -0,0 +1,18 @@ +//! Shared types for the per-user draft surface. +//! +//! Mirrors the `OtherDraftUser` type in `windmill-common::user_drafts` — +//! kept here so `windmill-types` row structs (ListableScript / ListableFlow / +//! ListableApp) can expose a typed `draft_users` field without taking a +//! dependency on `windmill-common`. The two structs serialize identically, +//! so the frontend doesn't notice. + +use serde::{Deserialize, Serialize}; + +/// One workspace user (or the legacy NULL-email row) with a per-user draft +/// at a given path. Used by the home-page list endpoints to feed the +/// avatar-circles inside the Draft badge. +#[derive(Debug, Serialize, Deserialize, Clone)] +pub struct DraftUserRef { + /// `None` represents a legacy workspace-level draft (no owner). + pub username: Option, +} diff --git a/backend/windmill-worker/Cargo.toml b/backend/windmill-worker/Cargo.toml index 2e75ac3632..14aeb734ed 100644 --- a/backend/windmill-worker/Cargo.toml +++ b/backend/windmill-worker/Cargo.toml @@ -21,6 +21,7 @@ bigquery = ["dep:gcp_auth"] benchmark = ["windmill-queue/benchmark", "windmill-common/benchmark"] parquet = ["windmill-common/parquet", "windmill-object-store/parquet"] flow_testing = [] +failpoints = [] cloud = [] sqlx = [] deno_core = ["dep:windmill-runtime-nativets"] @@ -38,7 +39,7 @@ java = ["dep:windmill-parser-java"] ruby = ["dep:windmill-parser-ruby"] rlang = ["dep:windmill-parser-r"] duckdb = ["dep:libloading"] -quickjs = ["windmill-jseval/quickjs"] +quickjs = ["windmill-jseval/quickjs", "windmill-queue/quickjs"] bedrock = ["windmill-ai/bedrock"] [dependencies] @@ -114,7 +115,8 @@ hmac.workspace = true pem = { workspace = true, optional = true } rsa = { workspace = true, optional = true } urlencoding.workspace = true -nix.workspace = true +# `fs` adds flock(2) for the cross-process Python install lock (shared cache mounts) +nix = { workspace = true, features = ["fs"] } bytes.workspace = true reqwest.workspace = true reqwest-middleware.workspace = true @@ -146,7 +148,7 @@ hyper-util = { workspace = true, optional = true } rcgen = { workspace = true, optional = true } [target.'cfg(windows)'.dependencies] -windows = { version = "0.61", features = ["Win32_System_JobObjects", "Win32_System_Threading"] } +windows = { version = "0.61", features = ["Win32_System_JobObjects", "Win32_System_Threading", "Win32_System_Diagnostics_ToolHelp"] } [dev-dependencies] tempfile.workspace = true diff --git a/backend/windmill-worker/nsjail/download_deps.py.sh b/backend/windmill-worker/nsjail/download_deps.py.sh index 13fc00ddf9..6b4cd8cfd8 100755 --- a/backend/windmill-worker/nsjail/download_deps.py.sh +++ b/backend/windmill-worker/nsjail/download_deps.py.sh @@ -2,7 +2,8 @@ INDEX_URL_ARG=$([ -z "$INDEX_URL" ] && echo ""|| echo "--index-url $INDEX_URL" ) EXTRA_INDEX_URL_ARG=$([ -z "$EXTRA_INDEX_URL" ] && echo ""|| echo "--extra-index-url $EXTRA_INDEX_URL" ) -TRUSTED_HOST_ARG=$([ -z "$TRUSTED_HOST" ] && echo "" || echo "--trusted-host $TRUSTED_HOST") +TRUSTED_HOST_ARG="" +for h in $TRUSTED_HOST; do TRUSTED_HOST_ARG="$TRUSTED_HOST_ARG --trusted-host $h"; done if [ ! -z "$INDEX_URL" ] then diff --git a/backend/windmill-worker/src/agent_workers.rs b/backend/windmill-worker/src/agent_workers.rs index 9bf0ea1834..5320c8ef6c 100644 --- a/backend/windmill-worker/src/agent_workers.rs +++ b/backend/windmill-worker/src/agent_workers.rs @@ -78,4 +78,22 @@ pub async fn get_datatable_resource_from_agent_http( .await } +/// Record a materialization outcome from an agent worker (no direct DB) via the +/// API, so `materialized_partition` state lands the same as on a Sql worker. +// Only called from the duckdb executor, which is itself `#[cfg(feature = "duckdb")]`. +#[cfg(feature = "duckdb")] +pub async fn record_materialization_from_agent_http( + client: &HttpClient, + w_id: &str, + req: &windmill_common::materialization::RecordMaterializationRequest, +) -> anyhow::Result<()> { + client + .post( + &format!("/api/w/{}/agent_workers/record_materialization", w_id), + None, + req, + ) + .await +} + pub const UPDATE_PING_URL: &str = "/api/agent_workers/update_ping"; diff --git a/backend/windmill-worker/src/ai/utils.rs b/backend/windmill-worker/src/ai/utils.rs index 353bab17a0..6e80551650 100644 --- a/backend/windmill-worker/src/ai/utils.rs +++ b/backend/windmill-worker/src/ai/utils.rs @@ -557,21 +557,20 @@ pub async fn load_mcp_tools( tracing::debug!("Loading MCP tools from resource: {}", config.resource_path); let path = config.resource_path.trim_start_matches("$res:"); - let mcp_resource = { - // Fetch the resource from database - let resource= sqlx::query_scalar!( - "SELECT value as \"value: sqlx::types::Json>\" FROM resource WHERE path = $1 AND workspace_id = $2", - &path, - &workspace_id - ) - .fetch_optional(db) - .await? - .ok_or_else(|| Error::NotFound(format!("Could not find the resource {}, update the resource path in the workspace settings", config.resource_path)))? - .ok_or_else(|| Error::BadRequest(format!("Empty resource value for {}", config.resource_path)))?; - - serde_json::from_str::(resource.0.get()) - .context("Failed to parse MCP resource")? - }; + // Load the resource through the job's permissioned (RLS + scope) path so + // a flow author cannot make the agent use an MCP resource their identity + // is not allowed to read (resources:read:{path}). Reading through the raw + // db pool here would bypass the authorization enforced by the regular MCP + // tools API (get_mcp_tools) and act as a confused deputy. + let mcp_resource = client + .get_resource_value::(path) + .await + .map_err(|e| { + Error::internal_err(format!( + "Failed to load MCP resource {}: {}", + config.resource_path, e + )) + })?; let resource_name = mcp_resource.name.clone(); diff --git a/backend/windmill-worker/src/ansible_executor.rs b/backend/windmill-worker/src/ansible_executor.rs index 7b0ae3b3ea..2758fe85ff 100644 --- a/backend/windmill-worker/src/ansible_executor.rs +++ b/backend/windmill-worker/src/ansible_executor.rs @@ -13,7 +13,7 @@ use tokio::process::Command; use uuid::Uuid; use windmill_common::{ error, - git_sync_oss::prepend_token_to_github_url, + git_sync_oss::{prepend_token_to_github_url, sanitize_git_url}, worker::{ is_allowed_file_location, split_python_requirements, to_raw_value, write_file, write_file_at_user_defined_location, Connection, PyVAlias, WORKER_CONFIG, @@ -22,7 +22,8 @@ use windmill_common::{ use windmill_queue::MiniPulledJob; use windmill_parser_yaml::{ - AnsibleRequirements, GitRepo, PreexistingAnsibleInventory, ResourceOrVariablePath, + validate_vault_id, AnsibleRequirements, GitRepo, PreexistingAnsibleInventory, + ResourceOrVariablePath, }; use windmill_queue::{append_logs, CanceledBy}; @@ -846,7 +847,7 @@ pub async fn get_git_repo_full_head_commit_hash( .first() .ok_or(anyhow!( "The HEAD commit hash was not found for repo `{}`", - &repo.url + sanitize_git_url(&repo.url) ))? .split_whitespace() .next() @@ -910,6 +911,11 @@ pub fn create_ansible_cfg( } if let Some(vault_ids) = reqs.as_ref().map(|r| &r.vault_id) { if !vault_ids.is_empty() { + // Defense in depth: entries are validated at parse time, but re-check here + // since they are interpolated raw into ansible.cfg (config-directive injection). + for vault_id in vault_ids { + validate_vault_id(vault_id)?; + } let password_files = vault_ids.join(","); passwords_cfg.push_str(&format!("vault_identity_list = {password_files}\n")); @@ -1248,7 +1254,12 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } else { clone_repo( &repo, @@ -1263,7 +1274,12 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } append_logs( @@ -1310,7 +1326,7 @@ pub async fn handle_ansible_job( append_logs( &job.id, &job.workspace_id, - format!("\nCloning {}...\n", &repo.url), + format!("\nCloning {}...\n", sanitize_git_url(&repo.url)), conn, ) .await; @@ -1332,13 +1348,18 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } else { if req_lockfiles.is_some() { append_logs( &job.id, &job.workspace_id, - format!("Warning: `{}` is using latest commit because the lockfile didn't store a commit hash for this repo. Updates to the repo could break the deployed playbook.\n", &repo.url), + format!("Warning: `{}` is using latest commit because the lockfile didn't store a commit hash for this repo. Updates to the repo could break the deployed playbook.\n", sanitize_git_url(&repo.url)), conn, ) .await; @@ -1356,13 +1377,22 @@ pub async fn handle_ansible_job( git_ssh_cmd, ) .await - .map_err(|e| anyhow!("Failed to clone git repo `{}`: {e}", repo.url))?; + .map_err(|e| { + anyhow!( + "Failed to clone git repo `{}`: {e}", + sanitize_git_url(&repo.url) + ) + })?; } append_logs( &job.id, &job.workspace_id, - format!("Cloned {} into {}\n", &repo.url, &repo.target_path), + format!( + "Cloned {} into {}\n", + sanitize_git_url(&repo.url), + &repo.target_path + ), conn, ) .await; @@ -1799,4 +1829,31 @@ mod tests { assert!(validate_relative_path("", "playbook").is_err()); assert!(validate_relative_path(" ", "playbook").is_err()); } + + #[test] + fn test_create_ansible_cfg_writes_valid_vault_id() { + let dir = tempfile::tempdir().unwrap(); + let job_dir = dir.path().to_str().unwrap(); + let reqs = AnsibleRequirements { + vault_id: vec!["dev@vault_pass.txt".to_string()], + ..Default::default() + }; + create_ansible_cfg(Some(&reqs), job_dir, false).unwrap(); + let cfg = std::fs::read_to_string(dir.path().join("ansible.cfg")).unwrap(); + assert!(cfg.contains("vault_identity_list = dev@vault_pass.txt")); + assert!(!cfg.contains("library")); + } + + #[test] + fn test_create_ansible_cfg_rejects_vault_id_injection() { + let dir = tempfile::tempdir().unwrap(); + let job_dir = dir.path().to_str().unwrap(); + let reqs = AnsibleRequirements { + vault_id: vec!["default@/tmp/wm/x\nlibrary = /tmp/wm/evil_modules".to_string()], + ..Default::default() + }; + // Defense-in-depth boundary: a poisoned entry must error before any config is written. + assert!(create_ansible_cfg(Some(&reqs), job_dir, false).is_err()); + assert!(!dir.path().join("ansible.cfg").exists()); + } } diff --git a/backend/windmill-worker/src/bigquery_executor.rs b/backend/windmill-worker/src/bigquery_executor.rs index bfe0874f83..8a80de25ef 100644 --- a/backend/windmill-worker/src/bigquery_executor.rs +++ b/backend/windmill-worker/src/bigquery_executor.rs @@ -378,6 +378,19 @@ pub async fn do_bigquery( .await .map_err(|e| Error::ExecutionErr(e.to_string()))?; + // Validate before it is interpolated into request URLs as a path segment + // (https://bigquery.googleapis.com/.../projects//...). + if project_id.is_empty() + || !project_id + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_' | ':')) + { + return Err(Error::ExecutionErr(format!( + "Invalid BigQuery project id '{}': only alphanumeric, '.', '-', '_' and ':' allowed", + project_id.chars().take(64).collect::() + ))); + } + let mut sig = parse_bigquery_sig(&query) .map_err(|x| Error::ExecutionErr(x.to_string()))? .args; diff --git a/backend/windmill-worker/src/common.rs b/backend/windmill-worker/src/common.rs index 1bef9e4c6d..a88ba67bb7 100644 --- a/backend/windmill-worker/src/common.rs +++ b/backend/windmill-worker/src/common.rs @@ -748,17 +748,20 @@ lazy_static! { const MEMORY_LIMIT_1CU: usize = 2 * 1024 * 1024 * 1024; /// Wrapper that holds a Windows Job Object handle alongside the child process. -/// The job object enforces memory limits and is closed when the child is dropped. +/// The job object carries KILL_ON_JOB_CLOSE and/or a memory limit. With +/// KILL_ON_JOB_CLOSE, the worker process dying closes the handle and the OS reaps the +/// whole child tree — preventing orphans when the worker is force-killed (e.g. a second +/// CTRL_BREAK_EVENT or Nomad's kill_timeout) before a job has drained. #[cfg(windows)] -struct MemoryLimitedChild { +struct WindowsJobChild { inner: Box, _job_handle: Win32JobHandle, } #[cfg(windows)] -impl std::fmt::Debug for MemoryLimitedChild { +impl std::fmt::Debug for WindowsJobChild { fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { - f.debug_struct("MemoryLimitedChild").finish() + f.debug_struct("WindowsJobChild").finish() } } @@ -781,7 +784,7 @@ impl Drop for Win32JobHandle { } #[cfg(windows)] -impl process_wrap::tokio::TokioChildWrapper for MemoryLimitedChild { +impl process_wrap::tokio::TokioChildWrapper for WindowsJobChild { fn inner(&self) -> &tokio::process::Child { self.inner.inner() } @@ -789,7 +792,11 @@ impl process_wrap::tokio::TokioChildWrapper for MemoryLimitedChild { self.inner.inner_mut() } fn into_inner(self: Box) -> tokio::process::Child { - self.inner.into_inner() + // Leak the job handle: with KILL_ON_JOB_CLOSE, closing the last handle reaps + // the (still-running) child, so extracting the inner Child must not close it. + let WindowsJobChild { inner, _job_handle } = *self; + std::mem::forget(_job_handle); + inner.into_inner() } fn start_kill(&mut self) -> std::io::Result<()> { self.inner.start_kill() @@ -805,9 +812,59 @@ impl process_wrap::tokio::TokioChildWrapper for MemoryLimitedChild { } } -/// Create a Windows Job Object with a memory limit and assign the process to it. +/// Resume all threads of a process created with CREATE_SUSPENDED. We create the child +/// suspended so it can be assigned to its job object before running any code; otherwise +/// a child that forks a helper at startup could create it before the assignment and +/// leave it outside the job (escaping KILL_ON_JOB_CLOSE reaping / the memory cap). +/// (Ported from process-wrap's resume_threads.) #[cfg(windows)] -fn apply_job_memory_limit(pid: u32, memory_limit: usize) -> Result { +fn resume_process(pid: u32) -> Result<(), std::io::Error> { + use windows::Win32::System::Diagnostics::ToolHelp::{ + CreateToolhelp32Snapshot, Thread32First, Thread32Next, TH32CS_SNAPTHREAD, THREADENTRY32, + }; + use windows::Win32::System::Threading::{OpenThread, ResumeThread, THREAD_SUSPEND_RESUME}; + + unsafe { + let snapshot = CreateToolhelp32Snapshot(TH32CS_SNAPTHREAD, 0).map_err(|e| { + std::io::Error::new( + std::io::ErrorKind::Other, + format!("CreateToolhelp32Snapshot: {e}"), + ) + })?; + let mut entry = THREADENTRY32 { + dwSize: std::mem::size_of::() as u32, + cntUsage: 0, + th32ThreadID: 0, + th32OwnerProcessID: 0, + tpBasePri: 0, + tpDeltaPri: 0, + dwFlags: 0, + }; + let mut res = Thread32First(snapshot, &mut entry); + while res.is_ok() { + if entry.th32OwnerProcessID == pid { + if let Ok(thread) = OpenThread(THREAD_SUSPEND_RESUME, false, entry.th32ThreadID) { + ResumeThread(thread); + let _ = windows::Win32::Foundation::CloseHandle(thread); + } + } + res = Thread32Next(snapshot, &mut entry); + } + let _ = windows::Win32::Foundation::CloseHandle(snapshot); + } + Ok(()) +} + +/// Create a Windows Job Object and assign the process to it, optionally with +/// KILL_ON_JOB_CLOSE (so the child tree is reaped when the worker drops the handle / +/// dies) and/or a memory limit. Assigning post-spawn (rather than via process-wrap's +/// suspend/resume JobObject wrap) keeps the dotnet-safe behavior that csharp relies on. +#[cfg(windows)] +fn assign_job_object( + pid: u32, + memory_limit: Option, + kill_on_close: bool, +) -> Result { use windows::Win32::System::JobObjects::*; use windows::Win32::System::Threading::{OpenProcess, PROCESS_SET_QUOTA, PROCESS_TERMINATE}; @@ -817,8 +874,13 @@ fn apply_job_memory_limit(pid: u32, memory_limit: usize) -> Result()), but + // get_wrap returns None mid-spawn (process-wrap 8.2.1 takes the wrappers out of + // `self` before running pre_spawn), so it silently overwrites and drops + // CREATE_NEW_PROCESS_GROUP. Job-object grouping is done post-spawn below instead, + // which also matches the dotnet-safe assignment csharp already relies on. + #[cfg(windows)] + if !*DISABLE_PROCESS_GROUP { + use process_wrap::tokio::CreationFlags; + use windows::Win32::System::Threading::{CREATE_NEW_PROCESS_GROUP, CREATE_SUSPENDED}; + // Also create the child suspended so it can be placed in its job object (below) + // before it runs any code; otherwise a child that forks a helper at startup could + // create it before the assignment and leave it outside the job. Resumed right + // after the job assignment. + cmd.wrap(CreationFlags(CREATE_NEW_PROCESS_GROUP | CREATE_SUSPENDED)); + } + if !*DISABLE_PROCESS_GROUP && !disable_process_group { #[cfg(unix)] { @@ -916,31 +1000,51 @@ pub async fn start_child_process( cmd.wrap(ProcessGroup::leader()); } - #[cfg(windows)] - { - cmd.wrap(JobObject); - } } let child: Box = cmd .spawn() .map_err(|err| tentatively_improve_error(err.into(), executable))?; + // On Windows, assign the child to a job object. KILL_ON_JOB_CLOSE makes the OS reap + // the whole child tree when the worker drops the handle or dies, so jobs aren't + // orphaned if the worker is force-killed (second CTRL_BREAK_EVENT, or Nomad exceeding + // kill_timeout) before they drain; it is gated on the DISABLE_PROCESS_GROUP escape + // hatch alongside CREATE_NEW_PROCESS_GROUP above. The LIMIT_WINDOWS_TO_1CU memory cap + // is independent of that hatch (it's its own opt-in), so it's applied whenever set. + // Assigning post-spawn does not touch the creation flags (so it composes with + // CREATE_NEW_PROCESS_GROUP) and is the dotnet-safe path csharp already uses. #[cfg(windows)] - if *windmill_common::worker::LIMIT_WINDOWS_TO_1CU { - if let Some(pid) = child.inner().id() { - match apply_job_memory_limit(pid, MEMORY_LIMIT_1CU) { - Ok(job_handle) => { - tracing::info!( - "Applied 2GB memory limit (LIMIT_WINDOWS_TO_1CU) to child process {pid}" - ); - return Ok(Box::new(MemoryLimitedChild { - inner: child, - _job_handle: job_handle, - })); + { + let kill_on_close = !*DISABLE_PROCESS_GROUP; + let memory_limit = + (*windmill_common::worker::LIMIT_WINDOWS_TO_1CU).then_some(MEMORY_LIMIT_1CU); + if kill_on_close || memory_limit.is_some() { + if let Some(pid) = child.inner().id() { + let assigned = assign_job_object(pid, memory_limit, kill_on_close); + // When kill_on_close, the child was created suspended (CREATE_SUSPENDED + // above) so it could be placed in the job before running. Resume it now — + // unconditionally of assign success, so a failed assign never leaves it hung. + if kill_on_close { + if let Err(e) = resume_process(pid) { + tracing::error!("Failed to resume child process {pid}: {e}"); + } } - Err(e) => { - tracing::warn!("Failed to apply memory limit to child process {pid}: {e}"); + match assigned { + Ok(job_handle) => { + if memory_limit.is_some() { + tracing::info!( + "Applied 2GB memory limit (LIMIT_WINDOWS_TO_1CU) to child process {pid}" + ); + } + return Ok(Box::new(WindowsJobChild { + inner: child, + _job_handle: job_handle, + })); + } + Err(e) => { + tracing::warn!("Failed to assign child process {pid} to job object: {e}"); + } } } } diff --git a/backend/windmill-worker/src/deno_executor.rs b/backend/windmill-worker/src/deno_executor.rs index c39e9e999e..a446548e1f 100644 --- a/backend/windmill-worker/src/deno_executor.rs +++ b/backend/windmill-worker/src/deno_executor.rs @@ -434,7 +434,11 @@ try {{ if let Some(ref npmrc_content) = npmrc { if !npmrc_content.trim().is_empty() { write_file(job_dir, ".npmrc", npmrc_content)?; - write_file(job_dir, "deno.json", "{}")?; + // minimumDependencyAge=0 opts out of Deno's supply-chain guard that rejects + // npm packages published within the last ~24h. Private/internal registries + // routinely serve just-published versions, so the guard would break them. + // Older Deno ignores the unknown field, so this is safe across versions. + write_file(job_dir, "deno.json", r#"{"minimumDependencyAge":"0"}"#)?; } } diff --git a/backend/windmill-worker/src/duckdb_executor.rs b/backend/windmill-worker/src/duckdb_executor.rs index 4a019be9ab..e4b94acabf 100644 --- a/backend/windmill-worker/src/duckdb_executor.rs +++ b/backend/windmill-worker/src/duckdb_executor.rs @@ -32,6 +32,470 @@ use crate::sql_utils::remove_comments; use windmill_common::client::AuthedClient; use windmill_object_store::DEFAULT_STORAGE; +// What a `// materialize` run records into `materialized_partition` once it +// finishes. `asset_path` is the full `/
` (the asset identity); +// `partition` is "" for an unpartitioned (whole-table) materialization. +struct MaterializeExec { + asset_kind: windmill_common::assets::AssetKind, + asset_path: String, + partition: String, + // Number of `// data_test` checks the codegen embedded. Enforcement recovers + // the per-test outcomes from the summary row; if it recovers fewer than this + // (e.g. an FFI serialization change drops the column), we fail loud rather + // than silently pass declared-but-unverified tests. + n_data_tests: usize, +} + +// Fetch and validate a custom data-test script's body. v1 custom tests are +// DuckDB scripts holding a single SELECT/CTE that returns the violating rows +// (dbt's singular-test convention); the worker embeds that query as a subquery +// check in the materialize connection (the single-statement constraint is +// enforced in sql_materialize.rs). Server workers only — agent (Http) workers +// have no script cache to read deployed content from. +async fn fetch_custom_test_body(conn: &Connection, w_id: &str, path: &str) -> Result { + let Connection::Sql(db) = conn else { + return Err(Error::ExecutionErr(format!( + "data_test custom `{path}`: custom tests require a server worker (not supported on \ + agent workers in v1)" + ))); + }; + let hash = windmill_common::get_latest_script_hash(db, path, w_id) + .await? + .ok_or_else(|| { + Error::ExecutionErr(format!( + "data_test custom `{path}`: no deployed script found at this path" + )) + })?; + let content = + crate::get_script_content_by_hash(&windmill_common::scripts::ScriptHash(hash), w_id, conn) + .await?; + if !matches!( + content.language, + Some(windmill_common::scripts::ScriptLang::DuckDb) + ) { + return Err(Error::ExecutionErr(format!( + "data_test custom `{path}`: must be a DuckDB script returning the violating rows \ + (got language {:?})", + content.language + ))); + } + Ok(content.content) +} + +// If `query` declares `// materialize `, return what to record plus, +// for the default managed mode, the rewritten managed-write SQL (in `manual` +// mode the script writes its own DDL, so the rewrite is `None`). The rewritten +// SQL contains a synthetic `ATTACH 'ducklake://' AS _wm_target` that the +// normal ATTACH-transform pass resolves to real credentials — the same path as +// the user's own ATTACH. `// data_test` lines append verifier probes that run +// against the freshly-materialized target and raise (failing the run) on +// violation. Returns `None` when there is no materialize annotation or the +// target isn't a ducklake (only ducklake is materialized in v1). +fn build_materialized_query( + query: &str, + partition_value: Option<&str>, + // Custom (`// data_test `) test bodies, pre-fetched by the caller + // (`fetch_custom_test_bodies`) so this stays pure/sync and unit-testable — + // the DB read is the only thing that needs a connection. Keyed by script path. + custom_test_bodies: &std::collections::HashMap, +) -> Result, MaterializeExec)>> { + use windmill_parser::asset_parser::{ + parse_pipeline_annotations, AssetKind as PAssetKind, DataTest, + }; + use windmill_parser::sql_materialize::{ + build_wrap_blocks, DataTestResolved, MaterializeStrategy, TARGET_ALIAS, + }; + + let ann = parse_pipeline_annotations(query); + let has_tests = !ann.data_tests.is_empty(); + let Some(m) = ann.materialize else { + // Data tests run *against the materialized asset*; without a + // `// materialize` target there is nothing to test. Fail loudly rather + // than silently skip the declared checks. + if has_tests { + return Err(Error::ExecutionErr( + "data_test: requires a `// materialize` target — data tests run against the \ + materialized asset" + .to_string(), + )); + } + return Ok(None); + }; + if m.target_kind != PAssetKind::Ducklake { + if has_tests { + return Err(Error::ExecutionErr( + "data_test: only `ducklake://` materialization targets support data tests in v1" + .to_string(), + )); + } + return Ok(None); + } + let partitioned = ann.partition.is_some(); + let partition = partition_value.unwrap_or("").to_string(); + // Partition *resolution* is enterprise; in its absence a partitioned + // materialize only runs with an explicit `partition` arg. Fail loudly rather + // than silently materialize the wrong (empty) slice. + if partitioned && partition.is_empty() { + return Err(Error::ExecutionErr( + "materialize: a `// partitioned` script ran with no resolved partition — pass an \ + explicit `partition` arg, or enable enterprise partition resolution" + .to_string(), + )); + } + // Convention: `ducklake:///
` — is the configured + // ducklake (resolved like a user ATTACH),
is the rest. + let (ducklake_name, table) = m + .target_path + .split_once('/') + .unwrap_or((m.target_path.as_str(), "")); + let meta = MaterializeExec { + asset_kind: windmill_common::assets::AssetKind::Ducklake, + asset_path: m.target_path.clone(), + partition: partition.clone(), + n_data_tests: ann.data_tests.len(), + }; + + // `{partition}` → escaped SQL literal substitution, applied to the managed + // SELECT, its setup, and any custom-test body so a partitioned test can + // filter by the active slice. Always a complete `'…'` literal (with `'` + // doubled) whether or not the author quoted it, so a run caller can't break + // out and alter statement boundaries. The pre-quoted `'{partition}'` form is + // matched first so it doesn't become `''…''`. No-op when unpartitioned. + let lit = format!("'{}'", partition.replace('\'', "''")); + let substitute = |s: &str| -> String { + if !partitioned { + return s.to_string(); + } + let tok = windmill_common::assets::PARTITION_TOKEN; + let quoted_tok = format!("'{tok}'"); + s.replace("ed_tok, &lit).replace(tok, &lit) + }; + + if m.manual { + // Escape hatch: the script owns its DDL. We can't reliably attach the + // managed target or know the partition column it wrote, so data tests + // are not generated for manual mode in v1. + if has_tests { + return Err(Error::ExecutionErr( + "data_test: not supported with `// materialize manual` in v1 — use managed \ + `// materialize`" + .to_string(), + )); + } + return Ok(Some((None, meta))); + } + if table.is_empty() { + return Err(Error::ExecutionErr(format!( + "materialize: target `ducklake://{}` has no table (use ducklake:///
)", + m.target_path + ))); + } + let mut plan = classify_wrap_or_err(query)?; + plan.output = substitute(&plan.output); + for s in plan.setup.iter_mut() { + *s = substitute(s); + } + let strategy = if m.append { + MaterializeStrategy::Append + } else if let Some(uk) = m.unique_key { + MaterializeStrategy::Merge { unique_key: uk } + } else { + MaterializeStrategy::Replace + }; + // Inline the partition as an escaped SQL literal (DuckLake has no bind for + // the partition column in our generated DDL). + let pval = lit.clone(); + let synthetic_attach = format!("ATTACH 'ducklake://{ducklake_name}' AS {TARGET_ALIAS};"); + + // Resolve data tests (fetch + partition-substitute custom bodies) so codegen + // can embed every check's violating-row count in the materialize summary. + // The summary then carries the full per-test breakdown back to the worker, + // which runs them all and decides pass/fail (no abort-on-first). Empty when + // there are no `// data_test` lines. + let mut resolved = Vec::with_capacity(ann.data_tests.len()); + for test in &ann.data_tests { + match test { + DataTest::Custom { path } => { + let raw = custom_test_bodies.get(path).ok_or_else(|| { + Error::ExecutionErr(format!( + "data_test custom `{path}`: body not fetched before codegen (internal)" + )) + })?; + resolved + .push(DataTestResolved::Custom { path: path.clone(), body: substitute(raw) }); + } + other => resolved.push(DataTestResolved::BuiltIn(other.clone())), + } + } + + let blocks = build_wrap_blocks( + &plan, + &synthetic_attach, + table, + &m.target_path, + "_wm_partition", + &pval, + partitioned, + strategy, + &resolved, + ) + .map_err(Error::ExecutionErr)?; + + Ok(Some((Some(blocks.join("\n")), meta))) +} + +// Fetch the deployed body of every `// data_test ` custom test declared in +// `query`, keyed by path, so the sync `build_materialized_query` can splice them +// in. The DB read is the only part of materialize codegen that needs a +// connection; isolating it here keeps the codegen pure and unit-testable. +// Server workers only (`fetch_custom_test_body` errors on agent workers). Empty +// when there are no custom tests. +async fn fetch_custom_test_bodies( + query: &str, + conn: &Connection, + w_id: &str, +) -> Result> { + use windmill_parser::asset_parser::{parse_pipeline_annotations, DataTest}; + let ann = parse_pipeline_annotations(query); + let mut bodies = std::collections::HashMap::new(); + for test in &ann.data_tests { + if let DataTest::Custom { path } = test { + if !bodies.contains_key(path) { + let body = fetch_custom_test_body(conn, w_id, path).await?; + bodies.insert(path.clone(), body); + } + } + } + Ok(bodies) +} + +// classify_wrap with the spec's actionable message turned into an executor error. +fn classify_wrap_or_err(query: &str) -> Result { + windmill_parser::sql_materialize::classify_wrap(query) + .map_err(|e| Error::ExecutionErr(e.message())) +} + +// Pull a named i64 field (`snapshot_id` / `rows`) out of the trailing summary +// read — which in wrap mode is the job result. Shape-tolerant (object / array / +// nested), returns None if absent (literal mode, or capture failed). +fn extract_i64(result: &RawValue, field: &str) -> Option { + fn find(v: &Value, field: &str) -> Option { + match v { + Value::Number(n) => n.as_i64(), + Value::Object(m) => m.get(field).and_then(|x| find(x, field)), + Value::Array(a) => a.iter().find_map(|x| find(x, field)), + _ => None, + } + } + find(&serde_json::from_str::(result.get()).ok()?, field) +} + +// One data test's outcome as carried by the materialize summary's `data_tests` +// column: its display name and how many rows violated it (0 = pass). +struct DataTestOutcome { + name: String, + violating: i64, +} + +// Pull the per-test breakdown out of the materialize summary result. The +// `data_tests` column is a DuckLake list-of-struct `[{test, violating}, …]`; +// the FFI may surface it as a nested JSON array or as a JSON string, so accept +// both. Returns empty when there are no tests (the column is absent). +fn extract_data_tests(result: &RawValue) -> Vec { + fn collect(v: &Value, out: &mut Vec) { + if let Value::Array(arr) = v { + for item in arr { + if let Value::Object(o) = item { + if let Some(Value::String(name)) = o.get("test") { + let violating = o + .get("violating") + .and_then(|x| x.as_i64().or_else(|| x.as_f64().map(|f| f as i64))) + .unwrap_or(0); + out.push(DataTestOutcome { name: name.clone(), violating }); + } + } + } + } + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("data_tests"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let mut out = Vec::new(); + let Ok(root) = serde_json::from_str::(result.get()) else { + return out; + }; + match find_field(&root) { + Some(arr @ Value::Array(_)) => collect(arr, &mut out), + // FFI serialized the list-of-struct as a JSON string — parse it. + Some(Value::String(s)) => { + if let Ok(parsed) = serde_json::from_str::(s) { + collect(&parsed, &mut out); + } + } + _ => {} + } + out +} + +// Pull the captured output schema out of the materialize summary's +// `output_schema` column (gap #2a): a list-of-struct `[{name, type}, …]` the +// codegen built from a `DESCRIBE`. Like `data_tests`, the FFI may surface it as +// a nested JSON array or a JSON string — accept both. Returns `None` when the +// column is absent (literal mode, manual mode, or capture failed) so the worker +// records the run without a schema rather than an empty one. +fn extract_schema( + result: &RawValue, +) -> Option> { + use windmill_common::materialization::SchemaColumn; + fn collect(v: &Value) -> Option> { + let Value::Array(arr) = v else { return None }; + let mut out = Vec::with_capacity(arr.len()); + for item in arr { + let o = item.as_object()?; + let name = o.get("name")?.as_str()?.to_string(); + let data_type = o.get("type")?.as_str()?.to_string(); + out.push(SchemaColumn { name, data_type }); + } + Some(out) + } + fn find_field(v: &Value) -> Option<&Value> { + match v { + Value::Object(o) => o.get("output_schema"), + Value::Array(a) => a.iter().find_map(find_field), + _ => None, + } + } + let root = serde_json::from_str::(result.get()).ok()?; + match find_field(&root)? { + arr @ Value::Array(_) => collect(arr), + // FFI serialized the list-of-struct as a JSON string — parse it. + Value::String(s) => collect(&serde_json::from_str::(s).ok()?), + _ => None, + } +} + +// Render the full pass/fail breakdown for a failed data-test run — every test, +// not just the first failure, so the user sees the whole picture in one place. +fn format_data_test_breakdown(asset_path: &str, tests: &[DataTestOutcome]) -> String { + let failed = tests.iter().filter(|t| t.violating > 0).count(); + let mut lines = vec![format!( + "data tests failed on {asset_path} ({failed}/{} failed):", + tests.len() + )]; + for t in tests { + if t.violating > 0 { + lines.push(format!(" ✗ {} — {} violating row(s)", t.name, t.violating)); + } else { + lines.push(format!(" ✓ {}", t.name)); + } + } + lines.join("\n") +} + +// Best-effort record of a materialization outcome. On a Sql connection it writes +// the row directly; on an agent worker (Http, no direct DB) it posts to the API +// so state lands the same way. Never fails the job — a lost row degrades the +// grid, not the run. +async fn record_mat( + conn: &Connection, + w_id: &str, + job_id: Uuid, + meta: &MaterializeExec, + status: windmill_common::materialization::MaterializationStatus, + snapshot_id: Option, + row_count: Option, + // Captured output schema (gap #2a). Only set on a successful materialize; + // when present, also upserts a `materialized_asset_schema` version. + schema: Option>, + error: Option<&str>, +) { + let req = windmill_common::materialization::RecordMaterializationRequest { + asset_kind: meta.asset_kind, + asset_path: meta.asset_path.clone(), + partition: meta.partition.clone(), + status, + snapshot_id, + row_count, + job_id: Some(job_id), + error: error.map(|e| e.to_string()), + schema: schema.clone(), + }; + let res: anyhow::Result<()> = match conn { + Connection::Sql(db) => { + let partition_res = windmill_common::materialization::record_materialization( + db, + w_id, + req.asset_kind, + &req.asset_path, + &req.partition, + req.status, + req.snapshot_id, + req.row_count, + req.job_id, + req.error.as_deref(), + ) + .await + .map_err(|e| anyhow::anyhow!("{e:#}")); + // Schema capture is a separate, independently best-effort write (its + // own transaction for the per-asset advisory lock); a failure here + // must not lose the partition row above. + if let Some(cols) = schema.as_ref() { + if let Err(e) = record_asset_schema_best_effort( + db, + w_id, + meta.asset_kind, + &meta.asset_path, + cols, + snapshot_id, + job_id, + ) + .await + { + tracing::warn!("failed to record captured asset schema: {e:#}"); + } + } + partition_res + } + Connection::Http(client) => { + crate::agent_workers::record_materialization_from_agent_http(client, w_id, &req).await + } + }; + if let Err(e) = res { + tracing::warn!("failed to record materialization state: {e:#}"); + } +} + +// Open a short transaction (needed for the per-asset advisory lock) and upsert +// the captured schema version. Isolated so its tx lifetime doesn't entangle the +// partition write. +async fn record_asset_schema_best_effort( + db: &windmill_common::DB, + w_id: &str, + asset_kind: windmill_common::assets::AssetKind, + asset_path: &str, + columns: &[windmill_common::materialization::SchemaColumn], + snapshot_id: Option, + job_id: Uuid, +) -> anyhow::Result<()> { + let mut tx = db.begin().await?; + windmill_common::materialization::record_asset_schema( + &mut tx, + w_id, + asset_kind, + asset_path, + columns, + snapshot_id, + Some(job_id), + ) + .await?; + tx.commit().await?; + Ok(()) +} + pub async fn do_duckdb( job: &MiniPulledJob, client: &AuthedClient, @@ -68,7 +532,54 @@ pub async fn do_duckdb( let mut hidden_passwords = hidden_passwords.clone(); let mut bigquery_credentials = None; + // Materialization (`// materialize`): rewrite a wrap script into managed + // DDL (its synthetic target ATTACH is resolved by the transform pass + // below, like the user's own ATTACH); a literal script is left as-is. + // `materialize` also carries what to record once the run finishes. + let partition_value: Option = job + .args + .as_ref() + .and_then(|a| a.0.get(windmill_common::partition::PARTITION_ARG)) + .and_then(|rv| serde_json::from_str::(rv.get()).ok()) + .filter(|s| !s.is_empty()); + let materialize = if query.contains("materialize") || query.contains("data_test") { + // Custom-test bodies need a DB read; fetch them first so the codegen + // itself stays pure/sync. + let custom_test_bodies = + fetch_custom_test_bodies(query, conn, &job.workspace_id).await?; + build_materialized_query(query, partition_value.as_deref(), &custom_test_bodies)? + } else { + None + }; + // Parse the signature from the ORIGINAL script: managed materialize wraps + // the trailing SELECT and strips line comments, which drops the + // `-- $name (type)` arg declarations while their `$name` references + // survive in the embedded SELECT. Parsing args here (pre-wrap) keeps them + // declared so they are still bound — and s3object args translated to + // `s3://` URIs — at run time. let sig = parse_duckdb_sig(query)?.args; + + let materialized_query; + let query: &str = match &materialize { + Some((Some(rewritten), _)) => { + materialized_query = rewritten.clone(); + &materialized_query + } + _ => query, + }; + + // Managed materialize generates its own trailing summary row (asset / + // rows / snapshot_id / data_tests), and data-test enforcement below reads + // the `data_tests` column off that row. The row shape is ours, not the + // user's — so force the full-last-row strategy regardless of any + // `// result_collection` annotation, which would otherwise reshape it + // (e.g. a scalar mode drops every column but the first) and silently + // bypass test enforcement. + let collection_strategy = if matches!(&materialize, Some((Some(_), _))) { + SqlResultCollectionStrategy::LastStatementAllRows + } else { + collection_strategy + }; let mut job_args = build_args_values(job, client, conn).await?; let reserved_variables = @@ -199,6 +710,20 @@ pub async fn do_duckdb( let (result, column_order) = match result { Ok(r) => r, Err(e) => { + if let Some((_, meta)) = &materialize { + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + None, + None, + None, + Some(&e.to_string()), + ) + .await; + } if let Some(s3_proxy_err) = S3_PROXY_LAST_ERRORS_CACHE.get(&client.token) { return Err(Error::ExecutionErr(format!( "{}\n\nS3 Related Error: {}", @@ -210,6 +735,86 @@ pub async fn do_duckdb( } }; + if let Some((_, meta)) = &materialize { + // In wrap mode the job result is the summary read (snapshot_id + + // rows + the per-test breakdown); in literal mode there is none. + let snapshot_id = extract_i64(&result, "snapshot_id"); + let row_count = extract_i64(&result, "rows"); + // Data tests all ran (every check counted in one query); decide + // pass/fail here. Any violation fails the run — the write is already + // committed (like dbt), so the slice is recorded `Failed` and the + // cascade stops. The error lists *every* test so the user sees the + // whole picture, not just the first failure. + let tests = extract_data_tests(&result); + // Captured output schema (gap #2a) — recorded only on the successful + // path below, not on the failure paths (a failed run shouldn't + // advance the asset's recorded schema version). Managed mode ONLY: + // in `// materialize manual` the result is the user's own query + // output (we generate no summary), so an `output_schema` field there + // is caller-shaped and must not be trusted — `materialize` is + // `Some((Some(_), _))` for managed, `Some((None, _))` for manual. + let is_managed = matches!(&materialize, Some((Some(_), _))); + let schema = if is_managed { + extract_schema(&result) + } else { + None + }; + // Defense-in-depth: codegen embedded `n_data_tests` checks, so the + // summary row must carry that many outcomes. Recovering fewer means + // the `data_tests` column was dropped/reshaped before we read it — + // fail loud rather than silently pass unverified tests. + if tests.len() < meta.n_data_tests { + let msg = format!( + "data tests on {}: expected {} test outcome(s) but recovered {} from the \ + result — aborting to avoid a silent pass", + meta.asset_path, + meta.n_data_tests, + tests.len() + ); + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + snapshot_id, + row_count, + None, + Some(&msg), + ) + .await; + return Err(Error::ExecutionErr(msg)); + } + if tests.iter().any(|t| t.violating > 0) { + let breakdown = format_data_test_breakdown(&meta.asset_path, &tests); + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Failed, + snapshot_id, + row_count, + None, + Some(&breakdown), + ) + .await; + return Err(Error::ExecutionErr(breakdown)); + } + record_mat( + conn, + &job.workspace_id, + job.id, + meta, + windmill_common::materialization::MaterializationStatus::Materialized, + snapshot_id, + row_count, + schema, + None, + ) + .await; + } + drop(bigquery_credentials); *column_order_ref = column_order; @@ -638,9 +1243,13 @@ async fn db_resource_to_attach_statements( db_type: &str, extra_args: Option<&str>, ) -> Result> { + // Escape single quotes: the connection string is built from resource fields + // (host/db/user/password) and embedded in a single-quoted DuckDB literal, so an + // unescaped quote in any field would otherwise break out of the ATTACH statement. + let conn_str = format_attach_db_conn_str(db_resource, db_type)?.replace('\'', "''"); let attach_str = format!( "ATTACH '{}' as {} (TYPE {}{});", - format_attach_db_conn_str(db_resource, db_type)?, + conn_str, ident_name, db_type, extra_args.unwrap_or("") @@ -690,13 +1299,18 @@ async fn transform_attach_ducklake( hidden_passwords.lock().unwrap().push(pwd.to_string()); } - let db_conn_str = format_attach_db_conn_str(ducklake.catalog_resource, db_type)?; + // Escape single quotes: db_conn_str, storage and data_path are embedded in + // single-quoted DuckDB literals below, so an unescaped quote in a resource + // field would break out of the ATTACH statement. + let db_conn_str = + format_attach_db_conn_str(ducklake.catalog_resource, db_type)?.replace('\'', "''"); let storage = ducklake .storage .storage .as_deref() - .unwrap_or(DEFAULT_STORAGE); - let data_path = ducklake.storage.path; + .unwrap_or(DEFAULT_STORAGE) + .replace('\'', "''"); + let data_path = ducklake.storage.path.replace('\'', "''"); let extra_args = if let Some(default_extra_args) = ducklake.extra_args { format!("{},{}", extra_args, default_extra_args) @@ -870,6 +1484,42 @@ mod tests { ); } + // Managed `// materialize` may take SQL args (e.g. an s3object uploaded on + // the run form). The wrap strips line comments — including the + // `-- $name (type)` declarations — so the executor parses the signature from + // the original script (done above, before the rewrite) while the `$name` + // references survive inside the wrapped SELECT. This pins both halves of that + // contract so a regression that drops either is caught. + #[test] + fn materialize_preserves_sql_args() { + let script = "-- materialize ducklake://main/rows\n\ + -- $file (s3object)\n\ + SELECT * FROM read_json_auto($file)"; + + // The signature is recoverable from the original (un-wrapped) script. + let sig = parse_duckdb_sig(script).expect("sig parses").args; + let file_arg = sig + .iter() + .find(|a| a.name == "file") + .expect("`$file` declared"); + assert_eq!(file_arg.otyp.as_deref(), Some("s3object")); + + // The wrapped query still references `$file`, so the parsed sig binds it. + // No custom data tests here, so no fetched bodies are needed. + let (rewritten, _) = + build_materialized_query(script, None, &std::collections::HashMap::new()) + .expect("materialize builds") + .expect("materialize present"); + let rewritten = rewritten.expect("managed mode rewrites the query"); + assert!( + rewritten.contains("$file"), + "wrapped query must keep the `$file` reference, got:\n{rewritten}" + ); + // The declaration comment is gone (wrap strips line comments) — which is + // exactly why the sig must come from the original, not the rewrite. + assert!(!rewritten.contains("-- $file")); + } + // Tests for parse_attach_db_resource function #[test] fn test_parse_attach_db_resource_postgres_res_prefix() { @@ -1228,4 +1878,79 @@ mod tests { let serialized = serde_json::to_string(&arg).unwrap(); assert!(serialized.contains("\"json_value\":{\"key\":\"value\"}")); } + + fn raw(s: &str) -> Box { + serde_json::from_str(s).unwrap() + } + + #[test] + fn extract_data_tests_parses_nested_array() { + // The real result shape: an array of one summary row carrying a nested + // `data_tests` array (how the FFI serialises the list-of-struct). + let r = raw( + r#"[{"rows":3,"snapshot_id":17,"materialized":"ducklake://a/b", + "data_tests":[{"test":"unique(order_id)","violating":0}, + {"test":"accepted_values(status)","violating":2}]}]"#, + ); + let out = extract_data_tests(&r); + assert_eq!(out.len(), 2); + assert_eq!(out[0].name, "unique(order_id)"); + assert_eq!(out[0].violating, 0); + assert_eq!(out[1].name, "accepted_values(status)"); + assert_eq!(out[1].violating, 2); + } + + #[test] + fn extract_data_tests_handles_string_encoded_and_absent() { + // Fallback: some serialisations surface the list-of-struct as a JSON string. + let s = raw(r#"{"data_tests":"[{\"test\":\"not_null(x)\",\"violating\":1}]"}"#); + let out = extract_data_tests(&s); + assert_eq!(out.len(), 1); + assert_eq!(out[0].name, "not_null(x)"); + assert_eq!(out[0].violating, 1); + // Absent column (no tests) -> empty, no panic. + assert!(extract_data_tests(&raw(r#"[{"rows":3}]"#)).is_empty()); + } + + #[test] + fn extract_schema_parses_nested_and_string_encoded() { + // Real shape: the summary row carries a nested `output_schema` + // list-of-struct from the DESCRIBE fold. + let r = raw( + r#"[{"materialized":"ducklake://a/b","rows":3,"snapshot_id":17, + "output_schema":[{"name":"order_id","type":"BIGINT"}, + {"name":"status","type":"VARCHAR"}]}]"#, + ); + let cols = extract_schema(&r).expect("schema present"); + assert_eq!(cols.len(), 2); + assert_eq!(cols[0].name, "order_id"); + assert_eq!(cols[0].data_type, "BIGINT"); + assert_eq!(cols[1].name, "status"); + assert_eq!(cols[1].data_type, "VARCHAR"); + // Fallback: FFI serialised the list-of-struct as a JSON string. + let s = raw(r#"{"output_schema":"[{\"name\":\"x\",\"type\":\"INTEGER\"}]"}"#); + let cols = extract_schema(&s).expect("schema present"); + assert_eq!(cols.len(), 1); + assert_eq!(cols[0].name, "x"); + assert_eq!(cols[0].data_type, "INTEGER"); + // Absent column (literal/manual mode) -> None, no panic. + assert!(extract_schema(&raw(r#"[{"rows":3}]"#)).is_none()); + } + + #[test] + fn format_data_test_breakdown_lists_all_with_marks() { + let tests = vec![ + DataTestOutcome { name: "unique(order_id)".into(), violating: 1 }, + DataTestOutcome { name: "not_null(user_id)".into(), violating: 0 }, + DataTestOutcome { name: "accepted_values(status)".into(), violating: 2 }, + ]; + let msg = format_data_test_breakdown("analytics/orders", &tests); + assert_eq!( + msg, + "data tests failed on analytics/orders (2/3 failed):\n \ + ✗ unique(order_id) — 1 violating row(s)\n \ + ✓ not_null(user_id)\n \ + ✗ accepted_values(status) — 2 violating row(s)" + ); + } } diff --git a/backend/windmill-worker/src/lib.rs b/backend/windmill-worker/src/lib.rs index f8b3edd069..9fa0b514c4 100644 --- a/backend/windmill-worker/src/lib.rs +++ b/backend/windmill-worker/src/lib.rs @@ -22,6 +22,12 @@ mod r_executor; mod ai; mod ai_executor; + +// Exposed for the MCP resource-authorization regression test +// (tests/mcp_resource_authz.rs): the AI agent worker must load MCP resources +// through the job's permissioned client, not the raw DB pool. +#[cfg(feature = "mcp")] +pub use ai::utils::{load_mcp_tools, McpResourceConfig}; mod bun_executor; pub mod common; mod config; diff --git a/backend/windmill-worker/src/pwsh_executor.rs b/backend/windmill-worker/src/pwsh_executor.rs index 1ce2eb09b3..c089930b42 100644 --- a/backend/windmill-worker/src/pwsh_executor.rs +++ b/backend/windmill-worker/src/pwsh_executor.rs @@ -21,13 +21,17 @@ const NSJAIL_CONFIG_RUN_POWERSHELL_CONTENT: &str = lazy_static::lazy_static! { static ref RE_POWERSHELL_IMPORTS: Regex = Regex::new(r#"^\s*Import-Module\s+(?:-Name\s+)?"?([^\s"]+)"?(?:\s+-RequiredVersion\s+"?([^\s"]+)"?)?"#).unwrap(); + // Module names are interpolated into PowerShell string literals in the install + // script, so they must not contain anything that could break out of a quoted + // string. Restrict to characters valid in real module names. + static ref RE_SAFE_MODULE_NAME: Regex = Regex::new(r"^[a-zA-Z0-9._-]+$").unwrap(); } use crate::{ common::{ build_args_map, build_command_with_isolation, get_reserved_variables, read_file, - read_file_content, resolve_nsjail_timeout, resolve_nsjail_tmp_mount_block, start_child_process, - MaybeLock, OccupancyMetrics, + read_file_content, resolve_nsjail_timeout, resolve_nsjail_tmp_mount_block, + start_child_process, MaybeLock, OccupancyMetrics, }, handle_child::handle_child, is_sandboxing_enabled, read_ee_registry_with_workspace_override, DISABLE_NUSER, HOME_ENV, @@ -264,12 +268,31 @@ struct ModuleRequest { version: Option, } +/// Reject module names that could break out of the PowerShell string literal they +/// are interpolated into (e.g. a name containing a single quote), preventing command +/// injection via crafted lock content or script imports (CWE-78). +fn validate_module_name(name: &str) -> Result<(), Error> { + if !RE_SAFE_MODULE_NAME.is_match(name) { + return Err(Error::internal_err(format!( + "Invalid PowerShell module name '{}': only alphanumeric, '.', '_' and '-' are allowed", + name.chars().take(50).collect::() + ))); + } + Ok(()) +} + /// Parse Import-Module statements from PowerShell code into module requests. +/// Imports with an invalid module name are skipped with a warning rather than +/// aborting the whole job. fn parse_script_imports(code: &str) -> Vec { let mut modules = Vec::new(); for line in code.lines() { for cap in RE_POWERSHELL_IMPORTS.captures_iter(line) { let name = cap.get(1).unwrap().as_str().to_string(); + if let Err(e) = validate_module_name(&name) { + tracing::warn!("Skipping PowerShell import with unsafe module name: {e}"); + continue; + } let version = cap.get(2).map(|m| m.as_str().to_string()); modules.push(ModuleRequest { name, version }); } @@ -293,6 +316,7 @@ fn parse_modules_json(content: &str) -> Result, Error> { })?; let mut result = Vec::new(); for (name, version) in modules { + validate_module_name(name)?; let version = match version { serde_json::Value::String(v) if v != "*" => Some(v.clone()), _ => None, @@ -511,13 +535,15 @@ pub async fn handle_powershell_job( let modules_list = modules_to_install .iter() .map(|module_req| { + // Defense-in-depth: escape single quotes so a name/version can never + // break out of the PowerShell string literal, even if validation is + // bypassed. Mirrors val_to_pwsh_param. + let name = module_req.name.replace("'", "''"); if let Some(version) = &module_req.version { - format!( - "@{{ Name = '{}'; Version = '{}' }}", - module_req.name, version - ) + let version = version.replace("'", "''"); + format!("@{{ Name = '{}'; Version = '{}' }}", name, version) } else { - format!("@{{ Name = '{}'; Version = $null }}", module_req.name) + format!("@{{ Name = '{}'; Version = $null }}", name) } }) .collect::>() @@ -528,13 +554,17 @@ pub async fn handle_powershell_job( .replace("{job_id}", &job.id.to_string()) .replace("{has_private_repo}", &format!("${has_private_repo}")) .replace("{has_credentials}", &format!("${has_credentials}")) + // Escape single quotes: these are interpolated into single-quoted + // PowerShell literals ($privateRepoUrl/$privateRepoPat) in the install + // script, so an unescaped quote in the configured repo URL/PAT would + // break out of the literal (same sink as the module names above). .replace( "{private_repo_url}", - &powershell_repo_url.unwrap_or_default(), + &powershell_repo_url.unwrap_or_default().replace("'", "''"), ) .replace( "{private_repo_pat}", - &powershell_repo_pat.unwrap_or_default(), + &powershell_repo_pat.unwrap_or_default().replace("'", "''"), ) .replace("{modules}", &modules_list); let mut cmd = Command::new(POWERSHELL_PATH.as_str()); @@ -1159,4 +1189,50 @@ Write-Host "Hello""#; versions.sort(); assert_eq!(versions, vec!["1.0.0", "1.655.0"]); } + + // --- module name sanitization (CWE-78) tests --- + + #[test] + fn test_validate_module_name_accepts_real_names() { + for name in [ + "PSWriteColor", + "ImportExcel", + "Az.Accounts", + "My_Module-1.0", + ] { + assert!(validate_module_name(name).is_ok(), "{name} should be valid"); + } + } + + #[test] + fn test_validate_module_name_rejects_injection() { + for name in [ + "Mod'; Remove-Item -Recurse / #", + "Mod' }; Invoke-Expression 'evil'; @{ Name = '", + "Mod with space", + "Mod\nImport-Module Evil", + "", + ] { + assert!( + validate_module_name(name).is_err(), + "{name:?} should be rejected" + ); + } + } + + #[test] + fn test_parse_modules_json_rejects_unsafe_name() { + let json = r#"{"modules": {"Mod'; Remove-Item /": "1.0.0"}}"#; + assert!(parse_modules_json(json).is_err()); + } + + #[test] + fn test_parse_script_imports_skips_unsafe_name() { + // Quoted import lets the regex capture a name containing a single quote; + // it must be silently dropped while the safe import is kept. + let code = "Import-Module \"Bad'Name\"\nImport-Module PSWriteColor"; + let modules = parse_script_imports(code); + assert_eq!(modules.len(), 1); + assert_eq!(modules[0].name, "PSWriteColor"); + } } diff --git a/backend/windmill-worker/src/python_executor.rs b/backend/windmill-worker/src/python_executor.rs index f74a523072..9ca79c6852 100644 --- a/backend/windmill-worker/src/python_executor.rs +++ b/backend/windmill-worker/src/python_executor.rs @@ -24,8 +24,10 @@ use tokio::{ use windmill_queue::MiniPulledJob; use uuid::Uuid; -#[cfg(all(feature = "enterprise", feature = "parquet", unix))] + +#[cfg(all(feature = "enterprise", feature = "parquet"))] use windmill_common::ee_oss::{get_license_plan, LicensePlan}; + use windmill_common::{ error::{ self, @@ -35,8 +37,8 @@ use windmill_common::{ scripts::ScriptLang, utils::calculate_hash, worker::{ - copy_dir_recursively, pad_string, split_python_requirements, write_file, Connection, - PyVAlias, PythonAnnotations, WORKER_CONFIG, + copy_dir_recursively, is_allowed_file_location, pad_string, split_python_requirements, + write_file, Connection, PyVAlias, PythonAnnotations, WORKER_CONFIG, }, }; @@ -60,6 +62,13 @@ lazy_static::lazy_static! { static ref PY_CONCURRENT_DOWNLOADS: usize = var("PY_CONCURRENT_DOWNLOADS").ok().map(|flag| flag.parse().unwrap_or(20)).unwrap_or(20); + // uv's HTTP request timeout (seconds). spawn_uv_install uses env_clear(), so a + // UV_HTTP_TIMEOUT set on the worker is dropped unless forwarded explicitly. + // Only forwarded when set; otherwise uv keeps its own default. Lets operators + // raise it for slow/contended private registries ("operation timed out"). + static ref UV_HTTP_TIMEOUT: Option = + var("UV_HTTP_TIMEOUT").ok().filter(|v| !v.is_empty()); + static ref NON_ALPHANUM_CHAR: Regex = regex::Regex::new(r"[^0-9A-Za-z=.-]").unwrap(); @@ -72,7 +81,7 @@ lazy_static::lazy_static! { static ref EPHEMERAL_TOKEN_CMD: Option = var("EPHEMERAL_TOKEN_CMD").ok(); } -#[cfg(all(feature = "enterprise", feature = "parquet", unix))] +#[cfg(all(feature = "enterprise", feature = "parquet"))] lazy_static::lazy_static! { static ref PIPTAR_UPLOAD_CHANNEL: tokio::sync::mpsc::UnboundedSender = { let (tx, rx) = tokio::sync::mpsc::unbounded_channel(); @@ -84,14 +93,14 @@ lazy_static::lazy_static! { }; } -#[cfg(all(feature = "enterprise", feature = "parquet", unix))] +#[cfg(all(feature = "enterprise", feature = "parquet"))] #[derive(Debug)] struct PiptarUploadTask { venv_path: String, cache_dir: String, } -#[cfg(all(feature = "enterprise", feature = "parquet", unix))] +#[cfg(all(feature = "enterprise", feature = "parquet"))] async fn handle_piptar_uploads(mut rx: tokio::sync::mpsc::UnboundedReceiver) { use crate::global_cache::build_tar_and_push; use windmill_object_store::get_object_store; @@ -137,10 +146,10 @@ pub fn has_relative_imports(content: &str) -> bool { RELATIVE_IMPORT_REGEX.is_match(content) } -#[cfg(all(feature = "enterprise", feature = "parquet", unix))] +#[cfg(all(feature = "enterprise", feature = "parquet"))] use crate::global_cache::pull_from_tar; -#[cfg(all(feature = "enterprise", feature = "parquet", unix))] +#[cfg(all(feature = "enterprise", feature = "parquet"))] use windmill_object_store::OBJECT_STORE_SETTINGS; use crate::{ @@ -361,7 +370,9 @@ pub async fn uv_pip_compile( args.extend(["--index-url", url]); } if let Some(host) = TRUSTED_HOST.as_ref() { - args.extend(["--trusted-host", host]); + host.split_whitespace().for_each(|h| { + args.extend(["--trusted-host", h]); + }); } if let Some(cert_path) = INDEX_CERT.as_ref() { args.extend(["--cert", cert_path]); @@ -660,10 +671,16 @@ pub fn compute_python_module_dir(script_path: &str) -> String { .replace("-", "_") .replace("@", "."); if dirs_full.len() > 0 { - dirs_full - .strip_prefix("/") - .unwrap_or(&dirs_full) - .to_string() + let dirs = dirs_full.strip_prefix("/").unwrap_or(&dirs_full); + // This directory is appended to job_dir and written to. Neutralize any + // `.`/`..` segment so the result stays a relative path inside job_dir: a + // Preview path is request-supplied and skips the DB `proper_id` CHECK that + // deployed runnables get, and the `@`->`.` rewrite above can also turn a + // segment like `@.` into `..`. + dirs.split('/') + .map(|seg| if seg == "." || seg == ".." { "_" } else { seg }) + .collect::>() + .join("/") } else { "tmp".to_string() } @@ -1664,6 +1681,10 @@ async fn prepare_wrapper( last }; let module_dir = format!("{}/{}", job_dir, dirs); + // Defense-in-depth: `dirs`/`last` derive from the (request-supplied for + // previews) script path. compute_python_module_dir already neutralizes `..`, + // but assert containment here too so the write can never escape job_dir. + is_allowed_file_location(job_dir, &format!("{dirs}/{last}.py"))?; tokio::fs::create_dir_all(format!("{module_dir}/")).await?; let _ = write_file(&module_dir, &format!("{last}.py"), inner_content)?; @@ -2036,6 +2057,33 @@ Returned from server: py_version - {:?}, py_version_v2 - {:?} lazy_static::lazy_static! { static ref PIP_SECRET_VARIABLE: Regex = Regex::new(r"\$\{PIP_SECRET:([^\s\}]+)\}").unwrap(); + + /// venv paths whose wheel RECORD this process has already verified against + /// disk. A cache entry is only damaged out-of-band (disk-pressure eviction, + /// an interrupted extraction on a shared cache volume, or a corrupt entry + /// that predates this worker), never spontaneously while we keep running, so + /// re-verifying it once per process is enough — every later reuse trusts the + /// in-memory marker and pays only the original single stat. + static ref VERIFIED_VENVS: tokio::sync::Mutex> = + tokio::sync::Mutex::new(HashSet::new()); + + // In-process locks serializing concurrent installs into the same shared + // `venv_p` cache dir; `uv --reinstall` removes a package's .dist-info/RECORD + // before rewriting it, so a sibling install/verify racing it corrupts the + // dir. Keyed by venv_p so distinct deps still install in parallel. + static ref PY_INSTALL_LOCKS: tokio::sync::Mutex>>> = + tokio::sync::Mutex::new(std::collections::HashMap::new()); +} + +/// Returns the in-process install lock for a given target cache dir, creating it +/// on first use. Idle entries (only the map holds a reference) are pruned each +/// call so the map stays bounded by the number of in-flight installs. +async fn get_venv_install_lock(venv_p: &str) -> Arc> { + let mut map = PY_INSTALL_LOCKS.lock().await; + map.retain(|_, v| Arc::strong_count(v) > 1); + map.entry(venv_p.to_string()) + .or_insert_with(|| Arc::new(tokio::sync::Mutex::new(()))) + .clone() } /// Spawn process of uv install @@ -2081,6 +2129,9 @@ async fn spawn_uv_install( if *NATIVE_CERT { vars.push(("UV_NATIVE_TLS", "true")); } + if let Some(timeout) = UV_HTTP_TIMEOUT.as_ref() { + vars.push(("UV_HTTP_TIMEOUT", timeout.as_str())); + } let _owner; if let Some(py_path) = py_path.as_ref() { @@ -2184,6 +2235,9 @@ async fn spawn_uv_install( let mut envs = vec![("PATH", PATH_ENV.as_str())]; envs.push(("HOME", HOME_ENV.as_str())); envs.push(("UV_INDEX_STRATEGY", uv_index_strategy)); + if let Some(timeout) = UV_HTTP_TIMEOUT.as_ref() { + envs.push(("UV_HTTP_TIMEOUT", timeout.as_str())); + } if let Some(mirror) = uv_python_install_mirror.as_ref() { envs.push(("UV_PYTHON_INSTALL_MIRROR", mirror)); } @@ -2192,7 +2246,9 @@ async fn spawn_uv_install( command_args.extend(["--index-url", url]); } if let Some(host) = TRUSTED_HOST.as_ref() { - command_args.extend(["--trusted-host", &host]); + host.split_whitespace().for_each(|h| { + command_args.extend(["--trusted-host", h]); + }); } if *NATIVE_CERT { command_args.extend(["--native-tls"]); @@ -2384,12 +2440,12 @@ pub async fn handle_python_reqs( instant: std::time::Instant, conn: &Connection, ) { - #[cfg(not(all(feature = "enterprise", feature = "parquet", unix)))] + #[cfg(not(all(feature = "enterprise", feature = "parquet")))] { (s3_pull, s3_push) = (false, false); } - #[cfg(all(feature = "enterprise", feature = "parquet", unix))] + #[cfg(all(feature = "enterprise", feature = "parquet"))] if OBJECT_STORE_SETTINGS.read().await.is_none() { (s3_pull, s3_push) = (false, false); } @@ -2473,8 +2529,53 @@ pub async fn handle_python_reqs( req.replace(' ', "").replace('/', "").replace(':', "") ); if metadata(venv_p.clone() + "/.valid.windmill").await.is_ok() { - req_paths.push(venv_p); - in_cache.push(req.to_string()); + // The .valid.windmill marker is written once at creation time, after + // verify_wheel_record passes on the install/pull paths. It is an empty + // file with no binding to the directory contents, so a file dropped + // out-of-band afterwards (disk-pressure eviction, interrupted tar + // extraction on a shared cache volume, or a corrupt entry that + // predates this worker) leaves the marker intact while the wheel is + // incomplete. Re-verify the RECORD once per process so such an entry + // is repaired rather than trusted; VERIFIED_VENVS makes every later + // reuse skip the scan and pay only the single stat above. + let already_verified = VERIFIED_VENVS.lock().await.contains(&venv_p); + let verify_res = if already_verified { + Ok(()) + } else { + verify_wheel_record(&venv_p).await + }; + match verify_res { + Ok(()) => { + if !already_verified { + VERIFIED_VENVS.lock().await.insert(venv_p.clone()); + } + req_paths.push(venv_p); + in_cache.push(req.to_string()); + } + Err(verify_err) => { + tracing::warn!( + workspace_id = %w_id, + job_id = %job_id, + "Local cache for {venv_p} failed wheel RECORD verification, will reinstall: {verify_err}" + ); + append_logs( + &job_id, + w_id, + format!( + "\n[!] cached wheel for {req} failed integrity check, reinstalling: {verify_err}\n" + ), + conn, + ) + .await; + if let Err(rm_err) = tokio::fs::remove_dir_all(&venv_p).await { + tracing::warn!( + workspace_id = %w_id, + "could not remove broken cache dir {venv_p}: {rm_err}" + ); + } + req_with_penv.push((req.to_string(), venv_p)); + } + } } else { // There is no valid or no wheel at all. Regardless of if there is content or not, we will overwrite it with --reinstall flag req_with_penv.push((req.to_string(), venv_p)); @@ -2644,7 +2745,7 @@ pub async fn handle_python_reqs( let mut handles = Vec::with_capacity(total_to_install); // let mem_peak_thread_safe = Arc::new(tokio::sync::Mutex::new(0)); - #[cfg(all(feature = "enterprise", feature = "parquet", unix))] + #[cfg(all(feature = "enterprise", feature = "parquet"))] let is_not_pro = !matches!(get_license_plan().await, LicensePlan::Pro); let total_time = std::time::Instant::now(); @@ -2692,7 +2793,7 @@ pub async fn handle_python_reqs( let pids = pids.clone(); let worker_dir = worker_dir.clone(); - #[cfg(all(feature = "enterprise", feature = "parquet", unix))] + #[cfg(all(feature = "enterprise", feature = "parquet"))] let py_version = py_version.clone(); handles.push(task::spawn(async move { @@ -2710,7 +2811,97 @@ pub async fn handle_python_reqs( ); let start = std::time::Instant::now(); - #[cfg(all(feature = "enterprise", feature = "parquet", unix))] + + // Lock the shared target dir (see PY_INSTALL_LOCKS). In-process lock + // first; only one task per dir then contends the cross-process file + // lock below. Both guards drop on every return path. + let venv_lock = get_venv_install_lock(&venv_p).await; + let _venv_guard = tokio::select! { + _ = kill_rx.recv() => { + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Err(Error::from(anyhow::anyhow!( + "install of {venv_p} canceled while waiting for venv lock" + ))); + } + guard = venv_lock.lock_owned() => guard, + }; + + // Cross-process advisory lock. Best-effort: if the filesystem doesn't + // support flock we log and proceed — verify_wheel_record + job retry + // still guard correctness, just without the dedup. + #[cfg(unix)] + let _venv_file_lock: Option = { + use std::os::unix::io::AsRawFd; + let lock_path = format!("{venv_p}.lock"); + if let Some(parent) = std::path::Path::new(&lock_path).parent() { + let _ = std::fs::create_dir_all(parent); + } + match std::fs::OpenOptions::new().create(true).write(true).open(&lock_path) { + Ok(f) => { + // Bounded wait: a holder that crashes releases the lock (the + // kernel drops it on fd close), but a live-but-stuck holder + // (e.g. uv wedged on a hung mount) would otherwise block us + // forever. After the cap, proceed degraded rather than hang — + // verify_wheel_record + retry still guard correctness. + const MAX_WAIT: std::time::Duration = std::time::Duration::from_secs(300); + let waited_since = std::time::Instant::now(); + loop { + match nix::fcntl::flock(f.as_raw_fd(), nix::fcntl::FlockArg::LockExclusiveNonblock) { + Ok(()) => break Some(f), + // EWOULDBLOCK == EAGAIN on Linux: another holder has the lock. + Err(nix::errno::Errno::EWOULDBLOCK) => { + if waited_since.elapsed() >= MAX_WAIT { + tracing::warn!( + workspace_id = %w_id, + "venv install lock {lock_path} still held after {}s, proceeding without cross-process install lock", + MAX_WAIT.as_secs() + ); + break Some(f); + } + tokio::select! { + _ = kill_rx.recv() => { + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Err(Error::from(anyhow::anyhow!( + "install of {venv_p} canceled while waiting for venv file lock" + ))); + } + _ = tokio::time::sleep(std::time::Duration::from_millis(200)) => {} + } + } + Err(e) => { + tracing::warn!( + workspace_id = %w_id, + "could not flock {lock_path}, proceeding without cross-process install lock: {e}" + ); + break Some(f); + } + } + } + } + Err(e) => { + tracing::warn!( + workspace_id = %w_id, + "could not open install lock file {lock_path}, proceeding without cross-process install lock: {e}" + ); + None + } + } + }; + + // Double-checked: another job (this process or another sharing the + // mount) may have installed this exact dep while we waited on the + // locks. Reuse it instead of reinstalling. + if metadata(format!("{venv_p}/.valid.windmill")).await.is_ok() { + print_success( + false, false, &job_id, &w_id, &req, req_tl, counter_arc, + total_to_install, start, &conn, + ) + .await; + pids.lock().await.get_mut(i).and_then(|e| e.take()); + return Ok(()); + } + + #[cfg(all(feature = "enterprise", feature = "parquet"))] if is_not_pro { if let Some(os) = windmill_object_store::get_object_store().await { tokio::select! { @@ -2892,10 +3083,10 @@ pub async fn handle_python_reqs( } }; - #[cfg(all(feature = "enterprise", feature = "parquet", unix))] + #[cfg(all(feature = "enterprise", feature = "parquet"))] let s3_push = is_not_pro; - #[cfg(not(all(feature = "enterprise", feature = "parquet", unix)))] + #[cfg(not(all(feature = "enterprise", feature = "parquet")))] let s3_push = false; if is_sandboxing_enabled() { @@ -2949,7 +3140,7 @@ pub async fn handle_python_reqs( ) .await; - #[cfg(all(feature = "enterprise", feature = "parquet", unix))] + #[cfg(all(feature = "enterprise", feature = "parquet"))] if s3_push { // Send to upload channel for sequential processing let upload_task = PiptarUploadTask { @@ -3351,6 +3542,17 @@ mod tests { assert_eq!(compute_python_module_dir("f/in/script"), "f/_in"); } + #[test] + fn test_compute_python_module_dir_neutralizes_traversal() { + // A Preview path skips the DB `proper_id` CHECK, so it can carry `..`. + // `..`/`.` segments must be neutralized so the dir stays inside job_dir. + let dirs = compute_python_module_dir("u/x/../../../../tmp/evil/payload"); + assert!(!dirs.split('/').any(|s| s == ".." || s == ".")); + assert_eq!(dirs, "u/x/_/_/_/_/tmp/evil"); + // The `@`->`.` rewrite must not be able to synthesize a `..` segment. + assert_eq!(compute_python_module_dir("u/@./script"), "u/_"); + } + #[test] fn test_compute_py_codegen_basic_args() { let code = "def main(x: str, y: int):\n return x\n"; @@ -3440,4 +3642,179 @@ mod tests { assert_eq!(kept, lines(&["# py: 3.11", "requests==2.0"])); assert_eq!(ignored, lines(&["pyyaml==6.0"])); } + + /// Materialize a fake installed wheel: every file in `files` is created, and + /// `record_entries` is written verbatim as the RECORD (so a test can list a + /// path in RECORD without creating it, to simulate out-of-band loss). + fn write_fake_wheel(root: &std::path::Path, files: &[&str], record_entries: &[&str]) { + for f in files { + let full = root.join(f); + std::fs::create_dir_all(full.parent().unwrap()).unwrap(); + std::fs::write(full, b"x").unwrap(); + } + let dist_info = root.join("pkg-1.0.0.dist-info"); + std::fs::create_dir_all(&dist_info).unwrap(); + std::fs::write(dist_info.join("RECORD"), record_entries.join("\n") + "\n").unwrap(); + } + + #[tokio::test] + async fn test_verify_wheel_record_ok_when_all_present() { + let dir = tempfile::tempdir().unwrap(); + write_fake_wheel( + dir.path(), + &["pkg/__init__.py", "pkg/mod.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "pkg/mod.py,sha256=bbb,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_ok()); + } + + #[tokio::test] + async fn test_verify_wheel_record_err_when_file_missing() { + let dir = tempfile::tempdir().unwrap(); + // RECORD lists pkg/mod.py but we never create it: the exact failure mode + // the customer hit (wmill/s3_reader.py present in RECORD, gone on disk). + write_fake_wheel( + dir.path(), + &["pkg/__init__.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "pkg/mod.py,sha256=bbb,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + let err = verify_wheel_record(dir.path().to_str().unwrap()) + .await + .unwrap_err(); + assert!(err.contains("pkg/mod.py"), "unexpected error: {err}"); + } + + #[tokio::test] + async fn test_verify_wheel_record_err_when_no_dist_info() { + let dir = tempfile::tempdir().unwrap(); + std::fs::write(dir.path().join("loose.py"), b"x").unwrap(); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_err()); + } + + #[tokio::test] + async fn test_verify_wheel_record_skips_absolute_and_escaping_entries() { + let dir = tempfile::tempdir().unwrap(); + // Absolute and `..` RECORD entries are not package-relative and must be + // skipped rather than reported missing. + write_fake_wheel( + dir.path(), + &["pkg/__init__.py"], + &[ + "pkg/__init__.py,sha256=aaa,1", + "/etc/passwd,sha256=ccc,1", + "../outside.py,sha256=ddd,1", + "pkg-1.0.0.dist-info/RECORD,,", + ], + ); + assert!(verify_wheel_record(dir.path().to_str().unwrap()) + .await + .is_ok()); + } + + // Regression tests for the concurrent-install guard. Two jobs installing the + // same uncached dep into the shared `venv_p` used to race uv's `--reinstall`, + // corrupting the on-disk wheel and failing with "Env installation did not + // succeed". The guard serializes those installs. + + #[tokio::test] + async fn test_venv_install_lock_serializes_same_path() { + use std::sync::atomic::{AtomicUsize, Ordering}; + // Same target path => one shared lock => no two tasks install at once. + let active = Arc::new(AtomicUsize::new(0)); + let max_seen = Arc::new(AtomicUsize::new(0)); + let mut handles = vec![]; + for _ in 0..8 { + let active = active.clone(); + let max_seen = max_seen.clone(); + handles.push(tokio::spawn(async move { + let lock = get_venv_install_lock("/cache/py/3.11/samedep==1.0").await; + let _g = lock.lock_owned().await; + let cur = active.fetch_add(1, Ordering::SeqCst) + 1; + max_seen.fetch_max(cur, Ordering::SeqCst); + // Yield so any concurrency would be observed by another task. + tokio::time::sleep(std::time::Duration::from_millis(5)).await; + active.fetch_sub(1, Ordering::SeqCst); + })); + } + for h in handles { + h.await.unwrap(); + } + assert_eq!( + max_seen.load(Ordering::SeqCst), + 1, + "installs into the same target dir must be serialized" + ); + } + + #[tokio::test] + async fn test_venv_install_lock_distinct_paths_are_independent() { + // Different target paths get different locks and never block each other. + let a = get_venv_install_lock("/cache/py/3.11/depA==1.0").await; + let b = get_venv_install_lock("/cache/py/3.11/depB==1.0").await; + let _ga = a.lock_owned().await; + // Holding depA's lock must not prevent acquiring depB's. + assert!( + b.try_lock().is_ok(), + "distinct deps must install in parallel" + ); + // Same path returns the same underlying lock. + let a2 = get_venv_install_lock("/cache/py/3.11/depA==1.0").await; + assert!( + a2.try_lock().is_err(), + "same target dir must map to the same lock" + ); + } + + #[cfg(unix)] + #[tokio::test] + async fn test_venv_file_lock_excludes_across_descriptions() { + // The cross-process layer: flock on a sibling `.lock` excludes a second + // independent open file description (i.e. another worker process) while + // held, and frees it on close. Mirrors the loop in handle_python_reqs. + use nix::fcntl::{flock, FlockArg}; + use std::os::unix::io::AsRawFd; + + let dir = std::env::temp_dir().join("wm_venv_lock_test"); + std::fs::create_dir_all(&dir).unwrap(); + let lock_path = dir.join("dep==1.0.lock"); + + let f1 = std::fs::OpenOptions::new() + .create(true) + .write(true) + .open(&lock_path) + .unwrap(); + flock(f1.as_raw_fd(), FlockArg::LockExclusiveNonblock).unwrap(); + + // A second descriptor (stand-in for another process) cannot take it. + let f2 = std::fs::OpenOptions::new() + .create(true) + .write(true) + .open(&lock_path) + .unwrap(); + assert_eq!( + flock(f2.as_raw_fd(), FlockArg::LockExclusiveNonblock), + Err(nix::errno::Errno::EWOULDBLOCK), + "a second holder must be blocked while the lock is held" + ); + + // Releasing the first lets the second acquire it. + drop(f1); + flock(f2.as_raw_fd(), FlockArg::LockExclusiveNonblock) + .expect("lock must be acquirable once the holder releases it"); + + drop(f2); + let _ = std::fs::remove_file(&lock_path); + } } diff --git a/backend/windmill-worker/src/python_versions.rs b/backend/windmill-worker/src/python_versions.rs index 0ae6b3462a..3a1f35a302 100644 --- a/backend/windmill-worker/src/python_versions.rs +++ b/backend/windmill-worker/src/python_versions.rs @@ -551,6 +551,13 @@ impl PyV { .stdout(Stdio::piped()) .stderr(Stdio::piped()); + if let Some(cert_path) = INDEX_CERT.as_ref() { + child_cmd.env("SSL_CERT_FILE", cert_path); + } + if *NATIVE_CERT { + child_cmd.env("UV_NATIVE_TLS", "true"); + } + if let Some(mirror) = UV_PYTHON_INSTALL_MIRROR.read().await.as_ref() { child_cmd.env("UV_PYTHON_INSTALL_MIRROR", mirror); } diff --git a/backend/windmill-worker/src/r_executor.rs b/backend/windmill-worker/src/r_executor.rs index 9b2bbc6874..0d4219c0df 100644 --- a/backend/windmill-worker/src/r_executor.rs +++ b/backend/windmill-worker/src/r_executor.rs @@ -314,6 +314,38 @@ struct RenvPackage { dependencies: Vec, } +/// Reject renv package names/versions that could break out of the R string +/// literal they are interpolated into (`renv::install("name@version", ...)`), +/// preventing command injection (CWE-78) via crafted renv.lock content. +/// CRAN package names are letters/digits/'.' starting with a letter; versions +/// are dotted numerics optionally with '-'/'_'/'+' separators. +fn validate_renv_package(name: &str, version: &str) -> Result<(), Error> { + let name_ok = name + .chars() + .next() + .map_or(false, |c| c.is_ascii_alphabetic()) + && name.chars().all(|c| c.is_ascii_alphanumeric() || c == '.'); + let version_ok = !version.is_empty() + && version + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_' | '+')); + if !name_ok || !version_ok { + return Err(Error::ExecutionErr(format!( + "Invalid renv package name '{}' / version '{}': name must be alphanumeric or '.', \ + version alphanumeric or '.-_+'", + name.chars().take(50).collect::(), + version.chars().take(50).collect::(), + ))); + } + Ok(()) +} + +/// Escape a value for safe interpolation into an R double-quoted string literal: +/// backslash and double-quote are the only metacharacters inside `"..."`. +fn escape_r_double_quoted(s: &str) -> String { + s.replace('\\', "\\\\").replace('"', "\\\"") +} + /// Parse renv.lock JSON and extract package info including dependency edges. fn parse_renv_lock(lockfile: &str) -> Result, Error> { let lock: serde_json::Value = serde_json::from_str(lockfile) @@ -378,6 +410,7 @@ fn parse_renv_lock(lockfile: &str) -> Result, Error> { // Skip renv itself — it's already loaded and reinstalling it while // loaded triggers a noisy "Restart your R session" message. if !pkg_name.is_empty() && !version.is_empty() && pkg_name != "renv" { + validate_renv_package(&pkg_name, &version)?; result.push(RenvPackage { name: pkg_name, version, repo_url, dependencies }); } } @@ -493,9 +526,9 @@ async fn install<'a>( r#"options(renv.verbose = {verbose_r}, renv.config.install.verbose = {install_verbose_r}, renv.config.restart.enabled = FALSE); renv::install("{pkg}@{version}", library = "{lib}", dependencies = FALSE)"#, verbose_r = verbose_r, install_verbose_r = install_verbose_r, - pkg = dependency.custom_payload.pkg, - version = dependency.custom_payload.version, - lib = install_lib, + pkg = escape_r_double_quoted(&dependency.custom_payload.pkg), + version = escape_r_double_quoted(&dependency.custom_payload.version), + lib = escape_r_double_quoted(&install_lib), ), // install.packages fallback (no version pinning): // &format!( @@ -730,3 +763,65 @@ tryCatch({{ spread = spread, )) } + +#[cfg(test)] +mod tests { + use super::{escape_r_double_quoted, parse_renv_lock, validate_renv_package}; + + #[test] + fn test_validate_renv_package_accepts_real() { + for (n, v) in [ + ("ggplot2", "3.4.4"), + ("data.table", "1.14.8"), + ("Rcpp", "1.0.11"), + ("renv", "1.0.3"), + ("pkg", "1.2-3"), + ("x", "0.9.8.9000"), + ] { + assert!( + validate_renv_package(n, v).is_ok(), + "{n}@{v} should be valid" + ); + } + } + + #[test] + fn test_validate_renv_package_rejects_injection() { + // name breakouts, version breakouts, leading non-letter, empty + for (n, v) in [ + ("ggplot2\", library=system(\"id\"))#", "1.0"), + ("ok", "1.0\"); system(\"id\"); (\""), + ("ok", "1.0\\\"x"), + ("9pkg", "1.0"), + ("pkg name", "1.0"), + ("ok", ""), + ] { + assert!( + validate_renv_package(n, v).is_err(), + "{n:?}@{v:?} should be rejected" + ); + } + } + + #[test] + fn test_escape_r_double_quoted() { + assert_eq!(escape_r_double_quoted(r#"a"b"#), r#"a\"b"#); + assert_eq!(escape_r_double_quoted(r"a\b"), r"a\\b"); + // backslash escaped before quote so \" cannot be reinterpreted + assert_eq!(escape_r_double_quoted(r#"\""#), r#"\\\""#); + } + + #[test] + fn test_parse_renv_lock_rejects_unsafe_name() { + let lock = r#"{"Packages": {"evil": {"Package": "evil\"); system(\"id\"); (\"", "Version": "1.0"}}}"#; + assert!(parse_renv_lock(lock).is_err()); + } + + #[test] + fn test_parse_renv_lock_accepts_clean() { + let lock = r#"{"Packages": {"ggplot2": {"Package": "ggplot2", "Version": "3.4.4"}}}"#; + let pkgs = parse_renv_lock(lock).unwrap(); + assert_eq!(pkgs.len(), 1); + assert_eq!(pkgs[0].name, "ggplot2"); + } +} diff --git a/backend/windmill-worker/src/result_processor.rs b/backend/windmill-worker/src/result_processor.rs index 60dce36d92..803c113540 100644 --- a/backend/windmill-worker/src/result_processor.rs +++ b/backend/windmill-worker/src/result_processor.rs @@ -35,9 +35,9 @@ use windmill_common::{ use windmill_common::bench::{BenchmarkInfo, BenchmarkIter}; use windmill_queue::{ - append_logs, get_mini_completed_job, is_pre_shaped_wm_failure_result, CanceledBy, FlowRunners, - JobCompleted, MiniCompletedJob, MiniPulledJob, ValidableJson, WrappedError, INIT_SCRIPT_TAG, - MANUAL_FAILURE_ERROR_NAME, + append_logs, asset_dispatch, get_mini_completed_job, is_pre_shaped_wm_failure_result, + CanceledBy, FlowRunners, JobCompleted, MiniCompletedJob, MiniPulledJob, ValidableJson, + WrappedError, INIT_SCRIPT_TAG, MANUAL_FAILURE_ERROR_NAME, }; use serde_json::{json, value::RawValue, Value}; @@ -900,13 +900,16 @@ pub async fn process_completed_job( )) })?; } else if let Some(preprocessed_args) = preprocessed_args { - // Update script args to preprocessed args - sqlx::query!( - "UPDATE v2_job SET args = $1, preprocessed = TRUE WHERE id = $2", - Json(preprocessed_args) as Json>>, - job.id + // Update script args to preprocessed args, but preserve a + // resolved pipeline `partition` (injected before the body ran + // by resolve_partition_for_job). Run identity is immutable — + // the preprocessor must not change or drop it, or the asset + // cascade would read no partition for this producer. + windmill_common::partition::merge_args_preserving_partition( + db, + job.id, + preprocessed_args, ) - .execute(db) .await?; } @@ -930,6 +933,12 @@ pub async fn process_completed_job( if job.kind == JobKind::DeploymentCallback { maybe_post_git_sync_pr_check(db, &job_id, &workspace_id, true, result.get()).await; } + + // Asset-trigger fan-out: best-effort, never propagates errors. + // Internal eligibility checks gate to top-level Script/Preview runs; + // see windmill_queue::asset_dispatch. + asset_dispatch::dispatch_asset_triggers(db, &job).await; + drop(job); add_time!(bench, "add_completed_job END"); @@ -1415,7 +1424,7 @@ pub async fn handle_job_error( db, &parent_job, mem_peak, - canceled_by, + canceled_by.clone(), e, worker_name, false, diff --git a/backend/windmill-worker/src/snowflake_executor.rs b/backend/windmill-worker/src/snowflake_executor.rs index 35051aee98..d3d4555f40 100644 --- a/backend/windmill-worker/src/snowflake_executor.rs +++ b/backend/windmill-worker/src/snowflake_executor.rs @@ -604,6 +604,24 @@ pub async fn do_snowflake( return Err(Error::BadRequest("Missing database argument".to_string())); }; + // Validate before it is interpolated into request URLs as the hostname + // (https://.snowflakecomputing.com/...). + if database.account_identifier.is_empty() + || !database + .account_identifier + .chars() + .all(|c| c.is_ascii_alphanumeric() || matches!(c, '.' | '-' | '_')) + { + return Err(Error::BadRequest(format!( + "Invalid Snowflake account identifier '{}': only alphanumeric, '.', '-' and '_' allowed", + database + .account_identifier + .chars() + .take(64) + .collect::() + ))); + } + let annotations = windmill_common::worker::SqlAnnotations::parse(query); let collection_strategy = if annotations.return_last_result { SqlResultCollectionStrategy::LastStatementAllRows diff --git a/backend/windmill-worker/src/worker.rs b/backend/windmill-worker/src/worker.rs index 8964beb6be..2335a78453 100644 --- a/backend/windmill-worker/src/worker.rs +++ b/backend/windmill-worker/src/worker.rs @@ -44,7 +44,7 @@ use windmill_common::{ schema::{should_validate_schema, SchemaValidator}, utils::{create_directory_async, WarnAfterExt}, worker::{ - make_pull_query, write_file, Connection, HttpClient, MAX_TIMEOUT, + is_allowed_file_location, make_pull_query, write_file, Connection, HttpClient, MAX_TIMEOUT, MIN_PERIODIC_SCRIPT_INTERVAL_SECONDS, ROOT_CACHE_DIR, ROOT_CACHE_NOMOUNT_DIR, WINDMILL_DIR, }, worker_group_job_stats::JobStatsMap, @@ -3808,6 +3808,9 @@ pub async fn handle_queued_job( worker_name, flow_runners, &killpill_rx, + // A freshly pulled flow job is being executed by a live worker; the prior + // step (if any) completed normally, so this is never unrecoverable here. + false, )) .warn_after_seconds(10) .await @@ -4301,6 +4304,114 @@ async fn try_validate_schema( Ok(()) } +/// Pipeline partition resolution at execution time. The script content is +/// already loaded for this job, so parsing the `// partitioned` annotation +/// here is free (no extra fetch / no DB column). The concrete partition +/// value is resolved exactly once — schedule fire-time for time kinds +/// (anchored on `scheduled_for`, NOT wall-clock, so a chain crossing +/// midnight stays coherent) or the triggering payload for `dynamic`. It is +/// then (a) injected into the in-memory args the body sees and (b) +/// persisted back to `v2_job.args` so the asset-dispatch cascade reads the +/// same value at completion and propagates it downstream (run identity is +/// immutable — never re-resolve once set). +/// +/// `Ok(Some(job))` = a value was injected (caller must use the returned +/// clone). `Ok(None)` = nothing to do (no `// partitioned`, or the +/// partition is already set: explicit / backfill / cascade-propagated, or +/// before the `start` anchor). `Err` fails the job with a clear message +/// (partitioned but unresolvable — e.g. `dynamic` with no payload). +async fn resolve_partition_for_job( + job: &MiniPulledJob, + code: &str, + conn: &Connection, +) -> error::Result<(Option, bool)> { + use windmill_common::partition::{resolve_partition, PARTITION_ARG}; + use windmill_parser::asset_parser::PartitionKind; + + // Only deployed scripts participate in asset pipelines. Cheap substring + // guard so the overwhelming majority of script jobs skip the annotation + // scan; when one might be present we parse *once* here and reuse the result + // for both `in_pipeline` (→ WM_PIPELINE env, read by the wmll.ducklake SDK to + // record state) and `partition` resolution — no second parse downstream. The + // bool is whether the script is a `// pipeline` member. + if !matches!(job.kind, JobKind::Script) + || !(code.contains("pipeline") || code.contains("partitioned")) + { + return Ok((None, false)); + } + let ann = windmill_parser::asset_parser::parse_pipeline_annotations(code); + let in_pipeline = ann.in_pipeline; + let Some(spec) = ann.partition else { + return Ok((None, in_pipeline)); + }; + + // Already resolved upstream — explicit run arg, backfill, or + // cascade-propagated (push_subscriber injects a top-level `partition`). + // Run identity is immutable: use it as-is, do not re-resolve. + let already_set = job.args.as_ref().is_some_and(|a| { + a.0.get(PARTITION_ARG) + .and_then(|v| serde_json::from_str::(v.get()).ok()) + .is_some_and(|s| !s.is_empty()) + }); + if already_set { + return Ok((None, in_pipeline)); + } + + // `dynamic` extracts from the triggering payload (the `trigger` object + // for a cascade/event hop, else the run args themselves). Time kinds + // ignore the payload. + let payload: Option = match &spec.kind { + PartitionKind::Dynamic { .. } => job.args.as_ref().map(|a| { + a.0.get("trigger") + .and_then(|t| serde_json::from_str::(t.get()).ok()) + .unwrap_or_else(|| { + serde_json::Value::Object( + a.0.iter() + .filter_map(|(k, v)| { + serde_json::from_str(v.get()).ok().map(|jv| (k.clone(), jv)) + }) + .collect(), + ) + }) + }), + _ => None, + }; + + let resolved = resolve_partition(&spec, job.scheduled_for, payload.as_ref()) + .map_err(|e| Error::ExecutionErr(format!("partition resolution failed: {e:#}")))?; + let Some(value) = resolved else { + // Before the `start` anchor: this run has no partition to + // materialize. v1 runs it without one (logged) rather than + // introducing a skip-the-queued-job mechanism. + tracing::warn!( + job_id = %job.id, + "partitioned script resolved to no partition (before start anchor); running without one" + ); + return Ok((None, in_pipeline)); + }; + + // Persist back so dispatch_asset_triggers (which reads the producer's + // completed v2_job.args) propagates the same value down the cascade. + if let Some(db) = conn.as_sql() { + windmill_common::partition::set_resolved_partition(db, job.id, &value).await?; + } else { + tracing::warn!( + job_id = %job.id, + "agent worker: resolved partition not persisted; downstream cascade will not propagate it" + ); + } + + // Inject into the in-memory args so the running body sees it. + let mut updated = job.clone(); + let mut map = updated.args.take().map(|j| j.0).unwrap_or_default(); + map.insert( + PARTITION_ARG.to_string(), + windmill_common::worker::to_raw_value(&value), + ); + updated.args = Some(Json(map)); + Ok((Some(updated), in_pipeline)) +} + #[tracing::instrument(level = "trace", skip_all)] async fn handle_code_execution_job( job: &MiniPulledJob, @@ -4456,6 +4567,19 @@ async fn handle_code_execution_job( ), }; + // Pipeline partition resolution: the content is now loaded, so resolve + // `// partitioned` (if any) and shadow `job` with a clone whose args + // carry the resolved `partition` for the rest of execution. + let _job_with_partition; + let (resolved_job, in_pipeline) = resolve_partition_for_job(job, code, conn).await?; + let job = match resolved_job { + Some(j) => { + _job_with_partition = j; + &_job_with_partition + } + None => job, + }; + // For preview jobs, extract modules from args._MODULES if not already set let modules = modules_from_data.clone().or_else(|| { job.args.as_ref().and_then(|args| { @@ -4501,48 +4625,153 @@ async fn handle_code_execution_job( lock, &modules, false, + in_pipeline, ) .await } +/// True when `path` contains only `Normal`/`CurDir` components, i.e. it cannot +/// escape the directory it is joined onto (no `..`, no absolute root, no Windows +/// drive prefix). +fn is_contained_relative_path(path: &str) -> bool { + use std::path::Component; + std::path::Path::new(path) + .components() + .all(|c| matches!(c, Component::Normal(_) | Component::CurDir)) +} + pub async fn write_module_files( job_dir: &str, modules: &std::collections::HashMap, base_dir: Option<&str>, ) -> error::Result<()> { + // base_dir is derived from the runnable path, which on a preview run can + // carry `..` traversal (it is not the validated module-map key). Reject it + // before it is used to build any write target, otherwise a module could + // escape job_dir and write arbitrary files. + if let Some(dir) = base_dir { + if !is_contained_relative_path(dir) { + return Err(error::Error::BadRequest(format!( + "Invalid module base directory (path traversal): {dir}" + ))); + } + } for (relpath, module) in modules { - // Reject path traversal attempts in module paths - if relpath.contains("..") { + // Reject path traversal attempts in module paths (the module-map key). + if !is_contained_relative_path(relpath) { tracing::warn!("Skipping module with path traversal: {relpath}"); continue; } - let full_path = match base_dir { - Some(dir) => format!("{}/{}/{}", job_dir, dir, relpath), - None => format!("{}/{}", job_dir, relpath), + let relpath_from_job_dir = match base_dir { + Some(dir) => format!("{}/{}", dir, relpath), + None => relpath.to_string(), }; - if let Some(parent) = std::path::Path::new(&full_path).parent() { + // Authoritative guard: resolve the path and assert it stays inside job_dir. + let full_path = is_allowed_file_location(job_dir, &relpath_from_job_dir)?; + if let Some(parent) = full_path.parent() { tokio::fs::create_dir_all(parent).await?; } // For Python modules, create __init__.py in each intermediate directory // between base_dir and the module's parent so that relative imports work. if let Some(dir) = base_dir { - let rel = std::path::Path::new(relpath); - let base = std::path::Path::new(job_dir).join(dir); - let mut current = base.clone(); - for component in rel.parent().into_iter().flat_map(|p| p.components()) { + let mut current = std::path::PathBuf::from(dir); + for component in std::path::Path::new(relpath) + .parent() + .into_iter() + .flat_map(|p| p.components()) + { current = current.join(component); - let init_py = current.join("__init__.py"); + let init_py = is_allowed_file_location( + job_dir, + ¤t.join("__init__.py").to_string_lossy(), + )?; if !init_py.exists() { tokio::fs::write(&init_py, "").await?; } } } - tracing::debug!("Writing module file: {full_path}"); + tracing::debug!("Writing module file: {}", full_path.display()); tokio::fs::write(&full_path, &module.content).await?; } Ok(()) } +#[cfg(test)] +mod write_module_files_tests { + use super::*; + use std::collections::HashMap; + use windmill_common::scripts::ScriptLang; + + fn module(content: &str) -> ScriptModule { + ScriptModule { content: content.to_string(), language: ScriptLang::Python3, lock: None } + } + + #[test] + fn contained_relative_path_rejects_traversal_and_absolute() { + assert!(is_contained_relative_path("u/admin/pkg")); + assert!(is_contained_relative_path("./pkg/sub")); + // A `..` in a filename is a valid name, not a traversal. + assert!(is_contained_relative_path("weird..name")); + + assert!(!is_contained_relative_path("u/x/../../../etc")); + assert!(!is_contained_relative_path("../escape")); + assert!(!is_contained_relative_path("/etc/cron.d/wm")); + } + + #[tokio::test] + async fn base_dir_traversal_is_rejected_and_writes_nothing() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + // Sentinel just outside job_dir that a successful traversal would create. + let outside = job.path().parent().unwrap().join("wm_escaped_marker"); + + let mut modules = HashMap::new(); + modules.insert( + "wm_escaped_marker".to_string(), + module("* * * * * root id\n"), + ); + + // base_dir derived from a preview path carrying `..` traversal. + let res = write_module_files(job_dir, &modules, Some("u/x/../../../../../..")).await; + assert!(res.is_err(), "traversal base_dir must be rejected"); + assert!(!outside.exists(), "no file may be written outside job_dir"); + } + + #[tokio::test] + async fn relpath_traversal_is_skipped() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + let outside = job.path().parent().unwrap().join("wm_relpath_escape.py"); + + let mut modules = HashMap::new(); + modules.insert("../wm_relpath_escape.py".to_string(), module("x = 1")); + + write_module_files(job_dir, &modules, None).await.unwrap(); + assert!(!outside.exists()); + } + + #[tokio::test] + async fn legitimate_modules_are_written_with_init_py() { + let job = tempfile::tempdir().unwrap(); + let job_dir = job.path().to_str().unwrap(); + + let mut modules = HashMap::new(); + modules.insert("pkg/sub/mod.py".to_string(), module("VALUE = 42")); + + write_module_files(job_dir, &modules, Some("u/admin")) + .await + .unwrap(); + + let base = job.path().join("u/admin"); + assert_eq!( + std::fs::read_to_string(base.join("pkg/sub/mod.py")).unwrap(), + "VALUE = 42" + ); + assert!(base.join("pkg/__init__.py").exists()); + assert!(base.join("pkg/sub/__init__.py").exists()); + } +} + pub async fn run_language_executor( job: &MiniPulledJob, conn: &Connection, @@ -4567,6 +4796,9 @@ pub async fn run_language_executor( lock: &Option, modules: &Option>, run_inline: bool, + // Whether the script is a `// pipeline` member (parsed once upstream) — sets + // WM_PIPELINE so the wmll.ducklake SDK helpers record materialization state. + in_pipeline: bool, ) -> error::Result> { // Defense-in-depth (GHSA-wxjq-w5pj-jqhx): the entrypoint override is // interpolated verbatim into a code position of the generated language @@ -4929,6 +5161,11 @@ mount {{ #[allow(unused_mut)] let mut envs = build_envs(envs.as_ref())?; + // Signal pipeline context to the script so the wmll.ducklake SDK helpers + // record materialization state (the grid/backfill) and skip it otherwise. + if in_pipeline { + envs.insert("WM_PIPELINE".to_string(), "true".to_string()); + } let Some(language) = language else { return Err(Error::ExecutionErr( @@ -5714,6 +5951,7 @@ pub fn init_worker_internal_server_inline_utils( &None, &None, true, + false, ) .await }) @@ -5795,6 +6033,7 @@ pub fn init_worker_internal_server_inline_utils( &content_info.lockfile, &content_info.modules, true, + false, ) .await }) diff --git a/backend/windmill-worker/src/worker_flow.rs b/backend/windmill-worker/src/worker_flow.rs index 699b8683ec..10e8218a77 100644 --- a/backend/windmill-worker/src/worker_flow.rs +++ b/backend/windmill-worker/src/worker_flow.rs @@ -28,7 +28,7 @@ use serde::{Deserialize, Serialize}; use serde_json::value::RawValue; use serde_json::{json, Value}; use sqlx::types::Json; -use sqlx::{FromRow, Postgres, Transaction}; +use sqlx::{Acquire, FromRow, Postgres, Transaction}; use tracing::instrument; use uuid::Uuid; use windmill_common::auth::get_job_perms; @@ -332,14 +332,14 @@ fn get_stop_after_if_data(stop_after_if: Option<&StopAfterIf>) -> (bool, Option< return (false, None, false); } -async fn get_id_ctx_for_expr( +async fn get_id_ctx_for_expr<'c>( expr: &str, flow: uuid::Uuid, - db: &DB, + e: impl sqlx::PgExecutor<'c>, status: &FlowStatus, ) -> error::Result> { if expr.contains("results.") || expr.contains("results[") || expr.contains("results?.") { - let flow_job = get_mini_pulled_job(db, &flow).await?; + let flow_job = get_mini_pulled_job(e, &flow).await?; if let Some(flow_job) = flow_job { Ok(Some(get_transform_context(&flow_job, "", &status))) } else { @@ -351,7 +351,7 @@ async fn get_id_ctx_for_expr( } async fn evaluate_stop_after_all_iters_if( - db: &DB, + tx: &mut Transaction<'_, Postgres>, stop_after_all_iters_if: &StopAfterIf, module_status: &FlowStatusModule, w_id: &str, @@ -366,9 +366,20 @@ async fn evaluate_stop_after_all_iters_if( flow: uuid::Uuid, status: &FlowStatus, ) -> error::Result<()> { + // Test hook (see test_stop_after_all_iters_if_db_error_isolated_by_savepoint): + // run a query that aborts this (savepoint) transaction so the test can verify the + // caller's savepoint keeps the outer status-update transaction committable. + #[cfg(feature = "failpoints")] + if stop_after_all_iters_if.expr == "__wm_failpoint_abort_tx__" { + sqlx::query("SELECT 1/0") + .execute(&mut **tx) + .await + .map_err(|e| Error::internal_err(format!("failpoint abort_tx: {e:#}")))?; + } + let iters_result = match &module_status { FlowStatusModule::InProgress { flow_jobs: Some(flow_jobs), .. } => { - Arc::new(retrieve_flow_jobs_results(db, w_id, flow_jobs).await?) + Arc::new(retrieve_flow_jobs_results(&mut **tx, w_id, flow_jobs).await?) } _ => { return Err(Error::internal_err(format!( @@ -379,7 +390,8 @@ async fn evaluate_stop_after_all_iters_if( *nresult = Some(iters_result.clone()); // as an optimization, we store the result of all jobs as when stop_early_after_all_iters evaluates to false, it would have to be computed (finished loop/branchall) - let id_ctx = get_id_ctx_for_expr(&stop_after_all_iters_if.expr, flow, db, status).await?; + let id_ctx = + get_id_ctx_for_expr(&stop_after_all_iters_if.expr, flow, &mut **tx, status).await?; let stop_early_after_all_iters = compute_bool_from_expr( &stop_after_all_iters_if.expr, @@ -976,8 +988,15 @@ pub async fn update_flow_status_after_job_completion_internal( .and_then(|x| x.stop_after_all_iters_if.as_ref()) { let args = from_result_to_args(args.as_ref().await.get_ref())?; - if let Err(e) = evaluate_stop_after_all_iters_if( - db, + // Isolate the reads in a savepoint on the same connection: the + // caller below swallows our error and keeps using `tx`, so a DB + // read failure must not leave the outer transaction aborted (that + // would fail the later commit). On error we roll back to the + // savepoint, matching the previous pool-read behaviour where a + // failed read left `tx` usable and the iteration was marked failed. + let mut sp = tx.begin().await?; + let eval_res = evaluate_stop_after_all_iters_if( + &mut sp, stop_after_all_iters_if, module_status, w_id, @@ -992,15 +1011,19 @@ pub async fn update_flow_status_after_job_completion_internal( flow, &old_status, ) - .await - { - tracing::error!("error evaluating stop_after_all_iters_if: {e:#}"); - stop_early = true; - skip_if_stop_early = false; - stop_early_err_msg = Some(format!( - "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", - stop_after_all_iters_if.expr - )); + .await; + match eval_res { + Ok(()) => sp.commit().await?, + Err(e) => { + let _ = sp.rollback().await; + tracing::error!("error evaluating stop_after_all_iters_if: {e:#}"); + stop_early = true; + skip_if_stop_early = false; + stop_early_err_msg = Some(format!( + "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", + stop_after_all_iters_if.expr + )); + } } } @@ -1054,7 +1077,7 @@ pub async fn update_flow_status_after_job_completion_internal( let r = sqlx::query_scalar!( "DELETE FROM parallel_monitor_lock WHERE parent_flow_id = $1 RETURNING last_ping", flow, - ).fetch_optional(db).await.map_err(|e| { + ).fetch_optional(&mut *tx).await.map_err(|e| { Error::internal_err(format!( "error while deleting parallel_monitor_lock: {e:#}" )) @@ -1198,8 +1221,12 @@ pub async fn update_flow_status_after_job_completion_internal( { let args = from_result_to_args(args.as_ref().await.get_ref())?; - if let Err(e) = evaluate_stop_after_all_iters_if( - db, + // See the matching savepoint comment above: isolate the reads so a + // DB read failure (whose error the caller swallows) cannot abort + // the outer transaction and break the later commit. + let mut sp = tx.begin().await?; + let eval_res = evaluate_stop_after_all_iters_if( + &mut sp, stop_after_all_iters_if, module_status, w_id, @@ -1214,14 +1241,18 @@ pub async fn update_flow_status_after_job_completion_internal( flow, &old_status, ) - .await - { - stop_early = true; - skip_if_stop_early = false; - stop_early_err_msg = Some(format!( - "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", - stop_after_all_iters_if.expr - )); + .await; + match eval_res { + Ok(()) => sp.commit().await?, + Err(e) => { + let _ = sp.rollback().await; + stop_early = true; + skip_if_stop_early = false; + stop_early_err_msg = Some(format!( + "Error evaluating stop_after_all_iters_if expression `{}`: {e:#}", + stop_after_all_iters_if.expr + )); + } } } } @@ -1279,7 +1310,11 @@ pub async fn update_flow_status_after_job_completion_internal( }), ) } else { - let inc = if continue_on_error { + // An unrecoverable failure (worker crash/OOM) must reach the error handler + // even on a continue_on_error step, so don't advance the step counter past + // the failed module — otherwise the flow would silently continue to the next + // step and hide the worker death. + let inc = if !unrecoverable && continue_on_error { let retry = current_module .as_ref() .and_then(|x| x.retry.clone()) @@ -1458,7 +1493,7 @@ pub async fn update_flow_status_after_job_completion_internal( match &new_status { Some(FlowStatusModule::Success { flow_jobs: Some(jobs), .. }) | Some(FlowStatusModule::Failure { flow_jobs: Some(jobs), .. }) => { - Arc::new(retrieve_flow_jobs_results(db, w_id, jobs).await?) + Arc::new(retrieve_flow_jobs_results(&mut *tx, w_id, jobs).await?) } _ => result.clone(), } @@ -1600,6 +1635,7 @@ pub async fn update_flow_status_after_job_completion_internal( windmill_common::runnable_settings::RunnableSettings { debouncing_settings: debouncing_hash, concurrency_settings: None, + retry_settings: None, }, db, ) @@ -1708,7 +1744,16 @@ pub async fn update_flow_status_after_job_completion_internal( _ if stop_early => stop_early_err_msg.is_some() && flow_value.failure_module.is_some(), // if stop_early_err_msg some, we want to trigger the error handler before stopping the flow, if any _ if flow_job.is_canceled() => false, true => !is_last_step, - false if unrecoverable => false, + // An unrecoverable failure (a step killed by a worker crash/OOM and surfaced by + // the zombie handler, or an error raised while updating the flow status itself) + // must not be retried or silently skipped, but it should still trigger the flow's + // error handler: an OOM/worker death is precisely when the error handler is expected + // to run. Continue the flow only to reach the failure module, never to retry. + false if unrecoverable => { + !is_failure_step + && !has_triggered_error_handler + && flow_value.failure_module.is_some() + } false if skip_seq_branch_failure || skip_loop_failures || continue_on_error => { !is_last_step } @@ -2059,6 +2104,7 @@ pub async fn update_flow_status_after_job_completion_internal( worker_name, flow_runners, &killpill_rx, + unrecoverable, )) .warn_after_seconds(10) .await @@ -2259,8 +2305,8 @@ async fn set_success_and_duration_in_flow_job_success<'c>( Ok(()) } -async fn retrieve_flow_jobs_results( - db: &DB, +async fn retrieve_flow_jobs_results<'c>( + e: impl sqlx::PgExecutor<'c>, w_id: &str, job_uuids: &Vec, ) -> error::Result> { @@ -2271,7 +2317,7 @@ async fn retrieve_flow_jobs_results( job_uuids.as_slice(), w_id ) - .fetch_all(db) + .fetch_all(e) .await? .into_iter() .map(|br| (br.id, br.result)) @@ -2749,6 +2795,10 @@ pub async fn handle_flow( worker_name: &str, flow_runners: Option>, killpill_rx: &tokio::sync::broadcast::Receiver<()>, + // The previous step failed unrecoverably (e.g. a worker crash/OOM surfaced by the + // zombie handler). The next pushed step can only be the error handler (failure + // module), and it must not be pinned to the dead worker via same_worker. + unrecoverable: bool, ) -> anyhow::Result<()> { let flow = flow_data.value(); @@ -2922,6 +2972,7 @@ pub async fn handle_flow( flow_runners.clone(), job_completed_tx.clone(), &killpill_rx, + unrecoverable, )) .warn_after_seconds(10) .await?; @@ -3104,6 +3155,10 @@ async fn push_next_flow_job( flow_runners: Option>, job_completed_tx: JobCompletedSender, killpill_rx: &tokio::sync::broadcast::Receiver<()>, + // The prior step failed unrecoverably (worker crash/OOM). The only step pushed + // from here is the error handler, which must run on a live worker rather than + // being pinned to the dead one via same_worker / dedicated runners. + unrecoverable: bool, ) -> error::Result { let job_root = flow_job .flow_innermost_root_job @@ -3268,7 +3323,7 @@ async fn push_next_flow_job( } // Compute and initialize last_job_result - let arc_last_job_result = if status_module.is_failure() { + let mut arc_last_job_result = if status_module.is_failure() { // if job is being retried, pass the result of its previous failure last_job_result.unwrap_or_else(|| Arc::new(to_raw_value(&json!("{}")))) } else if matches!(step, Step::Step { idx: 0, .. }) || step.is_preprocessor_step() { @@ -3657,7 +3712,10 @@ async fn push_next_flow_job( } }; - let retry = if matches!(&status_module, FlowStatusModule::Failure { .. },) { + // An unrecoverable failure (worker crash/OOM) must not be retried — the original worker + // and its state are gone — so skip retry evaluation and fall straight through to the + // failure module below. + let retry = if !unrecoverable && matches!(&status_module, FlowStatusModule::Failure { .. },) { let retry = &module.retry.clone().unwrap_or_default(); evaluate_retry( retry, @@ -3672,8 +3730,13 @@ async fn push_next_flow_job( None }; let get_args_from_id = match &status_module { + // `|| unrecoverable`: a worker crash/OOM routes to the failure module even on a + // continue_on_error step (whose failures are normally tolerated), matching the + // `unrecoverable` decision in update_flow_status_after_job_completion_internal. FlowStatusModule::Failure { job, .. } - if retry.as_ref().is_some() || !module.continue_on_error.is_some_and(|x| x) => + if retry.as_ref().is_some() + || !module.continue_on_error.is_some_and(|x| x) + || unrecoverable => { if let Some((fail_count, retry_in)) = retry { tracing::debug!( @@ -3699,6 +3762,30 @@ async fn push_next_flow_job( .context("update flow retry")?; status_module = FlowStatusModule::WaitingForPriorSteps { id: status_module.id() }; + + // The failed attempt's error has already been consumed by `evaluate_retry` + // above. Restore `previous_result` to the preceding step's result (or the + // flow args for the first step) so that predicates re-evaluated for the + // retry (skip_if, loop iterator expressions, ...) don't see the failed + // attempt's error instead. Like the suspend/restart path above, this + // falls back to `"{}"` when the preceding step has no Success status + // (e.g. it failed with continue_on_error). + if !matches!(step, Step::FailureStep) { + arc_last_job_result = if matches!(step, Step::Step { idx: 0, .. }) + || step.is_preprocessor_step() + { + Arc::new(to_raw_value(&flow_job.args)) + } else { + match get_previous_job_result(db, flow_job.workspace_id.as_str(), &status) + .warn_after_seconds(3) + .await? + { + None => Arc::new(to_raw_value(&json!("{}"))), + Some(previous_job_result) => Arc::new(previous_job_result), + } + }; + } + // we get the args from the last failed job status.retry.failed_jobs.last() /* Start the failure module ... */ @@ -3998,7 +4085,8 @@ async fn push_next_flow_job( .as_ref() .is_some_and(|fr| fr.job_id == flow_job.id); - let continue_with_runners = (start_runners || (flow_runners.is_some() && !do_not_pass_runners)) + let continue_with_runners = !unrecoverable + && (start_runners || (flow_runners.is_some() && !do_not_pass_runners)) && module.suspend.is_none() && module.sleep.is_none(); @@ -4007,8 +4095,13 @@ async fn push_next_flow_job( let job_same_worker = flow_job.same_worker && matches!(flow_job.kind, JobKind::Flow) && flow_job.runnable_id.is_some(); - let continue_on_same_worker = - (flow.same_worker || job_same_worker) && module.suspend.is_none() && module.sleep.is_none(); + // After an unrecoverable failure the original worker is gone, so the error handler + // step is pushed as a regular queued job (any live worker can pick it up) instead of + // being signaled to the dead worker via same_worker — which would strand it forever. + let continue_on_same_worker = !unrecoverable + && (flow.same_worker || job_same_worker) + && module.suspend.is_none() + && module.sleep.is_none(); /* Finally, push the job into the queue */ let mut uuids = vec![]; @@ -4265,7 +4358,7 @@ async fn push_next_flow_job( .filter(|t| !t.is_empty() && *t != flow_job.tag.as_str()) { check_tag_available_for_workspace_internal( - &db, + db, &flow_job.workspace_id, tag_str, email, @@ -6118,8 +6211,15 @@ fn needs_resume(flow: &FlowValue, status: &FlowStatus) -> Option<(Suspend, Uuid) return None; } - if let &FlowStatusModule::Success { job, .. } = status.modules.get(prev)? { - Some((suspend.unwrap(), job)) + if let &FlowStatusModule::Success { job, skipped, .. } = status.modules.get(prev)? { + // A step skipped via skip_if never ran, so its suspend/approval was never + // armed and no resume event will ever arrive. Gating the next step on it + // would park the flow forever. + if skipped { + None + } else { + Some((suspend.unwrap(), job)) + } } else { None } diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index 66f8277026..f8cc748bdb 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.723.0"; +export const VERSION = "v1.742.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ diff --git a/cli/build-npm.ts b/cli/build-npm.ts index 665021ffdc..5c9353dd21 100644 --- a/cli/build-npm.ts +++ b/cli/build-npm.ts @@ -28,6 +28,9 @@ rmSync(outDir, { recursive: true, force: true }); // Build with bun — bundle everything except esbuild (platform-specific binary), // svelte (optional, only needed for `wmill app bundle/dev`), and parser packages // (loaded at runtime via init() with readFileSync for the .wasm binary). +// esbuild-wasm is not a dependency at all: the host/binary-mismatch fallback in +// esbuild_loader.ts downloads and caches the whole esbuild-wasm package at +// runtime, so it stays out of the bundle and the published dependencies. console.log("Bundling with bun build..."); const buildResult = Bun.spawnSync([ "bun", "build", "src/main.ts", diff --git a/cli/src/commands/app/bundle.ts b/cli/src/commands/app/bundle.ts index e2f8553e22..07ee266145 100644 --- a/cli/src/commands/app/bundle.ts +++ b/cli/src/commands/app/bundle.ts @@ -6,6 +6,7 @@ import * as log from "../../core/log.ts"; import { colors } from "@cliffy/ansi/colors"; import * as windmillUtils from "@windmill-labs/shared-utils"; import { readTextFile, readTextFileSync } from "../../utils/utils.ts"; +import { getEsbuild, stopEsbuild } from "../../utils/esbuild_loader.ts"; export interface BundleOptions { entryPoint?: string; outDir?: string; @@ -170,8 +171,9 @@ export async function ensureNodeModules(appDir?: string): Promise { export async function createBundle( options: BundleOptions = {} ): Promise { - // Dynamically import esbuild - const esbuild = await import("esbuild"); + // Native esbuild with a transparent esbuild-wasm fallback on host/binary + // version mismatch (see esbuild_loader.ts). + const esbuild = await getEsbuild(); // Detect frameworks to determine default entry point. // Use the entryPoint's directory if provided, otherwise fall back to cwd. @@ -286,6 +288,10 @@ export async function createBundle( outfile, sourcemap, minify, + // Keep outputs in memory: esbuild-wasm cannot write to the filesystem + // ("write" option unavailable), and the dist files were discarded after the + // read anyway. Native esbuild supports write:false + outputFiles too. + write: false as const, define: { "process.env.NODE_ENV": production ? '"production"' : '"development"', }, @@ -307,29 +313,24 @@ export async function createBundle( log.info(colors.green("✅ Bundle created successfully")); - // Read the generated files - const jsPath = path.join(process.cwd(), outfile); - const cssPath = path.join(process.cwd(), outDir, "bundle.css"); + const outputFiles = result.outputFiles ?? []; + const jsFile = outputFiles.find((f) => f.path.endsWith(".js")); + const cssFile = outputFiles.find((f) => f.path.endsWith(".css")); - if (!fs.existsSync(jsPath)) { - throw new Error(`Expected JS bundle at ${jsPath} but file not found`); + if (!jsFile) { + throw new Error("Expected a JS bundle in esbuild output but none found"); } - const jsContent = readTextFileSync(jsPath); - const cssContent = fs.existsSync(cssPath) - ? readTextFileSync(cssPath) - : ""; - try { fs.rmSync(distDir, { recursive: true }); } catch { //ignore } - return { js: jsContent, css: cssContent }; - + return { js: jsFile.text, css: cssFile?.text ?? "" }; + } finally { - // Stop esbuild - await esbuild.stop(); + // Stop the native esbuild service so the process can exit (no-op for wasm). + await stopEsbuild(); } } diff --git a/cli/src/commands/app/dev.ts b/cli/src/commands/app/dev.ts index c73a8685c8..1a226ec5ef 100644 --- a/cli/src/commands/app/dev.ts +++ b/cli/src/commands/app/dev.ts @@ -437,7 +437,11 @@ async function dev(opts: DevOptions, appFolder?: string) { const rawApp = (await yamlParseFile(rawAppPath)) as any; const appPath = rawApp?.custom_path ?? "u/unknown/newapp"; - // Dynamically import esbuild only when the dev command is called + // Dynamically import esbuild only when the dev command is called. + // Native-only here (no esbuild-wasm fallback via getEsbuild): dev is a local + // interactive command that relies on context()/watch, whose semantics under + // wasm are untested. The host/binary-mismatch fallback covers the bundling + // paths that run on workers/CI via `wmill sync push`. const esbuild = await import("esbuild"); const host = opts.host ?? DEFAULT_HOST; diff --git a/cli/src/commands/datatable/datatable.ts b/cli/src/commands/datatable/datatable.ts index c86c971a90..1291d4cc13 100644 --- a/cli/src/commands/datatable/datatable.ts +++ b/cli/src/commands/datatable/datatable.ts @@ -41,6 +41,61 @@ async function run( await runCatalogQuery(opts, "datatable", name, sql); } +async function create( + opts: GlobalOptions & { resource?: string; force?: boolean }, + name?: string, +) { + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + const dtName = name ?? DEFAULT_DATATABLE_NAME; + + const existing = await wmill.listDataTables({ + workspace: workspace.workspaceId, + }); + if (existing.some((d) => d.name === dtName)) { + throw new Error(`Datatable '${dtName}' already exists in this workspace`); + } + // edit_datatable_config replaces the whole settings object, and fork + // metadata on existing datatables can't be read back through the API — + // so only touch a non-empty config when explicitly asked to. + if (existing.length > 0 && !opts.force) { + throw new Error( + `Workspace already has datatable(s): ${existing + .map((d) => d.name) + .join(", ")}. Re-run with --force to add '${dtName}' ` + + "(note: fork metadata on existing datatables is not preserved)", + ); + } + + const datatables: Record< + string, + { database: { resource_type: "postgresql" | "instance"; resource_path?: string } } + > = {}; + for (const d of existing) { + datatables[d.name] = { + database: { + resource_type: d.resource_type as "postgresql" | "instance", + resource_path: d.resource_path ?? undefined, + }, + }; + } + datatables[dtName] = opts.resource + ? { database: { resource_type: "postgresql", resource_path: opts.resource } } + : { database: { resource_type: "instance", resource_path: "datatable_db" } }; + + await wmill.editDataTableConfig({ + workspace: workspace.workspaceId, + requestBody: { settings: { datatables } }, + }); + log.info( + `Datatable '${dtName}' created (${ + opts.resource + ? `postgresql resource ${opts.resource}` + : "instance-backed" + }). Scripts can now use datatable://${dtName}.`, + ); +} + async function serve( opts: GlobalOptions & { port?: number; host?: string; password?: string }, ) { @@ -69,6 +124,20 @@ const command = new Command() "Output only the final result as JSON. Useful for scripting.", ) .action(run as any) + .command( + "create", + "register a datatable database in the workspace (default: instance-backed 'main') so scripts can use datatable://", + ) + .arguments("[name:string]") + .option( + "--resource ", + "Back the datatable with an existing postgresql resource path instead of the instance database", + ) + .option( + "--force", + "Allow adding to a workspace that already has datatables (fork metadata on existing ones is not preserved)", + ) + .action(create as any) .command( "serve", "Serve all datatables as a Postgres-wire endpoint (psql, DBeaver, pgAdmin); the client picks the datatable via the database name in its connection string", diff --git a/cli/src/commands/dev/dev.ts b/cli/src/commands/dev/dev.ts index 682cc8119d..f588f5841d 100644 --- a/cli/src/commands/dev/dev.ts +++ b/cli/src/commands/dev/dev.ts @@ -228,6 +228,33 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { const ignore = await ignoreF(opts); const codebases = await listSyncCodebases(opts); + // Resolve relative imports from local (not-yet-deployed) content so dev-page + // previews use locally-edited workspace scripts instead of the deployed + // versions. Diverged local scripts are uploaded to temp storage once here + // and the resulting path -> hash refs ride along on every preview run. + // Computed as a startup snapshot, like `wmill app dev`; restart `wmill dev` + // to pick up later edits to imported workspace scripts. Uses the "all" + // target because the previewed item can change at runtime (picker / + // loadWmPath), so there is no single anchor node. Degrades gracefully + // (undefined) on older backends without the /raw_temp endpoints. + let tempScriptRefs: Record | undefined = undefined; + { + const { buildPreviewTempScriptRefs } = await import( + "../generate-metadata/generate-metadata.ts" + ); + tempScriptRefs = await buildPreviewTempScriptRefs( + workspace, + opts, + codebases, + { kind: "all" }, + ); + if (tempScriptRefs) { + log.info( + `Resolved ${Object.keys(tempScriptRefs).length} locally-edited script(s) for preview relative imports (snapshot — restart wmill dev to refresh)` + ); + } + } + const changesTimeouts: Record> = {}; function watchChanges() { return new Promise((_resolve, _reject) => { @@ -314,6 +341,7 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { flow: localFlow, uriPath: localPath, path: wmFlowPath, + temp_script_refs: tempScriptRefs, }; log.info("Updated " + wmFlowPath); broadcastChanges(currentLastEdit); @@ -337,6 +365,7 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { language: lang, tag: typed?.tag, lock: typed?.lock, + temp_script_refs: tempScriptRefs, }; log.info("Updated " + wmPath); broadcastChanges(currentLastEdit); @@ -350,7 +379,7 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { language: string; tag?: string; lock?: string; - + temp_script_refs?: Record; }; type LastEditFlow = { @@ -358,6 +387,7 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { flow: OpenFlow; uriPath: string; path: string; + temp_script_refs?: Record; }; // Load a resource by its windmill path (e.g., "u/admin/my_script" or "f/my_flow") @@ -399,6 +429,7 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { flow: localFlow, uriPath: flowDir, path: wmPath, + temp_script_refs: tempScriptRefs, }; currentLastEdit = edit; return edit; @@ -421,6 +452,7 @@ export async function dev(opts: GlobalOptions & SyncOptions & DevOpts) { language: lang, tag: typed?.tag, lock: typed?.lock, + temp_script_refs: tempScriptRefs, }; currentLastEdit = edit; return edit; diff --git a/cli/src/commands/docs/docs.ts b/cli/src/commands/docs/docs.ts index fed91201e0..4ba7ae95d5 100644 --- a/cli/src/commands/docs/docs.ts +++ b/cli/src/commands/docs/docs.ts @@ -7,24 +7,16 @@ import { GlobalOptions } from "../../types.ts"; import { getHeaders } from "../../utils/utils.ts"; import { detectAuthGatewayChallenge } from "../../utils/http_guards.ts"; -interface DocContentItem { - title: string; +interface DocsSearchResult { url: string; - source?: { - content?: Array<{ text: string }>; - }; + title: string; + score: number; + snippets: string[]; } -interface InkeepResponse { - choices?: Array<{ - message?: { - content?: string; - }; - }>; -} - -interface ParsedContent { - content?: DocContentItem[]; +interface DocsSearchResponse { + text: string; + results: DocsSearchResult[]; } async function docs( @@ -34,7 +26,9 @@ async function docs( await requireLogin(opts); const workspace = await resolveWorkspace(opts); - const url = `${workspace.remote}api/inkeep`; + // The backend self-hosts the docs corpus and does the search, so this works + // against any instance (no windmill.dev egress required). + const url = `${workspace.remote}api/docs/search?query=${encodeURIComponent(query)}`; console.log(colors.bold(`\nSearching Windmill docs...\n`)); @@ -42,13 +36,11 @@ async function docs( let res: Response; try { res = await fetch(url, { - method: "POST", + method: "GET", headers: { - "Content-Type": "application/json", Authorization: `Bearer ${workspace.token}`, ...extraHeaders, }, - body: JSON.stringify({ query }), }); } catch (e) { throw new Error(`Network error connecting to ${workspace.remote}: ${e}`); @@ -56,54 +48,31 @@ async function docs( await detectAuthGatewayChallenge(res, url); - if (res.status === 403) { - log.info( - "Windmill documentation search is an Enterprise Edition feature. Please upgrade to use this command." - ); - return; - } - if (!res.ok) { throw new Error( `Documentation search failed: ${res.status} ${res.statusText}\n${await res.text()}` ); } - const data = (await res.json()) as InkeepResponse; - const raw = data.choices?.[0]?.message?.content; - - if (!raw) { - log.info("No documentation found for this query."); - return; - } - - let parsed: ParsedContent; - try { - parsed = JSON.parse(raw); - } catch { - throw new Error("Failed to parse documentation response."); - } - - const items = parsed.content ?? []; - - if (items.length === 0) { - log.info("No documentation found for this query."); - return; - } + const data = (await res.json()) as DocsSearchResponse; + const items = data.results ?? []; if (opts.json) { console.log(JSON.stringify(items, null, 2)); return; } + if (items.length === 0) { + log.info("No documentation found for this query."); + return; + } + for (const item of items) { console.log(colors.bold(colors.cyan(`📄 ${item.title}`))); if (item.url) { console.log(` ${colors.underline(item.url)}`); } - const text = item.source?.content?.[0]?.text; - if (text) { - const snippet = text.length > 500 ? text.slice(0, 500) + "..." : text; + for (const snippet of item.snippets ?? []) { console.log(` ${snippet}`); } console.log(); diff --git a/cli/src/commands/flow/flow.ts b/cli/src/commands/flow/flow.ts index c0e94c478d..71d90af6fe 100644 --- a/cli/src/commands/flow/flow.ts +++ b/cli/src/commands/flow/flow.ts @@ -135,6 +135,31 @@ function warnAboutLocalPathScriptDivergence( const alreadySynced: string[] = []; +// Collect every script/sub-flow step path in a flow value — recursively through loops, +// branches, and the failure/preprocessor modules — for workspace-path validation. Unlike +// `collectPathScriptPaths` this also includes `type: "flow"` sub-flow steps. +function collectStepPaths(flowValue: any): string[] { + const paths: string[] = []; + const walk = (modules: any[] | undefined) => { + for (const m of modules ?? []) { + const v = m?.value; + if (!v) continue; + if ((v.type === "script" || v.type === "flow") && typeof v.path === "string") { + paths.push(v.path); + } + walk(v.modules); + walk(v.default); + for (const b of v.branches ?? []) walk(b?.modules); + // AI-agent tools are step-like and can carry script paths too. + walk(v.tools); + } + }; + walk(flowValue?.modules); + if (flowValue?.failure_module) walk([flowValue.failure_module]); + if (flowValue?.preprocessor_module) walk([flowValue.preprocessor_module]); + return paths; +} + export async function pushFlow( workspace: string, remotePath: string, @@ -190,6 +215,21 @@ export async function pushFlow( ); } + // Reject script/sub-flow steps whose path is not a workspace path (u/, f/, g/ or hub/). + // A flow.yaml generated from a feature-branch checkout can carry absolute local paths + // (e.g. /tmp/.../ops/scripts/...); pushed, they silently mis-resolve at runtime (#9751). + // The backend re-validates the same rule for every step type, so this is a fail-fast. + const badStepPaths = collectStepPaths(localFlow.value).filter( + (p) => p !== "" && !/^(u|f|g|hub)\//.test(p) + ); + if (badStepPaths.length > 0) { + throw new Error( + `Cannot push flow ${remotePath}: step(s) reference non-workspace path(s): ${badStepPaths.join(", ")}. ` + + `Flow step paths must be workspace paths (u/, f/, g/ or hub/), not absolute or local filesystem paths. ` + + `This usually means flow.yaml was generated with paths from a checkout directory.` + ); + } + const hasOnBehalfOf = (localFlow as any).has_on_behalf_of ?? !!localFlow.on_behalf_of_email; delete (localFlow as any).has_on_behalf_of; diff --git a/cli/src/commands/flow/flow_metadata.ts b/cli/src/commands/flow/flow_metadata.ts index 5c3ca578c7..180491ea90 100644 --- a/cli/src/commands/flow/flow_metadata.ts +++ b/cli/src/commands/flow/flow_metadata.ts @@ -1,5 +1,7 @@ import { colors } from "@cliffy/ansi/colors"; import * as log from "../../core/log.ts"; +import { existsSync } from "node:fs"; +import { rm } from "node:fs/promises"; import * as path from "node:path"; import { sep as SEP } from "node:path"; import { stringify as yamlStringify } from "yaml"; @@ -17,7 +19,7 @@ import { filterWorkspaceDependenciesForScripts, } from "../../utils/metadata.ts"; import { ScriptLanguage } from "../../utils/script_common.ts"; -import { extractInlineScripts as extractInlineScriptsForFlows, extractCurrentMapping } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; +import { extractInlineScripts as extractInlineScriptsForFlows, extractCurrentMapping, legacyLockPathForContent } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; import { newPathAssigner } from "../../../windmill-utils-internal/src/path-utils/path-assigner.ts"; import { generateHash, getHeaders, readTextFile, writeIfChanged } from "../../utils/utils.ts"; @@ -344,6 +346,31 @@ export async function generateFlowLockInternal( process.cwd() + SEP + folder + SEP + "flow.yaml", yamlStringify(flowValue as Record, yamlOptions) ); + + // CLI versions between #8561 and the canonical-lock-name fix named lock + // files after the content path minus only its last dot segment (e.g. + // "x.inline_script.deno.lock"). The canonical name strips the full + // language extension ("x.inline_script.lock"), so remove the legacy file. + // Runs after the flow.yaml rewrite above so an interruption can't leave + // flow.yaml referencing an already-deleted legacy file. + for (const s of inlineScripts) { + if (s.is_lock) continue; + const legacyRelPath = legacyLockPathForContent(s.path, s.language); + if (!legacyRelPath) continue; + // A legacy name can coincide with another step's canonical file written + // in this run (e.g. deno step "x" vs a step whose summary is "x.deno") — + // never delete a path that is part of the current extraction. + if (inlineScripts.some((other) => other.path === legacyRelPath)) continue; + const legacyAbsPath = process.cwd() + SEP + folder + SEP + legacyRelPath; + if (existsSync(legacyAbsPath)) { + try { + await rm(legacyAbsPath); + log.info(colors.gray(`Removed legacy lock file ${legacyRelPath} (renamed to canonical name)`)); + } catch (e) { + log.info(colors.yellow(`Failed to remove legacy lock file ${legacyRelPath}: ${e}`)); + } + } + } } // In tree mode, workspace deps are tracked via the tree — exclude from hash diff --git a/cli/src/commands/generate-metadata/generate-metadata.ts b/cli/src/commands/generate-metadata/generate-metadata.ts index 76c82ed39f..8517ac0912 100644 --- a/cli/src/commands/generate-metadata/generate-metadata.ts +++ b/cli/src/commands/generate-metadata/generate-metadata.ts @@ -97,8 +97,10 @@ async function walkLocalAppItems( * or app), so a preview run resolves relative imports from not-yet-deployed * local content instead of the deployed scripts. Walks all local scripts so * transitive relative-import targets can be uploaded, then for flow/app adds - * that item's node. Degrades gracefully (returns undefined) on older backends - * without the /raw_temp endpoints. + * that item's node. The "all" target skips per-node filtering and returns refs + * for every uploaded script — used by `wmill dev`, where the previewed item + * changes at runtime. Degrades gracefully (returns undefined) on older + * backends without the /raw_temp endpoints. */ export async function buildPreviewTempScriptRefs( workspace: Workspace, @@ -107,7 +109,8 @@ export async function buildPreviewTempScriptRefs( target: | { kind: "script"; path: string } | { kind: "flow"; folder: string } - | { kind: "app"; folder: string; rawApp: boolean }, + | { kind: "app"; folder: string; rawApp: boolean } + | { kind: "all" }, ): Promise | undefined> { try { const rawWorkspaceDependencies = await getRawWorkspaceDependencies(true); @@ -129,8 +132,11 @@ export async function buildPreviewTempScriptRefs( ); } - let nodePath: string; - if (target.kind === "script") { + let nodePath: string | undefined; + if (target.kind === "all") { + // No anchor node — refs are collected tree-wide below + nodePath = undefined; + } else if (target.kind === "script") { nodePath = scriptPathToRemotePath(target.path); } else if (target.kind === "flow") { const folder = target.folder.endsWith(SEP) @@ -157,7 +163,9 @@ export async function buildPreviewTempScriptRefs( tree.propagateStaleness(); await uploadScripts(tree, workspace); - const refs = tree.getTempScriptRefs(nodePath); + const refs = nodePath !== undefined + ? tree.getTempScriptRefs(nodePath) + : tree.getAllTempScriptRefs(); return refs && Object.keys(refs).length > 0 ? refs : undefined; } catch (e) { // Degrade gracefully (preview still runs against deployed versions) but do @@ -791,7 +799,7 @@ async function rehashCommand( } const command = new Command() - .description("Generate metadata (locks, schemas) for all scripts, flows, and apps") + .description("Regenerate stale local locks and script schemas and refresh wmill-lock.yaml content hashes (scripts, flows, apps). Writes local files only, not a deploy. Run it after edits that add or remove imports or change a script's arguments, so the lock, the auto-generated UI schema, and wmill-lock.yaml stay in sync.") .arguments("[folder:string]") .option("--yes", "Skip confirmation prompt") .option("--dry-run", "Show what would be updated without making changes") @@ -815,9 +823,7 @@ const command = new Command() "rehash", new Command() .description( - "Trust on-disk content; rewrite wmill-lock.yaml hashes without backend " + - "trips or yaml/lock rewrites. Useful for bootstrapping missing lockfile " + - "entries or recovering from older-CLI hash drift." + "Refresh wmill-lock.yaml content hashes from the on-disk .lock and .script.yaml without re-resolving dependencies or hitting the backend. Use when those files are already correct and only the hashes need updating: bootstrapping missing entries or recovering from hash drift." ) .arguments("[folder:string]") .option("--skip-scripts", "Skip processing scripts") diff --git a/cli/src/commands/pipeline/boundedCascade.ts b/cli/src/commands/pipeline/boundedCascade.ts new file mode 100644 index 0000000000..27380d799a --- /dev/null +++ b/cli/src/commands/pipeline/boundedCascade.ts @@ -0,0 +1,241 @@ +// Bounded-cascade graph engine for `wmill pipeline run --to`. +// +// MIRROR of frontend/src/lib/components/assets/AssetGraph/boundedCascade.ts — +// the two have no shared import path (frontend is a separate package), so keep +// them in sync. The grammar is intentionally tiny: there is no dbt-style +// `--select` string. The user names a start (a schedule / manual root) and one +// or more end nodes; the run is the "path between" them: +// +// descendants(start) ∩ (ancestors(ends) ∪ ends) ∪ {start} +// +// Node ids: assets `${kind}:${path}` (e.g. `datatable:main/raw`); runnables +// `script:${path}`. Operates on the asset-graph payload shape used by +// pipeline.ts. + +export type BCGraph = { + runnables: { path: string; usage_kind: "script" | "flow" | "job" }[]; + assets: { kind: string; path: string }[]; + edges: { + runnable_kind: string; + runnable_path: string; + asset_kind: string; + asset_path: string; + access_type?: "r" | "w" | "rw"; + }[]; + triggers: ( + | { + trigger_kind: "asset"; + asset_kind: string; + asset_path: string; + runnable_kind: string; + runnable_path: string; + } + | { trigger_kind: string; runnable_kind: string; runnable_path: string } + )[]; +}; + +export const SCRIPT_PREFIX = "script:"; +export const scriptNodeId = (path: string): string => `${SCRIPT_PREFIX}${path}`; +export const isScriptNode = (id: string): boolean => id.startsWith(SCRIPT_PREFIX); +export const scriptPathOf = (id: string): string => id.slice(SCRIPT_PREFIX.length); +const assetNodeId = (kind: string, path: string): string => `${kind}:${path}`; + +// Native trigger kinds that fan out per-event — never bounded-run starts. +// `webhook` / `data_upload` have no trigger row in the graph payload, so a root +// whose only entry is one of those reads as a manual root. +const EVENT_TRIGGER_KINDS = new Set([ + "kafka", + "mqtt", + "nats", + "postgres", + "sqs", + "gcp", + "email", +]); + +/** Resolve an asset URI (`datatable://x`, `s3://b/k`, …) to its node id. */ +export function assetUriToNodeId(uri: string): string | undefined { + const m = uri.match(/^([a-z0-9_]+):\/\/(.+)$/i); + if (!m) return undefined; + const prefix = m[1].toLowerCase(); + const kind = prefix === "s3" ? "s3object" : prefix; + return `${kind}:${m[2]}`; +} + +export type LineageDag = { + down: Map>; + up: Map>; + nodes: Set; +}; + +function addEdge(dag: LineageDag, a: string, b: string) { + if (a === b) return; + dag.nodes.add(a); + dag.nodes.add(b); + (dag.down.get(a) ?? dag.down.set(a, new Set()).get(a)!).add(b); + (dag.up.get(b) ?? dag.up.set(b, new Set()).get(b)!).add(a); +} + +/** Unified upstream→downstream lineage DAG over scripts ∪ assets. */ +export function buildLineageDag(g: BCGraph): LineageDag { + const dag: LineageDag = { down: new Map(), up: new Map(), nodes: new Set() }; + for (const r of g.runnables ?? []) { + if (r.usage_kind === "script") dag.nodes.add(scriptNodeId(r.path)); + } + for (const a of g.assets ?? []) dag.nodes.add(assetNodeId(a.kind, a.path)); + for (const e of g.edges ?? []) { + if (e.runnable_kind !== "script") continue; + const aid = assetNodeId(e.asset_kind, e.asset_path); + const access = e.access_type ?? "r"; + if (access === "w" || access === "rw") { + addEdge(dag, scriptNodeId(e.runnable_path), aid); // producer + } else if (access === "r") { + addEdge(dag, aid, scriptNodeId(e.runnable_path)); // pure reader + } + } + for (const t of g.triggers ?? []) { + if (t.trigger_kind !== "asset" || t.runnable_kind !== "script") continue; + const at = t as Extract; + addEdge(dag, assetNodeId(at.asset_kind, at.asset_path), scriptNodeId(at.runnable_path)); + } + return dag; +} + +function closure(adj: Map>, start: string): Set { + const seen = new Set(); + const queue = [start]; + while (queue.length > 0) { + const cur = queue.shift()!; + for (const n of adj.get(cur) ?? []) { + if (seen.has(n)) continue; + seen.add(n); + queue.push(n); + } + } + // A cycle back to `start` would have re-added it; the contract excludes + // the node itself. + seen.delete(start); + return seen; +} + +export const descendants = (dag: LineageDag, n: string): Set => closure(dag.down, n); +export const ancestors = (dag: LineageDag, n: string): Set => closure(dag.up, n); + +export type BoundedResult = { + nodes: Set; + reachableEnds: string[]; + droppedEnds: string[]; +}; + +/** Path-between node set for `start` and `ends` (inclusive). */ +export function boundedSet(dag: LineageDag, start: string, ends: string[]): BoundedResult { + const downSet = new Set(descendants(dag, start)); + downSet.add(start); + const reachableEnds = ends.filter((e) => downSet.has(e)); + const droppedEnds = ends.filter((e) => !downSet.has(e)); + if (reachableEnds.length === 0) { + return { nodes: new Set([start]), reachableEnds, droppedEnds }; + } + const upClosure = new Set(); + for (const e of reachableEnds) { + upClosure.add(e); + for (const a of ancestors(dag, e)) upClosure.add(a); + } + const nodes = new Set(); + for (const n of downSet) if (upClosure.has(n)) nodes.add(n); + nodes.add(start); + return { nodes, reachableEnds, droppedEnds }; +} + +/** Script node ids eligible to start a bounded run. */ +export function validStarts(g: BCGraph): Set { + const subscribers = new Set(); + const scheduleScripts = new Set(); + const eventScripts = new Set(); + for (const t of g.triggers ?? []) { + if (t.runnable_kind !== "script") continue; + if (t.trigger_kind === "asset") subscribers.add(t.runnable_path); + else if (t.trigger_kind === "schedule") scheduleScripts.add(t.runnable_path); + else if (EVENT_TRIGGER_KINDS.has(t.trigger_kind)) eventScripts.add(t.runnable_path); + } + const out = new Set(); + for (const r of g.runnables ?? []) { + if (r.usage_kind !== "script") continue; + const p = r.path; + if (scheduleScripts.has(p)) out.add(scriptNodeId(p)); + else if (!subscribers.has(p) && !eventScripts.has(p)) out.add(scriptNodeId(p)); + } + return out; +} + +/** Project a node-id set to the script paths it contains. */ +export function scriptsOf(nodes: Iterable): string[] { + const out: string[] = []; + for (const id of nodes) if (isScriptNode(id)) out.push(scriptPathOf(id)); + return out; +} + +/** + * Resolve a CLI `--to` / `--from` token to a node id, or undefined if it + * matches nothing. Asset URIs (`kind://path`) resolve to the asset node; a bare + * token matches a runnable by exact path or by short (last-segment) name. + */ +export function resolveToken(g: BCGraph, token: string): string | undefined { + if (token.includes("://")) { + const id = assetUriToNodeId(token); + return id && g.assets.some((a) => `${a.kind}:${a.path}` === id) ? id : undefined; + } + const scripts = (g.runnables ?? []).filter((r) => r.usage_kind === "script"); + const exact = scripts.find((r) => r.path === token); + if (exact) return scriptNodeId(exact.path); + const byShort = scripts.filter((r) => (r.path.split("/").pop() ?? r.path) === token); + return byShort.length === 1 ? scriptNodeId(byShort[0].path) : undefined; +} + +/** + * Topological order of `scripts` over the in-set producer→subscriber edges + * (assets collapsed). Scripts on a cycle are returned in `cyclic` and excluded + * from `order`. Serial-run friendly: every script comes after its in-set + * upstreams. + */ +export function topoOrder( + g: BCGraph, + scripts: Set, +): { order: string[]; cyclic: string[] } { + const dag = buildLineageDag(g); + const down = new Map>(); + const indegree = new Map(); + for (const s of scripts) indegree.set(s, 0); + // One-hop (through a single asset) script→script edges, restricted to the set. + for (const s of scripts) { + const sid = scriptNodeId(s); + const oneHop = new Set(); + for (const asset of dag.down.get(sid) ?? []) { + for (const sub of dag.down.get(asset) ?? []) { + if (isScriptNode(sub)) { + const p = scriptPathOf(sub); + if (p !== s && scripts.has(p)) oneHop.add(p); + } + } + } + if (oneHop.size > 0) { + down.set(s, oneHop); + for (const p of oneHop) indegree.set(p, (indegree.get(p) ?? 0) + 1); + } + } + const ready = [...scripts].filter((s) => (indegree.get(s) ?? 0) === 0); + const remaining = new Map(indegree); + const order: string[] = []; + while (ready.length > 0) { + const n = ready.shift()!; + order.push(n); + for (const p of down.get(n) ?? []) { + const d = (remaining.get(p) ?? 0) - 1; + remaining.set(p, d); + if (d === 0) ready.push(p); + } + } + const orderedSet = new Set(order); + const cyclic = [...scripts].filter((s) => !orderedSet.has(s)); + return { order, cyclic }; +} diff --git a/cli/src/commands/pipeline/pipeline.ts b/cli/src/commands/pipeline/pipeline.ts new file mode 100644 index 0000000000..ac01da29a1 --- /dev/null +++ b/cli/src/commands/pipeline/pipeline.ts @@ -0,0 +1,506 @@ +import { Command } from "@cliffy/command"; +import { Table } from "@cliffy/table"; +import { colors } from "@cliffy/ansi/colors"; + +import { OpenAPI } from "../../../gen/index.ts"; +import * as wmill from "../../../gen/services.gen.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { resolveWorkspace } from "../../core/context.ts"; +import * as log from "../../core/log.ts"; +import { GlobalOptions } from "../../types.ts"; +import { + type BCGraph, + boundedSet, + buildLineageDag, + descendants, + resolveToken, + scriptNodeId, + scriptPathOf, + scriptsOf, + topoOrder, + validStarts, +} from "./boundedCascade.ts"; + +// Mirrors the asset-graph endpoint payload (backend/windmill-api-assets). +// TODO: the checked-in generated client (cli/gen, last regenerated 2025-04) +// predates these routes, so we raw-fetch and hand-roll the types. Once +// `cli/gen` is regenerated (run `cli/gen_wm_client.sh`, which is currently +// >700 openapi.yaml commits stale and would churn the whole client), replace +// `apiGet` + these types with the generated `wmill.getAssetsGraph(...)` +// (operationId getAssetsGraph) and `wmill.listPipelineFolders(...)` +// (operationId listPipelineFolders). +type GraphRunnable = { + path: string; + usage_kind: "script" | "flow" | "job"; + in_pipeline?: boolean; +}; +type GraphEdge = { + runnable_kind: string; + runnable_path: string; + asset_kind: string; + asset_path: string; + access_type?: "r" | "w" | "rw"; +}; +type GraphTrigger = + | { + trigger_kind: "asset"; + asset_kind: string; + asset_path: string; + runnable_kind: string; + runnable_path: string; + } + | { + trigger_kind: string; + path?: string; + runnable_kind: string; + runnable_path: string; + missing?: boolean; + }; +type AssetGraph = { + runnables: GraphRunnable[]; + assets: { kind: string; path: string }[]; + edges: GraphEdge[]; + triggers: GraphTrigger[]; +}; + +async function apiGet(path: string): Promise { + const response = await fetch(`${OpenAPI.BASE}${path}`, { + headers: { Authorization: `Bearer ${OpenAPI.TOKEN}` }, + }); + if (!response.ok) { + const body = await response.text(); + throw new Error(`GET ${path} -> ${response.status}: ${body}`); + } + return (await response.json()) as T; +} + +async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const items = await apiGet<{ folder: string; script_count: number }[]>( + `/w/${workspace.workspaceId}/assets/pipelines`, + ); + if (opts.json) { + console.log(JSON.stringify(items)); + } else if (items.length === 0) { + log.info( + "No pipelines in this workspace. Mark scripts with a `// pipeline` comment (plus `// on ` triggers) and push them into a folder.", + ); + } else { + new Table() + .header(["Folder", "Scripts"]) + .padding(2) + .border(true) + .body(items.map((p) => [`f/${p.folder}`, String(p.script_count)])) + .render(); + } +} + +const ASSET_KINDS = "s3object,ducklake,datatable,volume"; + +function assetUri(kind: string, path: string): string { + const prefix = kind === "s3object" ? "s3" : kind; + return `${prefix}://${path}`; +} + +function shortName(scriptPath: string): string { + return scriptPath.split("/").pop() ?? scriptPath; +} + +// Append to a multimap value, creating the bucket on first use. Avoids the +// O(n^2) spread-rebuild pattern (`map.set(k, [...(map.get(k) ?? []), v])`). +function pushTo(map: Map, key: K, val: V): void { + (map.get(key) ?? map.set(key, []).get(key)!).push(val); +} + +async function show( + opts: GlobalOptions & { json?: boolean }, + folder: string, +) { + if (opts.json) log.setSilent(true); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const f = folder.replace(/^f\//, "").replace(/\/$/, ""); + const graph = await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ); + if (opts.json) { + console.log(JSON.stringify(graph)); + return; + } + if (graph.runnables.length === 0) { + log.info( + `No pipeline scripts in f/${f}. Mark scripts with a \`// pipeline\` comment and push them.`, + ); + return; + } + + // Index the graph: writes per script, subscribers per asset, native + // trigger markers per script, asset subscriptions per script. + const writesByScript = new Map(); + for (const e of graph.edges) { + if (e.access_type === "w" || e.access_type === "rw") { + const uri = assetUri(e.asset_kind, e.asset_path); + pushTo(writesByScript, e.runnable_path, uri); + } + } + const subsByAsset = new Map(); + const subsByScript = new Map(); + const nativeByScript = new Map< + string, + { kind: string; path?: string; missing?: boolean }[] + >(); + for (const t of graph.triggers) { + if (t.trigger_kind === "asset") { + const at = t as Extract; + const uri = assetUri(at.asset_kind, at.asset_path); + pushTo(subsByAsset, uri, t.runnable_path); + pushTo(subsByScript, t.runnable_path, uri); + } else { + const nt = t as Exclude; + pushTo(nativeByScript, t.runnable_path, { + kind: nt.trigger_kind, + path: nt.path, + missing: nt.missing, + }); + } + } + + function triggerBadges(script: string): string { + const out: string[] = []; + for (const t of nativeByScript.get(script) ?? []) { + if (t.kind === "data_upload") { + out.push(colors.magenta("[data upload]")); + } else if (t.missing) { + out.push(colors.red(`[${t.kind} ✗ missing]`)); + } else { + out.push(colors.yellow(`[${t.kind}${t.path ? ` ${t.path}` : ""}]`)); + } + } + return out.length > 0 ? " " + out.join(" ") : ""; + } + + const printed = new Set(); + const lines: string[] = []; + + function printScript(script: string, prefix: string, extraOn?: string[]) { + const alsoOn = + extraOn && extraOn.length > 0 + ? colors.dim(` (also on: ${extraOn.join(", ")})`) + : ""; + if (printed.has(script)) { + lines.push( + `${prefix}${colors.bold(shortName(script))}${colors.dim(" ↻ shown above")}`, + ); + return; + } + printed.add(script); + lines.push(`${prefix}${colors.bold(shortName(script))}${triggerBadges(script)}${alsoOn}`); + const childPrefix = prefix.replace(/├─ $/, "│ ").replace(/└─ $/, " "); + const writes = [...(writesByScript.get(script) ?? [])].sort(); + writes.forEach((uri, i) => { + const lastAsset = i === writes.length - 1; + const assetBranch = lastAsset ? "└─▶ " : "├─▶ "; + lines.push(`${childPrefix}${assetBranch}${colors.cyan(uri)}`); + const assetChildPrefix = childPrefix + (lastAsset ? " " : "│ "); + const subs = [...(subsByAsset.get(uri) ?? [])].sort(); + subs.forEach((sub, j) => { + const branch = j === subs.length - 1 ? "└─ " : "├─ "; + const otherOn = (subsByScript.get(sub) ?? []).filter((u) => u !== uri); + printScript(sub, assetChildPrefix + branch, otherOn); + }); + }); + } + + // Roots: pipeline scripts that aren't subscribed to any asset — sources + // (data upload, schedule, webhook) and manual entries. + const roots = graph.runnables + .map((r) => r.path) + .filter((p) => !(subsByScript.get(p)?.length)) + .sort(); + + // UI-first markers (data_upload, webhook) have no trigger row — the + // graph endpoint's trigger enum (schedule/email/kafka/mqtt/nats/postgres/ + // sqs/gcp) can't surface them, so they only exist as `// on ` + // annotations in the script body. Roots are where sources matter, so fetch + // just those bodies and lift the marker kinds the canvas would show. + // + // DRIFT RISK: this regex + MARKER_KINDS is a divergent, partial copy of the + // canonical annotation parser. The proper fix is to have the graph endpoint + // emit these UI-only markers as trigger rows (a backend change), after which + // this whole Promise.all body-fetch can be deleted and read straight from + // the response. Until then, keep this list in sync with the canonical parser. + const MARKER_KINDS = ["data_upload", "webhook", "email"]; + await Promise.all( + roots.map(async (p) => { + const r = graph.runnables.find((x) => x.path === p); + if (r?.usage_kind !== "script") return; + try { + const script = await wmill.getScriptByPath({ + workspace: workspace.workspaceId, + path: p, + }); + const existing = nativeByScript.get(p) ?? []; + for (const line of (script.content ?? "").split("\n")) { + const m = line.match(/^\s*(?:\/\/|--|#)\s*on\s+(\w+)\s*$/); + if (!m) continue; + const kind = m[1]; + if (!MARKER_KINDS.includes(kind)) continue; + if (!existing.some((t) => t.kind === kind)) { + existing.push({ kind }); + } + } + if (existing.length > 0) nativeByScript.set(p, existing); + } catch { + // body fetch is best-effort enrichment only + } + }), + ); + + const scriptCount = graph.runnables.length; + const assetCount = graph.assets.length; + log.info( + colors.bold(`Pipeline f/${f}`) + + colors.dim(` — ${scriptCount} script${scriptCount === 1 ? "" : "s"} · ${assetCount} asset${assetCount === 1 ? "" : "s"}`), + ); + lines.push(""); + for (const root of roots) { + printScript(root, ""); + lines.push(""); + } + // Anything unreachable from the roots (e.g. cycles) still gets listed. + for (const r of graph.runnables) { + if (!printed.has(r.path)) { + printScript(r.path, ""); + lines.push(""); + } + } + console.log(lines.join("\n")); +} + +// Poll a launched job to a terminal state. Modest fixed cadence; capped so a +// wedged job can't hang the CLI forever. +async function waitJob(workspace: string, id: string): Promise { + const MAX_RETRIES = 6000; // ~10min at 100ms + for (let i = 0; i < MAX_RETRIES; i++) { + try { + const r = await wmill.getCompletedJobResultMaybe({ + workspace, + id, + getStarted: false, + }); + // A completed job without an explicit `success: true` is a failure + // (mirrors the frontend `waitJobTerminal`): the cascade only advances on + // a confirmed success. + if (r.completed) return r.success === true; + } catch { + // transient — retry + } + await new Promise((res) => setTimeout(res, 100)); + } + throw new Error(`Timed out waiting for job ${id}`); +} + +// Bounded-cascade run: start at a schedule / manual root, fan downstream, but +// stop at the `--to` end node(s). Scripts run in topological order; each is +// launched with `_wmill_skip_asset_dispatch` so the CLI owns the whole closure +// (the backend dispatcher never double-fires the deployed part). With no +// `--to`, runs the full read-aware downstream of `--from` (every descendant in +// the lineage DAG, pure readers included — broader than the canvas cascade, +// which dispatches subscribers only). +async function run( + opts: GlobalOptions & { + from?: string; + to?: string[]; + dryRun?: boolean; + json?: boolean; + }, + folder: string, +) { + if (opts.json) log.setSilent(true); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const f = folder.replace(/^f\//, "").replace(/\/$/, ""); + const graph = await apiGet( + `/w/${workspace.workspaceId}/assets/graph?folder=${encodeURIComponent(f)}&asset_kinds=${ASSET_KINDS}`, + ); + + // Resolve the start: explicit --from (must be a valid start) or the folder's + // sole valid start. + const starts = validStarts(graph); + let start: string; + if (opts.from) { + const resolved = resolveToken(graph, opts.from); + if (!resolved) { + // Distinguish "no match" from "ambiguous short name" (resolveToken + // returns undefined for both) so the hint is actionable. + const matches = graph.runnables.filter( + (r) => r.usage_kind === "script" && (r.path.split("/").pop() ?? r.path) === opts.from, + ); + if (matches.length > 1) { + throw new Error( + `--from '${opts.from}' matches multiple scripts (${matches.map((r) => r.path).sort().join(", ")}) — pass the full path.`, + ); + } + throw new Error(`--from '${opts.from}' matched no script in f/${f}.`); + } + if (!starts.has(resolved)) { + throw new Error( + `--from '${opts.from}' is not a valid bounded-run start. Starts must be schedule-triggered or manual roots; row-backed event triggers (kafka/mqtt/nats/postgres/sqs/gcp/email) fan out per-event and can't be bounded.`, + ); + } + start = resolved; + } else if (starts.size === 1) { + start = [...starts][0]; + } else if (starts.size === 0) { + throw new Error( + `No schedule or manual root in f/${f} to start a bounded run from.`, + ); + } else { + throw new Error( + `f/${f} has ${starts.size} possible starts — pass --from %sveltekit.head% @@ -56,7 +156,7 @@ /> { + copilotInfo.update((info) => ({ + ...info, + enabled: info.aiModels.length > 0 && !disabled + })) +}) + /** Strip the deprecated /thinking suffix from a configured model slot, if present. */ function stripModelSuffix(model: AIProviderModel | undefined): AIProviderModel | undefined { return model ? { ...model, model: stripLegacyThinkingSuffix(model.model) } : model @@ -107,7 +117,8 @@ export function setCopilotInfo(aiConfig: AIConfig) { }) copilotInfo.set({ - enabled: true, + // Providers are configured; the per-user opt-out is the only thing that can gate it off. + enabled: !get(aiUserDisabled), // Strip the deprecated /thinking suffix from the configured model slots too, // otherwise a workspace whose default still carries it sends an invalid model id. codeCompletionModel: stripModelSuffix(aiConfig.code_completion_model), @@ -193,3 +204,12 @@ export function getCombinedCustomPrompt(mode: string): string | undefined { return prompts.join('\n\n') } + +// Like getCombinedCustomPrompt but keeps the workspace and user slices separate so the +// Global system prompt can label them distinctly — only the user slice is editable by the +// update_user_instructions tool. +export function getCustomPromptParts(mode: string): { workspace?: string; user?: string } { + const workspace = get(copilotInfo).customPrompts?.[mode]?.trim() || undefined + const user = getUserCustomPrompts()[mode]?.trim() || undefined + return { workspace, user } +} diff --git a/frontend/src/lib/attachments/arrowTabNav.ts b/frontend/src/lib/attachments/arrowTabNav.ts new file mode 100644 index 0000000000..e98b7db34c --- /dev/null +++ b/frontend/src/lib/attachments/arrowTabNav.ts @@ -0,0 +1,59 @@ +import { tabbable } from 'tabbable' +import type { Attachment } from 'svelte/attachments' + +export interface ArrowTabNavOptions { + /** Which arrow-key pair walks the tab order. Default `'y'` (Up/Down). */ + axis?: 'x' | 'y' + /** Use this to layer custom keys (Enter, Escape …) */ + onKeyDown?: (e: KeyboardEvent) => void +} + +/** + * Map one axis of arrow keys to next/previous in Tab order, scoped to + * the attached element. Wraps around at the ends. Bails when the + * keypress originates inside a text input / textarea / contenteditable + * so the caret can still move with the arrows. + * + *
…
+ *
…
+ *
{ … } })}>…
+ */ +export function arrowTabNav(opts: ArrowTabNavOptions = {}): Attachment { + const axis = opts.axis ?? 'y' + const nextKey = axis === 'y' ? 'ArrowDown' : 'ArrowRight' + const prevKey = axis === 'y' ? 'ArrowUp' : 'ArrowLeft' + + return (node) => { + const handler = (e: KeyboardEvent) => { + // Run the consumer's handler first so they can preventDefault + // or do their own thing before we react to arrows. + opts.onKeyDown?.(e) + + if (e.key !== nextKey && e.key !== prevKey) return + + // Let editable fields keep their native caret behavior. + const t = e.target as HTMLElement | null + if (t && (t.tagName === 'INPUT' || t.tagName === 'TEXTAREA' || t.isContentEditable)) { + return + } + + const items = tabbable(node) + if (items.length === 0) return + + const active = document.activeElement as HTMLElement | null + const i = active ? items.indexOf(active) : -1 + const dir = e.key === nextKey ? 1 : -1 + // No tabbable currently focused: jump to the first (next) or last (prev). + const next = + i === -1 + ? items[dir === 1 ? 0 : items.length - 1] + : items[(i + dir + items.length) % items.length] + + e.preventDefault() + next?.focus() + } + + node.addEventListener('keydown', handler) + return () => node.removeEventListener('keydown', handler) + } +} diff --git a/frontend/src/lib/attachments/selectAndAdvanceTo.ts b/frontend/src/lib/attachments/selectAndAdvanceTo.ts new file mode 100644 index 0000000000..0de23572c5 --- /dev/null +++ b/frontend/src/lib/attachments/selectAndAdvanceTo.ts @@ -0,0 +1,49 @@ +import { tabbable } from 'tabbable' + +/** + * Build an Enter-key handler for `arrowTabNav`'s `onKeyDown` (or any + * `keydown` listener): activate whatever is focused (so its `onClick` + * fires), then move focus to the first tabbable inside the container + * returned by `getNext`. preventDefault suppresses the browser's own + * Enter→click so we don't double-fire. + * + * By default the handler bails inside ` - {:else if editorKind == 'json'} -
- {#await import('$lib/components/SimpleEditor.svelte')} - - {:then Module} - - {/await} -
- {:else if editorKind == 'yaml'} -
- {#await import('$lib/components/SimpleEditor.svelte')} - - {:then Module} - - {/await} -
- {/if} - + {#if isEncryptedDraftValue(variable.value)} + + (variable.value = '')} /> + {:else} +
+ + {#snippet children({ item })} + + + + {/snippet} + + {#if editorKind == 'plain'} + + {:else if editorKind == 'json'} +
+ {#await import('$lib/components/SimpleEditor.svelte')} + + {:then Module} + + {/await} +
+ {:else if editorKind == 'yaml'} +
+ {#await import('$lib/components/SimpleEditor.svelte')} + + {:then Module} + + {/await} +
+ {/if} +
+ {/if}
+ + + + + + + + + + + {#each partitions.current as p (p.partition)} + + + + + + + + {#if p.error} + + {/if} + {/each} + +
PartitionStatusSnapshotRowsMaterialized
{p.partition || '(whole table)'} + + {p.status} + + {p.snapshot_id ?? '—'}{p.row_count ?? '—'}{new Date(p.materialized_at).toLocaleString()}
{p.error}
+ {/if} + + + + diff --git a/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte b/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte new file mode 100644 index 0000000000..90e531e858 --- /dev/null +++ b/frontend/src/lib/components/assets/AssetGraph/PipelineActivityPanel.svelte @@ -0,0 +1,579 @@ + + + + +
+
+ + + Activity + {#if runningCount > 0} + + + {runningCount} running + + {:else if loading} + + {/if} + +
+ diff --git a/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte b/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte new file mode 100644 index 0000000000..26366325b5 --- /dev/null +++ b/frontend/src/lib/components/common/confirmationModal/DraftEditorModals.svelte @@ -0,0 +1,150 @@ + + +{#if enabled && workspace && path} + + {#if otherDraftsUsers.length > 0} + {#key path} + + {/key} + {/if} + {#if onLoadLatestDeploy} + + {/if} + OtherUserDraftLoad.confirmOverwrite(workspace, itemKind, path)} + onCanceled={() => OtherUserDraftLoad.dismissOverwriteModal(workspace, itemKind, path)} + > + + You're editing another user's draft. Saving this edit will overwrite your own draft at this + path. Continue? + + +{/if} diff --git a/frontend/src/lib/components/common/confirmationModal/DraftSyncConflictModal.svelte b/frontend/src/lib/components/common/confirmationModal/DraftSyncConflictModal.svelte new file mode 100644 index 0000000000..ca467fd571 --- /dev/null +++ b/frontend/src/lib/components/common/confirmationModal/DraftSyncConflictModal.svelte @@ -0,0 +1,92 @@ + + + +
+
+ +
+

+ Another tab, browser, or AI agent saved a newer version of this draft. Your autosave was + rejected to avoid overwriting their work. +

+ {#if conflictHandle.conflict} +

+ Server timestamp: {new Date(conflictHandle.conflict.serverTimestamp).toLocaleString()} +

+ {/if} +
+
+ +
+ + + +
+
+
diff --git a/frontend/src/lib/components/common/confirmationModal/LocalDraftStaleModal.svelte b/frontend/src/lib/components/common/confirmationModal/LocalDraftStaleModal.svelte deleted file mode 100644 index 94782f176e..0000000000 --- a/frontend/src/lib/components/common/confirmationModal/LocalDraftStaleModal.svelte +++ /dev/null @@ -1,125 +0,0 @@ - - - - -{#if open} - -{/if} diff --git a/frontend/src/lib/components/common/confirmationModal/MigrateLegacyDraftModal.svelte b/frontend/src/lib/components/common/confirmationModal/MigrateLegacyDraftModal.svelte new file mode 100644 index 0000000000..0dba022607 --- /dev/null +++ b/frontend/src/lib/components/common/confirmationModal/MigrateLegacyDraftModal.svelte @@ -0,0 +1,128 @@ + + + +
+ {#if confirmingAssign} +
+ +

+ You already have your own draft at {path}. + Assigning this legacy draft to yourself will + replace your current draft. This can't be undone. +

+
+
+ + +
+ {:else} +
+ +

+ This is a pre-migration workspace-level draft with no owner. As an admin you can delete + it, or assign it to yourself to keep editing it as your own draft. +

+
+
+ + + +
+ {/if} +
+
diff --git a/frontend/src/lib/components/common/confirmationModal/OtherUsersDraftsModal.svelte b/frontend/src/lib/components/common/confirmationModal/OtherUsersDraftsModal.svelte new file mode 100644 index 0000000000..49356d888e --- /dev/null +++ b/frontend/src/lib/components/common/confirmationModal/OtherUsersDraftsModal.svelte @@ -0,0 +1,219 @@ + + + +
+
+ +

+ Their drafts are independent of yours. For advanced collaboration, consider using workspace forks (EE) +

+
+ +
    + {#each otherDraftsUsers as owner (ownerKey(owner))} +
  • +
    +
    + + {ownerLabel(owner)} + + {#if !owner.username} + + Pre-migration workspace-scoped draft (no owner). Saved before drafts became + per-user — kept around so you can recover the content, but no current user owns + it. + + {/if} +
    + {#if owner.draft_saved_at} + + Last updated: {displayDate(owner.draft_saved_at)} + + {/if} +
    + {#if !draftOnly} + + {/if} + + {#if !owner.username && canMigrateLegacy} + + {/if} +
  • + {/each} +
+ +
+ +
+
+
+ + + + { + isOpen = false + await onReload?.() + }} +/> diff --git a/frontend/src/lib/components/common/confirmationModal/StaleDraftModal.svelte b/frontend/src/lib/components/common/confirmationModal/StaleDraftModal.svelte new file mode 100644 index 0000000000..cc027c6f12 --- /dev/null +++ b/frontend/src/lib/components/common/confirmationModal/StaleDraftModal.svelte @@ -0,0 +1,86 @@ + + + +
+
+ +
+

+ A newer version was deployed after you started editing. Your draft is based on the older + deploy. +

+

+ Draft saved {formatTs(draftSavedAt)} · Deployed {formatTs(deployedAt)} +

+
+
+ +
+ + +
+
+
diff --git a/frontend/src/lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte b/frontend/src/lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte index 55a78245be..6a9689b544 100644 --- a/frontend/src/lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte +++ b/frontend/src/lib/components/common/confirmationModal/UnsavedConfirmationModal.svelte @@ -2,8 +2,6 @@ import ConfirmationModal from './ConfirmationModal.svelte' import { beforeNavigate } from '$app/navigation' import { goto as gotoUrl } from '$app/navigation' - import Button from '../button/Button.svelte' - import type DiffDrawer from '$lib/components/DiffDrawer.svelte' import { cleanValueProperties, orderedJsonStringify, @@ -16,20 +14,28 @@ interface Props { getInitialAndModifiedValues?: GetInitialAndModifiedValues - diffDrawer?: DiffDrawer | undefined additionalExitAction?: () => void triggerOnSearchParamsChange?: boolean onDiscardChanges?: () => void tabMode?: boolean + /** Alternative dirty check. When provided it REPLACES the value diff: + * the modal engages whenever it returns true (the value-diff props can + * be omitted). The full-page editors pass the auto-save-off "parked + * unsaved changes" signal here. */ + hasUnsavedChanges?: () => boolean + /** Adds a line to the confirmation telling the user they can enable + * auto-save to persist a draft automatically. */ + showAutosaveTips?: boolean } let { getInitialAndModifiedValues = undefined, - diffDrawer = undefined, additionalExitAction = () => {}, triggerOnSearchParamsChange = false, onDiscardChanges = undefined, - tabMode = false + tabMode = false, + hasUnsavedChanges = undefined, + showAutosaveTips = false }: Props = $props() let savedValue: Value | undefined = $state(undefined) let modifiedValue: Value | undefined = $state(undefined) @@ -38,10 +44,15 @@ let open = $state(false) let goingTo: URL | undefined = $state(undefined) + // The modal is wired up when either dirty-detection mode is configured. + let dirtyDetectionActive = $derived(!!getInitialAndModifiedValues || !!hasUnsavedChanges) + // Mirrors the modal condition: dirty when values differ, or when either // value is missing (e.g. a never-saved draft). Also refreshes - // savedValue/modifiedValue for the diff drawer. - function hasUnsavedChanges(): boolean { + // savedValue/modifiedValue for the diff drawer. `hasUnsavedChanges`, when + // passed, short-circuits the value diff with the caller's own predicate. + function checkUnsavedChanges(): boolean { + if (hasUnsavedChanges) return hasUnsavedChanges() const state = getInitialAndModifiedValues?.() savedValue = state?.savedValue modifiedValue = state?.modifiedValue @@ -61,7 +72,7 @@ beforeNavigate(async (newNavigationState) => { if ( !bypassBeforeNavigate && - getInitialAndModifiedValues && + dirtyDetectionActive && newNavigationState.to && ((newNavigationState.to.url != page.url && newNavigationState.to.url.pathname !== newNavigationState.from?.url.pathname) || @@ -69,7 +80,7 @@ ) { goingTo = newNavigationState.to.url - if (hasUnsavedChanges()) { + if (checkUnsavedChanges()) { newNavigationState.cancel() open = true } else { @@ -84,7 +95,7 @@ }) function onBeforeUnload(event: BeforeUnloadEvent) { - if (!bypassBeforeNavigate && getInitialAndModifiedValues && hasUnsavedChanges()) { + if (!bypassBeforeNavigate && dirtyDetectionActive && checkUnsavedChanges()) { // Triggers the browser's native "leave site?" confirmation event.preventDefault() // Required by some browsers (legacy mechanism) @@ -125,37 +136,12 @@ >
Are you sure you want to discard the changes you have made? - {#if savedValue && modifiedValue && diffDrawer} - + {#if showAutosaveTips} + + Auto-save is off, so these changes are not saved as a draft. Enable auto-save (the cloud + icon in the editor toolbar) to persist your changes automatically, or press Ctrl/Cmd+S to + save the current draft before leaving. + {/if}
diff --git a/frontend/src/lib/components/common/index.ts b/frontend/src/lib/components/common/index.ts index 4dd5670ba0..32f402c643 100644 --- a/frontend/src/lib/components/common/index.ts +++ b/frontend/src/lib/components/common/index.ts @@ -16,6 +16,7 @@ export { default as TabContent } from './tabs/TabContent.svelte' export { default as Tabs } from './tabs/Tabs.svelte' export { default as Breadcrumb } from './breadcrumb/Breadcrumb.svelte' export { default as FileInput } from './fileInput/FileInput.svelte' +export { default as RadioCard } from './radioCard/RadioCard.svelte' export { default as Section } from '../Section.svelte' export { default as Url } from './Url.svelte' diff --git a/frontend/src/lib/components/common/modal/Modal2.svelte b/frontend/src/lib/components/common/modal/Modal2.svelte index 36fbb23c5f..c00bf758d0 100644 --- a/frontend/src/lib/components/common/modal/Modal2.svelte +++ b/frontend/src/lib/components/common/modal/Modal2.svelte @@ -8,6 +8,8 @@ import { X } from 'lucide-svelte' import List from '$lib/components/common/layout/List.svelte' import { fade } from 'svelte/transition' + import { zIndexes } from '$lib/zIndexes' + import { chatState } from '$lib/components/copilot/chat/sharedChatState.svelte' interface Props { title: string @@ -15,8 +17,15 @@ target?: string isOpen?: boolean fixedWidth?: 'xs' | 'sm' | 'md' | 'lg' | 'xl' | 'xxl' - fixedHeight?: 'xs' | 'sm' | 'md' | 'lg' | 'xl' | 'xxl' + /** `adaptive` sizes the modal to its content (no fixed height, + * still capped by max-h-screen-80). */ + fixedHeight?: 'xs' | 'sm' | 'md' | 'lg' | 'xl' | 'xxl' | 'adaptive' contentClasses?: string + /** Close when the user clicks outside the modal body. Default + * true. Set false when the caller stacks a child modal on top + * and clicks "outside" the child would otherwise propagate + * here and close the underlying modal. */ + closeOnOutsideClick?: boolean headerLeft?: import('svelte').Snippet headerRight?: import('svelte').Snippet children?: import('svelte').Snippet @@ -25,11 +34,15 @@ let { title, css = {}, - target = '', + // Forwarded to `Portal`. An empty string would hit + // `document.querySelector('')` and throw "The provided selector + // is empty" — match `Portal`'s own default instead. + target = 'body', isOpen = $bindable(false), fixedWidth = 'md', fixedHeight = 'md', contentClasses = '', + closeOnOutsideClick = true, headerLeft, headerRight, children @@ -49,7 +62,9 @@ md: '500px', lg: '720px', xl: '800px', - xxl: '1000px' + xxl: '1000px', + // Content-driven height — emit no `height:` rule at all. + adaptive: undefined } export function close() { @@ -61,6 +76,7 @@ } function handleKeyDown(event: KeyboardEvent) { + if (!isOpen) return if (event.key === 'Escape') { event.preventDefault() event.stopPropagation() @@ -71,6 +87,11 @@ function fadeFast(node: HTMLElement) { return fade(node, { duration: 200 }) } + + // Elevate above the AI chat panel (zIndexes.aiChat) while chat is open so + // the dialog isn't hidden behind it; otherwise keep the default modal + // stacking just above disposables (zIndexes.disposables). + const overlayZIndex = $derived(chatState.size > 0 ? zIndexes.aiChat + 1 : zIndexes.disposables + 10) @@ -78,20 +99,23 @@ {#if isOpen}
close() }} + use:clickOutside={{ + onClickOutside: () => closeOnOutsideClick && close() + }} >
diff --git a/frontend/src/lib/components/common/radioCard/RadioCard.svelte b/frontend/src/lib/components/common/radioCard/RadioCard.svelte new file mode 100644 index 0000000000..6cbb9c78a0 --- /dev/null +++ b/frontend/src/lib/components/common/radioCard/RadioCard.svelte @@ -0,0 +1,59 @@ + + + diff --git a/frontend/src/lib/components/common/table/AppRow.svelte b/frontend/src/lib/components/common/table/AppRow.svelte index bc84f0f884..f016fafd5c 100644 --- a/frontend/src/lib/components/common/table/AppRow.svelte +++ b/frontend/src/lib/components/common/table/AppRow.svelte @@ -3,13 +3,14 @@ import Dropdown from '$lib/components/DropdownV2.svelte' import type MoveDrawer from '$lib/components/MoveDrawer.svelte' import SharedBadge from '$lib/components/SharedBadge.svelte' + import DraftBadge from '$lib/components/DraftBadge.svelte' import type ShareModal from '$lib/components/ShareModal.svelte' - import { AppService, DraftService, type ListableApp } from '$lib/gen' + import { AppService, type ListableApp } from '$lib/gen' import { userStore, workspaceStore } from '$lib/stores' + import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { createEventDispatcher } from 'svelte' import Button from '../button/Button.svelte' import Row from './Row.svelte' - import DraftBadge from '$lib/components/DraftBadge.svelte' import InheritedLabels from '$lib/components/InheritedLabels.svelte' import Badge from '../badge/Badge.svelte' import { @@ -29,7 +30,7 @@ import { goto as gotoUrl } from '$app/navigation' import { page } from '$app/state' import type DeployWorkspaceDrawer from '$lib/components/DeployWorkspaceDrawer.svelte' - import { DELETE, copyToClipboard } from '$lib/utils' + import { copyToClipboard } from '$lib/utils' import AppDeploymentHistory from '$lib/components/apps/editor/AppDeploymentHistory.svelte' import { isDeployable } from '$lib/utils_deployable' import { getDeployUiSettings } from '$lib/components/home/deploy_ui' @@ -38,7 +39,7 @@ import { isCloudHosted } from '$lib/cloud' interface Props { - app: ListableApp & { has_draft?: boolean; draft_only?: boolean; canWrite: boolean } + app: ListableApp & { draft_only?: boolean; canWrite: boolean } marked: string | undefined shareModal: ShareModal moveDrawer: MoveDrawer @@ -65,11 +66,31 @@ const dispatch = createEventDispatcher() - let appExport: { open: (path: string) => void } | undefined = $state(undefined) + let appExport: { open: (path: string, rawApp?: boolean) => void } | undefined = $state(undefined) let appDeploymentHistory: AppDeploymentHistory | undefined = $state(undefined) async function loadAppJson() { - appExport?.open(app.path) + // Thread the row's `raw_app` flag so the JSON drawer's backend + // fetch picks the right draft kind on draft-only items (no + // deployed row to read the kind from server-side). + appExport?.open(app.path, !!app.raw_app) + } + + async function deleteApp(path: string): Promise { + // Draft-only items have no deployed row — the regular route would + // 404. Route the delete through the syncer instead; the `app` vs + // `raw_app` choice mirrors the row's own `raw_app` flag. + if (app.draft_only) { + await UserDraftDbSyncer.save({ + workspace: $workspaceStore ?? '', + itemKind: app.raw_app ? 'raw_app' : 'app', + path, + value: null, + immediate: true + }) + } else { + await AppService.deleteApp({ workspace: $workspaceStore ?? '', path }) + } } @@ -81,11 +102,13 @@ {/if} Raw {/if} - + dispatch('change')} + /> {#if app.labels?.length}
{#each app.labels.slice(0, 3) as label} @@ -132,7 +164,7 @@ variant="subtle" wrapperClasses="w-20" startIcon={{ icon: Pen }} - href="{base}/apps{app.raw_app ? '_raw' : ''}/edit/{app.path}?nodraft=true" + href="{base}/apps{app.raw_app ? '_raw' : ''}/edit/{app.path}" > Edit @@ -157,7 +189,7 @@ aiId={`app-row-dropdown-${app.summary?.length > 0 ? app.summary : app.path}`} aiDescription={`Open dropdown for app ${app.summary?.length > 0 ? app.summary : app.path} options`} items={async () => { - let { draft_only, canWrite, summary, execution_mode, path, has_draft } = app + let { draft_only, canWrite, summary, execution_mode, path } = app const canEdit = canWrite && showEditButton if (draft_only) { @@ -169,17 +201,20 @@ // TODO // @ts-ignore if (event?.shiftKey) { - await AppService.deleteApp({ workspace: $workspaceStore ?? '', path }) + await deleteApp(path) dispatch('change') } else { deleteConfirmedCallback = async () => { - await AppService.deleteApp({ workspace: $workspaceStore ?? '', path }) + await deleteApp(path) dispatch('change') } } }, type: 'delete', - disabled: !canEdit, + // A draft-only row is always the authed user's own draft (the + // list endpoint only surfaces own/legacy draft-only rows), so + // discarding it never requires write permission on the path. + disabled: !showEditButton, hide: $userStore?.operator }, { @@ -273,25 +308,6 @@ } ] : []), - ...(has_draft - ? [ - { - displayName: 'Delete Draft', - icon: Trash, - action: async () => { - await DraftService.deleteDraft({ - workspace: $workspaceStore ?? '', - path, - kind: 'app' - }) - dispatch('change') - }, - type: DELETE, - disabled: !canWrite, - hide: $userStore?.operator - } - ] - : []), { displayName: 'Delete', icon: Trash, @@ -299,11 +315,11 @@ // TODO // @ts-ignore if (event?.shiftKey) { - await AppService.deleteApp({ workspace: $workspaceStore ?? '', path }) + await deleteApp(path) dispatch('change') } else { deleteConfirmedCallback = async () => { - await AppService.deleteApp({ workspace: $workspaceStore ?? '', path }) + await deleteApp(path) dispatch('change') } } diff --git a/frontend/src/lib/components/common/table/FlowRow.svelte b/frontend/src/lib/components/common/table/FlowRow.svelte index aaed01e6d7..89a08b412b 100644 --- a/frontend/src/lib/components/common/table/FlowRow.svelte +++ b/frontend/src/lib/components/common/table/FlowRow.svelte @@ -5,16 +5,17 @@ import type MoveDrawer from '$lib/components/MoveDrawer.svelte' import ScheduleEditor from '$lib/components/triggers/schedules/ScheduleEditor.svelte' import SharedBadge from '$lib/components/SharedBadge.svelte' + import DraftBadge from '$lib/components/DraftBadge.svelte' import type ShareModal from '$lib/components/ShareModal.svelte' - import { FlowService, type Flow, DraftService } from '$lib/gen' + import { FlowService, type Flow } from '$lib/gen' import { userStore, workspaceStore } from '$lib/stores' + import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { createEventDispatcher } from 'svelte' import Badge from '../badge/Badge.svelte' import Button from '../button/Button.svelte' import Row from './Row.svelte' - import DraftBadge from '$lib/components/DraftBadge.svelte' import { sendUserToast } from '$lib/toast' - import { DELETE, copyToClipboard, isOwner } from '$lib/utils' + import { copyToClipboard, isOwner } from '$lib/utils' import { isDeployable } from '$lib/utils_deployable' import type DeployWorkspaceDrawer from '$lib/components/DeployWorkspaceDrawer.svelte' @@ -40,7 +41,13 @@ import { isCloudHosted } from '$lib/cloud' interface Props { - flow: Flow & { has_draft?: boolean; draft_only?: boolean; canWrite: boolean } + flow: Flow & { + draft_only?: boolean + is_draft?: boolean + draft_path?: string + draft_users?: { username?: string | null }[] + canWrite: boolean + } marked: string | undefined shareModal: ShareModal moveDrawer: MoveDrawer @@ -85,7 +92,20 @@ async function deleteFlow(path: string): Promise { try { - await FlowService.deleteFlowByPath({ workspace: $workspaceStore!, path }) + // Draft-only items have no deployed row to delete — the regular + // route would 404. Route the delete through the syncer so the + // per-user draft row is removed instead. + if (flow.draft_only) { + await UserDraftDbSyncer.save({ + workspace: $workspaceStore!, + itemKind: 'flow', + path, + value: null, + immediate: true + }) + } else { + await FlowService.deleteFlowByPath({ workspace: $workspaceStore!, path }) + } dispatch('change') sendUserToast(`Deleted flow ${path}`) } catch (err) { @@ -105,13 +125,13 @@ aiId={`flow-row-${flow.path}`} aiDescription={`Button to access the form to run the flow ${flow.summary ?? flow.path}`} href={flow.draft_only - ? `${base}/flows/edit/${flow.path}?nodraft=true` + ? `${base}/flows/edit/${flow.path}` : `${base}/flows/get/${flow.path}?workspace=${$workspaceStore}`} kind="flow" workspaceId={flow.workspace_id ?? $workspaceStore ?? ''} {marked} - path={flow.path} - summary={flow.summary} + path={flow.draft_path ?? flow.path} + summary={flow.is_draft ? `${flow.summary || flow.draft_path || flow.path}*` : flow.summary} {errorHandlerMuted} canFavorite={!flow.draft_only} {depth} @@ -122,7 +142,16 @@ archived {/if} - + dispatch('change')} + /> {#if flow.labels?.length}
{#each flow.labels.slice(0, 3) as label} @@ -154,7 +183,7 @@ wrapperClasses="w-20" unifiedSize="md" startIcon={{ icon: Pen }} - href="{base}/flows/edit/{flow.path}?nodraft=true" + href="{base}/flows/edit/{flow.path}" aiId={`edit-flow-button-${flow.summary?.length > 0 ? flow.summary : flow.path}`} aiDescription={`Edits the flow ${flow.summary?.length > 0 ? flow.summary : flow.path}`} > @@ -182,7 +211,7 @@ aiId={`flow-row-dropdown-${flow.summary?.length > 0 ? flow.summary : flow.path}`} aiDescription={`Open dropdown for flow ${flow.summary?.length > 0 ? flow.summary : flow.path} options`} items={async () => { - let { draft_only, path, archived, has_draft } = flow + let { draft_only, path, archived } = flow let owner = isOwner(path, $userStore, $workspaceStore) const canEdit = flow.canWrite && showEditButton if (draft_only) { @@ -201,7 +230,10 @@ } }, type: 'delete', - disabled: !owner, + // A draft-only row is always the authed user's own draft (the + // list endpoint only surfaces own/legacy draft-only rows), so + // discarding it never requires write permission on the path. + disabled: !showEditButton, hide: $userStore?.operator } ] @@ -295,25 +327,6 @@ disabled: !owner || !canEdit, hide: $userStore?.operator }, - ...(has_draft - ? [ - { - displayName: 'Delete Draft', - icon: Trash, - action: async () => { - await DraftService.deleteDraft({ - workspace: $workspaceStore ?? '', - path, - kind: 'flow' - }) - dispatch('change') - }, - type: DELETE, - disabled: !owner, - hide: $userStore?.operator - } - ] - : []), { displayName: 'Delete', icon: Trash, diff --git a/frontend/src/lib/components/common/table/RawAppRow.svelte b/frontend/src/lib/components/common/table/RawAppRow.svelte index e759a1d9fe..3f74356506 100644 --- a/frontend/src/lib/components/common/table/RawAppRow.svelte +++ b/frontend/src/lib/components/common/table/RawAppRow.svelte @@ -33,7 +33,7 @@ {} }: Props = $props() @@ -153,7 +165,12 @@ onkeydown={clickToSelect ? handleRowKeydown : undefined} > {#if isSelectable} - + + {:else if selectDisabledReason} + + + {#snippet text()}{selectDisabledReason}{/snippet} + {:else if alignWithSelectable}
{/if} @@ -208,7 +225,11 @@ {/if}
- {path} + {#if pathDisplay} + {@render pathDisplay()} + {:else} + {path} + {/if}
{/snippet} diff --git a/frontend/src/lib/components/common/table/RowIcon.svelte b/frontend/src/lib/components/common/table/RowIcon.svelte index 67a707ff1e..ec82773a4d 100644 --- a/frontend/src/lib/components/common/table/RowIcon.svelte +++ b/frontend/src/lib/components/common/table/RowIcon.svelte @@ -16,8 +16,10 @@ LayoutDashboard, Mail, Route, - Unplug + Unplug, + Workflow } from 'lucide-svelte' + import FileIcon from '$lib/components/raw_apps/FileIcon.svelte' interface Props { kind: @@ -25,6 +27,7 @@ | 'flow' | 'app' | 'raw_app' + | 'raw_app_file' | 'resource' | 'variable' | 'resource_type' @@ -51,12 +54,16 @@ | 'gcp_trigger' | 'azure_trigger' | 'email_trigger' + | 'data_pipeline' /** For 'trigger' kind, specifies the specific trigger type (routes, schedules, etc.) */ triggerKind?: string | undefined + /** For 'raw_app_file' kind: the file name/path, used to pick an + * extension-specific icon. */ + path?: string | undefined size?: number } - let { kind, triggerKind = undefined, size = 16 }: Props = $props() + let { kind, triggerKind = undefined, path = undefined, size = 16 }: Props = $props() // Map per-kind backend names (e.g. `kafka_trigger`) to the legacy short // names the icon switch already handles, so we don't have to duplicate cases. @@ -83,6 +90,8 @@ {:else if effectiveKind === 'app' || effectiveKind === 'raw_app'} + {:else if effectiveKind === 'raw_app_file'} + {:else if effectiveKind === 'script'} {:else if effectiveKind === 'variable'} @@ -117,6 +126,8 @@ {:else if effectiveKind === 'trigger'} + {:else if effectiveKind === 'data_pipeline'} + {:else}
{/if} diff --git a/frontend/src/lib/components/common/table/ScriptRow.svelte b/frontend/src/lib/components/common/table/ScriptRow.svelte index 2174776167..1d87d4c8d5 100644 --- a/frontend/src/lib/components/common/table/ScriptRow.svelte +++ b/frontend/src/lib/components/common/table/ScriptRow.svelte @@ -5,18 +5,19 @@ import type MoveDrawer from '$lib/components/MoveDrawer.svelte' import ScheduleEditor from '$lib/components/triggers/schedules/ScheduleEditor.svelte' import SharedBadge from '$lib/components/SharedBadge.svelte' + import DraftBadge from '$lib/components/DraftBadge.svelte' import type ShareModal from '$lib/components/ShareModal.svelte' - import { ScriptService, type Script, DraftService } from '$lib/gen' + import { ScriptService, type Script } from '$lib/gen' import { hubBaseUrlStore, userStore, workspaceStore } from '$lib/stores' + import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { createEventDispatcher } from 'svelte' import Badge from '../badge/Badge.svelte' import Button from '../button/Button.svelte' import Row from './Row.svelte' - import DraftBadge from '$lib/components/DraftBadge.svelte' import { sendUserToast } from '$lib/toast' - import { capitalize, copyToClipboard, DELETE, isOwner } from '$lib/utils' + import { capitalize, copyToClipboard, isOwner } from '$lib/utils' import { isDeployable } from '$lib/utils_deployable' import type DeployWorkspaceDrawer from '$lib/components/DeployWorkspaceDrawer.svelte' @@ -52,7 +53,13 @@ import { isCloudHosted } from '$lib/cloud' interface Props { - script: Script & { canWrite: boolean; use_codebase: boolean } + script: Script & { + canWrite: boolean + use_codebase: boolean + is_draft?: boolean + draft_path?: string + draft_users?: { username?: string | null }[] + } marked: string | undefined shareModal: ShareModal moveDrawer: MoveDrawer @@ -104,7 +111,20 @@ } async function deleteScript(path: string): Promise { - await ScriptService.deleteScriptByPath({ workspace: $workspaceStore!, path }) + // Draft-only items have no deployed row to delete — the regular + // route would 404. Route the delete through the syncer so the + // per-user draft row is removed instead. + if (script.draft_only) { + await UserDraftDbSyncer.save({ + workspace: $workspaceStore!, + itemKind: 'script', + path, + value: null, + immediate: true + }) + } else { + await ScriptService.deleteScriptByPath({ workspace: $workspaceStore!, path }) + } dispatch('change') sendUserToast(`Deleted script ${path}`) } @@ -127,8 +147,10 @@ : `${base}/scripts/get/${script.hash}?workspace=${$workspaceStore}`} kind="script" {marked} - path={script.path} - summary={script.summary} + path={script.draft_path ?? script.path} + summary={script.is_draft + ? `${script.summary || script.draft_path || script.path}*` + : script.summary} {errorHandlerMuted} workspaceId={$workspaceStore ?? ''} canFavorite={!script.draft_only} @@ -169,7 +191,16 @@ > {/if} - + dispatch('change')} + /> {#if script.labels?.length}
{#each script.labels.slice(0, 3) as label} @@ -265,7 +296,10 @@ } }, type: dlt, - disabled: !canEdit + // A draft-only row is always the authed user's own draft (the + // list endpoint only surfaces own/legacy draft-only rows), so + // discarding it never requires write permission on the path. + disabled: !showEditButton } ] } @@ -406,25 +440,6 @@ hide: $userStore?.operator }, - ...(script.has_draft - ? [ - { - displayName: 'Delete Draft', - icon: Trash, - action: async () => { - await DraftService.deleteDraft({ - workspace: $workspaceStore ?? '', - path: script.path, - kind: 'script' - }) - dispatch('change') - }, - type: DELETE, - disabled: !owner, - hide: $userStore?.operator - } - ] - : []), ...($userStore?.is_admin || $userStore?.is_super_admin ? [ { diff --git a/frontend/src/lib/components/copilot/autocomplete/Autocompletor.ts b/frontend/src/lib/components/copilot/autocomplete/Autocompletor.ts index b2484ff909..0ee2ad3f78 100644 --- a/frontend/src/lib/components/copilot/autocomplete/Autocompletor.ts +++ b/frontend/src/lib/components/copilot/autocomplete/Autocompletor.ts @@ -3,7 +3,8 @@ import { sleep } from '$lib/utils' import { editor as meditor, Position, languages, type IDisposable } from 'monaco-editor' import { LRUCache } from 'lru-cache' import { autocompleteRequest } from './request' -import { FIM_MAX_TOKENS, getModelContextWindow } from '../lib' +import { FIM_MAX_TOKENS } from '../lib' +import { getModelContextWindow } from '../modelConfig' import { setGlobalCSS } from '../shared' import { supportsAutocomplete } from '../utils' import { get } from 'svelte/store' diff --git a/frontend/src/lib/components/copilot/chat/AIButton.svelte b/frontend/src/lib/components/copilot/chat/AIButton.svelte index 139c524230..0234810a8b 100644 --- a/frontend/src/lib/components/copilot/chat/AIButton.svelte +++ b/frontend/src/lib/components/copilot/chat/AIButton.svelte @@ -1,6 +1,7 @@ -
+ +
+ {#if isDraggingFiles} +
+
+ + Drop files to attach +
+
+ {/if} {#if !hideHeader}
{/if}
@@ -508,6 +693,13 @@
{/if}
+ + {#if aiChatManager.mode === AIMode.GLOBAL} + + + {/if} {#if inputPreface} {@render inputPreface()} {/if} @@ -515,6 +707,7 @@ bind:this={aiChatInput} bind:selectedContext {availableContext} + showContext={aiChatManager.mode !== AIMode.GLOBAL} disabled={disabled || hasActiveUserQuestion} isFirstMessage={messages.length === 0} /> @@ -566,11 +759,76 @@ setShowing={(showing) => { if (!showing) close() }} + onSelectFile={(name) => { + aiChatInput?.insertFileMention(name) + close() + }} /> {/if} {/snippet} {/if} + {#if canAttachFiles} + [ + { displayName: 'Attach file', icon: FileText, action: () => linkFiles() }, + { + // A real (live) link needs the File System Access API; without it the + // folder is only snapshotted, so call it "Add folder", not "Link folder". + displayName: canUseFsAccess ? 'Link folder' : 'Add folder', + icon: Folder, + tooltip: canUseFsAccess + ? 'Linked live — the assistant reads the folder’s current files from disk and refreshes each turn.' + : 'Loaded as a snapshot — the folder’s files are copied into your browser (they won’t auto-update). For a live link that refreshes from disk, use a Chromium-based browser (Chrome, Edge).', + action: () => linkFolder() + } + ]} + placement="bottom-start" + fixedHeight={false} + > + {#snippet buttonReplacement()} + + {seg.att.content}{:else}{/if}{/each} -
- {/if} -
- {/if} - {#if message.role === 'user' && message.snapshot} -
- Saved {message.snapshot.type === 'flow' ? 'a flow' : 'an app'} snapshot - {seg.att.content}{:else}{/if}{/each} +
+ {/if} +
+ {/if} + {#if message.role === 'user' && message.snapshot} +
- Revert + Saved {message.snapshot.type === 'flow' ? 'a flow' : 'an app'} snapshot + +
+ {/if} +
+ {#if message.role === 'user' && message.error} +
+
{/if} -
-{#if message.role === 'user' && message.error} -
- -
{/if} diff --git a/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte new file mode 100644 index 0000000000..0a911134b6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/AIChatModelSettings.svelte @@ -0,0 +1,456 @@ + + +{#snippet externalLinkIcon()} + +{/snippet} + + + {#snippet buttonReplacement()} +
+ +
+ {/snippet} + {#snippet menu({ item, builders, close })} +
+ + + +
+
Model
+
+ {#each models as m (m.provider + m.model)} + selectModel(m)} + > + {m.model} + {#if m.model === providerModel.model && m.provider === providerModel.provider} + + {/if} + + {/each} +
+ +
+ {#if capability.supported} + + +
+ Thinking + {currentStop} +
+ {#if stops.length > 1} + +
+ selectReasoning(stops[+e.currentTarget.value])} + use:isolatePointer + class="lean-range no-default-style w-full" + aria-label="Reasoning effort" + /> +
+ {/if} +
+ {:else} + +
+
Thinking
+
Not supported by this model
+
+ {/if} +
+ {/snippet} +
+ + + + diff --git a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte b/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte deleted file mode 100644 index 683bf3c586..0000000000 --- a/frontend/src/lib/components/copilot/chat/AIChatSettingsMenu.svelte +++ /dev/null @@ -1,186 +0,0 @@ - - -{#snippet externalLinkIcon()} - -{/snippet} - - - {#snippet buttonReplacement()} - +
+
+ {#if expanded} +
+ {content} +
+ {/if} +
diff --git a/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts b/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts index a69d6670b4..a728d1c96b 100644 --- a/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/ContextManager.svelte.ts @@ -11,7 +11,7 @@ import type { ExtendedOpenFlow } from '$lib/components/flows/types' export interface ScriptOptions { lang: ScriptLang | 'bunnative' - code: string + getCode: () => string error: string | undefined args: Record path: string | undefined @@ -77,7 +77,10 @@ export default class ContextManager { } this.workspace = workspace this.selectedContext = currentlySelectedContext.filter( - (context) => context.type === 'workspace_script' || context.type === 'workspace_flow' + (context) => + context.type === 'workspace_script' || + context.type === 'workspace_flow' || + context.type === 'workspace_app' ) } @@ -158,6 +161,7 @@ export default class ContextManager { (c) => c.type === 'workspace_script' || c.type === 'workspace_flow' || + c.type === 'workspace_app' || newAvailableContext.some((ac) => ac.type === c.type && ac.title === c.title) ) .map((c) => @@ -192,7 +196,7 @@ export default class ContextManager { { type: 'code', title: this.getContextCodePath(scriptOptions) ?? '', - content: scriptOptions.code, + content: scriptOptions.getCode(), lang: scriptOptions.lang } ] @@ -209,22 +213,25 @@ export default class ContextManager { } } - if (scriptOptions.lastSavedCode && scriptOptions.lastSavedCode !== scriptOptions.code) { + if (scriptOptions.lastSavedCode && scriptOptions.lastSavedCode !== scriptOptions.getCode()) { newAvailableContext.push({ type: 'diff', title: 'diff_with_last_saved_draft', // can't use spaces in the title, because it will break the word match in the context text area hightlighting logic content: scriptOptions.lastSavedCode ?? '', - diff: diffLines(scriptOptions.lastSavedCode ?? '', scriptOptions.code), + diff: diffLines(scriptOptions.lastSavedCode ?? '', scriptOptions.getCode()), lang: scriptOptions.lang }) } - if (scriptOptions.lastDeployedCode && scriptOptions.lastDeployedCode !== scriptOptions.code) { + if ( + scriptOptions.lastDeployedCode && + scriptOptions.lastDeployedCode !== scriptOptions.getCode() + ) { newAvailableContext.push({ type: 'diff', title: 'diff_with_last_deployed_version', content: scriptOptions.lastDeployedCode ?? '', - diff: diffLines(scriptOptions.lastDeployedCode ?? '', scriptOptions.code), + diff: diffLines(scriptOptions.lastDeployedCode ?? '', scriptOptions.getCode()), lang: scriptOptions.lang }) } @@ -251,7 +258,7 @@ export default class ContextManager { { type: 'code', title: this.getContextCodePath(scriptOptions) ?? '', - content: scriptOptions.code, + content: scriptOptions.getCode(), lang: scriptOptions.lang, deletable: false }, @@ -277,19 +284,20 @@ export default class ContextManager { newSelectedContext = newSelectedContext .filter( (c) => - (c.type === 'code_piece' && scriptOptions.code.includes(c.content)) || + (c.type === 'code_piece' && scriptOptions.getCode().includes(c.content)) || c.type === 'code' || // Workspace references are user-picked via @-mention and not in // availableContext; preserve so badges survive editor refreshes. c.type === 'workspace_script' || c.type === 'workspace_flow' || + c.type === 'workspace_app' || newAvailableContext.some((ac) => ac.type === c.type && ac.title === c.title) ) .map((c) => { if (c.type === 'code') { return { ...c, - content: scriptOptions.code, + content: scriptOptions.getCode(), title: this.getContextCodePath(scriptOptions) } } @@ -403,7 +411,10 @@ export default class ContextManager { type: 'diff' as const, title: 'diff_with_last_deployed_version', content: this.scriptOptions.lastDeployedCode ?? '', - diff: diffLines(this.scriptOptions.lastDeployedCode ?? '', this.scriptOptions.code), + diff: diffLines( + this.scriptOptions.lastDeployedCode ?? '', + this.scriptOptions.getCode() + ), lang: this.scriptOptions.lang } ] @@ -428,21 +439,25 @@ export default class ContextManager { displayMessages: DisplayMessage[], dbSchemas: DBSchemas ): DisplayMessage[] { - return displayMessages.map((m) => ({ - ...m, - contextElements: - m.role !== 'tool' && m.contextElements - ? m.contextElements.map((c) => - c.type === 'db' - ? { - type: 'db', - title: c.title, - schema: dbSchemas[c.title] - } - : c - ) - : undefined - })) + return displayMessages.map((m) => { + // Only user/assistant messages carry contextElements; tool and summary + // messages pass through untouched. + if ((m.role === 'user' || m.role === 'assistant') && m.contextElements) { + return { + ...m, + contextElements: m.contextElements.map((c) => + c.type === 'db' + ? { + type: 'db' as const, + title: c.title, + schema: dbSchemas[c.title] + } + : c + ) + } + } + return m + }) } setSelectedModuleContext( diff --git a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte index 8c47cede58..c6f3268e50 100644 --- a/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte +++ b/frontend/src/lib/components/copilot/chat/ContextTextarea.svelte @@ -2,11 +2,14 @@ import autosize from '$lib/autosize' import { tick } from 'svelte' import type { ContextElement } from './context' + import { AIMode } from './AIChatManager.svelte' + import ChatCommandPicker from './ChatCommandPicker.svelte' import ChatContextPicker from './ChatContextPicker.svelte' import Portal from '$lib/components/Portal.svelte' import { zIndexes } from '$lib/zIndexes' import { twMerge } from 'tailwind-merge' - import { CHAT_INPUT_PADDING } from './aiChatManagerContext' + import { CHAT_INPUT_PADDING, getAiChatManager } from './aiChatManagerContext' + import { MENTION_RE, mentionTitle, formatMention } from './mention' import { createFloatingActions, createVirtualElement } from 'svelte-floating-ui' import { flip, offset, shift } from 'svelte-floating-ui/dom' import { @@ -50,10 +53,11 @@ onKeyDown = undefined }: Props = $props() - const MENTION_RE = /@[\w/.\-\[\]]+/g + const aiChatManager = getAiChatManager() + function extractMentions(text: string): Set { const out = new Set() - for (const m of text.matchAll(MENTION_RE)) out.add(m[0].slice(1)) + for (const m of text.matchAll(MENTION_RE)) out.add(mentionTitle(m[0])) return out } @@ -66,11 +70,22 @@ let showContextTooltip = $state(false) let contextTooltipWord = $state('') + let showCommandTooltip = $state(false) + let commandTooltipWord = $state('') let textarea = $state(undefined) let tooltipElement = $state(undefined) let chatContextPicker: ChatContextPicker | undefined = $state() + let chatCommandPicker: ChatCommandPicker | undefined = $state() + let commandSkillsRefreshInFlight = false - // Virtual reference anchored at the `@` that opened the mention (not the + const commandSkills = $derived( + aiChatManager.mode === AIMode.GLOBAL && aiChatManager.isSessionChat + ? aiChatManager.sessionCommands + : [] + ) + const activeTooltipWord = $derived(showContextTooltip ? contextTooltipWord : commandTooltipWord) + + // Virtual reference anchored at the trigger that opened the picker (not the // caret), so the picker stays put while the user types the query. // svelte-floating-ui's `createVirtualElement` takes a raw ClientRect and // wraps it in a function internally — re-`update()` on each anchor move. @@ -221,6 +236,18 @@ .replace(/'/g, ''') } + // Inverse of escapeHtml (& last so an escaped entity isn't double-decoded). Mentions + // are parsed out of the escaped HTML, so a title is un-escaped before the store lookup — + // else a filename like `R&D notes.md` (escaped to `R&D notes.md`) would never match. + function unescapeHtml(text: string) { + return text + .replace(/</g, '<') + .replace(/>/g, '>') + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/&/g, '&') + } + function getHighlightedText(text: string) { let html = escapeHtml(text) // Wrap collapsed-paste tokens as clickable chips. The span keeps the exact @@ -231,11 +258,13 @@ if (!att) return match return `${match}` }) - html = html.replace(/@[\w/.\-\[\]]+/g, (match) => { - const title = match.slice(1) + html = html.replace(MENTION_RE, (match) => { + const title = unescapeHtml(mentionTitle(match)) const inContext = availableContext.find((c) => c.title === title) || - selectedContext.find((c) => c.title === title) + selectedContext.find((c) => c.title === title) || + // Attached-file mentions (`@filename`) highlight just like context. + aiChatManager.attachedFiles.get(title) if (inContext) { return `${match}` } @@ -510,19 +539,33 @@ showContextTooltip = false } + function refreshCommandSkills() { + if (commandSkillsRefreshInFlight) return + commandSkillsRefreshInFlight = true + void aiChatManager.refreshGlobalSkills().finally(() => { + commandSkillsRefreshInFlight = false + }) + } + + function getCommandFilter(text: string): string | undefined { + if (aiChatManager.mode !== AIMode.GLOBAL || !aiChatManager.isSessionChat) return undefined + const match = /^\/([a-z0-9-]*)$/.exec(text) + return match?.[1] + } + function updateAnchorRect() { if (!textarea) return + const triggerWord = activeTooltipWord + if (!triggerWord) return try { - // Index of the `@` that started the current mention. handleInput - // only opens the picker when `contextTooltipWord` (= `@xxx`) is the - // LAST whitespace-separated word in `value`, so the `@` always sits - // at `value.length - contextTooltipWord.length`. - const atIndex = value.length - contextTooltipWord.length - const coords = getCaretCoordinates(textarea, atIndex) + // Inline `@` anchors to the last word; slash commands only open when + // `/...` is the whole input, so the trigger sits at index 0. + const triggerIndex = triggerWord.startsWith('/') ? 0 : value.length - triggerWord.length + const coords = getCaretCoordinates(textarea, triggerIndex) const rect = textarea.getBoundingClientRect() // getCaretCoordinates returns content-relative coords; subtract the - // textarea's own scroll so the anchor tracks the `@` once the input is - // capped (max-height) and scrolls internally. + // textarea's own scroll so the anchor tracks the trigger once the input + // is capped (max-height) and scrolls internally. anchorRect = new DOMRect( rect.left + coords.left - textarea.scrollLeft, rect.top + coords.top - textarea.scrollTop, @@ -542,6 +585,19 @@ function handleInput(e: Event) { textarea = e.target as HTMLTextAreaElement + const commandFilter = getCommandFilter(value) + if (commandFilter !== undefined) { + const wasShowing = showCommandTooltip + showCommandTooltip = true + commandTooltipWord = `/${commandFilter}` + showContextTooltip = false + contextTooltipWord = '' + if (!wasShowing) refreshCommandSkills() + return + } + showCommandTooltip = false + commandTooltipWord = '' + const words = value.split(/\s+/) const lastWord = words[words.length - 1] @@ -558,6 +614,12 @@ } } + function handleCommandSelection(skill: { name: string }) { + value = `/${skill.name} ` + showCommandTooltip = false + setTimeout(() => textarea?.focus(), 0) + } + function handleKeyDown(e: KeyboardEvent) { // Pass to parent first if provided if (onKeyDown) { @@ -569,6 +631,22 @@ return } + if (showCommandTooltip) { + if ( + e.key === 'ArrowDown' || + e.key === 'ArrowUp' || + e.key === 'Enter' || + e.key === 'Tab' || + e.key === 'Escape' + ) { + chatCommandPicker?.handleKeydown(e) + } + if (e.key === 'Enter') { + e.preventDefault() + } + return + } + if (showContextTooltip) { // Forward navigation keys to the picker so the textarea-focused // user can drive it. The picker preventDefault/stopPropagation's @@ -606,11 +684,11 @@ } $effect(() => { - // Re-track on every value change. The `@` position can shift when the - // user adds/deletes text BEFORE it (line wrap, etc.); the picker should - // follow. floating-ui's autoUpdate only fires on scroll/resize. + // Re-track on every value change. The trigger position can shift when + // the user adds/deletes text before it (line wrap, etc.); the picker + // should follow. floating-ui's autoUpdate only fires on scroll/resize. void value - if (showContextTooltip) updateAnchorRect() + if (showContextTooltip || showCommandTooltip) updateAnchorRect() }) $effect(() => { @@ -684,9 +762,9 @@ ondragstart={handlePasteDragStart} onscroll={(e) => { scrollTop = e.currentTarget.scrollTop - // Keep the `@` picker pinned to its anchor while the input scrolls + // Keep the picker pinned to its anchor while the input scrolls // internally (autoUpdate can't observe a virtual ref's scroll). - if (showContextTooltip) updateAnchorRect() + if (showContextTooltip || showCommandTooltip) updateAnchorRect() }} onblur={() => { setTimeout(() => { @@ -695,6 +773,7 @@ return } showContextTooltip = false + showCommandTooltip = false }, 200) }} {placeholder} @@ -708,7 +787,7 @@ >
-{#if showContextTooltip} +{#if showContextTooltip || showCommandTooltip}
- { - handleContextSelection(element) - }} - onSelectWorkspaceItem={(element) => { - onAddContext(element) - updateInstructionsWithContext(element) - showContextTooltip = false - setTimeout(() => textarea?.focus(), 0) - }} - externalFilter={contextTooltipWord.slice(1)} - autoFocus={false} - setShowing={(showing) => { - showContextTooltip = showing - }} - /> + {#if showCommandTooltip} + { + showCommandTooltip = showing + }} + /> + {:else} + { + handleContextSelection(element) + }} + onSelectWorkspaceItem={(element) => { + onAddContext(element) + updateInstructionsWithContext(element) + showContextTooltip = false + setTimeout(() => textarea?.focus(), 0) + }} + externalFilter={contextTooltipWord.slice(1)} + autoFocus={false} + setShowing={(showing) => { + showContextTooltip = showing + }} + onSelectFile={(name) => { + // Replace the in-progress `@word` with the chosen mention (bracketed if the + // filename has spaces, so the highlighter captures it whole). + const index = value.lastIndexOf('@') + value = (index !== -1 ? value.substring(0, index) : value) + `${formatMention(name)} ` + showContextTooltip = false + setTimeout(() => textarea?.focus(), 0) + }} + /> + {/if}
{/if} diff --git a/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte b/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte new file mode 100644 index 0000000000..80b317ba12 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/ContextUsageIndicator.svelte @@ -0,0 +1,91 @@ + + +{#if visible} + + +
+
+
+
+
+ {#snippet text()} +
+

Context usage

+

+ ~{formatTokenCount(usedTokens)}{contextWindow + ? ` / ${formatTokenCount(contextWindow)}` + : ''}{fillPct !== undefined ? ` (${fillPct}%)` : ''} +

+ {#if ratio !== undefined && ratio >= COMPACTION_TRIGGER_RATIO} +

History will be compacted soon to free up space.

+ {/if} + {#if canCompact} +

+ Type /compact to summarize and free up space now. +

+ {/if} +
+ {/snippet} +
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts b/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts index a2400b88df..9173e4c441 100644 --- a/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/HistoryManager.svelte.ts @@ -1,8 +1,17 @@ -import { openDB, type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' +import { type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' import type { DisplayMessage } from './shared' import { expanded, messageDraft } from './chatDraft' import { createLongHash } from '$lib/editorLangUtils' +import { userScopedDb, type UserScopedDbMigrateDeps } from '$lib/userScopedDb' import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs' +import type { PersistedContextUsage } from './tokenUsage' + +// Base IndexedDB name; userScopedDb namespaces the effective DB by the logged-in +// user's email so chat messages are never physically shared across users on a +// shared browser. The bare name is also the legacy (pre-namespacing) DB, claimed +// once on first login. +const DB_NAME = 'copilot-chat-history' + interface ChatSchema extends IDBSchema { chats: { key: string @@ -13,12 +22,73 @@ interface ChatSchema extends IDBSchema { title: string lastModified: number sessionId?: string + // New writes store the plain reported token count; chats persisted by + // earlier versions may still hold the legacy anchor object. + contextUsage?: PersistedContextUsage } } } +function createChatStore(db: IDBPDatabase): void { + if (!db.objectStoreNames.contains('chats')) { + db.createObjectStore('chats', { keyPath: 'id' }) + } +} + +// Shared across all HistoryManager instances. Each instance owns its own +// userScopedDb handle (see below), so without this the legacy claim could run +// concurrently in several instances on first login — and racing `deleteDB`s can +// block on each other's still-open legacy connections. Deduping to a single +// session-wide promise restores the "runs exactly once" guarantee the pre-factory +// module-level guard had. Reset on failure so a later instance can retry. +let legacyChatClaim: Promise | undefined + +async function migrateLegacyChatDb( + scopedDb: IDBPDatabase, + deps: UserScopedDbMigrateDeps +): Promise { + legacyChatClaim ??= claimLegacyChatDb(scopedDb, deps).catch((e) => { + legacyChatClaim = undefined + throw e + }) + return legacyChatClaim +} + +// One-shot claim of the pre-namespacing chat-history DB: when the user-scoped +// DB has no chats yet (first login on a previously single-user browser), copy +// every record from the legacy un-namespaced DB into it, then delete the legacy +// DB. Both session-tagged and untagged chats belong to the prior single browser +// user, so all are claimed. Subsequent users start with an empty DB. +async function claimLegacyChatDb( + scopedDb: IDBPDatabase, + { openDB, deleteDB }: UserScopedDbMigrateDeps +): Promise { + if ((await scopedDb.count('chats')) > 0) return + const legacy = await openDB(DB_NAME, 1, { upgrade: createChatStore }) + const legacyChats = await legacy.getAll('chats') + if (legacyChats.length > 0) { + const tx = scopedDb.transaction('chats', 'readwrite') + await Promise.all([...legacyChats.map((c) => tx.store.put(c)), tx.done]) + } + legacy.close() + await deleteDB(DB_NAME) +} + +// Test-only: reset the session-wide legacy-claim guard so suites can exercise +// the migration deterministically regardless of test order. +export function __resetLegacyChatClaimForTesting(): void { + legacyChatClaim = undefined +} + export default class HistoryManager { - private indexDB: IDBPDatabase | undefined = undefined + // Per-instance handle to the shared per-user DB lifecycle. There is one + // HistoryManager per AIChatManager (the singleton + one per session runtime), + // so the handle must be per-instance — not a module singleton. + private dbh = userScopedDb(DB_NAME, { + version: 1, + upgrade: createChatStore, + migrate: migrateLegacyChatDb + }) private savedChats: Record< string, @@ -29,6 +99,7 @@ export default class HistoryManager { id: string lastModified: number sessionId?: string + contextUsage?: PersistedContextUsage } > = $state({}) @@ -47,16 +118,13 @@ export default class HistoryManager { ) async init() { + // whenReady() is email-gated (returns undefined before the user is known — + // all callers run post-login, and the singleton re-inits via onUserChange), + // runs the legacy migration once, and reopens automatically on user change. + const db = await this.dbh.whenReady() + if (!db) return try { - this.indexDB = await openDB('copilot-chat-history', 1, { - upgrade(indexDB) { - if (!indexDB.objectStoreNames.contains('chats')) { - indexDB.createObjectStore('chats', { keyPath: 'id' }) - } - } - }) - - const chats = await this.indexDB.getAll('chats') + const chats = await db.getAll('chats') this.savedChats = chats.reduce( (acc, chat) => { acc[chat.id] = chat @@ -65,13 +133,12 @@ export default class HistoryManager { {} as typeof this.savedChats ) } catch (err) { - console.error('Could not open chat history database', err) - return {} + console.error('Could not load chat history', err) } } close() { - this.indexDB?.close() + this.dbh.close() } getCurrentChatId() { @@ -92,9 +159,10 @@ export default class HistoryManager { const snapshot = $state.snapshot(existing) const updated = { ...snapshot, sessionId } this.savedChats = { ...this.savedChats, [chatId]: updated } - if (this.indexDB) { - await this.indexDB.put('chats', updated) - } + // Resolve the DB via the handle (not a cached ref) so a write always lands + // in the current user's DB, even after an in-place user switch. + const db = await this.dbh.whenReady() + if (db) await db.put('chats', updated) } getPastChats() { @@ -105,11 +173,25 @@ export default class HistoryManager { return Object.values(this.savedChats) } - async saveChat(displayMessages: DisplayMessage[], messages: ChatCompletionMessageParam[]) { + async saveChat( + displayMessages: DisplayMessage[], + messages: ChatCompletionMessageParam[], + contextUsage?: number + ) { if (displayMessages.length > 0) { - // Expand any collapsed-paste tokens so the title is readable text, not - // the chip label + its zero-width id chars. - const title = expanded(messageDraft(displayMessages[0])).slice(0, 50) + // Compaction replaces the original first message with a summary boundary. + // Re-deriving the title would then shift it to the first surviving tail + // message, so once that boundary leads the transcript, keep the title + // computed before compaction. Otherwise derive it from the first message, + // expanding collapsed-paste tokens so it reads as text rather than the + // chip label + its zero-width id chars. + const existingTitle = this.savedChats[this.currentChatId]?.title + const title = + displayMessages[0].role === 'summary' && existingTitle !== undefined + ? existingTitle + : expanded( + messageDraft(displayMessages.find((m) => m.role !== 'summary') ?? displayMessages[0]) + ).slice(0, 50) // we don't want to save the snapshot in the history const updatedChat = { actualMessages: $state.snapshot(messages), @@ -120,21 +202,25 @@ export default class HistoryManager { title, id: this.currentChatId, lastModified: Date.now(), - ...(this.sessionId ? { sessionId: this.sessionId } : {}) + ...(this.sessionId ? { sessionId: this.sessionId } : {}), + ...(contextUsage !== undefined ? { contextUsage } : {}) } this.savedChats = { ...this.savedChats, [updatedChat.id]: updatedChat } - if (this.indexDB) { - await this.indexDB.put('chats', updatedChat) - } + const db = await this.dbh.whenReady() + if (db) await db.put('chats', updatedChat) } } - async save(displayMessages: DisplayMessage[], messages: ChatCompletionMessageParam[]) { - await this.saveChat(displayMessages, messages) + async save( + displayMessages: DisplayMessage[], + messages: ChatCompletionMessageParam[], + contextUsage?: number + ) { + await this.saveChat(displayMessages, messages, contextUsage) this.currentChatId = createLongHash() } @@ -142,7 +228,7 @@ export default class HistoryManager { this.savedChats = Object.fromEntries( Object.entries(this.savedChats).filter(([key]) => key !== id) ) - this.indexDB?.delete('chats', id) + void this.dbh.whenReady().then((db) => db?.delete('chats', id)) } loadPastChat(id: string) { diff --git a/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts b/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts new file mode 100644 index 0000000000..38fb7ecf0d --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/HistoryManager.test.ts @@ -0,0 +1,143 @@ +import { describe, it, expect, beforeEach, vi } from 'vitest' +import { IDBFactory } from 'fake-indexeddb' +import { openDB } from 'idb' + +// scopedKey resolves the email from userStore via a BROWSER-gated subscription. +vi.mock('esm-env', async (importOriginal) => ({ + ...(await importOriginal()), + BROWSER: true +})) + +import { userStore, type UserExt } from '$lib/stores' +import HistoryManager, { __resetLegacyChatClaimForTesting } from './HistoryManager.svelte' +import type { DisplayMessage } from './shared' +import type { ChatCompletionMessageParam } from 'openai/resources/index.mjs' + +function asUser(email: string): UserExt { + return { email, username: email.split('@')[0] } as unknown as UserExt +} + +type LegacyChat = { + id: string + actualMessages: unknown[] + displayMessages: unknown[] + title: string + lastModified: number + sessionId?: string +} + +async function seedLegacyChatDb(records: LegacyChat[]) { + const db = await openDB('copilot-chat-history', 1, { + upgrade(d) { + if (!d.objectStoreNames.contains('chats')) d.createObjectStore('chats', { keyPath: 'id' }) + } + }) + for (const r of records) await db.put('chats' as never, r as never) + db.close() +} + +beforeEach(() => { + ;(globalThis as any).indexedDB = new IDBFactory() + __resetLegacyChatClaimForTesting() + userStore.set(asUser('admin@test')) +}) + +async function countChats(dbName: string): Promise { + const db = await openDB(dbName) + if (!db.objectStoreNames.contains('chats')) { + db.close() + return 0 + } + const n = await db.count('chats' as never) + db.close() + return n +} + +describe('HistoryManager legacy chat-history migration', () => { + it('claims the legacy un-namespaced DB into the per-user DB, then deletes it', async () => { + await seedLegacyChatDb([ + { id: 'c1', actualMessages: [], displayMessages: [], title: 'Old chat', lastModified: 1 }, + { + id: 'c2', + actualMessages: [], + displayMessages: [], + title: 'Tagged chat', + lastModified: 2, + sessionId: 's9' + } + ]) + + const hm = new HistoryManager() + await hm.init() + + // Both session-tagged and untagged legacy chats are claimed. + expect( + hm + .getAllSavedChats() + .map((c) => c.id) + .sort() + ).toEqual(['c1', 'c2']) + + const names = (await indexedDB.databases()).map((d) => d.name) + expect(names).toContain('copilot-chat-history::admin@test') + // Bare legacy DB is gone so a later different user does not re-claim it. + expect(names).not.toContain('copilot-chat-history') + }) + + it('starts empty and does not throw when there is no legacy DB', async () => { + const hm = new HistoryManager() + await hm.init() + expect(hm.getAllSavedChats()).toEqual([]) + }) + + it('writes land in the current user DB after an in-place user switch', async () => { + const hm = new HistoryManager() + await hm.init() + + // Save under user A. + await hm.save( + [{ role: 'user', content: 'hello A' }] as DisplayMessage[], + [] as ChatCompletionMessageParam[] + ) + expect(await countChats('copilot-chat-history::admin@test')).toBe(1) + + // Switch identity in-place (no reload), then save again. The write must go + // to user B's DB, not A's stale handle. + userStore.set(asUser('other@test')) + await hm.save( + [{ role: 'user', content: 'hello B' }] as DisplayMessage[], + [] as ChatCompletionMessageParam[] + ) + + expect(await countChats('copilot-chat-history::other@test')).toBe(1) + // A's DB is untouched by the post-switch write. + expect(await countChats('copilot-chat-history::admin@test')).toBe(1) + }) +}) + +describe('HistoryManager title across compaction', () => { + it('keeps the original title once a summary boundary leads the transcript', async () => { + const hm = new HistoryManager() + await hm.init() + const id = hm.getCurrentChatId() + + // First save derives the title from the first user message. + await hm.save( + [{ role: 'user', content: 'original first question', index: 0 }] as DisplayMessage[], + [] as ChatCompletionMessageParam[] + ) + expect(hm.getAllSavedChats().find((c) => c.id === id)?.title).toBe('original first question') + + // After compaction the transcript leads with a summary boundary; deriving + // the title now would shift it to the surviving tail message. It must stay + // the title computed before compaction. + await hm.save( + [ + { role: 'summary', content: 'summary of the earlier conversation' }, + { role: 'user', content: 'a much later follow-up', index: 1 } + ] as DisplayMessage[], + [] as ChatCompletionMessageParam[] + ) + expect(hm.getAllSavedChats().find((c) => c.id === id)?.title).toBe('original first question') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/ProviderModelSelector.svelte b/frontend/src/lib/components/copilot/chat/ProviderModelSelector.svelte index efc14bd8b8..04449b1f11 100644 --- a/frontend/src/lib/components/copilot/chat/ProviderModelSelector.svelte +++ b/frontend/src/lib/components/copilot/chat/ProviderModelSelector.svelte @@ -58,11 +58,14 @@ selected: m.model === providerModel.model, action: () => { // Carry the effort onto the new model only if it supports that level - // ('off' is always valid); otherwise drop it so the model's default applies. + // ('off' only where the model can truly disable); otherwise drop it so + // the model's default applies. const carried = providerModel.reasoning const cap = getReasoningCapability(m.provider, m.model) const keep = - carried === REASONING_OFF || (carried !== undefined && cap.levels.includes(carried)) + carried === REASONING_OFF + ? cap.canDisable + : carried !== undefined && cap.levels.includes(carried) $copilotSessionModel = { ...m, ...(keep ? { reasoning: carried } : {}) } storeLocalSetting(COPILOT_SESSION_MODEL_SETTING_NAME, m.model) storeLocalSetting(COPILOT_SESSION_PROVIDER_SETTING_NAME, m.provider) @@ -93,7 +96,7 @@ {#if capability.supported} - [REASONING_OFF, ...capability.levels].map((level) => ({ + [...(capability.canDisable ? [REASONING_OFF] : []), ...capability.levels].map((level) => ({ displayName: level, selected: level === REASONING_OFF diff --git a/frontend/src/lib/components/copilot/chat/QueuedMessageChip.svelte b/frontend/src/lib/components/copilot/chat/QueuedMessageChip.svelte new file mode 100644 index 0000000000..3d40225360 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/QueuedMessageChip.svelte @@ -0,0 +1,33 @@ + + +{#if aiChatManager.queuedMessage} +
+
+

+ {aiChatManager.queuedMessage} +

+
+
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/anthropic.ts b/frontend/src/lib/components/copilot/chat/anthropic.ts index 753c0b04d1..48b068039f 100644 --- a/frontend/src/lib/components/copilot/chat/anthropic.ts +++ b/frontend/src/lib/components/copilot/chat/anthropic.ts @@ -78,16 +78,15 @@ export async function getAnthropicCompletion( const client = options?.anthropicClient ?? workspaceAIClients.getAnthropicClient() - // Adds output_config.effort + adaptive thinking (and strips temperature) when an - // effort is set; no-op otherwise. Returns the base shape unchanged when off. + // Adds output_config.effort + adaptive thinking when an effort is set; + // no-op otherwise. Returns the base shape unchanged when off. const anthropicParams = applyReasoningToConfig( { model: config.model, max_tokens: config.max_tokens as number, messages: anthropicMessages, ...(system && { system }), - ...(anthropicTools && { tools: anthropicTools }), - ...(typeof config.temperature === 'number' && { temperature: config.temperature }) + ...(anthropicTools && { tools: anthropicTools }) }, 'anthropic', options?.reasoningEffort diff --git a/frontend/src/lib/components/copilot/chat/api/core.ts b/frontend/src/lib/components/copilot/chat/api/core.ts index 4e47baea72..e4c2ab07c7 100644 --- a/frontend/src/lib/components/copilot/chat/api/core.ts +++ b/frontend/src/lib/components/copilot/chat/api/core.ts @@ -4,7 +4,6 @@ import type { } from 'openai/resources/index.mjs' import type { Tool } from '../shared' import { loadApiTools } from './apiTools' -import { getDocumentationTool } from '../navigator/core' import { userStore } from '$lib/stores' import { get } from 'svelte/store' @@ -14,13 +13,13 @@ You are Windmill's intelligent assistant, designed to interact with the platform Windmill is an open-source developer platform for building internal tools, API integrations, background jobs, workflows, and user interfaces. It offers a unified system where scripts are automatically turned into sharable UIs and can be composed into flows or embedded in custom applications. You have access to these tools: -1. Get documentation for user requests (get_documentation) +1. Search the documentation (search_docs) and read a documentation page (read_docs_page) 2. A comprehensive list of API endpoints to interact with the Windmill backend INSTRUCTIONS: - You can directly query, list, create, update, and delete various Windmill resources like scripts, flows, jobs, resources, variables, schedules, and workers through the provided API tools. - When users ask about specific data or want to perform operations, use the appropriate API endpoints to fulfill their requests. -- Use get_documentation to retrieve accurate information about features, concepts, and best practices when needed. +- Use search_docs (then read_docs_page on a returned Source URL) to retrieve accurate information about features, concepts, and best practices when needed. - Always present API results in a clear, readable format for the user. - If you need to make multiple related API calls to fulfill a request, do so systematically and explain what you're doing. - When showing lists of items, provide meaningful summaries rather than overwhelming the user with raw data. @@ -55,8 +54,6 @@ export async function getApiTools(): Promise[]> { return apiToolsCache } -export const apiTools: Tool<{}>[] = [getDocumentationTool] - export function prepareApiSystemMessage(customPrompt?: string): ChatCompletionSystemMessageParam { let content = CHAT_SYSTEM_PROMPT(get(userStore)?.username ?? '') diff --git a/frontend/src/lib/components/copilot/chat/ask/core.ts b/frontend/src/lib/components/copilot/chat/ask/core.ts index f9ba219599..36614eecb9 100644 --- a/frontend/src/lib/components/copilot/chat/ask/core.ts +++ b/frontend/src/lib/components/copilot/chat/ask/core.ts @@ -3,19 +3,23 @@ import type { ChatCompletionUserMessageParam } from 'openai/resources/index.mjs' import type { Tool } from '../shared' -import { getDocumentationTool } from '../navigator/core' +import { readDocsPageTool, searchDocsTool } from '../docs/core' export const CHAT_SYSTEM_PROMPT = ` You are Windmill's intelligent assistant, designed to answer questions about its functionality. It is your only purpose to help the user in the context of the windmill application. Windmill is an open-source developer platform for building internal tools, API integrations, background jobs, workflows, and user interfaces. It offers a unified system where scripts are automatically turned into sharable UIs and can be composed into flows or embedded in custom applications. You have access to these tools: -1. Get documentation for user requests (get_documentation) +1. Search the documentation (search_docs) +2. Read a documentation page (read_docs_page) INSTRUCTIONS: -- When user asks about something, use the get_documentation tool to retrieve accurate information about how to fulfill the user's request. -- Complete your response with precisions about how it works based on the documentation. Also drop a link to the relevant documentation if possible. -- If the user asks about something that you are unsure about, say that you are not sure about the answer and suggest to ask the question to the windmill team. +- Call search_docs FIRST with a few distinctive keywords from the user's question to find the most relevant documentation pages and matching snippets. +- If the snippets already answer the question, answer directly. Otherwise call read_docs_page with one of the returned Source URLs as its \`url\` argument to read the full page; if read_docs_page returns a list of section headings, call it again with the same \`url\` argument and a \`section\` argument to read the relevant section. +- If the first search returns nothing useful, retry with different or broader keywords before giving up. +- Answer based ONLY on what you find in the documentation. Do not invent features, flags, syntax, or behavior that you did not see in the docs. +- Always include the documentation URL(s) you consulted in your answer. Cite the exact "Source" URL shown in the search results (or the "Source page" URL at the top of a read page) — never reconstruct a URL from a link inside the page body. +- If the documentation does not cover the user's question, say so clearly rather than inventing an answer, and suggest asking the Windmill team. GENERAL PRINCIPLES: - Be concise but thorough @@ -23,7 +27,7 @@ GENERAL PRINCIPLES: - If you encounter an error or can't complete a request, explain why and suggest alternatives ` -export const askTools: Tool<{}>[] = [getDocumentationTool] +export const askTools: Tool<{}>[] = [searchDocsTool, readDocsPageTool] export function prepareAskSystemMessage(customPrompt?: string): ChatCompletionSystemMessageParam { let content = CHAT_SYSTEM_PROMPT diff --git a/frontend/src/lib/components/copilot/chat/chatLoop.test.ts b/frontend/src/lib/components/copilot/chat/chatLoop.test.ts index af619b93bf..9c1863ae54 100644 --- a/frontend/src/lib/components/copilot/chat/chatLoop.test.ts +++ b/frontend/src/lib/components/copilot/chat/chatLoop.test.ts @@ -1,7 +1,7 @@ import { beforeEach, describe, expect, it, vi } from 'vitest' import { randomUUID } from '$lib/utils/uuid' import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs' -import { runChatLoop, type ChatLoopConfig } from './chatLoop' +import { runChatLoop, truncateToToolPairedPrefix, type ChatLoopConfig } from './chatLoop' import type { ReasoningProviderModel } from '../reasoningRegistry' const mocks = vi.hoisted(() => ({ @@ -12,7 +12,7 @@ const mocks = vi.hoisted(() => ({ parseOpenAIResponsesCompletion: vi.fn(), getAnthropicCompletion: vi.fn(), parseAnthropicCompletion: vi.fn(), - resolveEffectiveReasoning: vi.fn() + resolveRequestReasoning: vi.fn() })) vi.mock('../lib', () => ({ @@ -22,7 +22,7 @@ vi.mock('../lib', () => ({ })) vi.mock('../reasoningRegistry', () => ({ - resolveEffectiveReasoning: mocks.resolveEffectiveReasoning + resolveRequestReasoning: mocks.resolveRequestReasoning })) vi.mock('./openai-responses', () => ({ @@ -83,7 +83,7 @@ describe('runChatLoop web search fallback', () => { mocks.providerSupportsWebSearch.mockImplementation( (provider) => provider === 'openai' || provider === 'anthropic' ) - mocks.resolveEffectiveReasoning.mockReturnValue(undefined) + mocks.resolveRequestReasoning.mockReturnValue(undefined) mocks.parseOpenAICompletion.mockResolvedValue({ shouldContinue: false, tokenUsage @@ -255,9 +255,7 @@ describe('runChatLoop web search fallback', () => { ) .mockResolvedValue({}) - await runChatLoop( - createConfig({ workspace, callbacks, modelProvider, onWebSearchUnavailable }) - ) + await runChatLoop(createConfig({ workspace, callbacks, modelProvider, onWebSearchUnavailable })) expect(mocks.getAnthropicCompletion).toHaveBeenCalledTimes(2) expect(mocks.getAnthropicCompletion.mock.calls[0][3]).toEqual( @@ -292,3 +290,112 @@ describe('runChatLoop web search fallback', () => { expect(callbacks.setToolStatus).not.toHaveBeenCalled() }) }) + +describe('runChatLoop lastIterationUsage', () => { + beforeEach(() => { + vi.resetAllMocks() + mocks.resolveRequestReasoning.mockReturnValue(undefined) + }) + + it('keeps the usage of the last completion that reported it', async () => { + const workspace = `workspace-${randomUUID()}` + mocks.getOpenAIResponsesCompletion.mockResolvedValue({}) + mocks.parseOpenAIResponsesCompletion + .mockResolvedValueOnce({ + shouldContinue: true, + tokenUsage: { prompt: 1000, completion: 50, total: 1050 } + }) + .mockResolvedValueOnce({ + shouldContinue: false, + tokenUsage: { prompt: 1200, completion: 80, total: 1280 } + }) + + const result = await runChatLoop({ ...createConfig({ workspace }), maxIterations: 2 }) + + expect(result.lastIterationUsage).toEqual({ prompt: 1200, completion: 80, total: 1280 }) + // the aggregate keeps summing across iterations + expect(result.tokenUsage).toEqual({ prompt: 2200, completion: 130, total: 2330 }) + }) + + it('ignores empty usage reports and returns null when none are real', async () => { + const workspace = `workspace-${randomUUID()}` + mocks.getOpenAIResponsesCompletion.mockResolvedValue({}) + mocks.parseOpenAIResponsesCompletion.mockResolvedValue({ + shouldContinue: false, + tokenUsage: { prompt: 0, completion: 0, total: 0 } + }) + + const result = await runChatLoop(createConfig({ workspace })) + + expect(result.lastIterationUsage).toBeNull() + }) +}) + +// Builders for the message shapes the chat loop accumulates. +const assistant = (content: string): ChatCompletionMessageParam => ({ role: 'assistant', content }) +const assistantTools = (...ids: string[]): ChatCompletionMessageParam => ({ + role: 'assistant', + content: '', + tool_calls: ids.map((id) => ({ + id, + type: 'function', + function: { name: 'do_thing', arguments: '{}' } + })) +}) +const tool = (id: string): ChatCompletionMessageParam => ({ + role: 'tool', + tool_call_id: id, + content: 'result' +}) +const user = (content: string): ChatCompletionMessageParam => ({ role: 'user', content }) + +describe('truncateToToolPairedPrefix', () => { + it('returns an empty array unchanged', () => { + expect(truncateToToolPairedPrefix([])).toEqual([]) + }) + + it('drops a trailing dangling tool_call (aborted before the result)', () => { + const msgs = [assistantTools('a')] + expect(truncateToToolPairedPrefix(msgs)).toEqual([]) + }) + + it('drops a partially-answered batch entirely (A answered, B missing)', () => { + const msgs = [assistantTools('a', 'b'), tool('a')] + expect(truncateToToolPairedPrefix(msgs)).toEqual([]) + }) + + it('keeps text + a completed round-trip, then drops the dangling tail', () => { + const msgs = [ + assistant('let me check'), + assistantTools('a'), + tool('a'), + assistant('more'), + assistantTools('b') // dangling + ] + expect(truncateToToolPairedPrefix(msgs)).toEqual([ + assistant('let me check'), + assistantTools('a'), + tool('a'), + assistant('more') + ]) + }) + + it('treats a user message as a valid boundary only when no tool calls are pending', () => { + const ok = [assistantTools('a'), tool('a'), user('next')] + expect(truncateToToolPairedPrefix(ok)).toEqual(ok) + + const dangling = [assistantTools('a'), user('next')] + expect(truncateToToolPairedPrefix(dangling)).toEqual([]) + }) + + it('leaves a valid full conversation unchanged (no loss on the normal path)', () => { + const msgs = [ + assistant('thinking'), + assistantTools('a', 'b'), + tool('a'), + tool('b'), + assistant('done') + ] + expect(truncateToToolPairedPrefix(msgs)).toEqual(msgs) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/chatLoop.ts b/frontend/src/lib/components/copilot/chat/chatLoop.ts index 46b8c881c0..0e5e4923d7 100644 --- a/frontend/src/lib/components/copilot/chat/chatLoop.ts +++ b/frontend/src/lib/components/copilot/chat/chatLoop.ts @@ -6,7 +6,7 @@ import type { ChatCompletionUserMessageParam } from 'openai/resources/chat/completions.mjs' import { getCompletion, parseOpenAICompletion, providerSupportsWebSearch } from '../lib' -import { resolveEffectiveReasoning, type ReasoningProviderModel } from '../reasoningRegistry' +import { resolveRequestReasoning, type ReasoningProviderModel } from '../reasoningRegistry' import { getAnthropicCompletion, parseAnthropicCompletion } from './anthropic' import { getOpenAIResponsesCompletion, parseOpenAIResponsesCompletion } from './openai-responses' import type { Tool, ToolCallbacks } from './shared' @@ -48,16 +48,60 @@ export interface ChatLoopConfig { onWebSearchUnavailable?: () => void /** Return a pending user message to inject between iterations, or undefined. */ getPendingUserMessage?: () => ChatCompletionUserMessageParam | undefined + /** + * Optional caller-owned accumulator for the messages produced this run — + * lets the caller recover partial output if the loop throws or is aborted. + */ + addedMessages?: ChatCompletionMessageParam[] /** Called before each iteration (e.g. to refresh tool schemas). */ onBeforeIteration?: (tools: Tool[], helpers: any) => Promise } export interface ChatLoopResult { addedMessages: ChatCompletionMessageParam[] + /** Sum of usage across all loop iterations (suitable for cost accounting). */ tokenUsage: ChatTokenUsage + lastIterationUsage: ChatTokenUsage | null hitMaxIterations: boolean } +/** + * Returns the longest prefix of `messages` that forms a valid request sequence: + * every assistant `tool_calls` batch must be fully answered by following tool + * messages before the next assistant turn. Used to commit the partial output of + * an aborted or failed turn as context for a follow-up, without leaving a + * dangling tool_call (which the provider APIs reject on the next request). + */ +export function truncateToToolPairedPrefix( + messages: ChatCompletionMessageParam[] +): ChatCompletionMessageParam[] { + let lastValidLen = 0 + let pending = new Set() + for (let i = 0; i < messages.length; i++) { + const m = messages[i] + if (m.role === 'assistant') { + // A new assistant turn while the previous tool batch is unanswered would + // be invalid — stop at the last known-good boundary. + if (pending.size > 0) break + const toolCalls = m.tool_calls ?? [] + if (toolCalls.length === 0) { + lastValidLen = i + 1 + } else { + pending = new Set(toolCalls.map((c) => c.id)) + } + } else if (m.role === 'tool') { + pending.delete(m.tool_call_id) + // Boundary is valid only once every tool_call in the batch is answered. + if (pending.size === 0) lastValidLen = i + 1 + } else { + // user/system message: a valid boundary only if no tool calls are pending. + if (pending.size > 0) break + lastValidLen = i + 1 + } + } + return messages.slice(0, lastValidLen) +} + const unsupportedWebSearchCache = new Set() const WEB_SEARCH_UNAVAILABLE_STATUS_CODES = new Set([400, 403, 404]) @@ -147,7 +191,11 @@ function shouldRetryWithoutWebSearch(err: unknown): boolean { return status === undefined || WEB_SEARCH_UNAVAILABLE_STATUS_CODES.has(status) } -function markWebSearchUnsupported(cacheKey: string, err: unknown, onWebSearchUnavailable?: () => void) { +function markWebSearchUnsupported( + cacheKey: string, + err: unknown, + onWebSearchUnavailable?: () => void +) { unsupportedWebSearchCache.add(cacheKey) console.warn('Native web search unavailable; retrying without web search:', err) onWebSearchUnavailable?.() @@ -167,11 +215,20 @@ export async function runChatLoop(config: ChatLoopConfig): Promise { + tokenUsage = addChatTokenUsage(tokenUsage, usage) + // Some providers/paths report no usage (prompt 0); keep the last real one. + if (usage && usage.prompt > 0) { + lastIterationUsage = usage + } + } + while (true) { if (maxIterations !== undefined && iterations >= maxIterations) { hitMaxIterations = true @@ -202,9 +259,10 @@ export async function runChatLoop(config: ChatLoopConfig): Promise t.def) - const parseOptions = { workspace } + const parseOptions = { workspace, provider: modelProvider.provider } if (isOpenAI) { const runOpenAIResponses = async (useWebSearch: boolean): Promise => { @@ -236,7 +294,7 @@ export async function runChatLoop(config: ChatLoopConfig): Promise => { - const completion = await getAnthropicCompletion( - messageParams, - abortController, - toolDefs, - { - forceModelProvider: modelProvider, - anthropicClient: clients.anthropic, - webSearch: useWebSearch, - reasoningEffort - } - ) + const completion = await getAnthropicCompletion(messageParams, abortController, toolDefs, { + forceModelProvider: modelProvider, + anthropicClient: clients.anthropic, + webSearch: useWebSearch, + reasoningEffort + }) if (!completion) { return true } @@ -326,7 +379,7 @@ export async function runChatLoop(config: ChatLoopConfig): Promise { + it('strips the analysis scratchpad and unwraps the summary block', () => { + const raw = ` +chronological thinking the model should not keep + + +1. Primary Request and Intent: build the thing +2. Pending Tasks: none +` + const formatted = formatCompactSummary(raw) + expect(formatted).not.toContain('chronological thinking') + expect(formatted).not.toContain('') + expect(formatted).not.toContain('') + expect(formatted).toContain('Primary Request and Intent: build the thing') + }) + + it('falls back to the trimmed raw text when the model omits the tags', () => { + expect(formatCompactSummary(' just a plain summary ')).toBe('just a plain summary') + }) + + it('keeps the summary even when there is no analysis block', () => { + expect(formatCompactSummary('only the summary')).toBe('only the summary') + }) + + it('collapses the blank-line runs left by stripping analysis', () => { + const raw = 'x\n\n\n\na\n\n\n\nb' + expect(formatCompactSummary(raw)).toBe('a\n\nb') + }) + + it('keeps the summary content when has no closing tag', () => { + const raw = '\n1. Primary Request and Intent: build the thing\n2. Pending Tasks: none' + const formatted = formatCompactSummary(raw) + expect(formatted).not.toContain('') + expect(formatted).toContain('Primary Request and Intent: build the thing') + expect(formatted).toContain('Pending Tasks: none') + }) + + it('drops the analysis scratchpad even when is left unclosed', () => { + const raw = + '\nchronological thinking the model should not keep\n\n\nthe real summary' + const formatted = formatCompactSummary(raw) + expect(formatted).not.toContain('chronological thinking') + expect(formatted).not.toContain('') + expect(formatted).not.toContain('') + expect(formatted).toBe('the real summary') + }) + + it('strips an orphaned closing summary tag', () => { + expect(formatCompactSummary('plain summary')).toBe('plain summary') + }) + + it('does not leak analysis scratchpad that mentions a literal tag', () => { + const raw = `scratchpad mentions before output +real summary` + const formatted = formatCompactSummary(raw) + expect(formatted).toBe('real summary') + expect(formatted).not.toContain('scratchpad') + expect(formatted).not.toContain('before output') + }) + + it('strips every analysis block, not just the first, when the summary is untagged', () => { + const raw = 'first\nkept one\nsecond\nkept two' + const formatted = formatCompactSummary(raw) + expect(formatted).not.toContain('first') + expect(formatted).not.toContain('second') + expect(formatted).not.toContain('') + expect(formatted).toContain('kept one') + expect(formatted).toContain('kept two') + }) +}) + +describe('buildSummaryMessageContent', () => { + it('embeds the summary and frames it as a continuation', () => { + const content = buildSummaryMessageContent('THE SUMMARY') + expect(content).toContain('THE SUMMARY') + expect(content).toContain('continued from a previous conversation') + expect(content).toContain('preserved verbatim') + }) +}) + +describe('getCompactionSummaryPrompt', () => { + it('asks for a structured, text-only summary', () => { + const prompt = getCompactionSummaryPrompt() + expect(prompt).toContain('detailed summary') + expect(prompt).toContain('') + expect(prompt).toContain('TEXT ONLY') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/compactionPrompt.ts b/frontend/src/lib/components/copilot/chat/compactionPrompt.ts new file mode 100644 index 0000000000..f785682dac --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/compactionPrompt.ts @@ -0,0 +1,138 @@ +// Summary-based compaction: when a conversation approaches the model's context +// window, the older prefix is replaced by an LLM-generated structured summary +// while the recent tail is kept verbatim. The summary precedes the kept tail, +// so it is written "up to" the point of compaction — newer messages the model +// does not see here will follow it. +// +// Prompt structure and the analysis/summary split are adapted from the +// reference compaction prompt used by coding agents. + +// Reinforce text-only output. The summary call is issued without tools, but a +// strong instruction keeps weaker models from narrating a tool call instead of +// producing the summary. +const NO_TOOLS_PREAMBLE = `CRITICAL: Respond with TEXT ONLY. Do NOT call any tools. + +- You already have all the context you need in the conversation above. +- Your entire response must be plain text: an block followed by a block. + +` + +const NO_TOOLS_TRAILER = + '\n\nREMINDER: Respond with plain text only — an block followed by a block.' + +// The block is a drafting scratchpad that formatCompactSummary() +// strips before the summary reaches context. +const SUMMARY_PROMPT = `Your task is to create a detailed summary of the conversation so far. This summary will be placed at the start of a continuing session; newer messages that build on this context will follow after it (you do not see them here). Summarize thoroughly so that someone reading only your summary and then the newer messages can fully understand what happened and continue the work without losing context. + +This is a conversation with Windmill's global workspace assistant. It inspects workspace items and authors them as per-user drafts — scripts, flows, apps, resources, variables, triggers, and schedules — then deploys those drafts and test-runs scripts and flows. It works with items by their workspace path (e.g. \`u/alice/sync_orders\`, \`f/team/my_flow\`); it does NOT edit files on a filesystem. Frame the summary in those terms. + +Before providing your final summary, wrap your analysis in tags to organize your thoughts. In your analysis: + +1. Chronologically analyze each message and section of the conversation. For each section thoroughly identify: + - The user's explicit requests and intents + - Your approach to addressing the user's requests + - Key decisions, technical concepts and code patterns + - Specific details: workspace item paths and kinds (script / flow / app / resource / variable / trigger / schedule), code snippets for runnables, and the exact actions taken (drafts created or updated, items deployed, test runs and their results) + - Errors you ran into and how you fixed them + - Specific user feedback, especially if the user told you to do something differently +2. Double-check for technical accuracy and completeness. + +Your summary should include the following sections: + +1. Primary Request and Intent: Capture all of the user's explicit requests and intents in detail. +2. Key Technical Concepts: List important technical concepts, technologies, and frameworks discussed. +3. Workspace Items and Code: Enumerate the workspace items inspected, drafted, updated, deployed, or test-run — each by its path and kind (script / flow / app / resource / variable / trigger / schedule) — noting what was done and why. Include full code snippets for runnables (script code, flow inline scripts, app inline runnables) wherever code was written or changed. +4. Errors and fixes: List errors encountered and how they were fixed, including any user feedback. +5. Problem Solving: Document problems solved and any ongoing troubleshooting efforts. +6. All user messages: List ALL user messages that are not tool results. These are critical for understanding the user's feedback and changing intent. +7. Pending Tasks: Outline any pending tasks you have explicitly been asked to work on. +8. Current Work: Describe precisely what was being worked on immediately before this summary, paying special attention to the most recent messages. Include item paths and code snippets where applicable. +9. Context for Continuing Work: Summarize any context, decisions, or state needed to understand and continue the work in subsequent messages — including which items are still drafts versus deployed and the exact paths involved. If there is a clear next step directly in line with the user's most recent explicit request, state it and include a direct quote from the most recent conversation showing where you left off. + +Structure your output like this: + + +[Your thought process, ensuring all points are covered thoroughly and accurately] + + + +1. Primary Request and Intent: + [Detailed description] + +2. Key Technical Concepts: + - [Concept] + +3. Workspace Items and Code: + - [path + kind, e.g. u/alice/sync_orders (script)] + - [What was done: read / draft created / draft updated / deployed / test-run result] + - [Why it matters] + - [Code snippet, for runnables] + +4. Errors and fixes: + - [Error]: [How you fixed it] + +5. Problem Solving: + [Description] + +6. All user messages: + - [Non-tool-result user message] + +7. Pending Tasks: + - [Task] + +8. Current Work: + [Precise description of current work] + +9. Context for Continuing Work: + [Key context, decisions, or state needed to continue] + + +Please provide your summary following this structure, ensuring precision and thoroughness.` + +/** Prompt sent as the final user message of the summarization request. */ +export function getCompactionSummaryPrompt(): string { + return NO_TOOLS_PREAMBLE + SUMMARY_PROMPT + NO_TOOLS_TRAILER +} + +/** + * Strips the drafting scratchpad and unwraps the block. + * Falls back to the trimmed raw text when the model didn't use the tags, so a + * well-formed-but-untagged summary is still usable. + */ +export function formatCompactSummary(raw: string): string { + // Strip the analysis scratchpad first: it precedes the summary and may itself + // mention / tokens that would otherwise be mistaken for the + // real summary boundary. + let formatted = raw.replace(/[\s\S]*?<\/analysis>/gi, '') + + const summaryMatch = formatted.match(/([\s\S]*?)<\/summary>/i) + if (summaryMatch) { + formatted = (summaryMatch[1] ?? '').trim() + } else { + // A truncated response or a weaker model sometimes opens without + // closing it. The text after the opener is still the summary, so keep it + // rather than leak the bare tag. + const openIdx = formatted.search(//i) + if (openIdx !== -1) { + formatted = formatted.slice(openIdx) + } + } + + // An orphaned opener or closer left by either branch must never reach the user. + formatted = formatted.replace(/<\/?(?:analysis|summary)>/gi, '') + + // Collapse the blank-line runs left behind by stripping the analysis block. + return formatted.replace(/\n{3,}/g, '\n\n').trim() +} + +/** + * Wraps a formatted summary as the content of the user message that replaces + * the summarized prefix in the conversation. + */ +export function buildSummaryMessageContent(formattedSummary: string): string { + return `This session is being continued from a previous conversation that ran out of context. The summary below covers the earlier portion of the conversation. Recent messages after the summary are preserved verbatim. + +${formattedSummary} + +Continue the conversation from where it left off. Do not re-introduce the summary or recap it to the user; pick up the work as if the break never happened.` +} diff --git a/frontend/src/lib/components/copilot/chat/context.ts b/frontend/src/lib/components/copilot/chat/context.ts index b2ab9e5f1c..cf41617f41 100644 --- a/frontend/src/lib/components/copilot/chat/context.ts +++ b/frontend/src/lib/components/copilot/chat/context.ts @@ -6,7 +6,8 @@ import { FileCode, Code2, TextSelect, - Table2 + Table2, + LayoutDashboard } from 'lucide-svelte' import BarsStaggered from '$lib/components/icons/BarsStaggered.svelte' import type { ScriptLang } from '$lib/gen/types.gen' @@ -25,7 +26,8 @@ export const ContextIconMap = { app_code_selection: TextSelect, app_datatable: Table2, workspace_script: Code2, - workspace_flow: BarsStaggered + workspace_flow: BarsStaggered, + workspace_app: LayoutDashboard // flow_module type is handled with FlowModuleIcon } @@ -228,6 +230,16 @@ export interface WorkspaceFlowElement { summary?: string } +/** Workspace app context element — reference to a (code-based) raw app in the + * workspace. Only raw apps are surfaced: the visual app builder produces a + * frontend, not a code unit the chat can act on. */ +export interface WorkspaceAppElement { + type: 'workspace_app' + path: string + title: string + summary?: string +} + export type ContextElement = ( | CodeElement | ErrorElement @@ -242,6 +254,7 @@ export type ContextElement = ( | AppDatatableElement | WorkspaceScriptElement | WorkspaceFlowElement + | WorkspaceAppElement ) & { deletable?: boolean } diff --git a/frontend/src/lib/components/copilot/chat/docs/core.ts b/frontend/src/lib/components/copilot/chat/docs/core.ts new file mode 100644 index 0000000000..38e9a600f0 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/docs/core.ts @@ -0,0 +1,113 @@ +import type { Tool } from '../shared' +import type { ChatCompletionTool } from 'openai/resources/index.mjs' +import { DocumentationService } from '$lib/gen' + +// The docs corpus is self-hosted by the backend (a vendored snapshot embedded in +// the binary). Search/ranking and page rendering happen server-side, so these +// tools are thin wrappers over /api/docs/* — see backend `windmill-api/src/docs`. +// The same endpoints back the MCP `searchDocs`/`readDocsPage` tools and the +// `wmill docs` CLI, so all three return identical content. + +const READ_DOCS_PAGE_TOOL: ChatCompletionTool = { + type: 'function', + function: { + name: 'read_docs_page', + description: + 'Fetch the raw markdown of a single Windmill documentation page. Provide the `url` of a page found via search_docs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section.', + parameters: { + type: 'object', + properties: { + url: { + type: 'string', + description: + 'The docs page to read, as a Source URL returned by search_docs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted.' + }, + section: { + type: 'string', + description: + 'Optional. A heading title from the page outline to read just that section instead of the full page.' + } + }, + required: ['url'] + } + } +} + +export const readDocsPageTool: Tool<{}> = { + def: READ_DOCS_PAGE_TOOL, + fn: async ({ args, toolId, toolCallbacks }) => { + const url = typeof args?.url === 'string' ? args.url : '' + const section = + typeof args?.section === 'string' && args.section.trim() ? args.section : undefined + toolCallbacks.setToolStatus(toolId, { + content: section ? `Reading docs section "${section}"...` : 'Reading documentation page...' + }) + try { + if (!url.trim()) { + return 'No documentation page URL was provided. Provide a `url` — e.g. a `Source` URL returned by search_docs.' + } + const res = await DocumentationService.readDocsPage({ url, section }) + toolCallbacks.setToolStatus(toolId, { content: 'Read documentation page' }) + return res.text + } catch (error) { + toolCallbacks.setToolStatus(toolId, { + content: 'Error reading documentation page', + error: 'Error reading documentation page' + }) + console.error('Error reading documentation page:', error) + const errorMessage = + error instanceof Error ? error.message : 'An error occurred while reading the documentation page' + return `Failed to read documentation page: ${errorMessage}, pursuing with the user request...` + } + } +} + +const SEARCH_DOCS_TOOL: ChatCompletionTool = { + type: 'function', + function: { + name: 'search_docs', + description: + 'Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call read_docs_page with a returned Source URL to read the full page or a section.', + parameters: { + type: 'object', + properties: { + query: { + type: 'string', + description: + 'Keywords to search for in the documentation body, e.g. "chromium worker tag" or "retry exponential backoff". Fewer, more distinctive words match better.' + } + }, + required: ['query'] + } + } +} + +export const searchDocsTool: Tool<{}> = { + def: SEARCH_DOCS_TOOL, + fn: async ({ args, toolId, toolCallbacks }) => { + const query = typeof args?.query === 'string' ? args.query.trim() : '' + toolCallbacks.setToolStatus(toolId, { + content: query ? `Searching documentation for "${query}"...` : 'Searching documentation...' + }) + try { + if (!query) { + return 'No search query was provided. Provide a `query` of one or more keywords.' + } + const res = await DocumentationService.searchDocs({ query }) + const count = res.results?.length ?? 0 + toolCallbacks.setToolStatus(toolId, { + content: count > 0 ? `Found ${count} matching page(s)` : 'No matching pages found' + }) + return res.text + } catch (error) { + toolCallbacks.setToolStatus(toolId, { + content: 'Error searching documentation', + error: 'Error searching documentation' + }) + console.error('Error searching documentation:', error) + const errorMessage = + error instanceof Error ? error.message : 'An error occurred while searching the documentation' + return `Failed to search documentation: ${errorMessage}, pursuing with the user request...` + } + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFileChip.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFileChip.svelte new file mode 100644 index 0000000000..52fcfd8b5a --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFileChip.svelte @@ -0,0 +1,61 @@ + + +
(showDelete = true)} + onmouseleave={() => (showDelete = false)} + role="listitem" + title={`${file.name} — ${detail}`} +> + + {file.name} +
diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte new file mode 100644 index 0000000000..50b961c090 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFilesBar.svelte @@ -0,0 +1,86 @@ + + +{#snippet chip(card: Card)} + {#if card.kind === 'folder'} + removeCard(card)} /> + {:else} + removeCard(card)} /> + {/if} +{/snippet} + +{#if cards.length > 0} +
+ {#each visible as card (card.key)} + {@render chip(card)} + {/each} + + {#if overflow.length > 0} + + {#snippet trigger()} +
+ +{overflow.length} +
+ {/snippet} + {#snippet content()} +
+ {#each overflow as card (card.key)} + {@render chip(card)} + {/each} +
+ {/snippet} +
+ {/if} + + {#if lockedCount > 0} + + {/if} +
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte new file mode 100644 index 0000000000..068447a878 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/AttachedFolderChip.svelte @@ -0,0 +1,48 @@ + + +
(showDelete = true)} + onmouseleave={() => (showDelete = false)} + role="listitem" + title={hoverList} +> + + {folder.name} +
diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts new file mode 100644 index 0000000000..124b390ec2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.svelte.ts @@ -0,0 +1,602 @@ +/** + * Session-scoped store of files/folders the user has linked to the GLOBAL AI chat. + * + * Persistence model (survives reload, keyed by session in ./attachedFilesDB): + * - FILES are always stored as a full-byte Blob snapshot — same on every browser, + * no permission re-grant, never "locked". + * - FOLDERS link as a live File System Access directory handle where the API exists + * (one record, re-enumerated live on restore — folder files are read through the + * handle, not copied). Where it doesn't (Firefox/Safari), a dropped/picked folder's + * files are snapshotted individually (each carrying its `folder`/`relPath`) so they + * regroup into the same folder chip on restore. + * + * Storage is bounded by the real browser quota (writes that exceed it are caught and + * the item simply isn't persisted — it stays usable for the session). Persistence is + * gated on the session being persisted (non-transient); links in a transient session + * are buffered and flushed on the first send. + */ +import { createLongHash } from '$lib/editorLangUtils' +import { buildLineIndex, isTextFile, type FileEntry } from './fileEngine' +import { + putItem, + deleteItem, + getItemsForSession, + ensurePersistentStorage, + type PersistedAttachedItem +} from './attachedFilesDB' +import { enumerateDir, isIgnoredPath, queryReadPermission, requestReadPermission } from './fsAccess' + +export type AttachedFileStatus = 'indexing' | 'ready' | 'error' | 'locked' | 'unavailable' + +export interface AttachedFile extends FileEntry { + size: number + status: AttachedFileStatus + error?: string + /** Top-level folder this file came from (first path segment), if part of a folder. */ + folder?: string + /** Persisted source-record id. Folder children share the folder's record id. */ + sourceId: string + /** Parent directory handle (folder children only) — used to re-grant / re-enumerate. */ + handle?: FileSystemDirectoryHandle + /** Relative path within the folder (folder children only) — stable key for refresh diffing. */ + relPath?: string + /** + * Internal: a single placeholder row standing in for a not-yet-expanded folder + * (locked/unavailable). Consumers should read `store.folders` instead of testing this. + */ + isFolderRoot?: boolean +} + +/** A linked folder as a first-class object — consumers read this instead of re-grouping rows. */ +export interface AttachedFolder { + name: string + /** Aggregate status (locked > unavailable > indexing > error > ready). */ + status: AttachedFileStatus + /** Child files; empty while the folder is locked/unavailable after a reload. */ + files: AttachedFile[] +} + +/** Aggregate a folder's rows (children + a possible placeholder) into one status. */ +function folderStatus(rows: AttachedFile[]): AttachedFileStatus { + for (const status of ['locked', 'unavailable', 'indexing', 'error'] as const) { + if (rows.some((f) => f.status === status)) return status + } + return 'ready' +} + +export interface AddFilesResult { + added: string[] + rejected: { name: string; reason: string }[] +} + +/** A file to link: a raw File, or `{ file, path? }` (path = relative display name). */ +export type FileToAttach = File | { file: File; path?: string } + +const EMPTY = new Blob([]) + +export class AttachedFilesStore { + files = $state([]) + + /** Session context, set by the runtime; persistence writes are gated on `#persisted`. */ + sessionId: string | undefined = undefined + #persisted = false + /** Records buffered while the session is transient (flushed on first send). */ + #pending: PersistedAttachedItem[] = [] + + list(): AttachedFile[] { + return this.files + } + get(name: string): AttachedFile | undefined { + // Resolve to a real file — a folder-root placeholder may share the folder's name. + return this.files.find((f) => f.name === name && !f.isFolderRoot) + } + readyFiles(): AttachedFile[] { + // Folder-root placeholders aren't real files — never expose them to the read/search tools. + return this.files.filter((f) => f.status === 'ready' && !f.isFolderRoot) + } + get count(): number { + return this.files.length + } + + /** Linked folders, children grouped and status aggregated (placeholder rows hidden). */ + folders: AttachedFolder[] = $derived.by(() => { + const byName = new Map() + for (const f of this.files) { + if (!f.folder) continue + const rows = byName.get(f.folder) + if (rows) rows.push(f) + else byName.set(f.folder, [f]) + } + return [...byName.entries()].map(([name, rows]) => ({ + name, + status: folderStatus(rows), + files: rows.filter((f) => !f.isFolderRoot) + })) + }) + + /** Files linked on their own (not as part of a folder). */ + standalone: AttachedFile[] = $derived.by(() => this.files.filter((f) => !f.folder)) + + /** Number of locked folders needing a re-grant. */ + get lockedCount(): number { + return this.folders.filter((f) => f.status === 'locked').length + } + + clear(): void { + this.files = [] + this.#pending = [] + } + + removeFile(name: string): void { + // Target the real file only — never a folder-root placeholder that happens to share + // the name (those are managed via removeFolder), else removing a same-named standalone + // file would also drop the folder's placeholder. + const f = this.files.find((x) => x.name === name && !x.isFolderRoot) + if (!f) return + this.files = this.files.filter((x) => !(x.name === name && !x.isFolderRoot)) + void this.#deleteRecord(f.sourceId) + } + + /** Remove every file linked as part of the given folder (and its persisted record). */ + removeFolder(folder: string): void { + const ids = new Set(this.files.filter((f) => f.folder === folder).map((f) => f.sourceId)) + this.files = this.files.filter((f) => f.folder !== folder) + for (const id of ids) void this.#deleteRecord(id) + } + + // ---------------------------------------------------------------- linking + + /** + * Link individual files — always stored as a Blob snapshot. Items carrying a folder + * path (`folder/sub/file`, from a dropped/picked folder) are grouped into a folder and + * have their junk paths (node_modules/.git/dotfiles) skipped; a loose single file is + * kept as-is (so an explicitly attached `.env` isn't filtered out). + */ + async addFiles(input: FileList | FileToAttach[]): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + + for (const item of Array.from(input as ArrayLike)) { + const file = item instanceof File ? item : item.file + const desired = + (item instanceof File ? '' : (item.path ?? '')) || + (file as File & { webkitRelativePath?: string }).webkitRelativePath || + file.name || + 'file' + + const folder = desired.includes('/') ? desired.split('/')[0] : undefined + if (folder && isIgnoredPath(desired)) continue // skip junk inside folders + + if (this.#isDuplicate(desired, file)) continue // silent no-op on re-link + + const reason = await this.#preflight(file) + if (reason) { + result.rejected.push({ name: desired, reason }) + continue + } + + const name = this.#uniqueName(desired) + const relPath = folder ? desired : undefined + const sourceId = createLongHash() + + this.#pushIndexing({ name, file, folder, sourceId, relPath }) + result.added.push(name) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'snapshot', + name, + folder, + relPath, + blob: file, + size: file.size, + lastModified: file.lastModified, + addedAt: Date.now() + }) + } + + return result + } + + /** + * Link a folder via a live directory handle (File System Access path only). + * Enumerates the handle internally (junk-filtered, capped) — the same walk used + * on restore and refresh, so callers never pre-enumerate. + */ + async addFolder(dirHandle: FileSystemDirectoryHandle): Promise { + const result: AddFilesResult = { added: [], rejected: [] } + const folder = dirHandle.name + const existing = this.files.filter((f) => f.folder === folder) + if (existing.length > 0) { + const placeholder = existing.length === 1 ? existing.find((f) => f.isFolderRoot) : undefined + if (placeholder) { + // Re-picking a folder that sits locked/unavailable after a reload is a natural + // recovery gesture — replace the stale link with the freshly-granted handle. + this.files = this.files.filter((f) => f.sourceId !== placeholder.sourceId) + void this.#deleteRecord(placeholder.sourceId) + } else { + // Same basename, possibly a different directory — surface it instead of a silent no-op. + result.rejected.push({ name: folder, reason: 'A folder with this name is already linked' }) + return result + } + } + + const files = await enumerateDir(dirHandle) + const sourceId = createLongHash() + for (const { file, path } of files) { + if (!(await this.#sniffText(file))) { + result.rejected.push({ name: path, reason: 'Not a text file' }) + continue + } + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + result.added.push(name) + } + // Keep the folder represented even when it links empty (or all-binary): a placeholder + // carries the handle so the chip stays and refreshFolders picks up files added later. + // Persist unconditionally so an empty-at-link folder also survives a reload. + this.#ensureFolderRow(sourceId, folder, dirHandle) + void this.#persist({ + id: sourceId, + sessionId: this.sessionId ?? '', + kind: 'dir-handle', + name: folder, + folder, + handle: dirHandle, + addedAt: Date.now() + }) + return result + } + + // ------------------------------------------------------------- persistence + + /** Set session context and load any persisted items for it (called on activation). */ + async restore(sessionId: string, persisted: boolean): Promise { + this.sessionId = sessionId + this.#persisted = persisted + this.files = [] + this.#pending = [] + + const items = await getItemsForSession(sessionId) + for (const item of items) { + try { + if (item.kind === 'snapshot') { + if (!item.blob) { + this.#pushPlaceholder(item, 'unavailable') + continue + } + this.#pushIndexing({ + name: item.name, + file: item.blob, + folder: item.folder, + relPath: item.relPath, + sourceId: item.id + }) + } else { + // dir-handle (folder) + const handle = item.handle as FileSystemDirectoryHandle + if ((await queryReadPermission(handle)) === 'granted') { + await this.#expandFolder(handle, item.id) + } else { + this.#pushPlaceholder(item, 'locked', true) + } + } + } catch { + this.#pushPlaceholder(item, 'unavailable', item.kind === 'dir-handle') + } + } + } + + /** Re-grant any locked folder handles. MUST be called within a user gesture (e.g. on send). */ + async regrantLocked(): Promise { + const sources = new Map() + for (const f of this.files) { + if (f.status === 'locked' && f.handle) sources.set(f.sourceId, f) + } + if (sources.size === 0) return + + // Kick off all permission requests within the gesture, then process. A rejected + // request (requestReadPermission never rejects, but stay defensive) counts as denied. + const decided = await Promise.all( + [...sources.values()].map((f) => + requestReadPermission(f.handle!).then( + (perm) => ({ f, perm }), + () => ({ f, perm: 'denied' as PermissionState }) + ) + ) + ) + for (const { f, perm } of decided) { + if (perm !== 'granted') continue + try { + await this.#expandFolder(f.handle as FileSystemDirectoryHandle, f.sourceId) + // Children are in — drop the locked placeholder row, then restore a ready + // placeholder if the folder came back empty/all-binary (else dropping the only + // handle-bearing row would unlink the folder and stop it ever refreshing). + this.files = this.files.filter((x) => !(x.sourceId === f.sourceId && x.isFolderRoot)) + this.#ensureFolderRow(f.sourceId, f.folder ?? f.name, f.handle as FileSystemDirectoryHandle) + } catch { + // Enumeration failed (folder moved/deleted on disk): drop any partially-added + // children and keep the placeholder so the chip shows "unavailable". + this.files = this.files.filter((x) => x.sourceId !== f.sourceId || x.isFolderRoot) + this.#patchSource(f.sourceId, { status: 'unavailable' }) + } + } + } + + /** Flush buffered links once the session becomes persistent (first send). */ + async flushPending(): Promise { + this.#persisted = true + if (!this.sessionId) return + const pending = this.#pending + this.#pending = [] + if (pending.length === 0) return + void ensurePersistentStorage() + for (const item of pending) { + try { + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file', e) + } + } + } + + async #persist(item: PersistedAttachedItem): Promise { + if (this.#persisted && this.sessionId) { + void ensurePersistentStorage() + try { + // A QuotaExceededError just means it won't survive a reload — the item + // stays usable for this session. Swallow + log rather than fail the link. + await putItem({ ...item, sessionId: this.sessionId }) + } catch (e) { + console.error('Could not persist linked file (kept for this session)', e) + } + } else { + this.#pending.push(item) + } + } + + async #deleteRecord(sourceId: string): Promise { + this.#pending = this.#pending.filter((p) => p.id !== sourceId) + if (this.#persisted) { + try { + await deleteItem(sourceId) + } catch { + /* ignore */ + } + } + } + + // ------------------------------------------------------------- internals + + /** Identical re-link (same name, or same File identity) → silent no-op. */ + #isDuplicate(desired: string, file: File): boolean { + return this.files.some( + (f) => + // Folder-root placeholders aren't real files — they must not block attaching a + // standalone file that happens to share the folder's name. + !f.isFolderRoot && + (f.name === desired || + // Identical re-drop at the SAME relative path (its row name may have been + // auto-suffixed). Keyed on the path, NOT the basename — otherwise two distinct + // files sharing a basename under different folder subdirs (proj/a/index.ts vs + // proj/b/index.ts) would be wrongly deduped and silently dropped. + ((f.relPath ?? f.name) === desired && + f.size === file.size && + f.file instanceof File && + f.file.lastModified === file.lastModified)) + ) + } + + /** Returns a rejection reason, or undefined if the file may be linked. */ + async #preflight(file: File): Promise { + if (!(await this.#sniffText(file))) return 'Not a text file' + return undefined + } + + async #sniffText(file: Blob): Promise { + try { + return await isTextFile(file) + } catch { + return false + } + } + + #pushIndexing(p: { + name: string + file: File | Blob + folder?: string + sourceId: string + handle?: FileSystemDirectoryHandle + relPath?: string + }): void { + this.files = [ + ...this.files, + { + name: p.name, + file: p.file, + size: p.file.size, + lineIndex: [], + lineCount: 0, + status: 'indexing', + folder: p.folder, + sourceId: p.sourceId, + handle: p.handle, + relPath: p.relPath + } + ] + void this.#indexFile(p.name, p.file) + } + + #pushPlaceholder( + item: PersistedAttachedItem, + status: AttachedFileStatus, + isFolderRoot = false + ): void { + this.files = [ + ...this.files, + { + name: item.name, + file: EMPTY, + size: item.size ?? 0, + lineIndex: [], + lineCount: 0, + status, + folder: item.folder, + sourceId: item.id, + handle: item.handle as FileSystemDirectoryHandle | undefined, + isFolderRoot + } + ] + } + + async #expandFolder(dirHandle: FileSystemDirectoryHandle, sourceId: string): Promise { + const folder = dirHandle.name + const children = await enumerateDir(dirHandle) + for (const { file, path } of children) { + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle: dirHandle, relPath: path }) + } + this.#ensureFolderRow(sourceId, folder, dirHandle) + } + + /** + * Re-enumerate granted folder handles to reflect on-disk changes since they were + * linked/last refreshed: added/removed/renamed files and content edits. Called on + * each send so the AI sees the folder's current state. Unchanged files are left as-is + * (diffed by relative path + lastModified); only changed files are re-indexed. + */ + async refreshFolders(): Promise { + const sources = new Map() + for (const f of this.files) { + // Include folder-root placeholders (an emptied folder keeps only its placeholder), + // else the source is lost and the folder never re-enumerates again. + if (f.folder && f.handle) { + sources.set(f.sourceId, { handle: f.handle, folder: f.folder }) + } + } + for (const [sourceId, { handle, folder }] of sources) { + try { + if ((await queryReadPermission(handle)) !== 'granted') continue + const children = await enumerateDir(handle) + await this.#reconcileFolder(sourceId, folder, handle, children) + } catch { + this.#patchSource(sourceId, { status: 'unavailable' }) + } + } + } + + async #reconcileFolder( + sourceId: string, + folder: string, + handle: FileSystemDirectoryHandle, + children: { file: File; path: string }[] + ): Promise { + const existing = new Map() + for (const f of this.files) if (f.sourceId === sourceId && f.relPath) existing.set(f.relPath, f) + const seen = new Set() + + for (const { file, path } of children) { + seen.add(path) + const cur = existing.get(path) + if (!cur) { + // newly added on disk + if (!(await this.#sniffText(file))) continue + const name = this.#uniqueName(path) + this.#pushIndexing({ name, file, folder, sourceId, handle, relPath: path }) + } else { + const curMod = cur.file instanceof File ? cur.file.lastModified : undefined + if (file.size !== cur.size || file.lastModified !== curMod) { + // content changed → re-read + re-index + this.#patch(cur.name, { file, size: file.size, status: 'indexing' }) + void this.#indexFile(cur.name, file) + } + } + } + // removed/renamed-away on disk → drop from memory + const removed = [...existing.values()].filter((f) => f.relPath && !seen.has(f.relPath)) + if (removed.length > 0) { + const names = new Set(removed.map((f) => f.name)) + this.files = this.files.filter((f) => !names.has(f.name)) + } + this.#ensureFolderRow(sourceId, folder, handle) + } + + /** + * Keep a linked folder represented even with no readable children: leave one + * handle-carrying placeholder row so the chip stays visible AND `refreshFolders` + * keeps the live source (without it, an emptied folder vanishes and never + * re-enumerates). Drop the placeholder as soon as real children exist again. + */ + #ensureFolderRow(sourceId: string, folder: string, handle: FileSystemDirectoryHandle): void { + const hasChild = this.files.some((f) => f.sourceId === sourceId && !f.isFolderRoot) + const hasPlaceholder = this.files.some((f) => f.sourceId === sourceId && f.isFolderRoot) + if (!hasChild && !hasPlaceholder) { + this.files = [ + ...this.files, + { + name: folder, + file: EMPTY, + size: 0, + lineIndex: [], + lineCount: 0, + status: 'ready', + folder, + sourceId, + handle, + isFolderRoot: true + } + ] + } else if (hasChild && hasPlaceholder) { + this.files = this.files.filter((f) => !(f.sourceId === sourceId && f.isFolderRoot)) + } + } + + async #indexFile(name: string, file: File | Blob): Promise { + try { + const { lineIndex, lineCount } = await buildLineIndex(file) + this.#patchFile(name, file, { lineIndex, lineCount, status: 'ready' }) + } catch (e) { + this.#patchFile(name, file, { + status: 'error', + error: e instanceof Error ? e.message : String(e) + }) + } + } + + #patch(name: string, changes: Partial): void { + this.files = this.files.map((f) => (f.name === name ? { ...f, ...changes } : f)) + } + /** + * Patch the row for `name` ONLY while it still holds the exact `file` we indexed. + * `buildLineIndex` is async and unawaited; between its start and finish the row's + * file can be swapped (remove + re-add a same-named file, or a folder refresh + * re-indexing an edited file). Without the identity check a stale completion would + * stamp the wrong lineIndex/lineCount on the new file, and read_file would then slice + * the new Blob with the old offsets. + */ + #patchFile(name: string, file: File | Blob, changes: Partial): void { + this.files = this.files.map((f) => + f.name === name && f.file === file ? { ...f, ...changes } : f + ) + } + #patchSource(sourceId: string, changes: Partial): void { + this.files = this.files.map((f) => (f.sourceId === sourceId ? { ...f, ...changes } : f)) + } + + #uniqueName(original: string): string { + // Uniqueness is only among real files — folder-root placeholders may share a name + // with a standalone file and must not push it to a "(2)" suffix. + const taken = (n: string) => this.files.some((f) => f.name === n && !f.isFolderRoot) + if (!taken(original)) return original + const dot = original.lastIndexOf('.') + const base = dot > 0 ? original.slice(0, dot) : original + const ext = dot > 0 ? original.slice(dot) : '' + let n = 2 + let candidate = `${base} (${n})${ext}` + while (taken(candidate)) { + n++ + candidate = `${base} (${n})${ext}` + } + return candidate + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts new file mode 100644 index 0000000000..c81bc1f9c4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFiles.test.ts @@ -0,0 +1,476 @@ +import { describe, it, expect, vi, beforeEach } from 'vitest' + +// Mock persistence + File System Access so we exercise the in-memory store logic. +vi.mock('./attachedFilesDB', () => ({ + putItem: vi.fn(async () => {}), + deleteItem: vi.fn(async () => {}), + getItemsForSession: vi.fn(async () => []), + ensurePersistentStorage: vi.fn(async () => {}) +})) + +const enumerateDirMock = vi.fn<(h: unknown) => Promise<{ file: File; path: string }[]>>() +vi.mock('./fsAccess', () => ({ + enumerateDir: (h: unknown) => enumerateDirMock(h), + isIgnoredPath: (p: string) => + p.split('/').some((s) => s.startsWith('.') || ['node_modules', 'dist', '.git'].includes(s)), + queryReadPermission: vi.fn(async () => 'granted'), + requestReadPermission: vi.fn(async () => 'granted') +})) + +// buildLineIndex is real by default; a single test flips to 'manual' to control +// completion ordering and exercise the stale-index race guard. +type BuildResult = { lineIndex: number[]; lineCount: number } +const buildDeferreds: Array<{ file: Blob; resolve: (r: BuildResult) => void }> = [] +let buildMode: 'real' | 'manual' = 'real' +vi.mock('./fileEngine', async (importOriginal) => { + const actual = await importOriginal() + return { + ...actual, + buildLineIndex: (file: Blob) => + buildMode === 'real' + ? actual.buildLineIndex(file) + : new Promise((resolve) => buildDeferreds.push({ file, resolve })) + } +}) + +import { AttachedFilesStore } from './attachedFiles.svelte' + +function file(name: string, content: string, lastModified = 1): File { + return new File([content], name, { type: 'text/plain', lastModified }) +} + +const dir = { kind: 'directory', name: 'proj' } as unknown as FileSystemDirectoryHandle + +async function settle(store: AttachedFilesStore) { + for (let i = 0; i < 100 && store.list().some((f) => f.status === 'indexing'); i++) { + await new Promise((r) => setTimeout(r, 2)) + } +} + +const names = (store: AttachedFilesStore) => + store + .list() + .map((f) => f.name) + .sort() + +describe('AttachedFilesStore', () => { + let store: AttachedFilesStore + beforeEach(async () => { + store = new AttachedFilesStore() + await store.restore('s1', false) + }) + + it('links and indexes individual files as snapshots', async () => { + await store.addFiles([file('a.txt', 'one\ntwo\n')]) + await settle(store) + const f = store.get('a.txt') + expect(f?.status).toBe('ready') + expect(f?.lineCount).toBe(2) + expect(f?.handle).toBeUndefined() // files never carry a handle + }) + + it('removes a file', async () => { + await store.addFiles([file('a.txt', 'x')]) + store.removeFile('a.txt') + expect(store.count).toBe(0) + }) + + it('links a folder via a directory handle (enumerating it internally)', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n'), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + expect(enumerateDirMock).toHaveBeenCalledWith(dir) + expect(names(store)).toEqual(['proj/app.ts', 'proj/old.ts']) + expect(store.get('proj/app.ts')?.folder).toBe('proj') + }) + + it('refreshFolders detects rename, add, edit, and delete', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n', 1), path: 'proj/app.ts' }, + { file: file('old.ts', 'y\n', 1), path: 'proj/old.ts' } + ]) + await store.addFolder(dir) + await settle(store) + + // On disk: app.ts edited (mtime bumped), old.ts renamed → new.ts, readme.md added. + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\nedited\n', 2), path: 'proj/app.ts' }, + { file: file('new.ts', 'y\n', 1), path: 'proj/new.ts' }, + { file: file('readme.md', '# hi\n', 1), path: 'proj/readme.md' } + ]) + await store.refreshFolders() + await settle(store) + + // old.ts dropped (renamed away); new.ts + readme.md added; app.ts kept. + expect(names(store)).toEqual(['proj/app.ts', 'proj/new.ts', 'proj/readme.md']) + // edited file re-indexed to its new content (2 lines). + expect(store.get('proj/app.ts')?.status).toBe('ready') + expect(store.get('proj/app.ts')?.lineCount).toBe(2) + }) + + it('exposes folders and standalone as structured views', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await store.addFolder(dir) + await store.addFiles([file('solo.txt', 'one\n')]) + await settle(store) + + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + expect(store.folders[0].status).toBe('ready') + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/app.ts', + 'proj/sub/b.ts' + ]) + expect(store.standalone.map((f) => f.name)).toEqual(['solo.txt']) + expect(store.lockedCount).toBe(0) + }) + + it('a locked folder surfaces as one folder with no files', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + + expect(s2.folders).toEqual([{ name: 'proj', status: 'locked', files: [] }]) + expect(s2.standalone).toEqual([]) + expect(s2.lockedCount).toBe(1) + }) + + it('re-picking a locked folder relinks it instead of silently no-oping', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.folders[0]?.status).toBe('locked') + + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + const result = await s2.addFolder(dir) + await settle(s2) + expect(result.added).toEqual(['proj/app.ts']) + expect(s2.folders).toHaveLength(1) + expect(s2.folders[0].status).toBe('ready') + }) + + it('rejects linking a second folder with the same name (visible, not silent)', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + const result = await store.addFolder(dir) + expect(result.added).toEqual([]) + expect(result.rejected[0]?.reason).toMatch(/already linked/) + expect(store.folders).toHaveLength(1) + }) + + it('regrant keeps the folder visible as unavailable when enumeration fails', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Permission re-granted, but the directory is gone from disk. + enumerateDirMock.mockRejectedValueOnce(new Error('directory removed')) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'unavailable', files: [] }]) + }) + + it('regrant of an empty folder keeps it linked and refreshing (not unlinked)', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 'src1', + sessionId: 's1', + kind: 'dir-handle', + name: 'proj', + folder: 'proj', + handle: dir, + addedAt: 0 + } + ]) + const { queryReadPermission } = await import('./fsAccess') + ;(queryReadPermission as ReturnType).mockResolvedValueOnce('prompt') + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + expect(s2.lockedCount).toBe(1) + + // Access re-granted, but the folder is currently empty — it must stay linked (ready + // placeholder), not vanish when the locked placeholder is dropped. + enumerateDirMock.mockResolvedValueOnce([]) + await s2.regrantLocked() + expect(s2.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(s2.lockedCount).toBe(0) + + // A file added afterward is picked up — the handle survived. + enumerateDirMock.mockResolvedValueOnce([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s2.refreshFolders() + await settle(s2) + expect(s2.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) + + it('removeFolder drops all of a folder’s files', async () => { + enumerateDirMock.mockResolvedValue([ + { file: file('app.ts', 'x\n'), path: 'proj/app.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/b.ts' } + ]) + await store.addFolder(dir) + store.removeFolder('proj') + expect(store.count).toBe(0) + }) + + it('snapshots a folder via addFiles (paths), grouping it and persisting folder + relPath', async () => { + const { putItem } = await import('./attachedFilesDB') + // A persisted (non-transient) session writes through to IndexedDB immediately. + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + expect(s.folders.map((f) => f.name)).toEqual(['proj']) + expect(s.folders[0].files.map((f) => f.relPath).sort()).toEqual(['proj/a.ts', 'proj/sub/b.ts']) + expect(s.standalone).toEqual([]) + const rec = (putItem as ReturnType).mock.calls + .map((c) => c[0]) + .find((r) => r.name === 'proj/a.ts') + expect(rec).toMatchObject({ kind: 'snapshot', folder: 'proj', relPath: 'proj/a.ts' }) + }) + + it('keeps same-basename files from different folder subdirs (dedup by path, not basename)', async () => { + // Two distinct files with the same basename, size and lastModified, different subdirs. + const res = await store.addFiles([ + { file: file('index.ts', 'a\n', 5), path: 'proj/a/index.ts' }, + { file: file('index.ts', 'a\n', 5), path: 'proj/b/index.ts' } + ]) + await settle(store) + expect(res.added.sort()).toEqual(['proj/a/index.ts', 'proj/b/index.ts']) + expect(store.folders[0].files.map((f) => f.relPath).sort()).toEqual([ + 'proj/a/index.ts', + 'proj/b/index.ts' + ]) + }) + + it('skips junk paths (node_modules/.git/dotfiles) inside a snapshotted folder', async () => { + const res = await store.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('dep.js', 'z\n'), path: 'proj/node_modules/dep.js' }, + { file: file('cfg', 'w\n'), path: 'proj/.git/config' } + ]) + await settle(store) + expect(res.added).toEqual(['proj/a.ts']) + expect(store.folders[0].files).toHaveLength(1) + }) + + it('keeps an explicitly attached standalone dotfile (filter is folder-only)', async () => { + const res = await store.addFiles([file('.env', 'SECRET=1\n')]) + await settle(store) + expect(res.added).toEqual(['.env']) + expect(store.standalone.map((f) => f.name)).toEqual(['.env']) + }) + + it('restores a snapshot folder grouped from its persisted folder/relPath', async () => { + const { getItemsForSession } = await import('./attachedFilesDB') + ;(getItemsForSession as ReturnType).mockResolvedValueOnce([ + { + id: 's-a', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/a.ts', + folder: 'proj', + relPath: 'proj/a.ts', + blob: file('a.ts', 'x\n'), + addedAt: 0 + }, + { + id: 's-b', + sessionId: 's1', + kind: 'snapshot', + name: 'proj/b.ts', + folder: 'proj', + relPath: 'proj/b.ts', + blob: file('b.ts', 'y\n'), + addedAt: 0 + } + ]) + const s2 = new AttachedFilesStore() + await s2.restore('s1', true) + await settle(s2) + expect(s2.folders.map((f) => f.name)).toEqual(['proj']) + expect(s2.folders[0].files).toHaveLength(2) + expect(s2.standalone).toEqual([]) + }) + + it('imposes no file-count cap on a folder', async () => { + enumerateDirMock.mockResolvedValue( + Array.from({ length: 150 }, (_, i) => ({ + file: file(`f${i}.ts`, 'x\n'), + path: `proj/f${i}.ts` + })) + ) + await store.addFolder(dir) + await settle(store) + expect(store.folders[0].files.length).toBe(150) + }) + + it('removeFolder deletes every snapshot record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + await s.addFiles([ + { file: file('a.ts', 'x\n'), path: 'proj/a.ts' }, + { file: file('b.ts', 'y\n'), path: 'proj/sub/b.ts' } + ]) + await settle(s) + const ids = s + .list() + .filter((f) => f.folder === 'proj') + .map((f) => f.sourceId) + expect(ids.length).toBe(2) + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + const deleted = (deleteItem as ReturnType).mock.calls.map((c) => c[0]) + for (const id of ids) expect(deleted).toContain(id) + }) + + it('a stale index completion does not corrupt a re-added same-named file', async () => { + buildMode = 'manual' + try { + const A = file('a.txt', 'AAA\n') + const B = file('a.txt', 'BBB\nBBB\nBBB\n') + await store.addFiles([A]) // row 'a.txt' (file A) → buildLineIndex(A) pending + store.removeFile('a.txt') + await store.addFiles([B]) // new row 'a.txt' (file B) → buildLineIndex(B) pending + + // The old (stale) index for A resolves last — it must NOT touch the row now holding B. + buildDeferreds.find((d) => d.file === A)!.resolve({ lineIndex: [0], lineCount: 99 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('indexing') + expect(store.get('a.txt')?.lineCount).not.toBe(99) + + // B's own index applies normally. + buildDeferreds.find((d) => d.file === B)!.resolve({ lineIndex: [0, 4, 8], lineCount: 3 }) + await Promise.resolve() + expect(store.get('a.txt')?.status).toBe('ready') + expect(store.get('a.txt')?.lineCount).toBe(3) + } finally { + buildMode = 'real' + buildDeferreds.length = 0 + } + }) + + it('removeFolder deletes the live folder record from storage (persisted session)', async () => { + const { deleteItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.addFolder(dir) + await settle(s) + const sourceId = s.list().find((f) => f.folder === 'proj')?.sourceId + ;(deleteItem as ReturnType).mockClear() + s.removeFolder('proj') + expect(s.count).toBe(0) + expect((deleteItem as ReturnType).mock.calls.map((c) => c[0])).toContain(sourceId) + }) + + it('an emptied live folder stays visible and refreshes when files return', async () => { + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await store.addFolder(dir) + await settle(store) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Folder emptied on disk → the last child is removed but the folder persists (placeholder). + enumerateDirMock.mockResolvedValue([]) + await store.refreshFolders() + await settle(store) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + expect(store.get('proj/app.ts')).toBeUndefined() + expect(store.readyFiles()).toEqual([]) // placeholder is never a tool target + + // A file added back on disk is picked up — the live source survived the empty state. + enumerateDirMock.mockResolvedValue([{ file: file('new.ts', 'y\n'), path: 'proj/new.ts' }]) + await store.refreshFolders() + await settle(store) + expect(store.folders[0].files.map((f) => f.relPath)).toEqual(['proj/new.ts']) + }) + + it('an empty-folder placeholder does not collide with a same-named standalone file', async () => { + enumerateDirMock.mockResolvedValue([]) // empty folder "proj" → creates a placeholder named "proj" + await store.addFolder(dir) + await settle(store) + + // A standalone file literally named "proj" must NOT be deduped by the placeholder. + const res = await store.addFiles([file('proj', 'hello\n')]) + await settle(store) + expect(res.added).toEqual(['proj']) + expect(store.standalone.map((f) => f.name)).toEqual(['proj']) + expect(store.folders.map((f) => f.name)).toEqual(['proj']) + + // Removing that standalone leaves the folder's placeholder intact. + store.removeFile('proj') + expect(store.standalone).toEqual([]) + expect(store.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + }) + + it('links an initially empty live folder (kept visible, persisted, refreshes)', async () => { + const { putItem } = await import('./attachedFilesDB') + const s = new AttachedFilesStore() + await s.restore('s1', true) + enumerateDirMock.mockResolvedValue([]) // folder is empty at link time + const res = await s.addFolder(dir) + await settle(s) + expect(res.added).toEqual([]) + expect(s.folders).toEqual([{ name: 'proj', status: 'ready', files: [] }]) + // persisted as a dir-handle so it survives a reload despite being empty + const persisted = (putItem as ReturnType).mock.calls.map((c) => c[0]) + expect(persisted.some((r) => r.kind === 'dir-handle' && r.folder === 'proj')).toBe(true) + + // a file added later is picked up — the source existed from the start. + enumerateDirMock.mockResolvedValue([{ file: file('app.ts', 'x\n'), path: 'proj/app.ts' }]) + await s.refreshFolders() + await settle(s) + expect(s.folders[0].files.map((f) => f.relPath)).toEqual(['proj/app.ts']) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts new file mode 100644 index 0000000000..9673e29edc --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.test.ts @@ -0,0 +1,28 @@ +import { describe, expect, it } from 'vitest' +import { + getItemsForSession, + putItem, + deleteItem, + deleteItemsForSession, + ensurePersistentStorage +} from './attachedFilesDB' + +// IndexedDB is unavailable in the node test env. The module must degrade gracefully +// (open fails → reads return [], writes/deletes are no-ops) rather than throwing. +describe('attachedFilesDB without IndexedDB', () => { + it('returns [] for reads', async () => { + expect(await getItemsForSession('s1')).toEqual([]) + }) + + it('does not throw on writes/deletes', async () => { + await expect( + putItem({ id: 'a', sessionId: 's1', kind: 'snapshot', name: 'x.txt', addedAt: 0 }) + ).resolves.toBeUndefined() + await expect(deleteItem('a')).resolves.toBeUndefined() + await expect(deleteItemsForSession('s1')).resolves.toBeUndefined() + }) + + it('does not throw when requesting persistent storage', async () => { + await expect(ensurePersistentStorage()).resolves.toBeUndefined() + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts new file mode 100644 index 0000000000..36f9ac4fbe --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/attachedFilesDB.ts @@ -0,0 +1,118 @@ +/** + * IndexedDB persistence for AI-chat linked files, keyed by session id. + * + * Two kinds of records survive a reload (see the persistence plan): + * - handle records ('file-handle' / 'dir-handle'): a re-grantable File System + * Access handle is stored (structured-clone), re-read live on restore. + * - 'snapshot' records: a full-byte Blob copy (fallback when the File System + * Access API is unavailable). + * + * Mirrors the `idb` usage in HistoryManager.svelte.ts. + */ +import { openDB, type DBSchema as IDBSchema, type IDBPDatabase } from 'idb' + +export type AttachedItemKind = 'snapshot' | 'dir-handle' + +export interface PersistedAttachedItem { + /** Stable record id. */ + id: string + sessionId: string + /** 'snapshot' = a file copied into IndexedDB; 'dir-handle' = a live folder handle. */ + kind: AttachedItemKind + /** Display name: relative path for files, folder name for dir-handle records. */ + name: string + /** Top-level folder (for grouping); equals `name` for dir-handle records. */ + folder?: string + /** Folder-relative path (snapshot folder children) — restores the folder grouping/tree. */ + relPath?: string + /** Live directory handle (for 'dir-handle'). */ + handle?: FileSystemDirectoryHandle + /** Full-content copy (for 'snapshot'). */ + blob?: Blob + size?: number + lastModified?: number + addedAt: number +} + +interface AttachedFilesSchema extends IDBSchema { + items: { + key: string + value: PersistedAttachedItem + indexes: { 'by-session': string } + } +} + +let dbPromise: Promise | undefined> | undefined + +function getDB(): Promise | undefined> { + if (!dbPromise) { + try { + dbPromise = openDB('copilot-attached-files', 1, { + upgrade(db) { + if (!db.objectStoreNames.contains('items')) { + const store = db.createObjectStore('items', { keyPath: 'id' }) + store.createIndex('by-session', 'sessionId') + } + } + }).catch((err) => { + console.error('Could not open attached-files database', err) + return undefined + }) + } catch (err) { + // IndexedDB unavailable (e.g. private mode / no DOM) — degrade gracefully. + console.error('Could not open attached-files database', err) + dbPromise = Promise.resolve(undefined) + } + } + return dbPromise +} + +export async function putItem(item: PersistedAttachedItem): Promise { + const db = await getDB() + await db?.put('items', item) +} + +export async function getItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return [] + try { + return await db.getAllFromIndex('items', 'by-session', sessionId) + } catch (err) { + console.error('Could not read attached files', err) + return [] + } +} + +export async function deleteItem(id: string): Promise { + const db = await getDB() + await db?.delete('items', id) +} + +export async function deleteItemsForSession(sessionId: string): Promise { + const db = await getDB() + if (!db) return + try { + const tx = db.transaction('items', 'readwrite') + const index = tx.store.index('by-session') + let cursor = await index.openCursor(sessionId) + while (cursor) { + await cursor.delete() + cursor = await cursor.continue() + } + await tx.done + } catch (err) { + console.error('Could not delete attached files for session', err) + } +} + +/** Ask the browser to keep our storage from being evicted (best-effort, once). */ +let persistRequested = false +export async function ensurePersistentStorage(): Promise { + if (persistRequested) return + persistRequested = true + try { + await navigator.storage?.persist?.() + } catch { + // best-effort; ignore + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts new file mode 100644 index 0000000000..8ef6d5b827 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.test.ts @@ -0,0 +1,286 @@ +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest' +import { + buildLineIndex, + readFile, + searchFiles, + searchFilesInWorker, + isTextFile, + numberLines, + type FileEntry +} from './fileEngine' + +function makeFile(content: string | Uint8Array, name = 'f.txt'): File { + return new File([content as BlobPart], name) +} + +async function makeEntry(content: string, name = 'f.txt'): Promise { + const file = makeFile(content, name) + const { lineIndex, lineCount } = await buildLineIndex(file) + return { name, file, lineIndex, lineCount } +} + +describe('buildLineIndex', () => { + it('counts lines without a trailing newline', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('does not count a single trailing newline as an extra line', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\nb\n')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 2]) + }) + + it('handles an empty file', async () => { + const { lineIndex, lineCount } = await buildLineIndex(makeFile('')) + expect(lineCount).toBe(0) + expect(lineIndex).toEqual([]) + }) + + it('handles CRLF line endings (offsets by byte)', async () => { + // bytes: a=0 \r=1 \n=2 b=3 → line starts at 0 and 3 + const { lineIndex, lineCount } = await buildLineIndex(makeFile('a\r\nb')) + expect(lineCount).toBe(2) + expect(lineIndex).toEqual([0, 3]) + }) + + it('counts lines correctly across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => `line ${i}`) + const { lineCount } = await buildLineIndex(makeFile(lines.join('\n'))) + expect(lineCount).toBe(5000) + }) +}) + +describe('readFile', () => { + it('reads a bounded window and reports pagination', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 2 }) + expect(res.text).toBe('l1\nl2\n') + expect(res.startLine).toBe(1) + expect(res.endLine).toBe(2) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=3') + }) + + it('reads the final line to end of file', async () => { + const entry = await makeEntry('l1\nl2\nl3') + const res = await readFile(entry, { startLine: 3 }) + expect(res.text).toBe('l3') + expect(res.truncated).toBe(false) + }) + + it('clamps the window to maxLines', async () => { + const entry = await makeEntry(Array.from({ length: 100 }, (_, i) => `l${i}`).join('\n')) + const res = await readFile(entry, { startLine: 1, maxLines: 10 }) + expect(res.endLine).toBe(10) + expect(res.truncated).toBe(true) + }) + + it('char-caps a degenerate single long line', async () => { + const entry = await makeEntry('x'.repeat(10000)) + const res = await readFile(entry, { maxChars: 8000 }) + expect(res.text.length).toBe(8000) + expect(res.truncated).toBe(true) + expect(res.note).toContain('8000 characters') + }) + + it('bounds the byte decode on a newline-sparse window (never reads past maxChars*4 bytes)', async () => { + // One 200k-char line; with maxChars=1000 only ≤4000 bytes are sliced before decode. + const entry = await makeEntry('y'.repeat(200_000)) + const res = await readFile(entry, { maxChars: 1000 }) + expect(res.text).toBe('y'.repeat(1000)) + expect(res.truncated).toBe(true) + }) + + it('clamps a start line beyond the end', async () => { + const entry = await makeEntry('l1\nl2') + const res = await readFile(entry, { startLine: 99 }) + expect(res.startLine).toBe(2) + expect(res.text).toBe('l2') + }) + + it('returns empty for an empty file', async () => { + const entry = await makeEntry('') + const res = await readFile(entry) + expect(res.text).toBe('') + expect(res.totalLines).toBe(0) + }) + + it('char-truncation inside the first line resumes the note at the next line', async () => { + // Line 1 exceeds maxChars; lines 2-3 follow. The window only returns line 1's prefix, + // so the note must report line 1 and point the model at line 2 (not claim lines 1-3). + const entry = await makeEntry('x'.repeat(20000) + '\nl2\nl3') + const res = await readFile(entry, { startLine: 1, endLine: 3, maxChars: 5000 }) + expect(res.endLine).toBe(1) + expect(res.totalLines).toBe(3) + expect(res.truncated).toBe(true) + expect(res.note).toContain('start_line=2') + }) + + it('char-truncation after some whole lines resumes at the truncated line', async () => { + const entry = await makeEntry('a\nb\n' + 'x'.repeat(20000) + '\nd') + const res = await readFile(entry, { startLine: 1, endLine: 4, maxChars: 5000 }) + expect(res.endLine).toBe(2) // a, b whole; line 3 (xxx) cut + expect(res.note).toContain('start_line=3') + // the partial line 3 must NOT leak into the body — it would contradict the note + expect(res.text).toBe('a\nb\n') + expect(numberLines(res.text, res.startLine)).toBe('1→a\n2→b') + }) +}) + +describe('searchFiles', () => { + it('finds matches with 1-based line numbers', async () => { + const entry = await makeEntry('alpha\nbeta\ngamma beta') + const res = await searchFiles([entry], 'beta') + expect(res.error).toBeUndefined() + expect(res.hits).toEqual([ + { file: 'f.txt', line: 2, text: 'beta' }, + { file: 'f.txt', line: 3, text: 'gamma beta' } + ]) + }) + + it('searches across multiple files', async () => { + const a = await makeEntry('needle here', 'a.txt') + const b = await makeEntry('nope\nneedle', 'b.txt') + const res = await searchFiles([a, b], 'needle') + expect(res.hits.map((h) => `${h.file}:${h.line}`)).toEqual(['a.txt:1', 'b.txt:2']) + }) + + it('restricts to a single file with pathFilter', async () => { + const a = await makeEntry('needle', 'a.txt') + const b = await makeEntry('needle', 'b.txt') + const res = await searchFiles([a, b], 'needle', { pathFilter: 'b.txt' }) + expect(res.hits).toEqual([{ file: 'b.txt', line: 1, text: 'needle' }]) + }) + + it('reports an unknown pathFilter as an error', async () => { + const a = await makeEntry('needle', 'a.txt') + const res = await searchFiles([a], 'needle', { pathFilter: 'missing.txt' }) + expect(res.error).toContain('missing.txt') + }) + + it('truncates at maxHits', async () => { + const entry = await makeEntry(Array.from({ length: 10 }, () => 'match').join('\n')) + const res = await searchFiles([entry], 'match', { maxHits: 3 }) + expect(res.hits.length).toBe(3) + expect(res.truncated).toBe(true) + }) + + it('supports case-insensitive flags', async () => { + const entry = await makeEntry('Hello\nworld') + const res = await searchFiles([entry], 'hello', { flags: 'i' }) + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'Hello' }]) + }) + + it('strips trailing CR from matched CRLF lines', async () => { + const entry = await makeEntry('foo\r\nbar') + const res = await searchFiles([entry], 'foo') + expect(res.hits).toEqual([{ file: 'f.txt', line: 1, text: 'foo' }]) + }) + + it('returns a friendly error for an invalid regex', async () => { + const entry = await makeEntry('anything') + const res = await searchFiles([entry], '(') + expect(res.error).toContain('Invalid regex') + expect(res.hits).toEqual([]) + }) + + it('matches across stream chunk boundaries', async () => { + const lines = Array.from({ length: 5000 }, (_, i) => (i === 4999 ? 'TARGET' : `line ${i}`)) + const entry = await makeEntry(lines.join('\n')) + const res = await searchFiles([entry], 'TARGET') + expect(res.hits).toEqual([{ file: 'f.txt', line: 5000, text: 'TARGET' }]) + }) + + it('a global flag does not drop matches via a stale lastIndex', async () => { + // `.test()` is stateful under the `g` flag; without resetting lastIndex, lines after + // the first match would be tested from a stale offset and silently miss. + const entry = await makeEntry('match\nmatch\nmatch') + const res = await searchFiles([entry], 'match', { flags: 'g' }) + expect(res.hits.map((h) => h.line)).toEqual([1, 2, 3]) + }) +}) + +describe('searchFilesInWorker', () => { + // A controllable stand-in for the search Worker — `reply`, `error`, or `hang` (never responds). + class MockWorker { + onmessage: ((e: MessageEvent) => void) | null = null + onerror: ((e: unknown) => void) | null = null + static mode: 'reply' | 'error' | 'hang' = 'reply' + static reply: unknown = { hits: [], truncated: false } + static terminated = false + constructor(_url: URL | string, _opts?: unknown) {} + postMessage(): void { + if (MockWorker.mode === 'reply') + queueMicrotask(() => this.onmessage?.({ data: MockWorker.reply } as MessageEvent)) + else if (MockWorker.mode === 'error') queueMicrotask(() => this.onerror?.({})) + // 'hang' → never responds, exercising the timeout path. + } + terminate(): void { + MockWorker.terminated = true + } + } + + beforeEach(() => { + MockWorker.terminated = false + vi.stubGlobal('Worker', MockWorker) + }) + afterEach(() => vi.unstubAllGlobals()) + + const entry: FileEntry = { name: 'a.txt', file: makeFile('x'), lineIndex: [], lineCount: 0 } + + it('resolves with the worker result and terminates the worker', async () => { + MockWorker.mode = 'reply' + MockWorker.reply = { hits: [{ file: 'a.txt', line: 1, text: 'x' }], truncated: false } + const res = await searchFilesInWorker([entry], 'x') + expect(res.hits).toEqual([{ file: 'a.txt', line: 1, text: 'x' }]) + expect(MockWorker.terminated).toBe(true) + }) + + it('times out on a non-responding (pathological) pattern and terminates the worker', async () => { + MockWorker.mode = 'hang' + const res = await searchFilesInWorker([entry], '^(a+)+$', {}, 20) + expect(res.error).toContain('timed out') + expect(MockWorker.terminated).toBe(true) + }) +}) + +describe('isTextFile', () => { + it('accepts UTF-8 text', async () => { + expect(await isTextFile(makeFile('hello © world'))).toBe(true) + }) + + it('accepts an empty file', async () => { + expect(await isTextFile(makeFile(''))).toBe(true) + }) + + it('rejects content with NUL bytes', async () => { + expect(await isTextFile(makeFile(new Uint8Array([104, 0, 105]), 'b.bin'))).toBe(false) + }) +}) + +describe('numberLines', () => { + it('prefixes each line with its absolute 1-based number', () => { + expect(numberLines('l3\nl4\n', 3)).toBe('3→l3\n4→l4') + }) + + it('right-aligns numbers to a common width', () => { + expect(numberLines('a\nb', 9)).toBe(' 9→a\n10→b') + }) + + it('numbers a final line that has no trailing newline', () => { + expect(numberLines('only', 5)).toBe('5→only') + }) + + it('matches a readFile window (numbers the returned lines, no phantom line)', async () => { + const file = makeFile('l1\nl2\nl3') + const { lineIndex, lineCount } = await buildLineIndex(file) + const res = await readFile( + { name: 'f.txt', file, lineIndex, lineCount }, + { startLine: 1, endLine: 2 } + ) + expect(numberLines(res.text, res.startLine)).toBe('1→l1\n2→l2') + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileEngine.ts b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts new file mode 100644 index 0000000000..5dbceb77e2 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileEngine.ts @@ -0,0 +1,430 @@ +/** + * Storage-agnostic streaming engine for reading and searching attached files. + * + * Files are kept as `File` handles (lazy references to bytes on disk). Nothing is + * decoded into the JS heap wholesale: we stream in chunks, so a large file never + * freezes the tab or blows up memory. The only per-file state held in RAM is a + * line-offset index (a flat array of byte offsets, ~8 bytes per line). + * + * Line semantics match `String.split('\n')` except a single trailing newline does + * NOT add an empty final line (so "a\nb\n" is 2 lines, like `wc -l`). Lines are + * 1-based in the public read/search API. + */ + +/** Minimal shape the engine needs. The attached-files store extends this with reactive status. */ +export interface FileEntry { + name: string + /** A File (live link) or a Blob (restored snapshot) — both stream/slice identically. */ + file: File | Blob + lineIndex: number[] + lineCount: number +} + +const CHUNK_NEWLINE = 0x0a // '\n' — in UTF-8 this byte never appears inside a multibyte sequence + +export const DEFAULT_READ_MAX_LINES = 200 +export const DEFAULT_READ_MAX_CHARS = 8000 +export const DEFAULT_SEARCH_MAX_HITS = 50 +/** + * Per-line cap on how much of a degenerate long line the regex is tested against. + * This bounds work for linear-time patterns; it does NOT prevent catastrophic + * backtracking — a nested-quantifier pattern can still go exponential within the + * capped prefix (search runs on the main thread, so that is a self-DoS of the tab). + */ +export const DEFAULT_SEARCH_LINE_SCAN_CAP = 100_000 +/** How much of a matching line we echo back, to keep search results bounded. */ +export const DEFAULT_SEARCH_LINE_ECHO_CAP = 500 + +/** + * Stream the file once and record the byte offset at which each line starts. + * Scans raw bytes for '\n' (no decode needed — 0x0A is unambiguous in UTF-8). + */ +export async function buildLineIndex( + file: Blob +): Promise<{ lineIndex: number[]; lineCount: number }> { + const fileSize = file.size + if (fileSize === 0) { + return { lineIndex: [], lineCount: 0 } + } + + const lineIndex: number[] = [0] + let offset = 0 + const reader = file.stream().getReader() + try { + while (true) { + const { done, value } = await reader.read() + if (done) break + const chunk = value as Uint8Array + for (let i = 0; i < chunk.length; i++) { + if (chunk[i] === CHUNK_NEWLINE) { + lineIndex.push(offset + i + 1) + } + } + offset += chunk.length + } + } finally { + reader.releaseLock() + } + + // A trailing newline points one past the end (a phantom empty line) — drop it. + if (lineIndex.length > 1 && lineIndex[lineIndex.length - 1] === fileSize) { + lineIndex.pop() + } + + return { lineIndex, lineCount: lineIndex.length } +} + +export interface ReadResult { + text: string + startLine: number + endLine: number + totalLines: number + truncated: boolean + note: string +} + +/** + * Read a bounded window of lines, reading only the relevant byte range from disk. + * Clamps the window to `maxLines` and the returned text to `maxChars` (protects + * against degenerate single-line files). Returns a self-describing pagination note. + */ +export async function readFile( + entry: FileEntry, + opts: { + startLine?: number + endLine?: number + maxLines?: number + maxChars?: number + } = {} +): Promise { + const totalLines = entry.lineCount + const maxLines = opts.maxLines ?? DEFAULT_READ_MAX_LINES + const maxChars = opts.maxChars ?? DEFAULT_READ_MAX_CHARS + + if (totalLines === 0) { + return { + text: '', + startLine: 0, + endLine: 0, + totalLines: 0, + truncated: false, + note: 'File is empty.' + } + } + + let start = opts.startLine ?? 1 + if (start < 1) start = 1 + if (start > totalLines) start = totalLines + + const requestedEnd = opts.endLine ?? start + maxLines - 1 + let end = requestedEnd + if (end < start) end = start + const cappedByLines = end - start + 1 > maxLines + if (cappedByLines) end = start + maxLines - 1 + if (end > totalLines) end = totalLines + + const byteStart = entry.lineIndex[start - 1] + const byteEnd = end < totalLines ? entry.lineIndex[end] : entry.file.size + // Bound the decode for newline-sparse files (minified JS, single-line JSONL): the + // window can span the whole file, but we only ever return maxChars characters, and + // a UTF-8 character is at most 4 bytes — so never materialize more than that. + const byteCap = byteStart + maxChars * 4 + const byteCapped = byteCap < byteEnd + + let text: string + try { + text = await entry.file.slice(byteStart, byteCapped ? byteCap : byteEnd).text() + } catch (e) { + throw new FileReadError(entry.name, e instanceof Error ? e.message : String(e)) + } + + let cappedByChars = byteCapped + if (text.length > maxChars) { + text = text.slice(0, maxChars) + cappedByChars = true + } + + // When the char cap truncates the window short of `end`, the text holds fewer lines + // than requested — so the note must report the last line actually returned and resume + // at the next unread one (otherwise it claims lines it didn't return and skips them). + let lastLine = end + let resumeAt: number | undefined = end < totalLines ? end + 1 : undefined + if (cappedByChars) { + const completeLines = (text.match(/\n/g) || []).length + if (completeLines >= 1) { + // lines start..start+completeLines-1 are whole; the next line was cut mid-content. + // Trim that partial line off the returned text so the body matches the note (and + // the model doesn't see a line the note says it'll get on the next read). + lastLine = start + completeLines - 1 + resumeAt = start + completeLines + text = text.slice(0, text.lastIndexOf('\n') + 1) + } else { + // the cap fell inside line `start` itself — it can't be returned in full, so + // advance past it rather than re-truncating the same line forever. + lastLine = start + resumeAt = start + 1 + } + if (resumeAt > totalLines) resumeAt = undefined + } + + const truncated = cappedByChars || resumeAt !== undefined + + let note = `Showing lines ${start}-${lastLine} of ${totalLines}.` + if (cappedByChars) { + note += ` Output truncated to ${maxChars} characters (line(s) very long).` + } + if (resumeAt !== undefined) { + note += ` Call read_file again with start_line=${resumeAt} for more.` + } + + return { text, startLine: start, endLine: lastLine, totalLines, truncated, note } +} + +/** + * Prefix each line of a read window with its absolute 1-based number (`→`), + * so the model can quote/reference exact lines. `startLine` is the window's first line. + */ +export function numberLines(text: string, startLine: number): string { + const lines = text.split('\n') + // readFile's window ends with the trailing newline of its last line when more lines + // follow, so split yields a phantom empty element — drop it before numbering. + if (lines.length > 1 && lines[lines.length - 1] === '') lines.pop() + const width = String(startLine + lines.length - 1).length + return lines.map((l, i) => `${String(startLine + i).padStart(width)}→${l}`).join('\n') +} + +export interface SearchHit { + file: string + line: number + text: string +} + +export interface SearchResult { + hits: SearchHit[] + truncated: boolean + error?: string +} + +/** + * Run a regex across one or more files, streaming each (no full-file load), and + * return matching lines with 1-based line numbers. Stops at `maxHits`. + */ +export async function searchFiles( + entries: FileEntry[], + pattern: string, + opts: { + flags?: string + pathFilter?: string + maxHits?: number + lineScanCap?: number + lineEchoCap?: number + } = {} +): Promise { + const maxHits = opts.maxHits ?? DEFAULT_SEARCH_MAX_HITS + const lineScanCap = opts.lineScanCap ?? DEFAULT_SEARCH_LINE_SCAN_CAP + const lineEchoCap = opts.lineEchoCap ?? DEFAULT_SEARCH_LINE_ECHO_CAP + + let regex: RegExp + try { + regex = new RegExp(pattern, opts.flags ?? '') + } catch (e) { + return { + hits: [], + truncated: false, + error: `Invalid regex: ${e instanceof Error ? e.message : String(e)}` + } + } + + const targets = opts.pathFilter ? entries.filter((e) => e.name === opts.pathFilter) : entries + if (opts.pathFilter && targets.length === 0) { + return { hits: [], truncated: false, error: `No attached file named "${opts.pathFilter}".` } + } + + const hits: SearchHit[] = [] + let truncated = false + + for (const entry of targets) { + if (hits.length >= maxHits) { + truncated = true + break + } + try { + await streamLines(entry.file, (line, lineNo) => { + // Bound backtracking on pathological long lines by only testing a prefix. + const scanned = line.length > lineScanCap ? line.slice(0, lineScanCap) : line + // `regex` may carry a caller-supplied `g`/`y` flag, which makes `.test()` + // stateful (it advances `lastIndex`) — reset so each line matches from 0. + regex.lastIndex = 0 + if (regex.test(scanned)) { + hits.push({ + file: entry.name, + line: lineNo, + text: line.length > lineEchoCap ? line.slice(0, lineEchoCap) + '…' : line + }) + } + return hits.length < maxHits // continue? + }) + } catch (e) { + return { + hits, + truncated, + error: `Error reading "${entry.name}": ${e instanceof Error ? e.message : String(e)}` + } + } + if (hits.length >= maxHits) { + truncated = true + break + } + } + + return { hits, truncated } +} + +/** + * Run `searchFiles` off the main thread. A model-supplied regex can backtrack + * catastrophically (e.g. /^(a+)+$/) and `regex.test()` can't be interrupted — so we run + * it in a Worker and `terminate()` it on a timeout, keeping the tab responsive instead of + * frozen. Falls back to a main-thread search where Workers aren't available (best effort). + */ +export function searchFilesInWorker( + entries: FileEntry[], + pattern: string, + opts: { flags?: string; pathFilter?: string; maxHits?: number } = {}, + timeoutMs = 3000 +): Promise { + let worker: Worker + try { + worker = new Worker(new URL('./searchWorker.ts', import.meta.url), { type: 'module' }) + } catch { + return searchFiles(entries, pattern, opts) + } + return new Promise((resolve) => { + const finish = (r: SearchResult) => { + clearTimeout(timer) + worker.terminate() + resolve(r) + } + const timer = setTimeout( + () => + finish({ + hits: [], + truncated: false, + error: 'Search timed out — the pattern is too expensive. Try a simpler regex.' + }), + timeoutMs + ) + worker.onmessage = (e: MessageEvent) => finish(e.data) + worker.onerror = () => { + // Worker script failed to load/run — fall back to a main-thread search. + clearTimeout(timer) + worker.terminate() + searchFiles(entries, pattern, opts).then(resolve) + } + worker.postMessage({ + files: entries.map((e) => ({ name: e.name, file: e.file })), + pattern, + flags: opts.flags, + pathFilter: opts.pathFilter, + maxHits: opts.maxHits + }) + }) +} + +export class FileReadError extends Error { + constructor( + public fileName: string, + message: string + ) { + super(message) + this.name = 'FileReadError' + } +} + +/** + * Max characters buffered for a single line while streaming. A newline-less file + * (e.g. minified JS) would otherwise accumulate wholesale in `buffer`; past this + * cap excess characters are dropped (the line's intact prefix is preserved) — + * harmless for search, which only tests/echoes a prefix far smaller than this. + */ +const MAX_LINE_BUFFER_CHARS = 1_000_000 + +/** + * Stream a file and invoke `onLine` for each line (1-based). A trailing newline + * does not produce an empty final line. `onLine` returns false to stop early. + * A trailing '\r' (CRLF) is stripped before the callback. Overlong lines are + * passed with at least their first MAX_LINE_BUFFER_CHARS characters intact; + * content beyond the cap may be dropped. + */ +async function streamLines( + file: Blob, + onLine: (line: string, lineNo: number) => boolean +): Promise { + const reader = file.stream().getReader() + const decoder = new TextDecoder('utf-8') + let buffer = '' + let lineNo = 0 + try { + while (true) { + const { done, value } = await reader.read() + if (done) { + buffer += decoder.decode() + break + } + buffer += decoder.decode(value as Uint8Array, { stream: true }) + let start = 0 + let nlIdx: number + while ((nlIdx = buffer.indexOf('\n', start)) !== -1) { + let line = buffer.slice(start, nlIdx) + if (line.endsWith('\r')) line = line.slice(0, -1) + start = nlIdx + 1 + lineNo++ + if (!onLine(line, lineNo)) return + } + buffer = buffer.slice(start) + // The remainder holds no newline — cap how much of an overlong line we keep. + // Dropped characters are line content only, so newline detection and line + // numbering in later chunks are unaffected. + if (buffer.length > MAX_LINE_BUFFER_CHARS) { + buffer = buffer.slice(0, MAX_LINE_BUFFER_CHARS) + } + } + if (buffer.length > 0) { + let line = buffer + if (line.endsWith('\r')) line = line.slice(0, -1) + lineNo++ + onLine(line, lineNo) + } + } finally { + reader.releaseLock() + } +} + +/** + * Sniff the first bytes of a file to decide whether it is text (UTF-8 decodable, + * no NUL bytes). Used to reject binary files at attach time. + */ +export async function isTextFile(file: Blob, sampleBytes = 8192): Promise { + if (file.size === 0) return true + const slice = file.slice(0, Math.min(sampleBytes, file.size)) + const buf = new Uint8Array(await slice.arrayBuffer()) + for (let i = 0; i < buf.length; i++) { + if (buf[i] === 0) return false // NUL byte → binary + } + try { + // `fatal` throws on invalid UTF-8. We may cut a multibyte char at the sample + // boundary, so only treat it as binary if the error is not at the very end. + new TextDecoder('utf-8', { fatal: true }).decode(buf) + return true + } catch { + // Could be a truncated trailing multibyte sequence — retry on a trimmed buffer. + if (buf.length >= 4) { + try { + new TextDecoder('utf-8', { fatal: true }).decode(buf.slice(0, buf.length - 3)) + return true + } catch { + return false + } + } + return false + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts new file mode 100644 index 0000000000..0bd45153af --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.test.ts @@ -0,0 +1,66 @@ +import { describe, expect, it, vi } from 'vitest' + +// '../shared' transitively imports monaco (CSS) which the node test env can't load. +// fileTools only needs createToolDef from it (called at module load), so stub it. +vi.mock('../shared', () => ({ + createToolDef: (_schema: unknown, name: string, description: string) => ({ name, description }) +})) + +import { searchFilesTool } from './fileTools' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Minimal store stub: searchFilesTool's empty-ready path only reads count/readyFiles/list. */ +function fakeStore(rows: Array>): AttachedFilesStore { + const files = rows as AttachedFile[] + return { + get count() { + return files.length + }, + readyFiles: () => files.filter((f) => f.status === 'ready' && !f.isFolderRoot), + list: () => files + } as unknown as AttachedFilesStore +} + +async function runSearch(store: AttachedFilesStore): Promise { + const res = await searchFilesTool.fn({ + args: { pattern: 'x' }, + helpers: { attachedFiles: store }, + toolId: 't', + toolCallbacks: { setToolStatus: () => {} } + } as any) + return res as string +} + +describe('search_files — attachments present but nothing readable', () => { + it('reports no searchable text for an empty/binary-only linked folder (only ready placeholders)', async () => { + // An empty/all-binary folder leaves a single `ready` placeholder row, filtered out of readyFiles(). + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'ready', isFolderRoot: true }])) + expect(msg).toMatch(/no searchable text files/i) + expect(msg).not.toMatch(/indexed/i) + }) + + it('tells the user to restore access when a restored folder is locked', async () => { + const msg = await runSearch(fakeStore([{ name: 'proj', status: 'locked', isFolderRoot: true }])) + expect(msg).toMatch(/restore access/i) + }) + + it('tells the user to re-link when files are unavailable', async () => { + const msg = await runSearch(fakeStore([{ name: 'gone.txt', status: 'unavailable' }])) + expect(msg).toMatch(/re-link/i) + }) + + it('still reports indexing while a file is genuinely indexing', async () => { + const msg = await runSearch(fakeStore([{ name: 'a.txt', status: 'indexing' }])) + expect(msg).toMatch(/still being indexed/i) + }) + + it('prefers the indexing message when an indexing file coexists with an empty folder', async () => { + const msg = await runSearch( + fakeStore([ + { name: 'proj', status: 'ready', isFolderRoot: true }, + { name: 'a.txt', status: 'indexing' } + ]) + ) + expect(msg).toMatch(/still being indexed/i) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fileTools.ts b/frontend/src/lib/components/copilot/chat/files/fileTools.ts new file mode 100644 index 0000000000..982b858154 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fileTools.ts @@ -0,0 +1,226 @@ +/** + * AI tools and system-prompt roster for files attached to the GLOBAL chat. + * + * The model is made aware of attached files via a metadata-only roster appended to + * the system message (see `appendAttachedFilesRoster`). Their contents are NEVER + * inlined — the model pulls only the slices it needs through these two read-only + * tools, which stream from disk via ./fileEngine. + */ +import { z } from 'zod' +import type { ChatCompletionSystemMessageParam } from 'openai/resources/chat/completions.mjs' +import { createToolDef, type Tool } from '../shared' +import { + readFile, + searchFilesInWorker, + numberLines, + FileReadError, + type SearchHit +} from './fileEngine' +import type { AttachedFile, AttachedFilesStore } from './attachedFiles.svelte' + +/** Slice of the GLOBAL tool helpers that exposes the attached-files store. */ +export interface AttachedFilesHelper { + attachedFiles?: AttachedFilesStore +} + +function storeFrom(helpers: unknown): AttachedFilesStore | undefined { + return (helpers as AttachedFilesHelper | undefined)?.attachedFiles +} + +/** + * For a specifically requested attached file, a message describing why it can't be read / + * searched yet (still indexing, locked, unavailable, errored) or that it isn't attached — + * or undefined when it's `ready`. Shared by read_file and search_files so both report the + * same accurate status instead of search_files claiming a non-ready file isn't attached. + */ +function notReadyMessage(store: AttachedFilesStore, file: string): string | undefined { + const entry = store.get(file) + if (entry?.status === 'ready') return undefined + if (entry?.status === 'indexing') + return `File "${file}" is still being indexed. Try again shortly.` + if (entry?.status === 'locked') + return `File "${file}" is locked after a reload. Ask the user to restore access (send a message, or click "Restore access").` + if (entry?.status === 'unavailable') + return `File "${file}" is no longer available (moved, deleted, or its local copy was evicted). Ask the user to re-link it.` + if (entry?.status === 'error') + return `File "${file}" failed to load: ${entry.error ?? 'unknown error'}.` + const names = store + .list() + .map((f) => f.name) + .join(', ') + return `No attached file named "${file}". Attached files: ${names || '(none)'}.` +} + +/** + * When attachments exist but none expose a readable target (`readyFiles()` is empty), + * explain the actual reason instead of always claiming files are still indexing. Empty + * or binary-only linked folders leave only `ready` placeholder rows (filtered out of + * `readyFiles`), while a locked/unavailable restore surfaces those statuses on the rows. + */ +function noReadyFilesMessage(store: AttachedFilesStore): string { + const statuses = new Set(store.list().map((f) => f.status)) + if (statuses.has('indexing')) return 'Attached files are still being indexed. Try again shortly.' + if (statuses.has('locked')) + return 'The attached files are locked after a reload. Ask the user to restore access (send a message, or click "Restore access").' + if (statuses.has('unavailable')) + return 'The attached files are no longer available (moved, deleted, or their local copies were evicted). Ask the user to re-link them.' + if (statuses.has('error')) return 'The attached files failed to load.' + return 'No searchable text files are attached (a linked folder may be empty or contain only non-text files).' +} + +function humanSize(bytes: number): string { + if (bytes < 1024) return `${bytes} B` + if (bytes < 1024 * 1024) return `${(bytes / 1024).toFixed(1)} KB` + return `${(bytes / (1024 * 1024)).toFixed(1)} MB` +} + +const searchFilesSchema = z.object({ + pattern: z.string().describe('JavaScript regular expression to search for.'), + file: z + .string() + .optional() + .describe( + 'Optional exact filename (as listed under "Attached files") to restrict the search to. Omit to search across all attached files.' + ), + ignore_case: z.boolean().optional().describe('Case-insensitive matching. Defaults to false.') +}) + +const searchFilesToolDef = createToolDef( + searchFilesSchema, + 'search_files', + 'Search the user-attached files with a regular expression and return matching lines with their line numbers. Use this to locate content before reading a specific window with read_file.' +) + +export const searchFilesTool: Tool<{}> = { + def: searchFilesToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = searchFilesSchema.parse(args) + // Validate a specifically requested file against the full store first, so a non-ready + // target reports its real status (indexing/locked/…) instead of "not attached". + if (parsed.file) { + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + } + const ready = store.readyFiles() + if (ready.length === 0) { + return noReadyFilesMessage(store) + } + toolCallbacks.setToolStatus(toolId, { + content: `Searching attached files for /${parsed.pattern}/...` + }) + + // Run in a Worker so a pathological model-supplied regex can't freeze the tab. + const result = await searchFilesInWorker(ready, parsed.pattern, { + flags: parsed.ignore_case ? 'i' : '', + pathFilter: parsed.file + }) + if (result.error) { + return `Error: ${result.error}` + } + const scope = parsed.file ? `"${parsed.file}"` : `${ready.length} file(s)` + if (result.hits.length === 0) { + return `No matches for /${parsed.pattern}/ in ${scope}.` + } + const body = result.hits.map((h: SearchHit) => `${h.file}:${h.line}: ${h.text}`).join('\n') + const header = `Found ${result.hits.length} match(es) in ${scope}:` + const footer = result.truncated + ? '\n\n(Stopped at the result limit — refine your pattern or pass a `file` to narrow the search.)' + : '' + return `${header}\n${body}${footer}` + } +} + +const readFileSchema = z.object({ + file: z.string().describe('Exact filename to read, as listed under "Attached files".'), + start_line: z.number().int().optional().describe('1-based first line to read. Defaults to 1.'), + end_line: z + .number() + .int() + .optional() + .describe('1-based last line to read. The window is capped at 200 lines.') +}) + +const readFileToolDef = createToolDef( + readFileSchema, + 'read_file', + 'Read a bounded window of lines from a user-attached file. Returns each line prefixed with its 1-based number (`→`) plus a pagination note. Files are not in context, so use this to inspect their contents.' +) + +export const readFileTool: Tool<{}> = { + def: readFileToolDef, + fn: async ({ args, helpers, toolId, toolCallbacks }) => { + const store = storeFrom(helpers) + if (!store || store.count === 0) { + return 'No files are attached to this conversation.' + } + const parsed = readFileSchema.parse(args) + const notReady = notReadyMessage(store, parsed.file) + if (notReady) return notReady + const entry = store.get(parsed.file)! + toolCallbacks.setToolStatus(toolId, { content: `Reading "${parsed.file}"...` }) + + try { + const res = await readFile(entry, { + startLine: parsed.start_line, + endLine: parsed.end_line + }) + return res.text ? `${res.note}\n\n${numberLines(res.text, res.startLine)}` : res.note + } catch (e) { + if (e instanceof FileReadError) { + return `Could not read "${parsed.file}": ${e.message}. The file may have been moved or deleted since it was attached.` + } + return `Error reading "${parsed.file}": ${e instanceof Error ? e.message : String(e)}` + } + } +} + +export const fileTools: Tool<{}>[] = [searchFilesTool, readFileTool] + +function rosterLine(f: AttachedFile): string { + if (f.status === 'indexing') return `- ${f.name} (indexing…)` + if (f.status === 'locked') return `- ${f.name} (locked — needs the user to restore access)` + if (f.status === 'unavailable') return `- ${f.name} (unavailable)` + if (f.status === 'error') return `- ${f.name} (failed to load)` + return `- ${f.name} — ${f.lineCount} lines, ${humanSize(f.size)}` +} + +/** Build the `## Attached files` system-prompt section (metadata only, never content). */ +export function buildAttachedFilesRoster(store: AttachedFilesStore): string { + const lines: string[] = [] + for (const folder of store.folders) { + // A locked/unavailable folder has no readable children — one line for the whole folder. + if (folder.status === 'locked') { + lines.push(`- ${folder.name} (locked — needs the user to restore access)`) + } else if (folder.status === 'unavailable') { + lines.push(`- ${folder.name} (unavailable)`) + } else { + lines.push(...folder.files.map(rosterLine)) + } + } + lines.push(...store.standalone.map(rosterLine)) + if (lines.length === 0) return '' + return [ + '## Attached files', + 'The user has attached the following files to this conversation. Their contents are NOT included here.', + 'Use the `search_files` tool to find content with a regex, and `read_file` to read a bounded window of lines.', + '', + lines.join('\n') + ].join('\n') +} + +/** + * Return a copy of the system message with the attached-files roster appended. + * Always derives from the provided base so the roster never accumulates across turns. + */ +export function appendAttachedFilesRoster( + base: ChatCompletionSystemMessageParam, + store: AttachedFilesStore +): ChatCompletionSystemMessageParam { + const roster = buildAttachedFilesRoster(store) + if (!roster || typeof base.content !== 'string') return base + return { ...base, content: `${base.content}\n\n${roster}` } +} diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts new file mode 100644 index 0000000000..e4536b6eac --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.test.ts @@ -0,0 +1,49 @@ +import { afterEach, describe, expect, it } from 'vitest' +import { hasFileSystemAccess, isIgnoredPath, pickDirectory } from './fsAccess' + +describe('hasFileSystemAccess', () => { + it('is false when the File System Access API is absent (node / Firefox / Safari today)', () => { + // The test env exposes none of showOpenFilePicker / showDirectoryPicker / + // DataTransferItem.getAsFileSystemHandle, so the gate must report false. + // The positive path (all three present) is exercised via browser verification. + expect(hasFileSystemAccess()).toBe(false) + }) +}) + +describe('pickDirectory', () => { + afterEach(() => { + delete (window as { showDirectoryPicker?: unknown }).showDirectoryPicker + }) + + it('returns undefined when the user dismisses the picker (AbortError)', async () => { + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('aborted', 'AbortError') + } + await expect(pickDirectory()).resolves.toBeUndefined() + }) + + it('rethrows any non-abort failure instead of silently no-oping', async () => { + // e.g. a policy that blocks the File System Access API, or a lost user-activation. + ;(window as { showDirectoryPicker?: unknown }).showDirectoryPicker = async () => { + throw new DOMException('blocked by policy', 'SecurityError') + } + await expect(pickDirectory()).rejects.toThrow(/blocked by policy/) + }) +}) + +describe('isIgnoredPath', () => { + it('keeps normal source paths', () => { + expect(isIgnoredPath('myproj/src/app.ts')).toBe(false) + expect(isIgnoredPath('README.md')).toBe(false) + }) + it('skips ignored directories', () => { + expect(isIgnoredPath('myproj/node_modules/lib/index.js')).toBe(true) + expect(isIgnoredPath('myproj/dist/bundle.js')).toBe(true) + expect(isIgnoredPath('a/target/x')).toBe(true) + }) + it('skips dotfiles and dotdirs', () => { + expect(isIgnoredPath('myproj/.env')).toBe(true) + expect(isIgnoredPath('myproj/.git/config')).toBe(true) + expect(isIgnoredPath('.DS_Store')).toBe(true) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/files/fsAccess.ts b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts new file mode 100644 index 0000000000..2c741c0988 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/fsAccess.ts @@ -0,0 +1,192 @@ +/** + * Thin wrappers over the File System Access API, used when available so linked + * files/folders can be re-read live after a reload (re-grantable handles). + * Capability is feature-detected (never browser-sniffed): the day Firefox/Safari + * ship the API, the handle path lights up automatically. + */ +const IGNORED_DIRS = new Set([ + 'node_modules', + 'dist', + 'build', + 'out', + 'target', + 'vendor', + 'coverage', + '__pycache__', + '.git', + '.svelte-kit', + '.next', + '.nuxt', + '.venv', + 'venv', + '.idea', + '.vscode', + '.turbo', + '.cache' +]) + +function isIgnoredSegment(name: string): boolean { + return name.startsWith('.') || IGNORED_DIRS.has(name) +} + +/** True if any segment of a relative path is a dotfile/dotdir or an ignored directory. */ +export function isIgnoredPath(path: string): boolean { + return path.split('/').some(isIgnoredSegment) +} + +type FSWindow = Window & { + showDirectoryPicker?: (opts?: { + mode?: 'read' | 'readwrite' + }) => Promise +} + +type FSDataTransferItem = DataTransferItem & { + getAsFileSystemHandle?: () => Promise +} + +/** + * True when the File System Access API needed for FOLDER linking is usable: + * the directory picker plus drag-drop handles. (Files never use the API — they're + * always snapshotted — so showOpenFilePicker is intentionally not required.) + */ +export function hasFileSystemAccess(): boolean { + return ( + typeof window !== 'undefined' && + 'showDirectoryPicker' in window && + typeof DataTransferItem !== 'undefined' && + 'getAsFileSystemHandle' in DataTransferItem.prototype + ) +} + +/** + * Open the directory picker. Returns undefined if the user dismisses it. + * Any other failure (a policy that blocks the File System Access API, a lost + * user-activation, etc.) is rethrown — swallowing it makes the picker silently + * never open, which is indistinguishable from a no-op and impossible to debug. + */ +export async function pickDirectory(): Promise { + const w = window as FSWindow + if (!w.showDirectoryPicker) return undefined + try { + return await w.showDirectoryPicker({ mode: 'read' }) + } catch (e) { + // AbortError means the user dismissed the dialog (and, under browser automation, + // that CDP intercepted the chooser) — a no-op, not a failure. + if (e instanceof DOMException && e.name === 'AbortError') return undefined + throw e + } +} + +/** + * Resolve File System Access handles from a drop's items. The `getAsFileSystemHandle` + * calls are kicked off synchronously (items are only valid during the drop event); + * the returned promise resolves the handles. + */ +export function handlesFromDataTransfer(dt: DataTransfer): Promise { + const pending = Array.from(dt.items) + .filter((it) => it.kind === 'file') + .map((it) => (it as FSDataTransferItem).getAsFileSystemHandle?.() ?? Promise.resolve(null)) + return Promise.all(pending).then((handles) => handles.filter((h): h is FileSystemHandle => !!h)) +} + +export function isFileHandle(h: FileSystemHandle): h is FileSystemFileHandle { + return h.kind === 'file' +} +export function isDirectoryHandle(h: FileSystemHandle): h is FileSystemDirectoryHandle { + return h.kind === 'directory' +} + +/** + * Recursively read a directory handle into a flat list of files with relative paths, + * skipping junk (dotfiles/dotdirs, node_modules, …). No file-count cap — the browser's + * memory/quota are the only limit. Used on link and on live re-enumeration after a reload. + */ +export async function enumerateDir( + dir: FileSystemDirectoryHandle +): Promise<{ file: File; path: string }[]> { + const out: { file: File; path: string }[] = [] + + async function walk(handle: FileSystemDirectoryHandle, prefix: string): Promise { + // @ts-ignore - values() is an async iterator in the File System Access API + for await (const entry of handle.values() as AsyncIterable) { + const path = `${prefix}/${entry.name}` + if (isIgnoredPath(path)) continue + if (isFileHandle(entry)) { + out.push({ file: await entry.getFile(), path }) + } else if (isDirectoryHandle(entry)) { + await walk(entry, path) + } + } + } + + await walk(dir, dir.name) + return out +} + +/** + * Recursively read dropped files AND folders via the legacy `webkitGetAsEntry` API — + * the fallback for browsers without the File System Access API (Firefox/Safari). Folder + * contents are snapshotted into the browser (no live handle). Each result `path` is + * folder-relative (`folder/sub/file` for a dropped folder, bare name for a loose file), + * junk paths skipped, no file-count cap. + * + * `webkitGetAsEntry()` is only valid synchronously during the drop event, so this MUST be + * called from the drop handler — its `.map(...)` runs before the first `await`, capturing + * the entries while the items are still live. + */ +export async function readDroppedEntries( + items: DataTransferItem[] +): Promise<{ file: File; path: string }[]> { + const roots = items + .map((it) => it.webkitGetAsEntry?.() ?? null) + .filter((e): e is FileSystemEntry => !!e) + const out: { file: File; path: string }[] = [] + for (const root of roots) await walkDropEntry(root, out) + return out +} + +async function walkDropEntry( + entry: FileSystemEntry, + out: { file: File; path: string }[] +): Promise { + const path = entry.fullPath.replace(/^\//, '') + if (isIgnoredPath(path)) return + if (entry.isFile) { + const fileEntry = entry as FileSystemFileEntry + const file = await new Promise((res, rej) => fileEntry.file(res, rej)) + out.push({ file, path }) + } else if (entry.isDirectory) { + const reader = (entry as FileSystemDirectoryEntry).createReader() + // readEntries yields in batches and returns [] once exhausted — loop until empty. + while (true) { + const batch = await new Promise((res, rej) => reader.readEntries(res, rej)) + if (batch.length === 0) break + for (const child of batch) await walkDropEntry(child, out) + } + } +} + +/** queryPermission without a user gesture; 'granted' | 'prompt' | 'denied'. Never rejects. */ +export async function queryReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - queryPermission is part of the File System Access API + return (await handle.queryPermission?.({ mode: 'read' })) ?? 'prompt' + } catch { + return 'prompt' + } +} + +/** + * requestPermission — MUST be called within a user gesture. Never rejects: the spec + * rejects with SecurityError when user activation is missing (e.g. a second prompt + * after the first consumed the gesture) — that maps to 'denied' here so callers can + * treat it as "still locked" instead of blowing up the send path. + */ +export async function requestReadPermission(handle: FileSystemHandle): Promise { + try { + // @ts-ignore - requestPermission is part of the File System Access API + return (await handle.requestPermission?.({ mode: 'read' })) ?? 'denied' + } catch { + return 'denied' + } +} diff --git a/frontend/src/lib/components/copilot/chat/files/searchWorker.ts b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts new file mode 100644 index 0000000000..68960801da --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/files/searchWorker.ts @@ -0,0 +1,38 @@ +/** + * Web Worker that runs `search_files` off the main thread. + * + * The regex is model-supplied and `RegExp.prototype.test()` can't be interrupted, so a + * catastrophic-backtracking pattern (e.g. /^(a+)+$/) would otherwise hang the whole tab. + * Running it here lets the caller `terminate()` this worker on a timeout instead. The + * matching itself reuses `searchFiles` from the engine (single source of truth). + */ +import { searchFiles, type FileEntry } from './fileEngine' + +interface SearchRequest { + files: { name: string; file: Blob }[] + pattern: string + flags?: string + pathFilter?: string + maxHits?: number +} + +self.onmessage = async (e: MessageEvent) => { + const { files, pattern, flags, pathFilter, maxHits } = e.data + // searchFiles only reads `name` + `file` (it streams); the index fields are unused here. + const entries: FileEntry[] = files.map((f) => ({ + name: f.name, + file: f.file, + lineIndex: [], + lineCount: 0 + })) + try { + const result = await searchFiles(entries, pattern, { flags, pathFilter, maxHits }) + ;(self as unknown as Worker).postMessage(result) + } catch (err) { + ;(self as unknown as Worker).postMessage({ + hits: [], + truncated: false, + error: err instanceof Error ? err.message : String(err) + }) + } +} diff --git a/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte b/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte index 37db9a90ba..b2496bbcda 100644 --- a/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte +++ b/frontend/src/lib/components/copilot/chat/flow/FlowAIChat.svelte @@ -135,7 +135,8 @@ await acceptPendingFlowEditsIfEnabled() }, getFlowInputsSchema: async () => { - return flowStore.val.schema ?? {} + const s = flowStore.val.schema ?? {} + return { type: 'object', properties: {}, required: [], ...s } }, updateExprsToSet: (id: string, inputTransforms: Record) => { diff --git a/frontend/src/lib/components/copilot/chat/global/core.test.ts b/frontend/src/lib/components/copilot/chat/global/core.test.ts index d6eb6803e4..a3b9c22499 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.test.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.test.ts @@ -26,6 +26,22 @@ vi.mock('@codingame/monaco-vscode-languages-service-override', () => ({ vi.mock('$lib/components/vscode', () => ({})) +// In-memory stand-in for the per-user draft backend. The chat now persists/reads +// drafts through DraftService (no in-tab cell in unit tests), so this Map is the +// source of truth the write/read tools round-trip against. `vi.hoisted` makes it +// available inside the hoisted `vi.mock` factory and the test body alike. +const { backendDrafts, serverTimestamps, failingWrites, failingReads } = vi.hoisted(() => ({ + backendDrafts: new Map(), + // Per-row server timestamp, only set by tests that want to simulate a + // concurrent writer advancing the row; otherwise empty, so the conflict + // branch in `updateDraft` stays inert for every pre-existing test. + serverTimestamps: new Map(), + // Keys whose `updateDraft` / `getDraftForUser` throw a non-404 (network/5xx); + // only set by the error-handling tests, empty otherwise. + failingWrites: new Set(), + failingReads: new Set() +})) + vi.mock('$lib/gen', async () => { const actual = await vi.importActual('$lib/gen') @@ -51,9 +67,6 @@ vi.mock('$lib/gen', async () => { getScriptByHash: vi.fn(async () => { throw new Error('getScriptByHash mock not configured') }), - getScriptByPathWithDraft: vi.fn(async () => { - throw new Error('getScriptByPathWithDraft mock not configured') - }), queryHubScripts: vi.fn(async () => []), getHubScriptContentByPath: vi.fn(async () => ''), listScripts: vi.fn(async () => []) @@ -109,8 +122,8 @@ vi.mock('$lib/gen', async () => { getFlowByPath: vi.fn(async () => { throw new Error('getFlowByPath mock not configured') }), - getFlowByPathWithDraft: vi.fn(async () => { - throw new Error('getFlowByPathWithDraft mock not configured') + getFlowVersion: vi.fn(async () => { + throw new Error('getFlowVersion mock not configured') }), getFlowLatestVersion: vi.fn(async () => ({ id: 1 })), listFlows: vi.fn(async () => []) @@ -131,8 +144,11 @@ vi.mock('$lib/gen', async () => { existsApp: vi.fn(async () => false), createAppRaw: vi.fn(async () => 'created'), updateAppRaw: vi.fn(async () => 'updated'), - getAppByPathWithDraft: vi.fn(async () => { - throw new Error('getAppByPathWithDraft mock not configured') + getAppByPath: vi.fn(async () => { + throw new Error('getAppByPath mock not configured') + }), + getAppByVersion: vi.fn(async () => { + throw new Error('getAppByVersion mock not configured') }), listApps: vi.fn(async () => []) }), @@ -149,6 +165,51 @@ vi.mock('$lib/gen', async () => { }), createVariable: vi.fn(async () => 'created'), updateVariable: vi.fn(async () => 'updated') + }), + FolderService: wrapService(actual.FolderService, { + createFolder: vi.fn(async () => 'created') + }), + DraftService: wrapService(actual.DraftService, { + updateDraft: vi.fn(async ({ kind, path, requestBody }: any) => { + const key = `${kind}:${path}` + if (failingWrites.has(key)) throw Object.assign(new Error('server error'), { status: 500 }) + // A non-force save whose last_sync no longer matches the row's + // server timestamp is rejected (optimistic concurrency). Inert + // unless a test set serverTimestamps for this key. + const serverTs = serverTimestamps.get(key) + if ( + !requestBody?.force && + requestBody?.last_sync != null && + serverTs != null && + requestBody.last_sync !== serverTs + ) { + return { status: 'conflict', current_timestamp: serverTs } + } + if (requestBody?.value == null) backendDrafts.delete(key) + else backendDrafts.set(key, requestBody.value) + return { status: 'saved', current_timestamp: '2026-06-15T00:00:00Z' } + }), + getDraftForUser: vi.fn(async ({ kind, path }: any) => { + const key = `${kind}:${path}` + if (failingReads.has(key)) throw Object.assign(new Error('server error'), { status: 500 }) + // 404-shaped (status) like the real ApiError, so the adapter's + // narrowed catch treats it as "no draft" rather than re-throwing. + if (!backendDrafts.has(key)) + throw Object.assign(new Error('no draft for that owner at that path'), { status: 404 }) + return { value: backendDrafts.get(key), created_at: '2026-06-15T00:00:00Z' } + }), + listDrafts: vi.fn(async () => + Array.from(backendDrafts.entries()).map(([key, value]) => { + const idx = key.indexOf(':') + return { + kind: key.slice(0, idx), + path: key.slice(idx + 1), + summary: (value as any)?.summary, + draft_only: true, + created_at: '2026-06-15T00:00:00Z' + } + }) + ) }) } }) @@ -160,6 +221,13 @@ vi.mock('./rawAppBundlerBridge', () => ({ })) })) +vi.mock('$lib/infer', async () => ({ + ...(await vi.importActual('$lib/infer')), + // Avoid the wasm parser in unit tests: the script deploy path infers the arg + // schema but tolerates failure, and these tests don't assert on the schema. + inferArgs: vi.fn(async () => {}) +})) + import { globalTools, globalToolsFor, @@ -172,11 +240,19 @@ import { setOpenPreviewHandler } from './core' import { UserDraft, __resetUserDraftForTesting } from '$lib/userDraft.svelte' -import { clearGlobalDrafts } from './userDraftAdapter' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' +import { + clearGlobalDrafts, + deleteGlobalDraft, + persistGlobalDraft, + readGlobalDraftValue, + saveGlobalAppDraft +} from './userDraftAdapter' import { bundleRawAppDraft } from './rawAppBundlerBridge' import { AppService, FlowService, + FolderService, HttpTriggerService, JobService, ResourceService, @@ -184,10 +260,23 @@ import { ScriptService, VariableService } from '$lib/gen' +import { userStore } from '$lib/stores' +import { get } from 'svelte/store' import type { Tool, ToolCallbacks } from '../shared' const WORKSPACE = 'global-core-test' +// Seed/read the backend draft store directly (keyed exactly like the syncer: +// `${itemKind}:${storagePath}`). Drop-in replacements for the old in-tab +// `UserDraft.save`/`UserDraft.get` round-trip the tests used before the drafts +// moved to the backend. Extra opts arg is ignored (kept for call-site parity). +function seedBackendDraft(kind: string, path: string, value: unknown, _opts?: unknown): void { + backendDrafts.set(`${kind}:${path}`, value) +} +function getBackendDraft(kind: string, path: string, _opts?: unknown): V | undefined { + return backendDrafts.get(`${kind}:${path}`) as V | undefined +} + const toolCallbacks: ToolCallbacks = { setToolStatus: vi.fn(), removeToolStatus: vi.fn() @@ -240,6 +329,10 @@ describe('global AI tools', () => { beforeEach(() => { __resetUserDraftForTesting() localStorage.clear() + backendDrafts.clear() + serverTimestamps.clear() + failingWrites.clear() + failingReads.clear() clearGlobalDrafts(WORKSPACE) vi.clearAllMocks() }) @@ -415,7 +508,7 @@ describe('global AI tools', () => { resource_type: 'postgresql' }) - expect(UserDraft.get('resource', 'f/resources/db', { workspace: WORKSPACE })).toEqual({ + expect(getBackendDraft('resource', 'f/resources/db', { workspace: WORKSPACE })).toEqual({ path: 'f/resources/db', description: 'existing database', args: { host: 'new.example.com', port: 5432 }, @@ -423,9 +516,6 @@ describe('global AI tools', () => { wsSpecific: true, resource_type: 'postgresql' }) - expect(UserDraft.getMeta('resource', 'f/resources/db', { workspace: WORKSPACE })).toEqual({ - remoteRev: '2026-05-22T09:30:00Z' - }) }) it('writes variable drafts in the editor UserDraft shape', async () => { @@ -450,22 +540,21 @@ describe('global AI tools', () => { description: 'new description' }) - expect(UserDraft.get('variable', 'f/secrets/api_key', { workspace: WORKSPACE })).toEqual({ - path: 'f/secrets/api_key', - variable: { - value: '', - is_secret: true, - description: 'new description' - }, - labels: ['prod'], - wsSpecific: true, - account: 123, - is_oauth: true, - expires_at: '2026-06-22T09:30:00Z' - }) - expect(UserDraft.getMeta('variable', 'f/secrets/api_key', { workspace: WORKSPACE })).toEqual({ - remoteRev: '2026-05-22T09:30:00Z' - }) + expect(getBackendDraft('variable', 'f/secrets/api_key', { workspace: WORKSPACE })).toEqual( + { + path: 'f/secrets/api_key', + variable: { + value: '', + is_secret: true, + description: 'new description' + }, + labels: ['prod'], + wsSpecific: true, + account: 123, + is_oauth: true, + expires_at: '2026-06-22T09:30:00Z' + } + ) expect(localStorageSnapshot()).not.toContain('new-secret-token') }) @@ -478,7 +567,7 @@ describe('global AI tools', () => { }) expect( - UserDraft.get('variable', 'f/secrets/api_key', { workspace: WORKSPACE }) + getBackendDraft('variable', 'f/secrets/api_key', { workspace: WORKSPACE }) ).toMatchObject({ path: 'f/secrets/api_key', variable: { @@ -505,12 +594,14 @@ describe('global AI tools', () => { ws_specific: false }) }) - expect(UserDraft.get('variable', 'f/secrets/api_key', { workspace: WORKSPACE })).toBeUndefined() + expect( + getBackendDraft('variable', 'f/secrets/api_key', { workspace: WORKSPACE }) + ).toBeUndefined() expect(localStorageSnapshot()).not.toContain('new-secret-token') }) it('does not deploy a secret variable draft when the ephemeral value is gone', async () => { - UserDraft.save( + seedBackendDraft( 'variable', 'f/secrets/api_key', { @@ -536,6 +627,178 @@ describe('global AI tools', () => { expect(VariableService.updateVariable).not.toHaveBeenCalled() }) + it('deploys every field of a script draft (not just content/summary)', async () => { + // The deploy delegates to the shared deployer, which reads the full persisted + // draft via getScriptByPath(getDraft) and deploys all of it. Config fields + // (tag/priority/schema/description/concurrency) were previously dropped, + // sourced from the deployed version instead. + seedBackendDraft( + 'script', + 'f/scripts/full', + { path: 'f/scripts/full', content: 'export async function main() {}', language: 'bun' }, + { workspace: WORKSPACE } + ) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ + hash: 1234, + path: 'f/scripts/full', + summary: 'Full script', + description: 'desc', + content: 'export async function main() {}', + schema: { foo: 'bar' }, + language: 'bun', + kind: 'script', + tag: 'custom-tag', + priority: 7, + concurrent_limit: 3, + draft_only: true + } as any) + + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/full' }) + + expect(ScriptService.createScript).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'f/scripts/full', + content: 'export async function main() {}', + summary: 'Full script', + description: 'desc', + schema: { foo: 'bar' }, + language: 'bun', + tag: 'custom-tag', + priority: 7, + concurrent_limit: 3, + parent_hash: 1234 + }) + }) + // Editor-only / server-managed draft keys must not leak into the deploy body. + const calls = vi.mocked(ScriptService.createScript).mock.calls + const body = calls[calls.length - 1][0].requestBody as any + expect(body.draft_only).toBeUndefined() + }) + + it('deploys every config field of a flow draft via createFlow', async () => { + seedBackendDraft( + 'flow', + 'f/flows/full', + { summary: 'Full flow', description: 'flow desc', value: { modules: [] }, schema: {} }, + { workspace: WORKSPACE } + ) + vi.mocked(FlowService.getFlowByPath).mockResolvedValueOnce({ + path: 'f/flows/full', + summary: 'Full flow', + description: 'flow desc', + value: { modules: [] }, + schema: { x: 1 }, + tag: 'flow-tag', + dedicated_worker: true + } as any) + + await callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/full' }) + + // No deployed flow row (existsFlowByPath defaults to false) → create. + expect(FlowService.createFlow).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ + path: 'f/flows/full', + summary: 'Full flow', + description: 'flow desc', + value: { modules: [] }, + schema: { x: 1 }, + tag: 'flow-tag', + dedicated_worker: true + }) + }) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + }) + + it('deploys an editor draft_only script at its chosen path, not its synthetic storage key', async () => { + // A new script created in the editor lives at a synthetic `u/{user}/draft_{uuid}` + // storage key while its chosen path is in the draft value. The chat addresses + // it by the chosen (display) path; deploy must resolve to the storage key so the + // shared deployer can read the draft via getScriptByPath, then deploy at the + // chosen path. Reading at the chosen path would 404. + const storageKey = 'u/admin/draft_abc123' + const chosenPath = 'f/team/chosen_path' + seedBackendDraft( + 'script', + storageKey, + { + path: chosenPath, + summary: 'New script', + description: '', + content: 'export async function main() {}', + schema: {}, + is_template: false, + language: 'bun', + kind: 'script' + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'script', + storagePath: storageKey, + effectivePath: chosenPath + }) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ + path: chosenPath, + summary: 'New script', + description: '', + content: 'export async function main() {}', + schema: {}, + language: 'bun', + kind: 'script' + } as any) + + const flushSpy = vi.spyOn(UserDraftDbSyncer, 'flush') + + await callGlobalTool('deploy_workspace_item', { type: 'script', path: chosenPath }) + + // Any pending editor autosave is flushed at the storage key before delegating, + // so the shared deployer reads the latest value, not a stale persisted draft. + expect(flushSpy).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, itemKind: 'script', path: storageKey }) + ) + // The draft is read at the STORAGE key (the chosen path would 404)… + expect(ScriptService.getScriptByPath).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, path: storageKey, getDraft: true }) + ) + // …and deployed at the chosen path. + expect(ScriptService.createScript).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: expect.objectContaining({ path: chosenPath }) + }) + }) + + it('aborts deploy when the pre-deploy draft flush hit a conflict', async () => { + // flush() resolves even when the save recorded a conflict; deploy must abort + // rather than publish the stale persisted draft. + seedBackendDraft( + 'script', + 'f/scripts/conflicted', + { + path: 'f/scripts/conflicted', + summary: '', + description: '', + content: 'export async function main() {}', + schema: {}, + is_template: false, + language: 'bun', + kind: 'script' + }, + { workspace: WORKSPACE } + ) + const conflictSpy = vi + .spyOn(UserDraftDbSyncer, 'getConflict') + .mockReturnValue({ conflict: { serverTimestamp: '2026', localLastSync: null } } as any) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/conflicted' }) + ).rejects.toThrow(/conflicting/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + conflictSpy.mockRestore() + }) + it('writes script drafts into UserDraft', async () => { const content = 'export async function main() {\n\treturn "hello"\n}' @@ -546,17 +809,17 @@ describe('global AI tools', () => { content }) - expect(UserDraft.get('script', 'f/scripts/hello', { workspace: WORKSPACE })).toMatchObject( - { - path: 'f/scripts/hello', - summary: 'Hello script', - language: 'bun', - content - } - ) + expect( + getBackendDraft('script', 'f/scripts/hello', { workspace: WORKSPACE }) + ).toMatchObject({ + path: 'f/scripts/hello', + summary: 'Hello script', + language: 'bun', + content + }) }) - it('applies path_prefix to local drafts before enforcing the result limit', async () => { + it('applies path_prefix to drafts before enforcing the result limit', async () => { await callGlobalTool('write_script', { path: 'f/other/outside', summary: 'Outside draft', @@ -586,7 +849,7 @@ describe('global AI tools', () => { }) it('lists and edits the live script editor draft through its effective path', async () => { - UserDraft.save( + seedBackendDraft( 'script', '', { @@ -624,17 +887,17 @@ describe('global AI tools', () => { new_string: 'return a * b' }) - expect(UserDraft.get('script', '', { workspace: WORKSPACE })).toMatchObject({ + expect(getBackendDraft('script', '', { workspace: WORKSPACE })).toMatchObject({ path: 'u/admin/amazed_script', content: 'export async function main(a: number, b: number) {\n\treturn a * b\n}' }) expect( - UserDraft.get('script', 'u/admin/amazed_script', { workspace: WORKSPACE }) + getBackendDraft('script', 'u/admin/amazed_script', { workspace: WORKSPACE }) ).toBeUndefined() }) it('lists and writes the live flow editor draft through its effective path', async () => { - UserDraft.save( + seedBackendDraft( 'flow', '', { @@ -672,16 +935,16 @@ describe('global AI tools', () => { modules: JSON.stringify([{ id: 'step', value: { type: 'identity' } }]) }) - expect(UserDraft.get('flow', '', { workspace: WORKSPACE })).toMatchObject({ + expect(getBackendDraft('flow', '', { workspace: WORKSPACE })).toMatchObject({ path: 'u/admin/live_flow', summary: 'Updated live flow', value: { modules: [{ id: 'step', value: { type: 'identity' } }] } }) - expect(UserDraft.get('flow', 'u/admin/live_flow', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('flow', 'u/admin/live_flow', { workspace: WORKSPACE })).toBeUndefined() }) it('writes the live raw app editor draft through its effective path', async () => { - UserDraft.save( + seedBackendDraft( 'raw_app', '', { @@ -705,16 +968,50 @@ describe('global AI tools', () => { content: 'export default function New() { return null }' }) - expect(UserDraft.get('raw_app', '', { workspace: WORKSPACE })).toMatchObject({ + expect(getBackendDraft('raw_app', '', { workspace: WORKSPACE })).toMatchObject({ files: { '/src/App.tsx': 'export default function App() { return null }', '/src/New.tsx': 'export default function New() { return null }' } }) - expect(UserDraft.get('raw_app', 'u/admin/live_app', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('raw_app', 'u/admin/live_app', { workspace: WORKSPACE })).toBeUndefined() }) - it('discards a local draft without deleting the workspace item', async () => { + it('does not echo the app value back to the model on write', async () => { + const sentinel = 'SENTINEL_DO_NOT_ECHO_DEADBEEF' + seedBackendDraft( + 'raw_app', + '', + { + summary: 'Echo check', + files: { '/src/App.tsx': `export default function App() { return '${sentinel}' }` }, + runnables: {}, + data: { tables: [] } + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: '', + effectivePath: 'u/admin/echo_app' + }) + + const raw = await callGlobalTool('write_app_file', { + path: 'u/admin/echo_app', + file_path: '/src/New.tsx', + content: 'export default function New() { return null }' + }) + + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(true) + expect(parsed.item).toBeUndefined() + // Neither the pre-existing file body nor the just-written one is resent. + expect(raw).not.toContain(sentinel) + expect(raw).not.toContain('function New') + }) + + it('discards a draft without deleting the workspace item', async () => { await callGlobalTool('write_script', { path: 'f/scripts/discard-me', summary: 'Temporary draft', @@ -722,7 +1019,9 @@ describe('global AI tools', () => { content: 'export async function main() { return 1 }' }) - expect(UserDraft.get('script', 'f/scripts/discard-me', { workspace: WORKSPACE })).toBeDefined() + expect( + getBackendDraft('script', 'f/scripts/discard-me', { workspace: WORKSPACE }) + ).toBeDefined() const raw = await callGlobalTool('discard_local_draft', { type: 'script', @@ -736,10 +1035,134 @@ describe('global AI tools', () => { }) expect(raw).toContain('The deployed workspace item was not changed') expect( - UserDraft.get('script', 'f/scripts/discard-me', { workspace: WORKSPACE }) + getBackendDraft('script', 'f/scripts/discard-me', { workspace: WORKSPACE }) ).toBeUndefined() }) + // Covers the conflict-on-save / override branch of `persistGlobalDraft` + // directly: a non-force save whose recorded baseline is older than the + // server row is rejected with `status:'conflict'`, and `override` (force) + // pushes our version through. NB: this targets persistGlobalDraft, not the + // write_* tools — those re-read the backend first (readGlobalDraftValue -> + // recordRemoteSync), which re-seeds the baseline and so can only surface a + // conflict when a live editor cell is mounted (not the case in unit tests). + it('persistGlobalDraft surfaces a conflict on a stale baseline and override forces it', async () => { + const path = 'f/scripts/conflicted' + const key = `script:${path}` + const v1 = { + path, + summary: 'v1', + description: '', + content: 'export function main() {}', + language: 'bun' + } + seedBackendDraft('script', path, v1) + // A concurrent writer advanced the row past the baseline we recorded. + serverTimestamps.set(key, '2026-06-15T00:01:00Z') + UserDraftDbSyncer.recordRemoteSync( + { workspace: WORKSPACE, itemKind: 'script', path }, + '2026-06-15T00:00:00Z' + ) + + const v2 = { ...v1, summary: 'v2', content: 'export function main() { return 1 }' } + const conflict = await persistGlobalDraft(WORKSPACE, 'script', path, v2) + expect(conflict.status).toBe('conflict') + if (conflict.status === 'conflict') { + expect(conflict.serverTimestamp).toBe('2026-06-15T00:01:00Z') + } + // The rejected write left the stored draft untouched. + expect(getBackendDraft('script', path, { workspace: WORKSPACE })).toMatchObject({ + summary: 'v1' + }) + + // override:true bypasses the check and persists our version. + const forced = await persistGlobalDraft(WORKSPACE, 'script', path, v2, { force: true }) + expect(forced.status).toBe('saved') + expect(getBackendDraft('script', path, { workspace: WORKSPACE })).toMatchObject({ + summary: 'v2', + content: 'export function main() { return 1 }' + }) + }) + + // A backend save failure (network/5xx) is recorded in the syncer's failure + // map, not thrown — persistGlobalDraft must report 'error', never 'saved'. + it('persistGlobalDraft reports an error (not saved) when the backend save fails', async () => { + const path = 'f/scripts/savefail' + failingWrites.add(`script:${path}`) + const v = { + path, + summary: 's', + description: '', + content: 'export function main() {}', + language: 'bun' + } + const res = await persistGlobalDraft(WORKSPACE, 'script', path, v) + expect(res.status).toBe('error') + if (res.status === 'error') expect(res.message).toBeTruthy() + // Nothing was persisted. + expect(getBackendDraft('script', path, { workspace: WORKSPACE })).toBeUndefined() + }) + + // A non-404 read failure must propagate, not collapse to "no draft" — else + // the write merge falls through to the deployed item, losing draft edits. + it('a non-404 backend read failure propagates instead of returning undefined', async () => { + const path = 'f/scripts/readfail' + failingReads.add(`script:${path}`) + await expect(readGlobalDraftValue(WORKSPACE, 'script', path)).rejects.toThrow() + }) + + // Raw-app writes go through saveGlobalAppDraft, which must carry the conflict + // status so write_app_* tools don't report a stale write as saved. + it('saveGlobalAppDraft surfaces a conflict on a stale baseline', async () => { + const path = 'u/admin/conflictedapp' + const key = `raw_app:${path}` + seedBackendDraft('raw_app', path, { summary: 'v1', files: {}, runnables: {} }) + serverTimestamps.set(key, '2026-06-15T00:01:00Z') + UserDraftDbSyncer.recordRemoteSync( + { workspace: WORKSPACE, itemKind: 'raw_app', path }, + '2026-06-15T00:00:00Z' + ) + const res = await saveGlobalAppDraft(WORKSPACE, path, { + summary: 'v2', + files: {}, + runnables: {} + } as any) + expect(res.status).toBe('conflict') + }) + + // A failed server delete must surface (throw), not silently report removed — + // the same guard the write path got, applied to the delete path. + it('deleteGlobalDraft throws when the server delete fails', async () => { + const path = 'f/scripts/delfail' + seedBackendDraft('script', path, { + path, + summary: 's', + content: 'export function main() {}', + language: 'bun' + }) + failingWrites.add(`script:${path}`) + await expect(deleteGlobalDraft(WORKSPACE, 'script', path)).rejects.toThrow() + }) + + // `override` is a tool-only conflict flag and must not leak into the persisted + // schedule draft value. + it('does not persist the tool-only override flag into a schedule draft', async () => { + await callGlobalTool('write_schedule', { + path: 'f/schedules/ov', + schedule: '0 0 9 * * *', + timezone: 'UTC', + script_path: 'f/scripts/run', + is_flow: false, + args: {}, + override: true + }) + const draft = getBackendDraft('trigger_schedule', 'f/schedules/ov', { + workspace: WORKSPACE + }) + expect(draft).toBeTruthy() + expect(draft).not.toHaveProperty('override') + }) + it('requires trigger_kind when discarding a trigger draft', async () => { await expect( callGlobalTool('discard_local_draft', { @@ -751,23 +1174,14 @@ describe('global AI tools', () => { it('preserves existing script metadata and seeds freshness on first script write', async () => { vi.mocked(ScriptService.existsScriptByPath).mockResolvedValueOnce(true) - vi.mocked(ScriptService.getScriptByPathWithDraft).mockResolvedValueOnce({ + vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ path: 'f/scripts/existing', hash: 'deployed-hash', - draft_created_at: '2026-05-22T10:00:00Z', summary: 'deployed summary', description: 'deployed description', content: 'old deployed content', language: 'bun', - kind: 'script', - draft: { - path: 'f/scripts/existing', - summary: 'db draft summary', - description: 'db draft description', - content: 'old draft content', - language: 'bun', - kind: 'script' - } + kind: 'script' } as any) await callGlobalTool('write_script', { @@ -778,25 +1192,275 @@ describe('global AI tools', () => { }) expect( - UserDraft.get('script', 'f/scripts/existing', { workspace: WORKSPACE }) + getBackendDraft('script', 'f/scripts/existing', { workspace: WORKSPACE }) ).toMatchObject({ path: 'f/scripts/existing', parent_hash: 'deployed-hash', summary: 'new summary', - description: 'db draft description', + description: 'deployed description', content: 'new content', language: 'bun' }) - expect(UserDraft.getMeta('script', 'f/scripts/existing', { workspace: WORKSPACE })).toEqual({ - remoteRev: 'deployed-hash', - remoteDraftRev: '2026-05-22T10:00:00Z' + }) + + describe('stale-draft deploy guard and rebase', () => { + // The suite's beforeEach only clears mock calls (not implementations), so + // restore the script-service mocks these tests override back to their factory + // defaults; otherwise a persistent resolved value leaks into later tests. + afterEach(() => { + vi.mocked(ScriptService.existsScriptByPath).mockResolvedValue(false) + vi.mocked(ScriptService.getScriptByPath).mockImplementation(async () => { + throw new Error('getScriptByPath mock not configured') + }) + vi.mocked(ScriptService.getScriptByHash).mockImplementation(async () => { + throw new Error('getScriptByHash mock not configured') + }) + vi.mocked(FlowService.existsFlowByPath).mockResolvedValue(false) + vi.mocked(FlowService.getFlowByPath).mockImplementation(async () => { + throw new Error('getFlowByPath mock not configured') + }) + vi.mocked(FlowService.getFlowVersion).mockImplementation(async () => { + throw new Error('getFlowVersion mock not configured') + }) + vi.mocked(FlowService.getFlowLatestVersion).mockResolvedValue({ id: 1 } as any) + vi.mocked(AppService.existsApp).mockResolvedValue(false) + vi.mocked(AppService.getAppByPath).mockImplementation(async () => { + throw new Error('getAppByPath mock not configured') + }) + vi.mocked(AppService.getAppByVersion).mockImplementation(async () => { + throw new Error('getAppByVersion mock not configured') + }) + }) + + function seedStaleScriptDraft(path: string, parentHash: string, content = 'draft content') { + seedBackendDraft('script', path, { + path, + summary: 's', + description: '', + content, + language: 'bun', + kind: 'script', + parent_hash: parentHash, + schema: {} + }) + } + + function mockDeployedScript(path: string, hash: string, content = 'latest deployed') { + vi.mocked(ScriptService.existsScriptByPath).mockResolvedValue(true) + vi.mocked(ScriptService.getScriptByPath).mockResolvedValue({ + path, + hash, + content, + language: 'bun', + summary: 's' + } as any) + } + + it('blocks deploying a script draft started from an older deployed version', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash') + mockDeployedScript('f/scripts/stale', 'new-hash') + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + }) + + it('deploys a stale script draft when force is set', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash') + mockDeployedScript('f/scripts/stale', 'new-hash') + + const result = JSON.parse( + await callGlobalTool('deploy_workspace_item', { + type: 'script', + path: 'f/scripts/stale', + force: true + }) + ) + expect(result.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + it('deploys a script draft that is based on the current deployed head', async () => { + seedStaleScriptDraft('f/scripts/fresh', 'head-hash') + mockDeployedScript('f/scripts/fresh', 'head-hash') + + const result = JSON.parse( + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/fresh' }) + ) + expect(result.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + it('rebase_draft discards the stale draft and surfaces the changes to replay', async () => { + seedStaleScriptDraft('f/scripts/stale', 'base-hash', 'base content\nmy added line\n') + mockDeployedScript('f/scripts/stale', 'new-hash', 'latest deployed content\n') + vi.mocked(ScriptService.getScriptByHash).mockResolvedValue({ + hash: 'base-hash', + content: 'base content\n', + language: 'bun' + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'script', path: 'f/scripts/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_hash).toBe('new-hash') + // The diff surfaces the draft's own change over its fork base. + expect(result.your_changes).toContain('my added line') + + // The stale draft is discarded (not reset), so a premature deploy fails + // cleanly rather than silently shipping the latest unchanged. + expect(getBackendDraft('script', 'f/scripts/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(ScriptService.createScript).not.toHaveBeenCalled() + + // Re-applying re-bases onto the current head; the deploy then passes. + await callGlobalTool('write_script', { + path: 'f/scripts/stale', + summary: 's', + language: 'bun', + content: 'latest deployed content\nmy added line\n' + }) + const deploy = JSON.parse( + await callGlobalTool('deploy_workspace_item', { type: 'script', path: 'f/scripts/stale' }) + ) + expect(deploy.success).toBe(true) + expect(ScriptService.createScript).toHaveBeenCalled() + }) + + function seedStaleFlowDraft(path: string, versionId: number, modules: any[] = []) { + seedBackendDraft('flow', path, { + path, + summary: 'f', + description: '', + version_id: versionId, + value: { modules }, + schema: {} + }) + } + + function mockDeployedFlow(path: string, versionId: number) { + vi.mocked(FlowService.existsFlowByPath).mockResolvedValue(true) + vi.mocked(FlowService.getFlowByPath).mockResolvedValue({ + path, + summary: 'f', + version_id: versionId, + value: { modules: [] }, + schema: {} + } as any) + } + + it('blocks deploying a flow draft started from an older deployed version', async () => { + seedStaleFlowDraft('f/flows/stale', 1) + mockDeployedFlow('f/flows/stale', 2) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + expect(FlowService.createFlow).not.toHaveBeenCalled() + }) + + it('rebase_draft discards the stale flow draft and surfaces the changes to replay', async () => { + seedStaleFlowDraft('f/flows/stale', 1, [{ id: 'a', value: { type: 'identity' } }]) + mockDeployedFlow('f/flows/stale', 2) + vi.mocked(FlowService.getFlowVersion).mockResolvedValue({ + value: { modules: [] } + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'flow', path: 'f/flows/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_version).toBe(2) + expect(result.your_changes).toContain('identity') + + // The stale draft is discarded, so a premature deploy fails cleanly. + expect(getBackendDraft('flow', 'f/flows/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'flow', path: 'f/flows/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(FlowService.updateFlow).not.toHaveBeenCalled() + }) + + function seedStaleAppDraft(path: string, parentVersion: number, file = 'old') { + seedBackendDraft('raw_app', path, { + summary: 'a', + files: { '/index.tsx': file }, + runnables: {}, + data: { tables: [] }, + parent_version: parentVersion + }) + } + + function mockDeployedApp(path: string, versionId: number, file = 'latest') { + vi.mocked(AppService.existsApp).mockResolvedValue(true) + vi.mocked(AppService.getAppByPath).mockResolvedValue({ + path, + summary: 'a', + versions: [versionId], + value: { files: { '/index.tsx': file }, runnables: {}, data: { tables: [] } }, + policy: { execution_mode: 'publisher' } + } as any) + } + + it('grafts the fork-base version onto a new app draft and keeps it through the save whitelist', async () => { + // No draft yet: the first app edit projects the deployed app into a draft. + // This exercises the runtime path types can't catch — the graft in + // appSourceToDraftValue AND survival through normalizeAppDraftValue's whitelist. + mockDeployedApp('f/apps/fresh', 5) + + await callGlobalTool('write_app_file', { + path: 'f/apps/fresh', + file_path: '/src/New.tsx', + content: 'export default function New() { return null }' + }) + + const draft = getBackendDraft('raw_app', 'f/apps/fresh', { workspace: WORKSPACE }) + expect(draft.parent_version).toBe(5) + }) + + it('blocks deploying an app draft started from an older deployed version', async () => { + seedStaleAppDraft('f/apps/stale', 1) + mockDeployedApp('f/apps/stale', 2) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/stale' }) + ).rejects.toThrow(/older deployed version/) + expect(AppService.createAppRaw).not.toHaveBeenCalled() + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + + it('rebase_draft discards the stale app draft and surfaces the changes to replay', async () => { + seedStaleAppDraft('f/apps/stale', 1, 'my-change') + mockDeployedApp('f/apps/stale', 2, 'latest-deployed') + vi.mocked(AppService.getAppByVersion).mockResolvedValue({ + value: { files: { '/index.tsx': 'base' }, runnables: {}, data: { tables: [] } } + } as any) + + const result = JSON.parse( + await callGlobalTool('rebase_draft', { type: 'app', path: 'f/apps/stale' }) + ) + expect(result.success).toBe(true) + expect(result.latest_version).toBe(2) + expect(result.your_changes).toContain('my-change') + + // The stale draft is discarded, so a premature deploy fails cleanly. + expect(getBackendDraft('raw_app', 'f/apps/stale', { workspace: WORKSPACE })).toBeUndefined() + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/stale' }) + ).rejects.toThrow(/No .*draft/) + expect(AppService.updateAppRaw).not.toHaveBeenCalled() }) }) it('preserves existing flow metadata and seeds freshness on first flow write', async () => { vi.mocked(FlowService.existsFlowByPath).mockResolvedValueOnce(true) vi.mocked(FlowService.getFlowLatestVersion).mockResolvedValueOnce({ id: 42 } as any) - vi.mocked(FlowService.getFlowByPathWithDraft).mockResolvedValueOnce({ + vi.mocked(FlowService.getFlowByPath).mockResolvedValueOnce({ path: 'f/flows/existing', summary: 'deployed summary', description: 'deployed description', @@ -805,19 +1469,7 @@ describe('global AI tools', () => { edited_by: 'admin', edited_at: '2026-05-22T09:00:00Z', archived: false, - extra_perms: {}, - draft_created_at: '2026-05-22T10:00:00Z', - draft: { - path: 'f/flows/existing', - summary: 'db draft summary', - description: 'db draft description', - value: { modules: [] }, - schema: { properties: { draft: { type: 'string' } } }, - edited_by: 'admin', - edited_at: '2026-05-22T09:30:00Z', - archived: false, - extra_perms: {} - } + extra_perms: {} } as any) await callGlobalTool('write_flow', { @@ -826,16 +1478,14 @@ describe('global AI tools', () => { modules: JSON.stringify([{ id: 'step', value: { type: 'identity' } }]) }) - expect(UserDraft.get('flow', 'f/flows/existing', { workspace: WORKSPACE })).toMatchObject({ + expect( + getBackendDraft('flow', 'f/flows/existing', { workspace: WORKSPACE }) + ).toMatchObject({ path: 'f/flows/existing', summary: 'new summary', - description: 'db draft description', + description: 'deployed description', value: { modules: [{ id: 'step', value: { type: 'identity' } }] } }) - expect(UserDraft.getMeta('flow', 'f/flows/existing', { workspace: WORKSPACE })).toEqual({ - remoteRev: 42, - remoteDraftRev: '2026-05-22T10:00:00Z' - }) }) it('preserves editor schedule fields when writing over an existing schedule', async () => { @@ -869,7 +1519,7 @@ describe('global AI tools', () => { }) expect( - UserDraft.get('trigger_schedule', 'f/schedules/nightly', { workspace: WORKSPACE }) + getBackendDraft('trigger_schedule', 'f/schedules/nightly', { workspace: WORKSPACE }) ).toMatchObject({ path: 'f/schedules/nightly', schedule: '0 15 0 * * *', @@ -884,7 +1534,7 @@ describe('global AI tools', () => { no_flow_overlap: true }) expect( - UserDraft.get('trigger_schedule', 'f/schedules/nightly', { workspace: WORKSPACE }) + getBackendDraft('trigger_schedule', 'f/schedules/nightly', { workspace: WORKSPACE }) ).not.toMatchObject({ edited_by: expect.anything() }) @@ -927,7 +1577,7 @@ describe('global AI tools', () => { } }) - const draft = UserDraft.get('trigger_http', 'f/routes/api', { workspace: WORKSPACE }) + const draft = getBackendDraft('trigger_http', 'f/routes/api', { workspace: WORKSPACE }) expect(draft).toMatchObject({ path: 'f/routes/api', script_path: 'f/flows/new', @@ -947,32 +1597,22 @@ describe('global AI tools', () => { }) it('seeds raw app draft metadata on first app write', async () => { - vi.mocked(AppService.getAppByPathWithDraft).mockResolvedValueOnce({ + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ path: 'f/apps/report', summary: 'deployed app', versions: [3, 4], - draft_created_at: '2026-05-22T10:30:00Z', value: { files: { '/src/App.tsx': 'deployed content' }, - runnables: {}, - data: { tables: [] } + runnables: { + main: { + type: 'inline', + inlineScript: { language: 'bun', content: 'export async function main() {}' } + } + }, + data: { tables: ['orders'], datatable: 'db', schema: 'public' } }, policy: { execution_mode: 'publisher' }, - custom_path: 'report', - draft: { - summary: 'saved app draft', - value: { - files: { '/src/App.tsx': 'draft content' }, - runnables: { - main: { - type: 'inline', - inlineScript: { language: 'bun', content: 'export async function main() {}' } - } - }, - data: { tables: ['orders'], datatable: 'db', schema: 'public' } - }, - policy: { execution_mode: 'anonymous' } - } + custom_path: 'report' } as any) await callGlobalTool('write_app_file', { @@ -981,11 +1621,11 @@ describe('global AI tools', () => { content: 'export default function New() { return null }' }) - const draft = UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE }) + const draft = getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE }) expect(draft).toMatchObject({ - summary: 'saved app draft', + summary: 'deployed app', files: { - '/src/App.tsx': 'draft content', + '/src/App.tsx': 'deployed content', '/src/New.tsx': 'export default function New() { return null }' }, runnables: { @@ -995,17 +1635,13 @@ describe('global AI tools', () => { } }, data: { tables: ['orders'], datatable: 'db', schema: 'public' }, - policy: { execution_mode: 'anonymous' }, + policy: { execution_mode: 'publisher' }, custom_path: 'report' }) - expect(UserDraft.getMeta('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toEqual({ - remoteRev: 4, - remoteDraftRev: '2026-05-22T10:30:00Z' - }) }) it('summarizes local raw app drafts in read_workspace_item', async () => { - UserDraft.save( + seedBackendDraft( 'raw_app', 'f/apps/local', { @@ -1055,39 +1691,31 @@ describe('global AI tools', () => { expect(item.value.backend[0]).not.toHaveProperty('content') }) - it('summarizes backend raw app drafts from the same source as file reads', async () => { - const appWithDraft = { + it('summarizes backend raw apps from the same source as file reads', async () => { + const deployedApp = { path: 'f/apps/report', summary: 'deployed app', versions: [5], value: { - files: { '/src/App.tsx': 'deployed content' }, - runnables: {}, - data: { tables: ['deployed'] } - }, - draft: { - summary: 'saved app draft', - value: { - files: { - '/src/App.tsx': 'draft content', - '/src/DraftOnly.tsx': 'draft-only content' - }, - runnables: { - main: { - type: 'inline', - inlineScript: { - language: 'bun', - content: 'export async function main() { return "draft" }' - } + files: { + '/src/App.tsx': 'deployed content', + '/src/Helper.tsx': 'helper content' + }, + runnables: { + main: { + type: 'inline', + inlineScript: { + language: 'bun', + content: 'export async function main() { return "deployed" }' } - }, - data: { tables: ['draft'] } - } + } + }, + data: { tables: ['deployed'] } } } - vi.mocked(AppService.getAppByPathWithDraft) - .mockResolvedValueOnce(appWithDraft as any) - .mockResolvedValueOnce(appWithDraft as any) + vi.mocked(AppService.getAppByPath) + .mockResolvedValueOnce(deployedApp as any) + .mockResolvedValueOnce(deployedApp as any) const raw = await callGlobalTool('read_workspace_item', { type: 'app', @@ -1095,15 +1723,14 @@ describe('global AI tools', () => { }) const item = JSON.parse(raw) - expect(raw).not.toContain('draft-only content') expect(item).toMatchObject({ type: 'app', path: 'f/apps/report', - summary: 'saved app draft', + summary: 'deployed app', value: { frontend: [ - { path: '/src/App.tsx', size: 'draft content'.length }, - { path: '/src/DraftOnly.tsx', size: 'draft-only content'.length } + { path: '/src/App.tsx', size: 'deployed content'.length }, + { path: '/src/Helper.tsx', size: 'helper content'.length } ], backend: [ expect.objectContaining({ @@ -1111,10 +1738,10 @@ describe('global AI tools', () => { name: 'main', type: 'inline', language: 'bun', - contentSize: 'export async function main() { return "draft" }'.length + contentSize: 'export async function main() { return "deployed" }'.length }) ], - data: { tables: ['draft'] } + data: { tables: ['deployed'] } }, isDraft: false }) @@ -1122,14 +1749,14 @@ describe('global AI tools', () => { await expect( callGlobalTool('read_app_file', { path: 'f/apps/report', - file_path: '/src/DraftOnly.tsx' + file_path: '/src/Helper.tsx' }) - ).resolves.toBe('draft-only content') - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + ).resolves.toBe('helper content') + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() }) - it('reads raw app files without creating a local draft', async () => { - vi.mocked(AppService.getAppByPathWithDraft).mockResolvedValueOnce({ + it('reads raw app files without creating a draft', async () => { + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ path: 'f/apps/report', summary: 'deployed app', versions: [5], @@ -1137,14 +1764,6 @@ describe('global AI tools', () => { files: { '/src/App.tsx': 'deployed content' }, runnables: {}, data: { tables: [] } - }, - draft: { - summary: 'saved app draft', - value: { - files: { '/src/App.tsx': 'draft content' }, - runnables: {}, - data: { tables: [] } - } } } as any) @@ -1153,12 +1772,379 @@ describe('global AI tools', () => { path: 'f/apps/report', file_path: '/src/App.tsx' }) - ).resolves.toBe('draft content') - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + ).resolves.toBe('deployed content') + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + }) + + const deployedAppWithFile = (filePath: string, content: string) => + ({ + path: 'f/apps/report', + summary: 'deployed app', + versions: [5], + value: { files: { [filePath]: content }, runnables: {}, data: {} } + }) as any + + it('truncates a large frontend file to a head slice with a paging annotation', async () => { + const lines = Array.from({ length: 2000 }, (_, i) => `line ${i + 1}`) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce( + deployedAppWithFile('/big.tsx', lines.join('\n')) + ) + + const result = await callGlobalTool('read_app_file', { + path: 'f/apps/report', + file_path: '/big.tsx' + }) + + expect(result).toContain('lines 1-1500 of 2000.') + expect(result).toContain('offset=1501') + expect(result).toContain('line 1500') + expect(result).not.toContain('line 1501') + }) + + it('returns the requested window when offset and limit are given', async () => { + const lines = Array.from({ length: 2000 }, (_, i) => `line ${i + 1}`) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce( + deployedAppWithFile('/big.tsx', lines.join('\n')) + ) + + const result = await callGlobalTool('read_app_file', { + path: 'f/apps/report', + file_path: '/big.tsx', + offset: 5, + limit: 3 + }) + + expect(result).toContain('lines 5-7 of 2000.') + expect(result).toContain('line 5\nline 6\nline 7') + expect(result).not.toContain('line 4') + expect(result).not.toContain('line 8') + }) + + it('truncates at the character budget for files with very long lines', async () => { + const bigLine = 'x'.repeat(30_000) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce( + deployedAppWithFile('/min.tsx', [bigLine, bigLine, bigLine].join('\n')) + ) + + const result = await callGlobalTool('read_app_file', { + path: 'f/apps/report', + file_path: '/min.tsx' + }) + + expect(result).toContain('lines 1-3 of 3, truncated to the first 50000 of 90002 chars.') + expect(result).toContain('the file is likely minified') + expect(result.split('\n\n')[1]).toHaveLength(50_000) + }) + + it('caps a single-line generated file at the character budget', async () => { + const bigLine = 'x'.repeat(60_000) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce( + deployedAppWithFile('/generated.js', bigLine) + ) + + const result = await callGlobalTool('read_app_file', { + path: 'f/apps/report', + file_path: '/generated.js' + }) + + expect(result).toContain('lines 1-1 of 1, truncated to the first 50000 of 60000 chars.') + expect(result).toContain('re-read with a smaller limit') + expect(result.split('\n\n')[1]).toBe('x'.repeat(50_000)) + }) + + it('reports an offset past the end of the file plainly', async () => { + const lines = Array.from({ length: 10 }, (_, i) => `line ${i + 1}`) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce( + deployedAppWithFile('/small.tsx', lines.join('\n')) + ) + + await expect( + callGlobalTool('read_app_file', { + path: 'f/apps/report', + file_path: '/small.tsx', + offset: 50 + }) + ).resolves.toBe('offset 50 is past the end of the file (10 lines).') + }) + + // Deterministic micro-benchmark: measures how much context the read_app_file cap + // saves over a realistic big-project read pattern, isolated from model + // nondeterminism. "Baseline" is the old behavior (whole file returned on every + // read); "actual" is the current line cap + char budget + paging. Asserting the + // ratio also guards against a future change silently weakening the savings. + it('micro-benchmark: the read cap cuts returned context for a realistic read pattern', async () => { + const bigContent = Array.from({ length: 5000 }, (_, i) => `const row${i} = ${i};`).join('\n') + const minified = 'a'.repeat(200_000) // single long line (e.g. a generated bundle) + const appValue = { + path: 'f/apps/report', + summary: 'big app', + versions: [5], + value: { files: { '/big.tsx': bigContent, '/min.js': minified }, runnables: {}, data: {} } + } as any + const fullSize: Record = { + '/big.tsx': bigContent.length, + '/min.js': minified.length + } + + // A plausible pass over a large app: read a big file head, page deeper into it, + // then hit a generated bundle (capped at the char budget). The old tool returned + // every file in full on every read. + const sequence = [ + { file_path: '/big.tsx' }, // 1. big file head (line cap) + { file_path: '/big.tsx', offset: 1501 }, // 2. next line chunk + { file_path: '/min.js' } // 3. minified bundle head (char budget) + ] + + let baselineChars = 0 + let actualChars = 0 + const perRead: number[] = [] + for (const read of sequence) { + baselineChars += fullSize[read.file_path] + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(appValue) + const out = await callGlobalTool('read_app_file', { path: 'f/apps/report', ...read }) + actualChars += out.length + perRead.push(out.length) + } + + const reductionPct = Math.round((1 - actualChars / baselineChars) * 100) + // Surfaced when the suite runs so the benchmark is readable, not just asserted. + // eslint-disable-next-line no-console + console.log( + `[read_app_file micro-benchmark] baseline=${baselineChars} chars, actual=${actualChars} chars ` + + `(per-read ${perRead.join(', ')}), reduction=${reductionPct}%` + ) + + // Each capped read is far smaller than the whole file it came from: + expect(perRead[0]).toBeLessThan(fullSize['/big.tsx']) // head slice < whole file + expect(perRead[1]).toBeLessThan(fullSize['/big.tsx']) // a paged line chunk too + expect(perRead[2]).toBeLessThan(51_000) // ~50k char budget + a short annotation + // Overall: well under half the bytes the old tool would have returned. + expect(actualChars).toBeLessThan(baselineChars * 0.5) + }) + + // A multi-file app: the revenue helper is referenced in three frontend files + // and one inline backend runnable; a generated file also mentions it (and must + // be excluded). Mirrors the analytics_dashboard fixture's "symbol spread". + const searchAppValue = () => + ({ + path: 'f/apps/report', + summary: 'search app', + versions: [5], + value: { + files: { + '/lib/aggregations.ts': 'export function computeRevenue(o) {\n return o.unitPrice\n}\n', + '/components/SummaryPanel.tsx': + 'import { computeRevenue } from "../lib/aggregations"\nconst total = computeRevenue(order)\n', + '/components/OrdersTable.tsx': 'const r = computeRevenue(row)\n// renders revenue\n', + '/styles.css': '.revenue { color: green }\n', + '/wmill.d.ts': 'declare function computeRevenue(o: any): number\n' + }, + runnables: { + computeSummary: { + type: 'inline', + inlineScript: { + language: 'bun', + content: 'export async function main() {\n return computeRevenue\n}\n' + } + } + }, + data: {} + } + }) as any + + it('greps across frontend files and inline runnables, returning file:line rows', async () => { + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(searchAppValue()) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'computeRevenue' + }) + + // header counts every match across the (non-generated) files, without echoing the query + expect(result).toMatch(/\d+ match(?:es)? in \d+ files?/) + // frontend rows use read_app_file's leading-slash addressing + expect(result).toContain('/lib/aggregations.ts') + expect(result).toContain('1: export function computeRevenue(o) {') + expect(result).toContain('/components/SummaryPanel.tsx') + // inline runnable rows use the backend//main. addressing + expect(result).toContain('backend/computeSummary/main.ts') + // generated files are never searched + expect(result).not.toContain('/wmill.d.ts') + }) + + it('matches case-insensitively', async () => { + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(searchAppValue()) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'COMPUTEREVENUE' // upper-case query still matches computeRevenue + }) + + expect(result).toContain('/lib/aggregations.ts') + expect(result).toContain('export function computeRevenue(o) {') + }) + + it('filters by a basename glob (matches nested files)', async () => { + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(searchAppValue()) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'computeRevenue', + file_glob: '*.tsx' + }) + + expect(result).toContain('/components/SummaryPanel.tsx') + expect(result).toContain('/components/OrdersTable.tsx') + expect(result).not.toContain('/lib/aggregations.ts') + expect(result).not.toContain('backend/computeSummary/main.ts') + }) + + it('filters by a path glob (e.g. backend/**)', async () => { + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(searchAppValue()) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'computeRevenue', + file_glob: 'backend/**' + }) + + expect(result).toContain('backend/computeSummary/main.ts') + expect(result).not.toContain('/lib/aggregations.ts') + }) + + it('reports zero matches with a hint instead of an empty result', async () => { + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(searchAppValue()) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'nonexistent_symbol_xyz' + }) + + expect(result).toContain('No matches') + expect(result).toContain('Try a broader') + }) + + it('truncates very long matching lines to keep results sparse', async () => { + const longLine = `const x = "${'q'.repeat(5000)} computeRevenue"` + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ + path: 'f/apps/report', + summary: 'app', + versions: [5], + value: { files: { '/min.js': longLine }, runnables: {}, data: {} } + } as any) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'computeRevenue' + }) + + expect(result).toContain('[line truncated]') + expect(result.length).toBeLessThan(1000) + }) + + it('caps the number of match rows and says it truncated', async () => { + const manyLines = Array.from({ length: 500 }, (_, i) => `hit ${i}`).join('\n') + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ + path: 'f/apps/report', + summary: 'app', + versions: [5], + value: { files: { '/big.tsx': manyLines }, runnables: {}, data: {} } + } as any) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'hit', + max_matches: 50 + }) + + expect(result).toContain('500 matches') + expect(result).toContain('showing the first 50') + // 50 capped match lines, each rendered with its fixed context window (deduped), + // so the body is bounded near max_matches and nowhere near the 500 total. + const rows = result.split('\n').filter((l) => /^\s+\d+: /.test(l)).length + expect(rows).toBeGreaterThanOrEqual(50) + expect(rows).toBeLessThan(60) + }) + + it('counts every file with a match, even matches past the render cap', async () => { + // The first (sorted) file exhausts max_matches; the later file's match falls + // past the cap but the symbol still lives there, so the header must count it. + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ + path: 'f/apps/report', + summary: 'app', + versions: [5], + value: { + files: { '/a.tsx': 'hit\nhit\nhit\nhit\nhit', '/b.tsx': 'hit' }, + runnables: {}, + data: {} + } + } as any) + + const result = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'hit', + max_matches: 3 + }) + + expect(result).toContain('6 matches in 2 files') + expect(result).toContain('showing the first 3') + }) + + // Deterministic micro-benchmark: how much context a single search_app call + // saves over locating a symbol by reading the candidate files whole. Baseline + // is the conservative "read only the files that actually contain the symbol" + // path (a model without search must read at least those in full); the real + // saving is larger because, lacking search, a model often reads non-matching + // files too. Isolated from model nondeterminism so it can gate regressions. + it('micro-benchmark: search_app locates a symbol far cheaper than reading files', async () => { + const fileBodies: Record = {} + // 8 component files, 3 of which reference the symbol, each ~120 lines. + for (let f = 0; f < 8; f++) { + const lines = Array.from({ length: 120 }, (_, i) => + f < 3 && i === 60 ? ` return computeRevenue(order${f})` : ` const v${i} = ${i} // padding` + ) + fileBodies[`/components/File${f}.tsx`] = lines.join('\n') + } + const appValue = { + path: 'f/apps/report', + summary: 'big app', + versions: [5], + value: { + files: fileBodies, + runnables: {}, + data: {} + } + } as any + + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce(appValue) + const searchOut = await callGlobalTool('search_app', { + path: 'f/apps/report', + query: 'computeRevenue' + }) + + // Baseline: the bytes a model must pull to gather the same locations by + // reading each matching file in full. + const matchingFiles = Object.entries(fileBodies).filter(([, body]) => + body.includes('computeRevenue') + ) + const baselineChars = matchingFiles.reduce((sum, [, body]) => sum + body.length, 0) + const actualChars = searchOut.length + const reductionPct = Math.round((1 - actualChars / baselineChars) * 100) + // eslint-disable-next-line no-console + console.log( + `[search_app micro-benchmark] baseline=${baselineChars} chars (read ${matchingFiles.length} files whole), ` + + `actual=${actualChars} chars (one search), reduction=${reductionPct}%` + ) + + // The search surfaced exactly the 3 locations… + expect(matchingFiles.length).toBe(3) + expect((searchOut.match(/computeRevenue/g) ?? []).length).toBeGreaterThanOrEqual(3) + // …at a tiny fraction of reading those files whole. + expect(actualChars).toBeLessThan(baselineChars * 0.15) }) it('does not persist a raw app draft when patch_app_file validation fails', async () => { - vi.mocked(AppService.getAppByPathWithDraft).mockResolvedValueOnce({ + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ path: 'f/apps/report', summary: 'deployed app', versions: [5], @@ -1178,11 +2164,11 @@ describe('global AI tools', () => { replace_all: false }) ).rejects.toThrow() - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() }) it('does not persist a raw app draft when delete_app_file validation fails', async () => { - vi.mocked(AppService.getAppByPathWithDraft).mockResolvedValueOnce({ + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ path: 'f/apps/report', summary: 'deployed app', versions: [5], @@ -1199,11 +2185,11 @@ describe('global AI tools', () => { file_path: '/src/Missing.tsx' }) ).rejects.toThrow('Frontend file "/src/Missing.tsx" not found in app "f/apps/report".') - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() }) it('does not persist a raw app draft when delete_app_runnable validation fails', async () => { - vi.mocked(AppService.getAppByPathWithDraft).mockResolvedValueOnce({ + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ path: 'f/apps/report', summary: 'deployed app', versions: [5], @@ -1225,11 +2211,11 @@ describe('global AI tools', () => { key: 'missing' }) ).rejects.toThrow('Backend runnable "missing" not found in app "f/apps/report".') - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() }) it('deploys a new raw app draft by bundling files and creating a raw app', async () => { - UserDraft.save( + seedBackendDraft( 'raw_app', 'f/apps/report', { @@ -1244,6 +2230,9 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + // getAppByPath resolves with no draft_path → deploy at the item's own path. + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + const raw = await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/report', @@ -1281,7 +2270,7 @@ describe('global AI tools', () => { } }) expect(AppService.updateAppRaw).not.toHaveBeenCalled() - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() expect(JSON.parse(raw)).toMatchObject({ success: true, type: 'app', @@ -1289,9 +2278,96 @@ describe('global AI tools', () => { }) }) + it('deploys an editor raw app draft at its draft_path, not its synthetic storage key', async () => { + // An editor-created draft_only raw app lives at a synthetic storage key with + // its chosen path in `draft_path`; deploy must resolve to the storage key, + // read draft_path, and create the app there — not at the synthetic key. + const storageKey = 'u/admin/draft_app999' + const chosenPath = 'f/team/chosen_app' + seedBackendDraft( + 'raw_app', + storageKey, + { + summary: 'Editor app', + files: { '/App.tsx': 'export default () => null' }, + runnables: {}, + data: { tables: [] }, + draft_path: chosenPath + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: storageKey, + effectivePath: chosenPath + }) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({ + draft: { draft_path: chosenPath } + } as any) + const flushSpy = vi.spyOn(UserDraftDbSyncer, 'flush') + + await callGlobalTool('deploy_workspace_item', { type: 'app', path: chosenPath }) + + // The draft is flushed at the storage key before the draft_path read, so a + // not-yet-saved editor rename isn't read stale. + expect(flushSpy).toHaveBeenCalledWith( + expect.objectContaining({ workspace: WORKSPACE, itemKind: 'raw_app', path: storageKey }) + ) + // draft_path is read from the backend draft at the storage key… + expect(AppService.getAppByPath).toHaveBeenCalledWith( + expect.objectContaining({ + workspace: WORKSPACE, + path: storageKey, + getDraft: true, + rawApp: true + }) + ) + // …and the app is created at the chosen path, not the synthetic key. + expect(AppService.createAppRaw).toHaveBeenCalledWith( + expect.objectContaining({ + formData: expect.objectContaining({ + app: expect.objectContaining({ path: chosenPath }) + }) + }) + ) + }) + + it('aborts a raw app deploy when the draft_path lookup fails (non-404)', async () => { + // A real lookup failure (network/5xx) must abort, not silently fall back to the + // storage path and deploy there. Only a 404 justifies the storage-path fallback. + seedBackendDraft( + 'raw_app', + 'u/admin/draft_appfail', + { + summary: 'Editor app', + files: { '/App.tsx': 'export default () => null' }, + runnables: {}, + data: { tables: [] }, + draft_path: 'f/team/chosen_app' + }, + { workspace: WORKSPACE } + ) + UserDraft.setLiveEditorDraft({ + workspace: WORKSPACE, + itemKind: 'raw_app', + storagePath: 'u/admin/draft_appfail', + effectivePath: 'f/team/chosen_app' + }) + vi.mocked(AppService.getAppByPath).mockRejectedValueOnce( + Object.assign(new Error('server error'), { status: 500 }) + ) + + await expect( + callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/team/chosen_app' }) + ).rejects.toThrow() + expect(AppService.createAppRaw).not.toHaveBeenCalled() + expect(AppService.updateAppRaw).not.toHaveBeenCalled() + }) + it('deploys an existing raw app draft by bundling files and updating the raw app', async () => { vi.mocked(AppService.existsApp).mockResolvedValueOnce(true) - UserDraft.save( + seedBackendDraft( 'raw_app', 'f/apps/report', { @@ -1305,6 +2381,8 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/report' @@ -1330,14 +2408,48 @@ describe('global AI tools', () => { } }) expect(AppService.createAppRaw).not.toHaveBeenCalled() - expect(UserDraft.get('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + expect(getBackendDraft('raw_app', 'f/apps/report', { workspace: WORKSPACE })).toBeUndefined() + }) + + it('forwards preserve_on_behalf_of when the deployed policy carries an on_behalf_of', async () => { + // Without the flag the backend resets the policy's on_behalf_of to the + // deploying user; this chat path has no on-behalf-of selector, so it must + // preserve whatever the carried policy already holds. + vi.mocked(AppService.existsApp).mockResolvedValueOnce(true) + seedBackendDraft( + 'raw_app', + 'f/apps/obo', + { + summary: 'On-behalf app', + files: { '/index.tsx': 'console.log("obo")' }, + runnables: {}, + data: { tables: [] }, + policy: { + execution_mode: 'publisher', + on_behalf_of: 'u/alice', + on_behalf_of_email: 'alice@windmill.dev' + } + }, + { workspace: WORKSPACE } + ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + + await callGlobalTool('deploy_workspace_item', { type: 'app', path: 'f/apps/obo' }) + + expect(AppService.updateAppRaw).toHaveBeenCalledWith( + expect.objectContaining({ + formData: expect.objectContaining({ + app: expect.objectContaining({ preserve_on_behalf_of: true }) + }) + }) + ) }) it('notifies the session preview (as raw_app) after deploying a raw app', async () => { const onDeployed = vi.fn() setDeployedInSessionHandler(onDeployed) try { - UserDraft.save( + seedBackendDraft( 'raw_app', 'f/apps/report', { @@ -1349,6 +2461,8 @@ describe('global AI tools', () => { { workspace: WORKSPACE } ) + vi.mocked(AppService.getAppByPath).mockResolvedValueOnce({} as any) + await callGlobalTool( 'deploy_workspace_item', { type: 'app', path: 'f/apps/report' }, @@ -1463,7 +2577,7 @@ describe('global AI tools', () => { expect(item.value.value).toBeUndefined() }) - it('test_run_script previews local draft script content by path', async () => { + it('test_run_script previews draft script content by path', async () => { const content = 'export async function main(name: string) {\n\treturn `hello ${name}`\n}' await callGlobalTool('write_script', { path: 'f/scripts/draft-test', @@ -1493,7 +2607,7 @@ describe('global AI tools', () => { expect(result).toContain('test logs') }) - it('test_run_script previews deployed script content when no local draft exists', async () => { + it('test_run_script previews deployed script content when no draft exists', async () => { vi.mocked(ScriptService.getScriptByPath).mockResolvedValueOnce({ path: 'f/scripts/deployed-test', summary: 'Deployed test script', @@ -1523,7 +2637,7 @@ describe('global AI tools', () => { }) }) - it('test_run_flow previews local draft flow content by path', async () => { + it('test_run_flow previews draft flow content by path', async () => { const modules = [{ id: 'start', value: { type: 'identity' } }] await callGlobalTool('write_flow', { path: 'f/flows/draft-test', @@ -1549,7 +2663,7 @@ describe('global AI tools', () => { }) }) - it('test_run_flow previews deployed flow content when no local draft exists', async () => { + it('test_run_flow previews deployed flow content when no draft exists', async () => { const modules = [{ id: 'deployed_start', value: { type: 'identity' } }] vi.mocked(FlowService.getFlowByPath).mockResolvedValueOnce({ path: 'f/flows/deployed-test', @@ -1580,7 +2694,7 @@ describe('global AI tools', () => { }) it('test_run_flow uses the live flow editor test hook when the active editor matches the path', async () => { - UserDraft.save( + seedBackendDraft( 'flow', '', { @@ -1622,7 +2736,7 @@ describe('global AI tools', () => { }) it('test_run_flow falls back to preview when the live flow editor test hook returns undefined', async () => { - UserDraft.save( + seedBackendDraft( 'flow', '', { @@ -1669,7 +2783,7 @@ describe('global AI tools', () => { }) }) - it('test_run_step previews rawscript steps from the local draft flow', async () => { + it('test_run_step previews rawscript steps from the draft flow', async () => { const content = 'export async function main(name: string) {\n\treturn name.toUpperCase()\n}' await callGlobalTool('write_flow', { path: 'f/flows/rawscript-step', @@ -1748,7 +2862,7 @@ describe('global AI tools', () => { }) }) - it('test_run_step previews local draft subflows for flow steps', async () => { + it('test_run_step previews draft subflows for flow steps', async () => { const nestedModules = [{ id: 'nested_start', value: { type: 'identity' } }] await callGlobalTool('write_flow', { path: 'f/flows/nested-draft', @@ -1934,14 +3048,62 @@ describe('global AI tools', () => { }) }) +describe('folder tools', () => { + beforeEach(() => { + vi.clearAllMocks() + userStore.set(undefined) + }) + afterEach(() => { + userStore.set(undefined) + }) + + it('create_folder requires confirmation', () => { + const tool = getGlobalTool('create_folder') + expect(tool.requiresConfirmation).toBe(true) + expect(tool.confirmationMessage).toBe('Create folder') + }) + + it('create_folder creates the folder and reflects it in the path context', async () => { + userStore.set({ username: 'bob', is_admin: false, folders: ['existing'] } as any) + const raw = await callGlobalTool('create_folder', { name: 'analytics', summary: 'team data' }) + + expect(vi.mocked(FolderService.createFolder)).toHaveBeenCalledWith({ + workspace: WORKSPACE, + requestBody: { name: 'analytics', summary: 'team data' } + }) + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(true) + expect(parsed.message).toContain('f/analytics') + expect((get(userStore) as any)?.folders).toContain('analytics') + }) + + it('create_folder rejects an invalid name without calling the API', async () => { + const raw = await callGlobalTool('create_folder', { name: 'bad name!' }) + expect(vi.mocked(FolderService.createFolder)).not.toHaveBeenCalled() + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(false) + expect(parsed.error).toContain('alphanumeric') + }) + + it('create_folder surfaces a backend error (e.g. name conflict)', async () => { + vi.mocked(FolderService.createFolder).mockRejectedValueOnce( + new Error('Folder already exists') + ) + const raw = await callGlobalTool('create_folder', { name: 'taken' }) + const parsed = JSON.parse(raw) + expect(parsed.success).toBe(false) + expect(parsed.error).toContain('Folder already exists') + }) +}) + describe('prepareGlobalSystemMessage', () => { it('keeps global chat draft instructions concise and user-facing', () => { const message = prepareGlobalSystemMessage() const content = message.content - expect(content).toContain('Draft tools create or update local drafts only') + expect(content).toContain('Draft tools create or update drafts only') expect(content).toContain( - 'Use discard_local_draft to remove an unsaved local draft, including the matching open editor draft' + 'Use discard_local_draft to remove a draft, including the matching open editor draft' ) expect(content).toContain( 'After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step' @@ -1953,12 +3115,110 @@ describe('prepareGlobalSystemMessage', () => { expect(content).not.toContain('frontend AI draft store') }) + it('honors user-supplied shared folder paths without asking first', () => { + const content = prepareGlobalSystemMessage(undefined, { + user: { username: 'admin', is_admin: true, folders: ['evals'] } + }).content as string + + expect(content).toContain( + 'If the user supplies a fully qualified `f//...` path, use that exact path' + ) + expect(content).toContain('Do not ask for folder confirmation') + expect(content).toContain('substitute a `u/admin/...` path unless a tool rejects it') + }) + + it('tells the model to create a folder only when the user explicitly asks', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).toContain( + 'create one with `create_folder` only when the user explicitly asks for a new folder' + ) + }) + + describe('folder guidance', () => { + const guidanceOf = (user: { + username: string + is_admin?: boolean + folders?: string[] + folders_read?: string[] + }) => prepareGlobalSystemMessage(undefined, { user }).content as string + + it('lists the writable folders for a non-admin', () => { + const content = guidanceOf({ + username: 'bob', + is_admin: false, + folders: ['marketing', 'data_engineering'], + folders_read: ['marketing', 'data_engineering'] + }) + expect(content).toContain( + 'Folders you can write to in this workspace: `f/marketing`, `f/data_engineering`.' + ) + expect(content).not.toContain('You can see but NOT write to') + }) + + it('flags read-only folders a non-admin cannot write to', () => { + const content = guidanceOf({ + username: 'bob', + is_admin: false, + folders: ['team_a'], + folders_read: ['team_a', 'team_b'] + }) + expect(content).toContain('Folders you can write to in this workspace: `f/team_a`.') + expect(content).toContain( + 'You can see but NOT write to: `f/team_b` — never create or deploy items there.' + ) + }) + + it('points a non-admin with no writable folders at the personal scope', () => { + const content = guidanceOf({ username: 'bob', is_admin: false, folders: [] }) + expect(content).toContain( + 'You have no shared folders you can write to in this workspace, so use `u/bob/`.' + ) + }) + + it('gives an admin permission-agnostic guidance with a non-exhaustive hint', () => { + const content = guidanceOf({ + username: 'admin', + is_admin: true, + folders: ['marketing', 'data_engineering'] + }) + expect(content).toContain('As a workspace admin you can write to any existing folder.') + expect(content).toContain( + 'Folders here include `f/marketing`, `f/data_engineering` (you can also write to others not listed).' + ) + expect(content).toContain( + 'If the user names a folder, use it; if they explicitly ask for a new folder, create it with `create_folder`; otherwise ask them which folder to use rather than guessing or creating one unprompted.' + ) + expect(content).not.toContain('Folders you can write to in this workspace') + }) + + it('omits the folder hint for an admin with no associated folders', () => { + const content = guidanceOf({ username: 'admin', is_admin: true, folders: [] }) + expect(content).toContain( + '- As a workspace admin you can write to any existing folder. If the user names a folder, use it; if they explicitly ask for a new folder, create it with `create_folder`; otherwise ask them which folder to use rather than guessing or creating one unprompted.' + ) + expect(content).not.toContain('Folders here include') + }) + + it('caps the folder list and notes the remainder', () => { + const folders = Array.from({ length: 45 }, (_, i) => `f${i}`) + const content = guidanceOf({ username: 'bob', is_admin: false, folders }) + expect(content).toContain('(+5 more)') + }) + + it('emits no folder guidance when no user is available', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).not.toContain('Folders you can write to in this workspace') + expect(content).not.toContain('As a workspace admin you can write to any existing folder') + expect(content).not.toContain('You have no shared folders you can write to') + }) + }) + it('exposes separate tools for discarding drafts and deleting workspace items', () => { const discard = getGlobalTool('discard_local_draft') const deleteItem = getGlobalTool('delete_workspace_item') expect(discard.def.function.description).toBe( - 'Discard a local draft only. Does not mutate deployed workspace items, but clears the matching open editor draft if one is mounted.' + 'Discard a draft only. Does not mutate deployed workspace items, but clears the matching open editor draft if one is mounted.' ) expect(deleteItem.def.function.description).toBe( 'Delete a deployed workspace item. Mutates the workspace.' @@ -2059,7 +3319,8 @@ describe('prepareGlobalSystemMessage', () => { const handler = vi.fn(() => ({ aiResult: 'runs output. Next step: call get_job_logs.', uiMessage: 'Listed 1 app run', - toolResult: '[{"job_id":"job-1","component":"backend.1","status":"completed","created_at":1718000000000,"started_at":1718000000000,"duration_ms":1000}]' + toolResult: + '[{"job_id":"job-1","component":"backend.1","status":"completed","created_at":1718000000000,"started_at":1718000000000,"duration_ms":1000}]' })) setListAppRunsHandler(handler) const result = await callGlobalTool('list_app_runs', {}, callbacks, { @@ -2078,7 +3339,8 @@ describe('prepareGlobalSystemMessage', () => { const handler = vi.fn(() => ({ aiResult: 'runs output', uiMessage: 'Listed app runs', - toolResult: '[{"job_id":"job-1","component":"backend.1","status":"completed","created_at":1718000000000,"started_at":1718000000000,"duration_ms":1000}]' + toolResult: + '[{"job_id":"job-1","component":"backend.1","status":"completed","created_at":1718000000000,"started_at":1718000000000,"duration_ms":1000}]' })) setListAppRunsHandler(handler) await callGlobalTool('list_app_runs', { limit: 5 }, toolCallbacks, { @@ -2123,6 +3385,156 @@ describe('session-only preview tools gating', () => { expect(on).toContain('get_app_runtime_logs') expect(on).toContain('list_app_runs') }) + + // The instruction headers are matched by their distinctive parenthetical so the + // guidance bullet (which references both block names) doesn't false-positive. + const WS_HEADER = 'WORKSPACE INSTRUCTIONS (configured by a workspace admin' + const USER_HEADER = "USER INSTRUCTIONS (this user's personal instructions" + + it('renders only the workspace block when given workspace instructions', () => { + const content = prepareGlobalSystemMessage({ workspace: 'Always be terse.' }).content as string + expect(content).toContain(WS_HEADER) + expect(content).toContain('Always be terse.') + expect(content).not.toContain(USER_HEADER) + }) + + it('renders the user block with the edit-tool mention when given user instructions', () => { + const content = prepareGlobalSystemMessage({ user: 'Prefer Bun for new scripts.' }) + .content as string + expect(content).toContain(USER_HEADER) + expect(content).toContain('update_user_instructions') + expect(content).toContain('Prefer Bun for new scripts.') + expect(content).not.toContain(WS_HEADER) + }) + + it('renders the workspace block before the user block when both are present', () => { + const content = prepareGlobalSystemMessage({ workspace: 'WS rule.', user: 'User rule.' }) + .content as string + expect(content).toContain(WS_HEADER) + expect(content.indexOf(USER_HEADER)).toBeGreaterThan(content.indexOf(WS_HEADER)) + }) + + it('omits both instruction headers when none are provided', () => { + const content = prepareGlobalSystemMessage().content as string + expect(content).not.toContain(WS_HEADER) + expect(content).not.toContain(USER_HEADER) + }) +}) + +describe('update_user_instructions', () => { + function makeHelpers(initial = '') { + let value = initial + return { + getUserInstructions: () => value, + setUserInstructions: vi.fn((v: string) => { + value = v + }) + } + } + + it('appends to empty instructions', async () => { + const helpers = makeHelpers('') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Prefer Bun for new scripts.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Prefer Bun for new scripts.') + expect(res).toContain('Added a personal instruction') + }) + + it('appends to existing instructions joined by a blank line', async () => { + const helpers = makeHelpers('Existing rule.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Another rule.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Existing rule.\n\nAnother rule.') + }) + + it('returns only a short confirmation, not the resulting instructions', async () => { + const helpers = makeHelpers('Existing rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'Another rule.' }, + toolCallbacks, + helpers + ) + expect(res).not.toContain('Existing rule.') + expect(res).not.toContain('Another rule.') + }) + + it('replaces an exact match', async () => { + const helpers = makeHelpers('Prefer Bun.\n\nUse tabs.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'Prefer Bun.', new_string: 'Prefer Deno.' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Prefer Deno.\n\nUse tabs.') + }) + + it('removes the matched text when new_string is empty', async () => { + const helpers = makeHelpers('Keep this.\n\nDrop this.') + await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: '\n\nDrop this.', new_string: '' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('Keep this.') + }) + + it('clears all instructions when the whole text is replaced with empty', async () => { + const helpers = makeHelpers('Only rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'Only rule.', new_string: '' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).toHaveBeenCalledWith('') + expect(res).toContain('Cleared your personal instructions') + }) + + it('errors without writing when old_string is not found, and echoes the current text for recovery', async () => { + const helpers = makeHelpers('Existing rule.') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'replace', old_string: 'missing', new_string: 'x' }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).not.toHaveBeenCalled() + expect(res).toContain('not found') + expect(res).toContain('Existing rule.') + }) + + it('rejects a result over the length cap without writing', async () => { + const helpers = makeHelpers('') + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'a'.repeat(5001) }, + toolCallbacks, + helpers + ) + expect(helpers.setUserInstructions).not.toHaveBeenCalled() + expect(res).toContain('over the 5000') + }) + + it('fails gracefully when the context does not provide instruction helpers', async () => { + const res = await callGlobalTool( + 'update_user_instructions', + { operation: 'append', text: 'x' }, + toolCallbacks, + {} + ) + expect(res).toContain('cannot modify user instructions') + }) }) describe('prepareGlobalUserMessage', () => { @@ -2160,15 +3572,23 @@ describe('prepareGlobalUserMessage', () => { path: 'f/flows/reporting', title: 'f/flows/reporting', summary: 'Reporting flow' + }, + { + type: 'workspace_app', + path: 'f/apps/dashboard', + title: 'f/apps/dashboard', + summary: 'Dashboard raw app' } ]) expect(message.content).toContain('## SELECTED CONTEXT') expect(message.content).toContain('- type: script, path: f/scripts/report') expect(message.content).toContain('- type: flow, path: f/flows/reporting') + expect(message.content).toContain('- type: raw_app, path: f/apps/dashboard') expect(message.content).toContain('## INSTRUCTIONS:\nUpdate these items') expect(message.content).not.toContain('Report script') expect(message.content).not.toContain('Reporting flow') + expect(message.content).not.toContain('Dashboard raw app') }) it('omits selected context section when no workspace item is selected', () => { diff --git a/frontend/src/lib/components/copilot/chat/global/core.ts b/frontend/src/lib/components/copilot/chat/global/core.ts index 7c3040e284..b85e271632 100644 --- a/frontend/src/lib/components/copilot/chat/global/core.ts +++ b/frontend/src/lib/components/copilot/chat/global/core.ts @@ -2,6 +2,7 @@ import { AppService, AzureTriggerService, FlowService, + FolderService, GcpTriggerService, HttpTriggerService, JobService, @@ -14,8 +15,10 @@ import { ScriptService, SqsTriggerService, VariableService, - WebsocketTriggerService + WebsocketTriggerService, + WorkspaceService } from '$lib/gen' +import { createTwoFilesPatch } from 'diff' import { $ScriptLang } from '$lib/gen/schemas.gen' import type { AppWithLastVersion, @@ -42,6 +45,7 @@ import { type FrameworkKey } from '$lib/components/raw_apps/templates' import { DEFAULT_DATA as DEFAULT_RAW_APP_DATA } from '$lib/components/raw_apps/dataTableRefUtils' +import { appSourceToDraftValue } from '$lib/components/raw_apps/rawAppDraftValue' import { applyEditableFlowJsonToFlow, buildEditableFlowJson, @@ -72,9 +76,12 @@ import { type ToolCallbacks, type ToolDisplayAction } from '../shared' +import { searchDocsTool, readDocsPageTool } from '../docs/core' import type { ContextElement } from '../context' import { getDatatableTools } from '../datatableTools' -import { UserDraft, type UserDraftMeta } from '$lib/userDraft.svelte' +import { fileTools } from '../files/fileTools' +import type { AttachedFilesStore } from '../files/attachedFiles.svelte' +import { UserDraft } from '$lib/userDraft.svelte' import { emptySchema } from '$lib/utils' import { inferArgs } from '$lib/infer' import { @@ -95,9 +102,10 @@ import { type WorkspaceItem, type WorkspaceItemType } from './workspaceItems' -import { buildFlowDeployRequestBody, buildScriptDeployRequestBody } from './deployRequests' import { userStore } from '$lib/stores' import { get } from 'svelte/store' +import { deployDraft as deployDraftToWorkspace } from '$lib/utils_draft_deploy' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { bundleRawAppDraft } from './rawAppBundlerBridge' import { clearEphemeralSecretVariableDraftValue, @@ -106,9 +114,11 @@ import { getGlobalDraft, getGlobalDraftStoragePath, listGlobalDrafts, + persistGlobalDraft, + readGlobalDraftValue, saveGlobalAppDraft, setEphemeralSecretVariableDraftValue, - triggerKindToUserDraftKind + type DraftPersistResult } from './userDraftAdapter' const ITEM_TYPES = [ @@ -129,10 +139,7 @@ const INSTRUCTION_SUBJECTS = [ // `datatable` is not a workspace item type, but the model can request the // datatable SDK reference (the wmill.datatable() runnable API) the same way. const INSTRUCTION_SUBJECTS_EXTRA = ['datatable'] as const -const ALL_INSTRUCTION_SUBJECTS = [ - ...INSTRUCTION_SUBJECTS, - ...INSTRUCTION_SUBJECTS_EXTRA -] as const +const ALL_INSTRUCTION_SUBJECTS = [...INSTRUCTION_SUBJECTS, ...INSTRUCTION_SUBJECTS_EXTRA] as const const MAX_LIST_LIMIT = 100 type ActiveGlobalEditorType = Extract type LiveEditorDraftKind = Parameters[0] @@ -141,10 +148,10 @@ const ACTIVE_GLOBAL_EDITOR_DRAFTS: readonly { itemKind: LiveEditorDraftKind type: ActiveGlobalEditorType }[] = [ - { itemKind: 'script', type: 'script' }, - { itemKind: 'flow', type: 'flow' }, - { itemKind: 'raw_app', type: 'app' } - ] + { itemKind: 'script', type: 'script' }, + { itemKind: 'flow', type: 'flow' }, + { itemKind: 'raw_app', type: 'app' } +] export type GlobalActiveEditorContext = { type: ActiveGlobalEditorType @@ -164,7 +171,7 @@ const scriptLangSchema = z.enum($ScriptLang.enum) const getInstructionsSchema = z.object({ subject: instructionSubjectSchema.describe( - "What to get authoring instructions for: a workspace item type (script, flow, resource, app) or \"datatable\" for the wmill.datatable() SQL SDK used inside runnables. Schedules, triggers, and variables don't need instructions — their tool schemas describe everything." + 'What to get authoring instructions for: a workspace item type (script, flow, resource, app) or "datatable" for the wmill.datatable() SQL SDK used inside runnables. Schedules, triggers, and variables don\'t need instructions — their tool schemas describe everything.' ), language: scriptLangSchema .optional() @@ -185,6 +192,43 @@ const askUserQuestionSchema = z.object({ .describe('Two to ten mutually exclusive proposed answer strings.') }) +// Matches the per-mode cap enforced by the prompt-settings UI (AIPromptsModal) and +// the backend workspace prompt (MAX_CUSTOM_PROMPT_LENGTH). +export const MAX_USER_INSTRUCTIONS_LENGTH = 5000 + +const updateUserInstructionsSchema = z.object({ + operation: z + .enum(['append', 'replace']) + .describe( + 'What to do with your personal Global instructions. \'append\' adds a new instruction to the end (use for "remember this" / "always do X"). \'replace\' performs an exact find-and-replace to edit or remove existing text (use for "change X" / "stop doing Y"); set new_string to "" to remove.' + ), + text: z + .string() + .min(1) + .optional() + .describe("Required when operation is 'append': the instruction to add. Ignored for 'replace'."), + old_string: z + .string() + .min(1) + .optional() + .describe( + "Required when operation is 'replace': exact text to find in your current personal instructions. Ignored for 'append'." + ), + new_string: z + .string() + .optional() + .describe( + "Required when operation is 'replace': replacement text. Use an empty string to delete the matched text. Ignored for 'append'." + ), + replace_all: z + .boolean() + .optional() + .default(false) + .describe( + "For operation 'replace': when true, replace every exact match; when false, old_string must match exactly once." + ) +}) + const listWorkspaceItemsSchema = z.object({ types: z .array(itemTypeSchema) @@ -214,11 +258,19 @@ const readWorkspaceItemSchema = z.object({ .describe('Required when type is trigger. Identifies which trigger service to call.') }) +const draftOverrideField = z + .boolean() + .optional() + .describe( + 'Overwrite the server draft even if it changed externally since you last read it (resolve a save conflict, your version wins).' + ) + const writeScriptSchema = z.object({ path: z.string().describe('Workspace path of the script, e.g. f/folder/name or u/user/name.'), summary: z.string().optional().describe('Short human-readable summary.'), language: scriptLangSchema.describe('Script language.'), - content: z.string().describe('Full script source code.') + content: z.string().describe('Full script source code.'), + override: draftOverrideField }) const readFlowModuleCodeSchema = z.object({ @@ -248,6 +300,12 @@ const setFlowModuleCodeSchema = z.object({ const writeFlowSchema = z.object({ path: z.string().describe('Workspace path of the flow, e.g. f/folder/name or u/user/name.'), summary: z.string().optional().describe('Short human-readable summary.'), + description: z + .string() + .optional() + .describe( + 'Longer human-readable description of what the flow does. Top-level flow metadata, separate from the modules — not part of the compact value patched by patch_flow_json.' + ), modules: z.string().describe('JSON string containing the complete flow modules array.'), schema: z .string() @@ -270,7 +328,8 @@ const writeFlowSchema = z.object({ .nullable() .describe( 'JSON string containing the optional array of semantic flow groups. Pass null to clear groups.' - ) + ), + override: draftOverrideField }) function parseOptionalJsonArg(value: unknown, field: string): unknown { @@ -313,7 +372,7 @@ function flowDraftAsEditableInput(flowDraft: FlowDraftValue): { } } -const writeScheduleSchema = scheduleRequestSchema +const writeScheduleSchema = scheduleRequestSchema.extend({ override: draftOverrideField }) const writeTriggerSchema = z.object({ kind: triggerKindSchema.describe('Trigger kind. Determines which fields are valid in config.'), @@ -331,12 +390,13 @@ const writeTriggerSchema = z.object({ ]) .describe( 'Full trigger configuration. Must include path, script_path, is_flow plus the kind-specific fields.' - ) + ), + override: draftOverrideField }) -const writeResourceSchema = resourceRequestSchema +const writeResourceSchema = resourceRequestSchema.extend({ override: draftOverrideField }) -const writeVariableSchema = variableRequestSchema +const writeVariableSchema = variableRequestSchema.extend({ override: draftOverrideField }) const searchResourceTypesSchema = z.object({ query: z.string().describe('Substring to match against resource type names.'), @@ -354,14 +414,8 @@ const getJobLogsSchema = z.object({ }) const listRunsSchema = z.object({ - path: z - .string() - .optional() - .describe('Filter to runs of this exact script or flow path.'), - created_by: z - .string() - .optional() - .describe('Filter by the username that started the run.'), + path: z.string().optional().describe('Filter to runs of this exact script or flow path.'), + created_by: z.string().optional().describe('Filter by the username that started the run.'), label: z.string().optional().describe('Filter by job label.'), success: z .boolean() @@ -387,7 +441,7 @@ const deleteWorkspaceItemSchema = z.object({ const discardLocalDraftSchema = z.object({ type: itemTypeSchema, - path: z.string().describe('Workspace path of the local draft to discard.'), + path: z.string().describe('Workspace path of the draft to discard.'), trigger_kind: triggerKindSchema .optional() .describe('Required when type is trigger. Must match the draft trigger kind.') @@ -402,7 +456,20 @@ const deployWorkspaceItemSchema = z.object({ deployment_message: z .string() .optional() - .describe('Optional deployment message recorded with the change.') + .describe('Optional deployment message recorded with the change.'), + force: z + .boolean() + .optional() + .describe( + 'Deploy even if the draft was started from an older deployed version, overwriting the version deployed since. Defaults to false; prefer calling rebase_draft first to keep the newer changes.' + ) +}) + +const rebaseDraftSchema = z.object({ + type: itemTypeSchema, + path: z + .string() + .describe('Workspace path of the draft to rebase onto the latest deployed version.') }) const editScriptSchema = z.object({ @@ -448,7 +515,7 @@ const testRunScriptSchema = z.object({ const testRunScriptToolDef = createToolDef( testRunScriptSchema, 'test_run_script', - 'Execute a preview-style test run of a script by path, preferring local draft content when it exists.', + 'Execute a preview-style test run of a script by path, preferring draft content when it exists.', { strict: false } ) @@ -460,7 +527,7 @@ const testRunFlowSchema = z.object({ const testRunFlowToolDef = createToolDef( testRunFlowSchema, 'test_run_flow', - 'Execute a preview-style test run of a flow by path, preferring local draft content when it exists.', + 'Execute a preview-style test run of a flow by path, preferring draft content when it exists.', { strict: false } ) @@ -473,7 +540,7 @@ const testRunStepSchema = z.object({ const testRunStepToolDef = createToolDef( testRunStepSchema, 'test_run_step', - 'Execute a test run of one step in a flow by path, preferring local draft flow/script content when it exists.', + 'Execute a test run of one step in a flow by path, preferring draft flow/script content when it exists.', { strict: false } ) @@ -517,6 +584,43 @@ const readAppFileSchema = z.object({ .string() .describe( 'Frontend file path like /index.tsx, or backend inline runnable path like backend//main.ts (or main.py).' + ), + offset: z + .number() + .int() + .min(1) + .optional() + .describe('1-based line number to start reading from. Use to page through a large file.'), + limit: z + .number() + .int() + .min(1) + .optional() + .describe( + 'Maximum number of lines to return. Large files are truncated by default; pass offset/limit to read a specific line range.' + ) +}) + +const searchAppSchema = z.object({ + path: z.string().describe('Workspace path of the app, e.g. f/folder/name.'), + query: z + .string() + .describe( + 'Literal substring to find across all app files (case-insensitive) — not a regex, so spaces and operators match verbatim. Returns matching file:line rows, not file bodies. To find call sites and skip similarly-named helpers, include the call paren (e.g. "formatCurrency(" matches calls but not "formatCurrencyPrecise"). Then read_app_file to inspect the ranges.' + ), + file_glob: z + .string() + .optional() + .describe( + 'Optional path filter. A pattern without "/" matches the file name anywhere (e.g. "*.tsx"); a pattern with "/" matches the full path (e.g. "/components/*", "backend/**"). Supports * (any chars except /), ** (any chars), and ?.' + ), + max_matches: z + .number() + .int() + .min(1) + .optional() + .describe( + 'Maximum number of matching lines to return (default 100, hard cap 200); each is shown with a few lines of surrounding context, so the output has more rows than this.' ) }) @@ -630,43 +734,117 @@ const initAppSchema = z.object({ ) }) +// Mirrors the backend VALID_FOLDER_NAME check so an invalid name fails before the +// network round-trip (the server enforces the same rule). +const VALID_FOLDER_NAME = /^[a-zA-Z_0-9-]+$/ + +const createFolderSchema = z.object({ + name: z + .string() + .describe( + 'Folder name — letters, digits, underscores or hyphens only. The folder becomes addressable as `f//`.' + ), + summary: z.string().optional().describe('Optional human-readable description of the folder.') +}) + +type FolderPromptContext = { folders: string[]; foldersRead: string[]; isAdmin: boolean } + +// Renders the folders the current user can act on into the system prompt so the +// model can pick an `f//...` path without a discovery round-trip (there +// is no folder-listing tool). For a non-admin, `folders` (from whoami) is exactly +// the writable set, so read-only folders are listed separately as off-limits. +// Admins bypass folder ACLs and can write anywhere, but `folders` only carries +// their explicitly-permissioned subset, so for admins it is offered as a +// non-exhaustive hint alongside permission-agnostic guidance (the complete set +// needs a folder-listing tool — follow-up). +// Capped so a folder-heavy workspace can't dominate the prompt. +function buildFolderGuidance(username: string, ctx?: FolderPromptContext): string { + if (!ctx) return '' + const MAX = 40 + const writable = ctx.folders ?? [] + const fmt = (names: string[]) => { + const shown = names + .slice(0, MAX) + .map((n) => `\`f/${n}\``) + .join(', ') + return names.length > MAX ? `${shown} (+${names.length - MAX} more)` : shown + } + if (ctx.isAdmin) { + const known = + writable.length > 0 + ? ` Folders here include ${fmt(writable)} (you can also write to others not listed).` + : '' + return `- As a workspace admin you can write to any existing folder.${known} If the user names a folder, use it; if they explicitly ask for a new folder, create it with \`create_folder\`; otherwise ask them which folder to use rather than guessing or creating one unprompted.` + } + const readOnly = (ctx.foldersRead ?? []).filter((f) => !writable.includes(f)) + const lines: string[] = [] + if (writable.length > 0) { + lines.push( + `- Folders you can write to in this workspace: ${fmt(writable)}. For shared/team work, pick the one whose purpose matches the request; if none clearly fits, ask which folder to use (askUserQuestion) rather than inventing a path. Use \`create_folder\` only when the user explicitly asks for a new folder.` + ) + } else { + lines.push( + `- You have no shared folders you can write to in this workspace, so use \`u/${username}/\`. If the user explicitly asks for a shared folder, create one with \`create_folder\` (you become an owner); otherwise ask before placing shared work rather than inventing an \`f//...\` path.` + ) + } + if (readOnly.length > 0) { + lines.push( + `- You can see but NOT write to: ${fmt(readOnly)} — never create or deploy items there.` + ) + } + return lines.join('\n') +} + const buildGlobalSystemPrompt = ( username: string, - previewTools: boolean -) => `You are Windmill's global workspace assistant. + previewTools: boolean, + folderCtx?: FolderPromptContext, + skills: AiSkillListItem[] = [] +) => { + const folderGuidance = buildFolderGuidance(username, folderCtx) + const folderGuidanceBlock = folderGuidance ? `\n${folderGuidance}` : '' + return `You are Windmill's global workspace assistant. The current user's workspace username is "${username}". -Use tools to inspect workspace items and create local drafts for scripts, flows, schedules, triggers, resources, variables, and raw apps. +Use tools to inspect workspace items and create per-user drafts (saved server-side, visible only to this user — not deployed) for scripts, flows, schedules, triggers, resources, variables, and raw apps. Path conventions: -- Every workspace path has exactly three segments and starts with one of two namespaces: - - \`u/${username}/\` — the current user's personal scope. Default for ad-hoc, exploratory, or scratch work. - - \`f//\` — a shared folder scope. The folder must already exist; bare \`f/\` is INVALID and will fail. -- When the user gives a bare name without a namespace prefix (e.g. "create a flow called myflow"), default to \`u/${username}/\`. Do NOT invent \`f/\` — that is a structurally invalid path. -- If the request implies shared / team work but doesn't name a specific folder (e.g. "the marketing flow"), ask which folder to use rather than guessing. Call \`list_workspace_items\` with \`type: ['folder']\` (or rely on the user's hint) before assuming a folder exists. -- Only use an \`f//\` path when the user explicitly named the folder or you confirmed it exists. +- A workspace path starts with one of two namespaces; its trailing may itself contain "/", so a path has three or more segments: + - \`u/${username}/\` — your personal scope. Default for ad-hoc, exploratory, or scratch work. + - \`f//\` — a shared folder scope; the must already exist (a bare \`f/\` with no folder segment is INVALID and will fail). +- If the user supplies a fully qualified \`f//...\` path, use that exact path; they have already chosen the folder. Do not ask for folder confirmation or substitute a \`u/${username}/...\` path unless a tool rejects it. +- Default a bare name with no namespace prefix (e.g. "create a flow called myflow") to \`u/${username}/\`. Never invent an \`f//...\` path for a folder that does not exist; create one with \`create_folder\` only when the user explicitly asks for a new folder.${folderGuidanceBlock} Rules: -- Draft tools create or update local drafts only; they do not deploy or mutate deployed workspace items. +- Draft tools create or update drafts only; they do not deploy or mutate deployed workspace items. - Use list_workspace_items to find items and read_workspace_item before changing an existing item. For triggers, pass trigger_kind. - If the user message includes an ACTIVE EDITOR section, treat it as the currently open item and use it for references like "this", "current", or "open editor". -- Use deploy_workspace_item only after the user explicitly asks to deploy. It persists a local draft to the workspace. -- Use discard_local_draft to remove an unsaved local draft, including the matching open editor draft. Use delete_workspace_item only to delete a deployed workspace item. +- Use deploy_workspace_item only after the user explicitly asks to deploy. It persists a draft to the workspace. +- Use discard_local_draft to remove a draft, including the matching open editor draft. Use delete_workspace_item only to delete a deployed workspace item. - Variable values are never readable. For secrets, create a secret variable and reference it from resources as "$var:path/to/variable". - Use search_resource_types before write_resource. - Use get_instructions before writing scripts, flows, resources, or apps. For scripts, pass the target language. -- After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step with representative args before reporting that it works. These tools prefer local drafts, so testing does not require deployment. +- After creating or editing a script or flow draft, run test_run_script, test_run_flow, or test_run_step with representative args before reporting that it works. These tools prefer drafts, so testing does not require deployment. - Use list_runs to find recent runs (optionally filtered by path, creator, label, or status), then get_job_logs with a returned id to inspect a specific run's logs — without starting a new test run. - When a required decision is ambiguous, use askUserQuestion with two to ten clear proposed answer strings instead of guessing. The user can also type a custom answer when none of the proposed answers fit. -- Keep context targeted.${previewTools - ? ` +- When the user asks you to remember a lasting preference, always/never do something, or change/stop a behavior going forward, call update_user_instructions to persist it. It edits only the USER INSTRUCTIONS block (not WORKSPACE INSTRUCTIONS). Keep each instruction concise; do not use it for one-off requests scoped to the current task. +- Keep context targeted.${ + previewTools + ? ` - After writing or substantially editing a script / flow / app draft, show it via open_preview(kind, path) so the user sees the editor and live preview right next to the chat. First check whether it is already shown: if unsure, call get_preview_status. Only call open_preview (or offer to) when no preview is open or it is showing a different item — don't re-open a preview already showing the item you just edited. - When debugging a running raw app, call get_app_runtime_logs to read the live preview's browser console output. It needs the raw app preview open (open_preview kind="raw_app"). - get_app_runtime_logs only shows the app's browser console. For the server-side logs of a backend runnable the app invoked (a backend. call), call list_app_runs to get that run's job_id from the live preview, then get_job_logs with it. Use this when a backend call errors or returns something unexpected.` - : '' + : '' } +Documentation: +- Use search_docs to look up how a Windmill feature works in the official documentation (a flag, concept, function, or "does Windmill support X") instead of guessing about product behavior. It returns matching doc snippets with their Source URL; call read_docs_page with a Source URL to read the full page (or a section, if it returns headings). Cite the Source URL when you rely on it. +- Complete your response with precisions about how it works based on the documentation. Also drop a link to the relevant documentation if possible. +- If the user asks about something that you are unsure about, say that you are not sure about the answer and suggest to ask the question to the windmill team. +- If the first search returns nothing useful, retry with different or broader keywords before giving up. +- If the documentation does not cover the user's question, say so clearly rather than inventing an answer, and suggest asking the Windmill team. + Flows: - read_workspace_item returns compact flow JSON. Inline script bodies appear as "inline_script.". - Use read_flow_module_code and set_flow_module_code for inline script bodies. @@ -684,7 +862,17 @@ Data Tables: - Use list_datatables to discover the available datatables and their tables. Reuse an existing table rather than creating a duplicate. If list_datatables reports none, this is a blocking prerequisite — tell the user to set up a datatable in their workspace settings and stop; do not assume a "main" datatable exists or call exec_datatable_sql. - Use get_datatable_table_schema only when you need a table's column names/types; list_datatables is enough for table-list or availability summaries. - Use exec_datatable_sql to explore data, run queries, mutate rows, or change schema (CREATE/ALTER/DROP). Creating a table is a normal CREATE TABLE statement — it appears in list_datatables afterward, with no registration step. -- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.` +- When writing runnable code (inline app runnables, scripts, flow modules) that reads or writes datatable data at runtime, it accesses a datatable via wmill.datatable(). Default to TypeScript (bun) unless the user asked for another language. Call get_instructions with subject "datatable" and language "bun" for the TypeScript SQL SDK reference (or language "python3" for Python) — it returns only that language so you get just what you need.${ + skills.length > 0 + ? ` + +Skills: +- Skills are reusable instruction sets curated for this workspace, each covering a specific kind of task. The available skills are listed below by name and description. +- When a user's request matches a skill's description, call read_skill with its exact name to load the full instructions BEFORE acting, then follow them. +${skills.map((s) => `- ${s.name}: ${s.description}`).join('\n')}` + : '' + }` +} const DEFAULT_LIST_TYPES = ['script', 'flow'] as const satisfies readonly WorkspaceItemType[] @@ -894,11 +1082,11 @@ function buildPersistedRunnable( ): PersistedRunnable { const fields = input.staticInputs ? Object.fromEntries( - Object.entries(input.staticInputs).map(([k, v]) => [ - k, - { type: 'static', value: v, fieldType: 'object' } - ]) - ) + Object.entries(input.staticInputs).map(([k, v]) => [ + k, + { type: 'static', value: v, fieldType: 'object' } + ]) + ) : (existing?.fields ?? {}) if (input.type === 'inline') { @@ -954,7 +1142,6 @@ type AppMetadata = { type LoadedAppDraftValue = { value: AppDraftValue - meta?: UserDraftMeta } function summarizeAppValue(value: AppDraftValue): AppMetadata { @@ -1063,73 +1250,32 @@ function getInlineRunnableContent( return { content: runnable.inlineScript?.content ?? '', runnable } } -function normalizeRawAppData(value: Record): AppDraftValue['data'] { - if (value.data?.creation) { - return { - tables: value.data.tables ?? [], - datatable: value.data.creation.datatable, - schema: value.data.creation.schema - } - } - if (value.data) { - return value.data - } - if (value.datatables) { - return { ...DEFAULT_RAW_APP_DATA, tables: value.datatables } - } - if (value.dataTableRefs) { - return { ...DEFAULT_RAW_APP_DATA, tables: value.dataTableRefs } - } - return { ...DEFAULT_RAW_APP_DATA } -} - -function appSourceToDraftValue(app: any, fallback?: any): AppDraftValue { - const value = (app.value ?? {}) as Record - return { - summary: app.summary ?? '', - files: { ...(value.files ?? {}) }, - runnables: { ...(value.runnables ?? {}) }, - data: normalizeRawAppData(value), - policy: app.policy ?? fallback?.policy, - custom_path: app.custom_path ?? fallback?.custom_path - } -} - -function appDraftMeta(app: { versions?: number[]; draft_created_at?: string }): UserDraftMeta { - return { - remoteRev: app.versions ? app.versions[app.versions.length - 1] : undefined, - remoteDraftRev: app.draft_created_at - } -} - async function loadAppValueForRead(path: string, workspace: string): Promise { - const draft = getGlobalDraft(workspace, 'app', path) + const draft = await getGlobalDraft(workspace, 'app', path) if (draft && draft.value && typeof draft.value === 'object' && 'files' in draft.value) { return draft.value as AppDraftValue } - const app = await AppService.getAppByPathWithDraft({ workspace, path }) - return appSourceToDraftValue(app.draft ?? app, app) + const app = await AppService.getAppByPath({ workspace, path }) + return appSourceToDraftValue(app, app) } async function loadAppDraftValue(path: string, workspace: string): Promise { - const draft = getGlobalDraft(workspace, 'app', path) + const draft = await getGlobalDraft(workspace, 'app', path) if (draft && draft.value && typeof draft.value === 'object' && 'files' in draft.value) { return { value: draft.value as AppDraftValue } } - const app = await AppService.getAppByPathWithDraft({ workspace, path }) - const value = appSourceToDraftValue(app.draft ?? app, app) - return { value, meta: appDraftMeta(app) } + const app = await AppService.getAppByPath({ workspace, path }) + return { value: appSourceToDraftValue(app, app) } } -function saveAppDraft( +async function saveAppDraft( workspace: string, path: string, - value: AppDraftValue, - meta?: UserDraftMeta -): WorkspaceItem { - return saveGlobalAppDraft(workspace, path, value, meta) + value: AppDraftValue +): Promise { + return saveGlobalAppDraft(workspace, path, value) } type TriggerLike = { path: string; summary?: string | null } @@ -1242,13 +1388,13 @@ async function readWorkspaceItem( switch (type) { case 'script': { // Prefer the DB draft (newer than the deployed version) when one exists. - const script = await ScriptService.getScriptByPathWithDraft({ workspace, path }) - return scriptToItem(script.draft ?? script, true) + const script = await ScriptService.getScriptByPath({ workspace, path, getDraft: true }) + return scriptToItem((script.draft as Script | undefined) ?? script, true) } case 'flow': { // Prefer the DB draft (newer than the deployed version) when one exists. - const flow = await FlowService.getFlowByPathWithDraft({ workspace, path }) - return flowToItem(flow.draft ?? flow, true) + const flow = await FlowService.getFlowByPath({ workspace, path, getDraft: true }) + return flowToItem((flow.draft as Flow | undefined) ?? flow, true) } case 'schedule': return scheduleToItem(await ScheduleService.getSchedule({ workspace, path }), true) @@ -1262,10 +1408,7 @@ async function readWorkspaceItem( true ) case 'resource': - return resourceToItem( - (await ResourceService.getResource({ workspace, path })) as ListableResource, - true - ) + return resourceToItem(await ResourceService.getResource({ workspace, path }), true) case 'variable': // Never expose the value, even when read directly. Pass decryptSecret=false // to avoid materializing secret values server-side. @@ -1274,8 +1417,8 @@ async function readWorkspaceItem( ) case 'app': { // Returns lightweight metadata only — file/runnable contents come via read_app_file. - const app = await AppService.getAppByPathWithDraft({ workspace, path }) - const value = appSourceToDraftValue(app.draft ?? app) + const app = await AppService.getAppByPath({ workspace, path }) + const value = appSourceToDraftValue(app) const metadata = summarizeAppValue(value) return { type: 'app', @@ -1391,7 +1534,7 @@ function getFlowInstructions(): string { - Global mode writes complete draft payloads only; it does not save, deploy, run, scaffold local files, or generate metadata. - Paths follow the conventions in the system prompt: default to \`u//\` when the user gave a bare name; only use \`f//\` when the folder is known to exist. Never invent a folder. -- \`write_flow\` mirrors flow mode's \`set_flow_json\`: pass \`path\`, optional \`summary\`, required \`modules\`, and optional \`schema\`, \`preprocessor_module\`, \`failure_module\`, and \`groups\`. The flow-structure arguments are JSON strings, matching the tool schema descriptions. +- \`write_flow\` mirrors flow mode's \`set_flow_json\`: pass \`path\`, optional \`summary\`, optional \`description\`, required \`modules\`, and optional \`schema\`, \`preprocessor_module\`, \`failure_module\`, and \`groups\`. \`summary\` and \`description\` are top-level flow metadata (not part of the compact value \`patch_flow_json\` edits); the flow-structure arguments are JSON strings, matching the tool schema descriptions. - \`read_workspace_item\` returns a compact flow \`value\` object with \`modules\`, \`schema\`, \`preprocessor_module\`, \`failure_module\`, and \`groups\`. - \`modules\` contains normal sequential modules. Use top-level \`preprocessor_module\` and \`failure_module\` for special modules; do not put \`preprocessor\` or \`failure\` in \`modules\`. - Every module needs a stable unique \`id\` and a useful \`summary\` when the schema supports it. @@ -1403,7 +1546,7 @@ function getFlowInstructions(): string { - \`read_workspace_item\` and \`patch_flow_json\` operate on a **compact view** of the flow: every rawscript module's \`value.content\` is replaced with the placeholder \`"inline_script."\` so inline script bodies don't bloat tool I/O. Schema, groups, preprocessor_module and failure_module are all shown in this view. - Inline rawscript content is **not** part of the JSON \`patch_flow_json\` sees. Edits to inline bodies happen via dedicated tools: - \`read_flow_module_code(path, module_id)\` — returns the raw inline script content for one module. - - \`set_flow_module_code(path, module_id, code)\` — overwrites that module's inline script content; saves to the local draft. + - \`set_flow_module_code(path, module_id, code)\` — overwrites that module's inline script content; saves to the draft. - Use \`patch_flow_json\` for *structural* edits: module ids, paths, input_transforms, branch arrangement, summaries, preprocessor/failure swaps, schema/groups. Use \`set_flow_module_code\` for changes inside a specific rawscript body. - \`write_flow\` is for full overwrites / create-from-scratch. Its \`modules\`, \`preprocessor_module\`, and \`failure_module\` arguments use **non-compact** flow modules (rawscript content is the actual code, not a placeholder). @@ -1426,7 +1569,8 @@ function getAppInstructions(): string { - \`/wmill.d.ts\` (or \`wmill.ts\`) is generated automatically from the backend runnables — never write it directly. - Inline runnables only support \`bun\` or \`python3\` in chat. Path runnables (\`script\`/\`flow\`/\`hubscript\`) reference an existing item. - Use \`deploy_workspace_item\` after explicit user deploy intent. The deploy tool bundles JS/CSS before saving the raw app. -- Use \`read_workspace_item\` with \`type: 'app'\` for a metadata summary (file paths and runnable list, no contents). Use \`read_app_file\` to read an individual file. +- Use \`read_workspace_item\` with \`type: 'app'\` for a metadata summary (file paths and runnable list, no contents). Use \`read_app_file\` to read an individual file; large files are truncated to a head slice, so pass \`offset\`/\`limit\` to page through the rest rather than re-reading the whole file. +- To find where a symbol or string lives across the app, call \`search_app\` (greps every frontend file and inline runnable, returns matching \`file:line\` rows) instead of reading files one by one — then \`read_app_file\` only the ranges you need. The loop is list (\`read_workspace_item\`) → locate (\`search_app\`) → inspect (\`read_app_file\` with \`offset\`/\`limit\`). - Note: the authoring reference below mentions the CLI on-disk layout (\`backend/.\`, \`raw_app.yaml\`, \`sql_to_apply/\`). That layout is only relevant for the terminal workflow — in chat, apps are addressed via the tool surface above. # Windmill raw app authoring reference @@ -1479,7 +1623,51 @@ function getInstructions(subject: InstructionSubject, language?: ScriptLang): st } } +export type AiSkillListItem = { name: string; description: string } + +/** Fetch the workspace's AI skills (name + description) for the global system prompt. */ +export async function loadWorkspaceSkills(workspace: string): Promise { + if (!workspace) return [] + try { + return await WorkspaceService.listAiSkills({ workspace }) + } catch (e) { + console.error('Failed to load AI skills', e) + return [] + } +} + +const readSkillSchema = z.object({ + name: z + .string() + .describe('The exact skill name as listed in the Skills section of the system prompt.') +}) + +export const readSkillTool: Tool<{}> = { + def: createToolDef( + readSkillSchema, + 'read_skill', + 'Load the full instructions for a workspace AI skill by name. Skills are listed in the system prompt under "Skills"; call this before acting on a task a skill covers, then follow its instructions.' + ), + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = readSkillSchema.parse(args) + toolCallbacks.setToolStatus(toolId, { content: `Reading skill "${parsed.name}"...` }) + try { + const skill = await WorkspaceService.getAiSkill({ workspace, name: parsed.name }) + toolCallbacks.setToolStatus(toolId, { content: `Read skill "${parsed.name}"` }) + return `Skill: ${skill.name}\nDescription: ${skill.description}\n\nInstructions:\n${skill.instructions}` + } catch (e) { + const msg = e instanceof Error ? e.message : String(e) + toolCallbacks.setToolStatus(toolId, { + content: `Error reading skill "${parsed.name}"`, + error: msg + }) + return `Failed to read skill "${parsed.name}": ${msg}. Check the name against the Skills list in the system prompt.` + } + } +} + export const globalTools: Tool<{}>[] = [ + readSkillTool, { def: createToolDef( getInstructionsSchema, @@ -1497,6 +1685,8 @@ export const globalTools: Tool<{}>[] = [ } }, createSearchHubScriptsTool(false), + searchDocsTool, + readDocsPageTool, { def: createToolDef( askUserQuestionSchema, @@ -1551,11 +1741,84 @@ export const globalTools: Tool<{}>[] = [ return selectedChoice } }, + { + def: createToolDef( + updateUserInstructionsSchema, + 'update_user_instructions', + 'Modify your own persistent personal instructions for the Global assistant. Use when the user asks you to remember a preference, always/never do something, or change/stop a behavior. Edits only the user-level instructions (the USER INSTRUCTIONS block, not workspace-level); changes persist in this browser and take effect on your next message.' + ), + showDetails: true, + fn: async ({ args, toolId, toolCallbacks, helpers }) => { + const parsed = updateUserInstructionsSchema.parse(args) + const h = helpers as GlobalToolHelpers + if (!h?.getUserInstructions || !h?.setUserInstructions) { + const message = 'This chat context cannot modify user instructions.' + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + + const current = h.getUserInstructions() + let next: string + if (parsed.operation === 'append') { + const text = parsed.text?.trim() + if (!text) { + const message = "operation 'append' requires a non-empty text." + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + next = current.trim() ? `${current.trim()}\n\n${text}` : text + } else { + if (parsed.old_string === undefined || parsed.new_string === undefined) { + const message = "operation 'replace' requires old_string and new_string." + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + try { + next = findAndReplace( + current, + parsed.old_string, + parsed.new_string, + parsed.replace_all ?? false, + 'personal instructions' + ).trim() + } catch (e) { + const detail = e instanceof Error ? e.message : String(e) + // Echo the current text on this rare recovery path so the model can rebuild a + // correct old_string. The system prompt's USER INSTRUCTIONS block can be stale + // within a turn that already made a successful edit, but `current` is always live. + const hint = current.trim() + ? `Your current personal instructions are:\n${current.trim()}` + : "You have no personal instructions yet; use operation 'append' to add one." + const message = `${detail} ${hint}` + toolCallbacks.setToolStatus(toolId, { content: detail, error: detail }) + return message + } + } + + if (next.length > MAX_USER_INSTRUCTIONS_LENGTH) { + const message = `Resulting instructions would be ${next.length} characters, over the ${MAX_USER_INSTRUCTIONS_LENGTH} limit. Make them more concise.` + toolCallbacks.setToolStatus(toolId, { content: message, error: message }) + return message + } + + h.setUserInstructions(next) + + const summary = next + ? parsed.operation === 'append' + ? 'Added a personal instruction' + : 'Updated your personal instructions' + : 'Cleared your personal instructions' + toolCallbacks.setToolStatus(toolId, { content: summary, result: summary }) + // Return only a short confirmation. The updated text is re-injected into the system + // prompt on the next iteration, so echoing it back here would waste context. + return `${summary}. It takes effect from your next message and is editable in the Global chat settings.` + } + }, { def: createToolDef( listWorkspaceItemsSchema, 'list_workspace_items', - 'List workspace items and local drafts. Returns metadata only.' + 'List workspace items and drafts. Returns metadata only.' ), fn: async ({ args, workspace, toolId, toolCallbacks }) => { const parsed = listWorkspaceItemsSchema.parse(args) @@ -1574,7 +1837,7 @@ export const globalTools: Tool<{}>[] = [ byKey.set(getWorkspaceItemKey(item.type, item.path, item.triggerKind), item) } - for (const draft of listGlobalDrafts(workspace)) { + for (const draft of await listGlobalDrafts(workspace)) { if (!types.includes(draft.type)) continue if (parsed.path_prefix && !draft.path.startsWith(parsed.path_prefix)) continue byKey.set(getWorkspaceItemKey(draft.type, draft.path, draft.triggerKind), { @@ -1597,7 +1860,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( readWorkspaceItemSchema, 'read_workspace_item', - 'Read one workspace item or local draft.' + 'Read one workspace item or draft.' ), fn: async ({ args, workspace, toolId, toolCallbacks }) => { const parsed = readWorkspaceItemSchema.parse(args) @@ -1606,10 +1869,10 @@ export const globalTools: Tool<{}>[] = [ toolCallbacks.setToolStatus(toolId, { content: message, error: message }) return JSON.stringify({ success: false, error: message }) } - const draft = getGlobalDraft(workspace, parsed.type, parsed.path, parsed.trigger_kind) + const draft = await getGlobalDraft(workspace, parsed.type, parsed.path, parsed.trigger_kind) if (draft) { toolCallbacks.setToolStatus(toolId, { - content: `Read local draft ${parsed.type} "${parsed.path}"` + content: `Read draft ${parsed.type} "${parsed.path}"` }) return JSON.stringify(serializeWorkspaceItemForRead(draft), null, 2) } @@ -1624,10 +1887,46 @@ export const globalTools: Tool<{}>[] = [ }, { def: createToolDef( - writeScriptSchema, - 'write_script', - 'Create or overwrite a local draft script.' + createFolderSchema, + 'create_folder', + 'Create a new shared folder (addressable as f//) in the workspace. The current user is added as an owner.' ), + requiresConfirmation: true, + confirmationMessage: 'Create folder', + showDetails: true, + fn: async ({ args, workspace, toolId, toolCallbacks }) => { + const parsed = createFolderSchema.parse(args) + if (!VALID_FOLDER_NAME.test(parsed.name)) { + const error = + 'Folder name can only contain alphanumeric characters, underscores, and hyphens.' + toolCallbacks.setToolStatus(toolId, { content: error, error }) + return JSON.stringify({ success: false, error }) + } + toolCallbacks.setToolStatus(toolId, { content: `Creating folder \`f/${parsed.name}\`...` }) + try { + await FolderService.createFolder({ + workspace, + requestBody: { name: parsed.name, summary: parsed.summary } + }) + // Reflect the new folder in the path-convention context for the rest of this + // session, matching FolderPicker's local update (avoids userStore.set()). + const user = get(userStore) + if (user) { + if (!user.folders) user.folders = [] + if (!user.folders.includes(parsed.name)) user.folders.push(parsed.name) + } + const message = `Created folder \`f/${parsed.name}\`. You can now write items to \`f/${parsed.name}/\`.` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ success: true, message }) + } catch (e) { + const error = e instanceof Error ? e.message : String(e) + toolCallbacks.setToolStatus(toolId, { content: `Error: ${error}`, error }) + return JSON.stringify({ success: false, error }) + } + } + }, + { + def: createToolDef(writeScriptSchema, 'write_script', 'Create or overwrite a draft script.'), showDetails: true, streamArguments: true, showFade: true, @@ -1637,7 +1936,7 @@ export const globalTools: Tool<{}>[] = [ } }, { - def: createToolDef(writeFlowSchema, 'write_flow', 'Create or overwrite a local draft flow.'), + def: createToolDef(writeFlowSchema, 'write_flow', 'Create or overwrite a draft flow.'), showDetails: true, streamArguments: true, showFade: true, @@ -1657,6 +1956,7 @@ export const globalTools: Tool<{}>[] = [ { path: parsed.path, summary: parsed.summary, + description: parsed.description, flow: editableFlowToDraftValue(editable) }, ctx @@ -1667,7 +1967,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( writeScheduleSchema, 'write_schedule', - 'Create or overwrite a local draft schedule.', + 'Create or overwrite a draft schedule.', { strict: false } ), showDetails: true, @@ -1682,7 +1982,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( writeTriggerSchema, 'write_trigger', - 'Create or overwrite a local draft trigger.', + 'Create or overwrite a draft trigger.', { strict: false } ), showDetails: true, @@ -1697,7 +1997,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( editScriptSchema, 'edit_script', - 'Find/replace exact text in a script and save a local draft.' + 'Find/replace exact text in a script and save a draft.' ), showDetails: true, streamArguments: true, @@ -1711,7 +2011,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( patchFlowJsonSchema, 'patch_flow_json', - 'Find/replace exact text in compact flow JSON and save a local draft.' + 'Find/replace exact text in compact flow JSON and save a draft.' ), showDetails: true, streamArguments: true, @@ -1817,18 +2117,32 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( deployWorkspaceItemSchema, 'deploy_workspace_item', - 'Deploy a local draft to the workspace. Mutates the workspace.', + 'Deploy a draft to the workspace. Mutates the workspace.', { strict: false } ), showDetails: true, showFade: true, requiresConfirmation: true, - confirmationMessage: 'Deploy local draft to workspace', + confirmationMessage: 'Deploy draft to workspace', fn: async (ctx) => { const parsed = deployWorkspaceItemSchema.parse(ctx.args) return deployDraft(parsed, { ...ctx, sessionId: sessionIdFromCtx(ctx) }) } }, + { + def: createToolDef( + rebaseDraftSchema, + 'rebase_draft', + 'Discard a stale script, flow, or app draft and return your changes as a diff to re-apply on the latest deployed version. Use when deploy_workspace_item reports the draft was started from an older deployed version.', + { strict: false } + ), + showDetails: true, + showFade: true, + fn: async (ctx) => { + const parsed = rebaseDraftSchema.parse(ctx.args) + return rebaseDraft(parsed, ctx) + } + }, { def: createToolDef( deleteWorkspaceItemSchema, @@ -1848,12 +2162,12 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( discardLocalDraftSchema, 'discard_local_draft', - 'Discard a local draft only. Does not mutate deployed workspace items, but clears the matching open editor draft if one is mounted.' + 'Discard a draft only. Does not mutate deployed workspace items, but clears the matching open editor draft if one is mounted.' ), showDetails: true, showFade: true, requiresConfirmation: true, - confirmationMessage: 'Discard local draft', + confirmationMessage: 'Discard draft', fn: async (ctx) => { const parsed = discardLocalDraftSchema.parse(ctx.args) return discardLocalDraft(parsed, ctx) @@ -1863,7 +2177,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( writeResourceSchema, 'write_resource', - 'Create or overwrite a local draft resource.', + 'Create or overwrite a draft resource.', { strict: false } ), showDetails: true, @@ -1878,7 +2192,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( writeVariableSchema, 'write_variable', - 'Create or overwrite a local draft variable.', + 'Create or overwrite a draft variable.', { strict: false } ), showDetails: true, @@ -1933,7 +2247,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( setFlowModuleCodeSchema, 'set_flow_module_code', - 'Overwrite inline script code in one flow module and save a local draft.' + 'Overwrite inline script code in one flow module and save a draft.' ), showDetails: true, streamArguments: true, @@ -1947,7 +2261,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( initAppSchema, 'init_app', - 'Initialize a local draft raw app from a framework template.', + 'Initialize a draft raw app from a framework template.', { strict: false } ), showDetails: true, @@ -1961,13 +2275,24 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( readAppFileSchema, 'read_app_file', - 'Read one raw app frontend file or inline backend runnable.' + 'Read one raw app frontend file or inline backend runnable. Large files are truncated to a head slice; pass offset/limit to page through the rest.' ), fn: async (ctx) => { const parsed = readAppFileSchema.parse(ctx.args) return readAppFile(parsed, ctx) } }, + { + def: createToolDef( + searchAppSchema, + 'search_app', + "Grep across all of a raw app's frontend files and inline backend runnables in one call. Returns matching file:line rows (capped), not file bodies — use it to locate a symbol or string before read_app_file instead of reading whole files one by one." + ), + fn: async (ctx) => { + const parsed = searchAppSchema.parse(ctx.args) + return searchApp(parsed, ctx) + } + }, { def: createToolDef( writeAppFileSchema, @@ -1997,7 +2322,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( patchAppFileSchema, 'patch_app_file', - 'Find/replace exact text in a raw app file and save a local draft.' + 'Find/replace exact text in a raw app file and save a draft.' ), showDetails: true, streamArguments: true, @@ -2066,7 +2391,7 @@ export const globalTools: Tool<{}>[] = [ const result = await getSessionRuntimeLogs(parsed.limit ?? 10, sessionIdFromCtx(ctx)) ctx.toolCallbacks.setToolStatus(ctx.toolId, { content: result.uiMessage, - result: result.toolResult, + result: result.toolResult }) return result.aiResult } @@ -2075,7 +2400,7 @@ export const globalTools: Tool<{}>[] = [ def: createToolDef( listAppRunsSchema, 'list_app_runs', - "List the backend runnable executions (jobs) the raw app preview currently open in this AI session has triggered, newest first." + 'List the backend runnable executions (jobs) the raw app preview currently open in this AI session has triggered, newest first.' ), showDetails: true, fn: async (ctx) => { @@ -2090,7 +2415,9 @@ export const globalTools: Tool<{}>[] = [ } }, // Workspace-scoped datatable tools (unrestricted: no whitelist, no creation policy) - ...getDatatableTools() + ...getDatatableTools(), + // Read-only tools over files the user attached to the conversation + ...fileTools ] // Tools that only make sense inside an AI session (they drive the session's @@ -2135,8 +2462,15 @@ type WriteDraftCtx = { // handlers below would route a backgrounded session's tool call to whatever // session the user happens to be viewing. export type SessionToolHelpers = { sessionId?: string } + export type GlobalToolHelpers = SessionToolHelpers & { testActiveFlow?: (args?: Record) => Promise + attachedFiles?: AttachedFilesStore + // Read/write the user-level Global instructions. `setUserInstructions` persists the + // value and rebuilds the system message so the change applies on the next chat-loop + // iteration. Backed by the update_user_instructions tool. + getUserInstructions?: () => string + setUserInstructions?: (instructions: string) => void } function sessionIdFromCtx(ctx: { helpers?: unknown }): string | undefined { @@ -2370,7 +2704,7 @@ function buildVariableDeployRequestBody( const secretValue = getEphemeralSecretVariableDraftValue(workspace, storagePath) if (secretValue === undefined) { throw new Error( - `Secret value for local draft variable "${path}" is no longer available because secret draft values are kept only in memory. Run write_variable again before deploying this secret.` + `Secret value for draft variable "${path}" is no longer available because secret draft values are kept only in memory. Run write_variable again before deploying this secret.` ) } @@ -2379,307 +2713,307 @@ function buildVariableDeployRequestBody( function startDraftWrite(ctx: WriteDraftCtx, type: WorkspaceItemType, path: string): void { ctx.toolCallbacks.setToolStatus(ctx.toolId, { - content: `Saving ${type} "${path}" to local storage…` + content: `Saving ${type} "${path}" as a draft…` }) } -function getRequiredGlobalDraft( - workspace: string, - type: WorkspaceItemType, - path: string, - triggerKind?: TriggerKind -): WorkspaceItem { - const draft = getGlobalDraft(workspace, type, path, triggerKind) - if (!draft) { - throw new Error(`Could not read written draft ${type} "${path}".`) +// Conflict / save-failure handling shared by the kind write tools and the app +// write tools. Returns the JSON tool-result for a non-saved persist, or undefined +// when the save succeeded (the caller then emits its own success payload). +function draftWriteFailure(result: DraftPersistResult, ctx: WriteDraftCtx): string | undefined { + const stored = result.item + if (result.status === 'conflict') { + ctx.toolCallbacks.setToolStatus(ctx.toolId, { + content: `Draft ${stored.type} "${stored.path}" changed externally`, + result: `Conflict` + }) + return JSON.stringify( + { + success: false, + conflict: true, + message: `The ${stored.type} draft "${stored.path}" changed externally since you last read it. Re-run this tool to merge onto the latest version, or pass override:true to overwrite. If an editor for it is open, a conflict dialog is also shown there.` + }, + null, + 2 + ) } - return draft + if (result.status === 'error') { + ctx.toolCallbacks.setToolStatus(ctx.toolId, { + content: `Failed to save ${stored.type} "${stored.path}"`, + result: `Save failed` + }) + return JSON.stringify( + { + success: false, + error: true, + message: `The ${stored.type} draft "${stored.path}" could NOT be saved (${result.message}). The change was not persisted — retry; do not assume it succeeded.` + }, + null, + 2 + ) + } + return undefined } -function finishDraftWrite(stored: WorkspaceItem, existed: boolean, ctx: WriteDraftCtx): string { - const verb = existed ? 'Updated' : 'Created' - - ctx.toolCallbacks.setToolStatus(ctx.toolId, { - content: `${verb} ${stored.type} "${stored.path}" in local storage`, - result: `Saved to local storage` - }) - const message = `${verb} ${stored.type} "${stored.path}" in local storage (a browser-only local draft, not a workspace draft). It was not deployed.` - // Don't echo the stored value back to the model — it just wrote that content, - // and echoing it doubles token usage on every edit. Use read_workspace_item to - // inspect the stored draft (e.g. after a merge with an existing base). +// App write tools build varied success messages but share the same conflict / +// save-failure handling; `onSaved` supplies the per-tool status + message. +function finishAppDraftWrite( + result: DraftPersistResult, + ctx: WriteDraftCtx, + onSaved: () => { content: string; message: string } +): string { + const failure = draftWriteFailure(result, ctx) + if (failure) return failure + const { content, message } = onSaved() + ctx.toolCallbacks.setToolStatus(ctx.toolId, { content, result: 'Saved as draft' }) return JSON.stringify({ success: true, message }, null, 2) } -async function writeScriptDraft( - args: { path: string; summary?: string; language: ScriptLang; content: string }, +function finishDraftWrite( + result: DraftPersistResult, + existed: boolean, ctx: WriteDraftCtx -): Promise { - const { workspace } = ctx - startDraftWrite(ctx, 'script', args.path) - const storagePath = getGlobalDraftStoragePath(workspace, 'script', args.path) +): string { + const failure = draftWriteFailure(result, ctx) + if (failure) return failure + const stored = result.item + const verb = existed ? 'Updated' : 'Created' + // Don't echo the flow value back: the model just sent it in the write call, + // so reflecting the (large) compact flow JSON only burns tokens. Variables + // echo a redacted item; everything else round-trips its small payload. + const serializedItem = + stored.type === 'flow' + ? undefined + : stored.type === 'variable' + ? serializeWorkspaceItemForRead(stored) + : stored - const existingDraft = UserDraft.get('script', storagePath, { workspace }) - const backendExists = existingDraft - ? false - : await ScriptService.existsScriptByPath({ workspace, path: args.path }) - - if (existingDraft) { - const draft: NewScript = { - ...structuredClone(existingDraft), - path: args.path, - summary: args.summary ?? existingDraft.summary, - content: args.content, - language: args.language - } - UserDraft.save('script', storagePath, draft, { workspace }) - } else if (backendExists) { - const existing = await ScriptService.getScriptByPathWithDraft({ - workspace, - path: args.path - }) - const base = (existing.draft ?? existing) as NewScript - const draft: NewScript = { - ...structuredClone(base), - parent_hash: existing.hash, - path: args.path, - summary: args.summary ?? base.summary, - content: args.content, - language: args.language - } - UserDraft.setDraftAndMeta( - 'script', - storagePath, - draft, - { remoteRev: existing.hash, remoteDraftRev: existing.draft_created_at }, - { workspace } - ) - } else { - const draft: NewScript = { - path: args.path, - summary: args.summary ?? '', - description: '', - content: args.content, - schema: emptySchema(), - is_template: false, - language: args.language, - kind: 'script' - } - UserDraft.save('script', storagePath, draft, { workspace }) - } - - return finishDraftWrite( - getRequiredGlobalDraft(workspace, 'script', args.path), - existingDraft !== undefined || backendExists, - ctx - ) -} - -async function writeFlowDraft( - args: { path: string; summary?: string; flow: FlowDraftValue }, - ctx: WriteDraftCtx -): Promise { - const { workspace } = ctx - startDraftWrite(ctx, 'flow', args.path) - const storagePath = getGlobalDraftStoragePath(workspace, 'flow', args.path) - - const draftValue = args.flow - const value = structuredClone(draftValue.value) - if (draftValue.groups !== undefined && draftValue.groups !== null) { - value.groups = structuredClone(draftValue.groups) - } - - const existingDraft = UserDraft.get('flow', storagePath, { workspace }) - const backendExists = existingDraft - ? false - : await FlowService.existsFlowByPath({ workspace, path: args.path }) - - if (existingDraft) { - const draft: Flow = { - ...structuredClone(existingDraft), - path: args.path, - summary: args.summary ?? existingDraft.summary, - value, - schema: draftValue.schema ?? existingDraft.schema - } - UserDraft.save('flow', storagePath, draft, { workspace }) - } else if (backendExists) { - const [existing, latestVersion] = await Promise.all([ - FlowService.getFlowByPathWithDraft({ workspace, path: args.path }), - FlowService.getFlowLatestVersion({ workspace, path: args.path }) - ]) - const base = (existing.draft ?? existing) as Flow - const draft: Flow = { - ...structuredClone(base), - path: args.path, - summary: args.summary ?? base.summary, - value, - schema: draftValue.schema ?? base.schema - } - UserDraft.setDraftAndMeta( - 'flow', - storagePath, - draft, - { remoteRev: latestVersion.id, remoteDraftRev: existing.draft_created_at }, - { workspace } - ) - } else { - const draft: Flow = { - path: args.path, - summary: args.summary ?? '', - value, - schema: draftValue.schema ?? emptySchema(), - edited_by: '', - edited_at: '', - archived: false, - extra_perms: {} - } - UserDraft.save('flow', storagePath, draft, { workspace }) - } - - return finishDraftWrite( - getRequiredGlobalDraft(workspace, 'flow', args.path), - existingDraft !== undefined || backendExists, - ctx - ) -} - -async function writeScheduleDraft(args: NewSchedule, ctx: WriteDraftCtx): Promise { - const { workspace } = ctx - startDraftWrite(ctx, 'schedule', args.path) - - const existingDraft = UserDraft.get('trigger_schedule', args.path, { - workspace + ctx.toolCallbacks.setToolStatus(ctx.toolId, { + content: `${verb} ${stored.type} "${stored.path}" as a draft`, + result: `Saved as draft` }) - const backendExists = existingDraft - ? false - : await ScheduleService.existsSchedule({ workspace, path: args.path }) - - const base = existingDraft - ? existingDraft - : backendExists - ? ((await ScheduleService.getSchedule({ - workspace, - path: args.path - })) as ScheduleDraftConfig) - : undefined - const draft = mergeDraftConfig(base, args as DraftConfig, args.path) - - UserDraft.save('trigger_schedule', args.path, draft, { workspace }) - - return finishDraftWrite( - getRequiredGlobalDraft(workspace, 'schedule', args.path), - existingDraft !== undefined || backendExists, - ctx + return JSON.stringify( + { + success: true, + message: `${verb} ${stored.type} "${stored.path}" as a per-user draft (saved server-side, visible only to this user — not a deployed workspace item). It was not deployed.`, + item: serializedItem + }, + null, + 2 ) } -async function writeTriggerDraft( - args: { kind: TriggerKind; config: unknown }, - ctx: WriteDraftCtx +// Per-draft-kind knowledge for the shared write skeleton. `fetchDeployed` returns +// the deployed item already shaped as a draft value (e.g. script with parent_hash) +// so `buildDraft` treats a draft base and a deployed base identically; a `base` of +// undefined is the create-from-scratch case. `beforePersist` is a kind-local side +// effect run after the value is built (only variable, for its in-memory secret). +type WriteSpec = { + probe: (workspace: string, path: string) => Promise + fetchDeployed: (workspace: string, path: string) => Promise + buildDraft: (base: T | undefined, args: A, path: string) => T | Promise + beforePersist?: (workspace: string, args: A) => void +} + +async function writeDraft( + spec: WriteSpec, + type: WorkspaceItemType, + path: string, + args: A, + ctx: WriteDraftCtx, + opts: { triggerKind?: TriggerKind; override?: boolean } = {} ): Promise { const { workspace } = ctx + startDraftWrite(ctx, type, path) + + const existingDraft = await readGlobalDraftValue(workspace, type, path, opts.triggerKind) + let base = existingDraft + let existed = existingDraft !== undefined + if (base === undefined && (await spec.probe(workspace, path))) { + base = await spec.fetchDeployed(workspace, path) + existed = true + } + + const draft = await spec.buildDraft(base, args, path) + spec.beforePersist?.(workspace, args) + + const result = await persistGlobalDraft(workspace, type, path, draft, { + triggerKind: opts.triggerKind, + force: opts.override + }) + return finishDraftWrite(result, existed, ctx) +} + +type ScriptDraftArgs = { + path: string + summary?: string + language: ScriptLang + content: string + override?: boolean +} + +const SCRIPT_SPEC: WriteSpec = { + probe: (workspace, path) => ScriptService.existsScriptByPath({ workspace, path }), + fetchDeployed: async (workspace, path) => { + const existing = await ScriptService.getScriptByPath({ workspace, path }) + return { ...(existing as unknown as NewScript), parent_hash: existing.hash } + }, + buildDraft: async (base, args, path) => { + const draft: NewScript = base + ? { + ...structuredClone(base), + path, + summary: args.summary ?? base.summary, + content: args.content, + language: args.language + } + : { + path, + summary: args.summary ?? '', + description: '', + content: args.content, + schema: emptySchema(), + is_template: false, + language: args.language, + kind: 'script' + } + // Infer the arg schema from the content at save time, like the editor does, + // so the persisted draft is the single source of truth at deploy. Keep the + // previous schema (or empty) on failure rather than blanking it. + try { + const schema = emptySchema() + await inferArgs(draft.language, draft.content, schema) + draft.schema = schema + } catch (e) { + console.error('Failed to infer script schema before saving draft', e) + } + return draft + } +} + +function writeScriptDraft(args: ScriptDraftArgs, ctx: WriteDraftCtx): Promise { + return writeDraft(SCRIPT_SPEC, 'script', args.path, args, ctx, { override: args.override }) +} + +type FlowDraftArgs = { + path: string + summary?: string + description?: string + flow: FlowDraftValue + override?: boolean +} + +const FLOW_SPEC: WriteSpec = { + probe: (workspace, path) => FlowService.existsFlowByPath({ workspace, path }), + fetchDeployed: (workspace, path) => FlowService.getFlowByPath({ workspace, path }), + buildDraft: (base, args, path) => { + const value = structuredClone(args.flow.value) + if (args.flow.groups !== undefined && args.flow.groups !== null) { + value.groups = structuredClone(args.flow.groups) + } + return base + ? { + ...structuredClone(base), + path, + summary: args.summary ?? base.summary, + description: args.description ?? base.description, + value, + schema: args.flow.schema ?? base.schema + } + : { + path, + summary: args.summary ?? '', + description: args.description ?? '', + value, + schema: args.flow.schema ?? emptySchema(), + edited_by: '', + edited_at: '', + archived: false, + extra_perms: {} + } + } +} + +function writeFlowDraft(args: FlowDraftArgs, ctx: WriteDraftCtx): Promise { + return writeDraft(FLOW_SPEC, 'flow', args.path, args, ctx, { override: args.override }) +} + +const SCHEDULE_SPEC: WriteSpec = { + probe: (workspace, path) => ScheduleService.existsSchedule({ workspace, path }), + fetchDeployed: async (workspace, path) => + (await ScheduleService.getSchedule({ workspace, path })) as ScheduleDraftConfig, + buildDraft: (base, args, path) => { + // `override` is a tool-only conflict-resolution flag, not schedule config — + // strip it so mergeDraftConfig doesn't clone it into the persisted draft. + const { override: _override, ...config } = args + return mergeDraftConfig(base, config as DraftConfig, path) + } +} + +function writeScheduleDraft( + args: NewSchedule & { override?: boolean }, + ctx: WriteDraftCtx +): Promise { + return writeDraft(SCHEDULE_SPEC, 'schedule', args.path, args, ctx, { override: args.override }) +} + +function triggerWriteSpec(kind: TriggerKind): WriteSpec { + const service = triggerServices[kind] + return { + probe: (workspace, path) => service.exists({ workspace, path }), + fetchDeployed: async (workspace, path) => + (await service.get({ workspace, path })) as TriggerDraftConfig, + buildDraft: (base, config, path) => mergeDraftConfig(base, config, path) + } +} + +function writeTriggerDraft( + args: { kind: TriggerKind; config: unknown; override?: boolean }, + ctx: WriteDraftCtx +): Promise { const config = args.config as TriggerDraftConfig - const path = config.path - const itemKind = triggerKindToUserDraftKind(args.kind) - startDraftWrite(ctx, 'trigger', path) - - const existingDraft = UserDraft.get(itemKind, path, { workspace }) - const backendExists = existingDraft - ? false - : await triggerServices[args.kind].exists({ workspace, path }) - - const base = existingDraft - ? existingDraft - : backendExists - ? ((await triggerServices[args.kind].get({ workspace, path })) as TriggerDraftConfig) - : undefined - const draft = mergeDraftConfig(base, config, path) - - UserDraft.save(itemKind, path, draft, { workspace }) - - return finishDraftWrite( - getRequiredGlobalDraft(workspace, 'trigger', path, args.kind), - existingDraft !== undefined || backendExists, - ctx - ) + return writeDraft(triggerWriteSpec(args.kind), 'trigger', config.path, config, ctx, { + triggerKind: args.kind, + override: args.override + }) } -async function writeResourceDraft(args: CreateResource, ctx: WriteDraftCtx): Promise { - const { workspace } = ctx - startDraftWrite(ctx, 'resource', args.path) - - const existingDraft = UserDraft.get('resource', args.path, { workspace }) - const backendExists = existingDraft - ? false - : await ResourceService.existsResource({ workspace, path: args.path }) - - if (existingDraft) { - UserDraft.save('resource', args.path, createResourceToDraftState(args, existingDraft), { - workspace - }) - } else if (backendExists) { - const existing = await ResourceService.getResource({ workspace, path: args.path }) - UserDraft.setDraftAndMeta( - 'resource', - args.path, - createResourceToDraftState(args, resourceToDraftState(existing)), - { remoteRev: existing.edited_at }, - { workspace } - ) - } else { - UserDraft.save('resource', args.path, createResourceToDraftState(args), { workspace }) - } - - return finishDraftWrite( - getRequiredGlobalDraft(workspace, 'resource', args.path), - existingDraft !== undefined || backendExists, - ctx - ) +const RESOURCE_SPEC: WriteSpec = { + probe: (workspace, path) => ResourceService.existsResource({ workspace, path }), + fetchDeployed: async (workspace, path) => + resourceToDraftState(await ResourceService.getResource({ workspace, path })), + buildDraft: (base, args) => createResourceToDraftState(args, base) } -async function writeVariableDraft(args: CreateVariable, ctx: WriteDraftCtx): Promise { - const { workspace } = ctx - startDraftWrite(ctx, 'variable', args.path) +function writeResourceDraft( + args: CreateResource & { override?: boolean }, + ctx: WriteDraftCtx +): Promise { + return writeDraft(RESOURCE_SPEC, 'resource', args.path, args, ctx, { override: args.override }) +} - const existingDraft = UserDraft.get('variable', args.path, { workspace }) - const backendExists = existingDraft - ? false - : await VariableService.existsVariable({ workspace, path: args.path }) +const VARIABLE_SPEC: WriteSpec = { + probe: (workspace, path) => VariableService.existsVariable({ workspace, path }), + fetchDeployed: async (workspace, path) => + variableToDraftState( + await VariableService.getVariable({ workspace, path, decryptSecret: false }) + ), + buildDraft: (base, args) => createVariableToDraftState(args, base), + beforePersist: (workspace, args) => syncEphemeralSecretVariableDraftValue(workspace, args) +} - if (existingDraft) { - UserDraft.save('variable', args.path, createVariableToDraftState(args, existingDraft), { - workspace - }) - } else if (backendExists) { - const existing = await VariableService.getVariable({ - workspace, - path: args.path, - decryptSecret: false - }) - UserDraft.setDraftAndMeta( - 'variable', - args.path, - createVariableToDraftState(args, variableToDraftState(existing)), - { remoteRev: existing.edited_at }, - { workspace } - ) - } else { - UserDraft.save('variable', args.path, createVariableToDraftState(args), { workspace }) - } - syncEphemeralSecretVariableDraftValue(workspace, args) - - return finishDraftWrite( - getRequiredGlobalDraft(workspace, 'variable', args.path), - existingDraft !== undefined || backendExists, - ctx - ) +function writeVariableDraft( + args: CreateVariable & { override?: boolean }, + ctx: WriteDraftCtx +): Promise { + return writeDraft(VARIABLE_SPEC, 'variable', args.path, args, ctx, { override: args.override }) } async function loadScriptForEdit( path: string, workspace: string ): Promise<{ content: string; language: ScriptLang; summary?: string }> { - const draft = getGlobalDraft(workspace, 'script', path) + const draft = await getGlobalDraft(workspace, 'script', path) if (draft) { if (typeof draft.value !== 'string' || !draft.language) { throw new Error(`Draft script "${path}" is missing content or language.`) @@ -2714,7 +3048,7 @@ async function loadFlowDraftValue( path: string, workspace: string ): Promise<{ flow: FlowDraftValue; summary?: string }> { - const draft = getGlobalDraft(workspace, 'flow', path) + const draft = await getGlobalDraft(workspace, 'flow', path) if (draft) { if (draft.value === undefined || typeof draft.value === 'string') { throw new Error(`Draft flow "${path}" has no value.`) @@ -2839,7 +3173,7 @@ async function loadScriptForFlowStep( moduleValue: { path: string; hash?: string }, workspace: string ): Promise<{ content: string; language: ScriptLang }> { - const draft = getGlobalDraft(workspace, 'script', moduleValue.path) + const draft = await getGlobalDraft(workspace, 'script', moduleValue.path) if (draft) { if (typeof draft.value !== 'string' || !draft.language) { throw new Error(`Draft script "${moduleValue.path}" is missing content or language.`) @@ -2857,7 +3191,7 @@ async function loadDraftFlowPreviewValue( path: string, workspace: string ): Promise { - if (!getGlobalDraft(workspace, 'flow', path)) { + if (!(await getGlobalDraft(workspace, 'flow', path))) { return undefined } const nestedFlow = await loadFlowDraftValue(path, workspace) @@ -2977,9 +3311,9 @@ async function initApp( const { workspace, toolId, toolCallbacks } = ctx const { path, summary, framework } = args - if (getGlobalDraft(workspace, 'app', path)) { + if (await getGlobalDraft(workspace, 'app', path)) { throw new Error( - `A local draft for app "${path}" already exists. Use write_app_file / write_app_runnable to modify it, or delete the existing draft first.` + `A draft for app "${path}" already exists. Use write_app_file / write_app_runnable to modify it, or delete the existing draft first.` ) } if (await AppService.existsApp({ workspace, path })) { @@ -2989,7 +3323,7 @@ async function initApp( } toolCallbacks.setToolStatus(toolId, { - content: `Saving app "${path}" to local storage (${framework} template)…` + content: `Saving app "${path}" draft (${framework} template)…` }) const template = FRAMEWORK_TEMPLATES[framework] @@ -2999,24 +3333,96 @@ async function initApp( runnables: { [STARTER_RUNNABLE_KEY]: { ...STARTER_RUNNABLE } } } await recomputeAppPolicy(value) - saveAppDraft(workspace, path, value) + const result = await saveAppDraft(workspace, path, value) + return finishAppDraftWrite(result, ctx, () => ({ + content: `Saved app "${path}" draft (${framework})`, + message: `Initialized a per-user draft app "${path}" from the ${framework} template with a starter runnable "${STARTER_RUNNABLE_KEY}" (saved server-side, not a deployed workspace item). Use write_app_file / write_app_runnable to evolve it.` + })) +} - toolCallbacks.setToolStatus(toolId, { - content: `Saved app "${path}" to local storage (${framework})`, - result: 'Saved to local storage' - }) - return JSON.stringify( - { - success: true, - message: `Initialized app "${path}" in local storage from the ${framework} template with files ${Object.keys(template).join(', ')} and a starter runnable "${STARTER_RUNNABLE_KEY}" (a browser-only local draft, not a workspace draft). Use write_app_file / write_app_runnable to evolve it.` - }, - null, - 2 - ) +// read_app_file caps: a large frontend file or inline runnable would otherwise +// enter context in full and persist for the rest of the session. Default to a +// head slice with a pointer to page further; the model widens with offset/limit. +// The char budget is a hard ceiling on a single read: a selected line window over +// it is truncated and the model is told to narrow the line range. There is no +// char-level paging, so a single line longer than the budget can't be read past — +// add paging here if minified/long-line files must be fully readable. +const READ_APP_FILE_DEFAULT_LINE_LIMIT = 1500 +const READ_APP_FILE_CHAR_BUDGET = 50_000 + +type AppFileSlice = { + body: string + startLine: number + endLine: number + requestedStartLine: number + totalLines: number + lineWindowChars: number + charTruncated: boolean + truncated: boolean +} + +function sliceAppFileForRead(content: string, offset?: number, limit?: number): AppFileSlice { + const lines = content.split('\n') + const totalLines = lines.length + const requestedStartLine = offset ?? 1 + const start = Math.min(Math.max(requestedStartLine - 1, 0), totalLines) + const lineLimit = limit ?? READ_APP_FILE_DEFAULT_LINE_LIMIT + const end = Math.min(start + lineLimit, totalLines) + const selectedBody = lines.slice(start, end).join('\n') + const lineWindowChars = selectedBody.length + const body = selectedBody.slice(0, READ_APP_FILE_CHAR_BUDGET) + const charTruncated = lineWindowChars > READ_APP_FILE_CHAR_BUDGET + + return { + body, + startLine: start + 1, + endLine: end, + requestedStartLine, + totalLines, + lineWindowChars, + charTruncated, + truncated: start > 0 || end < totalLines || charTruncated + } +} + +function formatAppFileReadRangeLabel(slice: AppFileSlice): string { + const lineRange = `lines ${slice.startLine}-${slice.endLine} of ${slice.totalLines}` + if (!slice.charTruncated) { + return lineRange + } + return `${lineRange}, truncated to the first ${READ_APP_FILE_CHAR_BUDGET} of ${slice.lineWindowChars} chars` +} + +// Small files (returned whole, starting at line 1) keep the raw body so +// patch_app_file's exact-match stays trivial; truncated/windowed reads get a +// one-line annotation describing the range (not part of the file). +function formatAppFileReadResult(slice: AppFileSlice): string { + // offset past the last line: report it plainly instead of a backwards range. + if (slice.requestedStartLine > slice.totalLines) { + return `offset ${slice.requestedStartLine} is past the end of the file (${slice.totalLines} lines).` + } + if (!slice.truncated && slice.startLine === 1) { + return slice.body + } + + let more = '' + if (slice.charTruncated) { + more = ` Reached the ${READ_APP_FILE_CHAR_BUDGET}-char limit; re-read with a smaller limit (fewer lines). If a single line exceeds the limit the file is likely minified and not readable this way.` + } else if (slice.endLine < slice.totalLines) { + more = ` Call read_app_file again with offset=${slice.endLine + 1} to continue.` + } + // No tool-name/path prefix: the model already has them from the call args. The + // range line orients it; the body follows after a blank line. + return `${formatAppFileReadRangeLabel(slice)}.${more}\n\n${slice.body}` } async function readAppFile( - args: { path: string; file_path: string }, + args: { + path: string + file_path: string + offset?: number + limit?: number + }, ctx: WriteDraftCtx ): Promise { const { workspace, toolId, toolCallbacks } = ctx @@ -3027,18 +3433,190 @@ async function readAppFile( const value = await loadAppValueForRead(args.path, workspace) + let content: string if (target.kind === 'frontend') { - const content = value.files[target.filePath] - if (content === undefined) { + const frontend = value.files[target.filePath] + if (frontend === undefined) { throw new Error(`Frontend file "${target.filePath}" not found in app "${args.path}".`) } - toolCallbacks.setToolStatus(toolId, { content: `Read ${target.filePath}` }) - return content + content = frontend + } else { + content = getInlineRunnableContent(value, target, args.path).content } - const { content } = getInlineRunnableContent(value, target, args.path) + const slice = sliceAppFileForRead(content, args.offset, args.limit) + toolCallbacks.setToolStatus(toolId, { content: `Read ${target.filePath}` }) - return content + return formatAppFileReadResult(slice) +} + +// search_app caps: a single query must stay sparse and cheap even when it hits a +// minified bundle or a 5k-line data module. Per-line and total-output caps bound +// the result the same way read_app_file's char budget bounds one file read; the +// match cap keeps a broad query from flooding context instead of locating it. +const SEARCH_APP_DEFAULT_MAX_MATCHES = 100 +const SEARCH_APP_MAX_MATCHES_CEILING = 200 +const SEARCH_APP_MAX_LINE_CHARS = 200 +const SEARCH_APP_TOTAL_CHAR_BUDGET = 12_000 +// Fixed surrounding-context window per match, kept off the tool schema to keep it +// lean. Bump if matches need more context than the line ± this. +const SEARCH_APP_CONTEXT_LINES = 2 + +type AppSearchableFile = { filePath: string; content: string } + +// The files search_app scans: frontend files (minus generated ones) plus inline +// backend runnables, each addressed exactly as read_app_file expects so a match +// row's path can be passed straight back to read_app_file. +function collectSearchableAppFiles(value: AppDraftValue): AppSearchableFile[] { + const files: AppSearchableFile[] = [] + for (const [filePath, content] of Object.entries(value.files)) { + if (GENERATED_APP_FILE_PATHS.has(filePath)) continue + if (typeof content === 'string') files.push({ filePath, content }) + } + for (const [key, runnable] of Object.entries(value.runnables)) { + const persisted = runnable as PersistedRunnable | undefined + const content = persisted?.inlineScript?.content + if (typeof content !== 'string') continue + files.push({ filePath: `backend/${key}/main.${getInlineScriptExtension(persisted)}`, content }) + } + return files +} + +// Minimal glob: * = any chars except '/', ** = any chars, ? = single non-slash. +// A pattern without '/' matches the file name only (ripgrep-style), so "*.tsx" +// finds nested files; a pattern with '/' matches the full path. +function appFileMatchesGlob(filePath: string, glob: string): boolean { + const hasSlash = glob.includes('/') + const subject = hasSlash ? filePath : filePath.slice(filePath.lastIndexOf('/') + 1) + const body = glob + .replace(/[.+^${}()|[\]\\]/g, '\\$&') + .replace(/\*\*/g, '\u0000') + .replace(/\*/g, '[^/]*') + .replace(/\u0000/g, '.*') + .replace(/\?/g, '[^/]') + try { + return new RegExp(`^${body}$`).test(subject) + } catch { + return false + } +} + +type AppSearchMatch = { filePath: string; line: number; text: string } + +async function searchApp( + args: { + path: string + query: string + file_glob?: string + max_matches?: number + }, + ctx: WriteDraftCtx +): Promise { + const { workspace, toolId, toolCallbacks } = ctx + const query = args.query + if (query.length === 0) { + throw new Error('search_app requires a non-empty query.') + } + toolCallbacks.setToolStatus(toolId, { + content: `Searching app "${args.path}" for "${query}"...` + }) + + const value = await loadAppValueForRead(args.path, workspace) + const maxMatches = Math.min( + args.max_matches ?? SEARCH_APP_DEFAULT_MAX_MATCHES, + SEARCH_APP_MAX_MATCHES_CEILING + ) + const contextLines = SEARCH_APP_CONTEXT_LINES + const needle = query.toLowerCase() + + let files = collectSearchableAppFiles(value).sort((a, b) => a.filePath.localeCompare(b.filePath)) + if (args.file_glob) { + files = files.filter((f) => appFileMatchesGlob(f.filePath, args.file_glob as string)) + } + + const matches: AppSearchMatch[] = [] + let totalMatchCount = 0 + // Cap on matching LINES, not pushed rows — each match expands to its context + // window, so counting rows would make `max_matches`/"showing the first N" wrong. + let renderedMatchCount = 0 + let fileCount = 0 + let truncated = false + for (const file of files) { + const lines = file.content.split('\n') + let fileHadMatch = false + for (let i = 0; i < lines.length; i++) { + if (!lines[i].toLowerCase().includes(needle)) continue + totalMatchCount++ + // Count the file on its first match, before the render cap, so the + // "N matches in M files" header counts every file with the symbol — not + // only the ones whose matches landed in the rendered slice (find-all-usages). + fileHadMatch = true + if (renderedMatchCount >= maxMatches) { + truncated = true + continue + } + renderedMatchCount++ + const lo = Math.max(0, i - contextLines) + const hi = Math.min(lines.length - 1, i + contextLines) + for (let j = lo; j <= hi; j++) { + matches.push({ filePath: file.filePath, line: j + 1, text: lines[j] }) + } + } + if (fileHadMatch) fileCount++ + } + + if (totalMatchCount === 0) { + toolCallbacks.setToolStatus(toolId, { content: `No matches for "${query}"` }) + return `No matches. Try a broader or differently-spelled term${ + args.file_glob ? ', or drop the file_glob' : '' + }.` + } + + // No tool-name/query prefix: the model already has them from the call args. + const header = `${totalMatchCount} match${ + totalMatchCount === 1 ? '' : 'es' + } in ${fileCount} file${fileCount === 1 ? '' : 's'}${ + truncated + ? ` (showing the first ${maxMatches}; narrow with file_glob or a more specific query)` + : '' + }` + + const out: string[] = [header] + let currentFile = '' + let budgetSpent = header.length + let budgetHit = false + const seen = new Set() + for (const m of matches) { + // context windows of adjacent matches overlap — show each source line once. + const dedupeKey = `${m.filePath}:${m.line}` + if (seen.has(dedupeKey)) continue + seen.add(dedupeKey) + const text = + m.text.length > SEARCH_APP_MAX_LINE_CHARS + ? `${m.text.slice(0, SEARCH_APP_MAX_LINE_CHARS)}… [line truncated]` + : m.text + const fileHeader = m.filePath === currentFile ? '' : `${m.filePath}\n` + const row = `${fileHeader} ${m.line}: ${text}` + if (budgetSpent + row.length + 1 > SEARCH_APP_TOTAL_CHAR_BUDGET) { + budgetHit = true + break + } + if (fileHeader) currentFile = m.filePath + out.push(row) + budgetSpent += row.length + 1 + } + if (budgetHit) { + out.push( + `… output truncated at the context budget — narrow with file_glob or a more specific query.` + ) + } + + toolCallbacks.setToolStatus(toolId, { + content: `Found ${totalMatchCount} match${totalMatchCount === 1 ? '' : 'es'} in ${fileCount} file${ + fileCount === 1 ? '' : 's' + }` + }) + return out.join('\n') } async function writeAppFile( @@ -3058,22 +3636,13 @@ async function writeAppFile( content: `Writing ${target.filePath} to app "${args.path}"...` }) - const { value, meta } = await loadAppDraftValue(args.path, workspace) + const { value } = await loadAppDraftValue(args.path, workspace) value.files = { ...value.files, [target.filePath]: args.content } - saveAppDraft(workspace, args.path, value, meta) - - toolCallbacks.setToolStatus(toolId, { + const result = await saveAppDraft(workspace, args.path, value) + return finishAppDraftWrite(result, ctx, () => ({ content: `Updated ${target.filePath} in app "${args.path}"`, - result: 'Saved to local storage' - }) - return JSON.stringify( - { - success: true, - message: `Updated app "${args.path}" in local storage with frontend file "${target.filePath}".` - }, - null, - 2 - ) + message: `Updated draft app "${args.path}" with frontend file "${target.filePath}".` + })) } async function deleteAppFile( @@ -3093,26 +3662,17 @@ async function deleteAppFile( content: `Deleting ${target.filePath} from app "${args.path}"...` }) - const { value, meta } = await loadAppDraftValue(args.path, workspace) + const { value } = await loadAppDraftValue(args.path, workspace) if (!(target.filePath in value.files)) { throw new Error(`Frontend file "${target.filePath}" not found in app "${args.path}".`) } const { [target.filePath]: _removed, ...remaining } = value.files value.files = remaining - saveAppDraft(workspace, args.path, value, meta) - - toolCallbacks.setToolStatus(toolId, { + const result = await saveAppDraft(workspace, args.path, value) + return finishAppDraftWrite(result, ctx, () => ({ content: `Removed ${target.filePath} from app "${args.path}"`, - result: 'Saved to local storage' - }) - return JSON.stringify( - { - success: true, - message: `Removed "${target.filePath}" from app "${args.path}" in local storage.` - }, - null, - 2 - ) + message: `Removed "${target.filePath}" from draft app "${args.path}".` + })) } async function patchAppFile( @@ -3142,7 +3702,7 @@ async function patchAppFile( content: `Patching ${target.filePath} in app "${path}"...` }) - const { value, meta } = await loadAppDraftValue(path, workspace) + const { value } = await loadAppDraftValue(path, workspace) let currentContent: string let runnable: PersistedRunnable | undefined @@ -3182,19 +3742,11 @@ async function patchAppFile( } } - saveAppDraft(workspace, path, value, meta) - toolCallbacks.setToolStatus(toolId, { + const result = await saveAppDraft(workspace, path, value) + return finishAppDraftWrite(result, ctx, () => ({ content: `Patched ${target.filePath} in app "${path}"`, - result: 'Saved to local storage' - }) - return JSON.stringify( - { - success: true, - message: `Patched "${target.filePath}" in app "${path}" in local storage.` - }, - null, - 2 - ) + message: `Patched "${target.filePath}" in draft app "${path}".` + })) } async function recomputeAppPolicy(value: AppDraftValue): Promise { @@ -3218,25 +3770,16 @@ async function writeAppRunnable( content: `Writing runnable "${key}" to app "${path}"...` }) - const { value, meta } = await loadAppDraftValue(path, workspace) + const { value } = await loadAppDraftValue(path, workspace) const existing = value.runnables[key] as PersistedRunnable | undefined const persisted = buildPersistedRunnable(input, existing) value.runnables = { ...value.runnables, [key]: persisted } await recomputeAppPolicy(value) - saveAppDraft(workspace, path, value, meta) - - toolCallbacks.setToolStatus(toolId, { + const result = await saveAppDraft(workspace, path, value) + return finishAppDraftWrite(result, ctx, () => ({ content: `Updated runnable "${key}" in app "${path}"`, - result: 'Saved to local storage' - }) - return JSON.stringify( - { - success: true, - message: `Updated app "${path}" in local storage with runnable "${key}".` - }, - null, - 2 - ) + message: `Updated draft app "${path}" with runnable "${key}".` + })) } async function deleteAppRunnable( @@ -3249,27 +3792,18 @@ async function deleteAppRunnable( content: `Removing runnable "${key}" from app "${path}"...` }) - const { value, meta } = await loadAppDraftValue(path, workspace) + const { value } = await loadAppDraftValue(path, workspace) if (!(key in value.runnables)) { throw new Error(`Backend runnable "${key}" not found in app "${path}".`) } const { [key]: _removed, ...remaining } = value.runnables value.runnables = remaining await recomputeAppPolicy(value) - saveAppDraft(workspace, path, value, meta) - - toolCallbacks.setToolStatus(toolId, { + const result = await saveAppDraft(workspace, path, value) + return finishAppDraftWrite(result, ctx, () => ({ content: `Removed runnable "${key}" from app "${path}"`, - result: 'Saved to local storage' - }) - return JSON.stringify( - { - success: true, - message: `Removed runnable "${key}" from app "${path}" in local storage.` - }, - null, - 2 - ) + message: `Removed runnable "${key}" from draft app "${path}".` + })) } const triggerLabels: Record = { @@ -3342,21 +3876,21 @@ async function discardLocalDraft( throw new Error('trigger_kind is required when discarding a trigger draft.') } - const draft = getGlobalDraft(workspace, type, path, triggerKind) + const draft = await getGlobalDraft(workspace, type, path, triggerKind) if (!draft) { - throw new Error(`No local draft found for ${type} "${path}".`) + throw new Error(`No draft found for ${type} "${path}".`) } - deleteGlobalDraft(workspace, type, path, triggerKind) + await deleteGlobalDraft(workspace, type, path, triggerKind) toolCallbacks.setToolStatus(toolId, { - content: `Discarded ${type} "${path}" from local storage`, - result: 'Discarded from local storage' + content: `Discarded ${type} "${path}" draft`, + result: 'Draft discarded' }) return JSON.stringify( { success: true, - message: `Discarded the local-storage draft for ${type} "${path}". The deployed workspace item was not changed.`, + message: `Discarded the draft for ${type} "${path}". The deployed workspace item was not changed.`, type, path, triggerKind @@ -3366,25 +3900,318 @@ async function discardLocalDraft( ) } +// A draft started from an older deploy would silently overwrite whatever was +// deployed since. Block the deploy and point the model at rebase_draft, unless it +// explicitly forces the overwrite. `base`/`head` undefined ⇒ can't tell ⇒ allow. +function assertDraftBasedOnLatest( + type: WorkspaceItemType, + path: string, + base: string | number | undefined, + head: string | number | undefined, + force: boolean | undefined +): void { + if (force || base == null || head == null || base === head) return + throw new Error( + `This ${type} draft "${path}" was started from an older deployed version (forked from ${base}, ` + + `latest is ${head}). Deploying now would overwrite the version deployed since. Call rebase_draft to ` + + `discard the stale draft and get your changes back as a diff, then re-apply them (the new draft ` + + `re-bases onto the latest version) and deploy. To deploy as-is and replace the newer version, call ` + + `deploy_workspace_item again with force: true.` + ) +} + +// Discard a stale draft and return its own changes (vs the fork base) as a diff so +// the model can re-apply them on the latest deploy. Discarding rather than resetting +// keeps the base pointer honest (the next write re-bases on the current head) and +// fails safe: a premature deploy hits "no draft" instead of silently shipping the +// latest unchanged. +async function rebaseDraft( + args: { type: WorkspaceItemType; path: string }, + ctx: WriteDraftCtx +): Promise { + switch (args.type) { + case 'script': + return rebaseScriptDraft(args.path, ctx) + case 'flow': + return rebaseFlowDraft(args.path, ctx) + case 'app': + return rebaseAppDraft(args.path, ctx) + default: + throw new Error('rebase_draft currently supports scripts, flows, and apps.') + } +} + +async function rebaseScriptDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'script', path) + if (!draft || typeof draft.value !== 'string' || !draft.language) { + throw new Error(`No script draft found for "${path}".`) + } + if (!(await ScriptService.existsScriptByPath({ workspace, path }))) { + throw new Error(`Script "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const latest = await ScriptService.getScriptByPath({ workspace, path }) + const baseHash = draft.parentHash + if (baseHash && baseHash === latest.hash) { + const message = `Draft "${path}" is already based on the latest deployed version (${latest.hash}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ success: true, alreadyLatest: true, latest_hash: latest.hash, message }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // Capture the draft's own changes (vs its fork base) BEFORE discarding — this + // diff is the only clean record of what to replay. Best-effort: if the base + // version is gone, diff against empty so the full draft is surfaced. + let baseContent = '' + if (baseHash) { + try { + baseContent = (await ScriptService.getScriptByHash({ workspace, hash: baseHash })).content + } catch (e) { + console.error(`rebase_draft: could not fetch base version ${baseHash} for "${path}"`, e) + } + } + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + baseContent, + draft.value, + '', + '' + ) + + // Discard the stale draft rather than resetting it to latest: the next write + // re-bases on the current head, and a premature deploy fails cleanly ("no + // draft") instead of silently shipping the latest unchanged and losing the work. + await deleteGlobalDraft(workspace, 'script', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a diff against the ` + + `version you forked from). Re-apply them with edit_script / write_script — the new draft will be ` + + `based on the latest deployed version (hash ${latest.hash}) — then deploy.`, + latest_hash: latest.hash, + your_changes: yourChanges + }, + null, + 2 + ) +} + +async function rebaseFlowDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'flow', path) + if (!draft || draft.value === undefined || typeof draft.value === 'string') { + throw new Error(`No flow draft found for "${path}".`) + } + if (!(await FlowService.existsFlowByPath({ workspace, path }))) { + throw new Error(`Flow "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const latest = await FlowService.getFlowByPath({ workspace, path }) + const baseVersion = draft.parentVersionId + if (baseVersion != null && baseVersion === latest.version_id) { + const message = `Draft "${path}" is already based on the latest deployed version (${latest.version_id}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ + success: true, + alreadyLatest: true, + latest_version: latest.version_id, + message + }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // The draft's own changes vs its fork-base flow value, as a JSON diff for the + // model to replay. Best-effort: skip if the base version can't be fetched. + let baseValue: unknown = {} + if (baseVersion != null) { + try { + baseValue = (await FlowService.getFlowVersion({ workspace, version: baseVersion })).value + } catch (e) { + console.error( + `rebase_draft: could not fetch base flow version ${baseVersion} for "${path}"`, + e + ) + } + } + const oursValue = (draft.value as FlowDraftValue).value + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + JSON.stringify(baseValue, null, 2), + JSON.stringify(oursValue, null, 2), + '', + '' + ) + + // Discard the stale draft (see rebaseScriptDraft): the next write re-bases on + // the current head, and a premature deploy fails cleanly instead of shipping + // the latest unchanged. + await deleteGlobalDraft(workspace, 'flow', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a JSON diff against ` + + `the version you forked from). Re-apply them with the flow edit tools — the new draft will be ` + + `based on the latest deployed version (version ${latest.version_id}) — then deploy.`, + latest_version: latest.version_id, + your_changes: yourChanges + }, + null, + 2 + ) +} + +async function rebaseAppDraft(path: string, ctx: WriteDraftCtx): Promise { + const { workspace, toolId, toolCallbacks } = ctx + + const draft = await getGlobalDraft(workspace, 'app', path) + if (!draft || !draft.value || typeof draft.value === 'string' || !('files' in draft.value)) { + throw new Error(`No app draft found for "${path}".`) + } + if (!(await AppService.existsApp({ workspace, path }))) { + throw new Error(`App "${path}" is not deployed; there is no newer version to rebase onto.`) + } + + const deployed = await AppService.getAppByPath({ workspace, path }) + const headVersion = deployed.versions?.[deployed.versions.length - 1] + const baseVersion = draft.parentVersionId + if (baseVersion != null && baseVersion === headVersion) { + const message = `Draft "${path}" is already based on the latest deployed version (${headVersion}).` + toolCallbacks.setToolStatus(toolId, { content: message }) + return JSON.stringify({ + success: true, + alreadyLatest: true, + latest_version: headVersion, + message + }) + } + + toolCallbacks.setToolStatus(toolId, { content: `Rebasing draft "${path}" onto latest...` }) + + // The draft's own changes vs its fork-base app source, as a JSON diff for the + // model to replay. Best-effort: skip if the base version can't be fetched. + const oursValue = draft.value as AppDraftValue + let baseSource: Pick = { + files: {}, + runnables: {}, + data: undefined + } + if (baseVersion != null) { + try { + const baseApp = await AppService.getAppByVersion({ workspace, id: baseVersion }) + const base = appSourceToDraftValue(baseApp, baseApp) + baseSource = { files: base.files, runnables: base.runnables, data: base.data } + } catch (e) { + console.error( + `rebase_draft: could not fetch base app version ${baseVersion} for "${path}"`, + e + ) + } + } + const yourChanges = createTwoFilesPatch( + 'fork-base', + 'your-draft', + JSON.stringify(baseSource, null, 2), + JSON.stringify( + { files: oursValue.files, runnables: oursValue.runnables, data: oursValue.data }, + null, + 2 + ), + '', + '' + ) + + // Discard the stale draft (see rebaseScriptDraft): the next write re-projects + // the deployed app into a fresh draft (re-pinning parent_version to the head), + // and a premature deploy fails cleanly instead of shipping the latest unchanged. + await deleteGlobalDraft(workspace, 'app', path) + + toolCallbacks.setToolStatus(toolId, { + content: `Discarded stale draft "${path}"`, + result: 'Rebased' + }) + return JSON.stringify( + { + success: true, + message: + `Discarded the stale draft for "${path}". Your changes are in "your_changes" (a JSON diff against ` + + `the version you forked from). Re-apply them with the app edit tools — the new draft will be based ` + + `on the latest deployed version (version ${headVersion}) — then deploy.`, + latest_version: headVersion, + your_changes: yourChanges + }, + null, + 2 + ) +} + +// Flush a draft's pending editor autosave, then verify it actually landed before +// the caller re-reads the persisted draft. `flush()` resolves even when the save +// recorded a conflict (server has a newer version) or failed (network/5xx) — it +// does not throw — so without this check a deploy could publish a stale/conflicting +// draft. Abort with a clear message instead. +async function flushDraftOrThrow( + query: Parameters[0], + label: string +): Promise { + await UserDraftDbSyncer.flush(query) + if (UserDraftDbSyncer.getConflict(query).conflict) { + throw new Error( + `Cannot deploy ${label}: the draft has a conflicting newer version on the server. Open it in the editor and resolve the conflict first.` + ) + } + const { state, failureMessage } = UserDraftDbSyncer.getState(query) + if (state === 'failed') { + throw new Error( + `Cannot deploy ${label}: saving the latest draft failed (${failureMessage ?? 'unknown error'}). Retry once the draft saves.` + ) + } +} + async function deployDraft( args: { type: WorkspaceItemType path: string trigger_kind?: TriggerKind deployment_message?: string + force?: boolean }, ctx: WriteDraftCtx ): Promise { const { workspace, toolId, toolCallbacks, sessionId } = ctx - const { type, path, trigger_kind: triggerKind, deployment_message: deploymentMessage } = args + const { + type, + path, + trigger_kind: triggerKind, + deployment_message: deploymentMessage, + force + } = args if (type === 'trigger' && !triggerKind) { throw new Error('trigger_kind is required when deploying a trigger.') } - const draft = getGlobalDraft(workspace, type, path, triggerKind) + const draft = await getGlobalDraft(workspace, type, path, triggerKind) if (!draft) { - throw new Error(`No local draft found for ${type} "${path}".`) + throw new Error(`No draft found for ${type} "${path}".`) } if (draft.value === undefined) { throw new Error(`Draft ${type} "${path}" has no value to deploy.`) @@ -3396,173 +4223,245 @@ async function deployDraft( let actions: ToolDisplayAction[] | undefined - switch (type) { - case 'script': { + if (type === 'script' || type === 'flow') { + // Promote the full persisted draft via the shared deploy module — the same + // "promote a draft to deployed" code the compare page's Review & Deploy uses. + // It deploys every field of the draft; the previous local builders dropped + // most config fields (tag, priority, schema, description, concurrency…), + // reading them from the already-deployed version instead. The other kinds + // below already deploy their draft value directly, so only script/flow need + // this. Scripts always create (with parent_hash); a flow on a deployed item + // updates, a draft-only flow (no flow row) is created. + // Address the draft by its STORAGE path: a draft_only item created in the + // editor lives at a synthetic `u/{user}/draft_{uuid}` key while its chosen + // path is held in the draft value. The shared deployer reads the draft via + // getScriptByPath/getFlowByPath at the path we pass (then deploys at the + // draft's own `path`), so passing the display/chosen path would 404. For a + // draft on a deployed item the storage path is just the item path. + const storagePath = getGlobalDraftStoragePath(workspace, type, path, triggerKind) + // The shared deployer re-reads the persisted DB draft, but an open editor's + // edit may still be parked in a debounced/disabled autosave. Flush it first so + // we deploy the latest value (not a stale persisted one) — and so the + // post-deploy draft delete doesn't drop an unsaved edit. `flush` always saves + // the parked value (like Ctrl/Cmd+S), since the user explicitly asked to deploy. + await flushDraftOrThrow({ workspace, itemKind: type, path: storagePath }, `${type} "${path}"`) + // Stale-draft guard: block when the draft was forked from an older deploy than + // the current head (unless force), pointing the model at rebase_draft. + if (type === 'script') { const existing = (await ScriptService.existsScriptByPath({ workspace, path })) ? await ScriptService.getScriptByPath({ workspace, path }) : undefined - const requestBody = buildScriptDeployRequestBody(path, draft, existing, deploymentMessage) - // Infer the arg schema from the content so it matches the code, like the editor does. - try { - const schema = emptySchema() - await inferArgs(requestBody.language, requestBody.content, schema) - requestBody.schema = schema - } catch (e) { - console.error('Failed to infer script schema before deploy', e) - } - await ScriptService.createScript({ workspace, requestBody }) - break - } - case 'flow': { - const flowDraft = draft.value as FlowDraftValue + assertDraftBasedOnLatest('script', path, draft.parentHash, existing?.hash, force) + } else { const existing = (await FlowService.existsFlowByPath({ workspace, path })) ? await FlowService.getFlowByPath({ workspace, path }) : undefined - const requestBody = buildFlowDeployRequestBody( - path, - draft.summary, - flowDraft, - existing, - deploymentMessage - ) - if (existing) { - await FlowService.updateFlow({ workspace, path, requestBody }) - } else { - await FlowService.createFlow({ workspace, requestBody }) - } - break + assertDraftBasedOnLatest('flow', path, draft.parentVersionId, existing?.version_id, force) } - case 'schedule': { - const requestBody = draft.value as any - if (await ScheduleService.existsSchedule({ workspace, path })) { - await ScheduleService.updateSchedule({ workspace, path, requestBody }) - } else { - await ScheduleService.createSchedule({ workspace, requestBody }) - } - actions = [createOpenScheduleAction(path, requestBody.is_flow ? 'flow' : 'script')] - break + const draftOnly = + type === 'flow' + ? !(await FlowService.existsFlowByPath({ workspace, path: storagePath })) + : false + const result = await deployDraftToWorkspace(type, storagePath, workspace, { + draftOnly, + deploymentMessage + }) + if (!result.success) { + throw new Error(result.error ?? `Failed to deploy ${type} "${path}".`) } - case 'trigger': { - const service = triggerServices[triggerKind!] - const requestBody = draft.value as { is_flow?: boolean } - if (await service.exists({ workspace, path })) { - await service.update({ workspace, path, requestBody }) - } else { - await service.create({ workspace, requestBody }) - } - actions = [ - createOpenTriggerAction(triggerKind!, path, requestBody.is_flow ? 'flow' : 'script') - ] - break - } - case 'resource': { - const requestBody = draft.value as any - if (await ResourceService.existsResource({ workspace, path })) { - await ResourceService.updateResource({ workspace, path, requestBody }) - } else { - await ResourceService.createResource({ workspace, requestBody }) - } - actions = [createOpenResourceAction(path)] - break - } - case 'variable': { - const requestBody = buildVariableDeployRequestBody( - workspace, - path, - draft.value as CreateVariable - ) - if (await VariableService.existsVariable({ workspace, path })) { - await VariableService.updateVariable({ workspace, path, requestBody }) - } else { - await VariableService.createVariable({ workspace, requestBody }) - } - actions = [createOpenVariableAction(path)] - break - } - case 'app': { - const appDraft = draft.value as AppDraftValue - const appValue: AppDraftValue = { - ...appDraft, - files: { ...(appDraft.files ?? {}) }, - runnables: { ...(appDraft.runnables ?? {}) }, - data: appDraft.data ?? { ...DEFAULT_RAW_APP_DATA } - } - await recomputeAppPolicy(appValue) - const policy = appValue.policy - if (!policy) { - throw new Error(`Draft app "${path}" has no policy to deploy.`) - } - - toolCallbacks.setToolStatus(toolId, { - content: `Bundling app "${path}"...` - }) - const bundle = await bundleRawAppDraft({ - workspace, - files: appValue.files, - onLog: (delta) => { - const lines = delta - .split('\n') - .map((line) => line.trim()) - .filter(Boolean) - const latest = lines[lines.length - 1] - if (latest) { - toolCallbacks.setToolStatus(toolId, { - content: `Bundling app "${path}"... ${latest}` - }) - } + } else { + switch (type) { + case 'schedule': { + const requestBody = draft.value as any + if (await ScheduleService.existsSchedule({ workspace, path })) { + await ScheduleService.updateSchedule({ workspace, path, requestBody }) + } else { + await ScheduleService.createSchedule({ workspace, requestBody }) } - }) - - toolCallbacks.setToolStatus(toolId, { - content: `Deploying app "${path}"...` - }) - const rawAppValue = { - files: appValue.files, - runnables: appValue.runnables, - data: appValue.data ?? { ...DEFAULT_RAW_APP_DATA } + actions = [createOpenScheduleAction(path, requestBody.is_flow ? 'flow' : 'script')] + break } - const summary = appValue.summary ?? draft.summary ?? '' - if (await AppService.existsApp({ workspace, path })) { - // Omit custom_path on update for now. The backend preserves it when absent, while - // sending it requires admin privileges; this chat deploy path does not yet mirror - // the raw app editor's user/admin-specific custom_path handling. - await AppService.updateAppRaw({ + case 'trigger': { + const service = triggerServices[triggerKind!] + const requestBody = draft.value as { is_flow?: boolean } + if (await service.exists({ workspace, path })) { + await service.update({ workspace, path, requestBody }) + } else { + await service.create({ workspace, requestBody }) + } + actions = [ + createOpenTriggerAction(triggerKind!, path, requestBody.is_flow ? 'flow' : 'script') + ] + break + } + case 'resource': { + const requestBody = draft.value as any + if (await ResourceService.existsResource({ workspace, path })) { + await ResourceService.updateResource({ workspace, path, requestBody }) + } else { + await ResourceService.createResource({ workspace, requestBody }) + } + actions = [createOpenResourceAction(path)] + break + } + case 'variable': { + // The chat keeps secret draft values only in memory (the DB draft + // stores `''`); buildVariableDeployRequestBody re-injects the ephemeral + // secret, so this can't go through the DB-reading shared deployer. + const requestBody = buildVariableDeployRequestBody( workspace, path, - formData: { - app: { - path, - value: rawAppValue, - summary, - policy, - deployment_message: deploymentMessage - }, - js: bundle.js, - css: bundle.css - } + draft.value as CreateVariable + ) + if (await VariableService.existsVariable({ workspace, path })) { + await VariableService.updateVariable({ workspace, path, requestBody }) + } else { + await VariableService.createVariable({ workspace, requestBody }) + } + actions = [createOpenVariableAction(path)] + break + } + case 'app': { + // Raw apps store a flat AppDraftValue (files/runnables at top level), + // not the deployed app's nested `value` shape the shared raw-app + // deployer reads, so they deploy through the chat's own bundle path. + const appDraft = draft.value as AppDraftValue + // Stale-draft guard: only fetch the deployed head when the draft records + // a fork base to compare against (pre-feature drafts have none). + if (draft.parentVersionId != null) { + const deployedApp = (await AppService.existsApp({ workspace, path })) + ? await AppService.getAppByPath({ workspace, path }) + : undefined + assertDraftBasedOnLatest( + 'app', + path, + draft.parentVersionId, + deployedApp?.versions?.[deployedApp.versions.length - 1], + force + ) + } + const appValue: AppDraftValue = { + ...appDraft, + files: { ...(appDraft.files ?? {}) }, + runnables: { ...(appDraft.runnables ?? {}) }, + data: appDraft.data ?? { ...DEFAULT_RAW_APP_DATA } + } + await recomputeAppPolicy(appValue) + const policy = appValue.policy + if (!policy) { + throw new Error(`Draft app "${path}" has no policy to deploy.`) + } + + toolCallbacks.setToolStatus(toolId, { + content: `Bundling app "${path}"...` }) - } else { - await AppService.createAppRaw({ + const bundle = await bundleRawAppDraft({ workspace, - formData: { - app: { - path, - value: rawAppValue, - summary, - policy, - deployment_message: deploymentMessage, - custom_path: appValue.custom_path - }, - js: bundle.js, - css: bundle.css + files: appValue.files, + onLog: (delta) => { + const lines = delta + .split('\n') + .map((line) => line.trim()) + .filter(Boolean) + const latest = lines[lines.length - 1] + if (latest) { + toolCallbacks.setToolStatus(toolId, { + content: `Bundling app "${path}"... ${latest}` + }) + } } + }) + + toolCallbacks.setToolStatus(toolId, { + content: `Deploying app "${path}"...` }) + const rawAppValue = { + files: appValue.files, + runnables: appValue.runnables, + data: appValue.data ?? { ...DEFAULT_RAW_APP_DATA } + } + const summary = appValue.summary ?? draft.summary ?? '' + // Deploy at the draft's chosen path. A draft_only raw app created in the + // editor lives at a synthetic `u/{user}/draft_{uuid}` storage key with its + // chosen path in the raw_app draft's `draft_path`; the chat's AppDraftValue + // doesn't carry it, so read it from the backend draft. For a chat-created app + // (real path, no draft_path) or a draft on a deployed app, the storage path + // is the deploy path. Same storage-path resolution as script/flow. + const storagePath = getGlobalDraftStoragePath(workspace, 'app', path) + // `draft_path` is read from the persisted backend draft below, but an + // editor rename may still be parked in a debounced/disabled autosave. + // Flush first (like script/flow) so we read the latest chosen path. + await flushDraftOrThrow( + { workspace, itemKind: 'raw_app', path: storagePath }, + `app "${path}"` + ) + let targetPath = storagePath + try { + const row = (await AppService.getAppByPath({ + workspace, + path: storagePath, + getDraft: true, + rawApp: true + })) as { draft?: { draft_path?: string; path?: string }; draft_path?: string } + targetPath = row?.draft?.draft_path ?? row?.draft?.path ?? row?.draft_path ?? storagePath + } catch (e) { + // Only a missing item (404) justifies falling back to the storage path; + // a real lookup failure (network/5xx) must abort rather than silently + // deploy to the wrong path. + if ((e as { status?: number } | null | undefined)?.status === 404) { + targetPath = storagePath + } else { + throw e + } + } + if (await AppService.existsApp({ workspace, path: targetPath })) { + // Omit custom_path on update for now. The backend preserves it when absent, while + // sending it requires admin privileges; this chat deploy path does not yet mirror + // the raw app editor's user/admin-specific custom_path handling. + await AppService.updateAppRaw({ + workspace, + path: targetPath, + formData: { + app: { + path: targetPath, + value: rawAppValue, + summary, + policy, + deployment_message: deploymentMessage, + // Preserve the policy's on_behalf_of: this chat deploy path has no + // on-behalf-of selector, so without the flag the backend resets it to + // the deploying user (gated server-side by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined + }, + js: bundle.js, + css: bundle.css + } + }) + } else { + await AppService.createAppRaw({ + workspace, + formData: { + app: { + path: targetPath, + value: rawAppValue, + summary, + policy, + deployment_message: deploymentMessage, + custom_path: appValue.custom_path, + // Preserve the policy's on_behalf_of (see update branch above). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined + }, + js: bundle.js, + css: bundle.css + } + }) + } + break } - break } } - deleteGlobalDraft(workspace, type, path, triggerKind, { preserveLiveDraft: true }) + await deleteGlobalDraft(workspace, type, path, triggerKind, { preserveLiveDraft: true }) // Reload the session preview if it's open on the deployed item. Map the // deploy type to the preview kind — a raw app deploys under 'app' but the @@ -3583,7 +4482,7 @@ async function deployDraft( return JSON.stringify( { success: true, - message: `Deployed local draft ${type} "${path}" to the workspace. Draft removed from the local draft system.`, + message: `Deployed draft ${type} "${path}" to the workspace. Draft removed.`, type, path, triggerKind @@ -3632,7 +4531,7 @@ async function deleteWorkspaceItem( break } - deleteGlobalDraft(workspace, type, path, triggerKind) + await deleteGlobalDraft(workspace, type, path, triggerKind) toolCallbacks.setToolStatus(toolId, { content: `Deleted ${type} "${path}"`, @@ -3641,7 +4540,7 @@ async function deleteWorkspaceItem( return JSON.stringify( { success: true, - message: `Deleted ${type} "${path}" from the workspace. Any matching local draft was also cleared.`, + message: `Deleted ${type} "${path}" from the workspace. Any matching draft was also cleared.`, type, path, triggerKind @@ -3652,13 +4551,36 @@ async function deleteWorkspaceItem( } export function prepareGlobalSystemMessage( - customPrompt?: string, - opts?: { previewTools?: boolean } + instructions?: { workspace?: string; user?: string }, + opts?: { + previewTools?: boolean + // Identity the path-convention guidance is built from. Production omits it + // (read from userStore); callers that must not touch the process-global + // store (the eval harness) pass it explicitly instead. + user?: { username: string; is_admin?: boolean; folders?: string[]; folders_read?: string[] } + skills?: AiSkillListItem[] + } ): ChatCompletionSystemMessageParam { - const username = get(userStore)?.username ?? '' - let content = buildGlobalSystemPrompt(username, opts?.previewTools ?? false) - if (customPrompt?.trim()) { - content = `${content}\n\nUSER GIVEN INSTRUCTIONS:\n${customPrompt.trim()}` + const user = opts?.user ?? get(userStore) + const username = user?.username ?? '' + const folderCtx: FolderPromptContext | undefined = user + ? { + folders: user.folders ?? [], + foldersRead: user.folders_read ?? user.folders ?? [], + isAdmin: user.is_admin ?? false + } + : undefined + let content = buildGlobalSystemPrompt( + username, + opts?.previewTools ?? false, + folderCtx, + opts?.skills ?? [] + ) + if (instructions?.workspace?.trim()) { + content = `${content}\n\nWORKSPACE INSTRUCTIONS (configured by a workspace admin, shared by everyone in this workspace — you cannot modify these):\n${instructions.workspace.trim()}` + } + if (instructions?.user?.trim()) { + content = `${content}\n\nUSER INSTRUCTIONS (this user's personal instructions — update them with the update_user_instructions tool when the user asks you to remember, change, or stop something):\n${instructions.user.trim()}` } return { @@ -3684,7 +4606,10 @@ export function prepareGlobalUserMessage( options: GlobalUserMessageOptions = {} ): ChatCompletionUserMessageParam { const selectedWorkspaceItems = selectedContext.filter( - (context) => context.type === 'workspace_script' || context.type === 'workspace_flow' + (context) => + context.type === 'workspace_script' || + context.type === 'workspace_flow' || + context.type === 'workspace_app' ) const activeEditor = options.activeEditor ?? @@ -3701,7 +4626,13 @@ export function prepareGlobalUserMessage( if (selectedWorkspaceItems.length > 0) { content += '## SELECTED CONTEXT\n' for (const context of selectedWorkspaceItems) { - content += `- type: ${context.type === 'workspace_script' ? 'script' : 'flow'}, path: ${context.path}\n` + const itemType = + context.type === 'workspace_script' + ? 'script' + : context.type === 'workspace_flow' + ? 'flow' + : 'raw_app' + content += `- type: ${itemType}, path: ${context.path}\n` } content += '\n' } diff --git a/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts b/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts deleted file mode 100644 index 88cbafe205..0000000000 --- a/frontend/src/lib/components/copilot/chat/global/deployRequests.test.ts +++ /dev/null @@ -1,165 +0,0 @@ -import { describe, expect, it } from 'vitest' -import type { Flow, NewScript, Script } from '$lib/gen/types.gen' -import { buildFlowDeployRequestBody, buildScriptDeployRequestBody } from './deployRequests' -import type { WorkspaceItem } from './workspaceItems' - -describe('global AI deploy request builders', () => { - it('preserves existing script metadata while replacing draft-controlled fields', () => { - const existing = { - hash: 'parent-hash', - path: 'f/demo/script', - summary: 'existing summary', - description: 'existing description', - content: 'old content', - schema: { properties: { name: { type: 'string' } } }, - is_template: true, - language: 'bun', - kind: 'script', - tag: 'node', - envs: ['ENV_A'], - concurrent_limit: 3, - concurrency_time_window_s: 60, - concurrency_key: 'key', - debounce_key: 'debounce', - debounce_delay_s: 5, - debounce_args_to_accumulate: ['ids'], - max_total_debouncing_time: 120, - max_total_debounces_amount: 4, - cache_ttl: 30, - cache_ignore_s3_path: true, - dedicated_worker: true, - ws_error_handler_muted: true, - priority: 9, - restart_unless_cancelled: true, - timeout: 300, - delete_after_secs: 600, - visible_to_runner_only: true, - auto_kind: 'script', - codebase: 'repo', - has_preprocessor: true, - on_behalf_of_email: 'deployer@example.com', - assets: [{ path: 's3://bucket/key', kind: 's3object' }], - modules: { 'helper.ts': { content: 'export const helper = 1', language: 'bun' } }, - labels: ['prod'], - lock: 'stale lock' - } as unknown as Script & Partial - const draft: WorkspaceItem = { - type: 'script', - path: 'f/demo/script', - summary: 'draft summary', - language: 'bun', - value: 'new content', - isDraft: true - } - - const requestBody = buildScriptDeployRequestBody('f/demo/script', draft, existing, 'ai deploy') - - expect(requestBody).toMatchObject({ - path: 'f/demo/script', - parent_hash: 'parent-hash', - summary: 'draft summary', - description: 'existing description', - content: 'new content', - schema: existing.schema, - tag: 'node', - envs: ['ENV_A'], - concurrent_limit: 3, - concurrency_time_window_s: 60, - concurrency_key: 'key', - debounce_key: 'debounce', - debounce_delay_s: 5, - debounce_args_to_accumulate: ['ids'], - max_total_debouncing_time: 120, - max_total_debounces_amount: 4, - cache_ttl: 30, - cache_ignore_s3_path: true, - dedicated_worker: true, - ws_error_handler_muted: true, - priority: 9, - restart_unless_cancelled: true, - timeout: 300, - delete_after_secs: 600, - visible_to_runner_only: true, - auto_kind: 'script', - codebase: 'repo', - has_preprocessor: true, - on_behalf_of_email: 'deployer@example.com', - preserve_on_behalf_of: true, - assets: existing.assets, - modules: existing.modules, - labels: ['prod'], - deployment_message: 'ai deploy' - }) - expect(requestBody.lock).toBeUndefined() - }) - - it('preserves existing flow metadata and uses draft value/schema overrides', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - value: { modules: [] }, - schema: { required: ['name'] }, - tag: 'python', - ws_error_handler_muted: true, - priority: 7, - dedicated_worker: true, - timeout: 60, - visible_to_runner_only: true, - on_behalf_of_email: 'deployer@example.com', - labels: ['critical'] - } as unknown as Flow - const draftValue = { - value: { modules: [{ id: 'step', value: { type: 'identity' } }] }, - schema: { properties: { name: { type: 'string' } } }, - groups: [{ start_id: 'step', end_id: 'step', summary: 'Group' }] - } - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - undefined, - draftValue as any, - existing, - 'ai deploy' - ) - - expect(requestBody).toMatchObject({ - path: 'f/demo/flow', - summary: 'existing summary', - description: 'existing description', - schema: draftValue.schema, - tag: 'python', - ws_error_handler_muted: true, - priority: 7, - dedicated_worker: true, - timeout: 60, - visible_to_runner_only: true, - on_behalf_of_email: 'deployer@example.com', - preserve_on_behalf_of: true, - labels: ['critical'], - deployment_message: 'ai deploy' - }) - expect(requestBody.value.modules).toHaveLength(1) - expect(requestBody.value.groups).toEqual(draftValue.groups) - }) - - it('falls back to existing flow schema when the draft has no schema', () => { - const existing = { - path: 'f/demo/flow', - summary: 'existing summary', - value: { modules: [] }, - schema: { properties: { existing: { type: 'boolean' } } } - } as unknown as Flow - - const requestBody = buildFlowDeployRequestBody( - 'f/demo/flow', - 'draft summary', - { value: { modules: [] }, schema: null, groups: null }, - existing, - undefined - ) - - expect(requestBody.summary).toBe('draft summary') - expect(requestBody.schema).toBe(existing.schema) - }) -}) diff --git a/frontend/src/lib/components/copilot/chat/global/deployRequests.ts b/frontend/src/lib/components/copilot/chat/global/deployRequests.ts deleted file mode 100644 index 9e779779f6..0000000000 --- a/frontend/src/lib/components/copilot/chat/global/deployRequests.ts +++ /dev/null @@ -1,101 +0,0 @@ -import type { Flow, NewScript, OpenFlowWPath, Script } from '$lib/gen/types.gen' -import type { FlowDraftValue, WorkspaceItem } from './workspaceItems' - -type ScriptWithDeployMetadata = Script & Partial> - -export type FlowDeployRequestBody = OpenFlowWPath & { - deployment_message?: string -} - -function preserveOnBehalfOf(email: string | undefined): true | undefined { - return email ? true : undefined -} - -export function buildScriptDeployRequestBody( - path: string, - draft: WorkspaceItem, - existing: Script | undefined, - deploymentMessage: string | undefined -): NewScript { - if (typeof draft.value !== 'string' || !draft.language) { - throw new Error(`Draft script "${path}" is missing content or language.`) - } - - const existingWithMetadata = existing as ScriptWithDeployMetadata | undefined - - return { - path, - summary: draft.summary ?? existing?.summary ?? '', - description: existing?.description ?? '', - content: draft.value, - parent_hash: existing?.hash, - schema: existing?.schema, - is_template: existing?.is_template, - language: draft.language, - kind: existing?.kind, - tag: existing?.tag, - envs: existing?.envs, - concurrent_limit: existing?.concurrent_limit, - concurrency_time_window_s: existing?.concurrency_time_window_s, - debounce_key: existing?.debounce_key, - debounce_delay_s: existing?.debounce_delay_s, - debounce_args_to_accumulate: existing?.debounce_args_to_accumulate, - max_total_debouncing_time: existing?.max_total_debouncing_time, - max_total_debounces_amount: existing?.max_total_debounces_amount, - cache_ttl: existing?.cache_ttl, - cache_ignore_s3_path: existingWithMetadata?.cache_ignore_s3_path, - dedicated_worker: existing?.dedicated_worker, - ws_error_handler_muted: existing?.ws_error_handler_muted, - priority: existing?.priority, - restart_unless_cancelled: existing?.restart_unless_cancelled, - timeout: existing?.timeout, - delete_after_secs: existing?.delete_after_secs, - deployment_message: deploymentMessage, - concurrency_key: existing?.concurrency_key, - visible_to_runner_only: existing?.visible_to_runner_only, - auto_kind: existing?.auto_kind, - codebase: existing?.codebase, - has_preprocessor: existing?.has_preprocessor, - on_behalf_of_email: existing?.on_behalf_of_email, - preserve_on_behalf_of: preserveOnBehalfOf(existing?.on_behalf_of_email), - assets: existingWithMetadata?.assets, - modules: existing?.modules, - labels: existing?.labels - } -} - -function flowValueWithDraftGroups(flowDraft: FlowDraftValue): FlowDraftValue['value'] { - if (flowDraft.groups === undefined) { - return flowDraft.value - } - return { - ...flowDraft.value, - groups: flowDraft.groups ?? undefined - } -} - -export function buildFlowDeployRequestBody( - path: string, - draftSummary: string | undefined, - flowDraft: FlowDraftValue, - existing: Flow | undefined, - deploymentMessage: string | undefined -): FlowDeployRequestBody { - return { - path, - summary: draftSummary ?? existing?.summary ?? '', - description: existing?.description ?? '', - value: flowValueWithDraftGroups(flowDraft), - schema: flowDraft.schema ?? existing?.schema ?? {}, - tag: existing?.tag, - ws_error_handler_muted: existing?.ws_error_handler_muted, - priority: existing?.priority, - dedicated_worker: existing?.dedicated_worker, - timeout: existing?.timeout, - visible_to_runner_only: existing?.visible_to_runner_only, - on_behalf_of_email: existing?.on_behalf_of_email, - preserve_on_behalf_of: preserveOnBehalfOf(existing?.on_behalf_of_email), - labels: existing?.labels, - deployment_message: deploymentMessage - } -} diff --git a/frontend/src/lib/components/copilot/chat/global/gate.ts b/frontend/src/lib/components/copilot/chat/global/gate.ts index 664f06ff39..df8035de87 100644 --- a/frontend/src/lib/components/copilot/chat/global/gate.ts +++ b/frontend/src/lib/components/copilot/chat/global/gate.ts @@ -11,8 +11,8 @@ * When the mode is ready to ship to everyone, replace every call to * `isGlobalAiEnabled()` with `true` and delete this file. The references are * intentionally narrow (chat mode visibility, custom prompt settings, the - * `change_mode` tool enum, and the `/global_drafts` dev route) so the rip-out - * is a small grep. + * `change_mode` tool enum, the AI skills workspace settings tab, and the + * `/global_drafts` dev route) so the rip-out is a small grep. */ const STORAGE_KEY = 'wm_dev_global_ai' diff --git a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts index 9a6d9125f4..87ca767904 100644 --- a/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts +++ b/frontend/src/lib/components/copilot/chat/global/userDraftAdapter.ts @@ -1,11 +1,10 @@ import type { Flow, NewSchedule, NewScript } from '$lib/gen/types.gen' +import { DraftService } from '$lib/gen' +import { get } from 'svelte/store' +import { userStore } from '$lib/stores' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { DEFAULT_DATA as DEFAULT_RAW_APP_DATA } from '$lib/components/raw_apps/dataTableRefUtils' -import { - UserDraft, - type UserDraftEntry, - type UserDraftItemKind, - type UserDraftMeta -} from '$lib/userDraft.svelte' +import { UserDraft, type UserDraftEntry, type UserDraftItemKind } from '$lib/userDraft.svelte' import { getWorkspaceItemKey, type AppDraftValue, @@ -67,7 +66,10 @@ function normalizeAppDraftValue(value: AppDraftValue): AppDraftValue { runnables: { ...(value.runnables ?? {}) }, data: value.data ?? { ...DEFAULT_RAW_APP_DATA }, policy: value.policy === undefined ? undefined : clone(value.policy), - custom_path: value.custom_path + custom_path: value.custom_path, + // Carry the fork-base version through the whitelist — it is dropped on every + // save otherwise, which would defeat the stale-draft check. + parent_version: value.parent_version } } @@ -136,6 +138,7 @@ function scriptDraftToWorkspaceItem(path: string, draft: NewScript): WorkspaceIt summary: draft.summary, language: draft.language, value: draft.content, + parentHash: draft.parent_hash, isDraft: true } } @@ -145,10 +148,15 @@ function flowDraftToWorkspaceItem(path: string, draft: Flow): WorkspaceItem { type: 'flow', path, summary: draft.summary, + // The persisted flow draft carries `version_id` (the deployed head it was + // forked from, pinned at fork by writeDraft/the editor) — the flow analog + // of a script's parent_hash. + parentVersionId: draft.version_id, value: { value: draft.value, schema: draft.schema ?? null, - groups: draft.value.groups ?? null + groups: draft.value.groups ?? null, + description: draft.description ?? null }, isDraft: true } @@ -160,6 +168,7 @@ function appDraftToWorkspaceItem(path: string, draft: AppDraftValue): WorkspaceI type: 'app', path, summary: value.summary, + parentVersionId: value.parent_version, value, isDraft: true } @@ -326,17 +335,178 @@ function getGlobalDraftSlot( return { itemKind, storagePath, displayPath, item } } -export function getGlobalDraft( +// Current user's persisted draft value (+ records the sync baseline so a later +// save detects external conflicts). undefined on 404 (no draft at that path). +async function fetchBackendDraftValue( + workspace: string, + itemKind: UserDraftItemKind, + storagePath: string +): Promise { + try { + const resp = await DraftService.getDraftForUser({ + workspace, + kind: itemKind as any, + path: storagePath, + username: get(userStore)?.username + }) + UserDraftDbSyncer.recordRemoteSync({ workspace, itemKind, path: storagePath }, resp.created_at) + return resp.value ?? undefined + } catch (e) { + // 404 = no draft for this owner at that path (the intended empty case). + // Anything else (403/500/network) MUST propagate: swallowing it would make + // the write merge fall through to the deployed item instead of the user's + // in-progress draft, silently overwriting their draft-only changes. + if ((e as { status?: number } | null | undefined)?.status === 404) return undefined + throw e + } +} + +// Draft VALUE for a write merge: cell-if-present (the user's freshest in-tab +// edits) else the current user's backend draft. +export async function readGlobalDraftValue( workspace: string, type: WorkspaceItemType, path: string, triggerKind?: TriggerKind -): WorkspaceItem | undefined { - return getGlobalDraftSlot(workspace, type, path, triggerKind)?.item +): Promise { + const itemKind = itemKindFor(type, triggerKind) + if (!itemKind) return undefined + const storagePath = resolveDraftStoragePath(workspace, itemKind, path) + const cell = UserDraft.get(itemKind, storagePath, { workspace }) + if (cell !== undefined) return cell + return (await fetchBackendDraftValue(workspace, itemKind, storagePath)) as V | undefined } -export function listGlobalDrafts(workspace: string): WorkspaceItem[] { +export type DraftPersistResult = + | { status: 'saved'; item: WorkspaceItem } + | { status: 'conflict'; item: WorkspaceItem; serverTimestamp?: string } + | { status: 'error'; item: WorkspaceItem; message: string } + +// Persist a built draft value. `UserDraft.seed` reflects it into an open editor's +// cell WITHOUT a double-POST (no-ops if no cell; its seedNextWrite suppresses the +// cell's autosave mirror), then the awaited immediate save is the single source of +// persistence + conflict detection against the shared baseline. force overwrites. +export async function persistGlobalDraft( + workspace: string, + type: WorkspaceItemType, + path: string, + value: unknown, + opts: { triggerKind?: TriggerKind; force?: boolean } = {} +): Promise { + const itemKind = itemKindFor(type, opts.triggerKind) + if (!itemKind) throw new Error(`Unsupported draft type "${type}".`) + const storagePath = resolveDraftStoragePath(workspace, itemKind, path) + UserDraft.seed(itemKind, storagePath, value, { workspace }) + await UserDraftDbSyncer.save({ + workspace, + itemKind, + path: storagePath, + value, + immediate: true, + force: opts.force + }) + const { displayPath, isLiveDraft } = liveDisplayPath(workspace, itemKind, storagePath) + const item = userDraftEntryToWorkspaceItem( + { workspace, itemKind, path: storagePath, value }, + displayPath, + isLiveDraft + ) + if (!item) throw new Error(`Could not synthesize ${type} draft "${path}".`) + // A failed save (network/5xx) is recorded in the syncer's failure map, not + // thrown — so check it before reporting success, else a write tool would tell + // the chat "saved" while the DB-backed source of truth was never updated. + const saveState = UserDraftDbSyncer.getState({ workspace, itemKind, path: storagePath }) + if (saveState.state === 'failed') { + return { status: 'error', item, message: saveState.failureMessage ?? 'Draft save failed' } + } + const conflict = opts.force + ? undefined + : UserDraftDbSyncer.getConflict({ workspace, itemKind, path: storagePath }).conflict + return conflict + ? { status: 'conflict', item, serverTimestamp: conflict.serverTimestamp } + : { status: 'saved', item } +} + +export async function getGlobalDraft( + workspace: string, + type: WorkspaceItemType, + path: string, + triggerKind?: TriggerKind +): Promise { + const slot = getGlobalDraftSlot(workspace, type, path, triggerKind) + if (slot) return slot.item + const itemKind = itemKindFor(type, triggerKind) + if (!itemKind) return undefined + const storagePath = resolveDraftStoragePath(workspace, itemKind, path) + const value = await fetchBackendDraftValue(workspace, itemKind, storagePath) + if (value === undefined || value === null) return undefined + const { displayPath, isLiveDraft } = liveDisplayPath(workspace, itemKind, storagePath) + return userDraftEntryToWorkspaceItem( + { workspace, itemKind, path: storagePath, value }, + displayPath, + isLiveDraft + ) +} + +// Maps a backend `listDrafts` metadata row (no value) to a lightweight item. +// The row's `path` is the storage path; remap it to the live editor's effective +// path (and flag it) when one is open on this key, matching the cell path. +function backendDraftRowToWorkspaceItem( + workspace: string, + row: { + kind: string + path: string + summary?: string + } +): WorkspaceItem | undefined { + if (!(GLOBAL_DRAFT_KINDS as readonly string[]).includes(row.kind)) return undefined + let type: WorkspaceItemType + let triggerKind: TriggerKind | undefined + switch (row.kind) { + case 'script': + case 'flow': + case 'resource': + case 'variable': + type = row.kind + break + case 'raw_app': + type = 'app' + break + case 'trigger_schedule': + type = 'schedule' + break + default: { + const tk = TRIGGER_KIND_BY_DRAFT_KIND[row.kind as UserDraftItemKind] + if (!tk) return undefined + type = 'trigger' + triggerKind = tk + } + } + const { displayPath, isLiveDraft } = liveDisplayPath( + workspace, + row.kind as UserDraftItemKind, + row.path + ) + return { + type, + path: displayPath, + summary: row.summary, + value: undefined, + isDraft: true, + triggerKind, + ...(isLiveDraft ? { isLiveDraft: true } : {}) + } +} + +export async function listGlobalDrafts(workspace: string): Promise { const drafts = new Map() + const rows = await DraftService.listDrafts({ workspace }) + for (const row of rows) { + const item = backendDraftRowToWorkspaceItem(workspace, row) + if (!item) continue + drafts.set(getWorkspaceItemKey(item.type, item.path, item.triggerKind), item) + } + // Overlay live in-tab cells (full values + the user's live edits); cell wins. for (const entry of UserDraft.list({ workspace, itemKinds: [...GLOBAL_DRAFT_KINDS] })) { const { displayPath, isLiveDraft } = liveDisplayPath(workspace, entry.itemKind, entry.path) const draft = userDraftEntryToWorkspaceItem(entry, displayPath, isLiveDraft) @@ -346,35 +516,27 @@ export function listGlobalDrafts(workspace: string): WorkspaceItem[] { return Array.from(drafts.values()) } -export function saveGlobalAppDraft( +export async function saveGlobalAppDraft( workspace: string, path: string, - value: AppDraftValue, - meta?: UserDraftMeta -): WorkspaceItem { - const storagePath = resolveDraftStoragePath(workspace, 'raw_app', path) - const normalized = normalizeAppDraftValue(value) - if (meta) { - UserDraft.setDraftAndMeta('raw_app', storagePath, normalized, meta, { workspace }) - } else { - UserDraft.save('raw_app', storagePath, normalized, { workspace }) - } - const stored = getGlobalDraft(workspace, 'app', path) - if (!stored) throw new Error(`Could not read written app draft "${path}".`) - return stored + value: AppDraftValue +): Promise { + // Return the full result (not just the item) so app write tools surface a + // conflict / save failure instead of reporting every stale write as saved. + return persistGlobalDraft(workspace, 'app', path, normalizeAppDraftValue(value), {}) } type DeleteGlobalDraftOptions = { preserveLiveDraft?: boolean } -export function deleteGlobalDraft( +export async function deleteGlobalDraft( workspace: string, type: WorkspaceItemType, path: string, triggerKind?: TriggerKind, options: DeleteGlobalDraftOptions = {} -): void { +): Promise { const itemKind = itemKindFor(type, triggerKind) if (!itemKind) return const storagePath = resolveDraftStoragePath(workspace, itemKind, path) @@ -384,6 +546,27 @@ export function deleteGlobalDraft( } else { UserDraft.clear(itemKind, storagePath, { workspace }) } + // `remove`/`clear` only debounce the delete; persist it now so a deploy/discard + // that the caller awaits has actually cleared the server draft on return. + await UserDraftDbSyncer.save({ + workspace, + itemKind, + path: storagePath, + value: null, + immediate: true + }) + // A failed (network/5xx) or conflicted delete is recorded in the syncer state, + // not thrown — surface it so callers don't report the draft as removed while + // the DB-backed source of truth still has it (same guard as the write path). + const state = UserDraftDbSyncer.getState({ workspace, itemKind, path: storagePath }) + if (state.state === 'failed') { + throw new Error(state.failureMessage ?? `Failed to delete draft "${path}".`) + } + if (UserDraftDbSyncer.getConflict({ workspace, itemKind, path: storagePath }).conflict) { + throw new Error( + `Draft "${path}" changed externally since you last read it; it was not removed. Re-read and retry.` + ) + } if (type === 'variable') clearEphemeralSecretVariableDraftValue(workspace, storagePath) } diff --git a/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts b/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts index acc2a5721a..a8196a5054 100644 --- a/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts +++ b/frontend/src/lib/components/copilot/chat/global/workspaceItems.ts @@ -28,6 +28,7 @@ export type FlowDraftValue = { value: FlowValue schema?: Record | null groups?: NonNullable | null + description?: string | null } export const TRIGGER_KINDS = [ @@ -71,6 +72,9 @@ export type AppDraftValue = { data?: any policy?: Policy custom_path?: string + // Fork base: the deployed app version this draft was started from, pinned at + // fork. The app analog of a script's parent_hash / a flow's version_id. + parent_version?: number } export type ResourceDraftState = { @@ -98,6 +102,11 @@ export type WorkspaceItem = { summary?: string language?: ScriptLang triggerKind?: TriggerKind + // Fork base: the deployed version this draft was started from, compared against + // the current deployed head to detect a stale draft. `parentHash` for scripts + // (script hash), `parentVersionId` for flows (flow_version id). + parentHash?: string + parentVersionId?: number value?: | string | FlowDraftValue diff --git a/frontend/src/lib/components/copilot/chat/mention.test.ts b/frontend/src/lib/components/copilot/chat/mention.test.ts new file mode 100644 index 0000000000..8d379ce3b4 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.test.ts @@ -0,0 +1,54 @@ +import { describe, expect, it } from 'vitest' +import { MENTION_RE, mentionTitle, formatMention } from './mention' + +describe('formatMention', () => { + it('leaves a simple name bare', () => { + expect(formatMention('app.ts')).toBe('@app.ts') + expect(formatMention('proj/sub/a.ts')).toBe('@proj/sub/a.ts') + }) + it('brackets a name containing whitespace', () => { + expect(formatMention('my file.txt')).toBe('@[my file.txt]') + expect(formatMention('my folder/a b.ts')).toBe('@[my folder/a b.ts]') + }) + it('brackets names with HTML-sensitive chars, parens, brackets', () => { + expect(formatMention('R&D notes.md')).toBe('@[R&D notes.md]') + expect(formatMention('a.txt')).toBe('@[a.txt]') + expect(formatMention('report(final).csv')).toBe('@[report(final).csv]') + }) +}) + +describe('mentionTitle', () => { + it('strips the @ from a bare mention', () => { + expect(mentionTitle('@app.ts')).toBe('app.ts') + }) + it('strips the @[ ] from a bracketed mention', () => { + expect(mentionTitle('@[my file.txt]')).toBe('my file.txt') + }) +}) + +describe('MENTION_RE', () => { + it('captures a bracketed (spaced) mention whole alongside bare ones', () => { + const tokens = [...'see @app.ts and @[my file.txt] ok'.matchAll(MENTION_RE)].map((m) => m[0]) + expect(tokens).toEqual(['@app.ts', '@[my file.txt]']) + expect(tokens.map(mentionTitle)).toEqual(['app.ts', 'my file.txt']) + }) + it('round-trips formatMention → MENTION_RE → mentionTitle for a spaced name', () => { + const name = 'my notes (v2).md' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips a name containing both whitespace and a closing bracket', () => { + const name = 'notes ] draft.md' + expect(formatMention(name)).toBe('@[notes \\] draft.md]') + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(m[0]).toBe('@[notes \\] draft.md]') + expect(mentionTitle(m[0])).toBe(name) + }) + + it('round-trips an HTML-sensitive name (highlighter handles HTML-escaping separately)', () => { + const name = 'a & c].txt' + const m = `x ${formatMention(name)} y`.match(MENTION_RE)! + expect(mentionTitle(m[0])).toBe(name) + }) +}) diff --git a/frontend/src/lib/components/copilot/chat/mention.ts b/frontend/src/lib/components/copilot/chat/mention.ts new file mode 100644 index 0000000000..65a2b3b7c6 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/mention.ts @@ -0,0 +1,31 @@ +/** + * `@mention` formatting shared between the chat input (which inserts mentions) and the + * textarea highlighter (which parses them) so the two never disagree. + * + * A simple name is inserted bare (`@app.ts`); a name containing whitespace is bracketed + * (`@[my file.txt]`) so it's captured whole instead of truncating at the first space. + */ + +/** + * Matches a mention token: a bracketed `@[name with spaces]` (where `\]` and `\\` are + * escaped, so a `]` inside the name doesn't end the token early) first, then a bare `@name`. + */ +export const MENTION_RE = /@\[(?:\\.|[^\]\\\r\n])*\]|@[\w/.\-\[\]]+/g + +/** The title of a mention token (`@name` or `@[name]`), brackets stripped and unescaped. */ +export function mentionTitle(token: string): string { + if (token.startsWith('@[') && token.endsWith(']')) { + return token.slice(2, -1).replace(/\\(.)/g, '$1') + } + return token.slice(1) +} + +/** Chars the bare `@name` regex matches without truncating; anything else needs brackets. */ +const BARE_SAFE = /^[\w/.\-]+$/ + +/** Format a name as a mention token. A bare `@name` only survives for simple names; anything + * with whitespace, HTML-sensitive chars (`< > &`), brackets, parens, etc. is bracketed (with + * `\` and `]` escaped) so the token is captured whole and round-trips through the parser. */ +export function formatMention(name: string): string { + return BARE_SAFE.test(name) ? `@${name}` : `@[${name.replace(/[\\\]]/g, '\\$&')}]` +} diff --git a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts index dfe3eb7565..1600d8fe9a 100644 --- a/frontend/src/lib/components/copilot/chat/monaco-adapter.ts +++ b/frontend/src/lib/components/copilot/chat/monaco-adapter.ts @@ -129,9 +129,17 @@ export class AIChatEditorHandler { const deletedChange = group.changes[0] const addedChange = group.changes[1] if (deletedChange.type === 'deleted' && addedChange.type === 'added_block') { - applyChange(this.editor, deletedChange) - addedChange.position.afterLineNumber = deletedChange.range.startLine - 1 - applyChange(this.editor, addedChange) + this.editor.executeEdits('chat', [ + { + range: { + startLineNumber: deletedChange.range.startLine, + startColumn: 1, + endLineNumber: deletedChange.range.endLine + 1, + endColumn: 0 + }, + text: addedChange.value + '\n' + } + ]) } else { throw new Error('Invalid group') } @@ -284,7 +292,7 @@ export class AIChatEditorHandler { }) if (!opts?.applyAll) { - ;({ collection, ids } = await displayVisualChanges( + ; ({ collection, ids } = await displayVisualChanges( 'editor-windmill-chat-style', this.editor, changes, diff --git a/frontend/src/lib/components/copilot/chat/navigator/core.ts b/frontend/src/lib/components/copilot/chat/navigator/core.ts index 6da158e6de..945f994862 100644 --- a/frontend/src/lib/components/copilot/chat/navigator/core.ts +++ b/frontend/src/lib/components/copilot/chat/navigator/core.ts @@ -4,6 +4,7 @@ import type { ChatCompletionUserMessageParam } from 'openai/resources/index.mjs' import { createSearchWorkspaceTool, createGetRunnableDetailsTool, type Tool } from '../shared' +import { readDocsPageTool, searchDocsTool } from '../docs/core' import { ResourceService } from '$lib/gen' import { workspaceStore } from '$lib/stores' import { get } from 'svelte/store' @@ -16,13 +17,14 @@ Windmill is an open-source developer platform for building internal tools, API i You have access to these tools: 1. View current buttons and inputs on the page (get_triggerable_components) 2. Execute buttons and inputs (trigger_component) -3. Get documentation for user requests (get_documentation) -4. Change the AI mode to the one specified (change_mode) -5. Search for scripts and flows in the workspace (search_workspace) -6. Get detailed information about a specific script or flow (get_runnable_details) +3. Search the documentation (search_docs) +4. Read a documentation page (read_docs_page) +5. Change the AI mode to the one specified (change_mode) +6. Search for scripts and flows in the workspace (search_workspace) +7. Get detailed information about a specific script or flow (get_runnable_details) INSTRUCTIONS: -- When users ask about application features or concepts, first use get_documentation internally to retrieve accurate information about how to fulfill the user's request. +- When users ask about application features or concepts, first use search_docs (with a few keywords) and, when a snippet is not enough, read_docs_page on a returned Source URL to retrieve accurate information about how to fulfill the user's request. - Then immediately use the available tools to guide the user through the application. Do not wait for the user's confirmation before taking action. - If you detect a confirmation modal that needs user confirmation, stop the navigation and let the user know that the action is pending confirmation. - Use get_triggerable_components to understand available options, and then trigger the components using trigger_component. Then wait a moment before rescanning the current page, and then continue with the next step. Do this 5 times max. @@ -59,30 +61,12 @@ When you complete the user's request, do not say "I created..." or "I updated... Example of good behavior: - User: "How can I set my AI providers?" -- You: +- You: - You: - You: - You: "" ` -const GET_DOCUMENTATION_TOOL: ChatCompletionTool = { - type: 'function', - function: { - name: 'get_documentation', - description: 'Get the documentation for the user request', - parameters: { - type: 'object', - properties: { - request: { - type: 'string', - description: 'The user request' - } - }, - required: ['request'] - } - } -} - // Tool definitions const GET_TRIGGERABLE_COMPONENTS_TOOL: ChatCompletionTool = { type: 'function', @@ -234,47 +218,6 @@ function triggerComponent(args: { id: string; value: string }): string { } } -async function getDocumentation(args: { request: string }): Promise { - const retrieval = await fetch('/api/inkeep', { - method: 'POST', - headers: { - 'Content-Type': 'application/json' - }, - body: JSON.stringify({ - query: args.request - }) - }) - - if (!retrieval.ok) { - const errorText = await retrieval.text() - throw new Error(errorText) - } - - const data = await retrieval.json() - if (!data.choices?.[0]?.message?.content) { - return 'No documentation found for this request' - } - - // Parse the raw response - const raw = data.choices[0].message.content - const parsed = JSON.parse(raw) - - // Clean up the response to include only essential information - if (parsed.content && Array.isArray(parsed.content)) { - const cleanedContent = parsed.content.map((item: any) => ({ - title: item.title, - url: item.url, - content: item.source?.content.map((c: any) => c.text).join('\n') || [] - })) - // Limit the response to 30000 characters max - const stringified = JSON.stringify({ content: cleanedContent }).slice(0, 30000) - - return stringified - } - - return data.choices[0].message.content -} - async function getAvailableResources(args: { resource_type: string }): Promise { const resources = await ResourceService.listResource({ workspace: get(workspaceStore) as string, @@ -318,27 +261,6 @@ const getCurrentPageNameTool: Tool<{}> = { } } -export const getDocumentationTool: Tool<{}> = { - def: GET_DOCUMENTATION_TOOL, - fn: async ({ args, toolId, toolCallbacks }) => { - toolCallbacks.setToolStatus(toolId, { content: 'Getting documentation...' }) - try { - const docResult = await getDocumentation(args) - toolCallbacks.setToolStatus(toolId, { content: 'Retrieved documentation' }) - return docResult - } catch (error) { - toolCallbacks.setToolStatus(toolId, { - content: 'Error getting documentation', - error: 'Error getting documentation' - }) - console.error('Error getting documentation:', error) - const errorMessage = - error instanceof Error ? error.message : 'An error occurred while getting documentation' - return `Failed to get documentation: ${errorMessage}, pursuing with the user request...` - } - } -} - const getAvailableResourcesTool: Tool<{}> = { def: GET_AVAILABLE_RESOURCES_TOOL, fn: async ({ args, toolId, toolCallbacks }) => { @@ -361,7 +283,8 @@ const getAvailableResourcesTool: Tool<{}> = { export const navigatorTools: Tool<{}>[] = [ getTriggerableComponentsTool, triggerComponentTool, - getDocumentationTool, + searchDocsTool, + readDocsPageTool, getCurrentPageNameTool, getAvailableResourcesTool, createSearchWorkspaceTool(), diff --git a/frontend/src/lib/components/copilot/chat/openai-responses.ts b/frontend/src/lib/components/copilot/chat/openai-responses.ts index e99591d379..eb1dd2b2d5 100644 --- a/frontend/src/lib/components/copilot/chat/openai-responses.ts +++ b/frontend/src/lib/components/copilot/chat/openai-responses.ts @@ -130,10 +130,6 @@ function convertCompletionConfigToResponsesConfig( responsesConfig.max_output_tokens = config.max_tokens } - // Keep other relevant fields - if (config.temperature !== undefined) { - responsesConfig.temperature = config.temperature - } if ('tools' in config && config.tools && config.tools.length > 0) { responsesConfig.tools = config.tools.map((tool) => { if (tool.type === 'function' && 'function' in tool) { diff --git a/frontend/src/lib/components/copilot/chat/openaiReasoning.ts b/frontend/src/lib/components/copilot/chat/openaiReasoning.ts index da809641a0..69d538a1a8 100644 --- a/frontend/src/lib/components/copilot/chat/openaiReasoning.ts +++ b/frontend/src/lib/components/copilot/chat/openaiReasoning.ts @@ -24,6 +24,45 @@ export function getReasoningContentDelta( return (delta as ChatCompletionDeltaWithReasoning).reasoning_content } +/** + * Mistral streams reasoning as structured content parts instead of a + * `reasoning_content` field: `delta.content` is an array of + * `{type: 'thinking', thinking: [{type: 'text', text}]}` chunks while the + * model thinks, then plain strings for the answer. Split a content delta + * into its reasoning and answer text (a plain-string delta is all answer). + */ +export function splitContentDelta(content: unknown): { reasoning: string; text: string } { + if (typeof content === 'string') { + return { reasoning: '', text: content } + } + if (!Array.isArray(content)) { + return { reasoning: '', text: '' } + } + let reasoning = '' + let text = '' + for (const part of content) { + if (part?.type === 'thinking' && Array.isArray(part.thinking)) { + for (const t of part.thinking) { + if (t?.type === 'text' && typeof t.text === 'string') { + reasoning += t.text + } + } + } else if (part?.type === 'text' && typeof part.text === 'string') { + text += part.text + } + } + return { reasoning, text } +} + +/** + * Whether the provider tolerates `reasoning_content` on input messages. + * DeepSeek wants the field replayed in tool-call loops and pass-through + * gateways ignore it, but Mistral hard-422s (`extra_forbidden`). + */ +function acceptsReasoningEcho(provider?: string): boolean { + return provider !== 'mistral' +} + export function buildAssistantTextMessage( content: string ): ChatCompletionAssistantMessageWithReasoning { @@ -36,16 +75,19 @@ export function buildAssistantTextMessage( export function buildAssistantToolCallMessage({ content, reasoning, - toolCalls + toolCalls, + provider }: { content: string reasoning: ReasoningContentState toolCalls: ChatCompletionMessageFunctionToolCall[] + provider?: string }): ChatCompletionAssistantMessageWithReasoning { + const echoReasoning = reasoning.hasReasoningContent && acceptsReasoningEcho(provider) return { role: 'assistant', ...(content || reasoning.hasReasoningContent ? { content } : {}), - ...(reasoning.hasReasoningContent ? { reasoning_content: reasoning.reasoningContent } : {}), + ...(echoReasoning ? { reasoning_content: reasoning.reasoningContent } : {}), tool_calls: toolCalls } } diff --git a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte index 3709eda53c..d0a4cba462 100644 --- a/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte +++ b/frontend/src/lib/components/copilot/chat/script/CodeDisplay.svelte @@ -17,6 +17,7 @@ import { AIMode } from '../AIChatManager.svelte' import { getAiChatManager } from '../aiChatManagerContext' import { Check, Play } from 'lucide-svelte' + import MermaidDisplay from './MermaidDisplay.svelte' const aiChatManager = getAiChatManager() @@ -90,7 +91,7 @@ if ( aiChatManager.mode !== AIMode.SCRIPT || !aiChatManager.scriptEditorApplyCode || - code === aiChatManager.scriptEditorOptions?.code + code === aiChatManager.scriptEditorOptions?.getCode() ) { return false } @@ -108,14 +109,18 @@
- + {#if language === 'mermaid'} + + {:else} + + {/if}
diff --git a/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte new file mode 100644 index 0000000000..06bcb63d98 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/script/MermaidDisplay.svelte @@ -0,0 +1,62 @@ + + +{#if showSvg} +
+ + {@html svg} +
+{:else} + +
{code}
+{/if} diff --git a/frontend/src/lib/components/copilot/chat/script/core.ts b/frontend/src/lib/components/copilot/chat/script/core.ts index 1e30bf50d9..a250cf2121 100644 --- a/frontend/src/lib/components/copilot/chat/script/core.ts +++ b/frontend/src/lib/components/copilot/chat/script/core.ts @@ -23,7 +23,7 @@ import { } from '../shared' import { createWorkspaceMutationTools } from '../workspaceTools' import { setupTypeAcquisition, type DepsToGet } from '$lib/ata' -import { getModelContextWindow } from '../../lib' +import { getModelContextWindow } from '../../modelConfig' import type { ReviewChangesOpts } from '../monaco-adapter' import { getCurrentModel } from '$lib/aiStore' import { getDbSchemas } from '$lib/components/apps/components/display/dbtable/metadata' @@ -97,7 +97,8 @@ export const SUPPORTED_CHAT_SCRIPT_LANGUAGES = [ 'powershell', 'csharp', 'java', - 'duckdb' + 'duckdb', + 'ansible' ] export function getLangContext( @@ -308,7 +309,11 @@ INSTRUCTIONS: export function prepareScriptSystemMessage( currentModel: AIProviderModel, language: ScriptLang | 'bunnative', - options: { isPreprocessor?: boolean; allowResourcesFetch?: boolean; workflowAsCode?: boolean } = {}, + options: { + isPreprocessor?: boolean + allowResourcesFetch?: boolean + workflowAsCode?: boolean + } = {}, customPrompt?: string ): ChatCompletionSystemMessageParam { let content = buildChatSystemPrompt(currentModel) diff --git a/frontend/src/lib/components/copilot/chat/shared.test.ts b/frontend/src/lib/components/copilot/chat/shared.test.ts index 56b6bc4d2b..9e53e3c40c 100644 --- a/frontend/src/lib/components/copilot/chat/shared.test.ts +++ b/frontend/src/lib/components/copilot/chat/shared.test.ts @@ -136,14 +136,22 @@ describe('buildContextString', () => { path: 'f/flows/reporting', title: 'f/flows/reporting', summary: 'Reporting flow' + }, + { + type: 'workspace_app', + path: 'f/apps/dashboard', + title: 'f/apps/dashboard', + summary: 'Dashboard raw app' } ]) expect(context).toContain('SELECTED WORKSPACE ITEMS:') expect(context).toContain('- type: script, path: f/scripts/report') expect(context).toContain('- type: flow, path: f/flows/reporting') + expect(context).toContain('- type: raw_app, path: f/apps/dashboard') expect(context).not.toContain('Report script') expect(context).not.toContain('Reporting flow') + expect(context).not.toContain('Dashboard raw app') expect(context).not.toContain('Code:') expect(context).not.toContain('Value:') }) diff --git a/frontend/src/lib/components/copilot/chat/shared.ts b/frontend/src/lib/components/copilot/chat/shared.ts index ccd63e5b90..b958283972 100644 --- a/frontend/src/lib/components/copilot/chat/shared.ts +++ b/frontend/src/lib/components/copilot/chat/shared.ts @@ -422,6 +422,11 @@ export function buildContextString(selectedContext: ContextElement[]): string { workspaceItemsContext = 'SELECTED WORKSPACE ITEMS:\n' } workspaceItemsContext += `- type: flow, path: ${context.path}\n` + } else if (context.type === 'workspace_app') { + if (!workspaceItemsContext) { + workspaceItemsContext = 'SELECTED WORKSPACE ITEMS:\n' + } + workspaceItemsContext += `- type: raw_app, path: ${context.path}\n` } } @@ -516,9 +521,31 @@ export type AssistantDisplayMessage = BaseDisplayMessage & { role: 'assistant' /** Summarized reasoning/thinking text streamed before the answer (Anthropic + compat providers). */ reasoning?: string + /** + * True only on the synthetic live message appended while tokens stream + * (see AIChat.svelte). Finalized messages never set it — without the flag, + * a reasoning-only message (thinking that led straight to a tool call) + * would look like it is still streaming forever. + */ + streaming?: boolean } -export type DisplayMessage = UserDisplayMessage | ToolDisplayMessage | AssistantDisplayMessage +/** + * Compaction boundary: replaces the summarized prefix in BOTH displayMessages + * and the API messages (where it is a plain user message). It carries no index + * because it is never a restart target — only the surviving tail's user + * messages are rewound to. + */ +export type SummaryDisplayMessage = { + role: 'summary' + content: string +} + +export type DisplayMessage = + | UserDisplayMessage + | ToolDisplayMessage + | AssistantDisplayMessage + | SummaryDisplayMessage // A tool message whose askUserQuestion is still awaiting an answer: the AI loop // is paused on the user. Drives the question card's interactivity, the @@ -914,7 +941,9 @@ export async function buildSchemaForTool( throw new Error(`Invalid flow inputs schema: ${invalidProperties.join(', ')}`) } - toolDef.function.parameters = { ...schema, additionalProperties: false } + // Anthropic requires input_schema.type to be present; flows with no inputs + // can produce a sparse schema (e.g. { order: [] }) lacking it. + toolDef.function.parameters = { type: 'object', ...schema, additionalProperties: false } // recursively normalize provider-incompatible schema fragments normalizeToolParameterSchema(toolDef.function.parameters) diff --git a/frontend/src/lib/components/copilot/chat/tokenUsage.ts b/frontend/src/lib/components/copilot/chat/tokenUsage.ts index d1b7a2e56b..51290f9269 100644 --- a/frontend/src/lib/components/copilot/chat/tokenUsage.ts +++ b/frontend/src/lib/components/copilot/chat/tokenUsage.ts @@ -4,6 +4,29 @@ export interface ChatTokenUsage { total: number } +/** + * Context usage persisted by earlier versions, which anchored the provider + * report to a message index and re-based it on system-prompt/tool changes. + * Usage is now a plain token count; old chats loaded from IndexedDB are + * collapsed to it via `normalizeContextUsage`. + */ +export interface LegacyContextTokenSnapshot { + tokens: number + atMessageIndex: number + overheadEstimate?: number +} + +export type PersistedContextUsage = number | LegacyContextTokenSnapshot + +export function normalizeContextUsage( + value: PersistedContextUsage | undefined +): number | undefined { + if (value === undefined) { + return undefined + } + return typeof value === 'number' ? value : value.tokens +} + export function emptyChatTokenUsage(): ChatTokenUsage { return { prompt: 0, completion: 0, total: 0 } } @@ -23,12 +46,17 @@ export function addChatTokenUsage( } } -export function anthropicUsageToChatTokenUsage(usage: { - input_tokens?: number | null - output_tokens?: number | null - cache_creation_input_tokens?: number | null - cache_read_input_tokens?: number | null -} | null | undefined): ChatTokenUsage { +export function anthropicUsageToChatTokenUsage( + usage: + | { + input_tokens?: number | null + output_tokens?: number | null + cache_creation_input_tokens?: number | null + cache_read_input_tokens?: number | null + } + | null + | undefined +): ChatTokenUsage { const prompt = (usage?.input_tokens ?? 0) + (usage?.cache_creation_input_tokens ?? 0) + @@ -42,11 +70,16 @@ export function anthropicUsageToChatTokenUsage(usage: { } } -export function openAIResponsesUsageToChatTokenUsage(usage: { - input_tokens?: number | null - output_tokens?: number | null - total_tokens?: number | null -} | null | undefined): ChatTokenUsage { +export function openAIResponsesUsageToChatTokenUsage( + usage: + | { + input_tokens?: number | null + output_tokens?: number | null + total_tokens?: number | null + } + | null + | undefined +): ChatTokenUsage { const prompt = usage?.input_tokens ?? 0 const completion = usage?.output_tokens ?? 0 @@ -57,11 +90,16 @@ export function openAIResponsesUsageToChatTokenUsage(usage: { } } -export function openAICompletionsUsageToChatTokenUsage(usage: { - prompt_tokens?: number | null - completion_tokens?: number | null - total_tokens?: number | null -} | null | undefined): ChatTokenUsage { +export function openAICompletionsUsageToChatTokenUsage( + usage: + | { + prompt_tokens?: number | null + completion_tokens?: number | null + total_tokens?: number | null + } + | null + | undefined +): ChatTokenUsage { const prompt = usage?.prompt_tokens ?? 0 const completion = usage?.completion_tokens ?? 0 diff --git a/frontend/src/lib/components/copilot/lib.test.ts b/frontend/src/lib/components/copilot/lib.test.ts index 15b09151b6..ccc8b85a65 100644 --- a/frontend/src/lib/components/copilot/lib.test.ts +++ b/frontend/src/lib/components/copilot/lib.test.ts @@ -7,10 +7,15 @@ import { describe, expect, it } from 'vitest' import { buildAssistantTextMessage, buildAssistantToolCallMessage, - getReasoningContentDelta + getReasoningContentDelta, + splitContentDelta } from './chat/openaiReasoning' import { parseFimCompletionChoice } from './fim' -import { getDefaultChatTemperature, modelDisallowsSamplingParams } from './modelConfig' +import { + getKnownModelContextWindow, + getModelContextWindow, + requiresMaxCompletionTokens +} from './modelConfig' import { supportsAutocomplete } from './utils' type AssistantMessageWithReasoning = ChatCompletionMessageParam & { @@ -21,91 +26,32 @@ type AssistantMessageWithReasoning = ChatCompletionMessageParam & { } describe('modelConfig', () => { - it('flags Fable 5 model IDs via includes matching', () => { - expect(modelDisallowsSamplingParams('claude-fable-5')).toBe(true) - expect(modelDisallowsSamplingParams('claude-fable-5@20260611')).toBe(true) - expect(modelDisallowsSamplingParams('claude-fable-5/thinking')).toBe(true) - expect(modelDisallowsSamplingParams('anthropic/claude-fable-5')).toBe(true) - }) - - it('flags Opus 4.7 model IDs via includes matching', () => { - expect(modelDisallowsSamplingParams('claude-opus-4-7')).toBe(true) - expect(modelDisallowsSamplingParams('claude-opus-4-7@20260416')).toBe(true) - expect(modelDisallowsSamplingParams('claude-opus-4-7/thinking')).toBe(true) - expect(modelDisallowsSamplingParams('anthropic/claude-opus-4-7')).toBe(true) - }) - - it('flags Opus 4.8 model IDs via includes matching', () => { - expect(modelDisallowsSamplingParams('claude-opus-4-8')).toBe(true) - expect(modelDisallowsSamplingParams('claude-opus-4-8@20260416')).toBe(true) - expect(modelDisallowsSamplingParams('claude-opus-4-8/thinking')).toBe(true) - expect(modelDisallowsSamplingParams('anthropic/claude-opus-4-8')).toBe(true) - }) - - it('omits deterministic temperature for Anthropic Opus 4.7 chat requests', () => { - expect( - getDefaultChatTemperature({ provider: 'anthropic', model: 'claude-opus-4-7' }) - ).toBeUndefined() - }) - - it('omits deterministic temperature for Anthropic Fable 5 chat requests', () => { - expect( - getDefaultChatTemperature({ provider: 'anthropic', model: 'claude-fable-5' }) - ).toBeUndefined() - }) - - it('omits deterministic temperature for non-anthropic providers carrying Opus 4.7 models', () => { - expect( - getDefaultChatTemperature({ provider: 'openrouter', model: 'anthropic/claude-opus-4-7' }) - ).toBeUndefined() - }) - - it('keeps deterministic temperature for older Anthropic models', () => { - expect(getDefaultChatTemperature({ provider: 'anthropic', model: 'claude-sonnet-4-6' })).toBe(0) - }) - it('flags gpt-5+ and o-series reasoning models via prefix matching', () => { - expect(modelDisallowsSamplingParams('gpt-5')).toBe(true) - expect(modelDisallowsSamplingParams('gpt-5.5')).toBe(true) - expect(modelDisallowsSamplingParams('gpt-5-mini')).toBe(true) - expect(modelDisallowsSamplingParams('o1')).toBe(true) - expect(modelDisallowsSamplingParams('o3')).toBe(true) - expect(modelDisallowsSamplingParams('o4-mini')).toBe(true) + expect(requiresMaxCompletionTokens('gpt-5')).toBe(true) + expect(requiresMaxCompletionTokens('gpt-5.5')).toBe(true) + expect(requiresMaxCompletionTokens('gpt-5-mini')).toBe(true) + expect(requiresMaxCompletionTokens('o1')).toBe(true) + expect(requiresMaxCompletionTokens('o3')).toBe(true) + expect(requiresMaxCompletionTokens('o4-mini')).toBe(true) // provider-prefixed identifiers (e.g. OpenRouter) match on the bare model id - expect(modelDisallowsSamplingParams('openai/gpt-5')).toBe(true) - expect(modelDisallowsSamplingParams('openai/o3')).toBe(true) + expect(requiresMaxCompletionTokens('openai/gpt-5')).toBe(true) + expect(requiresMaxCompletionTokens('openai/o3')).toBe(true) }) - it('keeps sampling params for non-reasoning models that merely share a prefix', () => { + it('does not require max_completion_tokens for non-reasoning models that merely share a prefix', () => { // gpt-4o starts with "gpt-" but not "gpt-5"; the "o" is mid-string, not a prefix - expect(modelDisallowsSamplingParams('gpt-4o')).toBe(false) - expect(modelDisallowsSamplingParams('gpt-4o-mini')).toBe(false) + expect(requiresMaxCompletionTokens('gpt-4o')).toBe(false) + expect(requiresMaxCompletionTokens('gpt-4o-mini')).toBe(false) // the provider prefix "openai/" must not be mistaken for an o-series model - expect(modelDisallowsSamplingParams('openai/gpt-4o')).toBe(false) + expect(requiresMaxCompletionTokens('openai/gpt-4o')).toBe(false) // the o-series match requires a digit after "o", so non-OpenAI ids that // start with "o" (Mistral open-* family, OpenRouter optimus-*/openchat-*) - // keep their deterministic temperature - expect(modelDisallowsSamplingParams('open-mistral-7b')).toBe(false) - expect(modelDisallowsSamplingParams('open-mixtral-8x7b')).toBe(false) - expect(modelDisallowsSamplingParams('open-mistral-nemo-2407')).toBe(false) - expect(modelDisallowsSamplingParams('optimus-alpha')).toBe(false) - expect(modelDisallowsSamplingParams('openchat/openchat-7b')).toBe(false) - }) - - it('keeps deterministic temperature for Mistral open-* models', () => { - expect(getDefaultChatTemperature({ provider: 'mistral', model: 'open-mixtral-8x7b' })).toBe(0) - }) - - it('omits deterministic temperature for gpt-5.5 routed through the customai gateway', () => { - expect(getDefaultChatTemperature({ provider: 'customai', model: 'gpt-5.5' })).toBeUndefined() - }) - - it('omits deterministic temperature for o-series models on the customai gateway', () => { - expect(getDefaultChatTemperature({ provider: 'customai', model: 'o3' })).toBeUndefined() - }) - - it('keeps deterministic temperature for gpt-4o on the customai gateway', () => { - expect(getDefaultChatTemperature({ provider: 'customai', model: 'gpt-4o' })).toBe(0) + // do not require max_completion_tokens + expect(requiresMaxCompletionTokens('open-mistral-7b')).toBe(false) + expect(requiresMaxCompletionTokens('open-mixtral-8x7b')).toBe(false) + expect(requiresMaxCompletionTokens('open-mistral-nemo-2407')).toBe(false) + expect(requiresMaxCompletionTokens('optimus-alpha')).toBe(false) + expect(requiresMaxCompletionTokens('openchat/openchat-7b')).toBe(false) }) }) @@ -222,4 +168,90 @@ describe('openaiReasoning', () => { tool_calls: [] }) }) + + it('omits reasoning_content for Mistral, which rejects it on input messages', () => { + const assistantMessage = buildAssistantToolCallMessage({ + content: '', + reasoning: { + hasReasoningContent: true, + reasoningContent: 'thinking trace' + }, + toolCalls: [], + provider: 'mistral' + }) as AssistantMessageWithReasoning + + expect(assistantMessage.reasoning_content).toBeUndefined() + // The content key is still present so the message stays well-formed. + expect(assistantMessage).toMatchObject({ role: 'assistant', content: '', tool_calls: [] }) + }) +}) + +describe('splitContentDelta', () => { + it('passes plain string deltas through as answer text', () => { + expect(splitContentDelta('hello')).toEqual({ reasoning: '', text: 'hello' }) + expect(splitContentDelta(null)).toEqual({ reasoning: '', text: '' }) + expect(splitContentDelta(undefined)).toEqual({ reasoning: '', text: '' }) + }) + + it('routes Mistral thinking parts to reasoning and text parts to the answer', () => { + expect( + splitContentDelta([ + { type: 'thinking', thinking: [{ type: 'text', text: 'step 1. ' }], closed: true }, + { type: 'thinking', thinking: [{ type: 'text', text: 'step 2.' }] }, + { type: 'text', text: '42' } + ]) + ).toEqual({ reasoning: 'step 1. step 2.', text: '42' }) + }) + + it('ignores malformed parts', () => { + expect(splitContentDelta([{ type: 'thinking' }, { type: 'text' }, 'junk', null])).toEqual({ + reasoning: '', + text: '' + }) + }) +}) + +describe('model context windows', () => { + it('maps Sonnet/Opus 4.6+ Claude models to the 1M window', () => { + expect(getKnownModelContextWindow('claude-sonnet-4-6')).toBe(1000000) + expect(getKnownModelContextWindow('claude-opus-4-6')).toBe(1000000) + expect(getKnownModelContextWindow('claude-opus-4-8')).toBe(1000000) + expect(getKnownModelContextWindow('anthropic.claude-sonnet-4-6-v1:0')).toBe(1000000) + }) + + it('keeps Haiku and older Claude models at 200K', () => { + expect(getKnownModelContextWindow('claude-haiku-4-5')).toBe(200000) + expect(getKnownModelContextWindow('global.anthropic.claude-haiku-4-5-20251001-v1:0')).toBe( + 200000 + ) + expect(getKnownModelContextWindow('claude-3-5-sonnet-latest')).toBe(200000) + expect(getKnownModelContextWindow('claude-sonnet-4-5-20250929')).toBe(200000) + expect(getKnownModelContextWindow('claude-opus-4-1')).toBe(200000) + // date-suffixed base ids without a minor version: the date must not be + // captured as the version + expect(getKnownModelContextWindow('claude-sonnet-4-20250514')).toBe(200000) + expect(getKnownModelContextWindow('anthropic.claude-sonnet-4-20250514-v1:0')).toBe(200000) + }) + + it('keeps base GPT-5 models at 400K while GPT-5.4+ get the 1M window', () => { + expect(getKnownModelContextWindow('gpt-5')).toBe(400000) + expect(getKnownModelContextWindow('gpt-5-mini')).toBe(400000) + expect(getKnownModelContextWindow('gpt-5.2')).toBe(400000) + expect(getKnownModelContextWindow('gpt-5.4')).toBe(1000000) + expect(getKnownModelContextWindow('gpt-5.5')).toBe(1000000) + }) + + it('maps recent Gemini and DeepSeek models to the 1M window', () => { + expect(getKnownModelContextWindow('gemini-3.1-pro')).toBe(1000000) + expect(getKnownModelContextWindow('gemini-3-flash')).toBe(1000000) + expect(getKnownModelContextWindow('gemini-2.5-flash')).toBe(1000000) + expect(getKnownModelContextWindow('deepseek-v4-pro')).toBe(1000000) + expect(getKnownModelContextWindow('deepseek-chat')).toBe(1000000) + expect(getKnownModelContextWindow('deepseek-reasoner')).toBe(1000000) + }) + + it('returns undefined for unrecognized models, 128K via the defaulting wrapper', () => { + expect(getKnownModelContextWindow('some-custom-model')).toBeUndefined() + expect(getModelContextWindow('some-custom-model')).toBe(128000) + }) }) diff --git a/frontend/src/lib/components/copilot/lib.ts b/frontend/src/lib/components/copilot/lib.ts index 2977da328a..ffa4e5db24 100644 --- a/frontend/src/lib/components/copilot/lib.ts +++ b/frontend/src/lib/components/copilot/lib.ts @@ -14,7 +14,7 @@ import Anthropic from '@anthropic-ai/sdk' import { get, type Writable } from 'svelte/store' import { OpenAPI, ResourceService, type Script } from '../../gen' import { EDIT_CONFIG, FIX_CONFIG, GEN_CONFIG } from './prompts' -import { getDefaultChatTemperature, modelDisallowsSamplingParams } from './modelConfig' +import { requiresMaxCompletionTokens } from './modelConfig' import { applyReasoningToConfig } from './reasoningRegistry' import { formatResourceTypes } from './utils' import { processToolCall, type Tool, type ToolCallbacks } from './chat/shared' @@ -34,6 +34,7 @@ import { import { buildAssistantTextMessage, buildAssistantToolCallMessage, + splitContentDelta, getReasoningContentDelta } from './chat/openaiReasoning' import { parseFimCompletionChoice } from './fim' @@ -286,21 +287,6 @@ export function getModelMaxTokens(provider: AIProvider, model: string) { return 8192 } -export function getModelContextWindow(model: string) { - if (model.includes('gpt-4.1') || model.includes('gemini')) { - return 1000000 - } else if (model.includes('gpt-5')) { - return 400000 - } else if (model.includes('gpt-4o') || model.includes('llama-3.3')) { - return 128000 - } else if (model.includes('claude') || model.includes('o4-mini') || model.includes('o3')) { - return 200000 - } else if (model.includes('codestral')) { - return 32000 - } else { - return 128000 - } -} function getModelSpecificConfig( modelProvider: AIProviderModel, @@ -315,10 +301,9 @@ function getModelSpecificConfig( // copilotInfo store may not be initialized in vitest } const maxTokens = customMaxTokensStore?.[modelKey] ?? defaultMaxTokens - const defaultTemperature = getDefaultChatTemperature(modelProvider) if ( (modelProvider.provider === 'openai' || modelProvider.provider === 'azure_openai') && - modelDisallowsSamplingParams(modelProvider.model) + requiresMaxCompletionTokens(modelProvider.model) ) { return { model: modelProvider.model, @@ -328,7 +313,6 @@ function getModelSpecificConfig( } else { return { model: modelProvider.model, - ...(defaultTemperature !== undefined ? { temperature: defaultTemperature } : {}), ...(tools && tools.length > 0 ? { tools } : {}), max_tokens: maxTokens } @@ -932,7 +916,7 @@ export async function getCompletion( } } : config, - 'completions', + provider === 'deepseek' ? 'deepseek' : provider === 'mistral' ? 'mistral' : 'completions', options?.reasoningEffort ) const completion = client.chat.completions.create(completionConfig, { @@ -965,7 +949,7 @@ export async function parseOpenAICompletion( tools: Tool[], helpers: any, _abortController?: AbortController, // unused, for signature compatibility with parseAnthropicCompletion - options?: { workspace?: string } + options?: { workspace?: string; provider?: string } ): Promise<{ shouldContinue: boolean; tokenUsage: ChatTokenUsage }> { const finalToolCalls: Record = {} let malformedFunctionCallError = false @@ -996,14 +980,19 @@ export async function parseOpenAICompletion( malformedFunctionCallError = true } + // Mistral nests reasoning inside structured content parts; split them out + // so a content delta never leaks "[object Object]" into the answer. + const structured = splitContentDelta(delta.content) const reasoningDelta = getReasoningContentDelta(delta) - if (typeof reasoningDelta === 'string') { + const reasoningText = + (typeof reasoningDelta === 'string' ? reasoningDelta : '') + structured.reasoning + if (typeof reasoningDelta === 'string' || structured.reasoning) { hasReasoningContent = true - reasoningContent += reasoningDelta - callbacks.onReasoningDelta?.(reasoningDelta) + reasoningContent += reasoningText + callbacks.onReasoningDelta?.(reasoningText) } - const contentDelta = delta.content + const contentDelta = structured.text if (contentDelta) { answer += contentDelta assistantContent += contentDelta @@ -1118,7 +1107,8 @@ export async function parseOpenAICompletion( hasReasoningContent, reasoningContent }, - toolCalls: normalizedToolCalls + toolCalls: normalizedToolCalls, + provider: options?.provider }) messages.push(toAdd) addedMessages.push(toAdd) diff --git a/frontend/src/lib/components/copilot/modelConfig.ts b/frontend/src/lib/components/copilot/modelConfig.ts index 7b228c6ba9..4ab08ce02c 100644 --- a/frontend/src/lib/components/copilot/modelConfig.ts +++ b/frontend/src/lib/components/copilot/modelConfig.ts @@ -1,28 +1,60 @@ -import type { AIProviderModel } from '$lib/gen' - -export function modelDisallowsSamplingParams(model: string) { +// gpt-5+ and o-series reasoning models reject the legacy `max_tokens` field on +// the OpenAI/Azure Chat Completions API and require `max_completion_tokens` +// instead. The check strips any provider prefix (e.g. OpenRouter's "openai/o3") +// so it matches the bare model id, and the o-series match requires a digit after +// the "o" (o1/o3/o4-mini) so it does not catch unrelated ids like Mistral's +// "open-mistral-*" or "optimus-*". +export function requiresMaxCompletionTokens(model: string) { const normalizedModel = model.toLowerCase() - // Strip any provider prefix (e.g. OpenRouter's "openai/o3") so the - // reasoning-model check matches the bare model id rather than the prefix. const baseModel = normalizedModel.split('/').pop() ?? normalizedModel - // gpt-5+ and o-series reasoning models reject sampling params such as - // temperature (only the default value is supported), regardless of which - // provider/gateway routes the request — so this must stay provider-agnostic. - // The o-series match requires a digit after the "o" (o1/o3/o4-mini) so it - // does not catch unrelated ids like Mistral's "open-mistral-*" or "optimus-*". - return ( - normalizedModel.includes('claude-fable-5') || - normalizedModel.includes('claude-opus-4-7') || - normalizedModel.includes('claude-opus-4-8') || - baseModel.startsWith('gpt-5') || - /^o\d/.test(baseModel) - ) + return baseModel.startsWith('gpt-5') || /^o\d/.test(baseModel) } -export function getDefaultChatTemperature(modelProvider: AIProviderModel): number | undefined { - if (modelDisallowsSamplingParams(modelProvider.model)) { - return undefined - } +// Context windows of the models we know, most specific entry first — the first +// name included in the model id wins, so provider-prefixed and date-suffixed +// ids (anthropic.claude-sonnet-4-6-...-v1:0, gpt-5.2-2026-01-01) still resolve. +// Conservative family fallbacks sit below the explicit entries; models not +// listed at all resolve to undefined, which disables auto-trimming and the +// indicator denominator. +const MODEL_CONTEXT_WINDOWS: [name: string, contextWindow: number][] = [ + // Anthropic — Sonnet/Opus 4.6+ ship a 1M window at standard pricing (GA); + // Haiku, older Claude models (3.x, 4.0, 4.1, 4.5) and date-suffixed Claude 4 + // base ids (claude-sonnet-4-20250514) fall through to 200K + ['claude-fable-5', 1_000_000], + ['claude-opus-4-8', 1_000_000], + ['claude-opus-4-7', 1_000_000], + ['claude-opus-4-6', 1_000_000], + ['claude-sonnet-4-6', 1_000_000], + ['claude', 200_000], + // OpenAI — gpt-5 covers the base family (-mini / -nano) and the 5.1/5.2 + // revisions, all 400K; only 5.4+ moved to 1M + ['gpt-5.5', 1_000_000], + ['gpt-5.4', 1_000_000], + ['gpt-5', 400_000], + ['gpt-4.1', 1_000_000], + ['gpt-4o', 128_000], + ['o4-mini', 200_000], + ['o3', 200_000], + // Google — the 2.5 / 3 / 3.1 Gemini families are all 1M + ['gemini-3.1', 1_000_000], + ['gemini-3', 1_000_000], + ['gemini-2.5', 1_000_000], + // DeepSeek — the V4 family is 1M; deepseek-chat / deepseek-reasoner are + // aliases of V4-Flash since April 2026 + ['deepseek-v4', 1_000_000], + ['deepseek-chat', 1_000_000], + ['deepseek-reasoner', 1_000_000], + ['deepseek', 128_000], + // Others + ['llama', 128_000], + ['codestral', 32_000] +] - return 0 +export function getKnownModelContextWindow(model: string): number | undefined { + return MODEL_CONTEXT_WINDOWS.find(([name]) => model.includes(name))?.[1] +} + +export function getModelContextWindow(model: string) { + // Trim/compaction logic needs a number; assume a conservative window when unknown. + return getKnownModelContextWindow(model) ?? 128000 } diff --git a/frontend/src/lib/components/copilot/reasoningRegistry.test.ts b/frontend/src/lib/components/copilot/reasoningRegistry.test.ts index 662a6430f2..42f0006a1d 100644 --- a/frontend/src/lib/components/copilot/reasoningRegistry.test.ts +++ b/frontend/src/lib/components/copilot/reasoningRegistry.test.ts @@ -4,6 +4,7 @@ import { getReasoningCapability, REASONING_OFF, resolveEffectiveReasoning, + resolveRequestReasoning, stripLegacyThinkingSuffix, supportsReasoning } from './reasoningRegistry' @@ -27,6 +28,11 @@ describe('supportsReasoning (static registry)', () => { }) it('excludes non-reasoning Anthropic models', () => { expect(supportsReasoning('anthropic', 'claude-3-5-haiku-latest')).toBe(false) + // Opus 4.5 predates adaptive thinking — offering effort would hard-400. + expect(supportsReasoning('anthropic', 'claude-opus-4-5')).toBe(false) + expect(supportsReasoning('aws_bedrock', 'eu.anthropic.claude-opus-4-5-20251101-v1:0')).toBe( + false + ) }) it('exposes the model-specific Anthropic effort ladder', () => { // Sonnet 4.6 / Opus 4.6: max but no xhigh @@ -45,17 +51,196 @@ describe('supportsReasoning (static registry)', () => { 'max' ]) }) + it('flags Claude models served through Bedrock, with the Anthropic ladder', () => { + expect(supportsReasoning('aws_bedrock', 'us.anthropic.claude-opus-4-6-v1')).toBe(true) + expect(supportsReasoning('aws_bedrock', 'anthropic.claude-sonnet-4-6-v1:0')).toBe(true) + expect(supportsReasoning('aws_bedrock', 'amazon.nova-pro-v1:0')).toBe(false) + expect(getReasoningCapability('aws_bedrock', 'us.anthropic.claude-opus-4-8-v1').levels).toEqual( + ['low', 'medium', 'high', 'xhigh', 'max'] + ) + }) + it('exposes the model-specific Gemini ladder', () => { + // Gemini 3+ Flash / Flash-Lite accept minimal; Pro does not. + expect(getReasoningCapability('googleai', 'gemini-3-flash-preview').levels).toEqual([ + 'minimal', + 'low', + 'medium', + 'high' + ]) + expect(getReasoningCapability('googleai', 'gemini-3.1-flash-lite').levels).toEqual([ + 'minimal', + 'low', + 'medium', + 'high' + ]) + expect(getReasoningCapability('googleai', 'gemini-3.1-pro-preview').levels).toEqual([ + 'low', + 'medium', + 'high' + ]) + // 2.5 uses budget tiers — no minimal, even on flash. + expect(getReasoningCapability('googleai', 'gemini-2.5-flash').levels).toEqual([ + 'low', + 'medium', + 'high' + ]) + }) + it('flags DeepSeek models with the two effective levels, excluding the chat alias', () => { + expect(supportsReasoning('deepseek', 'deepseek-v4-flash')).toBe(true) + expect(supportsReasoning('deepseek', 'deepseek-v4-pro')).toBe(true) + expect(supportsReasoning('deepseek', 'deepseek-reasoner')).toBe(true) + // `deepseek-chat` is the documented non-thinking mode — no knob. + expect(supportsReasoning('deepseek', 'deepseek-chat')).toBe(false) + // Only high/max are real; low/medium/xhigh are server-side aliases. + expect(getReasoningCapability('deepseek', 'deepseek-v4-flash').levels).toEqual(['high', 'max']) + }) it('flags OpenAI reasoning families, not gpt-4o', () => { expect(supportsReasoning('openai', 'gpt-5')).toBe(true) expect(supportsReasoning('openai', 'o3')).toBe(true) expect(supportsReasoning('openai', 'gpt-4o')).toBe(false) }) + it('exposes the model-specific OpenAI ladder', () => { + // gpt-5 has minimal; gpt-5.1+ dropped it; only gpt-5.5 adds xhigh. + expect(getReasoningCapability('openai', 'gpt-5').levels).toEqual([ + 'minimal', + 'low', + 'medium', + 'high' + ]) + expect(getReasoningCapability('openai', 'gpt-5.1').levels).toEqual(['low', 'medium', 'high']) + expect(getReasoningCapability('openai', 'gpt-5.5').levels).toEqual([ + 'low', + 'medium', + 'high', + 'xhigh' + ]) + expect(getReasoningCapability('openai', 'o3').levels).toEqual(['low', 'medium', 'high']) + }) + it('flags Mistral models verified to accept reasoning_effort, with the on/off ladder', () => { + expect(supportsReasoning('mistral', 'mistral-medium-3-5')).toBe(true) + expect(supportsReasoning('mistral', 'mistral-medium-latest')).toBe(true) + expect(supportsReasoning('mistral', 'mistral-small-latest')).toBe(true) + // These reject the param ("reasoning_effort is not enabled for this model"). + expect(supportsReasoning('mistral', 'mistral-large-latest')).toBe(false) + expect(supportsReasoning('mistral', 'magistral-medium-latest')).toBe(false) + expect(supportsReasoning('mistral', 'ministral-8b-latest')).toBe(false) + // 'high' is the only accepted effort token; off = omit the field. + expect(getReasoningCapability('mistral', 'mistral-medium-3-5').levels).toEqual(['high']) + expect(getReasoningCapability('mistral', 'mistral-medium-3-5').canDisable).toBe(true) + }) it('returns no levels for providers without a registry entry', () => { expect(getReasoningCapability('mistral', 'codestral-latest')).toEqual({ supported: false, - levels: [] + levels: [], + canDisable: false }) }) + it('only offers off where the model can truly disable thinking', () => { + // Gemini Pro enforces a thinking floor — no off option. + expect(getReasoningCapability('googleai', 'gemini-2.5-pro').canDisable).toBe(false) + expect(getReasoningCapability('googleai', 'gemini-3.1-pro-preview').canDisable).toBe(false) + // Flash / Flash-Lite can fully disable (budget 0 / minimal). + expect(getReasoningCapability('googleai', 'gemini-2.5-flash').canDisable).toBe(true) + expect(getReasoningCapability('googleai', 'gemini-3-flash-preview').canDisable).toBe(true) + // Claude doesn't think unless asked; DeepSeek has a disable param. + expect(getReasoningCapability('anthropic', 'claude-sonnet-4-6').canDisable).toBe(true) + expect(getReasoningCapability('deepseek', 'deepseek-v4-flash').canDisable).toBe(true) + // Fable thinking is always on — explicit disable 400s, omission is a no-op. + expect(getReasoningCapability('anthropic', 'claude-fable-5').canDisable).toBe(false) + expect( + getReasoningCapability('aws_bedrock', 'eu.anthropic.claude-fable-5-v1:0').canDisable + ).toBe(false) + // ...but the effort ladder is fully available on Fable. + expect(getReasoningCapability('anthropic', 'claude-fable-5').levels).toEqual([ + 'low', + 'medium', + 'high', + 'xhigh', + 'max' + ]) + // gpt-5.1+ accept effort 'none'; gpt-5 and o-series reason at medium + // by default and reject 'none' — omission isn't off. + expect(getReasoningCapability('openai', 'gpt-5.1').canDisable).toBe(true) + expect(getReasoningCapability('openai', 'gpt-5.5').canDisable).toBe(true) + expect(getReasoningCapability('openai', 'gpt-5').canDisable).toBe(false) + expect(getReasoningCapability('openai', 'o3').canDisable).toBe(false) + // OpenRouter's 'none' effort disables reasoning, but only where the + // underlying model can actually stop thinking — same family scoping + // as the levels. + expect(getReasoningCapability('openrouter', 'anthropic/claude-sonnet-4.6').canDisable).toBe( + true + ) + expect(getReasoningCapability('openrouter', 'google/gemini-2.5-flash').canDisable).toBe(true) + expect(getReasoningCapability('openrouter', 'deepseek/deepseek-v4-flash').canDisable).toBe(true) + expect(getReasoningCapability('openrouter', 'openai/gpt-5.1').canDisable).toBe(true) + expect(getReasoningCapability('openrouter', 'google/gemini-2.5-pro').canDisable).toBe(false) + expect(getReasoningCapability('openrouter', 'google/gemini-3.1-pro-preview').canDisable).toBe( + false + ) + expect(getReasoningCapability('openrouter', 'openai/o3').canDisable).toBe(false) + expect(getReasoningCapability('openrouter', 'openai/gpt-5-mini').canDisable).toBe(false) + expect(getReasoningCapability('openrouter', 'x-ai/grok-4').canDisable).toBe(false) + expect(getReasoningCapability('openrouter', 'deepseek/deepseek-r1').canDisable).toBe(false) + }) + it('forwards an explicit off as effort none through OpenRouter', () => { + expect( + resolveRequestReasoning({ + provider: 'openrouter', + model: 'google/gemini-2.5-flash', + reasoning: REASONING_OFF + }) + ).toBe('none') + }) + it('flags deepseek v4 models served through OpenRouter', () => { + expect(supportsReasoning('openrouter', 'deepseek/deepseek-v4-flash')).toBe(true) + expect(supportsReasoning('openrouter', 'deepseek/deepseek-r1')).toBe(true) + }) + it('scopes the OpenRouter ladder to the underlying model family', () => { + // Anthropic: all five levels are distinct budget ratios. + expect(getReasoningCapability('openrouter', 'anthropic/claude-sonnet-4.6').levels).toEqual([ + 'minimal', + 'low', + 'medium', + 'high', + 'xhigh' + ]) + // Gemini: thinkingLevel mapping; xhigh is clamped to high, so not offered. + expect(getReasoningCapability('openrouter', 'google/gemini-3-flash-preview').levels).toEqual([ + 'minimal', + 'low', + 'medium', + 'high' + ]) + expect(getReasoningCapability('openrouter', 'google/gemini-2.5-flash').levels).toEqual([ + 'low', + 'medium', + 'high' + ]) + // OpenAI: passed through verbatim — same per-model scoping as direct. + expect(getReasoningCapability('openrouter', 'openai/gpt-5-mini').levels).toEqual([ + 'minimal', + 'low', + 'medium', + 'high' + ]) + expect(getReasoningCapability('openrouter', 'openai/gpt-5.5').levels).toEqual([ + 'low', + 'medium', + 'high', + 'xhigh' + ]) + // DeepSeek: low/medium alias high and xhigh aliases max server-side. + expect(getReasoningCapability('openrouter', 'deepseek/deepseek-v4-flash').levels).toEqual([ + 'high', + 'xhigh' + ]) + // Unknown families keep the conservative common denominator. + expect(getReasoningCapability('openrouter', 'x-ai/grok-4').levels).toEqual([ + 'low', + 'medium', + 'high' + ]) + }) }) describe('resolveEffectiveReasoning', () => { @@ -89,6 +274,59 @@ describe('resolveEffectiveReasoning', () => { }) }) +describe('resolveRequestReasoning', () => { + it('matches resolveEffectiveReasoning for levels and defaults', () => { + expect(resolveRequestReasoning({ provider: 'googleai', model: 'gemini-2.5-pro' })).toBe('high') + expect(resolveRequestReasoning({ provider: 'openai', model: 'o3', reasoning: 'low' })).toBe( + 'low' + ) + }) + it('forwards an explicit off as the provider disable token on reasoning-by-default providers', () => { + expect( + resolveRequestReasoning({ + provider: 'googleai', + model: 'gemini-2.5-pro', + reasoning: REASONING_OFF + }) + ).toBe('none') + }) + it('forwards an explicit off for DeepSeek (thinks by default)', () => { + expect( + resolveRequestReasoning({ + provider: 'deepseek', + model: 'deepseek-v4-flash', + reasoning: REASONING_OFF + }) + ).toBe('none') + }) + it('forwards off as effort none on gpt-5.1+, but not on older OpenAI reasoning models', () => { + expect( + resolveRequestReasoning({ provider: 'openai', model: 'gpt-5.1', reasoning: REASONING_OFF }) + ).toBe('none') + expect( + resolveRequestReasoning({ provider: 'openai', model: 'o3', reasoning: REASONING_OFF }) + ).toBeUndefined() + }) + it('keeps off as undefined for providers without default-on reasoning', () => { + expect( + resolveRequestReasoning({ + provider: 'anthropic', + model: 'claude-sonnet-4-6', + reasoning: REASONING_OFF + }) + ).toBeUndefined() + }) + it('never sends a disable token for non-capable models', () => { + expect( + resolveRequestReasoning({ + provider: 'googleai', + model: 'gemini-2.0-flash', + reasoning: REASONING_OFF + }) + ).toBeUndefined() + }) +}) + describe('applyReasoningToConfig', () => { it('is a no-op when no effort is provided', () => { const config = { model: 'm', max_tokens: 10 } @@ -105,6 +343,30 @@ describe('applyReasoningToConfig', () => { reasoning: { effort: 'medium' } }) }) + it('adds reasoning_effort on the deepseek path for a level', () => { + expect(applyReasoningToConfig({ model: 'm' }, 'deepseek', 'max')).toMatchObject({ + reasoning_effort: 'max' + }) + }) + it('translates the deepseek off sentinel to the thinking-disabled param', () => { + const out = applyReasoningToConfig({ model: 'm' }, 'deepseek', 'none') as Record + expect(out.thinking).toEqual({ type: 'disabled' }) + expect(out.reasoning_effort).toBeUndefined() + }) + it('strips sampling params on the mistral path when reasoning is on', () => { + const out = applyReasoningToConfig( + { model: 'm', temperature: 0, max_tokens: 10 }, + 'mistral', + 'high' + ) as Record + expect(out.reasoning_effort).toBe('high') + expect(out.temperature).toBeUndefined() + expect(out.max_tokens).toBe(10) + // No effort -> config untouched, temperature kept. + expect( + applyReasoningToConfig({ model: 'm', temperature: 0 }, 'mistral', undefined) + ).toMatchObject({ temperature: 0 }) + }) it('adds adaptive thinking + output_config and strips sampling params for Anthropic', () => { const out = applyReasoningToConfig( { model: 'm', max_tokens: 10, temperature: 0 }, diff --git a/frontend/src/lib/components/copilot/reasoningRegistry.ts b/frontend/src/lib/components/copilot/reasoningRegistry.ts index 6d2af80cac..e1ce2467f2 100644 --- a/frontend/src/lib/components/copilot/reasoningRegistry.ts +++ b/frontend/src/lib/components/copilot/reasoningRegistry.ts @@ -40,14 +40,87 @@ function baseModelId(model: string): string { /** * Suggested effort levels per provider, sourced from each provider SDK's own - * vocabulary. These are the static fallback; dynamic enrichment (Anthropic, - * OpenRouter) can override with exact per-model levels. + * vocabulary. */ const PROVIDER_REASONING_LEVELS: Partial> = { - openai: ['minimal', 'low', 'medium', 'high'], - azure_openai: ['minimal', 'low', 'medium', 'high'], - openrouter: ['low', 'medium', 'high'], - googleai: ['low', 'medium', 'high'] + // DeepSeek accepts the full five-token vocabulary but only two levels are + // real: low/medium are server-mapped to high and xhigh to max — offering + // them would be a no-op knob. + deepseek: ['high', 'max'], + // Mistral's only effort token besides the 'none' disable is 'high' + // (anything else is rejected), so the knob is effectively on/off. + mistral: ['high'] +} + +/** + * OpenRouter validates effort against its own vocabulary + * (minimal..xhigh + none) and translates per underlying provider, so the + * real ladder depends on the model family: Anthropic gets all five as + * distinct budget ratios (minimal 10% .. xhigh 95% of max_tokens); Gemini + * maps to thinkingLevel with xhigh clamped to high (a no-op vs high); + * OpenAI gets the token passed through verbatim, so the per-model OpenAI + * scoping applies; DeepSeek server-maps low/medium to high and xhigh to max. + */ +function openrouterReasoningLevels(model: string): ReasoningEffort[] { + const m = model.toLowerCase() + const base = baseModelId(model) + if (/claude-(opus|sonnet)-4/.test(m)) { + return ['minimal', 'low', 'medium', 'high', 'xhigh'] + } + if (m.includes('gemini-')) { + return geminiReasoningLevels(m) + } + if (base.startsWith('gpt-5') || /^o\d/.test(base)) { + return openaiReasoningLevels(base) + } + if (m.includes('deepseek-v4')) { + return ['high', 'xhigh'] + } + return ['low', 'medium', 'high'] +} + +/** + * OpenAI's effort vocabulary is model-dependent: `minimal` exists on gpt-5 but + * not on gpt-5.1+, `xhigh` only on gpt-5.5; o-series take low/medium/high. + * An unsupported level is rejected, so scope the list to the model. + */ +function openaiReasoningLevels(model: string): ReasoningEffort[] { + const base = baseModelId(model) + if (/^gpt-5\.5/.test(base)) { + return ['low', 'medium', 'high', 'xhigh'] + } + if (/^gpt-5\./.test(base)) { + return ['low', 'medium', 'high'] + } + if (/^gpt-5/.test(base)) { + return ['minimal', 'low', 'medium', 'high'] + } + return ['low', 'medium', 'high'] +} + +/** + * Gemini's level ladder is model-dependent: Gemini 3+ Flash / Flash-Lite accept + * `minimal`, while 3.x Pro does not (and cannot disable thinking). Gemini 2.5 + * uses numeric budgets — the proxy maps the three tiers to budget values, so + * `minimal` is not offered there. + */ +function geminiReasoningLevels(model: string): ReasoningEffort[] { + const m = model.toLowerCase() + const isGemini3Plus = !m.includes('gemini-2.5') + if (isGemini3Plus && (m.includes('flash') || m.includes('lite'))) { + return ['minimal', 'low', 'medium', 'high'] + } + return ['low', 'medium', 'high'] +} + +/** + * Gemini Pro models cannot turn thinking off — the API enforces a floor + * (level `low` on 3.x Pro, a 128-token budget on 2.5 Pro), so an off option + * would silently mean "lowest". Flash / Flash-Lite can truly disable + * (budget 0 / level `minimal`). + */ +function geminiCanDisable(model: string): boolean { + return !model.toLowerCase().includes('pro') } /** @@ -65,31 +138,48 @@ function anthropicReasoningLevels(model: string): ReasoningEffort[] { /** * Conservative static predicate for whether a model accepts an effort knob. - * Kept tight to avoid 400s on models that reject reasoning params; dynamic - * enrichment widens it where the provider API exposes capability. + * Kept tight to avoid 400s on models that reject reasoning params. */ function supportsReasoningStatic(provider: AIProvider, model: string): boolean { const m = model.toLowerCase() const base = baseModelId(model) switch (provider) { case 'anthropic': - return /claude-opus-4-(5|6|7|8)/.test(m) || /claude-sonnet-4-6/.test(m) || m.includes('fable') + // Bedrock serves the same Claude models under prefixed ids + // (e.g. `us.anthropic.claude-opus-4-6-v1`), so match on the full string. + case 'aws_bedrock': + // 4.6+ only: Opus 4.5 rejects adaptive thinking (and, on Bedrock, + // the whole output_config surface) — live-verified hard 400. + return /claude-opus-4-(6|7|8)/.test(m) || /claude-sonnet-4-6/.test(m) || m.includes('fable') case 'openai': case 'azure_openai': return base.startsWith('gpt-5') || /^o\d/.test(base) case 'openrouter': - // best-effort markers; dynamic enrichment (supported_parameters) refines this + // Best-effort markers for models whose `supported_parameters` include + // `reasoning` in OpenRouter's catalog; OpenRouter translates the effort + // per underlying provider. return ( base.startsWith('gpt-5') || /^o\d/.test(base) || /claude-(opus|sonnet)-4/.test(m) || /gemini-(2\.5|3)/.test(m) || m.includes('deepseek-r') || + m.includes('deepseek-v4') || m.includes('grok-4') || m.includes(':thinking') ) case 'googleai': return /gemini-(2\.5|3)/.test(m) + case 'deepseek': + // All current API models take reasoning_effort (live-verified). The + // deprecated `deepseek-chat` alias is excluded: its documented meaning + // is "non-thinking mode", and sending an effort would silently flip it + // into thinking mode — picking that alias is itself an off choice. + return base.startsWith('deepseek') && base !== 'deepseek-chat' + case 'mistral': + // Only the ids verified to accept reasoning_effort; other models + // (large, magistral, ministral, pinned versions) reject the param. + return /^mistral-(small|medium)-latest$/.test(base) || base.startsWith('mistral-medium-3-5') default: return false } @@ -99,6 +189,12 @@ export type ReasoningCapability = { supported: boolean /** Suggested levels for the UI control. Empty when unsupported. */ levels: ReasoningEffort[] + /** + * Whether the model can truly turn reasoning off. When false the UI must + * not offer an off option — the provider would coerce it to the lowest + * level, making the switch a lie. + */ + canDisable: boolean } /** Resolve the reasoning capability of a model from the static registry. */ @@ -106,13 +202,64 @@ export function getReasoningCapability(provider: AIProvider, model: string): Rea const bareModel = stripLegacyThinkingSuffix(model) const supported = supportsReasoningStatic(provider, bareModel) if (!supported) { - return { supported: false, levels: [] } + return { supported: false, levels: [], canDisable: false } } const levels = - provider === 'anthropic' + provider === 'anthropic' || provider === 'aws_bedrock' ? anthropicReasoningLevels(bareModel) - : (PROVIDER_REASONING_LEVELS[provider] ?? ['low', 'medium', 'high']) - return { supported, levels } + : provider === 'googleai' + ? geminiReasoningLevels(bareModel) + : provider === 'openai' || provider === 'azure_openai' + ? openaiReasoningLevels(bareModel) + : provider === 'openrouter' + ? openrouterReasoningLevels(bareModel) + : (PROVIDER_REASONING_LEVELS[provider] ?? ['low', 'medium', 'high']) + return { supported, levels, canDisable: canDisableReasoning(provider, bareModel) } +} + +/** + * Whether selecting "off" truly disables reasoning for the model. Off is + * sent either as an explicit provider disable (see `explicitOffToken`) or by + * omitting the effort — which only works where the model doesn't reason by + * default. + */ +function canDisableReasoning(provider: AIProvider, model: string): boolean { + const m = model.toLowerCase() + const base = baseModelId(model) + switch (provider) { + case 'anthropic': + case 'aws_bedrock': + // Claude 4.6+ only think when asked, so omission is a real off — + // except Fable, where thinking is always on (explicit disable 400s). + return !m.includes('fable') + case 'googleai': + return geminiCanDisable(model) + case 'openai': + case 'azure_openai': + // gpt-5.1+ accept effort 'none'; gpt-5 and o-series reject it and + // reason at `medium` by default, so omission isn't off either. + return /^gpt-5\./.test(base) + case 'openrouter': + // 'none' is in OpenRouter's vocabulary, but the gateway can't + // disable a model whose upstream can't — scope off per underlying + // family, like the levels. + if (/claude-(opus|sonnet)-4/.test(m)) { + return true + } + if (m.includes('gemini-')) { + return geminiCanDisable(m) + } + if (base.startsWith('gpt-5') || /^o\d/.test(base)) { + return /^gpt-5\./.test(base) + } + if (m.includes('deepseek-v4')) { + return true + } + // grok-4, deepseek-r1 and :thinking variants reason unconditionally. + return false + default: + return true + } } export function supportsReasoning(provider: AIProvider, model: string): boolean { @@ -141,7 +288,68 @@ export function resolveEffectiveReasoning( : undefined } -export type ReasoningApiKind = 'anthropic' | 'responses' | 'completions' +/** + * Sentinel sent for the deepseek off case. It never reaches the wire as an + * effort: the 'deepseek' branch of `applyReasoningToConfig` translates it to + * the provider's `thinking: {type: "disabled"}` param (`reasoning_effort: + * "none"` is rejected by their API). + */ +export const DEEPSEEK_OFF_SENTINEL: ReasoningEffort = 'none' + +/** + * Disable token to forward when the user explicitly turns reasoning off on a + * model that reasons *by default* — omitting the field would silently keep + * the default-on behavior. Undefined means omission is the correct off. + */ +function explicitOffToken(provider: AIProvider, model: string): ReasoningEffort | undefined { + switch (provider) { + case 'googleai': + // Gemini 2.5/3 think by default (dynamic budget / level). The backend + // proxy maps 'none' to off on Flash, or the floor on Pro (only + // reachable via a stale persisted preference — see canDisableReasoning). + return 'none' + case 'deepseek': + return DEEPSEEK_OFF_SENTINEL + case 'openai': + case 'azure_openai': + // gpt-5.1+ reasoning is off only via the explicit 'none' effort + // (gpt-5.5 defaults to medium when the field is omitted). + return /^gpt-5\./.test(baseModelId(model)) ? 'none' : undefined + case 'openrouter': + // OpenRouter validates effort against xhigh..minimal|none and + // documents 'none' as disabling reasoning, translated per the + // underlying provider — more reliable than omission, which keeps + // reasoning-by-default models thinking. + return 'none' + default: + return undefined + } +} + +/** + * The effort to put on the wire for a given model selection. Same as + * `resolveEffectiveReasoning`, plus: an explicit user "off" on a + * reasoning-by-default provider resolves to that provider's disable token + * instead of undefined — omitting the field would silently keep the provider's + * default-on behavior, making the off switch a no-op. + */ +export function resolveRequestReasoning( + modelProvider: ReasoningProviderModel +): ReasoningEffort | undefined { + const effective = resolveEffectiveReasoning(modelProvider) + if (effective !== undefined) { + return effective + } + if ( + modelProvider.reasoning === REASONING_OFF && + supportsReasoning(modelProvider.provider, modelProvider.model) + ) { + return explicitOffToken(modelProvider.provider, stripLegacyThinkingSuffix(modelProvider.model)) + } + return undefined +} + +export type ReasoningApiKind = 'anthropic' | 'responses' | 'completions' | 'deepseek' | 'mistral' /** * Inject an effort level into a request config for the given completion path. @@ -185,5 +393,29 @@ export function applyReasoningToConfig>( ...config, reasoning_effort: effort } as unknown as T + case 'deepseek': + // Same completions dialect, except "off" is a separate `thinking` + // param — there is no effort token that disables thinking. + if (effort === DEEPSEEK_OFF_SENTINEL) { + return { + ...config, + thinking: { type: 'disabled' } + } as unknown as T + } + return { + ...config, + reasoning_effort: effort + } as unknown as T + case 'mistral': { + // Same completions dialect, but reasoning requests get stricter + // sampling validation ("top_p must be 1 when using greedy sampling" + // with our temperature 0) — strip sampling params like the + // Anthropic adaptive-thinking path does. + const { temperature: _t, top_p: _p, ...rest } = config as Record + return { + ...rest, + reasoning_effort: effort + } as unknown as T + } } } diff --git a/frontend/src/lib/components/custom_ui.ts b/frontend/src/lib/components/custom_ui.ts index 64c6221aea..a04c9ea4b5 100644 --- a/frontend/src/lib/components/custom_ui.ts +++ b/frontend/src/lib/components/custom_ui.ts @@ -59,10 +59,46 @@ export type PreviewPanelUi = { disableTracing?: boolean disableTriggerCaptures?: boolean disableTriggerButton?: boolean + disableJsonView?: boolean displayResult?: DisplayResultUi disableVariablePicker?: boolean disableDownload?: boolean tagLabel?: string + // Hide the args/SchemaForm pane entirely (use cases where the script + // is known to take no inputs — e.g. the pipeline editor's per-script + // preview). When set, the Test/Cancel button is rendered as a small + // floating affordance at the top-left of the preview area instead of + // inside the (now-absent) args column. + hideArgs?: boolean + // Render the LogPanel's logs/result as a left/right split instead of + // the default top/bottom. Pairs naturally with `hideArgs` when the + // preview area is the full bottom band. + logsResultSideBySide?: boolean + // Number of scripts that subscribe to assets this script writes (via + // `// on s3://…` etc.). When > 0, the Test button is rendered as a + // split: primary action runs just this step (cascade suppressed via + // `_wmill_skip_asset_dispatch`), and a caret exposes "Test + trigger N + // downstream" that lets the asset-dispatch hook fire downstream jobs. + // Undefined or 0 → plain Test button (no cascade UI). + downstreamSubscribers?: number + // Pairs with `hideArgs`: still hide the full args column, but when the + // script actually declares inputs render a compact SchemaForm between + // the floating Test button and the logs/result panel (e.g. a + // partitioned pipeline script that needs a `partition` arg to run). + // No-op when the script has no input properties. + argsAboveLogs?: boolean + // On mount, query the most recent top-level completed job for this + // script's path and load it into the preview pane so users immediately + // see the last run's logs/result instead of an empty panel. Used by the + // asset-graph details pane where selecting a script node should expose + // "what happened the last time this ran". No-op when a test is already + // in progress (we don't clobber a live run). + loadLastRunOnMount?: boolean + // Pipeline-only: when set, the Test split's caret popover gains a "Run + // downstream up to…" entry that calls this, letting the user bound the + // cascade from the script they're editing. Wired by the pipeline details + // pane only when the open script is a valid bounded-run start. + onBoundedRun?: () => void } export type EditorBarUi = { diff --git a/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts b/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts index c03695e0a3..c23a864a2b 100644 --- a/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts +++ b/frontend/src/lib/components/dbManagerDrawerModel.svelte.ts @@ -123,6 +123,7 @@ export function useDbManagerUriState(): DbManagerUriState { return { type: 'ducklake' as const, ducklake: parsed.path, + specificSchema: parsed.schema, specificTable: parsed.table } } @@ -161,6 +162,7 @@ export function useDbManagerUriState(): DbManagerUriState { params.dbm = buildDbm({ type: 'ducklake', path: nInput.ducklake, + schema: nInput.specificSchema, table: nInput.specificTable }) } diff --git a/frontend/src/lib/components/dbOps.ts b/frontend/src/lib/components/dbOps.ts index 8a528fd8e3..59d19ee172 100644 --- a/frontend/src/lib/components/dbOps.ts +++ b/frontend/src/lib/components/dbOps.ts @@ -1,5 +1,6 @@ import { getLanguageByResourceType, + ColumnIdentity, type ColumnDef, type TableMetadata } from './apps/components/display/dbtable/utils' @@ -45,12 +46,20 @@ export function dbTableOpsWithPreviewScripts({ input, tableKey, colDefs, - workspace + workspace, + whereClause, + version }: { input: DbInput tableKey: string colDefs: ColumnDef[] workspace: string + // Optional raw SQL predicate AND-ed into the read queries (count + rows). + // Caller-trusted — build it with escaped values. + whereClause?: string + // DuckLake time-travel: when set, reads are pinned to this catalog snapshot + // via `AT (VERSION => n)` (DuckDB/ducklake only). Read-only by nature. + version?: number }): IDbTableOps { const dbType = getDbType(input) const language = getLanguageByResourceType(dbType) @@ -67,7 +76,12 @@ export function dbTableOpsWithPreviewScripts({ tableKey, colDefs, getCount: async ({ quicksearch }) => { - const content = makeMarker('COUNT', { table: tableKey, columnDefs: colDefs }) + const content = makeMarker('COUNT', { + table: tableKey, + columnDefs: colDefs, + ...(whereClause ? { whereClause } : {}), + ...(version != undefined ? { version } : {}) + }) const result = await runScriptAndPollResult({ workspace, requestBody: { args: { ...dbArg, quicksearch }, language, content } @@ -79,7 +93,9 @@ export function dbTableOpsWithPreviewScripts({ const content = makeMarker('SELECT', { table: tableKey, columnDefs: colDefs, - fixPgIntTypes: true + fixPgIntTypes: true, + ...(whereClause ? { whereClause } : {}), + ...(version != undefined ? { version } : {}) }) let items = (await runScriptAndPollResult({ workspace, @@ -122,6 +138,90 @@ export function dbTableOpsWithPreviewScripts({ } } +export type DucklakeSnapshot = { + snapshot_id: number + // DuckLake returns this as microseconds-since-epoch serialized as a string + // (TIMESTAMP); callers must convert before formatting. + snapshot_time: string | number +} + +/** + * Column metadata of a ducklake table *at a specific snapshot*. The catalog's + * `information_schema` only reflects the current schema, so a time-travel read + * pinned to an older version must enumerate the columns that existed *then* — + * otherwise a column added in a later snapshot would break the `SELECT … AT + * (VERSION => n)`. `DESCRIBE SELECT * FROM … AT (VERSION => n)` gives exactly + * that. Returns minimal `ColumnDef`s (field + datatype) — enough for the + * read-only preview's SELECT/COUNT and grid headers. + */ +export async function fetchDucklakeColumnsAtVersion({ + workspace, + ducklake, + tableKey, + version +}: { + workspace: string + ducklake: string + tableKey: string + version: number +}): Promise { + // Quote each identifier part (schema.table) so a dotted/odd table name can't + // break the statement, and double single-quotes in the catalog name so it + // can't break out of the ATTACH string literal (mirrors the backend's + // `escape_sql_literal`). `version` is a number — injection-safe. + const quoted = tableKey + .split('.') + .map((p) => `"${p.replace(/"/g, '""')}"`) + .join('.') + const ducklakeLit = ducklake.replace(/'/g, "''") + const content = + `ATTACH 'ducklake://${ducklakeLit}' AS __dlv__; USE __dlv__; ` + + `DESCRIBE SELECT * FROM ${quoted} AT (VERSION => ${version});` + const rows = (await runScriptAndPollResult({ + workspace, + requestBody: { args: {}, language: 'duckdb', content } + })) as { column_name: string; column_type: string }[] + if (!Array.isArray(rows)) return [] + return rows.map((r) => ({ + field: r.column_name, + datatype: r.column_type, + defaultvalue: '', + isprimarykey: false, + isidentity: ColumnIdentity.No, + isnullable: 'YES' as const, + isenum: false + })) +} + +/** + * List a ducklake table's time-travel history, newest first. DuckLake snapshots + * are catalog-wide commits; passing `table` (schema-qualified, e.g. + * `main.events_daily`) scopes the list to snapshots where the table exists — + * otherwise an `AT (VERSION => n)` read could target a version predating the + * table's creation and error. Runs the `DUCKLAKE_SNAPSHOTS` marker as a duckdb + * preview job (server-side SQL build + ATTACH), so no raw SQL is constructed in + * the client. + */ +export async function fetchDucklakeSnapshots({ + workspace, + ducklake, + table +}: { + workspace: string + ducklake: string + table?: string +}): Promise { + const content = `-- WM_INTERNAL_DB_DUCKLAKE_SNAPSHOTS ${JSON.stringify({ + ducklake, + ...(table ? { table } : {}) + })}` + const rows = await runScriptAndPollResult({ + workspace, + requestBody: { args: {}, language: 'duckdb', content } + }) + return Array.isArray(rows) ? (rows as DucklakeSnapshot[]) : [] +} + export type IDbSchemaOps = { onDelete: (params: { tableKey: string; schema?: string }) => Promise onCreate: (params: { values: TableEditorValues; schema?: string }) => Promise @@ -297,35 +397,45 @@ export async function getDucklakeSchema({ args: {} } }) - let mainSchema = Array.isArray(result) && result.length && (result?.[0]?.['result'] ?? []) + let schemas = Array.isArray(result) && result.length && (result?.[0]?.['result'] ?? {}) // Safety for agent workers (duckdb ffi lib used to return JSON as stringified json) - if (typeof mainSchema === 'string') mainSchema = JSON.parse(mainSchema) + if (typeof schemas === 'string') schemas = JSON.parse(schemas) - if (!mainSchema) throw new Error('Failed to get Ducklake schema: ' + JSON.stringify(result)) - assert('mainSchema is an object', typeof mainSchema === 'object') + if (!schemas) throw new Error('Failed to get Ducklake schema: ' + JSON.stringify(result)) + assert('schemas is an object', typeof schemas === 'object') let schema: Omit = { - schema: { main: mainSchema }, - publicOnly: true, + schema: schemas, + publicOnly: false, lang: 'ducklake' } return { ...schema, stringified: stringifySchema(schema) } } +// Returns every schema in the ducklake (including empty ones, e.g. freshly created) +// as a nested map { schema: { table: { column: {...} } } }. const DUCKLAKE_GET_SCHEMA_QUERY = ` -SELECT json_group_object(table_name, table_data) AS result FROM ( +SELECT json_group_object(schema_name, COALESCE(schema_data, json_object())) AS result FROM ( SELECT - table_name, - json_group_object( - c.column_name, - json_object( - 'type', c.data_type, - 'default', c.column_default, - 'required', c.is_nullable == 'NO' + s.schema_name, + ( + SELECT json_group_object(table_name, table_data) FROM ( + SELECT + c.table_name, + json_group_object( + c.column_name, + json_object( + 'type', c.data_type, + 'default', c.column_default, + 'required', c.is_nullable == 'NO' + ) + ) AS table_data + FROM information_schema.columns c + WHERE c.table_catalog = '__ducklake__' AND c.table_schema = s.schema_name + GROUP BY c.table_name ) - ) AS table_data - FROM information_schema.columns c - WHERE table_catalog = '__ducklake__' AND table_schema = current_schema() - GROUP BY c.table_name + ) AS schema_data + FROM information_schema.schemata s + WHERE s.catalog_name = '__ducklake__' )` export function getDbType(input: DbInput): DbType { diff --git a/frontend/src/lib/components/dbTypes.ts b/frontend/src/lib/components/dbTypes.ts index 72ba4cedef..6a85617110 100644 --- a/frontend/src/lib/components/dbTypes.ts +++ b/frontend/src/lib/components/dbTypes.ts @@ -9,6 +9,7 @@ export type DbInput = | { type: 'ducklake' ducklake: string + specificSchema?: string specificTable?: string } diff --git a/frontend/src/lib/components/diff_drawer.ts b/frontend/src/lib/components/diff_drawer.ts index 108eddf8fc..c6e9b5cdf8 100644 --- a/frontend/src/lib/components/diff_drawer.ts +++ b/frontend/src/lib/components/diff_drawer.ts @@ -1,24 +1,24 @@ -import type { Value } from "$lib/utils" +import type { Value } from '$lib/utils' -export type DiffDrawerDiff = -| { - mode: 'normal' - deployed: Value - draft: Value | undefined - current: Value - defaultDiffType?: 'deployed' | 'draft' - button?: { text: string; onClick: () => void } -} -| { - mode: 'simple' - original: Value - current: Value - title: string - button?: { text: string; onClick: () => void } -} +export type DiffDrawerDiff = + | { + mode: 'normal' + deployed: Value + draft?: Value | undefined + current: Value + defaultDiffType?: 'deployed' | 'draft' + button?: { text: string; onClick: () => void } + } + | { + mode: 'simple' + original: Value + current: Value + title: string + button?: { text: string; onClick: () => void } + } export interface DiffDrawerI { - openDrawer: () => void - closeDrawer: () => void - setDiff: (diff: DiffDrawerDiff) => void -} \ No newline at end of file + openDrawer: () => void + closeDrawer: () => void + setDiff: (diff: DiffDrawerDiff) => void +} diff --git a/frontend/src/lib/components/flow_builder.ts b/frontend/src/lib/components/flow_builder.ts index 5d2493b930..48340e937e 100644 --- a/frontend/src/lib/components/flow_builder.ts +++ b/frontend/src/lib/components/flow_builder.ts @@ -1,7 +1,7 @@ -import type { OpenFlow } from '$lib/gen' +import type { Flow, OpenFlow } from '$lib/gen' import type { StateStore } from '$lib/utils' import type { FlowState } from './flows/flowState' -import type { FlowWithDraftAndDraftTriggers, Trigger } from './triggers/utils' +import type { Trigger } from './triggers/utils' import type { DiffDrawerI } from './diff_drawer' import type { FlowBuilderWhitelabelCustomUi } from './custom_ui' import type { ScheduleTrigger } from './triggers' @@ -17,14 +17,16 @@ export type FlowBuilderProps = { loading?: boolean flowStore: StateStore flowStateStore: StateStore - savedFlow?: FlowWithDraftAndDraftTriggers | undefined + savedFlow?: Flow & { no_deployed?: boolean } diffDrawer?: DiffDrawerI | undefined customUi?: FlowBuilderWhitelabelCustomUi disableAi?: boolean disabledFlowInputs?: boolean savedPrimarySchedule?: ScheduleTrigger | undefined // used to set the primary schedule in the legacy primaryScheduleStore version?: number | undefined - setSavedraftCb?: ((cb: () => void) => void) | undefined + /** flow_version the draft was forked from; when set, the deploy-time staleness + * check compares it (not the load-time head `version`) against the latest. */ + draftBaseVersion?: number | undefined draftTriggersFromUrl?: Trigger[] | undefined selectedTriggerIndexFromUrl?: number | undefined children?: import('svelte').Snippet @@ -34,23 +36,29 @@ export type FlowBuilderProps = { } noInitial?: boolean liveEditorDraftStoragePath?: string - onSaveInitial?: ({ path, id }: { path: string; id: string }) => void - onSaveDraft?: ({ - path, - savedAtNewPath, - newFlow - }: { - path: string - savedAtNewPath: boolean - newFlow: boolean - }) => void - onSaveDraftError?: ({ error }: { error: any }) => void - onSaveDraftOnlyAtNewPath?: ({ path, selectedId }: { path: string; selectedId: string }) => void + // Indicator-only draft key overrides. When the flow editor is embedded + // (e.g. the sessions preview) its autosave runs under a different + // (workspace, path) than `$workspaceStore`/`liveEditorDraftStoragePath` + // (a forked workspace, and a path this component doesn't own). These let + // the host point the `AutosaveIndicator` at the key its own autosave uses, + // WITHOUT repurposing `liveEditorDraftStoragePath` (which still drives this + // component's setLiveEditorDraft/flush). Undefined → fall back, so the + // full-page editor is unaffected. + autosaveWorkspace?: string + autosavePath?: string onDeploy?: ({ path }: { path: string }) => void onDeployError?: ({ error }: { error: any }) => void onDetails?: ({ path }: { path: string }) => void onHistoryRestore?: () => void onNavigate?: (item: WorkspaceItem) => void + // Threaded to the `AutosaveIndicator` popover so its "Reset to + // deployed" button can do the same thing the load-time toast offers. + onResetToDeployed?: () => void | Promise + // See ScriptBuilderProps — same semantics for the flow editor's + // indicator. + loadedFromDraft?: boolean + othersDraftsCount?: number + onOpenOthersDrafts?: () => void // Fired whenever a test run is started from the flow editor, with the // preview job id. Used by whitelabel embedders to track test jobs. onTestJob?: (e: { jobId: string }) => void diff --git a/frontend/src/lib/components/flows/CreateActionsApp.svelte b/frontend/src/lib/components/flows/CreateActionsApp.svelte index 5cc0df1502..7ae5e9850c 100644 --- a/frontend/src/lib/components/flows/CreateActionsApp.svelte +++ b/frontend/src/lib/components/flows/CreateActionsApp.svelte @@ -26,10 +26,10 @@ // Navigation to /apps_raw/add triggers a full page reload (for cross-origin isolation), // so the in-memory importStore would be lost. Use sessionStorage instead. sessionStorage.setItem('rawAppImport', JSON.stringify(parsed)) - await goto('/apps_raw/add?nodraft=true') + await goto('/apps_raw/add') } else { $importStore = parsed - await goto('/apps/add?nodraft=true') + await goto('/apps/add') } drawer?.closeDrawer?.() } @@ -40,51 +40,51 @@ function selectLowCode() { appTypeModalOpen = false - goto(`${base}/apps/add?nodraft=true`) + goto(`${base}/apps/add`) } function selectFullCode() { appTypeModalOpen = false - goto(`${base}/apps_raw/add?nodraft=true`) + goto(`${base}/apps_raw/add`) }
- -
+ }, + { + label: 'Import full-code app', + onClick: () => { + appKind = 'fullcode' + importType = 'yaml' + drawer?.toggleDrawer?.() + } + } + ]} + > +
App
+ +
- +
@@ -122,8 +122,8 @@

Build with React or Svelte with full control and a powerful AI agent. -

- Better for complex apps or apps that require full flexibility and control. +

+ Better for complex apps or apps that require full flexibility and control.

diff --git a/frontend/src/lib/components/flows/CreateActionsFlow.svelte b/frontend/src/lib/components/flows/CreateActionsFlow.svelte index dd975fcc72..ae848156f5 100644 --- a/frontend/src/lib/components/flows/CreateActionsFlow.svelte +++ b/frontend/src/lib/components/flows/CreateActionsFlow.svelte @@ -9,15 +9,12 @@ import { importFlowStore } from '$lib/components/flows/flowStore.svelte' import { importScriptStore } from '$lib/components/scripts/scriptStore.svelte' import Modal from '$lib/components/common/modal/Modal.svelte' - import Toggle from '$lib/components/Toggle.svelte' import Tabs from '$lib/components/common/tabs/Tabs.svelte' import Tab from '$lib/components/common/tabs/Tab.svelte' import { PythonIcon, TypeScriptIcon } from '$lib/components/common/languageIcons' import { Code2, Loader2, Plus } from 'lucide-svelte' import YAML from 'yaml' - const SKIP_FLOW_MODAL_KEY = 'windmill_skip_flow_modal' - let drawer: Drawer | undefined = $state(undefined) let wacDrawer: Drawer | undefined = $state(undefined) let pendingRaw: string | undefined = $state(undefined) @@ -26,14 +23,11 @@ let wacImportType: 'yaml' | 'json' = $state('yaml') let flowModalOpen = $state(false) let wacHovered = $state(false) - let skipModal = $state( - typeof localStorage !== 'undefined' && localStorage.getItem(SKIP_FLOW_MODAL_KEY) === 'true' - ) async function importRaw() { $importFlowStore = importType === 'yaml' ? YAML.parse(pendingRaw ?? '') : JSON.parse(pendingRaw ?? '') - await goto('/flows/add?nodraft=true') + await goto('/flows/add') drawer?.closeDrawer?.() } @@ -41,36 +35,32 @@ const parsed = wacImportType === 'yaml' ? YAML.parse(pendingWacRaw ?? '') : JSON.parse(pendingWacRaw ?? '') $importScriptStore = parsed - await goto(`${base}/scripts/add?import=true&nodraft=true`) + await goto(`${base}/scripts/add?import=true`) wacDrawer?.closeDrawer?.() } function handleFlowClick() { - if (skipModal) { - goto(`${base}/flows/add?nodraft=true`) - } else { - flowModalOpen = true - } + flowModalOpen = true } function selectFlowEditor() { flowModalOpen = false - goto(`${base}/flows/add?nodraft=true`) + goto(`${base}/flows/add`) } function selectWacPython() { flowModalOpen = false - goto(`${base}/scripts/add?nodraft=true&wac=python`) + goto(`${base}/scripts/add?wac=python`) } function selectWacTypescript() { flowModalOpen = false - goto(`${base}/scripts/add?nodraft=true&wac=typescript`) + goto(`${base}/scripts/add?wac=typescript`) } - function toggleSkipModal() { - skipModal = !skipModal - localStorage.setItem(SKIP_FLOW_MODAL_KEY, String(skipModal)) + function selectPipeline() { + flowModalOpen = false + goto(`${base}/pipeline`) } @@ -105,6 +95,10 @@ onClick: () => { wacDrawer?.toggleDrawer?.() } + }, + { + label: 'Pipeline (alpha)', + onClick: () => selectPipeline() } ]} > @@ -113,9 +107,11 @@
- + +
-
+
- -
- - Always use the Flow editor (skip this modal) -
diff --git a/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte b/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte index 30311e2fa3..0ce0843bbf 100644 --- a/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte +++ b/frontend/src/lib/components/flows/content/FlowEditorDrawer.svelte @@ -24,23 +24,14 @@ flowEditorDrawer?.openDrawer?.() try { - const flowWithDraft = await FlowService.getFlowByPathWithDraft({ + const backendFlow = await FlowService.getFlowByPath({ workspace: $workspaceStore!, path }) - savedFlow = { - ...structuredClone(flowWithDraft), - draft: flowWithDraft.draft - ? { - ...structuredClone(flowWithDraft.draft), - path: flowWithDraft.draft.path ?? flowWithDraft.path - } - : undefined - } as Flow & { draft?: Flow } + savedFlow = structuredClone(backendFlow) as Flow - // Use the draft if available, otherwise the deployed flow - flow = flowWithDraft.draft ?? flowWithDraft + flow = backendFlow await initFlow(flow, flowStore, flowStateStore) loading = false @@ -53,11 +44,7 @@ let callback: (() => void) | undefined = undefined let flowPath: string = $state('') let flow: Flow | undefined = $state(undefined) - let savedFlow: - | (Flow & { - draft?: Flow | undefined - }) - | undefined = $state(undefined) + let savedFlow: Flow | undefined = $state(undefined) let loading = $state(true) const flowStore: StateStore = $state({ diff --git a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte index 1bf578a081..46a5c78d32 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleComponent.svelte @@ -867,7 +867,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { @@ -931,7 +930,6 @@ bind:this={editor} class="h-full relative" code={flowModule.value.content} - syncExternalCode scriptLang={flowModule?.value?.language} automaticLayout={true} cmdEnterAction={async () => { diff --git a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte index 39aaaa66f2..62a3432995 100644 --- a/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte +++ b/frontend/src/lib/components/flows/content/FlowModuleTimeout.svelte @@ -108,9 +108,14 @@ {/if} -
- -

The timeout will be ignored when running "Test this step"

-
-
+ {#if flowModule.timeout && flowModule.timeout.type !== 'static'} +
+ +

+ A dynamic timeout expression is evaluated when running the full flow. It is ignored when + running "Test this step" — only a static timeout value applies there. +

+
+
+ {/if} diff --git a/frontend/src/lib/components/flows/flowModuleNextId.test.ts b/frontend/src/lib/components/flows/flowModuleNextId.test.ts new file mode 100644 index 0000000000..dcf007e925 --- /dev/null +++ b/frontend/src/lib/components/flows/flowModuleNextId.test.ts @@ -0,0 +1,53 @@ +import { describe, expect, it } from 'vitest' + +import type { OpenFlow } from '$lib/gen' +import type { FlowState } from './flowState' +import { nextId } from './flowModuleNextId' + +function flowWith(ids: string[]): OpenFlow { + return { + summary: '', + value: { + modules: ids.map((id) => ({ id, value: { type: 'identity' } as any })) + } + } as OpenFlow +} + +function stateWith(keys: string[]): FlowState { + return Object.fromEntries(keys.map((k) => [k, {}])) as FlowState +} + +describe('nextId', () => { + it('produces a, b, c, ... for a fresh flow', () => { + expect(nextId(stateWith(['failure']), flowWith([]))).toBe('a') + expect(nextId(stateWith(['a', 'failure']), flowWith(['a']))).toBe('b') + expect(nextId(stateWith(['a', 'b', 'c', 'failure']), flowWith(['a', 'b', 'c']))).toBe('d') + }) + + it('ignores the reserved failure/preprocessor keys always present in flowState', () => { + expect(nextId(stateWith(['failure', 'preprocessor']), flowWith([]))).toBe('a') + }) + + // Regression: copy ids ("z2"), subflow result keys and other non-canonical keys land in + // flowState; charsToNumber on them used to leak into the max and made new steps jump to + // garbage ids like "bzw". + it('is not poisoned by copy ids', () => { + const ids = ['a', 'b', 'c'] + const state = stateWith([...ids, 'c2', 'a2', 'z2', 'c10', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('d') + }) + + it('is not poisoned by subflow result keys', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'subflow:abcd', 'Result', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) + + // A step renamed to a long lowercase word ("process") is a valid base-26 string and would + // otherwise inflate the max; the length cutoff keeps such renames out of the sequence. + it('is not poisoned by renames to long lowercase words or underscored ids', () => { + const ids = ['a', 'b'] + const state = stateWith([...ids, 'process', 'my_step', 'failure']) + expect(nextId(state, flowWith(ids))).toBe('c') + }) +}) diff --git a/frontend/src/lib/components/flows/flowModuleNextId.ts b/frontend/src/lib/components/flows/flowModuleNextId.ts index e10c900982..48b2eb5ac2 100644 --- a/frontend/src/lib/components/flows/flowModuleNextId.ts +++ b/frontend/src/lib/components/flows/flowModuleNextId.ts @@ -1,19 +1,35 @@ import type { OpenFlow } from '$lib/gen' import { dfs } from './dfs' import type { FlowState } from './flowState' -import { charsToNumber, numberToChars } from './idUtils' +import { charsToNumber, forbiddenIds, numberToChars } from './idUtils' + +const reservedIds = new Set(forbiddenIds) + +// Returns the base-26 value of a key only if it is a short, auto-generated step id +// (a, b, ..., z, aa, ...). flowState/module-id keys also include copy ids ("a2"), subflow +// result keys ("subflow:..."), reserved keys and user-renamed ids; feeding those through +// charsToNumber yields meaningless (often huge) numbers that would poison id generation and +// make new steps jump to ids like "bzw". Short non-canonical keys are rejected via a +// round-trip check; longer keys are skipped entirely, which also leaves user renames to long +// lowercase words (e.g. "process") out of the sequence. +function autoIdNumber(key: string): number | undefined { + if (key.length >= 4 || reservedIds.has(key)) { + return undefined + } + const num = charsToNumber(key) + if (num < 0 || numberToChars(num) !== key) { + return undefined + } + return num +} // Computes the next available id export function nextId(flowState: FlowState, fullFlow: OpenFlow): string { const allIds = dfs(fullFlow.value.modules, (fm) => fm.id) const max = allIds.concat(Object.keys(flowState)).reduce((acc, key) => { - if (key.length >= 4) { - return acc - } else { - const num = charsToNumber(key) - return Math.max(acc, num + 1) - } + const num = autoIdNumber(key) + return num === undefined ? acc : Math.max(acc, num + 1) }, 0) return numberToChars(max) } diff --git a/frontend/src/lib/components/flows/map/VirtualItem.svelte b/frontend/src/lib/components/flows/map/VirtualItem.svelte index f7a08f1933..652cf4041d 100644 --- a/frontend/src/lib/components/flows/map/VirtualItem.svelte +++ b/frontend/src/lib/components/flows/map/VirtualItem.svelte @@ -9,7 +9,11 @@ import { Database, Square } from 'lucide-svelte' import FlowGraphPreviewButton from './FlowGraphPreviewButton.svelte' import type { Job } from '$lib/gen' - import { getNodeColorClasses, aiActionToNodeState } from '$lib/components/graph' + import { + getNodeColorClasses, + aiActionToNodeState, + type FlowNodeState + } from '$lib/components/graph' import { getGraphContext } from '$lib/components/graph/graphContext' interface Props { @@ -39,6 +43,9 @@ job?: Job showJobStatus?: boolean flowHasChanged?: boolean + /** When set, overrides the node outline with this run-state's colored outline. + * Used to mark the branch taken at runtime on branchone/branchall nodes. */ + borderState?: FlowNodeState } let { @@ -67,14 +74,13 @@ individualStepTests = false, job, showJobStatus = false, - flowHasChanged = false + flowHasChanged = false, + borderState = undefined }: Props = $props() const flowGraphContext = getGraphContext() - let isMultiSelected = $derived( - (flowGraphContext?.selectionManager?.selectedIds?.length ?? 0) > 1 - ) + let isMultiSelected = $derived((flowGraphContext?.selectionManager?.selectedIds?.length ?? 0) > 1) const outputPickerVisible = $derived( (nodeKind || (inputJson && Object.keys(inputJson).length > 0)) && editMode @@ -96,6 +102,10 @@ // AI action colors take priority over execution state, fallback to _VirtualItem const effectiveState = $derived(aiActionToNodeState(action) ?? outputType ?? '_VirtualItem') let colorClasses = $derived(getNodeColorClasses(effectiveState, selected)) + // The branch taken at runtime keeps its outline regardless of selection so it stays visible. + let outlineClasses = $derived( + borderState ? getNodeColorClasses(borderState, true).outline : colorClasses.outline + )
diff --git a/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts b/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts index 06a4972ee5..8aa7a1f69f 100644 --- a/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts +++ b/frontend/src/lib/components/flows/stepsInputArgs.svelte.ts @@ -100,9 +100,11 @@ export class StepsInputArgs { if (modules.length < 1) { return } + // dfs returns [step, immediate parent, ..., root]; the prop picker needs the + // immediate parent so nested loops resolve flow_input.iter to the innermost loop. let parentModule: FlowModule | undefined = undefined if (modules.length > 1) { - parentModule = modules[modules.length - 1] + parentModule = modules[1] } const stepPropPicker = getStepPropPicker( flowState, @@ -175,9 +177,11 @@ export class StepsInputArgs { if (modules.length < 1) { return } + // dfs returns [step, immediate parent, ..., root]; the prop picker needs the + // immediate parent so nested loops resolve flow_input.iter to the innermost loop. let parentModule: FlowModule | undefined = undefined if (modules.length > 1) { - parentModule = modules[modules.length - 1] + parentModule = modules[1] } const stepPropPicker = getStepPropPicker( flowState, diff --git a/frontend/src/lib/components/flows/types.ts b/frontend/src/lib/components/flows/types.ts index be0114ccc1..9b02f7224b 100644 --- a/frontend/src/lib/components/flows/types.ts +++ b/frontend/src/lib/components/flows/types.ts @@ -94,6 +94,10 @@ export type FlowEditorContext = { outputPickerOpenFns: Record void> preserveOnBehalfOf: Writable savedOnBehalfOfEmail: Writable + // Only set by the local dev page (Dev.svelte): path -> temp-storage hash of + // locally-edited workspace scripts, passed as temp_script_refs on preview + // runs so relative imports resolve from local (not-yet-deployed) content + devTempScriptRefs?: () => Record | undefined } export type FlowGraphAssetContext = StateStore<{ diff --git a/frontend/src/lib/components/flows/utils.svelte.ts b/frontend/src/lib/components/flows/utils.svelte.ts index f260813a1b..42ff5f57b9 100644 --- a/frontend/src/lib/components/flows/utils.svelte.ts +++ b/frontend/src/lib/components/flows/utils.svelte.ts @@ -184,7 +184,8 @@ export async function runFlowPreview( flow: OpenFlow & { tag?: string }, path: string, restartedFrom: RestartedFrom | undefined, - conversationId?: string | undefined + conversationId?: string | undefined, + tempScriptRefs?: Record ) { const newFlow = flow return await JobService.runFlowPreview({ @@ -194,7 +195,8 @@ export async function runFlowPreview( value: newFlow.value, path: path, tag: newFlow.tag, - restarted_from: restartedFrom + restarted_from: restartedFrom, + temp_script_refs: tempScriptRefs }, memoryId: conversationId }) diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte index 1f2623e552..9682d941b6 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchAllStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchAllStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchAllStartN['data'] id: string @@ -14,6 +15,10 @@ let { data, id }: Props = $props() const { selectionManager } = getGraphContext() + + let borderStatus = $derived( + computeBorderStatus(data.branchIndex, 'branchall', data.flowModuleState) + ) @@ -22,6 +27,7 @@ label={data.label} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data.eventHandlers.select(data.id)) }} diff --git a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte index 6333d32845..9902582ee0 100644 --- a/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte +++ b/frontend/src/lib/components/graph/renderers/nodes/BranchOneStart.svelte @@ -6,6 +6,7 @@ import { X } from 'lucide-svelte' import type { BranchOneStartN } from '../../graphBuilder.svelte' import { getGraphContext } from '../../graphContext' + import { computeBorderStatus } from '../utils' interface Props { data: BranchOneStartN['data'] id: string @@ -13,6 +14,12 @@ const { selectionManager } = getGraphContext() let { data, id }: Props = $props() + + // branchIndex is -1 for the default branch and 0-based for explicit branches; + // branchChosen is 0 for default and 1-based, hence the +1. + let borderStatus = $derived( + computeBorderStatus(data.branchIndex + 1, 'branchone', data.flowModuleState) + ) @@ -22,11 +29,12 @@ preLabel={data.preLabel} selectable selected={selectionManager && selectionManager.isNodeSelected(id)} + borderState={borderStatus} on:select={() => { setTimeout(() => data?.eventHandlers?.select(data.id)) }} /> - {#if data.insertable} + {#if data.insertable && data.branchIndex >= 0} + + {#if $open && active} +
+ {#if showDoc} + +
+
+
+ +
+
+
+

{active.label}

+ {#if active.badge} + + {active.badge.label} + + {/if} +
+

{active.tagline}

+
+
+ +

{active.description}

+ +
    + {#each active.bullets as bullet (bullet)} +
  • + + {bullet} +
  • + {/each} +
+ + +
+ {/if} + + +
+ {#snippet rowBody(option: Option, ac: (typeof accentClasses)[string])} +
+ +
+ + {option.label} + + {#if option.badge} + + {option.badge.label} + + {/if} + {/snippet} + {#each allOptions as option (option.key)} + {@const ac = accentClasses[option.accent]} + {@const rowClass = + 'w-full flex flex-row items-center gap-2.5 rounded-md px-2 py-1.5 text-left cursor-pointer transition-colors focus:outline-none data-[highlighted]:bg-surface-hover hover:bg-surface-hover'} + {#if option.variants} + + {#if $wacSubOpen} +
+ {#each option.variants ?? [] as variant (variant.label)} + {@const VariantIcon = variant.icon} + + {/each} +
+ {/if} + {:else} + + {/if} + {/each} + + +
+ + {#if $importSubOpen} +
+ {#each importActions as action (action.label)} + + {/each} +
+ {/if} + + {#if !showDoc} + + {/if} +
+
+ {/if} +
+ + + + importDrawer?.closeDrawer?.()}> + + + + {#snippet content()} +
+ {#key importType} + {#await import('$lib/components/SimpleEditor.svelte')} + + {:then Module} + + {/await} + {/key} +
+ {/snippet} +
+ {#snippet actions()} + + {/snippet} +
+
diff --git a/frontend/src/lib/components/home/ItemsList.svelte b/frontend/src/lib/components/home/ItemsList.svelte index 80b04fa455..b6ae577d4b 100644 --- a/frontend/src/lib/components/home/ItemsList.svelte +++ b/frontend/src/lib/components/home/ItemsList.svelte @@ -4,6 +4,7 @@ import Toggle from '$lib/components/Toggle.svelte' import { AppService, + AssetService, FlowService, type ListableApp, type Script, @@ -11,6 +12,7 @@ type Flow, type ListableRawApp } from '$lib/gen' + import { resource } from 'runed' import { userStore, workspaceStore } from '$lib/stores' import type uFuzzy from '@leeoniya/ufuzzy' import { @@ -43,6 +45,8 @@ import { getContext, tick, untrack } from 'svelte' import { triggerableByAI } from '$lib/actions/triggerableByAI.svelte' import TextInput from '../text_input/TextInput.svelte' + import { NetworkIcon } from 'lucide-svelte' + import { base } from '$lib/base' interface Props { filter?: string subtab?: 'flow' | 'script' | 'app' @@ -61,7 +65,6 @@ type?: U time?: number starred?: boolean - has_draft?: boolean hash?: string } @@ -70,6 +73,24 @@ type TableApp = TableItem type TableRawApp = TableItem + // Folders with ≥1 pipeline script (auto_kind='pipeline'). Used by + // TreeView to surface a "Pipeline" entry inside those folders. Cheap + // thanks to the partial index on script.auto_kind. + let pipelineFoldersRes = resource( + () => $workspaceStore, + async (ws) => { + if (!ws) return new Set() + try { + const rows = await AssetService.listPipelineFolders({ workspace: ws }) + return new Set(rows.map((r) => r.folder)) + } catch { + // Decorative tree entry — degrade to "no pipelines" on failure. + return new Set() + } + } + ) + let pipelineFolders = $derived(pipelineFoldersRes.current ?? new Set()) + let scripts: TableScript[] | undefined = $state() let flows: TableFlow[] | undefined = $state() let apps: TableApp[] | undefined = $state() @@ -241,9 +262,13 @@ async function showCode(path: string, summary: string) { viewCodeTitle = summary || path await viewCodeDrawer?.openDrawer() + // `getDraft: true` so draft-only scripts (no deployed row at this + // path) still return their content via the per-user draft overlay + // instead of 404'ing. script = await ScriptService.getScriptByPath({ workspace: $workspaceStore!, - path + path, + getDraft: true }) } @@ -304,11 +329,17 @@ let allLabels = $derived( Array.from(new Set(combinedItems?.flatMap((x) => itemLabels(x)) ?? [])).sort() ) + let prevWorkspace: string | undefined = undefined + // Clear filters only when the workspace actually changes. The initial + // resolution must be left alone so URL-loaded filter values (set by + // ListFilters.loadFilterFromUrl on mount) survive the async store settling. $effect(() => { - if ($workspaceStore) { + const ws = $workspaceStore + if (ws && prevWorkspace !== undefined && ws !== prevWorkspace) { ownerFilter = undefined labelFilter = undefined } + prevWorkspace = ws }) let preFilteredItems = $derived( ownerFilter != undefined @@ -475,7 +506,8 @@ if (menuItem) { if (e.key === 'ArrowUp' || e.key === 'ArrowDown') { const menu = menuItem.closest('[role="menu"]') - if (menu) { + // menus marked data-arrow-loop keep melt's cyclic wrap instead of exiting + if (menu && !menu.hasAttribute('data-arrow-loop')) { const items = Array.from(menu.querySelectorAll('[role="menuitem"]')) const idx = items.indexOf(menuItem) const isFirst = idx === 0 @@ -801,6 +833,7 @@ {items} {nbDisplayed} {collapseAll} + {pipelineFolders} isSearching={filter !== ''} on:scriptChanged={() => loadScripts(includeWithoutMain)} on:flowChanged={loadFlows} @@ -816,6 +849,17 @@ /> {:else}
+ {#if filter === ''} + {#each [...pipelineFolders].sort() as folder (folder)} + + + Pipeline · f/{folder} + + {/each} + {/if} {#each displayedItems as item, i (item.type + '/' + item.path + (item.hash ? '/' + item.hash : ''))} import TreeView from './TreeView.svelte' - import { ChevronDown, ChevronUp, Folder, FolderTree, User } from 'lucide-svelte' + import { ChevronDown, ChevronUp, Folder, FolderTree, NetworkIcon, User } from 'lucide-svelte' import Item from './Item.svelte' import type { FolderItem, ItemType, UserItem } from './treeViewUtils' import { twMerge } from 'tailwind-merge' import { pluralize } from '$lib/utils' + import { base } from '$lib/base' interface Props { item: ItemType | FolderItem | UserItem @@ -13,11 +14,24 @@ depth?: number showCode: (path: string, summary: string) => void isSearching?: boolean + pipelineFolders?: Set } - let { item, collapseAll, depth = 0, showCode, isSearching = false }: Props = $props() + let { + item, + collapseAll, + depth = 0, + showCode, + isSearching = false, + pipelineFolders + }: Props = $props() - const isFolder = (i: typeof item): i is FolderItem => i && 'folderName' in i + const isFolderItem = (i: typeof item): i is FolderItem => i && 'folderName' in i + let hasPipeline = $derived( + depth === 0 && isFolderItem(item) && (pipelineFolders?.has(item.folderName) ?? false) + ) + + const isFolder = isFolderItem const isUser = (i: typeof item): i is UserItem => i && 'username' in i let opened: boolean = $state(true) @@ -67,11 +81,25 @@
{#if opened || isSearching}
+ {#if hasPipeline && isFolder(item)} + + + + Pipeline + + {/if} {#each item.items.slice(0, showMax) as subItem, index ((subItem['path'] ? subItem['type'] + '__' + subItem['path'] + '__' + index : undefined) ?? 'folder__' + subItem['folderName'] + '__' + index)} } let { @@ -16,7 +17,8 @@ showCode, nbDisplayed = $bindable(), items, - isSearching = false + isSearching = false, + pipelineFolders }: Props = $props() let groupedItems: ReturnType | 'loading' = $state('loading') @@ -37,12 +39,13 @@
{:else}
- {#each groupedItems.slice(0, nbDisplayed) as item (item['folderName'] ?? 'user__' + item['username'])} + {#each groupedItems.slice(0, nbDisplayed) as item ('folderName' in item ? `f__${item.folderName}` : 'username' in item ? `u__${item.username}` : `i__${item.type}__${item.path}`)} {#if item} 15 && nbDisplayed < groupedItems.length} {nbDisplayed} root nodes out of {groupedItems.length} - (nbDisplayed += 30)}>load 30 more {/if} {/if} diff --git a/frontend/src/lib/components/home/treeViewUtils.ts b/frontend/src/lib/components/home/treeViewUtils.ts index 3196621a4d..1bbe013e46 100644 --- a/frontend/src/lib/components/home/treeViewUtils.ts +++ b/frontend/src/lib/components/home/treeViewUtils.ts @@ -5,7 +5,6 @@ type TableItem = T & { type?: U time?: number starred?: boolean - has_draft?: boolean } type TableScript = TableItem diff --git a/frontend/src/lib/components/icons/CoupaIcon.svelte b/frontend/src/lib/components/icons/CoupaIcon.svelte new file mode 100644 index 0000000000..4d4058fdac --- /dev/null +++ b/frontend/src/lib/components/icons/CoupaIcon.svelte @@ -0,0 +1,22 @@ + + + + + diff --git a/frontend/src/lib/components/icons/NetsuiteIcon.svelte b/frontend/src/lib/components/icons/NetsuiteIcon.svelte new file mode 100644 index 0000000000..768bbbc99e --- /dev/null +++ b/frontend/src/lib/components/icons/NetsuiteIcon.svelte @@ -0,0 +1,19 @@ + + + + + + diff --git a/frontend/src/lib/components/icons/OutreachIcon.svelte b/frontend/src/lib/components/icons/OutreachIcon.svelte new file mode 100644 index 0000000000..a41d123ecd --- /dev/null +++ b/frontend/src/lib/components/icons/OutreachIcon.svelte @@ -0,0 +1,38 @@ + + + + outreach-svg + + diff --git a/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte b/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte new file mode 100644 index 0000000000..fc9c2843ec --- /dev/null +++ b/frontend/src/lib/components/icons/WhatsappBusinessIcon.svelte @@ -0,0 +1,20 @@ + + + + + diff --git a/frontend/src/lib/components/icons/fileIcon.ts b/frontend/src/lib/components/icons/fileIcon.ts new file mode 100644 index 0000000000..fef893e398 --- /dev/null +++ b/frontend/src/lib/components/icons/fileIcon.ts @@ -0,0 +1,74 @@ +/** + * Resolve a file name (or relative path) to an icon by extension. Shared by the + * raw-app file tree and the AI-chat file attachments so both stay consistent. + */ +import { File, ImageIcon } from 'lucide-svelte' +import TypeScript from '../common/languageIcons/TypeScript.svelte' +import JavaScriptIcon from './JavaScriptIcon.svelte' +import JsonIcon from './JsonIcon.svelte' +import ReactIcon from './ReactIcon.svelte' +import SvelteIcon from './SvelteIcon.svelte' +import VueIcon from './VueIcon.svelte' +import CssIcon from './CssIcon.svelte' +import SassIcon from './SassIcon.svelte' +import LessIcon from './LessIcon.svelte' +import HtmlIcon from './HtmlIcon.svelte' +import MarkdownIcon from './MarkdownIcon.svelte' +import YamlIcon from './YamlIcon.svelte' + +export interface ResolvedFileIcon { + icon: any + className?: string +} + +/** Lowercased extension of a file name or path (basename only); '' if none. */ +export function getFileExtension(filename: string): string { + const base = filename.split('/').pop() ?? filename + const parts = base.split('.') + return parts.length > 1 ? parts[parts.length - 1].toLowerCase() : '' +} + +/** Icon (and optional color class) for a file, by extension. */ +export function getFileIcon(filename: string): ResolvedFileIcon { + switch (getFileExtension(filename)) { + case 'json': + return { icon: JsonIcon } + case 'tsx': + case 'jsx': + return { icon: ReactIcon } + case 'ts': + return { icon: TypeScript } + case 'js': + return { icon: JavaScriptIcon } + case 'svelte': + return { icon: SvelteIcon } + case 'vue': + return { icon: VueIcon } + case 'css': + return { icon: CssIcon } + case 'scss': + case 'sass': + return { icon: SassIcon } + case 'less': + return { icon: LessIcon } + case 'png': + case 'jpg': + case 'jpeg': + case 'gif': + case 'svg': + case 'webp': + case 'ico': + return { icon: ImageIcon, className: 'text-purple-500' } + case 'html': + case 'htm': + return { icon: HtmlIcon } + case 'md': + case 'markdown': + return { icon: MarkdownIcon } + case 'yaml': + case 'yml': + return { icon: YamlIcon } + default: + return { icon: File, className: 'text-tertiary' } + } +} diff --git a/frontend/src/lib/components/icons/index.ts b/frontend/src/lib/components/icons/index.ts index 56d77c810c..b39df4310e 100644 --- a/frontend/src/lib/components/icons/index.ts +++ b/frontend/src/lib/components/icons/index.ts @@ -26,6 +26,7 @@ import SendflakeIcon from './SendflakeIcon.svelte' import QRCodeIcon from './QRCodeIcon.svelte' import LinkedinIcon from './LinkedinIcon.svelte' import HubspotIcon from './HubspotIcon.svelte' +import CoupaIcon from './CoupaIcon.svelte' import DatadogIcon from './DatadogIcon.svelte' import DatabricksIcon from './DatabricksIcon.svelte' import AdobeAcrobatSignIcon from './AdobeAcrobatSignIcon.svelte' @@ -34,6 +35,7 @@ import TelegramIcon from './TelegramIcon.svelte' import FunkwhaleIcon from './FunkwhaleIcon.svelte' import GdocsIcon from './GdocsIcon.svelte' import NextcloudIcon from './NextcloudIcon.svelte' +import NetsuiteIcon from './NetsuiteIcon.svelte' import FaunadbIcon from './FaunadbIcon.svelte' import ClickhouseIcon from './ClickhouseIcon.svelte' import OpenaiIcon from './OpenaiIcon.svelte' @@ -70,6 +72,7 @@ import GraphqlIcon from './GraphqlIcon.svelte' import NocoDbIcon from './NocoDbIcon.svelte' import AzureIcon from './AzureIcon.svelte' import OktaIcon from './OktaIcon.svelte' +import OutreachIcon from './OutreachIcon.svelte' import Auth0Icon from './Auth0Icon.svelte' import MsSqlServerIcon from './MSSqlServerIcon.svelte' import AuthentikIcon from './AuthentikIcon.svelte' @@ -182,6 +185,7 @@ import TwitchIcon from './TwitchIcon.svelte' import TwitterIcon from './TwitterIcon.svelte' import VercelIcon from './VercelIcon.svelte' import WebflowIcon from './WebflowIcon.svelte' +import WhatsappBusinessIcon from './WhatsappBusinessIcon.svelte' import WooCommerceIcon from './WooCommerceIcon.svelte' import WordpressIcon from './WordpressIcon.svelte' import XataIcon from './XataIcon.svelte' @@ -248,6 +252,8 @@ export const APP_TO_ICON_COMPONENT = { hubspot: HubspotIcon, datadog: DatadogIcon, databricks: DatabricksIcon, + netsuite: NetsuiteIcon, + coupa: CoupaIcon, adobe_acrobat_sign: AdobeAcrobatSignIcon, stripe: StripeIcon, telegram: TelegramIcon, @@ -293,6 +299,7 @@ export const APP_TO_ICON_COMPONENT = { nocodb: NocoDbIcon, azure: AzureIcon, okta: OktaIcon, + outreach: OutreachIcon, auth0: Auth0Icon, authentik: AuthentikIcon, authelia: AutheliaIcon, @@ -407,6 +414,7 @@ export const APP_TO_ICON_COMPONENT = { twitter: TwitterIcon, vercel: VercelIcon, webflow: WebflowIcon, + whatsapp_business: WhatsappBusinessIcon, woocommerce: WooCommerceIcon, wordpress: WordpressIcon, xata: XataIcon, @@ -512,6 +520,7 @@ export { AzureIcon, MicrosoftIcon, OktaIcon, + OutreachIcon, Auth0Icon, AuthentikIcon, AutheliaIcon, diff --git a/frontend/src/lib/components/instanceSettings.ts b/frontend/src/lib/components/instanceSettings.ts index 6af6926c4f..81a7b841b7 100644 --- a/frontend/src/lib/components/instanceSettings.ts +++ b/frontend/src/lib/components/instanceSettings.ts @@ -443,7 +443,7 @@ export const settings: Record = { { label: 'Store audit logs in object storage', description: - 'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Pre-existing history is not backfilled: export starts from when the setting is enabled (transactions in flight at that moment may include a bounded set of just-prior rows). No audit log committed after enabling is ever skipped.', + 'When enabled and instance object storage is configured, audit logs are also exported as newline-delimited JSON to the dedicated logs/audit/ folder (partitioned by day). Export is incremental and runs off the hot path. Enabling (or re-enabling) anchors the export at ~now: while it stays enabled, every audit log committed from that point on is exported (transactions in flight at the moment of enabling may include a bounded set of just-prior rows). Pre-existing history, and any window during which export was disabled, are NOT exported by this cursor — use the opt-in backfill API to export a chosen historical range, back to when audit-log partitioning was introduced (older rows in the legacy audit table are not exported, and a window overlapping them is rejected): POST /settings/audit_logs_s3_backfill {from, to} (status at GET /settings/audit_logs_s3_backfill_status).', key: 'store_audit_logs_s3', fieldType: 'boolean', storage: 'setting', @@ -523,6 +523,14 @@ export const settings: Record = { fieldType: 'smtp_connect', storage: 'setting', ee_only: '' + }, + { + label: 'Disable workspace invite emails', + description: + 'Do not send email notifications when a user is invited or added to a workspace. Useful for automated workflows that add users programmatically.', + key: 'disable_workspace_invite_emails', + fieldType: 'boolean', + storage: 'setting' } ], 'Auth/OAuth/SAML': [ diff --git a/frontend/src/lib/components/meltComponents/Menu.svelte b/frontend/src/lib/components/meltComponents/Menu.svelte index 879df7bc2c..e97a3c4c15 100644 --- a/frontend/src/lib/components/meltComponents/Menu.svelte +++ b/frontend/src/lib/components/meltComponents/Menu.svelte @@ -24,6 +24,11 @@ menuClass?: string open?: boolean renderContent?: boolean + // Move the scroll/overflow onto an inner wrapper instead of the melt element. The + // melt element is the fixed-positioned containing block for any submenu, so overflow + // on it clips submenus that open to the side. Opt in only when using a submenu — the + // default keeps the existing single-element markup untouched for every other menu. + submenuSafe?: boolean classNames?: string triggr?: import('svelte').Snippet<[any]> children?: import('svelte').Snippet<[any]> @@ -42,6 +47,7 @@ menuClass = '', open = $bindable(false), renderContent = false, + submenuSafe = false, class: classNames = '', triggr, children @@ -60,6 +66,7 @@ //Melt const { elements: { trigger, menu: menuElement, item }, + builders, states } = menu @@ -106,15 +113,21 @@ use:melt={$menuElement} data-menu class={twMerge( - 'z-[6000] border w-56 origin-top-right rounded-md shadow-md focus:outline-none overflow-y-auto', + 'z-[6000] border w-56 origin-top-right rounded-md shadow-md focus:outline-none', + // Default: scroll on the melt element. submenuSafe moves it to the inner + // wrapper so a side-opening submenu isn't clipped by this element's overflow. + submenuSafe ? '' : 'overflow-y-auto', lightMode ? 'bg-surface-inverse' : 'bg-surface', invisible ? 'opacity-0' : '', menuClass )} onclick={bubble('click')} > -
- {@render children?.({ item, open })} +
+ {@render children?.({ item, open, builders })}
{/if} diff --git a/frontend/src/lib/components/meltComponents/MenuItemWrapper.svelte b/frontend/src/lib/components/meltComponents/MenuItemWrapper.svelte new file mode 100644 index 0000000000..02557abb71 --- /dev/null +++ b/frontend/src/lib/components/meltComponents/MenuItemWrapper.svelte @@ -0,0 +1,28 @@ + + + diff --git a/frontend/src/lib/components/otherUserDraftDiff.ts b/frontend/src/lib/components/otherUserDraftDiff.ts new file mode 100644 index 0000000000..f462a3bc83 --- /dev/null +++ b/frontend/src/lib/components/otherUserDraftDiff.ts @@ -0,0 +1,47 @@ +import { AppService, FlowService, ScriptService, type UserDraftItemKind } from '$lib/gen' +import type { Value } from '$lib/utils' +import { DEFAULT_DATA, extractDataConfig } from '$lib/components/raw_apps/dataTableRefUtils' + +/** + * Fetch the currently-deployed value for an item, in the same shape its draft + * is stored — so it can be the "original" side of a draft-vs-deployed diff. + * App drafts hold the bare `App` value (unwrap `.value`); raw-app drafts hold a + * flat bundle (`files`/`runnables`/`data`/`summary`/`policy`/`custom_path`), + * with `summary`/`policy`/`custom_path` living OUTSIDE `.value` on the deployed + * row — so we project the deployed app into that bundle (via the same + * `extractDataConfig` the editor uses) instead of diffing mismatched shapes. + * Only the cross-user-visible kinds have other-user drafts to diff. + */ +export async function fetchDeployedValueForDiff( + workspace: string, + itemKind: UserDraftItemKind, + path: string +): Promise { + switch (itemKind) { + case 'script': + return (await ScriptService.getScriptByPath({ + workspace, + path, + getDraft: false + })) as unknown as Value + case 'flow': + return (await FlowService.getFlowByPath({ workspace, path })) as unknown as Value + case 'app': + return (await AppService.getAppByPath({ workspace, path, getDraft: false })) + .value as unknown as Value + case 'raw_app': { + const app = await AppService.getAppByPath({ workspace, path, getDraft: false }) + const v = (app.value ?? {}) as any + return { + files: v.files, + runnables: v.runnables, + data: extractDataConfig(v) ?? { ...DEFAULT_DATA }, + summary: app.summary, + policy: app.policy, + custom_path: app.custom_path + } as unknown as Value + } + default: + throw new Error(`Cannot diff drafts of kind ${itemKind}`) + } +} diff --git a/frontend/src/lib/components/otherUserDraftLoad.svelte.ts b/frontend/src/lib/components/otherUserDraftLoad.svelte.ts new file mode 100644 index 0000000000..7ba5cde6fb --- /dev/null +++ b/frontend/src/lib/components/otherUserDraftLoad.svelte.ts @@ -0,0 +1,176 @@ +import { SvelteMap, SvelteSet } from 'svelte/reactivity' +import { goto } from '$lib/navigation' +import { base } from '$app/paths' +import { UserDraft, draftValuesEqual, type UserDraftItemKind } from '$lib/userDraft.svelte' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' + +/** + * Coordinates "Load another user's draft into the editor as if it were ours". + * + * Two cases, decided by the route once it knows whether WE already have a draft + * at this path (`is_draft` for us): + * - No own draft → the loaded value just becomes our draft (normal autosave). + * - Own draft → "overlay" mode: the loaded value is shown but NEVER saved + * (so our own draft on the server is untouched). The first edit prompts + * "overwrite your current draft?". Confirm persists the edited value as our + * draft; Reset restores our own draft. + * + * The save block is a hard per-key lock in the syncer (see `lockSync`) — it + * covers the reactive mirror AND the navigation / tab-death flush paths, so the + * foreign value can't leak onto the server through any route. + */ + +function keyOf(workspace: string, itemKind: UserDraftItemKind, path: string): string { + return `${workspace}/${itemKind}/${path}` +} + +export type PendingOtherUserDraftLoad = { + workspace: string + itemKind: UserDraftItemKind + path: string + value: unknown + ownerLabel: string +} + +type ActiveSession = { + workspace: string + itemKind: UserDraftItemKind + path: string + ownerLabel: string + /** The loaded value the editor shows. A blocked save whose value still + * equals this is a programmatic load-cascade write, not an edit — so the + * overwrite prompt fires only once the value actually diverges. */ + loadedValue: unknown + /** Reload our own draft into the editor (AutosaveIndicator's "Reset to draft"). */ + onResetToOwnDraft: () => void | Promise +} + +// One-shot handoff: set by the Load action, consumed by the editor's loader. +const pending = new SvelteMap() +// Live overlay sessions, keyed by (workspace, itemKind, path). +const active = new SvelteMap() +// Keys whose overwrite-confirmation modal is currently open. +const overwriteOpen = new SvelteSet() + +export function editRouteFor(itemKind: UserDraftItemKind, path: string): string { + switch (itemKind) { + case 'script': + return `${base}/scripts/edit/${path}` + case 'flow': + return `${base}/flows/edit/${path}` + case 'app': + return `${base}/apps/edit/${path}` + case 'raw_app': + return `${base}/apps_raw/edit/${path}` + default: + throw new Error(`Cannot load drafts of kind ${itemKind}`) + } +} + +export const OtherUserDraftLoad = { + /** + * Stage a load. The editor's loader picks it up via `takePending`. When + * `navigate`, route to the item's edit page (home-page entry point); the + * in-editor entry point reloads in place instead. + */ + stage( + workspace: string, + itemKind: UserDraftItemKind, + value: unknown, + path: string, + ownerLabel: string, + opts: { navigate: boolean } + ): void { + pending.set(keyOf(workspace, itemKind, path), { workspace, itemKind, path, value, ownerLabel }) + if (opts.navigate) goto(editRouteFor(itemKind, path)) + }, + + /** Consume the staged load for this key (returns + removes it). */ + takePending( + workspace: string, + itemKind: UserDraftItemKind, + path: string + ): PendingOtherUserDraftLoad | undefined { + const k = keyOf(workspace, itemKind, path) + const v = pending.get(k) + if (v) pending.delete(k) + return v + }, + + /** + * Enter overlay mode: lock all server saves for this key and remember how + * to restore our own draft. A blocked save opens the overwrite modal ONLY + * once the value diverges from `loadedValue` — so the programmatic load + * cascade (which writes back the same value) never trips the prompt, while + * the user's very first real edit does, with no timing window. + */ + beginOverlay(session: ActiveSession): void { + const k = keyOf(session.workspace, session.itemKind, session.path) + active.set(k, session) + UserDraftDbSyncer.lockSync( + { workspace: session.workspace, itemKind: session.itemKind, path: session.path }, + () => { + const current = UserDraft.get(session.itemKind, session.path, { + workspace: session.workspace + }) + if (current !== undefined && !draftValuesEqual(current, session.loadedValue)) { + this.requestOverwriteModal(session.workspace, session.itemKind, session.path) + } + } + ) + }, + + isActive(workspace: string, itemKind: UserDraftItemKind, path: string): boolean { + return active.has(keyOf(workspace, itemKind, path)) + }, + + getSession( + workspace: string, + itemKind: UserDraftItemKind, + path: string + ): ActiveSession | undefined { + return active.get(keyOf(workspace, itemKind, path)) + }, + + requestOverwriteModal(workspace: string, itemKind: UserDraftItemKind, path: string): void { + const k = keyOf(workspace, itemKind, path) + if (active.has(k)) overwriteOpen.add(k) + }, + + isOverwriteModalOpen(workspace: string, itemKind: UserDraftItemKind, path: string): boolean { + return overwriteOpen.has(keyOf(workspace, itemKind, path)) + }, + + /** Cancel: keep editing the loaded value, stay paused; re-prompt on the next edit. */ + dismissOverwriteModal(workspace: string, itemKind: UserDraftItemKind, path: string): void { + overwriteOpen.delete(keyOf(workspace, itemKind, path)) + }, + + /** Confirm: adopt the current (edited) value as our own draft and resume saving. */ + confirmOverwrite(workspace: string, itemKind: UserDraftItemKind, path: string): void { + const current = UserDraft.get(itemKind, path, { workspace }) + this.clear(workspace, itemKind, path) + if (current !== undefined) { + void UserDraftDbSyncer.save({ workspace, itemKind, path, value: current, immediate: true }) + } + }, + + /** Discard the loaded view and restore our own draft. */ + async resetToOwnDraft( + workspace: string, + itemKind: UserDraftItemKind, + path: string + ): Promise { + const session = active.get(keyOf(workspace, itemKind, path)) + this.clear(workspace, itemKind, path) + await session?.onResetToOwnDraft() + }, + + /** Exit overlay mode: unlock saves, drop the session, close the modal. */ + clear(workspace: string, itemKind: UserDraftItemKind, path: string): void { + const k = keyOf(workspace, itemKind, path) + active.delete(k) + overwriteOpen.delete(k) + UserDraftDbSyncer.unlockSync({ workspace, itemKind, path }) + } +} diff --git a/frontend/src/lib/components/raw_apps/FileIcon.svelte b/frontend/src/lib/components/raw_apps/FileIcon.svelte new file mode 100644 index 0000000000..10ff3f4868 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/FileIcon.svelte @@ -0,0 +1,88 @@ + + + +{#if spec} + {@const Icon = spec.icon} + +{/if} diff --git a/frontend/src/lib/components/raw_apps/FileTreeNode.svelte b/frontend/src/lib/components/raw_apps/FileTreeNode.svelte index 9819ed8796..3d428b9aaf 100644 --- a/frontend/src/lib/components/raw_apps/FileTreeNode.svelte +++ b/frontend/src/lib/components/raw_apps/FileTreeNode.svelte @@ -1,32 +1,11 @@
diff --git a/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte b/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte index 73abdf6c34..6cdcd6a427 100644 --- a/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppBackgroundRunner.svelte @@ -15,6 +15,26 @@ jobsById?: Record editor: boolean workspace: string + /** + * Restrict waitJob/getJob/streamJob to job ids launched by this app + * instance (WIN-2006): a SANDBOXED bundle must not read arbitrary + * workspace jobs through the credentialed bridge. Off for unsandboxed + * renders (the default, and editor preview) — there the bundle holds + * the same credential as the bridge, so gating adds nothing and would + * only break unsandboxed apps that poll persisted or runnable-returned + * job ids. + */ + gateJobIds?: boolean + /** + * Additional trusted message source beyond the bundle iframe: the + * detached preview window opened from the editor ("open preview in a + * separate window"). Its app bundle posts runnable requests to + * `window.opener` (this window), so the bridge must accept its + * `event.source` and reply to it. A getter so it tracks the live handle + * without a reactive prop. Editor-only — the detached window runs the + * same unsandboxed bundle as the inline preview. + */ + extraSourceWindow?: () => Window | null | undefined } let { @@ -24,16 +44,31 @@ jobs = $bindable([]), jobsById = $bindable({}), editor, - workspace + workspace, + gateJobIds = true, + extraSourceWindow }: Props = $props() + // Job ids launched by this app instance — see `gateJobIds`. + const launchedJobs = new Set() + let listener = async (event) => { - if (!iframe || event.source !== iframe.contentWindow) return + // Only accept messages from the bundle iframe (opaque origin) or the + // detached preview window we opened, so other frames/extensions can't + // drive the runnable bridge (WIN-2006). Reject unconditionally until the + // iframe is bound — never process a message from an unknown source. + const detachedWindow = extraSourceWindow?.() + const sourceWindow = event.source as Window | null + if (!iframe || !sourceWindow) return + if (sourceWindow !== iframe.contentWindow && sourceWindow !== detachedWindow) return const data = event.data + // Reply to whichever window sent the request (inline iframe or the + // detached preview), not a hardcoded target — otherwise the detached + // window's calls would hang waiting for a response routed elsewhere. function respond(o: object) { - iframe?.contentWindow?.postMessage({ type: data.type + 'Res', ...o, reqId: data.reqId }, '*') + sourceWindow?.postMessage({ type: data.type + 'Res', ...o, reqId: data.reqId }, '*') } async function respondWithResult(uuid: string) { let error = false @@ -115,6 +150,7 @@ }, undefined ) + launchedJobs.add(uuid) let job: JobById = { component: runnable_id, created_at: Date.now(), job: uuid } if (event.data.type == 'backendAsync') { let result = uuid @@ -134,14 +170,29 @@ console.error('No runnable found for', runnable_id) } } else if (event.data.type == 'waitJob') { + if (gateJobIds && !launchedJobs.has(data.jobId)) { + respond({ result: { message: 'Unknown job' }, error: true }) + return + } await respondWithResult(data.jobId) } else if (event.data.type == 'getJob') { + if (gateJobIds && !launchedJobs.has(data.jobId)) { + respond({ result: { message: 'Unknown job' }, error: true }) + return + } const job = await JobService.getJob({ workspace, id: data.jobId }) respond({ result: job }) } else if (event.data.type == 'streamJob') { // Stream job results using SSE const jobId = data.jobId const reqId = data.reqId + if (gateJobIds && !launchedJobs.has(jobId)) { + sourceWindow?.postMessage( + { type: 'streamJobRes', reqId, error: true, result: { message: 'Unknown job' } }, + '*' + ) + return + } const params = new URLSearchParams() params.set('fast', 'true') params.set('only_result', 'true') @@ -163,7 +214,7 @@ if (type === 'error') { eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, @@ -177,7 +228,7 @@ if (type === 'not_found') { eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, @@ -191,7 +242,7 @@ // Send stream update if there's new stream data if (update.new_result_stream !== undefined) { - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobUpdate', reqId, @@ -205,7 +256,7 @@ // Check if job is completed if (update.completed) { eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, @@ -223,7 +274,7 @@ eventSource.onerror = (error) => { console.warn('SSE stream error:', error) eventSource.close() - iframe?.contentWindow?.postMessage( + sourceWindow?.postMessage( { type: 'streamJobRes', reqId, diff --git a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte index b62e047e84..d93da9a46c 100644 --- a/frontend/src/lib/components/raw_apps/RawAppEditor.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppEditor.svelte @@ -29,7 +29,7 @@ import type { Modules } from './RawAppModules.svelte' import { isRunnableByName, isRunnableByPath } from '../apps/inputType' import { aiChatManager, AIMode } from '../copilot/chat/AIChatManager.svelte' - import { onMount, untrack } from 'svelte' + import { onMount, onDestroy, untrack } from 'svelte' import type { AppDatatableMetadata, LintResult, @@ -38,11 +38,19 @@ import { createAppSelectedContext, type AppCodeSelectionElement } from '../copilot/chat/context' import { rawAppLintStore } from './lintStore' import { dbSchemas } from '$lib/stores' - import { MousePointerSquareDashed, RefreshCw, Columns2, ChevronDown, Eye } from 'lucide-svelte' + import { + MousePointerSquareDashed, + RefreshCw, + Columns2, + ChevronDown, + Eye, + SquareArrowOutUpRight + } from 'lucide-svelte' import DraggableTabs, { type TabItem } from '$lib/components/common/tabs/DraggableTabs.svelte' import { runScriptAndPollResult } from '../jobs/utils' import { RawAppHistoryManager } from './RawAppHistoryManager.svelte' import { sendUserToast } from '$lib/utils' + import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { buildDataTableWhitelist, parseDataTableRef, @@ -63,6 +71,8 @@ summary?: string path: string newPath?: string | undefined + /** Initial labels for the app, threaded from the loaded app data. */ + labels?: string[] savedApp?: | { value: any @@ -71,16 +81,16 @@ summary: string policy: any draft_only?: boolean + /** No deployed counterpart exists (draft-only); disables Diff. */ + no_deployed?: boolean custom_path?: string + labels?: string[] } | undefined diffDrawer?: DiffDrawer | undefined onNavigate?: (item: import('$lib/components/workspacePicker').WorkspaceItem) => void /** Fired after a successful deploy; the session preview reloads on it. */ onDeploy?: (e: { path: string }) => void - /** Fired after a successful server-draft save; the session refreshes its - * draft-bar count on it (parity with the script/flow editors). */ - onSaveDraft?: (e: { path: string }) => void /** Initial collapsed state for the file/runnable sidebar. The user's * toggled preference is persisted under `sidebarStorageKey`; this prop * only seeds the very first open. */ @@ -90,6 +100,11 @@ * preference. */ sidebarStorageKey?: string liveEditorDraftStoragePath?: string + /** Indicator-only overrides forwarded to RawAppEditorHeader so the + * sessions preview's AutosaveIndicator watches the session's + * (workspace, path). Undefined on the full-page editor. */ + autosaveWorkspace?: string + autosavePath?: string /** Initial value for the "Split with Preview" tab-bar toggle. Defaults * to `true` (split mode, preview always pinned to the right). Set * `false` when the editor mounts inside a context that wants single- @@ -98,8 +113,25 @@ * still toggle the mode after mount; this prop only seeds the * initial state. */ defaultSplitWithPreview?: boolean + /** User-typed path when it differs from `savedApp.path`. The route injects + * it as `draft_path` so the home row shows the friendly name, not `draft_{uuid}`. */ + pendingDraftPath?: string | undefined + // Threaded to the AutosaveIndicator's "Reset to deployed" button. + onResetToDeployed?: () => void | Promise + // See ScriptBuilderProps — same indicator semantics. + loadedFromDraft?: boolean + othersDraftsCount?: number + onOpenOthersDrafts?: () => void onRuntimeLogRequester?: (requester: RawAppRuntimeLogRequester | undefined) => void onRunsProvider?: (provider: RawAppRunsProvider | undefined) => void + // Restoring an older deployment from the history drawer. A callback prop + // (not `on:restore` forwarding): forwarding a `createEventDispatcher` + // event up through these runes-mode components silently drops it. + onRestore?: (restoredApp: any) => void + // Deploy created the app at a new path; the page navigates to it. Callback + // prop for the same reason as `onRestore` — `on:savedNewAppPath` forwarding + // through these runes-mode components is dropped. + onSavedNewAppPath?: (path: string) => void } let { @@ -111,17 +143,26 @@ summary = $bindable(''), path, newPath = undefined, + labels = undefined, savedApp = $bindable(undefined), diffDrawer = undefined, onNavigate, onDeploy = undefined, - onSaveDraft = undefined, defaultSidebarCollapsed = false, sidebarStorageKey = 'raw-app-sidebar-collapsed', liveEditorDraftStoragePath = undefined, + autosaveWorkspace = undefined, + autosavePath = undefined, defaultSplitWithPreview = true, + pendingDraftPath = $bindable(undefined), + onResetToDeployed, + loadedFromDraft = false, + othersDraftsCount = 0, + onOpenOthersDrafts, onRuntimeLogRequester = undefined, - onRunsProvider = undefined + onRunsProvider = undefined, + onRestore, + onSavedNewAppPath }: Props = $props() export const version: number | undefined = undefined @@ -220,6 +261,10 @@ let previewIframe: HTMLIFrameElement | undefined = $state(undefined) let previewIframeLoaded = $state(false) let lastBuild: { css: string; js: string } | undefined = undefined + // Detached preview tab/window rendering the same app-preview bundle as the + // inline pane. Kept live-synced: every build is replayed into it until the + // user closes it. Not reactive — it's a window handle, not UI state. + let externalPreviewWindow: Window | null = null let inspectorEnabled = $state(false) let bundlerType: 'esbuild' | 'rolldown' = $state('esbuild') @@ -962,6 +1007,22 @@ } function listener(e: MessageEvent) { + // The detached preview window asks for the build every time it (re)loads, + // including a manual browser refresh — its app-preview.html shell starts + // blank and the one-shot `load` feed can't survive the tab reloading + // itself. Re-feed it here so it repaints. Gated to our own window handle + // AND a same-origin sender: the preview runs user app code that can + // navigate the window away, and a cross-origin doc must not be able to + // trigger a bundle replay (the build can carry app source/secrets). + if ( + e.data?.type === 'appPreviewReady' && + e.source === externalPreviewWindow && + e.origin === window.location.origin + ) { + feedExternalPreview() + return + } + // Two children speak to us now: the UI Builder iframe (source editor) // and the preview iframe (rendered user app). Gate by source so they // can't be confused or spoofed. @@ -977,6 +1038,7 @@ { type: 'preview', css: e.data.css, js: e.data.js }, '*' ) + syncExternalPreview() return } @@ -1077,6 +1139,67 @@ } } + function postToExternalPreview(msg: Record) { + if (!externalPreviewWindow || externalPreviewWindow.closed) { + externalPreviewWindow = null + return + } + // Restrict to our own origin: the detached window loads same-origin + // app-preview.html, but user app code can navigate it elsewhere — don't + // post the build (potential app source/secrets) to a cross-origin doc. + externalPreviewWindow.postMessage(msg, window.location.origin) + } + + function syncExternalPreview() { + if (lastBuild) { + postToExternalPreview({ type: 'preview', css: lastBuild.css, js: lastBuild.js }) + } + } + + // Full (re)feed of the detached window: theme first, then the build. Used + // when (re)attaching to a window — open, focus-reuse, load, handshake — so + // it always matches the editor's current state. Plain rebuilds use + // `syncExternalPreview` alone (the theme hasn't changed). + function feedExternalPreview() { + postToExternalPreview({ type: 'setDarkMode', dark: darkMode }) + syncExternalPreview() + } + + onDestroy(() => { + // Don't leave a detached preview behind when the editor unmounts: it + // would stop receiving builds and, once refreshed, has no opener to + // re-feed it — a permanently blank orphan. + if (externalPreviewWindow && !externalPreviewWindow.closed) externalPreviewWindow.close() + externalPreviewWindow = null + }) + + function openExternalPreview() { + // Reuse an already-open window instead of spawning duplicates. + if (externalPreviewWindow && !externalPreviewWindow.closed) { + externalPreviewWindow.focus() + feedExternalPreview() + return + } + // Scope the window name per app path so two open editors don't fight over + // (or take over / close) one shared OS-level preview window. + const win = window.open( + '/ui_builder/app-preview.html', + `windmillRawAppPreview:${encodeURIComponent(path)}` + ) + if (!win) { + sendUserToast('Could not open the preview window (popup blocked?)', true) + return + } + externalPreviewWindow = win + // Initial feed: fires once when the freshly opened tab loads. This is the + // only feed path against an app-preview.html that predates the + // `appPreviewReady` handshake, so the window isn't blank on first open + // regardless of the pinned UI Builder artifact. A manual refresh is + // covered separately by the handshake in `listener` (this listener is + // bound to the now-stale document and won't fire again). + win.addEventListener('load', () => feedExternalPreview()) + } + let getBundleResolve: (({ css, js }: { css: string; js: string }) => void) | undefined = undefined async function getBundle(): Promise<{ css: string; js: string }> { @@ -1206,6 +1329,7 @@ if (previewIframe && previewIframeLoaded) { previewIframe.contentWindow?.postMessage({ type: 'setDarkMode', dark: darkMode }, '*') } + postToExternalPreview({ type: 'setDarkMode', dark: darkMode }) }) $effect(() => { // Match VS Code's editor font size to Windmill's text-xs. @@ -1396,7 +1520,51 @@ return () => window.removeEventListener('keydown', onEscapeCapture, true) }) + // Force an immediate flush. No toast — the AutosaveIndicator narrates the + // result, and `flush` never rejects (postSave routes errors to the failures map). + function flushDraft() { + if (!$workspaceStore || !liveEditorDraftStoragePath) return + void UserDraftDbSyncer.flush({ + workspace: $workspaceStore, + itemKind: 'raw_app', + path: liveEditorDraftStoragePath + }) + } + + // The VS Code workbench iframe's keydowns don't bubble out, so the window + // handler can't see Ctrl/Cmd+S while editing code. Attach a capture listener + // inside the iframe per load (it dies with the iframe, so no leak). No + // preventDefault: VS Code's own save still runs; we just flush alongside it. + function attachIframeSaveShortcut() { + const win = iframe?.contentWindow + if (!win) return + win.addEventListener( + 'keydown', + (e: KeyboardEvent) => { + if ((e.ctrlKey || e.metaKey) && !e.shiftKey && (e.key === 's' || e.key === 'S')) { + flushDraft() + } + }, + true + ) + } + + // Monaco swallows Ctrl/Cmd+S in inline editors; Editor/SimpleEditor + // re-broadcast it as `wm-monaco-save-shortcut` (untyped, hence manual listener). + $effect(() => { + window.addEventListener('wm-monaco-save-shortcut', flushDraft) + return () => window.removeEventListener('wm-monaco-save-shortcut', flushDraft) + }) + function handleKeydown(e: KeyboardEvent) { + // Ctrl/Cmd + S — catch this BEFORE the input/Monaco guard below so + // the shortcut fires regardless of focus. + if ((e.ctrlKey || e.metaKey) && !e.shiftKey && (e.key === 's' || e.key === 'S')) { + e.preventDefault() + flushDraft() + return + } + // Skip when typing in an input, textarea, or Monaco editor. const classes = (e.target as HTMLElement | null)?.className if ( @@ -1432,6 +1600,8 @@ bind:jobsById {runnables} {path} + gateJobIds={false} + extraSourceWindow={() => externalPreviewWindow} />
{/if}
@@ -1689,6 +1867,14 @@ > + -
- {/snippet} - - - -{/if} closeSaveDrawer()}> {#snippet actions()}
{#if $enterpriseLicense && appPath != ''} @@ -975,17 +814,34 @@ {/snippet} - + +
- {/if} +
+ {/if} + {#if guarded} +
+ + Large file — {lineCount.toLocaleString()} lines. + + +
+ {:else} + {#await import('$lib/components/DiffEditor.svelte')} +
+ {:then Module} +
+ +
+ {/await} + {/if} + diff --git a/frontend/src/lib/components/raw_apps/RawAppPreview.svelte b/frontend/src/lib/components/raw_apps/RawAppPreview.svelte index 296590a0e4..9a70b162d7 100644 --- a/frontend/src/lib/components/raw_apps/RawAppPreview.svelte +++ b/frontend/src/lib/components/raw_apps/RawAppPreview.svelte @@ -2,8 +2,8 @@ import { type UserExt } from '$lib/stores' import RawAppBackgroundRunner from './RawAppBackgroundRunner.svelte' import type { Runnable } from './rawAppPolicy' - import { htmlContent } from './utils' - import { onMount, untrack } from 'svelte' + import { getContext, onMount, untrack } from 'svelte' + import { unsandboxedRawAppHtml } from './utils' interface Props { workspace: string @@ -17,43 +17,190 @@ let iframe = $state() as HTMLIFrameElement | undefined - // Get initial hash from parent URL to pass to iframe - let initialHash = $state('') + // Get initial hash from parent URL to pass to the iframe + let initialHash = '' - onMount(() => { - initialHash = window.location.hash || '' + // WIN-2006: unless the publisher opted into sandbox isolation, run the bundle + // same-origin with full access (the default); otherwise the opaque-origin sandbox. + const unsandboxedCtx = getContext<{ value: boolean }>('IS_APP_UNSANDBOXED') + let unsandboxed = $derived(unsandboxedCtx?.value ?? false) + // Unsandboxed (the default) must match the pre-isolation viewer exactly: NO + // sandbox attribute (a same-origin blob with full session — an attribute would + // only break leftover features like unsandboxed popups for OAuth flows, while + // adding no isolation). The sandboxed path keeps the restrictive attribute; the + // wrapper document's `CSP: sandbox` response header enforces the opaque origin + // regardless. + let sandboxAttr = $derived( + unsandboxed + ? undefined + : 'allow-scripts allow-forms allow-popups allow-popups-to-escape-sandbox allow-downloads allow-modals allow-top-navigation' + ) + + // WIN-2006: source of the bundle iframe. + // - DEFAULT (isolated): a real API URL serving a sandboxed, opaque-origin + // document (`CSP: sandbox` response header + the iframe sandbox attribute), + // so a malicious bundle can never reach the authenticated Windmill origin + // (no cookie, no window.parent, no token). Root-relative so it resolves + // against the real host even when this component itself runs inside an opaque + // viewer (where `location.origin` is "null"). Context is handed over via + // postMessage — never baked into the document, never a credential. + // - UNSANDBOXED (the default — publisher did not opt into isolation): a + // client-built blob: wrapper (same-origin with the SPA) loaded with `allow-same-origin`, + // so relative `fetch('/api/...')` and the session cookie work. The backend + // `.html` is ALWAYS sandboxed, so we must build the same-origin wrapper here + // rather than relax a real-origin endpoint a victim could be linked to. + let iframeSrc = $derived.by(() => { + if (!secret || typeof window === 'undefined') return undefined + if (unsandboxed) { + // untrack(user) so userStore refreshes don't regenerate the blob URL and + // reload the iframe (losing state); ctx is only needed for initial render. + // Always pass the wrapper object — pre-sandbox bundles rely on + // `window.ctx.workspace` even for anonymous viewers (ctx.ctx undefined). + const u = untrack(() => user) + const html = unsandboxedRawAppHtml( + workspace, + secret, + { ctx: u, workspace }, + window.location.origin, + window.location.hash || '' + ) + return URL.createObjectURL(new Blob([html], { type: 'text/html' })) + } + // `wm_coep` (embed-in-cross-origin-isolated-page opt-in) must be propagated + // to the wrapper document: under a COEP `require-corp` embedder, a nested + // document is only allowed to load if it asserts COEP itself, so the + // backend adds the header when the flag is present. + const coep = new URLSearchParams(window.location.search).has('wm_coep') ? '?wm_coep=1' : '' + return `/api/w/${workspace}/apps_u/get_data/v/${secret}.html${coep}` }) - // Use blob URL instead of srcDoc to give the iframe a proper origin. - // srcDoc iframes have "null" origin which breaks URL constructor in routers. - // untrack(user) so that userStore refreshes don't regenerate the blob URL - // and cause the iframe to fully reload (losing all state). - // The user context is only needed for initial render. - let blobUrl = $derived.by(() => { - if (!secret) return undefined - const u = untrack(() => user) - const baseUrl = typeof window !== 'undefined' ? window.location.origin : '' - const html = htmlContent(workspace, secret, { ctx: u, workspace }, baseUrl, initialHash) - const blob = new Blob([html], { type: 'text/html' }) - return URL.createObjectURL(blob) - }) - - // Cleanup blob URL when it changes or component unmounts + // Revoke blob: URLs (unsandboxed path) when they change or on unmount. $effect(() => { - const url = blobUrl + const url = iframeSrc return () => { - if (url) URL.revokeObjectURL(url) + if (url && url.startsWith('blob:')) URL.revokeObjectURL(url) + } + }) + + // Persistence for the bundle's (opaque-origin) localStorage, backed by a store + // scoped PER APP (keyed by workspace + app path) so one sandboxed app can't read + // or clobber another's (even two apps at the same path in different workspaces). On a real origin (workspace viewer, public page — even when + // that page sits inside someone else's iframe) it reads/writes real localStorage + // directly. Only inside an opaque frame (the Windmill embed viewer), where Web + // Storage throws, does it relay per-key ops up to the embedder, the persistence + // authority. `framed` therefore probes storage rather than just `window.parent`: + // an externally-embedded public page is framed too, but its parent is not the + // Windmill embedder and would never answer the relay (leaving the bundle without + // ctx). The snapshot is handed to the bundle before it evaluates so its + // localStorage is hydrated synchronously. + const SHARED_LS_KEY = `wm_apps_localstorage:${workspace}:${path}` + function storageAccessible(): boolean { + try { + localStorage.getItem(SHARED_LS_KEY) + return true + } catch (_) { + return false + } + } + const framed = typeof window !== 'undefined' && window.parent !== window && !storageAccessible() + let bundleStorage: Record | undefined = undefined + let pendingReady = false + + function readDirect(): Record { + try { + return JSON.parse(localStorage.getItem(SHARED_LS_KEY) || '{}') + } catch (_) { + return {} + } + } + + function applyDirectOp(d: any) { + try { + const s = readDirect() + if (d.op === 'set') s[d.key] = String(d.value) + else if (d.op === 'remove') delete s[d.key] + else if (d.op === 'clear') for (const k in s) delete s[k] + localStorage.setItem(SHARED_LS_KEY, JSON.stringify(s)) + } catch (_) {} + } + + function respondCtx() { + iframe?.contentWindow?.postMessage( + { + type: 'windmill:ctx', + // Same shape as the unsandboxed wrapper: always the object, so + // `window.ctx.workspace` works for anonymous viewers too. + ctx: { ctx: user, workspace }, + initialHash, + storage: { local: bundleStorage ?? {}, session: {} } + }, + '*' + ) + } + + onMount(() => { + initialHash = window.location.hash || '' + if (framed) { + // Pre-fetch the shared store from the embedder. + try { + window.parent.postMessage({ type: 'wm_ls_req' }, '*') + } catch (_) {} + // If the parent never answers (it isn't the Windmill embedder, e.g. an + // opaque context created by a third party), don't hold the bundle's ctx + // hostage: proceed with empty storage. Must beat the backend wrapper's + // own 1.5s no-ctx fallback. + const fallback = setTimeout(() => { + if (bundleStorage === undefined) { + bundleStorage = {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + } + }, 750) + return () => clearTimeout(fallback) } }) - // Listen for hash changes from iframe and update parent URL $effect(() => { function handleMessage(event: MessageEvent) { - console.log('[Parent] Received message:', event.data) - if (event.data?.type === 'windmill:hashchange') { - const newHash = event.data.hash || '' - console.log('[Parent] Updating hash to:', newHash) - // Update parent URL without triggering navigation + const data = event.data + // Shared-store hydration from the embedder (public mode only). + if (framed && event.source === window.parent && data?.type === 'wm_ls_hydrate') { + bundleStorage = data.data || {} + if (pendingReady) { + pendingReady = false + respondCtx() + } + return + } + // Everything else must come from the bundle iframe. + if (event.source !== iframe?.contentWindow) return + if (data?.type === 'windmill:ready') { + // Hand the bundle its context + shared storage before it evaluates. + if (!framed) { + bundleStorage = readDirect() + respondCtx() + } else if (bundleStorage !== undefined) { + respondCtx() + } else { + pendingReady = true + } + } else if (data?.type === 'wm_ls_op') { + // The bundle mutated localStorage — apply it to the shared store. + if (!framed) { + applyDirectOp(data) + } else { + try { + window.parent.postMessage( + { type: 'wm_ls_op', op: data.op, key: data.key, value: data.value }, + '*' + ) + } catch (_) {} + } + } else if (data?.type === 'windmill:hashchange') { + // Keep the parent URL hash in sync for shareable URLs. + const newHash = data.hash || '' if (window.location.hash !== newHash) { history.replaceState(null, '', newHash || window.location.pathname) } @@ -65,13 +212,29 @@ }) - + -{#if blobUrl} +{#if iframeSrc} + + {/if} diff --git a/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte b/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte new file mode 100644 index 0000000000..308c554891 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/RawAppTemplatePicker.svelte @@ -0,0 +1,399 @@ + + +{#if open} + + +
+
+

Summary

+ +
+ +
+

Framework

+
+ {#each templates as t, i} + + {/each} +
+
+ +
+

Data configuration

+ + {#if hasNoDatatables} + + You can still create an app, but for data storage you won't be able to use data tables + which are highly recommended. +
+ {#if $userStore?.is_admin} + Configure datatables in + workspace settings + to enable this feature. + {:else} + Ask your workspace admin to configure datatables in workspace settings to enable this + feature. + {/if} +
+ {:else} +
+
+ Default settings for new tables +
+
+
+ + +
+ {/if} +
+ {#if newSchemaAlreadyExists} + Schema "{newSchemaName}" already exists + {/if} +
+
+
+
+ +
+ +
+ +
+ dataTableDrawer?.openDrawer()} + onRemove={(index) => { + preWhitelistedTables = preWhitelistedTables.filter((_, i) => i !== index) + }} + /> +
+
+ {/if} + + +
+

+ + Start with AI + (optional) +

+ + {#if !isAiEnabled} + + You can still create an app manually but using AI is highly recommended. +
+ {#if $userStore?.is_admin} + Configure AI in + workspace settings + to enable this feature. + {:else} + Ask your workspace admin to configure AI in workspace settings to enable this feature. + {/if} +
+ {:else} +
+ +

+ Leave empty to start with a blank template, or describe your app to get AI assistance + right away. +

+
+ {/if} +
+ +
+ + {#if isAiEnabled} + + {/if} +
+ +
+{/if} + + { + preWhitelistedTables = [...preWhitelistedTables, ref] + }} +/> diff --git a/frontend/src/lib/components/raw_apps/dataTableRefUtils.ts b/frontend/src/lib/components/raw_apps/dataTableRefUtils.ts index b6f8c636d3..dfbfe6e4cd 100644 --- a/frontend/src/lib/components/raw_apps/dataTableRefUtils.ts +++ b/frontend/src/lib/components/raw_apps/dataTableRefUtils.ts @@ -25,6 +25,30 @@ export const DEFAULT_DATA: RawAppData = { schema: undefined } +/** + * Normalize a raw-app value's data config to `RawAppData`, handling the old + * nested `creation` shape and the legacy top-level `datatables`. Shared by the + * editor loader and the deployed-vs-draft diff so both project identically. + * Returns `undefined` when the value carries no data config. + */ +export function extractDataConfig(value: any): RawAppData | undefined { + if (value?.data) { + const d = value.data + // Handle old nested creation format + if (d.creation) { + return { + tables: d.tables ?? [], + datatable: d.creation.datatable, + schema: d.creation.schema + } + } + return d + } else if (value?.datatables) { + return { ...DEFAULT_DATA, tables: value.datatables } + } + return undefined +} + export type DataTableWhitelist = { datatables: Set allTablesDatatables: Set @@ -76,7 +100,12 @@ export function isDatatableTableAllowed( return true } - return whitelist.tables.get(datatableName)?.get(schemaName ?? 'public')?.has(tableName) ?? false + return ( + whitelist.tables + .get(datatableName) + ?.get(schemaName ?? 'public') + ?.has(tableName) ?? false + ) } /** diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts new file mode 100644 index 0000000000..300b5dde1e --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.test.ts @@ -0,0 +1,299 @@ +import { describe, expect, it } from 'vitest' +import { + parseRawAppDiff, + rawAppDiffToItems, + RAW_APP_METADATA_PATH, + type RawAppDiffEntry +} from './rawAppDiffUtils' + +function byPath(entries: RawAppDiffEntry[], path: string): RawAppDiffEntry | undefined { + return entries.find((e) => e.path === path) +} + +describe('parseRawAppDiff — files', () => { + it('detects added, removed and modified files, omits unchanged', () => { + const original = { + files: { 'index.html': '

hi

', 'styles.css': 'body{}', 'gone.js': 'x' } + } + const current = { + files: { 'index.html': '

hello

', 'styles.css': 'body{}', 'new.ts': 'y' } + } + const entries = parseRawAppDiff(original, current) + const paths = entries.map((e) => e.path).sort() + // styles.css unchanged → omitted + expect(paths).toEqual(['gone.js', 'index.html', 'new.ts']) + + expect(byPath(entries, 'index.html')?.status).toBe('modified') + expect(byPath(entries, 'index.html')?.lang).toBe('html') + expect(byPath(entries, 'gone.js')?.status).toBe('removed') + expect(byPath(entries, 'new.ts')?.status).toBe('added') + expect(byPath(entries, 'new.ts')?.lang).toBe('typescript') + }) + + it('carries original/current content on the right sides', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': 'old', 'b.txt': 'keep' } }, + { files: { 'a.txt': 'new', 'b.txt': 'keep' } } + ) + const a = byPath(entries, 'a.txt')! + expect(a.original).toBe('old') + expect(a.current).toBe('new') + }) +}) + +describe('parseRawAppDiff — runnables', () => { + it('emits per-runnable leaves for add/remove/modify', () => { + const original = { + runnables: { a: { path: 'u/x/a' }, b: { path: 'u/x/b' }, same: { path: 'u/x/same' } } + } + const current = { + runnables: { a: { path: 'u/x/a2' }, c: { path: 'u/x/c' }, same: { path: 'u/x/same' } } + } + const entries = parseRawAppDiff(original, current) + expect(byPath(entries, 'runnables/a')?.status).toBe('modified') + expect(byPath(entries, 'runnables/a')?.lang).toBe('yaml') + expect(byPath(entries, 'runnables/b')?.status).toBe('removed') + expect(byPath(entries, 'runnables/c')?.status).toBe('added') + // identical runnable omitted + expect(byPath(entries, 'runnables/same')).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — metadata', () => { + it('collapses summary/data/policy/custom_path into one app.yaml leaf', () => { + const entries = parseRawAppDiff( + { summary: 'old summary', custom_path: 'foo' }, + { summary: 'new summary', custom_path: 'foo' } + ) + const meta = byPath(entries, RAW_APP_METADATA_PATH) + expect(meta?.status).toBe('modified') + expect(meta?.lang).toBe('yaml') + expect(meta?.original).toContain('old summary') + expect(meta?.current).toContain('new summary') + }) + + it('omits app.yaml when no metadata field changed', () => { + const entries = parseRawAppDiff( + { files: { 'a.txt': '1' }, summary: 's' }, + { files: { 'a.txt': '2' }, summary: 's' } + ) + expect(byPath(entries, RAW_APP_METADATA_PATH)).toBeUndefined() + expect(entries).toHaveLength(1) + }) +}) + +describe('parseRawAppDiff — whole app added / removed', () => { + it('marks everything added when the original side is absent', () => { + const current = { + files: { 'index.html': '

hi

' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'brand new' + } + const entries = parseRawAppDiff(undefined, current) + expect(entries.every((e) => e.status === 'added')).toBe(true) + expect(byPath(entries, 'index.html')?.original).toBeUndefined() + expect(byPath(entries, RAW_APP_METADATA_PATH)?.status).toBe('added') + }) + + it('marks everything removed when the current side is absent', () => { + const original = { + files: { 'index.html': '

hi

' }, + runnables: { a: { path: 'u/x/a' } }, + summary: 'going away' + } + const entries = parseRawAppDiff(original, undefined) + expect(entries.every((e) => e.status === 'removed')).toBe(true) + expect(byPath(entries, 'index.html')?.current).toBeUndefined() + }) +}) + +describe('parseRawAppDiff — collisions', () => { + it('disambiguates synthesized paths against real files', () => { + const original = { files: { 'app.yaml': 'real-old' }, summary: 'sa' } + const current = { files: { 'app.yaml': 'real-new' }, summary: 'sb' } + const entries = parseRawAppDiff(original, current) + // The real file keeps the natural path. + const realFile = byPath(entries, 'app.yaml')! + expect(realFile.original).toBe('real-old') + // The metadata leaf is pushed to a non-colliding path. + const meta = byPath(entries, 'app.yaml~2')! + expect(meta.original).toContain('sa') + expect(meta.current).toContain('sb') + // No two entries share a path. + const paths = entries.map((e) => e.path) + expect(new Set(paths).size).toBe(paths.length) + }) +}) + +describe('parseRawAppDiff — input shapes', () => { + // getItemValue returns the deployed app row: files/runnables/data live under + // `value`; summary/policy/custom_path at the top level. + it('reads files/runnables/data from the `value` wrapper (app-row shape)', () => { + const original = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

a

' }, runnables: {}, data: {} } + } + const current = { + summary: 'classy', + policy: { execution_mode: 'viewer' }, + value: { files: { 'index.html': '

b

' }, runnables: {}, data: {} } + } + const entries = parseRawAppDiff(original, current) + const file = byPath(entries, 'index.html') + expect(file?.status).toBe('modified') + expect(file?.original).toBe('

a

') + expect(file?.current).toBe('

b

') + }) + + it('still handles the flat draft shape (files at top level)', () => { + const entries = parseRawAppDiff({ files: { 'a.ts': 'x' } }, { files: { 'a.ts': 'y' } }) + expect(byPath(entries, 'a.ts')?.status).toBe('modified') + }) + + it('prefers `value` over a stray top-level files key', () => { + const entries = parseRawAppDiff( + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'a' } } }, + { files: { 'top.ts': 'ignored' }, value: { files: { 'real.ts': 'b' } } } + ) + expect(byPath(entries, 'real.ts')?.status).toBe('modified') + expect(byPath(entries, 'top.ts')).toBeUndefined() + }) +}) + +describe('rawAppDiffToItems', () => { + const appPath = 'u/admin/classy_app' + + it('produces composite-pathed items with status-derived exists flags', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'a', '/gone.ts': 'x' }, runnables: { r: { p: 1 } } } }, + { value: { files: { '/App.tsx': 'b', '/new.ts': 'y' }, runnables: {} } } + ) + const byPath = (p: string) => items.find((i) => i.path === p) + + const app = byPath('u/admin/classy_app/App.tsx')! + expect(app.kind).toBe('raw_app_file') + expect(app.status).toBe('modified') + expect(app.exists_in_source).toBe(true) + expect(app.exists_in_fork).toBe(true) + expect(app.appPath).toBe(appPath) + expect((app as any).lang).toBe('typescript') + + const gone = byPath('u/admin/classy_app/gone.ts')! + expect(gone.status).toBe('removed') + expect(gone.exists_in_source).toBe(true) + expect(gone.exists_in_fork).toBe(false) + + const added = byPath('u/admin/classy_app/new.ts')! + expect(added.status).toBe('added') + expect(added.exists_in_source).toBe(false) + expect(added.exists_in_fork).toBe(true) + + // Runnables render as script/flow rows, not file leaves. + const runnable = byPath('u/admin/classy_app/runnables/r')! + expect(runnable.kind).toBe('script') + expect(runnable.status).toBe('removed') + }) + + it('renders an inline-script runnable as a script row with code hoisted', () => { + const mk = (code: string) => ({ + value: { + files: {}, + runnables: { + a: { name: 'a', type: 'inline', inlineScript: { content: code, language: 'bun' } } + } + } + }) + const items = rawAppDiffToItems(appPath, mk('old code'), mk('new code')) + const r = items.find((i) => i.path === `${appPath}/runnables/a`) + expect(r?.kind).toBe('script') + expect(r?.status).toBe('modified') + // content + language hoisted to the top level for the script-style viewer + const cur = (r as any).currentRaw + expect(cur.content).toBe('new code') + expect(cur.language).toBe('bun') + expect(cur.inlineScript.content).toBeUndefined() + }) + + it('picks the flow kind for a runType:flow runnable', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: {}, runnables: {} } }, + { value: { files: {}, runnables: { f: { runType: 'flow', path: 'u/x/f' } } } } + ) + const r = items.find((i) => i.path === `${appPath}/runnables/f`) + expect(r?.kind).toBe('flow') + expect(r?.status).toBe('added') + }) + + it('flags the metadata item and attaches whole-app YAML for the expand view', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: {} } }, + { summary: 'new', value: { files: {} } } + ) + const meta = items.find((i) => i.path === `${appPath}/${RAW_APP_METADATA_PATH}`) as any + expect(meta.kind).toBe('raw_app_file') + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlOriginal).toContain('old') + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named app.yaml', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { 'app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { 'app.yaml': 'real-new' } } } + ) + // Real file keeps the natural path and is NOT the metadata item. + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + expect(realFile.isMetadata).toBe(false) + expect(realFile.fullYamlCurrent).toBeUndefined() + // Synthesized metadata moved to app.yaml~2 but still carries the flag + YAML. + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(meta.isMetadata).toBe(true) + expect(meta.fullYamlCurrent).toContain('new') + }) + + it('keeps the metadata flag on the right item when a real file is named /app.yaml (leading slash)', () => { + const items = rawAppDiffToItems( + appPath, + { summary: 'old', value: { files: { '/app.yaml': 'real-old' } } }, + { summary: 'new', value: { files: { '/app.yaml': 'real-new' } } } + ) + const realFile = items.find((i) => i.path === `${appPath}/app.yaml`) as any + const meta = items.find((i) => i.path === `${appPath}/app.yaml~2`) as any + expect(realFile.isMetadata).toBe(false) + expect(meta.isMetadata).toBe(true) + // distinct composite paths → distinct row keys + expect(realFile.path).not.toBe(meta.path) + }) + + it('treats a file keyed /App.tsx on one side and App.tsx on the other as one item', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/App.tsx': 'old' } } }, + { value: { files: { 'App.tsx': 'new' } } } + ) + const files = items.filter((i) => i.path === `${appPath}/App.tsx`) + expect(files).toHaveLength(1) + expect(files[0].status).toBe('modified') + expect((files[0] as any).original).toBe('old') + expect((files[0] as any).current).toBe('new') + }) + + it('dedups a runnable leaf against a real file named runnables/', () => { + const items = rawAppDiffToItems( + appPath, + { value: { files: { '/runnables/foo': 'old' }, runnables: { foo: { p: 1 } } } }, + { value: { files: { '/runnables/foo': 'new' }, runnables: { foo: { p: 2 } } } } + ) + const realFile = items.find((i) => i.kind === 'raw_app_file')! + const runnable = items.find((i) => i.kind === 'script')! + expect(realFile.path).toBe(`${appPath}/runnables/foo`) + // Reserved away from the real file's composite path (slash-normalized). + expect(runnable.path).toBe(`${appPath}/runnables/foo~2`) + expect(runnable.path).not.toBe(realFile.path) + }) +}) diff --git a/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts new file mode 100644 index 0000000000..444e03a0a3 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDiffUtils.ts @@ -0,0 +1,383 @@ +import { extToLang } from '$lib/editorLangUtils' +import { cleanValueProperties, orderedYamlStringify, replaceFalseWithUndefined } from '$lib/utils' + +// A raw app rendered as a *folder of files* for diffing. Each entry is one +// virtual file: real `files` keep their natural path, runnables become +// `runnables/`, and the remaining app metadata collapses into a single +// `app.yaml` leaf. Only changed entries are emitted (unchanged are omitted). +export type RawAppDiffStatus = 'added' | 'removed' | 'modified' + +export interface RawAppDiffEntry { + /** Tree path. Real file path, `runnables/`, or `app.yaml`. */ + path: string + status: RawAppDiffStatus + /** Original (parent) side content. Undefined when status is `added`. */ + original?: string + /** Current (fork) side content. Undefined when status is `removed`. */ + current?: string + /** Monaco language id for the per-file diff editor. */ + lang: string + /** True for the synthesized metadata leaf. Tracked as a flag (not by matching + * `path === 'app.yaml'`) so it survives `reserveUnique` moving the leaf to + * `app.yaml~2` when a real file is literally named `app.yaml`. */ + isMetadata?: boolean +} + +// Loose shape — diff inputs come from `getItemValue` / drafts and aren't +// strictly typed, and arrive in TWO shapes: +// - the deployed app row (getAppByPath): `{ summary, policy, custom_path?, +// value: { files, runnables, data } }` — files/runnables/data nested under +// `value`. +// - the flat draft/runtime shape (RawAppDraft / RuntimeRawApp): everything at +// the top level. +// `normalizeRawApp` coalesces both. Anything not present is empty/absent. +export interface RawAppish { + value?: Record + files?: Record + runnables?: Record + summary?: unknown + data?: unknown + policy?: unknown + custom_path?: unknown + [k: string]: unknown +} + +interface NormalizedRawApp { + files: Record + runnables: Record + summary: unknown + data: unknown + policy: unknown + custom_path: unknown +} + +export const RAW_APP_METADATA_PATH = 'app.yaml' +const RUNNABLES_PREFIX = 'runnables/' +const METADATA_FIELDS = ['summary', 'data', 'policy', 'custom_path'] as const + +// Real file keys may carry a leading slash (`/App.tsx`) which `joinAppPath` +// strips. Collision reservation must compare in the stripped space, else a real +// `/app.yaml` and the synthetic `app.yaml` leaf both become `/app.yaml`. +const stripLeadingSlash = (p: string) => p.replace(/^\/+/, '') + +function isObject(v: unknown): v is Record { + return !!v && typeof v === 'object' +} + +function asFileMap(f: unknown): Record { + if (!isObject(f)) return {} + const out: Record = {} + for (const [k, v] of Object.entries(f)) { + // Canonicalize the key (strip the leading slash `joinAppPath` would strip + // anyway) so the same file keyed `/App.tsx` on one side and `App.tsx` on the + // other is treated as one file — not two leaves colliding at the same + // composite path. Coerce non-string content so the diff editor gets a string. + out[stripLeadingSlash(k)] = typeof v === 'string' ? v : String(v ?? '') + } + return out +} + +// Coalesce the two raw-app shapes (app row with a `value` wrapper vs flat draft) +// into a canonical view. Returns undefined when the whole app is absent. +// +// Per-field precedence is deliberately asymmetric and mirrors the app-row shape: +// the editor payload (`files`/`runnables`/`data`) lives under `value`, so those +// prefer `value` first; the row-level metadata (`summary`/`policy`/`custom_path`) +// lives at the top level, so those prefer `raw` first. Each falls back to the +// other side so a flat draft (everything top-level) still normalizes correctly. +function normalizeRawApp(raw: RawAppish | undefined): NormalizedRawApp | undefined { + if (!isObject(raw)) return undefined + const value = isObject(raw.value) ? raw.value : undefined + return { + files: asFileMap(value?.files ?? raw.files), + runnables: isObject(value?.runnables ?? raw.runnables) + ? ((value?.runnables ?? raw.runnables) as Record) + : {}, + summary: raw.summary ?? value?.summary, + data: value?.data ?? raw.data, + policy: raw.policy ?? value?.policy, + custom_path: raw.custom_path ?? value?.custom_path + } +} + +// The serialized metadata blob for one side, or undefined when the whole app +// is absent (so the `app.yaml` leaf reads as added/removed). +function metadataYaml(app: NormalizedRawApp | undefined): string | undefined { + if (!app) return undefined + const meta: Record = {} + for (const field of METADATA_FIELDS) { + if (app[field] !== undefined) meta[field] = app[field] + } + return orderedYamlStringify(meta) +} + +function extOf(path: string): string { + const base = path.split('/').pop() ?? path + const dot = base.lastIndexOf('.') + return dot >= 0 ? base.slice(dot + 1).toLowerCase() : '' +} + +function diffStatus(o: string | undefined, c: string | undefined): RawAppDiffStatus | undefined { + if (o === undefined && c !== undefined) return 'added' + if (o !== undefined && c === undefined) return 'removed' + if (o !== c) return 'modified' + return undefined +} + +// Reserve a synthesized path, disambiguating against already-taken paths so a +// real file literally named `app.yaml` or under `runnables/` never collides +// with a synthesized leaf. +function reserveUnique(path: string, taken: Set): string { + if (!taken.has(path)) { + taken.add(path) + return path + } + let i = 2 + while (taken.has(`${path}~${i}`)) i++ + const p = `${path}~${i}` + taken.add(p) + return p +} + +/** + * Diff two raw-app objects into a flat list of changed virtual files. Either + * side may be undefined (whole app added or removed). Consumers build a tree + * from the returned paths with `buildFileTree`. + */ +export function parseRawAppDiff( + original: RawAppish | undefined, + current: RawAppish | undefined, + opts: { includeRunnables?: boolean } = {} +): RawAppDiffEntry[] { + const { includeRunnables = true } = opts + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oFiles = oApp?.files ?? {} + const cFiles = cApp?.files ?? {} + const oRunnables = oApp?.runnables ?? {} + const cRunnables = cApp?.runnables ?? {} + + // Reserve every real file path (changed or not) up front so synthesized + // runnable/metadata paths can dodge collisions — slash-normalized so a real + // `/app.yaml` or `/runnables/x` is seen as colliding with the synthetic leaf. + const taken = new Set( + [...Object.keys(oFiles), ...Object.keys(cFiles)].map(stripLeadingSlash) + ) + + const entries: RawAppDiffEntry[] = [] + + // Real files. + for (const path of [...new Set([...Object.keys(oFiles), ...Object.keys(cFiles)])].sort()) { + const o = Object.prototype.hasOwnProperty.call(oFiles, path) ? oFiles[path] : undefined + const c = Object.prototype.hasOwnProperty.call(cFiles, path) ? cFiles[path] : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ path, status, original: o, current: c, lang: extToLang(extOf(path)) }) + } + + // Runnables → one YAML leaf each under `runnables/`. Consumers that render + // runnables as script/flow rows (rawAppDiffToItems) skip these and diff the + // runnable objects themselves. + const runnableNames = includeRunnables + ? [...new Set([...Object.keys(oRunnables), ...Object.keys(cRunnables)])].sort() + : [] + for (const name of runnableNames) { + const inO = Object.prototype.hasOwnProperty.call(oRunnables, name) + const inC = Object.prototype.hasOwnProperty.call(cRunnables, name) + const o = inO ? orderedYamlStringify(oRunnables[name]) : undefined + const c = inC ? orderedYamlStringify(cRunnables[name]) : undefined + const status = diffStatus(o, c) + if (!status) continue + entries.push({ + path: reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken), + status, + original: o, + current: c, + lang: 'yaml' + }) + } + + // Remaining metadata → single `app.yaml` leaf. + const oMeta = metadataYaml(oApp) + const cMeta = metadataYaml(cApp) + const metaStatus = diffStatus(oMeta, cMeta) + if (metaStatus) { + entries.push({ + path: reserveUnique(RAW_APP_METADATA_PATH, taken), + status: metaStatus, + original: oMeta, + current: cMeta, + lang: 'yaml', + isMetadata: true + }) + } + + return entries +} + +// A raw-app file rendered as a standalone diff item, shaped like a +// WorkspaceItemDiff (so it flows through the existing list / tree / search / +// count machinery) plus the embedded diff payload. `kind: 'raw_app_file'` +// keeps it distinct from the backend kinds. The composite `path` +// (`/`) nests it under the app's folder in the tree. +export interface RawAppFileItem { + kind: 'raw_app_file' + path: string + /** Friendly composite path (`/`) for tree display only; + * `path` stays storage-keyed so a never-deployed draft still loads/edits via + * its `…/draft_` path. Defaults to `path` when no friendly path differs. */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + /** Workspace path of the owning raw app (for the edit link). */ + appPath: string + status: RawAppDiffStatus + original?: string + current?: string + lang: string + /** True for the synthesized `app.yaml` metadata item. */ + isMetadata: boolean + /** Whole serialized app (both sides) — only on the metadata item, for its + * optional "expand to full YAML" view. */ + fullYamlOriginal?: string + fullYamlCurrent?: string +} + +// A raw-app runnable rendered as a script/flow item (what it actually is). It +// carries the reshaped runnable object per side so the normal script-style +// diff (Content + Metadata) renders it — inline code shows with proper syntax +// highlighting instead of a YAML blob. `kind` drives the row icon/label +// (script vs flow); the body is always rendered script-style. +export interface RawAppRunnableItem { + kind: 'script' | 'flow' + path: string + /** Friendly composite path for tree display only (see RawAppFileItem). */ + displayPath?: string + ahead: number + behind: number + has_changes: boolean + exists_in_source: boolean + exists_in_fork: boolean + appPath: string + status: RawAppDiffStatus + /** Reshaped runnable (content/language hoisted) for the diff viewer. */ + originalRaw?: unknown + currentRaw?: unknown +} + +export type RawAppSyntheticItem = RawAppFileItem | RawAppRunnableItem + +function joinAppPath(appPath: string, filePath: string): string { + // File keys may carry a leading slash (e.g. `/App.tsx`); strip it so the + // composite path has single separators and splits cleanly in the tree. + return `${appPath}/${filePath.replace(/^\/+/, '')}` +} + +// The whole serialized app for one side, matching the previous YAML escape +// hatch (cleaned + ordered). Undefined when the app is absent on that side. +function wholeAppYaml(raw: RawAppish | undefined): string | undefined { + if (!isObject(raw)) return undefined + return orderedYamlStringify(cleanValueProperties(replaceFalseWithUndefined(raw))) +} + +// Hoist an inline runnable's code + language to the top level so the +// script-style diff viewer (which reads top-level `content`/`language`) shows +// the code in the Content tab and the rest as Metadata. Path-referencing +// runnables (no inline script) just fall through to a YAML metadata diff. +function reshapeRunnable(runnable: unknown): unknown { + if (!isObject(runnable)) return runnable + const inline = isObject(runnable.inlineScript) ? runnable.inlineScript : undefined + if (!inline) return runnable + return { + ...runnable, + content: inline.content, + language: inline.language, + // Drop the now-hoisted code so it isn't duplicated in the Metadata tab. + inlineScript: { ...inline, content: undefined } + } +} + +// Runnables become script/flow rows; `runType: 'flow'` picks the flow icon. +function runnableKind(...sides: unknown[]): 'script' | 'flow' { + return sides.some((s) => isObject(s) && s.runType === 'flow') ? 'flow' : 'script' +} + +/** + * Expand a raw-app diff into standalone items for `appPath`: + * - files + the `app.yaml` metadata leaf → `RawAppFileItem`s (the metadata item + * also carries the whole-app YAML for its expand view); + * - runnables → `RawAppRunnableItem`s (script/flow rows). + */ +export function rawAppDiffToItems( + appPath: string, + original: RawAppish | undefined, + current: RawAppish | undefined, + displayAppPath: string = appPath +): RawAppSyntheticItem[] { + // Files + metadata (runnables handled separately as script/flow rows). + const fileItems = parseRawAppDiff(original, current, { includeRunnables: false }).map( + (e): RawAppFileItem => ({ + kind: 'raw_app_file', + path: joinAppPath(appPath, e.path), + displayPath: joinAppPath(displayAppPath, e.path), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: e.status !== 'added', + exists_in_fork: e.status !== 'removed', + appPath, + status: e.status, + original: e.original, + current: e.current, + lang: e.lang, + isMetadata: e.isMetadata ?? false + }) + ) + const metaItem = fileItems.find((i) => i.isMetadata) + if (metaItem) { + metaItem.fullYamlOriginal = wholeAppYaml(original) + metaItem.fullYamlCurrent = wholeAppYaml(current) + } + + // Runnables → script/flow rows, diffed on their object value. + const oApp = normalizeRawApp(original) + const cApp = normalizeRawApp(current) + const oRun = oApp?.runnables ?? {} + const cRun = cApp?.runnables ?? {} + // Reserve each runnable's composite leaf against the real file paths, mirroring + // parseRawAppDiff, so a real file literally named `runnables/` can't yield + // a second leaf at the same composite path. Normalize the leading slash (which + // joinAppPath strips) so `/runnables/x` and `runnables/x` are seen as equal. + const taken = new Set( + [...Object.keys(oApp?.files ?? {}), ...Object.keys(cApp?.files ?? {})].map(stripLeadingSlash) + ) + const runnableItems: RawAppRunnableItem[] = [] + for (const name of [...new Set([...Object.keys(oRun), ...Object.keys(cRun)])].sort()) { + const inO = Object.prototype.hasOwnProperty.call(oRun, name) + const inC = Object.prototype.hasOwnProperty.call(cRun, name) + const oStr = inO ? orderedYamlStringify(oRun[name]) : undefined + const cStr = inC ? orderedYamlStringify(cRun[name]) : undefined + const status = diffStatus(oStr, cStr) + if (!status) continue + const rel = reserveUnique(`${RUNNABLES_PREFIX}${name}`, taken) + runnableItems.push({ + kind: runnableKind(oRun[name], cRun[name]), + path: joinAppPath(appPath, rel), + displayPath: joinAppPath(displayAppPath, rel), + ahead: 0, + behind: 0, + has_changes: true, + exists_in_source: status !== 'added', + exists_in_fork: status !== 'removed', + appPath, + status, + originalRaw: inO ? reshapeRunnable(oRun[name]) : undefined, + currentRaw: inC ? reshapeRunnable(cRun[name]) : undefined + }) + } + + return [...fileItems, ...runnableItems] +} diff --git a/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts b/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts new file mode 100644 index 0000000000..35ff0805d3 --- /dev/null +++ b/frontend/src/lib/components/raw_apps/rawAppDraftValue.ts @@ -0,0 +1,58 @@ +/** + * Shared projection of a raw-app *source* into the flat `AppDraftValue` the + * deploy paths consume. The single source for BOTH the global AI chat + * (`copilot/chat/global/core.ts`) and the Review & Deploy page + * (`rawAppDeploy.ts`), so both bundle the identical shape. It must read a + * draft's top-level `files` (a `RawAppDraft` carries them there, not under + * `value.files`) — otherwise the bundle is empty and deploy fails with "Raw app + * bundle requires /index.ts". + */ +import type { AppDraftValue } from '$lib/components/copilot/chat/global/workspaceItems' +import { DEFAULT_DATA as DEFAULT_RAW_APP_DATA } from '$lib/components/raw_apps/dataTableRefUtils' + +/** Collapse the historical `data` shapes a raw-app source might carry into the + * canonical `AppDraftValue['data']`. */ +export function normalizeRawAppData(value: Record): AppDraftValue['data'] { + if (value.data?.creation) { + return { + tables: value.data.tables ?? [], + datatable: value.data.creation.datatable, + schema: value.data.creation.schema + } + } + if (value.data) { + return value.data + } + if (value.datatables) { + return { ...DEFAULT_RAW_APP_DATA, tables: value.datatables } + } + if (value.dataTableRefs) { + return { ...DEFAULT_RAW_APP_DATA, tables: value.dataTableRefs } + } + return { ...DEFAULT_RAW_APP_DATA } +} + +/** + * Project a raw-app source into a flat `AppDraftValue`. Handles both shapes: + * - a deployed app nests its source under `value` (`app.value.files`); + * - a `RawAppDraft` already carries `files`/`runnables`/`data` at the top level. + * Falling back to the object itself when there's no nested `value` keeps a + * draft's bundle from being dropped. + */ +export function appSourceToDraftValue(app: any, fallback?: any): AppDraftValue { + const value = (app.value ?? app) as Record + return { + summary: app.summary ?? '', + files: { ...(value.files ?? {}) }, + runnables: { ...(value.runnables ?? {}) }, + data: normalizeRawAppData(value), + policy: app.policy ?? fallback?.policy, + custom_path: app.custom_path ?? fallback?.custom_path, + // Pin the fork base: a deployed app exposes `versions` (head = last); an + // existing draft already carries `parent_version` — preserve it. + parent_version: + app.parent_version ?? + (Array.isArray(app.versions) ? app.versions[app.versions.length - 1] : undefined) ?? + fallback?.parent_version + } +} diff --git a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts index f33c6e7eb6..16f588c993 100644 --- a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts +++ b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts @@ -19,10 +19,11 @@ export async function updateRawAppPolicy( ) ).filter((entry): entry is [string, TriggerableV2] => entry != null) const triggerables_v2 = Object.fromEntries(entries) - return { + const next: Policy = { ...currentPolicy, triggerables_v2 } + return next } type RunnableWithInlineScript = RunnableWithFields & { diff --git a/frontend/src/lib/components/raw_apps/utils.test.ts b/frontend/src/lib/components/raw_apps/utils.test.ts index b8ca6d207d..7d087a61ec 100644 --- a/frontend/src/lib/components/raw_apps/utils.test.ts +++ b/frontend/src/lib/components/raw_apps/utils.test.ts @@ -1,9 +1,11 @@ import { describe, expect, it } from 'vitest' import { + canonicalRawAppDiffValue, formatRuntimeLogsForChat, genWmillTs, normalizeRawAppRuntimeLogs, + stripRawAppDiffNoise, type Runnable } from './utils' @@ -61,3 +63,100 @@ describe('normalizeRawAppRuntimeLogs', () => { expect(formatRuntimeLogsForChat(entries)).toBe('[06:13:20.000] LOG: ready') }) }) + +// A deployed raw-app row as returned by getAppByPath: nested `value`, plus the +// server-managed columns and a recomputed inline-script lock. +function deployedRow() { + return { + id: 42, + raw_app: true, + is_draft: false, + created_at: '2024-01-01', + created_by: 'admin', + versions: [1, 2], + extra_perms: { 'u/admin': true }, + summary: 'app', + path: 'u/admin/app', + policy: { execution_mode: 'publisher' }, + value: { + files: { '/App.tsx': 'export default 1' }, + runnables: { + a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun', lock: 'deps\n' } } + } + } + } +} + +describe('stripRawAppDiffNoise', () => { + it('drops server-managed columns, nulls inline locks and canonicalizes data', () => { + const cleaned = stripRawAppDiffNoise(deployedRow()) + + for (const key of [ + 'raw_app', + 'id', + 'created_at', + 'created_by', + 'versions', + 'extra_perms', + 'is_draft' + ]) { + expect(cleaned).not.toHaveProperty(key) + } + expect(cleaned.value.runnables.a.inlineScript.lock).toBeUndefined() + // absent `data` is canonicalized to the default empty shape + expect(cleaned.value.data).toEqual({ tables: [], datatable: undefined, schema: undefined }) + }) + + it('does not mutate the input (live editor state)', () => { + const input = deployedRow() + stripRawAppDiffNoise(input) + expect(input.raw_app).toBe(true) + expect(input.value.runnables.a.inlineScript.lock).toBe('deps\n') + }) + + it('handles the flat editor/draft shape (files/runnables top-level)', () => { + const flat = { + summary: 'app', + files: { '/App.tsx': 'x' }, + runnables: { + a: { type: 'inline', inlineScript: { content: 'm()', language: 'bun', lock: 'l' } } + } + } + const cleaned = stripRawAppDiffNoise(flat) + expect(cleaned.runnables.a.inlineScript.lock).toBeUndefined() + expect(cleaned.data).toEqual({ tables: [], datatable: undefined, schema: undefined }) + }) +}) + +describe('canonicalRawAppDiffValue', () => { + it('collapses a nested deployed row and a flat draft to an identical value when content matches', () => { + const deployed = deployedRow() + // The flat draft shape a raw app autosaves: top-level files/runnables/data, + // no server columns, lock cleared on edit. + const draft = { + summary: 'app', + files: { '/App.tsx': 'export default 1' }, + runnables: { a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun' } } }, + data: { tables: [] }, + policy: { execution_mode: 'publisher' } + } + + expect(canonicalRawAppDiffValue(deployed)).toEqual(canonicalRawAppDiffValue(draft)) + }) + + it('still surfaces a real change (summary edit)', () => { + const deployed = deployedRow() + const draft = { + summary: 'app EDITED', + files: { '/App.tsx': 'export default 1' }, + runnables: { a: { type: 'inline', inlineScript: { content: 'main()', language: 'bun' } } }, + data: { tables: [] } + } + + const a = canonicalRawAppDiffValue(deployed) + const b = canonicalRawAppDiffValue(draft) + expect(a).not.toEqual(b) + expect(a.summary).toBe('app') + expect(b.summary).toBe('app EDITED') + }) +}) diff --git a/frontend/src/lib/components/raw_apps/utils.ts b/frontend/src/lib/components/raw_apps/utils.ts index 53d84fd97c..c0fba48e18 100644 --- a/frontend/src/lib/components/raw_apps/utils.ts +++ b/frontend/src/lib/components/raw_apps/utils.ts @@ -3,6 +3,8 @@ import type { Schema } from '../../common' import { schemaToTsType } from '../../schema' import { isRunnableByName, isRunnableByPath, type RunnableWithFields } from '../apps/inputType' import type { InlineScript } from '../apps/sharedTypes' +import { stateSnapshot } from '$lib/svelte5Utils.svelte' +import { appSourceToDraftValue, normalizeRawAppData } from './rawAppDraftValue' // export type RunnableWithFields = any @@ -15,6 +17,66 @@ export type RawApp = { files: string[] } +// Server-managed columns the deployed app row (getAppByPath) carries but the +// editor's current value never does — leaving them in renders as spurious diff. +const RAW_APP_DEPLOYED_METADATA_KEYS = [ + 'raw_app', + 'id', + 'created_at', + 'created_by', + 'versions', + 'extra_perms', + 'is_draft' +] as const + +/** + * Normalize a raw-app value before a deployed-vs-current diff (or unsaved-change + * comparison). Three sources of spurious post-deploy diff: + * - the deployed row carries server-managed columns (`raw_app`, timestamps, …) + * that the editor's current value lacks; + * - inline-script `lock`s are recomputed server-side at every deploy and the + * editor clears them on edit, so the editor value and the freshly deployed + * one always diverge on `lock` even though the user changed nothing there; + * - the deployed value omits an empty `data` while the editor always carries + * the default `{ tables: [] }`, so even an untouched app reads as changed. + * All three must be neutralized symmetrically on both sides. Returns a deep + * clone; never mutates the input (the current side is live editor state). + */ +export function stripRawAppDiffNoise>(value: T): T { + const cloned = structuredClone(stateSnapshot(value)) as Record + for (const key of RAW_APP_DEPLOYED_METADATA_KEYS) { + delete cloned[key] + } + // Runnables/data live under `.value` on a deployed row and on the editor's + // diff value alike, but a flat draft shape carries them top-level. + const source = cloned.value ?? cloned + const runnables = source.runnables + if (runnables && typeof runnables === 'object') { + for (const k of Object.keys(runnables)) { + const inlineScript = runnables[k]?.inlineScript + if (inlineScript && inlineScript.lock != undefined) { + inlineScript.lock = undefined + } + } + } + // Canonicalize `data` so an absent and a default-empty `data` compare equal. + source.data = normalizeRawAppData(source) + return cloned as T +} + +/** + * Canonical raw-app value for diffing a *draft* against a *deployed* row. On top + * of the noise stripped by stripRawAppDiffNoise, the two also differ in shape: a + * deployed row nests its source under `value`, whereas a draft carries + * `files`/`runnables`/`data` at the top level. `appSourceToDraftValue` collapses + * both onto the same flat field set first. Use this for the session/compare + * draft diff so it matches the editor's Diff button (which shares + * stripRawAppDiffNoise). + */ +export function canonicalRawAppDiffValue(source: Record) { + return stripRawAppDiffNoise(appSourceToDraftValue(source)) +} + export type RawAppRuntimeLogLevel = 'log' | 'info' | 'warn' | 'error' | 'debug' export type RawAppRuntimeLogEntry = { level: RawAppRuntimeLogLevel @@ -79,65 +141,50 @@ export function formatAppRunsForChat(runs: RawAppRunSummary[]): string { return JSON.stringify(runs, null, 2) } -export function htmlContent( +// The sandboxed (isolated) raw-app wrapper is generated server-side and served as +// a sandboxed, opaque-origin document (see `get_raw_app_data` in the backend +// `apps.rs`, WIN-2006) — a blob: URL cannot carry the `CSP: sandbox` response +// header that enforces isolation, so the wrapper must come from the backend. +// +// The function below is used ONLY for the unsandboxed path (the default — the +// publisher did not opt into sandbox isolation). It is loaded as a blob: URL — +// same-origin with the SPA — so, with `allow-same-origin`, the bundle runs with +// the viewer's full session. Crucially this is an in-memory blob, not a +// real-origin endpoint, so it is not a URL an attacker can navigate a logged-in +// victim to in order to gain isolation-bypassing access — the backend `.html` +// document stays sandboxed whenever the publisher did opt in. +export function unsandboxedRawAppHtml( workspace: string, - secret: string | undefined, + secret: string, ctx: any, - baseUrl: string = '', - initialHash: string = '' + baseUrl: string, + initialHash: string ) { return ` - App Preview + App diff --git a/frontend/src/lib/components/runs/DispatchEventsButton.svelte b/frontend/src/lib/components/runs/DispatchEventsButton.svelte new file mode 100644 index 0000000000..8edae8efc6 --- /dev/null +++ b/frontend/src/lib/components/runs/DispatchEventsButton.svelte @@ -0,0 +1,46 @@ + + +{#if list.length > 0} + + {#snippet trigger()} + + {/snippet} + {#snippet content()} +
+ +
+ {/snippet} +
+{/if} diff --git a/frontend/src/lib/components/runs/DispatchEventsPanel.svelte b/frontend/src/lib/components/runs/DispatchEventsPanel.svelte new file mode 100644 index 0000000000..5f9ee2d582 --- /dev/null +++ b/frontend/src/lib/components/runs/DispatchEventsPanel.svelte @@ -0,0 +1,23 @@ + + +{#if list.length > 0} +
+

Dispatch

+
+ +
+
+{/if} diff --git a/frontend/src/lib/components/runs/DispatchEventsTable.svelte b/frontend/src/lib/components/runs/DispatchEventsTable.svelte new file mode 100644 index 0000000000..828437fee6 --- /dev/null +++ b/frontend/src/lib/components/runs/DispatchEventsTable.svelte @@ -0,0 +1,98 @@ + + + + + + + + + + + + + {#each events as ev (ev.created_at + ev.subscriber_path + ev.asset_path)} + + + + + + + {/each} + +
OutcomeSubscriberAssetDetail
+ {#if ev.outcome === 'dispatched'} + + + + {:else if ev.outcome === 'join_pending'} + + join pending + + {:else} + + skipped + + {/if} + + {ev.subscriber_path} + + {ev.asset_kind}://{ev.asset_path} + + {#if ev.outcome === 'dispatched' && ev.child_job_id} + + {ev.child_job_id.slice(0, 8)}… + + + {#if ev.partition} + + · partition {ev.partition} + {/if} + {#if ev.debounce_s && ev.debounce_s > 0} + · debounced {ev.debounce_s}s + {/if} + {:else if ev.outcome === 'join_pending'} + {ev.received_inputs ?? 0}/{ev.required_inputs ?? 0} inputs received + {#if ev.partition} + + for partition {ev.partition} + {/if} + {:else} + {reasonLabel(ev.reason)} + {/if} +
diff --git a/frontend/src/lib/components/runs/RunsQueue.svelte b/frontend/src/lib/components/runs/RunsQueue.svelte index b0c5ed5adb..2cd2914878 100644 --- a/frontend/src/lib/components/runs/RunsQueue.svelte +++ b/frontend/src/lib/components/runs/RunsQueue.svelte @@ -1,7 +1,7 @@ + +{#if self} +
+
{HANDLER_LABEL[self.kind]}
+
+ {#if self.handledJob} + + {/if} + {#if self.schedulePath} + + for schedule {self.schedulePath} + + {/if} +
+
+{:else} + {#if retries.length > 1} +
+
Retries ({retries.length - 1})
+
+ {#each retries as attempt, i (attempt.id)} + + {/each} +
+
+ {/if} + {#if handlers.length > 0} +
+
Handlers
+
+ {#each handlers as handler (handler.id)} + + {/each} +
+
+ {/if} +{/if} diff --git a/frontend/src/lib/components/runs/useDispatchEvents.svelte.ts b/frontend/src/lib/components/runs/useDispatchEvents.svelte.ts new file mode 100644 index 0000000000..f9a148f7f4 --- /dev/null +++ b/frontend/src/lib/components/runs/useDispatchEvents.svelte.ts @@ -0,0 +1,24 @@ +import { resource } from 'runed' +import { JobService, type ListDispatchEventsResponse } from '$lib/gen' + +export type DispatchEvent = ListDispatchEventsResponse[number] + +// Shared loader for "the jobs this run dispatched". Both the inline button and +// the run-detail panel render the same list, so keep the fetch in one place. +// Returns a getter for the (never-undefined) event list — empty while loading, +// when there's nothing to load, or before workspace/jobId are known. +export function useDispatchEvents( + workspace: () => string, + jobId: () => string +): { readonly list: DispatchEvent[] } { + const events = resource( + () => ({ workspace: workspace(), jobId: jobId() }), + async ({ workspace, jobId }) => + workspace && jobId ? await JobService.listDispatchEvents({ workspace, id: jobId }) : [] + ) + return { + get list() { + return events.current ?? [] + } + } +} diff --git a/frontend/src/lib/components/scriptEditor/LogPanel.svelte b/frontend/src/lib/components/scriptEditor/LogPanel.svelte index 1f31d7f12d..008a987999 100644 --- a/frontend/src/lib/components/scriptEditor/LogPanel.svelte +++ b/frontend/src/lib/components/scriptEditor/LogPanel.svelte @@ -112,6 +112,14 @@ let forceJson = $state(false) let isWac = $derived(!!previewJob?.workflow_as_code_status) + // Hide the tab strip when only the "Logs & Result" tab would render — + // avoids a single-item bar in embedded contexts (e.g. asset graph pane). + let visibleTabCount = $derived( + 1 + + (customUi?.disableHistory !== true ? 1 : 0) + + (showCaptures && customUi?.disableTriggerCaptures !== true ? 1 : 0) + + (customUi?.disableTracing !== true ? 1 : 0) + ) let wacDone = $derived( previewJob?.type == 'CompletedJob' || (previewJob != undefined && !previewIsLoading && !!previewJob.workflow_as_code_status) @@ -140,7 +148,11 @@
- + 1 ? 'flex-none' : 'hidden'} + > {#if customUi?.disableHistory !== true} @@ -169,7 +181,7 @@
{:else} - + ('script', ...)` + * handle is keyed by. Distinct from `initialPath` for new drafts — + * `initialPath` is the displayed/editor path (empty for new), while + * this is the URL path the draft is persisted under (`u/{user}/ + * draft_{uuid}`). Used to bracket the bootstrap `initContent` write + * with `UserDraft.stopSync` / `restartSync` so the template seed + * doesn't POST before the user's first real edit. Default to `''` + * for backwards compat with callers that don't manage drafts; the + * stop/restart pair is a no-op on a non-live entry. + */ + userDraftPath?: string + /** + * Workspace + path the AutosaveIndicator watches for sync state. Default + * (undefined) falls back to `$workspaceStore` / `userDraftPath` — the + * full-page editor. The sessions preview sets these to the session's + * (possibly forked) workspace and target path, where autosave is owned by + * `SessionEditorTarget`/`useUserDraftSync`, so the indicator must watch that + * key rather than the global store + the unset `userDraftPath`. + */ + autosaveWorkspace?: string + autosavePath?: string template?: | 'docker' | 'bunnative' @@ -29,7 +51,7 @@ export interface ScriptBuilderProps { showMeta?: boolean neverShowMeta?: boolean diffDrawer?: DiffDrawerI | undefined - savedScript?: NewScriptWithDraftAndDraftTriggers | undefined + savedScript?: (Script | NewScript) & { no_deployed?: boolean } searchParams?: URLSearchParams disableHistoryChange?: boolean customUi?: ScriptBuilderWhitelabelCustomUi @@ -41,12 +63,8 @@ export interface ScriptBuilderProps { // the deployed item) — consumers should skip post-deploy navigation when set. onDeploy?: (e: { path: string; hash: string; stay: boolean }) => void onDeployError?: (e: { path: string; error: any }) => void - onSaveInitial?: (e: { path: string; hash: string }) => void onHistoryRestore?: () => void - onSaveDraftOnlyAtNewPath?: (e: { path: string }) => void - onSaveDraft?: (e: { path: string; savedAtNewPath: boolean; script: NewScript }) => void onSeeDetails?: (e: { path: string }) => void - onSaveDraftError?: (e: { path: string; error: any }) => void onNavigate?: (item: WorkspaceItem) => void // Fired whenever a test run is started from the script editor, with the // preview job id. Used by whitelabel embedders to track test jobs. @@ -59,4 +77,19 @@ export interface ScriptBuilderProps { // overwrite it. Used by the session preview, which opens AI-created scripts // as new but with a path the AI already assigned. initialPathChosen?: boolean + // Threaded to the `AutosaveIndicator` popover so its "Reset to + // deployed" button can do the same thing the load-time toast offers. + // Routes pass their own re-load-without-draft callback here; omit on + // callers (session preview, embedded SDK) that shouldn't surface the + // action at all. + onResetToDeployed?: () => void | Promise + // Triggers the AutosaveIndicator's on-mount "Loaded from draft" hint + // (with a one-shot green flash) the first time it flips to true. + loadedFromDraft?: boolean + // Non-zero when other workspace users have a draft at this path. + // Drives both the indicator's hint label ("Others are working on + // this script") and the popover's "See others' drafts" button. + othersDraftsCount?: number + // Wired by the route to flip the OtherUsersDraftsModal open. + onOpenOthersDrafts?: () => void } diff --git a/frontend/src/lib/components/scripts/CreateActionsScript.svelte b/frontend/src/lib/components/scripts/CreateActionsScript.svelte index 5abd8235f2..48480918e0 100644 --- a/frontend/src/lib/components/scripts/CreateActionsScript.svelte +++ b/frontend/src/lib/components/scripts/CreateActionsScript.svelte @@ -14,7 +14,7 @@ unifiedSize="lg" variant="accent" startIcon={{ icon: Plus }} - href="{base}/scripts/add?nodraft=true" + href="{base}/scripts/add" endIcon={{ icon: Code2 }} > Script diff --git a/frontend/src/lib/components/search/GlobalSearchModal.svelte b/frontend/src/lib/components/search/GlobalSearchModal.svelte index 21b79ea6b9..19e71530e6 100644 --- a/frontend/src/lib/components/search/GlobalSearchModal.svelte +++ b/frontend/src/lib/components/search/GlobalSearchModal.svelte @@ -472,7 +472,6 @@ type?: U time?: number starred?: boolean - has_draft?: boolean } // interface SelectableSearchMenuItem { diff --git a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte b/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte index 9fa83be7fc..0716398131 100644 --- a/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte +++ b/frontend/src/lib/components/sessions/DraftDiffDrawer.svelte @@ -7,6 +7,27 @@ import { getDraftDiffValues, type DraftKind } from '$lib/utils_draft_deploy' import { getDraftItems } from '$lib/workspaceDrafts.svelte' + // Draft rows carry the user-draft itemKind (`trigger_schedule`, `trigger_http`…), + // but the shared row icon/label and the edit-link builder speak the deploy-style + // kinds (`schedule`, `http_trigger`…) the fork drawer and compare page use. Map + // to deploy-style for display, and back for the draft-value getter. + const DEPLOY_KIND_BY_DRAFT_KIND: Partial> = { + trigger_schedule: 'schedule', + trigger_http: 'http_trigger', + trigger_websocket: 'websocket_trigger', + trigger_kafka: 'kafka_trigger', + trigger_nats: 'nats_trigger', + trigger_postgres: 'postgres_trigger', + trigger_mqtt: 'mqtt_trigger', + trigger_sqs: 'sqs_trigger', + trigger_gcp: 'gcp_trigger', + trigger_azure: 'azure_trigger', + trigger_email: 'email_trigger' + } + const DRAFT_KIND_BY_DEPLOY_KIND = Object.fromEntries( + Object.entries(DEPLOY_KIND_BY_DRAFT_KIND).map(([d, p]) => [p, d]) + ) as Record + // Thin wrapper: supplies the deployed ↔ draft data source (server `draft` // table, same as the compare page) to the generic WorkspaceDiffDrawer. // Read-only, mirroring ForkDiffDrawer; deploy/discard live on the Review page. @@ -37,9 +58,21 @@ rows = items.map((it) => { // Raw apps must surface as `raw_app` so the row's edit link points at the // raw-app editor (mirrors CompareDrafts); `getDraftItems` carries the flag. - const kind = it.raw_app ? 'raw_app' : it.kind + const baseKind = it.raw_app ? 'raw_app' : it.kind + const kind = DEPLOY_KIND_BY_DRAFT_KIND[baseKind] ?? baseKind donly[`${kind}/${it.path}`] = it.draft_only - return { kind, path: it.path, status: it.draft_only ? 'added' : 'modified' } + // A never-deployed app/raw_app is parked at a synthetic `…/draft_` + // storage path with the user's typed name in `draft_path`; show that + // (matches the home list) while `path` stays the storage key for loading. + // `summary` comes straight from the draft row, so it shows for every kind + // up front instead of only after the diff value loads. + return { + kind, + path: it.path, + displayPath: it.draft_path ?? it.path, + summary: it.summary, + status: it.draft_only ? 'added' : 'modified' + } }) draftOnlyByKey = donly } catch (e) { @@ -53,8 +86,11 @@ async function loadValues(d: DiffRow): Promise<{ before: unknown; after: unknown }> { const draftOnly = draftOnlyByKey[`${d.kind}/${d.path}`] ?? false - // getDraftDiffValues works on the draft_type kind ('app' for raw apps too). - const kind: DraftKind = d.kind === 'raw_app' ? 'app' : (d.kind as DraftKind) + // getDraftDiffValues keys on the draft itemKind: `raw_app` must stay + // `raw_app` (the helper sends rawApp:true only for that exact kind, which a + // never-deployed raw app needs, else it hits the normal app endpoint and + // 404s). Only the trigger display kinds map back from their deploy-style names. + const kind = (DRAFT_KIND_BY_DEPLOY_KIND[d.kind] ?? d.kind) as DraftKind const { deployed, draft } = await getDraftDiffValues(kind, d.path, workspaceId, draftOnly) // draft_only items have never been deployed → render as "added" (empty // before), matching how the fork drawer renders added items. diff --git a/frontend/src/lib/components/sessions/FlowEditorView.svelte b/frontend/src/lib/components/sessions/FlowEditorView.svelte index 9613b4c25a..fb98e05496 100644 --- a/frontend/src/lib/components/sessions/FlowEditorView.svelte +++ b/frontend/src/lib/components/sessions/FlowEditorView.svelte @@ -2,9 +2,12 @@ import FlowBuilder from '$lib/components/FlowBuilder.svelte' import DiffDrawer from '$lib/components/DiffDrawer.svelte' import type { WorkspaceItem } from '$lib/components/workspacePicker' + import type { Flow } from '$lib/gen' import type { SessionRuntime } from './sessionRuntime.svelte' import SessionEditorTarget from './SessionEditorTarget.svelte' import { sendUserToast } from '$lib/toast' + import { UserDraft } from '$lib/userDraft.svelte' + import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import { invalidateWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' let { @@ -26,22 +29,40 @@ let selectedId = $state('settings-metadata') let diffDrawer: DiffDrawer | undefined = $state() - // In a session pane, "restore" just reloads from the current state — the - // session target stays put. The Diff drawer's primary use here is viewing - // the diff; restore is best-effort. - async function restoreFromCurrentTarget() { + // Restore actions for the diff drawer. A `loadFlow`-based handler is a no-op: + // loadFlow early-returns on the already-loaded path (and would re-read the + // local draft anyway). Instead reset the live UserDraft cell to the target + // baseline — useUserDraftSync's inbound effect then syncs the editor preview. + // Mirrors ScriptEditorView. + async function restoreDeployed() { + const saved = runtime.savedFlow.val + if (!saved) { + sendUserToast('Could not restore to deployed', true) + return + } diffDrawer?.closeDrawer() - await runtime.loadFlow(workspaceId, path) + // Drop the user's per-user draft too, so "deployed" sticks across a + // reload. The syncer's `value: null` POST is the canonical per-user + // delete; fire-and-forget since the in-memory reset below is what the + // preview reflects immediately. + if (saved.is_draft) { + saved.is_draft = false + UserDraftDbSyncer.save({ + workspace: workspaceId, + itemKind: 'flow', + path: saved.path, + value: null + }).catch((e) => console.error('restoreDeployed: draft delete failed', e)) + invalidateWorkspaceDrafts(workspaceId) + } + const deployed = structuredClone($state.snapshot(saved)) as Flow & { draft?: unknown } + delete deployed.draft + UserDraft.discard('flow', path, deployed, { workspace: workspaceId }) } {#if runtime.savedFlow.val} - + {/if} {#snippet editor()} + has its own AI chat in the left pane, so the toggle is redundant here. + newFlow: a draft-only flow has a synthesized `savedFlow` (no_deployed=true) + but no deployed row, so it must deploy via createFlow — treating it as + !newFlow would updateFlow the draft_ path and 404 "Flow not found". + initialPath: a brand-new flow is stored under a `draft_` path with + its intended name in `draft_path`; seed the builder from `draft_path` + (as the full-page editor does) so the Path widget and deploy use the + friendly name rather than creating a flow literally named draft_. --> { - runtime.scheduleForkComparisonRefresh() - // Saving a draft adds/keeps a pending draft — refresh the Draft Count. - invalidateWorkspaceDrafts(workspaceId) - }} onDeploy={() => { // FlowBuilder has no deploy toast and the session stays put, so toast // here, then sync the preview to deployed (pulls the new locks + version_id). diff --git a/frontend/src/lib/components/sessions/RawAppEditorView.svelte b/frontend/src/lib/components/sessions/RawAppEditorView.svelte index 8b2d435daa..3e600e5114 100644 --- a/frontend/src/lib/components/sessions/RawAppEditorView.svelte +++ b/frontend/src/lib/components/sessions/RawAppEditorView.svelte @@ -1,9 +1,11 @@ {#if runtime.savedRawApp.val} - + {/if} {#snippet editor()} {#if runtime.rawApp.val} + { // Sync the preview to deployed (raw apps deploy only from this editor). runtime.syncPreviewWithDeployed(workspaceId, 'raw_app', e.path) // Deploying clears the item's pending draft — refresh the Draft Count. invalidateWorkspaceDrafts(workspaceId) }} - onSaveDraft={() => { - // Saving a server draft adds/updates a draft — refresh the Draft Count so - // the session draft bar appears/updates immediately (parity with script/flow). - invalidateWorkspaceDrafts(workspaceId) - }} defaultSidebarCollapsed sidebarStorageKey="raw-app-sidebar-collapsed-preview" defaultSplitWithPreview={false} diff --git a/frontend/src/lib/components/sessions/ScriptEditorView.svelte b/frontend/src/lib/components/sessions/ScriptEditorView.svelte index 2e6fc4210f..563ddf9e55 100644 --- a/frontend/src/lib/components/sessions/ScriptEditorView.svelte +++ b/frontend/src/lib/components/sessions/ScriptEditorView.svelte @@ -3,8 +3,9 @@ import DiffDrawer from '$lib/components/DiffDrawer.svelte' import type { WorkspaceItem } from '$lib/components/workspacePicker' import type { SessionRuntime } from './sessionRuntime.svelte' - import { DraftService, ScriptService, type NewScript } from '$lib/gen' + import type { NewScript } from '$lib/gen' import { UserDraft } from '$lib/userDraft.svelte' + import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import SessionEditorTarget from './SessionEditorTarget.svelte' import { sendUserToast } from '$lib/toast' import { invalidateWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' @@ -41,39 +42,37 @@ return } diffDrawer?.closeDrawer() - // Drop the backend (DB) draft too, so "deployed" sticks across a reload. - if (saved.draft) { - try { - await DraftService.deleteDraft({ workspace: workspaceId, kind: 'script', path: saved.path }) - saved.draft = undefined - // Server draft gone — refresh the session draft-bar count immediately - // instead of waiting for an AI turn-end / tab-refocus signal. - invalidateWorkspaceDrafts(workspaceId) - } catch (e: any) { - sendUserToast(`Could not delete draft: ${e?.body ?? e}`, true) - return - } + // Drop the user's per-user draft too, so "deployed" sticks across + // a reload. The overlay sets `is_draft: true` when a draft exists + // for the authed user; the syncer's `value: null` POST is the + // canonical per-user delete. + // + // Fire-and-forget: every read here (`saved`, the snapshot we build + // below, the UserDraft.discard write) is purely in-memory, so we + // don't need the DELETE to have landed to finish the restore. We + // flip `is_draft` optimistically so the UI matches the new intent + // immediately. A failed DELETE only matters across a hard reload + // before it lands — log and move on. + if (saved.is_draft) { + saved.is_draft = false + UserDraftDbSyncer.save({ + workspace: workspaceId, + itemKind: 'script', + path: saved.path, + value: null + }).catch((e) => console.error('restoreDeployed: draft delete failed', e)) + // Per-user draft gone — refresh the session draft-bar count immediately + // instead of waiting for an AI turn-end / tab-refocus signal. + invalidateWorkspaceDrafts(workspaceId) } const deployed = structuredClone($state.snapshot(saved)) as NewScript & { draft?: unknown } delete deployed.draft UserDraft.discard('script', path, deployed, { workspace: workspaceId }) } - - async function restoreDraft() { - const backendDraft = runtime.savedScript.val?.draft as NewScript | undefined - if (!backendDraft) { - sendUserToast('Could not restore to draft', true) - return - } - diffDrawer?.closeDrawer() - UserDraft.discard('script', path, structuredClone($state.snapshot(backendDraft)), { - workspace: workspaceId - }) - } {#if runtime.savedScript.val} - + {/if} { - runtime.scheduleForkComparisonRefresh() - // Saving a draft adds/keeps a pending draft — refresh the Draft Count. - invalidateWorkspaceDrafts(workspaceId) - // Re-pin parent_hash to the latest version so the next Deploy's conflict - // check (which runs before deploy, while the session stays mounted) - // doesn't misfire. - try { - const latest = await ScriptService.getScriptLatestVersion({ - workspace: workspaceId, - path: e.path - }) - const cur = runtime.scriptStore.val - if (latest?.script_hash && cur) cur.parent_hash = latest.script_hash - } catch (err) { - console.error('Failed to sync parent_hash after save draft', err) - } - }} onDeploy={(e) => { // Fires on every deploy (primary, "Deploy & Stay here", and lib — we // ignore e.stay since the session always stays). Toast, then sync the diff --git a/frontend/src/lib/components/sessions/SessionDraftBar.svelte b/frontend/src/lib/components/sessions/SessionDraftBar.svelte index dcc5640f2c..d23070151a 100644 --- a/frontend/src/lib/components/sessions/SessionDraftBar.svelte +++ b/frontend/src/lib/components/sessions/SessionDraftBar.svelte @@ -7,6 +7,7 @@ import DraftDiffDrawer from './DraftDiffDrawer.svelte' import SessionDiffButton from './SessionDiffButton.svelte' import { useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' + import Tooltip from '$lib/components/meltComponents/Tooltip.svelte' let { session }: { session: Session } = $props() @@ -58,7 +59,13 @@ >
- {count} draft{count === 1 ? '' : 's'} + + {count} draft{count === 1 ? '' : 's'} + {#snippet text()} + Tracks all unsaved draft changes in this workspace — including edits made outside this + chat (e.g. in the editor), not only changes made by the assistant. + {/snippet} +
drawer?.open()} /> diff --git a/frontend/src/lib/components/sessions/SessionEditorTarget.svelte b/frontend/src/lib/components/sessions/SessionEditorTarget.svelte index 55bd697bb9..f4c29188f9 100644 --- a/frontend/src/lib/components/sessions/SessionEditorTarget.svelte +++ b/frontend/src/lib/components/sessions/SessionEditorTarget.svelte @@ -49,7 +49,7 @@ function buildCodec(): DraftSyncCodec { if (kind === 'flow') return makeFlowCodec(runtime) - if (kind === 'script') return makeScriptCodec(runtime) + if (kind === 'script') return makeScriptCodec(runtime, () => path) return makeRawAppCodec(runtime) } diff --git a/frontend/src/lib/components/sessions/SessionFilterMenu.svelte b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte new file mode 100644 index 0000000000..e5e67d1567 --- /dev/null +++ b/frontend/src/lib/components/sessions/SessionFilterMenu.svelte @@ -0,0 +1,77 @@ + + + + +{#if $subOpen} +
+ +
+
+ + Include sessions from every workspace. +
+
+ + {#if archivedCount > 0} + + {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + + {/if} +
+
+
+{/if} diff --git a/frontend/src/lib/components/sessions/SessionForkBar.svelte b/frontend/src/lib/components/sessions/SessionForkBar.svelte index b1e74b43eb..18d36a755c 100644 --- a/frontend/src/lib/components/sessions/SessionForkBar.svelte +++ b/frontend/src/lib/components/sessions/SessionForkBar.svelte @@ -68,6 +68,10 @@ const forkStatus = $derived(deriveForkStatus(session, $userWorkspaces, comparison)) const isUnavailable = $derived(forkStatus === 'unavailable') + // The committed workspace is gone, so we can't read its parent to know if + // it was a fork — fall back to the fork-id convention for the wording. + const committedIsFork = $derived(committedId?.startsWith('wm-fork-') ?? false) + $effect(() => { if (!runtime || !committedId || !parentWorkspaceId) return void runtime.ensureForkComparison(parentWorkspaceId, committedId) @@ -111,15 +115,19 @@ {#if committedId && isUnavailable} - + chat input is disabled by SessionWrapper while this is shown. Shown even + for an archived session — unarchiving in place can't help when the + workspace is gone, so move/discard is the only real recovery path. -->
- The fork has been archived or deleted + The {committedIsFork ? 'fork' : 'workspace'} has been archived or deleted Move this session to another workspace, or discard it. {committedId} diff --git a/frontend/src/lib/components/sessions/SessionPicker.svelte b/frontend/src/lib/components/sessions/SessionPicker.svelte index 31c463e79f..a3273ddceb 100644 --- a/frontend/src/lib/components/sessions/SessionPicker.svelte +++ b/frontend/src/lib/components/sessions/SessionPicker.svelte @@ -22,8 +22,9 @@ import { createSession, deriveForkStatus, - getEffectiveWorkspaceId, + deleteSessionsForWorkspace, isForkSession, + reconcileAfterWorkspaceChange, renameSession, selectSession, sessionState, @@ -31,7 +32,7 @@ syncWorkspaceTo, type Session } from './sessionState.svelte' - import { forgetSessionSeen, unreadCountFor } from './sessionUnread.svelte' + import { unreadCountFor } from './sessionUnread.svelte' import Popover from '$lib/components/meltComponents/Popover.svelte' import Toggle from '$lib/components/Toggle.svelte' import { @@ -41,14 +42,15 @@ removeSession } from './sessionRuntime.svelte' import SessionStatusDot from './SessionStatusDot.svelte' + import SessionFilterMenu from './SessionFilterMenu.svelte' import { Menu, Menubar, MenuItem } from '$lib/components/meltComponents' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' - import { visibleWorkspaceIds } from './sessionScope.svelte' import { isGlobalAiEnabled } from '$lib/components/copilot/chat/global/gate' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' + import { currentWorkspaceRootId, workspaceRootId } from './sessionScope.svelte' // Look up the cached fork comparison for a session through its runtime // (if any). The deriveForkStatus helper handles the "no runtime yet" @@ -101,40 +103,81 @@ 'boolean' ) const showArchived = useLocalStorageValue('windmill_sessions_show_archived', false, 'boolean') + // Off by default: the list is scoped to the current workspace family. Turn on + // to include sessions from every workspace (grouped by family) — handy when + // switching sessions across workspaces without switching workspace first. + const showAllWorkspaces = useLocalStorageValue( + 'windmill_sessions_show_all_workspaces', + false, + 'boolean' + ) let listRoot: HTMLDivElement | undefined = $state() - // Sessions visible in the current workspace (active workspace + its - // forks). Drafts (no committed workspace) are scoped by their - // pending workspace pick — set at create time to the workspace the - // user was in. Archived sessions are filtered out unless the user - // has opted in via the filter popover. + // A session's family root: the stored grouping id, else derived live. + function sessionRootOf(s: Session): string | undefined { + return ( + s.workspace_root_id ?? + workspaceRootId(s.workspace_id ?? s.pending_workspace_id, $userWorkspaces) + ) + } + + // Flat list passing the archive + scope filters. Grouping for display happens + // in `sessionGroups`; this flat view drives the runtime / fork-comparison + // effects, the unread total, and keyboard navigation. const visibleSessions = $derived( sessionState.sessions.filter((s) => { - // Transient (not-yet-sent) sessions live as their own page but - // don't clutter the sidebar list. if (s.transient) return false + // The open session always stays in the list, ignoring both filters. + if (s.id === sessionState.currentSessionId) return true if (s.archived && !showArchived.val) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - // Unavailable sessions (committed workspace was deleted / - // archived / access revoked) stay visible everywhere so the - // user can resolve them — move, archive, or delete. They'd - // otherwise be permanently hidden the moment their workspace - // disappeared. - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false + if (!showAllWorkspaces.val) { + const currentRoot = $currentWorkspaceRootId + if (currentRoot && sessionRootOf(s) !== currentRoot) return false + } + return true }) ) + + // Sessions grouped by workspace family for display, each group newest-first. + // Family order is stable (by most-recent activity) and deliberately NOT tied + // to the current workspace: pinning the active family first reshuffled the + // whole list on every workspace switch, which is disorienting. + const sessionGroups = $derived.by(() => { + const byRoot = new Map() + for (const s of visibleSessions) { + const root = sessionRootOf(s) ?? s.workspace_id ?? s.pending_workspace_id ?? '' + const arr = byRoot.get(root) + if (arr) arr.push(s) + else byRoot.set(root, [s]) + } + const groups = [...byRoot.entries()].map(([rootId, sessions]) => { + sessions.sort((a, b) => b.createdAt - a.createdAt) + return { + rootId, + name: $userWorkspaces.find((w) => w.id === rootId)?.name || rootId || 'Workspace', + sessions, + mostRecent: sessions[0]?.createdAt ?? 0 + } + }) + groups.sort((a, b) => b.mostRecent - a.mostRecent) + return groups + }) + + // Family labels are redundant when scoped to the current workspace (a single + // family) — show them when including all workspaces, and also if the + // active-session override surfaces a second family while scoped (avoids + // ambiguity). + const showGroupHeaders = $derived(showAllWorkspaces.val || sessionGroups.length > 1) + const archivedCount = $derived( sessionState.sessions.filter((s) => { if (!s.archived || s.transient) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false + if (showAllWorkspaces.val) return true + const currentRoot = $currentWorkspaceRootId + return ( + !currentRoot || sessionRootOf(s) === currentRoot || s.id === sessionState.currentSessionId + ) }).length ) @@ -264,12 +307,12 @@ if (!session) return const wasActive = sessionState.currentSessionId === session.id removeSession(session.id) - forgetSessionSeen(session.id) if (forkToDelete) { try { await WorkspaceService.deleteWorkspace({ workspace: forkToDelete }) + await deleteSessionsForWorkspace(forkToDelete) sendUserToast(`Deleted forked workspace ${forkToDelete}`) - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to delete fork ${forkToDelete}: ${e?.body ?? e}`, true) } @@ -319,12 +362,15 @@ {#if !globalEnabled} - + {:else if isCollapsed}
{#snippet children({ createMenu })} - + {#snippet triggr({ trigger })}
{/snippet} - {#snippet children({ item })} + {#snippet children({ item, builders })}
@@ -353,47 +399,66 @@
- {#each visibleSessions as session (session.id)} - {@const runtime = getRuntime(session.id)} - {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} - {@const isSelected = - onSessionsPage && session.id === sessionState.currentSessionId} - {@const unread = unreadFor(session)} - {@const draft = hasDraft(session)} - activate(session)} - {item} - > - - 0 ? 'font-semibold text-primary' : '' - )} + +
+
+ {#each sessionGroups as group (group.rootId)} + {#if showGroupHeaders} +
- {session.summary ?? 'Untitled session'} - - {#if draft || unread > 0} - - {#if draft} - - {/if} - {#if unread > 0} - - {unread > 9 ? '9+' : unread} - - {/if} + {group.name} +
+ {/if} + {#each group.sessions as session (session.id)} + {@const runtime = getRuntime(session.id)} + {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} + {@const isSelected = + onSessionsPage && session.id === sessionState.currentSessionId} + {@const unread = unreadFor(session)} + {@const draft = hasDraft(session)} + activate(session)} + {item} + > + + 0 ? 'font-semibold text-primary' : '' + )} + > + {session.summary ?? 'Untitled session'} - {/if} - + {#if draft || unread > 0} + + {#if draft} + + {/if} + {#if unread > 0} + + {unread > 9 ? '9+' : unread} + + {/if} + + {/if} + + {/each} {/each}
@@ -434,7 +499,8 @@ type="button" title="Filter sessions" aria-label="Filter sessions" - class="inline-flex items-center justify-center w-5 h-5 rounded text-tertiary hover:bg-surface-hover hover:text-primary {showArchived.val + class="inline-flex items-center justify-center w-5 h-5 rounded text-tertiary hover:bg-surface-hover hover:text-primary {showArchived.val || + showAllWorkspaces.val ? 'text-emphasis' : ''}" > @@ -443,18 +509,30 @@ {/snippet} {#snippet content()}
- - {#if archivedCount > 0} +
+ - {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + Include sessions from every workspace. - {/if} +
+
+ + {#if archivedCount > 0} + + {archivedCount} archived session{archivedCount === 1 ? '' : 's'} + + {/if} +
{/snippet} @@ -477,119 +555,137 @@ role="listbox" tabindex="-1" > - {#each visibleSessions as session (session.id)} - {@const runtime = getRuntime(session.id)} - {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} - {@const isSelected = onSessionsPage && session.id === sessionState.currentSessionId} - {@const isEditing = editingId === session.id} - {@const unread = unreadFor(session)} - {@const draft = hasDraft(session)} -
- {#if isEditing} - - - - { - if (e.key === 'Enter') commitRename() - else if (e.key === 'Escape') cancelRename() - }} - onblur={commitRename} - placeholder="Untitled session" - autofocus - spellcheck="false" - class="flex-1 min-w-0 bg-transparent border-0 outline-none text-xs font-normal text-primary" - /> - - {:else} - -
- startRename(session) - }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => setSessionArchived(session.id, true) - }, - { - displayName: 'Delete', - icon: Trash2, - type: 'delete', - action: () => (pendingDelete = session) - } - ]} + {#each sessionGroups as group (group.rootId)} + {#if showGroupHeaders} +
+ {group.name} +
+ {/if} + {#each group.sessions as session (session.id)} + {@const runtime = getRuntime(session.id)} + {@const status = runtime ? getSessionChatStatus(runtime) : 'idle'} + {@const isSelected = onSessionsPage && session.id === sessionState.currentSessionId} + {@const isEditing = editingId === session.id} + {@const unread = unreadFor(session)} + {@const draft = hasDraft(session)} +
+ {#if isEditing} + + + + { + if (e.key === 'Enter') commitRename() + else if (e.key === 'Escape') cancelRename() + }} + onblur={commitRename} + placeholder="Untitled session" + autofocus + spellcheck="false" + class="flex-1 min-w-0 bg-transparent border-0 outline-none text-xs font-normal text-primary" + /> + + {:else} +
- {/if} -
+ {/if} + +
+ startRename(session) + }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailableFork(session) + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => setSessionArchived(session.id, true) + } + ]), + { + displayName: 'Delete', + icon: Trash2, + type: 'delete', + action: () => (pendingDelete = session) + } + ]} + > + {#snippet buttonReplacement()} + + + + {/snippet} + +
+ {/if} +
+ {/each} {/each}
{/if} diff --git a/frontend/src/lib/components/sessions/SessionWrapper.svelte b/frontend/src/lib/components/sessions/SessionWrapper.svelte index 830cf1d2bd..027089d4ae 100644 --- a/frontend/src/lib/components/sessions/SessionWrapper.svelte +++ b/frontend/src/lib/components/sessions/SessionWrapper.svelte @@ -7,7 +7,7 @@ import ConfirmationModal from '$lib/components/common/confirmationModal/ConfirmationModal.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { AIChatManager } from '$lib/components/copilot/chat/AIChatManager.svelte' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' import Toggle from '$lib/components/Toggle.svelte' @@ -32,9 +32,11 @@ import SessionDraftBar from './SessionDraftBar.svelte' import { createSession, + deleteSessionsForWorkspace, getEffectiveWorkspaceId, moveSessionToNewFork, moveSessionToWorkspace, + reconcileAfterWorkspaceChange, renameSession, selectSession, sessionState, @@ -101,16 +103,6 @@ let archiveConfirmOpen = $state(false) let archiveAlsoFork = $state(false) - async function refreshWorkspaceList() { - // Match the SidebarContent.deleteFork pattern: replace the in-memory - // list rather than nulling it. See B1 fix. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } - } - async function handleConfirmedDelete() { deleteConfirmOpen = false if (!session) return @@ -125,8 +117,9 @@ if (forkToDelete) { try { await WorkspaceService.deleteWorkspace({ workspace: forkToDelete }) + await deleteSessionsForWorkspace(forkToDelete) sendUserToast(`Deleted forked workspace ${forkToDelete}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to delete fork ${forkToDelete}: ${e?.body ?? e}`, true) } @@ -150,7 +143,7 @@ try { await WorkspaceService.archiveWorkspace({ workspace: forkToArchive }) sendUserToast(`Archived forked workspace ${forkToArchive}`) - await refreshWorkspaceList() + await reconcileAfterWorkspaceChange() } catch (e: any) { sendUserToast(`Failed to archive fork ${forkToArchive}: ${e?.body ?? e}`, true) } @@ -280,6 +273,29 @@ is position:fixed, so it doesn't count as a flex item — no stray gap when only one bar shows. -->
+ {#if session.archived && !isUnavailable} + +
+
+ + This session is archived +
+ +
+ {/if} moveAndActivate(workspaceId)} @@ -295,8 +311,13 @@ sessions have their own empty-state affordances above. --> {#snippet sessionEmptyHint()}{/snippet} + - +
summaryInput?.edit() }, - session.archived - ? { - displayName: 'Unarchive', - icon: ArchiveRestore, - action: () => setSessionArchived(session.id, false) - } - : { - displayName: 'Archive', - icon: Archive, - action: () => archiveAndReset() - }, + ...(session.archived + ? // No Unarchive when the workspace is gone — it can't persist + // (putSession guard) and reconcile would re-archive it. + isUnavailable + ? [] + : [ + { + displayName: 'Unarchive', + icon: ArchiveRestore, + action: () => setSessionArchived(session.id, false) + } + ] + : [ + { + displayName: 'Archive', + icon: Archive, + action: () => archiveAndReset() + } + ]), { displayName: 'Delete', icon: Trash2, @@ -398,17 +427,19 @@ hideHeader hideModeSelector wideLayout - forceDisabled={isUnavailable} + forceDisabled={isUnavailable || !!session.archived} forceDisabledMessage={isUnavailable ? 'This session is linked to a workspace that no longer exists. Move it or discard it from the banner above to keep working.' - : ''} + : session.archived + ? 'This session is archived. Unarchive it from the banner above to keep working.' + : ''} emptyHint={sessionEmptyHint} {inputPreface} />
{#if hasEditor && session.target} - +
`) show its friendly typed path + * while keys, value-loading and edit links stay keyed on `path`. */ + displayPath?: string + /** Summary supplied by the data source. Preferred over the one derived + * from the loaded diff value, and shown before that value loads. */ + summary?: string }
@@ -33,35 +42,52 @@
{ - updateSetting(codeCompletionSessionEnabled, e.detail, 'codeCompletionSessionEnabled') + updateAiEnabled(e.detail) }} - checked={$codeCompletionSessionEnabled} + checked={!$aiUserDisabled} options={{ - right: 'Code completion', - rightTooltip: 'AI completion in the code editors' + right: 'Windmill AI', + rightTooltip: + 'Enable Windmill AI for your account on this device. Turning this off hides the AI chat, code completion, metadata completion and flow step input completion.' }} /> - { - updateSetting(metadataCompletionEnabled, e.detail, 'metadataCompletionEnabled') - }} - checked={$metadataCompletionEnabled} - options={{ - right: 'Metadata completion', - rightTooltip: 'AI completion for summaries and descriptions' - }} - /> - { - updateSetting(stepInputCompletionEnabled, e.detail, 'stepInputCompletionEnabled') - }} - checked={$stepInputCompletionEnabled} - options={{ - right: 'Flow step input completion', - rightTooltip: 'AI completion for flow step inputs' - }} - /> +
+ { + updateSetting(codeCompletionSessionEnabled, e.detail, 'codeCompletionSessionEnabled') + }} + checked={$codeCompletionSessionEnabled} + options={{ + right: 'Code completion', + rightTooltip: 'AI completion in the code editors' + }} + /> + + { + updateSetting(metadataCompletionEnabled, e.detail, 'metadataCompletionEnabled') + }} + checked={$metadataCompletionEnabled} + options={{ + right: 'Metadata completion', + rightTooltip: 'AI completion for summaries and descriptions' + }} + /> + { + updateSetting(stepInputCompletionEnabled, e.detail, 'stepInputCompletionEnabled') + }} + checked={$stepInputCompletionEnabled} + options={{ + right: 'Flow step input completion', + rightTooltip: 'AI completion for flow step inputs' + }} + /> +
diff --git a/frontend/src/lib/components/sidebar/MultiplayerMenu.svelte b/frontend/src/lib/components/sidebar/MultiplayerMenu.svelte index 29fe62d2ff..ebbd0b24b1 100644 --- a/frontend/src/lib/components/sidebar/MultiplayerMenu.svelte +++ b/frontend/src/lib/components/sidebar/MultiplayerMenu.svelte @@ -1,5 +1,6 @@ + +
+
+ {path} +
+ + Triggers whose runnable is part of a data pipeline are managed from the pipeline editor. + The script path is fixed; edit the pipeline to reassign or remove the trigger. + +
diff --git a/frontend/src/lib/components/triggers/TriggerRunnablePicker.svelte b/frontend/src/lib/components/triggers/TriggerRunnablePicker.svelte new file mode 100644 index 0000000000..208645182f --- /dev/null +++ b/frontend/src/lib/components/triggers/TriggerRunnablePicker.svelte @@ -0,0 +1,59 @@ + + +{#if fixedScriptPath != ''} + +{:else} +

+ {promptText} +

+
+ + {@render createButton?.()} +
+{/if} diff --git a/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte index 3293a171da..e628c7cc99 100644 --- a/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/azure/AzureTriggerEditorInner.svelte @@ -135,11 +135,16 @@ itemKind = isFlow ? 'flow' : 'script' edit = true dirtyPath = false - await loadTrigger(defaultValues) + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultValues) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getAzureConfig())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultValues) { initialConfig = structuredClone($state.snapshot(getAzureConfig())) } - originalConfig = structuredClone($state.snapshot(getAzureConfig())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load Azure trigger: ${err.body}`, true) @@ -184,19 +189,31 @@ } } - async function loadTrigger(defaultConfig?: Record): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return + return { overlay: undefined, noDeployed: false } } try { const s = await AzureTriggerService.getAzureTrigger({ workspace: $workspaceStore!, - path: initialPath + path: initialPath, + getDraft: true }) - loadTriggerConfig(s) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined + } } catch (error) { sendUserToast(`Could not load Azure trigger: ${error.body}`, true) + return { overlay: undefined, noDeployed: false } } } diff --git a/frontend/src/lib/components/triggers/email/EmailTriggerEditor.svelte b/frontend/src/lib/components/triggers/email/EmailTriggerEditor.svelte index 77014d0dac..dd035afb2e 100644 --- a/frontend/src/lib/components/triggers/email/EmailTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/email/EmailTriggerEditor.svelte @@ -11,10 +11,10 @@ let { onUpdate = undefined, customSaveBehavior }: Props = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte index 436cd16762..34cf4e92c3 100644 --- a/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/email/EmailTriggerEditorInner.svelte @@ -3,8 +3,7 @@ import Drawer from '$lib/components/common/drawer/Drawer.svelte' import DrawerContent from '$lib/components/common/drawer/DrawerContent.svelte' import Path from '$lib/components/Path.svelte' - import Required from '$lib/components/Required.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import { EmailTriggerService, type ErrorHandler, @@ -117,7 +116,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultConfig?: Partial + defaultConfig?: Partial, + fixedScriptPath_?: string ) { drawerLoading = true let loader = setTimeout(() => { @@ -131,12 +131,21 @@ edit = true dirtyPath = false dirtyLocalPart = false - await loadTrigger(defaultConfig) + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + // Form holds DEPLOYED here. Capture `originalConfig` as the + // deployed baseline so `hasChanged` (= current != originalConfig) + // fires whenever a draft exists, not only after the user edits. + originalConfig = structuredClone($state.snapshot(getEmailTriggerConfig())) as NewEmailTrigger + if (draftOverlay) loadTriggerConfig(draftOverlay as Partial) if (!defaultConfig) { // If the email trigger is loaded from the backend, we to set the initial config - initialConfig = structuredClone($state.snapshot(getEmailTriggerConfig())) + initialConfig = structuredClone($state.snapshot(getEmailTriggerConfig())) as NewEmailTrigger } - originalConfig = structuredClone($state.snapshot(getEmailTriggerConfig())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load email trigger: ${err}`, true) @@ -206,17 +215,31 @@ preservePermissionedAs = !!cfg?.permissioned_as } - async function loadTrigger(defaultConfig?: Partial): Promise { + /** + * Apply the deployed config to the form, then return the saved-draft + * overlay (if any) so the caller can capture the deployed-only form + * state as `originalConfig` BEFORE applying the draft. See + * `NatsTriggerEditorInner` for the rationale. + */ + async function loadTrigger( + defaultConfig?: Partial + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await EmailTriggerService.getEmailTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await EmailTriggerService.getEmailTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } @@ -365,23 +388,16 @@ {#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
-
- - + + {#snippet createButton()} {#if emptyString(script_path)} {/if} -
-
+ {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/gcp/GcpTriggerEditor.svelte b/frontend/src/lib/components/triggers/gcp/GcpTriggerEditor.svelte index 2a7c31512a..7a645e6886 100644 --- a/frontend/src/lib/components/triggers/gcp/GcpTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/gcp/GcpTriggerEditor.svelte @@ -5,10 +5,10 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte index 868d4ff69e..f2a22c359c 100644 --- a/frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/gcp/GcpTriggerEditorInner.svelte @@ -18,8 +18,7 @@ type TriggerMode } from '$lib/gen' import Section from '$lib/components/Section.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' - import Required from '$lib/components/Required.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import GcpTriggerEditorConfigSection from './GcpTriggerEditorConfigSection.svelte' import { untrack, type Snippet } from 'svelte' import TriggerEditorToolbar from '../TriggerEditorToolbar.svelte' @@ -128,7 +127,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultValues?: Record + defaultValues?: Record, + fixedScriptPath_?: string ) { drawerLoading = true try { @@ -137,11 +137,17 @@ itemKind = isFlow ? 'flow' : 'script' edit = true dirtyPath = false - await loadTrigger(defaultValues) + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultValues) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getGcpConfig())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultValues) { initialConfig = structuredClone($state.snapshot(getGcpConfig())) } - originalConfig = structuredClone($state.snapshot(getGcpConfig())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load GCP Pub/Sub trigger: ${err.body}`, true) @@ -188,20 +194,31 @@ } } - async function loadTrigger(defaultConfig?: Record): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - try { - const s = await GcpTriggerService.getGcpTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - loadTriggerConfig(s) - } catch (error) { - sendUserToast(`Could not load GCP Pub/Sub trigger: ${error.body}`, true) + return { overlay: undefined, noDeployed: false } + } + try { + const s = await GcpTriggerService.getGcpTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } + } catch (error) { + sendUserToast(`Could not load GCP Pub/Sub trigger: ${error.body}`, true) + return { overlay: undefined, noDeployed: false } } } @@ -458,32 +475,28 @@ {#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
- - {#if emptyString(script_path)} - - {/if} -
+ + {#snippet createButton()} + {#if emptyString(script_path)} + + {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/http/RouteEditorInner.svelte b/frontend/src/lib/components/triggers/http/RouteEditorInner.svelte index 3f281be23b..d29a3d0cab 100644 --- a/frontend/src/lib/components/triggers/http/RouteEditorInner.svelte +++ b/frontend/src/lib/components/triggers/http/RouteEditorInner.svelte @@ -230,12 +230,17 @@ edit = true dirtyPath = false dirtyRoutePath = false - await loadTrigger(defaultConfig) + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getRouteConfig())) as NewHttpTrigger + if (draftOverlay) loadTriggerConfig(draftOverlay as Partial) if (!defaultConfig) { // If the route is loaded from the backend, we to set the initial config - initialConfig = structuredClone($state.snapshot(getRouteConfig())) + initialConfig = structuredClone($state.snapshot(getRouteConfig())) as NewHttpTrigger } - originalConfig = structuredClone($state.snapshot(getRouteConfig())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load route: ${err}`, true) @@ -339,17 +344,26 @@ preservePermissionedAs = !!cfg?.permissioned_as } - async function loadTrigger(defaultConfig?: Partial): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Partial + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await HttpTriggerService.getHttpTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await HttpTriggerService.getHttpTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } diff --git a/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditor.svelte b/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditor.svelte index dac8ec1dc7..1dba5ad4cd 100644 --- a/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditor.svelte @@ -5,10 +5,10 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditorInner.svelte index 293421db42..afdd9d7746 100644 --- a/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/kafka/KafkaTriggerEditorInner.svelte @@ -4,8 +4,7 @@ import Drawer from '$lib/components/common/drawer/Drawer.svelte' import DrawerContent from '$lib/components/common/drawer/DrawerContent.svelte' import Path from '$lib/components/Path.svelte' - import Required from '$lib/components/Required.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import { KafkaTriggerService, type ErrorHandler, type Retry, type TriggerMode } from '$lib/gen' import { usedTriggerKinds, userStore, workspaceStore } from '$lib/stores' import { canWrite, capitalize, emptyString, sendUserToast } from '$lib/utils' @@ -147,7 +146,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultConfig?: Record + defaultConfig?: Record, + fixedScriptPath_?: string ) { let loadingTimeout = setTimeout(() => { showLoading = true @@ -159,11 +159,17 @@ itemKind = isFlow ? 'flow' : 'script' edit = true dirtyPath = false - await loadTrigger(defaultConfig) + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getSaveCfg())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultConfig) { initialConfig = structuredClone($state.snapshot(getSaveCfg())) } - originalConfig = structuredClone($state.snapshot(getSaveCfg())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load Kafka trigger: ${err}`, true) @@ -245,16 +251,26 @@ preservePermissionedAs = !!cfg?.permissioned_as } - async function loadTrigger(defaultConfig?: Record): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await KafkaTriggerService.getKafkaTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await KafkaTriggerService.getKafkaTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } @@ -507,32 +523,27 @@ {#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
- - {#if emptyString(script_path)} - - {/if} -
+ + {#snippet createButton()} + {#if emptyString(script_path)} + + {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditor.svelte b/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditor.svelte index ac74859255..ab96f0a909 100644 --- a/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditor.svelte @@ -5,10 +5,10 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditorInner.svelte index f91ac8b23d..561608bd9f 100644 --- a/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/mqtt/MqttTriggerEditorInner.svelte @@ -4,8 +4,7 @@ import Drawer from '$lib/components/common/drawer/Drawer.svelte' import DrawerContent from '$lib/components/common/drawer/DrawerContent.svelte' import Path from '$lib/components/Path.svelte' - import Required from '$lib/components/Required.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import { usedTriggerKinds, userStore, workspaceStore } from '$lib/stores' import { canWrite, capitalize, emptyString, sendUserToast } from '$lib/utils' import { withForkConflictRetry } from '$lib/utils/forkConflict' @@ -143,7 +142,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultConfig?: Record + defaultConfig?: Record, + fixedScriptPath_?: string ) { let loadingTimeout = setTimeout(() => { showLoading = true @@ -155,11 +155,17 @@ itemKind = isFlow ? 'flow' : 'script' edit = true dirtyPath = false - await loadTrigger(defaultConfig) + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getSaveCfg())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultConfig) { initialConfig = structuredClone($state.snapshot(getSaveCfg())) } - originalConfig = structuredClone($state.snapshot(getSaveCfg())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load mqtt trigger: ${err.body}`, true) @@ -244,20 +250,31 @@ } } - async function loadTrigger(defaultConfig?: Record): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { try { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await MqttTriggerService.getMqttTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await MqttTriggerService.getMqttTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } catch (error) { sendUserToast(`Could not load mqtt trigger: ${error.body}`, true) + return { overlay: undefined, noDeployed: false } } } @@ -481,34 +498,29 @@ {#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
- - {#if emptyString(script_path)} - - {/if} -
+ + {#snippet createButton()} + {#if emptyString(script_path)} + + {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/nats/NatsTriggerEditor.svelte b/frontend/src/lib/components/triggers/nats/NatsTriggerEditor.svelte index da06a82898..2beb6cb815 100644 --- a/frontend/src/lib/components/triggers/nats/NatsTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/nats/NatsTriggerEditor.svelte @@ -5,10 +5,10 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/nats/NatsTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/nats/NatsTriggerEditorInner.svelte index 60aa6c4478..4450cd6891 100644 --- a/frontend/src/lib/components/triggers/nats/NatsTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/nats/NatsTriggerEditorInner.svelte @@ -3,8 +3,7 @@ import Drawer from '$lib/components/common/drawer/Drawer.svelte' import DrawerContent from '$lib/components/common/drawer/DrawerContent.svelte' import Path from '$lib/components/Path.svelte' - import Required from '$lib/components/Required.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import { NatsTriggerService, type ErrorHandler, type Retry, type TriggerMode } from '$lib/gen' import { usedTriggerKinds, userStore, workspaceStore } from '$lib/stores' import { canWrite, capitalize, emptyString, sendUserToast } from '$lib/utils' @@ -131,7 +130,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultConfig?: Record + defaultConfig?: Record, + fixedScriptPath_?: string ) { let loadingTimeout = setTimeout(() => { showLoading = true @@ -143,11 +143,23 @@ itemKind = isFlow ? 'flow' : 'script' edit = true dirtyPath = false - await loadTrigger(defaultConfig) + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + // At this point the form holds the DEPLOYED config (or + // `defaultConfig` for new triggers). Capture `originalConfig` + // here so `hasChanged` (= `current != originalConfig`) compares + // against the deployed baseline; if a draft exists, applying + // the overlay below makes `current != originalConfig` fire + // the "unsaved changes" banner immediately. + originalConfig = structuredClone($state.snapshot(getSaveCfg())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultConfig) { initialConfig = structuredClone($state.snapshot(getSaveCfg())) } - originalConfig = structuredClone($state.snapshot(getSaveCfg())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load nats trigger: ${err}`, true) @@ -227,16 +239,34 @@ preservePermissionedAs = !!cfg?.permissioned_as } - async function loadTrigger(defaultConfig?: Record): Promise { + /** + * Apply the deployed config to the form, then return the saved-draft + * overlay (if any) so the caller can capture the deployed-only form + * state as `originalConfig` BEFORE applying the draft. The + * "unsaved changes" banner compares `current` vs `originalConfig`, + * so capturing originalConfig from the deployed-only form makes the + * banner fire whenever a draft is present (instead of only after + * the user starts editing on top of the draft). + */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await NatsTriggerService.getNatsTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await NatsTriggerService.getNatsTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } @@ -463,33 +493,28 @@
{#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
- - {#if emptyString(script_path)} - - {/if} -
+ + {#snippet createButton()} + {#if emptyString(script_path)} + + {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditor.svelte b/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditor.svelte index e73becdd56..3165abe9dc 100644 --- a/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditor.svelte @@ -5,10 +5,10 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditorInner.svelte index 3fc92d5e79..6420d5f9b1 100644 --- a/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/postgres/PostgresTriggerEditorInner.svelte @@ -4,7 +4,7 @@ import DrawerContent from '$lib/components/common/drawer/DrawerContent.svelte' import Path from '$lib/components/Path.svelte' import Required from '$lib/components/Required.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import { PostgresTriggerService, type ErrorHandler, @@ -231,7 +231,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultConfig?: Record + defaultConfig?: Record, + fixedScriptPath_?: string ) { let loadingTimeout = setTimeout(() => { showLoading = true @@ -250,11 +251,15 @@ relations = [] transaction_to_track = [] tab = 'basic' - await loadTrigger(defaultConfig) + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers have no deployed row, so saving must CREATE (update 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getSaveCfg())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultConfig) { initialConfig = structuredClone($state.snapshot(getSaveCfg())) } - originalConfig = structuredClone($state.snapshot(getSaveCfg())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load postgres trigger: ${err.body}`, true) @@ -367,28 +372,55 @@ preservePermissionedAs = !!cfg?.permissioned_as } - async function loadTrigger(defaultConfig?: Record): Promise { + /** + * Apply the deployed config to the form, then return the saved-draft overlay + * so the caller captures `originalConfig` BEFORE applying the draft (see + * `NatsTriggerEditorInner.loadTrigger`). Postgres wrinkle: the publication is + * keyed by resource/name the draft may have changed, so it's fetched using + * the effective values to reflect the draft's view. + */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) if (defaultConfig?.publication) { transaction_to_track = [...defaultConfig.publication.transaction_to_track] relations = defaultConfig.publication.table_to_track ?? [] } - return - } else { - const s = await PostgresTriggerService.getPostgresTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - - const publication_data = await PostgresTriggerService.getPostgresPublication({ - path: s.postgres_resource_path, - workspace: $workspaceStore!, - publication: s.publication_name - }) - - loadTriggerConfig({ ...s, publication: publication_data }) + return { overlay: undefined, noDeployed: false } } + const s = await PostgresTriggerService.getPostgresTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + // Draft-only: synthesized stand-in, no row, so saving must CREATE. + const noDeployed = !!(s as any)?.no_deployed + + // Deployed config + publication become the `originalConfig` baseline. + const deployedPublication = await PostgresTriggerService.getPostgresPublication({ + path: deployedTrigger.postgres_resource_path, + workspace: $workspaceStore!, + publication: deployedTrigger.publication_name + }) + loadTriggerConfig({ ...deployedTrigger, publication: deployedPublication }) + + if (!draftFromBackend) return { overlay: undefined, noDeployed } + + // Refetch the publication for the draft's keys if they differ. + const effective = { ...deployedTrigger, ...draftFromBackend } + const effectivePublication = + effective.postgres_resource_path === deployedTrigger.postgres_resource_path && + effective.publication_name === deployedTrigger.publication_name + ? deployedPublication + : await PostgresTriggerService.getPostgresPublication({ + path: effective.postgres_resource_path, + workspace: $workspaceStore!, + publication: effective.publication_name + }) + return { overlay: { ...effective, publication: effectivePublication }, noDeployed } } function getCaptureConfig() { @@ -639,43 +671,39 @@ {#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
- - - {#if emptyString(script_path) && is_flow === false} -
- -
- {/if} -
+ + {#snippet createButton()} + {#if emptyString(script_path) && is_flow === false} +
+ +
+ {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/schedules/ScheduleEditor.svelte b/frontend/src/lib/components/triggers/schedules/ScheduleEditor.svelte index c75e215e79..a0010829a5 100644 --- a/frontend/src/lib/components/triggers/schedules/ScheduleEditor.svelte +++ b/frontend/src/lib/components/triggers/schedules/ScheduleEditor.svelte @@ -5,20 +5,21 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( is_flow: boolean, initial_script_path?: string, - schedule_path?: string + schedule_path?: string, + fixedScriptPath?: string ) { open = true await tick() - drawer?.openNew(is_flow, initial_script_path, undefined, schedule_path) + drawer?.openNew(is_flow, initial_script_path, undefined, schedule_path, fixedScriptPath) } let drawer: ScheduleEditorInner | undefined = $state() diff --git a/frontend/src/lib/components/triggers/schedules/ScheduleEditorInner.svelte b/frontend/src/lib/components/triggers/schedules/ScheduleEditorInner.svelte index 891b931536..0d7198bf86 100644 --- a/frontend/src/lib/components/triggers/schedules/ScheduleEditorInner.svelte +++ b/frontend/src/lib/components/triggers/schedules/ScheduleEditorInner.svelte @@ -8,6 +8,7 @@ import LabelsInput from '$lib/components/LabelsInput.svelte' import Required from '$lib/components/Required.svelte' import ScriptPicker from '$lib/components/ScriptPicker.svelte' + import PipelineLockedRunnableInfo from '$lib/components/triggers/PipelineLockedRunnableInfo.svelte' import ErrorOrRecoveryHandler from '$lib/components/ErrorOrRecoveryHandler.svelte' import Toggle from '$lib/components/Toggle.svelte' import Tooltip from '$lib/components/Tooltip.svelte' @@ -96,6 +97,10 @@ let dynamicSkipPath: string | undefined = $state(undefined) let script_path = $state('') let initialScriptPath = $state('') + // When non-empty, the drawer was opened from the pipeline editor for an + // already-bound script. We swap the runnable ScriptPicker for a read-only + // viewer so the trigger can't be silently reassigned off the pipeline. + let fixedScriptPath = $state('') let runnable: Script | Flow | undefined = $state() let args: Record = $state({}) let loading = $state(false) @@ -142,7 +147,12 @@ deployed: () => initialConfig }) - export async function openEdit(ePath: string, isFlow: boolean, defaultCfg?: Record) { + export async function openEdit( + ePath: string, + isFlow: boolean, + defaultCfg?: Record, + fixedScriptPath_?: string + ) { let loadingTimeout = setTimeout(() => { showLoading = true }, 100) // Do not show loading spinner for the first 100ms @@ -152,11 +162,16 @@ initialPath = ePath itemKind = isFlow ? 'flow' : 'script' path = defaultCfg?.path ?? ePath - await loadSchedule(defaultCfg) - edit = true + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadSchedule(defaultCfg) + // Draft-only schedules have no deployed row, so saving must CREATE (update 404s). + edit = !noDeployed if (!defaultCfg) { + // Form holds DEPLOYED here; capture it as `initialConfig` so the + // dirty check / banner fires whenever a saved draft exists. initialConfig = structuredClone($state.snapshot(getScheduleCfg())) } + if (draftOverlay) await loadScheduleCfg(draftOverlay) await draftSync.maybeRestore() } finally { clearTimeout(loadingTimeout) @@ -277,8 +292,11 @@ nis_flow: boolean, initial_script_path?: string, defaultValues?: Schedule, - schedule_path?: string + schedule_path?: string, + fixedScriptPath_?: string, + opts: { getDraft?: boolean } = {} ) { + const getDraft = opts.getDraft ?? true let loadingTimeout = setTimeout(() => { showLoading = true }, 100) // Do not show loading spinner for the first 100ms @@ -286,10 +304,17 @@ try { let s: Schedule | undefined if (schedule_path) { - s = await ScheduleService.getSchedule({ + const resp = await ScheduleService.getSchedule({ workspace: $workspaceStore!, - path: schedule_path + path: schedule_path, + getDraft }) + // `.draft` holds the saved Schedule; layer it over the deployed + // fields so the form assignments below see the last-saved state. + const { draft: draftFromBackend, ...deployedSchedule } = resp as any + s = draftFromBackend + ? ({ ...deployedSchedule, ...draftFromBackend } as Schedule) + : (deployedSchedule as Schedule) initNewPath = true } else if (defaultValues) { s = defaultValues @@ -304,6 +329,7 @@ initialConfig = undefined itemKind = (s?.is_flow ?? nis_flow) ? 'flow' : 'script' initialScriptPath = initial_script_path ?? '' + fixedScriptPath = fixedScriptPath_ ?? '' path = initNewPath ? '' : (defaultValues?.path ?? (trigger?.isPrimary ? initialScriptPath : '')) @@ -452,19 +478,36 @@ } } - async function loadSchedule(defaultCfg?: Record): Promise { - if (!defaultCfg) { - try { - const s = await ScheduleService.getSchedule({ - workspace: $workspaceStore!, - path: initialPath - }) - await loadScheduleCfg(s) - } catch (err) { - sendUserToast(`Could not load schedule: ${err}`, true) - } - } else { + /** + * Apply the deployed config to the form, then return the saved-draft overlay + * so the caller captures `initialConfig` from the deployed-only form BEFORE + * applying the draft, making the banner fire whenever a draft is present. + */ + async function loadSchedule( + defaultCfg?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { + if (defaultCfg) { await loadScheduleCfg(defaultCfg) + return { overlay: undefined, noDeployed: false } + } + try { + const s = await ScheduleService.getSchedule({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedSchedule } = s as any + await loadScheduleCfg(deployedSchedule) + return { + overlay: draftFromBackend + ? ({ ...deployedSchedule, ...draftFromBackend } as Record) + : undefined, + // Draft-only: synthesized stand-in, no row, so saving must CREATE. + noDeployed: !!(s as any).no_deployed + } + } catch (err) { + sendUserToast(`Could not load schedule: ${err}`, true) + return { overlay: undefined, noDeployed: false } } } @@ -880,7 +923,9 @@
{#if !hideTarget} - {#if !edit} + {#if fixedScriptPath != ''} + + {:else if !edit}

Pick a script or flow to be triggered by the schedule

diff --git a/frontend/src/lib/components/triggers/sqs/SqsTriggerEditor.svelte b/frontend/src/lib/components/triggers/sqs/SqsTriggerEditor.svelte index f60dfc83a5..e27f78e792 100644 --- a/frontend/src/lib/components/triggers/sqs/SqsTriggerEditor.svelte +++ b/frontend/src/lib/components/triggers/sqs/SqsTriggerEditor.svelte @@ -5,10 +5,10 @@ let { onUpdate }: { onUpdate?: (path?: string) => void } = $props() let open = $state(false) - export async function openEdit(ePath: string, isFlow: boolean) { + export async function openEdit(ePath: string, isFlow: boolean, fixedScriptPath?: string) { open = true await tick() - drawer?.openEdit(ePath, isFlow) + drawer?.openEdit(ePath, isFlow, undefined, fixedScriptPath) } export async function openNew( diff --git a/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte index b202c2d6b2..6ca2d13ddd 100644 --- a/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/sqs/SqsTriggerEditorInner.svelte @@ -17,8 +17,7 @@ } from '$lib/gen' import SqsTriggerEditorConfigSection from './SqsTriggerEditorConfigSection.svelte' import Section from '$lib/components/Section.svelte' - import ScriptPicker from '$lib/components/ScriptPicker.svelte' - import Required from '$lib/components/Required.svelte' + import TriggerRunnablePicker from '$lib/components/triggers/TriggerRunnablePicker.svelte' import { untrack, type Snippet } from 'svelte' import TriggerEditorToolbar from '../TriggerEditorToolbar.svelte' import PermissionedAsLine from '../PermissionedAsLine.svelte' @@ -126,7 +125,8 @@ export async function openEdit( ePath: string, isFlow: boolean, - defaultConfig?: Record + defaultConfig?: Record, + fixedScriptPath_?: string ) { let loadingTimeout = setTimeout(() => { showLoading = true @@ -138,14 +138,21 @@ itemKind = isFlow ? 'flow' : 'script' edit = true dirtyPath = false - await loadTrigger(defaultConfig) - // Snapshot the *backend* config as the baseline before overlaying - // any local autosave, so hasChanged / onConfigChange correctly - // flag the local edits as unsaved changes. + fixedScriptPath = fixedScriptPath_ ?? '' + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + // Snapshot the *deployed* config as the baseline before + // overlaying the saved draft, so hasChanged compares + // draft-vs-deployed and the banner fires whenever a draft + // exists. + originalConfig = structuredClone($state.snapshot(getSaveCfg())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultConfig) { initialConfig = structuredClone($state.snapshot(getSaveCfg())) } - originalConfig = structuredClone($state.snapshot(getSaveCfg())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load sqs trigger: ${err.body}`, true) @@ -221,20 +228,31 @@ } } - async function loadTrigger(defaultConfig?: Record): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { try { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await SqsTriggerService.getSqsTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await SqsTriggerService.getSqsTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } catch (error) { sendUserToast(`Could not load SQS trigger: ${error.body}`, true) + return { overlay: undefined, noDeployed: false } } } @@ -459,34 +477,30 @@ {#if !hideTarget}
-

- Pick a script or flow to be triggered -

-
- - {#if emptyString(script_path)} - - {/if} -
+ + {#snippet createButton()} + {#if emptyString(script_path)} + + {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts b/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts index 88c1d62232..aba27de99f 100644 --- a/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts +++ b/frontend/src/lib/components/triggers/useTriggerDraftSync.svelte.ts @@ -1,8 +1,24 @@ import { untrack } from 'svelte' -import { UserDraft, localDraftDiffers, type UserDraftItemKind } from '$lib/userDraft.svelte' +import { deepEqual } from 'fast-equals' +import { UserDraft, normalizeDraftForCompare, type UserDraftItemKind } from '$lib/userDraft.svelte' +import { setLocalDraftHint } from '$lib/localDraftHints.svelte' type Cfg = Record +/** + * Whether `a` differs from `b` after `normalizeDraftForCompare` (JSON + * round-trip to drop `undefined`-valued keys, plus ignored deploy-directive + * fields). Nullish `a` returns `false` ("no draft" = "no divergence"). A + * `true` result narrows `a` to non-nullish `V`. + */ +function cfgDiffers(a: V | undefined | null, b: V | undefined): a is V { + if (a === undefined || a === null) return false + return !deepEqual( + normalizeDraftForCompare(a), + b === undefined ? undefined : normalizeDraftForCompare(b) + ) +} + export interface TriggerDraftSyncOptions { /** UserDraft item kind for this trigger, e.g. `'trigger_postgres'`. */ itemKind: UserDraftItemKind @@ -66,7 +82,7 @@ export interface TriggerDraftSync { * - **persist-effect**: writes form edits back through the handle, dropping * the draft when the form is back at the deployed baseline. * - * Both effect bodies are `untrack`ed and gated by `localDraftDiffers` + * Both effect bodies are `untrack`ed and gated by `cfgDiffers` * idempotence so they can't feed back into each other. Must be called once * during component init (it registers `useMany` + two `$effect`s). */ @@ -85,12 +101,18 @@ export function useTriggerDraftSync(opts: TriggerDraftSyncOptions): TriggerDraft const hasDraft = $derived( !opts.drawerLoading() && opts.deployed() != null && - localDraftDiffers(opts.getCfg() as Cfg, opts.deployed() as Cfg) + cfgDiffers(opts.getCfg() as Cfg, opts.deployed() as Cfg) ) - function discard(path: string, fallback: Cfg | undefined): void { + /** `auto: true` marks a discard from the reactive persist-effect (not an + * explicit user action), so it respects the "Enable auto-save" toggle — else + * with autosave off the editor would delete server drafts while writing none. */ + function discard(path: string, fallback: Cfg | undefined, auto = false): void { + // `UserDraft.discard` POSTs `value: null`, which clears the list-page + // `*` hint via the syncer. No explicit clear needed. UserDraft.discard(opts.itemKind, path, fallback, { - workspace: opts.workspace() ?? undefined + workspace: opts.workspace() ?? undefined, + auto }) } @@ -99,12 +121,33 @@ export function useTriggerDraftSync(opts: TriggerDraftSyncOptions): TriggerDraft const d = handle?.draft if (opts.drawerLoading() || d == null) return untrack(() => { - if (localDraftDiffers(d, opts.getCfg() as Cfg)) { + if (cfgDiffers(d, opts.getCfg() as Cfg)) { void opts.applyCfg(d) } }) }) + /** Deferred + revalidated auto-discard. "At baseline but a draft exists" + * also occurs transiently during programmatic churn (list pages fire + * `openEdit` twice per row click), where an immediate discard would delete a + * draft the user never touched. Re-checking after a settle window keeps the + * legit revert case while the churn case reconciles before the timer fires. */ + let discardTimer: ReturnType | undefined + function scheduleAutoDiscard() { + if (discardTimer) return + discardTimer = setTimeout(() => { + discardTimer = undefined + if (opts.drawerLoading()) return + const cfg = opts.getCfg() + const deployed = opts.deployed() + const h = handle + if (!h || cfg == null) return + if (!cfgDiffers(cfg, deployed) && cfgDiffers(h.draft, deployed)) { + discard(opts.path(), deployed, true) + } + }, 600) + } + // persist-effect: form edits → handle; drop the draft when back at the // deployed baseline. $effect(() => { @@ -115,14 +158,30 @@ export function useTriggerDraftSync(opts: TriggerDraftSyncOptions): TriggerDraft const h = handle if (!h) return const deployed = opts.deployed() - if (localDraftDiffers(cfg, deployed)) { - if (localDraftDiffers(cfg, h.draft)) h.draft = cfg - } else { - discard(opts.path(), deployed) + if (cfgDiffers(cfg, deployed)) { + if (cfgDiffers(cfg, h.draft)) h.draft = cfg + } else if (cfgDiffers(h.draft, deployed)) { + // Only when a draft actually exists to drop: `h.draft` equals + // `deployed` right after a discard or the post-load seed. + scheduleAutoDiscard() } }) }) + // The list-page `*` hint is owned by UserDraftDbSyncer. This effect only + // CLEARS it (never sets): while settled and at baseline, drop any stale + // hint, so a draft discarded in another tab disappears on reopen. + $effect(() => { + const ws = opts.workspace() + const p = opts.path() + const loading = opts.drawerLoading() + const dirty = hasDraft + untrack(() => { + if (!ws || !p || loading) return + if (!dirty) setLocalDraftHint(ws, opts.itemKind, p, false) + }) + }) + return { get draft() { return handle?.draft @@ -138,9 +197,22 @@ export function useTriggerDraftSync(opts: TriggerDraftSyncOptions): TriggerDraft }, async maybeRestore() { const d = handle?.draft - if (!localDraftDiffers(d, opts.getCfg() as Cfg)) return - // Overlay the local autosave on the just-loaded backend config. - await opts.applyCfg(d) + if (cfgDiffers(d, opts.getCfg() as Cfg)) { + // Overlay the local autosave on the just-loaded backend config. + await opts.applyCfg(d) + } + // Adopt the post-load form state as the cell's baseline without + // POSTing, consuming the entry's one-shot first-write seed guard. + // Trigger drawers never write the cell programmatically on open, so + // without this the guard would swallow the user's first edit. + const ws = opts.workspace() + const p = opts.path() + const cfg = opts.getCfg() + if (ws && p && cfg != null) { + UserDraft.seed(opts.itemKind, p, structuredClone($state.snapshot(cfg)) as Cfg, { + workspace: ws + }) + } }, async resetToDeployed(path: string) { const deployedCfg = structuredClone($state.snapshot(opts.deployed())) as Cfg diff --git a/frontend/src/lib/components/triggers/utils.ts b/frontend/src/lib/components/triggers/utils.ts index 711d84aea9..231de5cea4 100644 --- a/frontend/src/lib/components/triggers/utils.ts +++ b/frontend/src/lib/components/triggers/utils.ts @@ -81,7 +81,8 @@ export const jobTriggerKinds: JobTriggerKind[] = [ 'gcp', 'azure', 'google', - 'github' + 'github', + 'asset' ] export type Trigger = { @@ -650,71 +651,8 @@ export function sortTriggers(triggers: Trigger[]): Trigger[] { }) } -export type FlowWithDraftAndDraftTriggers = Flow & { - draft?: Flow & { - draft_triggers?: Trigger[] - } -} - -export type NewScriptWithDraftAndDraftTriggers = NewScript & { - draft?: NewScript & { draft_triggers?: Trigger[] } - hash: string -} - -// Get rid of deployed triggers from the saved flow in the case there is a match with a deployed trigger -export function filterDraftTriggers( - savedValue: FlowWithDraftAndDraftTriggers | NewScriptWithDraftAndDraftTriggers, - triggersState: Triggers -): FlowWithDraftAndDraftTriggers | NewScriptWithDraftAndDraftTriggers { - const deployedTriggers = triggersState.triggers.filter((t) => !t.draftConfig && !t.isDraft) - - // Early return if no deployed triggers or no draft triggers to filter - if (deployedTriggers.length === 0 || !savedValue?.draft?.draft_triggers?.length) { - return savedValue - } - - const deployedTriggerKeys = new Set(deployedTriggers.map((t) => `${t.path}:${t.type}`)) - - const originalSavedDraftTriggers = savedValue.draft.draft_triggers - const keptTriggers: Trigger[] = [] - const removedTriggers: Trigger[] = [] - - // Single pass to separate kept vs removed triggers - for (const savedTrigger of originalSavedDraftTriggers) { - const triggerKey = `${savedTrigger.draftConfig?.path}:${savedTrigger.type}` - if (deployedTriggerKeys.has(triggerKey)) { - removedTriggers.push(savedTrigger) - } else { - keptTriggers.push(savedTrigger) - } - } - - // Early return if nothing was filtered - if (removedTriggers.length === 0) { - return savedValue - } - - // Update saved value - const newSavedValue = { - ...savedValue, - draft: { - ...savedValue.draft, - draft_triggers: keptTriggers.length > 0 ? keptTriggers : undefined - } - } as typeof savedValue - - const removedTriggerKeys = new Set(removedTriggers.map((t) => `${t.draftConfig?.path}:${t.type}`)) - - // Remove filtered triggers from triggersState - triggersState.setTriggers( - triggersState.triggers.filter((trigger) => { - const triggerKey = `${trigger.draftConfig?.path}:${trigger.type}` - return !removedTriggerKeys.has(triggerKey) - }) - ) - - return newSavedValue -} +export type FlowWithDraftAndDraftTriggers = Flow +export type NewScriptWithDraftAndDraftTriggers = NewScript & { hash?: string } export function getHandlerType(scriptPath: string): ErrorHandler { const handlerMap = { diff --git a/frontend/src/lib/components/triggers/webhook/WebhookEditor.svelte b/frontend/src/lib/components/triggers/webhook/WebhookEditor.svelte new file mode 100644 index 0000000000..031da4aacf --- /dev/null +++ b/frontend/src/lib/components/triggers/webhook/WebhookEditor.svelte @@ -0,0 +1,66 @@ + + +{#if open} + (open = false)}> + drawer?.closeDrawer()}> + + + +{/if} diff --git a/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte b/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte index 0567071bae..77b5149933 100644 --- a/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte +++ b/frontend/src/lib/components/triggers/websocket/WebsocketTriggerEditorInner.svelte @@ -187,11 +187,16 @@ edit = true dirtyPath = false dirtyUrl = false - await loadTrigger(defaultConfig) + const { overlay: draftOverlay, noDeployed } = await loadTrigger(defaultConfig) + // Draft-only triggers open as "new trigger prefilled from the + // draft" — no deployed row exists, so saving must CREATE (the + // update endpoint 404s). + edit = !noDeployed + originalConfig = structuredClone($state.snapshot(getSaveCfg())) + if (draftOverlay) loadTriggerConfig(draftOverlay) if (!defaultConfig) { initialConfig = structuredClone($state.snapshot(getSaveCfg())) } - originalConfig = structuredClone($state.snapshot(getSaveCfg())) await draftSync.maybeRestore() } catch (err) { sendUserToast(`Could not load websocket trigger: ${err}`, true) @@ -302,16 +307,26 @@ } } - async function loadTrigger(defaultConfig?: Record): Promise { + /** See `NatsTriggerEditorInner.loadTrigger` for the rationale. */ + async function loadTrigger( + defaultConfig?: Record + ): Promise<{ overlay: Record | undefined; noDeployed: boolean }> { if (defaultConfig) { loadTriggerConfig(defaultConfig) - return - } else { - const s = await WebsocketTriggerService.getWebsocketTrigger({ - workspace: $workspaceStore!, - path: initialPath - }) - loadTriggerConfig(s) + return { overlay: undefined, noDeployed: false } + } + const s = await WebsocketTriggerService.getWebsocketTrigger({ + workspace: $workspaceStore!, + path: initialPath, + getDraft: true + }) + const { draft: draftFromBackend, ...deployedTrigger } = (s ?? {}) as any + loadTriggerConfig(deployedTrigger) + return { + noDeployed: !!(s as any)?.no_deployed, + overlay: draftFromBackend + ? ({ ...deployedTrigger, ...draftFromBackend } as Record) + : undefined } } diff --git a/frontend/src/lib/components/usePageDraftSync.svelte.ts b/frontend/src/lib/components/usePageDraftSync.svelte.ts new file mode 100644 index 0000000000..8ca2b0beee --- /dev/null +++ b/frontend/src/lib/components/usePageDraftSync.svelte.ts @@ -0,0 +1,120 @@ +import { untrack } from 'svelte' +import { UserDraft, type UserDraftHandle, type UserDraftItemKind } from '$lib/userDraft.svelte' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' + +/** + * Page-level draft orchestration for the full-page editors (scripts / flows / + * apps / apps_raw) — the page analogue of `useTriggerDraftSync`. Owns the + * per-path autosave handle (re-keyed on navigation), the live-editor-draft + * registry entry (home-page "edit draft" links), `recordRemoteSync` (a method + * so pages can't forget it and silently downgrade conflict detection to + * overwrite-as-fresh), the deployed-baseline `seed`, and the draft `remove`. + * The entity-specific backend load and new-draft template stay in the page. + */ +export interface PageDraftSyncOptions { + itemKind: UserDraftItemKind + /** Reactive draft storage path. `''` (e.g. viewing a historical hash) + * releases the handle and skips registry/sync work. */ + path: () => string + /** Reactive workspace (`$workspaceStore`). */ + workspace: () => string | undefined + /** Reactive effective path for the live-editor-draft registry (the + * draft's own `path`, used by home-page deep links). Omit to skip + * registry registration entirely (e.g. read-only hash views). */ + effectivePath?: () => string | undefined + /** Predicate: is the value about to autosave back at the deployed + * baseline? When true the syncer POSTs a delete instead of a + * baseline-equal draft, so editing back to deployed clears the draft + * instead of leaving a no-op behind. MUST read the deployed baseline + * reactively (it's captured once per re-keyed acquire) and use + * `draftValuesEqual` so it can't disagree with the "unsaved changes" + * banner. Return false for draft-only items (no deployed baseline). */ + discardIf?: (val: V) => boolean + /** Seeds the cell on first acquire without POSTing (the syncer's seed + * guard swallows it). Use when the value is already in hand at mount (an + * embedder providing the item) instead of assigning `draft` after load. + * Pass a STABLE reference (read it under `untrack`). */ + defaultValue?: V +} + +export interface PageDraftSync { + /** The live autosave handle. `draft` is the editor's bind target. */ + readonly handle: UserDraftHandle + get draft(): V | undefined + set draft(value: V | undefined) + /** Load `value` as the deployed/template baseline WITHOUT POSTing it + * (see `UserDraft.seed`). */ + seedBaseline(value: V): void + /** After a backend load, record the server's `draft_saved_at` so the + * next autosave attaches a matching `last_sync` and the server can + * reject stale writes. `undefined` clears it (no draft existed). */ + recordRemoteSync(draftSavedAt: string | undefined): void + /** Drop the draft (server row + local cell) — restore-to-deployed and + * post-deploy cleanup. */ + remove(): void +} + +export function usePageDraftSync(opts: PageDraftSyncOptions): PageDraftSync { + // One handle, re-keyed on (workspace, path); `''` path releases it. + // `canBeDisabled` because these editors carry the "Enable auto-save" toggle. + const handle = UserDraft.useReactive(() => ({ + itemKind: opts.itemKind, + path: opts.path(), + workspace: opts.workspace(), + canBeDisabled: true, + defaultValue: opts.defaultValue, + discardIf: opts.discardIf + })) + + // Live-editor-draft registry: lets the home-page "edit draft" link resolve + // this open editor. Re-registers on path change; cleared on teardown. + $effect(() => { + if (!opts.effectivePath) return + const ws = opts.workspace() + const p = opts.path() + const eff = opts.effectivePath() + if (!ws || !p) return + UserDraft.setLiveEditorDraft({ + workspace: ws, + itemKind: opts.itemKind, + storagePath: p, + effectivePath: eff || p + }) + return () => UserDraft.clearLiveEditorDraft(opts.itemKind, { workspace: ws, storagePath: p }) + }) + + return { + get handle() { + return handle + }, + get draft() { + return handle.draft + }, + set draft(value: V | undefined) { + handle.draft = value + }, + seedBaseline(value: V) { + const ws = opts.workspace() + const p = opts.path() + if (!ws || !p) return + UserDraft.seed(opts.itemKind, p, value, { workspace: ws }) + }, + recordRemoteSync(draftSavedAt: string | undefined) { + const ws = opts.workspace() + const p = opts.path() + if (!ws || !p) return + untrack(() => + UserDraftDbSyncer.recordRemoteSync( + { workspace: ws, itemKind: opts.itemKind, path: p }, + draftSavedAt + ) + ) + }, + remove() { + const ws = opts.workspace() + const p = opts.path() + if (!ws || !p) return + UserDraft.remove(opts.itemKind, p, { workspace: ws }) + } + } +} diff --git a/frontend/src/lib/components/workspace/WorkspaceCard.svelte b/frontend/src/lib/components/workspace/WorkspaceCard.svelte index c4ac967b50..851743b56b 100644 --- a/frontend/src/lib/components/workspace/WorkspaceCard.svelte +++ b/frontend/src/lib/components/workspace/WorkspaceCard.svelte @@ -6,6 +6,7 @@ import type { UserWorkspace } from '$lib/stores' import { superadmin } from '$lib/stores' import { WorkspaceService } from '$lib/gen' + import { reconcileAfterWorkspaceChange } from '$lib/components/sessions/sessionState.svelte' import { pluralize } from '$lib/utils' import WorkspaceIcon from './WorkspaceIcon.svelte' import WorkspaceCard from './WorkspaceCard.svelte' @@ -64,6 +65,8 @@ if (onUnarchive) { await WorkspaceService.unarchiveWorkspace({ workspace: workspace.id }) await onUnarchive(workspace.id) + // Restore sessions auto-archived when this workspace was archived. + await reconcileAfterWorkspaceChange() } } diff --git a/frontend/src/lib/components/workspaceSettings/AISettings.svelte b/frontend/src/lib/components/workspaceSettings/AISettings.svelte index 2080ecd332..f44059e395 100644 --- a/frontend/src/lib/components/workspaceSettings/AISettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/AISettings.svelte @@ -13,6 +13,8 @@ import { supportsAutocomplete } from '../copilot/utils' import TestAiKey from '../copilot/TestAIKey.svelte' import Label from '../Label.svelte' + import AiSkillsSettings from './AiSkillsSettings.svelte' + import { isGlobalAiEnabled } from '../copilot/chat/global/gate' import SettingsPageHeader from '../settings/SettingsPageHeader.svelte' import ResourcePicker from '../ResourcePicker.svelte' import Toggle from '../Toggle.svelte' @@ -587,6 +589,10 @@
{/if} + + {#if promptScope === 'workspace' && isGlobalAiEnabled()} + + {/if}
+ import { onMount } from 'svelte' + import YAML from 'yaml' + import Button from '../common/button/Button.svelte' + import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte' + import SettingCard from '../instanceSettings/SettingCard.svelte' + import Label from '../Label.svelte' + import autosize from '$lib/autosize' + import { workspaceStore } from '$lib/stores' + import { sendUserToast } from '$lib/toast' + import { WorkspaceService } from '$lib/gen' + import { FolderUp, Plus, Trash2 } from 'lucide-svelte' + + type SkillListItem = { name: string; description: string } + type SkillUpload = { name: string; description: string; instructions: string } + + // `//SKILL.md` is 3 path segments; SKILL.md files nested deeper + // are likely vendored/incidental and are skipped so importing a parent dir + // doesn't sweep in unrelated skills. + const MAX_SKILL_DEPTH = 3 + const MAX_SKILLS_PER_IMPORT = 50 + const MAX_SKILLS_PER_WORKSPACE = 100 + // `name` + `description` mirror the Claude SKILL.md spec (counted in + // characters); the body is a byte-bounded payload. Keep these in sync with + // backend `validate_skill`. + const MAX_SKILL_NAME_LENGTH = 64 + const MAX_SKILL_DESCRIPTION_LENGTH = 1_024 + const MAX_SKILL_INSTRUCTIONS_LENGTH = 64 * 1024 + const SKILL_NAME_PATTERN = /^[a-z0-9-]+$/ + const textEncoder = new TextEncoder() + const SAMPLE_SKILL_PLACEHOLDER = + '---\nname: my-skill\ndescription: what this skill helps with\n---\n\n# My skill\n\nInstructions for the assistant…' + + let skills: SkillListItem[] = $state([]) + let uploading: boolean = $state(false) + let pasteContent: string = $state('') + let dirInput: HTMLInputElement | undefined = $state(undefined) + let toDelete: string | undefined = $state(undefined) + let pendingImport: SkillUpload[] | undefined = $state(undefined) + let pendingSkipped: string[] = $state([]) + let listRequestId = 0 + + let pendingNamesPreview = $derived.by(() => { + const p = pendingImport ?? [] + const shown = p + .slice(0, 12) + .map((s) => s.name) + .join(', ') + return p.length > 12 ? `${shown}, … (+${p.length - 12} more)` : shown + }) + + async function loadList(workspace: string | undefined) { + const requestId = ++listRequestId + if (!workspace) { + skills = [] + return + } + try { + const loaded = await WorkspaceService.listAiSkills({ workspace }) + if (requestId === listRequestId && workspace === $workspaceStore) { + skills = loaded + } + } catch (e) { + if (requestId === listRequestId && workspace === $workspaceStore) { + sendUserToast(`Failed to load skills: ${e}`, true) + } + } + } + + /** Split a SKILL.md into its frontmatter `name`/`description` and the markdown body. */ + function parseSkillMd(raw: string): { + name: string | undefined + description: string | undefined + instructions: string + } { + const text = raw.replace(/^/, '') + const fm = /^---\s*\r?\n([\s\S]*?)\r?\n---\s*\r?\n?/.exec(text) + if (!fm) { + return { name: undefined, description: undefined, instructions: text.trim() } + } + let name: string | undefined + let description: string | undefined + try { + const data = YAML.parse(fm[1]) ?? {} + if (typeof data?.name === 'string') name = data.name.trim() + if (typeof data?.description === 'string') description = data.description.trim() + } catch { + // Malformed frontmatter — fall through so the skill is reported as + // invalid rather than silently dropped. + } + return { name, description, instructions: text.slice(fm[0].length).trim() } + } + + function validateParsedSkill(skill: SkillUpload): string | undefined { + if ([...skill.name].length > MAX_SKILL_NAME_LENGTH) { + return `name is longer than ${MAX_SKILL_NAME_LENGTH} characters` + } + if (!SKILL_NAME_PATTERN.test(skill.name)) { + return `name ${JSON.stringify(skill.name)} must only contain lowercase letters, digits or '-'` + } + if ([...skill.description].length > MAX_SKILL_DESCRIPTION_LENGTH) { + return `description is longer than ${MAX_SKILL_DESCRIPTION_LENGTH} characters` + } + if (textEncoder.encode(skill.instructions).byteLength > MAX_SKILL_INSTRUCTIONS_LENGTH) { + return `body is longer than ${MAX_SKILL_INSTRUCTIONS_LENGTH} bytes` + } + } + + /** + * Turn a map of `relativePath -> content` (from an imported folder) into skills. + * A skill is any `SKILL.md`; its id is the name of the folder holding it. + */ + function collectSkills(files: Record): { + skills: SkillUpload[] + skipped: string[] + } { + const collected: SkillUpload[] = [] + const skipped: string[] = [] + for (const [path, content] of Object.entries(files)) { + const segments = path.split('/') + if (segments[segments.length - 1]?.toLowerCase() !== 'skill.md') continue + const name = segments.length >= 2 ? segments[segments.length - 2] : '' + const { description, instructions } = parseSkillMd(content) + if (!name) { + skipped.push(`${path} (SKILL.md must live in a named folder)`) + } else if (!description) { + skipped.push(`${name} (missing frontmatter description)`) + } else if (!instructions) { + skipped.push(`${name} (empty body)`) + } else { + const parsed = { name, description, instructions } + const validationError = validateParsedSkill(parsed) + if (validationError) { + skipped.push(`${name} (${validationError})`) + } else { + collected.push(parsed) + } + } + } + return { skills: collected, skipped } + } + + async function uploadSkills(parsed: SkillUpload[], skipped: string[] = []) { + const workspace = $workspaceStore + if (!workspace || parsed.length === 0) { + sendUserToast( + `No valid skill found.${skipped.length ? ` Skipped: ${skipped.join(', ')}` : ''}`, + true + ) + return false + } + if (parsed.length > MAX_SKILLS_PER_IMPORT) { + sendUserToast(`Cannot add more than ${MAX_SKILLS_PER_IMPORT} skills at a time.`, true) + return false + } + // Uploads upsert, so only names not already stored count toward the cap. + const existingNames = new Set(skills.map((s) => s.name)) + const newCount = parsed.filter((s) => !existingNames.has(s.name)).length + if (skills.length + newCount > MAX_SKILLS_PER_WORKSPACE) { + sendUserToast(`This workspace can store at most ${MAX_SKILLS_PER_WORKSPACE} skills.`, true) + return false + } + uploading = true + try { + await WorkspaceService.uploadAiSkills({ + workspace, + requestBody: { skills: parsed } + }) + let message = `Added ${parsed.length} skill(s)` + if (skipped.length) message += `; skipped ${skipped.length}: ${skipped.join(', ')}` + sendUserToast(message) + await loadList(workspace) + return true + } catch (e) { + sendUserToast(`Failed to add skills: ${e}`, true) + return false + } finally { + uploading = false + } + } + + async function addPastedSkill() { + const { name, description, instructions } = parseSkillMd(pasteContent) + if (!name) { + sendUserToast('The pasted SKILL.md needs a `name` in its frontmatter.', true) + return + } + if (!description) { + sendUserToast('The pasted SKILL.md needs a `description` in its frontmatter.', true) + return + } + if (!instructions) { + sendUserToast('The pasted SKILL.md has an empty body.', true) + return + } + const parsed = { name, description, instructions } + const validationError = validateParsedSkill(parsed) + if (validationError) { + sendUserToast(`The pasted SKILL.md ${validationError}.`, true) + return + } + if (await uploadSkills([parsed])) { + pasteContent = '' + } + } + + async function onDirSelected(event: Event) { + const target = event.target as HTMLInputElement + const files = Array.from(target.files ?? []) + // Reset early so re-selecting the same folder re-fires `change`. + if (dirInput) dirInput.value = '' + + // Pick SKILL.md files within the depth limit BEFORE reading any content, + // so a huge tree never gets read in full. + const skipped: string[] = [] + const eligible: File[] = [] + for (const f of files) { + const path = f.webkitRelativePath || f.name + const segments = path.split('/') + if (segments[segments.length - 1]?.toLowerCase() !== 'skill.md') continue + if (segments.length > MAX_SKILL_DEPTH) { + skipped.push(`${path} (nested deeper than ${MAX_SKILL_DEPTH} folder levels)`) + continue + } + eligible.push(f) + } + + if (eligible.length === 0) { + sendUserToast( + `No SKILL.md found within ${MAX_SKILL_DEPTH} folder levels.${ + skipped.length ? ` Skipped ${skipped.length} deeper file(s).` : '' + }`, + true + ) + return + } + if (eligible.length > MAX_SKILLS_PER_IMPORT) { + sendUserToast( + `Found ${eligible.length} skills in this folder; imports are limited to ${MAX_SKILLS_PER_IMPORT} at a time.`, + true + ) + return + } + + const map: Record = {} + for (const f of eligible) { + map[f.webkitRelativePath || f.name] = await f.text() + } + const { skills: parsed, skipped: parseSkipped } = collectSkills(map) + const allSkipped = [...skipped, ...parseSkipped] + if (parsed.length === 0) { + sendUserToast( + `No valid skill found.${allSkipped.length ? ` Skipped: ${allSkipped.join(', ')}` : ''}`, + true + ) + return + } + // Confirm before writing — the import can pull in several skills at once. + pendingSkipped = allSkipped + pendingImport = parsed + } + + async function deleteSkill(name: string) { + const workspace = $workspaceStore + if (!workspace) return + try { + await WorkspaceService.deleteAiSkill({ workspace, name }) + sendUserToast(`Deleted skill ${name}`) + await loadList(workspace) + } catch (e) { + sendUserToast(`Failed to delete skill: ${e}`, true) + } + } + + onMount(() => { + return workspaceStore.subscribe((workspace) => { + toDelete = undefined + pendingImport = undefined + pendingSkipped = [] + void loadList(workspace) + }) + }) + + + +
+ + + + + {#if skills.length > 0} +
+ {#each skills as skill (skill.name)} +
+
+
{skill.name}
+
{skill.description}
+
+
+ {/each} +
+ {/if} +
+
+ + { + const toImport = pendingImport + const skipped = pendingSkipped + pendingImport = undefined + pendingSkipped = [] + if (toImport) await uploadSkills(toImport, skipped) + }} + onCanceled={() => { + pendingImport = undefined + pendingSkipped = [] + }} +> + + Add {pendingImport?.length} skill(s) to the AI chat? + {pendingNamesPreview} + {#if pendingSkipped.length} +
{pendingSkipped.length} file(s) will be skipped. + {/if} +
+
+ + { + const name = toDelete + toDelete = undefined + if (name) await deleteSkill(name) + }} + onCanceled={() => (toDelete = undefined)} +> + + Delete the skill {toDelete}? The AI chat will no longer be able to use it. + + diff --git a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte index 024d94e8e6..30b050a9f6 100644 --- a/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte +++ b/frontend/src/lib/components/workspaceSettings/CreateWorkspaceInner.svelte @@ -22,6 +22,7 @@ import { sendUserToast } from '$lib/toast' import TestAIKey from '$lib/components/copilot/TestAIKey.svelte' import { switchWorkspace } from '$lib/storeUtils' + import { deleteSessionsForWorkspace } from '$lib/components/sessions/sessionState.svelte' import { isCloudHosted } from '$lib/cloud' import ToggleButtonGroup from '$lib/components/common/toggleButton-v2/ToggleButtonGroup.svelte' import ToggleButton from '$lib/components/common/toggleButton-v2/ToggleButton.svelte' @@ -100,6 +101,14 @@ deletingExistingFork = true try { await WorkspaceService.deleteWorkspace({ workspace: prefixedId }) + // Drop local sessions bound to this id so they don't resurface (or + // auto-unarchive) against a new fork recreated under the same id. + // Fire-and-forget: neither a slow nor a failing IndexedDB op should + // block the delete/reuse flow (cleanup completes long before the UI + // could create a session in a recreated fork). + void deleteSessionsForWorkspace(prefixedId).catch((e) => + console.error(`Session cleanup for reused fork id ${prefixedId} failed`, e) + ) sendUserToast(`Permanently deleted workspace ${prefixedId}`) deleteExistingForkOpen = false await validateName(id) diff --git a/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte b/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte index eab008fef6..156d11ce95 100644 --- a/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte +++ b/frontend/src/lib/components/workspaceSettings/CustomInstanceDbWizardModal.svelte @@ -146,7 +146,7 @@ title: 'Database name is valid', status: status?.logs.valid_dbname, description: - 'The database name must be alphanumeric (underscores allowed) and cannot be named the same as the Windmill database (usually "windmill")' + 'The database name must be alphanumeric (underscores and hyphens allowed) and cannot be named the same as the Windmill database (usually "windmill")' }, { title: diff --git a/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte b/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte index 60794c6923..af6fe29e58 100644 --- a/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte +++ b/frontend/src/lib/components/workspaceSettings/DataTableSettings.svelte @@ -41,8 +41,6 @@ } return s } - - let DEFAULT_DATATABLE_DB_NAME = 'datatable_db' @@ -144,7 +174,7 @@ - {#each tableRows as tableRow, idx} + {#each tableRows as tableRow} {#if tableRow[0] === null} @@ -161,27 +191,27 @@
{#if tableRow[1].resourceType === 'filesystem'} - - {/if} + + {/if}
{#if tableRow[1].resourceType === 'filesystem'} @@ -195,6 +225,7 @@ class="flex-1" bind:value={tableRow[1].resourcePath} resourceType={tableRow[1].resourceType} + error={emptyString(tableRow[1].resourcePath)} /> {/if}
@@ -223,19 +254,15 @@ class="cursor-not-allowed" > {#snippet trigger()} - - - - {/snippet} + + {/snippet} {#snippet content()} - - {#if emptyString(tableRow[1].resourcePath)} - Please select a storage resource - {:else if isDirty(tableRow[0])} - Please save your changes - {/if} - - {/snippet} + {#if emptyString(tableRow[1].resourcePath)} + Please select a storage resource + {:else if isDirty(tableRow[0])} + Please save your changes + {/if} + {/snippet} {:else} { if (s3ResourceSettings.secondaryStorage) { - s3ResourceSettings.secondaryStorage.splice(idx - 1, 1) - s3ResourceSettings.secondaryStorage = [...s3ResourceSettings.secondaryStorage] + const realIdx = s3ResourceSettings.secondaryStorage.findIndex( + (s) => s === tableRow + ) + if (realIdx !== -1) { + s3ResourceSettings.secondaryStorage.splice(realIdx, 1) + s3ResourceSettings.secondaryStorage = [...s3ResourceSettings.secondaryStorage] + } } }} /> + {:else if (s3ResourceSettings.secondaryStorage?.length ?? 0) === 0} + {/if} @@ -295,7 +329,16 @@ {/snippet}
- {#if !s3ResourceSettings.resourcePath} + {#if !showPrimaryRow} + + {:else if !s3ResourceSettings.resourcePath} onDiscard?.()} saveLabel="Save storage settings" diff --git a/frontend/src/lib/components/workspaceSettings/utils.svelte.ts b/frontend/src/lib/components/workspaceSettings/utils.svelte.ts index 3d5d70afec..6925be9c04 100644 --- a/frontend/src/lib/components/workspaceSettings/utils.svelte.ts +++ b/frontend/src/lib/components/workspaceSettings/utils.svelte.ts @@ -7,3 +7,31 @@ export let isCustomInstanceDbEnabled = derived( [superadmin], ([superadmin_]) => superadmin_ && !isCloudHosted() ) + +// Postgres caps identifiers at 63 bytes; the backend rejects longer db names. +const MAX_INSTANCE_DB_NAME_LEN = 63 + +// Builds a default instance database name scoped to the workspace (e.g. `dt_myworkspace`), +// appending `_1`, `_2`... until an unused name is found. Workspace ids may contain hyphens, +// which are not valid in unquoted postgres identifiers, so they are replaced with underscores. +// The result is truncated to keep it within the postgres identifier length limit. +export function getUnusedInstanceDbName( + prefix: string, + workspaceId: string, + usedNames: Iterable +): string { + const used = new Set(usedNames) + const base = `${prefix}_${workspaceId.toLowerCase().replace(/-/g, '_')}`.slice( + 0, + MAX_INSTANCE_DB_NAME_LEN + ) + if (!used.has(base)) return base + let i = 1 + let candidate: string + do { + const suffix = `_${i}` + candidate = base.slice(0, MAX_INSTANCE_DB_NAME_LEN - suffix.length) + suffix + i++ + } while (used.has(candidate)) + return candidate +} diff --git a/frontend/src/lib/components/workspaceSettings/utils.test.ts b/frontend/src/lib/components/workspaceSettings/utils.test.ts new file mode 100644 index 0000000000..68115e6a62 --- /dev/null +++ b/frontend/src/lib/components/workspaceSettings/utils.test.ts @@ -0,0 +1,45 @@ +import { describe, expect, it } from 'vitest' + +import { getUnusedInstanceDbName } from './utils.svelte' + +describe('getUnusedInstanceDbName', () => { + it('scopes the name to the workspace with the given prefix', () => { + expect(getUnusedInstanceDbName('dt', 'myworkspace', [])).toBe('dt_myworkspace') + expect(getUnusedInstanceDbName('dl', 'myworkspace', [])).toBe('dl_myworkspace') + }) + + it('lowercases and replaces hyphens with underscores', () => { + expect(getUnusedInstanceDbName('dt', 'My-Team', [])).toBe('dt_my_team') + }) + + it('appends an incrementing suffix when the name is already used', () => { + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc'])).toBe('dt_abc_1') + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc', 'dt_abc_1'])).toBe('dt_abc_2') + }) + + it('skips over already-used suffixed names', () => { + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc', 'dt_abc_2'])).toBe('dt_abc_1') + expect(getUnusedInstanceDbName('dt', 'abc', ['dt_abc', 'dt_abc_1', 'dt_abc_2'])).toBe( + 'dt_abc_3' + ) + }) + + it('accepts any iterable of used names', () => { + expect(getUnusedInstanceDbName('dt', 'abc', new Set(['dt_abc']))).toBe('dt_abc_1') + }) + + it('truncates the base name to the postgres 63-char identifier limit', () => { + const longId = 'w'.repeat(100) + const name = getUnusedInstanceDbName('dt', longId, []) + expect(name.length).toBe(63) + expect(name.startsWith('dt_')).toBe(true) + }) + + it('keeps the result within 63 chars even when appending a suffix', () => { + const longId = 'w'.repeat(100) + const base = getUnusedInstanceDbName('dt', longId, []) // length 63 + const name = getUnusedInstanceDbName('dt', longId, [base]) + expect(name.length).toBeLessThanOrEqual(63) + expect(name.endsWith('_1')).toBe(true) + }) +}) diff --git a/frontend/src/lib/debouncerByKey.svelte.ts b/frontend/src/lib/debouncerByKey.svelte.ts new file mode 100644 index 0000000000..b584deca12 --- /dev/null +++ b/frontend/src/lib/debouncerByKey.svelte.ts @@ -0,0 +1,88 @@ +/** + * Per-key debouncer with a max-wait ceiling (keyed lodash + * `debounce(fn, { wait, maxWait })`). Within a chain the latest `fn` wins and + * the timer is pushed `debounceMs` out, capped at `chainStart + maxDebounceMs` + * so a steady trickle can't defer a save forever. The fire ends the chain; the + * next `schedule` starts a fresh one. Keys independent; task errors are logged + * and swallowed. + */ +import { SvelteSet } from 'svelte/reactivity' + +export type DebouncedTask = () => unknown | Promise + +export type DebouncerByKey = { + /** Replace `key`'s pending task with `fn` and set/extend the timer to + * `min(now + debounceMs, chainStart + maxDebounceMs)`. */ + schedule(key: string, fn: DebouncedTask): void + /** Drop `key`'s pending task and timer (returns whether there was one). + * Use to hand a key to an imperative path (e.g. an immediate save). */ + cancel(key: string): boolean + /** Reactively whether `key` has a queued task (SvelteSet-backed). */ + isPending(key: string): boolean +} + +type Entry = { + timer: ReturnType + task: DebouncedTask + /** Wall-clock ms of the chain's first schedule; the max-wait ceiling + * is measured from here, not "now". */ + chainStart: number +} + +export function createDebouncerByKey(opts: { + debounceMs: number + maxDebounceMs: number +}): DebouncerByKey { + const { debounceMs, maxDebounceMs } = opts + const entries = new Map() + // Reactive mirror of `entries`' keys, kept in lock-step (SvelteSet for + // per-key `isPending` subscriptions). + const pendingKeys = new SvelteSet() + + function fire(key: string): void { + const entry = entries.get(key) + if (!entry) return + entries.delete(key) + // Drop from `pendingKeys` before running: the task synchronously flips + // the key to "running" in the coalescing runner, so there's no gap. + pendingKeys.delete(key) + try { + const result = entry.task() + if (result && typeof (result as Promise).then === 'function') { + ;(result as Promise).catch((e) => + console.error('debouncerByKey: task rejected', e) + ) + } + } catch (e) { + console.error('debouncerByKey: task threw', e) + } + } + + function schedule(key: string, fn: DebouncedTask): void { + const now = Date.now() + const existing = entries.get(key) + const chainStart = existing?.chainStart ?? now + const fireAt = Math.min(now + debounceMs, chainStart + maxDebounceMs) + const delay = Math.max(0, fireAt - now) + + if (existing) clearTimeout(existing.timer) + const timer = setTimeout(() => fire(key), delay) + entries.set(key, { timer, task: fn, chainStart }) + pendingKeys.add(key) + } + + function cancel(key: string): boolean { + const existing = entries.get(key) + if (!existing) return false + clearTimeout(existing.timer) + entries.delete(key) + pendingKeys.delete(key) + return true + } + + function isPending(key: string): boolean { + return pendingKeys.has(key) + } + + return { schedule, cancel, isPending } +} diff --git a/frontend/src/lib/debouncerByKey.test.ts b/frontend/src/lib/debouncerByKey.test.ts new file mode 100644 index 0000000000..f6a51c95c5 --- /dev/null +++ b/frontend/src/lib/debouncerByKey.test.ts @@ -0,0 +1,87 @@ +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest' +import { createDebouncerByKey } from './debouncerByKey.svelte' + +describe('createDebouncerByKey', () => { + beforeEach(() => { + vi.useFakeTimers() + }) + afterEach(() => { + vi.useRealTimers() + }) + + it('fires the task after debounceMs', () => { + const d = createDebouncerByKey({ debounceMs: 1000, maxDebounceMs: 5000 }) + const fn = vi.fn() + d.schedule('k', fn) + expect(d.isPending('k')).toBe(true) + vi.advanceTimersByTime(999) + expect(fn).not.toHaveBeenCalled() + vi.advanceTimersByTime(1) + expect(fn).toHaveBeenCalledTimes(1) + expect(d.isPending('k')).toBe(false) + }) + + it('keeps only the latest task and pushes the timer on each schedule', () => { + const d = createDebouncerByKey({ debounceMs: 1000, maxDebounceMs: 10000 }) + const first = vi.fn() + const second = vi.fn() + d.schedule('k', first) + vi.advanceTimersByTime(800) + d.schedule('k', second) // resets the 1000ms window, replaces task + vi.advanceTimersByTime(999) + expect(second).not.toHaveBeenCalled() + vi.advanceTimersByTime(1) + expect(first).not.toHaveBeenCalled() // replaced — never ran + expect(second).toHaveBeenCalledTimes(1) + }) + + it('caps the chain at maxDebounceMs from its start under a constant trickle', () => { + const d = createDebouncerByKey({ debounceMs: 1000, maxDebounceMs: 2500 }) + const fn = vi.fn() + // Re-schedule every 500ms — without the ceiling the 1000ms window + // would never elapse, but maxDebounceMs forces a fire by t=2500. + d.schedule('k', fn) // chainStart = 0 + for (let t = 500; t <= 2500; t += 500) { + vi.advanceTimersByTime(500) + if (t < 2500) d.schedule('k', fn) + } + expect(fn).toHaveBeenCalledTimes(1) // fired at the 2500ms ceiling + }) + + it('starts a fresh chain (and ceiling) after a fire', () => { + const d = createDebouncerByKey({ debounceMs: 1000, maxDebounceMs: 2000 }) + const fn = vi.fn() + d.schedule('k', fn) + vi.advanceTimersByTime(1000) + expect(fn).toHaveBeenCalledTimes(1) + // Next schedule is a new chain — its own full debounce applies. + d.schedule('k', fn) + vi.advanceTimersByTime(999) + expect(fn).toHaveBeenCalledTimes(1) + vi.advanceTimersByTime(1) + expect(fn).toHaveBeenCalledTimes(2) + }) + + it('cancel stops a pending task and reports whether there was one', () => { + const d = createDebouncerByKey({ debounceMs: 1000, maxDebounceMs: 5000 }) + const fn = vi.fn() + d.schedule('k', fn) + expect(d.cancel('k')).toBe(true) + expect(d.isPending('k')).toBe(false) + vi.advanceTimersByTime(5000) + expect(fn).not.toHaveBeenCalled() + expect(d.cancel('k')).toBe(false) // nothing left to cancel + }) + + it('keeps keys independent', () => { + const d = createDebouncerByKey({ debounceMs: 1000, maxDebounceMs: 5000 }) + const a = vi.fn() + const b = vi.fn() + d.schedule('a', a) + d.schedule('b', b) + d.cancel('a') + vi.advanceTimersByTime(1000) + expect(a).not.toHaveBeenCalled() + expect(b).toHaveBeenCalledTimes(1) + }) +}) diff --git a/frontend/src/lib/draftAddRedirect.ts b/frontend/src/lib/draftAddRedirect.ts new file mode 100644 index 0000000000..2f9e381ed1 --- /dev/null +++ b/frontend/src/lib/draftAddRedirect.ts @@ -0,0 +1,32 @@ +import { redirect } from '@sveltejs/kit' +import { base } from '$app/paths' +import { getUsernameForNamespace } from '$lib/userNamespace' +import { randomUUID } from '$lib/utils/uuid' + +/** + * Shared `load` for every `/{scripts,flows,apps,apps_raw}/add` route. Doing the + * redirect in `load` (not `onMount`) avoids painting a blank frame first. + * + * Mints a fresh `u//draft_` path and 307s to + * `{base}/{editPrefix}/?new_draft=true&`. `new_draft=true` + * tells the edit route to seed an empty editor instead of 404-ing. The hash is + * carried over too — fork / handler-template buttons encode a payload into it + * that the edit route's `new_draft` branch consumes. `randomUUID` not + * `crypto.randomUUID` (WebCrypto is absent on non-secure origins). + */ +export function makeDraftAddLoad(editPrefix: string) { + return ({ url }: { url: URL }) => { + const username = getUsernameForNamespace() + // Underscores not dashes — path segments are `[a-zA-Z0-9_]` words and + // downstream consumers treat `-` as foreign. + const uuid = randomUUID().replaceAll('-', '_') + const params = new URLSearchParams(url.searchParams) + params.set('new_draft', 'true') + // `url.hash` is unavailable in `load`; read `window.location` instead + // (safe — the app is SPA-only, `ssr = false`). On client-side nav it + // points at the PREVIOUS page, but every hash-payload producer arrives + // as a full page load, so the hash is correct. + const hash = typeof window !== 'undefined' ? window.location.hash : '' + redirect(307, `${base}/${editPrefix}/u/${username}/draft_${uuid}?${params.toString()}${hash}`) + } +} diff --git a/frontend/src/lib/editorLangUtils.ts b/frontend/src/lib/editorLangUtils.ts index 3152eff946..5706ad8bcb 100644 --- a/frontend/src/lib/editorLangUtils.ts +++ b/frontend/src/lib/editorLangUtils.ts @@ -85,6 +85,22 @@ export function extToLang(ext: string) { return 'graphql' case 'css': return 'css' + case 'scss': + return 'scss' + case 'less': + return 'less' + case 'html': + case 'htm': + return 'html' + case 'md': + case 'markdown': + return 'markdown' + case 'xml': + return 'xml' + case 'svg': + return 'xml' + case 'txt': + return 'plaintext' case 'yml': return 'ansible' case 'cs': diff --git a/frontend/src/lib/encryptedDraft.ts b/frontend/src/lib/encryptedDraft.ts new file mode 100644 index 0000000000..c0062058a7 --- /dev/null +++ b/frontend/src/lib/encryptedDraft.ts @@ -0,0 +1,10 @@ +/** Marker prefix for draft secret values the backend encrypted at rest + * with the workspace key (mirrors `ENCRYPTED_DRAFT_PREFIX` in + * `backend/windmill-common/src/user_drafts.rs`). The plaintext cannot be + * recovered client-side — deploying sends the marker as-is and the + * deploy endpoints decrypt it server-side. */ +export const ENCRYPTED_DRAFT_PREFIX = '$encrypted:' + +export function isEncryptedDraftValue(v: unknown): boolean { + return typeof v === 'string' && v.startsWith(ENCRYPTED_DRAFT_PREFIX) +} diff --git a/frontend/src/lib/forLater.ts b/frontend/src/lib/forLater.ts index b3ce8001e2..281cbe1e92 100644 --- a/frontend/src/lib/forLater.ts +++ b/frontend/src/lib/forLater.ts @@ -21,8 +21,8 @@ export async function computeDrift() { } } -export function forLater(scheduledString: string): boolean { - return getDbClockNow() < subtractSeconds(new Date(scheduledString), 5) +export function forLater(scheduled: string | number | Date): boolean { + return getDbClockNow() < subtractSeconds(new Date(scheduled), 5) } const limit = pLimit(1) diff --git a/frontend/src/lib/infer.ts b/frontend/src/lib/infer.ts index 7c8b83a64f..a70dd7c745 100644 --- a/frontend/src/lib/infer.ts +++ b/frontend/src/lib/infer.ts @@ -68,6 +68,7 @@ import wasmUrlWac from 'windmill-parser-wasm-wac/windmill_parser_wasm_bg.wasm?ur import { workspaceStore } from './stores.js' import { argSigToJsonSchemaType } from 'windmill-utils-internal' import { type AssetWithAccessType } from './components/assets/lib.js' +import { type ColumnLineage } from './components/assets/AssetGraph/parsePipelineAnnotations' const loadSchemaLastRun = writable< | [ @@ -169,6 +170,10 @@ type InferAssetsResult = assets: AssetWithAccessType[] sql_queries?: InferAssetsSqlQueryDetails[] columns?: Record + // Body-inferred column lineage (DuckDB SQL AST). Present once the + // `windmill-parser-wasm-asset` package is rebuilt with the inference; + // the spread below already forwards it from the parser output. + column_lineage?: ColumnLineage[] } | { status: 'error' diff --git a/frontend/src/lib/localDraftHints.svelte.ts b/frontend/src/lib/localDraftHints.svelte.ts new file mode 100644 index 0000000000..c0c12de22a --- /dev/null +++ b/frontend/src/lib/localDraftHints.svelte.ts @@ -0,0 +1,41 @@ +/** + * Optimistic "this item has a draft" overrides for the list-page `*` suffix. + * + * The list pages render `*` from the server's `is_draft`, which only updates on + * refetch. An open editor knows the live truth and publishes it here; pages read + * `getLocalDraftHint(...) ?? is_draft`, so the editor overrides the stale flag in + * both directions immediately. SvelteMap-backed, so readers re-render on a flip. + * + * Hints PERSIST past editor teardown (the divergence/discard is synced + * server-side) and are corrected, not expired, on the next editor settle. + */ +import { SvelteMap } from 'svelte/reactivity' +import type { UserDraftItemKind } from '$lib/gen' + +const hints = new SvelteMap() + +function key(workspace: string, kind: UserDraftItemKind, path: string): string { + return `${workspace}/${kind}/${path}` +} + +export function setLocalDraftHint( + workspace: string, + kind: UserDraftItemKind, + path: string, + on: boolean +): void { + if (!workspace || !path) return + hints.set(key(workspace, kind, path), on) +} + +/** The editor-observed draft state, or `undefined` when no editor has + * published an opinion — callers fall back to the server flag: + * `getLocalDraftHint(...) ?? is_draft`. */ +export function getLocalDraftHint( + workspace: string | undefined, + kind: UserDraftItemKind, + path: string +): boolean | undefined { + if (!workspace) return undefined + return hints.get(key(workspace, kind, path)) +} diff --git a/frontend/src/lib/logoutRedirect.test.ts b/frontend/src/lib/logoutRedirect.test.ts index ca961f147d..a42a86e6ef 100644 --- a/frontend/src/lib/logoutRedirect.test.ts +++ b/frontend/src/lib/logoutRedirect.test.ts @@ -1,5 +1,33 @@ import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' -import { toSameOriginRelativePath } from './logoutRedirect' +import { isValidLogoutRedirect, toSameOriginRelativePath } from './logoutRedirect' + +describe('isValidLogoutRedirect', () => { + beforeEach(() => { + vi.stubGlobal('window', { location: { origin: 'http://localhost:3000' } }) + }) + + afterEach(() => { + vi.unstubAllGlobals() + }) + + it('accepts same-origin absolute URLs', () => { + expect(isValidLogoutRedirect('http://localhost:3000/')).toBe(true) + expect(isValidLogoutRedirect('http://localhost:3000/runs?workspace=foo')).toBe(true) + }) + + it('accepts root-relative paths', () => { + expect(isValidLogoutRedirect('/foo')).toBe(true) + }) + + it('accepts windmill.dev hosts', () => { + expect(isValidLogoutRedirect('https://app.windmill.dev/foo')).toBe(true) + }) + + it('rejects cross-origin and protocol-relative', () => { + expect(isValidLogoutRedirect('https://evil.com/')).toBe(false) + expect(isValidLogoutRedirect('//evil.com')).toBe(false) + }) +}) describe('toSameOriginRelativePath', () => { beforeEach(() => { diff --git a/frontend/src/lib/logoutRedirect.ts b/frontend/src/lib/logoutRedirect.ts index 6416eef1e5..8160dfd66d 100644 --- a/frontend/src/lib/logoutRedirect.ts +++ b/frontend/src/lib/logoutRedirect.ts @@ -7,6 +7,13 @@ export function isValidLogoutRedirect(url: string): boolean { } try { const parsed = new URL(url) + // Same-origin absolute URLs are always safe: the user can only be sent + // back to the site they're already on, which is never an open redirect. + // Mirrors toSameOriginRelativePath()'s same-origin handling, and lets a + // self-hosted instance redirect back to its own app after OAuth login. + if (typeof window !== 'undefined' && parsed.origin === window.location.origin) { + return true + } const host = parsed.hostname if (host === 'windmill.dev' || host.endsWith('.windmill.dev')) { return true diff --git a/frontend/src/lib/mcpEndpointTools.ts b/frontend/src/lib/mcpEndpointTools.ts index b86119477e..8a46163009 100644 --- a/frontend/src/lib/mcpEndpointTools.ts +++ b/frontend/src/lib/mcpEndpointTools.ts @@ -17,25 +17,53 @@ export interface EndpointTool { export const mcpEndpointTools: EndpointTool[] = [ { - name: "queryDocumentation", - description: "query Windmill AI documentation assistant (EE only)", + name: "searchDocs", + description: "Full-text search across the entire Windmill documentation. Provide one or more keywords; returns the most relevant docs pages, each with its Source URL and short matching snippets. Use this FIRST to find relevant pages by their content (a flag, function, error message, config key or concept). If the snippets answer the question, answer directly; otherwise call readDocsPage with a returned Source URL to read more.", instructions: "", - path: "/inkeep", - method: "POST", + path: "/docs/search", + method: "GET", pathParamsSchema: undefined, - queryParamsSchema: undefined, - bodySchema: { + queryParamsSchema: { "type": "object", "properties": { "query": { "type": "string", - "description": "The documentation query to send to the AI assistant" + "description": "Keywords to search for in the documentation body, e.g. \"chromium worker tag\" or \"retry exponential backoff\". Fewer, more distinctive words match better." } }, "required": [ "query" ] }, + bodySchema: undefined, + pathFieldRenames: undefined, + queryFieldRenames: undefined, + bodyFieldRenames: undefined + }, + { + name: "readDocsPage", + description: "Fetch the markdown of a single Windmill documentation page. Provide the `url` of a page found via searchDocs (its Source URL). If the page is large, this returns its list of section headings instead of the full content; call again with the `section` argument set to one of those headings to read that section.", + instructions: "", + path: "/docs/page", + method: "GET", + pathParamsSchema: undefined, + queryParamsSchema: { + "type": "object", + "properties": { + "url": { + "type": "string", + "description": "The docs page to read, as a Source URL returned by searchDocs (e.g. https://www.windmill.dev/docs/core_concepts/jobs). A bare path (e.g. /docs/core_concepts/jobs) is also accepted." + }, + "section": { + "type": "string", + "description": "Optional. A heading title from the page outline to read just that section instead of the full page." + } + }, + "required": [ + "url" + ] +}, + bodySchema: undefined, pathFieldRenames: undefined, queryFieldRenames: undefined, bodyFieldRenames: undefined @@ -94,6 +122,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -179,6 +210,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the variable (body parameter)" @@ -220,6 +254,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "include_encrypted": { "type": "boolean", "description": "ask to include the encrypted value if secret and decrypt secret is not true (default: false)\n" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -270,6 +308,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft variables whose path has no\ndeployed variable. Synthesized rows carry `draft_only: true`\nso the home page can render a \"Draft\" badge.\n" } }, "required": [] @@ -319,6 +361,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "items": { "type": "string" } + }, + "ws_specific": { + "type": "boolean" } }, "required": [ @@ -393,6 +438,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "string" } }, + "ws_specific": { + "type": "boolean" + }, "path__body": { "type": "string", "description": "The path to the resource (body parameter)" @@ -424,7 +472,16 @@ export const mcpEndpointTools: EndpointTool[] = [ "path" ] }, - queryParamsSchema: undefined, + queryParamsSchema: { + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +}, bodySchema: undefined, pathFieldRenames: undefined, queryFieldRenames: undefined, @@ -479,6 +536,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft resources whose path has\nno deployed resource. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] @@ -602,7 +663,7 @@ export const mcpEndpointTools: EndpointTool[] = [ { name: "createScript", description: "create script: Creates a new script when the path does not already exist.\nCreates a new version of an existing script when called with the same path and the current `parent_hash`", - instructions: "To create a script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language. For TypeScript, use 'bun' unless deno-specific APIs are needed.", + instructions: "To create a NEW script, specify the path (e.g., 'f/my_folder/my_script'), the content (source code), and the language, and leave parent_hash unset. For TypeScript, use 'bun' unless deno-specific APIs are needed. To UPDATE an existing script, do NOT delete and recreate it: call this tool with the same path and set parent_hash to the script's current hash, which you can read from the `hash` field returned by getScriptByPath. This creates a new version while preserving the script's history.", path: "/w/{workspace}/scripts/create", method: "POST", pathParamsSchema: undefined, @@ -613,6 +674,9 @@ export const mcpEndpointTools: EndpointTool[] = [ "path": { "type": "string" }, + "parent_hash": { + "type": "string" + }, "summary": { "type": "string" }, @@ -725,6 +789,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -853,6 +921,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "properties": { "with_starred_info": { "type": "boolean" + }, + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." } }, "required": [] @@ -1190,6 +1262,14 @@ export const mcpEndpointTools: EndpointTool[] = [ "language" ] } + }, + "temp_script_refs": { + "type": "object", + "nullable": true, + "description": "Map of relative-import script path -> temp storage hash so the preview job resolves those imports from not-yet-deployed local content instead of the deployed script", + "additionalProperties": { + "type": "string" + } } }, "required": [ @@ -1462,6 +1542,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "boolean", "description": "filter on successful jobs" }, + "status": { + "type": "string", + "description": "filter on the exact completed job status. Unlike `success=true` (which also matches `skipped`), `status=success` matches only `success`.. Possible values: success, failure, canceled, skipped" + }, "all_workspaces": { "type": "boolean", "description": "get jobs from all workspaces (only valid if request come from the `admins` workspace)" @@ -1470,6 +1554,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "type": "boolean", "description": "is not a scheduled job" }, + "excludes_entrypoint_override": { + "type": "boolean", + "description": "exclude jobs that were started with a `_ENTRYPOINT_OVERRIDE` arg (e.g. dynamic-select helper runs and preprocessor previews)" + }, "broad_filter": { "type": "string", "description": "broad search across multiple fields (case-insensitive substring match on path, tag, schedule path, trigger kind, label)" @@ -1508,6 +1596,10 @@ export const mcpEndpointTools: EndpointTool[] = [ }, "no_code": { "type": "boolean" + }, + "approval_token": { + "type": "string", + "description": "Approval token granting read access to the job when not logged in. The token must be the one issued for this job's flow (i.e. the flow id used when generating the approval URL)." } }, "required": [] @@ -1562,7 +1654,7 @@ export const mcpEndpointTools: EndpointTool[] = [ "properties": { "path": { "type": "string", - "description": "The unique path identifier for this schedule" + "description": "The unique Windmill path for this schedule. Must be of the form `u//` or `f//`." }, "schedule": { "type": "string", @@ -2002,7 +2094,16 @@ export const mcpEndpointTools: EndpointTool[] = [ "path" ] }, - queryParamsSchema: undefined, + queryParamsSchema: { + "type": "object", + "properties": { + "get_draft": { + "type": "boolean", + "description": "When true, overlay the authed user's draft (if any) onto the deployed payload." + } + }, + "required": [] +}, bodySchema: undefined, pathFieldRenames: undefined, queryFieldRenames: undefined, @@ -2061,6 +2162,10 @@ export const mcpEndpointTools: EndpointTool[] = [ "label": { "type": "string", "description": "Filter by label" + }, + "include_draft_only": { + "type": "boolean", + "description": "When true, append per-user draft schedules whose path has\nno deployed schedule. Synthesized rows carry\n`draft_only: true`.\n" } }, "required": [] diff --git a/frontend/src/lib/newDraftFlag.test.ts b/frontend/src/lib/newDraftFlag.test.ts new file mode 100644 index 0000000000..bd84e1cdb6 --- /dev/null +++ b/frontend/src/lib/newDraftFlag.test.ts @@ -0,0 +1,101 @@ +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest' + +// Service layer mocked: a "saved" response drives the syncer's confirmed-save +// path, which is what `onSaved` / `stripNewDraftFlagOnSave` hang off of. +const updateDraft = vi.fn(async (..._args: any[]) => ({ + status: 'saved' as const, + current_timestamp: '2020-01-01T00:00:00Z' +})) + +vi.mock('./gen', () => ({ + DraftService: { updateDraft: (...a: unknown[]) => updateDraft(...(a as [])) } +})) +vi.mock('./gen/core/OpenAPI', () => ({ OpenAPI: { BASE: '' } })) +vi.mock('./localDraftHints.svelte', () => ({ setLocalDraftHint: vi.fn() })) + +import { UserDraftDbSyncer } from './userDraftDbSyncer.svelte' +import { stripNewDraftFlagOnSave } from './newDraftFlag' + +/** Minimal `window` stand-in: `history.replaceState` rewrites `location.href`, + * mirroring what jsdom does, so the helper's strip is observable. */ +function stubWindow(href: string): { current: () => string } { + const win: any = { + location: { href }, + history: { + state: null as unknown, + replaceState(state: unknown, _title: string, url: string) { + this.state = state + win.location.href = new URL(url, win.location.href).toString() + } + } + } + vi.stubGlobal('window', win) + return { current: () => win.location.href } +} + +afterEach(() => { + vi.unstubAllGlobals() + vi.clearAllMocks() + updateDraft.mockResolvedValue({ status: 'saved', current_timestamp: '2020-01-01T00:00:00Z' }) +}) + +describe('UserDraftDbSyncer.onSaved', () => { + it('fires after a confirmed non-delete save', async () => { + const q = { workspace: 'w', itemKind: 'script' as const, path: 'u/me/draft_a' } + const listener = vi.fn() + UserDraftDbSyncer.onSaved(q, listener) + await UserDraftDbSyncer.save({ ...q, value: { content: 'x' }, immediate: true }) + expect(listener).toHaveBeenCalledTimes(1) + }) + + it('does NOT fire on a delete (value: null) save', async () => { + const q = { workspace: 'w', itemKind: 'script' as const, path: 'u/me/draft_b' } + const listener = vi.fn() + UserDraftDbSyncer.onSaved(q, listener) + await UserDraftDbSyncer.save({ ...q, value: null, immediate: true }) + expect(listener).not.toHaveBeenCalled() + }) + + it('stops firing after unsubscribe', async () => { + const q = { workspace: 'w', itemKind: 'script' as const, path: 'u/me/draft_c' } + const listener = vi.fn() + const unsub = UserDraftDbSyncer.onSaved(q, listener) + unsub() + await UserDraftDbSyncer.save({ ...q, value: { content: 'x' }, immediate: true }) + expect(listener).not.toHaveBeenCalled() + }) +}) + +describe('stripNewDraftFlagOnSave', () => { + beforeEach(() => { + stubWindow('http://localhost/scripts/edit/u/me/draft_d?new_draft=true&template=foo') + }) + + it('keeps ?new_draft until a save lands, then strips only that flag', async () => { + const q = { workspace: 'w', itemKind: 'script' as const, path: 'u/me/draft_d' } + stripNewDraftFlagOnSave(q) + // Before any save the flag is untouched. + expect(window.location.href).toContain('new_draft=true') + await UserDraftDbSyncer.save({ ...q, value: { content: 'x' }, immediate: true }) + expect(window.location.href).not.toContain('new_draft') + // Sibling seeding params are preserved. + expect(window.location.href).toContain('template=foo') + }) + + it('does not strip on a delete save', async () => { + const q = { workspace: 'w', itemKind: 'script' as const, path: 'u/me/draft_e' } + stubWindow('http://localhost/scripts/edit/u/me/draft_e?new_draft=true') + stripNewDraftFlagOnSave(q) + await UserDraftDbSyncer.save({ ...q, value: null, immediate: true }) + expect(window.location.href).toContain('new_draft=true') + }) + + it('cleanup unsubscribes so a later save does not strip', async () => { + const q = { workspace: 'w', itemKind: 'script' as const, path: 'u/me/draft_f' } + stubWindow('http://localhost/scripts/edit/u/me/draft_f?new_draft=true') + const cleanup = stripNewDraftFlagOnSave(q) + cleanup() + await UserDraftDbSyncer.save({ ...q, value: { content: 'x' }, immediate: true }) + expect(window.location.href).toContain('new_draft=true') + }) +}) diff --git a/frontend/src/lib/newDraftFlag.ts b/frontend/src/lib/newDraftFlag.ts new file mode 100644 index 0000000000..77d399acfa --- /dev/null +++ b/frontend/src/lib/newDraftFlag.ts @@ -0,0 +1,31 @@ +import { UserDraftDbSyncer, type UserDraftLastSyncQuery } from '$lib/userDraftDbSyncer.svelte' + +/** + * Defer stripping `?new_draft=true` from the URL until the draft's first save + * is CONFIRMED by the backend. The `/{scripts,flows,apps,apps_raw}/add` + * redirect lands the editor on an unsaved `u//draft_` path; + * stripping the flag on load (the old behavior) meant a refresh before any + * edit fell through to the `getByPath` fetch for a row that doesn't exist yet + * → 404 "not found". Keeping the flag until `onSaved` fires lets a pre-edit + * refresh re-enter the new-draft seeding branch, while a post-save refresh + * loads the now-persisted draft. + * + * Returns a cleanup to drop the listener on navigation / unmount; the listener + * also self-unsubscribes after the first save. + */ +export function stripNewDraftFlagOnSave(query: UserDraftLastSyncQuery): () => void { + let unsub: (() => void) | undefined + unsub = UserDraftDbSyncer.onSaved(query, () => { + unsub?.() + unsub = undefined + if (typeof window === 'undefined') return + const url = new URL(window.location.href) + if (url.searchParams.get('new_draft') !== 'true') return + url.searchParams.delete('new_draft') + window.history.replaceState(window.history.state, '', url.toString()) + }) + return () => { + unsub?.() + unsub = undefined + } +} diff --git a/frontend/src/lib/rawAppDeploy.ts b/frontend/src/lib/rawAppDeploy.ts index 0ed87a1b09..b13951c5f8 100644 --- a/frontend/src/lib/rawAppDeploy.ts +++ b/frontend/src/lib/rawAppDeploy.ts @@ -6,9 +6,8 @@ * This mirrors how the global AI chat deploys raw apps * (`copilot/chat/global/core.ts` → deployDraft, case 'app'): read the item with * its draft, normalise to an AppDraftValue, recompute the policy, bundle the - * files, then createAppRaw/updateAppRaw. The two pure transforms - * (appSourceToDraftValue / normalizeRawAppData) are re-implemented here to avoid - * importing the heavy chat module. + * files, then createAppRaw/updateAppRaw. The source→AppDraftValue projection is + * shared via `rawAppDraftValue` so the two deploy paths can't drift. */ import { get } from 'svelte/store' import { AppService } from '$lib/gen' @@ -18,32 +17,7 @@ import { bundleRawAppDraft } from '$lib/components/copilot/chat/global/rawAppBun import type { AppDraftValue } from '$lib/components/copilot/chat/global/workspaceItems' import { updateRawAppPolicy } from '$lib/components/raw_apps/rawAppPolicy' import { DEFAULT_DATA as DEFAULT_RAW_APP_DATA } from '$lib/components/raw_apps/dataTableRefUtils' - -function normalizeRawAppData(value: Record): AppDraftValue['data'] { - if (value.data?.creation) { - return { - tables: value.data.tables ?? [], - datatable: value.data.creation.datatable, - schema: value.data.creation.schema - } - } - if (value.data) return value.data - if (value.datatables) return { ...DEFAULT_RAW_APP_DATA, tables: value.datatables } - if (value.dataTableRefs) return { ...DEFAULT_RAW_APP_DATA, tables: value.dataTableRefs } - return { ...DEFAULT_RAW_APP_DATA } -} - -function appSourceToDraftValue(app: any, fallback?: any): AppDraftValue { - const value = (app.value ?? {}) as Record - return { - summary: app.summary ?? '', - files: { ...(value.files ?? {}) }, - runnables: { ...(value.runnables ?? {}) }, - data: normalizeRawAppData(value), - policy: app.policy ?? fallback?.policy, - custom_path: app.custom_path ?? fallback?.custom_path - } -} +import { appSourceToDraftValue } from '$lib/components/raw_apps/rawAppDraftValue' /** * Promote a raw app's draft to deployed. Throws on failure (caller wraps into a @@ -55,10 +29,15 @@ export async function deployRawAppDraft( path: string, deploymentMessage?: string ): Promise { - const app = await AppService.getAppByPathWithDraft({ workspace, path }) + // `rawApp: true` so a never-deployed raw app (no `app` row) resolves to + // the raw_app draft kind server-side instead of 404ing. + const app = await AppService.getAppByPath({ workspace, path, getDraft: true, rawApp: true }) const draft = (app as any).draft - // Honor a renamed draft path; the URL `path` below stays the existing item key. - const targetPath = draft?.path ?? path + // Deploy at the draft's intended path. A raw-app draft carries the user-typed + // path in `draft_path` (a never-deployed app is parked at a synthetic + // `u/{user}/draft_{uuid}` storage key); the URL `path` below stays that storage + // key. Falls back to `path` for an unrenamed draft on a deployed app. + const targetPath = draft?.draft_path ?? draft?.path ?? path const value = appSourceToDraftValue(draft ?? app, app) const policy = (await updateRawAppPolicy( @@ -96,7 +75,11 @@ export async function deployRawAppDraft( summary, policy, deployment_message: deploymentMessage, - custom_path: isAdmin ? (value.custom_path ?? '') : undefined + custom_path: isAdmin ? (value.custom_path ?? '') : undefined, + // Preserve the policy's on_behalf_of: this draft-deploy path has no + // on-behalf-of selector, so without the flag the backend resets it to + // the deploying user (gated server-side by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined }, js: bundle.js, css: bundle.css @@ -112,7 +95,9 @@ export async function deployRawAppDraft( summary, policy, deployment_message: deploymentMessage, - custom_path: value.custom_path + custom_path: value.custom_path, + // Preserve the policy's on_behalf_of (see update branch above). + preserve_on_behalf_of: policy.on_behalf_of ? true : undefined }, js: bundle.js, css: bundle.css diff --git a/frontend/src/lib/stores.ts b/frontend/src/lib/stores.ts index ce7b25ace6..3e3ab817eb 100644 --- a/frontend/src/lib/stores.ts +++ b/frontend/src/lib/stores.ts @@ -27,6 +27,7 @@ export interface UserExt { groups: string[] pgroups: string[] folders: string[] + folders_read: string[] folders_owners: string[] is_service_account?: boolean impersonating_email?: string @@ -130,6 +131,13 @@ export const codeCompletionSessionEnabled = writable( getLocalSetting(CODE_COMPLETION_SETTING_NAME) != 'false' ) +export const AI_USER_DISABLED_SETTING_NAME = 'aiUserDisabled' +// Master per-user (per-device) opt-out for all Windmill AI features. Initialized at +// module load so it applies on startup, not only once the settings panel mounts. +export const aiUserDisabled = writable( + getLocalSetting(AI_USER_DISABLED_SETTING_NAME) === 'true' +) + export const usedTriggerKinds = writable([]) export let globalDbManagerDrawer: StateStore = { val: undefined } @@ -178,6 +186,10 @@ export interface SQLSchema { schema: SQLBaseSchema publicOnly: boolean | undefined stringified: string + /** MySQL only: the connection's default database (`DATABASE()`), surfaced by the + * introspection script. Lets the table picker render the default db's tables + * unprefixed even when the connection can also see other (non-system) schemas. */ + defaultDb?: string } export interface GraphqlSchema { diff --git a/frontend/src/lib/svelte5Utils.svelte.ts b/frontend/src/lib/svelte5Utils.svelte.ts index aad16431cb..d688ec1502 100644 --- a/frontend/src/lib/svelte5Utils.svelte.ts +++ b/frontend/src/lib/svelte5Utils.svelte.ts @@ -600,11 +600,21 @@ export function useLocalStorageValue( * across long editing sessions. */ transformBeforePersist?: (val: T) => T + /** + * Register a `$effect` that walks the stored value on every access and + * persists when any nested field mutated. Required for callers that + * mutate the value in place (`s.foo = bar`) rather than reassigning + * via the setter. Setter-only callers (e.g. a flat string slot) should + * leave this `false` — the `$effect` requires a Svelte effect scope, + * so enabling it forces the hook to be called from inside a component. + */ + reactToNestedUpdates?: boolean } ): { val: T; skipNextWriteOnce(): void; setWithoutPersist(newVal: T): void } { const saveInitialValue = options?.saveInitialValue ?? true const debounceMs = options?.debounce ?? 0 const transformBeforePersist = options?.transformBeforePersist + const reactToNestedUpdates = options?.reactToNestedUpdates ?? false const serialize = (val: T) => typ === 'string' || typ === 'number' || typ === 'boolean' ? String(val) : JSON.stringify(val) const deserialize = (val: string): T => { @@ -672,17 +682,19 @@ export function useLocalStorageValue( pendingValue = undefined } - $effect(() => { - readFieldsRecursively(s) - const next = s === undefined ? undefined : serialize(s) - if (next === lastSerialized) return - lastSerialized = next - if (skipNextWrite) { - skipNextWrite = false - return - } - schedulePersist(s) - }) + if (reactToNestedUpdates) { + $effect(() => { + readFieldsRecursively(s) + const next = s === undefined ? undefined : serialize(s) + if (next === lastSerialized) return + lastSerialized = next + if (skipNextWrite) { + skipNextWrite = false + return + } + schedulePersist(s) + }) + } return { get val() { diff --git a/frontend/src/lib/test-setup.ts b/frontend/src/lib/test-setup.ts index 6b82fcefb6..01cacdc912 100644 --- a/frontend/src/lib/test-setup.ts +++ b/frontend/src/lib/test-setup.ts @@ -2,6 +2,10 @@ * Vitest setup file to mock browser globals for testing */ +// Provides a real in-memory IndexedDB (indexedDB / IDBKeyRange / structuredClone) +// under the node test environment so the IndexedDB-backed session list and +// chat-history stores can be exercised in unit tests. +import 'fake-indexeddb/auto' import { vi } from 'vitest' // Mock localStorage @@ -67,7 +71,8 @@ Object.defineProperty(globalThis, 'sessionStorage', { if (typeof (globalThis as any).window === 'undefined') { Object.defineProperty(globalThis, 'window', { value: globalThis, - writable: true + writable: true, + configurable: true }) } diff --git a/frontend/src/lib/tutorials/config.ts b/frontend/src/lib/tutorials/config.ts index 455c4c50e3..082dc0473d 100644 --- a/frontend/src/lib/tutorials/config.ts +++ b/frontend/src/lib/tutorials/config.ts @@ -67,7 +67,7 @@ export const TUTORIALS_CONFIG: Record = { title: 'Build a flow', description: 'Learn how to build workflows in Windmill with our interactive tutorial.', onClick: () => { - window.location.href = `${base}/flows/add?tutorial=flow-live-tutorial&nodraft=true` + window.location.href = `${base}/flows/add?tutorial=flow-live-tutorial` }, index: 2, active: true, @@ -81,7 +81,7 @@ export const TUTORIALS_CONFIG: Record = { title: 'Fix a broken flow', description: 'Learn how to monitor and debug your script and flow executions.', onClick: () => { - window.location.href = `${base}/flows/add?tutorial=troubleshoot-flow&nodraft=true` + window.location.href = `${base}/flows/add?tutorial=troubleshoot-flow` }, index: 3, active: true, @@ -131,7 +131,7 @@ export const TUTORIALS_CONFIG: Record = { title: 'Background runnables', description: 'Learn how to create and use background runnables in your apps.', onClick: () => { - window.location.href = `${base}/apps/add?tutorial=backgroundrunnables&nodraft=true` + window.location.href = `${base}/apps/add?tutorial=backgroundrunnables` }, index: 4, active: true, @@ -145,7 +145,7 @@ export const TUTORIALS_CONFIG: Record = { title: 'Connection', description: 'Learn how to connect component inputs to outputs in your apps.', onClick: () => { - window.location.href = `${base}/apps/add?tutorial=connection&nodraft=true` + window.location.href = `${base}/apps/add?tutorial=connection` }, index: 5, active: true, diff --git a/frontend/src/lib/userDraft.svelte.ts b/frontend/src/lib/userDraft.svelte.ts index 69a9bba5d8..8437a1b87c 100644 --- a/frontend/src/lib/userDraft.svelte.ts +++ b/frontend/src/lib/userDraft.svelte.ts @@ -2,8 +2,14 @@ import { get } from 'svelte/store' import { onDestroy, untrack } from 'svelte' import { deepEqual } from 'fast-equals' import { workspaceStore } from './stores' -import { useLocalStorageValue } from './svelte5Utils.svelte' +import { readFieldsRecursively } from './utils' +import { UserDraftDbSyncer } from './userDraftDbSyncer.svelte' +import type { UserDraftItemKind } from './gen' +export type { UserDraftItemKind } + +// Runtime mirror of the generated `UserDraftItemKind` union. `satisfies` +// + the exhaustiveness check below make a drift between the two a type error. export const USER_DRAFT_ITEM_KINDS = [ 'script', 'flow', @@ -28,79 +34,26 @@ export const USER_DRAFT_ITEM_KINDS = [ 'trigger_cli', 'trigger_nextcloud', 'trigger_google', - 'trigger_github' -] as const + 'trigger_github', + 'data_pipeline' +] as const satisfies readonly UserDraftItemKind[] -export type UserDraftItemKind = (typeof USER_DRAFT_ITEM_KINDS)[number] +// Reverse direction: every union member must appear in the array above. +type _AssertKindsExhaustive = + Exclude extends never ? true : never +const _: _AssertKindsExhaustive = true +void _ export type UserDraftOptions = { workspace?: string } -export type UserDraftUseOptions = UserDraftOptions & { - /** - * Initial value used when localStorage holds no draft for this - * (workspace, itemKind, path). It is *not* eagerly persisted — the first - * actual mutation is what writes to localStorage. - */ - defaultValue?: V -} - export type UserDraftListOptions = UserDraftOptions & { itemKinds?: readonly UserDraftItemKind[] } -/** - * A single (kind, path, workspace) tuple that `useMany` should hold a handle - * for. The shape mirrors `use()`'s arguments, just bundled into one object - * so a getter can return a list of them. - */ -export type UserDraftSpec = { - itemKind: UserDraftItemKind - path: string - workspace?: string - defaultValue?: V -} - -/** - * Snapshot of the remote item's freshness at the moment the local draft was - * written. Used by editor routes to detect that the remote has moved on - * (someone else deployed, or saved a DB draft) so we can warn the user - * before they push stale changes. - * - * - `remoteRev`: the deployed version's id/hash/timestamp at draft creation. - * - `remoteDraftRev`: the DB-draft `created_at` at draft creation, only set - * for kinds that have a DB-draft (`script`, `flow`, `app`, `raw_app`). - */ -export type UserDraftMeta = { - remoteRev?: string | number - remoteDraftRev?: string | number -} - -/** - * The shape of what we actually persist. Wrapping the value lets us add - * metadata (timestamps, originating user, schema version, ...) later - * without breaking existing entries. - * - * `lastWrittenAt` is the unix-ms timestamp of the most recent write - * (setter call or deep mutation flush). It's the GC signal — - * `gcUserDrafts` sweeps entries that haven't been touched in N days. - * Set at every persist via `useLocalStorageValue`'s `transformBeforePersist`, - * `UserDraft.save`'s direct-write fallback, and `persistDirect`. Missing - * (undefined) on entries written before this field was introduced; - * `gcUserDrafts` backfills them on first sighting. - */ -type StoredDraft = { value: V; lastWrittenAt?: number } & UserDraftMeta - -function stamp(stored: StoredDraft | undefined): StoredDraft | undefined { - if (stored === undefined) return undefined - return { ...stored, lastWrittenAt: Date.now() } -} - type DraftState = { - val: StoredDraft | undefined - skipNextWriteOnce(): void - setWithoutPersist(newVal: StoredDraft | undefined): void + val: V | undefined } type DraftEntry = { @@ -110,12 +63,27 @@ type DraftEntry = { path: string state: DraftState /** - * Tears down the `$effect.root` scope that owns the entry's - * `useLocalStorageValue` reactivity — its `$state` cell and the persist - * `$effect` deep-mutation loop. Called when the refcount hits 0. - * - * `undefined` only when the test runtime's broken `$effect.root` forced - * us through the fallback path (see `acquireEntry`). + * Single-shot: skip the next reactive POST. Set by callers that already + * pushed the right thing (e.g. `discard`'s explicit `value: null` POST) + * before the reactive write, so the effect doesn't fire a duplicate. + */ + skipNextSync: boolean + /** + * Sticky `skipNextSync`: while true the sync effect updates local state + * but never POSTs. For programmatic bootstrap mutations (e.g. seeding + * `initialCode`). Toggled via `stopSync` / `restartSync`. + */ + syncSuspended: boolean + /** + * Single-shot: the next cell write is a programmatic SEED (baseline load + * / new-draft template), not a user edit. The effect adopts it as the + * baseline and skips the POST. Like `syncSuspended` but scoped to one + * write, so there's no suspension to forget to resume. Set via `seed`. + */ + seedNextWrite: boolean + /** + * Tears down the entry's `$effect.root` scope; called at refcount 0. + * `undefined` only on the test-runtime fallback path (see `acquireEntry`). */ destroyRoot?: () => void } @@ -125,9 +93,6 @@ export type UserDraftEntry = { itemKind: UserDraftItemKind path: string value: V | undefined - meta: UserDraftMeta - persisted: boolean - live: boolean } export type LiveEditorDraft = { @@ -150,6 +115,40 @@ export type ClearLiveEditorDraftOptions = UserDraftOptions & { const entries = new Map() const liveEditorDrafts = new Map() +/** + * Map keys whose entry should start `syncSuspended` on acquire. Lets + * callers `stopSync` BEFORE the editor has mounted (and called `use`). + * Consumed by `acquireEntry`; cleared by the matching `restartSync`. */ +const pendingSuspensions = new Set() + +/** + * Synchronous read-through cache for values written via `save` while no live + * editor entry exists. That branch persists through the debounced + * `UserDraftDbSyncer` (async, fire-and-forget), so without this a same-tab + * `save(...)` followed by `get(...)` would miss its own write: the global AI + * chat writes a draft then immediately reads it back to return the result and + * would otherwise throw "Could not read written draft". A live entry shadows + * the cache (the entry is authoritative) and a release drops the key; a delete + * (`remove`/`discard`) evicts it. + */ +const writtenCache = new Map< + string, + { workspace: string; itemKind: UserDraftItemKind; path: string; val: unknown } +>() + +function rememberWrite( + workspace: string, + itemKind: UserDraftItemKind, + path: string, + val: unknown +): void { + const mk = mapKey(workspace, itemKind, path) + if (val === undefined) { + writtenCache.delete(mk) + } else { + writtenCache.set(mk, { workspace, itemKind, path, val: snapshotDraftValue(val) }) + } +} function resolveWorkspace(opts?: UserDraftOptions): string { const ws = opts?.workspace ?? get(workspaceStore) @@ -161,115 +160,14 @@ function resolveWorkspace(opts?: UserDraftOptions): string { return ws } -function wrap(value: V | undefined, meta?: UserDraftMeta): StoredDraft | undefined { - if (value === undefined) return undefined - const out: StoredDraft = { value } - if (meta?.remoteRev !== undefined) out.remoteRev = meta.remoteRev - if (meta?.remoteDraftRev !== undefined) out.remoteDraftRev = meta.remoteDraftRev - return out -} - -function unwrap(stored: StoredDraft | undefined): V | undefined { - return stored?.value -} - -function extractMeta(stored: StoredDraft | undefined): UserDraftMeta { - if (!stored) return {} - const meta: UserDraftMeta = {} - if (stored.remoteRev !== undefined) meta.remoteRev = stored.remoteRev - if (stored.remoteDraftRev !== undefined) meta.remoteDraftRev = stored.remoteDraftRev - return meta -} - -/** - * Compares the rev metadata recorded against the local draft to the current - * backend revs. Returns the staleness cause, or `null` when the local draft - * is still based on the latest backend state we know about. - * - * - Entries with no recorded meta (legacy entries written before this field - * existed) report `null` — we can't tell if they're stale, and we'd rather - * trust the local autosave than spam the user with false positives. - * - DB-draft staleness wins over deployed-version staleness: a remote DB - * draft is the more recent state to reconcile against. - * - If a DB draft existed when the local autosave was created but now no - * longer exists on the remote (someone discarded it), we report `version` - * because the deployed version is now the canonical "latest saved". - */ -export type UserDraftStalenessCause = 'draft' | 'version' - -export function checkStaleness( - meta: UserDraftMeta, - currentRev: string | number | undefined, - currentDraftRev?: string | number | undefined -): UserDraftStalenessCause | null { - if (meta.remoteRev === undefined && meta.remoteDraftRev === undefined) return null - if (meta.remoteDraftRev !== currentDraftRev) { - return currentDraftRev !== undefined ? 'draft' : 'version' - } - if (currentRev !== undefined && meta.remoteRev !== currentRev) return 'version' - return null -} - -/** - * Synchronous localStorage write, bypassing the entry's debounced setter - * and its first-write skip. See `setMeta({ force: true })`. - */ -function persistDirect(key: string, value: V | undefined, meta: UserDraftMeta): void { - try { - const next = stamp(wrap(value, meta)) - if (next === undefined) { - localStorage.removeItem(key) - } else { - localStorage.setItem(key, JSON.stringify(next)) - } - } catch (e) { - console.error('UserDraft: localStorage write failed', e) - } -} - -function readPersisted(key: string): StoredDraft | undefined { - try { - const raw = localStorage.getItem(key) - if (raw == null || raw === 'undefined') return undefined - const parsed = JSON.parse(raw) - // Defensive: ignore pre-wrapping payloads (no `.value`). - if (parsed == null || typeof parsed !== 'object' || !('value' in parsed)) return undefined - return parsed as StoredDraft - } catch (e) { - console.error('UserDraft: localStorage read failed', e) - return undefined - } -} - function mapKey(workspace: string, itemKind: UserDraftItemKind, path: string): string { return `${workspace}/${itemKind}/${path}` } -function localStorageKey(workspace: string, itemKind: UserDraftItemKind, path: string): string { - return `userdraft/w/${workspace}/${itemKind}/${path}` -} - function liveEditorDraftKey(workspace: string, itemKind: UserDraftItemKind): string { return `${workspace}/${itemKind}` } -function parseLocalStorageKey( - key: string, - workspace: string, - itemKinds: readonly UserDraftItemKind[] -): { itemKind: UserDraftItemKind; path: string } | undefined { - const prefix = `userdraft/w/${workspace}/` - if (!key.startsWith(prefix)) return undefined - const rest = key.slice(prefix.length) - for (const itemKind of itemKinds) { - const kindPrefix = `${itemKind}/` - if (rest.startsWith(kindPrefix)) { - return { itemKind, path: rest.slice(kindPrefix.length) } - } - } - return undefined -} - function snapshotDraftValue(value: V | undefined): V | undefined { if (value === undefined) return undefined try { @@ -283,69 +181,64 @@ function snapshotDraftValue(value: V | undefined): V | undefined { } } -export type UserDraftHandle = { - get draft(): V | undefined - set draft(value: V | undefined) - /** - * Read the rev metadata stored alongside the current draft. Empty object - * if the entry has no draft or no rev was ever recorded. - */ - get meta(): UserDraftMeta - /** - * Set value AND rev metadata in one write (no extra persist). Later - * `draft = X` writes preserve the rev metadata. - */ - setDraftAndMeta(value: V | undefined, meta: UserDraftMeta): void - /** - * Update rev metadata without touching the value. `{ force: true }` also - * persists synchronously — use when this may be the entry's first write, - * else the ack is lost on remount. - */ - setMeta(meta: UserDraftMeta, opts?: { force?: boolean }): void -} +/** + * Top-level fields IGNORED when diffing a draft against its deployed + * baseline. `permissioned_as` / `preserve_permissioned_as` are deploy + * run-as directives, not draft content, and the editor round-trips them + * asymmetrically (`preserve_…` rebuilt as `!!cfg.permissioned_as` on load + * but `|| undefined` on build) — keeping them produces a phantom banner. + * + * The rest are server-managed read-time metadata that ride along on the + * loaded deployed payload but never appear in the editor's draft content, so + * comparing them would mask a true baseline match: + * `draft_saved_at` (the draft's own save time), `edited_at` (deploy time), + * `edited_by` (deploy author), `workspace_id`, `version_id` (deployed version), + * and `is_draft` (backend presence flag). + */ +const DRAFT_COMPARE_IGNORED_FIELDS = [ + 'permissioned_as', + 'preserve_permissioned_as', + 'extra_perms', + 'draft_saved_at', + 'edited_at', + 'edited_by', + 'workspace_id', + 'version_id', + 'parent_version', + 'is_draft' +] as const /** - * JSON round-trip normalization. localStorage persistence stringifies the - * draft, which silently drops keys whose value is `undefined`, turns `Date` - * into a string, etc. A freshly-built config object (e.g. a trigger editor's - * `getXConfig()`) keeps those `undefined`-valued keys, so a raw - * `deepEqual(persistedDraft, freshConfig)` reports spurious differences - * (`{ a: undefined }` ≠ `{}`). Normalize BOTH sides through the same - * round-trip before comparing. Returns the input unchanged if it can't be - * serialized (e.g. a cyclic structure) — better a false "differs" than a - * throw inside a load/effect path. + * Normalize one side of a draft-vs-baseline comparison: JSON round-trip + * (drafts are stored as json server-side, which strips `undefined` keys, + * so `{ labels: undefined }` and `{}` must compare equal) and drop the + * ignored fields above. Returns the input unchanged if unserializable. */ -export function normalizeForCompare(value: V | undefined): V | undefined { - if (value === undefined) return undefined +export function normalizeDraftForCompare(value: V): V { try { - return JSON.parse(JSON.stringify(value)) as V + const v = JSON.parse(JSON.stringify(value)) + if (v !== null && typeof v === 'object' && !Array.isArray(v)) { + for (const f of DRAFT_COMPARE_IGNORED_FIELDS) delete v[f] + } + return v as V } catch { return value } } /** - * Whether the persisted local autosave (`localDraft`, as returned by - * `UserDraft.get`) meaningfully differs from the freshly-built - * `currentConfig`. Editor restore guards use this to decide whether to - * overlay the local autosave and toast. - * - * Returns `false` when there is no local draft. Normalizes both sides (see - * `normalizeForCompare`) so a draft that round-trips equal to the deployed - * config — e.g. one written by merely opening then closing the editor with - * no edits — is correctly treated as "no meaningful draft" instead of - * spuriously triggering a restore on every reopen. - * - * Typed as a guard: a `true` result narrows `localDraft` to non-nullish - * `V`, mirroring the `localCfg && …` narrowing it replaces so call sites - * can pass the draft straight into `loadXConfig(...)` without re-checking. + * Normalized deep equality for draft values — THE "diverges from deployed + * baseline" check. Editors must use it for BOTH their "unsaved changes" + * banner and the `discardIf` predicate so the two can't disagree. */ -export function localDraftDiffers( - localDraft: V | undefined | null, - currentConfig: V -): localDraft is V { - if (localDraft === undefined || localDraft === null) return false - return !deepEqual(normalizeForCompare(localDraft), normalizeForCompare(currentConfig)) +export function draftValuesEqual(a: unknown, b: unknown): boolean { + if (a === undefined || b === undefined) return a === b + return deepEqual(normalizeDraftForCompare(a), normalizeDraftForCompare(b)) +} + +export type UserDraftHandle = { + get draft(): V | undefined + set draft(value: V | undefined) } export const UserDraft = { @@ -354,70 +247,22 @@ export const UserDraft = { const mk = mapKey(ws, itemKind, path) const entry = entries.get(mk) if (entry) { - // Static writes are external mutations. Update live observers and - // force the storage slot to match, even if the live entry still has - // its initial-write skip armed. - const current = untrack(() => entry.state.val as StoredDraft | undefined) - const meta = extractMeta(current) - entry.state.setWithoutPersist(wrap(value, meta)) - persistDirect(localStorageKey(ws, itemKind, path), value, meta) - return + // The reactive effect in `acquireEntry` observes this write + // and POSTs it. + entry.state.val = value + } else { + // No live handle: remember the value so a same-tab read-after-write + // observes it synchronously (the syncer POST below is debounced), + // then persist. The next editor mount re-fetches from the backend. + rememberWrite(ws, itemKind, path, value) + void UserDraftDbSyncer.save({ workspace: ws, itemKind, path, value }) } - // No live handle: preserve any persisted meta so the staleness - // signal survives a write while the editor is closed. - const existing = readPersisted(localStorageKey(ws, itemKind, path)) - try { - localStorage.setItem( - localStorageKey(ws, itemKind, path), - JSON.stringify(stamp(wrap(value, extractMeta(existing)))) - ) - } catch (e) { - console.error('UserDraft.save: localStorage write failed', e) - } - }, - - setDraftAndMeta( - itemKind: UserDraftItemKind, - path: string, - value: V | undefined, - meta: UserDraftMeta, - opts?: UserDraftOptions - ): void { - const ws = resolveWorkspace(opts) - const mk = mapKey(ws, itemKind, path) - const entry = entries.get(mk) - if (entry) { - // Static writes represent explicit external draft mutations. A - // freshly acquired live entry may still have the initial-write skip - // armed, so force the storage slot to match the live value. - entry.state.setWithoutPersist(wrap(value, meta)) - persistDirect(localStorageKey(ws, itemKind, path), value, meta) - return - } - persistDirect(localStorageKey(ws, itemKind, path), value, meta) }, /** - * Autosave gate: persist `value` only when it differs (after - * `normalizeForCompare`) from the `deployed` baseline; otherwise remove - * any draft. Without this, opening and closing an editor with no edits - * would leave a no-op draft that `has()` / restore guards treat as - * unsaved work. + * Current draft value from the in-memory cell. `undefined` when no + * editor has mounted a handle for this key in this tab. */ - saveIfChanged( - itemKind: UserDraftItemKind, - path: string, - value: V, - deployed: V | undefined, - opts?: UserDraftOptions - ): void { - if (deepEqual(normalizeForCompare(value), normalizeForCompare(deployed))) { - UserDraft.remove(itemKind, path, opts) - } else { - UserDraft.save(itemKind, path, value, opts) - } - }, - get( itemKind: UserDraftItemKind, path: string, @@ -426,127 +271,130 @@ export const UserDraft = { const ws = resolveWorkspace(opts) const mk = mapKey(ws, itemKind, path) const entry = entries.get(mk) - if (entry) { - return snapshotDraftValue(unwrap(entry.state.val as StoredDraft | undefined)) - } - return snapshotDraftValue(unwrap(readPersisted(localStorageKey(ws, itemKind, path)))) + if (entry) return snapshotDraftValue(entry.state.val as V | undefined) + const cached = writtenCache.get(mk) + if (cached) return snapshotDraftValue(cached.val as V | undefined) + return undefined }, /** - * Update the rev metadata for an entry without touching the value, and - * persist immediately. Used by editor routes that don't hold a live - * handle (apps, raw apps) — they read the local draft via `UserDraft.get` - * and the handle is created later inside the child editor. - * - * No-op when the entry has no draft to attach meta to. - */ - saveMeta( - itemKind: UserDraftItemKind, - path: string, - meta: UserDraftMeta, - opts?: UserDraftOptions - ): void { - const ws = resolveWorkspace(opts) - const mk = mapKey(ws, itemKind, path) - const entry = entries.get(mk) - if (entry) { - const current = untrack(() => entry.state.val as StoredDraft | undefined) - if (current === undefined) return - entry.state.val = wrap(current.value, meta) - } - const existing = readPersisted(localStorageKey(ws, itemKind, path)) - if (existing === undefined) return - persistDirect(localStorageKey(ws, itemKind, path), existing.value, meta) - }, - - /** - * Read the rev metadata for the entry. Returns an empty object if there - * is no entry. Useful for staleness checks before reading the draft. - */ - getMeta(itemKind: UserDraftItemKind, path: string, opts?: UserDraftOptions): UserDraftMeta { - const ws = resolveWorkspace(opts) - const mk = mapKey(ws, itemKind, path) - const entry = entries.get(mk) - if (entry) return extractMeta(entry.state.val as StoredDraft | undefined) - return extractMeta(readPersisted(localStorageKey(ws, itemKind, path))) - }, - - /** - * Whether a draft currently exists for (workspace, itemKind, path). - * Falls back to the persisted localStorage entry when no live handle is - * registered. Useful for distinguishing "first visit" from "returning - * visit with unsaved local changes". + * Whether a draft exists for this key in the in-memory cache. False when + * no editor has mounted a handle for it yet. */ has(itemKind: UserDraftItemKind, path: string, opts?: UserDraftOptions): boolean { const ws = resolveWorkspace(opts) const mk = mapKey(ws, itemKind, path) const entry = entries.get(mk) if (entry) return entry.state.val !== undefined - return readPersisted(localStorageKey(ws, itemKind, path)) !== undefined + return writtenCache.get(mk)?.val !== undefined }, remove(itemKind: UserDraftItemKind, path: string, opts?: UserDraftOptions): void { const ws = resolveWorkspace(opts) - try { - localStorage.removeItem(localStorageKey(ws, itemKind, path)) - } catch (e) { - console.error('UserDraft.remove: localStorage remove failed', e) + const mk = mapKey(ws, itemKind, path) + const entry = entries.get(mk) + if (entry) { + // Clear the cell so live observers see the delete; arm + // `skipNextSync` since we POST the `null` explicitly below. + entry.skipNextSync = true + entry.state.val = undefined } + writtenCache.delete(mk) + void UserDraftDbSyncer.save({ workspace: ws, itemKind, path, value: null }) }, clear(itemKind: UserDraftItemKind, path: string, opts?: UserDraftOptions): void { UserDraft.discard(itemKind, path, undefined, opts) }, + /** + * Suspend reactive sync for this key: writes still update the cell and + * subscribers but don't POST. Use to bracket programmatic bootstrap + * mutations (seeding `initialCode`, app init) so they don't appear as + * the user's "first edit". + * + * Safe BEFORE the entry is live (queued, applied on acquire). MUST pair + * every `stopSync` with a `restartSync` — forgetting silently disables + * autosave for the rest of the session. + */ + stopSync(itemKind: UserDraftItemKind, path: string, opts?: UserDraftOptions): void { + const ws = resolveWorkspace(opts) + const mk = mapKey(ws, itemKind, path) + const entry = entries.get(mk) + if (entry) entry.syncSuspended = true + else pendingSuspensions.add(mk) + }, + + /** + * Resume reactive sync after `stopSync`. Subsequent differing writes + * POST normally; writes made during the suspension are dropped from the + * server's view (the cell still reflects them). Also clears a queued + * (pre-acquire) suspension. + */ + restartSync(itemKind: UserDraftItemKind, path: string, opts?: UserDraftOptions): void { + const ws = resolveWorkspace(opts) + const mk = mapKey(ws, itemKind, path) + pendingSuspensions.delete(mk) + const entry = entries.get(mk) + if (entry) entry.syncSuspended = false + }, + + /** + * Load `value` into the cell as a SEED (baseline reload / new-draft + * template) that must NOT POST as the user's edit. Reactive readers + * update immediately; the sync effect adopts it as the new baseline. + * + * Prefer over the `stopSync`/`restartSync` bracket for a single write — + * scoped to exactly this write, nothing to forget to resume. The bracket + * is still needed when a write fans out across components (e.g. an + * editor's `initContent` cascading into the bound value). + * + * No-op if the entry isn't live yet (acquire via `use`/`useMany` first). + */ + seed(itemKind: UserDraftItemKind, path: string, value: V, opts?: UserDraftOptions): void { + const ws = resolveWorkspace(opts) + const mk = mapKey(ws, itemKind, path) + const entry = entries.get(mk) + if (!entry) return + entry.seedNextWrite = true + entry.state.val = snapshotDraftValue(value) + }, + + /** + * Currently-mounted live entries for `workspace` (in-tab only — for a + * workspace-wide view call `DraftService` directly). + */ list(opts?: UserDraftListOptions): UserDraftEntry[] { const ws = resolveWorkspace(opts) const itemKinds = opts?.itemKinds ?? USER_DRAFT_ITEM_KINDS - const out = new Map>() - - if (typeof localStorage !== 'undefined') { - const keys: string[] = [] - for (let i = 0; i < localStorage.length; i++) { - const key = localStorage.key(i) - if (key != null && key.startsWith(`userdraft/w/${ws}/`)) keys.push(key) - } - for (const key of keys) { - const parsed = parseLocalStorageKey(key, ws, itemKinds) - if (!parsed) continue - const stored = readPersisted(key) - if (stored === undefined) continue - out.set(mapKey(ws, parsed.itemKind, parsed.path), { - workspace: ws, - itemKind: parsed.itemKind, - path: parsed.path, - value: snapshotDraftValue(unwrap(stored)), - meta: extractMeta(stored), - persisted: true, - live: false - }) - } - } - + const out: UserDraftEntry[] = [] + const seen = new Set() for (const entry of entries.values()) { if (entry.workspace !== ws || !itemKinds.includes(entry.itemKind)) continue - const stored = untrack(() => entry.state.val as StoredDraft | undefined) - const mk = mapKey(entry.workspace, entry.itemKind, entry.path) - if (stored === undefined) { - out.delete(mk) - continue - } - const existing = out.get(mk) - out.set(mk, { + const val = untrack(() => entry.state.val as V | undefined) + if (val === undefined) continue + seen.add(mapKey(entry.workspace, entry.itemKind, entry.path)) + out.push({ workspace: entry.workspace, itemKind: entry.itemKind, path: entry.path, - value: snapshotDraftValue(unwrap(stored)), - meta: extractMeta(stored), - persisted: existing?.persisted ?? false, - live: true + value: snapshotDraftValue(val) }) } - - return Array.from(out.values()) + // Drafts written without a live entry (e.g. global AI chat) live only in + // `writtenCache`; surface them too so the list matches what `get` returns. + for (const cached of writtenCache.values()) { + if (cached.workspace !== ws || !itemKinds.includes(cached.itemKind)) continue + const mk = mapKey(cached.workspace, cached.itemKind, cached.path) + if (seen.has(mk)) continue + out.push({ + workspace: cached.workspace, + itemKind: cached.itemKind, + path: cached.path, + value: snapshotDraftValue(cached.val as V | undefined) + }) + } + return out }, setLiveEditorDraft(spec: LiveEditorDraftSpec): void { @@ -578,74 +426,145 @@ export const UserDraft = { }, /** - * Like `remove`, but also resets any live handle's `draft` to - * `fallback` in-memory (so reactive readers see it immediately) and - * skips re-persisting it, leaving the LS slot empty until the next real - * edit. Pass the deployed baseline as `fallback`. + * Like `remove`, but also resets any live handle's `draft` to `fallback` + * in-memory (reactive readers see it at once). The explicit `value: null` + * POST is the canonical delete; the cell update is UI convenience. * - * `fallback` is deep-cloned before being installed — otherwise a caller - * who passes their own live `$state` baseline (e.g. resource/variable - * editors' `initialStates[ws]`) would end up with `handle.draft` and the - * baseline pointing at the *same* proxy; subsequent edits would mutate - * both sides in lock-step and the dirty check would never fire. + * `fallback` MUST be deep-cloned: otherwise a caller passing their own + * live `$state` baseline (e.g. `initialStates[ws]`) would share a proxy + * with `handle.draft`, so edits mutate both sides and the dirty check + * never fires. */ discard( itemKind: UserDraftItemKind, path: string, fallback: V | undefined, - opts?: UserDraftOptions + opts?: UserDraftOptions & { + /** Mark the `value: null` POST as a reactive autosave (subject to + * the auto-save toggle) instead of an explicit action. Set by the + * trigger persist-effect's at-baseline discard; explicit discards + * leave it unset. */ + auto?: boolean + } ): void { const ws = resolveWorkspace(opts) const mk = mapKey(ws, itemKind, path) const entry = entries.get(mk) const safeFallback = snapshotDraftValue(fallback) if (entry) { - // Drop any queued debounced write owned by this live entry before - // resetting the in-memory value. Otherwise a timer from the old - // entry can outlive unmount and later delete a freshly written - // draft for the same key. - entry.state.setWithoutPersist(wrap(safeFallback) as StoredDraft | undefined) - } - try { - localStorage.removeItem(localStorageKey(ws, itemKind, path)) - } catch (e) { - console.error('UserDraft.discard: localStorage remove failed', e) + entry.skipNextSync = true + entry.state.val = safeFallback } + // The draft is deleted server-side (the `null` POST below); the fallback + // only resets the live handle's UI. Drop the cache so a no-entry read + // reports "no draft" rather than the discarded value. + writtenCache.delete(mk) + void UserDraftDbSyncer.save({ workspace: ws, itemKind, path, value: null, auto: opts?.auto }) }, use( itemKind: UserDraftItemKind, path: string, - opts?: UserDraftUseOptions + opts?: UserDraftOptions & { + /** See the `useMany` spec field. Default `false`. */ + canBeDisabled?: boolean + /** See the `useMany` spec field. Captured once on first acquire. */ + discardIf?: (val: V) => boolean + } ): UserDraftHandle { - // `use()` is a single-spec wrapper around `useMany`. We untrack the - // getter so that reactive opts (e.g. `$workspaceStore`) are captured - // once at call time — the current `use()` contract is "the handle - // stays bound to this workspace until the component unmounts." Use - // `useMany` directly if you want spec changes to release/acquire - // entries as you go. + // Single-spec wrapper around `useMany`. `untrack` captures reactive + // opts (e.g. `$workspaceStore`) once: the handle stays bound to this + // workspace until unmount. For reactive `(kind, path)` use `useReactive`. const handles = UserDraft.useMany(() => untrack(() => [ { itemKind, path, workspace: opts?.workspace, - defaultValue: opts?.defaultValue + canBeDisabled: opts?.canBeDisabled, + discardIf: opts?.discardIf } ]) ) return handles[0] }, - useMany(getSpecs: () => UserDraftSpec[]): UserDraftHandle[] { - // Reactive handles array, reconciled against the latest `getSpecs()` - // output. Indices line up with the spec array. Handles for the same - // (workspace, kind, path) tuple are reused across reconciles so - // callers can capture a reference and keep it alive — only the - // underlying entry's refcount moves. + /** + * Reactive single-spec variant of `use()`. `getSpec` is read inside the + * `useMany` reconcile, so a changed `(workspace, kind, path)` releases the + * old entry and acquires a new one. The returned object is a stable proxy + * forwarding `draft` to the current handle, so `bind:` lvalues survive + * re-keying. Use for reactive paths (`/scripts/edit/[...path]`). + */ + useReactive( + getSpec: () => { + itemKind: UserDraftItemKind + path: string + workspace?: string + canBeDisabled?: boolean + /** See the `useMany` spec field. Seeds the cell on first acquire + * without POSTing. Pass a STABLE reference (read it under `untrack`) + * — it's consumed once, so re-reading reactive state here only churns + * the reconcile. */ + defaultValue?: V + /** See the `useMany` spec field. Captured per re-keyed acquire. */ + discardIf?: (val: V) => boolean + } + ): UserDraftHandle { + const handles = UserDraft.useMany(() => [getSpec()]) + return { + get draft(): V | undefined { + return handles[0]?.draft + }, + set draft(value: V | undefined) { + const h = handles[0] + if (h) h.draft = value + } + } + }, + + useMany( + getSpecs: () => { + itemKind: UserDraftItemKind + path: string + workspace?: string + /** + * Value the entry's cell is seeded with on first acquire. Swallowed + * by the syncer's seed guard so it never POSTs. Ignored if the + * entry already exists (refcount > 0, e.g. another live handle + * keeps its value). + */ + defaultValue?: V + /** + * Predicate: is the value about to autosave back at the deployed + * baseline? When true, the mirror POSTs a delete instead of a + * baseline-equal copy — keeping it would leave `is_draft` (and the + * list `*`) stuck on. MUST use the same comparison as the "unsaved + * changes" banner (`draftValuesEqual`) so the two can't disagree. + * Return false for draft-only items (no deployed copy — deleting on + * equality would destroy the item). Captured on first acquire. + */ + discardIf?: (val: V) => boolean + /** + * Subject autosaves to the auto-save toggle. Only the full-page + * editors (script / flow / app / raw app) opt in; while off their + * keystroke saves park for Ctrl/Cmd+S. Default `false`: drawer + * editors always sync. Captured on first acquire. + */ + canBeDisabled?: boolean + }[] + ): UserDraftHandle[] { + // Handles array reconciled against `getSpecs()`, indices aligned. + // Same-key handles are reused across reconciles so callers can hold + // a stable reference — only the entry's refcount moves. const handles = $state[]>([]) const acquired = new Set() const handleCache = new Map>() + // `defaultValue` reference last used to seed each detached (empty-path) + // handle. The reference is stable within an editing session but swapped + // for a fresh clone each time the caller restarts (e.g. reopening the + // new-item drawer) — so a change here means "re-seed", not "live edit". + const detachedSeeds = new Map() function reconcile() { const specs = getSpecs() @@ -653,12 +572,52 @@ export const UserDraft = { const next: UserDraftHandle[] = [] for (const spec of specs) { - const ws = spec.workspace ?? resolveWorkspace() - const mk = mapKey(ws, spec.itemKind, spec.path) + // Resolve the workspace WITHOUT throwing: a reactive caller (e.g. an + // SDK editor mounted before login) may not have one yet. An absent + // workspace is handled like an empty path below, so the handle + // re-keys into a real entry once the workspace resolves. + const ws = spec.workspace ?? get(workspaceStore) ?? undefined + const mk = mapKey(ws ?? '', spec.itemKind, spec.path) + + // No workspace yet, or empty path = no draftable item (e.g. a + // read-only historical-hash view that still binds an editor value). + // Acquiring would mirror edits into an unroutable + // `POST /drafts/update/kind/` (permanent "Save failed"). + // Hand out a detached, local-only handle instead. + if (!ws || !spec.path) { + seen.add(mk) + let handle = handleCache.get(mk) + // Drop the cached handle when the caller hands in a fresh + // `defaultValue` reference (reopening the new-item drawer seeds a + // new clone) so the rebuilt handle re-seeds instead of replaying + // the previous session's edits. Stable reference within a session + // means live edits are never clobbered. + if (handle && detachedSeeds.get(mk) !== spec.defaultValue) { + handleCache.delete(mk) + handle = undefined + } + if (!handle) { + // Seed with `defaultValue` so consumers (e.g. the new-variable + // drawer, whose path is empty until the user types one) get a + // populated cell to bind their form to instead of `undefined`. + handle = makeDetachedHandle(spec.defaultValue) + handleCache.set(mk, handle) + detachedSeeds.set(mk, spec.defaultValue) + } + next.push(handle) + continue + } seen.add(mk) if (!acquired.has(mk)) { - acquireEntry(ws, spec.itemKind, spec.path, spec.defaultValue) + acquireEntry( + ws, + spec.itemKind, + spec.path, + spec.defaultValue, + spec.discardIf as ((val: unknown) => boolean) | undefined, + spec.canBeDisabled ?? false + ) acquired.add(mk) } let handle = handleCache.get(mk) @@ -677,24 +636,52 @@ export const UserDraft = { } } - // Skip no-op mutations (handles are cached by mapKey, so an - // unchanged spec set yields reference-equal arrays). `untrack` so - // this effect doesn't subscribe to its own `handles` write — - // otherwise it self-loops (`effect_update_depth_exceeded`). - // Downstream notification still propagates. + // Detached handles (empty-path) live only in `handleCache` — they're + // never in `acquired`. Drop any that fell out of the specs so they + // don't leak and a later reappearance rebuilds from scratch. + for (const mk of [...handleCache.keys()]) { + if (!acquired.has(mk) && !seen.has(mk)) { + handleCache.delete(mk) + detachedSeeds.delete(mk) + } + } + + // Skip no-op mutations (cached handles → reference-equal arrays). + // `untrack` so this effect doesn't subscribe to its own `handles` + // write — otherwise it self-loops (`effect_update_depth_exceeded`). untrack(() => { const unchanged = handles.length === next.length && handles.every((h, i) => h === next[i]) if (!unchanged) handles.splice(0, handles.length, ...next) }) } - // Synchronous initial reconcile so single-spec callers (`use()`) get a - // populated `handles[0]` before the function returns. Reactive reads - // inside `getSpecs()` here are intentionally not tracked — the - // `$effect` below picks up any subsequent dependency changes. + // Synchronous initial reconcile so single-spec callers get a populated + // `handles[0]` before returning. `untrack` here — the `$effect` below + // picks up subsequent changes. untrack(reconcile) $effect(reconcile) onDestroy(() => { + // Flush each entry's pending autosave BEFORE releasing it. SPA + // nav doesn't fire `pagehide`, so a debounced edit would silently + // vanish when the editor unmounts mid-typing. Fire-and-forget — + // the POST rides the runner's own lifetime, which outlives this + // component, so destroying the cell here doesn't cancel it. + // + // `honorAutosaveToggle`: this unmount flush is an implicit autosave, + // so a toggle-aware handle whose auto-save is off must NOT persist on + // leave — the editor's UnsavedConfirmationModal warns the user instead. + for (const mk of acquired) { + const entry = entries.get(mk) + if (!entry) continue + void UserDraftDbSyncer.flush( + { + workspace: entry.workspace, + itemKind: entry.itemKind, + path: entry.path + }, + { honorAutosaveToggle: true } + ) + } for (const mk of acquired) releaseEntry(mk) acquired.clear() handleCache.clear() @@ -708,7 +695,9 @@ function acquireEntry( workspace: string, itemKind: UserDraftItemKind, path: string, - defaultValue: unknown + defaultValue?: unknown, + discardIf?: (val: unknown) => boolean, + canBeDisabled = false ): void { const mk = mapKey(workspace, itemKind, path) const existing = entries.get(mk) @@ -716,40 +705,112 @@ function acquireEntry( existing.count++ return } - // `useLocalStorageValue`'s internal persist `$effect` would otherwise - // parent to `useMany`'s reconcile effect and be torn down on the next - // reconcile. `$effect.root` gives the entry its own scope, disposed only - // by `releaseEntry`. - const useLocalStorageOptions = { - // First value is the baseline (don't persist it); coalesce edits. - saveInitialValue: false, - debounce: 500, - // Stamp `lastWrittenAt` at persist time so deep mutations also bump - // the GC clock (the setter doesn't re-run for those). - transformBeforePersist: stamp - } as const + // Seed the cell with `defaultValue` (deep-cloned). Swallowed by + // `skipNextWrite` below — it never POSTs. + const seed = defaultValue !== undefined ? snapshotDraftValue(defaultValue) : undefined + // `$effect.root` gives the entry its own scope, disposed only by + // `releaseEntry`. Without it the sync `$effect` would parent to + // `useMany`'s reconcile effect and die on the next reconcile. let stateRef: DraftState | undefined const destroyRoot = $effect.root(() => { - stateRef = useLocalStorageValue | undefined>( - localStorageKey(workspace, itemKind, path), - wrap(defaultValue), - undefined, - useLocalStorageOptions - ) + const cell = $state<{ val: unknown }>({ val: seed }) + stateRef = cell as DraftState + // Mirror every observable change of `cell.val` to the syncer. + // `readFieldsRecursively` walks the value so deep mutations + // (`handle.draft.content = '...'`) re-fire the effect — reading + // `cell.val` alone only subscribes to the proxy root. + // + // `lastSerialized` + `skipNextWrite` dedup no-op updates and treat + // the FIRST change after mount as the seed/restore (no POST), so + // landing on `?new_draft` doesn't sync until the user edits. + // + // `cell.val === undefined` is the delete signal (`value: null`). + // `skipNextSync` lets callers that already POSTed (`discard`, + // `remove`) suppress the duplicate fire from their own write. + let lastSerialized: string | undefined = undefined + let skipNextWrite = true + $effect(() => { + const val = cell.val + if (val !== undefined) readFieldsRecursively(val) + const next = val === undefined ? undefined : JSON.stringify(val) + if (next === lastSerialized) { + // No-op write. If a `seed` re-seeded the value already in the + // cell, its one-shot flag consumed nothing — defuse it here or + // it lingers and swallows the user's NEXT edit. (`skipNextWrite` + // stays armed: an undefined-seeded cell's initial run lands + // here, and page editors rely on it to swallow their load write.) + const e = entries.get(mk) + if (e?.seedNextWrite) e.seedNextWrite = false + return + } + lastSerialized = next + if (skipNextWrite) { + skipNextWrite = false + // One write consumes BOTH one-shot guards: a `seed` on a fresh + // entry (still armed with the first-write skip) must not leave + // `seedNextWrite` behind to swallow the user's first edit. + const fresh = entries.get(mk) + if (fresh?.seedNextWrite) fresh.seedNextWrite = false + return + } + const entry = entries.get(mk) + // `seed` write: baseline already advanced above; don't POST. + if (entry?.seedNextWrite) { + entry.seedNextWrite = false + return + } + if (entry?.skipNextSync) { + entry.skipNextSync = false + return + } + // `syncSuspended` swallows the POST but `lastSerialized` advanced + // above, so only writes made during suspension are dropped — the + // first change after resume is still detected. + if (entry?.syncSuspended) return + // At the deployed baseline → sync a delete, not a baseline-equal + // copy. `untrack` so reactive reads in the predicate (the editor's + // post-deploy baseline) don't re-fire the mirror. + const atBaseline = untrack(() => val !== undefined && (discardIf?.(val) ?? false)) + void UserDraftDbSyncer.save({ + workspace, + itemKind, + path, + value: val === undefined || atBaseline ? null : val, + // Reactive keystroke mirror — `auto`, so suppressed while the + // auto-save toggle is off (for `canBeDisabled` handles). + auto: true, + canBeDisabled + }) + }) }) if (stateRef) { - entries.set(mk, { count: 1, workspace, itemKind, path, state: stateRef, destroyRoot }) + entries.set(mk, { + count: 1, + workspace, + itemKind, + path, + state: stateRef, + skipNextSync: false, + syncSuspended: pendingSuspensions.delete(mk), + seedNextWrite: false, + destroyRoot + }) return } // Fallback for the vitest runtime where `$effect.root`'s callback isn't - // invoked. Unreachable in production (Svelte runs it synchronously). - const state = useLocalStorageValue | undefined>( - localStorageKey(workspace, itemKind, path), - wrap(defaultValue), - undefined, - useLocalStorageOptions - ) - entries.set(mk, { count: 1, workspace, itemKind, path, state }) + // invoked (unreachable in production). No sync effect — writes in tests + // stay in-memory. + const fallback = $state<{ val: unknown }>({ val: seed }) + entries.set(mk, { + count: 1, + workspace, + itemKind, + path, + state: fallback as DraftState, + skipNextSync: false, + syncSuspended: pendingSuspensions.delete(mk), + seedNextWrite: false + }) } function releaseEntry(mk: string): void { @@ -757,105 +818,49 @@ function releaseEntry(mk: string): void { if (!entry) return entry.count-- if (entry.count <= 0) { + // The live entry was authoritative while mounted; once gone, drop any + // cached write for this key so a later read falls back to the server + // rather than a value the editor may have changed in the meantime. + writtenCache.delete(mk) entry.destroyRoot?.() entries.delete(mk) } } +/** + * Local-only handle for empty-path specs: a reactive cell that supports + * `bind:` but is wired to nothing (no entry, no sync, no POSTs). For views + * that bind an editor value with no draftable item behind it. + */ +function makeDetachedHandle(defaultValue?: V): UserDraftHandle { + let val = $state(snapshotDraftValue(defaultValue)) + return { + get draft(): V | undefined { + return val + }, + set draft(value: V | undefined) { + val = value + } + } +} + function makeHandle( workspace: string, itemKind: UserDraftItemKind, path: string ): UserDraftHandle { - // The handle reads `entries.get(mk)` on every access. The entry it points - // at is stable as long as the refcount stays > 0 (which `useMany` keeps - // the case for as long as a spec references it). If the refcount drops to - // 0 and the entry is destroyed, reads return `undefined` rather than - // throwing — the consumer should already have been torn down by that point. + // Reads `entries.get(mk)` on every access; the entry is stable while + // refcount > 0. After destruction, reads return `undefined` rather than + // throwing — the consumer should already be torn down by then. const mk = mapKey(workspace, itemKind, path) const stateOf = (): DraftState | undefined => entries.get(mk)?.state return { get draft(): V | undefined { - return unwrap(stateOf()?.val as StoredDraft | undefined) + return stateOf()?.val as V | undefined }, set draft(value: V | undefined) { - // Preserve existing rev metadata on a value edit. `untrack` the - // read: callers often set this from inside a `$effect` mirroring - // `$state` into the handle; a tracked read would subscribe that - // effect to the cell it's about to write (self-loop → - // effect_update_depth_exceeded). const state = stateOf() - if (!state) return - const current = untrack(() => state.val as StoredDraft | undefined) - state.val = wrap(value, extractMeta(current)) - }, - get meta(): UserDraftMeta { - return extractMeta(stateOf()?.val as StoredDraft | undefined) - }, - setDraftAndMeta(value: V | undefined, meta: UserDraftMeta): void { - const state = stateOf() - if (!state) return - state.val = wrap(value, meta) - }, - setMeta(meta: UserDraftMeta, opts?: { force?: boolean }): void { - // Read under `untrack` for the same reason as `set draft` above — - // avoid making any surrounding effect re-fire on the write below. - const state = stateOf() - if (!state) return - const current = untrack(() => state.val as StoredDraft | undefined) - if (current === undefined) return - state.val = wrap(current.value, meta) - if (opts?.force) { - persistDirect(localStorageKey(workspace, itemKind, path), current.value, meta) - } - } - } -} - -/** - * Default GC retention window: 30 days. Entries that haven't been touched - * (no setter call, no deep-mutation persist) for this long are swept on - * the next `gcUserDrafts` invocation. - */ -export const USER_DRAFT_GC_MAX_AGE_MS = 30 * 24 * 60 * 60 * 1000 - -/** - * Sweep stale UserDraft entries from localStorage. Walks every - * `userdraft/w/...` key, checks its `lastWrittenAt` stamp, and removes - * any entry older than `maxAgeMs`. - * - * Entries written before `lastWrittenAt` was introduced lack the field; - * we backfill them to `now()` on first sighting so they participate in - * the next sweep cycle rather than getting wiped immediately. - * - * Safe to call on every load and on a timer (e.g. every 30 min) — live - * entries get their stamp refreshed on every persist, so the sweep only - * touches truly stale records. - */ -export function gcUserDrafts(maxAgeMs: number = USER_DRAFT_GC_MAX_AGE_MS): void { - if (typeof localStorage === 'undefined') return - const now = Date.now() - const cutoff = now - maxAgeMs - const keys: string[] = [] - for (let i = 0; i < localStorage.length; i++) { - const k = localStorage.key(i) - if (k != null && k.startsWith('userdraft/w/')) keys.push(k) - } - for (const key of keys) { - try { - const raw = localStorage.getItem(key) - if (raw == null) continue - const parsed = JSON.parse(raw) - if (parsed == null || typeof parsed !== 'object') continue - if (typeof parsed.lastWrittenAt !== 'number') { - // Pre-GC-feature entry. Backfill so the next sweep can decide. - parsed.lastWrittenAt = now - localStorage.setItem(key, JSON.stringify(parsed)) - continue - } - if (parsed.lastWrittenAt < cutoff) localStorage.removeItem(key) - } catch (e) { - console.error('UserDraft GC: failed to inspect', key, e) + if (state) state.val = value } } } @@ -864,4 +869,5 @@ export function gcUserDrafts(maxAgeMs: number = USER_DRAFT_GC_MAX_AGE_MS): void export function __resetUserDraftForTesting(): void { entries.clear() liveEditorDrafts.clear() + writtenCache.clear() } diff --git a/frontend/src/lib/userDraft.test.ts b/frontend/src/lib/userDraft.test.ts deleted file mode 100644 index 05679b823b..0000000000 --- a/frontend/src/lib/userDraft.test.ts +++ /dev/null @@ -1,1145 +0,0 @@ -import { describe, it, expect, beforeEach, vi } from 'vitest' - -// Capture onDestroy callbacks so we can simulate component teardown without -// a real component context. -const onDestroyCallbacks: Array<() => void> = [] - -vi.mock('svelte', async (importOriginal) => { - const actual = (await importOriginal()) as Record - return { - ...actual, - onDestroy: (fn: () => void) => { - onDestroyCallbacks.push(fn) - } - } -}) - -// Imported AFTER vi.mock so the module sees the mocked onDestroy. -const { UserDraft, normalizeForCompare, localDraftDiffers, __resetUserDraftForTesting } = - await import('./userDraft.svelte') -const { workspaceStore } = await import('./stores') -const { deleteGlobalDraft } = await import('./components/copilot/chat/global/userDraftAdapter') - -function flushDestroyCallbacks(): void { - const callbacks = onDestroyCallbacks.splice(0, onDestroyCallbacks.length) - for (const cb of callbacks) cb() -} - -// UserDraft.use debounces localStorage writes by 500 ms via -// useLocalStorageValue. Tests assert localStorage state synchronously after -// writes, so we use fake timers and call this helper to fast-forward past -// the debounce window before each assertion. -function flushPersist(): void { - vi.runAllTimers() -} - -// Helper: localStorage payloads are always wrapped as { value: } so -// future metadata fields can be added without breaking existing entries. -function wrapped(value: V): string { - return JSON.stringify({ value }) -} - -// Helper: read a localStorage entry, strip the GC `lastWrittenAt` stamp so -// assertions can stay focused on value + rev metadata. Real entries always -// carry `lastWrittenAt` once written; the GC tests below assert on it -// directly via `localStorage.getItem`. -function storedShape(key: string): string | null { - const raw = localStorage.getItem(key) - if (raw == null) return null - const parsed = JSON.parse(raw) - delete parsed.lastWrittenAt - return JSON.stringify(parsed) -} - -beforeEach(() => { - __resetUserDraftForTesting() - onDestroyCallbacks.length = 0 - localStorage.clear() - workspaceStore.set('test_ws') - vi.useFakeTimers() -}) - -describe('UserDraft.save / get / remove (no observers)', () => { - it('save writes a wrapped { value } payload under the workspace-scoped key', () => { - UserDraft.save('flow', 'u/me/myflow', { hello: 'world' }) - - expect(storedShape('userdraft/w/test_ws/flow/u/me/myflow')).toBe(wrapped({ hello: 'world' })) - }) - - it('get reads from a wrapped localStorage payload when no observer is registered', () => { - localStorage.setItem('userdraft/w/test_ws/script/u/me/script1', wrapped('code')) - - expect(UserDraft.get('script', 'u/me/script1')).toBe('code') - }) - - it('get returns undefined when nothing is stored', () => { - expect(UserDraft.get('flow', 'u/me/missing')).toBeUndefined() - }) - - it('get returns undefined when the stored payload is malformed', () => { - localStorage.setItem('userdraft/w/test_ws/flow/u/me/bad', 'not-json') - expect(UserDraft.get('flow', 'u/me/bad')).toBeUndefined() - }) - - it('get returns undefined when the stored payload is unwrapped (pre-migration entry)', () => { - // Drafts written before the wrapping was introduced look like the raw - // value rather than { value: ... }. They must be ignored rather than - // surface as undefined-shaped drafts. - localStorage.setItem('userdraft/w/test_ws/flow/u/me/raw', JSON.stringify({ hello: 'world' })) - expect(UserDraft.get('flow', 'u/me/raw')).toBeUndefined() - expect(UserDraft.has('flow', 'u/me/raw')).toBe(false) - }) - - it('remove clears the localStorage entry', () => { - UserDraft.save('app', 'u/me/app1', { grid: [] }) - expect(localStorage.getItem('userdraft/w/test_ws/app/u/me/app1')).not.toBeNull() - - UserDraft.remove('app', 'u/me/app1') - expect(localStorage.getItem('userdraft/w/test_ws/app/u/me/app1')).toBeNull() - }) - - it('uses the workspace from opts when provided', () => { - UserDraft.save('flow', 'u/me/f', 1, { workspace: 'other_ws' }) - - expect(storedShape('userdraft/w/other_ws/flow/u/me/f')).toBe(wrapped(1)) - // Default workspace key must remain empty. - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/f')).toBeNull() - }) - - it('supports trigger kinds as item kinds', () => { - UserDraft.save('trigger_kafka', 'u/me/topic1', { brokers: ['localhost:9092'] }) - - expect(storedShape('userdraft/w/test_ws/trigger_kafka/u/me/topic1')).toBe( - wrapped({ brokers: ['localhost:9092'] }) - ) - }) - - it('throws when neither opts.workspace nor $workspaceStore is set', () => { - workspaceStore.set(undefined) - expect(() => UserDraft.save('flow', 'u/me/x', 1)).toThrow(/no workspace/) - }) -}) - -describe('UserDraft live editor draft registry', () => { - it('stores the live editor storage path and effective path per workspace and kind', () => { - UserDraft.setLiveEditorDraft({ - itemKind: 'script', - storagePath: '', - effectivePath: 'u/me/generated_script' - }) - - expect(UserDraft.getLiveEditorDraft('script')).toEqual({ - workspace: 'test_ws', - itemKind: 'script', - storagePath: '', - effectivePath: 'u/me/generated_script' - }) - }) - - it('keeps live editor registrations isolated by workspace', () => { - UserDraft.setLiveEditorDraft({ - workspace: 'ws_a', - itemKind: 'flow', - storagePath: '', - effectivePath: 'u/me/a' - }) - UserDraft.setLiveEditorDraft({ - workspace: 'ws_b', - itemKind: 'flow', - storagePath: '', - effectivePath: 'u/me/b' - }) - - expect(UserDraft.getLiveEditorDraft('flow', { workspace: 'ws_a' })?.effectivePath).toBe( - 'u/me/a' - ) - expect(UserDraft.getLiveEditorDraft('flow', { workspace: 'ws_b' })?.effectivePath).toBe( - 'u/me/b' - ) - }) - - it('clears only the matching live editor storage path when provided', () => { - UserDraft.setLiveEditorDraft({ - itemKind: 'raw_app', - storagePath: '', - effectivePath: 'u/me/live_app' - }) - - UserDraft.clearLiveEditorDraft('raw_app', { storagePath: 'u/me/other' }) - expect(UserDraft.getLiveEditorDraft('raw_app')).toBeDefined() - - UserDraft.clearLiveEditorDraft('raw_app', { storagePath: '' }) - expect(UserDraft.getLiveEditorDraft('raw_app')).toBeUndefined() - }) - - // Pins the slot-collision contract behind the SessionWrapper - // `isActiveSession` gate. Without the gate, two warm-mounted session - // editors on the same (workspace, kind) — e.g. `/sessions` keeping 3 - // warm sessions and two of them have script editors open in the same - // workspace — both call setLiveEditorDraft and the hidden one can - // clobber the visible one's claim. The fix in - // {Script,Flow,RawApp}EditorView returns early when `isActiveSession` - // is false, so only the visible session writes to the slot. - it('collides per (workspace, kind) when two callers both set — the hidden session can hijack', () => { - UserDraft.setLiveEditorDraft({ - workspace: 'ws_collide', - itemKind: 'script', - storagePath: 'session_a_path', - effectivePath: 'u/me/a' - }) - UserDraft.setLiveEditorDraft({ - workspace: 'ws_collide', - itemKind: 'script', - storagePath: 'session_b_path', - effectivePath: 'u/me/b' - }) - // Last write wins — exactly the bug Codex flagged: a hidden warm - // session B mounted after the active A overwrites A's claim. - expect(UserDraft.getLiveEditorDraft('script', { workspace: 'ws_collide' })).toMatchObject({ - storagePath: 'session_b_path', - effectivePath: 'u/me/b' - }) - }) - - it('with the active-session gate, only the visible session claims the slot', () => { - // Simulate the effect bodies in {Script,Flow,RawApp}EditorView: each - // returns early when isActiveSession is false. Session A is active, - // Session B is warm-mounted but hidden. - function registerIfActive(opts: { - isActive: boolean - workspace: string - storagePath: string - effectivePath: string - }) { - if (!opts.isActive) return - UserDraft.setLiveEditorDraft({ - workspace: opts.workspace, - itemKind: 'flow', - storagePath: opts.storagePath, - effectivePath: opts.effectivePath - }) - } - registerIfActive({ - isActive: true, - workspace: 'ws_gate', - storagePath: 'session_a_path', - effectivePath: 'u/me/a' - }) - registerIfActive({ - isActive: false, - workspace: 'ws_gate', - storagePath: 'session_b_path', - effectivePath: 'u/me/b' - }) - expect(UserDraft.getLiveEditorDraft('flow', { workspace: 'ws_gate' })).toMatchObject({ - storagePath: 'session_a_path', - effectivePath: 'u/me/a' - }) - }) - - it('cleanup on the deactivating session is a no-op once the new active session has claimed the slot', () => { - // Active-session swap: A becomes hidden (cleanup runs), B becomes - // active and registers. Even if Svelte flushes B's effect before - // A's cleanup, A's cleanup is keyed on its own storagePath and is - // guarded so it doesn't clobber B's claim. Verified here by running - // the cleanups in reverse order. - UserDraft.setLiveEditorDraft({ - workspace: 'ws_swap', - itemKind: 'raw_app', - storagePath: 'session_b_path', - effectivePath: 'u/me/b' - }) - // A's cleanup runs after — should be a no-op. - UserDraft.clearLiveEditorDraft('raw_app', { - workspace: 'ws_swap', - storagePath: 'session_a_path' - }) - expect(UserDraft.getLiveEditorDraft('raw_app', { workspace: 'ws_swap' })).toMatchObject({ - storagePath: 'session_b_path' - }) - }) - - it('can remove persisted global draft storage without blanking the live editor', () => { - const draft = { path: 'u/me/live_script', content: 'export async function main() {}' } - localStorage.setItem('userdraft/w/test_ws/script/', wrapped(draft)) - const handle = UserDraft.use('script', '') - UserDraft.setLiveEditorDraft({ - itemKind: 'script', - storagePath: '', - effectivePath: 'u/me/live_script' - }) - - deleteGlobalDraft('test_ws', 'script', 'u/me/live_script', undefined, { - preserveLiveDraft: true - }) - flushPersist() - - expect(handle.draft).toEqual(draft) - expect(localStorage.getItem('userdraft/w/test_ws/script/')).toBeNull() - }) -}) - -describe('UserDraft.use() — observer sync', () => { - it('loads the existing localStorage value on first use', () => { - localStorage.setItem('userdraft/w/test_ws/flow/u/me/loaded', wrapped('preloaded')) - - const handle = UserDraft.use('flow', 'u/me/loaded') - expect(handle.draft).toBe('preloaded') - }) - - it('two handles on the same key share the same underlying state', () => { - const a = UserDraft.use('flow', 'u/me/shared') - const b = UserDraft.use('flow', 'u/me/shared') - - a.draft = 42 - expect(b.draft).toBe(42) - - b.draft = 99 - expect(a.draft).toBe(99) - }) - - it('save() propagates to live use() handles and persists immediately', () => { - const handle = UserDraft.use('flow', 'u/me/observed') - expect(handle.draft).toBeUndefined() - - UserDraft.save('flow', 'u/me/observed', 7) - expect(handle.draft).toBe(7) - expect(storedShape('userdraft/w/test_ws/flow/u/me/observed')).toBe(wrapped(7)) - - UserDraft.save('flow', 'u/me/observed', 9) - expect(handle.draft).toBe(9) - expect(storedShape('userdraft/w/test_ws/flow/u/me/observed')).toBe(wrapped(9)) - }) - - it('get() returns a cloneable snapshot of live handle values', () => { - const handle = UserDraft.use<{ path: string; nested: { value: number } }>('script', '') - handle.draft = { path: 'u/me/live', nested: { value: 1 } } - - const draft = UserDraft.get<{ path: string; nested: { value: number } }>('script', '') - expect(draft).toEqual({ path: 'u/me/live', nested: { value: 1 } }) - expect(draft).not.toBe(handle.draft) - expect(() => structuredClone(draft)).not.toThrow() - }) - - it('remove() clears localStorage without touching the in-memory handle', () => { - // Seed localStorage so the live handle initialises from it. - localStorage.setItem('userdraft/w/test_ws/flow/u/me/removed', wrapped(1)) - const handle = UserDraft.use('flow', 'u/me/removed') - expect(handle.draft).toBe(1) - - UserDraft.remove('flow', 'u/me/removed') - // Live handle keeps its current value — remove() only wipes the - // persisted side. This is what lets callers run UserDraft.remove - // during navigation without flickering the editor UI. - expect(handle.draft).toBe(1) - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/removed')).toBeNull() - }) - - it('discard() clears LS, resets the handle to the fallback, and does NOT re-persist', () => { - // Seed: handle holds a divergent local autosave. - localStorage.setItem('userdraft/w/test_ws/flow/u/me/discard', wrapped('local-edit')) - const handle = UserDraft.use('flow', 'u/me/discard') - expect(handle.draft).toBe('local-edit') - - // Reset to a known backend baseline. - UserDraft.discard('flow', 'u/me/discard', 'backend-baseline') - flushPersist() - - // In-memory handle reflects the fallback immediately. - expect(handle.draft).toBe('backend-baseline') - // LS is cleared and stays cleared — the fallback must NOT round-trip - // back into storage (that would make the next reload "restore" the - // fallback as if it were a real autosave). - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/discard')).toBeNull() - }) - - it('discard() with undefined fallback clears both LS and in-memory state', () => { - localStorage.setItem('userdraft/w/test_ws/flow/u/me/wipe', wrapped('local-edit')) - const handle = UserDraft.use('flow', 'u/me/wipe') - expect(handle.draft).toBe('local-edit') - - UserDraft.discard('flow', 'u/me/wipe', undefined) - flushPersist() - - expect(handle.draft).toBeUndefined() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/wipe')).toBeNull() - }) - - it('the second write through the handle setter persists to localStorage', () => { - const handle = UserDraft.use('flow', 'u/me/setter') - - // First write is the baseline — not persisted. - handle.draft = 'initial' - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/setter')).toBeNull() - - // Second (and onwards) persists. - handle.draft = 'persisted' - flushPersist() - expect(storedShape('userdraft/w/test_ws/flow/u/me/setter')).toBe(wrapped('persisted')) - }) - - it('setting handle.draft = undefined after edits removes the localStorage entry', () => { - const handle = UserDraft.use('flow', 'u/me/clear') - handle.draft = 'initial' // baseline, not persisted - handle.draft = 'edited' // persisted - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/clear')).not.toBeNull() - - handle.draft = undefined - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/clear')).toBeNull() - expect(handle.draft).toBeUndefined() - }) - - it('two handles in different workspaces are isolated', () => { - const a = UserDraft.use('flow', 'u/me/iso', { workspace: 'ws_a' }) - const b = UserDraft.use('flow', 'u/me/iso', { workspace: 'ws_b' }) - - a.draft = 1 - b.draft = 2 - - expect(a.draft).toBe(1) - expect(b.draft).toBe(2) - }) - - it('save() falls back to localStorage when no handle is registered', () => { - UserDraft.save('flow', 'u/me/noobs', 'fallback') - // First use() afterwards loads the persisted value. - const handle = UserDraft.use('flow', 'u/me/noobs') - expect(handle.draft).toBe('fallback') - }) -}) - -describe('UserDraft.use() — defaultValue', () => { - it('returns defaultValue when localStorage has no entry', () => { - const handle = UserDraft.use('flow', 'u/me/withdefault', { defaultValue: 'fallback' }) - - expect(handle.draft).toBe('fallback') - }) - - it('does not persist the defaultValue on first read', () => { - UserDraft.use('flow', 'u/me/lazyDefault', { defaultValue: 'fallback' }) - - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/lazyDefault')).toBeNull() - }) - - it('localStorage value wins over defaultValue', () => { - localStorage.setItem('userdraft/w/test_ws/flow/u/me/overridden', wrapped('persisted')) - - const handle = UserDraft.use('flow', 'u/me/overridden', { - defaultValue: 'fallback' - }) - - expect(handle.draft).toBe('persisted') - }) - - it('second write through the setter persists even though defaultValue was set', () => { - const handle = UserDraft.use('flow', 'u/me/writeDefault', { - defaultValue: 'fallback' - }) - - // First write is the initial-value baseline. - handle.draft = 'initial' - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/writeDefault')).toBeNull() - - handle.draft = 'modified' - flushPersist() - expect(storedShape('userdraft/w/test_ws/flow/u/me/writeDefault')).toBe(wrapped('modified')) - }) -}) - -describe('UserDraft — empty path (new-item drafts persist across reloads)', () => { - it('use() with empty path persists subsequent edits to localStorage', () => { - const handle = UserDraft.use('flow', '', { defaultValue: 0 }) - - // First write under saveInitialValue=false counts as the baseline and - // is skipped — only the user's subsequent edits persist. - handle.draft = 99 - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/flow/')).toBeNull() - handle.draft = 100 - flushPersist() - // The "+ Flow / + Script / …" buttons are expected to call - // `UserDraft.remove(kind, '')` to wipe before navigating; an - // unguarded /add reload therefore restores the previous session. - expect(storedShape('userdraft/w/test_ws/flow/')).toBe(wrapped(100)) - }) - - it('two handles with empty path share state per workspace', () => { - const a = UserDraft.use('flow', '') - const b = UserDraft.use('flow', '') - - a.draft = 1 - expect(b.draft).toBe(1) - - b.draft = 2 - expect(a.draft).toBe(2) - }) - - it('save() with empty path writes to localStorage when no handle is live', () => { - UserDraft.save('flow', '', 5) - expect(storedShape('userdraft/w/test_ws/flow/')).toBe(wrapped(5)) - }) - - it('get() with empty path falls back to localStorage when no handle is live', () => { - localStorage.setItem('userdraft/w/test_ws/flow/', wrapped(11)) - expect(UserDraft.get('flow', '')).toBe(11) - }) - - it('remove() with empty path clears localStorage', () => { - localStorage.setItem('userdraft/w/test_ws/flow/', wrapped(1)) - UserDraft.remove('flow', '') - expect(localStorage.getItem('userdraft/w/test_ws/flow/')).toBeNull() - }) -}) - -describe('UserDraft — rev metadata for staleness checks', () => { - it('setDraftAndMeta atomically stores value + rev, and the first write is still skipped', () => { - const handle = UserDraft.use('flow', 'u/me/atomic') - - // Single atomic write — under saveInitialValue=false this counts as the - // initial baseline and shouldn't hit localStorage yet. - handle.setDraftAndMeta('backendValue', { - remoteRev: 42, - remoteDraftRev: '2026-01-01T00:00:00Z' - }) - expect(handle.draft).toBe('backendValue') - expect(handle.meta).toEqual({ remoteRev: 42, remoteDraftRev: '2026-01-01T00:00:00Z' }) - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/atomic')).toBeNull() - - // A subsequent user edit persists *with* the rev metadata. - handle.draft = 'userEdit' - flushPersist() - expect(storedShape('userdraft/w/test_ws/flow/u/me/atomic')).toBe( - JSON.stringify({ - value: 'userEdit', - remoteRev: 42, - remoteDraftRev: '2026-01-01T00:00:00Z' - }) - ) - }) - - it('setMeta updates only the rev fields, preserving the value', () => { - const handle = UserDraft.use('flow', 'u/me/setmeta') - handle.setDraftAndMeta('initial', { remoteRev: 1 }) // baseline, not persisted - handle.draft = 'edited' // persisted with remoteRev: 1 - - handle.setMeta({ remoteRev: 2 }) - expect(handle.draft).toBe('edited') - expect(handle.meta).toEqual({ remoteRev: 2 }) - flushPersist() - expect(storedShape('userdraft/w/test_ws/flow/u/me/setmeta')).toBe( - JSON.stringify({ value: 'edited', remoteRev: 2 }) - ) - }) - - it('handle.draft setter preserves rev metadata across user edits', () => { - const handle = UserDraft.use<{ count: number }>('flow', 'u/me/preserve') - handle.setDraftAndMeta({ count: 0 }, { remoteRev: 'v1' }) - handle.draft = { count: 1 } // first edit, persisted - handle.draft = { count: 2 } // another edit - - expect(handle.meta).toEqual({ remoteRev: 'v1' }) - flushPersist() - expect(storedShape('userdraft/w/test_ws/flow/u/me/preserve')).toBe( - JSON.stringify({ value: { count: 2 }, remoteRev: 'v1' }) - ) - }) - - it('UserDraft.getMeta reads from localStorage when no live handle exists', () => { - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/getmeta', - JSON.stringify({ value: 'x', remoteRev: 7, remoteDraftRev: '2026-01-02' }) - ) - expect(UserDraft.getMeta('flow', 'u/me/getmeta')).toEqual({ - remoteRev: 7, - remoteDraftRev: '2026-01-02' - }) - }) - - it('UserDraft.getMeta returns empty object when there is no entry', () => { - expect(UserDraft.getMeta('flow', 'u/me/none')).toEqual({}) - }) - - it('UserDraft.save preserves persisted rev metadata when no live handle exists', () => { - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/savepreserve', - JSON.stringify({ value: 'old', remoteRev: 5 }) - ) - UserDraft.save('flow', 'u/me/savepreserve', 'new') - - expect(storedShape('userdraft/w/test_ws/flow/u/me/savepreserve')).toBe( - JSON.stringify({ value: 'new', remoteRev: 5 }) - ) - }) - - it('UserDraft.save persists immediately when a live handle exists', () => { - const handle = UserDraft.use('flow', 'u/me/live-save') - - UserDraft.save('flow', 'u/me/live-save', 'external') - - expect(handle.draft).toBe('external') - expect(storedShape('userdraft/w/test_ws/flow/u/me/live-save')).toBe(wrapped('external')) - }) - - it('UserDraft.save preserves live rev metadata while forcing persistence', () => { - const handle = UserDraft.use('flow', 'u/me/live-save-meta') - handle.setDraftAndMeta('baseline', { remoteRev: 5 }) - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/live-save-meta')).toBeNull() - - UserDraft.save('flow', 'u/me/live-save-meta', 'external') - - expect(handle.draft).toBe('external') - expect(storedShape('userdraft/w/test_ws/flow/u/me/live-save-meta')).toBe( - JSON.stringify({ value: 'external', remoteRev: 5 }) - ) - }) - - it('handle.meta is empty for a draft persisted without rev (forward compat with older entries)', () => { - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/legacy', - JSON.stringify({ value: 'no-rev' }) - ) - const handle = UserDraft.use('flow', 'u/me/legacy') - expect(handle.draft).toBe('no-rev') - expect(handle.meta).toEqual({}) - }) - - it('setMeta({ force: true }) persists immediately, bypassing the first-write skip', () => { - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/forceack', - JSON.stringify({ value: 'edited', remoteRev: 'v1' }) - ) - const handle = UserDraft.use('flow', 'u/me/forceack') - - // Without force, this is the entry's first state mutation and gets - // swallowed by saveInitialValue=false — localStorage would still - // hold the old remoteRev. - handle.setMeta({ remoteRev: 'v2' }, { force: true }) - - expect(handle.meta).toEqual({ remoteRev: 'v2' }) - expect(storedShape('userdraft/w/test_ws/flow/u/me/forceack')).toBe( - JSON.stringify({ value: 'edited', remoteRev: 'v2' }) - ) - }) -}) - -describe('checkStaleness', () => { - let checkStaleness: ( - meta: { remoteRev?: string | number; remoteDraftRev?: string | number }, - currentRev: string | number | undefined, - currentDraftRev?: string | number | undefined - ) => 'draft' | 'version' | null - - beforeEach(async () => { - // Re-import to dodge ESM caching surprises across test files. - ;({ checkStaleness } = await import('./userDraft.svelte')) - }) - - it('returns null for legacy entries with no recorded rev', () => { - expect(checkStaleness({}, 'h1', '2026-01-01')).toBeNull() - }) - - it('returns null when meta matches current revs exactly', () => { - expect(checkStaleness({ remoteRev: 'h1', remoteDraftRev: 'd1' }, 'h1', 'd1')).toBeNull() - expect(checkStaleness({ remoteRev: 'h1' }, 'h1', undefined)).toBeNull() - }) - - it('returns "draft" when a newer DB draft was pushed on the remote', () => { - expect(checkStaleness({ remoteRev: 'h1', remoteDraftRev: 'd1' }, 'h1', 'd2')).toBe('draft') - }) - - it('returns "draft" when the remote gained a DB draft that we didn\'t baseline against', () => { - expect(checkStaleness({ remoteRev: 'h1' }, 'h1', 'd1')).toBe('draft') - }) - - it('returns "version" when the deployed rev moved and draft revs match', () => { - expect(checkStaleness({ remoteRev: 'h1' }, 'h2', undefined)).toBe('version') - }) - - it('returns "version" when the baseline draft was deleted on the remote (no current draft)', () => { - expect(checkStaleness({ remoteRev: 'h1', remoteDraftRev: 'd1' }, 'h1', undefined)).toBe( - 'version' - ) - }) - - it('prefers "draft" over "version" when both have changed', () => { - expect(checkStaleness({ remoteRev: 'h1', remoteDraftRev: 'd1' }, 'h2', 'd2')).toBe('draft') - }) -}) - -describe('UserDraft.use() — reference counting & cleanup', () => { - it('destroys the entry when the last handle is released', () => { - // First handle acquires the entry. - const a = UserDraft.use('flow', 'u/me/ref') - a.draft = 1 // baseline write — not persisted - - // Second handle increments the count. - const b = UserDraft.use('flow', 'u/me/ref') - expect(b.draft).toBe(1) - - // onDestroy for both handles got registered. - expect(onDestroyCallbacks.length).toBe(2) - - // Releasing one handle keeps the entry alive — save() still updates handle a. - const firstCb = onDestroyCallbacks.shift()! - firstCb() - - UserDraft.save('flow', 'u/me/ref', 2) - expect(a.draft).toBe(2) - // External save() calls persist immediately, even with a live handle. - expect(storedShape('userdraft/w/test_ws/flow/u/me/ref')).toBe(wrapped(2)) - - // Releasing the second handle drops the entry; subsequent save() - // must go straight to localStorage rather than mutating in-memory - // state (which no longer exists). - const secondCb = onDestroyCallbacks.shift()! - secondCb() - - UserDraft.save('flow', 'u/me/ref', 3) - // UserDraft.save without a live entry writes synchronously. - expect(storedShape('userdraft/w/test_ws/flow/u/me/ref')).toBe(wrapped(3)) - }) - - it('a fresh use() after cleanup re-reads the latest persisted value', () => { - const a = UserDraft.use('flow', 'u/me/cycle') - a.draft = 'initial' // baseline — not persisted - a.draft = 'edited' // persisted (after debounce) - flushPersist() - flushDestroyCallbacks() - - // After all handles release, a brand-new use() must pick up the - // value persisted to localStorage from the previous round. - const b = UserDraft.use('flow', 'u/me/cycle') - expect(b.draft).toBe('edited') - }) - - it('coalesces a typing storm into a single localStorage write per 500 ms window', () => { - const handle = UserDraft.use('flow', 'u/me/debounce') - handle.draft = 'baseline' // first write — skipped under saveInitialValue=false - - // Three quick edits inside the 500 ms window: in-memory updates every - // time, but localStorage stays untouched until the timer fires. - handle.draft = 'one' - vi.advanceTimersByTime(100) - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/debounce')).toBeNull() - handle.draft = 'two' - vi.advanceTimersByTime(100) - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/debounce')).toBeNull() - handle.draft = 'three' - expect(handle.draft).toBe('three') - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/debounce')).toBeNull() - - // After the window elapses, only the latest value lands. - vi.advanceTimersByTime(500) - expect(storedShape('userdraft/w/test_ws/flow/u/me/debounce')).toBe(wrapped('three')) - }) -}) - -describe('UserDraft.useMany()', () => { - it('acquires one handle per spec in the synchronous initial reconcile', () => { - // `useMany`'s sync reconcile populates handles[0..] before returning, - // so callers (and `use()`'s 1-len wrapper) can use them immediately - // without waiting for an `$effect` tick. - const handles = UserDraft.useMany(() => [ - { itemKind: 'flow', path: 'u/me/many', workspace: 'a' }, - { itemKind: 'flow', path: 'u/me/many', workspace: 'b' } - ]) - expect(handles.length).toBe(2) - - // Each spec gets its own entry in the workspace-keyed store. - handles[0].draft = 0 // baseline - handles[0].draft = 1 // persisted - handles[1].draft = 0 - handles[1].draft = 9 - flushPersist() - expect(storedShape('userdraft/w/a/flow/u/me/many')).toBe(wrapped(1)) - expect(storedShape('userdraft/w/b/flow/u/me/many')).toBe(wrapped(9)) - - // One component-level onDestroy releases every acquired entry. - expect(onDestroyCallbacks.length).toBe(1) - }) -}) - -describe('gcUserDrafts', () => { - let gcUserDrafts: (maxAgeMs?: number) => void - let USER_DRAFT_GC_MAX_AGE_MS: number - const DAY = 24 * 60 * 60 * 1000 - - beforeEach(async () => { - ;({ gcUserDrafts, USER_DRAFT_GC_MAX_AGE_MS } = await import('./userDraft.svelte')) - }) - - it('sweeps entries whose lastWrittenAt is older than the cutoff', () => { - vi.setSystemTime(new Date('2026-06-01T00:00:00Z')) - const old = Date.now() - 31 * DAY - const fresh = Date.now() - 1 * DAY - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/old', - JSON.stringify({ value: 1, lastWrittenAt: old }) - ) - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/fresh', - JSON.stringify({ value: 2, lastWrittenAt: fresh }) - ) - // Unrelated keys are left alone. - localStorage.setItem('some_other_key', 'unrelated') - - gcUserDrafts() - - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/old')).toBeNull() - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/fresh')).not.toBeNull() - expect(localStorage.getItem('some_other_key')).toBe('unrelated') - }) - - it('backfills lastWrittenAt on entries lacking it, instead of sweeping them immediately', () => { - // Pre-GC-feature entry (legacy migration output, or just an old entry - // from earlier in this PR's lifecycle): no `lastWrittenAt`. First GC - // pass should stamp it as "now" rather than wipe it on sight. - localStorage.setItem('userdraft/w/test_ws/flow/u/me/legacy', JSON.stringify({ value: 'data' })) - vi.setSystemTime(new Date('2026-06-01T00:00:00Z')) - - gcUserDrafts() - - const raw = localStorage.getItem('userdraft/w/test_ws/flow/u/me/legacy') - expect(raw).not.toBeNull() - const parsed = JSON.parse(raw!) - expect(parsed.lastWrittenAt).toBe(Date.now()) - expect(parsed.value).toBe('data') - }) - - it('exposes a 30-day default retention window', () => { - expect(USER_DRAFT_GC_MAX_AGE_MS).toBe(30 * 24 * 60 * 60 * 1000) - }) - - it('respects a custom maxAgeMs', () => { - vi.setSystemTime(new Date('2026-06-01T00:00:00Z')) - localStorage.setItem( - 'userdraft/w/test_ws/flow/u/me/two_hours_ago', - JSON.stringify({ value: 1, lastWrittenAt: Date.now() - 2 * 60 * 60 * 1000 }) - ) - - gcUserDrafts(60 * 60 * 1000) // 1h cutoff - - expect(localStorage.getItem('userdraft/w/test_ws/flow/u/me/two_hours_ago')).toBeNull() - }) -}) - -describe('normalizeForCompare', () => { - it('returns undefined for undefined input', () => { - expect(normalizeForCompare(undefined)).toBeUndefined() - }) - - it('drops keys whose value is undefined (mirrors JSON.stringify persistence)', () => { - const out = normalizeForCompare({ a: 1, b: undefined, c: { d: undefined, e: 2 } }) - expect(out).toEqual({ a: 1, c: { e: 2 } }) - expect(Object.keys(out as object)).not.toContain('b') - expect(Object.keys((out as any).c)).not.toContain('d') - }) - - it('falls back to the original value when not serializable (cyclic)', () => { - const cyclic: any = { a: 1 } - cyclic.self = cyclic - expect(normalizeForCompare(cyclic)).toBe(cyclic) - }) -}) - -describe('localDraftDiffers', () => { - it('returns false when there is no local draft', () => { - expect(localDraftDiffers(undefined, { a: 1 })).toBe(false) - expect(localDraftDiffers(null, { a: 1 })).toBe(false) - }) - - it('treats a draft that round-trips equal to the config as NOT differing', () => { - // The Schedule bug: getXCfg() emits conditionally-undefined keys, but - // the persisted draft went through JSON.stringify which dropped them. - const freshCfg = { path: 'u/me/s', schedule: '0 0 * * *', on_failure: undefined } - const persisted = JSON.parse(JSON.stringify(freshCfg)) // { path, schedule } - expect(localDraftDiffers(persisted, freshCfg)).toBe(false) - }) - - it('returns true for a genuine difference', () => { - expect(localDraftDiffers({ a: 1 }, { a: 2 })).toBe(true) - expect(localDraftDiffers({ a: 1, extra: 'x' }, { a: 1 })).toBe(true) - }) -}) - -describe('UserDraft.saveIfChanged', () => { - const KEY = 'userdraft/w/test_ws/trigger_schedule/u/me/s' - - it('does not persist a draft equal to the deployed baseline', () => { - const deployed = { path: 'u/me/s', schedule: '0 0 * * *', on_failure: undefined } - // value is the post-load reactive cfg — same shape, undefined keys present - UserDraft.saveIfChanged('trigger_schedule', 'u/me/s', { ...deployed }, deployed) - expect(localStorage.getItem(KEY)).toBeNull() - }) - - it('treats a value that round-trips equal to deployed as unchanged', () => { - const deployed = { path: 'u/me/s', schedule: '0 0 * * *', on_failure: undefined } - const value = JSON.parse(JSON.stringify(deployed)) // { path, schedule } - UserDraft.saveIfChanged('trigger_schedule', 'u/me/s', value, deployed) - expect(localStorage.getItem(KEY)).toBeNull() - }) - - it('persists when the value differs from the deployed baseline', () => { - const deployed = { path: 'u/me/s', schedule: '0 0 * * *' } - const value = { path: 'u/me/s', schedule: '5 0 * * *' } - UserDraft.saveIfChanged('trigger_schedule', 'u/me/s', value, deployed) - expect(storedShape(KEY)).toBe(wrapped(value)) - }) - - it('removes a pre-existing draft once the value reverts to deployed', () => { - const deployed = { path: 'u/me/s', schedule: '0 0 * * *' } - UserDraft.save('trigger_schedule', 'u/me/s', { path: 'u/me/s', schedule: '5 0 * * *' }) - expect(localStorage.getItem(KEY)).not.toBeNull() - UserDraft.saveIfChanged('trigger_schedule', 'u/me/s', { ...deployed }, deployed) - expect(localStorage.getItem(KEY)).toBeNull() - }) - - it('persists when there is no deployed baseline (undefined)', () => { - const value = { path: 'u/me/s', schedule: '0 0 * * *' } - UserDraft.saveIfChanged('trigger_schedule', 'u/me/s', value, undefined) - expect(storedShape(KEY)).toBe(wrapped(value)) - }) -}) - -describe('UserDraft.list / clear / setDraftAndMeta', () => { - it('enumerates persisted-only drafts for the requested workspace and kinds', () => { - UserDraft.setDraftAndMeta('script', 'f/a', { path: 'f/a', content: 'a' }, { remoteRev: 'h1' }) - UserDraft.setDraftAndMeta( - 'flow', - 'f/b', - { path: 'f/b', value: { modules: [] } }, - { remoteRev: 2 }, - { workspace: 'other_ws' } - ) - UserDraft.setDraftAndMeta('resource', 'f/c', { path: 'f/c' }, {}) - - expect(UserDraft.list({ itemKinds: ['script'] })).toEqual([ - { - workspace: 'test_ws', - itemKind: 'script', - path: 'f/a', - value: { path: 'f/a', content: 'a' }, - meta: { remoteRev: 'h1' }, - persisted: true, - live: false - } - ]) - expect(UserDraft.list({ workspace: 'other_ws' })).toEqual([ - expect.objectContaining({ - workspace: 'other_ws', - itemKind: 'flow', - path: 'f/b', - persisted: true, - live: false - }) - ]) - }) - - it('keeps multiple path-addressed drafts and the empty-path scratch draft distinct', () => { - UserDraft.setDraftAndMeta('script', '', { path: '', content: 'scratch' }, {}) - UserDraft.setDraftAndMeta('script', 'f/new-a', { path: 'f/new-a', content: 'a' }, {}) - UserDraft.setDraftAndMeta('script', 'f/new-b', { path: 'f/new-b', content: 'b' }, {}) - - const entries = UserDraft.list<{ path: string; content: string }>({ itemKinds: ['script'] }) - - expect(entries).toHaveLength(3) - expect(entries).toEqual( - expect.arrayContaining([ - expect.objectContaining({ - itemKind: 'script', - path: '', - value: { path: '', content: 'scratch' } - }), - expect.objectContaining({ - itemKind: 'script', - path: 'f/new-a', - value: { path: 'f/new-a', content: 'a' } - }), - expect.objectContaining({ - itemKind: 'script', - path: 'f/new-b', - value: { path: 'f/new-b', content: 'b' } - }) - ]) - ) - }) - - it('enumerates live-only drafts before the debounce persists them', () => { - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/live') - handle.setDraftAndMeta({ path: 'f/live', content: 'live' }, { remoteRev: 'h1' }) - - expect(localStorage.getItem('userdraft/w/test_ws/script/f/live')).toBeNull() - expect(UserDraft.list({ itemKinds: ['script'] })).toEqual([ - { - workspace: 'test_ws', - itemKind: 'script', - path: 'f/live', - value: { path: 'f/live', content: 'live' }, - meta: { remoteRev: 'h1' }, - persisted: false, - live: true - } - ]) - }) - - it('dedupes entries that are both persisted and live', () => { - UserDraft.setDraftAndMeta( - 'script', - 'f/both', - { path: 'f/both', content: 'persisted' }, - { - remoteRev: 'h1' - } - ) - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/both') - handle.draft = { path: 'f/both', content: 'live' } - - expect(UserDraft.list({ itemKinds: ['script'] })).toEqual([ - { - workspace: 'test_ws', - itemKind: 'script', - path: 'f/both', - value: { path: 'f/both', content: 'live' }, - meta: { remoteRev: 'h1' }, - persisted: true, - live: true - } - ]) - }) - - it('clear removes persisted storage and live state without re-persisting', () => { - UserDraft.setDraftAndMeta( - 'script', - 'f/clear', - { path: 'f/clear', content: 'x' }, - { - remoteRev: 'h1' - } - ) - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/clear') - expect(handle.draft).toEqual({ path: 'f/clear', content: 'x' }) - - UserDraft.clear('script', 'f/clear') - flushPersist() - - expect(handle.draft).toBeUndefined() - expect(localStorage.getItem('userdraft/w/test_ws/script/f/clear')).toBeNull() - expect(UserDraft.list({ itemKinds: ['script'] })).toEqual([]) - }) - - it('clear cancels pending debounced live writes', () => { - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/pending-clear') - handle.draft = { path: 'f/pending-clear', content: 'initial' } - handle.draft = { path: 'f/pending-clear', content: 'pending' } - - UserDraft.clear('script', 'f/pending-clear') - expect(handle.draft).toBeUndefined() - expect(localStorage.getItem('userdraft/w/test_ws/script/f/pending-clear')).toBeNull() - - flushPersist() - expect(localStorage.getItem('userdraft/w/test_ws/script/f/pending-clear')).toBeNull() - }) - - it('clear does not let an old debounced remove delete a later direct write', () => { - const key = 'userdraft/w/test_ws/script/f/rewrite-after-clear' - const handle = UserDraft.use<{ path: string; content: string }>( - 'script', - 'f/rewrite-after-clear' - ) - handle.draft = { path: 'f/rewrite-after-clear', content: 'initial' } - handle.draft = { path: 'f/rewrite-after-clear', content: 'pending' } - - UserDraft.clear('script', 'f/rewrite-after-clear') - flushDestroyCallbacks() - UserDraft.setDraftAndMeta( - 'script', - 'f/rewrite-after-clear', - { path: 'f/rewrite-after-clear', content: 'new' }, - {} - ) - - flushPersist() - expect(storedShape(key)).toBe(wrapped({ path: 'f/rewrite-after-clear', content: 'new' })) - }) - - it('list hides persisted drafts when a live handle has cleared the value', () => { - UserDraft.setDraftAndMeta( - 'script', - 'f/live-clear', - { path: 'f/live-clear', content: 'persisted' }, - {} - ) - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/live-clear') - handle.draft = { path: 'f/live-clear', content: 'edited' } - handle.draft = undefined - - expect(localStorage.getItem('userdraft/w/test_ws/script/f/live-clear')).not.toBeNull() - expect(UserDraft.list({ itemKinds: ['script'] })).toEqual([]) - }) - - it('setDraftAndMeta updates live handles atomically and preserves metadata on later draft writes', () => { - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/meta') - - UserDraft.setDraftAndMeta( - 'script', - 'f/meta', - { path: 'f/meta', content: 'first' }, - { - remoteRev: 'h1', - remoteDraftRev: 'd1' - } - ) - handle.draft = { path: 'f/meta', content: 'second' } - - expect(handle.draft).toEqual({ path: 'f/meta', content: 'second' }) - expect(handle.meta).toEqual({ remoteRev: 'h1', remoteDraftRev: 'd1' }) - expect(UserDraft.list({ itemKinds: ['script'] })[0]).toEqual( - expect.objectContaining({ - value: { path: 'f/meta', content: 'second' }, - meta: { remoteRev: 'h1', remoteDraftRev: 'd1' } - }) - ) - }) - - it('static setDraftAndMeta persists first writes even when a live handle exists', () => { - const handle = UserDraft.use<{ path: string; content: string }>('script', 'f/static-live') - - UserDraft.setDraftAndMeta( - 'script', - 'f/static-live', - { path: 'f/static-live', content: 'first' }, - { remoteRev: 'h1' } - ) - - expect(handle.draft).toEqual({ path: 'f/static-live', content: 'first' }) - expect(storedShape('userdraft/w/test_ws/script/f/static-live')).toBe( - JSON.stringify({ - value: { path: 'f/static-live', content: 'first' }, - remoteRev: 'h1' - }) - ) - }) - - it('lists live drafts with runtime-only values without throwing', () => { - const handle = UserDraft.use>('script', 'f/runtime') - handle.draft = { - path: 'f/runtime', - content: 'x', - callback: () => 'not serializable' - } - - expect(() => UserDraft.list({ itemKinds: ['script'] })).not.toThrow() - expect(UserDraft.list({ itemKinds: ['script'] })).toEqual([ - expect.objectContaining({ - itemKind: 'script', - path: 'f/runtime', - value: { path: 'f/runtime', content: 'x' } - }) - ]) - }) -}) diff --git a/frontend/src/lib/userDraftDbMigration.test.ts b/frontend/src/lib/userDraftDbMigration.test.ts new file mode 100644 index 0000000000..622401c2b5 --- /dev/null +++ b/frontend/src/lib/userDraftDbMigration.test.ts @@ -0,0 +1,185 @@ +import { describe, it, expect, beforeEach, vi } from 'vitest' + +// Service layer is mocked: the migration must dedup against the deployed value +// without making real network calls. +const updateDraft = vi.fn(async (..._args: any[]) => ({ status: 'created' as const })) +const getScriptByPath = vi.fn() +const getFlowByPath = vi.fn() +const getAppByPath = vi.fn() + +vi.mock('./gen', () => ({ + DraftService: { updateDraft: (...a: unknown[]) => updateDraft(...(a as [])) }, + ScriptService: { getScriptByPath: (...a: unknown[]) => getScriptByPath(...(a as [])) }, + FlowService: { getFlowByPath: (...a: unknown[]) => getFlowByPath(...(a as [])) }, + AppService: { getAppByPath: (...a: unknown[]) => getAppByPath(...(a as [])) } +})) + +// `migrateApp` mutates an App in place; the deployed fixtures below are already +// in migrated shape, so a no-op keeps the dedup comparison exact. +vi.mock('./components/apps/migrateApp', () => ({ migrateApp: vi.fn() })) +vi.mock('./toast', () => ({ sendUserToast: vi.fn() })) +vi.mock('./userNamespace', () => ({ getUsernameForNamespace: () => 'me' })) +vi.mock('./utils/uuid', () => ({ randomUUID: () => 'fixed-uuid' })) + +import { migrateUserDraftsToDb } from './userDraftDbMigration' + +function lsKey(kind: string, path: string): string { + return `userdraft/w/main/${kind}/${path}` +} + +function setDraft(kind: string, path: string, value: unknown): string { + const key = lsKey(kind, path) + localStorage.setItem(key, JSON.stringify({ value, lastWrittenAt: 123 })) + return key +} + +beforeEach(() => { + localStorage.clear() + vi.clearAllMocks() + updateDraft.mockResolvedValue({ status: 'created' }) +}) + +describe('migrateUserDraftsToDb dedup', () => { + it('drops a draft deep-equal to the deployed script without uploading it', async () => { + const deployed = { path: 'u/me/s', summary: 'hi', content: 'x', language: 'bun' } + getScriptByPath.mockResolvedValue(deployed) + const key = setDraft('script', 'u/me/s', { ...deployed }) + + await migrateUserDraftsToDb() + + expect(getScriptByPath).toHaveBeenCalledWith({ + workspace: 'main', + path: 'u/me/s', + getDraft: false + }) + expect(updateDraft).not.toHaveBeenCalled() + expect(localStorage.getItem(key)).toBeNull() + }) + + it('uploads a draft that differs from the deployed script', async () => { + getScriptByPath.mockResolvedValue({ + path: 'u/me/s', + summary: 'hi', + content: 'x', + language: 'bun' + }) + const key = setDraft('script', 'u/me/s', { + path: 'u/me/s', + summary: 'hi', + content: 'EDITED', + language: 'bun' + }) + + await migrateUserDraftsToDb() + + expect(updateDraft).toHaveBeenCalledTimes(1) + expect(localStorage.getItem(key)).toBeNull() + }) + + it('treats `{ field: undefined }` and an absent field as equal (json normalization)', async () => { + // The draft table stores JSON, which strips `undefined` keys — the + // comparison must too, or a draft that only differs by an undefined key + // would never dedup. + getFlowByPath.mockResolvedValue({ summary: 'f', value: { modules: [] } }) + const key = setDraft('flow', 'u/me/f', { + summary: 'f', + value: { modules: [] }, + labels: undefined + }) + + await migrateUserDraftsToDb() + + expect(updateDraft).not.toHaveBeenCalled() + expect(localStorage.getItem(key)).toBeNull() + }) + + it('ignores server-managed metadata fields on the deployed flow payload', async () => { + // The deployed flow carries read-time metadata (workspace_id, edited_by, + // version_id, is_draft, timestamps) that the editor's draft content never + // holds — they must not block the dedup. + getFlowByPath.mockResolvedValue({ + workspace_id: 'admins', + path: 'u/me/f', + summary: 'f', + value: { modules: [] }, + edited_by: 'admin@windmill.dev', + edited_at: '2026-01-01T00:00:00Z', + archived: false, + schema: {}, + extra_perms: {}, + version_id: 2, + is_draft: false, + draft_saved_at: '2026-01-01T00:00:01Z' + }) + const key = setDraft('flow', 'u/me/f', { + path: 'u/me/f', + summary: 'f', + value: { modules: [] }, + archived: false, + schema: {} + }) + + await migrateUserDraftsToDb() + + expect(updateDraft).not.toHaveBeenCalled() + expect(localStorage.getItem(key)).toBeNull() + }) + + it('compares app drafts against the deployed `.value`', async () => { + const appValue = { + grid: [], + fullscreen: false, + unusedInlineScripts: [], + hiddenInlineScripts: [] + } + getAppByPath.mockResolvedValue({ value: { ...appValue } }) + const key = setDraft('app', 'u/me/a', { ...appValue }) + + await migrateUserDraftsToDb() + + expect(getAppByPath).toHaveBeenCalledWith({ + workspace: 'main', + path: 'u/me/a', + getDraft: false + }) + expect(updateDraft).not.toHaveBeenCalled() + expect(localStorage.getItem(key)).toBeNull() + }) + + it('uploads when there is no deployed item (fetch rejects)', async () => { + getScriptByPath.mockRejectedValue(new Error('404')) + const key = setDraft('script', 'u/me/new', { path: 'u/me/new', content: 'x' }) + + await migrateUserDraftsToDb() + + expect(updateDraft).toHaveBeenCalledTimes(1) + expect(localStorage.getItem(key)).toBeNull() + }) + + it('skips the deployed fetch for a pathless /add draft and uploads at a minted path', async () => { + // A legacy `/add` autosave has an empty path; there is no deployed item to + // dedup against, so it uploads to a freshly minted `u/{user}/draft_{uuid}`. + setDraft('script', '', { path: '', content: 'x' }) + + await migrateUserDraftsToDb() + + expect(getScriptByPath).not.toHaveBeenCalled() + expect(updateDraft).toHaveBeenCalledTimes(1) + expect(updateDraft.mock.calls[0][0]).toMatchObject({ + kind: 'script', + // `mintDraftAddPath` dashes→underscores (path segments are word chars). + path: 'u/me/draft_fixed_uuid' + }) + }) + + it('does not dedup unsupported kinds (e.g. variable) — uploads as before', async () => { + const key = setDraft('variable', 'u/me/v', { value: 'secret' }) + + await migrateUserDraftsToDb() + + expect(getScriptByPath).not.toHaveBeenCalled() + expect(getAppByPath).not.toHaveBeenCalled() + expect(updateDraft).toHaveBeenCalledTimes(1) + expect(localStorage.getItem(key)).toBeNull() + }) +}) diff --git a/frontend/src/lib/userDraftDbMigration.ts b/frontend/src/lib/userDraftDbMigration.ts new file mode 100644 index 0000000000..4c3079ea94 --- /dev/null +++ b/frontend/src/lib/userDraftDbMigration.ts @@ -0,0 +1,289 @@ +/** + * One-off migration from the localStorage UserDraft autosave to the + * DB-backed `draft` table. Reads the workspace-scoped + * `userdraft/w/{workspace}/{kind}/{path}` keys (written by the editor during + * the interim LS-backed phase, so the embedded workspace is correct), POSTing + * each to `/drafts/update` and clearing the source key only on success — so + * it's idempotent without a sentinel; failed entries retry next mount. Not + * workspace-gated: keys embed their own workspace and the token covers all of + * them, so gating would orphan other-workspace entries. + * + * Before uploading, each draft is compared against its deployed version + * (script / flow / app); a draft that's deep-equal to what's deployed carries + * no changes, so it's dropped instead of migrated (no error). + */ + +import { AppService, DraftService, FlowService, ScriptService } from './gen' +import type { UserDraftItemKind } from './gen' +import type { App } from './components/apps/types' +import { migrateApp } from './components/apps/migrateApp' +import { sendUserToast } from './toast' +import { draftValuesEqual } from './userDraft.svelte' +import { + openDraftMigrationErrorModal, + reportDraftMigrationError +} from './userDraftMigrationErrors.svelte' +import { getUsernameForNamespace } from './userNamespace' +import { randomUUID } from './utils/uuid' + +// Mirror of `USER_DRAFT_ITEM_KINDS`, inlined to avoid importing the reactive +// runtime. The `_Exhaustive` assertion below fails compilation if this list +// drifts from the OpenAPI schema's kinds. +const ITEM_KINDS = [ + 'script', + 'flow', + 'app', + 'raw_app', + 'resource', + 'variable', + 'trigger_schedule', + 'trigger_webhook', + 'trigger_default_email', + 'trigger_email', + 'trigger_http', + 'trigger_websocket', + 'trigger_postgres', + 'trigger_kafka', + 'trigger_nats', + 'trigger_mqtt', + 'trigger_sqs', + 'trigger_gcp', + 'trigger_azure', + 'trigger_poll', + 'trigger_cli', + 'trigger_nextcloud', + 'trigger_google', + 'trigger_github', + 'data_pipeline' +] as const satisfies readonly UserDraftItemKind[] + +type _Exhaustive = + Exclude extends never ? true : never +const _: _Exhaustive = true +void _ + +const KEY_PREFIX = 'userdraft/w/' + +type ParsedKey = { + key: string + workspace: string + itemKind: UserDraftItemKind + path: string +} + +/** + * Split a `userdraft/w/{workspace}/{kind}/{path}` key into its parts, or + * `undefined` for keys that don't match or have an unrecognized kind. + * `path` is `''` only for a legacy `/add` autosave (key ended `.../{kind}/`); + * the caller mints a fresh slot for those. An empty `path` here is always that + * case, never garbage, since a non-matching key returns `undefined`. + */ +function parseKey(key: string): ParsedKey | undefined { + if (!key.startsWith(KEY_PREFIX)) return undefined + const rest = key.slice(KEY_PREFIX.length) + const firstSlash = rest.indexOf('/') + if (firstSlash <= 0) return undefined + const workspace = rest.slice(0, firstSlash) + const afterWorkspace = rest.slice(firstSlash + 1) + for (const kind of ITEM_KINDS) { + const kindPrefix = `${kind}/` + if (afterWorkspace.startsWith(kindPrefix)) { + const path = afterWorkspace.slice(kindPrefix.length) + return { key, workspace, itemKind: kind, path } + } + } + return undefined +} + +/** + * Mint a fresh `u/{user}/draft_{uuid}` slot for a pathless legacy `/add` + * autosave, matching the editors' `/add` redirect convention (`makeDraftAddLoad`). + * Underscores not dashes (path segments are `[a-zA-Z0-9_]` words); `randomUUID` + * not `crypto.randomUUID` (WebCrypto is absent on non-secure origins). + */ +function mintDraftAddPath(): string { + const username = getUsernameForNamespace() + const uuid = randomUUID().replaceAll('-', '_') + return `u/${username}/draft_${uuid}` +} + +/** + * Extract `value` and `lastWrittenAt` from the LS payload (`{ value, lastWrittenAt?, ... }`). + * `lastWrittenAt` rides along as `last_sync` so a fresher server draft wins. + * `undefined` when the slot is empty / unparseable / wrong shape. + */ +function readPayload(key: string): { value: unknown; lastWrittenAt?: number } | undefined { + try { + const raw = localStorage.getItem(key) + if (raw == null || raw === 'undefined') return undefined + const parsed = JSON.parse(raw) + if (parsed == null || typeof parsed !== 'object' || !('value' in parsed)) return undefined + const lastWrittenAt = (parsed as { lastWrittenAt?: unknown }).lastWrittenAt + return { + value: (parsed as { value: unknown }).value, + lastWrittenAt: typeof lastWrittenAt === 'number' ? lastWrittenAt : undefined + } + } catch { + return undefined + } +} + +/** + * Fetch the deployed value for a draft so the migration can drop a draft that + * carries no changes (deep-equal to what's already deployed) instead of + * uploading a no-op that would light up the "unsaved" badge. Returns the + * comparable deployed payload, or `undefined` when there's nothing to compare + * against: an unsupported kind, a pathless (minted `/add`) draft, or a fetch + * miss (404 — the path is draft-only, so the draft is genuinely new). `getDraft` + * is forced off so we compare against the deployed baseline, not our own draft. + */ +async function fetchDeployedValue( + workspace: string, + kind: UserDraftItemKind, + path: string +): Promise { + if (!path) return undefined + try { + switch (kind) { + case 'script': + return await ScriptService.getScriptByPath({ workspace, path, getDraft: false }) + case 'flow': + return await FlowService.getFlowByPath({ workspace, path, getDraft: false }) + case 'app': { + // The app autosave stores the inner `App`, not the `AppWithLastVersion` + // wrapper getAppByPath returns — compare against `.value`. Run + // `migrateApp` so the deployed value matches the editor-migrated draft + // (AppEditor `migrateApp`s `stateApp` on mount); without this an app + // whose deployed row predates those field migrations never dedups. + const app = await AppService.getAppByPath({ workspace, path, getDraft: false }) + const value = (app as { value?: App }).value + if (value) migrateApp(value) + return value + } + default: + return undefined + } + } catch { + // No deployed item at this path (or the fetch failed) — nothing to dedup + // against, so the caller proceeds to upload the draft. + return undefined + } +} + +function collectKeys(): string[] { + const keys: string[] = [] + for (let i = 0; i < localStorage.length; i++) { + const k = localStorage.key(i) + if (k != null && k.startsWith(KEY_PREFIX)) keys.push(k) + } + return keys +} + +/** + * Push every LS `userdraft/...` entry to `/drafts/update`, clearing each on + * success; failures stay in LS and retry next mount. + * + * `lastWrittenAt` rides as `last_sync` so the server rejects the upload when its + * own draft is fresher (the user may have edited the same path from another + * browser since this LS write). Missing `lastWrittenAt` uses epoch 0 (insert if + * absent, else yield). A `conflict` response means the server won; drop the LS + * copy. Never throws — the caller is fire-and-forget. + */ +export async function migrateUserDraftsToDb(): Promise { + if (typeof localStorage === 'undefined') return + const keys = collectKeys() + if (keys.length === 0) return + + // Parse up front so we only announce the migration when there's a real + // `userdraft/...` entry to upload (and can drop unparseable junk first). + const toMigrate: { + key: string + parsed: ParsedKey + path: string + value: unknown + lastWrittenAt?: number + }[] = [] + for (const key of keys) { + const parsed = parseKey(key) + if (!parsed) continue + const payload = readPayload(key) + if (payload === undefined) { + // Unparseable or empty — clear so we don't keep retrying it. + try { + localStorage.removeItem(key) + } catch { + // ignore + } + continue + } + // A legacy `/add` autosave has no path — mint a fresh slot so it lands + // as a regular draft-only item instead of being dropped. + const path = parsed.path === '' ? mintDraftAddPath() : parsed.path + toMigrate.push({ + key, + parsed, + path, + value: payload.value, + lastWrittenAt: payload.lastWrittenAt + }) + } + if (toMigrate.length === 0) return + + // Legacy drafts detected — tell the user the one-off upload is running, with + // an escape hatch to the modal where any failures show up as they happen. + sendUserToast('Migrating local storage drafts ...', 'info', [ + { label: 'See more', callback: openDraftMigrationErrorModal } + ]) + + for (const { key, parsed, path, value, lastWrittenAt } of toMigrate) { + try { + // Dedup: if the draft is deep-equal to the deployed version it carries + // no changes — drop it (no error) instead of uploading a no-op draft. + // Fetches against `parsed.path` (the real item path); minted `/add` + // drafts have `parsed.path === ''` and so are never deduped. + const deployed = await fetchDeployedValue(parsed.workspace, parsed.itemKind, parsed.path) + if (deployed !== undefined && draftValuesEqual(value, deployed)) { + try { + localStorage.removeItem(key) + } catch { + // Best-effort; a stale LS entry is harmless — it re-dedups next mount. + } + continue + } + // Preserve the draft's original age: stamp `created_at` with the LS + // write time (epoch 0 when unknown) so migrated drafts don't all + // resurface to the top as freshly created. Same value as `last_sync`, + // which still drives the conflict check. + const writtenAt = new Date(lastWrittenAt ?? 0).toISOString() + const res = await DraftService.updateDraft({ + workspace: parsed.workspace, + kind: parsed.itemKind, + path, + requestBody: { value, last_sync: writtenAt, created_at: writtenAt } + }) + if (res.status === 'conflict') { + console.info( + `UserDraft LS→DB migration: server draft for ${path} is fresher, dropping LS copy` + ) + } + try { + localStorage.removeItem(key) + } catch { + // Best-effort. If LS removal fails the next mount retries + // the save (the conflict rule keeps it idempotent). + } + } catch (e) { + // Leave the LS entry in place — the next mount tries again — but + // surface it so the user isn't silently stuck, with an escape + // hatch to drop the un-migratable draft. + console.error('UserDraft LS→DB migration: failed for', key, e) + reportDraftMigrationError({ + key, + workspace: parsed.workspace, + itemKind: parsed.itemKind, + path, + value + }) + } + } +} diff --git a/frontend/src/lib/userDraftDbSyncer.svelte.ts b/frontend/src/lib/userDraftDbSyncer.svelte.ts new file mode 100644 index 0000000000..a75f6b6837 --- /dev/null +++ b/frontend/src/lib/userDraftDbSyncer.svelte.ts @@ -0,0 +1,634 @@ +import { SvelteMap } from 'svelte/reactivity' +import { DraftService, type UserDraftItemKind } from './gen' +import { OpenAPI } from './gen/core/OpenAPI' +import { createCoalescingKeyedRunner } from './coalescingRunner.svelte' +import { createDebouncerByKey } from './debouncerByKey.svelte' +import { setLocalDraftHint } from './localDraftHints.svelte' + +/** + * Per-draft baseline timestamp attached as `last_sync` on the next save so + * the backend can reject stale writes (`created_at <= last_sync`). + * + * MUST be tab-local, not localStorage: each tab tracks its own baseline. + * Sharing it would let tab-1's save advance tab-2's `last_sync` to a fresh + * timestamp, so tab-2's next save would pass the server's WHERE check and + * clobber tab-1 — the exact conflict this feature prevents. The cost: a + * fresh tab has no `last_sync` (first save lands unconditionally), but the + * editor's `recordRemoteSync` reseeds from the server timestamp on load + * before any user edit can save. + */ +type DraftLastSyncEntry = { lastSync: string } +const lastSyncMap = new Map() + +/** Must match `mapKey` in `userDraft.svelte.ts`. */ +function draftKey(workspace: string, itemKind: UserDraftItemKind, path: string): string { + return `${workspace}/${itemKind}/${path}` +} + +/** + * Top-level script fields stripped before a draft is persisted. `hash` is the + * deployed version's identity (server-managed, re-supplied from the deployed row + * on load) and `assets` is re-derived from the script content by the editor — + * neither is draft content, so saving them bloats the row and resurfaces as + * fork/workspace diff noise. The editing object carries them because + * `getScriptByPath` returns the full DB row (since #9351). + * + * `CLEANED_VALUE_KEYS` (frontend/src/lib/utils.ts) strips a SUPERSET of these + * from the diff view (it also drops `inherited_labels` and other bookkeeping + * keys). The two lists are deliberately NOT the same: the diff hides every + * non-content key, while this sanitizer only trims the two that meaningfully + * bloat the persisted draft. Just keep the hash/assets entries here in step + * with that set; the rest may diverge. + */ +const SCRIPT_DRAFT_OMITTED_FIELDS = ['hash', 'assets'] as const + +/** + * Drop server-managed / re-derived fields a draft must not persist. Runs at the + * single persistence chokepoint (`save`) so every path — reactive autosave, + * Ctrl/Cmd+S flush, the `pagehide` keepalive — sends the same trimmed payload. + * Only scripts carry these fields; other kinds pass through untouched. + */ +export function sanitizeDraftValueForSave( + itemKind: UserDraftItemKind, + value: unknown | null +): unknown | null { + if ( + itemKind !== 'script' || + value === null || + typeof value !== 'object' || + Array.isArray(value) + ) { + return value + } + const obj = value as Record + if (!SCRIPT_DRAFT_OMITTED_FIELDS.some((f) => f in obj)) return value + const clone = { ...obj } + for (const f of SCRIPT_DRAFT_OMITTED_FIELDS) delete clone[f] + return clone +} + +function getLastSyncEntry(key: string): DraftLastSyncEntry | undefined { + return lastSyncMap.get(key) +} + +function setLastSync( + workspace: string, + itemKind: UserDraftItemKind, + path: string, + lastSync: string +): void { + lastSyncMap.set(draftKey(workspace, itemKind, path), { lastSync }) +} + +function clearLastSync(workspace: string, itemKind: UserDraftItemKind, path: string): void { + lastSyncMap.delete(draftKey(workspace, itemKind, path)) +} + +export type UserDraftDbSyncerSaveOpts = { + workspace: string + itemKind: UserDraftItemKind + path: string + /** `null` signals a delete — same conflict rules as an upsert. */ + value: unknown | null + /** Bypass the debouncer: cancel any pending autosave for this key (it + * would otherwise overwrite what we send), route through the coalescing + * runner to preserve ordering against an in-flight POST, and resolve + * the returned promise only once the POST lands. Use for + * `await save(...); read-the-server` flows where a fire-and-forget save + * would race the next read. */ + immediate?: boolean + /** Skip the optimistic-concurrency check and overwrite the server row. + * Used by the conflict-resolution UI ("Overwrite the remote"). Default + * `false`: autosaves attach `last_sync` and respect a reject. */ + force?: boolean + /** Save came from the reactive autosave mirror, not an explicit user + * action (Ctrl/Cmd+S flush, discard, fork, overwrite). */ + auto?: boolean + /** Source handle opts into the "Enable auto-save" toggle. Only the + * full-page editors (script / flow / app / raw app) set it; their + * `auto` saves are suppressed while the toggle is off (latest opts + * still park in `pendingSaveOpts` for an explicit flush). Drawer + * editors (variables / resources / triggers) leave it unset and always + * sync. Explicit saves always go through. */ + canBeDisabled?: boolean +} + +/** + * Snapshot of a rejected save. `localLastSync` is what we sent (or `null` + * if never synced); `serverTimestamp` is the row's current `created_at`, + * surfaced so the resolution UI can show how recent the conflict was. + */ +export type DraftConflictInfo = { + serverTimestamp: string + localLastSync: string | null +} + +export type UserDraftLastSyncQuery = { + workspace: string + itemKind: UserDraftItemKind + path: string +} + +/** + * Autosave lifecycle for a single draft: + * - `saving`: a POST is in flight (coalescing runner busy). + * - `pending`: a change is queued in the debouncer, not yet fired. + * - `failed`: the last POST threw (network / 5xx) and no later attempt + * succeeded. Conflicts go through the modal, not here. + * - `none`: in sync (or nothing happened). + * Render priority `saving` > `pending` > `failed` > `none`: an active + * retry outranks the prior failure so the indicator shows it in-flight. + */ +export type UserDraftSyncState = 'none' | 'pending' | 'saving' | 'failed' + +export type UserDraftStateHandle = { + /** Reactive: re-runs as the draft moves through the pipeline. */ + readonly state: UserDraftSyncState + /** When `state === 'failed'`, the message from the thrown error. + * `undefined` otherwise. */ + readonly failureMessage: string | undefined + /** Reactive: bumped each time `flush()` completes, INCLUDING the no-op + * path. Lets the indicator flash "Saved" on Ctrl/Cmd+S even when the + * autosave already landed (otherwise the shortcut is silent then). */ + readonly flushCount: number +} + +/** + * Two-stage pipeline per draft key. The debouncer collapses keystroke + * bursts (1500ms reset, 10000ms ceiling so sustained typing still flushes + * within 10s). When it fires, `opts` goes to the coalescing runner, which + * keeps at most one POST in flight per key plus one "latest" follow-up — + * newer submissions replace any prior pending, so the server never sees + * stale-then-fresh out of order. + * + * Imperative awaits (delete-then-refetch) MUST NOT rely on `save()`'s + * promise: it resolves when the work is queued, not when the POST lands. + * Use the `immediate` bypass for those. + */ +const debouncer = createDebouncerByKey({ debounceMs: 1500, maxDebounceMs: 10000 }) +const runner = createCoalescingKeyedRunner() + +/** + * "Enable auto-save" preference (AutosaveIndicator popover toggle). + * Browser-wide and persisted. While off, `auto: true` saves and the + * unload keepalive flush are suppressed — nothing leaves the tab except + * explicit saves (latest opts still park in `pendingSaveOpts` for flush). + */ +const AUTOSAVE_ENABLED_LS_KEY = 'userDraftAutosaveEnabled' +function readAutosaveEnabled(): boolean { + try { + return localStorage.getItem(AUTOSAVE_ENABLED_LS_KEY) !== 'false' + } catch { + return true + } +} +let autosaveEnabledState = $state(readAutosaveEnabled()) + +/** + * Latest unconfirmed `save` opts per draft key. The unload flush fires a + * `keepalive` POST for each entry. Cleared by `postSave` on success, but + * only when the entry is still the same object the success was for — a + * newer `save()` during the in-flight POST must survive for the next round. + */ +const pendingSaveOpts = new Map() + +/** + * Keys whose saves are HARD-blocked: while editing another user's loaded draft + * the foreign value must never reach the server through ANY path (reactive + * mirror, explicit flush, the pagehide keepalive flush). The value is the + * "blocked save attempted" callback — the first such attempt is the user's + * first edit, which the overlay UI turns into an "overwrite?" prompt. + */ +const syncLocked = new Map void) | undefined>() + +/** + * Conflict snapshots, populated when the server rejects a save (row + * `created_at` newer than our `last_sync`). Read via `getConflict(query)` + * to drive the resolution modal. + */ +const conflicts = new SvelteMap() + +/** + * Draft keys whose last save threw (network / 5xx) → extracted error + * message. Cleared on the next success. Drives the AutosaveIndicator's + * "Save failed" label so a silent failure can't masquerade as "Saved". + * Conflicts are tracked separately and don't populate this map. + */ +const failures = new SvelteMap() + +/** + * Reactive per-key counter bumped every time `flush()` completes — even + * when there was nothing to flush. See `UserDraftStateHandle.flushCount`. + */ +const flushes = new SvelteMap() + +/** + * Per-key listeners fired when a save for that key LANDS on the server + * (`status === 'saved'` with a non-null value — the draft now exists + * server-side). Distinct from `save()` resolving, which only means the work + * was queued. Used to defer the `?new_draft` URL strip until the first + * autosave is confirmed (see `stripNewDraftFlagOnSave`). + */ +const saveListeners = new Map void>>() + +/** + * Best-effort error → readable string. The generated client wraps HTTP + * failures as `ApiError` (`body` / `statusText`); raw fetch errors are a + * plain `Error`. Falls back to `String(e)` to avoid `[object Object]`. + */ +function formatSaveError(e: unknown): string { + if (e == null) return 'Unknown error' + if (typeof e === 'string') return e + const obj = e as Record + const body = obj.body + if (typeof body === 'string' && body) return body + if (body && typeof body === 'object') { + const inner = body.error?.message ?? body.message ?? body.error + if (typeof inner === 'string' && inner) return inner + } + if (typeof obj.message === 'string' && obj.message) return obj.message + if (typeof obj.statusText === 'string' && obj.statusText) return obj.statusText + return String(e) +} + +async function postSave(opts: UserDraftDbSyncerSaveOpts): Promise { + const key = draftKey(opts.workspace, opts.itemKind, opts.path) + const lastSync = getLastSyncEntry(key)?.lastSync + try { + const resp = await DraftService.updateDraft({ + workspace: opts.workspace, + kind: opts.itemKind, + path: opts.path, + requestBody: { + value: opts.value as any, + // Force-saves skip the conflict check; autosaves attach + // `last_sync` so the server can reject stale writes. Omitting + // `last_sync` (first-ever save) hits the backend's "treat as + // fresh" branch. + last_sync: opts.force ? undefined : lastSync, + force: opts.force ?? false + } + }) + if (resp.status === 'conflict') { + // Someone advanced the row past our `last_sync`. Park the + // snapshot for the UI; do NOT touch `lastSync` — the next save + // retries from the same baseline so the conflict persists until + // resolved. + conflicts.set(key, { + serverTimestamp: resp.current_timestamp, + localLastSync: lastSync ?? null + }) + return + } + // resp.status === 'saved' — advance lastSync (or drop on delete). + if (opts.value === null) { + clearLastSync(opts.workspace, opts.itemKind, opts.path) + } else { + setLastSync(opts.workspace, opts.itemKind, opts.path, resp.current_timestamp) + } + // postSave is the only place a draft's server-side existence changes, + // so it's the single source for the list pages' `*` hint + // (value !== null → exists). Every delete path clears the hint for + // free instead of maintaining a separate source of truth. + setLocalDraftHint(opts.workspace, opts.itemKind, opts.path, opts.value !== null) + conflicts.delete(key) + failures.delete(key) + // Clear pending only if it's still the opts we just saved — a + // newer `save()` that arrived during the POST replaces the entry + // and must survive for the next flush / debouncer round. + if (pendingSaveOpts.get(key) === opts) pendingSaveOpts.delete(key) + // Notify `onSaved` subscribers only for a real persisted draft, never a + // delete: stripping `?new_draft` after a `value: null` save would point a + // refresh at a row that no longer exists. + if (opts.value !== null) { + const listeners = saveListeners.get(key) + if (listeners) for (const l of [...listeners]) l() + } + } catch (e) { + console.error('UserDraftDbSyncer.save failed', e) + // Leave pending opts in place so the next attempt retries the same + // payload — we don't pretend the edit landed when it didn't. + failures.set(key, formatSaveError(e)) + } +} + +/** Keys whose `lastSync` was made stale by a `pagehide` keepalive flush + * (the POST advances `created_at` but its response is unreadable). + * Consumed by the `pageshow` handler: on a bfcache restore the SAME + * document comes back alive, and a save carrying the pre-flush + * `last_sync` would conflict against the user's own keepalive write. */ +const staleSyncAfterHideFlush = new Set() + +/** + * True-unload flush on `pagehide`. Uses `keepalive` so the request commits + * after the JS context is torn down (response discarded). Bypasses the + * debouncer/runner — both are async-scheduled and won't run post-hide. The + * keepalive body is capped (~64KB in Chrome); oversized payloads are + * rejected and lost (logged, still better than dropping every pending + * save). The POST advances `created_at` but we can't read it to update + * `lastSync` — fine here, the next mount reseeds via `recordRemoteSync`. + */ +function flushOnPageHide(): void { + if (pendingSaveOpts.size === 0) return + for (const [key, opts] of pendingSaveOpts) { + // Editing another user's loaded draft: never flush the foreign value. + if (syncLocked.has(key)) continue + // Auto-save off: page-editor opts are dropped with the page; + // drawer-kind pendings (no `canBeDisabled`) still flush. + if (!autosaveEnabledState && opts.auto && opts.canBeDisabled) continue + debouncer.cancel(key) + try { + // `encodeURI` (not `encodeURIComponent`), mirroring the + // generated client, so slashes in the path pass through. + const url = + OpenAPI.BASE + + `/w/${encodeURI(opts.workspace)}` + + `/drafts/update/${encodeURI(opts.itemKind)}` + + `/${encodeURI(opts.path)}` + const lastSync = getLastSyncEntry(key)?.lastSync + void fetch(url, { + method: 'POST', + credentials: 'include', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({ + value: opts.value, + last_sync: opts.force ? undefined : lastSync, + force: opts.force ?? false + }), + keepalive: true + }).catch((e) => { + console.error('UserDraftDbSyncer: keepalive flush failed', e) + }) + // POST advanced the row past `lastSync` and we can't read the + // response — mark the key so a bfcache restore drops it. + staleSyncAfterHideFlush.add(key) + } catch (e) { + console.error('UserDraftDbSyncer: keepalive flush threw', e) + } + } + pendingSaveOpts.clear() +} + +if (typeof document !== 'undefined') { + // Only `pagehide` means the document is truly going away. Do NOT use + // `visibilitychange → hidden`: it fires on every tab switch with the + // page surviving, where the debounced POST still fires normally — a + // flush there would spuriously conflict against the user's own write. + window.addEventListener('pagehide', flushOnPageHide) + // bfcache restore: drop the keepalive-stale entries so the next save + // omits `last_sync` (first-push). Safe — the server copy it overwrites + // is this document's own flush. Without this, that save is rejected as + // a conflict and the modal opens against the user's own write. + window.addEventListener('pageshow', (e: PageTransitionEvent) => { + if (!e.persisted) return + for (const key of staleSyncAfterHideFlush) { + lastSyncMap.delete(key) + } + staleSyncAfterHideFlush.clear() + }) +} + +/** + * Server-side persistence for `UserDraft`. Every write goes through the + * debouncer + coalescing runner above so autosave spam can't become one + * POST per keystroke or out-of-order writes under slow networks. + * + * Concurrency: every save attaches the per-tab `last_sync` (from + * `recordRemoteSync` or the prior success). The server rejects a stale + * `last_sync` and `postSave` surfaces a `DraftConflictInfo` for the modal. + */ +export const UserDraftDbSyncer = { + /** + * Reactive autosave-state handle for a draft. The key is captured at + * call time; if the consumer's `(workspace, itemKind, path)` can change, + * recompute the handle in a `$derived`. + */ + getState(query: UserDraftLastSyncQuery): UserDraftStateHandle { + const key = draftKey(query.workspace, query.itemKind, query.path) + return { + get state(): UserDraftSyncState { + if (runner.isRunning(key)) return 'saving' + if (debouncer.isPending(key)) return 'pending' + if (failures.has(key)) return 'failed' + return 'none' + }, + get failureMessage(): string | undefined { + // Suppress while saving/pending: an in-flight retry must not + // show the stale message from the failure it's retrying. + if (runner.isRunning(key) || debouncer.isPending(key)) return undefined + return failures.get(key) + }, + get flushCount(): number { + return flushes.get(key) ?? 0 + } + } + }, + + async save(opts: UserDraftDbSyncerSaveOpts): Promise { + // Trim non-draft fields once, here, so the parked opts the unload + // keepalive flush replays carry the same payload as the live POST. + opts = { ...opts, value: sanitizeDraftValueForSave(opts.itemKind, opts.value) } + const key = draftKey(opts.workspace, opts.itemKind, opts.path) + // Hard lock (editing another user's loaded draft): block EVERY save path + // for this key — no parking, no POST. Notify the overlay so the first + // blocked attempt (the user's first edit) can prompt before overwriting. + if (syncLocked.has(key)) { + syncLocked.get(key)?.() + return + } + // Park the latest opts BEFORE the pipeline so the unload flush has + // something to send even if the page hides before the debouncer fires. + pendingSaveOpts.set(key, opts) + if (opts.immediate) { + // Drop the queued autosave — firing it after our POST would + // re-save the pre-delete value. + debouncer.cancel(key) + runner.cancel(key) + await runner.submitAndWait(key, () => postSave(opts)) + return + } + // Auto-save off: opts stay parked (above) for an explicit flush but + // never schedule a POST. Only for handles that opted into the toggle. + if (opts.auto && opts.canBeDisabled && !autosaveEnabledState) return + // Optimistically light the list-page `*` so it tracks the editor's + // unsaved-changes banner without waiting for the debounced POST; + // postSave reconciles to the confirmed server state. + if (opts.value !== null) { + setLocalDraftHint(opts.workspace, opts.itemKind, opts.path, true) + } + debouncer.schedule(key, () => { + runner.submit(key, () => postSave(opts)) + }) + }, + + /** "Enable auto-save" preference — see the module-level doc. Turning it + * back ON re-schedules every parked draft so edits made while off catch + * up without waiting for the next keystroke. */ + get autosaveEnabled(): boolean { + return autosaveEnabledState + }, + set autosaveEnabled(enabled: boolean) { + autosaveEnabledState = enabled + try { + localStorage.setItem(AUTOSAVE_ENABLED_LS_KEY, String(enabled)) + } catch {} + if (enabled) { + for (const [key, opts] of pendingSaveOpts) { + debouncer.schedule(key, () => { + runner.submit(key, () => postSave(opts)) + }) + } + } + }, + + /** + * Seed the per-tab `last_sync` after an editor reads a draft from the + * server. Pass the response's `draft_saved_at` so the next save sends a + * matching `last_sync`; pass `undefined` when no draft existed (next + * save omits `last_sync`, the backend's first-push branch). + */ + recordRemoteSync(query: UserDraftLastSyncQuery, draftSavedAt: string | undefined): void { + const key = draftKey(query.workspace, query.itemKind, query.path) + if (draftSavedAt) { + setLastSync(query.workspace, query.itemKind, query.path, draftSavedAt) + } else { + clearLastSync(query.workspace, query.itemKind, query.path) + } + // Back in sync with the server: clear any conflict / failure. + conflicts.delete(key) + failures.delete(key) + }, + + /** + * Hard-block every save for this key (editing another user's loaded draft). + * Cancels any in-flight/queued autosave and drops parked opts so a pending + * flush can't fire the user's own value either. `onBlockedAttempt` fires on + * each subsequent blocked save — the overlay uses it to detect the first + * edit. MUST pair with `unlockSync`. + */ + lockSync(query: UserDraftLastSyncQuery, onBlockedAttempt?: () => void): void { + const key = draftKey(query.workspace, query.itemKind, query.path) + syncLocked.set(key, onBlockedAttempt) + debouncer.cancel(key) + runner.cancel(key) + pendingSaveOpts.delete(key) + }, + + /** Release a `lockSync`; subsequent saves go through normally. */ + unlockSync(query: UserDraftLastSyncQuery): void { + syncLocked.delete(draftKey(query.workspace, query.itemKind, query.path)) + }, + + /** + * Subscribe to CONFIRMED, non-delete saves for a draft key — fired after + * the POST lands on the server, not when `save()` queues it. Returns an + * unsubscribe. Backs `stripNewDraftFlagOnSave`. + */ + onSaved(query: UserDraftLastSyncQuery, listener: () => void): () => void { + const key = draftKey(query.workspace, query.itemKind, query.path) + let set = saveListeners.get(key) + if (!set) { + set = new Set() + saveListeners.set(key, set) + } + set.add(listener) + return () => { + const s = saveListeners.get(key) + if (!s) return + s.delete(listener) + if (s.size === 0) saveListeners.delete(key) + } + }, + + /** Reactive conflict snapshot (if any) for a draft. */ + getConflict(query: UserDraftLastSyncQuery): { + readonly conflict: DraftConflictInfo | undefined + } { + const key = draftKey(query.workspace, query.itemKind, query.path) + return { + get conflict() { + return conflicts.get(key) + } + } + }, + + /** + * Clear the conflict snapshot. `recordRemoteSync` does this implicitly + * on a fresh read, so the only standalone use is "dismiss without + * resolving". + */ + clearConflict(query: UserDraftLastSyncQuery): void { + conflicts.delete(draftKey(query.workspace, query.itemKind, query.path)) + }, + + /** + * Force-save: bypass the `last_sync` check and overwrite the server row + * (conflict modal's "Overwrite the remote"). Resolves only after the + * POST lands so the caller can `await` before navigating / refetching. + */ + async overwrite(opts: Omit): Promise { + await this.save({ ...opts, immediate: true, force: true }) + }, + + /** + * Flush the draft's queued autosave NOW (explicit Ctrl/Cmd+S). Re-submits + * the parked opts with `immediate: true` and resolves only after the POST + * lands, so callers can `await flush(...); show "Saved"`. + * + * No-op when nothing is pending. "No pending" does NOT mean "nothing to + * save" — Monaco may hold unmaterialized text; flush the editor + * (`Editor.flushPendingChanges()`) and await `tick()` first so its + * bind:code reaches our save() before this. + * + * `honorAutosaveToggle` makes the flush respect the "Enable auto-save" + * preference: a toggle-aware autosave (`auto` + `canBeDisabled`) stays + * parked while auto-save is off, so the edit is NOT persisted. The editor's + * unmount uses it (leaving with auto-save off must not silently save — + * the UnsavedConfirmationModal warns instead); explicit Ctrl/Cmd+S omits it + * and always saves. + */ + async flush( + query: UserDraftLastSyncQuery, + opts?: { honorAutosaveToggle?: boolean } + ): Promise { + const key = draftKey(query.workspace, query.itemKind, query.path) + try { + const parked = pendingSaveOpts.get(key) + if (!parked) return + if (opts?.honorAutosaveToggle && !autosaveEnabledState && parked.auto && parked.canBeDisabled) + return + await this.save({ ...parked, immediate: true }) + } finally { + // Signal the indicator even on the no-op path so Ctrl/Cmd+S + // shows "Saved" even when the autosave already landed. + flushes.set(key, (flushes.get(key) ?? 0) + 1) + } + }, + + /** + * Whether this draft has content edits parked but unsaved because auto-save + * is off — the signal the full-page editors' UnsavedConfirmationModal uses + * to warn before leaving. True only when auto-save is disabled AND a + * toggle-aware (`auto` + `canBeDisabled`) write carrying content is parked; + * a parked delete (`value: null` from deploy / discard / reset) is not + * unsaved content. Read imperatively (e.g. in `beforeNavigate`), not + * reactively. + */ + hasUnsavedDisabledChanges(query: UserDraftLastSyncQuery): boolean { + if (autosaveEnabledState) return false + const parked = pendingSaveOpts.get(draftKey(query.workspace, query.itemKind, query.path)) + return !!parked && parked.auto === true && parked.canBeDisabled === true && parked.value != null + }, + + /** + * Drop a draft's parked-but-unsaved autosave WITHOUT POSTing — the user + * chose to discard the auto-save-off edits on leave. Also cancels any + * queued debounce so turning auto-save back on can't resurrect them + * (the `autosaveEnabled` setter re-schedules every parked entry). + */ + dropPending(query: UserDraftLastSyncQuery): void { + const key = draftKey(query.workspace, query.itemKind, query.path) + pendingSaveOpts.delete(key) + debouncer.cancel(key) + } +} diff --git a/frontend/src/lib/userDraftLegacyMigration.test.ts b/frontend/src/lib/userDraftLegacyMigration.test.ts index 7069929897..f7963e55a0 100644 --- a/frontend/src/lib/userDraftLegacyMigration.test.ts +++ b/frontend/src/lib/userDraftLegacyMigration.test.ts @@ -1,6 +1,6 @@ import { describe, it, expect, beforeEach } from 'vitest' import { - migrateLegacyUserDrafts, + purgeLegacyUserDrafts, __resetUserDraftLegacyMigrationForTesting } from './userDraftLegacyMigration' @@ -8,18 +8,12 @@ function encodeLegacy(value: unknown): string { return btoa(encodeURIComponent(JSON.stringify(value))) } -function wrapped(value: V): string { - return JSON.stringify({ value }) -} - -// Read a migrated entry, strip the GC `lastWrittenAt` stamp so assertions -// can match the `{ value }` shape regardless of when the migration ran. -function storedShape(key: string): string | null { - const raw = localStorage.getItem(key) - if (raw == null) return null - const parsed = JSON.parse(raw) - delete parsed.lastWrittenAt - return JSON.stringify(parsed) +const legacyApp = { + grid: [], + fullscreen: false, + theme: undefined, + unusedInlineScripts: [], + hiddenInlineScripts: [] } beforeEach(() => { @@ -27,197 +21,110 @@ beforeEach(() => { __resetUserDraftLegacyMigrationForTesting() }) -describe('migrateLegacyUserDrafts', () => { - it('migrates a legacy app draft to the workspace-scoped key with a { value } wrapper', () => { - // Shape mirrors what the legacy AppEditor wrote: `encodeState($appStore)`, - // i.e. the inner App value, not the wrapping AppWithLastVersion. - const legacyApp = { - grid: [], - fullscreen: false, - theme: undefined, - unusedInlineScripts: [], - hiddenInlineScripts: [] - } +describe('purgeLegacyUserDrafts', () => { + it('drops a recognised legacy app draft without re-creating it under any key', () => { localStorage.setItem('app-u/me/dashboard', encodeLegacy(legacyApp)) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app-u/me/dashboard')).toBeNull() - expect(storedShape('userdraft/w/main/app/u/me/dashboard')).toBe(wrapped(legacyApp)) + // The workspace-blind key is gone, NOT promoted to a guessed workspace. + expect(localStorage.getItem('userdraft/w/main/app/u/me/dashboard')).toBeNull() }) - it('migrates a legacy empty-path app draft (the `app` literal key)', () => { - const legacyApp = { - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - } + it('drops the empty-path legacy keys (`app` / `flow` / `rawapp` literals)', () => { localStorage.setItem('app', encodeLegacy(legacyApp)) + localStorage.setItem('flow', encodeLegacy({ flow: { summary: '', value: { modules: [] } } })) + localStorage.setItem('rawapp', encodeLegacy({ files: {}, runnables: {}, data: {} })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app')).toBeNull() - expect(storedShape('userdraft/w/main/app/')).toBe(wrapped(legacyApp)) + expect(localStorage.getItem('flow')).toBeNull() + expect(localStorage.getItem('rawapp')).toBeNull() }) - it('migrates a legacy flow draft and strips the view-state envelope', () => { - const flow = { summary: 'f', value: { modules: [] }, path: 'u/me/myflow' } - const legacyBundle = { - flow, - path: 'u/me/myflow', - selectedId: 'settings', - draft_triggers: [{ id: 't1' }], - selected_trigger: null, - loadedFromHistory: undefined - } - localStorage.setItem('flow-u/me/myflow', encodeLegacy(legacyBundle)) + it('drops recognised legacy flow and raw-app drafts', () => { + localStorage.setItem( + 'flow-u/me/myflow', + encodeLegacy({ flow: { summary: 'f', value: { modules: [] } }, selectedId: 'settings' }) + ) + localStorage.setItem( + 'rawapp-u/me/site', + encodeLegacy({ files: { 'index.tsx': 'x' }, runnables: {}, data: {} }) + ) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('flow-u/me/myflow')).toBeNull() - // Only the inner Flow survives; the view-state envelope is dropped. - expect(storedShape('userdraft/w/main/flow/u/me/myflow')).toBe(wrapped(flow)) - }) - - it('migrates a legacy raw-app draft, defaulting the new `summary` field', () => { - const legacy = { - files: { 'index.tsx': 'export default () => null' }, - runnables: {}, - data: { tables: [] } - } - localStorage.setItem('rawapp-u/me/site', encodeLegacy(legacy)) - - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('rawapp-u/me/site')).toBeNull() - expect(storedShape('userdraft/w/main/raw_app/u/me/site')).toBe( - wrapped({ ...legacy, summary: '' }) - ) }) - it('preserves an existing new-format entry instead of overwriting it', () => { - // Old and new both exist for the same item — the new one is presumed - // fresher. - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - const existingNew = wrapped({ value: 'new' }) - localStorage.setItem('userdraft/w/main/app/u/me/dash', existingNew) + it('leaves the workspace-scoped interim keys untouched (migrateUserDraftsToDb owns those)', () => { + const interim = JSON.stringify({ value: { modules: [] } }) + localStorage.setItem('userdraft/w/main/flow/u/me/keep', interim) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('app-u/me/dash')).toBeNull() - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBe(existingNew) - }) - - it('is idempotent — the second invocation is a no-op', () => { - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).not.toBeNull() - - // Drop the migrated entry to detect any re-migration attempt. - localStorage.removeItem('userdraft/w/main/app/u/me/dash') - // Drop the source too, so re-running couldn't even find a source. - // (The sentinel alone should be enough; this just clarifies the intent.) - migrateLegacyUserDrafts('main') - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBeNull() - }) - - it('skips entirely when no workspace is available', () => { - localStorage.setItem( - 'app-u/me/dash', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) - migrateLegacyUserDrafts('') - - expect(localStorage.getItem('app-u/me/dash')).not.toBeNull() - }) - - it('handles malformed legacy payloads without throwing', () => { - localStorage.setItem('app-u/me/garbled', 'not-base64!!!') - expect(() => migrateLegacyUserDrafts('main')).not.toThrow() - // Migration didn't migrate, didn't crash — leaves the entry alone. - expect(localStorage.getItem('app-u/me/garbled')).toBe('not-base64!!!') + expect(localStorage.getItem('userdraft/w/main/flow/u/me/keep')).toBe(interim) }) it('leaves keys whose path does not match the legacy `u|f/owner/name` shape alone', () => { - // A future feature or neighbouring code might pick a key like - // `app-recent` for its own purposes. The path doesn't look like a - // Windmill item path, so the migration must skip it. + // `app-recent` / `app-some_other_app` look like the legacy prefix but the + // suffix isn't a Windmill item path — a future feature might own them. localStorage.setItem('app-recent', 'whatever') localStorage.setItem('app-some_other_app', 'whatever') - // `flow-u/me/foo` matches the shape and would be migrated, but the - // payload also needs to look like a Windmill draft (asserted below). - localStorage.setItem('flow-u/me/foo', encodeLegacy({ flow: { value: { modules: [] } } })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() expect(localStorage.getItem('app-recent')).toBe('whatever') expect(localStorage.getItem('app-some_other_app')).toBe('whatever') - expect(localStorage.getItem('userdraft/w/main/flow/u/me/foo')).not.toBeNull() }) - it('skips legacy-shaped keys whose payload does not look like a Windmill draft', () => { - // `app-u/me/dash` matches LEGACY_PATH_SHAPE and decodes to valid JSON, - // but none of the App-shape fields (grid/fullscreen/theme/ - // unusedInlineScripts/hiddenInlineScripts) are present. Treat it as - // unrelated and leave it untouched. - const unrelated = encodeLegacy({ random: 'data', count: 7 }) - localStorage.setItem('app-u/me/dash', unrelated) + it('leaves legacy-shaped keys whose payload does not look like a Windmill draft', () => { + // Matches LEGACY_PATH_SHAPE and decodes to valid JSON, but carries none of + // the App/flow draft fields — treat as unrelated, do not delete. + const unrelatedApp = encodeLegacy({ random: 'data', count: 7 }) + localStorage.setItem('app-u/me/dash', unrelatedApp) const unrelatedFlow = encodeLegacy({ stepsState: {} }) localStorage.setItem('flow-u/me/bar', unrelatedFlow) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('app-u/me/dash')).toBe(unrelated) - expect(localStorage.getItem('userdraft/w/main/app/u/me/dash')).toBeNull() + expect(localStorage.getItem('app-u/me/dash')).toBe(unrelatedApp) expect(localStorage.getItem('flow-u/me/bar')).toBe(unrelatedFlow) - expect(localStorage.getItem('userdraft/w/main/flow/u/me/bar')).toBeNull() }) - it('migrates multiple legacy entries in a single invocation', () => { - localStorage.setItem( - 'app-u/me/a', - encodeLegacy({ - grid: [], - fullscreen: false, - unusedInlineScripts: [], - hiddenInlineScripts: [] - }) - ) + it('leaves a malformed (non-base64) legacy payload alone and does not throw', () => { + localStorage.setItem('app-u/me/garbled', 'not-base64!!!') + + expect(() => purgeLegacyUserDrafts()).not.toThrow() + expect(localStorage.getItem('app-u/me/garbled')).toBe('not-base64!!!') + }) + + it('is idempotent — once the sentinel is set, a later legacy key survives', () => { + localStorage.setItem('app-u/me/a', encodeLegacy(legacyApp)) + purgeLegacyUserDrafts() + expect(localStorage.getItem('app-u/me/a')).toBeNull() + + // A key written after the first run is NOT swept (the sentinel short-circuits). + localStorage.setItem('app-u/me/b', encodeLegacy(legacyApp)) + purgeLegacyUserDrafts() + expect(localStorage.getItem('app-u/me/b')).not.toBeNull() + }) + + it('purges multiple legacy entries in a single invocation', () => { + localStorage.setItem('app-u/me/a', encodeLegacy(legacyApp)) localStorage.setItem( 'flow-u/me/b', - encodeLegacy({ flow: { summary: '', value: { modules: [] }, path: 'u/me/b' } }) - ) - localStorage.setItem( - 'rawapp-u/me/c', - encodeLegacy({ files: {}, runnables: {}, data: { tables: [] } }) + encodeLegacy({ flow: { summary: '', value: { modules: [] } } }) ) + localStorage.setItem('rawapp-u/me/c', encodeLegacy({ files: {}, runnables: {}, data: {} })) - migrateLegacyUserDrafts('main') + purgeLegacyUserDrafts() - expect(localStorage.getItem('userdraft/w/main/app/u/me/a')).not.toBeNull() - expect(localStorage.getItem('userdraft/w/main/flow/u/me/b')).not.toBeNull() - expect(localStorage.getItem('userdraft/w/main/raw_app/u/me/c')).not.toBeNull() + expect(localStorage.getItem('app-u/me/a')).toBeNull() + expect(localStorage.getItem('flow-u/me/b')).toBeNull() + expect(localStorage.getItem('rawapp-u/me/c')).toBeNull() }) }) diff --git a/frontend/src/lib/userDraftLegacyMigration.ts b/frontend/src/lib/userDraftLegacyMigration.ts index d4ee04ee61..4239bc0e8a 100644 --- a/frontend/src/lib/userDraftLegacyMigration.ts +++ b/frontend/src/lib/userDraftLegacyMigration.ts @@ -1,22 +1,29 @@ /** - * One-off migration from the pre-UserDraft localStorage autosave entries to - * the workspace-scoped `userdraft/w/{ws}/{kind}/{path}` format. + * One-shot purge of the pre-UserDraft browser-local autosave keys. * - * Legacy keys (global, not workspace-scoped — assumed to belong to the user's - * current workspace at migration time): + * The original autosave (pre-#9121) wrote workspace-BLIND keys: * - * `flow` / `flow-{path}` base64 of `encodeState({ flow, path, selectedId, draft_triggers, ... })` - * `app` / `app-{path}` base64 of `encodeState(App)` - * `rawapp` / `rawapp-{path}` base64 of `encodeState({ files, runnables, data })` + * `flow` / `flow-{path}` base64 of `encodeState({ flow, path, selectedId, draft_triggers, ... })` + * `app` / `app-{path}` base64 of `encodeState(App)` + * `rawapp` / `rawapp-{path}` base64 of `encodeState({ files, runnables, data })` * - * Target keys: `userdraft/w/{workspace}/{flow|app|raw_app}/{path}` storing - * `JSON.stringify({ value: })`. + * Neither the key nor the decoded value records a workspace (the value carries + * only workspace-agnostic item paths like `u/me/x`), so these drafts cannot be + * attributed to the workspace they were edited in. The current editors are + * DB-backed and never read these keys, so they are dead data with one dangerous + * property: promoting them to the DB would force a GUESS of the workspace, which + * mis-files drafts into whatever workspace happened to be active when the + * migration first ran (a single global sentinel gates it). We therefore drop + * them instead of migrating them. + * + * Only keys that BOTH match the legacy path shape AND decode to a plausible + * legacy draft are removed; unrelated look-alikes (`app-recent`, garbage, + * non-Windmill payloads) are left untouched. The workspace-scoped interim keys + * (`userdraft/w/{ws}/...`, written by the editor with the correct workspace) + * are NOT touched here — `migrateUserDraftsToDb` still pushes those to the DB. * * Idempotent: writes a sentinel under `MIGRATION_FLAG` after the first run so - * subsequent invocations are no-ops. Existing new-format entries are never - * overwritten — when both an old and a new entry exist for the same item, the - * old one is simply dropped on the assumption that the new entry is the more - * recent edit. + * subsequent invocations are no-ops. * * This file is intentionally standalone — it does not import from * `userDraft.svelte.ts` so the new code stays uncluttered by the legacy @@ -71,10 +78,9 @@ function decodeLegacyState(raw: string): unknown { * Per-kind shape gate. The legacy keys (`app-foo`, `flow-foo`, ...) are * unusual enough that nothing else in the codebase has used them, but * matching `LEGACY_PATH_SHAPE` doesn't prove the payload is actually a - * Windmill draft (any base64-of-JSON could pass). Promoting a stray payload - * would silently surface as a phantom "Restored from local storage" toast - * on the next edit, so we reject anything that doesn't carry the fields the - * legacy writers actually produced. + * Windmill draft (any base64-of-JSON could pass). We only delete keys we can + * positively recognise as legacy drafts, so a stray look-alike that happens to + * use this key shape is left untouched rather than silently dropped. */ function isPlausibleLegacyValue(kind: LegacyKind, decoded: unknown): boolean { if (decoded == null || typeof decoded !== 'object') return false @@ -103,32 +109,6 @@ function isPlausibleLegacyValue(kind: LegacyKind, decoded: unknown): boolean { } } -function transformLegacyValue(kind: LegacyKind, decoded: unknown): unknown { - const obj = decoded as Record - switch (kind) { - case 'flow': - // The legacy bundle wrapped the Flow alongside view-state fields - // (selectedId, draft_triggers, ...). The new entry stores only the - // Flow — the view-state lives elsewhere or is re-derived. - return obj.flow - case 'app': - // Legacy stored the App directly. - return obj - case 'raw_app': - // Legacy bundle missed the `summary` field that the new editor adds. - return { - files: obj.files ?? {}, - runnables: obj.runnables ?? {}, - data: obj.data ?? {}, - summary: typeof obj.summary === 'string' ? obj.summary : '' - } - } -} - -function newKey(workspace: string, kind: LegacyKind, path: string): string { - return `userdraft/w/${workspace}/${kind}/${path}` -} - function listLocalStorageKeys(): string[] { const out: string[] = [] for (let i = 0; i < localStorage.length; i++) { @@ -139,16 +119,11 @@ function listLocalStorageKeys(): string[] { } /** - * Run the legacy → new-format migration. Idempotent: returns immediately if a - * previous run completed (signalled by `MIGRATION_FLAG`). - * - * The migration is workspace-scoped because the legacy keys had no notion of - * workspace — we treat the caller's current workspace as the owner of any - * surviving legacy entries. + * Remove the workspace-blind legacy autosave keys (see file header). Idempotent: + * returns immediately if a previous run completed (signalled by `MIGRATION_FLAG`). */ -export function migrateLegacyUserDrafts(workspace: string): void { +export function purgeLegacyUserDrafts(): void { if (typeof localStorage === 'undefined') return - if (!workspace) return if (localStorage.getItem(MIGRATION_FLAG) !== null) return try { @@ -157,32 +132,18 @@ export function migrateLegacyUserDrafts(workspace: string): void { if (!match) continue const raw = localStorage.getItem(key) if (raw == null) continue - - try { - const decoded = decodeLegacyState(raw) - if (!isPlausibleLegacyValue(match.newKind, decoded)) continue - const value = transformLegacyValue(match.newKind, decoded) - const target = newKey(workspace, match.newKind, match.path) - if (value !== undefined && localStorage.getItem(target) == null) { - // `lastWrittenAt` makes the migrated entry visible to - // `gcUserDrafts`. We stamp it as "now" so a freshly-migrated - // autosave gets the full retention window — sweeping it - // immediately on the first GC pass would lose work the - // legacy migration just rescued. - localStorage.setItem(target, JSON.stringify({ value, lastWrittenAt: Date.now() })) - } - localStorage.removeItem(key) - } catch (e) { - console.error('UserDraft legacy migration: failed to migrate', key, e) - } + // Only drop keys we can positively recognise as legacy Windmill + // drafts; leave unrelated or unparseable look-alikes in place. + if (!isPlausibleLegacyValue(match.newKind, decodeLegacyState(raw))) continue + localStorage.removeItem(key) } localStorage.setItem(MIGRATION_FLAG, new Date().toISOString()) } catch (e) { - console.error('UserDraft legacy migration: aborted', e) + console.error('UserDraft legacy purge: aborted', e) } } -/** Test-only: clear the sentinel so the migration can re-run. */ +/** Test-only: clear the sentinel so the purge can re-run. */ export function __resetUserDraftLegacyMigrationForTesting(): void { try { localStorage.removeItem(MIGRATION_FLAG) diff --git a/frontend/src/lib/userDraftMigrationErrors.svelte.ts b/frontend/src/lib/userDraftMigrationErrors.svelte.ts new file mode 100644 index 0000000000..4059c4f990 --- /dev/null +++ b/frontend/src/lib/userDraftMigrationErrors.svelte.ts @@ -0,0 +1,75 @@ +/** + * Reactive registry of drafts that `migrateUserDraftsToDb` could not push to + * the server. The migration runs on every layout mount, so a persistently + * un-migratable draft would re-fail (and re-report) each time — keying by the + * LS key dedupes those repeats. A SINGLE toast fires on the empty→non-empty + * transition (never per-failure, never when there's nothing wrong); its action + * opens `DraftMigrationErrorModal`, which reads `list` live so failures that + * surface while the modal is already open just appear in place. + */ +import { SvelteMap } from 'svelte/reactivity' +import type { UserDraftItemKind } from '$lib/gen' +import { sendUserToast } from './toast' + +export type DraftMigrationError = { + /** The source `userdraft/...` localStorage key — identity and delete target. */ + key: string + workspace: string + itemKind: UserDraftItemKind + path: string + /** The draft payload, surfaced verbatim by the modal's "View JSON". */ + value: unknown +} + +const errors = new SvelteMap() +let modalOpen = $state(false) + +export const draftMigrationErrors = { + get list(): DraftMigrationError[] { + return [...errors.values()] + }, + get modalOpen(): boolean { + return modalOpen + }, + set modalOpen(open: boolean) { + modalOpen = open + } +} + +/** Open the modal listing the failed migrations. */ +export function openDraftMigrationErrorModal(): void { + modalOpen = true +} + +/** + * Record a failed draft migration. Idempotent per `key`; the toast only fires + * on the first failure of a batch (empty→non-empty) and is suppressed when the + * modal is already open, since the user is already resolving issues there. + */ +export function reportDraftMigrationError(error: DraftMigrationError): void { + if (errors.has(error.key)) return + const wasEmpty = errors.size === 0 + errors.set(error.key, error) + if (wasEmpty && !modalOpen) { + sendUserToast('Some local storage drafts could not be migrated', 'error', [ + { label: 'Resolve issues', callback: openDraftMigrationErrorModal } + ]) + } +} + +/** Drop the un-migratable draft from localStorage and clear its error entry. */ +export function deleteDraftMigrationError(key: string): void { + try { + localStorage.removeItem(key) + } catch { + // Best-effort; the entry leaves the list regardless. + } + errors.delete(key) +} + +/** Drop every un-migratable draft at once. */ +export function deleteAllDraftMigrationErrors(): void { + for (const key of [...errors.keys()]) { + deleteDraftMigrationError(key) + } +} diff --git a/frontend/src/lib/userDraftSanitize.test.ts b/frontend/src/lib/userDraftSanitize.test.ts new file mode 100644 index 0000000000..1d722a4948 --- /dev/null +++ b/frontend/src/lib/userDraftSanitize.test.ts @@ -0,0 +1,54 @@ +import { describe, it, expect, vi } from 'vitest' + +// Importing the syncer pulls in the generated client and its localStorage / +// pagehide wiring; stub the same surfaces the other syncer test does so the +// pure-function import stays side-effect free. +vi.mock('./gen', () => ({ + DraftService: { updateDraft: vi.fn() } +})) +vi.mock('./gen/core/OpenAPI', () => ({ OpenAPI: { BASE: '' } })) +vi.mock('./localDraftHints.svelte', () => ({ setLocalDraftHint: vi.fn() })) + +import { sanitizeDraftValueForSave } from './userDraftDbSyncer.svelte' + +describe('sanitizeDraftValueForSave', () => { + it('strips hash and assets from a script draft', () => { + const value = { + path: 'u/me/foo', + summary: 'hi', + content: 'export function main() {}', + hash: '123456789', + assets: [{ path: 's3://bucket/x', kind: 's3object' }] + } + const cleaned = sanitizeDraftValueForSave('script', value) as any + expect(cleaned).not.toHaveProperty('hash') + expect(cleaned).not.toHaveProperty('assets') + expect(cleaned.content).toBe('export function main() {}') + expect(cleaned.summary).toBe('hi') + }) + + it('does not mutate the input object', () => { + const value = { path: 'u/me/foo', hash: 'h', assets: [] } + sanitizeDraftValueForSave('script', value) + expect(value).toHaveProperty('hash', 'h') + expect(value).toHaveProperty('assets') + }) + + it('returns the same reference when no omitted field is present', () => { + const value = { path: 'u/me/foo', summary: 'hi' } + expect(sanitizeDraftValueForSave('script', value)).toBe(value) + }) + + it('leaves non-script kinds untouched even if they carry hash/assets', () => { + const value = { hash: 'h', assets: [] } + expect(sanitizeDraftValueForSave('flow', value)).toBe(value) + expect(sanitizeDraftValueForSave('app', value)).toBe(value) + }) + + it('passes through null (the delete signal) and non-objects', () => { + expect(sanitizeDraftValueForSave('script', null)).toBeNull() + expect(sanitizeDraftValueForSave('script', 'hello')).toBe('hello') + const arr = [1, 2, 3] + expect(sanitizeDraftValueForSave('script', arr)).toBe(arr) + }) +}) diff --git a/frontend/src/lib/userDraftToast.ts b/frontend/src/lib/userDraftToast.ts index 7e3e48d211..dbb16b9c3d 100644 --- a/frontend/src/lib/userDraftToast.ts +++ b/frontend/src/lib/userDraftToast.ts @@ -1,31 +1,91 @@ -/** - * "Restored from local storage" toast, shown when an editor reopens on a - * local autosave that differs from the backend. Owns only the wording and - * which reset actions are offered; the reset side-effects live at each call - * site (route-specific state). - */ +/** Toast helpers for when a per-user draft is loaded from the server. */ import { sendUserToast } from '$lib/toast' +import { UserDraft } from '$lib/userDraft.svelte' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' +import type { UserDraftItemKind } from '$lib/gen' -export type RestoreFromLocalActions = { - /** Drop the local autosave, apply the backend DB draft. Offered when `hasBackendDraft`. */ - onResetToSavedDraft?: () => void | Promise - /** Drop the local autosave, load the deployed version. Offered when `hasDeployed`. */ - onResetToDeployed?: () => void | Promise +/** + * Run a "reset to deployed" with autosave suspended: the reset's own + * wipe-and-reload writes would otherwise re-create the draft just deleted. + * Sync re-arms on the user's next interaction, not via `tick()` — editors + * keep cascading writes after remount (Monaco acks, schema re-infer, iframe + * acks) that would resurrect the draft. Shared by the load-time toast and + * the AutosaveIndicator popover. + */ +export async function runResetToDeployed(opts: { + workspace: string + itemKind: UserDraftItemKind + path: string + onResetToDeployed: () => void | Promise +}): Promise { + UserDraft.stopSync(opts.itemKind, opts.path, { workspace: opts.workspace }) + UserDraftDbSyncer.save({ + workspace: opts.workspace, + itemKind: opts.itemKind, + path: opts.path, + value: null + }).catch((e) => console.error('Reset to deployed: draft delete failed', e)) + try { + await opts.onResetToDeployed() + } catch (e: any) { + sendUserToast(`Could not reset to deployed: ${e?.body ?? e}`, true) + } finally { + armRestartOnFirstInteraction(opts.workspace, opts.itemKind, opts.path) + } } -/** Show the toast with up to two reset actions, gated by what the backend has. */ -export function notifyRestoredFromLocal( - hasBackendDraft: boolean, - hasDeployed: boolean, - { onResetToSavedDraft, onResetToDeployed }: RestoreFromLocalActions +/** + * Post-deploy draft cleanup. Deletes the draft at its CANONICAL slot key + * (the path the handle is keyed on, NOT the just-typed deploy path, which + * differs for draft-only items) with sync muted and the delete flushed. + * Mute + flush both matter: `remove` only QUEUES the delete in the + * debouncer, and editors still mounted through the navigation (AppEditor, + * RawAppEditor) keep mirroring their value — one such write displaces the + * queued delete and re-saves the draft. Sync re-arms on next interaction. + */ +export function discardDraftAfterDeploy(opts: { + workspace: string + itemKind: UserDraftItemKind + path: string +}): void { + UserDraft.stopSync(opts.itemKind, opts.path, { workspace: opts.workspace }) + UserDraft.remove(opts.itemKind, opts.path, { workspace: opts.workspace }) + void UserDraftDbSyncer.flush({ + workspace: opts.workspace, + itemKind: opts.itemKind, + path: opts.path + }) + armRestartOnFirstInteraction(opts.workspace, opts.itemKind, opts.path) +} + +/** + * Defer `UserDraft.restartSync` to the first user interaction, with a 5s + * fallback so the suspension can't leak when the user navigates away + * untouched. Brackets bootstrap mutations that would otherwise POST as the + * "first edit" (template fills, conflict reloads, reset-to-deployed); pair + * with `UserDraft.stopSync` at the bootstrap start. + */ +export function armRestartOnFirstInteraction( + workspace: string, + itemKind: UserDraftItemKind, + path: string ): void { - const actions: Array<{ label: string; callback: () => void | Promise }> = [] - if (hasBackendDraft && onResetToSavedDraft) { - actions.push({ label: 'Reset to saved draft', callback: onResetToSavedDraft }) + let restarted = false + const restart = () => { + if (restarted) return + restarted = true + document.removeEventListener('keydown', restart, true) + document.removeEventListener('input', restart, true) + document.removeEventListener('pointerdown', restart, true) + clearTimeout(fallback) + UserDraft.restartSync(itemKind, path, { workspace }) } - if (hasDeployed && onResetToDeployed) { - actions.push({ label: 'Reset to deployed', callback: onResetToDeployed }) - } - if (actions.length === 0) return - sendUserToast('Restored from local storage', false, actions) + // Three events: keydown misses pointer-driven UI (sliders, pickers, the + // path popover OK); pointerdown misses pure-keyboard Monaco edits. + document.addEventListener('keydown', restart, { capture: true, once: true }) + document.addEventListener('input', restart, { capture: true, once: true }) + document.addEventListener('pointerdown', restart, { capture: true, once: true }) + // Fallback if the user navigates away before interacting, else the + // suspension leaks and autosave stays off next visit. + const fallback = setTimeout(restart, 5000) } diff --git a/frontend/src/lib/userNamespace.ts b/frontend/src/lib/userNamespace.ts new file mode 100644 index 0000000000..4839e46f6e --- /dev/null +++ b/frontend/src/lib/userNamespace.ts @@ -0,0 +1,31 @@ +import { get } from 'svelte/store' +import { userStore } from '$lib/stores' + +/** + * Workspace username for the authed user, suitable as the `u/{X}/...` + * namespace in editor draft paths. + * + * The `/add` → `/edit/u/{username}/draft_{uuid}` redirects in each + * editor's `+page.ts` run during SvelteKit's load phase — BEFORE the + * (logged) layout's async `getUserExt` populates `userStore`. Reading + * `get(userStore)?.username` from there returns `undefined` on every + * fresh nav, falling back to the `'me'` placeholder and producing + * `u/me/draft_{uuid}` paths instead of the real namespace. + * + * The layout writes `username` to `localStorage` on every successful + * `getUserExt`, so this helper: + * 1. Reads `userStore` first (live value when it IS populated, e.g. + * same-tab re-navs after the first load). + * 2. Falls back to `localStorage` (warm-cache from the previous + * session — covers the load-phase race on cold reload). + * 3. Falls back to `'me'` (true first-ever load with no session yet). + */ +export function getUsernameForNamespace(): string { + const live = get(userStore)?.username + if (live) return live + try { + const cached = localStorage.getItem('username') + if (cached) return cached + } catch {} + return 'me' +} diff --git a/frontend/src/lib/userScopedDb.test.ts b/frontend/src/lib/userScopedDb.test.ts new file mode 100644 index 0000000000..f97ec7cc09 --- /dev/null +++ b/frontend/src/lib/userScopedDb.test.ts @@ -0,0 +1,107 @@ +import { describe, it, expect, beforeEach, vi } from 'vitest' +import { IDBFactory } from 'fake-indexeddb' +import { openDB, type DBSchema, type IDBPDatabase } from 'idb' + +// scopedKey resolves the email from userStore via a BROWSER-gated subscription. +vi.mock('esm-env', async (importOriginal) => ({ + ...(await importOriginal()), + BROWSER: true +})) + +import { userStore, type UserExt } from '$lib/stores' +import { userScopedDb, type UserScopedDbMigrateDeps } from './userScopedDb' + +interface TestSchema extends DBSchema { + items: { key: string; value: { id: string; v: number } } +} + +function upgrade(db: IDBPDatabase) { + if (!db.objectStoreNames.contains('items')) { + db.createObjectStore('items', { keyPath: 'id' }) + } +} + +function asUser(email: string): UserExt { + return { email, username: email.split('@')[0] } as unknown as UserExt +} + +beforeEach(() => { + // Fresh in-memory IndexedDB per test for isolation. + ;(globalThis as any).indexedDB = new IDBFactory() + userStore.set(undefined) +}) + +describe('userScopedDb', () => { + it('returns undefined while no user is logged in', async () => { + const dbh = userScopedDb('t', { version: 1, upgrade }) + expect(await dbh.whenReady()).toBeUndefined() + }) + + it('isolates data between users and restores it on return', async () => { + const dbh = userScopedDb('t', { version: 1, upgrade }) + + userStore.set(asUser('a@x.com')) + const dbA = await dbh.whenReady() + await dbA!.put('items', { id: 'i1', v: 1 }) + + // Switch user: whenReady reopens the other user's (empty) DB — A's record + // is not visible. + userStore.set(asUser('b@y.com')) + const dbB = await dbh.whenReady() + expect(await dbB!.count('items')).toBe(0) + await dbB!.put('items', { id: 'i2', v: 2 }) + + // Back to A: their record is intact, B's is not present. + userStore.set(asUser('a@x.com')) + const dbA2 = await dbh.whenReady() + expect((await dbA2!.getAll('items')).map((x) => x.id)).toEqual(['i1']) + }) + + it('runs migrate once per scoped name and claims+deletes the legacy DB', async () => { + // Seed a legacy (un-namespaced) DB, mirroring the chat-history pattern. + const legacy = await openDB('t', 1, { upgrade }) + await legacy.put('items', { id: 'legacy1', v: 9 }) + legacy.close() + + const migrate = vi.fn(async (db: IDBPDatabase, deps: UserScopedDbMigrateDeps) => { + if ((await db.count('items')) > 0) return + const src = await deps.openDB('t', 1, { upgrade }) + const all = await src.getAll('items') + const tx = db.transaction('items', 'readwrite') + await Promise.all([...all.map((x) => tx.store.put(x)), tx.done]) + src.close() + await deps.deleteDB('t') + }) + + const dbh = userScopedDb('t', { version: 1, upgrade, migrate }) + userStore.set(asUser('a@x.com')) + + const db = await dbh.whenReady() + expect((await db!.getAll('items')).map((x) => x.id)).toEqual(['legacy1']) + // Legacy bare DB was deleted. + const names = (await indexedDB.databases()).map((d) => d.name) + expect(names).not.toContain('t') + expect(names).toContain('t::a@x.com') + + // migrate is gated to once per scoped name even across repeated whenReady. + await dbh.whenReady() + expect(migrate).toHaveBeenCalledTimes(1) + }) + + it('degrades to undefined (no throw) when the DB cannot be opened', async () => { + const failingOpen = vi.fn(async () => { + throw new Error('blocked') + }) as unknown as typeof openDB + const dbh = userScopedDb('t', { version: 1, upgrade, openDB: failingOpen }) + userStore.set(asUser('a@x.com')) + expect(await dbh.whenReady()).toBeUndefined() + }) + + it('clears the handle on logout', async () => { + const dbh = userScopedDb('t', { version: 1, upgrade }) + userStore.set(asUser('a@x.com')) + expect(await dbh.whenReady()).toBeDefined() + userStore.set(undefined) + expect(await dbh.whenReady()).toBeUndefined() + }) +}) diff --git a/frontend/src/lib/userScopedDb.ts b/frontend/src/lib/userScopedDb.ts new file mode 100644 index 0000000000..4b7242f35a --- /dev/null +++ b/frontend/src/lib/userScopedDb.ts @@ -0,0 +1,103 @@ +import { openDB as idbOpenDB, deleteDB as idbDeleteDB, type DBSchema, type IDBPDatabase } from 'idb' +import { scopedKey } from '$lib/userScopedStorage' + +// Per-user IndexedDB lifecycle, shared by the session list and the copilot +// chat-history stores. The effective DB name is the base name namespaced by the +// logged-in user's email (scopedKey), so two users on a shared browser never +// touch the same physical database. +// +// `whenReady()` is name-aware: it computes the current scoped name on every call +// and transparently closes + reopens when the email changes, so the handle +// self-heals on user switch WITHOUT subscribing to onUserChange. That matters +// because there is one handle per HistoryManager instance (singleton + one per +// session runtime) — a per-instance subscription would leak callbacks. + +export interface UserScopedDbMigrateDeps { + openDB: typeof idbOpenDB + deleteDB: typeof idbDeleteDB +} + +export interface UserScopedDbOptions { + version: number + upgrade: (db: IDBPDatabase) => void + // Invoked once per scoped name right after a successful open. The fn owns its + // own "already migrated / not applicable" gate (e.g. checking a store's + // count) — claim-then-delete legacy data lives here. + migrate?: (db: IDBPDatabase, deps: UserScopedDbMigrateDeps) => Promise + // Injectable for tests (defaults to the real idb implementations). + openDB?: typeof idbOpenDB + deleteDB?: typeof idbDeleteDB +} + +export interface UserScopedDb { + // Resolves to the open DB for the current user, or undefined when no user is + // logged in yet or the open failed (degrade to in-memory; never rejects). + whenReady(): Promise | undefined> + close(): void +} + +export function userScopedDb( + baseName: string, + opts: UserScopedDbOptions +): UserScopedDb { + const openDB = opts.openDB ?? idbOpenDB + const deleteDB = opts.deleteDB ?? idbDeleteDB + const migratedNames = new Set() + + let openName: string | undefined + let openPromise: Promise | undefined> | undefined + + function closeCurrent() { + const prev = openPromise + if (prev) void prev.then((db) => db?.close()).catch(() => {}) + openPromise = undefined + openName = undefined + } + + async function open(name: string): Promise | undefined> { + try { + const db = await openDB(name, opts.version, { + upgrade(database) { + opts.upgrade(database) + } + }) + if (opts.migrate && !migratedNames.has(name)) { + migratedNames.add(name) + try { + await opts.migrate(db, { openDB, deleteDB }) + } catch (e) { + // A failed migration is non-fatal: the (open) DB is still usable, so + // we log and return it — unlike a failed open below, which yields + // undefined. Worst case the legacy claim is missed, not the store. + console.error(`userScopedDb(${baseName}): migration failed`, e) + } + } + return db + } catch (e) { + // Failed open (blocked / corrupt / private-browsing): degrade to + // in-memory by resolving undefined (callers no-op their writes). The + // undefined is cached for this name so we don't hammer the open. + console.error(`userScopedDb(${baseName}): could not open database`, e) + return undefined + } + } + + return { + whenReady() { + const name = scopedKey(baseName) + if (!name) { + closeCurrent() + return Promise.resolve(undefined) + } + if (name !== openName) { + closeCurrent() + openName = name + openPromise = open(name) + } + return openPromise! + }, + close() { + closeCurrent() + } + } +} diff --git a/frontend/src/lib/userScopedStorage.test.ts b/frontend/src/lib/userScopedStorage.test.ts new file mode 100644 index 0000000000..bc25734171 --- /dev/null +++ b/frontend/src/lib/userScopedStorage.test.ts @@ -0,0 +1,81 @@ +import { describe, it, expect, beforeEach, vi } from 'vitest' + +// The userStore subscription is gated on BROWSER; the vitest "server" env +// reports BROWSER=false. +vi.mock('esm-env', async (importOriginal) => ({ + ...(await importOriginal()), + BROWSER: true +})) + +import { userStore, type UserExt } from '$lib/stores' +import { + scopedKey, + getCurrentUserEmail, + onUserChange, + migrateLegacyLocalStorage +} from './userScopedStorage' + +function asUser(email: string): UserExt { + return { email, username: email.split('@')[0] } as unknown as UserExt +} + +beforeEach(() => { + localStorage.clear() + userStore.set(undefined) +}) + +describe('scopedKey / getCurrentUserEmail', () => { + it('returns undefined while no user is logged in', () => { + expect(getCurrentUserEmail()).toBeUndefined() + expect(scopedKey('windmill_sessions')).toBeUndefined() + }) + + it('namespaces by email and isolates distinct users', () => { + userStore.set(asUser('a@x.com')) + expect(scopedKey('windmill_sessions')).toBe('windmill_sessions::a@x.com') + userStore.set(asUser('b@y.com')) + expect(scopedKey('windmill_sessions')).toBe('windmill_sessions::b@y.com') + }) +}) + +describe('onUserChange', () => { + it('fires immediately with the current email, then on every change', () => { + userStore.set(asUser('a@x.com')) + const calls: Array<[string | undefined, string | undefined]> = [] + onUserChange((email, prev) => calls.push([email, prev])) + // Immediate fire with current email, prev undefined. + expect(calls).toEqual([['a@x.com', undefined]]) + userStore.set(asUser('b@y.com')) + expect(calls).toEqual([ + ['a@x.com', undefined], + ['b@y.com', 'a@x.com'] + ]) + // No-op when the email is unchanged. + userStore.set(asUser('b@y.com')) + expect(calls).toHaveLength(2) + }) +}) + +describe('migrateLegacyLocalStorage', () => { + it('claims a legacy key into the target and deletes the legacy copy', () => { + localStorage.setItem('ai-chat-autonomy-mode', 'yolo') + migrateLegacyLocalStorage('ai-chat-autonomy-mode', 'ai-chat-autonomy-mode::a@x.com') + expect(localStorage.getItem('ai-chat-autonomy-mode::a@x.com')).toBe('yolo') + expect(localStorage.getItem('ai-chat-autonomy-mode')).toBeNull() + }) + + it('does not overwrite an existing target', () => { + localStorage.setItem('ai-chat-autonomy-mode', 'yolo') + localStorage.setItem('ai-chat-autonomy-mode::a@x.com', 'acceptedit') + migrateLegacyLocalStorage('ai-chat-autonomy-mode', 'ai-chat-autonomy-mode::a@x.com') + expect(localStorage.getItem('ai-chat-autonomy-mode::a@x.com')).toBe('acceptedit') + // Legacy left untouched since the target was already populated. + expect(localStorage.getItem('ai-chat-autonomy-mode')).toBe('yolo') + }) + + it('is a no-op when the target key is undefined (no user)', () => { + localStorage.setItem('ai-chat-autonomy-mode', 'yolo') + migrateLegacyLocalStorage('ai-chat-autonomy-mode', undefined) + expect(localStorage.getItem('ai-chat-autonomy-mode')).toBe('yolo') + }) +}) diff --git a/frontend/src/lib/userScopedStorage.ts b/frontend/src/lib/userScopedStorage.ts new file mode 100644 index 0000000000..905d609b61 --- /dev/null +++ b/frontend/src/lib/userScopedStorage.ts @@ -0,0 +1,85 @@ +import { BROWSER } from 'esm-env' +import { userStore } from '$lib/stores' +import { getLocalSetting, storeLocalSetting } from '$lib/utils' + +// Browser-persisted AI-session state (the `/sessions` list, unread markers, +// chat-history IndexedDB, autonomy mode) must be scoped to the logged-in +// instance user so a shared browser never leaks one user's data to the next. +// +// The scoping identity is the instance user **email** — the only identity +// stable across a fork family (a single session deliberately spans multiple +// workspace_ids; `username` varies per workspace, `email` does not). +// +// The email arrives asynchronously (userStore is populated after the layout's +// getUserExt), while several consumers initialise eagerly at module import. +// This module owns the single subscription and lets each surface register a +// hydrate+migrate callback that fires once on registration (covering late +// registrants whose email already resolved) and again on every email change. + +type UserChangeCallback = (email: string | undefined, prevEmail: string | undefined) => void + +let currentEmail: string | undefined = undefined +const callbacks = new Set() + +if (BROWSER) { + // userStore.subscribe fires synchronously with the current value (undefined + // at import time) and again whenever the user resolves / changes. Logout is + // a full page reload, and email is constant across workspace switches, so in + // practice the only transition is the initial undefined → email hydration. + userStore.subscribe((u) => { + const next = u?.email + if (next === currentEmail) return + const prev = currentEmail + currentEmail = next + for (const cb of callbacks) { + try { + cb(next, prev) + } catch (e) { + console.error('userScopedStorage: onUserChange callback failed', e) + } + } + }) +} + +// Current logged-in user's email, or undefined before it resolves / when +// logged out. +export function getCurrentUserEmail(): string | undefined { + return currentEmail +} + +// Namespace a base storage key (localStorage key or IndexedDB name) by the +// current user's email. Returns undefined when no user is known — callers must +// treat that as "do not read/write" so we never touch a browser-global key. +export function scopedKey(base: string): string | undefined { + if (!currentEmail) return undefined + return `${base}::${currentEmail}` +} + +// Register a callback invoked whenever the scoping email changes. Fired once +// immediately with the current email (prevEmail undefined) so a surface that +// registers after the email already resolved still hydrates. +export function onUserChange(cb: UserChangeCallback): void { + callbacks.add(cb) + try { + cb(currentEmail, undefined) + } catch (e) { + console.error('userScopedStorage: initial onUserChange callback failed', e) + } +} + +// One-shot claim of pre-namespacing data: if the scoped key is empty and a +// legacy un-namespaced key exists, move it under the scoped key and delete the +// legacy copy. The first user to log in on a previously single-user browser +// keeps their data; subsequent users start clean. +export function migrateLegacyLocalStorage(legacyKey: string, targetKey: string | undefined): void { + if (!BROWSER || !targetKey) return + try { + if (getLocalSetting(targetKey) != null) return + const legacy = getLocalSetting(legacyKey) + if (legacy == null) return + storeLocalSetting(targetKey, legacy) + storeLocalSetting(legacyKey, undefined) + } catch (e) { + console.error('userScopedStorage: legacy localStorage migration failed', e) + } +} diff --git a/frontend/src/lib/utils.test.ts b/frontend/src/lib/utils.test.ts index d5e1ba0c89..dc7e52e9f4 100644 --- a/frontend/src/lib/utils.test.ts +++ b/frontend/src/lib/utils.test.ts @@ -1,5 +1,39 @@ import { describe, it, expect } from 'vitest' -import { getQueryStmtCountHeuristic } from './utils' +import { + cleanValueProperties, + getQueryStmtCountHeuristic, + parseDbInputFromAssetSyntax +} from './utils' + +describe('parseDbInputFromAssetSyntax', () => { + it('parses a table path', () => { + expect(parseDbInputFromAssetSyntax('ducklake://main/orders')).toEqual({ + type: 'ducklake', + ducklake: 'main', + specificTable: 'orders', + specificSchema: undefined + }) + }) + + it('parses a schema-qualified table path', () => { + expect(parseDbInputFromAssetSyntax('ducklake://main/analytics.orders')).toEqual({ + type: 'ducklake', + ducklake: 'main', + specificTable: 'orders', + specificSchema: 'analytics' + }) + }) + + it('handles a catalog-only path without throwing (no table segment)', () => { + // e.g. `// materialize ducklake` → `ducklake://main` — must not throw. + expect(parseDbInputFromAssetSyntax('ducklake://main')).toEqual({ + type: 'ducklake', + ducklake: 'main', + specificTable: undefined, + specificSchema: undefined + }) + }) +}) describe('getQueryStmtCountHeuristic', () => { describe('basic statements', () => { @@ -283,3 +317,76 @@ DELETE FROM logs WHERE timestamp < NOW() - INTERVAL '30 days' }) }) }) + +describe('cleanValueProperties', () => { + const serverManagedKeys = [ + 'parent_hash', + 'hash', + 'assets', + 'inherited_labels', + 'draft', + 'draft_only', + 'draft_saved_at', + 'draft_created_at', + 'is_draft', + 'other_drafts_users', + 'created_at', + 'created_by', + 'workspace_id', + 'parent_hashes', + 'lock_error_logs' + ] + + it('strips every server-managed bookkeeping key', () => { + const input: any = { summary: 'hi' } + for (const key of serverManagedKeys) { + input[key] = 'noise' + } + const cleaned = cleanValueProperties(input) as any + for (const key of serverManagedKeys) { + expect(cleaned).not.toHaveProperty(key) + } + }) + + it('preserves user-editable keys', () => { + const input: any = { + summary: 'my script', + description: 'does things', + content: 'export function main() {}', + schema: { properties: { x: { type: 'string' } } }, + language: 'bun', + created_at: '2024-01-01' + } + const cleaned = cleanValueProperties(input) as any + expect(cleaned.summary).toBe('my script') + expect(cleaned.description).toBe('does things') + expect(cleaned.content).toBe('export function main() {}') + expect(cleaned.schema).toEqual({ properties: { x: { type: 'string' } } }) + expect(cleaned.language).toBe('bun') + expect(cleaned).not.toHaveProperty('created_at') + }) + + it('preserves lock so version-to-version diffs still surface lockfile changes', () => { + const cleaned = cleanValueProperties({ summary: 'hi', lock: 'resolved deps' } as any) as any + expect(cleaned.lock).toBe('resolved deps') + }) + + it('preserves extra_perms so folder workspace/fork diffs still surface permission changes', () => { + const cleaned = cleanValueProperties({ + summary: 'hi', + extra_perms: { 'u/foo': true } + } as any) as any + expect(cleaned.extra_perms).toEqual({ 'u/foo': true }) + }) + + it('returns non-object values unchanged', () => { + expect(cleanValueProperties('hello' as any)).toBe('hello') + expect(cleanValueProperties(42 as any)).toBe(42) + }) + + it('does not mutate the input object', () => { + const input: any = { summary: 'hi', created_at: '2024-01-01' } + cleanValueProperties(input) + expect(input).toHaveProperty('created_at') + }) +}) diff --git a/frontend/src/lib/utils.ts b/frontend/src/lib/utils.ts index 2e6302fb6c..fdca33fa22 100644 --- a/frontend/src/lib/utils.ts +++ b/frontend/src/lib/utils.ts @@ -1190,8 +1190,10 @@ export function isCodeInjection(expr: string | undefined): boolean { // app logic via the `query` context. Only params we actually own are listed // here — the `wm_` prefix is a naming convention, not a reserved namespace, so // we don't strip it wholesale (that would break apps reading their own `wm_*` -// params). `wm_coep` is a transport flag for cross-origin isolation headers. -export const WINDMILL_RESERVED_QUERY_PARAMS = new Set(['wm_coep']) +// params). `wm_coep` is a transport flag for cross-origin isolation headers; +// `wm_embed`/`wm_embedder_origin` are the opaque app viewer transport params +// (see PublicAppFrame). +export const WINDMILL_RESERVED_QUERY_PARAMS = new Set(['wm_coep', 'wm_embed', 'wm_embedder_origin']) export function urlParamsToObject( params: URLSearchParams, @@ -1296,13 +1298,47 @@ function replaceFalseWithUndefinedRec(obj: any) { return obj } +// Keys that are server-managed bookkeeping, never user-editable, and therefore +// must not surface in any value diff or unsaved-change comparison. `getScriptByPath` +// (and the flow/app equivalents) return the full DB row, so the editing object +// carries these while the deployed side is fetched trimmed — leaving them in would +// render as spurious metadata diff. +// +// `hash` is the deployed version's identity, `assets` is re-derived from the +// script content by the editor, and `inherited_labels` is computed at read time +// from the parent folder — none is editable content, so all three are noise in a +// fork/workspace or version diff. +// +// `lock` and `extra_perms` are deliberately NOT in this set: both are legitimate, +// user-meaningful fields in some diff contexts (lockfile changes in version-to-version +// diffs, folder sharing-permission changes in workspace/fork diffs). The script-editor +// noise they would otherwise cause is stripped at the source instead (the deployed side +// in `ScriptBuilder.syncWithDeployed`, the current side in `ScriptBuilder.openDiffDrawer`). +const CLEANED_VALUE_KEYS = new Set([ + 'parent_hash', + 'hash', + 'assets', + 'inherited_labels', + 'draft', + 'draft_only', + 'draft_saved_at', + 'draft_created_at', + 'is_draft', + 'other_drafts_users', + 'created_at', + 'created_by', + 'workspace_id', + 'parent_hashes', + 'lock_error_logs' +]) + export function cleanValueProperties(obj: Value) { if (typeof obj !== 'object') { return obj } else { let newObj: any = {} for (const key of Object.keys(obj)) { - if (key !== 'parent_hash' && key !== 'draft' && key !== 'draft_only') { + if (!CLEANED_VALUE_KEYS.has(key)) { newObj[key] = structuredClone(stateSnapshot(obj[key])) } } @@ -2084,17 +2120,27 @@ export function pick(obj: T, keys: readonly export function parseDbInputFromAssetSyntax(path: string): DbInput | null { const [p1, _p2] = path.split('://') - const [p2, _p3] = _p2.split('/') - const [p3, p4] = _p3.split('.') + const [p2, _p3] = (_p2 ?? '').split('/') + // `_p3` is undefined for a catalog-only path (e.g. `ducklake://main`, no + // table segment) — guard the split so the helper returns a table-less input + // instead of throwing. + const [p3, p4] = (_p3 ?? '').split('.') + const specificTable = p4 || p3 || undefined + const specificSchema = p4 ? p3 : undefined return p1 === 'ducklake' - ? { type: 'ducklake', ducklake: p2 || 'main', specificTable: p4 ?? p3 } + ? { + type: 'ducklake', + ducklake: p2 || 'main', + specificTable, + specificSchema + } : p1 === 'datatable' ? { type: 'database', resourcePath: `datatable://${p2 || 'main'}`, resourceType: 'postgresql', - specificTable: p4 ?? p3, - specificSchema: p4 ? p3 : undefined + specificTable, + specificSchema } : null } diff --git a/frontend/src/lib/utils/editInFork.ts b/frontend/src/lib/utils/editInFork.ts index 2a0768694d..aa6d653961 100644 --- a/frontend/src/lib/utils/editInFork.ts +++ b/frontend/src/lib/utils/editInFork.ts @@ -9,13 +9,13 @@ export function buildForkEditUrl(itemType: ItemType, itemPath: string): string { editPath = `${base}/scripts/edit/${itemPath}` break case 'flow': - editPath = `${base}/flows/edit/${itemPath}?nodraft=true` + editPath = `${base}/flows/edit/${itemPath}` break case 'app': - editPath = `${base}/apps/edit/${itemPath}?nodraft=true` + editPath = `${base}/apps/edit/${itemPath}` break case 'raw_app': - editPath = `${base}/apps_raw/edit/${itemPath}?nodraft=true` + editPath = `${base}/apps_raw/edit/${itemPath}` break } return `${base}/user/fork_workspace?rd=${encodeURIComponent(editPath)}` diff --git a/frontend/src/lib/utils_deployable.ts b/frontend/src/lib/utils_deployable.ts index 98aeb8b82d..5f49eb5c49 100644 --- a/frontend/src/lib/utils_deployable.ts +++ b/frontend/src/lib/utils_deployable.ts @@ -43,6 +43,9 @@ export type Kind = // Legacy generic kind used by the cross-workspace `DeployWorkspace` UI, // which carries the trigger sub-kind in `additionalInformation`. | 'trigger' + // A data-pipeline draft bundle (drafts list only — not a deployable item; + // opens the pipeline view instead). + | 'data_pipeline' export const ALL_DEPLOYABLE: WorkspaceDeployUISettings = { include_path: [], @@ -334,7 +337,7 @@ export async function getTriggersDeployData( * `TRIGGER_COMPARE_IGNORE` and `stripTriggerOrScheduleRuntimeFields` in the * shared deploy module. */ -const TRIGGER_RUNTIME_IGNORE = new Set([ +export const TRIGGER_RUNTIME_IGNORE = new Set([ 'workspace_id', 'edited_by', 'edited_at', diff --git a/frontend/src/lib/utils_draft_deploy.ts b/frontend/src/lib/utils_draft_deploy.ts index 8ce4624fc1..2b08b379cf 100644 --- a/frontend/src/lib/utils_draft_deploy.ts +++ b/frontend/src/lib/utils_draft_deploy.ts @@ -13,30 +13,156 @@ * a draft on an already-deployed item just deletes the draft row. */ import { get, writable } from 'svelte/store' -import { ScriptService, FlowService, AppService, DraftService } from '$lib/gen' +import { + DraftService, + ScriptService, + FlowService, + AppService, + VariableService, + ResourceService, + ScheduleService, + HttpTriggerService, + WebsocketTriggerService, + PostgresTriggerService, + KafkaTriggerService, + NatsTriggerService, + MqttTriggerService, + SqsTriggerService, + GcpTriggerService, + AzureTriggerService, + EmailTriggerService, + type UserDraftItemKind +} from '$lib/gen' +import { UserDraftDbSyncer } from '$lib/userDraftDbSyncer.svelte' import type { DeployResult } from '$lib/utils_workspace_deploy' +import { TRIGGER_RUNTIME_IGNORE } from '$lib/utils_deployable' import { deployRawAppDraft } from '$lib/rawAppDeploy' +import { canonicalRawAppDiffValue } from '$lib/components/raw_apps/utils' import { invalidateWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' +import { setLocalDraftHint } from '$lib/localDraftHints.svelte' import { userStore } from '$lib/stores' import { deployTriggers, type Trigger } from '$lib/components/triggers/utils' +import { saveScheduleFromCfg } from '$lib/components/flows/scheduleUtils' +import { saveHttpRouteFromCfg } from '$lib/components/triggers/http/utils' +import { saveWebsocketTriggerFromCfg } from '$lib/components/triggers/websocket/utils' +import { savePostgresTriggerFromCfg } from '$lib/components/triggers/postgres/utils' +import { saveKafkaTriggerFromCfg } from '$lib/components/triggers/kafka/utils' +import { saveNatsTriggerFromCfg } from '$lib/components/triggers/nats/utils' +import { saveMqttTriggerFromCfg } from '$lib/components/triggers/mqtt/utils' +import { saveSqsTriggerFromCfg } from '$lib/components/triggers/sqs/utils' +import { saveGcpTriggerFromCfg } from '$lib/components/triggers/gcp/utils' +import { saveAzureTriggerFromCfg } from '$lib/components/triggers/azure/utils' +import { saveEmailTriggerFromCfg } from '$lib/components/triggers/email/utils' -export type DraftKind = 'script' | 'flow' | 'app' +export type DraftKind = UserDraftItemKind + +/** Kind → "get by path with draft overlay" call, for kinds whose draft is the + * editor's flat config (all but script/flow/app, handled below). Feature-gated + * trigger services 404 when the backend lacks the kind; the caller surfaces it. */ +const OVERLAY_GETTERS: Partial< + Record Promise> +> = { + variable: (workspace, path) => + VariableService.getVariable({ workspace, path, decryptSecret: false, getDraft: true }), + resource: (workspace, path) => ResourceService.getResource({ workspace, path, getDraft: true }), + trigger_schedule: (workspace, path) => + ScheduleService.getSchedule({ workspace, path, getDraft: true }), + trigger_http: (workspace, path) => + HttpTriggerService.getHttpTrigger({ workspace, path, getDraft: true }), + trigger_websocket: (workspace, path) => + WebsocketTriggerService.getWebsocketTrigger({ workspace, path, getDraft: true }), + trigger_postgres: (workspace, path) => + PostgresTriggerService.getPostgresTrigger({ workspace, path, getDraft: true }), + trigger_kafka: (workspace, path) => + KafkaTriggerService.getKafkaTrigger({ workspace, path, getDraft: true }), + trigger_nats: (workspace, path) => + NatsTriggerService.getNatsTrigger({ workspace, path, getDraft: true }), + trigger_mqtt: (workspace, path) => + MqttTriggerService.getMqttTrigger({ workspace, path, getDraft: true }), + trigger_sqs: (workspace, path) => + SqsTriggerService.getSqsTrigger({ workspace, path, getDraft: true }), + trigger_gcp: (workspace, path) => + GcpTriggerService.getGcpTrigger({ workspace, path, getDraft: true }), + trigger_azure: (workspace, path) => + AzureTriggerService.getAzureTrigger({ workspace, path, getDraft: true }), + trigger_email: (workspace, path) => + EmailTriggerService.getEmailTrigger({ workspace, path, getDraft: true }) +} + +/** Strip the per-user draft-overlay metadata, returning `{deployed, draft}`. */ +function splitOverlay(r: any): { deployed: any; draft: any } { + const { + draft, + is_draft: _i, + draft_saved_at: _c, + no_deployed: _n, + other_drafts_users: _o, + ...deployed + } = r + return { deployed, draft: draft ?? deployed } +} export interface DraftDiffValues { deployed: unknown draft: unknown } -// Empty-but-valid "deployed" shapes for draft_only items (which have never been -// deployed). Using a fully-empty `{}` breaks the flow graph diff (it needs -// `value.modules`) and leaves the drawer spinning — so each kind gets a minimal -// valid shape, making the whole draft show as "all new". -const EMPTY_DEPLOYED: Record unknown> = { +// Empty-but-valid "deployed" shapes for draft_only items. A bare `{}` breaks +// the flow graph diff (needs `value.modules`) and hangs the drawer, so each +// listed kind gets a minimal shape; unlisted kinds fall back to `{}`. +const EMPTY_DEPLOYED: Partial unknown>> = { script: (draft) => ({ content: '', language: draft?.language, schema: {} }), flow: () => ({ summary: '', value: { modules: [] }, schema: {} }), app: () => ({ summary: '', value: {}, policy: {} }) } +// Schedule & trigger rows drop the same runtime/server-managed fields as the +// fork/compare path so the diff shows only config changes — reuse that set +// (the authoritative mirror of the backend `TRIGGER_COMPARE_IGNORE`). +function stripScheduleTriggerRuntime(row: any): Record { + if (!row || typeof row !== 'object') return {} + return Object.fromEntries(Object.entries(row).filter(([k]) => !TRIGGER_RUNTIME_IGNORE.has(k))) +} + +/** + * Project a variable/resource/schedule/trigger value — given in either its + * deployed backend-row shape or its draft editor-state shape — onto one + * canonical field set, so the deployed and draft sides of a diff are comparable + * and read as labeled rows instead of structural noise (variable `variable.value` + * vs `value`, resource `args` vs `value`, schedule/trigger runtime fields). This + * matches the shaping the compare page applies via `getItemValue`. `isDraft` + * selects the editor-state field names; secret variable values are masked. + */ +function canonicalizeDraftDiffValue(kind: DraftKind, raw: any, isDraft: boolean): unknown { + if (!raw || typeof raw !== 'object') return raw ?? {} + if (kind === 'variable') { + // draft: { variable: { value, is_secret, description }, labels, wsSpecific } + // deployed row: { value, is_secret, description, labels, ws_specific } + const v = isDraft ? (raw.variable ?? {}) : raw + const is_secret = !!v.is_secret + return { + value: is_secret ? '' : (v.value ?? ''), + is_secret, + description: v.description ?? '', + labels: raw.labels ?? undefined, + ws_specific: (isDraft ? raw.wsSpecific : raw.ws_specific) ?? undefined + } + } + if (kind === 'resource') { + // draft: { args, description, resource_type, labels, wsSpecific } + // deployed row: { value, description, resource_type, labels, ws_specific } + return { + value: (isDraft ? raw.args : raw.value) ?? {}, + description: raw.description ?? '', + resource_type: raw.resource_type ?? undefined, + labels: raw.labels ?? undefined, + ws_specific: (isDraft ? raw.wsSpecific : raw.ws_specific) ?? undefined + } + } + // schedule + triggers: same field names on both sides — drop runtime noise. + return stripScheduleTriggerRuntime(raw) +} + /** * Fetch the deployed value and the draft value for an item, for the DiffDrawer * (`mode: 'simple'`, original = deployed, current = draft). For a `draft_only` @@ -54,18 +180,59 @@ export async function getDraftDiffValues( // draft-table row (e.g. a flow created via createFlow(draft_only: true), like // `u/admin/new`). There `draft` is null, so the draft side must fall back to // the row's own value — otherwise the diff "after" is empty and nothing shows. + // Strip overlay metadata (is_draft / draft_saved_at / no_deployed / + // other_drafts_users) from the deployed side so the diff doesn't show the + // per-user markers as noise. if (kind === 'script') { - const r = (await ScriptService.getScriptByPathWithDraft({ workspace, path })) as any - const { draft, draft_created_at: _c, hash: _h, ...deployed } = r + const r = (await ScriptService.getScriptByPath({ workspace, path, getDraft: true })) as any + const { + draft, + is_draft: _i, + draft_saved_at: _c, + no_deployed: _n, + other_drafts_users: _o, + hash: _h, + ...deployed + } = r const draftValue = draft ?? deployed - return { deployed: draftOnly ? EMPTY_DEPLOYED.script(draftValue) : deployed, draft: draftValue } + return { + deployed: draftOnly ? EMPTY_DEPLOYED.script!(draftValue) : deployed, + draft: draftValue + } } else if (kind === 'flow') { - const r = (await FlowService.getFlowByPathWithDraft({ workspace, path })) as any - const { draft, draft_created_at: _c, ...deployed } = r - const draftValue = draft ?? deployed - return { deployed: draftOnly ? EMPTY_DEPLOYED.flow(draftValue) : deployed, draft: draftValue } - } else { - const r = (await AppService.getAppByPathWithDraft({ workspace, path })) as any + const r = (await FlowService.getFlowByPath({ workspace, path, getDraft: true })) as any + const { + draft, + is_draft: _i, + draft_saved_at: _c, + no_deployed: _n, + other_drafts_users: _o, + version_id: _v, + ...deployed + } = r + // Strip the draft's pinned base `version_id` (which differs from the deployed + // head for a stale draft) so it never renders as a spurious diff line. + const { version_id: _dv, ...draftValue } = (draft ?? deployed) as any + return { deployed: draftOnly ? EMPTY_DEPLOYED.flow!(draftValue) : deployed, draft: draftValue } + } else if (kind === 'app' || kind === 'raw_app') { + // A never-deployed raw app has no `app` row; the backend resolves the + // draft kind from `rawApp`, so it MUST be set or the lookup 404s. + const r = (await AppService.getAppByPath({ + workspace, + path, + getDraft: true, + rawApp: kind === 'raw_app' + })) as any + if (kind === 'raw_app' || r.raw_app === true) { + // Raw-app drafts are stored flat (files/runnables/data top-level) while the + // deployed row nests them under `value`, and deployed inline scripts carry + // server-recomputed locks. Canonicalize both onto the same shape with the + // post-deploy noise stripped — the same module the editor's Diff button uses. + return { + deployed: draftOnly ? canonicalRawAppDiffValue({}) : canonicalRawAppDiffValue(r), + draft: canonicalRawAppDiffValue(r.draft ?? r) + } + } const deployed = { summary: r.summary, value: r.value, @@ -73,8 +240,25 @@ export async function getDraftDiffValues( path: r.path, custom_path: r.custom_path } - const draftValue = r.draft ?? deployed - return { deployed: draftOnly ? EMPTY_DEPLOYED.app(draftValue) : deployed, draft: draftValue } + // Strip the draft's pinned fork-base `parent_version` (the deployed allowlist + // above already omits it) so it never renders as a spurious diff line. + const { parent_version: _pv, ...draftValue } = (r.draft ?? deployed) as any + return { deployed: draftOnly ? EMPTY_DEPLOYED.app!(draftValue) : deployed, draft: draftValue } + } else { + // Variables / resources / schedules / triggers: one overlay GET yields + // both sides, but the draft side is the editor's state shape while the + // deployed side is the backend row — they diverge enough to make a raw + // diff pure noise. Canonicalize both onto a shared field set (same shaping + // the compare page's `getItemValue` applies) so only real changes show. + const getter = OVERLAY_GETTERS[kind] + if (!getter) { + throw new Error(`Draft diff not supported for kind ${kind}`) + } + const { deployed, draft } = splitOverlay(await getter(workspace, path)) + return { + deployed: draftOnly ? {} : canonicalizeDraftDiffValue(kind, deployed, false), + draft: canonicalizeDraftDiffValue(kind, draft, true) + } } } @@ -108,16 +292,19 @@ export async function deployDraft( kind: DraftKind, path: string, workspace: string, - draftOnly = false, - rawApp = false + opts: { draftOnly?: boolean; rawApp?: boolean; deploymentMessage?: string } = {} ): Promise { + const { draftOnly = false, rawApp = false, deploymentMessage } = opts try { - if (kind === 'app' && rawApp) { - // Raw apps bundle their source files to js/css and deploy via the - // raw-app endpoints — same as the global AI chat's deploy. - await deployRawAppDraft(workspace, path) + if (kind === 'raw_app' || (kind === 'app' && rawApp)) { + // Raw apps bundle their source files and deploy via the raw-app + // endpoints. Reached as `kind === 'raw_app'` (Review & Deploy) or + // `kind === 'app'` + `rawApp` (editor). Must route here: the + // visual-app branch would `updateApp` with no `value` (RawAppDraft + // has none) and silently drop the draft's files. + await deployRawAppDraft(workspace, path, deploymentMessage) } else if (kind === 'script') { - const r = (await ScriptService.getScriptByPathWithDraft({ workspace, path })) as any + const r = (await ScriptService.getScriptByPath({ workspace, path, getDraft: true })) as any const d = r.draft ?? r // Drop editor-only / server-managed keys; deploy as a real (non-draft) version. const { draft_triggers: draftTriggers, draft_only: _o, ...rest } = d @@ -126,16 +313,24 @@ export async function deployDraft( // the editor: createScript at the new path with parent_hash links lineage). await ScriptService.createScript({ workspace, - requestBody: { ...rest, path: scriptPath, parent_hash: r.hash } + requestBody: { + ...rest, + path: scriptPath, + parent_hash: r.hash, + deployment_message: deploymentMessage + } }) // Then deploy any draft trigger edits, so they aren't dropped with the draft. await deployDraftTriggers(draftTriggers, workspace, scriptPath, true) } else if (kind === 'flow') { - const r = (await FlowService.getFlowByPathWithDraft({ workspace, path })) as any + const r = (await FlowService.getFlowByPath({ workspace, path, getDraft: true })) as any const d = r.draft ?? r const requestBody = { - // Honor a renamed draft path; the URL `path` stays the existing item key. - path: d.path ?? path, + // Deploy at the draft's intended path: flow/app/raw-app drafts keep the + // user-typed path in `draft_path` (a never-deployed item is parked at a + // synthetic `u/{user}/draft_{uuid}` storage key). The URL `path` stays + // that storage key. + path: d.draft_path ?? d.path ?? path, summary: d.summary ?? '', description: d.description ?? '', value: d.value, @@ -145,72 +340,249 @@ export async function deployDraft( ws_error_handler_muted: d.ws_error_handler_muted, visible_to_runner_only: d.visible_to_runner_only, on_behalf_of_email: d.on_behalf_of_email, - labels: d.labels + labels: d.labels, + deployment_message: deploymentMessage + } + // Draft-only flows have NO flow row (they live solely in the + // draft table), so they deploy via createFlow; a draft on a + // deployed flow updates it. + if (draftOnly) { + await FlowService.createFlow({ workspace, requestBody }) + } else { + await FlowService.updateFlow({ workspace, path, requestBody }) } - // A draft (draft_only or on a deployed flow) always has a flow row, so - // updateFlow is correct in both cases — it promotes a draft_only flow to - // a real deployed version (clearing the flag). createFlow would 400 - // "Flow already exists". - await FlowService.updateFlow({ workspace, path, requestBody }) // Then deploy any draft trigger edits, so they aren't dropped with the draft. - await deployDraftTriggers(d.draft_triggers, workspace, d.path ?? path, draftOnly) - } else { - const r = (await AppService.getAppByPathWithDraft({ workspace, path })) as any - const d = r.draft ?? { - value: r.value, - summary: r.summary, - policy: r.policy, - path: r.path, - custom_path: r.custom_path - } - // custom_path requires admin on app update. Non-admins send undefined so - // the backend preserves the existing route (no RequireAdmin 403). For - // admins, fall back to the *deployed* route (`r.custom_path`) when the - // draft doesn't carry one — the visual-app draft value usually omits - // custom_path, and sending `''` would clear the existing route. An - // explicit '' in the draft still clears (`'' ?? x === ''`). + await deployDraftTriggers( + d.draft_triggers, + workspace, + d.draft_path ?? d.path ?? path, + draftOnly + ) + } else if (kind === 'app') { + // `raw_app` is handled above; only visual apps reach here. + const r = (await AppService.getAppByPath({ workspace, path, getDraft: true })) as any + // A visual-app draft is stored as the *bare* app value (grid/theme/..., + // plus a `draft_path` when the path was renamed) — NOT wrapped in + // { value, summary, policy } like script/flow drafts. So the deploy value + // is the draft object itself; fall back to the deployed value when there's + // no draft. `draft_path` and `summary` are draft-only fields mirrored onto + // the App value (the editor drops them on deploy), so strip them from the + // value and apply them as the deploy path / summary column. + const draft = r.draft as Record | undefined + const { draft_path: draftPath, summary: draftSummary, ...appValue } = draft ?? r.value ?? {} + // Policy isn't carried in the app draft, so it comes from the deployed app + // (or a default). custom_path requires admin on update; non-admins send + // undefined so the backend preserves the existing route. The draft has no + // custom_path, so admins fall back to the deployed route (`''` when none). const isAdmin = !!(get(userStore)?.is_admin || get(userStore)?.is_super_admin) + const policy = r.policy ?? { execution_mode: 'publisher' } const requestBody = { - value: d.value, - summary: d.summary ?? '', - policy: d.policy, - path: d.path ?? path, - custom_path: isAdmin ? (d.custom_path ?? r.custom_path) : undefined + value: appValue, + summary: draftSummary ?? r.summary ?? '', + policy, + // Honor the draft's intended path; `draft_path` holds the user-typed path + // for a never-deployed app parked at a `u/{user}/draft_{uuid}` storage key. + path: draftPath ?? r.path ?? path, + custom_path: isAdmin ? (r.custom_path ?? '') : undefined, + deployment_message: deploymentMessage, + // The draft carries no on-behalf-of selector — the policy comes straight + // from the deployed app. Preserve its on_behalf_of (the backend resets it + // to the deploying user without this flag, gated by can_preserve_on_behalf_of). + preserve_on_behalf_of: policy?.on_behalf_of ? true : undefined } - // Same as flows: a draft always has an app row, so updateApp promotes a - // draft_only app (clearing the flag); createApp would 400 "already exists". - await AppService.updateApp({ workspace, path, requestBody }) + // Same as flows: draft-only apps have no app row → create; + // drafts on a deployed app update it. + if (draftOnly) { + await AppService.createApp({ workspace, requestBody }) + } else { + await AppService.updateApp({ workspace, path, requestBody }) + } + } else if (kind === 'variable') { + const { deployed, draft: d } = splitOverlay(await OVERLAY_GETTERS.variable!(workspace, path)) + // VariableEditor's `VariableState` draft shape: + // { path, variable: { value, is_secret, description }, labels?, wsSpecific } + if (draftOnly) { + await VariableService.createVariable({ + workspace, + requestBody: { + path: d.path ?? path, + value: d.variable?.value ?? '', + is_secret: !!d.variable?.is_secret, + description: d.variable?.description ?? '', + labels: d.labels, + ws_specific: d.wsSpecific + } + }) + } else { + await VariableService.updateVariable({ + workspace, + path, + requestBody: { + path: d.path !== path ? d.path : undefined, + // '' = untouched secret value; sending it would blank the secret. + value: d.variable?.value === '' ? undefined : d.variable?.value, + is_secret: d.variable?.is_secret, + description: d.variable?.description, + labels: d.labels, + ws_specific: d.wsSpecific + } + }) + } + void deployed + } else if (kind === 'resource') { + const { deployed, draft: d } = splitOverlay(await OVERLAY_GETTERS.resource!(workspace, path)) + // ResourceEditor's `ResourceState` draft shape: + // { path, description, args, resource_type?, labels?, wsSpecific } + if (draftOnly) { + await ResourceService.createResource({ + workspace, + requestBody: { + path: d.path ?? path, + value: d.args ?? {}, + description: d.description ?? '', + resource_type: d.resource_type ?? deployed.resource_type, + labels: d.labels, + ws_specific: d.wsSpecific + } + }) + } else { + await ResourceService.updateResource({ + workspace, + path, + requestBody: { + path: d.path ?? path, + value: d.args ?? {}, + description: d.description ?? '', + labels: d.labels, + ws_specific: d.wsSpecific + } + }) + } + } else if (kind === 'trigger_schedule') { + const { draft: d } = splitOverlay(await OVERLAY_GETTERS.trigger_schedule!(workspace, path)) + // The schedule editor's draft IS the cfg shape `saveScheduleFromCfg` + // consumes — same save the editor's Deploy button runs. + const ok = await saveScheduleFromCfg({ ...d, path: d.path ?? path }, !draftOnly, workspace) + if (!ok) { + return { success: false, error: 'Schedule save failed' } + } + } else if (kind in TRIGGER_SAVERS) { + const getter = OVERLAY_GETTERS[kind]! + const { draft: d } = splitOverlay(await getter(workspace, path)) + const isAdmin = !!(get(userStore)?.is_admin || get(userStore)?.is_super_admin) + const ok = await TRIGGER_SAVERS[kind]!( + path, + { ...d, path: d.path ?? path }, + !draftOnly, + workspace, + isAdmin + ) + if (!ok) { + return { success: false, error: 'Trigger save failed' } + } + } else { + return { success: false, error: `Deploy not supported for draft kind ${kind}` } } + // Delete the draft at its STORAGE path (the row key, = the `path` arg). + // Two reasons it must happen here for every kind, mirroring the editors' + // post-deploy `discardDraftAfterDeploy(draftPath)`: + // - Drawer kinds (variable / resource / triggers) aren't deleted by + // their create/update endpoints at all. + // - script/flow/app/raw_app DO delete server-side, but only the draft at + // the *deployed* path (`d.path`). A renamed draft_only item lives at a + // synthetic `u/{user}/draft_{uuid}` storage path ≠ `d.path`, so its + // draft row survives the deploy and keeps listing. Deleting the + // storage-path draft removes it (a no-op when the server already did). + await UserDraftDbSyncer.save({ + workspace, + itemKind: kind, + path, + value: null, + immediate: true + }) // Mutated the workspace's Server Drafts — refresh every mounted reader. invalidateWorkspaceDrafts(workspace) + // For script/flow/app the server-side delete bypasses UserDraftDbSyncer, + // so the syncer-owned hint won't auto-clear — clear it explicitly. + // (Idempotent: the drawer-kind delete above already cleared it.) + setLocalDraftHint(workspace, kind, path, false) return { success: true } } catch (e: any) { return { success: false, error: e?.body ?? e?.message ?? String(e) } } } +/** Kind → editor save helper for the standalone trigger kinds, all sharing the + * `(initialPath, cfg, edit, workspace, isAdmin?)` shape. The throwaway + * `usedTriggerKinds` store only feeds the editors' kind-usage UI. */ +const TRIGGER_SAVERS: Partial< + Record< + DraftKind, + ( + initialPath: string, + cfg: Record, + edit: boolean, + workspace: string, + isAdmin: boolean + ) => Promise + > +> = { + trigger_http: (p, cfg, edit, ws, isAdmin) => + saveHttpRouteFromCfg(p, cfg, edit, ws, isAdmin, writable([])), + trigger_websocket: (p, cfg, edit, ws) => + saveWebsocketTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_postgres: (p, cfg, edit, ws) => + savePostgresTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_kafka: (p, cfg, edit, ws) => + saveKafkaTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_nats: (p, cfg, edit, ws) => + saveNatsTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_mqtt: (p, cfg, edit, ws) => + saveMqttTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_sqs: (p, cfg, edit, ws) => + saveSqsTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_gcp: (p, cfg, edit, ws) => + saveGcpTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_azure: (p, cfg, edit, ws) => + saveAzureTriggerFromCfg(p, cfg, edit, ws, writable([])), + trigger_email: (p, cfg, edit, ws, isAdmin) => + saveEmailTriggerFromCfg(p, cfg, edit, ws, isAdmin, writable([])) +} + /** - * Discard a draft. For `draft_only` items the item exists only as a draft, so - * delete the whole item; otherwise delete just the draft row. + * Discard a draft. Draft-only items exist only as a draft-table row, so + * deleting that row is the whole discard in every case. `save({ value: null })` + * is the canonical "drop my draft for this path" POST. + * + * A legacy draft (workspace-level, `email IS NULL`) isn't owned by the authed + * user, so the email-scoped syncer delete can't reach it. Discard it via a + * direct `legacy` delete instead — then clear the local hint + invalidate as + * the syncer path would. */ export async function discardDraft( kind: DraftKind, path: string, workspace: string, - draftOnly = false + _draftOnly = false, + legacy = false ): Promise { try { - if (draftOnly) { - if (kind === 'script') { - await ScriptService.deleteScriptByPath({ workspace, path }) - } else if (kind === 'flow') { - await FlowService.deleteFlowByPath({ workspace, path }) - } else { - await AppService.deleteApp({ workspace, path }) - } - } else { - await DraftService.deleteDraft({ workspace, path, kind }) + if (legacy) { + await DraftService.updateDraft({ + workspace, + kind, + path, + requestBody: { value: null, legacy: true } + }) + setLocalDraftHint(workspace, kind, path, false) + invalidateWorkspaceDrafts(workspace) + return { success: true } } + // postSave clears the syncer-owned `*` hint on the delete. `immediate` + // so the await resolves after the POST lands — else it resolves at + // enqueue time and the invalidate below refetches before the delete, + // re-listing the just-discarded draft. + await UserDraftDbSyncer.save({ workspace, itemKind: kind, path, value: null, immediate: true }) invalidateWorkspaceDrafts(workspace) return { success: true } } catch (e: any) { diff --git a/frontend/src/lib/workspaceDrafts.svelte.ts b/frontend/src/lib/workspaceDrafts.svelte.ts index 475fbb9894..90a7cf9542 100644 --- a/frontend/src/lib/workspaceDrafts.svelte.ts +++ b/frontend/src/lib/workspaceDrafts.svelte.ts @@ -4,10 +4,9 @@ * simply that list's length — never a separate query. This is what makes the * count reliable: count ≡ list, by construction. * - * Behind this seam the list is currently assembled from the three version-aware - * list endpoints (scripts/flows/apps with `include_draft_only`). A single - * `GET /w/{ws}/drafts/items` endpoint can replace `getDraftItems` later without - * touching any consumer. + * Backed by `GET /w/{ws}/drafts/list` — one query over the `draft` table on + * the server, covering every kind (scripts, flows, apps, variables, resources, + * schedules, triggers) with a per-kind `draft_only` flag. * * Reactivity: `useWorkspaceDrafts(() => ws)` is a component-scoped `runed` * resource — it fetches on mount and when `ws` changes, and is disposed on @@ -16,84 +15,57 @@ * every *mounted* consumer re-fetches after a Server-Draft mutation. */ import { resource } from 'runed' -import { ScriptService, FlowService, AppService } from '$lib/gen' +import { DraftService, type UserDraftItemKind } from '$lib/gen' -export type DraftKind = 'script' | 'flow' | 'app' +export type DraftKind = UserDraftItemKind export interface DraftItem { kind: DraftKind path: string summary?: string + /** User-typed friendly path (from the draft JSON's `draft_path`) when it + * differs from the storage `path` — e.g. a never-deployed item parked at + * `u/{user}/draft_{uuid}`. Display this instead of `path` when present. */ + draft_path?: string /** Never deployed — exists only as a draft. */ draft_only: boolean + /** Legacy workspace-level draft (email NULL) predating the per-user drafts + * migration. Not tied to any user, so anyone with access to the path sees it. */ + legacy_draft: boolean /** App is a raw app (deploys via the raw-app endpoints). Always false for non-apps. */ raw_app: boolean + /** Current user may deploy/discard this draft — matches the server-side check. + * Defaults to true when the field is absent (older backend) so a frontend + * running ahead of the API doesn't disable every action; the deploy/discard + * endpoints enforce permission regardless. */ + can_write: boolean + /** Draft authors at this (path, kind); populated only for the shared + * full-page-editor kinds (script/flow/app/raw_app). Feeds the badge circles. */ + draft_users?: { username?: string | null }[] + /** The row is the current user's own draft (or the legacy no-owner row), so + * they can deploy/discard it. Always true in the default listing; only the + * `allUsers` listing surfaces other users' rows as `false` (view-only). + * Defaults to true when the field is absent (older backend). */ + mine: boolean } -/** The one place the "is this a deployable Draft Item?" rule lives on the - * frontend: a pending draft on a deployed item (`has_draft`) OR a never-deployed - * `draft_only` item. Mirrors the backend `count_drafts` predicate. */ -/** The list-endpoint fields this module reads. Kept as a narrow local interface - * (rather than `any`) so the count predicate isn't typed against `any`. NOTE: - * `openapi.yaml`'s `ListableApp` still omits `has_draft`/`draft_only` (the backend - * struct returns them) — the proper fix is to add them to the spec and regenerate - * the client; until then this interface documents the contract relied on. */ -interface DraftListEntry { - path: string - summary?: string - has_draft?: boolean - draft_only?: boolean - raw_app?: boolean -} - -// The list endpoints are paginated; without paging, drafts past the first page -// would be silently missing from the count/list (and "Deploy all"). Page through -// with a generous page size until a short page signals the end. -const DRAFT_LIST_PER_PAGE = 100 - -async function listAllPages( - fetchPage: (page: number, perPage: number) => Promise -): Promise { - const all: DraftListEntry[] = [] - for (let page = 1; ; page++) { - const batch = await fetchPage(page, DRAFT_LIST_PER_PAGE) - all.push(...batch) - if (batch.length < DRAFT_LIST_PER_PAGE) break - } - return all -} - -export async function getDraftItems(workspace: string): Promise { - const [scripts, flows, apps] = await Promise.all([ - listAllPages((page, perPage) => - ScriptService.listScripts({ workspace, includeDraftOnly: true, page, perPage }) - ), - listAllPages((page, perPage) => - FlowService.listFlows({ workspace, includeDraftOnly: true, page, perPage }) - ), - listAllPages((page, perPage) => - AppService.listApps({ workspace, includeDraftOnly: true, page, perPage }) - ) - ]) - const items: DraftItem[] = [] - const push = (kind: DraftKind, list: DraftListEntry[]) => { - for (const it of list) { - if (it.has_draft || it.draft_only) { - items.push({ - kind, - path: it.path, - summary: it.summary, - draft_only: !!it.draft_only, - raw_app: !!it.raw_app - }) - } - } - } - push('script', scripts) - push('flow', flows) - push('app', apps) - items.sort((a, b) => a.path.localeCompare(b.path)) - return items +export async function getDraftItems( + workspace: string, + allUsers: boolean = false +): Promise { + const rows = await DraftService.listDrafts({ workspace, allUsers: allUsers || undefined }) + return rows.map((r) => ({ + kind: r.kind, + path: r.path, + summary: r.summary, + draft_path: r.draft_path, + draft_only: r.draft_only, + legacy_draft: r.legacy_draft, + raw_app: r.kind === 'raw_app', + can_write: r.can_write ?? true, + draft_users: r.draft_users, + mine: r.mine ?? true + })) } // Per-workspace invalidation version. Bumping it changes the resource key for @@ -118,13 +90,16 @@ export interface WorkspaceDraftsHandle { * Re-fetches on mount, when `workspace` changes, and when * `invalidateWorkspaceDrafts(workspace)` is called while mounted. */ -export function useWorkspaceDrafts(workspace: () => string | undefined): WorkspaceDraftsHandle { +export function useWorkspaceDrafts( + workspace: () => string | undefined, + allUsers: () => boolean = () => false +): WorkspaceDraftsHandle { const res = resource( () => { const ws = workspace() - return { ws, v: ws ? (versions[ws] ?? 0) : 0 } + return { ws, all: allUsers(), v: ws ? (versions[ws] ?? 0) : 0 } }, - async ({ ws }) => (ws ? getDraftItems(ws) : []) + async ({ ws, all }) => (ws ? getDraftItems(ws, all) : []) ) return { get items() { diff --git a/frontend/src/routes/(root)/(logged)/+layout.svelte b/frontend/src/routes/(root)/(logged)/+layout.svelte index e6b1fd1626..e91ad87583 100644 --- a/frontend/src/routes/(root)/(logged)/+layout.svelte +++ b/frontend/src/routes/(root)/(logged)/+layout.svelte @@ -58,8 +58,9 @@ import GlobalSearchModal from '$lib/components/search/GlobalSearchModal.svelte' import MenuButton from '$lib/components/sidebar/MenuButton.svelte' import { loadProtectionRules } from '$lib/workspaceProtectionRules.svelte' - import { migrateLegacyUserDrafts } from '$lib/userDraftLegacyMigration' - import { gcUserDrafts } from '$lib/userDraft.svelte' + import { purgeLegacyUserDrafts } from '$lib/userDraftLegacyMigration' + import { migrateUserDraftsToDb } from '$lib/userDraftDbMigration' + import DraftMigrationErrorModal from '$lib/components/DraftMigrationErrorModal.svelte' import { setContext, untrack } from 'svelte' import { base } from '$app/paths' import { Menubar } from '$lib/components/meltComponents' @@ -136,6 +137,16 @@ if (!deepEqual(user, $userStore)) { userStore.set(user) } + // Persist the workspace username so the synchronous `/add` → + // `/edit/u/{user}/draft_{uuid}` redirects (in each editor's + // `+page.ts`) can land on the right namespace without waiting + // for this async layout fetch. Without this they fall back to + // the `'me'` placeholder on every fresh nav. + try { + if (user?.username) localStorage.setItem('username', user.username) + } catch (e) { + console.error('Could not persist username to local storage', e) + } if (isCloudHosted() && user?.is_admin) { isPremiumStore.set(await WorkspaceService.getIsPremium({ workspace })) } @@ -153,8 +164,9 @@ const toPath = navigation.to?.url.pathname if (toPath && (toPath.startsWith('/apps_raw/add') || toPath.startsWith('/apps_raw/edit'))) { const currentPath = navigation.from?.url.pathname - // Reload if we're not on an apps_raw path, or if we're on /apps/get_raw/ (viewing a raw app) - // The /apps/get_raw/ path doesn't have cross-origin isolation headers, so we need to reload + // Reload if we're not on an apps_raw path, or if we're on the raw app viewer + // (/apps_raw/get/): the viewer doesn't have cross-origin isolation headers, so + // we need a full reload to fetch them for the editor. if (!currentPath?.startsWith('/apps_raw/') || currentPath?.startsWith('/apps_raw/get/')) { navigation.cancel() window.location.href = navigation.to!.url.href @@ -424,18 +436,20 @@ $effect(() => { $workspaceStore && untrack(() => onLoad()) }) + // One-shot UserDraft migration. `purgeLegacyUserDrafts` drops the oldest + // workspace-blind `flow` / `app-…` / `rawapp-…` LS autosave keys (they + // can't be attributed to a workspace, so promoting them would mis-file + // drafts). `migrateUserDraftsToDb` then pushes the workspace-scoped + // `userdraft/w/{ws}/{kind}/{path}` keys — written by the editor with the + // correct workspace — onto the server-side draft table, clearing LS on + // success. $effect(() => { - if ($workspaceStore) untrack(() => migrateLegacyUserDrafts($workspaceStore!)) - }) - // Sweep UserDraft entries that haven't been touched in 30 days. Runs - // once on mount and on a 30-min timer so a single very long session - // also clears out stale autosaves over time. Live entries stamp - // `lastWrittenAt` on every persist, so the sweep only touches truly - // dormant records. - $effect(() => { - gcUserDrafts() - const interval = setInterval(() => gcUserDrafts(), 30 * 60 * 1000) - return () => clearInterval(interval) + if ($workspaceStore && $userStore) { + untrack(() => { + purgeLegacyUserDrafts() + void migrateUserDraftsToDb() + }) + } }) $effect(() => { innerWidth && untrack(() => changeCollapsed()) @@ -490,6 +504,7 @@ + {#if page.status == 404} {:else if $userStore} diff --git a/frontend/src/routes/(root)/(logged)/+page.svelte b/frontend/src/routes/(root)/(logged)/+page.svelte index b070bd5fef..e326bc87f7 100644 --- a/frontend/src/routes/(root)/(logged)/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/+page.svelte @@ -1,13 +1,11 @@ - -{#if value} -
- {#key value} - { - goto(`/apps/edit/${path}`) - }} - {summary} - app={value} - path={''} - {policy} - fromHub={hubId != null} - newApp={true} - replaceStateFn={(path) => replaceState(path, page.state)} - gotoFn={(path, opt) => goto(path, opt)} - > - {#snippet unsavedConfirmationModal({ - diffDrawer, - additionalExitAction, - getInitialAndModifiedValues - })} - - {/snippet} - - {/key} -
-{/if} + diff --git a/frontend/src/routes/(root)/(logged)/apps/add/+page.ts b/frontend/src/routes/(root)/(logged)/apps/add/+page.ts new file mode 100644 index 0000000000..52ecb242b7 --- /dev/null +++ b/frontend/src/routes/(root)/(logged)/apps/add/+page.ts @@ -0,0 +1,6 @@ +import { makeDraftAddLoad } from '$lib/draftAddRedirect' +import type { PageLoad } from './$types' + +export const prerender = false + +export const load: PageLoad = makeDraftAddLoad('apps/edit') diff --git a/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte index e2e1029d0e..7eb5d30106 100644 --- a/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/edit/[...path]/+page.svelte @@ -1,227 +1,356 @@ - - + + + UserDraftDbSyncer.hasUnsavedDisabledChanges({ + workspace: $workspaceStore ?? '', + itemKind: 'app', + path + })} + onDiscardChanges={() => + UserDraftDbSyncer.dropPending({ + workspace: $workspaceStore ?? '', + itemKind: 'app', + path + })} +/> + { + // AppEditor's `stateApp` is captured at mount and ignores prop changes, + // so `redraw++` remounts it against the fresh `app`. + await loadApp() + redraw++ + }} + getLocalDraft={() => app?.value} + bind:othersModalOpen + {draftSavedAt} + {deployedAt} + {draftBaseVersion} + {deployedHeadVersion} + onLoadLatestDeploy={async () => { + if (!$workspaceStore) return + await runResetToDeployed({ + workspace: $workspaceStore, + itemKind: 'app', + path, + onResetToDeployed: reloadDeployed + }) + }} /> {#key redraw} {#if app}
{ goto(`/apps/edit/${url}`) if (app) { app.path = url } }} - on:restore={onRestore} + {onRestore} summary={app.summary} app={app.value} - newPath={app.path} + labels={app.labels} + {deployedBaseline} + newPath={app.value?.draft_path ?? app.path} path={page.params.path ?? ''} policy={app.policy} bind:savedApp {diffDrawer} version={app.versions ? app.versions[app.versions.length - 1] : undefined} - newApp={false} - initialRevs={currentRevs} + newApp={isNewApp} replaceStateFn={(path) => replaceState(path, page.state)} gotoFn={(path, opt) => goto(path, opt)} - > - {#snippet unsavedConfirmationModal({ - diffDrawer, - additionalExitAction, - getInitialAndModifiedValues - })} - - {/snippet} - + onResetToDeployed={reloadDeployed} + {loadedFromDraft} + othersDraftsCount={otherDraftsUsers.length} + onOpenOthersDrafts={() => (othersModalOpen = true)} + />
{/if} {/key} diff --git a/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte index 737ebb2e33..0af3cdeded 100644 --- a/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/get/[...path]/+page.svelte @@ -1,91 +1,30 @@ -{#if app} - {#key app} -
- { - goto(path) - }} - gotoFn={(path, opt) => { - goto(path, opt) - }} - /> - {#if can_write && !hideEditBtn} -
- -
- {/if} -
+ +{#if workspace && path} + + {#key `${workspace}/${path}`} + {/key} {:else} diff --git a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js deleted file mode 100644 index 5b680f2fd0..0000000000 --- a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `App ${params.path}` } - } -} diff --git a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte index db7b7da001..dbba9ce790 100644 --- a/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps/get_raw/[version]/[...path]/+page.svelte @@ -1,41 +1,17 @@ - -
- -{#if !loaded} - -{/if} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.js b/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.js deleted file mode 100644 index 1f2d07eb57..0000000000 --- a/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load() { - return { - stuff: { title: `New Raw App` } - } -} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.svelte index 716000215f..e54568a9ed 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.svelte @@ -1,688 +1,5 @@ - - -{#if templatePicker} - -
- -
-

Summary

- -
- - -
-

Framework

-
- {#each templates as t, i} - - {/each} -
-
- - -
-

Data configuration

- - {#if hasNoDatatables} - - You can still create an app, but for data storage you won't be able to use data tables - which are highly recommended. -
- - {#if $userStore?.is_admin} - Configure datatables in - workspace settings - to enable this feature. - {:else} - Ask your workspace admin to configure datatables in workspace settings to enable this - feature. - {/if} -
- {:else} -
- -
- Default settings for new tables -
-
-
- - -
- {/if} -
- {#if newSchemaAlreadyExists} - Schema "{newSchemaName}" already exists - {/if} -
-
-
-
- - -
- -
- - -
- dataTableDrawer?.openDrawer()} - onRemove={(index) => { - preWhitelistedTables = preWhitelistedTables.filter((_, i) => i !== index) - }} - /> -
-
- {/if} -
- - -
-

- - Start with AI - (optional) -

- - {#if !isAiEnabled} - - You can still create an app manually but using AI is highly recommended. -
- {#if $userStore?.is_admin} - Configure AI in - workspace settings - - to enable this feature. - {:else} - Ask your workspace admin to configure AI in workspace settings to enable this feature. - {/if} -
- {:else} -
- -

- Leave empty to start with a blank template, or describe your app to get AI assistance - right away. -

-
- {/if} -
- - -
- - {#if isAiEnabled} - - {/if} -
-
- -{/if} -{#key reloadCounter} - { - goto(`/apps_raw/edit/${event.detail}`) - }} - bind:files - bind:runnables - bind:data - {policy} - path={''} - liveEditorDraftStoragePath="" - bind:summary - newApp - /> -{/key} - - { - preWhitelistedTables = [...preWhitelistedTables, ref] - }} -/> + diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.ts b/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.ts new file mode 100644 index 0000000000..ea96e67846 --- /dev/null +++ b/frontend/src/routes/(root)/(logged)/apps_raw/add/+page.ts @@ -0,0 +1,6 @@ +import { makeDraftAddLoad } from '$lib/draftAddRedirect' +import type { PageLoad } from './$types' + +export const prerender = false + +export const load: PageLoad = makeDraftAddLoad('apps_raw/edit') diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte index 84ce23e7a5..6227776ec4 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/edit/[...path]/+page.svelte @@ -1,15 +1,11 @@ - - + + + UserDraftDbSyncer.hasUnsavedDisabledChanges({ + workspace: $workspaceStore ?? '', + itemKind: 'raw_app', + path + })} + onDiscardChanges={() => + UserDraftDbSyncer.dropPending({ + workspace: $workspaceStore ?? '', + itemKind: 'raw_app', + path + })} /> + loadApp()} + getLocalDraft={() => draftSync.draft} + bind:othersModalOpen + {draftSavedAt} + {deployedAt} + onLoadLatestDeploy={async () => { + // stopSync-bracketed; see /scripts/edit's restoreDeployed for the race. + if (!$workspaceStore) return + await runResetToDeployed({ + workspace: $workspaceStore, + itemKind: 'raw_app', + path, + onResetToDeployed: reloadDeployed + }) + }} +/> + + {#if files} {#key redraw}
{ - UserDraft.remove('raw_app', path) - goto(`/apps_raw/edit/${event.detail}`) - newPath = event.detail + onSavedNewAppPath={(savedPath) => { + draftSync.remove() + goto(`/apps_raw/edit/${savedPath}`) + newPath = savedPath }} - on:restore={onRestore} + {onRestore} bind:files bind:runnables bind:data bind:summary + bind:pendingDraftPath {newPath} + {labels} path={page.params.path ?? ''} liveEditorDraftStoragePath={path} {policy} bind:savedApp {diffDrawer} - newApp={false} + newApp={isNewApp} + onResetToDeployed={reloadDeployed} + {loadedFromDraft} + othersDraftsCount={otherDraftsUsers.length} + onOpenOthersDrafts={() => (othersModalOpen = true)} />
{/key} diff --git a/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte index eefa19fcd3..990e74ed8c 100644 --- a/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/apps_raw/get/[...path]/+page.svelte @@ -1,65 +1,26 @@ -
- {#if !$workspaceStore || !$userStore || !app} - - {:else} - - {/if} - {#if can_write && !hideEditBtn} -
- -
- {/if} -
+{#if workspace && path} + + {#key `${workspace}/${path}`} + + {/key} +{:else} + +{/if} diff --git a/frontend/src/routes/(root)/(logged)/azure_triggers/+page.svelte b/frontend/src/routes/(root)/(logged)/azure_triggers/+page.svelte index a8d0b566b8..2d03b85398 100644 --- a/frontend/src/routes/(root)/(logged)/azure_triggers/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/azure_triggers/+page.svelte @@ -1,4 +1,5 @@ - - - - { - UserDraft.remove('flow', '') - if ($workspaceStore) invalidate($workspaceStore, 'flow') - goto(`/flows/edit/${e.path}?selected=${e.id}`) - }} - onDeploy={(e) => { - UserDraft.remove('flow', '') - if ($workspaceStore) invalidate($workspaceStore, 'flow') - goto(`/flows/get/${e.path}?workspace=${$workspaceStore}`) - }} - onDetails={(e) => { - goto(`/flows/get/${e.path}?workspace=${$workspaceStore}`) - }} - onNavigate={(item) => goto(editPathFor(item))} - {initialPath} - {pathStoreInit} - liveEditorDraftStoragePath="" - bind:this={flowBuilder} - newFlow - {initialArgs} - {flowStore} - {flowStateStore} - {selectedId} - {loading} - {draftTriggersFromUrl} - {selectedTriggerIndexFromUrl} - noInitial -> - - + diff --git a/frontend/src/routes/(root)/(logged)/flows/add/+page.ts b/frontend/src/routes/(root)/(logged)/flows/add/+page.ts new file mode 100644 index 0000000000..153c2255df --- /dev/null +++ b/frontend/src/routes/(root)/(logged)/flows/add/+page.ts @@ -0,0 +1,6 @@ +import { makeDraftAddLoad } from '$lib/draftAddRedirect' +import type { PageLoad } from './$types' + +export const prerender = false + +export const load: PageLoad = makeDraftAddLoad('flows/edit') diff --git a/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte index b979a09758..e73af57ee2 100644 --- a/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/flows/edit/[...path]/+page.svelte @@ -1,35 +1,34 @@ - - + + + UserDraftDbSyncer.hasUnsavedDisabledChanges({ + workspace: $workspaceStore ?? '', + itemKind: 'flow', + path: flowDraftPath + })} + onDiscardChanges={() => + UserDraftDbSyncer.dropPending({ + workspace: $workspaceStore ?? '', + itemKind: 'flow', + path: flowDraftPath + })} +/> + loadFlow()} + getLocalDraft={() => draftSync.draft} + bind:othersModalOpen + {draftSavedAt} + {deployedAt} + {draftBaseVersion} + deployedHeadVersion={version} + onLoadLatestDeploy={async () => { + // stopSync-bracketed; see /scripts/edit's restoreDeployed for the race. + if (!$workspaceStore) return + await runResetToDeployed({ + workspace: $workspaceStore, + itemKind: 'flow', + path: flowDraftPath, + onResetToDeployed: async () => { + draftSync.draft = undefined + await loadFlow({ getDraft: false }) + } + }) + }} /> {#if notFound}
@@ -404,25 +536,36 @@ {:else if renderEditor} { - UserDraft.remove('flow', flowDraftPath) + // stopSync-bracketed immediate delete; see /scripts/edit's restoreDeployed. + if ($workspaceStore) { + discardDraftAfterDeploy({ + workspace: $workspaceStore, + itemKind: 'flow', + path: flowDraftPath + }) + } if ($workspaceStore) invalidate($workspaceStore, 'flow') goto(`/flows/get/${e.path}?workspace=${$workspaceStore}`) }} onDetails={(e) => { goto(`/flows/get/${e.path}?workspace=${$workspaceStore}`) }} - onSaveDraftOnlyAtNewPath={(e) => { - goto(`/flows/edit/${e.path}?selected=${e.selectedId}`) - }} onHistoryRestore={() => { loadFlow() }} + onResetToDeployed={async () => { + draftSync.draft = undefined + await loadFlow({ getDraft: false }) + }} + {loadedFromDraft} + othersDraftsCount={otherDraftsUsers.length} + onOpenOthersDrafts={() => (othersModalOpen = true)} onNavigate={(item) => goto(editPathFor(item))} {flowStore} {flowStateStore} - initialPath={page.params.path ?? ''} + bind:initialPath={flowInitialPath} liveEditorDraftStoragePath={flowDraftPath} - newFlow={false} + newFlow={isNewFlow} {selectedId} {initialArgs} {loading} @@ -433,11 +576,7 @@ {draftTriggersFromUrl} {selectedTriggerIndexFromUrl} {version} + {draftBaseVersion} {loadedFromHistoryFromUrl} - > - - + /> {/if} diff --git a/frontend/src/routes/(root)/(logged)/flows/get/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/flows/get/[...path]/+page.svelte index 088563b640..8741a8b93b 100644 --- a/frontend/src/routes/(root)/(logged)/flows/get/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/flows/get/[...path]/+page.svelte @@ -329,7 +329,7 @@ onClick: async () => { const app = createAppFromFlow(flow.path, flow.schema) $importStore = JSON.parse(JSON.stringify(app)) - await goto('/apps/add?nodraft=true') + await goto('/apps/add') }, unifiedSize: 'md', variant: 'subtle', @@ -341,7 +341,7 @@ buttons.push({ label: 'Edit', buttonProps: { - href: `${base}/flows/edit/${path}?nodraft=true`, + href: `${base}/flows/edit/${path}`, variant: 'accent', unifiedSize: 'md', disabled: !can_write || !showEditButtons, diff --git a/frontend/src/routes/(root)/(logged)/folders/+page.svelte b/frontend/src/routes/(root)/(logged)/folders/+page.svelte index 1e488d3d13..0a55acfe69 100644 --- a/frontend/src/routes/(root)/(logged)/folders/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/folders/+page.svelte @@ -17,6 +17,7 @@ import { Pen, Trash, Plus } from 'lucide-svelte' import Head from '$lib/components/table/Head.svelte' import Row from '$lib/components/table/Row.svelte' + import Badge from '$lib/components/common/badge/Badge.svelte' import { untrack } from 'svelte' type FolderW = Folder & { canWrite: boolean } @@ -135,6 +136,7 @@ Name + Labels Scripts Flows Apps @@ -149,7 +151,7 @@ {#if folders === undefined} {#each new Array(4) as _} - + @@ -157,7 +159,7 @@ {:else} {#if folders.length === 0} - +
No folders yet, create one
@@ -165,7 +167,7 @@ {/if} - {#each folders as { name, extra_perms, owners, canWrite, summary } (name)} + {#each folders as { name, extra_perms, owners, canWrite, summary, labels } (name)} { @@ -180,6 +182,27 @@ {summary} {/if}
+ + {#if labels?.length} +
+ {#each labels.slice(0, 3) as label} + {label} + {/each} + {#if labels.length > 3} + 'Label: ' + l) + .join('\n')}>+{labels.length - 3} + {/if} +
+ {/if} +
diff --git a/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte b/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte index 2710167325..624db4061e 100644 --- a/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/forks/compare/+page.svelte @@ -2,9 +2,14 @@ import CompareWorkspaces from '$lib/components/CompareWorkspaces.svelte' import CompareDrafts from '$lib/components/CompareDrafts.svelte' import { WorkspaceService, type WorkspaceComparison } from '$lib/gen' + import { + archiveSessionsForWorkspace, + deleteSessionsForWorkspace, + reconcileAfterWorkspaceChange + } from '$lib/components/sessions/sessionState.svelte' import { useWorkspaceDrafts } from '$lib/workspaceDrafts.svelte' import { page } from '$app/state' - import { userWorkspaces, usersWorkspaceStore, workspaceStore } from '$lib/stores' + import { userWorkspaces, workspaceStore } from '$lib/stores' import { onDestroy, untrack } from 'svelte' import CenteredPage from '$lib/components/CenteredPage.svelte' import PageHeader from '$lib/components/PageHeader.svelte' @@ -143,14 +148,9 @@ let acting = $state(false) async function afterForkGone() { - // Mirror SidebarContent.deleteFork (B1): refresh the workspace list - // rather than letting `clearStores()` null it, then land the user on - // the parent if still accessible. - try { - usersWorkspaceStore.set(await WorkspaceService.listUserWorkspaces()) - } catch (e) { - console.error('Failed to refresh workspaces', e) - } + // The workspace list was already refreshed by reconcileAfterWorkspaceChange + // (so the just-removed fork is gone from it); land the user on the parent if + // it's still accessible. if (parentWorkspaceId && $userWorkspaces.find((w) => w.id === parentWorkspaceId)) { switchWorkspace(parentWorkspaceId) await goto('/') @@ -166,6 +166,15 @@ try { await WorkspaceService.archiveWorkspace({ workspace: currentWorkspaceId }) sendUserToast(`Archived fork ${currentWorkspaceId}`) + // Client session cleanup is best-effort: a local IndexedDB failure must + // not falsely report the (already successful) archive as failed, nor + // block navigation away from the now-archived fork. + try { + await archiveSessionsForWorkspace(currentWorkspaceId) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after fork archive failed', e) + } await afterForkGone() } catch (e: any) { sendUserToast(`Failed to archive fork: ${e?.body ?? e}`, true) @@ -181,6 +190,15 @@ try { await WorkspaceService.deleteWorkspace({ workspace: currentWorkspaceId }) sendUserToast(`Deleted fork ${currentWorkspaceId}`) + // Client session cleanup is best-effort: a local IndexedDB failure must + // not abort the redirect after a successful delete, leaving the user on + // the now-deleted workspace path. + try { + await deleteSessionsForWorkspace(currentWorkspaceId) + await reconcileAfterWorkspaceChange() + } catch (e) { + console.error('Session cleanup after fork delete failed', e) + } await afterForkGone() } catch (e: any) { sendUserToast(`Failed to delete fork: ${e?.body ?? e}`, true) diff --git a/frontend/src/routes/(root)/(logged)/gcp_triggers/+page.svelte b/frontend/src/routes/(root)/(logged)/gcp_triggers/+page.svelte index bb65a27065..6e59e7be5e 100644 --- a/frontend/src/routes/(root)/(logged)/gcp_triggers/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/gcp_triggers/+page.svelte @@ -1,4 +1,5 @@ + + + Pipeline editor — Windmill + + +
+
+
+ +

+ {$userStore?.operator ? 'Pipelines' : 'Pipeline editor'} +

+ · no folder selected +
+
+ +
+
+ + Pick a folder to open its pipeline. + +
+
+
+ + + diff --git a/frontend/src/routes/(root)/(logged)/scripts/add/+page.js b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.js similarity index 51% rename from frontend/src/routes/(root)/(logged)/scripts/add/+page.js rename to frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.js index 695ebc3035..f92a72894f 100644 --- a/frontend/src/routes/(root)/(logged)/scripts/add/+page.js +++ b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.js @@ -1,5 +1,5 @@ export function load() { return { - stuff: { title: `New Script` } + stuff: { title: 'Pipeline editor' } } } diff --git a/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte new file mode 100644 index 0000000000..3056245e51 --- /dev/null +++ b/frontend/src/routes/(root)/(logged)/pipeline/[folder]/+page.svelte @@ -0,0 +1,2815 @@ + + + + Pipeline · {folder} — Windmill + + +
+
+
+ + {:else} + + {#snippet buttonReplacement()} + + + f/{folder} + + + {/snippet} + + {/if} + {#if summary.length > 0} + · {summary.join(' · ')} + {/if} +
+ {#if !isOperator} + +
+ setMode(m)} /> + {#if mode === 'view' && drafts.size > 0} + + + {/if} +
+ {/if} +
+ {#if mode === 'edit' && saveErrors.size > 0} + + + {#snippet trigger()} + + {/snippet} + {#snippet content()} +
+ Save errors +
+ {#each [...saveErrors.entries()] as [path, message]} +
+ {path} + + {message} + +
+ {/each} +
+
+ {/snippet} +
+ {/if} + {#if mode === 'edit' && drafts.size > 0} + + {#if $workspaceStore} + + {/if} + + {/if} + {#if mode === 'view'} + + {/if} +
+
+ +
+ {#if graphRes.loading && !graphRes.current} +
+ + Loading pipeline… +
+ {:else if graphRes.error} +
+ Failed to load pipeline: {graphRes.error.message} +
+ {:else} + + +
+ + {#if boundPick} + +
+ +
+ + {boundPickEnds.size === 0 + ? 'Click end node(s) to bound the run' + : `${boundScripts.length} script${boundScripts.length === 1 ? '' : 's'} up to ${boundPickEnds.size} end${boundPickEnds.size === 1 ? '' : 's'}`} + + + from {boundPickStart ? shortPath(boundPickStart) : ''} + +
+ + +
+ {/if} + {#if mode === 'edit'} + + + {/if} + {#if prefetchingAssets} +
+ + Parsing assets… +
+ {/if} + {#if mode !== 'edit' || selection != undefined || activeDraftPath != undefined} + +
+
+ {/if} +
+ {#if detailsPaneOpen && $workspaceStore} + + {#if mode !== 'edit' && selection == undefined && activeDraftPath == undefined} + + (activityHoverPaths = p ?? [])} + onSelectRun={(p) => (activitySelectPaths = p ?? [])} + /> + {:else} + setMode('edit')} + canRunByPath={openScriptHasDataUpload} + onRunByPath={runByPathLegit} + selection={activeDraft ? undefined : selection} + selectionProducers={activeDraft ? [] : selectionProducers} + selectionColumnGraph={activeDraft ? EMPTY_COLUMN_GRAPH : columnGraph} + {schemaCanEvolve} + {runsRefreshKey} + {runsPendingJobId} + {activeRunnable} + downstreamSubscribers={editedScriptDownstreamCount} + onStartBoundedRun={openScriptPath && + validStartPaths.has(openScriptPath) && + lineageDownstreamPaths.has(openScriptPath) + ? () => startBoundedRun(openScriptPath!) + : undefined} + onRunCompleted={() => { + activeRunnable = undefined + activeRunnableJobId = undefined + }} + onTestStateChange={(running) => { + // Bridge: ScriptEditor's Test button triggers the + // same canvas-level "is running" hint as the + // per-node Run button. The currently-edited script + // is whichever path is open in the pane (active + // draft, or the persisted-script selection). + const openPath = openScriptPath + if (running && openPath) { + activeRunnable = { kind: 'script', path: openPath } + // Mark the tested runnable as launched-from-here so the + // folder poll's catch-up pulse won't re-flash its edge a + // poll-interval after a fast job already finished (the + // edge is animated zero-latency by `activeRunnable`, and + // the test loader clears that the instant it completes). + // Also upgrades to the fast poll so the badge lands sooner. + activeRunnables.arm(`script:${openPath}`) + // Editor Test path clears via its own callbacks, not + // the job-id effect — drop any stale tracked id so a + // prior canvas run's completion can't clear this hint. + activeRunnableJobId = undefined + } else if (!running && activeRunnable?.path === openPath) { + activeRunnable = undefined + activeRunnableJobId = undefined + } + }} + {requestRemoveSignal} + {requestRunSignal} + {requestRunCascadeSignal} + focusUploadSignal={focusDataUploadSignal} + draftScript={activeDraft?.script} + {pathPrefix} + onDraftPathChange={renameDraft} + workspace={$workspaceStore} + onAnnotationsChange={handleAnnotationsChange} + onAssetsChange={handleAssetsChange} + onContentChange={handleContentChange} + onDraftPersist={handleDraftPersist} + onclose={() => { + // Close dismisses the pane but preserves drafts so + // the user can come back to them. Discarding is + // via the explicit "Discard" button in the pane. + selection = undefined + activeDraftPath = undefined + liveAnnotations = EMPTY_LIVE_ANNOTATIONS + }} + onHide={() => (panelHidden = true)} + onDiscard={() => { + if (activeDraftPath) discardDraft(activeDraftPath) + }} + onDraftSaved={async (savedPath) => { + // Snapshot the preview's promise while the draft + // overlay still exists (dropped from `drafts` below). + const predicted = predictCascadeFacts([savedPath]) + // Drop the now-deployed draft and hand focus to its + // persisted runnable so the pane stays open on the + // same script. `discardDraft` would clear + // activeDraftPath without setting selection — the + // canvas would deselect and the view reset on the + // next refetch. + const nextDrafts = new Map(drafts) + nextDrafts.delete(savedPath) + drafts = nextDrafts + if (activeDraftPath === savedPath) { + selection = { + kind: 'runnable', + runnable_kind: 'script', + path: savedPath + } + activeDraftPath = undefined + } + clearSaveError(savedPath) + await graphRes.refetch() + reportDeployDrift(predicted) + }} + onPersistedSaved={async (savedPath) => { + // Snapshot before the refetch replaces the base graph + // — the live editor overlay is the prediction here. + const predicted = predictCascadeFacts([savedPath]) + // Refresh the asset graph so the rows the deploy + // just inserted (from the body-asset write list we + // pass at save time) make it into base.edges. The + // in-memory `inferredWritesByPath` overlay + // dedupes against base, so the edge stays put + // instead of flickering when the ScriptEditor + // remounts on the new hash. + await graphRes.refetch() + reportDeployDrift(predicted) + }} + onScriptRenamed={async (oldPath, newPath) => { + // Repoint the selection at the new path before the + // graph refetches so the pane stays focused on the + // same script. Order matters: update selection + // first, then refetch — otherwise the resource + // driving the pane would briefly resolve to nothing. + if (selection?.kind === 'runnable' && selection.path === oldPath) { + selection = { ...selection, path: newPath } + } + await graphRes.refetch() + }} + onScriptRemoved={async (removedPath) => { + // Drop every path-keyed overlay / cache entry + // pointing at the now-archived runnable so + // resolveGraph doesn't keep emitting lineage + // edges or missing-trigger placeholders against + // a script that no longer exists. Without this + // the inferred writes / annotation maps would + // keep dragging phantom nodes onto the canvas + // until the next folder change. + forgetPath(removedPath) + await graphRes.refetch() + }} + /> + {/if} + + {/if} +
+ {/if} +
+
+ + + + + graphRes.refetch()} +/> + +{#if leaveModalOpen} + + +{/if} diff --git a/frontend/src/routes/(root)/(logged)/postgres_triggers/+page.svelte b/frontend/src/routes/(root)/(logged)/postgres_triggers/+page.svelte index 31e3c57b6b..a98ed34bac 100644 --- a/frontend/src/routes/(root)/(logged)/postgres_triggers/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/postgres_triggers/+page.svelte @@ -1,4 +1,5 @@ - - - - -{#if scriptHandle.draft} - { - // "Deploy & Stay here" / lib: stay on the editor (just confirm). - if (e.stay) { - sendUserToast('Deployed') - return - } - goto(`/scripts/get/${e.hash}?workspace=${$workspaceStore}`) - }} - onSaveInitial={(e) => { - goto(`/scripts/edit/${e.path}`) - }} - onNavigate={(item) => goto(editPathFor(item))} - searchParams={page.url.searchParams} - bind:script={scriptHandle.draft} - {showMeta} - > - - -{:else} - -{/if} + diff --git a/frontend/src/routes/(root)/(logged)/scripts/add/+page.ts b/frontend/src/routes/(root)/(logged)/scripts/add/+page.ts new file mode 100644 index 0000000000..05237ef0db --- /dev/null +++ b/frontend/src/routes/(root)/(logged)/scripts/add/+page.ts @@ -0,0 +1,6 @@ +import { makeDraftAddLoad } from '$lib/draftAddRedirect' +import type { PageLoad } from './$types' + +export const prerender = false + +export const load: PageLoad = makeDraftAddLoad('scripts/edit') diff --git a/frontend/src/routes/(root)/(logged)/scripts/edit/[...path]/+page.svelte b/frontend/src/routes/(root)/(logged)/scripts/edit/[...path]/+page.svelte index b76c03a804..2fb31f5e41 100644 --- a/frontend/src/routes/(root)/(logged)/scripts/edit/[...path]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/scripts/edit/[...path]/+page.svelte @@ -1,95 +1,44 @@ - - + + + UserDraftDbSyncer.hasUnsavedDisabledChanges({ + workspace: $workspaceStore ?? '', + itemKind: 'script', + path: draftPath + })} + onDiscardChanges={() => + UserDraftDbSyncer.dropPending({ + workspace: $workspaceStore ?? '', + itemKind: 'script', + path: draftPath + })} /> - - loadScript()} + getLocalDraft={() => draftSync.draft} + bind:othersModalOpen + {draftSavedAt} + {deployedAt} + onLoadLatestDeploy={async () => { + // stopSync-bracketed; see restoreDeployed for the race. + if (!$workspaceStore) return + await runResetToDeployed({ + workspace: $workspaceStore, + itemKind: 'script', + path: draftPath, + onResetToDeployed: async () => { + draftSync.draft = undefined + await loadScript({ getDraft: false }) + } + }) + }} /> -{#if scriptHandle.draft && renderEditor} +{#if draftSync.draft && renderEditor} (othersModalOpen = true)} + onResetToDeployed={async () => { + draftSync.draft = undefined + await loadScript({ getDraft: false }) + }} onDeploy={(e) => { // "Deploy & Stay here" / lib: stay on the editor (just confirm). if (e.stay) { sendUserToast('Deployed') return } - UserDraft.remove('script', draftPath) + // stopSync-bracketed immediate delete; see restoreDeployed for the race. + if ($workspaceStore) { + discardDraftAfterDeploy({ + workspace: $workspaceStore, + itemKind: 'script', + path: draftPath + }) + } if ($workspaceStore) invalidate($workspaceStore, 'script') goto(`/scripts/get/${e.hash}?workspace=${$workspaceStore}`) }} - onSaveInitial={(e) => { - goto(`/scripts/edit/${e.path}`) - }} onSeeDetails={(e) => { goto(`/scripts/get/${e.path}?workspace=${$workspaceStore}`) }} onNavigate={(item) => goto(editPathFor(item))} - > - - + /> {/if} diff --git a/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte b/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte index df37a946f9..24bf997c50 100644 --- a/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/scripts/get/[...hash]/+page.svelte @@ -422,7 +422,7 @@ onClick: async () => { const app = createAppFromScript(script.path, script.schema) $importStore = JSON.parse(JSON.stringify(app)) - await goto('/apps/add?nodraft=true') + await goto('/apps/add') }, disabled: !showEditButtons, unifiedSize: 'md', diff --git a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte index a733c325d6..1d6f3f45ec 100644 --- a/frontend/src/routes/(root)/(logged)/sessions/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/sessions/+page.svelte @@ -7,7 +7,6 @@ import SessionWrapper from '$lib/components/sessions/SessionWrapper.svelte' import { createSession, - getEffectiveWorkspaceId, selectSession, sessionState, syncWorkspaceTo @@ -19,7 +18,6 @@ promoteEditorWarm } from '$lib/components/sessions/sessionRuntime.svelte' import { markSessionSeen } from '$lib/components/sessions/sessionUnread.svelte' - import { visibleWorkspaceIds } from '$lib/components/sessions/sessionScope.svelte' import { isGlobalAiEnabled } from '$lib/components/copilot/chat/global/gate' import { userWorkspaces } from '$lib/stores' @@ -46,21 +44,10 @@ untrack(() => syncWorkspaceTo(ws)) }) - // Resolve the active session if its effective workspace is in scope - // (active workspace + its forks). Unavailable sessions — committed to - // a workspace that no longer exists — also resolve so the user can - // land on the move/discard banner instead of hitting "Session not - // found". - const activeSession = $derived( - sessionState.sessions.find((s) => { - if (s.name !== sessionName) return false - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false - }) - ) + // sessionState.sessions holds every local session for the user. Resolve by + // name without applying the sidebar root filter so an open chat survives + // workspace switches. + const activeSession = $derived(sessionState.sessions.find((s) => s.name === sessionName)) // Touch the runtime for the active session so it gets created on first visit // and the pane shows up. Subsequent renders find it via listRuntimes(). @@ -94,20 +81,15 @@ }) }) - // Warm = has a live runtime (module-scoped) AND its workspace is in - // scope (or its workspace is unavailable — those sessions still need - // to render the move/discard banner instead of vanishing on us). + // Warm = sessions that currently have a live (module-scoped) runtime. The + // picker eagerly creates runtimes for its visible sessions, so this tracks + // whatever the picker shows — the current family, or every family when + // "Show all workspaces" is on. Runtimes whose session record isn't loaded + // resolve to undefined here and drop out. const warmSessions = $derived( listRuntimes() .map((r) => sessionState.sessions.find((s) => s.id === r.sessionId)) .filter((s): s is NonNullable => s != null) - .filter((s) => { - const ws = getEffectiveWorkspaceId(s) - if (!ws) return false - if ($visibleWorkspaceIds.has(ws)) return true - if (s.workspace_id && !$userWorkspaces.find((w) => w.id === s.workspace_id)) return true - return false - }) ) // Promote the active session in the LRU. Mutations untracked so the effect diff --git a/frontend/src/routes/(root)/(logged)/sqs_triggers/+page.svelte b/frontend/src/routes/(root)/(logged)/sqs_triggers/+page.svelte index b70730bd8c..586d2077d2 100644 --- a/frontend/src/routes/(root)/(logged)/sqs_triggers/+page.svelte +++ b/frontend/src/routes/(root)/(logged)/sqs_triggers/+page.svelte @@ -1,4 +1,5 @@ - { + { + refresh = requestTokenRefresh loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte new file mode 100644 index 0000000000..d0af10b433 --- /dev/null +++ b/frontend/src/routes/app_embed/[workspace]/[...path]/+page.svelte @@ -0,0 +1,99 @@ + + + { + refresh = requestTokenRefresh + loadApp() + }} +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/approve/[workspace]/[job]/+page.svelte b/frontend/src/routes/approve/[workspace]/[job]/+page.svelte index 9f07bc8d47..6b88204c08 100644 --- a/frontend/src/routes/approve/[workspace]/[job]/+page.svelte +++ b/frontend/src/routes/approve/[workspace]/[job]/+page.svelte @@ -34,6 +34,9 @@ let loading = $state(false) let valid = $state(true) let actionTaken: 'approved' | 'denied' | undefined = $state(undefined) + // Whether the current visitor is a logged-in member of this workspace — if so we can + // surface a clear, ready-to-use run-details link (the view token grants them access). + let isWorkspaceMember = $state(false) let pollInterval: number | undefined = undefined let scheduleEditor: ScheduleEditor | undefined = $state(undefined) @@ -57,6 +60,7 @@ } loadData() pollInterval = setInterval(loadData, 2000) + getUserExt(page.params.workspace ?? '').then((u) => (isWorkspaceMember = !!u)) }) onDestroy(() => { @@ -146,6 +150,18 @@ // strips the approval details (form, description, args, approvers) and getJob is denied. // Hide the empty detail scaffolding and show only the sign-in / not-authorized state. let isLocked = $derived(!!approvalInfo?.user_auth_required && !approvalInfo?.can_approve) + // Carry the share-read-link token so a workspace-member approver can open the run + // details of a flow they don't otherwise have read access to. Build from the route + // params (not `job`): `getJob` is fire-and-forget and gets denied for exactly the + // approver this link serves, leaving `job` undefined — and `page.params.job` is the + // flow id the token is minted for anyway. + let runDetailsHref = $derived.by(() => { + let url = `${base}/run/${page.params.job}?workspace=${page.params.workspace}` + if (approvalInfo?.view_token) { + url += `&view_token=${encodeURIComponent(approvalInfo.view_token)}` + } + return url + }) let isWac = $derived(!!(job as any)?.workflow_as_code_status) let filteredArgs = $derived.by(() => { if (!job?.args) return job?.args @@ -346,14 +362,21 @@ {#if !isLocked}
- - Open run details (require auth) - + {#if isWorkspaceMember} + + {:else} + + Open run details (require auth) + + {/if}
{/if} diff --git a/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte b/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte index 340624ba06..5a2f0b4746 100644 --- a/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte +++ b/frontend/src/routes/approve/[workspace]/[job]/[resume]/[hmac]/+page.svelte @@ -26,6 +26,10 @@ let job: Job | undefined = $state(undefined) let currentApprovers: { resume_id: number; approver: string }[] = $state([]) + let viewToken: string | undefined = $state(undefined) + // Whether the current visitor is a logged-in member of this workspace — if so we can + // surface a clear, ready-to-use run-details link (the view token grants them access). + let isWorkspaceMember = $state(false) let approver = page.url.searchParams.get('approver') ?? undefined let completed: boolean = $state(false) @@ -60,6 +64,7 @@ } getJob() timeout = setInterval(getJob, 1000) + getUserExt(page.params.workspace ?? '').then((u) => (isWorkspaceMember = !!u)) }) onDestroy(() => { @@ -101,6 +106,7 @@ }) job = suspendedJobFlow.job as Job currentApprovers = suspendedJobFlow.approvers + viewToken = suspendedJobFlow.view_token } async function resume() { @@ -143,6 +149,15 @@ .includes(new Number(page.params.resume ?? '').valueOf()) ) let approvalStep = $derived.by(() => (job?.flow_status?.step ?? 1) - 1) + // Carry the share-read-link token so a workspace-member approver can open the run + // details of a flow they don't otherwise have read access to. + let runDetailsHref = $derived.by(() => { + let url = `${base}/run/${job?.id}?workspace=${job?.workspace_id}` + if (viewToken) { + url += `&view_token=${encodeURIComponent(viewToken)}` + } + return url + }) let schema = $derived.by( () => job?.raw_flow?.modules?.[approvalStep]?.suspend?.resume_form?.schema ?? dynamicSchema ) @@ -290,13 +305,21 @@
- Open run details (require auth) + {#if isWorkspaceMember} + + {:else} + Open run details (require auth) + {/if}
{#if job && job.raw_flow && !completed}

Flow details

diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js deleted file mode 100644 index 42a8b51427..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.js +++ /dev/null @@ -1,5 +0,0 @@ -export function load({ params }) { - return { - stuff: { title: `Public App` } - } -} diff --git a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte b/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte deleted file mode 100644 index 38563cfbd9..0000000000 --- a/frontend/src/routes/apps_raw/[workspace]/[...version]/+page.svelte +++ /dev/null @@ -1,11 +0,0 @@ - - - diff --git a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte index c661aa48c6..089b95c97d 100644 --- a/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte +++ b/frontend/src/routes/public/[workspace]/[...secret]/+page.svelte @@ -4,21 +4,20 @@ import { AppService, OpenAPI, type AppWithLastVersion } from '$lib/gen' import { userStore } from '$lib/stores' - import { setContext } from 'svelte' import { setLicense } from '$lib/enterpriseUtils' import { getUserExt } from '$lib/user' - import { sendUserToast } from '$lib/toast' import { page } from '$app/state' + import { base } from '$lib/base' import PublicApp from '$lib/components/apps/editor/PublicApp.svelte' + import PublicAppFrame from '$lib/components/apps/editor/PublicAppFrame.svelte' let app: (AppWithLastVersion & { value: any }) | undefined = $state(undefined) let notExists = $state(false) let noPermission = $state(false) - let jwtError = $state(false) - function parseSecret(secret: string): { secret: string; jwt: string } { + function parseSecret(secret: string): { secret: string; jwt: string | undefined } { const parts = secret.split('/') return { secret: parts[0], @@ -27,18 +26,59 @@ } const parsedSecret = parseSecret(page.params.secret ?? '') + const workspace = page.params.workspace ?? '' + // URL for the opaque viewer iframe: the share URL WITHOUT the trailing JWT + // segment. The JWT is a viewer credential (broader and longer-lived than the + // scoped embed token) consumed here on the embedder side only — it must never + // appear in the iframe's own location, where app-authored code could read it. + // Captured once (not reactively): the embedder mirrors the app's hash/query + // back onto this page's URL, and re-deriving the src from it would reload the + // app on its every navigation. + const viewerUrl = `${base}/public/${workspace}/${parsedSecret.secret}${page.url.search}${page.url.hash}` + + let refresh: (() => void) | undefined + + // Embedder side: validate access (using the main session cookie or the shared + // JWT) and mint a scoped embed token for the opaque iframe (WIN-2006). + async function fetchEmbedToken(): Promise<{ token?: string }> { + if (parsedSecret.jwt) { + OpenAPI.TOKEN = 'jwt_ext_' + parsedSecret.jwt + } + const headers: Record = {} + if (typeof OpenAPI.TOKEN === 'string' && OpenAPI.TOKEN) { + headers['Authorization'] = `Bearer ${OpenAPI.TOKEN}` + } + const res = await fetch( + `${OpenAPI.BASE}/w/${workspace}/apps_u/embed_token/${parsedSecret.secret}`, + { headers } + ) + if (!res.ok) { + const err: any = new Error('Failed to fetch embed token') + err.status = res.status + throw err + } + return await res.json() + } + + // Viewer side: load the app + user using the embed token handed to the iframe. async function loadApp() { + try { + userStore.set(await getUserExt(workspace)) + } catch (e) { + console.warn('Anonymous user') + } try { app = await AppService.getPublicAppBySecret({ - workspace: page.params.workspace ?? '', + workspace, path: parsedSecret.secret }) noPermission = false notExists = false } catch (e) { if (e.status == 401) { - noPermission = true + // Embed token missing/expired — ask the embedder for a fresh one. + refresh?.() } else { notExists = true } @@ -47,42 +87,25 @@ if (BROWSER) { setLicense() - loadAll() - } - - function loadAll() { - console.log('loadAll') - loadUser().then(() => { - loadApp() - }) - } - - async function loadUser() { - if (parsedSecret.jwt) { - const token = 'jwt_ext_' + parsedSecret.jwt - OpenAPI.TOKEN = token - setContext<{ token?: string }>('AuthToken', { token }) - jwtError = false - } - try { - userStore.set(await getUserExt(page.params.workspace ?? '')) - if (!$userStore && parsedSecret.jwt) { - jwtError = true - sendUserToast('Could not authentify user with jwt token', true) - } - } catch (e) { - console.warn('Anonymous user') - } } - { - loadAll() + { + refresh = requestTokenRefresh + loadApp() }} -> +> + {#snippet viewer()} + loadApp()} + > + {/snippet} + diff --git a/frontend/src/routes/test_dev/+layout.svelte b/frontend/src/routes/test_dev/+layout.svelte new file mode 100644 index 0000000000..d94739a497 --- /dev/null +++ b/frontend/src/routes/test_dev/+layout.svelte @@ -0,0 +1,8 @@ + + + +{@render children()} diff --git a/frontend/src/routes/test_dev/TestDevHeader.svelte b/frontend/src/routes/test_dev/TestDevHeader.svelte new file mode 100644 index 0000000000..9c2bb3defb --- /dev/null +++ b/frontend/src/routes/test_dev/TestDevHeader.svelte @@ -0,0 +1,143 @@ + + +
+ + + + + + + + + +
+ {#if $userStore} + ● + {$userStore.username} @ {$workspaceStore} + {:else if token} + ● + token set, no user + {:else} + ● + not authenticated + {/if} +
+
diff --git a/frontend/src/routes/test_dev/sdk_app/+page.svelte b/frontend/src/routes/test_dev/sdk_app/+page.svelte new file mode 100644 index 0000000000..11b3201c32 --- /dev/null +++ b/frontend/src/routes/test_dev/sdk_app/+page.svelte @@ -0,0 +1,22 @@ + + + diff --git a/frontend/src/routes/test_dev/sdk_flow/+page.svelte b/frontend/src/routes/test_dev/sdk_flow/+page.svelte index b6aa68da3e..45cc6e32a3 100644 --- a/frontend/src/routes/test_dev/sdk_flow/+page.svelte +++ b/frontend/src/routes/test_dev/sdk_flow/+page.svelte @@ -1,16 +1,10 @@