mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-11 00:06:06 +00:00
Merge branch 'main' into tl/workspace-to-hub
This commit is contained in:
Ruben Fiszel
@@ -0,0 +1,50 @@
|
||||
---
|
||||
name: local-review-codex
|
||||
description: Run the CI Codex PR review locally against this branch's unpushed work (committed + uncommitted) before pushing. Same policy, model, and reasoning effort as the codex-pr-review GitHub action.
|
||||
---
|
||||
|
||||
# Local Codex Review (pre-push)
|
||||
|
||||
Runs the exact same review Codex performs in CI (`.github/workflows/codex-pr-review.yml`),
|
||||
but locally and scoped to work you have not pushed yet — so you catch what CI would flag
|
||||
before the PR exists. Use this before `git push` on a non-trivial change.
|
||||
|
||||
**Correspondence with CI** — identical:
|
||||
- Policy: `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test coverage).
|
||||
- Model: `gpt-5.6-sol`, `model_reasoning_effort="xhigh"`.
|
||||
- Output: markdown starting with `## Codex Review`, findings tagged P0 / P1 / P2 with file:line.
|
||||
|
||||
**Differences from CI** — local-only:
|
||||
- Scope is the current branch vs `main` at the merge-base, **including uncommitted changes** (CI reviews a pushed PR diff).
|
||||
- Sandbox is `read-only` (CI uses `danger-full-access` on an ephemeral runner). Codex reads the diff and files but cannot modify your working tree.
|
||||
- Fresh context is inherent: `codex exec` is a separate cold process, so it does not anchor on the current chat session — the same reason `local-review` insists on a subagent.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- `codex` CLI **>= 0.144.1** installed and authed (`codex login` or `OPENAI_API_KEY`). Older CLIs reject `gpt-5.6-sol` with "requires a newer version of Codex". Upgrade with `npm install --global @openai/codex@0.144.1` (may need `sudo` for a global install). Keep this in sync with the pin in `.github/workflows/codex-pr-review.yml`.
|
||||
- `git fetch` the base ref if it's stale, so the merge-base is accurate.
|
||||
|
||||
## Run
|
||||
|
||||
```bash
|
||||
bash .agents/skills/local-review-codex/run.sh # review vs main (default)
|
||||
bash .agents/skills/local-review-codex/run.sh <base> # review vs a different base ref
|
||||
```
|
||||
|
||||
Invoke with `bash` (or run the executable directly) — the script needs Bash for
|
||||
`set -o pipefail`; `sh` is Dash on Debian/Ubuntu and would fail. If `main` isn't a
|
||||
local branch (e.g. a fresh single-branch checkout), the runner falls back to
|
||||
`origin/main` automatically.
|
||||
|
||||
The script computes `BASE_SHA = git merge-base HEAD <base>`, feeds Codex `REVIEW.md` plus a
|
||||
diff context pointing at `git diff <BASE_SHA>` (which folds in uncommitted edits), and prints
|
||||
the review. It writes only temp files — nothing lands in the working tree.
|
||||
|
||||
## Relaying the result
|
||||
|
||||
Print the Codex output verbatim. Do not re-summarize or filter it — the value of a cold Codex
|
||||
pass is surfacing what the current session would rationalize away. Then decide with the user
|
||||
whether to address findings before pushing.
|
||||
|
||||
For a Claude-native review instead, use `local-review` (branch-diff-reviewer subagent). This
|
||||
skill is the Codex counterpart; run both for independent perspectives.
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/usr/bin/env bash
|
||||
# Local Codex review — mirrors the .github/workflows/codex-pr-review.yml CI job,
|
||||
# but scoped to this branch's unpushed work (committed + uncommitted) so you can
|
||||
# review before pushing. Same policy (REVIEW.md), same model (gpt-5.6-sol) and
|
||||
# reasoning effort (xhigh) as CI. Runs read-only: Codex cannot modify your tree.
|
||||
#
|
||||
# Usage: run.sh [BASE_REF] (BASE_REF defaults to "main")
|
||||
set -euo pipefail
|
||||
|
||||
BASE_REF="${1:-main}"
|
||||
REPO_ROOT="$(git rev-parse --show-toplevel)"
|
||||
cd "$REPO_ROOT"
|
||||
|
||||
if ! command -v codex >/dev/null 2>&1; then
|
||||
echo "codex CLI not found. Install with: npm install --global @openai/codex@0.144.1" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Resolve the base to a concrete commit, preferring a local ref but falling back to
|
||||
# the remote-tracking ref — checkouts (CI, single-branch clones) often have only
|
||||
# origin/main, not a local main.
|
||||
if git rev-parse --verify --quiet "${BASE_REF}^{commit}" >/dev/null; then
|
||||
BASE_COMMITISH="$BASE_REF"
|
||||
elif git rev-parse --verify --quiet "origin/${BASE_REF}^{commit}" >/dev/null; then
|
||||
BASE_COMMITISH="origin/${BASE_REF}"
|
||||
else
|
||||
echo "Base ref '$BASE_REF' not found as '$BASE_REF' or 'origin/$BASE_REF'. Try: git fetch origin $BASE_REF" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Diff from the merge-base so only this branch's changes are reviewed. Using the
|
||||
# base SHA with a single-ref `git diff` also folds in uncommitted working-tree edits,
|
||||
# but `git diff` never sees untracked files — those are gathered separately below so
|
||||
# brand-new files (a whole new module, a new skill dir) are not silently skipped.
|
||||
BASE_SHA="$(git merge-base HEAD "$BASE_COMMITISH")"
|
||||
HEAD_SHA="$(git rev-parse HEAD)"
|
||||
UNTRACKED="$(git ls-files --others --exclude-standard)"
|
||||
|
||||
if [ "$BASE_SHA" = "$HEAD_SHA" ] && git diff --quiet "$BASE_SHA" && [ -z "$UNTRACKED" ]; then
|
||||
echo "No changes vs $BASE_REF — nothing to review." >&2
|
||||
exit 0
|
||||
fi
|
||||
|
||||
PROMPT="$(mktemp)"
|
||||
OUT="$(mktemp)"
|
||||
trap 'rm -f "$PROMPT" "$OUT"' EXIT
|
||||
|
||||
# REVIEW.md is the shared policy CI feeds Codex. Append the local output-format
|
||||
# and diff context inline (CI reads these from a generated context file; inlining
|
||||
# keeps the working tree clean — no scratch files land in the repo).
|
||||
cat REVIEW.md > "$PROMPT"
|
||||
cat >> "$PROMPT" <<EOF
|
||||
|
||||
# Codex output format
|
||||
|
||||
- This is a pre-push LOCAL review of unpushed work; there is no PR yet.
|
||||
- Inspect the changes by running the diff commands in the review context below.
|
||||
- Untracked files do NOT appear in \`git diff\`. Review every untracked path listed below by reading it directly (\`cat\`) — treat its entire contents as newly added.
|
||||
- Return markdown starting with \`## Codex Review\`.
|
||||
- Tag each finding with a severity (P0 / P1 / P2), file path, and line number when known confidently.
|
||||
|
||||
# Review context
|
||||
|
||||
Local review (pre-push): current branch vs $BASE_REF
|
||||
Base SHA: $BASE_SHA
|
||||
Head SHA: $HEAD_SHA (plus any uncommitted working-tree changes)
|
||||
|
||||
Changed commits command:
|
||||
git log --oneline $BASE_SHA..HEAD
|
||||
|
||||
Changed files command:
|
||||
git diff --stat $BASE_SHA
|
||||
|
||||
Full review diff command (tracked changes, includes uncommitted edits):
|
||||
git diff --unified=0 $BASE_SHA
|
||||
|
||||
Untracked files (NOT in the diff above — read each one directly, it is entirely new):
|
||||
$(if [ -n "$UNTRACKED" ]; then printf '%s\n' "$UNTRACKED"; else echo "(none)"; fi)
|
||||
EOF
|
||||
|
||||
codex exec \
|
||||
-C "$REPO_ROOT" \
|
||||
-m gpt-5.6-sol \
|
||||
-c 'model_reasoning_effort="xhigh"' \
|
||||
-s read-only \
|
||||
-o "$OUT" \
|
||||
- < "$PROMPT"
|
||||
|
||||
echo
|
||||
echo "===== Codex review ====="
|
||||
cat "$OUT"
|
||||
@@ -0,0 +1 @@
|
||||
../../../.agents/skills/local-review-codex/SKILL.md
|
||||
@@ -57,7 +57,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
|
||||
@@ -69,7 +69,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
|
||||
@@ -23,7 +23,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache: false
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
- name: cargo check
|
||||
working-directory: ./backend
|
||||
timeout-minutes: 16
|
||||
@@ -44,7 +44,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache: false
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
- name: cargo check
|
||||
working-directory: ./backend
|
||||
timeout-minutes: 16
|
||||
@@ -81,7 +81,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache: false
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
- name: cargo check
|
||||
working-directory: ./backend
|
||||
timeout-minutes: 16
|
||||
@@ -118,7 +118,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
- name: Fix stale v8 build cache
|
||||
working-directory: ./backend
|
||||
run: |
|
||||
|
||||
@@ -50,7 +50,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- uses: actions/setup-dotnet@v4
|
||||
with:
|
||||
@@ -174,13 +174,17 @@ jobs:
|
||||
# binary link spikes several hundred MB of transient I/O. Capping at
|
||||
# 8 trades ~25% wall time for headroom on the ~75GB runner disk.
|
||||
CARGO_BUILD_JOBS: 8
|
||||
# backend/Cargo.toml sets split-debuginfo = "unpacked", which on
|
||||
# windows-msvc is coerced to "packed": every test-binary link spawns
|
||||
# the mspdbsrv.exe PDB type server and writes a large .pdb. CI needs
|
||||
# no debug info, so disable PDB generation for the dev/test profiles
|
||||
# here (avoids both LNK1318 type-server limit and PDB disk usage).
|
||||
CARGO_PROFILE_DEV_SPLIT_DEBUGINFO: "off"
|
||||
CARGO_PROFILE_TEST_SPLIT_DEBUGINFO: "off"
|
||||
# backend/Cargo.toml leaves profile.dev at the default debug = 2 for
|
||||
# the (large) windmill workspace crates; that debuginfo is emitted
|
||||
# into every object file and embedded in each test binary, and on
|
||||
# windows-msvc also spawns the mspdbsrv.exe PDB type server. Across a
|
||||
# full --all --features build it is the dominant consumer of the
|
||||
# ~63GB free on the runner disk (LNK1180 / disk-full during linking).
|
||||
# CI needs no debug info, so drop it entirely for the dev/test
|
||||
# profiles here. debug = 0 supersedes the previous split-debuginfo=off
|
||||
# knob (no debuginfo => no .pdb and no LNK1318 type-server limit).
|
||||
CARGO_PROFILE_DEV_DEBUG: "0"
|
||||
CARGO_PROFILE_TEST_DEBUG: "0"
|
||||
# Tests' poll-time stack frames (deep nested async fn chains in
|
||||
# debug builds) reach ~1.8MB. 4MB gives ~2x headroom against flaky
|
||||
# overflows under parallel-test contention.
|
||||
|
||||
@@ -90,7 +90,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
- name: Fix stale v8 build cache
|
||||
working-directory: ./backend
|
||||
run: |
|
||||
|
||||
@@ -33,7 +33,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- name: Substitute EE code
|
||||
shell: bash
|
||||
|
||||
@@ -166,7 +166,7 @@ jobs:
|
||||
|
||||
- name: Install Codex CLI
|
||||
if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
|
||||
run: npm install --global @openai/codex@0.128.0
|
||||
run: npm install --global @openai/codex@0.144.1
|
||||
|
||||
- name: Configure Codex auth
|
||||
if: steps.codex_config.outputs.enabled == 'true' && steps.pr.outputs.skip != 'true'
|
||||
@@ -280,7 +280,7 @@ jobs:
|
||||
cat REVIEW.md .github/codex/pr-review.prompt.md > /tmp/codex-prompt.md
|
||||
codex exec \
|
||||
-C "$GITHUB_WORKSPACE" \
|
||||
-m gpt-5.5 \
|
||||
-m gpt-5.6-sol \
|
||||
-c 'model_reasoning_effort="xhigh"' \
|
||||
-s danger-full-access \
|
||||
-o codex-final-message.md \
|
||||
|
||||
@@ -80,7 +80,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- name: Install xmlsec and gssapi build-time deps
|
||||
run: |
|
||||
|
||||
@@ -121,7 +121,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- uses: oven-sh/setup-bun@v2
|
||||
with:
|
||||
|
||||
@@ -35,7 +35,7 @@ jobs:
|
||||
- uses: actions-rust-lang/setup-rust-toolchain@v1
|
||||
with:
|
||||
cache-workspaces: backend
|
||||
toolchain: 1.93.0
|
||||
toolchain: 1.97.0
|
||||
|
||||
- name: Substitute EE code
|
||||
shell: bash
|
||||
|
||||
@@ -16,7 +16,7 @@ Open-source platform for internal tools, workflows, API integrations, background
|
||||
- **Backend patterns**: use the `rust-backend` skill when writing Rust code
|
||||
- **Frontend patterns**: use the `svelte-frontend` skill when writing Svelte code. Do NOT edit svelte files unless you have read that skill.
|
||||
- **Frontend UUIDs**: do not call `crypto.randomUUID()` in frontend code. Import `randomUUID` from `$lib/utils/uuid` instead.
|
||||
- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi.
|
||||
- **Code review**: review the current PR or branch against the shared review policy in `REVIEW.md` (severity triage, public-surface checklist, AGENTS.md compliance, test-coverage assessment). The skill at `.agents/skills/local-review/SKILL.md` orchestrates it. All three CLIs auto-discover the same SKILL — Claude reads `.claude/skills/` (symlinked to the canonical `.agents/skills/` file), Codex and Pi read `.agents/skills/` directly. Invoke with `/local-review` in Claude Code, `$local-review` (or `/skills` selector) in Codex, or `pi --skill local-review` / `/skill:local-review` in Pi. For a Codex-driven pass that mirrors the `codex-pr-review` GitHub action against your unpushed work (committed + uncommitted) before you push, use `/local-review-codex` (`.agents/skills/local-review-codex/`) — same `REVIEW.md` policy, `gpt-5.6-sol`, `xhigh` reasoning; requires the `codex` CLI >= 0.144.1.
|
||||
- **Domain guides**: `.claude/skills/native-trigger/` and `frontend/tutorial-system-guide.mdc`
|
||||
- **Brand/UI guidelines**: `frontend/brand-guidelines.md`
|
||||
- **CLI commands**: when adding/modifying/removing a command, subcommand, option, or description in `cli/src/commands/`, run `python system_prompts/generate.py` to refresh `system_prompts/auto-generated/` and `cli/src/guidance/skills.gen.ts`. The CLI docs the agents use to operate `wmill` are derived from the source — stale generated files give agents the wrong flags.
|
||||
|
||||
+111
@@ -1,5 +1,116 @@
|
||||
# Changelog
|
||||
|
||||
## [1.756.0](https://github.com/windmill-labs/windmill/compare/v1.755.0...v1.756.0) (2026-07-12)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* **triggers:** serve binary HTTP-route responses via base64 transfer encoding ([#10058](https://github.com/windmill-labs/windmill/issues/10058)) ([29f4cd4](https://github.com/windmill-labs/windmill/commit/29f4cd4b6f58a29b83b84a7a9b8a439d20ade00e)), closes [#5986](https://github.com/windmill-labs/windmill/issues/5986)
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* replicate all secrets on fork when external backend is configured ([#10060](https://github.com/windmill-labs/windmill/issues/10060)) ([92b7f37](https://github.com/windmill-labs/windmill/commit/92b7f375a90de2f78565ca06a13c79ff04eda44d))
|
||||
* **sessions:** sync AI-session preview with workspace edits + stop phantom autosave (WIN-2160) ([#10061](https://github.com/windmill-labs/windmill/issues/10061)) ([5cde2d5](https://github.com/windmill-labs/windmill/commit/5cde2d5b6746be9f2d0be3a98ecdaf08777a6395))
|
||||
|
||||
## [1.755.0](https://github.com/windmill-labs/windmill/compare/v1.754.0...v1.755.0) (2026-07-11)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add per-workspace job-retention override ([#10050](https://github.com/windmill-labs/windmill/issues/10050)) ([ff774c4](https://github.com/windmill-labs/windmill/commit/ff774c46bff4bff1c532e512b163225aa7c41c11))
|
||||
* **apps:** authorize deployed-app S3 reads on-behalf of the author for logged-in viewers ([#10048](https://github.com/windmill-labs/windmill/issues/10048)) ([1e192f2](https://github.com/windmill-labs/windmill/commit/1e192f2d864b8a4671e900726972737406bc388a))
|
||||
* **mcp:** add multi-workspace MCP tokens via the gateway endpoint ([#10043](https://github.com/windmill-labs/windmill/issues/10043)) ([8343203](https://github.com/windmill-labs/windmill/commit/8343203ec2cea28a2ffd5b4ac636497e861fa3ce))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* clearer errors on auto-draft save failure (WIN-2157) ([#10053](https://github.com/windmill-labs/windmill/issues/10053)) ([04eb7dd](https://github.com/windmill-labs/windmill/commit/04eb7ddd3906c28bec1711e276473a87c7b9500f))
|
||||
* **docker:** pin ansible tool interpreter to a persistent path ([#10054](https://github.com/windmill-labs/windmill/issues/10054)) ([6f49a1f](https://github.com/windmill-labs/windmill/commit/6f49a1f6a904442fcae9bb703f095b0a3ef61268))
|
||||
* enforce read authorization when signing S3 objects ([#10049](https://github.com/windmill-labs/windmill/issues/10049)) ([5844c32](https://github.com/windmill-labs/windmill/commit/5844c32ac5d08081b3de7f3d11b8b98eb1e1ad9a))
|
||||
* **frontend:** don't re-seed empty editor on stale ?new_draft after draft exists ([#10044](https://github.com/windmill-labs/windmill/issues/10044)) ([e668193](https://github.com/windmill-labs/windmill/commit/e668193a93b4a7df50459b31f2dc5f9a9b23d0fe))
|
||||
* **frontend:** keep draft autosave alive after AI-session round-trip ([#10052](https://github.com/windmill-labs/windmill/issues/10052)) ([7d02d9a](https://github.com/windmill-labs/windmill/commit/7d02d9a1e47760f287a71f6e09cd6fe45efb5635))
|
||||
* **frontend:** mint draft path for new SDK builder items so autosave attaches ([#10056](https://github.com/windmill-labs/windmill/issues/10056)) ([a89b896](https://github.com/windmill-labs/windmill/commit/a89b896ce5638f42f334055f9ffe6971b047aa84))
|
||||
* **frontend:** show nested restart button for subflows nested in containers ([#10042](https://github.com/windmill-labs/windmill/issues/10042)) ([3b07817](https://github.com/windmill-labs/windmill/commit/3b0781761b70667c5961bcb15d41c907716fa9e7))
|
||||
* **frontend:** show optimistic user message and fork-creation label before beforeSend ([#10037](https://github.com/windmill-labs/windmill/issues/10037)) ([1c88242](https://github.com/windmill-labs/windmill/commit/1c88242849a02b927f59e0a67c4b4707371b784f))
|
||||
* keep agent-worker server job-completed processor alive & self-healing ([#10033](https://github.com/windmill-labs/windmill/issues/10033)) ([ab38e14](https://github.com/windmill-labs/windmill/commit/ab38e1418e67be8bcc37391bb21d2f86d1ca3fc6))
|
||||
|
||||
## [1.754.0](https://github.com/windmill-labs/windmill/compare/v1.753.0...v1.754.0) (2026-07-10)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add multi-select mode to copilot askUserQuestion ([#10016](https://github.com/windmill-labs/windmill/issues/10016)) ([7569798](https://github.com/windmill-labs/windmill/commit/756979852c3245d06c5c73eb60e9a09fd59635c5))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* accept bunnative language in AI chat flow step validation ([#10030](https://github.com/windmill-labs/windmill/issues/10030)) ([5a460db](https://github.com/windmill-labs/windmill/commit/5a460dbec6e2b81e01aa2c36cb504dde4ff6b24a))
|
||||
* **backend:** propagate script timeout when restarting perpetual scripts ([#10029](https://github.com/windmill-labs/windmill/issues/10029)) ([6c521e9](https://github.com/windmill-labs/windmill/commit/6c521e9d87724e43ebe3b7fce8b30a3671ef3d89))
|
||||
* **frontend:** name the draft in AI chat test-run confirmation ([#10024](https://github.com/windmill-labs/windmill/issues/10024)) ([9036ac7](https://github.com/windmill-labs/windmill/commit/9036ac789f358103b8d639a6b94708c452f52f90))
|
||||
* **frontend:** open new script/flow/app in AI session (not-found + friendly tab) ([#10028](https://github.com/windmill-labs/windmill/issues/10028)) ([0353569](https://github.com/windmill-labs/windmill/commit/03535691d607d8a9d1c1fa1c9c284fa3b6051d40))
|
||||
* **frontend:** persist forked "Copy of X" script drafts ([#10021](https://github.com/windmill-labs/windmill/issues/10021)) ([c537d45](https://github.com/windmill-labs/windmill/commit/c537d45e4982f30a44026d6644d5340e56f16ef9))
|
||||
* **frontend:** persist per-session preview panel resize width ([#10031](https://github.com/windmill-labs/windmill/issues/10031)) ([5387076](https://github.com/windmill-labs/windmill/commit/5387076c1c6fb35a99843aadd2055d3ac381d6cf))
|
||||
* **frontend:** scope raw-app, flow and script editors to the session workspace ([#10015](https://github.com/windmill-labs/windmill/issues/10015)) ([c000bbc](https://github.com/windmill-labs/windmill/commit/c000bbca283f5d61cff8a39458764b2b2dd2b58f))
|
||||
* resolve fork family/picker for superadmin visiting a non-member workspace ([#10023](https://github.com/windmill-labs/windmill/issues/10023)) ([368fd2d](https://github.com/windmill-labs/windmill/commit/368fd2d9e4b3ffb66e64934d9a622eea291cde5a))
|
||||
* scope AI-session flow/script editors to the session workspace ([#10025](https://github.com/windmill-labs/windmill/issues/10025)) ([c5060a1](https://github.com/windmill-labs/windmill/commit/c5060a1e9af5a704e90f92d325abf29626ecd28a))
|
||||
* **security:** drop --allow-run from Deno sandbox (GHSA-gj6h-vw66-mr8f) ([#10039](https://github.com/windmill-labs/windmill/issues/10039)) ([c029d6d](https://github.com/windmill-labs/windmill/commit/c029d6dcde44a3d16dee23a80afad920a0535b73))
|
||||
* **security:** remove git from Deno sandbox allow-run (GHSA-gj6h-vw66-mr8f) ([#10038](https://github.com/windmill-labs/windmill/issues/10038)) ([689b20a](https://github.com/windmill-labs/windmill/commit/689b20a4704a3dda8d9437b4793c0d16eb1f780f))
|
||||
* session preview tab labels, splitter hover, and diff-drawer sizing ([#10008](https://github.com/windmill-labs/windmill/issues/10008)) ([c139eed](https://github.com/windmill-labs/windmill/commit/c139eed631548113b843b466f5505bf6a01f17d3))
|
||||
* **sessions:** open test pane when enabling debug so the debug UI is visible ([#9998](https://github.com/windmill-labs/windmill/issues/9998)) ([d7a9b46](https://github.com/windmill-labs/windmill/commit/d7a9b46ab95108c7669b47b7c4be6d8c7964a9e6))
|
||||
* sync theme into session page preview iframes on toggle ([#10018](https://github.com/windmill-labs/windmill/issues/10018)) ([3704d00](https://github.com/windmill-labs/windmill/commit/3704d00956dea3b8e562a894d5330e052a753cb3))
|
||||
|
||||
|
||||
### Performance Improvements
|
||||
|
||||
* index v2_job(parent_job) to speed up run child-job listing ([#10034](https://github.com/windmill-labs/windmill/issues/10034)) ([9feda57](https://github.com/windmill-labs/windmill/commit/9feda57c15bddc7ef481579b73636b88c2a143c5))
|
||||
* skip redundant retry-chain job query for successful top-level scripts ([#10035](https://github.com/windmill-labs/windmill/issues/10035)) ([15f9e9b](https://github.com/windmill-labs/windmill/commit/15f9e9b48fc326aa3d776191aabaed22f4c41e74))
|
||||
|
||||
## [1.753.0](https://github.com/windmill-labs/windmill/compare/v1.752.0...v1.753.0) (2026-07-08)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* AI chat background jobs tray with detach, approval and preview ([#9982](https://github.com/windmill-labs/windmill/issues/9982)) ([286da00](https://github.com/windmill-labs/windmill/commit/286da005ef2faad1d193640f74f8e96999358707))
|
||||
* condensed top bar for session preview editors ([#10011](https://github.com/windmill-labs/windmill/issues/10011)) ([b847ca2](https://github.com/windmill-labs/windmill/commit/b847ca2bc7f06f494aa802d4350d6f032ba2bb58))
|
||||
* **db-health:** add connection sizing guidance ([#10014](https://github.com/windmill-labs/windmill/issues/10014)) ([f28ea9c](https://github.com/windmill-labs/windmill/commit/f28ea9cb991bbda32ed1b7a37a5f1b3552a589a8))
|
||||
* **sessions:** scoped preview refresh + multi-target live editors + pipeline preview ([#10006](https://github.com/windmill-labs/windmill/issues/10006)) ([32c398f](https://github.com/windmill-labs/windmill/commit/32c398f27de8cd5b1478ef60d247c13705b6b50f))
|
||||
* shared tab system, universal markdown code blocks, subtle scrollbars ([#10003](https://github.com/windmill-labs/windmill/issues/10003)) ([a00ee51](https://github.com/windmill-labs/windmill/commit/a00ee5196b2c013e9672ab029f5477079ac5da21))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* bump bundled Go CLIs to patched versions to clear image CVEs ([#9996](https://github.com/windmill-labs/windmill/issues/9996)) ([d467161](https://github.com/windmill-labs/windmill/commit/d467161117444d7d9b18def627e90d9622512e02))
|
||||
* name the offending item when a fork fails on a NUL escape ([#10013](https://github.com/windmill-labs/windmill/issues/10013)) ([99d0047](https://github.com/windmill-labs/windmill/commit/99d00475156def6faad255c4e728923253f9169f))
|
||||
* preserve worker group tag override on 'Run again' ([#10004](https://github.com/windmill-labs/windmill/issues/10004)) ([c4cb2f3](https://github.com/windmill-labs/windmill/commit/c4cb2f373b6361f0f3ce6b1c8e32a4c010207760))
|
||||
* replicate external secret backend secrets when forking a workspace ([#10007](https://github.com/windmill-labs/windmill/issues/10007)) ([f65fe7b](https://github.com/windmill-labs/windmill/commit/f65fe7bf585d353f7d88746e947d68e2f351e516))
|
||||
* session preview editors and picker dropdown overflow ([#10010](https://github.com/windmill-labs/windmill/issues/10010)) ([fb12b23](https://github.com/windmill-labs/windmill/commit/fb12b23e0169ba2cdcf454a27dcf814a2caf26b3))
|
||||
|
||||
## [1.752.0](https://github.com/windmill-labs/windmill/compare/v1.751.0...v1.752.0) (2026-07-07)
|
||||
|
||||
|
||||
### Features
|
||||
|
||||
* add fork_parent_workspace claim to OIDC tokens for fork workspaces ([#9987](https://github.com/windmill-labs/windmill/issues/9987)) ([7efeae2](https://github.com/windmill-labs/windmill/commit/7efeae26d821b10667b6e3edd220468f6ae48936))
|
||||
* add SQL migrations for data tables ([#9693](https://github.com/windmill-labs/windmill/issues/9693)) ([e47aeda](https://github.com/windmill-labs/windmill/commit/e47aedac0a4af40dd697d5fc4d54dd3c8efe9ab8))
|
||||
* **cli:** clarify fork-branch workspace auto-targeting in output ([#9988](https://github.com/windmill-labs/windmill/issues/9988)) ([88c2d0e](https://github.com/windmill-labs/windmill/commit/88c2d0e8e32c218787c01daed80ef41efc39dd11))
|
||||
* open runs/schedules pages from AI chat in session preview tabs ([#9976](https://github.com/windmill-labs/windmill/issues/9976)) ([4bb82ad](https://github.com/windmill-labs/windmill/commit/4bb82ad6cdb62eae7b69b1054714333e54558632))
|
||||
* **raw-apps:** runtime-error overlay + AI import-React instruction ([#9966](https://github.com/windmill-labs/windmill/issues/9966)) ([8df613b](https://github.com/windmill-labs/windmill/commit/8df613b4d2f88765f49cc988a894ca323c4ec4f7))
|
||||
* **sessions:** v2 unified sidebar with family/fork scoping and preview router ([#9816](https://github.com/windmill-labs/windmill/issues/9816)) ([9503190](https://github.com/windmill-labs/windmill/commit/95031903ebe223dc03b49a6bcd3e4ee67cefc4bb))
|
||||
* smooth bursty AI chat streaming with a typewriter reveal ([#9991](https://github.com/windmill-labs/windmill/issues/9991)) ([a6276b5](https://github.com/windmill-labs/windmill/commit/a6276b590082d06480434a8ea002c335ea1cfb59))
|
||||
* update base image to debian 13 (trixie) ([#9973](https://github.com/windmill-labs/windmill/issues/9973)) ([c5c1ead](https://github.com/windmill-labs/windmill/commit/c5c1eadeb18e509a98d1e787206c0438417683fc))
|
||||
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
* **ai-agent:** align agent_actions_success with agent_actions for mcp and websearch ([#9983](https://github.com/windmill-labs/windmill/issues/9983)) ([87f8d46](https://github.com/windmill-labs/windmill/commit/87f8d46aafffd5e88a336192c51e0c95ff2e6f18))
|
||||
* **ai:** flow writer builds approval steps as scripts, not identity ([#9985](https://github.com/windmill-labs/windmill/issues/9985)) ([6b01caa](https://github.com/windmill-labs/windmill/commit/6b01caaf26a4f0a08f643db4e22a70e27d0dc554))
|
||||
* clear old path asset usage when renaming a script ([#9979](https://github.com/windmill-labs/windmill/issues/9979)) ([927b8d0](https://github.com/windmill-labs/windmill/commit/927b8d064f693384978184992b8f8a1cd708e711))
|
||||
* **cli:** auto-derive cascade triggers in --local pipeline graph ([#9978](https://github.com/windmill-labs/windmill/issues/9978)) ([edfe7b4](https://github.com/windmill-labs/windmill/commit/edfe7b415af6670c5855b7a0b52db4c1f7781964))
|
||||
* **pipelines:** live materialize/dataset editing — stale graph, phantom drafts, stale Save-all deploys ([#9990](https://github.com/windmill-labs/windmill/issues/9990)) ([f7efb64](https://github.com/windmill-labs/windmill/commit/f7efb646bf1f2e132d1e3ff031b142383ae01c5e))
|
||||
* **sessions:** auto-rename regression + preview-panel and fork nits ([#9993](https://github.com/windmill-labs/windmill/issues/9993)) ([804178f](https://github.com/windmill-labs/windmill/commit/804178f5e1c904c3f8e35e2b660f33c78964c6eb))
|
||||
* **sessions:** scope fork session Edits to session-edited items only ([#9989](https://github.com/windmill-labs/windmill/issues/9989)) ([7046dc6](https://github.com/windmill-labs/windmill/commit/7046dc6dfb474ef49313377855bb2bd60294e25a))
|
||||
|
||||
## [1.751.0](https://github.com/windmill-labs/windmill/compare/v1.750.0...v1.751.0) (2026-07-06)
|
||||
|
||||
|
||||
|
||||
+4
-4
@@ -1,5 +1,5 @@
|
||||
ARG DEBIAN_IMAGE=debian:trixie-slim
|
||||
ARG RUST_IMAGE=rust:1.93-slim-trixie
|
||||
ARG RUST_IMAGE=rust:1.97-slim-trixie
|
||||
|
||||
FROM debian:trixie-slim AS nsjail
|
||||
|
||||
@@ -135,8 +135,8 @@ FROM ${DEBIAN_IMAGE}
|
||||
|
||||
ARG TARGETPLATFORM
|
||||
ARG POWERSHELL_VERSION=7.5.0
|
||||
ARG KUBECTL_VERSION=1.28.7
|
||||
ARG HELM_VERSION=3.14.3
|
||||
ARG KUBECTL_VERSION=1.36.2
|
||||
ARG HELM_VERSION=3.21.2
|
||||
# NOTE: If changing, also change go version in workspace dependencies template at WorkspaceDependenciesEditor.svelte
|
||||
ARG GO_VERSION=1.26.0
|
||||
ARG APP=/usr/src/app
|
||||
@@ -310,7 +310,7 @@ COPY --from=nsjail /nsjail/nsjail /bin/nsjail
|
||||
|
||||
# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
|
||||
# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
|
||||
ARG CRANE_VERSION=v0.20.6
|
||||
ARG CRANE_VERSION=v0.21.7
|
||||
RUN arch="$(dpkg --print-architecture)"; \
|
||||
case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
|
||||
wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
|
||||
|
||||
@@ -359,6 +359,8 @@
|
||||
- request_approval
|
||||
- finalize_purchase
|
||||
topLevelStepTypes:
|
||||
- id: request_approval
|
||||
type: [rawscript, script]
|
||||
- id: finalize_purchase
|
||||
type: rawscript
|
||||
schemaRequiredPaths:
|
||||
@@ -373,6 +375,7 @@
|
||||
judgeChecklist:
|
||||
- "the flow includes an approval step named `request_approval`"
|
||||
- "`request_approval` pauses the flow and asks the approver for a comment"
|
||||
- "`request_approval` is a real script step that generates approval/resume URLs (e.g. via `getResumeUrls`) so approvers receive an actionable link, not a no-op passthrough (identity) step"
|
||||
- one approval is enough to continue
|
||||
- "the flow includes a final step named `finalize_purchase`"
|
||||
- "`finalize_purchase` returns an approved status object after approval"
|
||||
|
||||
@@ -872,6 +872,207 @@
|
||||
- fetches the logs for the requested job id
|
||||
- explains the failure from the returned logs (connection refused to the upstream API)
|
||||
|
||||
# --- Page navigation (open_page) ---
|
||||
# The assistant should take the user to a Windmill page (Runs/Schedules) with the
|
||||
# right filters via open_page, rather than describing where to click or dumping the
|
||||
# data. No draft is produced, so the global judge is skipped and we validate the
|
||||
# tool call and its arguments.
|
||||
|
||||
- id: global-openpage1-runs-failed-of-script
|
||||
prompt: |-
|
||||
Take me to the failed runs of the script at f/evals/global/greet_user so I can see what's going wrong.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- open_page
|
||||
forbiddenToolsUsed:
|
||||
- write_script
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: open_page
|
||||
field: page
|
||||
stringIncludesAnyOf:
|
||||
- runs
|
||||
- tool: open_page
|
||||
field: status
|
||||
stringIncludesAnyOf:
|
||||
- failure
|
||||
- tool: open_page
|
||||
field: path
|
||||
stringIncludesAnyOf:
|
||||
- f/evals/global/greet_user
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- opens the Runs page filtered to the failed runs of f/evals/global/greet_user
|
||||
- applies both the failure status and the script path as filters
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
- id: global-openpage2-runs-of-schedule
|
||||
prompt: |-
|
||||
Open the runs page filtered to the jobs triggered by the schedule f/evals/global/nightly_digest.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- open_page
|
||||
forbiddenToolsUsed:
|
||||
- write_script
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: open_page
|
||||
field: page
|
||||
stringIncludesAnyOf:
|
||||
- runs
|
||||
- tool: open_page
|
||||
field: schedule_path
|
||||
stringIncludesAnyOf:
|
||||
- f/evals/global/nightly_digest
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- opens the Runs page filtered to jobs triggered by the f/evals/global/nightly_digest schedule
|
||||
- passes the schedule path as the filter
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
- id: global-openpage3-open-schedule
|
||||
prompt: |-
|
||||
Open the schedule f/evals/global/nightly_digest so I can review and edit it.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- open_page
|
||||
forbiddenToolsUsed:
|
||||
- write_schedule
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: open_page
|
||||
field: page
|
||||
stringIncludesAnyOf:
|
||||
- schedules
|
||||
- tool: open_page
|
||||
field: open
|
||||
stringIncludesAnyOf:
|
||||
- f/evals/global/nightly_digest
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- opens the Schedules page and targets the f/evals/global/nightly_digest schedule for editing
|
||||
- passes the schedule path so the editor opens on it
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
- id: global-openpage4-workspace-settings-tab
|
||||
prompt: |-
|
||||
Take me to the Git sync configuration for this workspace.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- open_page
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: open_page
|
||||
field: page
|
||||
stringIncludesAnyOf:
|
||||
- workspace_settings
|
||||
- tool: open_page
|
||||
field: tab
|
||||
stringIncludesAnyOf:
|
||||
- git_sync
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- opens the Workspace settings page on the git_sync tab
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
- id: global-openpage5-audit-logs-user
|
||||
prompt: |-
|
||||
Open the audit logs filtered to actions performed by the user admin.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- open_page
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: open_page
|
||||
field: page
|
||||
stringIncludesAnyOf:
|
||||
- audit_logs
|
||||
- tool: open_page
|
||||
field: username
|
||||
stringIncludesAnyOf:
|
||||
- admin
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- opens the Audit logs page filtered to the admin user
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
- id: global-openpage6-triggers-kind
|
||||
prompt: |-
|
||||
Take me to the Kafka triggers for this workspace.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- open_page
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: open_page
|
||||
field: page
|
||||
stringIncludesAnyOf:
|
||||
- triggers
|
||||
- tool: open_page
|
||||
field: trigger_kind
|
||||
stringIncludesAnyOf:
|
||||
- kafka
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- opens the Kafka triggers page
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
- id: global-closepage1-close-runs-tab
|
||||
prompt: |-
|
||||
You just opened the runs page for me in the side panel. Close that tab, I'm done looking at it.
|
||||
runtime:
|
||||
maxTurns: 6
|
||||
validate:
|
||||
draftCountExactly: 0
|
||||
toolExpect:
|
||||
requiredToolsUsed:
|
||||
- close_page
|
||||
forbiddenToolsUsed:
|
||||
- deploy_workspace_item
|
||||
- delete_workspace_item
|
||||
toolCallArgs:
|
||||
- tool: close_page
|
||||
field: match
|
||||
stringIncludesAnyOf:
|
||||
- runs
|
||||
skipJudge: true
|
||||
judgeChecklist:
|
||||
- closes the runs preview tab in the side panel
|
||||
- does not write, deploy, or delete anything
|
||||
|
||||
# --- Documentation search (search_docs) ---
|
||||
# Pure product-knowledge questions: the assistant should consult the docs via
|
||||
# search_docs and answer conversationally, not draft or mutate anything. No
|
||||
|
||||
@@ -47,7 +47,7 @@ export interface FlowValidationSpec {
|
||||
}>;
|
||||
topLevelStepTypes?: Array<{
|
||||
id: string;
|
||||
type: string;
|
||||
type: string | string[];
|
||||
}>;
|
||||
moduleRules?: Array<{
|
||||
id: string;
|
||||
|
||||
@@ -1378,11 +1378,14 @@ function validateFlowRequirements(
|
||||
continue;
|
||||
}
|
||||
|
||||
const allowedTypes = Array.isArray(requiredStep.type)
|
||||
? requiredStep.type
|
||||
: [requiredStep.type];
|
||||
checks.push(
|
||||
check(
|
||||
`${requiredStep.id} type matches required`,
|
||||
getModuleType(module) === requiredStep.type,
|
||||
`expected ${requiredStep.type}, got ${getModuleType(module) ?? "(missing)"}`
|
||||
allowedTypes.includes(getModuleType(module) ?? ""),
|
||||
`expected ${allowedTypes.join(" or ")}, got ${getModuleType(module) ?? "(missing)"}`
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT workspace_id as \"workspace_id!\", MIN(completed_at) as oldest\n FROM v2_job_completed\n WHERE workspace_id = ANY($1::text[])\n GROUP BY workspace_id",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "workspace_id!",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "oldest",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "056231b872a080a257c034489127e253678631e091b9e4223842648a4475dbbf"
|
||||
}
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n (SELECT MIN(completed_at) FROM v2_job_completed) as true_oldest,\n (SELECT MIN(completed_at) FROM v2_job_completed\n WHERE workspace_id <> ALL($1::text[])) as global_oldest,\n (SELECT COUNT(*) FROM v2_job_completed) as total",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "true_oldest",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "global_oldest",
|
||||
"type_info": "Timestamptz"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "total",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "0d373b2600273763c002b1e75f743e1b981f7884df99f033838c4336bea2b051"
|
||||
}
|
||||
+8
-2
@@ -1,12 +1,17 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT restart_unless_cancelled FROM script WHERE hash = $1 AND workspace_id = $2",
|
||||
"query": "SELECT restart_unless_cancelled, timeout FROM script WHERE hash = $1 AND workspace_id = $2",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "restart_unless_cancelled",
|
||||
"type_info": "Bool"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "timeout",
|
||||
"type_info": "Int4"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
@@ -16,8 +21,9 @@
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "1d27895aa42ccbb542479b19baefd62790205b529ab0d8af36f18c470e8bb838"
|
||||
"hash": "1debd472c9ffd2fc78877484f93db51f9aabed54f9894eda8ad610053ad76ce6"
|
||||
}
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.workspace_id = $5\n AND jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND jc.completed_at >= COALESCE($4::timestamptz, '-infinity'::timestamptz)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"UuidArray",
|
||||
"Timestamptz",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "20d62e1937c855cc6887597b3860e991cb539320f4c8321af94416f702108e2e"
|
||||
}
|
||||
+35
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT\n COUNT(*) FILTER (WHERE worker LIKE $1) as \"live_workers!\",\n COUNT(DISTINCT worker_instance) FILTER (WHERE worker LIKE $1) as \"live_instances!\",\n COUNT(*) FILTER (WHERE worker LIKE $2) as \"live_agent_workers!\"\n FROM worker_ping\n WHERE ping_at > now() - interval '30 seconds'",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "live_workers!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "live_instances!",
|
||||
"type_info": "Int8"
|
||||
},
|
||||
{
|
||||
"ordinal": 2,
|
||||
"name": "live_agent_workers!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null,
|
||||
null,
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "4556f04f9da4adffb296b9c45bd97c9af65dd72a682d9d704a2dafb563001f83"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT id, name FROM workspace WHERE deleted = false ORDER BY name",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "name",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "67c405ff2bfd68119dbd5e2edc91fde70711b2fb8ec6826411cc7d74687d5bcb"
|
||||
}
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COUNT(*) FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($2::text[] IS NULL OR workspace_id NOT IN (\n SELECT u FROM unnest($2::text[]) AS u WHERE u IS NOT NULL\n ))",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "73a3e00e61b33c8ba683ea03f314a229f3dc601b4592d052d3956d5e54d69463"
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT setting::bigint as \"v!\" FROM pg_settings WHERE name = 'superuser_reserved_connections'",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "v!",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": []
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "80e233d7db456ec7030486b009b6cfbf0fa45c2463b861c824fb9f1c0c46192f"
|
||||
}
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT COUNT(*) FROM v2_job_completed\n WHERE workspace_id = $1\n AND completed_at <= now() - ($2::bigint::text || ' s')::interval",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "count",
|
||||
"type_info": "Int8"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text",
|
||||
"Int8"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
null
|
||||
]
|
||||
},
|
||||
"hash": "8c1ae47f01c3d5f66faf2f1835c612cea9f6c69d664ceb89094fd544dc63af9d"
|
||||
}
|
||||
-30
@@ -1,30 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($3::timestamptz IS NULL OR completed_at >= $3)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Timestamptz"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "a2d4a8aedb15e9faf0a2512fa4241a9e9f2d5a56e12a9d6805ca58ff40f61614"
|
||||
}
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT workspace_id, scopes FROM token WHERE token_hash = $1 AND (expiration > NOW() OR expiration IS NULL)",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "workspace_id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "scopes",
|
||||
"type_info": "TextArray"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
true,
|
||||
true
|
||||
]
|
||||
},
|
||||
"hash": "a460f0ca8f23a8eb9d808b5edd6e0cde0e125f8ed426bd784dd7b92e1d21dfdf"
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE workspace_id = $4\n AND completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND completed_at >= COALESCE($3::timestamptz, '-infinity'::timestamptz)\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Timestamptz",
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "b326b02adf2245369173fbb1b24386515290c33ea7cedfe2a1092690749ca49c"
|
||||
}
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT workspace.id, workspace.name\n FROM workspace\n JOIN usr ON usr.workspace_id = workspace.id\n WHERE usr.email = $1 AND usr.disabled = false AND workspace.deleted = false\n ORDER BY workspace.name",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Varchar"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "name",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "b3b06ec52fde4b8264c6307c24b046cd3af17c5ce0d4426153b3065b2faaa781"
|
||||
}
|
||||
+22
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "SELECT path FROM variable\n WHERE workspace_id = $1 AND is_secret = true AND value != ''",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "path",
|
||||
"type_info": "Varchar"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Text"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "b6de5fbd5fa89e9a8ab8a61982d4f6e02030d8581a7e8c1f0673c7f6dae78fb3"
|
||||
}
|
||||
-31
@@ -1,31 +0,0 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"UuidArray",
|
||||
"Timestamptz"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "c033a690fde04da79745e850b72fa7cfd861f1dcad88c7ea75a0a8b014ec1f75"
|
||||
}
|
||||
+32
@@ -0,0 +1,32 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT jc.id FROM v2_job_completed jc\n LEFT JOIN v2_job j ON j.id = jc.id\n WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND jc.completed_at >= COALESCE($4::timestamptz, '-infinity'::timestamptz)\n AND ($5::text[] IS NULL OR jc.workspace_id NOT IN (\n SELECT u FROM unnest($5::text[]) AS u WHERE u IS NOT NULL\n ))\n AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (\n SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL\n )\n ORDER BY jc.completed_at ASC\n LIMIT $2\n FOR UPDATE OF jc SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"UuidArray",
|
||||
"Timestamptz",
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "c89a6cb3b2d5aad2d5e406b5d85149d3ee4cd59ae9d7b154a6d0fcab438b09e6"
|
||||
}
|
||||
+31
@@ -0,0 +1,31 @@
|
||||
{
|
||||
"db_name": "PostgreSQL",
|
||||
"query": "DELETE FROM v2_job_completed\n WHERE id IN (\n SELECT id FROM v2_job_completed\n WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval\n AND completed_at >= COALESCE($3::timestamptz, '-infinity'::timestamptz)\n AND ($4::text[] IS NULL OR workspace_id NOT IN (\n SELECT u FROM unnest($4::text[]) AS u WHERE u IS NOT NULL\n ))\n ORDER BY completed_at ASC\n LIMIT $2\n FOR UPDATE SKIP LOCKED\n )\n RETURNING id, completed_at",
|
||||
"describe": {
|
||||
"columns": [
|
||||
{
|
||||
"ordinal": 0,
|
||||
"name": "id",
|
||||
"type_info": "Uuid"
|
||||
},
|
||||
{
|
||||
"ordinal": 1,
|
||||
"name": "completed_at",
|
||||
"type_info": "Timestamptz"
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Left": [
|
||||
"Int8",
|
||||
"Int8",
|
||||
"Timestamptz",
|
||||
"TextArray"
|
||||
]
|
||||
},
|
||||
"nullable": [
|
||||
false,
|
||||
false
|
||||
]
|
||||
},
|
||||
"hash": "e9c3e9b9dd6a54c50a7f6639b7922d59d970aa87c0d74ca461f1dea83972ccd9"
|
||||
}
|
||||
@@ -18,6 +18,86 @@
|
||||
- **Running data pipelines (DuckLake) from source**: see the section below — a plain build
|
||||
advertises the `duckdb` tag but cannot execute DuckDB scripts and has no working S3 proxy.
|
||||
|
||||
## Cargo features & running the dev backend
|
||||
|
||||
The dev backend runs under `cargo watch` and is launched by default with **only
|
||||
`--features quickjs`** (see the tmux backend pane). That baseline compiles fast but
|
||||
**deliberately omits most functionality** — notably S3/object storage, the S3 proxy, all
|
||||
EE code, MCP, and every non-JS language runtime. A running server never gains a feature you
|
||||
didn't compile in: feature-gated routes 404 or return a `"requires <feature>"` stub. So if
|
||||
you touch code behind a feature gate, or need to *exercise* such a feature at runtime, you
|
||||
MUST **restart the backend with the appropriate features** for what you're working on.
|
||||
|
||||
### Restarting the dev backend with the right features
|
||||
|
||||
The backend runs in tmux pane 1 as `cargo watch -x "run --features <…>"`. To restart it with a
|
||||
different feature set — scope kills by pid/cwd, **never** `pkill -f target/debug/windmill` (it
|
||||
kills every sibling worktree's backend):
|
||||
|
||||
1. Stop the current run: `tmux send-keys -t <pane1> C-c`, then kill *this worktree's*
|
||||
`cargo-watch` pid (find it via `/proc/<pid>/cwd`).
|
||||
2. Relaunch in the same pane so it inherits the shell's `DATABASE_URL` etc.; the pane env's
|
||||
`PORT` may be stale, so set it explicitly:
|
||||
```bash
|
||||
export PORT=$BACKEND_PORT
|
||||
cargo watch -x "run --features enterprise,private,parquet,quickjs"
|
||||
```
|
||||
3. Wait for `health check completed` in the pane before hitting the API.
|
||||
|
||||
cargo-watch only re-runs on a file change, so after an idle/failed run `touch README.md` (from
|
||||
`backend/`, where the watch runs) is a cheap retrigger (touching a `.rs` forces a full rebuild).
|
||||
|
||||
### What each feature gate does (the ones you'll actually toggle)
|
||||
|
||||
`backend/Cargo.toml` `[features]` is the source of truth; this is the practical dev map. Combine
|
||||
only what you need — build time scales with the set.
|
||||
|
||||
| Feature | Enables | Need it for |
|
||||
|---|---|---|
|
||||
| `quickjs` | Embedded JS engine for inline JS eval (the default dev baseline). | Keep in every dev set. |
|
||||
| `private` | Compiles the `*_ee.rs` files (symlinked from `windmill-ee-private`). Gates **all** EE code, including the real S3 helpers, the S3 proxy, and advanced S3 permission checks. | Any EE code path, S3/object storage. |
|
||||
| `enterprise` | EE business logic (autoscaling, SAML hooks, advanced S3 rule **enforcement**, WAP, forks, …). Pulls in `license`. | Running EE features. Advanced S3 permission rules only take effect with this. |
|
||||
| `license` | License-key/plan plumbing (`LICENSE_KEY`). Pulled in by `enterprise`. Having the feature compiled does **not** require a license *key* at runtime — CE defaults to a free plan and most EE paths still run keyless. | License-gated behavior. |
|
||||
| `parquet` | S3/object-storage support: the `job_helpers/*` and `apps_u/*` S3 endpoints, parquet/CSV preview, workspace large-file storage. Without it those routes return `"requires parquet"`. | Anything touching S3/object storage or datasets. |
|
||||
| `duckdb` | DuckDB script executor (also needs the FFI dylib — see above). | DuckDB scripts, DuckLake. |
|
||||
| `python` `rust` `php` `java` `ruby` `csharp` `nu` `deno_core` `mysql` `mssql` `bigquery` `oracledb` `rlang` | Each enables that language/DB runtime for job execution. | Running jobs in that language. |
|
||||
| `mcp` | MCP gateway routes (baseline `quickjs` does NOT include it → MCP routes 404). | MCP work. |
|
||||
| `websocket` `http_trigger` `kafka` `nats` `mqtt_trigger` `sqs_trigger` `gcp_trigger` `azure_trigger` `postgres_trigger` `native_trigger` | Each native trigger kind; none on by default (creating one 404s without its feature). | Working on / exercising that trigger. |
|
||||
| `no_auth` | Treats every request as an admin superadmin (`CLOUD_HOSTED`-guarded). | Local auth-free experiments only. |
|
||||
|
||||
Convenience bundles (`ce`, `ee`, `oss`, …) exist in `[features]` but are heavy — prefer the
|
||||
minimal explicit set for dev.
|
||||
|
||||
**Common combinations** (run from `backend/`):
|
||||
|
||||
| Goal | `--features` |
|
||||
|---|---|
|
||||
| Plain dev baseline (JS eval only) | `quickjs` |
|
||||
| S3 / object storage / datasets (CE) | `quickjs,private,parquet` |
|
||||
| S3 + EE (advanced S3 rules, on-behalf app reads, WAP, forks) | `quickjs,enterprise,private,parquet` |
|
||||
| DuckLake / DuckDB (CE) | `quickjs,duckdb,parquet,private` (+ build the FFI) |
|
||||
| + Python jobs | append `,python` |
|
||||
|
||||
## Workspace object storage in dev — use the local filesystem
|
||||
|
||||
For a dev workspace you don't need MinIO/S3: use the built-in **`FilesystemStorage`** large-file
|
||||
storage (a root path on local disk). It is intentionally hidden from the settings-UI storage
|
||||
dropdown (dev-only), so set it via the API. Requires the backend built with `parquet` (+ `private`
|
||||
for the real S3 helpers, + `enterprise` if you want advanced permission rules enforced):
|
||||
|
||||
```bash
|
||||
curl -X POST "$BASE/api/w/<ws>/workspaces/edit_large_file_storage_config" \
|
||||
-H "Authorization: Bearer <admin-token>" -H "Content-Type: application/json" \
|
||||
-d '{"large_file_storage":{"type":"FilesystemStorage","root_path":"/abs/writable/dir",
|
||||
"public_resource":false,"advanced_permissions":null,"secondary_storage":{}}}'
|
||||
```
|
||||
|
||||
Optional `advanced_permissions` (EE) is a list of `{"pattern":"<glob>","allow":"read[,write,delete,list]"}`
|
||||
rules: admins bypass them, non-admins are confined to matching grants. Uploads/reads then flow
|
||||
through the normal `job_helpers/*` (viewer-scoped) and `apps_u/*` (app-author on-behalf) S3
|
||||
endpoints. Caveat: direct DuckDB access rejects filesystem stores (`"Filesystem is not supported
|
||||
in DuckDB"`) — DuckLake/datatable go through the S3 proxy instead, which works.
|
||||
|
||||
## Running data pipelines (DuckLake) from source
|
||||
|
||||
DuckLake pipelines need **both** the right cargo features **and** the prebuilt DuckDB FFI. A
|
||||
|
||||
Generated
+117
-118
@@ -1890,18 +1890,18 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "bytemuck"
|
||||
version = "1.25.0"
|
||||
version = "1.25.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c8efb64bd706a16a1bdde310ae86b351e4d21550d98d056f22f8a7f7a2183fec"
|
||||
checksum = "d6aedf8ae72766347502cf3cb4f41cf5e9cc37d28bee90f1fdaaae15f9cf9424"
|
||||
dependencies = [
|
||||
"bytemuck_derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "bytemuck_derive"
|
||||
version = "1.10.2"
|
||||
version = "1.11.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f9abbd1bc6865053c427f7198e6af43bfdedc55ab791faed4fbd361d789575ff"
|
||||
checksum = "f65693059b6b9c588b9f62fed1cedbf0a8b805631457ea162d68f0de186f3de5"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -1916,9 +1916,9 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"
|
||||
|
||||
[[package]]
|
||||
name = "bytes"
|
||||
version = "1.12.0"
|
||||
version = "1.12.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8ae3f5d315924270530207e2a68396c3cc547f6dca3fbdca317cfb1a51edb593"
|
||||
checksum = "fc652a48c352aef3ea3aed32080501cf3ef6ed5da78602a020c991775b0aff04"
|
||||
dependencies = [
|
||||
"serde",
|
||||
]
|
||||
@@ -2057,9 +2057,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.2.66"
|
||||
version = "1.2.67"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f5d6cac793997bd970000024b2934968efe83b382de4fdcf4fcb46b6ee4ad996"
|
||||
checksum = "e17dd265a7d0f31ef544e1b20e03add05d3b45b491b633b10d67145d2acc1a38"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"jobserver",
|
||||
@@ -6757,9 +6757,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "lru"
|
||||
version = "0.18.0"
|
||||
version = "0.18.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "8a860605968fce16869fd239cf4237a82f3ac470723415db603b0e8b6c8d4fb9"
|
||||
checksum = "0b6180140927ee907000b0aa540091f6ea512ead4447c92b8fc35bc72788a5a6"
|
||||
dependencies = [
|
||||
"hashbrown 0.17.1",
|
||||
]
|
||||
@@ -7042,9 +7042,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "memchr"
|
||||
version = "2.8.2"
|
||||
version = "2.8.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "88904434abc2901f197fe8cc55f0445e7ded921dba5911dad2e2b39b48e663c4"
|
||||
checksum = "cf8baf1c55e62ffcace7a9f06f4bd9cd3f0c4beb022d3b367256b91b87513d98"
|
||||
|
||||
[[package]]
|
||||
name = "memmap2"
|
||||
@@ -7243,7 +7243,7 @@ dependencies = [
|
||||
"futures-sink",
|
||||
"futures-util",
|
||||
"keyed_priority_queue",
|
||||
"lru 0.18.0",
|
||||
"lru 0.18.1",
|
||||
"mysql_common",
|
||||
"native-tls",
|
||||
"pem 3.0.6",
|
||||
@@ -7628,11 +7628,10 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "num-iter"
|
||||
version = "0.1.45"
|
||||
version = "0.1.46"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "1429034a0490724d0075ebb2bc9e875d6503c3cf69e235a8941aa757d83ef5bf"
|
||||
checksum = "c92800bd69a1eac91786bcfe9da64a897eb72911b8dc3095decbd07429e8048b"
|
||||
dependencies = [
|
||||
"autocfg",
|
||||
"num-integer",
|
||||
"num-traits",
|
||||
]
|
||||
@@ -9371,9 +9370,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "regex"
|
||||
version = "1.12.4"
|
||||
version = "1.13.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f1292b7759ae1cb9ec195452d1390a074f0cd8541ab7a5a8c31cd6db45d4a6ba"
|
||||
checksum = "2a0e75113e14dc5acb068cd0786884f214f1312650a3d36d269f5c4f3cdee8a2"
|
||||
dependencies = [
|
||||
"aho-corasick",
|
||||
"memchr",
|
||||
@@ -9383,9 +9382,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "regex-automata"
|
||||
version = "0.4.14"
|
||||
version = "0.4.15"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "6e1dd4122fc1595e8162618945476892eefca7b88c52820e74af6262213cae8f"
|
||||
checksum = "1f388202e4b80542a0921078cc23b6333bcf1409c1e3f86404cae4766a6131db"
|
||||
dependencies = [
|
||||
"aho-corasick",
|
||||
"memchr",
|
||||
@@ -9842,9 +9841,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustc-demangle"
|
||||
version = "0.1.27"
|
||||
version = "0.1.28"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b50b8869d9fc858ce7266cce0194bd74df58b9d0e3f6df3a9fc8eb470d95c09d"
|
||||
checksum = "b74b56ffa8bb2830709a538c2cbcae9aa062db0d2a42563bfb09bdaae44020eb"
|
||||
|
||||
[[package]]
|
||||
name = "rustc-hash"
|
||||
@@ -10149,9 +10148,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustversion"
|
||||
version = "1.0.22"
|
||||
version = "1.0.23"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b39cdef0fa800fc44525c84ccb54a029961a8215f9619753635a9c0d2538d46d"
|
||||
checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
|
||||
|
||||
[[package]]
|
||||
name = "ryu"
|
||||
@@ -10161,9 +10160,9 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
|
||||
|
||||
[[package]]
|
||||
name = "ryu-js"
|
||||
version = "1.0.2"
|
||||
version = "1.0.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "dd29631678d6fb0903b69223673e122c32e9ae559d0960a38d574695ebc0ea15"
|
||||
checksum = "04d056b875a9d2e6cb9a61d127afee9ac5999b9f87bcb32079d1318e505be714"
|
||||
|
||||
[[package]]
|
||||
name = "safetensors"
|
||||
@@ -10664,9 +10663,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "sha1"
|
||||
version = "0.10.6"
|
||||
version = "0.10.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba"
|
||||
checksum = "a978451301f4db1d02937a4ab3ccce137717b81826e79b7d49ffe3244a13c3b8"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
"cpufeatures 0.2.17",
|
||||
@@ -11191,9 +11190,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "sse-stream"
|
||||
version = "0.2.3"
|
||||
version = "0.2.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f3962b63f038885f15bce2c6e02c0e7925c072f1ac86bb60fd44c5c6b762fb72"
|
||||
checksum = "39f24a9b78c40b90817bbcd1821c74ddfd74916aadd29403d001532a9195532d"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"futures-util",
|
||||
@@ -12108,9 +12107,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "thread_local"
|
||||
version = "1.1.9"
|
||||
version = "1.1.10"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f60246a4944f24f6e018aa17cdeffb7818b76356965d03b07d6a9886e8962185"
|
||||
checksum = "1ad99c4c6d32803332c548b1af0540b357b3f5fc0be8f6c6bfe8b2e6ae784070"
|
||||
dependencies = [
|
||||
"cfg-if",
|
||||
]
|
||||
@@ -12249,9 +12248,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "tinyvec"
|
||||
version = "1.11.0"
|
||||
version = "1.12.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3e61e67053d25a4e82c844e8424039d9745781b3fc4f32b8d55ed50f5f667ef3"
|
||||
checksum = "bb4ebadaa0af04fab11ae01eb5f9fdb5f9c5b875506e210e71c07873528baa7f"
|
||||
dependencies = [
|
||||
"tinyvec_macros",
|
||||
]
|
||||
@@ -13746,7 +13745,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
@@ -13828,7 +13827,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-ai"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"async-stream",
|
||||
"async-trait",
|
||||
@@ -13861,7 +13860,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-alerting"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -13874,7 +13873,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"argon2",
|
||||
@@ -14012,7 +14011,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-agent-workers"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14035,7 +14034,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-assets"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14050,7 +14049,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-auth"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum 0.8.9",
|
||||
@@ -14076,7 +14075,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-client"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"reqwest 0.12.28",
|
||||
"serde",
|
||||
@@ -14086,7 +14085,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-configs"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14103,7 +14102,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-debug"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"base64 0.22.1",
|
||||
@@ -14125,7 +14124,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-embeddings"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum 0.8.9",
|
||||
@@ -14148,7 +14147,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-flow-conversations"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14164,7 +14163,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-flows"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14185,7 +14184,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-groups"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14206,7 +14205,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-inputs"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14220,7 +14219,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-integration-tests"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
@@ -14255,7 +14254,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-jobs"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum 0.8.9",
|
||||
@@ -14280,7 +14279,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-npm-proxy"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"flate2",
|
||||
@@ -14298,7 +14297,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-openapi"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum 0.8.9",
|
||||
@@ -14320,7 +14319,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-schedule"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14340,7 +14339,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-scripts"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14377,7 +14376,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-settings"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum 0.8.9",
|
||||
@@ -14405,7 +14404,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-sse"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"lazy_static",
|
||||
"serde",
|
||||
@@ -14417,7 +14416,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-users"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"argon2",
|
||||
"axum 0.8.9",
|
||||
@@ -14442,7 +14441,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-workers"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14456,7 +14455,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-api-workspaces"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"axum 0.8.9",
|
||||
"chrono",
|
||||
@@ -14491,7 +14490,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-audit"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"lazy_static",
|
||||
@@ -14505,7 +14504,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-autoscaling"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum 0.8.9",
|
||||
@@ -14524,7 +14523,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-common"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"aes-gcm",
|
||||
"aho-corasick",
|
||||
@@ -14626,7 +14625,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-dep-map"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"chrono",
|
||||
"itertools 0.14.0",
|
||||
@@ -14645,7 +14644,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-git-sync"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"regex",
|
||||
"serde",
|
||||
@@ -14660,7 +14659,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-indexer"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"astral-tokio-tar",
|
||||
@@ -14684,7 +14683,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-jseval"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"futures",
|
||||
@@ -14701,7 +14700,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-macros"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"itertools 0.14.0",
|
||||
"lazy_static",
|
||||
@@ -14717,7 +14716,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-mcp"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -14738,7 +14737,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-native-triggers"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -14769,7 +14768,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-oauth"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"arc-swap",
|
||||
@@ -14794,7 +14793,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-object-store"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-stream",
|
||||
@@ -14828,7 +14827,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-operator"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"futures",
|
||||
@@ -14846,7 +14845,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"convert_case 0.6.0",
|
||||
"serde",
|
||||
@@ -14855,7 +14854,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-bash"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -14867,7 +14866,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-csharp"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -14879,7 +14878,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-go"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"gosyn",
|
||||
@@ -14891,7 +14890,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-graphql"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -14903,7 +14902,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-java"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -14915,7 +14914,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-nu"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"nu-parser",
|
||||
@@ -14926,7 +14925,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-php"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"itertools 0.14.0",
|
||||
@@ -14937,7 +14936,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"itertools 0.14.0",
|
||||
@@ -14949,7 +14948,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py-asset"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"rustpython-ast",
|
||||
@@ -14960,7 +14959,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py-imports"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-recursion",
|
||||
@@ -14982,7 +14981,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-r"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -14994,7 +14993,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ruby"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -15008,7 +15007,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-rust"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"convert_case 0.6.0",
|
||||
@@ -15025,7 +15024,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-sql"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -15038,7 +15037,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-sql-asset"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde",
|
||||
@@ -15050,7 +15049,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ts"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -15068,7 +15067,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ts-asset"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde-wasm-bindgen",
|
||||
@@ -15084,7 +15083,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-wac"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"rustpython-ast",
|
||||
@@ -15100,7 +15099,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-yaml"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde",
|
||||
@@ -15111,7 +15110,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-queue"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-recursion",
|
||||
@@ -15150,7 +15149,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-runtime-nativets"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"const_format",
|
||||
@@ -15189,7 +15188,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-sql-datatype-parser-wasm"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"getrandom 0.3.4",
|
||||
"wasm-bindgen",
|
||||
@@ -15200,7 +15199,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-store"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-recursion",
|
||||
@@ -15234,7 +15233,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-test-utils"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15258,7 +15257,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15291,7 +15290,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-azure"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15324,7 +15323,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-email"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15344,7 +15343,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-gcp"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15378,7 +15377,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-http"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15414,7 +15413,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-kafka"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15437,7 +15436,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-mqtt"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15461,7 +15460,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-nats"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-nats",
|
||||
@@ -15485,7 +15484,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-postgres"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15520,7 +15519,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-sqs"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15548,7 +15547,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-trigger-websocket"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-trait",
|
||||
@@ -15573,7 +15572,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-types"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"bitflags 2.13.0",
|
||||
@@ -15592,7 +15591,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-worker"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-once-cell",
|
||||
@@ -15702,7 +15701,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-worker-volumes"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"bytes",
|
||||
"futures",
|
||||
@@ -16428,18 +16427,18 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "zerocopy"
|
||||
version = "0.8.53"
|
||||
version = "0.8.54"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "75726053136156d419e285b9b7eddaaea9e3fea6ce32eed44a89901f0bd98de1"
|
||||
checksum = "b7cbbc0a705a0fd05cc3676525980d2bf5a9bc4adac6d6475209a7887cf59d19"
|
||||
dependencies = [
|
||||
"zerocopy-derive",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "zerocopy-derive"
|
||||
version = "0.8.53"
|
||||
version = "0.8.54"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "4714fd92cf900833d49538023a9b3915155210801d1c1169eba513b2addefd71"
|
||||
checksum = "e2e817b7b52d0c7358d3246da9d69935ebb18116b2b102b4230dac079b4862f5"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -16520,9 +16519,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "zlib-rs"
|
||||
version = "0.6.5"
|
||||
version = "0.6.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "5431d5661c32445236631278f27946e444ddafe4684cac70b185272d4f9c52d5"
|
||||
checksum = "b142a20ec14a91d5bc708c1dc21b080c550113d8aa77afa29635673a65dd02c5"
|
||||
|
||||
[[package]]
|
||||
name = "zmij"
|
||||
|
||||
+2
-2
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "windmill"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
authors.workspace = true
|
||||
edition.workspace = true
|
||||
|
||||
@@ -87,7 +87,7 @@ members = [
|
||||
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
|
||||
edition = "2021"
|
||||
|
||||
|
||||
@@ -1 +1 @@
|
||||
27672e37df5d9dfde94f19963d5ffcdf8dd5448c
|
||||
2ba6a2a75b6fc97858b306b2c98ada481e363c10
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
DROP INDEX IF EXISTS ix_v2_job_parent_job;
|
||||
@@ -0,0 +1,11 @@
|
||||
-- Partial index for listing a run's child jobs (flow steps, loop iterations,
|
||||
-- native-retry attempts, schedule handlers) via the `parent_job = ?` filter on
|
||||
-- /jobs/list and /jobs/completed/list. Without it, Postgres walks the whole
|
||||
-- workspace (workspace_id, created_at) timeline filtering row-by-row for the
|
||||
-- parent. Children of one parent are few, so (parent_job, created_at DESC)
|
||||
-- returns them directly and serves both ASC and DESC orderings.
|
||||
-- Partial on parent_job IS NOT NULL keeps it small (root jobs are the majority).
|
||||
-- Created CONCURRENTLY via the OVERRIDDEN_MIGRATIONS rewrite in windmill-api/src/db.rs.
|
||||
CREATE INDEX IF NOT EXISTS ix_v2_job_parent_job
|
||||
ON v2_job (parent_job, created_at DESC)
|
||||
WHERE parent_job IS NOT NULL;
|
||||
+24
-24
@@ -6191,7 +6191,7 @@ checksum = "712e227841d057c1ee1cd2fb22fa7e5a5461ae8e48fa2ca79ec42cfc1931183f"
|
||||
|
||||
[[package]]
|
||||
name = "windmill-common"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"aho-corasick",
|
||||
"anyhow",
|
||||
@@ -6272,7 +6272,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-macros"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -6284,7 +6284,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"convert_case",
|
||||
"serde",
|
||||
@@ -6293,7 +6293,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-bash"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6305,7 +6305,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-csharp"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -6317,7 +6317,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-go"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"gosyn",
|
||||
@@ -6329,7 +6329,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-graphql"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6341,7 +6341,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-java"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -6353,7 +6353,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-nu"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"nu-parser",
|
||||
@@ -6364,7 +6364,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-php"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"itertools 0.14.0",
|
||||
@@ -6375,7 +6375,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"itertools 0.14.0",
|
||||
@@ -6387,7 +6387,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py-asset"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"rustpython-ast",
|
||||
@@ -6398,7 +6398,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-py-imports"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"async-recursion",
|
||||
@@ -6420,7 +6420,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-r"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde_json",
|
||||
@@ -6432,7 +6432,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ruby"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6446,7 +6446,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-rust"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"convert_case",
|
||||
@@ -6463,7 +6463,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-sql"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6476,7 +6476,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-sql-asset"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde",
|
||||
@@ -6488,7 +6488,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ts"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"lazy_static",
|
||||
@@ -6506,7 +6506,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-ts-asset"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde-wasm-bindgen",
|
||||
@@ -6522,7 +6522,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-wac"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"rustpython-ast",
|
||||
@@ -6538,7 +6538,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-wasm"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"getrandom 0.2.17",
|
||||
@@ -6570,7 +6570,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-parser-yaml"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"serde",
|
||||
@@ -6581,7 +6581,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "windmill-types"
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"bitflags",
|
||||
|
||||
@@ -12,7 +12,7 @@ resolver = "2"
|
||||
members = ["."]
|
||||
|
||||
[workspace.package]
|
||||
version = "1.751.0"
|
||||
version = "1.756.0"
|
||||
edition = "2021"
|
||||
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
|
||||
|
||||
|
||||
@@ -3,9 +3,11 @@ use windmill_common::{
|
||||
get_database_url, DatabaseUrl,
|
||||
};
|
||||
|
||||
pub const DEFAULT_MAX_CONNECTIONS_SERVER: u32 = 50;
|
||||
pub const DEFAULT_MAX_CONNECTIONS_WORKER: u32 = 5;
|
||||
pub const DEFAULT_MAX_CONNECTIONS_INDEXER: u32 = 5;
|
||||
// Single source of truth in windmill_common so the DB-health sizing guidance
|
||||
// (windmill-api/src/db_health.rs) and the actual pool sizing here can't drift.
|
||||
pub use windmill_common::{
|
||||
DEFAULT_MAX_CONNECTIONS_INDEXER, DEFAULT_MAX_CONNECTIONS_SERVER, DEFAULT_MAX_CONNECTIONS_WORKER,
|
||||
};
|
||||
#[cfg(feature = "operator")]
|
||||
pub const DEFAULT_MAX_CONNECTIONS_OPERATOR: u32 = 2;
|
||||
|
||||
|
||||
+11
-6
@@ -56,11 +56,11 @@ use windmill_common::{
|
||||
PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING, POWERSHELL_REPO_URL_SETTING,
|
||||
PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING,
|
||||
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RESTART_COORDINATION_SETTING,
|
||||
RETENTION_PERIOD_SECS_SETTING, RUBY_REPOS_SETTING, SAML_METADATA_SETTING,
|
||||
SANDBOX_IMAGE_CACHE_MAX_MB_SETTING, SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING,
|
||||
SANDBOX_IMAGE_MAX_SIZE_MB_SETTING, SANDBOX_IMAGE_PULL_POLICY_SETTING,
|
||||
SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING, SMTP_SETTING,
|
||||
STORE_AUDIT_LOGS_S3_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
|
||||
RETENTION_PERIOD_SECS_OVERRIDES_SETTING, RETENTION_PERIOD_SECS_SETTING, RUBY_REPOS_SETTING,
|
||||
SAML_METADATA_SETTING, SANDBOX_IMAGE_CACHE_MAX_MB_SETTING,
|
||||
SANDBOX_IMAGE_DEFAULT_REGISTRY_SETTING, SANDBOX_IMAGE_MAX_SIZE_MB_SETTING,
|
||||
SANDBOX_IMAGE_PULL_POLICY_SETTING, SANDBOX_REGISTRY_AUTH_SETTING, SCIM_TOKEN_SETTING,
|
||||
SMTP_SETTING, STORE_AUDIT_LOGS_S3_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
|
||||
UV_EXCLUDE_NEWER_SETTING, UV_INDEX_STRATEGY_SETTING, UV_PYTHON_INSTALL_MIRROR_SETTING,
|
||||
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
|
||||
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
|
||||
@@ -124,7 +124,7 @@ use windmill_worker::{
|
||||
use crate::monitor::{
|
||||
initial_load, load_disable_password_login, load_fork_workspace_tag_append_fork_suffix,
|
||||
load_keep_job_dir, load_metrics_debug_enabled, load_preview_tags_override,
|
||||
load_require_preexisting_user, load_tag_per_workspace_enabled,
|
||||
load_require_preexisting_user, load_retention_period_overrides, load_tag_per_workspace_enabled,
|
||||
load_tag_per_workspace_workspaces, load_workspace_fairness_duration_secs,
|
||||
load_workspace_fairness_enabled, load_workspace_fairness_max_percent,
|
||||
load_workspace_fairness_min_total, monitor_db, reload_app_workspaced_route_setting,
|
||||
@@ -1881,6 +1881,11 @@ async fn process_notify_event(
|
||||
}
|
||||
TIMEOUT_WAIT_RESULT_SETTING => reload_timeout_wait_result_setting(conn).await,
|
||||
RETENTION_PERIOD_SECS_SETTING => reload_retention_period_setting(conn).await,
|
||||
RETENTION_PERIOD_SECS_OVERRIDES_SETTING => {
|
||||
if let Err(e) = load_retention_period_overrides(db).await {
|
||||
tracing::error!("Error loading per-workspace retention overrides: {e:#}");
|
||||
}
|
||||
}
|
||||
AUDIT_LOG_RETENTION_DAYS_SETTING => {
|
||||
reload_audit_log_retention_days_setting(conn).await
|
||||
}
|
||||
|
||||
+441
-111
@@ -94,7 +94,8 @@ use windmill_common::{
|
||||
},
|
||||
KillpillSender, AUDIT_LOG_RETENTION_DAYS, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE,
|
||||
CRITICAL_ALERTS_ON_TOKEN_EXPIRY, CRITICAL_ALERT_MUTE_UI_ENABLED, CRITICAL_ERROR_CHANNELS, DB,
|
||||
DEFAULT_HUB_BASE_URL, HUB_BASE_URL, JOB_RETENTION_SECS, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
|
||||
DEFAULT_HUB_BASE_URL, HUB_BASE_URL, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED, METRICS_DEBUG_ENABLED, METRICS_ENABLED,
|
||||
MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED,
|
||||
SERVICE_LOG_RETENTION_SECS, STORE_AUDIT_LOGS_S3,
|
||||
};
|
||||
@@ -265,6 +266,12 @@ pub async fn initial_load(
|
||||
tracing::error!("Error loading preview tags override: {e:#}");
|
||||
}
|
||||
|
||||
// Load per-workspace retention overrides before the first cleanup tick so a fresh server
|
||||
// never sweeps globally without honoring configured longer-retention workspaces.
|
||||
if let Err(e) = load_retention_period_overrides(db).await {
|
||||
tracing::error!("Error loading per-workspace retention overrides: {e:#}");
|
||||
}
|
||||
|
||||
// Workspace fairness (cloud-only). Load the percentage/duration/min knobs
|
||||
// *before* the enabled flag so that `load_workspace_fairness_enabled` reads
|
||||
// current values when re-storing the pull queries.
|
||||
@@ -1339,68 +1346,73 @@ pub async fn delete_expired_items(db: &DB) -> () {
|
||||
),
|
||||
}
|
||||
|
||||
let job_retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
|
||||
if job_retention_secs > 0 {
|
||||
let batch_size = *JOB_CLEANUP_BATCH_SIZE;
|
||||
let max_batches = *JOB_CLEANUP_MAX_BATCHES;
|
||||
let cleanup_start = Instant::now();
|
||||
let mut total_deleted = 0u64;
|
||||
let mut batch_num = 0i32;
|
||||
// Watermark carried across batches so each one resumes after the rows the previous batch
|
||||
// already processed instead of re-scanning the (potentially undeletable) oldest prefix.
|
||||
let mut completed_at_floor: Option<DateTime<Utc>> = None;
|
||||
// Per-workspace retention overrides (EE-only; the cache is always empty on CE). A workspace may
|
||||
// keep jobs LONGER or SHORTER than the instance-wide window. Phase 1 sweeps globally on the
|
||||
// instance window but excludes override workspaces; Phase 2 sweeps each override workspace on its
|
||||
// own window (a sargable `workspace_id = $w` scan). The override count is capped small
|
||||
// (`MAX_RETENTION_OVERRIDE_WORKSPACES`), so Phase 2's per-workspace fan-out stays bounded.
|
||||
//
|
||||
// Deliberate simplicity/scale trade-off: a LONGER or keep-forever override lets that workspace's
|
||||
// old rows accumulate at the front of the completed_at index, and Phase 1's first batch each tick
|
||||
// scans past that retained prefix (an index scan, thanks to the sargable floor — not a Seq Scan)
|
||||
// before reaching a deletable row. This is only material at extreme scale (millions of retained
|
||||
// rows on one busy keep-forever workspace); we accept it rather than carrying a cross-tick
|
||||
// watermark, given overrides are a capped, targeted escape hatch.
|
||||
//
|
||||
// Gate the whole sweep on a confirmed-known override set: if the load never succeeded (e.g. a
|
||||
// startup DB hiccup, or malformed data), the empty cache is "unknown", not "no overrides", and
|
||||
// sweeping globally would delete jobs a longer-retention workspace configured. Retry the load
|
||||
// once here (on CE the flag is already set at startup, so this is a no-op), and skip the whole
|
||||
// job-cleanup phase this tick if still unknown — it runs again shortly.
|
||||
if !JOB_RETENTION_SECS_OVERRIDES_LOADED.load(std::sync::atomic::Ordering::Relaxed) {
|
||||
if let Err(e) = load_retention_period_overrides(db).await {
|
||||
tracing::error!("Error (re)loading per-workspace retention overrides: {e:#}");
|
||||
}
|
||||
}
|
||||
if !JOB_RETENTION_SECS_OVERRIDES_LOADED.load(std::sync::atomic::Ordering::Relaxed) {
|
||||
tracing::error!(
|
||||
"Skipping job retention cleanup this cycle: per-workspace overrides not yet loaded"
|
||||
);
|
||||
} else {
|
||||
let job_retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
|
||||
// `load_full` (owned Arc) rather than `load` (Guard): the sweep below holds this across many
|
||||
// `.await`s, and an arc_swap Guard is not meant to be held for long.
|
||||
let retention_overrides = JOB_RETENTION_SECS_OVERRIDES.load_full();
|
||||
let override_workspace_ids: Vec<String> = retention_overrides.keys().cloned().collect();
|
||||
|
||||
// Process batches until no more expired jobs or max batches reached
|
||||
loop {
|
||||
if max_batches > 0 && batch_num >= max_batches {
|
||||
tracing::debug!(
|
||||
"Job cleanup: reached max batches limit ({}), will continue next iteration",
|
||||
max_batches
|
||||
);
|
||||
break;
|
||||
// Phase 1: global sweep with the instance window, skipping override workspaces.
|
||||
if job_retention_secs > 0 {
|
||||
run_retention_cleanup(
|
||||
db,
|
||||
job_retention_secs,
|
||||
RetentionScope::GlobalExcluding(&override_workspace_ids),
|
||||
)
|
||||
.await;
|
||||
|
||||
// Clean up concurrency keys separately (not tied to specific job IDs). Kept global on
|
||||
// the instance window — concurrency keys are short-lived and not worth per-workspace
|
||||
// scoping.
|
||||
if let Err(e) = sqlx::query!(
|
||||
"DELETE FROM concurrency_key WHERE ended_at <= now() - ($1::bigint::text || ' s')::interval",
|
||||
job_retention_secs
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
{
|
||||
tracing::error!("Error deleting custom concurrency key: {:?}", e);
|
||||
}
|
||||
}
|
||||
|
||||
// Each batch runs in its own transaction to avoid long-running locks
|
||||
let batch_result =
|
||||
delete_expired_jobs_batch(db, job_retention_secs, batch_size, completed_at_floor)
|
||||
// Phase 2: each override workspace swept on its own window. A window of 0 means "keep
|
||||
// forever" for that workspace, so it is excluded from Phase 1 above and skipped here. The
|
||||
// override count is capped at MAX_RETENTION_OVERRIDE_WORKSPACES (enforced at write time), so
|
||||
// this loop runs a bounded number of scoped sweeps per pass.
|
||||
for (w_id, retention_secs) in retention_overrides.iter() {
|
||||
if *retention_secs > 0 {
|
||||
run_retention_cleanup(db, *retention_secs, RetentionScope::OnlyWorkspace(w_id))
|
||||
.await;
|
||||
|
||||
match batch_result {
|
||||
Ok((deleted_count, max_completed_at)) => {
|
||||
if deleted_count == 0 {
|
||||
// No more expired jobs to delete
|
||||
break;
|
||||
}
|
||||
completed_at_floor = max_completed_at.or(completed_at_floor);
|
||||
total_deleted += deleted_count as u64;
|
||||
batch_num += 1;
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Error in job cleanup batch {}: {:?}", batch_num, e);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if total_deleted > 0 {
|
||||
tracing::info!(
|
||||
"Job cleanup completed: deleted {} jobs in {} batches, took {:?}",
|
||||
total_deleted,
|
||||
batch_num,
|
||||
cleanup_start.elapsed()
|
||||
);
|
||||
}
|
||||
|
||||
// Clean up concurrency keys separately (not tied to specific job IDs)
|
||||
if let Err(e) = sqlx::query!(
|
||||
"DELETE FROM concurrency_key WHERE ended_at <= now() - ($1::bigint::text || ' s')::interval",
|
||||
job_retention_secs
|
||||
)
|
||||
.execute(db)
|
||||
.await
|
||||
{
|
||||
tracing::error!("Error deleting custom concurrency key: {:?}", e);
|
||||
}
|
||||
}
|
||||
|
||||
match windmill_common::trashbin::delete_expired_trash(db).await {
|
||||
@@ -1546,11 +1558,18 @@ pub async fn check_expiring_tokens(db: &DB) {
|
||||
///
|
||||
/// Returns `(jobs deleted in this batch, max completed_at deleted)`. The caller feeds the
|
||||
/// returned watermark back in as `completed_at_floor` for the next batch.
|
||||
///
|
||||
/// `only_workspace` and `exclude_workspaces` implement the per-workspace retention override and are
|
||||
/// mutually exclusive: Phase 1 passes `exclude_workspaces` (skip override workspaces, sweep the
|
||||
/// rest), Phase 2 passes `only_workspace` (sweep just that workspace on its own window). Both `None`
|
||||
/// reproduces the plain global sweep exactly. See `run_retention_cleanup` / `delete_expired_items`.
|
||||
async fn delete_expired_jobs_batch(
|
||||
db: &DB,
|
||||
job_retention_secs: i64,
|
||||
batch_size: i64,
|
||||
completed_at_floor: Option<DateTime<Utc>>,
|
||||
only_workspace: Option<&str>,
|
||||
exclude_workspaces: Option<&[String]>,
|
||||
) -> error::Result<(usize, Option<DateTime<Utc>>)> {
|
||||
let mut tx = db.begin().await?;
|
||||
|
||||
@@ -1571,65 +1590,142 @@ async fn delete_expired_jobs_batch(
|
||||
// max(completed_at) deleted by the previous batch. Re-applying it as `completed_at >= floor`
|
||||
// lets each batch resume after the rows the previous batch already processed instead of
|
||||
// re-scanning them. This matters when the oldest rows are undeletable (children of a
|
||||
// still-active root flow): without the floor the `ORDER BY completed_at ASC` scan walks that
|
||||
// same protected prefix on every batch, turning a cleanup run quadratic in prefix size.
|
||||
// still-active root flow, or override workspaces excluded from the global sweep): without the
|
||||
// floor the `ORDER BY completed_at ASC` scan walks that same protected/retained prefix on every
|
||||
// batch, turning a cleanup run quadratic in prefix size.
|
||||
// Floor only ever skips rows the current run already deleted, was protecting, or skip-locked —
|
||||
// all correctly deferred to the next run, identical to the unbounded scan's semantics.
|
||||
//
|
||||
// It is applied as `completed_at >= COALESCE($floor, '-infinity')`, NOT `$floor IS NULL OR
|
||||
// completed_at >= $floor`: the `OR ... IS NULL` disjunction is non-sargable, so the planner
|
||||
// cannot use the floor as an index lower bound and falls back to a Seq Scan of the whole table —
|
||||
// walking the entire prefix regardless of the floor. The COALESCE sentinel keeps a single cached
|
||||
// query while making the bound a plain range predicate the completed_at / composite index drives.
|
||||
//
|
||||
// Use FOR UPDATE SKIP LOCKED to avoid contention between replicas; ORDER BY completed_at
|
||||
// deletes oldest jobs first.
|
||||
let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() {
|
||||
// Common case: no old root flow is still running, so nothing is protected and the
|
||||
// v2_job join (a PK lookup per candidate) is pure overhead — skip it entirely.
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT id FROM v2_job_completed
|
||||
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND ($3::timestamptz IS NULL OR completed_at >= $3)
|
||||
ORDER BY completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
} else {
|
||||
// Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`:
|
||||
// the subquery form lets the planner build a one-time hashed SubPlan and apply it as a
|
||||
// filter on the ordered index scan, giving O(1) membership per candidate instead of a
|
||||
// per-row linear array scan (which degrades sharply when many root jobs are active). The
|
||||
// `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT jc.id FROM v2_job_completed jc
|
||||
LEFT JOIN v2_job j ON j.id = jc.id
|
||||
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)
|
||||
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
|
||||
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
|
||||
)
|
||||
ORDER BY jc.completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE OF jc SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
&active_root_job_ids,
|
||||
completed_at_floor,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
// Two orthogonal choices drive which DELETE we run:
|
||||
// - `only_workspace`: Some => a single-workspace (Phase 2) sweep. We bind `workspace_id = $n`
|
||||
// directly (no `OR $n IS NULL` guard) so the composite `(workspace_id, completed_at)` index
|
||||
// can drive the ordered scan — a sargable equality the OR-form would defeat. `None` => a
|
||||
// global (Phase 1) sweep that instead excludes override workspaces via a hashed `NOT IN
|
||||
// (SELECT ... unnest($exclude))` SubPlan (same one-time-hash trick as the active-root
|
||||
// exclusion below): O(1) membership per candidate, vs `<> ALL($exclude)`'s per-row linear
|
||||
// array scan which degrades sharply once many workspaces have overrides.
|
||||
// - `active_root_job_ids.is_empty()`: skip the `v2_job` join entirely when nothing is
|
||||
// protected (a PK lookup per candidate is pure overhead in the common case).
|
||||
let (deleted_jobs, max_completed_at) = match only_workspace {
|
||||
Some(w_id) if active_root_job_ids.is_empty() => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT id FROM v2_job_completed
|
||||
WHERE workspace_id = $4
|
||||
AND completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND completed_at >= COALESCE($3::timestamptz, '-infinity'::timestamptz)
|
||||
ORDER BY completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
w_id,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
Some(w_id) => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT jc.id FROM v2_job_completed jc
|
||||
LEFT JOIN v2_job j ON j.id = jc.id
|
||||
WHERE jc.workspace_id = $5
|
||||
AND jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND jc.completed_at >= COALESCE($4::timestamptz, '-infinity'::timestamptz)
|
||||
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
|
||||
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
|
||||
)
|
||||
ORDER BY jc.completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE OF jc SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
&active_root_job_ids,
|
||||
completed_at_floor,
|
||||
w_id,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
None if active_root_job_ids.is_empty() => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT id FROM v2_job_completed
|
||||
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND completed_at >= COALESCE($3::timestamptz, '-infinity'::timestamptz)
|
||||
AND ($4::text[] IS NULL OR workspace_id NOT IN (
|
||||
SELECT u FROM unnest($4::text[]) AS u WHERE u IS NOT NULL
|
||||
))
|
||||
ORDER BY completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
exclude_workspaces,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
None => {
|
||||
// Active-root exclusion uses `NOT IN (SELECT ... unnest($3))` rather than `!= ALL($3)`:
|
||||
// the subquery form lets the planner build a one-time hashed SubPlan and apply it as a
|
||||
// filter on the ordered index scan, giving O(1) membership per candidate instead of a
|
||||
// per-row linear array scan (which degrades sharply when many root jobs are active). The
|
||||
// `u IS NOT NULL` guard sidesteps NOT IN's null-trap semantics ($3 holds non-null PK ids).
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT jc.id FROM v2_job_completed jc
|
||||
LEFT JOIN v2_job j ON j.id = jc.id
|
||||
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND jc.completed_at >= COALESCE($4::timestamptz, '-infinity'::timestamptz)
|
||||
AND ($5::text[] IS NULL OR jc.workspace_id NOT IN (
|
||||
SELECT u FROM unnest($5::text[]) AS u WHERE u IS NOT NULL
|
||||
))
|
||||
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
|
||||
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
|
||||
)
|
||||
ORDER BY jc.completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE OF jc SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
&active_root_job_ids,
|
||||
completed_at_floor,
|
||||
exclude_workspaces,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
};
|
||||
|
||||
let deleted_count = deleted_jobs.len();
|
||||
@@ -1704,6 +1800,189 @@ async fn delete_expired_jobs_batch(
|
||||
Ok((deleted_count, max_completed_at))
|
||||
}
|
||||
|
||||
/// Which workspaces a retention cleanup run targets.
|
||||
#[derive(Debug)]
|
||||
enum RetentionScope<'a> {
|
||||
/// Sweep every workspace except the listed ones (they run in their own Phase-2 pass).
|
||||
GlobalExcluding(&'a [String]),
|
||||
/// Sweep only this single workspace, on its own retention window.
|
||||
OnlyWorkspace(&'a str),
|
||||
}
|
||||
|
||||
/// Drives the batched job-retention delete for a given `retention_secs` window and `scope`.
|
||||
/// Preserves the per-run `completed_at_floor` watermark across batches (see
|
||||
/// `delete_expired_jobs_batch`). Returns the number of jobs deleted.
|
||||
///
|
||||
/// `JOB_CLEANUP_MAX_BATCHES` bounds the batches per call, i.e. per scope. A full cleanup cycle can
|
||||
/// therefore run up to `(1 + n_override_workspaces) * max_batches` batches; the override count is
|
||||
/// capped at `MAX_RETENTION_OVERRIDE_WORKSPACES`, and any residue is picked up on the next tick.
|
||||
async fn run_retention_cleanup(db: &DB, retention_secs: i64, scope: RetentionScope<'_>) -> u64 {
|
||||
let (only_workspace, exclude_workspaces): (Option<&str>, Option<&[String]>) = match &scope {
|
||||
// An empty exclusion list binds as NULL so the guard short-circuits to the plain sweep.
|
||||
RetentionScope::GlobalExcluding(ids) => {
|
||||
(None, if ids.is_empty() { None } else { Some(*ids) })
|
||||
}
|
||||
RetentionScope::OnlyWorkspace(w_id) => (Some(*w_id), None),
|
||||
};
|
||||
|
||||
let batch_size = *JOB_CLEANUP_BATCH_SIZE;
|
||||
let max_batches = *JOB_CLEANUP_MAX_BATCHES;
|
||||
let cleanup_start = Instant::now();
|
||||
let mut total_deleted = 0u64;
|
||||
let mut batch_num = 0i32;
|
||||
// Watermark carried across batches so each one resumes after the rows the previous batch
|
||||
// already processed instead of re-scanning the (potentially undeletable) oldest prefix.
|
||||
let mut completed_at_floor: Option<DateTime<Utc>> = None;
|
||||
|
||||
// Process batches until no more expired jobs or max batches reached
|
||||
loop {
|
||||
if max_batches > 0 && batch_num >= max_batches {
|
||||
tracing::debug!(
|
||||
"Job cleanup ({scope:?}): reached max batches limit ({max_batches}), will continue next iteration"
|
||||
);
|
||||
break;
|
||||
}
|
||||
|
||||
// Each batch runs in its own transaction to avoid long-running locks
|
||||
let batch_result = delete_expired_jobs_batch(
|
||||
db,
|
||||
retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
only_workspace,
|
||||
exclude_workspaces,
|
||||
)
|
||||
.await;
|
||||
|
||||
match batch_result {
|
||||
Ok((deleted_count, max_completed_at)) => {
|
||||
if deleted_count == 0 {
|
||||
// No more expired jobs to delete
|
||||
break;
|
||||
}
|
||||
completed_at_floor = max_completed_at.or(completed_at_floor);
|
||||
total_deleted += deleted_count as u64;
|
||||
batch_num += 1;
|
||||
}
|
||||
Err(e) => {
|
||||
tracing::error!("Error in job cleanup batch {batch_num} ({scope:?}): {e:?}");
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if total_deleted > 0 {
|
||||
tracing::info!(
|
||||
"Job cleanup completed ({scope:?}): deleted {total_deleted} jobs in {batch_num} batches, took {:?}",
|
||||
cleanup_start.elapsed()
|
||||
);
|
||||
}
|
||||
|
||||
total_deleted
|
||||
}
|
||||
|
||||
/// Parses the raw `{workspace_id: seconds}` global-setting object into an override map. Returns
|
||||
/// `Err` (with the offending workspace) if ANY value is not a non-negative integer, so the caller
|
||||
/// can keep the last-good map instead of dropping just that entry — dropping a longer-retention
|
||||
/// entry would let the Phase-1 global window delete its jobs, and a negative value would silently
|
||||
/// become keep-forever (Phase 2 only sweeps `> 0`).
|
||||
#[cfg(feature = "enterprise")]
|
||||
fn parse_retention_overrides(
|
||||
map: serde_json::Map<String, serde_json::Value>,
|
||||
) -> std::result::Result<std::collections::HashMap<String, i64>, String> {
|
||||
use windmill_common::global_settings::MAX_RETENTION_OVERRIDE_WORKSPACES;
|
||||
if map.len() > MAX_RETENTION_OVERRIDE_WORKSPACES {
|
||||
return Err(format!(
|
||||
"at most {MAX_RETENTION_OVERRIDE_WORKSPACES} per-workspace retention overrides are allowed, got {}",
|
||||
map.len()
|
||||
));
|
||||
}
|
||||
let mut overrides = std::collections::HashMap::with_capacity(map.len());
|
||||
for (w_id, v) in map {
|
||||
match v.as_i64() {
|
||||
Some(secs) if secs >= 0 => {
|
||||
overrides.insert(w_id, secs);
|
||||
}
|
||||
_ => {
|
||||
return Err(format!(
|
||||
"override for '{w_id}' must be a non-negative integer number of seconds, got {v}"
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(overrides)
|
||||
}
|
||||
|
||||
/// Loads the per-workspace retention overrides from the `retention_period_secs_overrides` global
|
||||
/// setting (a JSON `{workspace_id: secs}` object) into the in-memory `JOB_RETENTION_SECS_OVERRIDES`
|
||||
/// cache, so the cleanup sweep reads them without a per-tick DB query. Enterprise-only — CE leaves
|
||||
/// the cache empty so the sweep behaves exactly as before.
|
||||
///
|
||||
/// On a load error, unexpected value shape, or malformed data the previous map is kept but
|
||||
/// `JOB_RETENTION_SECS_OVERRIDES_LOADED` is set to FALSE, marking the cache unknown. Clobbering the
|
||||
/// map to empty would let the global sweep delete jobs a workspace asked to keep longer; leaving the
|
||||
/// flag TRUE would keep the stale (possibly shorter) policy in force after a lengthened/added
|
||||
/// override fails to refresh, deleting those jobs prematurely. Marking it unknown makes the sweep
|
||||
/// fail closed — it skips and the monitor retries the load next tick until a confirmed-current state
|
||||
/// loads. `LOADED` is set true only on a valid map, explicit unset (`Ok(None)`), or CE's no-op.
|
||||
pub async fn load_retention_period_overrides(db: &DB) -> error::Result<()> {
|
||||
#[cfg(not(feature = "enterprise"))]
|
||||
{
|
||||
let _ = db;
|
||||
// Overrides are EE-only; empty is the correct, fully-known state on CE.
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
#[cfg(feature = "enterprise")]
|
||||
{
|
||||
use windmill_common::global_settings::RETENTION_PERIOD_SECS_OVERRIDES_SETTING;
|
||||
let value =
|
||||
load_value_from_global_settings(db, RETENTION_PERIOD_SECS_OVERRIDES_SETTING).await;
|
||||
match value {
|
||||
Ok(Some(serde_json::Value::Object(map))) => match parse_retention_overrides(map) {
|
||||
Ok(overrides) => {
|
||||
JOB_RETENTION_SECS_OVERRIDES.store(std::sync::Arc::new(overrides));
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED
|
||||
.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
// Malformed persisted value: we can't confirm the current override set. Keep the
|
||||
// last-good map but mark the cache unknown so the sweep fails closed (skips) and
|
||||
// retries, rather than deleting with a stale — possibly shorter — policy.
|
||||
Err(reason) => {
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED
|
||||
.store(false, std::sync::atomic::Ordering::Relaxed);
|
||||
tracing::error!(
|
||||
"Malformed per-workspace retention overrides, gating cleanup until it loads: {reason}"
|
||||
);
|
||||
}
|
||||
},
|
||||
Ok(None) => {
|
||||
// Explicit unset is a known state: no overrides.
|
||||
JOB_RETENTION_SECS_OVERRIDES
|
||||
.store(std::sync::Arc::new(std::collections::HashMap::new()));
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED
|
||||
.store(true, std::sync::atomic::Ordering::Relaxed);
|
||||
}
|
||||
// Unexpected shape / read failure: mark unknown so a lengthened or added override that
|
||||
// failed to refresh can't be missed by a sweep still running the previous policy.
|
||||
Ok(Some(other)) => {
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED
|
||||
.store(false, std::sync::atomic::Ordering::Relaxed);
|
||||
tracing::error!(
|
||||
"Per-workspace retention overrides setting is not a JSON object (got {other}); gating cleanup until it loads"
|
||||
);
|
||||
}
|
||||
Err(e) => {
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED
|
||||
.store(false, std::sync::atomic::Ordering::Relaxed);
|
||||
tracing::error!(
|
||||
"Error loading per-workspace retention overrides, gating cleanup until it loads: {e:#}"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn delete_log_files_from_disk_and_store(
|
||||
paths_to_delete: Vec<String>,
|
||||
tmp_dir: &str,
|
||||
@@ -4734,3 +5013,54 @@ async fn manage_audit_partitions(db: &DB, retention_days: i64) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(test, feature = "enterprise"))]
|
||||
mod retention_overrides_tests {
|
||||
use super::parse_retention_overrides;
|
||||
use serde_json::json;
|
||||
|
||||
fn obj(v: serde_json::Value) -> serde_json::Map<String, serde_json::Value> {
|
||||
v.as_object().unwrap().clone()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn parses_valid_map() {
|
||||
let m = parse_retention_overrides(obj(json!({"a": 3600, "b": 0}))).unwrap();
|
||||
assert_eq!(m.get("a"), Some(&3600));
|
||||
assert_eq!(m.get("b"), Some(&0)); // 0 = keep forever, allowed
|
||||
assert_eq!(m.len(), 2);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn empty_map_is_ok() {
|
||||
assert!(parse_retention_overrides(obj(json!({})))
|
||||
.unwrap()
|
||||
.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_negative() {
|
||||
// A negative value must not silently become keep-forever; the whole map is rejected.
|
||||
assert!(parse_retention_overrides(obj(json!({"a": 3600, "b": -1}))).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_non_integer() {
|
||||
assert!(parse_retention_overrides(obj(json!({"a": "3600"}))).is_err());
|
||||
assert!(parse_retention_overrides(obj(json!({"a": 3600.5}))).is_err());
|
||||
assert!(parse_retention_overrides(obj(json!({"a": null}))).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rejects_too_many_overrides() {
|
||||
use windmill_common::global_settings::MAX_RETENTION_OVERRIDE_WORKSPACES;
|
||||
let at_cap: serde_json::Map<_, _> = (0..MAX_RETENTION_OVERRIDE_WORKSPACES)
|
||||
.map(|i| (format!("ws_{i}"), json!(3600)))
|
||||
.collect();
|
||||
assert!(parse_retention_overrides(at_cap.clone()).is_ok());
|
||||
let over_cap: serde_json::Map<_, _> = (0..MAX_RETENTION_OVERRIDE_WORKSPACES + 1)
|
||||
.map(|i| (format!("ws_{i}"), json!(3600)))
|
||||
.collect();
|
||||
assert!(parse_retention_overrides(over_cap).is_err());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -22,7 +22,7 @@ fn bun_code(code: &str) -> RawCode {
|
||||
.into(),
|
||||
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
|
||||
modules: None,
|
||||
tag: None,
|
||||
tag: None,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
//! Deployed-app S3 reads authorize on-behalf of the app author and are confined
|
||||
//! to app provenance (declared keys or recent job outputs): a viewer cannot read
|
||||
//! an arbitrary `file_key` as the author. Requires the `parquet` feature — the
|
||||
//! real `apps_u/*` S3 handlers are gated on it.
|
||||
//!
|
||||
//! `base` fixture: test-user (admin, SECRET_TOKEN); test-user-2 (non-admin,
|
||||
//! SECRET_TOKEN_2, no S3 folder permission).
|
||||
#![cfg(feature = "parquet")]
|
||||
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
const ADMIN_TOKEN: &str = "SECRET_TOKEN";
|
||||
const USER_TOKEN: &str = "SECRET_TOKEN_2";
|
||||
const APP: &str = "u/test-user/s3onbehalf";
|
||||
const DECLARED: &str = "provenance/allowed.csv";
|
||||
const NON_PROVENANCE: &str = "evil/secret.csv";
|
||||
|
||||
fn client() -> reqwest::Client {
|
||||
reqwest::Client::new()
|
||||
}
|
||||
|
||||
fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
|
||||
builder.header("Authorization", format!("Bearer {}", token))
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_deployed_app_s3_onbehalf_provenance(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
let ws = format!("http://localhost:{port}/api/w/test-workspace");
|
||||
|
||||
// `on_behalf_of` is auto-set to the creator (admin) for an anonymous app, so
|
||||
// the app reads S3 as that author; `DECLARED` is the only allowlisted key.
|
||||
let resp = authed(client().post(format!("{ws}/apps/create")), ADMIN_TOKEN)
|
||||
.json(&json!({
|
||||
"path": APP,
|
||||
"summary": "s3 onbehalf test",
|
||||
"value": {},
|
||||
"policy": {
|
||||
"execution_mode": "anonymous",
|
||||
"triggerables": {},
|
||||
"allowed_s3_keys": [{ "s3_path": DECLARED }]
|
||||
}
|
||||
}))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 201, "app create: {}", resp.text().await?);
|
||||
|
||||
// GET an app-scoped S3 route as `token`. No workspace storage is configured,
|
||||
// so a request that clears the provenance gate fails later at the storage
|
||||
// lookup (or the CE OSS stub), never with "File restricted" — which is what
|
||||
// lets these assertions distinguish "gate passed" from "gate denied".
|
||||
let get = |route: &str, token: &'static str| {
|
||||
let url = format!("{ws}/apps_u/{route}");
|
||||
authed(client().get(url), token).send()
|
||||
};
|
||||
let denied = |body: &str| body.contains("File restricted");
|
||||
|
||||
// download_s3_file: author-on-behalf allowed for the declared key, denied for
|
||||
// a key the app never declared (the confused-deputy guard).
|
||||
let body = get(&format!("download_s3_file/{APP}?s3={DECLARED}"), USER_TOKEN)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(!denied(&body), "declared key must clear the gate: {body}");
|
||||
let body = get(
|
||||
&format!("download_s3_file/{APP}?s3={NON_PROVENANCE}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(denied(&body), "non-provenance key must be denied: {body}");
|
||||
|
||||
// load_table_count and load_csv_preview enforce the same gate. The preview's
|
||||
// numeric `limit`/`offset` must deserialize (regression: a flattened query
|
||||
// struct 400s on them under serde_urlencoded).
|
||||
let body = get(
|
||||
&format!("load_table_count/{APP}?file_key={DECLARED}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(
|
||||
!denied(&body),
|
||||
"table_count declared key must clear the gate: {body}"
|
||||
);
|
||||
let body = get(
|
||||
&format!("load_table_count/{APP}?file_key={NON_PROVENANCE}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(
|
||||
denied(&body),
|
||||
"table_count non-provenance key must be denied: {body}"
|
||||
);
|
||||
|
||||
let resp = get(
|
||||
&format!("load_csv_preview/{APP}?file_key={DECLARED}&limit=5&offset=0"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let body = resp.text().await?;
|
||||
assert_ne!(status, 400, "numeric limit/offset must deserialize: {body}");
|
||||
assert!(
|
||||
!denied(&body),
|
||||
"csv_preview declared key must clear the gate: {body}"
|
||||
);
|
||||
|
||||
// load_file_preview: `read_bytes_from` / `read_bytes_length` are required.
|
||||
let resp = get(
|
||||
&format!("load_file_preview/{APP}?file_key={DECLARED}"),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?;
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
400,
|
||||
"file_preview without byte range must 400: {}",
|
||||
resp.text().await?
|
||||
);
|
||||
let body = get(
|
||||
&format!(
|
||||
"load_file_preview/{APP}?file_key={DECLARED}&read_bytes_from=0&read_bytes_length=4096"
|
||||
),
|
||||
USER_TOKEN,
|
||||
)
|
||||
.await?
|
||||
.text()
|
||||
.await?;
|
||||
assert!(
|
||||
!denied(&body),
|
||||
"file_preview declared key must clear the gate: {body}"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,106 @@
|
||||
//! Regression test: renaming a script clears the OLD path's static asset
|
||||
//! usage rows.
|
||||
//!
|
||||
//! A script deploy persists its producer/consumer asset lineage as `asset`
|
||||
//! rows keyed by `usage_path = <script path>`, `usage_kind = 'script'`. A
|
||||
//! rename is a deploy with a new `path` and a `parent_hash` pointing at the
|
||||
//! version being renamed. The new path's rows are (re)written by the deploy,
|
||||
//! but the old path's rows must be cleared — otherwise the renamed script
|
||||
//! keeps lingering in the asset graph at a path where it no longer exists.
|
||||
//!
|
||||
//! The clear used to be attempted with the NEW (not-yet-inserted) script hash,
|
||||
//! which resolved to no path and cleared nothing.
|
||||
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
fn client() -> reqwest::Client {
|
||||
reqwest::Client::new()
|
||||
}
|
||||
|
||||
fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
|
||||
builder.header("Authorization", format!("Bearer {}", token))
|
||||
}
|
||||
|
||||
fn new_script_with_asset(path: &str, content: &str) -> serde_json::Value {
|
||||
json!({
|
||||
"path": path,
|
||||
"summary": "",
|
||||
"description": "",
|
||||
"content": content,
|
||||
"language": "bash",
|
||||
"assets": [{
|
||||
"path": "u/test-user/my_db",
|
||||
"kind": "resource",
|
||||
"access_type": "rw"
|
||||
}]
|
||||
})
|
||||
}
|
||||
|
||||
async fn asset_usage_paths(db: &Pool<Postgres>) -> anyhow::Result<Vec<String>> {
|
||||
Ok(sqlx::query_scalar(
|
||||
"SELECT usage_path FROM asset \
|
||||
WHERE usage_kind = 'script' AND path = $1 AND workspace_id = $2 \
|
||||
ORDER BY usage_path",
|
||||
)
|
||||
.bind("u/test-user/my_db")
|
||||
.bind("test-workspace")
|
||||
.fetch_all(db)
|
||||
.await?)
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_rename_clears_old_path_asset_usage(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
let base = format!("http://localhost:{port}/api/w/test-workspace");
|
||||
|
||||
let original_path = "u/test-user/asset_rename_orig";
|
||||
let renamed_path = "u/test-user/asset_rename_renamed";
|
||||
|
||||
// 1. Deploy a script that produces (rw) a resource asset.
|
||||
let resp = authed(
|
||||
client().post(format!("{base}/scripts/create")),
|
||||
"SECRET_TOKEN",
|
||||
)
|
||||
.json(&new_script_with_asset(original_path, "# v1"))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 201, "create should succeed");
|
||||
let original_hash: String = resp.text().await?;
|
||||
|
||||
assert_eq!(
|
||||
asset_usage_paths(&db).await?,
|
||||
vec![original_path.to_string()],
|
||||
"asset usage should be recorded at the original path"
|
||||
);
|
||||
|
||||
// 2. Rename: deploy at a new path with parent_hash = v1.
|
||||
let mut rename = new_script_with_asset(renamed_path, "# v2");
|
||||
rename["parent_hash"] = json!(original_hash);
|
||||
let resp = authed(
|
||||
client().post(format!("{base}/scripts/create")),
|
||||
"SECRET_TOKEN",
|
||||
)
|
||||
.json(&rename)
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(
|
||||
resp.status(),
|
||||
201,
|
||||
"rename should succeed: {}",
|
||||
resp.text().await?
|
||||
);
|
||||
|
||||
// 3. Only the new path retains the asset usage; the old path is cleared.
|
||||
assert_eq!(
|
||||
asset_usage_paths(&db).await?,
|
||||
vec![renamed_path.to_string()],
|
||||
"after rename the old path's asset usage must be cleared and only the new path kept"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -0,0 +1,124 @@
|
||||
//! Regression test for the `sign_s3_objects` permission bypass.
|
||||
//!
|
||||
//! Invariant: minting an S3 read signature (`apps/sign_s3_objects`) requires the
|
||||
//! CALLER to hold `S3Permission::READ` on the key. The signature is a transferable
|
||||
//! bearer capability (`validate_s3_signature` only checks HMAC + expiry), so a
|
||||
//! caller must not be able to sign a key they cannot themselves read — otherwise
|
||||
//! any workspace member could bypass the advanced S3 permission rules.
|
||||
//!
|
||||
//! Pinned against a FilesystemStorage LFS whose advanced permissions grant a
|
||||
//! non-admin READ on `allowed/*` but nothing on `secret/*`:
|
||||
//! - the non-admin CAN sign `allowed/*` (authorized), and the minted signature
|
||||
//! validates end-to-end through the presigned s3_proxy fetch route;
|
||||
//! - the non-admin CANNOT sign `secret/*` (bypass closed);
|
||||
//! Advanced S3 permissions are an enterprise feature, so this test requires the
|
||||
//! `enterprise` + `private` + `parquet` features.
|
||||
#![cfg(all(feature = "enterprise", feature = "private", feature = "parquet"))]
|
||||
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
fn client() -> reqwest::Client {
|
||||
reqwest::Client::new()
|
||||
}
|
||||
|
||||
fn authed(builder: reqwest::RequestBuilder, token: &str) -> reqwest::RequestBuilder {
|
||||
builder.header("Authorization", format!("Bearer {}", token))
|
||||
}
|
||||
|
||||
/// Configure the workspace LFS as a filesystem store rooted at `root_path`, with
|
||||
/// an advanced permission rule granting READ on `allowed/*` to everyone the glob
|
||||
/// matches (non-admins included). No rule covers `secret/*`, so it is denied.
|
||||
async fn configure_lfs(db: &Pool<Postgres>, root_path: &str) -> anyhow::Result<()> {
|
||||
let lfs_config = json!({
|
||||
"type": "FilesystemStorage",
|
||||
"root_path": root_path,
|
||||
"public_resource": null,
|
||||
"advanced_permissions": [
|
||||
{ "pattern": "allowed/*", "allow": "read" }
|
||||
]
|
||||
});
|
||||
sqlx::query!(
|
||||
"UPDATE workspace_settings SET large_file_storage = $1 WHERE workspace_id = $2",
|
||||
lfs_config,
|
||||
"test-workspace"
|
||||
)
|
||||
.execute(db)
|
||||
.await?;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
#[sqlx::test(fixtures("base"))]
|
||||
async fn test_sign_s3_objects_enforces_read_authz(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
initialize_tracing().await;
|
||||
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
let base = format!("http://localhost:{port}/api/w/test-workspace");
|
||||
|
||||
let storage_dir = tempfile::tempdir()?;
|
||||
let storage_root = storage_dir.path().to_string_lossy().to_string();
|
||||
configure_lfs(&db, &storage_root).await?;
|
||||
|
||||
// A real object so the signed fetch can stream bytes end-to-end.
|
||||
let allowed_dir = storage_dir.path().join("allowed");
|
||||
std::fs::create_dir_all(&allowed_dir)?;
|
||||
std::fs::write(allowed_dir.join("file.txt"), b"authorized payload")?;
|
||||
|
||||
// ---- CORE REGRESSION: a non-admin (test-user-2) may NOT sign a key they have
|
||||
// no READ permission on. Before the fix this returned a valid signature.
|
||||
let resp = authed(
|
||||
client().post(format!("{base}/apps/sign_s3_objects")),
|
||||
"SECRET_TOKEN_2",
|
||||
)
|
||||
.json(&json!({ "s3_objects": [{ "s3": "secret/file.txt" }] }))
|
||||
.send()
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let body = resp.text().await?;
|
||||
assert!(
|
||||
!status.is_success(),
|
||||
"non-admin must NOT be able to sign a key they cannot read (bypass): {status} {body}"
|
||||
);
|
||||
|
||||
// ---- NO OVER-BLOCKING: the same non-admin CAN sign a key their advanced
|
||||
// permissions allow them to read.
|
||||
let resp = authed(
|
||||
client().post(format!("{base}/apps/sign_s3_objects")),
|
||||
"SECRET_TOKEN_2",
|
||||
)
|
||||
.json(&json!({ "s3_objects": [{ "s3": "allowed/file.txt" }] }))
|
||||
.send()
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let signed: serde_json::Value = resp.json().await?;
|
||||
assert!(
|
||||
status.is_success(),
|
||||
"non-admin must be able to sign a key they can read: {status} {signed}"
|
||||
);
|
||||
let presigned = signed[0]["presigned"]
|
||||
.as_str()
|
||||
.expect("authorized sign must return a presigned string")
|
||||
.to_string();
|
||||
|
||||
// ---- END-TO-END: the minted signature is accepted by the fetch-side gate.
|
||||
// Hit the presigned s3_proxy route (default storage) and confirm it
|
||||
// streams the object rather than rejecting the signature.
|
||||
let fetch_url = format!("{base}/s3_proxy/_default_/allowed/file.txt?{presigned}");
|
||||
let resp = client().get(&fetch_url).send().await?;
|
||||
let status = resp.status();
|
||||
let body = resp.bytes().await?;
|
||||
assert!(
|
||||
status.is_success(),
|
||||
"signed fetch of an authorized key must succeed end-to-end: {status} {:?}",
|
||||
String::from_utf8_lossy(&body)
|
||||
);
|
||||
assert_eq!(
|
||||
body.as_ref(),
|
||||
b"authorized payload",
|
||||
"signed fetch must stream the authorized object's bytes"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
@@ -541,6 +541,16 @@ pub fn require_owner_of_path(authed: &ApiAuthed, path: &str) -> Result<()> {
|
||||
}
|
||||
if !path.is_empty() {
|
||||
let splitted = path.split("/").collect::<Vec<&str>>();
|
||||
// A valid path is at least `<kind>/<name>` (e.g. `u/alice/...`,
|
||||
// `f/folder/...`). Guard the `splitted[1]` accesses below so a
|
||||
// malformed single-segment path returns a clear error instead of
|
||||
// panicking with an out-of-bounds index.
|
||||
if splitted.len() < 2 {
|
||||
return Err(Error::BadRequest(format!(
|
||||
"Invalid path '{}': a valid path starts with 'u/<user>/' or 'f/<folder>/'",
|
||||
path
|
||||
)));
|
||||
}
|
||||
if splitted[0] == "u" {
|
||||
if splitted[1] == authed.username {
|
||||
Ok(())
|
||||
@@ -1131,6 +1141,26 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
// Regression for WIN-2157: a malformed single-segment path (e.g. a draft
|
||||
// saved at a bare `u`) must return a clear error, not panic on the
|
||||
// `splitted[1]` index. Non-admins reach this branch (admins short-circuit).
|
||||
#[test]
|
||||
fn require_owner_of_path_rejects_malformed_path_without_panicking() {
|
||||
let alice = ApiAuthed { username: "alice".into(), ..Default::default() };
|
||||
for path in ["u", "f", "g", "nonsense"] {
|
||||
let err =
|
||||
require_owner_of_path(&alice, path).expect_err("malformed path must be rejected");
|
||||
assert!(
|
||||
matches!(err, Error::BadRequest(_)),
|
||||
"expected BadRequest for '{path}', got {err:?}"
|
||||
);
|
||||
}
|
||||
// A well-formed foreign path returns the owner error, not a malformed one.
|
||||
assert!(require_owner_of_path(&alice, "u/bob/script").is_err());
|
||||
// The user's own namespace resolves.
|
||||
assert!(require_owner_of_path(&alice, "u/alice/script").is_ok());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn predicate_no_scopes_allows_all() {
|
||||
let authed = authed_with_scopes(None);
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
//! End-to-end regression test for WIN-2161.
|
||||
//!
|
||||
//! Reproduces, through real product code, the state after a database-to-external
|
||||
//! migration: a secret that was created under the database backend and then
|
||||
//! *migrated* to an external backend (Azure Key Vault). Migration writes the
|
||||
//! plaintext to the store but
|
||||
//! leaves the encrypted ciphertext in `variable.value` (it never rewrites it to
|
||||
//! a `$azure_kv:` marker). The bug: `clone_variables` only replicated
|
||||
//! marker-valued secrets, so forking left the migrated secret unreplicated and
|
||||
//! reads in the fork failed with "not found in Azure Key Vault".
|
||||
//!
|
||||
//! This drives the real `/migrate_secrets_to_azure_kv`, `/create_fork` and
|
||||
//! `variables/get_value` endpoints against a local Azure Key Vault emulator
|
||||
//! (lowkey-vault), which the `AzureKeyVaultBackend` talks to via its
|
||||
//! static-token / self-signed-cert emulator mode.
|
||||
//!
|
||||
//! Run it:
|
||||
//! ```bash
|
||||
//! podman run -d --name lowkey -p 8443:8443 \
|
||||
//! -e LOWKEY_ARGS="--LOWKEY_VAULT_NAMES=default" \
|
||||
//! docker.io/nagyesta/lowkey-vault:7.3.0
|
||||
//!
|
||||
//! RUN_AZURE_KV_TESTS=1 cargo test -p windmill-api-integration-tests \
|
||||
//! --features private,enterprise --test fork_secret_replication_azure -- --nocapture
|
||||
//! ```
|
||||
|
||||
#[cfg(all(feature = "private", feature = "enterprise"))]
|
||||
mod azure_fork {
|
||||
use serde_json::json;
|
||||
use sqlx::{Pool, Postgres};
|
||||
use windmill_common::variables::{build_crypt, encrypt};
|
||||
use windmill_test_utils::*;
|
||||
|
||||
fn client() -> reqwest::Client {
|
||||
reqwest::Client::new()
|
||||
}
|
||||
|
||||
fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder {
|
||||
builder.header("Authorization", "Bearer SECRET_TOKEN")
|
||||
}
|
||||
|
||||
fn vault_url() -> String {
|
||||
std::env::var("AZURE_KV_URL").unwrap_or_else(|_| "https://localhost:8443".to_string())
|
||||
}
|
||||
|
||||
/// The Azure settings for the emulator: a static token switches the backend
|
||||
/// into emulator mode (no Entra ID, self-signed certs accepted).
|
||||
fn azure_settings() -> serde_json::Value {
|
||||
json!({
|
||||
"vault_url": vault_url(),
|
||||
"tenant_id": "emulator-tenant",
|
||||
"client_id": "emulator-client",
|
||||
"token": "emulator-token",
|
||||
})
|
||||
}
|
||||
|
||||
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
|
||||
async fn migrated_secret_is_replicated_on_fork(db: Pool<Postgres>) -> anyhow::Result<()> {
|
||||
if std::env::var("RUN_AZURE_KV_TESTS").as_deref() != Ok("1") {
|
||||
eprintln!("skipping: set RUN_AZURE_KV_TESTS=1 and start lowkey-vault to run");
|
||||
return Ok(());
|
||||
}
|
||||
initialize_tracing().await;
|
||||
|
||||
// The Azure KV emulator persists across runs; derive unique names per run
|
||||
// so a secret written by a previous run can't mask a regression.
|
||||
let suffix = uuid::Uuid::new_v4().simple().to_string();
|
||||
let short = &suffix[..8];
|
||||
let source_ws = "test-workspace";
|
||||
let path = format!("u/test-user/db_password_{short}");
|
||||
let path = path.as_str();
|
||||
let plaintext = "s3cr3t-value";
|
||||
|
||||
let ciphertext = {
|
||||
let mc = build_crypt(&db, source_ws).await?;
|
||||
encrypt(&mc, plaintext)
|
||||
};
|
||||
sqlx::query(
|
||||
"INSERT INTO variable (workspace_id, path, value, is_secret, description, extra_perms)
|
||||
VALUES ($1, $2, $3, true, '', '{}')",
|
||||
)
|
||||
.bind(source_ws)
|
||||
.bind(path)
|
||||
.bind(&ciphertext)
|
||||
.execute(&db)
|
||||
.await?;
|
||||
|
||||
sqlx::query(
|
||||
"INSERT INTO global_settings (name, value) VALUES ('secret_backend', $1)
|
||||
ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value",
|
||||
)
|
||||
.bind(json!({
|
||||
"type": "AzureKeyVault",
|
||||
"vault_url": vault_url(),
|
||||
"tenant_id": "emulator-tenant",
|
||||
"client_id": "emulator-client",
|
||||
"token": "emulator-token",
|
||||
}))
|
||||
.execute(&db)
|
||||
.await?;
|
||||
|
||||
let server = ApiServer::start(db.clone()).await?;
|
||||
let port = server.addr.port();
|
||||
|
||||
let resp = authed(client().post(format!(
|
||||
"http://localhost:{port}/api/settings/migrate_secrets_to_azure_kv"
|
||||
)))
|
||||
.json(&azure_settings())
|
||||
.send()
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let body = resp.text().await?;
|
||||
assert_eq!(status, 200, "migrate_secrets_to_azure_kv failed: {body}");
|
||||
let report: serde_json::Value = serde_json::from_str(&body)?;
|
||||
assert!(
|
||||
report["migrated_count"].as_i64().unwrap_or(0) >= 1,
|
||||
"expected at least one migrated secret: {report}"
|
||||
);
|
||||
|
||||
// Assert the source resolves before forking, so a fork-read failure is
|
||||
// attributable to replication rather than a broken seed.
|
||||
let resp = authed(client().get(format!(
|
||||
"http://localhost:{port}/api/w/{source_ws}/variables/get_value/{path}"
|
||||
)))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 200, "source read: {}", resp.text().await?);
|
||||
assert_eq!(resp.json::<String>().await?, plaintext);
|
||||
|
||||
let fork_ws = format!("wm-fork-az{short}");
|
||||
let fork_ws = fork_ws.as_str();
|
||||
let resp = authed(client().post(format!(
|
||||
"http://localhost:{port}/api/w/{source_ws}/workspaces/create_fork"
|
||||
)))
|
||||
.json(&json!({ "id": fork_ws, "name": "Azure Fork Test" }))
|
||||
.send()
|
||||
.await?;
|
||||
assert_eq!(resp.status(), 200, "create_fork: {}", resp.text().await?);
|
||||
|
||||
// The fork resolves the secret only if it was replicated under the fork's
|
||||
// own workspace-id key in the external store.
|
||||
let resp = authed(client().get(format!(
|
||||
"http://localhost:{port}/api/w/{fork_ws}/variables/get_value/{path}"
|
||||
)))
|
||||
.send()
|
||||
.await?;
|
||||
let status = resp.status();
|
||||
let body = resp.text().await?;
|
||||
assert_eq!(
|
||||
status, 200,
|
||||
"forked secret must resolve, got {status}: {body}"
|
||||
);
|
||||
assert_eq!(
|
||||
serde_json::from_str::<String>(&body)?,
|
||||
plaintext,
|
||||
"fork should return the replicated plaintext"
|
||||
);
|
||||
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
@@ -12,6 +12,7 @@ use axum::{
|
||||
response::{IntoResponse, Response},
|
||||
Json,
|
||||
};
|
||||
use base64::Engine as _;
|
||||
use http::{HeaderMap, HeaderName, HeaderValue};
|
||||
use hyper::StatusCode;
|
||||
use serde::Deserialize;
|
||||
@@ -254,6 +255,8 @@ pub struct WindmillCompositeResult {
|
||||
windmill_content_type: Option<String>,
|
||||
#[serde(alias = "wm_headers")]
|
||||
windmill_headers: Option<HashMap<String, String>>,
|
||||
#[serde(alias = "wm_content_transfer_encoding")]
|
||||
windmill_content_transfer_encoding: Option<String>,
|
||||
result: Option<Box<RawValue>>,
|
||||
}
|
||||
|
||||
@@ -375,11 +378,13 @@ pub fn result_to_response(result: Box<RawValue>, success: bool) -> error::Result
|
||||
windmill_status_code,
|
||||
windmill_content_type,
|
||||
windmill_headers,
|
||||
windmill_content_transfer_encoding,
|
||||
result: result_value,
|
||||
}) => {
|
||||
if windmill_content_type.is_none()
|
||||
&& windmill_status_code.is_none()
|
||||
&& windmill_headers.is_none()
|
||||
&& windmill_content_transfer_encoding.is_none()
|
||||
{
|
||||
return Ok((
|
||||
if success {
|
||||
@@ -425,18 +430,54 @@ pub fn result_to_response(result: Box<RawValue>, success: bool) -> error::Result
|
||||
let serialized_json_result = result_value
|
||||
.map(|val| val.get().to_owned())
|
||||
.unwrap_or_else(String::new);
|
||||
let serialized_result =
|
||||
serde_json::from_str::<String>(serialized_json_result.as_str())
|
||||
.ok()
|
||||
.unwrap_or(serialized_json_result);
|
||||
let parsed_string =
|
||||
serde_json::from_str::<String>(serialized_json_result.as_str()).ok();
|
||||
let result_is_json_string = parsed_string.is_some();
|
||||
let serialized_result = parsed_string.unwrap_or(serialized_json_result);
|
||||
headers.insert(
|
||||
http::header::CONTENT_TYPE,
|
||||
HeaderValue::from_str(content_type.as_str()).map_err(|err| {
|
||||
Error::internal_err(format!("Invalid content type {content_type}: {err}"))
|
||||
})?,
|
||||
);
|
||||
// Invalid base64 is a hard error, never a silent fallback to the encoded text.
|
||||
match windmill_content_transfer_encoding.as_deref() {
|
||||
Some("base64") => {
|
||||
// Only a JSON string carries base64; a number/bool/null/array/object
|
||||
// must not have its raw JSON text decoded into arbitrary bytes.
|
||||
if !result_is_json_string {
|
||||
return Err(Error::ExecutionErr(
|
||||
"windmill_content_transfer_encoding \"base64\" requires result \
|
||||
to be a base64-encoded string"
|
||||
.to_string(),
|
||||
));
|
||||
}
|
||||
let decoded = base64::engine::general_purpose::STANDARD
|
||||
.decode(serialized_result.as_bytes())
|
||||
.map_err(|err| {
|
||||
Error::ExecutionErr(format!(
|
||||
"windmill_content_transfer_encoding is \"base64\" but the \
|
||||
result is not valid base64: {err}"
|
||||
))
|
||||
})?;
|
||||
return Ok((status_code_or_default, headers, decoded).into_response());
|
||||
}
|
||||
Some(other) => {
|
||||
return Err(Error::ExecutionErr(format!(
|
||||
"Unsupported windmill_content_transfer_encoding \"{other}\" \
|
||||
(only \"base64\" is supported)"
|
||||
)));
|
||||
}
|
||||
None => {}
|
||||
}
|
||||
return Ok((status_code_or_default, headers, serialized_result).into_response());
|
||||
}
|
||||
if windmill_content_transfer_encoding.is_some() {
|
||||
return Err(Error::ExecutionErr(
|
||||
"windmill_content_transfer_encoding requires windmill_content_type to be set"
|
||||
.to_string(),
|
||||
));
|
||||
}
|
||||
if let Some(result_value) = result_value {
|
||||
return Ok((status_code_or_default, headers, Json(result_value)).into_response());
|
||||
} else {
|
||||
@@ -960,3 +1001,106 @@ pub async fn push_script_job_by_path_into_queue<'c>(
|
||||
Ok((uuid, resolved_delete_secs, None))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod result_to_response_tests {
|
||||
use super::*;
|
||||
|
||||
fn raw(json: &str) -> Box<RawValue> {
|
||||
serde_json::from_str(json).expect("valid json")
|
||||
}
|
||||
|
||||
async fn body_bytes(resp: Response) -> Vec<u8> {
|
||||
axum::body::to_bytes(resp.into_body(), usize::MAX)
|
||||
.await
|
||||
.expect("read body")
|
||||
.to_vec()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn base64_result_is_decoded_to_raw_bytes() {
|
||||
// 0x00 0x01 0x02 0xFF is not valid UTF-8, so it can only survive as bytes.
|
||||
let bytes = vec![0u8, 1, 2, 255];
|
||||
let b64 = base64::engine::general_purpose::STANDARD.encode(&bytes);
|
||||
let resp = result_to_response(
|
||||
raw(&format!(
|
||||
r#"{{"wm_content_type":"application/pdf","wm_content_transfer_encoding":"base64","result":"{b64}"}}"#
|
||||
)),
|
||||
true,
|
||||
)
|
||||
.expect("response");
|
||||
|
||||
assert_eq!(resp.status(), StatusCode::OK);
|
||||
assert_eq!(
|
||||
resp.headers().get(http::header::CONTENT_TYPE).unwrap(),
|
||||
"application/pdf"
|
||||
);
|
||||
assert_eq!(body_bytes(resp).await, bytes);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn invalid_base64_is_a_hard_error() {
|
||||
let res = result_to_response(
|
||||
raw(
|
||||
r#"{"wm_content_type":"application/pdf","wm_content_transfer_encoding":"base64","result":"not valid base64!!"}"#,
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert!(res.is_err(), "invalid base64 must not silently fall back");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn base64_mode_rejects_non_string_results() {
|
||||
// A number/bool whose raw JSON text happens to be valid base64 (right length,
|
||||
// base64 alphabet) must not be decoded into bytes — it must be a hard error.
|
||||
for result in ["12345678", "true", "null", "[1,2,3]"] {
|
||||
let res = result_to_response(
|
||||
raw(&format!(
|
||||
r#"{{"wm_content_type":"application/octet-stream","wm_content_transfer_encoding":"base64","result":{result}}}"#
|
||||
)),
|
||||
true,
|
||||
);
|
||||
assert!(
|
||||
res.is_err(),
|
||||
"base64 mode must reject non-string result: {result}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unsupported_transfer_encoding_is_rejected() {
|
||||
let res = result_to_response(
|
||||
raw(
|
||||
r#"{"wm_content_type":"text/plain","wm_content_transfer_encoding":"gzip","result":"x"}"#,
|
||||
),
|
||||
true,
|
||||
);
|
||||
assert!(res.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn transfer_encoding_without_content_type_is_rejected() {
|
||||
let res = result_to_response(
|
||||
raw(r#"{"wm_content_transfer_encoding":"base64","result":"aGk="}"#),
|
||||
true,
|
||||
);
|
||||
assert!(res.is_err());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn string_result_is_still_served_verbatim() {
|
||||
// Regression: without a transfer encoding, a string result is sent as-is
|
||||
// (quotes stripped), not base64-decoded.
|
||||
let resp = result_to_response(
|
||||
raw(r#"{"wm_content_type":"text/html","result":"<h1>hi</h1>"}"#),
|
||||
true,
|
||||
)
|
||||
.expect("response");
|
||||
|
||||
assert_eq!(
|
||||
resp.headers().get(http::header::CONTENT_TYPE).unwrap(),
|
||||
"text/html"
|
||||
);
|
||||
assert_eq!(body_bytes(resp).await, b"<h1>hi</h1>");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1124,8 +1124,6 @@ async fn create_script_internal<'c>(
|
||||
.execute(&mut *tx)
|
||||
.await?;
|
||||
|
||||
clear_static_asset_usage_by_script_hash(&mut *tx, &w_id, hash).await?;
|
||||
|
||||
r
|
||||
}
|
||||
}?;
|
||||
@@ -1989,15 +1987,17 @@ async fn create_script_internal<'c>(
|
||||
// discovered by the graph endpoint directly from the per-kind trigger
|
||||
// tables, so `trigger_spec_to_row` returns None for those.
|
||||
clear_script_triggers(&mut *tx, &w_id, &ns.path, AssetUsageKind::Script).await?;
|
||||
// On rename, also drop the OLD path's trigger rows. clear is keyed by
|
||||
// path (no by-hash variant), and only `ns.path` is wiped above — without
|
||||
// this, stale `// on` edges for the old path keep matching producers and
|
||||
// would trigger a script later recreated at that path even if it has no
|
||||
// annotation (P1). (Producer/asset rows for the old path are already
|
||||
// cleared via clear_static_asset_usage_by_script_hash on the parent.)
|
||||
// On rename, also drop the OLD path's trigger and producer/asset rows.
|
||||
// Both clears are keyed by path, and only `ns.path` is (re)written above
|
||||
// (script_triggers here, asset rows via replace_static_asset_usage) — the
|
||||
// renamed script no longer lives at the old path, so without this its
|
||||
// stale `// on` edges keep matching producers (and would trigger a script
|
||||
// later recreated at that path with no annotation, P1) and its producer
|
||||
// rows keep it lingering in the asset graph.
|
||||
if let Some(ref old) = p_path_opt {
|
||||
if old != &ns.path {
|
||||
clear_script_triggers(&mut *tx, &w_id, old, AssetUsageKind::Script).await?;
|
||||
clear_static_asset_usage(&mut *tx, &w_id, old, AssetUsageKind::Script).await?;
|
||||
}
|
||||
}
|
||||
for spec in &pipeline_triggers {
|
||||
|
||||
@@ -61,7 +61,8 @@ use windmill_common::{
|
||||
AI_CONFIG_SETTING, APP_WORKSPACED_ROUTE_SETTING, AUTOMATE_USERNAME_CREATION_SETTING,
|
||||
CRITICAL_ALERT_MUTE_UI_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING, DISABLE_HUB_SETTING,
|
||||
EMAIL_DOMAIN_SETTING, ENV_SETTINGS, HTTP_ROUTE_WORKSPACED_ROUTE_SETTING,
|
||||
HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING, RUFF_CONFIG_SETTING,
|
||||
HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING, MAX_RETENTION_OVERRIDE_WORKSPACES,
|
||||
RETENTION_PERIOD_SECS_OVERRIDES_SETTING, RUFF_CONFIG_SETTING,
|
||||
WORKSPACE_FAIRNESS_DURATION_SECS_SETTING, WORKSPACE_FAIRNESS_ENABLED_SETTING,
|
||||
WORKSPACE_FAIRNESS_MAX_PERCENT_SETTING, WORKSPACE_FAIRNESS_MIN_TOTAL_SETTING,
|
||||
WS_BASE_URL_SETTING,
|
||||
@@ -1047,6 +1048,38 @@ async fn run_setting_pre_write_hook(
|
||||
}
|
||||
}
|
||||
}
|
||||
RETENTION_PERIOD_SECS_OVERRIDES_SETTING => {
|
||||
// Reject a malformed map at write time so it can never be persisted. A persisted bad
|
||||
// value (negative or non-integer) would fail to parse on the next server start and,
|
||||
// because the loader fails closed (skips cleanup until a known-good value is read),
|
||||
// silently disable ALL job-retention cleanup indefinitely. This shape check must stay in
|
||||
// sync with `parse_retention_overrides` in backend/src/monitor.rs.
|
||||
match value {
|
||||
// Clearing (delete row) is handled by the caller; allow it through.
|
||||
serde_json::Value::Null => {}
|
||||
serde_json::Value::String(s) if s.trim().is_empty() => {}
|
||||
serde_json::Value::Object(map) => {
|
||||
if map.len() > MAX_RETENTION_OVERRIDE_WORKSPACES {
|
||||
return Err(error::Error::BadRequest(format!(
|
||||
"retention_period_secs_overrides: at most {MAX_RETENTION_OVERRIDE_WORKSPACES} per-workspace overrides are allowed, got {}",
|
||||
map.len()
|
||||
)));
|
||||
}
|
||||
for (ws, v) in map {
|
||||
if !v.as_i64().is_some_and(|secs| secs >= 0) {
|
||||
return Err(error::Error::BadRequest(format!(
|
||||
"retention_period_secs_overrides: override for '{ws}' must be a non-negative integer number of seconds, got {v}"
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {
|
||||
return Err(error::Error::BadRequest(
|
||||
"retention_period_secs_overrides must be a JSON object of {workspace_id: seconds}".to_string(),
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
_ => {}
|
||||
}
|
||||
Ok(())
|
||||
|
||||
@@ -30,7 +30,10 @@ use windmill_common::error::{self};
|
||||
use windmill_common::jobs::delete_jobs;
|
||||
use windmill_common::tracing_init::{LOGS_SERVICE, TMP_WINDMILL_LOGS_SERVICE};
|
||||
use windmill_common::worker::WINDMILL_DIR;
|
||||
use windmill_common::{DB, INSTANCE_NAME, JOB_RETENTION_SECS, SERVICE_LOG_RETENTION_SECS};
|
||||
use windmill_common::{
|
||||
DB, INSTANCE_NAME, JOB_RETENTION_SECS, JOB_RETENTION_SECS_OVERRIDES,
|
||||
JOB_RETENTION_SECS_OVERRIDES_LOADED, SERVICE_LOG_RETENTION_SECS,
|
||||
};
|
||||
|
||||
use windmill_object_store::object_store_reexports::{
|
||||
ObjectStore, ObjectStoreError, Path as ObjectPath,
|
||||
@@ -321,29 +324,103 @@ async fn cleanup_job_logs(
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
) -> error::Result<()> {
|
||||
let retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
|
||||
if retention_secs <= 0 {
|
||||
|
||||
// Per-workspace retention overrides (EE). Honor them exactly like the periodic monitor sweep:
|
||||
// Phase 1 deletes on the instance window but EXCLUDES override workspaces, Phase 2 deletes each
|
||||
// override workspace on its own window. Fail closed if the override set was never loaded (e.g.
|
||||
// manual cleanup triggered right after startup) — sweeping globally with an unknown override set
|
||||
// would delete jobs a longer-retention workspace asked to keep.
|
||||
if !JOB_RETENTION_SECS_OVERRIDES_LOADED.load(std::sync::atomic::Ordering::Relaxed) {
|
||||
tracing::warn!(
|
||||
"log cleanup: per-workspace retention overrides not yet loaded; skipping job log cleanup this run"
|
||||
);
|
||||
return Ok(());
|
||||
}
|
||||
let overrides = JOB_RETENTION_SECS_OVERRIDES.load_full();
|
||||
let override_ids: Vec<String> = overrides.keys().cloned().collect();
|
||||
let exclude: Option<&[String]> = if override_ids.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(&override_ids)
|
||||
};
|
||||
|
||||
let total: i64 = sqlx::query_scalar!(
|
||||
"SELECT COUNT(*) FROM v2_job_completed
|
||||
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval",
|
||||
retention_secs,
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await?
|
||||
.unwrap_or(0);
|
||||
// Upfront total for the progress bar: Phase-1 candidates (instance window, excluding overrides)
|
||||
// plus Phase-2 candidates (each override on its own window). Collapsed to `processed` at the end.
|
||||
let mut total: i64 = if retention_secs > 0 {
|
||||
sqlx::query_scalar!(
|
||||
"SELECT COUNT(*) FROM v2_job_completed
|
||||
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND ($2::text[] IS NULL OR workspace_id NOT IN (
|
||||
SELECT u FROM unnest($2::text[]) AS u WHERE u IS NOT NULL
|
||||
))",
|
||||
retention_secs,
|
||||
exclude,
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await?
|
||||
.unwrap_or(0)
|
||||
} else {
|
||||
0
|
||||
};
|
||||
for (w_id, secs) in overrides.iter() {
|
||||
if *secs > 0 {
|
||||
total += sqlx::query_scalar!(
|
||||
"SELECT COUNT(*) FROM v2_job_completed
|
||||
WHERE workspace_id = $1
|
||||
AND completed_at <= now() - ($2::bigint::text || ' s')::interval",
|
||||
w_id,
|
||||
secs,
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await?
|
||||
.unwrap_or(0);
|
||||
}
|
||||
}
|
||||
|
||||
session.update(|p| p.total_jobs = total as u64).await;
|
||||
|
||||
if total <= 0 {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Phase 1: instance window, excluding override workspaces.
|
||||
if retention_secs > 0 {
|
||||
run_job_log_cleanup_phase(session, db, store, retention_secs, None, exclude).await?;
|
||||
}
|
||||
// Phase 2: each override workspace on its own window (0 = keep forever, skipped).
|
||||
for (w_id, secs) in overrides.iter() {
|
||||
if *secs > 0 {
|
||||
run_job_log_cleanup_phase(session, db, store, *secs, Some(w_id), None).await?;
|
||||
}
|
||||
}
|
||||
|
||||
// Collapse the total to what we actually processed — the upfront count includes jobs whose root
|
||||
// is still active (protected from deletion), so without this the progress bar would get stuck.
|
||||
session.update(|p| p.total_jobs = p.processed_jobs).await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Runs the batched job+log delete loop for one retention scope (`only_workspace` / `exclude`),
|
||||
/// deleting the returned log blobs from storage and updating progress. See `cleanup_job_logs`.
|
||||
async fn run_job_log_cleanup_phase(
|
||||
session: &Session,
|
||||
db: &DB,
|
||||
store: &Arc<dyn ObjectStore>,
|
||||
retention_secs: i64,
|
||||
only_workspace: Option<&str>,
|
||||
exclude_workspaces: Option<&[String]>,
|
||||
) -> error::Result<()> {
|
||||
let mut completed_at_floor: Option<DateTime<Utc>> = None;
|
||||
loop {
|
||||
let (deleted_count, rel_paths, max_completed_at) =
|
||||
delete_expired_jobs_batch(db, retention_secs, JOB_BATCH, completed_at_floor).await?;
|
||||
let (deleted_count, rel_paths, max_completed_at) = delete_expired_jobs_batch(
|
||||
db,
|
||||
retention_secs,
|
||||
JOB_BATCH,
|
||||
completed_at_floor,
|
||||
only_workspace,
|
||||
exclude_workspaces,
|
||||
)
|
||||
.await?;
|
||||
|
||||
if deleted_count == 0 {
|
||||
break;
|
||||
@@ -369,12 +446,6 @@ async fn cleanup_job_logs(
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
// Collapse the total to what we actually processed — the upfront count
|
||||
// includes jobs whose root is still active (protected from deletion), so
|
||||
// without this the progress bar would get stuck at e.g. 3/44.
|
||||
session.update(|p| p.total_jobs = p.processed_jobs).await;
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -386,6 +457,8 @@ async fn delete_expired_jobs_batch(
|
||||
job_retention_secs: i64,
|
||||
batch_size: i64,
|
||||
completed_at_floor: Option<DateTime<Utc>>,
|
||||
only_workspace: Option<&str>,
|
||||
exclude_workspaces: Option<&[String]>,
|
||||
) -> error::Result<(usize, Vec<String>, Option<DateTime<Utc>>)> {
|
||||
let mut tx = db.begin().await?;
|
||||
|
||||
@@ -401,53 +474,119 @@ async fn delete_expired_jobs_batch(
|
||||
|
||||
// `completed_at_floor` carries a watermark across batches so each one resumes after the rows
|
||||
// the previous batch processed instead of re-scanning the (potentially undeletable) oldest
|
||||
// prefix; the empty-active-roots branch skips the v2_job join entirely. See
|
||||
// backend/src/monitor.rs::delete_expired_jobs_batch for the full rationale.
|
||||
let (deleted_jobs, max_completed_at) = if active_root_job_ids.is_empty() {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT id FROM v2_job_completed
|
||||
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND ($3::timestamptz IS NULL OR completed_at >= $3)
|
||||
ORDER BY completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
} else {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT jc.id FROM v2_job_completed jc
|
||||
LEFT JOIN v2_job j ON j.id = jc.id
|
||||
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND ($4::timestamptz IS NULL OR jc.completed_at >= $4)
|
||||
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
|
||||
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
|
||||
)
|
||||
ORDER BY jc.completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE OF jc SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
&active_root_job_ids,
|
||||
completed_at_floor,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
// prefix; the empty-active-roots branch skips the v2_job join entirely. Applied as
|
||||
// `completed_at >= COALESCE($floor, '-infinity')` — the `$floor IS NULL OR ...` form is
|
||||
// non-sargable and forces a Seq Scan. `only_workspace` / `exclude_workspaces` scope the sweep for
|
||||
// the per-workspace retention override (Phase 1 global excluding override workspaces, Phase 2
|
||||
// per-override) — same 4-arm shape and index rationale as
|
||||
// backend/src/monitor.rs::delete_expired_jobs_batch (see there for the full rationale).
|
||||
let (deleted_jobs, max_completed_at) = match only_workspace {
|
||||
Some(w_id) if active_root_job_ids.is_empty() => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT id FROM v2_job_completed
|
||||
WHERE workspace_id = $4
|
||||
AND completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND completed_at >= COALESCE($3::timestamptz, '-infinity'::timestamptz)
|
||||
ORDER BY completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
w_id,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
Some(w_id) => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT jc.id FROM v2_job_completed jc
|
||||
LEFT JOIN v2_job j ON j.id = jc.id
|
||||
WHERE jc.workspace_id = $5
|
||||
AND jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND jc.completed_at >= COALESCE($4::timestamptz, '-infinity'::timestamptz)
|
||||
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
|
||||
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
|
||||
)
|
||||
ORDER BY jc.completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE OF jc SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
&active_root_job_ids,
|
||||
completed_at_floor,
|
||||
w_id,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
None if active_root_job_ids.is_empty() => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT id FROM v2_job_completed
|
||||
WHERE completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND completed_at >= COALESCE($3::timestamptz, '-infinity'::timestamptz)
|
||||
AND ($4::text[] IS NULL OR workspace_id NOT IN (
|
||||
SELECT u FROM unnest($4::text[]) AS u WHERE u IS NOT NULL
|
||||
))
|
||||
ORDER BY completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
completed_at_floor,
|
||||
exclude_workspaces,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
None => {
|
||||
let rows = sqlx::query!(
|
||||
"DELETE FROM v2_job_completed
|
||||
WHERE id IN (
|
||||
SELECT jc.id FROM v2_job_completed jc
|
||||
LEFT JOIN v2_job j ON j.id = jc.id
|
||||
WHERE jc.completed_at <= now() - ($1::bigint::text || ' s')::interval
|
||||
AND jc.completed_at >= COALESCE($4::timestamptz, '-infinity'::timestamptz)
|
||||
AND ($5::text[] IS NULL OR jc.workspace_id NOT IN (
|
||||
SELECT u FROM unnest($5::text[]) AS u WHERE u IS NOT NULL
|
||||
))
|
||||
AND COALESCE(j.root_job, j.flow_innermost_root_job, jc.id) NOT IN (
|
||||
SELECT u FROM unnest($3::uuid[]) AS u WHERE u IS NOT NULL
|
||||
)
|
||||
ORDER BY jc.completed_at ASC
|
||||
LIMIT $2
|
||||
FOR UPDATE OF jc SKIP LOCKED
|
||||
)
|
||||
RETURNING id, completed_at",
|
||||
job_retention_secs,
|
||||
batch_size,
|
||||
&active_root_job_ids,
|
||||
completed_at_floor,
|
||||
exclude_workspaces,
|
||||
)
|
||||
.fetch_all(&mut *tx)
|
||||
.await?;
|
||||
let max = rows.iter().map(|r| r.completed_at).max();
|
||||
(rows.into_iter().map(|r| r.id).collect::<Vec<Uuid>>(), max)
|
||||
}
|
||||
};
|
||||
|
||||
let deleted_count = deleted_jobs.len();
|
||||
@@ -542,15 +681,28 @@ async fn cleanup_s3_orphans(
|
||||
let job_retention_secs = JOB_RETENTION_SECS.load(std::sync::atomic::Ordering::Relaxed);
|
||||
let now = Utc::now();
|
||||
// Service logs always have a retention (hardcoded SERVICE_LOG_RETENTION_SECS),
|
||||
// so we scan for service-log orphans regardless of JOB_RETENTION_SECS. Job-log
|
||||
// orphans, by contrast, can only be considered expired relative to
|
||||
// JOB_RETENTION_SECS; when that is disabled we skip the job branch entirely.
|
||||
// so we scan for service-log orphans regardless of JOB_RETENTION_SECS.
|
||||
let service_cutoff = now - chrono::Duration::seconds(SERVICE_LOG_RETENTION_SECS);
|
||||
let job_cutoff = if job_retention_secs > 0 {
|
||||
Some(now - chrono::Duration::seconds(job_retention_secs))
|
||||
} else {
|
||||
None
|
||||
};
|
||||
|
||||
// Job-log orphans are only considered once past a job's effective retention window. That window
|
||||
// is the instance one OR, for an override workspace (EE), its own — and jobs orphan their logs as
|
||||
// soon as the SHORTEST applicable window elapses. Since this scan applies a single cutoff (the S3
|
||||
// path carries only the job id, not the workspace), use the MINIMUM positive window across the
|
||||
// instance window and every positive override so no window's orphans are missed. Crucially this
|
||||
// also covers a `0` (keep-forever) instance window that still has positive overrides — the case
|
||||
// where a plain global-only cutoff would skip the job branch entirely and orphan those logs
|
||||
// forever. Keep-forever windows (0) contribute nothing: their jobs are never deleted. Overrides
|
||||
// are folded in only once the cache is a known state; otherwise we fall back to the instance
|
||||
// window alone and the next run picks up any override-only orphans once the cache loads.
|
||||
let mut min_positive_window = (job_retention_secs > 0).then_some(job_retention_secs);
|
||||
if JOB_RETENTION_SECS_OVERRIDES_LOADED.load(std::sync::atomic::Ordering::Relaxed) {
|
||||
for w in JOB_RETENTION_SECS_OVERRIDES.load_full().values().copied() {
|
||||
if w > 0 {
|
||||
min_positive_window = Some(min_positive_window.map_or(w, |m| m.min(w)));
|
||||
}
|
||||
}
|
||||
}
|
||||
let job_cutoff = min_positive_window.map(|w| now - chrono::Duration::seconds(w));
|
||||
|
||||
let logs_prefix = ObjectPath::from("logs/");
|
||||
let mut stream = store.list(Some(&logs_prefix));
|
||||
|
||||
@@ -69,8 +69,9 @@ use windmill_types::s3::LargeFileStorage;
|
||||
|
||||
use hyper::StatusCode;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sqlx::{FromRow, Postgres, Transaction};
|
||||
use sqlx::{FromRow, Postgres, Row, Transaction};
|
||||
use windmill_common::oauth2::InstanceEvent;
|
||||
use windmill_common::secret_backend::{get_secret_backend, is_vault_backend_configured};
|
||||
use windmill_common::utils::not_found_if_none;
|
||||
|
||||
lazy_static::lazy_static! {
|
||||
@@ -1928,6 +1929,38 @@ mod tests {
|
||||
format!("text={}...", "é".repeat(27))
|
||||
);
|
||||
}
|
||||
|
||||
// A real NUL can't live in Rust source, so build `{"k":"<n backslashes>u0000"}`
|
||||
// by repeating backslashes: an ODD run before `u0000` is a genuine NUL escape,
|
||||
// an EVEN run is an escaped backslash then the literal text "u0000".
|
||||
fn nul_json(backslashes: usize) -> String {
|
||||
format!(r#"{{"k":"{}u0000"}}"#, "\\".repeat(backslashes))
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nul_escape_detected_only_for_odd_backslash_runs() {
|
||||
// 1 backslash: `\u0000` — a genuine NUL escape.
|
||||
assert!(json_text_has_nul_escape(&nul_json(1)));
|
||||
// 3 backslashes: escaped backslash + genuine NUL escape.
|
||||
assert!(json_text_has_nul_escape(&nul_json(3)));
|
||||
// 2 backslashes: escaped backslash then literal "u0000" (e.g. minified JS) — safe.
|
||||
assert!(!json_text_has_nul_escape(&nul_json(2)));
|
||||
// 0 backslashes: the bare token "u0000" — safe.
|
||||
assert!(!json_text_has_nul_escape(&nul_json(0)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn nul_escape_ignores_clean_values() {
|
||||
assert!(!json_text_has_nul_escape(
|
||||
r#"{"files":{"/index.tsx":"hello"}}"#
|
||||
));
|
||||
assert!(!json_text_has_nul_escape(""));
|
||||
// A later genuine NUL is still caught even after an earlier even (safe) run.
|
||||
assert!(json_text_has_nul_escape(&format!(
|
||||
r#"{{"a":"x{b}{b}u0000y","b":"z{b}u0000"}}"#,
|
||||
b = "\\"
|
||||
)));
|
||||
}
|
||||
}
|
||||
|
||||
/// Resolve a source string to PgDatabase credentials with user-scoped permission checks.
|
||||
@@ -4115,6 +4148,7 @@ async fn create_workspace(
|
||||
// their drafts would dangle as orphans.
|
||||
async fn clone_workspace_data(
|
||||
tx: &mut Transaction<'_, Postgres>,
|
||||
db: &DB,
|
||||
source_workspace_id: &str,
|
||||
target_workspace_id: &str,
|
||||
authed_email: &str,
|
||||
@@ -4146,8 +4180,8 @@ async fn clone_workspace_data(
|
||||
// Clone resources
|
||||
clone_resources(tx, source_workspace_id, target_workspace_id).await?;
|
||||
|
||||
// Clone variables with re-encryption
|
||||
clone_variables(tx, source_workspace_id, target_workspace_id).await?;
|
||||
// Clone variables (including external secret backend replication)
|
||||
clone_variables(tx, db, source_workspace_id, target_workspace_id).await?;
|
||||
|
||||
// Clone scripts with new hashes
|
||||
clone_scripts(tx, source_workspace_id, target_workspace_id).await?;
|
||||
@@ -4652,6 +4686,7 @@ async fn clone_resources(
|
||||
|
||||
async fn clone_variables(
|
||||
tx: &mut Transaction<'_, Postgres>,
|
||||
db: &DB,
|
||||
source_workspace_id: &str,
|
||||
target_workspace_id: &str,
|
||||
) -> Result<()> {
|
||||
@@ -4666,6 +4701,51 @@ async fn clone_variables(
|
||||
.execute(&mut **tx)
|
||||
.await?;
|
||||
|
||||
// With an external backend the secret lives in the store under (workspace_id,
|
||||
// path), so the row copy above leaves the fork pointing at keys that don't
|
||||
// exist. Replicate every secret, not just marker-valued ones: migration writes
|
||||
// to the store without rewriting `value` to a `$...:` marker.
|
||||
if is_vault_backend_configured(db).await? {
|
||||
let secret_variables = sqlx::query!(
|
||||
"SELECT path FROM variable
|
||||
WHERE workspace_id = $1 AND is_secret = true AND value != ''",
|
||||
target_workspace_id,
|
||||
)
|
||||
.fetch_all(&mut **tx)
|
||||
.await?;
|
||||
|
||||
let backend = get_secret_backend(db).await?;
|
||||
for variable in secret_variables {
|
||||
match backend
|
||||
.get_secret(source_workspace_id, &variable.path)
|
||||
.await
|
||||
{
|
||||
Ok(plain_value) => {
|
||||
backend
|
||||
.set_secret(target_workspace_id, &variable.path, &plain_value)
|
||||
.await
|
||||
.map_err(|e| {
|
||||
Error::internal_err(format!(
|
||||
"Failed to replicate secret variable {} to the external secret backend for the forked workspace: {e}",
|
||||
variable.path
|
||||
))
|
||||
})?;
|
||||
}
|
||||
// The source secret is unreadable (e.g. deleted out-of-band from
|
||||
// the external store), so the variable is equally broken in the
|
||||
// source workspace — don't let it block forking.
|
||||
Err(e) => {
|
||||
tracing::warn!(
|
||||
workspace_id = %source_workspace_id,
|
||||
path = %variable.path,
|
||||
error = %e,
|
||||
"Could not read secret variable from the external secret backend while forking; the forked variable will not resolve"
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -5485,6 +5565,94 @@ async fn enforce_fork_depth(
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// True if `raw` (the text form of a `json` value) contains a genuine `\u0000`
|
||||
/// NUL escape: a `u0000` preceded by an ODD run of backslashes. Mirrors the
|
||||
/// parity rule in `strip_null_chars` (windmill-api `apps.rs`) — an even run
|
||||
/// (`\\u0000`) is an escaped backslash then the literal text "u0000" (common in
|
||||
/// minified JS) and is jsonb-safe. A genuine NUL is exactly what the
|
||||
/// `json`→`jsonb` re-encode in `clone_apps` / `clone_flows` rejects with
|
||||
/// SQLSTATE 22P05.
|
||||
fn json_text_has_nul_escape(raw: &str) -> bool {
|
||||
let bytes = raw.as_bytes();
|
||||
let mut search_from = 0;
|
||||
while let Some(rel) = raw[search_from..].find("u0000") {
|
||||
let at = search_from + rel;
|
||||
let mut backslashes = 0;
|
||||
let mut j = at;
|
||||
while j > 0 && bytes[j - 1] == b'\\' {
|
||||
backslashes += 1;
|
||||
j -= 1;
|
||||
}
|
||||
if backslashes % 2 == 1 {
|
||||
return true;
|
||||
}
|
||||
search_from = at + 5;
|
||||
}
|
||||
false
|
||||
}
|
||||
|
||||
/// SQLSTATE 22P05 (`untranslatable_character`) is what Postgres raises for
|
||||
/// "unsupported Unicode escape sequence" when a `json` value carrying a genuine
|
||||
/// `\u0000` is re-encoded to `jsonb` — the exact failure the per-row `json`
|
||||
/// clones (`clone_apps`, `clone_flows`) hit when a source item holds a NUL.
|
||||
fn is_unsupported_unicode_escape(e: &Error) -> bool {
|
||||
matches!(
|
||||
e,
|
||||
Error::SqlErr { error, .. }
|
||||
if error.as_database_error().and_then(|d| d.code()).as_deref() == Some("22P05")
|
||||
)
|
||||
}
|
||||
|
||||
/// After a fork clone aborts on a NUL escape, locate the offending source items
|
||||
/// so the error can name them. Reads the committed source workspace on the pool
|
||||
/// (the clone transaction is already poisoned and unusable). Only the `json`
|
||||
/// columns re-encoded to `jsonb` during the clone can trigger the failure:
|
||||
/// `app_version.value` (clone_apps) and `flow_version.schema` (clone_flows).
|
||||
/// Best-effort — returns an empty list rather than erroring if a probe query
|
||||
/// fails, so the caller can still surface a generic message.
|
||||
async fn find_nul_escape_locations(db: &DB, workspace_id: &str) -> Vec<String> {
|
||||
let mut apps: std::collections::BTreeSet<String> = Default::default();
|
||||
if let Ok(rows) = sqlx::query(
|
||||
"SELECT a.path AS path, av.value::text AS value
|
||||
FROM app_version av JOIN app a ON a.id = av.app_id
|
||||
WHERE a.workspace_id = $1 AND av.value IS NOT NULL",
|
||||
)
|
||||
.bind(workspace_id)
|
||||
.fetch_all(db)
|
||||
.await
|
||||
{
|
||||
for row in rows {
|
||||
let value: String = row.get("value");
|
||||
if json_text_has_nul_escape(&value) {
|
||||
apps.insert(row.get::<String, _>("path"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let mut flows: std::collections::BTreeSet<String> = Default::default();
|
||||
if let Ok(rows) = sqlx::query(
|
||||
"SELECT fv.path AS path, fv.schema::text AS schema
|
||||
FROM flow_version fv
|
||||
WHERE fv.workspace_id = $1 AND fv.schema IS NOT NULL",
|
||||
)
|
||||
.bind(workspace_id)
|
||||
.fetch_all(db)
|
||||
.await
|
||||
{
|
||||
for row in rows {
|
||||
let schema: String = row.get("schema");
|
||||
if json_text_has_nul_escape(&schema) {
|
||||
flows.insert(row.get::<String, _>("path"));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
apps.into_iter()
|
||||
.map(|p| format!("app: {p}"))
|
||||
.chain(flows.into_iter().map(|p| format!("flow: {p}")))
|
||||
.collect()
|
||||
}
|
||||
|
||||
async fn create_workspace_fork(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
@@ -5635,7 +5803,39 @@ async fn create_workspace_fork(
|
||||
.await?;
|
||||
|
||||
// Clone all data from the parent workspace using Rust implementation
|
||||
clone_workspace_data(&mut tx, &parent_workspace_id, &forked_id, &authed.email).await?;
|
||||
if let Err(e) = clone_workspace_data(
|
||||
&mut tx,
|
||||
&db,
|
||||
&parent_workspace_id,
|
||||
&forked_id,
|
||||
&authed.email,
|
||||
)
|
||||
.await
|
||||
{
|
||||
// A genuine `\u0000` in a source `json` value (`app_version.value` /
|
||||
// `flow_version.schema`) aborts the clone when it is re-encoded to jsonb:
|
||||
// Postgres raises 22P05 with only "unsupported Unicode escape sequence"
|
||||
// and no hint at which item. Pinpoint the offenders so the user can fix
|
||||
// them — re-saving strips the NUL, and the usual source is a binary file
|
||||
// (e.g. `.DS_Store`) accidentally bundled into a raw app.
|
||||
if is_unsupported_unicode_escape(&e) {
|
||||
drop(tx); // release the poisoned connection before probing on the pool
|
||||
let locations = find_nul_escape_locations(&db, &parent_workspace_id).await;
|
||||
let where_clause = if locations.is_empty() {
|
||||
"The offending item could not be pinpointed — check recently edited apps and flows."
|
||||
.to_string()
|
||||
} else {
|
||||
format!("Offending item(s):\n - {}", locations.join("\n - "))
|
||||
};
|
||||
return Err(Error::BadRequest(format!(
|
||||
"Cannot fork workspace '{parent_workspace_id}': an item contains a NUL character \
|
||||
(\\u0000) that Postgres cannot store as jsonb. Re-save the item to remove it \
|
||||
(the editor strips NUL automatically), or delete the offending binary/character \
|
||||
from its source (often a file like .DS_Store bundled into a raw app). {where_clause}"
|
||||
)));
|
||||
}
|
||||
return Err(e);
|
||||
}
|
||||
|
||||
// Clone triggers and schedules unconditionally, always with mode='disabled' /
|
||||
// enabled=false. Disabled rows have no side effects (no listener
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
openapi: "3.0.3"
|
||||
|
||||
info:
|
||||
version: 1.751.0
|
||||
version: 1.756.0
|
||||
title: Windmill API
|
||||
|
||||
contact:
|
||||
@@ -11924,7 +11924,7 @@ paths:
|
||||
|
||||
/w/{workspace}/apps/sign_s3_objects:
|
||||
post:
|
||||
summary: sign s3 objects, to be used by anonymous users in public apps
|
||||
summary: sign s3 objects (caller must have S3 read permission on each key); the signed URLs can then be used by anonymous users in public apps
|
||||
operationId: signS3Objects
|
||||
tags:
|
||||
- app
|
||||
@@ -12121,6 +12121,247 @@ paths:
|
||||
schema:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/apps_u/load_file_metadata/{path}:
|
||||
get:
|
||||
summary: Load metadata of an s3 file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadFileMetadata
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: FileMetadata
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/WindmillFileMetadata"
|
||||
|
||||
/w/{workspace}/apps_u/load_file_preview/{path}:
|
||||
get:
|
||||
summary: Load a preview of an s3 file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadFilePreview
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: file_size_in_bytes
|
||||
in: query
|
||||
schema:
|
||||
type: integer
|
||||
- name: file_mime_type
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: csv_separator
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: csv_has_header
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
- name: read_bytes_from
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: integer
|
||||
- name: read_bytes_length
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: integer
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: FilePreview
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/WindmillFilePreview"
|
||||
|
||||
/w/{workspace}/apps_u/load_parquet_preview/{path}:
|
||||
get:
|
||||
summary: Load a preview of a parquet file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadParquetPreview
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: offset
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: sort_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: sort_desc
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
- name: search_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: search_term
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Parquet Preview
|
||||
content:
|
||||
application/json: {}
|
||||
|
||||
/w/{workspace}/apps_u/load_csv_preview/{path}:
|
||||
get:
|
||||
summary: Load a preview of a csv file on-behalf of the app author (deployed app)
|
||||
operationId: appLoadCsvPreview
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: offset
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: limit
|
||||
in: query
|
||||
schema:
|
||||
type: number
|
||||
- name: sort_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: sort_desc
|
||||
in: query
|
||||
schema:
|
||||
type: boolean
|
||||
- name: search_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: search_term
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: csv_separator
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Csv Preview
|
||||
content:
|
||||
application/json: {}
|
||||
|
||||
/w/{workspace}/apps_u/load_table_count/{path}:
|
||||
get:
|
||||
summary: Load the table row count on-behalf of the app author (deployed app)
|
||||
operationId: appLoadTableCount
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: search_col
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: search_term
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: Table count
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
count:
|
||||
type: number
|
||||
|
||||
/w/{workspace}/apps_u/download_s3_parquet_file_as_csv/{path}:
|
||||
get:
|
||||
summary: Download a parquet s3 file as csv on-behalf of the app author (deployed app)
|
||||
operationId: appDownloadS3ParquetFileAsCsv
|
||||
tags:
|
||||
- app
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/WorkspaceId"
|
||||
- $ref: "#/components/parameters/Path"
|
||||
- name: file_key
|
||||
in: query
|
||||
required: true
|
||||
schema:
|
||||
type: string
|
||||
- name: storage
|
||||
in: query
|
||||
schema:
|
||||
type: string
|
||||
responses:
|
||||
"200":
|
||||
description: The downloaded file
|
||||
content:
|
||||
text/csv:
|
||||
schema:
|
||||
type: string
|
||||
|
||||
/w/{workspace}/jobs/run/f/{path}:
|
||||
post:
|
||||
summary: run flow by path
|
||||
|
||||
@@ -26,6 +26,7 @@ use crate::{
|
||||
job_helpers_oss::{
|
||||
download_s3_file_internal, get_random_file_name, get_s3_resource,
|
||||
get_workspace_s3_resource_and_check_paths, upload_file_from_req, DownloadFileQuery,
|
||||
LoadCountQuery, LoadFileMetadataQuery, LoadFilePreviewQuery, LoadPreviewQuery,
|
||||
},
|
||||
users::fetch_api_authed_from_permissioned_as,
|
||||
};
|
||||
@@ -140,6 +141,18 @@ pub fn unauthed_service() -> Router {
|
||||
.route("/upload_s3_file/{*path}", post(upload_s3_file_from_app))
|
||||
.route("/delete_s3_file", delete(delete_s3_file_from_app))
|
||||
.route("/download_s3_file/{*path}", get(download_s3_file_from_app))
|
||||
.route(
|
||||
"/download_s3_parquet_file_as_csv/{*path}",
|
||||
get(app_download_s3_parquet_file_as_csv),
|
||||
)
|
||||
.route("/load_file_metadata/{*path}", get(app_load_file_metadata))
|
||||
.route("/load_file_preview/{*path}", get(app_load_file_preview))
|
||||
.route("/load_table_count/{*path}", get(app_load_table_count))
|
||||
.route(
|
||||
"/load_parquet_preview/{*path}",
|
||||
get(app_load_parquet_preview),
|
||||
)
|
||||
.route("/load_csv_preview/{*path}", get(app_load_csv_preview))
|
||||
.route("/public_app/{secret}", get(get_public_app_by_secret))
|
||||
.route("/embed_token/{secret}", get(get_app_embed_token))
|
||||
.route("/public_resource/{*path}", get(get_public_resource))
|
||||
@@ -3266,6 +3279,7 @@ struct S3TokenRequestBody {
|
||||
}
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn sign_s3_objects(
|
||||
authed: ApiAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path(w_id): Path<String>,
|
||||
Json(body): Json<S3TokenRequestBody>,
|
||||
@@ -3273,6 +3287,22 @@ async fn sign_s3_objects(
|
||||
let workspace_key = get_workspace_key(&w_id, &db).await?;
|
||||
|
||||
let futures = body.s3_objects.into_iter().map(|s3_object| async {
|
||||
// The signature this mints is a transferable bearer capability: `validate_s3_signature`
|
||||
// only checks the HMAC and expiry, so anyone who obtains the string can read this key.
|
||||
// Authorize the CALLER's own read permission before signing — otherwise any workspace
|
||||
// member (operators included) could mint a signature for any key and bypass the advanced
|
||||
// S3 permission rules. This is the fix; do NOT move the check to validation time.
|
||||
let db_with_opt_authed = DbWithOptAuthed::from_authed(&authed, db.clone(), None);
|
||||
get_workspace_s3_resource_and_check_paths(
|
||||
&db_with_opt_authed,
|
||||
Some(&authed),
|
||||
&w_id,
|
||||
s3_object.storage.clone(),
|
||||
&[(&s3_object.s3, S3Permission::READ)],
|
||||
None,
|
||||
)
|
||||
.await?;
|
||||
|
||||
let exp = (chrono::Utc::now() + chrono::Duration::hours(12)).timestamp();
|
||||
let mut message = format!("file_key={}&exp={}", s3_object.s3.clone(), exp);
|
||||
if let Some(ref storage) = s3_object.storage {
|
||||
@@ -3808,8 +3838,9 @@ async fn check_if_allowed_to_access_s3_file_from_app(
|
||||
path: &str,
|
||||
policy: &Policy,
|
||||
) -> Result<()> {
|
||||
// if anonymous, check that the file was the result of an app script ran by an anonymous user in the last 3 hours
|
||||
// otherwise, if logged in, allow any file (TODO: change that when we implement better s3 policy)
|
||||
let is_app_embed = opt_authed.as_ref().is_some_and(|authed| {
|
||||
windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref())
|
||||
});
|
||||
|
||||
if file_query.sig.is_some() {
|
||||
#[cfg(feature = "private")]
|
||||
@@ -3829,19 +3860,17 @@ async fn check_if_allowed_to_access_s3_file_from_app(
|
||||
return Err(Error::InternalErr(
|
||||
"Internal error: signature validation is not supported in open source mode".to_string(),
|
||||
));
|
||||
} else if opt_authed.as_ref().is_some_and(|authed| {
|
||||
!windmill_api_auth::scopes::has_app_embed_sentinel(authed.scopes.as_deref())
|
||||
}) {
|
||||
// A normal logged-in caller (editor / full session) may fetch any file they
|
||||
// can reach. An app embed token also carries an identity but represents
|
||||
// untrusted app JS, so it falls through to the allowlist below instead of
|
||||
// this bypass — otherwise the app could read arbitrary S3 keys the
|
||||
// viewer/on-behalf identity can see, beyond its own declared keys/outputs.
|
||||
} else if matches!(policy.execution_mode, ExecutionMode::Viewer) && !is_app_embed {
|
||||
// Viewer mode: the on-behalf identity IS the viewer, so the downstream
|
||||
// get_workspace_s3_resource_and_check_paths already bounds the read by
|
||||
// their own perms — no provenance gate (it would over-restrict). Embed
|
||||
// tokens are excluded (untrusted app JS stays confined below).
|
||||
Ok(())
|
||||
} else {
|
||||
// Anonymous viewer, or an app embed token: confine to the app's declared S3
|
||||
// keys, or files produced by THIS app's own component runs. The producing
|
||||
// identity is the embed viewer for a token, else `anonymous`.
|
||||
// Author-mode (Anonymous/Publisher) or embed token: confine to the app's
|
||||
// declared keys or files it recently produced. Without this gate a
|
||||
// logged-in viewer could launder the author's S3 perms via an arbitrary
|
||||
// file_key (confused deputy). Producing identity = caller else `anonymous`.
|
||||
let creator = opt_authed
|
||||
.as_ref()
|
||||
.map(|authed| authed.username.clone())
|
||||
@@ -3966,6 +3995,258 @@ async fn download_s3_file_from_app(
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
fn app_s3_file_query(s3: String, storage: Option<String>) -> AppS3FileQuery {
|
||||
AppS3FileQuery {
|
||||
s3,
|
||||
storage,
|
||||
sig: None,
|
||||
#[cfg(feature = "private")]
|
||||
exp: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Shared entry for every app-scoped (`apps_u/*`) S3 display op: scope-confine an
|
||||
/// app embed token, resolve the on-behalf identity per `execution_mode`, then run
|
||||
/// the provenance gate (`check_if_allowed_to_access_s3_file_from_app`) once before
|
||||
/// dispatching to the S3 helpers.
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_s3_on_behalf_and_provenance(
|
||||
db: &DB,
|
||||
path: &str,
|
||||
w_id: &str,
|
||||
opt_authed: &Option<ApiAuthed>,
|
||||
file_query: &AppS3FileQuery,
|
||||
) -> Result<crate::db::OptJobAuthed> {
|
||||
if let Some(authed) = opt_authed.as_ref() {
|
||||
check_scopes(authed, || format!("apps:read:{}", path))?;
|
||||
}
|
||||
let (on_behalf_authed, policy) =
|
||||
get_on_behalf_authed_from_app(db, path, w_id, opt_authed, None).await?;
|
||||
check_if_allowed_to_access_s3_file_from_app(db, opt_authed, file_query, w_id, path, &policy)
|
||||
.await?;
|
||||
Ok(crate::db::OptJobAuthed { authed: on_behalf_authed, job_id: None })
|
||||
}
|
||||
|
||||
// The app-scoped display ops carry the app path in the URL and everything else
|
||||
// (file_key + op args) in the query, so they avoid a second `{*path}` wildcard.
|
||||
// `LoadCountQuery` / `LoadPreviewQuery` don't include the file key (it's a path
|
||||
// param on the raw `job_helpers/*` route), so restate their fields here with the
|
||||
// file key added. Do NOT `#[serde(flatten)]` the inner struct: axum's `Query`
|
||||
// uses `serde_urlencoded`, which cannot deserialize a flattened field's typed
|
||||
// (numeric/bool) values and 400s on `limit`/`offset` — the fields must be
|
||||
// declared directly on the outer struct.
|
||||
#[cfg(feature = "parquet")]
|
||||
#[derive(Deserialize)]
|
||||
struct AppLoadCountQuery {
|
||||
file_key: String,
|
||||
search_col: Option<String>,
|
||||
search_term: Option<String>,
|
||||
storage: Option<String>,
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
impl AppLoadCountQuery {
|
||||
fn into_inner(self) -> (String, LoadCountQuery) {
|
||||
(
|
||||
self.file_key,
|
||||
LoadCountQuery {
|
||||
search_col: self.search_col,
|
||||
search_term: self.search_term,
|
||||
storage: self.storage,
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
#[derive(Deserialize)]
|
||||
struct AppLoadPreviewQuery {
|
||||
file_key: String,
|
||||
limit: Option<u32>,
|
||||
offset: Option<i64>,
|
||||
sort_col: Option<String>,
|
||||
sort_desc: Option<bool>,
|
||||
search_col: Option<String>,
|
||||
search_term: Option<String>,
|
||||
storage: Option<String>,
|
||||
csv_separator: Option<String>,
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
impl AppLoadPreviewQuery {
|
||||
fn into_inner(self) -> (String, LoadPreviewQuery) {
|
||||
(
|
||||
self.file_key,
|
||||
LoadPreviewQuery {
|
||||
limit: self.limit,
|
||||
offset: self.offset,
|
||||
sort_col: self.sort_col,
|
||||
sort_desc: self.sort_desc,
|
||||
search_col: self.search_col,
|
||||
search_term: self.search_term,
|
||||
storage: self.storage,
|
||||
csv_separator: self.csv_separator,
|
||||
},
|
||||
)
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_download_s3_parquet_file_as_csv(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<DownloadFileQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let file_query = app_s3_file_query(query.file_key.clone(), query.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
crate::job_helpers_oss::download_s3_parquet_file_as_csv_internal(
|
||||
job_authed,
|
||||
&db,
|
||||
None,
|
||||
&w_id,
|
||||
DownloadFileQuery {
|
||||
file_key: query.file_key,
|
||||
s3_resource_path: None,
|
||||
storage: query.storage,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_file_metadata(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<LoadFileMetadataQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let file_query = app_s3_file_query(query.file_key.clone(), query.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp =
|
||||
crate::job_helpers_oss::load_file_metadata_internal(job_authed, &db, &w_id, query).await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_file_preview(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<LoadFilePreviewQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let file_query = app_s3_file_query(query.file_key.clone(), query.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp =
|
||||
crate::job_helpers_oss::load_file_preview_internal(job_authed, &db, &w_id, query).await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_table_count(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<AppLoadCountQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let (file_key, inner) = query.into_inner();
|
||||
let file_query = app_s3_file_query(file_key.clone(), inner.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp =
|
||||
crate::job_helpers_oss::load_table_count_internal(job_authed, &db, &w_id, file_key, inner)
|
||||
.await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_parquet_preview(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<AppLoadPreviewQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let (file_key, inner) = query.into_inner();
|
||||
let file_query = app_s3_file_query(file_key.clone(), inner.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp = crate::job_helpers_oss::load_preview_internal(
|
||||
job_authed, &db, &w_id, file_key, inner, true,
|
||||
)
|
||||
.await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(feature = "parquet")]
|
||||
async fn app_load_csv_preview(
|
||||
OptAuthed(opt_authed): OptAuthed,
|
||||
Extension(db): Extension<DB>,
|
||||
Path((w_id, path)): Path<(String, StripPath)>,
|
||||
Query(query): Query<AppLoadPreviewQuery>,
|
||||
) -> Result<Response> {
|
||||
let path = path.to_path();
|
||||
let (file_key, inner) = query.into_inner();
|
||||
let file_query = app_s3_file_query(file_key.clone(), inner.storage.clone());
|
||||
let job_authed =
|
||||
app_s3_on_behalf_and_provenance(&db, &path, &w_id, &opt_authed, &file_query).await?;
|
||||
let resp = crate::job_helpers_oss::load_preview_internal(
|
||||
job_authed, &db, &w_id, file_key, inner, false,
|
||||
)
|
||||
.await?;
|
||||
Ok(Json(resp).into_response())
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_download_s3_parquet_file_as_csv() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_file_metadata() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_file_preview() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_table_count() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_parquet_preview() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
#[cfg(not(feature = "parquet"))]
|
||||
async fn app_load_csv_preview() -> Result<()> {
|
||||
Err(Error::BadRequest(
|
||||
"This endpoint requires the parquet feature to be enabled".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
fn get_on_behalf_of(policy: &Policy) -> Result<(String, String)> {
|
||||
let permissioned_as = policy
|
||||
.on_behalf_of
|
||||
|
||||
@@ -90,6 +90,9 @@ lazy_static::lazy_static! {
|
||||
(20260614075900, include_str!(
|
||||
"../../migrations/20260614075900_dedup_folder_labels.up.sql"
|
||||
).replace("SET search_path = public", "SET search_path FROM CURRENT").to_string()),
|
||||
(20260710073406, include_str!(
|
||||
"../../migrations/20260710073406_index_v2_job_parent_job.up.sql"
|
||||
).replace("CREATE INDEX", "CREATE INDEX CONCURRENTLY")),
|
||||
].into_iter().collect();
|
||||
}
|
||||
|
||||
|
||||
@@ -15,6 +15,7 @@ use axum::{
|
||||
use serde::{Deserialize, Serialize};
|
||||
|
||||
use windmill_common::error::JsonResult;
|
||||
use windmill_common::{JOB_RETENTION_SECS_OVERRIDES, JOB_RETENTION_SECS_OVERRIDES_LOADED};
|
||||
|
||||
use crate::db::{ApiAuthed, DB};
|
||||
use crate::utils::require_super_admin;
|
||||
@@ -96,8 +97,35 @@ pub struct ConnectionPoolInfo {
|
||||
pub pg_total_connections: i64,
|
||||
pub pg_active_connections: i64,
|
||||
pub pg_idle_connections: i64,
|
||||
pub pg_superuser_reserved_connections: i64,
|
||||
pub status: HealthLevel,
|
||||
pub message: String,
|
||||
/// Connection sizing guidance derived from the live Windmill fleet.
|
||||
pub sizing: ConnectionSizingInfo,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct ConnectionSizingInfo {
|
||||
/// Live DB-connected worker processes (distinct worker_instance pinged recently).
|
||||
pub live_worker_instances: i64,
|
||||
/// Live individual DB-connected workers across all instances.
|
||||
pub live_workers: i64,
|
||||
/// Live agent workers (HTTP-only, hold no postgres connections; excluded from the estimate).
|
||||
pub live_agent_workers: i64,
|
||||
/// Effective per-server pool ceiling: DATABASE_CONNECTIONS if set, else DEFAULT_MAX_CONNECTIONS_SERVER.
|
||||
pub server_pool_size: i64,
|
||||
/// Effective per-worker-instance pool ceiling (single-worker baseline; grows +1 per extra worker unless DATABASE_CONNECTIONS is set).
|
||||
pub worker_pool_size: i64,
|
||||
/// The DATABASE_CONNECTIONS override, if this server has one set (caps every process's pool).
|
||||
pub database_connections_override: Option<i64>,
|
||||
/// Estimated peak connections opened by all live worker instances.
|
||||
pub estimated_worker_connections: i64,
|
||||
/// Recommended max_connections floor (workers + one server + headroom).
|
||||
pub recommended_max_connections: i64,
|
||||
/// Per-additional-server increment to add to the recommendation.
|
||||
pub per_server_increment: i64,
|
||||
/// Human-readable sizing explanation.
|
||||
pub message: String,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
@@ -264,10 +292,50 @@ async fn fetch_database_size(db: &DB) -> windmill_common::error::Result<Database
|
||||
}
|
||||
|
||||
async fn fetch_job_retention(db: &DB) -> windmill_common::error::Result<JobRetentionInfo> {
|
||||
let job_row =
|
||||
sqlx::query!("SELECT MIN(completed_at) as oldest, COUNT(*) as total FROM v2_job_completed")
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
// Per-workspace retention overrides (EE) make "oldest completed job vs the instance retention"
|
||||
// wrong as a single global signal: each override workspace has its own effective window, so its
|
||||
// intentionally-retained jobs must be judged against that window — not the instance one. We
|
||||
// therefore compute the ratio per scope and report the worst:
|
||||
// - global scope: oldest job across all non-override workspaces vs the instance retention;
|
||||
// - each override workspace with a *positive* window: its own oldest job vs its own window;
|
||||
// - keep-forever (0) overrides: excluded entirely — their jobs are retained forever by design,
|
||||
// so there is no window to fall behind on.
|
||||
// The majority (no-override) path keeps the original index-driven `MIN(completed_at)` with no
|
||||
// performance change; the override paths use the completed_at / (workspace_id, completed_at)
|
||||
// indexes and only run when overrides exist.
|
||||
let overrides = JOB_RETENTION_SECS_OVERRIDES.load_full();
|
||||
let overrides_active = !overrides.is_empty()
|
||||
&& JOB_RETENTION_SECS_OVERRIDES_LOADED.load(std::sync::atomic::Ordering::Relaxed);
|
||||
|
||||
// `true_oldest` is the real table minimum across every workspace — reported verbatim in the
|
||||
// public `oldest_completed_at` field so the UI's "Oldest job" label stays honest. `global_oldest`
|
||||
// excludes override workspaces and drives only the global health ratio (override workspaces are
|
||||
// judged against their own window below).
|
||||
type OptTs = Option<chrono::DateTime<chrono::Utc>>;
|
||||
let (true_oldest, global_oldest, total): (OptTs, OptTs, i64) = if !overrides_active {
|
||||
let r = sqlx::query!(
|
||||
"SELECT MIN(completed_at) as oldest, COUNT(*) as total FROM v2_job_completed"
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
(r.oldest, r.oldest, r.total.unwrap_or(0))
|
||||
} else {
|
||||
// `MIN(...) WHERE workspace_id <> ALL(...)` is still driven by the completed_at index
|
||||
// (ascending scan, early-stop at the first non-override row); the plain `MIN(...)` is an
|
||||
// index-only scan. Both are cheap.
|
||||
let override_ids: Vec<String> = overrides.keys().cloned().collect();
|
||||
let r = sqlx::query!(
|
||||
"SELECT
|
||||
(SELECT MIN(completed_at) FROM v2_job_completed) as true_oldest,
|
||||
(SELECT MIN(completed_at) FROM v2_job_completed
|
||||
WHERE workspace_id <> ALL($1::text[])) as global_oldest,
|
||||
(SELECT COUNT(*) FROM v2_job_completed) as total",
|
||||
&override_ids,
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
(r.true_oldest, r.global_oldest, r.total.unwrap_or(0))
|
||||
};
|
||||
|
||||
let retention_row =
|
||||
sqlx::query!("SELECT value FROM global_settings WHERE name = 'retention_period_secs'")
|
||||
@@ -277,48 +345,103 @@ async fn fetch_job_retention(db: &DB) -> windmill_common::error::Result<JobReten
|
||||
let retention_period_secs: Option<i64> =
|
||||
retention_row.map(|r| r.value).and_then(|v| v.as_i64());
|
||||
|
||||
let oldest = job_row.oldest;
|
||||
let total = job_row.total.unwrap_or(0);
|
||||
// Oldest job per positive-window override workspace (one grouped seek on the
|
||||
// `(workspace_id, completed_at)` index; only workspaces that actually have rows come back).
|
||||
let positive_override_ids: Vec<String> = if overrides_active {
|
||||
overrides
|
||||
.iter()
|
||||
.filter(|(_, &secs)| secs > 0)
|
||||
.map(|(ws, _)| ws.clone())
|
||||
.collect()
|
||||
} else {
|
||||
Vec::new()
|
||||
};
|
||||
let mut per_workspace_oldest: Vec<(String, chrono::DateTime<chrono::Utc>)> = Vec::new();
|
||||
if !positive_override_ids.is_empty() {
|
||||
let rows = sqlx::query!(
|
||||
"SELECT workspace_id as \"workspace_id!\", MIN(completed_at) as oldest
|
||||
FROM v2_job_completed
|
||||
WHERE workspace_id = ANY($1::text[])
|
||||
GROUP BY workspace_id",
|
||||
&positive_override_ids,
|
||||
)
|
||||
.fetch_all(db)
|
||||
.await?;
|
||||
for r in rows {
|
||||
if let Some(oldest) = r.oldest {
|
||||
per_workspace_oldest.push((r.workspace_id, oldest));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let (status, message) = if let (Some(oldest_ts), Some(retention_secs)) =
|
||||
(oldest, retention_period_secs)
|
||||
{
|
||||
let age_secs: i64 = (chrono::Utc::now() - oldest_ts).num_seconds();
|
||||
let ratio = if retention_secs > 0 {
|
||||
age_secs as f64 / retention_secs as f64
|
||||
} else {
|
||||
0.0
|
||||
};
|
||||
// Evaluate each scope independently and report the WORST. Each scope contributes a candidate
|
||||
// (severity, level, message); the max-severity candidate wins. This keeps the global scope's
|
||||
// "no retention configured" warning visible even when a healthy override would otherwise mask it.
|
||||
let now = chrono::Utc::now();
|
||||
let ratio_status = |scope: String, ratio: f64| -> (u8, HealthLevel, String) {
|
||||
if ratio <= 2.0 {
|
||||
(
|
||||
0,
|
||||
HealthLevel::Green,
|
||||
format!(
|
||||
"Oldest job is {:.1}x the retention period. Cleanup is keeping up.",
|
||||
ratio
|
||||
),
|
||||
format!("{scope} is {ratio:.1}x the retention period. Cleanup is keeping up."),
|
||||
)
|
||||
} else if ratio <= 5.0 {
|
||||
(
|
||||
1,
|
||||
HealthLevel::Yellow,
|
||||
format!(
|
||||
"Oldest job is {:.1}x the retention period. Cleanup may be falling behind.",
|
||||
ratio
|
||||
"{scope} is {ratio:.1}x the retention period. Cleanup may be falling behind."
|
||||
),
|
||||
)
|
||||
} else {
|
||||
(HealthLevel::Red, format!("Oldest job is {:.1}x the retention period. Consider reducing retention or investigating cleanup.", ratio))
|
||||
(2, HealthLevel::Red, format!("{scope} is {ratio:.1}x the retention period. Consider reducing retention or investigating cleanup."))
|
||||
}
|
||||
} else if oldest.is_some() && retention_period_secs.is_none() {
|
||||
(
|
||||
};
|
||||
|
||||
let mut candidates: Vec<(u8, HealthLevel, String)> = Vec::new();
|
||||
// Global scope: judged against the instance retention, or flagged when non-override jobs exist
|
||||
// (`global_oldest` is `Some`) but no positive instance retention is configured. A `0` instance
|
||||
// retention means keep-forever globally, so it contributes no candidate.
|
||||
match (global_oldest, retention_period_secs) {
|
||||
(Some(oldest_ts), Some(retention_secs)) if retention_secs > 0 => {
|
||||
let ratio = (now - oldest_ts).num_seconds() as f64 / retention_secs as f64;
|
||||
candidates.push(ratio_status("Oldest job".to_string(), ratio));
|
||||
}
|
||||
(Some(_), None) => candidates.push((
|
||||
1,
|
||||
HealthLevel::Yellow,
|
||||
"No retention_period_secs configured. Old jobs will accumulate.".to_string(),
|
||||
)),
|
||||
_ => {}
|
||||
}
|
||||
// Each positive-window override, judged against its own window.
|
||||
for (ws, oldest_ts) in &per_workspace_oldest {
|
||||
if let Some(&window) = overrides.get(ws) {
|
||||
if window > 0 {
|
||||
let ratio = (now - *oldest_ts).num_seconds() as f64 / window as f64;
|
||||
candidates.push(ratio_status(format!("Workspace {ws} oldest job"), ratio));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
let (status, message) = if let Some((_, level, message)) = candidates
|
||||
.into_iter()
|
||||
.max_by_key(|(severity, _, _)| *severity)
|
||||
{
|
||||
(level, message)
|
||||
} else if total > 0 {
|
||||
// Jobs exist but none produced a candidate: every completed job lives in a keep-forever
|
||||
// scope (instance or override), so it is retained by design rather than overdue.
|
||||
(
|
||||
HealthLevel::Green,
|
||||
"Completed jobs are within their configured retention windows.".to_string(),
|
||||
)
|
||||
} else {
|
||||
(HealthLevel::Green, "No completed jobs found.".to_string())
|
||||
};
|
||||
|
||||
Ok(JobRetentionInfo {
|
||||
oldest_completed_at: oldest,
|
||||
oldest_completed_at: true_oldest,
|
||||
total_completed_jobs: total,
|
||||
retention_period_secs,
|
||||
status,
|
||||
@@ -379,6 +502,13 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result<Connec
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
|
||||
let reserved = sqlx::query_scalar!(
|
||||
r#"SELECT setting::bigint as "v!" FROM pg_settings WHERE name = 'superuser_reserved_connections'"#
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await
|
||||
.unwrap_or(0);
|
||||
|
||||
let stats_row = sqlx::query!(
|
||||
r#"SELECT
|
||||
COUNT(*) as "total!",
|
||||
@@ -390,6 +520,41 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result<Connec
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
|
||||
// Live Windmill worker fleet: each worker pings worker_ping every ~5s; a
|
||||
// window of 30s tolerates a missed ping without counting dead workers.
|
||||
// Agent workers (name prefix "ag-") talk to the API over HTTP and hold no
|
||||
// postgres pool, so they're excluded from the connection estimate and only
|
||||
// reported for context; regular DB-connected workers use the "wk-" prefix.
|
||||
let db_worker_pattern = format!("{}-%", windmill_common::utils::WORKER_NAME_PREFIX);
|
||||
let agent_worker_pattern = format!("{}-%", windmill_common::utils::AGENT_WORKER_NAME_PREFIX);
|
||||
let fleet = sqlx::query!(
|
||||
r#"SELECT
|
||||
COUNT(*) FILTER (WHERE worker LIKE $1) as "live_workers!",
|
||||
COUNT(DISTINCT worker_instance) FILTER (WHERE worker LIKE $1) as "live_instances!",
|
||||
COUNT(*) FILTER (WHERE worker LIKE $2) as "live_agent_workers!"
|
||||
FROM worker_ping
|
||||
WHERE ping_at > now() - interval '30 seconds'"#,
|
||||
db_worker_pattern,
|
||||
agent_worker_pattern,
|
||||
)
|
||||
.fetch_one(db)
|
||||
.await?;
|
||||
|
||||
// Matches how db_connect.rs reads it: when DATABASE_CONNECTIONS is set it caps
|
||||
// every process's pool (server, indexer, worker) regardless of worker count.
|
||||
let database_connections_override = std::env::var("DATABASE_CONNECTIONS")
|
||||
.ok()
|
||||
.and_then(|n| n.parse::<i64>().ok())
|
||||
.filter(|n| *n > 0);
|
||||
|
||||
let sizing = compute_connection_sizing(
|
||||
fleet.live_workers,
|
||||
fleet.live_instances,
|
||||
fleet.live_agent_workers,
|
||||
reserved,
|
||||
database_connections_override,
|
||||
);
|
||||
|
||||
let pg_max = max_row;
|
||||
let pg_total = stats_row.total;
|
||||
let pg_active = stats_row.active;
|
||||
@@ -438,11 +603,105 @@ async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result<Connec
|
||||
pg_total_connections: pg_total,
|
||||
pg_active_connections: pg_active,
|
||||
pg_idle_connections: pg_idle,
|
||||
pg_superuser_reserved_connections: reserved,
|
||||
status,
|
||||
message,
|
||||
sizing,
|
||||
})
|
||||
}
|
||||
|
||||
/// Estimate how many postgres connections the live Windmill fleet can open and
|
||||
/// derive a recommended `max_connections` floor.
|
||||
///
|
||||
/// Pool sizing mirrors `db_connect.rs`: when `DATABASE_CONNECTIONS` is set it
|
||||
/// caps *every* process's pool (server, indexer, worker) at that value, so each
|
||||
/// worker instance opens up to that many connections. Otherwise each worker
|
||||
/// instance shares a pool of `DEFAULT_MAX_CONNECTIONS_WORKER + (workers - 1)`
|
||||
/// (fleet ceiling `(worker_pool - 1) * instances + workers`) and each server
|
||||
/// opens up to `DEFAULT_MAX_CONNECTIONS_SERVER`.
|
||||
///
|
||||
/// `database_connections_override` is this server's `DATABASE_CONNECTIONS`, our
|
||||
/// best proxy for the fleet's config. Servers do not ping `worker_ping`, so we
|
||||
/// can't count them — the recommendation assumes one server and exposes the
|
||||
/// per-server increment so the operator can add capacity for the rest.
|
||||
///
|
||||
/// `live_agent_workers` is reported for context only: agent workers reach the
|
||||
/// API over HTTP and open no postgres connections, so they never contribute to
|
||||
/// the estimate.
|
||||
fn compute_connection_sizing(
|
||||
live_workers: i64,
|
||||
live_instances: i64,
|
||||
live_agent_workers: i64,
|
||||
reserved: i64,
|
||||
database_connections_override: Option<i64>,
|
||||
) -> ConnectionSizingInfo {
|
||||
// Never recommend below this floor: postgres defaults to 100 and headroom
|
||||
// for growth/bursts/psql is cheap, so 200 is a safe baseline for any fleet.
|
||||
const MIN_RECOMMENDED_MAX_CONNECTIONS: i64 = 200;
|
||||
|
||||
let default_server_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_SERVER as i64;
|
||||
let default_worker_pool = windmill_common::DEFAULT_MAX_CONNECTIONS_WORKER as i64;
|
||||
|
||||
let (server_pool, worker_pool_size, estimated_worker_connections) =
|
||||
match database_connections_override {
|
||||
// Override caps every process identically; per-instance pool is the override.
|
||||
Some(n) => (n, n, n * live_instances),
|
||||
None => (
|
||||
default_server_pool,
|
||||
default_worker_pool,
|
||||
(default_worker_pool - 1) * live_instances + live_workers,
|
||||
),
|
||||
};
|
||||
|
||||
// Workers + one server, plus 20% headroom and the superuser reserve, so the
|
||||
// recommendation leaves room for psql/monitoring sessions and bursts, then
|
||||
// floored at MIN_RECOMMENDED_MAX_CONNECTIONS.
|
||||
let base = estimated_worker_connections + server_pool;
|
||||
let recommended = ((((base as f64) * 1.20).ceil() as i64) + reserved.max(3))
|
||||
.max(MIN_RECOMMENDED_MAX_CONNECTIONS);
|
||||
|
||||
let pool_source = if database_connections_override.is_some() {
|
||||
format!("DATABASE_CONNECTIONS={server_pool}")
|
||||
} else {
|
||||
"defaults, configurable via DATABASE_CONNECTIONS".to_string()
|
||||
};
|
||||
|
||||
let message = if live_instances == 0 {
|
||||
format!(
|
||||
"No live workers detected. Each Windmill server and worker instance opens up to {server_pool} connections ({pool_source}). Size max_connections as (servers + worker instances) × {server_pool} + ~20% headroom, and at least {MIN_RECOMMENDED_MAX_CONNECTIONS}."
|
||||
)
|
||||
} else {
|
||||
let per_instance = if database_connections_override.is_some() {
|
||||
format!("each instance up to {worker_pool_size}")
|
||||
} else {
|
||||
format!("each instance up to {worker_pool_size}, +1 per extra worker")
|
||||
};
|
||||
let agent_note = if live_agent_workers > 0 {
|
||||
format!(
|
||||
" ({live_agent_workers} agent worker(s) excluded — they use HTTP, not postgres connections.)"
|
||||
)
|
||||
} else {
|
||||
String::new()
|
||||
};
|
||||
format!(
|
||||
"{live_workers} live worker(s) across {live_instances} instance(s) can open up to ~{estimated_worker_connections} connections ({per_instance}; {pool_source}). Each Windmill server adds up to {server_pool}. Recommended max_connections ≥ {recommended} for a single server; add {server_pool} per additional server.{agent_note}"
|
||||
)
|
||||
};
|
||||
|
||||
ConnectionSizingInfo {
|
||||
live_worker_instances: live_instances,
|
||||
live_workers,
|
||||
live_agent_workers,
|
||||
server_pool_size: server_pool,
|
||||
worker_pool_size,
|
||||
database_connections_override,
|
||||
estimated_worker_connections,
|
||||
recommended_max_connections: recommended,
|
||||
per_server_increment: server_pool,
|
||||
message,
|
||||
}
|
||||
}
|
||||
|
||||
async fn fetch_table_maintenance(
|
||||
db: &DB,
|
||||
) -> windmill_common::error::Result<Vec<TableMaintenanceInfo>> {
|
||||
@@ -638,3 +897,91 @@ async fn fetch_datatables(db: &DB) -> windmill_common::error::Result<Vec<Datatab
|
||||
result.sort_by(|a, b| b.size_bytes.cmp(&a.size_bytes));
|
||||
Ok(result)
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::compute_connection_sizing;
|
||||
|
||||
#[test]
|
||||
fn no_workers_reports_defaults_and_floors_at_200() {
|
||||
let s = compute_connection_sizing(0, 0, 0, 3, None);
|
||||
assert_eq!(s.live_workers, 0);
|
||||
assert_eq!(s.live_worker_instances, 0);
|
||||
assert_eq!(s.live_agent_workers, 0);
|
||||
assert_eq!(s.estimated_worker_connections, 0);
|
||||
assert_eq!(s.server_pool_size, 50);
|
||||
assert_eq!(s.worker_pool_size, 5);
|
||||
assert_eq!(s.database_connections_override, None);
|
||||
assert_eq!(s.per_server_increment, 50);
|
||||
// ceil((0 + 50) * 1.20) + 3 = 63, floored up to 200.
|
||||
assert_eq!(s.recommended_max_connections, 200);
|
||||
assert!(s.message.contains("No live workers"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn single_worker_single_instance_floors_at_200() {
|
||||
let s = compute_connection_sizing(1, 1, 0, 3, None);
|
||||
// (5 - 1) * 1 instance + 1 worker = 5
|
||||
assert_eq!(s.estimated_worker_connections, 5);
|
||||
// ceil((5 + 50) * 1.20) + 3 = 66 + 3 = 69, floored up to 200.
|
||||
assert_eq!(s.recommended_max_connections, 200);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multi_worker_instances_sum_per_instance_pools() {
|
||||
// Two instances, 5 workers total: pools are (4 + w_i) summed = 4*2 + 5 = 13.
|
||||
let s = compute_connection_sizing(5, 2, 0, 3, None);
|
||||
assert_eq!(s.estimated_worker_connections, 13);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn large_fleet_exceeds_floor_with_margin_and_reserve() {
|
||||
// 300 workers across 10 instances: (5-1)*10 + 300 = 340 worker connections.
|
||||
let s = compute_connection_sizing(300, 10, 0, 3, None);
|
||||
assert_eq!(s.estimated_worker_connections, 340);
|
||||
// ceil((340 + 50) * 1.20) + 3 = ceil(468.0) + 3 = 471, above the 200 floor.
|
||||
assert_eq!(s.recommended_max_connections, 471);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn reserved_above_default_widens_recommendation() {
|
||||
// Big enough fleet that the floor doesn't mask the reserved contribution.
|
||||
let base = compute_connection_sizing(300, 10, 0, 3, None);
|
||||
let high = compute_connection_sizing(300, 10, 0, 20, None);
|
||||
assert_eq!(
|
||||
high.recommended_max_connections - base.recommended_max_connections,
|
||||
17
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn database_connections_override_caps_every_process() {
|
||||
// DATABASE_CONNECTIONS=100 means each instance opens up to 100, not the
|
||||
// default 5-based estimate. 5 instances -> 500 worker connections.
|
||||
let s = compute_connection_sizing(20, 5, 0, 3, Some(100));
|
||||
assert_eq!(s.server_pool_size, 100);
|
||||
assert_eq!(s.worker_pool_size, 100);
|
||||
assert_eq!(s.database_connections_override, Some(100));
|
||||
assert_eq!(s.estimated_worker_connections, 500);
|
||||
assert_eq!(s.per_server_increment, 100);
|
||||
// ceil((500 + 100) * 1.20) + 3 = 720 + 3 = 723.
|
||||
assert_eq!(s.recommended_max_connections, 723);
|
||||
assert!(s.message.contains("DATABASE_CONNECTIONS=100"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn agent_workers_are_excluded_from_the_estimate() {
|
||||
// 50 agent workers alongside 2 DB workers/2 instances: only the DB
|
||||
// workers count toward connections; the agent count is reported.
|
||||
let s = compute_connection_sizing(2, 2, 50, 3, None);
|
||||
assert_eq!(s.live_agent_workers, 50);
|
||||
// (5 - 1) * 2 + 2 = 10, agent workers contribute nothing.
|
||||
assert_eq!(s.estimated_worker_connections, 10);
|
||||
let without_agents = compute_connection_sizing(2, 2, 0, 3, None);
|
||||
assert_eq!(
|
||||
s.recommended_max_connections,
|
||||
without_agents.recommended_max_connections
|
||||
);
|
||||
assert!(s.message.contains("50 agent worker(s) excluded"));
|
||||
}
|
||||
}
|
||||
|
||||
@@ -120,7 +120,11 @@ async fn list_drafts(
|
||||
.await
|
||||
{
|
||||
Ok(()) => true,
|
||||
Err(Error::NotAuthorized(_)) => false,
|
||||
// A stored draft can sit at an unwritable path — unauthorized,
|
||||
// or malformed (`BadRequest`; the `draft` table has no path
|
||||
// constraint). Either way it's not writable, and one bad row
|
||||
// must not 400 the whole listing.
|
||||
Err(Error::NotAuthorized(_)) | Err(Error::BadRequest(_)) => false,
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
out.push(row);
|
||||
@@ -745,8 +749,17 @@ async fn require_can_write_path(
|
||||
return Ok(());
|
||||
}
|
||||
}
|
||||
// A path without a recognized namespace prefix (u/, f/, g/) can never be
|
||||
// writable — no namespace rule and no deployed row can apply — so report it
|
||||
// as malformed rather than as a plain permission denial.
|
||||
if !(path.starts_with("u/") || path.starts_with("f/") || path.starts_with("g/")) {
|
||||
return Err(Error::BadRequest(format!(
|
||||
"Invalid path '{path}': a valid path starts with 'u/<user>/', 'f/<folder>/' or 'g/<group>/'"
|
||||
)));
|
||||
}
|
||||
Err(Error::NotAuthorized(format!(
|
||||
"you don't have write permission on {path}"
|
||||
"You don't have write permission on '{path}'. It must be in your own 'u/{}/' namespace, or in a folder ('f/<folder>/') or group ('g/<group>/') you can write to.",
|
||||
authed.username
|
||||
)))
|
||||
}
|
||||
|
||||
|
||||
@@ -217,6 +217,130 @@ pub struct DeleteS3FileQuery {
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
// Stubs for the app-scoped S3 display ops (mirrors the EE `*_internal` helpers +
|
||||
// their query/response structs). Only compiled for a CE build with `parquet` but
|
||||
// without `private`; the real implementations live in `job_helpers_ee.rs`.
|
||||
#[cfg(all(feature = "parquet", not(feature = "private")))]
|
||||
mod app_s3_display_stubs {
|
||||
use super::*;
|
||||
use serde::Serialize;
|
||||
use serde_json::value::RawValue;
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadFileMetadataQuery {
|
||||
pub file_key: String,
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct LoadFileMetadataResponse {}
|
||||
|
||||
// Mirror the EE query's required/optional fields so the CE build enforces the
|
||||
// same query contract (e.g. the mandatory byte range) at the extraction layer.
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadFilePreviewQuery {
|
||||
pub storage: Option<String>,
|
||||
pub file_key: String,
|
||||
pub file_size_in_bytes: Option<u64>,
|
||||
pub file_mime_type: Option<String>,
|
||||
pub csv_separator: Option<String>,
|
||||
pub csv_has_header: Option<bool>,
|
||||
pub read_bytes_from: u64,
|
||||
pub read_bytes_length: u64,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct LoadFilePreviewResponse {}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadCountQuery {
|
||||
pub search_col: Option<String>,
|
||||
pub search_term: Option<String>,
|
||||
pub storage: Option<String>,
|
||||
}
|
||||
|
||||
#[derive(Serialize)]
|
||||
pub struct TableCount {}
|
||||
|
||||
#[derive(Deserialize)]
|
||||
#[allow(dead_code)]
|
||||
pub struct LoadPreviewQuery {
|
||||
pub limit: Option<u32>,
|
||||
pub offset: Option<i64>,
|
||||
pub sort_col: Option<String>,
|
||||
pub sort_desc: Option<bool>,
|
||||
pub search_col: Option<String>,
|
||||
pub search_term: Option<String>,
|
||||
pub storage: Option<String>,
|
||||
pub csv_separator: Option<String>,
|
||||
}
|
||||
|
||||
pub async fn load_file_metadata_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_query: LoadFileMetadataQuery,
|
||||
) -> error::Result<LoadFileMetadataResponse> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_file_preview_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_query: LoadFilePreviewQuery,
|
||||
) -> error::Result<LoadFilePreviewResponse> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_table_count_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_file_key: String,
|
||||
_query: LoadCountQuery,
|
||||
) -> error::Result<TableCount> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn load_preview_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_w_id: &str,
|
||||
_file_key: String,
|
||||
_query: LoadPreviewQuery,
|
||||
_is_parquet: bool,
|
||||
) -> error::Result<Box<RawValue>> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
|
||||
pub async fn download_s3_parquet_file_as_csv_internal(
|
||||
_authed: OptJobAuthed,
|
||||
_db: &DB,
|
||||
_user_db: Option<UserDB>,
|
||||
_w_id: &str,
|
||||
_query: DownloadFileQuery,
|
||||
) -> error::Result<Response> {
|
||||
Err(error::Error::internal_err(
|
||||
"Not implemented in Windmill's Open Source repository".to_string(),
|
||||
))
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(all(feature = "parquet", not(feature = "private")))]
|
||||
pub use app_s3_display_stubs::*;
|
||||
|
||||
#[cfg(not(feature = "private"))]
|
||||
pub async fn get_workspace_s3_resource_and_check_paths<'c>(
|
||||
_db_with_opt_authed: &DbWithOptAuthed<'c, ApiAuthed>,
|
||||
|
||||
@@ -484,8 +484,13 @@ pub async fn run_server(
|
||||
add_www_authenticate_header, add_www_authenticate_header_gateway,
|
||||
extract_workspace_from_token,
|
||||
};
|
||||
let (mcp_router, mcp_cancellation_token) =
|
||||
setup_mcp_server(db.clone(), user_db, _base_internal_url.clone()).await?;
|
||||
let (mcp_router, mcp_cancellation_token) = setup_mcp_server(
|
||||
db.clone(),
|
||||
user_db,
|
||||
_base_internal_url.clone(),
|
||||
auth_cache.clone(),
|
||||
)
|
||||
.await?;
|
||||
// Workspace-scoped MCP router
|
||||
let workspaced_mcp_router = mcp_router
|
||||
.clone()
|
||||
|
||||
@@ -10,10 +10,11 @@ use windmill_common::{db::UserDB, utils::StripPath, DB};
|
||||
use windmill_mcp::common::schema::enrich_resource_schemas;
|
||||
use windmill_mcp::common::transform::apply_key_transformation;
|
||||
use windmill_mcp::common::types::{
|
||||
FlowInfo, HubScriptInfo, ResourceInfo, ResourceType, SchemaType, ScriptInfo,
|
||||
FlowInfo, HubScriptInfo, ResourceInfo, ResourceType, SchemaType, ScriptInfo, WorkspaceInfo,
|
||||
};
|
||||
use windmill_mcp::server::{BackendResult, EndpointTool, ErrorData, McpBackend};
|
||||
|
||||
use crate::auth::AuthCache;
|
||||
use crate::db::ApiAuthed;
|
||||
use crate::jobs::{
|
||||
run_wait_result_flow_by_path_internal, run_wait_result_script_by_path_internal, RunJobQuery,
|
||||
@@ -31,7 +32,8 @@ use std::time::Duration;
|
||||
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use windmill_mcp::server::{
|
||||
LocalSessionManager, Runner, StreamableHttpServerConfig, StreamableHttpService,
|
||||
LocalSessionManager, McpToken, MultiWorkspaceMcp, Runner, StreamableHttpServerConfig,
|
||||
StreamableHttpService,
|
||||
};
|
||||
use windmill_mcp::WorkspaceId;
|
||||
|
||||
@@ -53,11 +55,17 @@ pub struct WindmillBackend {
|
||||
pub db: DB,
|
||||
pub user_db: UserDB,
|
||||
pub base_internal_url: String,
|
||||
pub auth_cache: Arc<AuthCache>,
|
||||
}
|
||||
|
||||
impl WindmillBackend {
|
||||
pub fn new(db: DB, user_db: UserDB, base_internal_url: String) -> Self {
|
||||
Self { db, user_db, base_internal_url }
|
||||
pub fn new(
|
||||
db: DB,
|
||||
user_db: UserDB,
|
||||
base_internal_url: String,
|
||||
auth_cache: Arc<AuthCache>,
|
||||
) -> Self {
|
||||
Self { db, user_db, base_internal_url, auth_cache }
|
||||
}
|
||||
}
|
||||
|
||||
@@ -305,6 +313,8 @@ impl McpBackend for WindmillBackend {
|
||||
args_map,
|
||||
&endpoint_tool.body_schema,
|
||||
&endpoint_tool.body_field_renames,
|
||||
&endpoint_tool.path_params_schema,
|
||||
&endpoint_tool.query_params_schema,
|
||||
);
|
||||
|
||||
// Create and execute request
|
||||
@@ -338,6 +348,57 @@ impl McpBackend for WindmillBackend {
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_accessible_workspaces(
|
||||
&self,
|
||||
auth: &ApiAuthed,
|
||||
) -> BackendResult<Vec<WorkspaceInfo>> {
|
||||
// A superadmin can act in every workspace and often has no explicit `usr`
|
||||
// membership row (matching resolve_workspace_auth, which authorizes any
|
||||
// workspace for a superadmin), so list them all. Everyone else is limited
|
||||
// to the workspaces they are a member of.
|
||||
let workspaces = if auth.is_admin {
|
||||
sqlx::query_as!(
|
||||
WorkspaceInfo,
|
||||
"SELECT id, name FROM workspace WHERE deleted = false ORDER BY name",
|
||||
)
|
||||
.fetch_all(&self.db)
|
||||
.await
|
||||
} else {
|
||||
sqlx::query_as!(
|
||||
WorkspaceInfo,
|
||||
"SELECT workspace.id, workspace.name
|
||||
FROM workspace
|
||||
JOIN usr ON usr.workspace_id = workspace.id
|
||||
WHERE usr.email = $1 AND usr.disabled = false AND workspace.deleted = false
|
||||
ORDER BY workspace.name",
|
||||
auth.email,
|
||||
)
|
||||
.fetch_all(&self.db)
|
||||
.await
|
||||
};
|
||||
|
||||
workspaces.map_err(|e| ErrorData::internal_error(e.to_string(), None))
|
||||
}
|
||||
|
||||
async fn resolve_workspace_auth(
|
||||
&self,
|
||||
token: &str,
|
||||
workspace_id: &str,
|
||||
) -> BackendResult<ApiAuthed> {
|
||||
self.auth_cache
|
||||
.get_authed(Some(workspace_id.to_string()), token)
|
||||
.await
|
||||
.ok_or_else(|| {
|
||||
ErrorData::invalid_params(
|
||||
format!(
|
||||
"Access denied: token owner is not a member of workspace '{}'",
|
||||
workspace_id
|
||||
),
|
||||
None,
|
||||
)
|
||||
})
|
||||
}
|
||||
|
||||
fn all_endpoint_tools(&self) -> Vec<EndpointTool> {
|
||||
all_tools()
|
||||
}
|
||||
@@ -401,37 +462,67 @@ pub async fn add_www_authenticate_header(
|
||||
}
|
||||
}
|
||||
|
||||
/// Middleware for gateway: extract workspace_id from the Bearer token in the DB
|
||||
/// and inject it as WorkspaceId extension so the MCP runner can use it.
|
||||
/// Extract the bearer token from either the `Authorization` header or the
|
||||
/// `?token=` query parameter (MCP clients commonly pass it in the URL).
|
||||
fn extract_gateway_token(request: &Request<axum::body::Body>) -> Option<String> {
|
||||
if let Some(token) = request
|
||||
.headers()
|
||||
.get(axum::http::header::AUTHORIZATION)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
.and_then(|h| h.strip_prefix("Bearer "))
|
||||
{
|
||||
return Some(token.to_string());
|
||||
}
|
||||
request.uri().query().and_then(|q| {
|
||||
url::form_urlencoded::parse(q.as_bytes())
|
||||
.find(|(k, _)| k == "token")
|
||||
.map(|(_, v)| v.into_owned())
|
||||
})
|
||||
}
|
||||
|
||||
/// Middleware for gateway: resolve the MCP session mode from the Bearer token in
|
||||
/// the DB. A token bound to a workspace injects `WorkspaceId` (single-workspace
|
||||
/// mode). A workspace-less MCP token (`workspace_id IS NULL` with an `mcp:` scope)
|
||||
/// injects `MultiWorkspaceMcp` + `McpToken`, putting the runner in
|
||||
/// multi-workspace mode where tools take an explicit `workspace_id` argument.
|
||||
pub async fn extract_workspace_from_token(
|
||||
Extension(db): Extension<DB>,
|
||||
mut request: Request<axum::body::Body>,
|
||||
next: Next,
|
||||
) -> Response {
|
||||
if let Some(auth_header) = request
|
||||
.headers()
|
||||
.get(axum::http::header::AUTHORIZATION)
|
||||
.and_then(|v| v.to_str().ok())
|
||||
{
|
||||
if let Some(token) = auth_header.strip_prefix("Bearer ") {
|
||||
let t_hash = hash_token(token);
|
||||
match sqlx::query_scalar!(
|
||||
"SELECT workspace_id FROM token WHERE token_hash = $1 AND workspace_id IS NOT NULL AND (expiration > NOW() OR expiration IS NULL)",
|
||||
t_hash
|
||||
)
|
||||
.fetch_optional(&db)
|
||||
.await
|
||||
{
|
||||
Ok(Some(Some(workspace_id))) => {
|
||||
if let Some(token) = extract_gateway_token(&request) {
|
||||
let t_hash = hash_token(&token);
|
||||
match sqlx::query!(
|
||||
"SELECT workspace_id, scopes FROM token WHERE token_hash = $1 AND (expiration > NOW() OR expiration IS NULL)",
|
||||
t_hash
|
||||
)
|
||||
.fetch_optional(&db)
|
||||
.await
|
||||
{
|
||||
Ok(Some(row)) => match row.workspace_id {
|
||||
Some(workspace_id) => {
|
||||
request
|
||||
.extensions_mut()
|
||||
.insert(GatewayWorkspaceId(workspace_id.clone()));
|
||||
request.extensions_mut().insert(WorkspaceId(workspace_id));
|
||||
}
|
||||
Ok(_) => {}
|
||||
Err(e) => {
|
||||
tracing::error!("Gateway token workspace lookup failed: {}", e);
|
||||
None => {
|
||||
// Only enter multi-workspace mode for genuine MCP tokens; a
|
||||
// full-privilege global token without mcp scope is rejected
|
||||
// by the runner's mcp-scope check anyway.
|
||||
let is_mcp = row
|
||||
.scopes
|
||||
.as_deref()
|
||||
.is_some_and(|s| s.iter().any(|scope| scope.starts_with("mcp:")));
|
||||
if is_mcp {
|
||||
request.extensions_mut().insert(MultiWorkspaceMcp);
|
||||
request.extensions_mut().insert(McpToken(token));
|
||||
}
|
||||
}
|
||||
},
|
||||
Ok(None) => {}
|
||||
Err(e) => {
|
||||
tracing::error!("Gateway token workspace lookup failed: {}", e);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -472,11 +563,12 @@ pub async fn setup_mcp_server(
|
||||
db: DB,
|
||||
user_db: UserDB,
|
||||
base_internal_url: String,
|
||||
auth_cache: Arc<AuthCache>,
|
||||
) -> anyhow::Result<(Router, CancellationToken)> {
|
||||
let cancellation_token = CancellationToken::new();
|
||||
let session_manager = Arc::new(LocalSessionManager::default());
|
||||
|
||||
let backend = WindmillBackend::new(db, user_db, base_internal_url);
|
||||
let backend = WindmillBackend::new(db, user_db, base_internal_url, auth_cache);
|
||||
let runner = Runner::new(backend);
|
||||
|
||||
let service_config = StreamableHttpServerConfig {
|
||||
|
||||
@@ -412,12 +412,50 @@ pub fn build_request_body(
|
||||
args_map: &serde_json::Map<String, Value>,
|
||||
body_schema: &Option<Value>,
|
||||
body_field_renames: &Option<Value>,
|
||||
path_params_schema: &Option<Value>,
|
||||
query_params_schema: &Option<Value>,
|
||||
) -> Option<Value> {
|
||||
if method == "GET" {
|
||||
return None;
|
||||
}
|
||||
|
||||
let schema = body_schema.as_ref()?;
|
||||
|
||||
let has_declared_props = schema
|
||||
.get("properties")
|
||||
.and_then(|p| p.as_object())
|
||||
.map(|o| !o.is_empty())
|
||||
.unwrap_or(false);
|
||||
|
||||
// Pass-through body: the schema declares no explicit properties (e.g.
|
||||
// runScriptByPath / runFlowByPath, whose body is `additionalProperties: true`
|
||||
// and carries the script/flow arguments verbatim). Forward every argument
|
||||
// that isn't already consumed by a path or query parameter — without this the
|
||||
// request body would be empty and parameterized runs would lose their args.
|
||||
if !has_declared_props {
|
||||
if schema.get("type").and_then(|t| t.as_str()) != Some("object") {
|
||||
return None;
|
||||
}
|
||||
let consumed: std::collections::HashSet<&str> = [path_params_schema, query_params_schema]
|
||||
.into_iter()
|
||||
.filter_map(|s| s.as_ref())
|
||||
.filter_map(|s| s.get("properties").and_then(|p| p.as_object()))
|
||||
.flat_map(|props| props.keys().map(|k| k.as_str()))
|
||||
.collect();
|
||||
|
||||
let body_map: serde_json::Map<String, Value> = args_map
|
||||
.iter()
|
||||
.filter(|(k, v)| !consumed.contains(k.as_str()) && !v.is_null())
|
||||
.map(|(k, v)| (k.clone(), v.clone()))
|
||||
.collect();
|
||||
|
||||
return if body_map.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(Value::Object(body_map))
|
||||
};
|
||||
}
|
||||
|
||||
let props = schema.get("properties")?.as_object()?;
|
||||
|
||||
let body_map: serde_json::Map<String, Value> = props
|
||||
@@ -540,6 +578,65 @@ mod tests {
|
||||
use super::*;
|
||||
use serde_json::json;
|
||||
|
||||
#[test]
|
||||
fn build_request_body_passthrough_forwards_script_args_minus_path() {
|
||||
// runScriptByPath-shaped body: additionalProperties, no declared props.
|
||||
// `path` is a path param and must be excluded; the rest are the script's
|
||||
// arguments and must be forwarded verbatim.
|
||||
let body_schema = Some(json!({ "type": "object", "additionalProperties": true }));
|
||||
let path_schema = Some(json!({
|
||||
"type": "object",
|
||||
"properties": { "path": { "type": "string" } },
|
||||
"required": ["path"]
|
||||
}));
|
||||
let args: serde_json::Map<String, Value> = json!({
|
||||
"path": "u/admin/my_script",
|
||||
"name": "alice",
|
||||
"count": 3
|
||||
})
|
||||
.as_object()
|
||||
.unwrap()
|
||||
.clone();
|
||||
|
||||
let body = build_request_body("POST", &args, &body_schema, &None, &path_schema, &None)
|
||||
.expect("passthrough body should be built");
|
||||
let obj = body.as_object().unwrap();
|
||||
assert_eq!(obj.get("name"), Some(&json!("alice")));
|
||||
assert_eq!(obj.get("count"), Some(&json!(3)));
|
||||
assert!(
|
||||
!obj.contains_key("path"),
|
||||
"path param must be excluded from body"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_request_body_declared_props_only_forwards_declared() {
|
||||
// Endpoints with explicit properties keep the strict declared-only behavior.
|
||||
let body_schema = Some(json!({
|
||||
"type": "object",
|
||||
"properties": { "value": { "type": "string" } },
|
||||
"required": ["value"]
|
||||
}));
|
||||
let args: serde_json::Map<String, Value> = json!({ "value": "x", "sneaky": "y" })
|
||||
.as_object()
|
||||
.unwrap()
|
||||
.clone();
|
||||
let body = build_request_body("POST", &args, &body_schema, &None, &None, &None).unwrap();
|
||||
let obj = body.as_object().unwrap();
|
||||
assert_eq!(obj.get("value"), Some(&json!("x")));
|
||||
assert!(
|
||||
!obj.contains_key("sneaky"),
|
||||
"undeclared args must be dropped"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn build_request_body_get_has_no_body() {
|
||||
let body_schema = Some(json!({ "type": "object", "additionalProperties": true }));
|
||||
let args: serde_json::Map<String, Value> = json!({ "a": 1 }).as_object().unwrap().clone();
|
||||
assert!(build_request_body("GET", &args, &body_schema, &None, &None, &None).is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn validate_path_param_value_accepts_legitimate_windmill_paths() {
|
||||
for ok in [
|
||||
|
||||
@@ -14,6 +14,12 @@ pub const WS_BASE_URL_SETTING: &str = "ws_base_url";
|
||||
pub const OAUTH_SETTING: &str = "oauths";
|
||||
pub const AI_CONFIG_SETTING: &str = "ai_config";
|
||||
pub const RETENTION_PERIOD_SECS_SETTING: &str = "retention_period_secs";
|
||||
pub const RETENTION_PERIOD_SECS_OVERRIDES_SETTING: &str = "retention_period_secs_overrides";
|
||||
/// Upper bound on how many per-workspace retention overrides may be configured. The periodic monitor
|
||||
/// sweeps each override workspace in its own transaction every pass, so this keeps a pass bounded
|
||||
/// (and the feature is a targeted escape hatch for a handful of special workspaces, not a bulk knob).
|
||||
/// Enforced at write time and defensively on load.
|
||||
pub const MAX_RETENTION_OVERRIDE_WORKSPACES: usize = 10;
|
||||
pub const AUDIT_LOG_RETENTION_DAYS_SETTING: &str = "audit_log_retention_days";
|
||||
pub const STORE_AUDIT_LOGS_S3_SETTING: &str = "store_audit_logs_s3";
|
||||
/// `background_task_state.name` for the audit-log → object-store export cursor.
|
||||
|
||||
@@ -260,6 +260,16 @@ lazy_static::lazy_static! {
|
||||
pub static ref CRITICAL_ALERTS_ON_DB_OVERSIZE: arc_swap::ArcSwap<Option<f32>> = arc_swap::ArcSwap::from_pointee(None);
|
||||
|
||||
pub static ref JOB_RETENTION_SECS: AtomicI64 = AtomicI64::new(0);
|
||||
/// Per-workspace overrides of `JOB_RETENTION_SECS` (EE-only), keyed by workspace_id, in seconds.
|
||||
/// Sourced from the `retention_period_secs_overrides` global setting and cached here so the
|
||||
/// cleanup sweep reads it without a per-tick DB query. A workspace may be given a longer OR
|
||||
/// shorter window than the instance-wide value; `0` means "keep forever" for that workspace.
|
||||
pub static ref JOB_RETENTION_SECS_OVERRIDES: arc_swap::ArcSwap<std::collections::HashMap<String, i64>> = arc_swap::ArcSwap::from_pointee(std::collections::HashMap::new());
|
||||
/// Whether `JOB_RETENTION_SECS_OVERRIDES` has ever been loaded successfully (a valid map, an
|
||||
/// explicit unset, or CE's no-op). Until then the empty cache is "unknown, not confirmed empty",
|
||||
/// so the retention sweep must NOT run globally — that would delete jobs a longer-retention
|
||||
/// workspace configured before its override could be read.
|
||||
pub static ref JOB_RETENTION_SECS_OVERRIDES_LOADED: AtomicBool = AtomicBool::new(false);
|
||||
pub static ref AUDIT_LOG_RETENTION_DAYS: AtomicI64 = AtomicI64::new(0);
|
||||
|
||||
pub static ref MONITOR_LOGS_ON_OBJECT_STORE: AtomicBool = AtomicBool::new(false);
|
||||
|
||||
@@ -61,6 +61,8 @@ pub struct JobClaim {
|
||||
pub email: String,
|
||||
pub workspace: String,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub fork_parent_workspace: Option<String>,
|
||||
#[serde(skip_serializing_if = "Option::is_none")]
|
||||
pub end_user_email: Option<String>,
|
||||
}
|
||||
|
||||
|
||||
@@ -23,6 +23,26 @@ pub struct McpScopeConfig {
|
||||
}
|
||||
|
||||
impl McpScopeConfig {
|
||||
/// Whether the token grants access to *any* concrete resource of this type by
|
||||
/// path. Used to decide whether to advertise the run-by-path tools in
|
||||
/// multi-workspace mode (a `mcp:scripts:*`-only token should see
|
||||
/// `runScriptByPath` even without an endpoint scope). `mcp:all` grants
|
||||
/// everything; `mcp:favorites` does NOT — favorites are an enumerated set the
|
||||
/// caller can only reach through the per-item tools, not by naming an
|
||||
/// arbitrary path, so it grants nothing here (mirrors `is_allowed`, which
|
||||
/// returns false for a favorites token).
|
||||
pub fn has_any(&self, resource_type: &str) -> bool {
|
||||
if self.all {
|
||||
return true;
|
||||
}
|
||||
match resource_type {
|
||||
"script" => !self.scripts.is_empty(),
|
||||
"flow" => !self.flows.is_empty(),
|
||||
"endpoint" => !self.endpoints.is_empty(),
|
||||
_ => false,
|
||||
}
|
||||
}
|
||||
|
||||
/// Check if a resource is allowed based on its type and path
|
||||
pub fn is_allowed(&self, resource_type: &str, path: &str) -> bool {
|
||||
if self.all {
|
||||
@@ -324,6 +344,29 @@ mod tests {
|
||||
parse_mcp_scopes(&scopes.iter().map(|s| s.to_string()).collect::<Vec<_>>()).unwrap()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_has_any() {
|
||||
// mcp:all grants everything by path.
|
||||
assert!(cfg(&["mcp:all"]).has_any("script"));
|
||||
|
||||
// mcp:favorites grants NO arbitrary-path access (favorites are reached
|
||||
// via per-item tools, not by naming a path) — matches is_allowed.
|
||||
let fav = cfg(&["mcp:favorites"]);
|
||||
assert!(!fav.has_any("script"));
|
||||
assert!(!fav.has_any("flow"));
|
||||
assert!(!fav.is_allowed("script", "f/anything/x"));
|
||||
|
||||
// Granular: only the resource types with at least one pattern.
|
||||
let scripts_only = cfg(&["mcp:scripts:f/team/*"]);
|
||||
assert!(scripts_only.has_any("script"));
|
||||
assert!(!scripts_only.has_any("flow"));
|
||||
assert!(!scripts_only.has_any("endpoint"));
|
||||
|
||||
let endpoints_only = cfg(&["mcp:endpoints:runScriptByPath"]);
|
||||
assert!(!endpoints_only.has_any("script"));
|
||||
assert!(endpoints_only.has_any("endpoint"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_contains_subset_and_widening() {
|
||||
// mcp:all contains anything.
|
||||
|
||||
@@ -15,6 +15,27 @@ use sqlx::FromRow;
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct WorkspaceId(pub String);
|
||||
|
||||
/// Marker extension inserted by the gateway middleware when an MCP token has no
|
||||
/// bound workspace (`workspace_id IS NULL`). Signals the runner to operate in
|
||||
/// multi-workspace mode: tools take an explicit `workspace_id` argument and the
|
||||
/// per-workspace auth is resolved on demand from the raw token.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct MultiWorkspaceMcp;
|
||||
|
||||
/// Raw bearer token wrapper for Axum extensions. In multi-workspace mode the
|
||||
/// runner needs the raw token to re-resolve auth for each requested workspace.
|
||||
#[derive(Clone, Debug)]
|
||||
pub struct McpToken(pub String);
|
||||
|
||||
/// Summary of a workspace the caller can access, returned by the
|
||||
/// `list_workspaces` tool in multi-workspace mode.
|
||||
#[derive(Serialize, Debug, Clone)]
|
||||
#[cfg_attr(feature = "server", derive(FromRow))]
|
||||
pub struct WorkspaceInfo {
|
||||
pub id: String,
|
||||
pub name: String,
|
||||
}
|
||||
|
||||
/// Hub API response structure
|
||||
#[derive(Serialize, Deserialize, Debug)]
|
||||
pub struct HubResponse {
|
||||
|
||||
@@ -9,7 +9,7 @@ use serde_json::Value;
|
||||
use std::collections::HashMap;
|
||||
|
||||
use crate::common::types::{
|
||||
FlowInfo, HubScriptInfo, ResourceInfo, ResourceType, SchemaType, ScriptInfo,
|
||||
FlowInfo, HubScriptInfo, ResourceInfo, ResourceType, SchemaType, ScriptInfo, WorkspaceInfo,
|
||||
};
|
||||
use crate::server::endpoints::EndpointTool;
|
||||
|
||||
@@ -159,6 +159,27 @@ pub trait McpBackend: Send + Sync + Clone + 'static {
|
||||
args: Value,
|
||||
) -> BackendResult<Value>;
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Multi-workspace support
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
/// List the workspaces the caller (identified by `auth`) can access. Used by
|
||||
/// the `list_workspaces` tool exposed in multi-workspace mode.
|
||||
async fn list_accessible_workspaces(
|
||||
&self,
|
||||
auth: &Self::Auth,
|
||||
) -> BackendResult<Vec<WorkspaceInfo>>;
|
||||
|
||||
/// Resolve a workspace-specific auth for `workspace_id` from the raw bearer
|
||||
/// `token`. Returns an error if the token's owner is not a member of the
|
||||
/// workspace. Used in multi-workspace mode to authorize per-workspace tool
|
||||
/// calls (the base auth carries no workspace-specific permissions).
|
||||
async fn resolve_workspace_auth(
|
||||
&self,
|
||||
token: &str,
|
||||
workspace_id: &str,
|
||||
) -> BackendResult<Self::Auth>;
|
||||
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
// Endpoint Tools
|
||||
// ─────────────────────────────────────────────────────────────────
|
||||
|
||||
@@ -73,6 +73,86 @@ pub fn endpoint_tool_to_mcp_tool(tool: &EndpointTool) -> Tool {
|
||||
}
|
||||
}
|
||||
|
||||
/// Convert an endpoint tool to an MCP tool for multi-workspace mode.
|
||||
///
|
||||
/// Endpoints whose path is workspace-scoped (`/w/{workspace}/...`) gain a
|
||||
/// required `workspace_id` argument — in multi-workspace mode there is no
|
||||
/// ambient workspace, so the caller must name the target workspace explicitly.
|
||||
/// Global endpoints (e.g. docs search) are returned unchanged.
|
||||
pub fn endpoint_tool_to_mcp_tool_multi(tool: &EndpointTool) -> Tool {
|
||||
let mut mcp_tool = endpoint_tool_to_mcp_tool(tool);
|
||||
|
||||
if !tool.path.contains("{workspace}") {
|
||||
return mcp_tool;
|
||||
}
|
||||
|
||||
let mut schema = (*mcp_tool.input_schema).clone();
|
||||
|
||||
if let Some(props) = schema.get_mut("properties").and_then(|p| p.as_object_mut()) {
|
||||
props.insert(
|
||||
"workspace_id".to_string(),
|
||||
serde_json::json!({
|
||||
"type": "string",
|
||||
"description": "Target workspace id (from list_workspaces)."
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
match schema.get_mut("required").and_then(|r| r.as_array_mut()) {
|
||||
Some(req) => {
|
||||
if !req.iter().any(|v| v.as_str() == Some("workspace_id")) {
|
||||
req.insert(0, serde_json::Value::String("workspace_id".to_string()));
|
||||
}
|
||||
}
|
||||
None => {
|
||||
schema.insert("required".to_string(), serde_json::json!(["workspace_id"]));
|
||||
}
|
||||
}
|
||||
|
||||
// Surface the requirement in the prose description too (the schema is
|
||||
// authoritative, but some models/clients lean on the text). Kept terse — this
|
||||
// repeats across every workspace-scoped tool in the list.
|
||||
if let Some(desc) = mcp_tool.description.take() {
|
||||
mcp_tool.description = Some(format!("{desc} Requires `workspace_id`.").into());
|
||||
} else {
|
||||
mcp_tool.description = Some("Requires `workspace_id`.".into());
|
||||
}
|
||||
|
||||
mcp_tool.input_schema = Arc::new(schema);
|
||||
mcp_tool
|
||||
}
|
||||
|
||||
/// Build the synthetic `list_workspaces` tool exposed only in multi-workspace
|
||||
/// mode. It takes no arguments and returns the workspaces the token can access.
|
||||
pub fn list_workspaces_tool() -> Tool {
|
||||
let schema = serde_json::json!({
|
||||
"type": "object",
|
||||
"properties": {},
|
||||
"required": []
|
||||
});
|
||||
|
||||
Tool {
|
||||
name: Cow::Borrowed("list_workspaces"),
|
||||
description: Some(
|
||||
"List the Windmill workspaces this token can access. Use the returned workspace ids as the `workspace_id` argument of the other tools."
|
||||
.into(),
|
||||
),
|
||||
input_schema: Arc::new(schema.as_object().unwrap().clone()),
|
||||
title: Some("List accessible workspaces".to_string()),
|
||||
output_schema: None,
|
||||
icons: None,
|
||||
annotations: Some(ToolAnnotations {
|
||||
title: Some("List accessible workspaces".to_string()),
|
||||
read_only_hint: Some(true),
|
||||
destructive_hint: Some(false),
|
||||
idempotent_hint: Some(true),
|
||||
open_world_hint: Some(false),
|
||||
}),
|
||||
meta: None,
|
||||
execution: None,
|
||||
}
|
||||
}
|
||||
|
||||
/// Create appropriate annotations for endpoint tools based on HTTP method
|
||||
fn create_endpoint_annotations(tool: &EndpointTool) -> ToolAnnotations {
|
||||
let method = tool.method.as_ref();
|
||||
@@ -116,3 +196,119 @@ fn merge_schema_into(
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod tests {
|
||||
use super::*;
|
||||
|
||||
fn tool(name: &'static str, path: &'static str) -> EndpointTool {
|
||||
EndpointTool {
|
||||
name: Cow::Borrowed(name),
|
||||
description: Cow::Borrowed("desc"),
|
||||
instructions: Cow::Borrowed(""),
|
||||
path: Cow::Borrowed(path),
|
||||
method: Cow::Borrowed("GET"),
|
||||
path_params_schema: None,
|
||||
query_params_schema: Some(serde_json::json!({
|
||||
"type": "object",
|
||||
"properties": { "starred_only": { "type": "boolean" } },
|
||||
"required": []
|
||||
})),
|
||||
body_schema: None,
|
||||
path_field_renames: None,
|
||||
query_field_renames: None,
|
||||
body_field_renames: None,
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multi_injects_required_workspace_id_for_workspaced_tool() {
|
||||
let mcp =
|
||||
endpoint_tool_to_mcp_tool_multi(&tool("listScripts", "/w/{workspace}/scripts/list"));
|
||||
let props = mcp
|
||||
.input_schema
|
||||
.get("properties")
|
||||
.unwrap()
|
||||
.as_object()
|
||||
.unwrap();
|
||||
assert!(
|
||||
props.contains_key("workspace_id"),
|
||||
"workspace_id must be added as a property"
|
||||
);
|
||||
// pre-existing param is preserved
|
||||
assert!(props.contains_key("starred_only"));
|
||||
let required = mcp
|
||||
.input_schema
|
||||
.get("required")
|
||||
.unwrap()
|
||||
.as_array()
|
||||
.unwrap();
|
||||
assert!(
|
||||
required.iter().any(|v| v.as_str() == Some("workspace_id")),
|
||||
"workspace_id must be required"
|
||||
);
|
||||
assert!(
|
||||
mcp.description
|
||||
.as_deref()
|
||||
.unwrap_or_default()
|
||||
.contains("workspace_id"),
|
||||
"description must mention the workspace_id requirement"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multi_leaves_global_tool_unchanged() {
|
||||
let global = tool("searchDocs", "/docs/search");
|
||||
let plain = endpoint_tool_to_mcp_tool(&global);
|
||||
let mcp = endpoint_tool_to_mcp_tool_multi(&global);
|
||||
assert_eq!(
|
||||
mcp.description, plain.description,
|
||||
"global tool description must be unchanged"
|
||||
);
|
||||
let props = mcp
|
||||
.input_schema
|
||||
.get("properties")
|
||||
.unwrap()
|
||||
.as_object()
|
||||
.unwrap();
|
||||
assert!(
|
||||
!props.contains_key("workspace_id"),
|
||||
"global tools (no {{workspace}} in path) must not gain a workspace_id arg"
|
||||
);
|
||||
let required = mcp
|
||||
.input_schema
|
||||
.get("required")
|
||||
.unwrap()
|
||||
.as_array()
|
||||
.unwrap();
|
||||
assert!(required.iter().all(|v| v.as_str() != Some("workspace_id")));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn multi_does_not_duplicate_workspace_id() {
|
||||
// Even if run twice, workspace_id stays a single required entry.
|
||||
let once = endpoint_tool_to_mcp_tool_multi(&tool("listFlows", "/w/{workspace}/flows/list"));
|
||||
let required = once
|
||||
.input_schema
|
||||
.get("required")
|
||||
.unwrap()
|
||||
.as_array()
|
||||
.unwrap();
|
||||
let count = required
|
||||
.iter()
|
||||
.filter(|v| v.as_str() == Some("workspace_id"))
|
||||
.count();
|
||||
assert_eq!(
|
||||
count, 1,
|
||||
"workspace_id must appear exactly once in required"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_workspaces_tool_has_no_params() {
|
||||
let t = list_workspaces_tool();
|
||||
assert_eq!(t.name.as_ref(), "list_workspaces");
|
||||
let required = t.input_schema.get("required").unwrap().as_array().unwrap();
|
||||
assert!(required.is_empty());
|
||||
}
|
||||
}
|
||||
|
||||
@@ -11,8 +11,12 @@ pub mod runner;
|
||||
pub mod tools;
|
||||
|
||||
// Re-export main types
|
||||
pub use crate::common::types::{McpToken, MultiWorkspaceMcp, WorkspaceInfo};
|
||||
pub use backend::{BackendResult, McpAuth, McpBackend};
|
||||
pub use endpoints::{endpoint_tool_to_mcp_tool, is_endpoint_read_only, EndpointTool};
|
||||
pub use endpoints::{
|
||||
endpoint_tool_to_mcp_tool, endpoint_tool_to_mcp_tool_multi, is_endpoint_read_only,
|
||||
list_workspaces_tool, EndpointTool,
|
||||
};
|
||||
pub use runner::Runner;
|
||||
pub use tools::create_tool_from_item;
|
||||
|
||||
|
||||
@@ -9,9 +9,11 @@ use crate::common::transform::{
|
||||
extract_hub_version_id_from_hashed, extract_path_prefix_from_hashed, parse_tool_prefix,
|
||||
reverse_transform, reverse_transform_key,
|
||||
};
|
||||
use crate::common::types::{ResourceInfo, ToolableItem, WorkspaceId};
|
||||
use crate::common::types::{McpToken, MultiWorkspaceMcp, ResourceInfo, ToolableItem, WorkspaceId};
|
||||
use crate::server::backend::{McpAuth, McpBackend};
|
||||
use crate::server::endpoints::endpoint_tool_to_mcp_tool;
|
||||
use crate::server::endpoints::{
|
||||
endpoint_tool_to_mcp_tool, endpoint_tool_to_mcp_tool_multi, list_workspaces_tool,
|
||||
};
|
||||
use crate::server::tools::create_tool_from_item;
|
||||
use rmcp::handler::server::ServerHandler;
|
||||
use rmcp::model::{
|
||||
@@ -61,16 +63,28 @@ impl<B: McpBackend> Clone for Runner<B> {
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether the request targets one bound workspace or spans every workspace the
|
||||
/// token can access.
|
||||
enum McpMode {
|
||||
/// A single workspace, resolved from the URL path or the token's bound
|
||||
/// workspace. Tools operate against this workspace implicitly.
|
||||
Single(String),
|
||||
/// The token has no bound workspace (`workspace_id IS NULL`). Tools take an
|
||||
/// explicit `workspace_id` argument; the wrapped value is the raw bearer
|
||||
/// token, used to re-resolve auth per requested workspace.
|
||||
Multi(String),
|
||||
}
|
||||
|
||||
impl<B: McpBackend> Runner<B> {
|
||||
/// Create a new Runner with the given backend
|
||||
pub fn new(backend: B) -> Self {
|
||||
Self { backend: Arc::new(backend) }
|
||||
}
|
||||
|
||||
/// Extract authentication and workspace from request context
|
||||
/// Extract authentication and the workspace mode from request context
|
||||
fn extract_context(
|
||||
context: &RequestContext<RoleServer>,
|
||||
) -> Result<(B::Auth, String), ErrorData> {
|
||||
) -> Result<(B::Auth, McpMode), ErrorData> {
|
||||
let http_parts = context.extensions.get::<HttpParts>().ok_or_else(|| {
|
||||
tracing::error!("http::request::Parts not found");
|
||||
ErrorData::internal_error("http::request::Parts not found", None)
|
||||
@@ -81,15 +95,6 @@ impl<B: McpBackend> Runner<B> {
|
||||
ErrorData::internal_error("Auth extension not found", None)
|
||||
})?;
|
||||
|
||||
let workspace_id = http_parts
|
||||
.extensions
|
||||
.get::<WorkspaceId>()
|
||||
.ok_or_else(|| {
|
||||
tracing::error!("WorkspaceId not found");
|
||||
ErrorData::internal_error("WorkspaceId not found", None)
|
||||
})
|
||||
.map(|w_id| w_id.0.clone())?;
|
||||
|
||||
// Validate MCP scope
|
||||
if !auth.has_mcp_scope() {
|
||||
tracing::error!("Unauthorized: missing mcp scope");
|
||||
@@ -99,7 +104,39 @@ impl<B: McpBackend> Runner<B> {
|
||||
));
|
||||
}
|
||||
|
||||
Ok((auth.clone(), workspace_id))
|
||||
let mode = if http_parts.extensions.get::<MultiWorkspaceMcp>().is_some() {
|
||||
let token = http_parts.extensions.get::<McpToken>().ok_or_else(|| {
|
||||
tracing::error!("MultiWorkspaceMcp set but McpToken missing");
|
||||
ErrorData::internal_error("MCP token not found for multi-workspace session", None)
|
||||
})?;
|
||||
McpMode::Multi(token.0.clone())
|
||||
} else {
|
||||
let workspace_id = http_parts
|
||||
.extensions
|
||||
.get::<WorkspaceId>()
|
||||
.ok_or_else(|| {
|
||||
tracing::error!("WorkspaceId not found");
|
||||
ErrorData::internal_error("WorkspaceId not found", None)
|
||||
})
|
||||
.map(|w_id| w_id.0.clone())?;
|
||||
McpMode::Single(workspace_id)
|
||||
};
|
||||
|
||||
Ok((auth.clone(), mode))
|
||||
}
|
||||
}
|
||||
|
||||
/// The run-by-path endpoint tools execute an arbitrary script/flow named by a
|
||||
/// `path` argument. In multi-workspace mode they are the only way to run
|
||||
/// scripts/flows, so their authorization must honor the `mcp:scripts:` /
|
||||
/// `mcp:flows:` path scopes (not the generic endpoint scope) — otherwise a
|
||||
/// granular token could run items outside its allowed paths. Returns the scope
|
||||
/// resource type ("script"/"flow") for these endpoints, `None` otherwise.
|
||||
fn run_by_path_scope_kind(endpoint_name: &str) -> Option<&'static str> {
|
||||
match endpoint_name {
|
||||
"runScriptByPath" => Some("script"),
|
||||
"runFlowByPath" => Some("flow"),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -137,16 +174,99 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
context: RequestContext<RoleServer>,
|
||||
) -> Result<ListToolsResult, ErrorData> {
|
||||
let (auth, workspace_id) = Self::extract_context(&context)?;
|
||||
let (auth, mode) = Self::extract_context(&context)?;
|
||||
|
||||
// Parse MCP scopes to determine what to expose
|
||||
let scopes = auth.scopes().unwrap_or(&[]);
|
||||
let scope_config =
|
||||
parse_mcp_scopes(scopes).map_err(|e| ErrorData::internal_error(e, None))?;
|
||||
|
||||
let favorites_only = scope_config.favorites;
|
||||
let read_only = auth.read_only();
|
||||
|
||||
match mode {
|
||||
McpMode::Single(workspace_id) => {
|
||||
self.list_tools_single(&auth, &workspace_id, &scope_config, read_only)
|
||||
.await
|
||||
}
|
||||
// Multi-workspace: expose the generic endpoint tools (each taking an
|
||||
// explicit workspace_id) plus list_workspaces. Per-workspace scripts
|
||||
// and flows are intentionally not enumerated here — doing so across
|
||||
// every workspace would overload the tool list; callers run them via
|
||||
// runScriptByPath / runFlowByPath with a workspace_id instead.
|
||||
McpMode::Multi(_) => Ok(self.list_tools_multi(&scope_config, read_only)),
|
||||
}
|
||||
}
|
||||
|
||||
async fn call_tool(
|
||||
&self,
|
||||
request: CallToolRequestParams,
|
||||
context: RequestContext<RoleServer>,
|
||||
) -> Result<CallToolResult, ErrorData> {
|
||||
let (auth, mode) = Self::extract_context(&context)?;
|
||||
|
||||
// Parse MCP scopes for authorization
|
||||
let scopes = auth.scopes().unwrap_or(&[]);
|
||||
let scope_config =
|
||||
parse_mcp_scopes(scopes).map_err(|e| ErrorData::internal_error(e, None))?;
|
||||
let read_only = auth.read_only();
|
||||
|
||||
let args = request.arguments.map(Value::Object).unwrap_or(Value::Null);
|
||||
|
||||
match mode {
|
||||
McpMode::Single(workspace_id) => {
|
||||
self.call_tool_single(
|
||||
&auth,
|
||||
&workspace_id,
|
||||
&scope_config,
|
||||
read_only,
|
||||
request.name,
|
||||
args,
|
||||
)
|
||||
.await
|
||||
}
|
||||
McpMode::Multi(token) => {
|
||||
self.call_tool_multi(&auth, &token, &scope_config, read_only, request.name, args)
|
||||
.await
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_resources(
|
||||
&self,
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
_context: RequestContext<RoleServer>,
|
||||
) -> Result<ListResourcesResult, ErrorData> {
|
||||
Ok(ListResourcesResult { resources: vec![], next_cursor: None, meta: None })
|
||||
}
|
||||
|
||||
async fn list_prompts(
|
||||
&self,
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
_context: RequestContext<RoleServer>,
|
||||
) -> Result<ListPromptsResult, ErrorData> {
|
||||
Ok(ListPromptsResult::default())
|
||||
}
|
||||
|
||||
async fn list_resource_templates(
|
||||
&self,
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
_context: RequestContext<RoleServer>,
|
||||
) -> Result<ListResourceTemplatesResult, ErrorData> {
|
||||
Ok(ListResourceTemplatesResult::default())
|
||||
}
|
||||
}
|
||||
|
||||
impl<B: McpBackend> Runner<B> {
|
||||
/// List tools for a single, bound workspace (URL-path or token-bound).
|
||||
async fn list_tools_single(
|
||||
&self,
|
||||
auth: &B::Auth,
|
||||
workspace_id: &str,
|
||||
scope_config: &crate::common::scope::McpScopeConfig,
|
||||
read_only: bool,
|
||||
) -> Result<ListToolsResult, ErrorData> {
|
||||
let favorites_only = scope_config.favorites;
|
||||
|
||||
let mut tools = Vec::new();
|
||||
|
||||
// Read-only tokens cannot run scripts/flows/hub-scripts (running is a
|
||||
@@ -155,10 +275,10 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
if !read_only {
|
||||
let (scripts, flows, resource_types, hub_scripts) = tokio::try_join!(
|
||||
self.backend
|
||||
.list_scripts(&auth, &workspace_id, favorites_only, None),
|
||||
.list_scripts(auth, workspace_id, favorites_only, None),
|
||||
self.backend
|
||||
.list_flows(&auth, &workspace_id, favorites_only, None),
|
||||
self.backend.list_resource_types(&auth, &workspace_id),
|
||||
.list_flows(auth, workspace_id, favorites_only, None),
|
||||
self.backend.list_resource_types(auth, workspace_id),
|
||||
async {
|
||||
if let Some(ref apps) = scope_config.hub_apps {
|
||||
self.backend.list_hub_scripts(Some(apps)).await
|
||||
@@ -199,7 +319,7 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
.map(|rt| {
|
||||
let backend = self.backend.clone();
|
||||
let auth = auth.clone();
|
||||
let workspace_id = workspace_id.clone();
|
||||
let workspace_id = workspace_id.to_string();
|
||||
async move {
|
||||
backend
|
||||
.list_resources(&auth, &workspace_id, &rt)
|
||||
@@ -257,25 +377,20 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
Ok(ListToolsResult { tools, next_cursor: None, meta: None })
|
||||
}
|
||||
|
||||
async fn call_tool(
|
||||
/// Handle a tool call for a single, bound workspace.
|
||||
async fn call_tool_single(
|
||||
&self,
|
||||
request: CallToolRequestParams,
|
||||
context: RequestContext<RoleServer>,
|
||||
auth: &B::Auth,
|
||||
workspace_id: &str,
|
||||
scope_config: &crate::common::scope::McpScopeConfig,
|
||||
read_only: bool,
|
||||
name: std::borrow::Cow<'static, str>,
|
||||
args: Value,
|
||||
) -> Result<CallToolResult, ErrorData> {
|
||||
let (auth, workspace_id) = Self::extract_context(&context)?;
|
||||
|
||||
// Parse MCP scopes for authorization
|
||||
let scopes = auth.scopes().unwrap_or(&[]);
|
||||
let scope_config =
|
||||
parse_mcp_scopes(scopes).map_err(|e| ErrorData::internal_error(e, None))?;
|
||||
let read_only = auth.read_only();
|
||||
|
||||
let args = request.arguments.map(Value::Object).unwrap_or(Value::Null);
|
||||
|
||||
// Check if this is an endpoint tool
|
||||
let endpoint_tools = self.backend.all_endpoint_tools();
|
||||
for endpoint_tool in &endpoint_tools {
|
||||
if endpoint_tool.name.as_ref() == request.name {
|
||||
if endpoint_tool.name.as_ref() == name.as_ref() {
|
||||
// Validate endpoint scope
|
||||
if scope_config.granular
|
||||
&& !scope_config.is_allowed("endpoint", &endpoint_tool.name)
|
||||
@@ -301,7 +416,7 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
// This is an endpoint tool, call via backend
|
||||
let result = self
|
||||
.backend
|
||||
.call_endpoint(&auth, &workspace_id, endpoint_tool, args)
|
||||
.call_endpoint(auth, workspace_id, endpoint_tool, args)
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?;
|
||||
|
||||
@@ -319,53 +434,50 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
return Err(ErrorData::internal_error(
|
||||
format!(
|
||||
"Access denied: tool '{}' runs a script/flow and this token is restricted to read-only operations",
|
||||
request.name
|
||||
name
|
||||
),
|
||||
None,
|
||||
));
|
||||
}
|
||||
|
||||
// Resolve the tool name to (type, path, is_hub)
|
||||
let (type_str, is_hub, is_hashed) = parse_tool_prefix(&request.name).map_err(|e| {
|
||||
let (type_str, is_hub, is_hashed) = parse_tool_prefix(name.as_ref()).map_err(|e| {
|
||||
ErrorData::internal_error(format!("Failed to parse tool name: {}", e), None)
|
||||
})?;
|
||||
|
||||
let (tool_type, path, is_hub) = if !is_hashed {
|
||||
reverse_transform(&request.name).map_err(|e| {
|
||||
reverse_transform(name.as_ref()).map_err(|e| {
|
||||
ErrorData::internal_error(format!("Failed to parse tool name: {}", e), None)
|
||||
})?
|
||||
} else if is_hub {
|
||||
let version_id = extract_hub_version_id_from_hashed(&request.name).map_err(|e| {
|
||||
let version_id = extract_hub_version_id_from_hashed(name.as_ref()).map_err(|e| {
|
||||
ErrorData::internal_error(format!("Failed to extract hub version_id: {}", e), None)
|
||||
})?;
|
||||
(type_str, version_id, true)
|
||||
} else {
|
||||
let path_prefix = extract_path_prefix_from_hashed(&request.name);
|
||||
let path_prefix = extract_path_prefix_from_hashed(name.as_ref());
|
||||
let favorites_only = scope_config.favorites;
|
||||
let matched_path = if type_str == "script" {
|
||||
find_matching_path(
|
||||
self.backend
|
||||
.list_scripts(&auth, &workspace_id, favorites_only, path_prefix.as_deref())
|
||||
.list_scripts(auth, workspace_id, favorites_only, path_prefix.as_deref())
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?,
|
||||
&request.name,
|
||||
name.as_ref(),
|
||||
)
|
||||
} else {
|
||||
find_matching_path(
|
||||
self.backend
|
||||
.list_flows(&auth, &workspace_id, favorites_only, path_prefix.as_deref())
|
||||
.list_flows(auth, workspace_id, favorites_only, path_prefix.as_deref())
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?,
|
||||
&request.name,
|
||||
name.as_ref(),
|
||||
)
|
||||
};
|
||||
|
||||
let matched_path = matched_path.ok_or_else(|| {
|
||||
ErrorData::internal_error(
|
||||
format!(
|
||||
"No {} found matching hashed tool name '{}'",
|
||||
type_str, request.name
|
||||
),
|
||||
format!("No {} found matching hashed tool name '{}'", type_str, name),
|
||||
None,
|
||||
)
|
||||
})?;
|
||||
@@ -396,7 +508,7 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?
|
||||
} else {
|
||||
self.backend
|
||||
.get_item_schema(&auth, &workspace_id, &path, tool_type)
|
||||
.get_item_schema(auth, workspace_id, &path, tool_type)
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?
|
||||
};
|
||||
@@ -422,11 +534,11 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
// Execute script or flow
|
||||
let result = if tool_type == "script" {
|
||||
self.backend
|
||||
.run_script(&auth, &workspace_id, &script_or_flow_path, transformed_args)
|
||||
.run_script(auth, workspace_id, &script_or_flow_path, transformed_args)
|
||||
.await
|
||||
} else {
|
||||
self.backend
|
||||
.run_flow(&auth, &workspace_id, &script_or_flow_path, transformed_args)
|
||||
.run_flow(auth, workspace_id, &script_or_flow_path, transformed_args)
|
||||
.await
|
||||
};
|
||||
|
||||
@@ -443,27 +555,181 @@ impl<B: McpBackend> ServerHandler for Runner<B> {
|
||||
}
|
||||
}
|
||||
|
||||
async fn list_resources(
|
||||
/// List tools for a multi-workspace session: the synthetic `list_workspaces`
|
||||
/// tool plus every generic endpoint tool, each taking an explicit
|
||||
/// `workspace_id` argument.
|
||||
fn list_tools_multi(
|
||||
&self,
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
_context: RequestContext<RoleServer>,
|
||||
) -> Result<ListResourcesResult, ErrorData> {
|
||||
Ok(ListResourcesResult { resources: vec![], next_cursor: None, meta: None })
|
||||
scope_config: &crate::common::scope::McpScopeConfig,
|
||||
read_only: bool,
|
||||
) -> ListToolsResult {
|
||||
let mut tools = vec![list_workspaces_tool()];
|
||||
|
||||
let endpoint_tools = self.backend.all_endpoint_tools();
|
||||
for endpoint_tool in endpoint_tools {
|
||||
// Run-by-path tools are gated by script/flow scope (they run an
|
||||
// arbitrary path); every other endpoint by the endpoint scope.
|
||||
let allowed = match run_by_path_scope_kind(&endpoint_tool.name) {
|
||||
Some(kind) => scope_config.has_any(kind),
|
||||
None => {
|
||||
!scope_config.granular
|
||||
|| scope_config.is_allowed("endpoint", &endpoint_tool.name)
|
||||
}
|
||||
};
|
||||
if !allowed {
|
||||
continue;
|
||||
}
|
||||
if read_only && !crate::server::is_endpoint_read_only(&endpoint_tool) {
|
||||
continue;
|
||||
}
|
||||
|
||||
tools.push(endpoint_tool_to_mcp_tool_multi(&endpoint_tool));
|
||||
}
|
||||
|
||||
ListToolsResult { tools, next_cursor: None, meta: None }
|
||||
}
|
||||
|
||||
async fn list_prompts(
|
||||
/// Handle a tool call for a multi-workspace session. `base_auth` is the
|
||||
/// workspace-less identity derived from the token; per-workspace auth is
|
||||
/// resolved on demand from `token` for the workspace named in the args.
|
||||
async fn call_tool_multi(
|
||||
&self,
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
_context: RequestContext<RoleServer>,
|
||||
) -> Result<ListPromptsResult, ErrorData> {
|
||||
Ok(ListPromptsResult::default())
|
||||
}
|
||||
base_auth: &B::Auth,
|
||||
token: &str,
|
||||
scope_config: &crate::common::scope::McpScopeConfig,
|
||||
read_only: bool,
|
||||
name: std::borrow::Cow<'static, str>,
|
||||
args: Value,
|
||||
) -> Result<CallToolResult, ErrorData> {
|
||||
if name.as_ref() == "list_workspaces" {
|
||||
let workspaces = self
|
||||
.backend
|
||||
.list_accessible_workspaces(base_auth)
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?;
|
||||
return Ok(CallToolResult::success(vec![Content::text(
|
||||
serde_json::to_string_pretty(&workspaces).unwrap_or_else(|_| "[]".to_string()),
|
||||
)]));
|
||||
}
|
||||
|
||||
async fn list_resource_templates(
|
||||
&self,
|
||||
_request: Option<PaginatedRequestParams>,
|
||||
_context: RequestContext<RoleServer>,
|
||||
) -> Result<ListResourceTemplatesResult, ErrorData> {
|
||||
Ok(ListResourceTemplatesResult::default())
|
||||
// Only endpoint tools are exposed in multi-workspace mode; scripts and
|
||||
// flows are run through the runScriptByPath / runFlowByPath endpoints.
|
||||
let endpoint_tools = self.backend.all_endpoint_tools();
|
||||
let endpoint_tool = endpoint_tools
|
||||
.iter()
|
||||
.find(|t| t.name.as_ref() == name.as_ref())
|
||||
.ok_or_else(|| {
|
||||
ErrorData::invalid_params(
|
||||
format!(
|
||||
"Unknown tool '{}' in multi-workspace mode. Available tools are list_workspaces and the generic API endpoint tools (run scripts/flows via runScriptByPath / runFlowByPath).",
|
||||
name
|
||||
),
|
||||
None,
|
||||
)
|
||||
})?;
|
||||
|
||||
// Authorize the tool. Run-by-path endpoints (runScriptByPath /
|
||||
// runFlowByPath) run an arbitrary `path` and must be checked against the
|
||||
// script/flow scope for that path — the endpoint scope alone would let a
|
||||
// granular token run items outside its allowed paths.
|
||||
match run_by_path_scope_kind(&endpoint_tool.name) {
|
||||
Some(kind) => {
|
||||
let path = args
|
||||
.get("path")
|
||||
.and_then(|v| v.as_str())
|
||||
.filter(|s| !s.is_empty())
|
||||
.ok_or_else(|| {
|
||||
ErrorData::invalid_params(
|
||||
format!(
|
||||
"Missing required 'path' argument for tool '{}'.",
|
||||
endpoint_tool.name
|
||||
),
|
||||
None,
|
||||
)
|
||||
})?;
|
||||
// No `granular` gate: is_allowed already encodes every mode —
|
||||
// true for mcp:all, pattern-matched for granular scopes, and
|
||||
// false for mcp:favorites (a favorites token can't run an
|
||||
// arbitrary path, only its enumerated favorites).
|
||||
if !scope_config.is_allowed(kind, path) {
|
||||
return Err(ErrorData::internal_error(
|
||||
format!("Access denied: {} '{}' not in token scope", kind, path),
|
||||
None,
|
||||
));
|
||||
}
|
||||
}
|
||||
None => {
|
||||
if scope_config.granular
|
||||
&& !scope_config.is_allowed("endpoint", &endpoint_tool.name)
|
||||
{
|
||||
return Err(ErrorData::internal_error(
|
||||
format!(
|
||||
"Access denied: endpoint '{}' not in token scope",
|
||||
endpoint_tool.name
|
||||
),
|
||||
None,
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
if read_only && !crate::server::is_endpoint_read_only(endpoint_tool) {
|
||||
return Err(ErrorData::internal_error(
|
||||
format!(
|
||||
"Access denied: endpoint '{}' is not read-only and this token is restricted to read-only operations",
|
||||
endpoint_tool.name
|
||||
),
|
||||
None,
|
||||
));
|
||||
}
|
||||
|
||||
// Workspace-scoped endpoints need an explicit target workspace and a
|
||||
// per-workspace auth; global endpoints (e.g. docs) use the base identity.
|
||||
let needs_workspace = endpoint_tool.path.contains("{workspace}");
|
||||
let (workspace_id, resolved_auth) = if needs_workspace {
|
||||
let workspace_id = args
|
||||
.get("workspace_id")
|
||||
.and_then(|v| v.as_str())
|
||||
.filter(|s| !s.is_empty())
|
||||
.ok_or_else(|| {
|
||||
ErrorData::invalid_params(
|
||||
format!(
|
||||
"Missing required 'workspace_id' argument for tool '{}'. Call list_workspaces to see the workspaces you can access.",
|
||||
endpoint_tool.name
|
||||
),
|
||||
None,
|
||||
)
|
||||
})?
|
||||
.to_string();
|
||||
|
||||
let resolved = self
|
||||
.backend
|
||||
.resolve_workspace_auth(token, &workspace_id)
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?;
|
||||
(workspace_id, resolved)
|
||||
} else {
|
||||
(String::new(), base_auth.clone())
|
||||
};
|
||||
|
||||
// `workspace_id` is a synthetic argument only this layer understands; the
|
||||
// target workspace is passed to call_endpoint separately. Strip it so it
|
||||
// can't leak into a pass-through request body (e.g. runScriptByPath, whose
|
||||
// body forwards all remaining args as the script's arguments).
|
||||
let mut args = args;
|
||||
if let Value::Object(map) = &mut args {
|
||||
map.remove("workspace_id");
|
||||
}
|
||||
|
||||
let result = self
|
||||
.backend
|
||||
.call_endpoint(&resolved_auth, &workspace_id, endpoint_tool, args)
|
||||
.await
|
||||
.map_err(|e| ErrorData::internal_error(e.message, None))?;
|
||||
|
||||
Ok(CallToolResult::success(vec![Content::text(
|
||||
truncate_tool_result(
|
||||
serde_json::to_string_pretty(&result).unwrap_or_else(|_| "{}".to_string()),
|
||||
),
|
||||
)]))
|
||||
}
|
||||
}
|
||||
|
||||
@@ -921,8 +921,8 @@ lazy_static::lazy_static! {
|
||||
pub static ref GLOBAL_ERROR_HANDLER_PATH_IN_ADMINS_WORKSPACE: Option<String> = std::env::var("GLOBAL_ERROR_HANDLER_PATH_IN_ADMINS_WORKSPACE").ok();
|
||||
pub static ref MAX_RESULT_SIZE_MB: usize = std::env::var("MAX_RESULT_SIZE_MB").unwrap_or("500".to_string()).parse().unwrap_or(500);
|
||||
|
||||
// Cache for restart_unless_cancelled flag - keyed by (hash, workspace_id)
|
||||
static ref RESTART_UNLESS_CANCELLED_CACHE: Cache<(i64, String), bool> = Cache::new(10000);
|
||||
// Cache for perpetual-restart settings (restart_unless_cancelled, timeout) - keyed by (hash, workspace_id)
|
||||
static ref RESTART_UNLESS_CANCELLED_CACHE: Cache<(i64, String), (bool, Option<i32>)> = Cache::new(10000);
|
||||
|
||||
// Cache for workspace error handler settings with 60s TTL
|
||||
// Key: workspace_id, Value: (error_handler, error_handler_extra_args, error_handler_muted_on_cancel, error_handler_muted_on_user_path, expiry_timestamp)
|
||||
@@ -1538,21 +1538,27 @@ async fn restart_job_if_perpetual_inner(
|
||||
) -> Result<(), Error> {
|
||||
let cache_key = (hash.0, queued_job.workspace_id.clone());
|
||||
|
||||
let restart = if let Some(cached) = RESTART_UNLESS_CANCELLED_CACHE.get(&cache_key) {
|
||||
let (restart, script_timeout) = if let Some(cached) =
|
||||
RESTART_UNLESS_CANCELLED_CACHE.get(&cache_key)
|
||||
{
|
||||
cached
|
||||
} else {
|
||||
let restart = sqlx::query_scalar!(
|
||||
"SELECT restart_unless_cancelled FROM script WHERE hash = $1 AND workspace_id = $2",
|
||||
let row = sqlx::query!(
|
||||
"SELECT restart_unless_cancelled, timeout FROM script WHERE hash = $1 AND workspace_id = $2",
|
||||
hash.0,
|
||||
&queued_job.workspace_id
|
||||
)
|
||||
.fetch_optional(db)
|
||||
.await?
|
||||
.flatten()
|
||||
.unwrap_or(false);
|
||||
.await?;
|
||||
|
||||
RESTART_UNLESS_CANCELLED_CACHE.insert(cache_key, restart);
|
||||
restart
|
||||
let restart = row
|
||||
.as_ref()
|
||||
.and_then(|r| r.restart_unless_cancelled)
|
||||
.unwrap_or(false);
|
||||
let script_timeout = row.and_then(|r| r.timeout);
|
||||
|
||||
RESTART_UNLESS_CANCELLED_CACHE.insert(cache_key, (restart, script_timeout));
|
||||
(restart, script_timeout)
|
||||
};
|
||||
|
||||
if restart {
|
||||
@@ -1623,7 +1629,7 @@ async fn restart_job_if_perpetual_inner(
|
||||
None,
|
||||
true,
|
||||
Some(queued_job.tag.clone()),
|
||||
None,
|
||||
script_timeout,
|
||||
None,
|
||||
queued_job.priority,
|
||||
None,
|
||||
|
||||
@@ -198,6 +198,10 @@ async fn execute_mcp_tool_call(
|
||||
arguments: arguments.clone(),
|
||||
});
|
||||
|
||||
if let Some(parent_job) = ctx.parent_job {
|
||||
update_flow_status_module_with_actions(ctx.db, parent_job, actions).await?;
|
||||
}
|
||||
|
||||
match tool_result {
|
||||
Ok(result) => {
|
||||
let result_str =
|
||||
@@ -227,6 +231,10 @@ async fn execute_mcp_tool_call(
|
||||
stream_event_processor.send(event, final_events_str).await?;
|
||||
}
|
||||
|
||||
if let Some(parent_job) = ctx.parent_job {
|
||||
update_flow_status_module_with_actions_success(ctx.db, parent_job, true).await?;
|
||||
}
|
||||
|
||||
// Add tool message to conversation if chat_input_enabled
|
||||
let content = format!("Used {} tool", tool_call.function.name);
|
||||
add_tool_message_to_chat(ctx, None, &content, true).await;
|
||||
@@ -259,6 +267,10 @@ async fn execute_mcp_tool_call(
|
||||
stream_event_processor.send(event, final_events_str).await?;
|
||||
}
|
||||
|
||||
if let Some(parent_job) = ctx.parent_job {
|
||||
update_flow_status_module_with_actions_success(ctx.db, parent_job, false).await?;
|
||||
}
|
||||
|
||||
// Add tool message to conversation if chat_input_enabled
|
||||
add_tool_message_to_chat(ctx, None, &error_msg, false).await;
|
||||
}
|
||||
|
||||
@@ -1030,6 +1030,11 @@ pub async fn run_agent(
|
||||
// Add websearch tool message if websearch was used
|
||||
if used_websearch {
|
||||
actions.push(AgentAction::WebSearch {});
|
||||
if let Some(parent_job) = parent_job {
|
||||
update_flow_status_module_with_actions(db, parent_job, &actions).await?;
|
||||
update_flow_status_module_with_actions_success(db, parent_job, true)
|
||||
.await?;
|
||||
}
|
||||
messages.push(OpenAIMessage {
|
||||
role: "tool".to_string(),
|
||||
content: Some(OpenAIContent::Text(
|
||||
|
||||
@@ -481,7 +481,15 @@ try {{
|
||||
args.push("--allow-write=./");
|
||||
args.push("--allow-env");
|
||||
args.push("--allow-import");
|
||||
args.push("--allow-run=git,/usr/bin/chromium");
|
||||
// Deliberately NO --allow-run: unlike every other language, deno jobs
|
||||
// are never nsjail-wrapped, so the Deno permission model is the *only*
|
||||
// sandbox boundary. Any allowed binary that can spawn a subprocess
|
||||
// therefore escapes it entirely — git via hook configs
|
||||
// (`-c core.fsmonitor=<cmd>`) and chromium via subprocess-launcher flags
|
||||
// (`--renderer-cmd-prefix` / `--gpu-launcher`) both coerce /bin/sh and
|
||||
// defeat the guarantee (GHSA-gj6h-vw66-mr8f). Omitting the flag denies
|
||||
// all subprocess execution. Admins who accept the risk (e.g. puppeteer)
|
||||
// can re-add specific binaries via DENO_FLAGS.
|
||||
} else {
|
||||
args.push("-A");
|
||||
}
|
||||
|
||||
@@ -300,6 +300,12 @@ pub fn start_background_processor(
|
||||
worker_name: String,
|
||||
killpill_tx: KillpillSender,
|
||||
is_dedicated_worker: bool,
|
||||
// True when this processor runs inside the agent-worker API server, relaying
|
||||
// completions on behalf of many remote agent workers. Such a processor must
|
||||
// never kill itself: dropping its receiver would disconnect the shared
|
||||
// job-completed channel and make every future /send_result fail until the
|
||||
// whole server is restarted.
|
||||
is_agent_server: bool,
|
||||
stats_map: JobStatsMap,
|
||||
) -> JoinHandle<()> {
|
||||
tokio::spawn(async move {
|
||||
@@ -376,6 +382,7 @@ pub fn start_background_processor(
|
||||
jc.job.kind,
|
||||
JobKind::Dependencies | JobKind::FlowDependencies
|
||||
);
|
||||
let jc_id = jc.job.id;
|
||||
#[cfg(feature = "benchmark")]
|
||||
let bench_job_id = jc.job.id;
|
||||
#[cfg(feature = "benchmark")]
|
||||
@@ -403,9 +410,20 @@ pub fn start_background_processor(
|
||||
.await;
|
||||
|
||||
if is_init_script && !final_success {
|
||||
tracing::error!("init script errored, exiting");
|
||||
killpill_tx.send();
|
||||
break;
|
||||
if is_agent_server {
|
||||
// The failed init script belongs to a remote agent
|
||||
// worker, not to this server. That worker handles its
|
||||
// own restart; killing the server relay here would
|
||||
// strand every other agent worker's completions.
|
||||
tracing::error!(
|
||||
job_id = %jc_id,
|
||||
"agent worker init script errored; failure recorded, keeping server bg processor alive"
|
||||
);
|
||||
} else {
|
||||
tracing::error!("init script errored, exiting");
|
||||
killpill_tx.send();
|
||||
break;
|
||||
}
|
||||
}
|
||||
if is_dependency_job && is_dedicated_worker {
|
||||
tracing::error!("Dedicated worker executed a dependency job, a new script has been deployed. Exiting expecting to be restarted.");
|
||||
|
||||
@@ -2269,6 +2269,7 @@ pub async fn run_worker(
|
||||
worker_name.clone(),
|
||||
killpill_tx.clone(),
|
||||
is_dedicated_worker,
|
||||
false,
|
||||
stats_map,
|
||||
)),
|
||||
_ => None,
|
||||
|
||||
+1
-1
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
|
||||
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
|
||||
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
|
||||
|
||||
export const VERSION = "v1.751.0";
|
||||
export const VERSION = "v1.756.0";
|
||||
|
||||
export async function login(email: string, password: string): Promise<string> {
|
||||
return await windmill.UserService.login({
|
||||
|
||||
@@ -341,6 +341,72 @@ function normalizeRetry(retry: ParseAssetsRaw["retry"]): ParseAssetsRaw["retry"]
|
||||
return { ...retry, delay: retry.delay.replace(/^delay=/, "") };
|
||||
}
|
||||
|
||||
// Read-asset kinds whose read auto-derives a cascade trigger edge inside a
|
||||
// `// pipeline`. Mirror of backend `is_auto_trigger_kind` (windmill-common
|
||||
// assets.rs) / frontend `AUTO_TRIGGER_KINDS` (resolveGraph.ts) — ducklake
|
||||
// tables and s3 objects only; resource/datatable/volume stay explicit-`// on`.
|
||||
const AUTO_TRIGGER_KINDS = new Set(["ducklake", "s3object"]);
|
||||
|
||||
// Asset-URI prefixes accepted by `// mute <asset>`, in lockstep with the
|
||||
// canonical `parse_asset_syntax` / frontend `ASSET_PREFIXES`. Non-derivable
|
||||
// kinds are parsed too (their muted key simply never matches a derived edge),
|
||||
// so a mute line is never REinterpreted differently from the deploy path.
|
||||
const MUTE_ASSET_PREFIXES: [string, string][] = [
|
||||
["s3://", "s3object"],
|
||||
["res://", "resource"],
|
||||
["$res:", "resource"],
|
||||
["ducklake://", "ducklake"],
|
||||
["datatable://", "datatable"],
|
||||
["volume://", "volume"],
|
||||
];
|
||||
|
||||
// `// mute <asset>` / `// mute all` from the LEADING comment header, as
|
||||
// `<kind>:<path>` keys. Parsed locally because the pinned wasm asset parser
|
||||
// predates the mute annotations; mirrors the canonical parsers (Rust
|
||||
// `parse_pipeline_annotations`, frontend parsePipelineAnnotations.ts): any of
|
||||
// the three comment prefixes is accepted regardless of language, blank lines
|
||||
// are skipped, scanning stops at the first non-comment line, and `mute` must
|
||||
// be a complete word (`// muted for now` never matches).
|
||||
export function parseMuteAnnotations(content: string): {
|
||||
muteAll: boolean;
|
||||
muted: Set<string>;
|
||||
} {
|
||||
const muted = new Set<string>();
|
||||
let muteAll = false;
|
||||
for (const rawLine of content.split("\n")) {
|
||||
const line = rawLine.trimStart();
|
||||
if (line === "") continue;
|
||||
let rest: string;
|
||||
if (line.startsWith("//")) rest = line.slice(2);
|
||||
else if (line.startsWith("--")) rest = line.slice(2);
|
||||
else if (line.startsWith("#")) rest = line.slice(1);
|
||||
else break;
|
||||
rest = rest.trimStart();
|
||||
if (!rest.startsWith("mute")) continue;
|
||||
const after = rest.slice("mute".length);
|
||||
if (after !== "" && !/^\s/.test(after)) continue;
|
||||
const arg = after.trim();
|
||||
if (arg === "all") {
|
||||
muteAll = true;
|
||||
continue;
|
||||
}
|
||||
for (const [prefix, kind] of MUTE_ASSET_PREFIXES) {
|
||||
if (arg.startsWith(prefix)) {
|
||||
// S3 canonicalization as in `parse_asset_syntax`: strip every leading
|
||||
// slash so `s3:///key` (default storage) mutes the same node as the
|
||||
// inferred bare `key`.
|
||||
const p =
|
||||
kind === "s3object"
|
||||
? arg.slice(prefix.length).replace(/^\/+/, "")
|
||||
: arg.slice(prefix.length);
|
||||
muted.add(`${kind}:${p}`);
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
return { muteAll, muted };
|
||||
}
|
||||
|
||||
// Comment prefix for `volume:` annotations. Deliberately NOT `commentPrefix`
|
||||
// above (which returns `--` for SQL): volume annotations are only recognized for
|
||||
// the languages the backend/frontend recognize them for — mirrors
|
||||
@@ -685,6 +751,49 @@ export async function buildLocalPipelineGraph(args: {
|
||||
runnable_path: s.path,
|
||||
});
|
||||
}
|
||||
// Auto-derived cascade edges (backend `derive_pipeline_asset_trigger_refs`,
|
||||
// frontend `deriveAutoAssetTriggers`): a pipeline script's read-only
|
||||
// ducklake/s3 input wires its cascade trigger straight from the body read,
|
||||
// so `// on <asset>` is only needed for edges inference can't see. Skipped:
|
||||
// assets this script also writes — including the `// materialize` target and
|
||||
// its scd2 `_current` companion, whose writes the body SELECT doesn't
|
||||
// express — plus `// mute <asset>` opt-outs and explicit `// on` (which wins
|
||||
// the dedup); `// mute all` opts the script out of derivation entirely.
|
||||
// Ambiguous access (no `access_type`) fails safe and derives nothing.
|
||||
const mute = parseMuteAnnotations(s.content);
|
||||
if (!mute.muteAll) {
|
||||
const skip = new Set<string>(mute.muted);
|
||||
for (const a of out.assets ?? []) {
|
||||
if (a.access_type === "w" || a.access_type === "rw") {
|
||||
skip.add(`${a.kind}:${a.path}`);
|
||||
}
|
||||
}
|
||||
if (mat) {
|
||||
skip.add(`${mat.target_kind}:${mat.target_path}`);
|
||||
if (mat.scd2 && !mat.manual) {
|
||||
skip.add(`${mat.target_kind}:${mat.target_path}_current`);
|
||||
}
|
||||
}
|
||||
for (const t of out.triggers ?? []) {
|
||||
if (t.kind === "asset") {
|
||||
const at = t as { kind: "asset"; asset_kind: string; path: string };
|
||||
skip.add(`${at.asset_kind}:${at.path}`);
|
||||
}
|
||||
}
|
||||
for (const a of out.assets ?? []) {
|
||||
if (a.access_type !== "r" || !AUTO_TRIGGER_KINDS.has(a.kind)) continue;
|
||||
const key = `${a.kind}:${a.path}`;
|
||||
if (skip.has(key)) continue;
|
||||
skip.add(key);
|
||||
triggers.push({
|
||||
trigger_kind: "asset",
|
||||
asset_kind: a.kind,
|
||||
asset_path: a.path,
|
||||
runnable_kind: "script",
|
||||
runnable_path: s.path,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
const macroEdges = buildMacroEdges(all, libMacros, runnables);
|
||||
|
||||
@@ -131,41 +131,6 @@ async function createWorkspaceFork(
|
||||
return;
|
||||
}
|
||||
|
||||
// When we're converting the current branch into the fork branch, default
|
||||
// the fork's name/id to that branch — almost always what you want, and it
|
||||
// keeps the fork branch named after the work you already have
|
||||
// (wm-fork/<base>/<branch>). Interactive: pre-fill the prompt (press enter
|
||||
// to accept). Non-interactive (`--yes`): use it automatically.
|
||||
const branchDefaultId = renameCurrent ? branchToForkId(currentBranch) : undefined;
|
||||
const interactive = process.stdin.isTTY && opts.yes !== true;
|
||||
|
||||
if (workspaceName === undefined) {
|
||||
if (branchDefaultId && !interactive) {
|
||||
workspaceName = branchDefaultId;
|
||||
log.info(`Naming the fork after the current branch: \`${workspaceName}\``);
|
||||
} else {
|
||||
workspaceName = await Input.prompt({
|
||||
message: "Name this forked workspace:",
|
||||
default: branchDefaultId,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
if (!workspaceId) {
|
||||
// The id (unlike the display name) must be a valid slug — derive it from
|
||||
// the name rather than using the free-form name verbatim.
|
||||
const idDefault = branchToForkId(workspaceName);
|
||||
if (branchDefaultId && !interactive) {
|
||||
workspaceId = idDefault;
|
||||
} else {
|
||||
workspaceId = await Input.prompt({
|
||||
message: `Enter the ID of this forked workspace, it will then be prefixed by ${WM_FORK_PREFIX}. It will also determine the branch name`,
|
||||
default: idDefault,
|
||||
suggestions: [idDefault],
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
const token = workspace.token;
|
||||
|
||||
if (!token) {
|
||||
@@ -178,6 +143,62 @@ async function createWorkspaceFork(
|
||||
remote.endsWith("/") ? remote.substring(0, remote.length - 1) : remote
|
||||
);
|
||||
|
||||
// Default the fork's display name to "<parent>'s fork", using the parent
|
||||
// workspace's actual name (fall back to the local profile name / id if the
|
||||
// lookup fails). It's only a default — always overridable.
|
||||
let parentName = workspace.name;
|
||||
try {
|
||||
const fetched = await wmill.getWorkspaceName({ workspace: workspace.workspaceId });
|
||||
if (fetched) parentName = fetched;
|
||||
} catch {
|
||||
// Non-fatal: keep the local profile name / id as the fallback.
|
||||
}
|
||||
// Cap the auto default at the workspace.name limit (varchar(50)): a valid
|
||||
// parent name can be up to 50 chars, so appending "'s fork" would otherwise
|
||||
// push the default over the limit and trip the length guard below.
|
||||
const forkSuffix = "'s fork";
|
||||
const nameBase = parentName || workspace.workspaceId;
|
||||
const defaultForkName =
|
||||
nameBase.length + forkSuffix.length <= 50
|
||||
? `${nameBase}${forkSuffix}`
|
||||
: `${nameBase.slice(0, 50 - forkSuffix.length).trimEnd()}${forkSuffix}`;
|
||||
|
||||
// The fork branch (and thus the id) stays named after the work you already
|
||||
// have when converting the current branch into the fork branch
|
||||
// (wm-fork/<base>/<branch>); the display name is independent.
|
||||
const branchDefaultId = renameCurrent ? branchToForkId(currentBranch) : undefined;
|
||||
const interactive = process.stdin.isTTY && opts.yes !== true;
|
||||
|
||||
if (workspaceName === undefined) {
|
||||
if (interactive) {
|
||||
workspaceName = await Input.prompt({
|
||||
message: "Friendly name for the forked workspace (shown in the UI, may contain spaces):",
|
||||
default: defaultForkName,
|
||||
});
|
||||
} else {
|
||||
workspaceName = defaultForkName;
|
||||
log.info(`Naming the fork \`${workspaceName}\` (override with the workspace_name argument).`);
|
||||
}
|
||||
}
|
||||
|
||||
if (!workspaceId) {
|
||||
// The id must be a valid slug. Prefer the branch-derived id when renaming a
|
||||
// working branch (keeps id/branch aligned). Otherwise slugify the name —
|
||||
// but for the auto default name ("<parent>'s fork") slugify "<parent>-fork"
|
||||
// instead, so the id/branch is `<parent>-fork` rather than `<parent>-s-fork`.
|
||||
const idBasis = workspaceName === defaultForkName ? `${parentName}-fork` : workspaceName;
|
||||
const idDefault = branchDefaultId ?? branchToForkId(idBasis);
|
||||
if (interactive) {
|
||||
workspaceId = await Input.prompt({
|
||||
message: `Id for the forked workspace (a slug: no spaces or special characters). It will be prefixed with '${WM_FORK_PREFIX}-' and also determines the git branch name. The suggested default is normalized from the name (or the branch when converting one into the fork branch)`,
|
||||
default: idDefault,
|
||||
suggestions: [idDefault],
|
||||
});
|
||||
} else {
|
||||
workspaceId = idDefault;
|
||||
}
|
||||
}
|
||||
|
||||
log.info(colors.blue(`Creating forked workspace: ${workspaceName}...`));
|
||||
|
||||
const trueWorkspaceId = `${WM_FORK_PREFIX}-${workspaceId}`;
|
||||
@@ -390,7 +411,7 @@ async function createWorkspaceFork(
|
||||
|
||||
log.info(`${checkoutHint}
|
||||
|
||||
When doing operations on the forked workspace, it will use the remote setup in the workspaces section for the branch it was forked from.
|
||||
While on the branch \`${newBranchName}\`, every wmill command (sync pull/push, script run, ...) automatically targets the fork workspace \`${trueWorkspaceId}\` — no --workspace flag or profile switch needed. The remote comes from the base branch \`${clonedBranchName}\`'s entry in wmill.yaml's workspaces section, and auth is reused from that workspace's saved profile.
|
||||
|
||||
To merge changes back to the parent workspace, you can:
|
||||
- Use the CLI: ` + colors.white(`git checkout ${newBranchName} && wmill workspace merge`) + `
|
||||
|
||||
@@ -86,6 +86,29 @@ export async function getWorkspaceByName(
|
||||
return undefined;
|
||||
}
|
||||
|
||||
// When the current git branch is a fork branch (wm-fork/<base>/<id>), commands
|
||||
// resolve the fork workspace from the branch name and ignore the active
|
||||
// profile — surface that wherever we display the active workspace, so users
|
||||
// don't act on the wrong "Active:" line.
|
||||
async function forkBranchAutoTargetNote(): Promise<string | undefined> {
|
||||
const {
|
||||
getCurrentGitBranch,
|
||||
getOriginalBranchForWorkspaceForks,
|
||||
getWorkspaceIdForWorkspaceForkFromBranchName,
|
||||
} = await import("../../utils/git.ts");
|
||||
const branch = getCurrentGitBranch();
|
||||
if (!branch || !getOriginalBranchForWorkspaceForks(branch)) {
|
||||
return undefined;
|
||||
}
|
||||
const forkWorkspaceId = getWorkspaceIdForWorkspaceForkFromBranchName(branch);
|
||||
if (!forkWorkspaceId) {
|
||||
return undefined;
|
||||
}
|
||||
return (
|
||||
`Note: you are on fork branch \`${branch}\` — wmill commands automatically target the fork workspace \`${forkWorkspaceId}\` here, regardless of the active profile. Use --workspace to override.`
|
||||
);
|
||||
}
|
||||
|
||||
export async function list(opts: GlobalOptions) {
|
||||
const workspaces = await allWorkspaces(opts.configDir);
|
||||
const activeName = await getActiveWorkspaceName(opts);
|
||||
@@ -107,6 +130,11 @@ export async function list(opts: GlobalOptions) {
|
||||
.render();
|
||||
|
||||
log.info("Active: " + colors.green.bold(activeName || "none"));
|
||||
|
||||
const forkNote = await forkBranchAutoTargetNote();
|
||||
if (forkNote) {
|
||||
log.info(colors.yellow(forkNote));
|
||||
}
|
||||
}
|
||||
|
||||
async function switchC(opts: GlobalOptions, workspaceName: string) {
|
||||
@@ -140,6 +168,10 @@ async function switchC(opts: GlobalOptions, workspaceName: string) {
|
||||
`Switched to workspace ${workspaceName} (${workspace?.workspaceId} on ${workspace?.remote})`
|
||||
)
|
||||
);
|
||||
const forkNote = await forkBranchAutoTargetNote();
|
||||
if (forkNote) {
|
||||
log.info(colors.yellow(forkNote));
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -821,7 +853,15 @@ const command = new Command()
|
||||
.option("--workspace <name:string>", "Workspace to unbind")
|
||||
.action((opts) => bind(opts as any, false))
|
||||
.command("fork")
|
||||
.description("Create a forked workspace")
|
||||
.description(
|
||||
`Create a forked workspace from its parent workspace.
|
||||
|
||||
The parent is resolved from your current git branch, not from the active profile: run this from a git repo checked out on the branch mapped to the parent workspace in wmill.yaml's \`workspaces:\` section (a fork branch of it resolves to the same parent). \`wmill workspace switch\` does not change which workspace is forked.
|
||||
|
||||
Arguments (omit both to be prompted interactively):
|
||||
[workspace_name] Friendly display name for the fork, shown in the UI. May contain spaces, so quote it in the shell (e.g. "My Fork"). Max 50 chars. Defaults to "<parent workspace name>'s fork".
|
||||
[workspace_id] Id for the fork. Must be a slug (no spaces or special characters) and is automatically prefixed with \`wm-fork-\`, so pass just the bare slug (e.g. \`my-fork\` becomes \`wm-fork-my-fork\`). This id also determines the fork's git branch name. Defaults to a slug derived from the name — or, when you are converting an existing branch into the fork branch, from that branch.`
|
||||
)
|
||||
.arguments("[workspace_name:string] [workspace_id:string]")
|
||||
.option(
|
||||
"--create-workspace-name <workspace_name:string>",
|
||||
|
||||
@@ -405,7 +405,9 @@ export async function validateBranchConfiguration(
|
||||
|
||||
let currentBranch: string | null;
|
||||
if (originalBranchIfForked) {
|
||||
log.info(
|
||||
// The fork targeting itself is announced by tryResolveBranchWorkspace;
|
||||
// this validation detail stays at debug to avoid a near-duplicate line.
|
||||
log.debug(
|
||||
`Workspace fork detected from branch name \`${rawBranch}\`. Validating workspace configuration using original branch \`${originalBranchIfForked}\``
|
||||
);
|
||||
currentBranch = originalBranchIfForked;
|
||||
@@ -563,7 +565,7 @@ export async function getEffectiveSettings(
|
||||
|
||||
const branch = originalBranchIfForked ?? rawGitBranch;
|
||||
if (originalBranchIfForked) {
|
||||
log.info(
|
||||
log.debug(
|
||||
`Using overrides from original branch \`${originalBranchIfForked}\``
|
||||
);
|
||||
}
|
||||
|
||||
@@ -10,4 +10,4 @@ export const WM_FORK_PREFIX = "wm-fork";
|
||||
// (e.g. utils.ts) can read it without importing main.ts and creating a circular
|
||||
// dependency (main → workspace → utils → main) that triggers a TDZ.
|
||||
// Re-exported from main.ts for backwards compatibility.
|
||||
export const VERSION = "1.751.0";
|
||||
export const VERSION = "1.756.0";
|
||||
|
||||
+54
-55
@@ -326,12 +326,9 @@ export async function tryResolveBranchWorkspace(
|
||||
workspaceIdIfForked =
|
||||
getWorkspaceIdForWorkspaceForkFromBranchName(rawBranch);
|
||||
|
||||
// The "matched via fork branch" reason logged below already explains the
|
||||
// base-branch lookup, so no extra message here.
|
||||
const branchToLookup = originalBranchIfForked ?? rawBranch;
|
||||
if (originalBranchIfForked) {
|
||||
log.infoStderr(
|
||||
`Using original branch \`${originalBranchIfForked}\` for finding workspace from workspaces section in wmill.yaml`
|
||||
);
|
||||
}
|
||||
|
||||
const match = findWorkspaceByGitBranch(config.workspaces, branchToLookup);
|
||||
if (match) {
|
||||
@@ -370,9 +367,9 @@ export async function tryResolveBranchWorkspace(
|
||||
reason = `matched current git branch '${rawBranch}'`;
|
||||
}
|
||||
|
||||
log.infoStderr(
|
||||
`Using workspace '${wsName}' (${reason}) → ${workspaceId} on ${baseUrl}`
|
||||
);
|
||||
// Printed as part of the single final targeting line on the happy paths;
|
||||
// logged up front only when an interactive flow needs the context first.
|
||||
const workspaceLine = `Using workspace '${wsName}' (${reason}) → ${workspaceId} on ${baseUrl}`;
|
||||
|
||||
let normalizedBaseUrl: string;
|
||||
try {
|
||||
@@ -390,27 +387,34 @@ export async function tryResolveBranchWorkspace(
|
||||
(w) => w.remote === normalizedBaseUrl && w.workspaceId === workspaceId
|
||||
);
|
||||
|
||||
// Every branch below must flow into the shared fork handling at the end —
|
||||
// returning a profile directly would hand back the parent workspace profile
|
||||
// on a fork branch.
|
||||
let selectedProfile: Workspace;
|
||||
let profileNote: string;
|
||||
// Set when workspaceLine was already printed as context for the interactive
|
||||
// profile-creation flow, so it isn't printed a second time at the end.
|
||||
let workspaceLinePrinted = false;
|
||||
|
||||
if (matchingProfiles.length === 0) {
|
||||
// No matching profile exists - prompt to create one
|
||||
return await createWorkspaceProfileInteractively(
|
||||
log.infoStderr(workspaceLine);
|
||||
workspaceLinePrinted = true;
|
||||
const created = await createWorkspaceProfileInteractively(
|
||||
normalizedBaseUrl,
|
||||
workspaceId,
|
||||
wsName,
|
||||
opts,
|
||||
{ rawBranch: rawBranch ?? wsName, isForked: !!originalBranchIfForked }
|
||||
);
|
||||
}
|
||||
|
||||
// Handle multiple profiles
|
||||
let selectedProfile: Workspace;
|
||||
|
||||
if (matchingProfiles.length === 1) {
|
||||
if (!created) {
|
||||
return undefined;
|
||||
}
|
||||
selectedProfile = created;
|
||||
profileNote = `profile '${selectedProfile.name}'`;
|
||||
} else if (matchingProfiles.length === 1) {
|
||||
selectedProfile = matchingProfiles[0];
|
||||
log.infoStderr(
|
||||
colors.green(
|
||||
`Using workspace profile '${selectedProfile.name}' for workspace '${wsName}' with workspace id \`${workspaceId}\``
|
||||
)
|
||||
);
|
||||
profileNote = `profile '${selectedProfile.name}'`;
|
||||
} else {
|
||||
const lastUsedName = await getLastUsedProfile(
|
||||
wsName,
|
||||
@@ -418,50 +422,45 @@ export async function tryResolveBranchWorkspace(
|
||||
workspaceId,
|
||||
opts.configDir
|
||||
);
|
||||
const lastUsedProfile = lastUsedName
|
||||
? matchingProfiles.find((p) => p.name === lastUsedName)
|
||||
: undefined;
|
||||
|
||||
if (lastUsedName) {
|
||||
const lastUsedProfile = matchingProfiles.find(
|
||||
(p) => p.name === lastUsedName
|
||||
if (lastUsedProfile) {
|
||||
selectedProfile = lastUsedProfile;
|
||||
profileNote = `last used profile '${selectedProfile.name}'`;
|
||||
} else {
|
||||
// selectFromMultipleProfiles prints its own context header, and the
|
||||
// final summary line below names the chosen profile.
|
||||
selectedProfile = await selectFromMultipleProfiles(
|
||||
matchingProfiles,
|
||||
normalizedBaseUrl,
|
||||
workspaceId,
|
||||
`workspace '${wsName}'`,
|
||||
opts.configDir
|
||||
);
|
||||
if (lastUsedProfile) {
|
||||
log.infoStderr(
|
||||
colors.green(
|
||||
`Using workspace profile '${lastUsedProfile.name}' for workspace '${wsName}' (last used)`
|
||||
)
|
||||
);
|
||||
return lastUsedProfile;
|
||||
}
|
||||
|
||||
await setLastUsedProfile(
|
||||
wsName,
|
||||
normalizedBaseUrl,
|
||||
workspaceId,
|
||||
selectedProfile.name,
|
||||
opts.configDir
|
||||
);
|
||||
profileNote = `profile '${selectedProfile.name}'`;
|
||||
}
|
||||
|
||||
selectedProfile = await selectFromMultipleProfiles(
|
||||
matchingProfiles,
|
||||
normalizedBaseUrl,
|
||||
workspaceId,
|
||||
`workspace '${wsName}'`,
|
||||
opts.configDir
|
||||
);
|
||||
|
||||
await setLastUsedProfile(
|
||||
wsName,
|
||||
normalizedBaseUrl,
|
||||
workspaceId,
|
||||
selectedProfile.name,
|
||||
opts.configDir
|
||||
);
|
||||
|
||||
log.infoStderr(
|
||||
colors.green(
|
||||
`Using workspace profile '${selectedProfile.name}' for workspace '${wsName}'`
|
||||
)
|
||||
);
|
||||
}
|
||||
|
||||
if (workspaceIdIfForked) {
|
||||
selectedProfile.name = `${selectedProfile.name}/${workspaceIdIfForked}`;
|
||||
selectedProfile.workspaceId = workspaceIdIfForked;
|
||||
log.infoStderr(
|
||||
`Using fork workspace \`${workspaceIdIfForked}\` (parent: \`${workspaceId}\`) from branch \`${rawBranch}\``
|
||||
colors.green(
|
||||
`Automatically targeting fork workspace \`${workspaceIdIfForked}\` (fork of \`${workspaceId}\` on ${baseUrl}, ${profileNote}), resolved from git branch \`${rawBranch}\`. Use --workspace to override.`
|
||||
)
|
||||
);
|
||||
} else if (!workspaceLinePrinted) {
|
||||
log.infoStderr(`${workspaceLine} (${profileNote})`);
|
||||
}
|
||||
|
||||
return selectedProfile;
|
||||
@@ -556,7 +555,7 @@ export async function resolveWorkspace(
|
||||
return workspace;
|
||||
} else {
|
||||
log.infoStderr(
|
||||
`Found an active workspace \`${workspace.name}\` but the branch name indicates this is a forked workspace. Ignoring active workspace and trying to resolve the correct workspace from the branch name \`${branch}\`. Use --workspace to override.`
|
||||
`You are on fork branch \`${branch}\`, which takes precedence over the active workspace profile \`${workspace.name}\`: resolving the fork workspace from the branch name instead. Use --workspace to override.`
|
||||
);
|
||||
}
|
||||
} else if (opts.workspace) {
|
||||
|
||||
@@ -177,6 +177,8 @@ Just run \`wmill workspace fork\` — it adapts to where you are:
|
||||
|
||||
For non-interactive runs from a working branch, pass \`--from-branch <base>\` to skip the prompts. The CLI refuses to rename a base branch.
|
||||
|
||||
While a \`wm-fork/<base>/<id>\` branch is checked out, every wmill command automatically targets the fork workspace (resolved from the branch name), reusing the base branch workspace's remote (from wmill.yaml) and its saved profile's auth — no \`--workspace\` flag or profile switch needed. Pass \`--workspace\` to target a different workspace explicitly.
|
||||
|
||||
Merge a fork back into its parent with \`wmill workspace merge\` (or the Merge UI on the fork's home page). Full reference: https://www.windmill.dev/docs/advanced/workspace_forks
|
||||
|
||||
## Debugging Jobs
|
||||
|
||||
File diff suppressed because one or more lines are too long
@@ -0,0 +1,93 @@
|
||||
import { describe, expect, test } from "bun:test";
|
||||
import { mkdtemp, rm, writeFile } from "node:fs/promises";
|
||||
import { execSync } from "node:child_process";
|
||||
import os from "node:os";
|
||||
import path from "node:path";
|
||||
import { stringify as yamlStringify } from "yaml";
|
||||
|
||||
import { tryResolveBranchWorkspace } from "../src/core/context.ts";
|
||||
import { setLastUsedProfile } from "../src/core/branch-profiles.ts";
|
||||
import { getWorkspaceConfigFilePath } from "../windmill-utils-internal/src/config/config.ts";
|
||||
import type { GlobalOptions } from "../src/types.ts";
|
||||
import type { Workspace } from "../src/commands/workspace/workspace.ts";
|
||||
|
||||
const BASE_URL = "http://localhost:9999/";
|
||||
const PARENT_WORKSPACE_ID = "parent";
|
||||
|
||||
// Fork-branch workspace resolution: every profile-selection path in
|
||||
// tryResolveBranchWorkspace must rewrite the returned profile to the fork
|
||||
// workspace id derived from the wm-fork/<base>/<id> branch — a path that
|
||||
// returns the parent profile untouched silently targets the parent workspace.
|
||||
async function withForkBranchSetup(
|
||||
profiles: Workspace[],
|
||||
fn: (opts: GlobalOptions) => Promise<void>,
|
||||
): Promise<void> {
|
||||
const repoDir = await mkdtemp(path.join(os.tmpdir(), "wmill_fork_repo_"));
|
||||
const configDir = await mkdtemp(path.join(os.tmpdir(), "wmill_fork_conf_"));
|
||||
const originalCwd = process.cwd();
|
||||
try {
|
||||
execSync(`git init -q -b wm-fork/main/myfork`, { cwd: repoDir });
|
||||
execSync(`git config user.email test@example.com`, { cwd: repoDir });
|
||||
execSync(`git config user.name test`, { cwd: repoDir });
|
||||
|
||||
await writeFile(
|
||||
path.join(repoDir, "wmill.yaml"),
|
||||
yamlStringify({
|
||||
workspaces: {
|
||||
main: { baseUrl: BASE_URL, workspaceId: PARENT_WORKSPACE_ID },
|
||||
},
|
||||
}),
|
||||
);
|
||||
execSync(`git add wmill.yaml && git commit -q -m init`, { cwd: repoDir });
|
||||
|
||||
const remotesPath = await getWorkspaceConfigFilePath(configDir);
|
||||
await writeFile(
|
||||
remotesPath,
|
||||
profiles.map((p) => JSON.stringify(p)).join("\n") + "\n",
|
||||
);
|
||||
|
||||
process.chdir(repoDir);
|
||||
await fn({ configDir } as GlobalOptions);
|
||||
} finally {
|
||||
process.chdir(originalCwd);
|
||||
await rm(repoDir, { recursive: true, force: true });
|
||||
await rm(configDir, { recursive: true, force: true });
|
||||
}
|
||||
}
|
||||
|
||||
function profile(name: string): Workspace {
|
||||
return {
|
||||
name,
|
||||
remote: BASE_URL,
|
||||
workspaceId: PARENT_WORKSPACE_ID,
|
||||
token: `token-${name}`,
|
||||
};
|
||||
}
|
||||
|
||||
describe("tryResolveBranchWorkspace on a fork branch", () => {
|
||||
test("single matching profile targets the fork workspace", async () => {
|
||||
await withForkBranchSetup([profile("prod")], async (opts) => {
|
||||
const ws = await tryResolveBranchWorkspace(opts);
|
||||
expect(ws?.workspaceId).toEqual("wm-fork-myfork");
|
||||
expect(ws?.token).toEqual("token-prod");
|
||||
});
|
||||
});
|
||||
|
||||
test("last-used profile among multiple still targets the fork workspace", async () => {
|
||||
await withForkBranchSetup(
|
||||
[profile("prod"), profile("prod-alt")],
|
||||
async (opts) => {
|
||||
await setLastUsedProfile(
|
||||
"main",
|
||||
BASE_URL,
|
||||
PARENT_WORKSPACE_ID,
|
||||
"prod-alt",
|
||||
opts.configDir,
|
||||
);
|
||||
const ws = await tryResolveBranchWorkspace(opts);
|
||||
expect(ws?.workspaceId).toEqual("wm-fork-myfork");
|
||||
expect(ws?.token).toEqual("token-prod-alt");
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -3,7 +3,10 @@ import { mkdtempSync, mkdirSync, writeFileSync, rmSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import { join } from "node:path";
|
||||
|
||||
import { buildLocalPipelineGraph } from "../src/commands/pipeline/localGraph.ts";
|
||||
import {
|
||||
buildLocalPipelineGraph,
|
||||
parseMuteAnnotations,
|
||||
} from "../src/commands/pipeline/localGraph.ts";
|
||||
|
||||
// Build a throwaway workspace tree with `f/<folder>/<file>` scripts and a
|
||||
// wmill.yaml at the root, then assert the graph the wasm-backed builder derives.
|
||||
@@ -290,6 +293,120 @@ test("HD-2: an scd2 `history` producer writes both `<dim>` and `<dim>_current` s
|
||||
);
|
||||
});
|
||||
|
||||
test("auto-derived cascade triggers: a body ducklake read wires the edge, incl. the scd2 `_current` view", async () => {
|
||||
// Backend parity (#9963 `derive_pipeline_asset_trigger_refs`): inside a
|
||||
// `// pipeline`, a read-only ducklake/s3 body read derives its cascade
|
||||
// trigger — no `// on` needed. The scd2 `_current` companion is both written
|
||||
// by the producer AND a derivable read for its consumer, so the full chain
|
||||
// stg → dim → consumer must connect without a single explicit trigger.
|
||||
await withFolder(
|
||||
{
|
||||
"stg.duckdb.sql":
|
||||
`-- pipeline\n-- materialize ducklake://main/stg\nSELECT 1 AS id;\n`,
|
||||
"dim.duckdb.sql":
|
||||
`-- pipeline\n-- materialize ducklake://main/dim key=id history\nATTACH 'ducklake' AS dl;\nUSE dl;\nSELECT * FROM stg;\n`,
|
||||
"consume.duckdb.sql":
|
||||
`-- pipeline\nATTACH 'ducklake' AS dl;\nUSE dl;\nSELECT * FROM dim_current;\n`,
|
||||
},
|
||||
async (root, folder) => {
|
||||
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
|
||||
const ats = graph.triggers.filter((t) => t.trigger_kind === "asset") as Extract<
|
||||
(typeof graph.triggers)[number],
|
||||
{ trigger_kind: "asset" }
|
||||
>[];
|
||||
expect(
|
||||
ats.map((t) => `${t.asset_path}->${t.runnable_path}`).sort(),
|
||||
).toEqual(["main/dim_current->f/mypipe/consume", "main/stg->f/mypipe/dim"]);
|
||||
// the schema-level `main` read (ambiguous access) must NOT derive a trigger
|
||||
expect(ats.some((t) => t.asset_path === "main")).toBe(false);
|
||||
// the scd2 producer still carries the `_current` companion write the
|
||||
// derived consumer edge resolves against (deployed-graph parity)
|
||||
expect(graph.edges).toContainEqual({
|
||||
runnable_kind: "script",
|
||||
runnable_path: "f/mypipe/dim",
|
||||
asset_kind: "ducklake",
|
||||
asset_path: "main/dim_current",
|
||||
access_type: "w",
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test("`// mute <asset>` suppresses one derived trigger; `// mute all` opts the script out", async () => {
|
||||
await withFolder(
|
||||
{
|
||||
// reads two tables, mutes one → only the unmuted read derives
|
||||
"partial.duckdb.sql":
|
||||
`-- pipeline\n-- mute ducklake://main/lookup\nATTACH 'ducklake' AS dl;\nUSE dl;\nSELECT * FROM lookup JOIN facts USING (id);\n`,
|
||||
// mute all: body read derives nothing, the explicit `// on` still stands
|
||||
"optout.duckdb.sql":
|
||||
`-- pipeline\n-- mute all\n-- on ducklake://main/manual\nATTACH 'ducklake' AS dl;\nUSE dl;\nSELECT * FROM facts;\n`,
|
||||
},
|
||||
async (root, folder) => {
|
||||
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
|
||||
const ats = graph.triggers.filter((t) => t.trigger_kind === "asset") as Extract<
|
||||
(typeof graph.triggers)[number],
|
||||
{ trigger_kind: "asset" }
|
||||
>[];
|
||||
expect(
|
||||
ats.map((t) => `${t.asset_path}->${t.runnable_path}`).sort(),
|
||||
).toEqual(["main/facts->f/mypipe/partial", "main/manual->f/mypipe/optout"]);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test("derived triggers dedup against explicit `// on` and never self-trigger a materialize producer", async () => {
|
||||
await withFolder(
|
||||
{
|
||||
// explicit `// on` for an asset the body also reads → exactly one trigger
|
||||
"explicit.duckdb.sql":
|
||||
`-- pipeline\n-- on ducklake://main/src debounce=60s\nATTACH 'ducklake' AS dl;\nUSE dl;\nSELECT * FROM src;\n`,
|
||||
// an incremental model reading its own materialize target must not
|
||||
// cascade on itself (the deploy path upgrades that read to rw)
|
||||
"incremental.duckdb.sql":
|
||||
`-- pipeline\n-- materialize ducklake://main/inc key=id\nATTACH 'ducklake' AS dl;\nUSE dl;\nSELECT * FROM inc;\n`,
|
||||
},
|
||||
async (root, folder) => {
|
||||
const { graph } = await buildLocalPipelineGraph({ root, folder, defaultTs: "bun" });
|
||||
const ats = graph.triggers.filter((t) => t.trigger_kind === "asset") as Extract<
|
||||
(typeof graph.triggers)[number],
|
||||
{ trigger_kind: "asset" }
|
||||
>[];
|
||||
expect(
|
||||
ats.map((t) => `${t.asset_path}->${t.runnable_path}`),
|
||||
).toEqual(["main/src->f/mypipe/explicit"]);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
test("parseMuteAnnotations mirrors the canonical annotation grammar", () => {
|
||||
// Any comment prefix regardless of language, header-only scan, complete-word
|
||||
// keyword, s3 leading-slash canonicalization — in lockstep with the Rust
|
||||
// `parse_pipeline_annotations` / frontend parsePipelineAnnotations.ts.
|
||||
const all3 = parseMuteAnnotations(
|
||||
`// mute ducklake://main/a\n-- mute datatable://main/b\n# mute s3:///lead/slash\nSELECT 1;\n// mute ducklake://main/body\n`,
|
||||
);
|
||||
expect(all3.muteAll).toBe(false);
|
||||
// all three prefixes accepted; s3 triple-slash canonicalizes to the bare key;
|
||||
// the line PAST the first non-comment line is ignored (header-only)
|
||||
expect([...all3.muted].sort()).toEqual([
|
||||
"datatable:main/b",
|
||||
"ducklake:main/a",
|
||||
"s3object:lead/slash",
|
||||
]);
|
||||
|
||||
// `mute` must be a complete word, and prose args are not asset URIs
|
||||
const prose = parseMuteAnnotations(
|
||||
`// muted for now\n// mutex ducklake://main/x\n// mute for now\n`,
|
||||
);
|
||||
expect(prose.muteAll).toBe(false);
|
||||
expect(prose.muted.size).toBe(0);
|
||||
|
||||
// `mute all` sets the opt-out; blank header lines are skipped, not a stop
|
||||
const optout = parseMuteAnnotations(`-- pipeline\n\n-- mute all\nSELECT 1;\n`);
|
||||
expect(optout.muteAll).toBe(true);
|
||||
});
|
||||
|
||||
test("a bare `.sql` (ambiguous dialect) is skipped, not a build-aborting crash", async () => {
|
||||
// `inferContentTypeFromFilePath` throws on a dialect-less `.sql`; one such file
|
||||
// must not abort the whole graph build (it also wedged `pipeline dev` at start).
|
||||
|
||||
+10
-2
@@ -6,14 +6,22 @@ RUN apt-get update && apt-get install -y curl gnupg2
|
||||
RUN curl "https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.0-1_all.deb" -o cuda.deb && \
|
||||
dpkg -i cuda.deb && rm cuda.deb
|
||||
|
||||
RUN apt-get update -y && \
|
||||
# NVIDIA's CUDA apt repo signing key carries a SHA1 self-binding signature,
|
||||
# which the Debian trixie base image's Sequoia-based apt verifier (sqv) rejects
|
||||
# as of 2026-02-01, leaving the repo treated as unsigned. Re-enable SHA1 via a
|
||||
# scoped crypto policy applied only to the apt runs that touch the CUDA repo.
|
||||
RUN printf '[hash_algorithms.sha1]\ncollision_resistance = "always"\nsecond_preimage_resistance = "always"\n' > /etc/apt-nvidia-sqv-policy.toml
|
||||
|
||||
RUN export SEQUOIA_CRYPTO_POLICY=/etc/apt-nvidia-sqv-policy.toml && \
|
||||
apt-get update -y && \
|
||||
apt-get install -y --no-install-recommends \
|
||||
cuda-cudart-12-2 cuda-nvcc-12-2 cuda-nvrtc-12-2 \
|
||||
libcudnn8 libcublas-12-2 && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install FFmpeg if needed
|
||||
RUN apt-get update && \
|
||||
RUN export SEQUOIA_CRYPTO_POLICY=/etc/apt-nvidia-sqv-policy.toml && \
|
||||
apt-get update && \
|
||||
apt-get install -y ffmpeg && \
|
||||
rm -rf /var/lib/apt/lists/*
|
||||
|
||||
|
||||
@@ -1,12 +1,17 @@
|
||||
FROM ghcr.io/windmill-labs/windmill:dev
|
||||
|
||||
# Rust
|
||||
COPY --from=rust:1.93.0 /usr/local/cargo /usr/local/cargo
|
||||
COPY --from=rust:1.93.0 /usr/local/rustup /usr/local/rustup
|
||||
COPY --from=rust:1.97.0 /usr/local/cargo /usr/local/cargo
|
||||
COPY --from=rust:1.97.0 /usr/local/rustup /usr/local/rustup
|
||||
RUN RUSTUP_HOME=/usr/local/rustup CARGO_HOME=/usr/local/cargo /usr/local/cargo/bin/cargo install cargo-sweep --version ^0.7
|
||||
|
||||
# Ansible
|
||||
RUN uv tool install ansible && [ -d "$(uv tool dir)/ansible/bin/" ] && find "$(uv tool dir)/ansible/bin/" -mindepth 1 -maxdepth 1 -type f -executable -regextype posix-extended -regex '^((.+/)?)[^.]+' -print0 | xargs -0 ln -s -t "$UV_TOOL_BIN_DIR/" || true
|
||||
# UV_PYTHON_INSTALL_DIR defaults to /tmp/windmill/cache/py_runtime, which is an
|
||||
# ephemeral runtime cache (fresh volume/tmpfs, and pruned by the worker). Installing
|
||||
# ansible there leaves its venv interpreter as a dangling symlink at runtime, so every
|
||||
# ansible-* executable fails with ENOENT ("ansible-galaxy not found"). Pin the tool's
|
||||
# interpreter to a persistent image path so the install stays self-contained.
|
||||
RUN UV_PYTHON_INSTALL_DIR=/usr/local/uv/py uv tool install ansible && [ -d "$(uv tool dir)/ansible/bin/" ] && find "$(uv tool dir)/ansible/bin/" -mindepth 1 -maxdepth 1 -type f -executable -regextype posix-extended -regex '^((.+/)?)[^.]+' -print0 | xargs -0 ln -sf -t "$UV_TOOL_BIN_DIR/" || true
|
||||
|
||||
# C#
|
||||
RUN wget https://dot.net/v1/dotnet-install.sh -O dotnet-install.sh \
|
||||
|
||||
@@ -20,12 +20,17 @@ RUN if [ "$TARGETPLATFORM" = "linux/arm64" ]; then \
|
||||
FROM ghcr.io/windmill-labs/windmill-ee:dev
|
||||
|
||||
# Rust
|
||||
COPY --from=rust:1.93.0 /usr/local/cargo /usr/local/cargo
|
||||
COPY --from=rust:1.93.0 /usr/local/rustup /usr/local/rustup
|
||||
COPY --from=rust:1.97.0 /usr/local/cargo /usr/local/cargo
|
||||
COPY --from=rust:1.97.0 /usr/local/rustup /usr/local/rustup
|
||||
RUN RUSTUP_HOME=/usr/local/rustup CARGO_HOME=/usr/local/cargo /usr/local/cargo/bin/cargo install cargo-sweep --version ^0.7
|
||||
|
||||
# Ansible
|
||||
RUN uv tool install ansible && [ -d "$(uv tool dir)/ansible/bin/" ] && find "$(uv tool dir)/ansible/bin/" -mindepth 1 -maxdepth 1 -type f -executable -regextype posix-extended -regex '^((.+/)?)[^.]+' -print0 | xargs -0 ln -s -t "$UV_TOOL_BIN_DIR/" || true
|
||||
# UV_PYTHON_INSTALL_DIR defaults to /tmp/windmill/cache/py_runtime, which is an
|
||||
# ephemeral runtime cache (fresh volume/tmpfs, and pruned by the worker). Installing
|
||||
# ansible there leaves its venv interpreter as a dangling symlink at runtime, so every
|
||||
# ansible-* executable fails with ENOENT ("ansible-galaxy not found"). Pin the tool's
|
||||
# interpreter to a persistent image path so the install stays self-contained.
|
||||
RUN UV_PYTHON_INSTALL_DIR=/usr/local/uv/py uv tool install ansible && [ -d "$(uv tool dir)/ansible/bin/" ] && find "$(uv tool dir)/ansible/bin/" -mindepth 1 -maxdepth 1 -type f -executable -regextype posix-extended -regex '^((.+/)?)[^.]+' -print0 | xargs -0 ln -sf -t "$UV_TOOL_BIN_DIR/" || true
|
||||
# dotnet SDK
|
||||
RUN wget https://dot.net/v1/dotnet-install.sh -O dotnet-install.sh \
|
||||
&& chmod +x dotnet-install.sh \
|
||||
|
||||
@@ -92,7 +92,7 @@ COPY --from=nsjail /nsjail/nsjail /bin/nsjail
|
||||
|
||||
# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
|
||||
# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
|
||||
ARG CRANE_VERSION=v0.20.6
|
||||
ARG CRANE_VERSION=v0.21.7
|
||||
RUN arch="$(dpkg --print-architecture)"; \
|
||||
case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
|
||||
wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
|
||||
|
||||
@@ -92,7 +92,7 @@ COPY --from=nsjail /nsjail/nsjail /bin/nsjail
|
||||
|
||||
# crane: pulls + flattens images for the sandboxed container runtime (`# sandbox <image>`).
|
||||
# Single static binary — no daemon/store/root needed. See docs/docker-v2-runtime.md.
|
||||
ARG CRANE_VERSION=v0.20.6
|
||||
ARG CRANE_VERSION=v0.21.7
|
||||
RUN arch="$(dpkg --print-architecture)"; \
|
||||
case "$arch" in amd64) crane_arch=x86_64 ;; arm64) crane_arch=arm64 ;; *) echo >&2 "error: unsupported arch '$arch' for crane"; exit 1 ;; esac; \
|
||||
wget -O /tmp/crane.tgz "https://github.com/google/go-containerregistry/releases/download/${CRANE_VERSION}/go-containerregistry_Linux_${crane_arch}.tar.gz" \
|
||||
|
||||
Generated
+114
-53
@@ -1,12 +1,12 @@
|
||||
{
|
||||
"name": "@windmill-labs/components",
|
||||
"version": "1.751.0",
|
||||
"version": "1.756.0",
|
||||
"lockfileVersion": 3,
|
||||
"requires": true,
|
||||
"packages": {
|
||||
"": {
|
||||
"name": "@windmill-labs/components",
|
||||
"version": "1.751.0",
|
||||
"version": "1.756.0",
|
||||
"hasInstallScript": true,
|
||||
"license": "AGPL-3.0",
|
||||
"dependencies": {
|
||||
@@ -291,6 +291,7 @@
|
||||
"integrity": "sha512-9NhCeYjq9+3uxgdtp20LSiJXJvN0FeCtNGpJxuMFZ1Kv3cWUNb6DOhJwUvcVCzKGR66cw4njwM6hrJLqgOwbcw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/helper-validator-identifier": "^7.28.5",
|
||||
"js-tokens": "^4.0.0",
|
||||
@@ -306,6 +307,7 @@
|
||||
"integrity": "sha512-qSs4ifwzKJSV39ucNjsvc6WVHs6b7S03sOh2OcHF9UHfVPqWWALUsNUVzhSBiItjRZoLHx7nIarVjqKVusUZ1Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=6.9.0"
|
||||
}
|
||||
@@ -863,6 +865,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": "^14 || ^16 || >=18"
|
||||
},
|
||||
@@ -1389,7 +1392,6 @@
|
||||
"integrity": "sha512-Jer+M7DgIwT5IHfTayb4Iw/fkkxWNmC/mqn/nMh9JrbPbkxmyabfLQnhJ+JDn5HK77f84j34lubO3iqFtYAfMg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@floating-ui/core": "^1.3.1",
|
||||
"@floating-ui/dom": "^1.4.5",
|
||||
@@ -1546,7 +1548,6 @@
|
||||
"resolved": "https://registry.npmjs.org/@popperjs/core/-/core-2.11.8.tgz",
|
||||
"integrity": "sha512-P1st0aksCrn9sGZhp8GMYwBnQsbvAWsZAX44oXNNvLHGqAOcoVxmjZiohstwQ7SqKnbR47akdNi+uleWD8+g6A==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"funding": {
|
||||
"type": "opencollective",
|
||||
"url": "https://opencollective.com/popperjs"
|
||||
@@ -1911,7 +1912,6 @@
|
||||
"integrity": "sha512-iAIPEahFgDJJyvz8g0jP08KvqnM6JvdW8YfsygZ+pMeMvyM2zssWMltcsotETvjSZ82G3VlitgDtBIvpQSZrTA==",
|
||||
"devOptional": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@standard-schema/spec": "^1.0.0",
|
||||
"@sveltejs/acorn-typescript": "^1.0.5",
|
||||
@@ -2007,7 +2007,6 @@
|
||||
"integrity": "sha512-ILXmxC7HAsnkK2eslgPetrqqW1BKSL7LktsFgqzNj83MaivMGZzluWq32m25j2mDOjmSKX7GGWahePhuEs7P/g==",
|
||||
"devOptional": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"deepmerge": "^4.3.1",
|
||||
"magic-string": "^0.30.21",
|
||||
@@ -2469,7 +2468,8 @@
|
||||
"resolved": "https://registry.npmjs.org/@types/minimist/-/minimist-1.2.5.tgz",
|
||||
"integrity": "sha512-hov8bUuiLiyFPGyFPE1lwWhmzYbirOXQNNo40+y3zow8aFVTeyn3VWL0VFFfdNddA8S4Vf0Tc062rzyNr7Paag==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/@types/ms": {
|
||||
"version": "2.1.0",
|
||||
@@ -2482,7 +2482,8 @@
|
||||
"resolved": "https://registry.npmjs.org/@types/normalize-package-data/-/normalize-package-data-2.4.4.tgz",
|
||||
"integrity": "sha512-37i+OaWTh9qeK4LSHPsyRC7NahnGotNuZvjLSgcPzblpHB3rrCJxAOgI5gCdKm7coonsaX1Of0ILiTcnZjbfxA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/@types/semver": {
|
||||
"version": "7.7.1",
|
||||
@@ -2551,7 +2552,6 @@
|
||||
"integrity": "sha512-VlJEV0fOQ7BExOsHYAGrgbEiZoi8D+Bl2+f6V2RrXerRSylnp+ZBHmPvaIa8cz0Ajx7WO7Z5RqfgYg7ED1nRhA==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@typescript-eslint/scope-manager": "5.62.0",
|
||||
"@typescript-eslint/types": "5.62.0",
|
||||
@@ -3079,7 +3079,6 @@
|
||||
"resolved": "https://registry.npmjs.org/acorn/-/acorn-8.15.0.tgz",
|
||||
"integrity": "sha512-NZyJarBfL7nWwIq+FDL6Zp/yHEhePMNnnJ0y3qfieCrmNvYct8uvtiV41UvlSe6apAfk0fY1FbWx+NwfmpvtTg==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"acorn": "bin/acorn"
|
||||
},
|
||||
@@ -3133,7 +3132,6 @@
|
||||
"resolved": "https://registry.npmjs.org/ajv/-/ajv-8.17.1.tgz",
|
||||
"integrity": "sha512-B/gBuNg5SiMTrPkC+A2+cW0RszwxYmn6VYxB/inlBStS5nx6xHIt/ehKRhIMhqusl7a8LjQoZnjCs5vhwxOQ1g==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"fast-deep-equal": "^3.1.3",
|
||||
"fast-uri": "^3.0.1",
|
||||
@@ -3279,6 +3277,7 @@
|
||||
"integrity": "sha512-3CYzex9M9FGQjCGMGyi6/31c8GJbgb0qGyrx5HWxPd0aCwh4cB2YjMb2Xf9UuoogrMrlO9cTqnB5rI5GHZTcUA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -3306,6 +3305,7 @@
|
||||
"integrity": "sha512-Z7tMw1ytTXt5jqMcOP+OQteU1VuNK9Y02uuJtKQ1Sv69jXQKKg5cibLwGJow8yzZP+eAc18EmLGPal0bp36rvQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
@@ -3386,7 +3386,8 @@
|
||||
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-2.0.0.tgz",
|
||||
"integrity": "sha512-1ugUSr8BHXRnK23KfuYS+gVMC3LB8QGH9W1iGtDPsNWoQbgtXSExkBu2aDR4epiGWZOjZsj6lDl/N/AqqTC3UA==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/base64-js": {
|
||||
"version": "1.5.1",
|
||||
@@ -3513,7 +3514,6 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"baseline-browser-mapping": "^2.8.9",
|
||||
"caniuse-lite": "^1.0.30001746",
|
||||
@@ -3711,6 +3711,7 @@
|
||||
"integrity": "sha512-Rjs1H+A9R+Ig+4E/9oyB66UC5Mj9Xq3N//vcLf2WzgdTi/3gUu3Z9KoqmlrEG4VuuLK8wJHofxzdQXz/knhiYg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"camelcase": "^6.3.0",
|
||||
"map-obj": "^4.1.0",
|
||||
@@ -3730,6 +3731,7 @@
|
||||
"integrity": "sha512-Gmy6FhYlCY7uOElZUSbxo2UCDH8owEk996gkbrpsgGtrJLM3J7jGxl9Ic7Qwwj4ivOE5AWZWRMecDdF7hqGjFA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -3743,6 +3745,7 @@
|
||||
"integrity": "sha512-WuyALRjWPDGtt/wzJiadO5AXY+8hZ80hVpe6MyivgraREW751X3SbhRvG3eLKOYN+8VEvqLcf3wdnt44Z4S4SA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -3756,6 +3759,7 @@
|
||||
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
|
||||
"dev": true,
|
||||
"license": "(MIT OR CC0-1.0)",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -3864,7 +3868,6 @@
|
||||
"resolved": "https://registry.npmjs.org/chart.js/-/chart.js-4.5.1.tgz",
|
||||
"integrity": "sha512-GIjfiT9dbmHRiYi6Nl2yFCq7kkwdkp1W/lp2J99rX0yo9tgJGn3lKQATztIjb5tVtevcBtIdICNWqlq5+E8/Pw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@kurkle/color": "^0.3.0"
|
||||
},
|
||||
@@ -4141,6 +4144,7 @@
|
||||
"integrity": "sha512-kcZ6+W5QzcJ3P1Mt+83OUv/oHFqZHIx8DuxG6eZ5RGMERoLqp4BuGjhHLYGK+Kf5XVkQvqBSmAy/nGWN3qDgEA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"import-fresh": "^3.3.0",
|
||||
"js-yaml": "^4.1.0",
|
||||
@@ -4205,6 +4209,7 @@
|
||||
"integrity": "sha512-8HFEBPKhOpJPEPu70wJJetjKta86Gw9+CCyCnB3sui2qQfOvRyqBy4IKLKKAwdMpWb2lHXWk9Wb4Z6AmaUT1Pg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
@@ -4392,7 +4397,6 @@
|
||||
"resolved": "https://registry.npmjs.org/cytoscape/-/cytoscape-3.34.0.tgz",
|
||||
"integrity": "sha512-62rNSrioXw93uliKFBwjukeQyeWwH2PqDrTac31r2P6464u3AUvTk0xS4LVvT251g7IgkFunrI48ZEZGjywSOg==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10"
|
||||
}
|
||||
@@ -4815,7 +4819,6 @@
|
||||
"resolved": "https://registry.npmjs.org/d3-selection/-/d3-selection-3.0.0.tgz",
|
||||
"integrity": "sha512-fmTRWbNMmsmWq6xJV8D19U/gw/bwrHfNXxrIN+HfZgnzqTHp9jOmKMhsTUjXOJnZOdZY9Q28y4yebKzqDKlxlQ==",
|
||||
"license": "ISC",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
}
|
||||
@@ -4915,7 +4918,6 @@
|
||||
"resolved": "https://registry.npmjs.org/date-fns/-/date-fns-2.30.0.tgz",
|
||||
"integrity": "sha512-fnULvOpxnC5/Vg3NCiWelDsLiUc9bRwAPs/+LfTLNvetFCtCTN+yQz15C/fs4AwX1R9K5GLtLfn8QW+dWisaAw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/runtime": "^7.21.0"
|
||||
},
|
||||
@@ -4956,6 +4958,7 @@
|
||||
"integrity": "sha512-VfxadyCECXgQlkoEAjeghAr5gY3Hf+IKjKb+X8tGVDtveCjN+USwprd2q3QXBR9T1+x2DG0XZF5/w+7HAtSaXA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -4969,6 +4972,7 @@
|
||||
"integrity": "sha512-WiPxgEirIV0/eIOMcnFBA3/IJZAZqKnwAwWyvvdi4lsr1WCN22nhdf/3db3DoZcUjTV2SqfzIwNyp6y2xs3nmg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"decamelize": "^1.1.0",
|
||||
"map-obj": "^1.0.0"
|
||||
@@ -4986,6 +4990,7 @@
|
||||
"integrity": "sha512-z2S+W9X73hAUUki+N+9Za2lBlun89zigOyGrsax+KUQ6wKW4ZoWpEYBkGhQjwAjjDCkWxhY0VKEhk8wzY7F5cA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -4996,6 +5001,7 @@
|
||||
"integrity": "sha512-7N/q3lyZ+LVCp7PzuxrJr4KMbBE2hW7BT7YNia330OFxIf4d3r5zVpicP2650l7CPN6RM9zOJRl3NGpqSiw3Eg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -5489,6 +5495,7 @@
|
||||
"integrity": "sha512-sqQamAnR14VgCr1A618A3sGrygcpK+HEbenA/HiEAkkUwcZIIB/tgWqHFxWgOyDh4nB4JCRimh79dR5Ywc9MDQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"is-arrayish": "^0.2.1"
|
||||
}
|
||||
@@ -5586,7 +5593,6 @@
|
||||
"deprecated": "This version is no longer supported. Please see https://eslint.org/version-support for other options.",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@eslint-community/eslint-utils": "^4.2.0",
|
||||
"@eslint-community/regexpp": "^4.6.1",
|
||||
@@ -6143,6 +6149,7 @@
|
||||
"integrity": "sha512-eRnCtTTtGZFpQCwhJiUOuxPQWRXVKYDn0b2PeHfXL6/Zi53SLAzAHfVhVWK2AryC/WH05kGfxhFIPvTF0SXQzg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">= 4.9.1"
|
||||
}
|
||||
@@ -6502,6 +6509,7 @@
|
||||
"integrity": "sha512-NGbfmJBp9x8IxyJSd1P+otYK8vonoJactOogrVfFRIAEY1ukil8RSKDz2Yo7wh1oihl51l/r6W4epkeKJHqL8A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"global-prefix": "^3.0.0"
|
||||
},
|
||||
@@ -6515,6 +6523,7 @@
|
||||
"integrity": "sha512-awConJSVCHVGND6x3tmMaKcQvwXLhjdkmomy2W+Goaui8YPgYgXJZewhg3fWC+DlfqqQuWg8AwqjGTD2nAPVWg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"ini": "^1.3.5",
|
||||
"kind-of": "^6.0.2",
|
||||
@@ -6530,6 +6539,7 @@
|
||||
"integrity": "sha512-HxJdYWq1MTIQbJ3nw0cqssHoTNU267KlrDuGZ1WYlxDStUtKUhOaJmh112/TZmHxxUfuJqPXSOm7tDyas0OSIQ==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"isexe": "^2.0.0"
|
||||
},
|
||||
@@ -6579,7 +6589,8 @@
|
||||
"resolved": "https://registry.npmjs.org/globjoin/-/globjoin-0.1.4.tgz",
|
||||
"integrity": "sha512-xYfnw62CKG8nLkZBfWbhWwDw02CHty86jfPcc2cr3ZfeuK9ysoVPPEUxf21bAD/rWAgk52SuBrLJlefNy8mvFg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/gopd": {
|
||||
"version": "1.2.0",
|
||||
@@ -6662,6 +6673,7 @@
|
||||
"integrity": "sha512-VIZB+ibDhx7ObhAe7OVtoEbuP4h/MuOTHJ+J8h/eBXotJYl0fBgR72xDFCKgIh22OJZIOVNxBMWuhAr10r8HdA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=6"
|
||||
}
|
||||
@@ -6898,6 +6910,7 @@
|
||||
"integrity": "sha512-kyCuEOWjJqZuDbRHzL8V93NzQhwIB71oFWSyzVo+KPZI+pnQPPxucdkrOZvkLRnrf5URsQM+IJ09Dw29cRALIA==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"lru-cache": "^6.0.0"
|
||||
},
|
||||
@@ -6911,6 +6924,7 @@
|
||||
"integrity": "sha512-Jo6dJ04CmSjuznwJSS3pUeWmd/H0ffTlkXXgwZi+eq1UCmqQwCh+eLsYOYCwY991i2Fah4h1BEMCx4qThGbsiA==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"yallist": "^4.0.0"
|
||||
},
|
||||
@@ -6923,7 +6937,8 @@
|
||||
"resolved": "https://registry.npmjs.org/yallist/-/yallist-4.0.0.tgz",
|
||||
"integrity": "sha512-3wdGidZyq5PB084XLES5TpOSRA3wjXAlIWMhum2kRcv/41Sn2emQ0dycQW4uZXLejwKvg6EsvbdlVL+FYEct7A==",
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
"license": "ISC",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/html-tags": {
|
||||
"version": "3.3.1",
|
||||
@@ -6931,6 +6946,7 @@
|
||||
"integrity": "sha512-ztqyC3kLto0e9WbNp0aeP+M3kTt+nbaIveGmUxAtZa+8iFgKLUOD4YKM5j+f3QD89bra7UeumolZHKuOXnTmeQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
},
|
||||
@@ -7026,6 +7042,7 @@
|
||||
"integrity": "sha512-rKtvo6a868b5Hu3heneU+L4yEQ4jYKLtjpnPeUdK7h0yzXGmyBTypknlkCvHFBqfX9YlorEiMM6Dnq/5atfHkw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
@@ -7056,6 +7073,7 @@
|
||||
"integrity": "sha512-m6FAo/spmsW2Ab2fU35JTYwtOKa2yAwXSwgjSv1TJzh4Mh7mC3lzAOVLBprb72XsTrgkEIsl7YrFNAiDiRhIGg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
@@ -7125,7 +7143,8 @@
|
||||
"resolved": "https://registry.npmjs.org/is-arrayish/-/is-arrayish-0.2.1.tgz",
|
||||
"integrity": "sha512-zz06S8t0ozoDXMG+ube26zeCTNXcKIPJZJi8hBrF4idCLms4CG9QtK7qBl1boi5ODzFpjswb5JPmHCbMpjaYzg==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/is-binary-path": {
|
||||
"version": "2.1.0",
|
||||
@@ -7230,6 +7249,7 @@
|
||||
"integrity": "sha512-yvkRyxmFKEOQ4pNXCmJG5AEQNlXJS5LaONXo5/cLdTZdWvsZ1ioJEonLGAosKlMWE8lwUy/bJzMjcw8az73+Fg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -7240,6 +7260,7 @@
|
||||
"integrity": "sha512-VRSzKkbMm5jMDoKLbltAkFQ5Qr7VDiTFGXxYFXXowVj387GeGNOCsOH6Msy00SGZ3Fp84b1Naa1psqgcCIEP5Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -7344,7 +7365,8 @@
|
||||
"resolved": "https://registry.npmjs.org/js-tokens/-/js-tokens-4.0.0.tgz",
|
||||
"integrity": "sha512-RdJUflcE3cUzKiMqQgsCu06FPu9UdIJO0beYbPhHN4k6apgJtifcoCtT9bcxOpYBtpD2kCM6Sbzg4CausW/PKQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/js-yaml": {
|
||||
"version": "4.1.0",
|
||||
@@ -7380,7 +7402,8 @@
|
||||
"resolved": "https://registry.npmjs.org/json-parse-even-better-errors/-/json-parse-even-better-errors-2.3.1.tgz",
|
||||
"integrity": "sha512-xyFwyhro/JEof6Ghe2iz2NcXoj2sloNsWr/XsERDK/oiPCfaNhl5ONfp+jQdAZRQQ0IJWNzH9zIZF7li91kh2w==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/json-refs": {
|
||||
"version": "3.0.15",
|
||||
@@ -7577,6 +7600,7 @@
|
||||
"integrity": "sha512-dcS1ul+9tmeD95T+x28/ehLgd9mENa3LsvDTtzm3vyBEO7RPptvAD+t44WVXaUjTBRcrpFeFlC8WCruUR456hw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=0.10.0"
|
||||
}
|
||||
@@ -8182,7 +8206,8 @@
|
||||
"resolved": "https://registry.npmjs.org/lodash.truncate/-/lodash.truncate-4.4.2.tgz",
|
||||
"integrity": "sha512-jttmRe7bRse52OsWIMDLaXxWqRAmtIUccAQ3garviCqJjafXOfNMO0yMfNpdD6zbGaTU0P5Nz7e7gAT6cKmJRw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/lodash.uniq": {
|
||||
"version": "4.5.0",
|
||||
@@ -8250,6 +8275,7 @@
|
||||
"integrity": "sha512-hdN1wVrZbb29eBGiGjJbeP8JbKjq1urkHJ/LIP/NY48MZ1QVXUsQBV1G1zvYFHn1XE06cwjBsOI2K3Ulnj1YXQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
},
|
||||
@@ -8300,6 +8326,7 @@
|
||||
"integrity": "sha512-APMBEanjybaPzUrfqU0IMU5I0AswKMH7k8OTLs0vvV4KZpExkTkY87nR/zpbuTPj+gARop7aGUbl11pnDfW6xg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"funding": {
|
||||
"type": "github",
|
||||
"url": "https://github.com/sponsors/wooorm"
|
||||
@@ -8540,6 +8567,7 @@
|
||||
"integrity": "sha512-/d+PQ4GKmGvM9Bee/DPa8z3mXs/pkvJE2KEThngVNOqtmljC6K7NMPxtc2JeZYTmpWb9k/TmxjeL18ez3h7vCw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/minimist": "^1.2.2",
|
||||
"camelcase-keys": "^7.0.0",
|
||||
@@ -8567,6 +8595,7 @@
|
||||
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
|
||||
"dev": true,
|
||||
"license": "(MIT OR CC0-1.0)",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -9301,6 +9330,7 @@
|
||||
"integrity": "sha512-Q4r8ghd80yhO/0j1O3B2BjweX3fiHg9cdOwjJd2J76Q135c+NDxGCqdYKQ1SKBuFfgWbAUzBfvYjPUEeNgqN1A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"arrify": "^1.0.1",
|
||||
"is-plain-obj": "^1.1.0",
|
||||
@@ -9366,7 +9396,6 @@
|
||||
"resolved": "https://registry.npmjs.org/@codingame/monaco-vscode-editor-api/-/monaco-vscode-editor-api-25.0.0.tgz",
|
||||
"integrity": "sha512-uiY06RTWFo2WZdh6OybkLlDhuG+8LlkjUDpr9/wW55uucqHo4X8fx4XKEtD98cscC+6FKQkbG2yyUiOJ/npHOw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@codingame/monaco-vscode-api": "25.0.0"
|
||||
}
|
||||
@@ -9603,6 +9632,7 @@
|
||||
"integrity": "sha512-p2W1sgqij3zMMyRC067Dg16bfzVH+w7hyegmpIvZ4JNjqtGOVAIvLmjBx3yP7YTe9vKJgkoNOPjwQGogDoMXFA==",
|
||||
"dev": true,
|
||||
"license": "BSD-2-Clause",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"hosted-git-info": "^4.0.1",
|
||||
"is-core-module": "^2.5.0",
|
||||
@@ -9954,6 +9984,7 @@
|
||||
"integrity": "sha512-ayCKvm/phCGxOkYRSCM82iDwct8/EonSEgCSxWxD7ve6jHggsFl4fZVQBPRNgQoKiuV/odhFrGzQXZwbifC8Rg==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@babel/code-frame": "^7.0.0",
|
||||
"error-ex": "^1.3.1",
|
||||
@@ -10132,9 +10163,9 @@
|
||||
"license": "ISC"
|
||||
},
|
||||
"node_modules/picomatch": {
|
||||
"version": "2.3.1",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.1.tgz",
|
||||
"integrity": "sha512-JU3teHTNjmE2VCGFzuY8EXzCDVwEqB2a8fsIvwaStHhAWJEeVd1o1QD80CU6+ZdEXXSLbSsuLwJjkCBWqRQUVA==",
|
||||
"version": "2.3.2",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-2.3.2.tgz",
|
||||
"integrity": "sha512-V7+vQEJ06Z+c5tSye8S+nHUfI51xoXIXjHQ99cQtKUkQqqO1kO/KCJUfZXuB47h/YBlDhah2H3hdUGXn8ie0oA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -10275,7 +10306,6 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"nanoid": "^3.3.11",
|
||||
"picocolors": "^1.1.1",
|
||||
@@ -10464,7 +10494,6 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"lilconfig": "^3.0.0",
|
||||
"yaml": "^2.3.4"
|
||||
@@ -10854,7 +10883,8 @@
|
||||
"resolved": "https://registry.npmjs.org/postcss-resolve-nested-selector/-/postcss-resolve-nested-selector-0.1.6.tgz",
|
||||
"integrity": "sha512-0sglIs9Wmkzbr8lQwEyIzlDOOC9bGmfVKcJTaxv3vMmd3uo4o4DerC3En0bnmgceeql9BfC8hRkp7cg0fjdVqw==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/postcss-safe-parser": {
|
||||
"version": "6.0.0",
|
||||
@@ -11030,7 +11060,6 @@
|
||||
"integrity": "sha512-I7AIg5boAr5R0FFtJ6rCfD+LFsWHp81dolrFD8S79U9tb8Az2nGrJncnMSnys+bpQJfRUzqs9hnA81OAA3hCuQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"prettier": "bin/prettier.cjs"
|
||||
},
|
||||
@@ -11334,6 +11363,7 @@
|
||||
"integrity": "sha512-X1Fu3dPuk/8ZLsMhEj5f4wFAF0DWoK7qhGJvgaijocXxBmSToKfbFtqbxMO7bVjNA1dmE5huAzjXj/ey86iw9Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@types/normalize-package-data": "^2.4.0",
|
||||
"normalize-package-data": "^3.0.2",
|
||||
@@ -11353,6 +11383,7 @@
|
||||
"integrity": "sha512-snVCqPczksT0HS2EC+SxUndvSzn6LRCwpfSvLrIfR5BKDQQZMaI6jPRC9dYvYFDRAuFEAnkwww8kBBNE/3VvzQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"find-up": "^5.0.0",
|
||||
"read-pkg": "^6.0.0",
|
||||
@@ -11371,6 +11402,7 @@
|
||||
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
|
||||
"dev": true,
|
||||
"license": "(MIT OR CC0-1.0)",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -11384,6 +11416,7 @@
|
||||
"integrity": "sha512-yGSza74xk0UG8k+pLh5oeoYirvIiWo5t0/o3zHHAO2tRDiZcxWP7fywNlXhqb6/r6sWvwi+RsyQMWhVLe4BVuA==",
|
||||
"dev": true,
|
||||
"license": "(MIT OR CC0-1.0)",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
},
|
||||
@@ -11426,6 +11459,7 @@
|
||||
"integrity": "sha512-tYkDkVVtYkSVhuQ4zBgfvciymHaeuel+zFKXShfDnFP5SyVEP7qo70Rf1jTOTCx3vGNAbnEi/xFkcfQVMIBWag==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"indent-string": "^5.0.0",
|
||||
"strip-indent": "^4.0.0"
|
||||
@@ -12066,6 +12100,7 @@
|
||||
"integrity": "sha512-qMCMfhY040cVHT43K9BFygqYbUPFZKHOg7K73mtTWJRb8pyP3fzf4Ixd5SzdEJQ6MRUg/WBnOLxghZtKKurENQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"ansi-styles": "^4.0.0",
|
||||
"astral-regex": "^2.0.0",
|
||||
@@ -12142,6 +12177,7 @@
|
||||
"integrity": "sha512-kN9dJbvnySHULIluDHy32WHRUu3Og7B9sbY7tsFLctQkIqnMh3hErYgdMjTYuqmcXX+lK5T1lnUt3G7zNswmZA==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"spdx-expression-parse": "^3.0.0",
|
||||
"spdx-license-ids": "^3.0.0"
|
||||
@@ -12152,7 +12188,8 @@
|
||||
"resolved": "https://registry.npmjs.org/spdx-exceptions/-/spdx-exceptions-2.5.0.tgz",
|
||||
"integrity": "sha512-PiU42r+xO4UbUS1buo3LPJkjlO7430Xn5SVAhdpzzsPHsjbYVflnnFdATgabnLude+Cqu25p6N+g2lw/PFsa4w==",
|
||||
"dev": true,
|
||||
"license": "CC-BY-3.0"
|
||||
"license": "CC-BY-3.0",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/spdx-expression-parse": {
|
||||
"version": "3.0.1",
|
||||
@@ -12160,6 +12197,7 @@
|
||||
"integrity": "sha512-cbqHunsQWnJNE6KhVSMsMeH5H/L9EpymbzqTQ3uLwNCLZ1Q481oWaofqH7nO6V07xlXwY6PhQdQ2IedWx/ZK4Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"spdx-exceptions": "^2.1.0",
|
||||
"spdx-license-ids": "^3.0.0"
|
||||
@@ -12170,7 +12208,8 @@
|
||||
"resolved": "https://registry.npmjs.org/spdx-license-ids/-/spdx-license-ids-3.0.23.tgz",
|
||||
"integrity": "sha512-CWLcCCH7VLu13TgOH+r8p1O/Znwhqv/dbb6lqWy67G+pT1kHmeD/+V36AVb/vq8QMIQwVShJ6Ssl5FPh0fuSdw==",
|
||||
"dev": true,
|
||||
"license": "CC0-1.0"
|
||||
"license": "CC0-1.0",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/sprintf-js": {
|
||||
"version": "1.0.3",
|
||||
@@ -12264,6 +12303,7 @@
|
||||
"integrity": "sha512-SlyRoSkdh1dYP0PzclLE7r0M9sgbFKKMFXpFRUMNuKhQSbC6VQIGzq3E0qsfvGJaUFJPGv6Ws1NZ/haTAjfbMA==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
@@ -12289,7 +12329,8 @@
|
||||
"resolved": "https://registry.npmjs.org/style-search/-/style-search-0.1.0.tgz",
|
||||
"integrity": "sha512-Dj1Okke1C3uKKwQcetra4jSuk0DqbzbYtXipzFlFMZtowbF1x7BKJwB9AayVMyFARvU8EDrZdcax4At/452cAg==",
|
||||
"dev": true,
|
||||
"license": "ISC"
|
||||
"license": "ISC",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/style-to-object": {
|
||||
"version": "0.4.4",
|
||||
@@ -12338,6 +12379,7 @@
|
||||
"integrity": "sha512-78O4c6IswZ9TzpcIiQJIN49K3qNoXTM8zEJzhaTE/xRTCZswaovSEVIa/uwbOltZrk16X4jAxjaOhzz/hTm1Kw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@csstools/css-parser-algorithms": "^2.3.1",
|
||||
"@csstools/css-tokenizer": "^2.2.0",
|
||||
@@ -12420,6 +12462,7 @@
|
||||
}
|
||||
],
|
||||
"license": "MIT-0",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": "^14 || ^16 || >=18"
|
||||
},
|
||||
@@ -12433,6 +12476,7 @@
|
||||
"integrity": "sha512-TfW7/1iI4Cy7Y8L6iqNdZQVvdXn0f8B4QcIXmkIbtTIe/Okm/nSlHb4IwGzRVOd3WfSieCgvf5cMzEfySAIl0g==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"flat-cache": "^3.2.0"
|
||||
},
|
||||
@@ -12445,7 +12489,8 @@
|
||||
"resolved": "https://registry.npmjs.org/known-css-properties/-/known-css-properties-0.29.0.tgz",
|
||||
"integrity": "sha512-Ne7wqW7/9Cz54PDt4I3tcV+hAyat8ypyOGzYRJQfdxnnjeWsTxt1cy8pjvvKeI5kfXuyvULyeeAvwvvtAX3ayQ==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
"license": "MIT",
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/stylelint/node_modules/postcss-selector-parser": {
|
||||
"version": "6.1.2",
|
||||
@@ -12468,6 +12513,7 @@
|
||||
"integrity": "sha512-qYg9KP24dD5qka9J47d0aVky0N+b4fTU89LN9iDnjB5waksiC49rvMB0PrUJQGoTmH50XPiqOvAjDfaijGxYZw==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=8"
|
||||
}
|
||||
@@ -12608,6 +12654,7 @@
|
||||
"integrity": "sha512-zFObLMyZeEwzAoKCyu1B91U79K2t7ApXuQfo8OuxwXLDgcKxuwM+YvcbIhm6QWqz7mHUH1TVytR1PwVVjEuMig==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"has-flag": "^4.0.0",
|
||||
"supports-color": "^7.0.0"
|
||||
@@ -12637,7 +12684,6 @@
|
||||
"resolved": "https://registry.npmjs.org/svelte/-/svelte-5.53.5.tgz",
|
||||
"integrity": "sha512-YkqERnF05g8KLdDZwZrF8/i1eSbj6Eoat8Jjr2IfruZz9StLuBqo8sfCSzjosNKd+ZrQ8DkKZDjpO5y3ht1Pow==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@jridgewell/remapping": "^2.3.4",
|
||||
"@jridgewell/sourcemap-codec": "^1.5.0",
|
||||
@@ -12735,6 +12781,21 @@
|
||||
}
|
||||
}
|
||||
},
|
||||
"node_modules/svelte-check/node_modules/picomatch": {
|
||||
"version": "4.0.5",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz",
|
||||
"integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"optional": true,
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/jonschlinkert"
|
||||
}
|
||||
},
|
||||
"node_modules/svelte-eslint-parser": {
|
||||
"version": "0.43.0",
|
||||
"resolved": "https://registry.npmjs.org/svelte-eslint-parser/-/svelte-eslint-parser-0.43.0.tgz",
|
||||
@@ -12964,7 +13025,8 @@
|
||||
"version": "1.0.0",
|
||||
"resolved": "https://registry.npmjs.org/svg-tags/-/svg-tags-1.0.0.tgz",
|
||||
"integrity": "sha512-ovssysQTa+luh7A5Weu3Rta6FJlFBBbInjOh722LIt6klpU2/HtdUbszju/G4devcvk8PGt7FCLv5wftu3THUA==",
|
||||
"dev": true
|
||||
"dev": true,
|
||||
"peer": true
|
||||
},
|
||||
"node_modules/svgo": {
|
||||
"version": "3.3.2",
|
||||
@@ -13015,6 +13077,7 @@
|
||||
"integrity": "sha512-9kY+CygyYM6j02t5YFHbNz2FN5QmYGv9zAjVp4lCDjlCw7amdckXlEt/bjMhUIfj4ThGRE4gCUH5+yGnNuPo5A==",
|
||||
"dev": true,
|
||||
"license": "BSD-3-Clause",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"ajv": "^8.0.1",
|
||||
"lodash.truncate": "^4.4.2",
|
||||
@@ -13042,7 +13105,6 @@
|
||||
"integrity": "sha512-6A2rnmW5xZMdw11LYjhcI5846rt9pbLSabY5XPxo+XWdxwZaFEn47Go4NzFiHu9sNNmr/kXivP1vStfvMaK1GQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"@alloc/quick-lru": "^5.2.0",
|
||||
"arg": "^5.0.2",
|
||||
@@ -13290,12 +13352,11 @@
|
||||
}
|
||||
},
|
||||
"node_modules/tinyglobby/node_modules/picomatch": {
|
||||
"version": "4.0.4",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
|
||||
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
|
||||
"version": "4.0.5",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz",
|
||||
"integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==",
|
||||
"devOptional": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
@@ -13358,6 +13419,7 @@
|
||||
"integrity": "sha512-jRKj0n0jXWo6kh62nA5TEh3+4igKDXLvzBJcPpiizP7oOolUrYIxmVBG9TOtHYFHoddUk6YvAkGeGoSVTXfQXQ==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=12"
|
||||
},
|
||||
@@ -13475,7 +13537,6 @@
|
||||
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"bin": {
|
||||
"tsc": "bin/tsc",
|
||||
"tsserver": "bin/tsserver"
|
||||
@@ -13712,6 +13773,7 @@
|
||||
"integrity": "sha512-DpKm2Ui/xN7/HQKCtpZxoRWBhZ9Z0kqtygG8XCgNQ8ZlDnxuQmWhj566j8fN4Cu3/JmbhsDo7fcAJq4s9h27Ew==",
|
||||
"dev": true,
|
||||
"license": "Apache-2.0",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"spdx-correct": "^3.0.0",
|
||||
"spdx-expression-parse": "^3.0.0"
|
||||
@@ -13765,7 +13827,6 @@
|
||||
"integrity": "sha512-MFtjBYgzmSxmgA4RAfjIyXWpGe1oALnjgUTzzV7QLx/TKxCzjtMH6Fd9/eVK+5Fg1qNoz5VAwsmMs/NofrmJvw==",
|
||||
"devOptional": true,
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"lightningcss": "^1.32.0",
|
||||
"picomatch": "^4.0.4",
|
||||
@@ -13870,9 +13931,9 @@
|
||||
}
|
||||
},
|
||||
"node_modules/vite/node_modules/picomatch": {
|
||||
"version": "4.0.4",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.4.tgz",
|
||||
"integrity": "sha512-QP88BAKvMam/3NxH6vj2o21R6MjxZUAd6nlwAS/pnGvN9IVLocLHxGYIzFhg6fUQ+5th6P4dv4eW9jX3DSIj7A==",
|
||||
"version": "4.0.5",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz",
|
||||
"integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==",
|
||||
"devOptional": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -14006,9 +14067,9 @@
|
||||
"license": "MIT"
|
||||
},
|
||||
"node_modules/vitest/node_modules/picomatch": {
|
||||
"version": "4.0.3",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.3.tgz",
|
||||
"integrity": "sha512-5gTmgEY/sqK6gFXLIsQNH19lWb4ebPDLA4SdLP7dsWkIXHWlG66oPuVvXSGFPppYZz8ZDZq0dYYrbHfBCVUb1Q==",
|
||||
"version": "4.0.5",
|
||||
"resolved": "https://registry.npmjs.org/picomatch/-/picomatch-4.0.5.tgz",
|
||||
"integrity": "sha512-RvwwcruNjI1ncT5xRakeyS9Lf8lcItv34KD+aif+VH9kduAyfYBipGh12274xtenIPZ119/R9BdTBa8gAwSh0A==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"engines": {
|
||||
@@ -14473,6 +14534,7 @@
|
||||
"integrity": "sha512-+QU2zd6OTD8XWIJCbffaiQeH9U73qIqafo1x6V1snCWYGJf6cVE0cDR4D8xRzcEnfI21IFrUPzPGtcPf8AC+Rw==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"imurmurhash": "^0.1.4",
|
||||
"signal-exit": "^4.0.1"
|
||||
@@ -14668,6 +14730,7 @@
|
||||
"integrity": "sha512-y11nGElTIV+CT3Zv9t7VKl+Q3hTQoT9a1Qzezhhl6Rp21gJ/IVTW7Z3y9EWXhuUBC2Shnf+DX0antecpAwSP8w==",
|
||||
"dev": true,
|
||||
"license": "ISC",
|
||||
"peer": true,
|
||||
"engines": {
|
||||
"node": ">=10"
|
||||
}
|
||||
@@ -14687,7 +14750,6 @@
|
||||
"resolved": "https://registry.npmjs.org/yjs/-/yjs-13.6.27.tgz",
|
||||
"integrity": "sha512-OIDwaflOaq4wC6YlPBy2L6ceKeKuF7DeTxx+jPzv1FHn9tCZ0ZwSRnUBxD05E3yed46fv/FWJbvR+Ud7x0L7zw==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"dependencies": {
|
||||
"lib0": "^0.2.99"
|
||||
},
|
||||
@@ -14723,7 +14785,6 @@
|
||||
"resolved": "https://registry.npmjs.org/zod/-/zod-4.1.12.tgz",
|
||||
"integrity": "sha512-JInaHOamG8pt5+Ey8kGmdcAcg3OL9reK8ltczgHTAwNhMys/6ThXHityHxVV2p3fkw/c+MAvBHFVYHFZDmjMCQ==",
|
||||
"license": "MIT",
|
||||
"peer": true,
|
||||
"funding": {
|
||||
"url": "https://github.com/sponsors/colinhacks"
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user