diff --git a/.agents/skills/native-trigger/SKILL.md b/.agents/skills/native-trigger/SKILL.md index 026c1900bf..781e200d0c 100644 --- a/.agents/skills/native-trigger/SKILL.md +++ b/.agents/skills/native-trigger/SKILL.md @@ -1,3 +1,8 @@ +--- +name: native-trigger +description: Guidance for adding native trigger services to Windmill. Use when implementing or modifying native trigger integrations across the backend and frontend. +--- + # Skill: Adding Native Trigger Services This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications. diff --git a/.claude/review-prompt.md b/.claude/review-prompt.md new file mode 100644 index 0000000000..6814089bea --- /dev/null +++ b/.claude/review-prompt.md @@ -0,0 +1,25 @@ +# Code Review Instructions + +Review this pull request and provide comprehensive feedback. + +## Focus Areas + +- **Code quality and best practices** — does the code follow established patterns? +- **Potential bugs or issues** — will this code work correctly in all cases? +- **Performance considerations** — are there unnecessary allocations, N+1 queries, or bottlenecks? +- **Security implications** — injection, auth bypass, data exposure? + +## CLAUDE.md Compliance + +Read all relevant CLAUDE.md files (root and in directories containing changed files). Check each rule against the changed code. Quote the exact rule when flagging a violation. + +## Review Guidelines + +- Provide detailed feedback using inline comments for specific issues +- Use top-level comments for general observations or praise +- Only flag issues introduced by this PR, not pre-existing problems +- Self-validate each finding: "Is this definitely a real issue?" If uncertain, discard it + +## Testing Instructions + +At the end of your review, add complete instructions to reproduce the added changes through the app interface. These instructions will be given to a tester so they can verify the changes. It should be a short descriptive text (not a step-by-step or a list) on how to navigate the app (what page, what action, what input, etc.) to see the changes. diff --git a/.claude/skills/local-review/SKILL.md b/.claude/skills/local-review/SKILL.md index f14f5608db..0399ad7294 100644 --- a/.claude/skills/local-review/SKILL.md +++ b/.claude/skills/local-review/SKILL.md @@ -6,53 +6,24 @@ description: Code review a pull request for bugs and CLAUDE.md compliance. MUST # Local Code Review Skill -Review a pull request for real bugs and CLAUDE.md compliance violations. This review targets HIGH SIGNAL issues only. - -## Review Philosophy - -- **Only flag issues you are certain about.** If you are not sure an issue is real, do not flag it. False positives erode trust and waste reviewer time. -- Think like a senior engineer doing a final review — flag things that would cause incidents, not things that are merely imperfect. - -## What to Flag - -- Code that won't compile or parse (syntax errors, type errors, missing imports) -- Code that will definitely produce wrong results regardless of inputs -- Clear, unambiguous CLAUDE.md violations (quote the exact rule being violated) -- Security issues in introduced code (injection, auth bypass, data exposure) -- Incorrect logic that will fail in production - -## What NOT to Flag - -- Code style or quality concerns -- Potential issues that depend on specific inputs or runtime state -- Subjective suggestions or improvements -- Pre-existing issues not introduced by this PR -- Pedantic nitpicks a senior engineer wouldn't flag -- Issues a linter or type checker will catch -- General quality concerns unless explicitly prohibited in CLAUDE.md -- Issues silenced via lint ignore comments +Run the same review locally that the GitHub Claude Auto Review action runs on PRs. The shared review instructions live in `.claude/review-prompt.md` — read that file first and follow its instructions. ## Execution Steps -1. **Determine the PR scope**: +1. **Read `.claude/review-prompt.md`** for the review criteria and focus areas + +2. **Determine the PR scope**: - If an argument is provided, use it as the PR number or branch - Otherwise, detect from the current branch vs main - Run `gh pr view` if a PR exists, or use `git diff main...HEAD` -2. **Find relevant CLAUDE.md files**: - - Read the root `CLAUDE.md` - - Check for CLAUDE.md files in directories containing changed files - 3. **Get the diff and metadata**: - `gh pr diff` or `git diff main...HEAD` for the full diff - `gh pr view` or `git log main..HEAD --oneline` for context 4. **Read changed files** where the diff alone is insufficient to understand context -5. **Review for**: - - CLAUDE.md compliance — check each rule against the changed code - - Bugs and logic errors — will this code work correctly? - - Security issues — injection, auth, data exposure in new code +5. **Apply the review instructions from `.claude/review-prompt.md`** 6. **Self-validate each finding**: Before reporting, ask yourself: - "Is this definitely a real issue, not a false positive?" diff --git a/.claude/skills/native-trigger/SKILL.md b/.claude/skills/native-trigger/SKILL.md index 026c1900bf..781e200d0c 100644 --- a/.claude/skills/native-trigger/SKILL.md +++ b/.claude/skills/native-trigger/SKILL.md @@ -1,3 +1,8 @@ +--- +name: native-trigger +description: Guidance for adding native trigger services to Windmill. Use when implementing or modifying native trigger integrations across the backend and frontend. +--- + # Skill: Adding Native Trigger Services This skill provides comprehensive guidance for adding new native trigger services to Windmill. Native triggers allow external services (like Nextcloud, Google Drive, etc.) to trigger Windmill scripts/flows via webhooks or push notifications. diff --git a/.claude/skills/pr/SKILL.md b/.claude/skills/pr/SKILL.md index ab67b58748..2c7bd691ca 100644 --- a/.claude/skills/pr/SKILL.md +++ b/.claude/skills/pr/SKILL.md @@ -61,12 +61,13 @@ Generated with [Claude Code](https://claude.com/claude-code) 1. Run `git status` to check for uncommitted changes 2. Run `git log main..HEAD --oneline` to see all commits in this branch 3. Run `git diff main...HEAD` to see the full diff against main -4. Check if remote branch exists and is up to date: +4. **Run `/local-review`** before creating the PR. If issues are found, fix them and commit before proceeding. Do not skip this step. +5. Check if remote branch exists and is up to date: ```bash git rev-parse --abbrev-ref --symbolic-full-name @{u} 2>/dev/null || echo "no upstream" ``` -5. Push to remote if needed: `git push -u origin HEAD` -6. Create draft PR using gh CLI: +6. Push to remote if needed: `git push -u origin HEAD` +7. Create draft PR using gh CLI: ```bash gh pr create --draft --title ": " --body "$(cat <<'EOF' ## Summary @@ -85,7 +86,7 @@ Generated with [Claude Code](https://claude.com/claude-code) EOF )" ``` -7. Return the PR URL to the user +8. Return the PR URL to the user ## EE Companion PR (when `*_ee.rs` files were modified) diff --git a/.github/workflows/benchmark.yml b/.github/workflows/benchmark.yml index 195821b2dd..d420ff1f00 100644 --- a/.github/workflows/benchmark.yml +++ b/.github/workflows/benchmark.yml @@ -290,6 +290,49 @@ jobs: path: | *.json + benchmark_wac: + runs-on: ubicloud-standard-8 + services: + postgres: + image: postgres + env: + POSTGRES_DB: windmill + POSTGRES_PASSWORD: changeme + POSTGRES_INITDB_ARGS: "-c shared_buffers=2GB -c work_mem=32MB -c effective_cache_size=4GB" + options: >- + --health-cmd pg_isready --health-interval 10s --health-timeout 5s + --health-retries 5 + --shm-size=2g + windmill: + image: ghcr.io/windmill-labs/windmill-ee:main + env: + DATABASE_URL: postgres://postgres:changeme@postgres:5432/windmill + LICENSE_KEY: ${{ secrets.WM_LICENSE_KEY_CI }} + WORKER_GROUP: main + WORKER_TAGS: deno,bun,go,python3,bash,dependency,flow,nativets + options: >- + --pull always --health-interval 10s --health-timeout 5s + --health-retries 5 --health-cmd "curl + http://localhost:8000/api/version" + ports: + - 8000:8000 + steps: + - uses: denoland/setup-deno@v2 + with: + deno-version: v2.x + - name: benchmark + timeout-minutes: 30 + run: deno run -A -r + https://raw.githubusercontent.com/windmill-labs/windmill/${GITHUB_REF##ref/head/}/benchmarks/benchmark_suite.ts + -c + https://raw.githubusercontent.com/windmill-labs/windmill/${GITHUB_REF##ref/head/}/benchmarks/suite_wac.json + - name: Save benchmark results + uses: actions/upload-artifact@v4 + with: + name: benchmark_wac + path: | + *.json + benchmark_graphs: runs-on: ubicloud needs: @@ -297,6 +340,7 @@ jobs: - benchmark_dedicated - benchmark_4workers - benchmark_8workers + - benchmark_wac steps: - uses: denoland/setup-deno@v2 with: diff --git a/.github/workflows/cli-tests.yml b/.github/workflows/cli-tests.yml index 9c87a249a3..237a5ff555 100644 --- a/.github/workflows/cli-tests.yml +++ b/.github/workflows/cli-tests.yml @@ -1,6 +1,7 @@ name: CLI Tests on: + workflow_dispatch: push: branches: [main] paths: diff --git a/.github/workflows/pr-ready-review.yml b/.github/workflows/pr-ready-review.yml index 25553bcb2c..78c0c3e045 100644 --- a/.github/workflows/pr-ready-review.yml +++ b/.github/workflows/pr-ready-review.yml @@ -22,6 +22,15 @@ jobs: with: fetch-depth: 1 + - name: Read review prompt + id: review-prompt + run: | + { + echo 'REVIEW_PROMPT<> "$GITHUB_ENV" + - name: Automatic PR Review uses: anthropics/claude-code-action@v1 with: @@ -31,18 +40,7 @@ jobs: REPO: ${{ github.repository }} PR NUMBER: ${{ github.event.pull_request.number }} - Please review this pull request and provide comprehensive feedback. - - Focus on: - - Code quality and best practices - - Potential bugs or issues - - Performance considerations - - Security implications - - Provide detailed feedback using inline comments for specific issues. - Use top-level comments for general observations or praise. - - At the end of your review, add complete instructions to reproduce the added changes through the app interface. These instructions will be given to a tester so he can verify the changes. It should be a short descriptive text (not a step by step or a list) on how to navigate the app (what page, what action, what input, etc) to see the changes. + ${{ env.REVIEW_PROMPT }} claude_args: | --allowedTools "mcp__github_inline_comment__create_inline_comment,Bash(gh pr comment:*),Bash(gh pr diff:*),Bash(gh pr view:*)" --model opus diff --git a/.github/workflows/rust-client-check.yml b/.github/workflows/rust-client-check.yml index 4ee5575071..32f06f5b85 100644 --- a/.github/workflows/rust-client-check.yml +++ b/.github/workflows/rust-client-check.yml @@ -18,10 +18,7 @@ jobs: runs-on: ubicloud-standard-8 steps: - uses: actions/checkout@v4 - - uses: cachix/install-nix-action@v20 - with: - extra_nix_config: | - experimental-features = nix-command flakes + - uses: cachix/install-nix-action@v31 - name: Check rust client builds run: cd rust-client && nix develop ../ --command ./dev.nu --check timeout-minutes: 16 \ No newline at end of file diff --git a/.github/workflows/rust_on_release.yml b/.github/workflows/rust_on_release.yml index 5594cc08bb..e23549e5e7 100644 --- a/.github/workflows/rust_on_release.yml +++ b/.github/workflows/rust_on_release.yml @@ -10,10 +10,7 @@ jobs: runs-on: ubicloud-standard-8 steps: - uses: actions/checkout@v4 - - uses: cachix/install-nix-action@v20 - with: - extra_nix_config: | - experimental-features = nix-command flakes + - uses: cachix/install-nix-action@v31 - run: cd rust-client && nix develop ../ --command ./dev.nu --check --publish env: CRATES_IO_TOKEN: ${{ secrets.CRATES_IO_TOKEN }} diff --git a/.webmux.yaml b/.webmux.yaml index 6a765fbb12..c41d0aa699 100644 --- a/.webmux.yaml +++ b/.webmux.yaml @@ -55,11 +55,13 @@ profiles: - id: backend kind: command split: right - command: ROOT="$(git rev-parse --show-toplevel)"; cd "$ROOT/backend" && cargo watch -x "run ${CARGO_FEATURES:+--features $CARGO_FEATURES}" + workingDir: backend + command: PORT=${BACKEND_PORT:-8000} cargo watch -x "run ${CARGO_FEATURES:+--features $CARGO_FEATURES}" - id: frontend kind: command split: bottom - command: ROOT="$(git rev-parse --show-toplevel)"; cd "$ROOT/frontend" && npm run generate-backend-client && npm run dev -- --host 0.0.0.0 + workingDir: frontend + command: npm run generate-backend-client && REMOTE=${REMOTE:-http://localhost:${BACKEND_PORT:-8000}} npm run dev -- --port ${FRONTEND_PORT:-3000} --host 0.0.0.0 frontendOnly: runtime: host @@ -82,7 +84,8 @@ profiles: - id: frontend kind: command split: right - command: ROOT="$(git rev-parse --show-toplevel)"; cd "$ROOT/frontend" && npm run generate-backend-client && npm run dev -- --host 0.0.0.0 + workingDir: frontend + command: npm run generate-backend-client && npm run dev -- --port ${FRONTEND_PORT:-3000} --host 0.0.0.0 agentOnly: runtime: host diff --git a/CHANGELOG.md b/CHANGELOG.md index 07701ce7ce..2e8710b1be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,221 @@ # Changelog +## [1.670.0](https://github.com/windmill-labs/windmill/compare/v1.669.1...v1.670.0) (2026-03-31) + + +### Features + +* add OR logic support to kafka/websocket trigger filters ([#8580](https://github.com/windmill-labs/windmill/issues/8580)) ([3876902](https://github.com/windmill-labs/windmill/commit/3876902a7be798fd5ef208bc5756b28fb55e569e)) +* expose getJob and getJobLogs as MCP tools ([#8632](https://github.com/windmill-labs/windmill/issues/8632)) ([cd8edcd](https://github.com/windmill-labs/windmill/commit/cd8edcd94f2bf44c3e771000cb0bbad08accc0e7)) +* support multiline secrets in resource password fields ([#8637](https://github.com/windmill-labs/windmill/issues/8637)) ([26050f9](https://github.com/windmill-labs/windmill/commit/26050f96c34f14826298760174a45f3559d3266c)) +* support sensitive/secret fields for non-string types ([#8635](https://github.com/windmill-labs/windmill/issues/8635)) ([375fb66](https://github.com/windmill-labs/windmill/commit/375fb66abe2d1861b53dc2b36d2cf0e2eb82c3a8)) + + +### Bug Fixes + +* cap input history per_page to 100 on cloud ([#8624](https://github.com/windmill-labs/windmill/issues/8624)) ([8e973c8](https://github.com/windmill-labs/windmill/commit/8e973c892d768be2da2e6b4b7af9e40b62333052)) +* compute highest workspace role across all instance groups ([#8633](https://github.com/windmill-labs/windmill/issues/8633)) ([92b9ac7](https://github.com/windmill-labs/windmill/commit/92b9ac72c5fc9a5085fcb2e9d835ccbb53bcd4b0)) +* Ducklake UI Nits ([#8628](https://github.com/windmill-labs/windmill/issues/8628)) ([ef1757f](https://github.com/windmill-labs/windmill/commit/ef1757f5d747e513d201eb6fa48918dba8248abe)) +* preserve flow notes/groups and field ordering in generate-metadata ([#8641](https://github.com/windmill-labs/windmill/issues/8641)) ([#8642](https://github.com/windmill-labs/windmill/issues/8642)) ([52a04d2](https://github.com/windmill-labs/windmill/commit/52a04d210f476f4598007f67770bc6520b045950)) +* remove timeout on python client httpx to prevent ducklake query timeouts ([#8636](https://github.com/windmill-labs/windmill/issues/8636)) ([c5fccd2](https://github.com/windmill-labs/windmill/commit/c5fccd2f69ad8a6e46c514cf89b9aa21b380e6fe)) +* resolve missing form schema for nested suspend steps in FlowNode sub-flows ([#8643](https://github.com/windmill-labs/windmill/issues/8643)) ([12ea7e7](https://github.com/windmill-labs/windmill/commit/12ea7e74237560a9dfc99b6bc1338e3343b57640)) +* smarter secret masking based on secret length ([#8629](https://github.com/windmill-labs/windmill/issues/8629)) ([bfc2aef](https://github.com/windmill-labs/windmill/commit/bfc2aefdb8ab92b7284de7f9e485a5504502d944)) + +## [1.669.1](https://github.com/windmill-labs/windmill/compare/v1.669.0...v1.669.1) (2026-03-30) + + +### Bug Fixes + +* avoid doubled /oauth2 path in Okta custom authorization server URLs ([#8620](https://github.com/windmill-labs/windmill/issues/8620)) ([4817913](https://github.com/windmill-labs/windmill/commit/4817913f0cab49980bfeb442089631d7953955ff)) +* improve db health UI text and prevent label wrapping ([d532c1d](https://github.com/windmill-labs/windmill/commit/d532c1d470fcb0ef02ebc5342ad1cf22e58b1f4d)) + +## [1.669.0](https://github.com/windmill-labs/windmill/compare/v1.668.5...v1.669.0) (2026-03-30) + + +### Features + +* WAC workflow diagram visualization via WASM ([#8604](https://github.com/windmill-labs/windmill/issues/8604)) ([abc6b12](https://github.com/windmill-labs/windmill/commit/abc6b12d6815edc4dda3ddf5f0572ecedcb670dd)) + + +### Bug Fixes + +* add path traversal check in service_logs get_log_file endpoint ([#8605](https://github.com/windmill-labs/windmill/issues/8605)) ([5f2d3e6](https://github.com/windmill-labs/windmill/commit/5f2d3e6812f01fe6194bcfd976970a6e3c4186cc)) +* cast DuckDB IS_NULLABLE to string in metadata query ([#8607](https://github.com/windmill-labs/windmill/issues/8607)) ([f3012ee](https://github.com/windmill-labs/windmill/commit/f3012ee7ccc7a8947b5f6bd7c7df77984437f91e)) +* enable S3 bundle cache for PHP previews without lock file ([#8608](https://github.com/windmill-labs/windmill/issues/8608)) ([ee62315](https://github.com/windmill-labs/windmill/commit/ee6231590ed91063f104e6d054b52e88b569986f)) +* enforce workspace isolation on flow resume endpoint ([#8612](https://github.com/windmill-labs/windmill/issues/8612)) ([33032ed](https://github.com/windmill-labs/windmill/commit/33032ed297cf9ea867388d4ea2ece607c9d36dc7)) +* handle DuckDB boolean types in ColumnDef deserializers ([#8610](https://github.com/windmill-labs/windmill/issues/8610)) ([22da5bd](https://github.com/windmill-labs/windmill/commit/22da5bd9ea1ca000cfab3eecf1e3fb0fc01200cb)) +* use route_service instead of fallback_service for MCP router ([#8614](https://github.com/windmill-labs/windmill/issues/8614)) ([98934d5](https://github.com/windmill-labs/windmill/commit/98934d59c552325fcf88c016e31ae977970e8c9a)) + +## [1.668.5](https://github.com/windmill-labs/windmill/compare/v1.668.4...v1.668.5) (2026-03-29) + + +### Bug Fixes + +* add per-IP and per-account brute force protection on login endpoint ([#8601](https://github.com/windmill-labs/windmill/issues/8601)) ([06bbe7b](https://github.com/windmill-labs/windmill/commit/06bbe7b94bfb846bd73aaf6abdc83e4c14e70adc)) +* add timestamp validation to webhook signature verification ([#8596](https://github.com/windmill-labs/windmill/issues/8596)) ([74fba2a](https://github.com/windmill-labs/windmill/commit/74fba2abf3dc68b682777c01da360258786fded8)) +* disable workspace webhook events when CLOUD_HOSTED ([#8598](https://github.com/windmill-labs/windmill/issues/8598)) ([be7fbeb](https://github.com/windmill-labs/windmill/commit/be7fbeb8b1f31d15e33b0783b2a504d6a01e532e)) +* harden login rate limiting with CLOUD_HOSTED gating and memory eviction ([#8602](https://github.com/windmill-labs/windmill/issues/8602)) ([754b88a](https://github.com/windmill-labs/windmill/commit/754b88a52c4e76421cb21c1eed87ad9d8385e9aa)) +* prevent SSRF and local file read via git repository resource URLs ([#8600](https://github.com/windmill-labs/windmill/issues/8600)) ([845db72](https://github.com/windmill-labs/windmill/commit/845db72b7344fb87ac9c5e24697750549665c7bf)) +* rename snippet param to avoid svelte compiler shadowing bug in asset usages drawer ([#8595](https://github.com/windmill-labs/windmill/issues/8595)) ([8c770a2](https://github.com/windmill-labs/windmill/commit/8c770a206a3b0704642c0bda2ab2aeb199d8af3f)) +* require mcp: scope for MCP endpoints instead of blanket bypass ([#8597](https://github.com/windmill-labs/windmill/issues/8597)) ([f5fc9f8](https://github.com/windmill-labs/windmill/commit/f5fc9f8485d2ec3e20f8b451305195446b90e5a3)) +* use constant-time comparison for API key and basic auth validation ([#8593](https://github.com/windmill-labs/windmill/issues/8593)) ([b4d1f2a](https://github.com/windmill-labs/windmill/commit/b4d1f2aac789306c2e35e123ac93e12c47c26f99)) +* validate JSON before sql_builder bind to prevent injection via JSONB queries ([#8599](https://github.com/windmill-labs/windmill/issues/8599)) ([970e859](https://github.com/windmill-labs/windmill/commit/970e859a410b0144847a1a30d7059955effdd402)) + +## [1.668.4](https://github.com/windmill-labs/windmill/compare/v1.668.3...v1.668.4) (2026-03-29) + + +### Bug Fixes + +* update git sync version to latest cli ([0549f68](https://github.com/windmill-labs/windmill/commit/0549f682fe14f4d4b2f67941362ed2cc29d974a1)) + +## [1.668.3](https://github.com/windmill-labs/windmill/compare/v1.668.2...v1.668.3) (2026-03-28) + + +### Bug Fixes + +* **cli:** phantom diffs, flow safety, trigger DX, lint watch, error clarity ([#8588](https://github.com/windmill-labs/windmill/issues/8588)) ([c6ce319](https://github.com/windmill-labs/windmill/commit/c6ce3197a72ceeffd702cf2263b1074ecbf1ca33)) + +## [1.668.2](https://github.com/windmill-labs/windmill/compare/v1.668.1...v1.668.2) (2026-03-28) + + +### Bug Fixes + +* **cli:** app push crash, lint path, push --message, run validation, history timestamps ([#8585](https://github.com/windmill-labs/windmill/issues/8585)) ([f40cdaf](https://github.com/windmill-labs/windmill/commit/f40cdaf43453d2643800ed730d6abe6873bbe8e7)) + +## [1.668.1](https://github.com/windmill-labs/windmill/compare/v1.668.0...v1.668.1) (2026-03-28) + + +### Bug Fixes + +* **cli:** fix 13 CLI bugs — exit codes, sync tar fallback, variable encryption, JSON output ([#8582](https://github.com/windmill-labs/windmill/issues/8582)) ([38acaa3](https://github.com/windmill-labs/windmill/commit/38acaa3653728bf9e0ae6f746edf433703b4ab63)) + +## [1.668.0](https://github.com/windmill-labs/windmill/compare/v1.667.0...v1.668.0) (2026-03-28) + + +### Features + +* add DB health diagnostic dashboard for superadmins ([#8574](https://github.com/windmill-labs/windmill/issues/8574)) ([9ceab73](https://github.com/windmill-labs/windmill/commit/9ceab730d7def09c2b46527f8a586789d14f2ce0)) +* **cli:** add job, group, audit, token commands and schedule enable/disable ([#8581](https://github.com/windmill-labs/windmill/issues/8581)) ([d29cb23](https://github.com/windmill-labs/windmill/commit/d29cb234dbff07473b911e5e75e362def8a47650)) +* IAM RDS auth for PostgreSQL worker resources ([#8573](https://github.com/windmill-labs/windmill/issues/8573)) ([56253c0](https://github.com/windmill-labs/windmill/commit/56253c04cb679c58d00750da699a6cb62ed52aca)) + + +### Bug Fixes + +* add Authority Key Identifier to MITM proxy leaf certs ([#8576](https://github.com/windmill-labs/windmill/issues/8576)) ([ce2e6c8](https://github.com/windmill-labs/windmill/commit/ce2e6c8c015110d0385e6afecdc8313aabca1364)) +* Improve CLI developer experience: error handling, sync workflow, JSON output, workspace forks ([#8578](https://github.com/windmill-labs/windmill/issues/8578)) ([501a4ff](https://github.com/windmill-labs/windmill/commit/501a4ff2a94510145952686d24ccc639781beefe)) +* trigger capture filter and focus issues ([#8579](https://github.com/windmill-labs/windmill/issues/8579)) ([820f28f](https://github.com/windmill-labs/windmill/commit/820f28f8799f8dad5cfab94b51ac9921d664f04a)) + +## [1.667.0](https://github.com/windmill-labs/windmill/compare/v1.666.0...v1.667.0) (2026-03-27) + + +### Features + +* add schedule support to CLI branch-specific items ([#8570](https://github.com/windmill-labs/windmill/issues/8570)) ([b592996](https://github.com/windmill-labs/windmill/commit/b592996eee98ddb664f1b007b95a2096d5d4e3a6)) +* add workspace-level service accounts ([#8560](https://github.com/windmill-labs/windmill/issues/8560)) ([3959fe8](https://github.com/windmill-labs/windmill/commit/3959fe82974f5f0383e94fd83a5d78fe4212d56a)) +* **cli:** generate commented wmill.yaml and add config reference command ([#8546](https://github.com/windmill-labs/windmill/issues/8546)) ([d06b426](https://github.com/windmill-labs/windmill/commit/d06b42613f73c4a7b31c990be22b0c97efab2666)) +* DB-coordinated graceful restart staggering for settings changes ([#8555](https://github.com/windmill-labs/windmill/issues/8555)) ([2f32675](https://github.com/windmill-labs/windmill/commit/2f326758013dd1f1e6ae732e5784a32f1fb6e4bd)) +* improve-replay-ui ([#8250](https://github.com/windmill-labs/windmill/issues/8250)) ([c0aafee](https://github.com/windmill-labs/windmill/commit/c0aafee9a9923d5dc2fa3b99da4378e923933a06)) +* support multiple folder selection in MCP scope selector ([#8557](https://github.com/windmill-labs/windmill/issues/8557)) ([ad19ac9](https://github.com/windmill-labs/windmill/commit/ad19ac9b37b04591c921f93f180bdda961af6cef)) + + +### Bug Fixes + +* **cli:** preserve inline script files during flow generate-locks ([#8561](https://github.com/windmill-labs/windmill/issues/8561)) ([a8b651d](https://github.com/windmill-labs/windmill/commit/a8b651da9ff86766119e14c0b61652be8a7b453a)) +* emit 0 for OTEL queue metrics when tag queue is empty ([#8559](https://github.com/windmill-labs/windmill/issues/8559)) ([79cc4a9](https://github.com/windmill-labs/windmill/commit/79cc4a92d88486c999799826bd0c9663767103f5)) +* handle inline script deletion in sync push + flow new nonDottedPaths ([#8553](https://github.com/windmill-labs/windmill/issues/8553)) ([943fe9c](https://github.com/windmill-labs/windmill/commit/943fe9c6cc9b046e24007e45b5c37afc4804256a)) +* include importer_kind in dependency debounce key to prevent cross-kind collisions ([#8567](https://github.com/windmill-labs/windmill/issues/8567)) ([bc7007b](https://github.com/windmill-labs/windmill/commit/bc7007bb4265e1f1375c1f0678b74325882a4e92)) +* multi-script dedicated workers race on shared job_dir ([#8551](https://github.com/windmill-labs/windmill/issues/8551)) ([#8569](https://github.com/windmill-labs/windmill/issues/8569)) ([63a3573](https://github.com/windmill-labs/windmill/commit/63a3573951d1f724cc63728ed973d039a5468072)) +* preserve notes on nodes inside collapsed groups ([#8552](https://github.com/windmill-labs/windmill/issues/8552)) ([0fb1153](https://github.com/windmill-labs/windmill/commit/0fb115304afc49812420e9ce24e5048502621059)) +* sanitize flow step summaries for filesystem-safe names ([#8554](https://github.com/windmill-labs/windmill/issues/8554)) ([e15bfbf](https://github.com/windmill-labs/windmill/commit/e15bfbf91ee1517432a6861ebb48e129485006aa)) +* use admin db pool in get_copilot_settings_state ([#8564](https://github.com/windmill-labs/windmill/issues/8564)) ([70f3ee5](https://github.com/windmill-labs/windmill/commit/70f3ee5ed4470e9993be822874f2b38e83a96611)) + + +### Performance Improvements + +* enable bun bundle caching for WAC v2 scripts ([#8556](https://github.com/windmill-labs/windmill/issues/8556)) ([ab868e9](https://github.com/windmill-labs/windmill/commit/ab868e9ebceadaa55e54770d9d59dc5524da13ff)) + +## [1.666.0](https://github.com/windmill-labs/windmill/compare/v1.665.0...v1.666.0) (2026-03-26) + + +### Features + +* add PDF input support to AI agent ([#8525](https://github.com/windmill-labs/windmill/issues/8525)) ([e44504c](https://github.com/windmill-labs/windmill/commit/e44504c6e93e7a4ee94ced03ab626b79a4fd0754)) + + +### Bug Fixes + +* add relative imports to the dependency list in deploymentUI ([#8548](https://github.com/windmill-labs/windmill/issues/8548)) ([d760ea5](https://github.com/windmill-labs/windmill/commit/d760ea5eaf4dc33007f1fd3e5e07b86925a0aa11)) +* filter null entries in FileUpload initialValue to prevent s3 access error ([#8544](https://github.com/windmill-labs/windmill/issues/8544)) ([1a73012](https://github.com/windmill-labs/windmill/commit/1a73012e0737a6ebea8307013dc0f79982269d91)) +* pass pre-bound TcpListener to run_server to fix Windows CI test race ([#8542](https://github.com/windmill-labs/windmill/issues/8542)) ([d7f4b95](https://github.com/windmill-labs/windmill/commit/d7f4b950ce6e966ed1b410e03d48fe96bc036e73)) +* resolve parent_hash race condition in sync push with auto_parent ([#8545](https://github.com/windmill-labs/windmill/issues/8545)) ([71549c3](https://github.com/windmill-labs/windmill/commit/71549c3db053bcc209c7065ac8cd42f1e8047cc3)) +* upload_s3_file not working in VS Code extension ([#8547](https://github.com/windmill-labs/windmill/issues/8547)) ([1fa4d91](https://github.com/windmill-labs/windmill/commit/1fa4d919b30ac9eff2d1789fba2695450ba115e7)) + +## [1.665.0](https://github.com/windmill-labs/windmill/compare/v1.664.0...v1.665.0) (2026-03-26) + + +### Features + +* add instance setting to enforce workspace prefix for HTTP routes ([#8528](https://github.com/windmill-labs/windmill/issues/8528)) ([9b3e558](https://github.com/windmill-labs/windmill/commit/9b3e558d84f15052e9c32695a467f8ef7e4ad1f5)) +* add trashbin system for soft-deleting items ([#8519](https://github.com/windmill-labs/windmill/issues/8519)) ([69ce946](https://github.com/windmill-labs/windmill/commit/69ce946241d98ea90bc7135d44ca0c87f928be88)) +* mask sensitive values in job logs ([#8520](https://github.com/windmill-labs/windmill/issues/8520)) ([0885d8c](https://github.com/windmill-labs/windmill/commit/0885d8c986f13ac210e4db3ad38febe9be391ba4)) +* move basic git sync from EE to CE with runtime user count gating ([#8493](https://github.com/windmill-labs/windmill/issues/8493)) ([79d2bd5](https://github.com/windmill-labs/windmill/commit/79d2bd51a00654162754046308d7670242120df6)) +* runner groups for shared-process multi-script dedicated workers ([#8434](https://github.com/windmill-labs/windmill/issues/8434)) ([c28314f](https://github.com/windmill-labs/windmill/commit/c28314f424ea0e04b86565ce88e6c91e0df1a0cf)) +* SCIM user deprovisioning (active:false) + instance-level user disable ([#8484](https://github.com/windmill-labs/windmill/issues/8484)) ([0bd7568](https://github.com/windmill-labs/windmill/commit/0bd756839c0261f255111d62088bdaaecb838085)) +* show groups and notes in flow status viewer ([#8535](https://github.com/windmill-labs/windmill/issues/8535)) ([167084a](https://github.com/windmill-labs/windmill/commit/167084a0ebe73384fa0d31f0b24017a47686a072)) + + +### Bug Fixes + +* auto-generate datatable SDK reference for app mode system prompt ([#8522](https://github.com/windmill-labs/windmill/issues/8522)) ([8a32322](https://github.com/windmill-labs/windmill/commit/8a32322c187ccc60ec7eafb61a9678f267a82282)) +* consider wmill.yaml environments alias in git sync ([#8532](https://github.com/windmill-labs/windmill/issues/8532)) ([b7475c7](https://github.com/windmill-labs/windmill/commit/b7475c73094a28f520f798f6cb1a0c6b4807ccb7)) +* GitHub Enterprise Server support for self-managed GitHub Apps ([#8507](https://github.com/windmill-labs/windmill/issues/8507)) ([935fb44](https://github.com/windmill-labs/windmill/commit/935fb44c848b8bf9430b5600dd3c3bedb2f89efd)) +* raw apps bundle not found during deployment error ([#8515](https://github.com/windmill-labs/windmill/issues/8515)) ([34e3115](https://github.com/windmill-labs/windmill/commit/34e3115bcbd19a8e0b6f483435586a2ab43d0a8e)) +* require admin for workspace encryption key export ([#8523](https://github.com/windmill-labs/windmill/issues/8523)) ([0317668](https://github.com/windmill-labs/windmill/commit/031766808945aefc926f0836d011c0b2a5d2243d)) +* restrict logout redirect to whitelisted domains ([#8524](https://github.com/windmill-labs/windmill/issues/8524)) ([4c8edd5](https://github.com/windmill-labs/windmill/commit/4c8edd5e944d77ed2d41c2b87171c1115c0fdcdc)) +* serve index disk storage sizes from /srch/ endpoint ([#8511](https://github.com/windmill-labs/windmill/issues/8511)) ([e3620e0](https://github.com/windmill-labs/windmill/commit/e3620e074e1bdb46b2b8d732f35a91d300589663)) +* use /apps_raw/get/ redirect URL for raw apps set as workspace default ([#8508](https://github.com/windmill-labs/windmill/issues/8508)) ([85c52e2](https://github.com/windmill-labs/windmill/commit/85c52e2cded10606cc895d0d3b717e13c69bc9b3)) +* use resource-level scope overrides during OAuth2 token refresh ([#8540](https://github.com/windmill-labs/windmill/issues/8540)) ([55ad0ff](https://github.com/windmill-labs/windmill/commit/55ad0ff5c499c33b766f47c6f32ba5d3eeb14763)) + +## [1.664.0](https://github.com/windmill-labs/windmill/compare/v1.663.0...v1.664.0) (2026-03-24) + + +### Features + +* add instance-level AI settings ([#8453](https://github.com/windmill-labs/windmill/issues/8453)) ([db5e036](https://github.com/windmill-labs/windmill/commit/db5e03610da325288d53afdbca94b9cbfc7ceace)) +* add selfApproval option to WAC + inline approval buttons ([#8440](https://github.com/windmill-labs/windmill/issues/8440)) ([d578e40](https://github.com/windmill-labs/windmill/commit/d578e40101a838d3dffda14157cf72ee4d5a93c0)) +* flow group nodes with collapsible groups ([#8075](https://github.com/windmill-labs/windmill/issues/8075)) ([81eb446](https://github.com/windmill-labs/windmill/commit/81eb446eee359f44374b81320690e5345fd08c15)) + + +### Bug Fixes + +* add GIT_SSL_CAINFO to tracing proxy env vars ([#8502](https://github.com/windmill-labs/windmill/issues/8502)) ([bdfd5d5](https://github.com/windmill-labs/windmill/commit/bdfd5d57261a4bb760fc57ad41ee56aff9b9c0af)) +* create parent dirs and accept 'python' alias in script bootstrap ([#8497](https://github.com/windmill-labs/windmill/issues/8497)) ([7f27d99](https://github.com/windmill-labs/windmill/commit/7f27d996accb3c3b471d1c50df397867d89c738a)) + +## [1.663.0](https://github.com/windmill-labs/windmill/compare/v1.662.0...v1.663.0) (2026-03-24) + + +### Features + +* add summary field for native triggers ([#8476](https://github.com/windmill-labs/windmill/issues/8476)) ([5089a45](https://github.com/windmill-labs/windmill/commit/5089a458819abbc6f241bc354bebb91520bd1a52)) +* add typed request body to OpenAPI spec generation ([#8481](https://github.com/windmill-labs/windmill/issues/8481)) ([37ebaf4](https://github.com/windmill-labs/windmill/commit/37ebaf4d0ac342703498733f97778a552f979f6a)) +* **cli:** better stale scripts detection [#3](https://github.com/windmill-labs/windmill/issues/3) ([#8480](https://github.com/windmill-labs/windmill/issues/8480)) ([9643006](https://github.com/windmill-labs/windmill/commit/9643006f1e90b991b334bb58caf62301bc26d09d)) +* Debounce node ([#8324](https://github.com/windmill-labs/windmill/issues/8324)) ([5d1c54d](https://github.com/windmill-labs/windmill/commit/5d1c54d9b33d6ff6f2c98481a2740d1e7629cdfa)) +* surface permissioned_as selector in trigger editor UI ([#8475](https://github.com/windmill-labs/windmill/issues/8475)) ([f035b53](https://github.com/windmill-labs/windmill/commit/f035b538bbd786445526339f88be8f33a3628105)) + + +### Bug Fixes + +* clean up stale dependency map entries for renamed scripts ([#8492](https://github.com/windmill-labs/windmill/issues/8492)) ([47c0c36](https://github.com/windmill-labs/windmill/commit/47c0c363f4fc1d9af7efd07ea172e32989ce50d2)) +* **cli:** add Svelte 5 event delegation guidance and safe push to raw-app skill ([#8466](https://github.com/windmill-labs/windmill/issues/8466)) ([911df95](https://github.com/windmill-labs/windmill/commit/911df958e78d2dab9823dfa7d7e5c9824fc2d565)) +* Fix worker panic when job_isolation changed to unshare at runtime ([#8490](https://github.com/windmill-labs/windmill/issues/8490)) ([cbe47c0](https://github.com/windmill-labs/windmill/commit/cbe47c0b6c22f79452d020777e481ee26970f25b)) +* improve SQS retries ([3c8d351](https://github.com/windmill-labs/windmill/commit/3c8d351c9722a089133871019d27cf3bc3cdc159)) +* Move database manager SQL queries to backend ([#8306](https://github.com/windmill-labs/windmill/issues/8306)) ([aa30fd2](https://github.com/windmill-labs/windmill/commit/aa30fd252dcf40233d191c43a6293fb9feabf010)) +* prevent SQL injection in job query parameters ([#8494](https://github.com/windmill-labs/windmill/issues/8494)) ([54f5a19](https://github.com/windmill-labs/windmill/commit/54f5a19377e9df712e18f85f896e21b1776981ed)) +* respect NO_COLOR env variable for stdout log output ([#8483](https://github.com/windmill-labs/windmill/issues/8483)) ([f329ee7](https://github.com/windmill-labs/windmill/commit/f329ee7aaefbae0ad344743c40825440a936bd30)) +* show effective isolation level on workers page ([#8491](https://github.com/windmill-labs/windmill/issues/8491)) ([37886ed](https://github.com/windmill-labs/windmill/commit/37886edda1443293806a9b1b810196b72e076b12)) +* skip debounce arg accumulation when batch table is empty (CE) ([#8485](https://github.com/windmill-labs/windmill/issues/8485)) ([010753c](https://github.com/windmill-labs/windmill/commit/010753c73ac85237af50acadf9c08567b1bc993c)) +* stop_after_if with empty error_message prevents flow from stopping ([#8464](https://github.com/windmill-labs/windmill/issues/8464)) ([1503bf9](https://github.com/windmill-labs/windmill/commit/1503bf948e3340b8a6933d71885f8f2cb8dc1867)) + ## [1.662.0](https://github.com/windmill-labs/windmill/compare/v1.661.0...v1.662.0) (2026-03-20) diff --git a/backend/.sqlx/query-077467cd813d5af161cb1cc232724f26984822d4c28ba36c0a9331273b10edc0.json b/backend/.sqlx/query-077467cd813d5af161cb1cc232724f26984822d4c28ba36c0a9331273b10edc0.json new file mode 100644 index 0000000000..0f5113bac5 --- /dev/null +++ b/backend/.sqlx/query-077467cd813d5af161cb1cc232724f26984822d4c28ba36c0a9331273b10edc0.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO email_to_igroup (email, igroup) VALUES ('alice@example.com', 'admins') ON CONFLICT DO NOTHING", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "077467cd813d5af161cb1cc232724f26984822d4c28ba36c0a9331273b10edc0" +} diff --git a/backend/.sqlx/query-07770a002a49428c4f956cfc7262d6b6792ae5b97ed90b0ee07d17480b2dffe2.json b/backend/.sqlx/query-07770a002a49428c4f956cfc7262d6b6792ae5b97ed90b0ee07d17480b2dffe2.json new file mode 100644 index 0000000000..145fdc1229 --- /dev/null +++ b/backend/.sqlx/query-07770a002a49428c4f956cfc7262d6b6792ae5b97ed90b0ee07d17480b2dffe2.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT setting::bigint as \"max!\" FROM pg_settings WHERE name = 'max_connections'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "max!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "07770a002a49428c4f956cfc7262d6b6792ae5b97ed90b0ee07d17480b2dffe2" +} diff --git a/backend/.sqlx/query-08522e494e34f4ecae21460262bf0ed3c5a197dd744c87cb760aaf47001febbd.json b/backend/.sqlx/query-08522e494e34f4ecae21460262bf0ed3c5a197dd744c87cb760aaf47001febbd.json new file mode 100644 index 0000000000..31eaa4bbd3 --- /dev/null +++ b/backend/.sqlx/query-08522e494e34f4ecae21460262bf0ed3c5a197dd744c87cb760aaf47001febbd.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM trashbin WHERE id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [] + }, + "hash": "08522e494e34f4ecae21460262bf0ed3c5a197dd744c87cb760aaf47001febbd" +} diff --git a/backend/.sqlx/query-6f9386dfcb4c201525722aee3caa25bf2f3a35d90f7354c7d3aef8a3538a03a7.json b/backend/.sqlx/query-1048d1c95270ce1f36c02bce31a2bc8a88935c613bd213b7156299811377db8e.json similarity index 57% rename from backend/.sqlx/query-6f9386dfcb4c201525722aee3caa25bf2f3a35d90f7354c7d3aef8a3538a03a7.json rename to backend/.sqlx/query-1048d1c95270ce1f36c02bce31a2bc8a88935c613bd213b7156299811377db8e.json index 69af249a3f..577b2b3826 100644 --- a/backend/.sqlx/query-6f9386dfcb4c201525722aee3caa25bf2f3a35d90f7354c7d3aef8a3538a03a7.json +++ b/backend/.sqlx/query-1048d1c95270ce1f36c02bce31a2bc8a88935c613bd213b7156299811377db8e.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n INSERT INTO native_trigger (\n external_id,\n workspace_id,\n service_name,\n script_path,\n is_flow,\n webhook_token_hash,\n service_config\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7\n )\n ON CONFLICT (external_id, workspace_id, service_name)\n DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, error = NULL, updated_at = NOW()\n ", + "query": "\n INSERT INTO native_trigger (\n external_id,\n workspace_id,\n service_name,\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n summary\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7, $8\n )\n ON CONFLICT (external_id, workspace_id, service_name)\n DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, summary = $8, error = NULL, updated_at = NOW()\n ", "describe": { "columns": [], "parameters": { @@ -21,10 +21,11 @@ "Varchar", "Bool", "Varchar", - "Jsonb" + "Jsonb", + "Varchar" ] }, "nullable": [] }, - "hash": "6f9386dfcb4c201525722aee3caa25bf2f3a35d90f7354c7d3aef8a3538a03a7" + "hash": "1048d1c95270ce1f36c02bce31a2bc8a88935c613bd213b7156299811377db8e" } diff --git a/backend/.sqlx/query-05027983ffdb11824190543754d0be922e1463d2046753cf80377369a90013ab.json b/backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json similarity index 83% rename from backend/.sqlx/query-05027983ffdb11824190543754d0be922e1463d2046753cf80377369a90013ab.json rename to backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json index 53a3863587..d2c85b0e53 100644 --- a/backend/.sqlx/query-05027983ffdb11824190543754d0be922e1463d2046753cf80377369a90013ab.json +++ b/backend/.sqlx/query-115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT $1 OFFSET $2", + "query": "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT $1 OFFSET $2", "describe": { "columns": [ { @@ -57,6 +57,11 @@ "ordinal": 10, "name": "role_source", "type_info": "Varchar" + }, + { + "ordinal": 11, + "name": "disabled", + "type_info": "Bool" } ], "parameters": { @@ -76,8 +81,9 @@ true, null, false, + false, false ] }, - "hash": "05027983ffdb11824190543754d0be922e1463d2046753cf80377369a90013ab" + "hash": "115a9cb44d0a41952c08dc36e0331410d32a8d672cfa4929e9e3763c51daa1bc" } diff --git a/backend/.sqlx/query-143acebe5d815c5d828013ebe46274f891f953c75f821499552ab7794f75063d.json b/backend/.sqlx/query-143acebe5d815c5d828013ebe46274f891f953c75f821499552ab7794f75063d.json new file mode 100644 index 0000000000..3514370ec8 --- /dev/null +++ b/backend/.sqlx/query-143acebe5d815c5d828013ebe46274f891f953c75f821499552ab7794f75063d.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') as \"exists!\"", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists!", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "143acebe5d815c5d828013ebe46274f891f953c75f821499552ab7794f75063d" +} diff --git a/backend/.sqlx/query-bac545933a627a62b7845d8aab80702443285e4d1d11e5a0f4cd2a3d4add51bb.json b/backend/.sqlx/query-15014ce696cf2af4f719a537a4e3ca5b322cc130a35a91f8b8854f5ebdf25ad2.json similarity index 84% rename from backend/.sqlx/query-bac545933a627a62b7845d8aab80702443285e4d1d11e5a0f4cd2a3d4add51bb.json rename to backend/.sqlx/query-15014ce696cf2af4f719a537a4e3ca5b322cc130a35a91f8b8854f5ebdf25ad2.json index a90b2b8398..66b8c431da 100644 --- a/backend/.sqlx/query-bac545933a627a62b7845d8aab80702443285e4d1d11e5a0f4cd2a3d4add51bb.json +++ b/backend/.sqlx/query-15014ce696cf2af4f719a537a4e3ca5b322cc130a35a91f8b8854f5ebdf25ad2.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND external_id = $3\n ", + "query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at,\n summary\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND external_id = $3\n ", "describe": { "columns": [ { @@ -62,6 +62,11 @@ "ordinal": 9, "name": "updated_at", "type_info": "Timestamptz" + }, + { + "ordinal": 10, + "name": "summary", + "type_info": "Varchar" } ], "parameters": { @@ -91,8 +96,9 @@ true, true, false, - false + false, + true ] }, - "hash": "bac545933a627a62b7845d8aab80702443285e4d1d11e5a0f4cd2a3d4add51bb" + "hash": "15014ce696cf2af4f719a537a4e3ca5b322cc130a35a91f8b8854f5ebdf25ad2" } diff --git a/backend/.sqlx/query-1721f8b52ea265c0537fd7c742deddf0afbe5cf0d81b15e487c411ae169d3a89.json b/backend/.sqlx/query-1721f8b52ea265c0537fd7c742deddf0afbe5cf0d81b15e487c411ae169d3a89.json new file mode 100644 index 0000000000..ba4ad17175 --- /dev/null +++ b/backend/.sqlx/query-1721f8b52ea265c0537fd7c742deddf0afbe5cf0d81b15e487c411ae169d3a89.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT igroup FROM email_to_igroup WHERE email = 'alice@example.com'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "igroup", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false + ] + }, + "hash": "1721f8b52ea265c0537fd7c742deddf0afbe5cf0d81b15e487c411ae169d3a89" +} diff --git a/backend/.sqlx/query-192ddae8c3c82a8f099a4944483024d9826a328bf0416c22daf06fff5ced08f6.json b/backend/.sqlx/query-192ddae8c3c82a8f099a4944483024d9826a328bf0416c22daf06fff5ced08f6.json new file mode 100644 index 0000000000..dc7c41cfd3 --- /dev/null +++ b/backend/.sqlx/query-192ddae8c3c82a8f099a4944483024d9826a328bf0416c22daf06fff5ced08f6.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM token WHERE email = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "192ddae8c3c82a8f099a4944483024d9826a328bf0416c22daf06fff5ced08f6" +} diff --git a/backend/.sqlx/query-1bc77ad29b9c68b1d339b85158bc3592deb61d1111d1430ddd2879b72e6424ef.json b/backend/.sqlx/query-1bc77ad29b9c68b1d339b85158bc3592deb61d1111d1430ddd2879b72e6424ef.json new file mode 100644 index 0000000000..0930bdf1b8 --- /dev/null +++ b/backend/.sqlx/query-1bc77ad29b9c68b1d339b85158bc3592deb61d1111d1430ddd2879b72e6424ef.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT ws.default_app AS default_app_path, av.raw_app AS \"default_app_raw: Option\"\n FROM workspace_settings ws\n LEFT JOIN app ON app.path = ws.default_app AND app.workspace_id = ws.workspace_id\n LEFT JOIN app_version av ON av.id = app.versions[array_upper(app.versions, 1)]\n WHERE ws.workspace_id = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "default_app_path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "default_app_raw: Option", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + true, + false + ] + }, + "hash": "1bc77ad29b9c68b1d339b85158bc3592deb61d1111d1430ddd2879b72e6424ef" +} diff --git a/backend/.sqlx/query-6aabe704395c9be30c86d15a5d22f3509b4fcea56227b019588837132b64d58b.json b/backend/.sqlx/query-1cf8597b9d37ec5a924aff8cbc0a05768ed9a679ba908ab16497a9bd55578ba1.json similarity index 75% rename from backend/.sqlx/query-6aabe704395c9be30c86d15a5d22f3509b4fcea56227b019588837132b64d58b.json rename to backend/.sqlx/query-1cf8597b9d37ec5a924aff8cbc0a05768ed9a679ba908ab16497a9bd55578ba1.json index 6e1b36a97c..b4a9f19f45 100644 --- a/backend/.sqlx/query-6aabe704395c9be30c86d15a5d22f3509b4fcea56227b019588837132b64d58b.json +++ b/backend/.sqlx/query-1cf8597b9d37ec5a924aff8cbc0a05768ed9a679ba908ab16497a9bd55578ba1.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT usr.*, password.super_admin, password.name FROM usr LEFT JOIN password ON usr.email = password.email Where usr.username = $1 AND workspace_id = $2\n ", + "query": "SELECT usr.*, COALESCE(password.super_admin, false) as \"super_admin!\", password.name FROM usr LEFT JOIN password ON usr.email = password.email Where usr.username = $1 AND workspace_id = $2\n ", "describe": { "columns": [ { @@ -50,11 +50,16 @@ }, { "ordinal": 9, - "name": "super_admin", + "name": "is_service_account", "type_info": "Bool" }, { "ordinal": 10, + "name": "super_admin!", + "type_info": "Bool" + }, + { + "ordinal": 11, "name": "name", "type_info": "Varchar" } @@ -76,8 +81,9 @@ true, true, false, + null, true ] }, - "hash": "6aabe704395c9be30c86d15a5d22f3509b4fcea56227b019588837132b64d58b" + "hash": "1cf8597b9d37ec5a924aff8cbc0a05768ed9a679ba908ab16497a9bd55578ba1" } diff --git a/backend/.sqlx/query-1d995dd5a094631ae96c16d68026fdeb22714af38162e87c02b052a5b8ec2645.json b/backend/.sqlx/query-1d995dd5a094631ae96c16d68026fdeb22714af38162e87c02b052a5b8ec2645.json new file mode 100644 index 0000000000..4ae3cea3ff --- /dev/null +++ b/backend/.sqlx/query-1d995dd5a094631ae96c16d68026fdeb22714af38162e87c02b052a5b8ec2645.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM trashbin WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [] + }, + "hash": "1d995dd5a094631ae96c16d68026fdeb22714af38162e87c02b052a5b8ec2645" +} diff --git a/backend/.sqlx/query-1dd73eff0e89b84c0316af2760a136afdd19dc34f9f31c4f9de6b0f74bc386a6.json b/backend/.sqlx/query-1dd73eff0e89b84c0316af2760a136afdd19dc34f9f31c4f9de6b0f74bc386a6.json new file mode 100644 index 0000000000..987acc6d1b --- /dev/null +++ b/backend/.sqlx/query-1dd73eff0e89b84c0316af2760a136afdd19dc34f9f31c4f9de6b0f74bc386a6.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n schemaname || '.' || relname as \"table_name!\",\n pg_total_relation_size(relid) as \"total_size_bytes!\",\n pg_size_pretty(pg_total_relation_size(relid)) as \"total_size_pretty!\"\n FROM pg_catalog.pg_statio_user_tables\n ORDER BY pg_total_relation_size(relid) DESC\n LIMIT 15", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "table_name!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "total_size_bytes!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "total_size_pretty!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "1dd73eff0e89b84c0316af2760a136afdd19dc34f9f31c4f9de6b0f74bc386a6" +} diff --git a/backend/.sqlx/query-21f4840f60e8310d7b7efcba7483e69e4ef8821c6cbf3b4f296b3853d95692af.json b/backend/.sqlx/query-21f4840f60e8310d7b7efcba7483e69e4ef8821c6cbf3b4f296b3853d95692af.json new file mode 100644 index 0000000000..a78e67067f --- /dev/null +++ b/backend/.sqlx/query-21f4840f60e8310d7b7efcba7483e69e4ef8821c6cbf3b4f296b3853d95692af.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) FROM usr WHERE workspace_id = $1 AND disabled = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "21f4840f60e8310d7b7efcba7483e69e4ef8821c6cbf3b4f296b3853d95692af" +} diff --git a/backend/.sqlx/query-23b9c862d050b00aaa332527b62ef901cd3c417b9f3af03f35009213143bd443.json b/backend/.sqlx/query-23b9c862d050b00aaa332527b62ef901cd3c417b9f3af03f35009213143bd443.json new file mode 100644 index 0000000000..d6946e80d0 --- /dev/null +++ b/backend/.sqlx/query-23b9c862d050b00aaa332527b62ef901cd3c417b9f3af03f35009213143bd443.json @@ -0,0 +1,28 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT email, disabled FROM password WHERE email = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "email", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "disabled", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + false + ] + }, + "hash": "23b9c862d050b00aaa332527b62ef901cd3c417b9f3af03f35009213143bd443" +} diff --git a/backend/.sqlx/query-250a4e3f1a1f95296f7075bf8780e9c7407e89c8f7636484895e99f5a5e71297.json b/backend/.sqlx/query-250a4e3f1a1f95296f7075bf8780e9c7407e89c8f7636484895e99f5a5e71297.json new file mode 100644 index 0000000000..6fd286898a --- /dev/null +++ b/backend/.sqlx/query-250a4e3f1a1f95296f7075bf8780e9c7407e89c8f7636484895e99f5a5e71297.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr (workspace_id, username, email, is_admin, operator, added_via)\n VALUES ($1, 'alice', 'alice@example.com', false, true, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "250a4e3f1a1f95296f7075bf8780e9c7407e89c8f7636484895e99f5a5e71297" +} diff --git a/backend/.sqlx/query-26e62b4509e44a7548957ad4ef217fd46bc03d5dca19344cd3bf7b131fa40ed2.json b/backend/.sqlx/query-26e62b4509e44a7548957ad4ef217fd46bc03d5dca19344cd3bf7b131fa40ed2.json new file mode 100644 index 0000000000..2ab912644a --- /dev/null +++ b/backend/.sqlx/query-26e62b4509e44a7548957ad4ef217fd46bc03d5dca19344cd3bf7b131fa40ed2.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value FROM global_settings WHERE name = 'retention_period_secs'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false + ] + }, + "hash": "26e62b4509e44a7548957ad4ef217fd46bc03d5dca19344cd3bf7b131fa40ed2" +} diff --git a/backend/.sqlx/query-2ba03e555d2e09dbd0e2ae5ddfd9a268a675bdb23615c78904cebe7f1e31f400.json b/backend/.sqlx/query-2ba03e555d2e09dbd0e2ae5ddfd9a268a675bdb23615c78904cebe7f1e31f400.json new file mode 100644 index 0000000000..3f4b2f53ab --- /dev/null +++ b/backend/.sqlx/query-2ba03e555d2e09dbd0e2ae5ddfd9a268a675bdb23615c78904cebe7f1e31f400.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr (workspace_id, username, email, is_admin, operator, added_via)\n VALUES ($1, 'alice', 'alice@example.com', true, false, $2)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "2ba03e555d2e09dbd0e2ae5ddfd9a268a675bdb23615c78904cebe7f1e31f400" +} diff --git a/backend/.sqlx/query-2d4ccf3ee19a70cbb5bd034c74703bbb30f217cd3673821e11bae3bf9f925720.json b/backend/.sqlx/query-2d4ccf3ee19a70cbb5bd034c74703bbb30f217cd3673821e11bae3bf9f925720.json new file mode 100644 index 0000000000..007c6fdcb2 --- /dev/null +++ b/backend/.sqlx/query-2d4ccf3ee19a70cbb5bd034c74703bbb30f217cd3673821e11bae3bf9f925720.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n ws.workspace_id as \"workspace_id!\",\n dt.key as \"name!\",\n dt.value->>'table_name' as \"table_name\"\n FROM workspace_settings ws,\n jsonb_each(ws.datatable) dt\n WHERE dt.value->>'resource_type' = 'instance'\n AND dt.value->>'table_name' IS NOT NULL", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id!", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "name!", + "type_info": "Text" + }, + { + "ordinal": 2, + "name": "table_name", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + null, + null + ] + }, + "hash": "2d4ccf3ee19a70cbb5bd034c74703bbb30f217cd3673821e11bae3bf9f925720" +} diff --git a/backend/.sqlx/query-2d95191e899d60385b32f36f2e38137e4173a34c54344ee522745640d48b8813.json b/backend/.sqlx/query-2d95191e899d60385b32f36f2e38137e4173a34c54344ee522745640d48b8813.json new file mode 100644 index 0000000000..d31420ada0 --- /dev/null +++ b/backend/.sqlx/query-2d95191e899d60385b32f36f2e38137e4173a34c54344ee522745640d48b8813.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n COUNT(*) as \"total!\",\n COUNT(*) FILTER (WHERE state = 'active') as \"active!\",\n COUNT(*) FILTER (WHERE state = 'idle') as \"idle!\"\n FROM pg_stat_activity\n WHERE backend_type = 'client backend'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "total!", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "active!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "idle!", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null + ] + }, + "hash": "2d95191e899d60385b32f36f2e38137e4173a34c54344ee522745640d48b8813" +} diff --git a/backend/.sqlx/query-30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa.json b/backend/.sqlx/query-30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa.json new file mode 100644 index 0000000000..ca31cbc9ca --- /dev/null +++ b/backend/.sqlx/query-30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa.json @@ -0,0 +1,44 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n table_name as \"table_name!\",\n SUM(live_tuples)::bigint as \"live_tuples!\",\n SUM(dead_tuples)::bigint as \"dead_tuples!\",\n MAX(last_autovacuum) as \"last_autovacuum\",\n MAX(last_autoanalyze) as \"last_autoanalyze\"\n FROM (\n SELECT\n CASE\n WHEN i.inhparent IS NOT NULL THEN schemaname || '.' || p.relname\n ELSE schemaname || '.' || s.relname\n END as table_name,\n COALESCE(n_live_tup, 0) as live_tuples,\n COALESCE(n_dead_tup, 0) as dead_tuples,\n last_autovacuum,\n last_autoanalyze\n FROM pg_stat_user_tables s\n LEFT JOIN pg_class c ON c.relname = s.relname AND c.relnamespace = (\n SELECT oid FROM pg_namespace WHERE nspname = s.schemaname\n )\n LEFT JOIN pg_inherits i ON i.inhrelid = c.oid\n LEFT JOIN pg_class p ON p.oid = i.inhparent\n ) sub\n GROUP BY table_name\n ORDER BY SUM(dead_tuples) DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "table_name!", + "type_info": "Text" + }, + { + "ordinal": 1, + "name": "live_tuples!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "dead_tuples!", + "type_info": "Int8" + }, + { + "ordinal": 3, + "name": "last_autovacuum", + "type_info": "Timestamptz" + }, + { + "ordinal": 4, + "name": "last_autoanalyze", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null, + null, + null + ] + }, + "hash": "30930bfb0513f1a70194a900011b2e890bc4146bb0419210cd76743cacda8bfa" +} diff --git a/backend/.sqlx/query-32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9.json b/backend/.sqlx/query-32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9.json deleted file mode 100644 index 0a2976f868..0000000000 --- a/backend/.sqlx/query-32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9.json +++ /dev/null @@ -1,28 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT parent_job, flow_step_id FROM v2_job WHERE id = $1", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "parent_job", - "type_info": "Uuid" - }, - { - "ordinal": 1, - "name": "flow_step_id", - "type_info": "Varchar" - } - ], - "parameters": { - "Left": [ - "Uuid" - ] - }, - "nullable": [ - true, - true - ] - }, - "hash": "32ca7941db013dacd2479962fa9ed5c8c64daec45ba820a6c8f7d7ab76cc40c9" -} diff --git a/backend/.sqlx/query-3453c0b7dd3c4d2c9bc639f379901741955502c9345e82a9b7fbbf3d3c7ab517.json b/backend/.sqlx/query-3453c0b7dd3c4d2c9bc639f379901741955502c9345e82a9b7fbbf3d3c7ab517.json new file mode 100644 index 0000000000..5893b08c81 --- /dev/null +++ b/backend/.sqlx/query-3453c0b7dd3c4d2c9bc639f379901741955502c9345e82a9b7fbbf3d3c7ab517.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM trashbin WHERE expires_at <= now()", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "3453c0b7dd3c4d2c9bc639f379901741955502c9345e82a9b7fbbf3d3c7ab517" +} diff --git a/backend/.sqlx/query-359cd29f531d263a8cf7205e0869229a610767087f01e0154be8da0620fa114b.json b/backend/.sqlx/query-359cd29f531d263a8cf7205e0869229a610767087f01e0154be8da0620fa114b.json new file mode 100644 index 0000000000..23cf739f90 --- /dev/null +++ b/backend/.sqlx/query-359cd29f531d263a8cf7205e0869229a610767087f01e0154be8da0620fa114b.json @@ -0,0 +1,38 @@ +{ + "db_name": "PostgreSQL", + "query": "WITH all_audit AS (SELECT username, operation, timestamp FROM audit_partitioned UNION ALL SELECT username, operation, timestamp FROM audit),\n active_users as (SELECT distinct username as email FROM all_audit WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n active_authors as (SELECT distinct email FROM usr WHERE usr.operator IS false AND email IN (SELECT email FROM active_users)),\n active_authors_agg as (SELECT array_agg(email) as authors FROM active_authors),\n active_ops_agg as (SELECT array_agg(email) as operators from active_users WHERE email NOT IN (SELECT email FROM active_authors))\n SELECT active_authors_agg.authors, active_ops_agg.operators, array_length(active_authors_agg.authors, 1) as author_count, array_length(active_ops_agg.operators, 1) as operator_count FROM active_authors_agg, active_ops_agg", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "authors", + "type_info": "VarcharArray" + }, + { + "ordinal": 1, + "name": "operators", + "type_info": "VarcharArray" + }, + { + "ordinal": 2, + "name": "author_count", + "type_info": "Int4" + }, + { + "ordinal": 3, + "name": "operator_count", + "type_info": "Int4" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null, + null, + null + ] + }, + "hash": "359cd29f531d263a8cf7205e0869229a610767087f01e0154be8da0620fa114b" +} diff --git a/backend/.sqlx/query-384f5e9b2ab8e430141e28ea58854cbcfbcf96fd2adbf0513ce942cfe9bceaf0.json b/backend/.sqlx/query-384f5e9b2ab8e430141e28ea58854cbcfbcf96fd2adbf0513ce942cfe9bceaf0.json new file mode 100644 index 0000000000..cb033816d1 --- /dev/null +++ b/backend/.sqlx/query-384f5e9b2ab8e430141e28ea58854cbcfbcf96fd2adbf0513ce942cfe9bceaf0.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_database_size(current_database()) as size_bytes, pg_size_pretty(pg_database_size(current_database())) as size_pretty", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "size_bytes", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "size_pretty", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "384f5e9b2ab8e430141e28ea58854cbcfbcf96fd2adbf0513ce942cfe9bceaf0" +} diff --git a/backend/.sqlx/query-3bd4f38a1629a69ddda622b6b436198b47c2fe1a507358d49f05031e7beedab6.json b/backend/.sqlx/query-3bd4f38a1629a69ddda622b6b436198b47c2fe1a507358d49f05031e7beedab6.json new file mode 100644 index 0000000000..0c71ee8ad2 --- /dev/null +++ b/backend/.sqlx/query-3bd4f38a1629a69ddda622b6b436198b47c2fe1a507358d49f05031e7beedab6.json @@ -0,0 +1,34 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT workspace_id,\n auto_invite->'instance_groups_roles' as instance_groups_roles,\n auto_invite->'instance_groups' as instance_groups_json\n FROM workspace_settings\n WHERE auto_invite->'instance_groups' ? $1\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "instance_groups_roles", + "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "instance_groups_json", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false, + null, + null + ] + }, + "hash": "3bd4f38a1629a69ddda622b6b436198b47c2fe1a507358d49f05031e7beedab6" +} diff --git a/backend/.sqlx/query-446404eda9b9632c9a1384af6bf2f88594825dbaa647290a58bd63df61b531a7.json b/backend/.sqlx/query-446404eda9b9632c9a1384af6bf2f88594825dbaa647290a58bd63df61b531a7.json new file mode 100644 index 0000000000..9768a13f3d --- /dev/null +++ b/backend/.sqlx/query-446404eda9b9632c9a1384af6bf2f88594825dbaa647290a58bd63df61b531a7.json @@ -0,0 +1,65 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, workspace_id, item_kind, item_path, item_data, deleted_by, deleted_at, expires_at\n FROM trashbin\n WHERE workspace_id = $1 AND id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "item_kind", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "item_path", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "item_data", + "type_info": "Jsonb" + }, + { + "ordinal": 5, + "name": "deleted_by", + "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "deleted_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 7, + "name": "expires_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + false, + false, + false + ] + }, + "hash": "446404eda9b9632c9a1384af6bf2f88594825dbaa647290a58bd63df61b531a7" +} diff --git a/backend/.sqlx/query-51c3274a8092d80503a6b97ef3896cc3ba1957042a48ac5f9629ada25b3e78ef.json b/backend/.sqlx/query-51c3274a8092d80503a6b97ef3896cc3ba1957042a48ac5f9629ada25b3e78ef.json new file mode 100644 index 0000000000..fa4a6fc50e --- /dev/null +++ b/backend/.sqlx/query-51c3274a8092d80503a6b97ef3896cc3ba1957042a48ac5f9629ada25b3e78ef.json @@ -0,0 +1,61 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, workspace_id, item_kind, item_path, deleted_by, deleted_at, expires_at\n FROM trashbin\n WHERE workspace_id = $1 AND item_kind = $2\n ORDER BY deleted_at DESC\n LIMIT $3 OFFSET $4", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "item_kind", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "item_path", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "deleted_by", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "deleted_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 6, + "name": "expires_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Int8", + "Int8" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + false, + false + ] + }, + "hash": "51c3274a8092d80503a6b97ef3896cc3ba1957042a48ac5f9629ada25b3e78ef" +} diff --git a/backend/.sqlx/query-52d765c87cb8da0ca71fb53156820e383a998a54c95355bb85fe7e762a0d9765.json b/backend/.sqlx/query-52d765c87cb8da0ca71fb53156820e383a998a54c95355bb85fe7e762a0d9765.json new file mode 100644 index 0000000000..16ae512f37 --- /dev/null +++ b/backend/.sqlx/query-52d765c87cb8da0ca71fb53156820e383a998a54c95355bb85fe7e762a0d9765.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT DISTINCT imported_path as \"imported_path!\"\n FROM dependency_map\n WHERE workspace_id = $1\n AND importer_path = $2\n AND imported_path NOT LIKE 'dependencies/%'\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "imported_path!", + "type_info": "Varchar" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "52d765c87cb8da0ca71fb53156820e383a998a54c95355bb85fe7e762a0d9765" +} diff --git a/backend/.sqlx/query-544a02447bb2cbe8354a5c4ae93685848af38a3461257a9734c43cbd7bd905cb.json b/backend/.sqlx/query-544a02447bb2cbe8354a5c4ae93685848af38a3461257a9734c43cbd7bd905cb.json new file mode 100644 index 0000000000..2086242e9c --- /dev/null +++ b/backend/.sqlx/query-544a02447bb2cbe8354a5c4ae93685848af38a3461257a9734c43cbd7bd905cb.json @@ -0,0 +1,35 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT email, is_service_account, disabled FROM usr WHERE username = $1 AND workspace_id = $2", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "email", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "is_service_account", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "disabled", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "544a02447bb2cbe8354a5c4ae93685848af38a3461257a9734c43cbd7bd905cb" +} diff --git a/backend/.sqlx/query-ed1a053c7b22d9cb69767be40d33f3be67b6160cd258c86b8e8f22a6d601afd0.json b/backend/.sqlx/query-5d160ba4958583f1ad42de846c544d8d8e81e1b54925a0c5f2cedc1817d99a1b.json similarity index 54% rename from backend/.sqlx/query-ed1a053c7b22d9cb69767be40d33f3be67b6160cd258c86b8e8f22a6d601afd0.json rename to backend/.sqlx/query-5d160ba4958583f1ad42de846c544d8d8e81e1b54925a0c5f2cedc1817d99a1b.json index b7c642ef12..8fcffe364a 100644 --- a/backend/.sqlx/query-ed1a053c7b22d9cb69767be40d33f3be67b6160cd258c86b8e8f22a6d601afd0.json +++ b/backend/.sqlx/query-5d160ba4958583f1ad42de846c544d8d8e81e1b54925a0c5f2cedc1817d99a1b.json @@ -1,11 +1,11 @@ { "db_name": "PostgreSQL", - "query": "SELECT default_app FROM workspace_settings WHERE workspace_id = $1", + "query": "SELECT igroup FROM email_to_igroup WHERE email = $1", "describe": { "columns": [ { "ordinal": 0, - "name": "default_app", + "name": "igroup", "type_info": "Varchar" } ], @@ -15,8 +15,8 @@ ] }, "nullable": [ - true + false ] }, - "hash": "ed1a053c7b22d9cb69767be40d33f3be67b6160cd258c86b8e8f22a6d601afd0" + "hash": "5d160ba4958583f1ad42de846c544d8d8e81e1b54925a0c5f2cedc1817d99a1b" } diff --git a/backend/.sqlx/query-5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c.json b/backend/.sqlx/query-5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c.json index 09775dcc3a..79625b6baf 100644 --- a/backend/.sqlx/query-5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c.json +++ b/backend/.sqlx/query-5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c.json @@ -47,6 +47,11 @@ "ordinal": 8, "name": "added_via", "type_info": "Jsonb" + }, + { + "ordinal": 9, + "name": "is_service_account", + "type_info": "Bool" } ], "parameters": { @@ -63,7 +68,8 @@ false, false, true, - true + true, + false ] }, "hash": "5d6adbe21b9f8dd984d1bfc750fb81763d8650c1316bb0b20816f1a5d61a678c" diff --git a/backend/.sqlx/query-60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b.json b/backend/.sqlx/query-60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b.json index 3a635ab004..ed09f2833f 100644 --- a/backend/.sqlx/query-60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b.json +++ b/backend/.sqlx/query-60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b.json @@ -47,6 +47,11 @@ "ordinal": 8, "name": "added_via", "type_info": "Jsonb" + }, + { + "ordinal": 9, + "name": "is_service_account", + "type_info": "Bool" } ], "parameters": { @@ -64,7 +69,8 @@ false, false, true, - true + true, + false ] }, "hash": "60b3a59805d463a61eed68072d1ea032b00fc9bd7a6db22f530f67eb9730fa3b" diff --git a/backend/.sqlx/query-62e8e443cf063fcb30799d9c8971c00d761d54811936deb87a0315ca9cdc9769.json b/backend/.sqlx/query-62e8e443cf063fcb30799d9c8971c00d761d54811936deb87a0315ca9cdc9769.json new file mode 100644 index 0000000000..f4a2546900 --- /dev/null +++ b/backend/.sqlx/query-62e8e443cf063fcb30799d9c8971c00d761d54811936deb87a0315ca9cdc9769.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT is_admin, operator, added_via FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "is_admin", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "operator", + "type_info": "Bool" + }, + { + "ordinal": 2, + "name": "added_via", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + false, + true + ] + }, + "hash": "62e8e443cf063fcb30799d9c8971c00d761d54811936deb87a0315ca9cdc9769" +} diff --git a/backend/.sqlx/query-cc269052ffc1e613d7edc31f0f7bb84f6e6301ad1afb028813105a121a69fa7e.json b/backend/.sqlx/query-63c48fde8c0c0fff9abffc3be27e9948556b636b70b818cc31c2d50921a27366.json similarity index 76% rename from backend/.sqlx/query-cc269052ffc1e613d7edc31f0f7bb84f6e6301ad1afb028813105a121a69fa7e.json rename to backend/.sqlx/query-63c48fde8c0c0fff9abffc3be27e9948556b636b70b818cc31c2d50921a27366.json index 2354b265a8..0ba2d18257 100644 --- a/backend/.sqlx/query-cc269052ffc1e613d7edc31f0f7bb84f6e6301ad1afb028813105a121a69fa7e.json +++ b/backend/.sqlx/query-63c48fde8c0c0fff9abffc3be27e9948556b636b70b818cc31c2d50921a27366.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT client, refresh_token, grant_type, cc_client_id, cc_client_secret, cc_token_url FROM account WHERE workspace_id = $1 AND id = $2", + "query": "SELECT client, refresh_token, grant_type, cc_client_id, cc_client_secret, cc_token_url, scopes FROM account WHERE workspace_id = $1 AND id = $2", "describe": { "columns": [ { @@ -32,6 +32,11 @@ "ordinal": 5, "name": "cc_token_url", "type_info": "Varchar" + }, + { + "ordinal": 6, + "name": "scopes", + "type_info": "TextArray" } ], "parameters": { @@ -46,8 +51,9 @@ false, true, true, + true, true ] }, - "hash": "cc269052ffc1e613d7edc31f0f7bb84f6e6301ad1afb028813105a121a69fa7e" + "hash": "63c48fde8c0c0fff9abffc3be27e9948556b636b70b818cc31c2d50921a27366" } diff --git a/backend/.sqlx/query-7e01ef5799168c0fc2779d42ce352827e2fda6711c0a1b104ca6435ddb14b47d.json b/backend/.sqlx/query-66e2f8468ba64f22b7a7caa18639d7c833ac2ec573bd89d878b5c8b1afc74d3a.json similarity index 54% rename from backend/.sqlx/query-7e01ef5799168c0fc2779d42ce352827e2fda6711c0a1b104ca6435ddb14b47d.json rename to backend/.sqlx/query-66e2f8468ba64f22b7a7caa18639d7c833ac2ec573bd89d878b5c8b1afc74d3a.json index 7c8a5bc156..15681aab51 100644 --- a/backend/.sqlx/query-7e01ef5799168c0fc2779d42ce352827e2fda6711c0a1b104ca6435ddb14b47d.json +++ b/backend/.sqlx/query-66e2f8468ba64f22b7a7caa18639d7c833ac2ec573bd89d878b5c8b1afc74d3a.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT\n workspace_id,\n auto_invite->'instance_groups_roles' as instance_groups_roles\n FROM workspace_settings\n WHERE\n auto_invite->'instance_groups' IS NOT NULL\n AND auto_invite->'instance_groups' ? $1\n ", + "query": "\n SELECT\n workspace_id,\n auto_invite->'instance_groups_roles' as instance_groups_roles,\n auto_invite->'instance_groups' as instance_groups_json\n FROM workspace_settings\n WHERE\n auto_invite->'instance_groups' IS NOT NULL\n AND auto_invite->'instance_groups' ? $1\n ", "describe": { "columns": [ { @@ -12,6 +12,11 @@ "ordinal": 1, "name": "instance_groups_roles", "type_info": "Jsonb" + }, + { + "ordinal": 2, + "name": "instance_groups_json", + "type_info": "Jsonb" } ], "parameters": { @@ -21,8 +26,9 @@ }, "nullable": [ false, + null, null ] }, - "hash": "7e01ef5799168c0fc2779d42ce352827e2fda6711c0a1b104ca6435ddb14b47d" + "hash": "66e2f8468ba64f22b7a7caa18639d7c833ac2ec573bd89d878b5c8b1afc74d3a" } diff --git a/backend/.sqlx/query-68c19cb0e18b94870bbe81f9aab92ba37da67cd2a56834c9d1378eab7551284d.json b/backend/.sqlx/query-68c19cb0e18b94870bbe81f9aab92ba37da67cd2a56834c9d1378eab7551284d.json new file mode 100644 index 0000000000..de69c0fc8c --- /dev/null +++ b/backend/.sqlx/query-68c19cb0e18b94870bbe81f9aab92ba37da67cd2a56834c9d1378eab7551284d.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE kafka_trigger\n SET\n kafka_resource_path = $1,\n group_id = $2,\n topics = $3,\n filters = $4,\n filter_logic = $5,\n auto_offset_reset = $6,\n auto_commit = $7,\n script_path = $8,\n path = $9,\n is_flow = $10,\n edited_by = $11,\n permissioned_as = $12,\n edited_at = now(),\n server_id = NULL,\n error = NULL,\n error_handler_path = $15,\n error_handler_args = $16,\n retry = $17\n WHERE\n workspace_id = $13 AND path = $14\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "VarcharArray", + "JsonbArray", + "Varchar", + "Varchar", + "Bool", + "Varchar", + "Varchar", + "Bool", + "Varchar", + "Varchar", + "Text", + "Text", + "Varchar", + "Jsonb", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "68c19cb0e18b94870bbe81f9aab92ba37da67cd2a56834c9d1378eab7551284d" +} diff --git a/backend/.sqlx/query-68d1370fa02f4fe585684a91e898c4aed45e6b8f409bb33c2681f92265922040.json b/backend/.sqlx/query-68d1370fa02f4fe585684a91e898c4aed45e6b8f409bb33c2681f92265922040.json new file mode 100644 index 0000000000..cd1d6810cd --- /dev/null +++ b/backend/.sqlx/query-68d1370fa02f4fe585684a91e898c4aed45e6b8f409bb33c2681f92265922040.json @@ -0,0 +1,24 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM usr WHERE workspace_id = $1 AND (username = $2 OR email = $3))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "68d1370fa02f4fe585684a91e898c4aed45e6b8f409bb33c2681f92265922040" +} diff --git a/backend/.sqlx/query-942c0abb55c910862fd45d3fa56a4eb6729f1a658101bda2d0b0fca96b3cfee5.json b/backend/.sqlx/query-6948eb5aabf82f2f4a08dd4410eb472080ecab3ed652912397245e5216ae0389.json similarity index 60% rename from backend/.sqlx/query-942c0abb55c910862fd45d3fa56a4eb6729f1a658101bda2d0b0fca96b3cfee5.json rename to backend/.sqlx/query-6948eb5aabf82f2f4a08dd4410eb472080ecab3ed652912397245e5216ae0389.json index e3ca43fbd0..a6f1f5f7cc 100644 --- a/backend/.sqlx/query-942c0abb55c910862fd45d3fa56a4eb6729f1a658101bda2d0b0fca96b3cfee5.json +++ b/backend/.sqlx/query-6948eb5aabf82f2f4a08dd4410eb472080ecab3ed652912397245e5216ae0389.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n INSERT INTO websocket_trigger (\n workspace_id,\n path,\n url,\n script_path,\n is_flow,\n mode,\n filters,\n initial_messages,\n url_runnable_args,\n edited_by,\n can_return_message,\n can_return_error_result,\n permissioned_as,\n edited_at,\n error_handler_path,\n error_handler_args,\n retry\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, now(), $14, $15, $16\n )\n ", + "query": "\n INSERT INTO websocket_trigger (\n workspace_id,\n path,\n url,\n script_path,\n is_flow,\n mode,\n filters,\n filter_logic,\n initial_messages,\n url_runnable_args,\n edited_by,\n can_return_message,\n can_return_error_result,\n permissioned_as,\n edited_at,\n error_handler_path,\n error_handler_args,\n retry\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, now(), $15, $16, $17\n )\n ", "describe": { "columns": [], "parameters": { @@ -23,6 +23,7 @@ } }, "JsonbArray", + "Varchar", "JsonbArray", "Jsonb", "Varchar", @@ -36,5 +37,5 @@ }, "nullable": [] }, - "hash": "942c0abb55c910862fd45d3fa56a4eb6729f1a658101bda2d0b0fca96b3cfee5" + "hash": "6948eb5aabf82f2f4a08dd4410eb472080ecab3ed652912397245e5216ae0389" } diff --git a/backend/.sqlx/query-a0a545fda5f3ebea0113d5daaf13358c964d9fb0f41bf2a1c834305b4d2398f2.json b/backend/.sqlx/query-6a8f4ed9946bb2a3c5e90695c90b70aa2e83fcb5aa0c953febdd9bac2d95bbec.json similarity index 57% rename from backend/.sqlx/query-a0a545fda5f3ebea0113d5daaf13358c964d9fb0f41bf2a1c834305b4d2398f2.json rename to backend/.sqlx/query-6a8f4ed9946bb2a3c5e90695c90b70aa2e83fcb5aa0c953febdd9bac2d95bbec.json index 99d8d2a181..c6d3374639 100644 --- a/backend/.sqlx/query-a0a545fda5f3ebea0113d5daaf13358c964d9fb0f41bf2a1c834305b4d2398f2.json +++ b/backend/.sqlx/query-6a8f4ed9946bb2a3c5e90695c90b70aa2e83fcb5aa0c953febdd9bac2d95bbec.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n INSERT INTO kafka_trigger (\n workspace_id,\n path,\n kafka_resource_path,\n group_id,\n topics,\n filters,\n auto_offset_reset,\n auto_commit,\n script_path,\n is_flow,\n mode,\n edited_by,\n permissioned_as,\n edited_at,\n error_handler_path,\n error_handler_args,\n retry\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, now(), $14, $15, $16\n )\n ", + "query": "\n INSERT INTO kafka_trigger (\n workspace_id,\n path,\n kafka_resource_path,\n group_id,\n topics,\n filters,\n filter_logic,\n auto_offset_reset,\n auto_commit,\n script_path,\n is_flow,\n mode,\n edited_by,\n permissioned_as,\n edited_at,\n error_handler_path,\n error_handler_args,\n retry\n ) VALUES (\n $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, now(), $15, $16, $17\n )\n ", "describe": { "columns": [], "parameters": { @@ -12,6 +12,7 @@ "VarcharArray", "JsonbArray", "Varchar", + "Varchar", "Bool", "Varchar", "Bool", @@ -36,5 +37,5 @@ }, "nullable": [] }, - "hash": "a0a545fda5f3ebea0113d5daaf13358c964d9fb0f41bf2a1c834305b4d2398f2" + "hash": "6a8f4ed9946bb2a3c5e90695c90b70aa2e83fcb5aa0c953febdd9bac2d95bbec" } diff --git a/backend/.sqlx/query-6d070f476538aa6fcd6227fe5312561a7d098f2af5287e1e6c339e15080378be.json b/backend/.sqlx/query-6d070f476538aa6fcd6227fe5312561a7d098f2af5287e1e6c339e15080378be.json new file mode 100644 index 0000000000..6d6acec840 --- /dev/null +++ b/backend/.sqlx/query-6d070f476538aa6fcd6227fe5312561a7d098f2af5287e1e6c339e15080378be.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT pg_advisory_xact_lock(hashtext($1 || '/' || $2))", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "pg_advisory_xact_lock", + "type_info": "Void" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "6d070f476538aa6fcd6227fe5312561a7d098f2af5287e1e6c339e15080378be" +} diff --git a/backend/.sqlx/query-1a69ef11a3f361f105c2a8af7b7fa182f3953150ade1756259b31a50e9308fce.json b/backend/.sqlx/query-6dafcc89668fb0e5740f23264b515b1724c36031da39d53ae6c329a479bdf8aa.json similarity index 82% rename from backend/.sqlx/query-1a69ef11a3f361f105c2a8af7b7fa182f3953150ade1756259b31a50e9308fce.json rename to backend/.sqlx/query-6dafcc89668fb0e5740f23264b515b1724c36031da39d53ae6c329a479bdf8aa.json index 492fffe8be..ea974dae42 100644 --- a/backend/.sqlx/query-1a69ef11a3f361f105c2a8af7b7fa182f3953150ade1756259b31a50e9308fce.json +++ b/backend/.sqlx/query-6dafcc89668fb0e5740f23264b515b1724c36031da39d53ae6c329a479bdf8aa.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND script_path = $3\n AND is_flow = $4\n LIMIT 1\n ", + "query": "\n SELECT\n external_id,\n workspace_id,\n service_name AS \"service_name!: ServiceName\",\n script_path,\n is_flow,\n webhook_token_hash,\n service_config,\n error,\n created_at,\n updated_at,\n summary\n FROM\n native_trigger\n WHERE\n workspace_id = $1\n AND service_name = $2\n AND script_path = $3\n AND is_flow = $4\n LIMIT 1\n ", "describe": { "columns": [ { @@ -62,6 +62,11 @@ "ordinal": 9, "name": "updated_at", "type_info": "Timestamptz" + }, + { + "ordinal": 10, + "name": "summary", + "type_info": "Varchar" } ], "parameters": { @@ -92,8 +97,9 @@ true, true, false, - false + false, + true ] }, - "hash": "1a69ef11a3f361f105c2a8af7b7fa182f3953150ade1756259b31a50e9308fce" + "hash": "6dafcc89668fb0e5740f23264b515b1724c36031da39d53ae6c329a479bdf8aa" } diff --git a/backend/.sqlx/query-6f941e4454f736b32eaef80cdfb9582d6e75af3dc159e7c5f12497d3957f1eef.json b/backend/.sqlx/query-6f941e4454f736b32eaef80cdfb9582d6e75af3dc159e7c5f12497d3957f1eef.json new file mode 100644 index 0000000000..8446e14aa4 --- /dev/null +++ b/backend/.sqlx/query-6f941e4454f736b32eaef80cdfb9582d6e75af3dc159e7c5f12497d3957f1eef.json @@ -0,0 +1,12 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr (workspace_id, username, email, is_admin, operator)\n VALUES ('ws-multi-group', 'alice', 'alice@example.com', true, false)", + "describe": { + "columns": [], + "parameters": { + "Left": [] + }, + "nullable": [] + }, + "hash": "6f941e4454f736b32eaef80cdfb9582d6e75af3dc159e7c5f12497d3957f1eef" +} diff --git a/backend/.sqlx/query-79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073.json b/backend/.sqlx/query-79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073.json deleted file mode 100644 index f990932367..0000000000 --- a/backend/.sqlx/query-79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "79b82ae996fba2e2ab53fcf84c108cb1ca21fbdba3373af54fadf1f4af324073" -} diff --git a/backend/.sqlx/query-7c5db0b3bd1dd1f766e1841ca620871a468033e05b6e0188ea4775b63fc66e84.json b/backend/.sqlx/query-7c5db0b3bd1dd1f766e1841ca620871a468033e05b6e0188ea4775b63fc66e84.json new file mode 100644 index 0000000000..304027071a --- /dev/null +++ b/backend/.sqlx/query-7c5db0b3bd1dd1f766e1841ca620871a468033e05b6e0188ea4775b63fc66e84.json @@ -0,0 +1,40 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n c.relname as \"table_name!\",\n pg_total_relation_size(c.oid) as \"size_bytes!\",\n pg_size_pretty(pg_total_relation_size(c.oid)) as \"size_pretty!\",\n COALESCE(c.reltuples, 0) as \"estimated_rows!\"\n FROM pg_class c\n JOIN pg_namespace n ON n.oid = c.relnamespace\n WHERE n.nspname = 'public' AND c.relname = ANY($1)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "table_name!", + "type_info": "Name" + }, + { + "ordinal": 1, + "name": "size_bytes!", + "type_info": "Int8" + }, + { + "ordinal": 2, + "name": "size_pretty!", + "type_info": "Text" + }, + { + "ordinal": 3, + "name": "estimated_rows!", + "type_info": "Float4" + } + ], + "parameters": { + "Left": [ + "NameArray" + ] + }, + "nullable": [ + false, + null, + null, + null + ] + }, + "hash": "7c5db0b3bd1dd1f766e1841ca620871a468033e05b6e0188ea4775b63fc66e84" +} diff --git a/backend/.sqlx/query-b1bd088c2e1aca3104bede7d0953369b6b17ad3ad62692ae6f2303be890e6391.json b/backend/.sqlx/query-870e1c3f0dc1aaa07ac74a2e37721ce352ad4fb67d36c19dce09d841e36f85dd.json similarity index 68% rename from backend/.sqlx/query-b1bd088c2e1aca3104bede7d0953369b6b17ad3ad62692ae6f2303be890e6391.json rename to backend/.sqlx/query-870e1c3f0dc1aaa07ac74a2e37721ce352ad4fb67d36c19dce09d841e36f85dd.json index bda29ecd5f..2b2fe3c8e6 100644 --- a/backend/.sqlx/query-b1bd088c2e1aca3104bede7d0953369b6b17ad3ad62692ae6f2303be890e6391.json +++ b/backend/.sqlx/query-870e1c3f0dc1aaa07ac74a2e37721ce352ad4fb67d36c19dce09d841e36f85dd.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO account (workspace_id, client, expires_at, refresh_token, grant_type, cc_client_id, cc_client_secret, cc_token_url, mcp_server_url) VALUES ($1, $2, now() + ($3 || ' seconds')::interval, $4, $5, $6, $7, $8, $9) RETURNING id", + "query": "INSERT INTO account (workspace_id, client, expires_at, refresh_token, grant_type, cc_client_id, cc_client_secret, cc_token_url, mcp_server_url, scopes) VALUES ($1, $2, now() + ($3 || ' seconds')::interval, $4, $5, $6, $7, $8, $9, $10) RETURNING id", "describe": { "columns": [ { @@ -19,12 +19,13 @@ "Varchar", "Varchar", "Varchar", - "Text" + "Text", + "TextArray" ] }, "nullable": [ false ] }, - "hash": "b1bd088c2e1aca3104bede7d0953369b6b17ad3ad62692ae6f2303be890e6391" + "hash": "870e1c3f0dc1aaa07ac74a2e37721ce352ad4fb67d36c19dce09d841e36f85dd" } diff --git a/backend/.sqlx/query-87ee10d8ba5ba281781f23e5390190fb90df980a19c900452f9b1a19c3620e30.json b/backend/.sqlx/query-87ee10d8ba5ba281781f23e5390190fb90df980a19c900452f9b1a19c3620e30.json new file mode 100644 index 0000000000..aaad5153bd --- /dev/null +++ b/backend/.sqlx/query-87ee10d8ba5ba281781f23e5390190fb90df980a19c900452f9b1a19c3620e30.json @@ -0,0 +1,32 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT\n route_path,\n workspace_id,\n http_method::TEXT AS \"http_method!\"\n FROM\n http_trigger\n WHERE\n workspaced_route IS FALSE\n AND route_path_key IN (\n SELECT\n route_path_key\n FROM\n http_trigger\n WHERE\n workspaced_route IS FALSE\n GROUP BY\n route_path_key, http_method\n HAVING COUNT(*) > 1\n )\n ORDER BY route_path_key\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "route_path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "http_method!", + "type_info": "Text" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + false, + null + ] + }, + "hash": "87ee10d8ba5ba281781f23e5390190fb90df980a19c900452f9b1a19c3620e30" +} diff --git a/backend/.sqlx/query-88a467f3c943b134a81ac69c3c6686d1ce1ff2f5aafc15ff1b63cfa86c09c4f0.json b/backend/.sqlx/query-88a467f3c943b134a81ac69c3c6686d1ce1ff2f5aafc15ff1b63cfa86c09c4f0.json new file mode 100644 index 0000000000..6ee3c89e40 --- /dev/null +++ b/backend/.sqlx/query-88a467f3c943b134a81ac69c3c6686d1ce1ff2f5aafc15ff1b63cfa86c09c4f0.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT is_admin, operator FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "is_admin", + "type_info": "Bool" + }, + { + "ordinal": 1, + "name": "operator", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false, + false + ] + }, + "hash": "88a467f3c943b134a81ac69c3c6686d1ce1ff2f5aafc15ff1b63cfa86c09c4f0" +} diff --git a/backend/.sqlx/query-8b25c4252da77cd2fe1b3916b518251dbb3c6d4c095efa015823f0324ab27d7f.json b/backend/.sqlx/query-8b25c4252da77cd2fe1b3916b518251dbb3c6d4c095efa015823f0324ab27d7f.json new file mode 100644 index 0000000000..9995bb1b51 --- /dev/null +++ b/backend/.sqlx/query-8b25c4252da77cd2fe1b3916b518251dbb3c6d4c095efa015823f0324ab27d7f.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO trashbin (workspace_id, item_kind, item_path, item_data, deleted_by)\n VALUES ($1, $2, $3, $4, $5) RETURNING id", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Text", + "Jsonb", + "Varchar" + ] + }, + "nullable": [ + false + ] + }, + "hash": "8b25c4252da77cd2fe1b3916b518251dbb3c6d4c095efa015823f0324ab27d7f" +} diff --git a/backend/.sqlx/query-8bd266705fc8272f3d8941922ad7d18161eb6f5ec1ba9f1b55feffe8b6518c67.json b/backend/.sqlx/query-8bd266705fc8272f3d8941922ad7d18161eb6f5ec1ba9f1b55feffe8b6518c67.json new file mode 100644 index 0000000000..fc86915946 --- /dev/null +++ b/backend/.sqlx/query-8bd266705fc8272f3d8941922ad7d18161eb6f5ec1ba9f1b55feffe8b6518c67.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE password SET disabled = $1 WHERE email = $2", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Bool", + "Text" + ] + }, + "nullable": [] + }, + "hash": "8bd266705fc8272f3d8941922ad7d18161eb6f5ec1ba9f1b55feffe8b6518c67" +} diff --git a/backend/.sqlx/query-92fb6afe3b7041b2954340094c08e702fc1577d3fa4ff1ff2f1e089971ff5e32.json b/backend/.sqlx/query-92fb6afe3b7041b2954340094c08e702fc1577d3fa4ff1ff2f1e089971ff5e32.json new file mode 100644 index 0000000000..49fd50d7e6 --- /dev/null +++ b/backend/.sqlx/query-92fb6afe3b7041b2954340094c08e702fc1577d3fa4ff1ff2f1e089971ff5e32.json @@ -0,0 +1,60 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT id, workspace_id, item_kind, item_path, deleted_by, deleted_at, expires_at\n FROM trashbin\n WHERE workspace_id = $1\n ORDER BY deleted_at DESC\n LIMIT $2 OFFSET $3", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id", + "type_info": "Int8" + }, + { + "ordinal": 1, + "name": "workspace_id", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "item_kind", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "item_path", + "type_info": "Text" + }, + { + "ordinal": 4, + "name": "deleted_by", + "type_info": "Varchar" + }, + { + "ordinal": 5, + "name": "deleted_at", + "type_info": "Timestamptz" + }, + { + "ordinal": 6, + "name": "expires_at", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Text", + "Int8", + "Int8" + ] + }, + "nullable": [ + false, + false, + false, + false, + false, + false, + false + ] + }, + "hash": "92fb6afe3b7041b2954340094c08e702fc1577d3fa4ff1ff2f1e089971ff5e32" +} diff --git a/backend/.sqlx/query-950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb.json b/backend/.sqlx/query-950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb.json deleted file mode 100644 index 9ec97dbc82..0000000000 --- a/backend/.sqlx/query-950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb.json +++ /dev/null @@ -1,46 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH RECURSIVE chain AS (\n SELECT\n j.id,\n j.parent_job,\n j.flow_step_id,\n 1 AS depth\n FROM v2_job j\n WHERE j.id = $1\n UNION ALL\n SELECT\n pj.id,\n pj.parent_job,\n pj.flow_step_id,\n c.depth + 1\n FROM chain c\n JOIN v2_job pj ON pj.id = c.parent_job\n WHERE c.parent_job IS NOT NULL\n )\n SELECT\n c.id,\n c.parent_job,\n c.flow_step_id,\n EXISTS(SELECT 1 FROM v2_job_queue q WHERE q.id = c.parent_job) AS \"parent_in_queue!\",\n EXISTS(\n SELECT 1 FROM v2_job sib\n WHERE sib.parent_job = c.parent_job\n AND sib.id != c.id\n AND sib.id IN (SELECT sq.id FROM v2_job_queue sq)\n ) AS \"has_other_active_siblings!\"\n FROM chain c\n WHERE c.depth >= 1\n ORDER BY c.depth ASC\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "id", - "type_info": "Uuid" - }, - { - "ordinal": 1, - "name": "parent_job", - "type_info": "Uuid" - }, - { - "ordinal": 2, - "name": "flow_step_id", - "type_info": "Varchar" - }, - { - "ordinal": 3, - "name": "parent_in_queue!", - "type_info": "Bool" - }, - { - "ordinal": 4, - "name": "has_other_active_siblings!", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Uuid" - ] - }, - "nullable": [ - null, - null, - null, - null, - null - ] - }, - "hash": "950f364c9fa3c680eea895558a559f29220c08e94e1822e3bcb5c6ed6aa7d2bb" -} diff --git a/backend/.sqlx/query-975099ff6b07718ea94bcb5f84a4414c59199964cee53ef2ee6b35a78cf0c49a.json b/backend/.sqlx/query-975099ff6b07718ea94bcb5f84a4414c59199964cee53ef2ee6b35a78cf0c49a.json new file mode 100644 index 0000000000..012316fd4a --- /dev/null +++ b/backend/.sqlx/query-975099ff6b07718ea94bcb5f84a4414c59199964cee53ef2ee6b35a78cf0c49a.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value FROM global_settings WHERE name = 'ai_config'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false + ] + }, + "hash": "975099ff6b07718ea94bcb5f84a4414c59199964cee53ef2ee6b35a78cf0c49a" +} diff --git a/backend/.sqlx/query-d1dcc7fc8a1e1bc4dad263ec5163a94fca9dd95cc3b26b33611eab9d2a261141.json b/backend/.sqlx/query-97cf826b271cf064182382c924188fee392ed9cff6ae446abc86170984304a25.json similarity index 69% rename from backend/.sqlx/query-d1dcc7fc8a1e1bc4dad263ec5163a94fca9dd95cc3b26b33611eab9d2a261141.json rename to backend/.sqlx/query-97cf826b271cf064182382c924188fee392ed9cff6ae446abc86170984304a25.json index 4a0592701c..644a9eb8f2 100644 --- a/backend/.sqlx/query-d1dcc7fc8a1e1bc4dad263ec5163a94fca9dd95cc3b26b33611eab9d2a261141.json +++ b/backend/.sqlx/query-97cf826b271cf064182382c924188fee392ed9cff6ae446abc86170984304a25.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT args as \"args: sqlx::types::Json>\"\n FROM v2_job\n WHERE id = $1", + "query": "SELECT args as \"args: sqlx::types::Json>\"\n FROM v2_job\n WHERE id = $1", "describe": { "columns": [ { @@ -18,5 +18,5 @@ true ] }, - "hash": "d1dcc7fc8a1e1bc4dad263ec5163a94fca9dd95cc3b26b33611eab9d2a261141" + "hash": "97cf826b271cf064182382c924188fee392ed9cff6ae446abc86170984304a25" } diff --git a/backend/.sqlx/query-98033aae3182bde22d5b2ff08ef6e8a4f8f3a9bf04238b33e9caf46836df73d9.json b/backend/.sqlx/query-98033aae3182bde22d5b2ff08ef6e8a4f8f3a9bf04238b33e9caf46836df73d9.json deleted file mode 100644 index 7dd6e9ac5d..0000000000 --- a/backend/.sqlx/query-98033aae3182bde22d5b2ff08ef6e8a4f8f3a9bf04238b33e9caf46836df73d9.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "98033aae3182bde22d5b2ff08ef6e8a4f8f3a9bf04238b33e9caf46836df73d9" -} diff --git a/backend/.sqlx/query-726e956cfcd3ac7c07abeecdf92cf0996efe7fa7b671ac2b3b000ead0ea307de.json b/backend/.sqlx/query-9a1483a81f5b086e0765d3d69483e29b09f66090e1f9d394564c16d921d2e66c.json similarity index 52% rename from backend/.sqlx/query-726e956cfcd3ac7c07abeecdf92cf0996efe7fa7b671ac2b3b000ead0ea307de.json rename to backend/.sqlx/query-9a1483a81f5b086e0765d3d69483e29b09f66090e1f9d394564c16d921d2e66c.json index 4baf932023..e751b8fc7c 100644 --- a/backend/.sqlx/query-726e956cfcd3ac7c07abeecdf92cf0996efe7fa7b671ac2b3b000ead0ea307de.json +++ b/backend/.sqlx/query-9a1483a81f5b086e0765d3d69483e29b09f66090e1f9d394564c16d921d2e66c.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by s.created_at DESC", + "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg, s.created_at as created_at\n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by s.created_at DESC", "describe": { "columns": [ { @@ -12,6 +12,11 @@ "ordinal": 1, "name": "deployment_msg", "type_info": "Text" + }, + { + "ordinal": 2, + "name": "created_at", + "type_info": "Timestamptz" } ], "parameters": { @@ -22,8 +27,9 @@ }, "nullable": [ false, - true + true, + false ] }, - "hash": "726e956cfcd3ac7c07abeecdf92cf0996efe7fa7b671ac2b3b000ead0ea307de" + "hash": "9a1483a81f5b086e0765d3d69483e29b09f66090e1f9d394564c16d921d2e66c" } diff --git a/backend/.sqlx/query-d5661c7557cf3a8dee7cf799cd364d21d38edb827d2c08b0ca7d72311b78d574.json b/backend/.sqlx/query-a32d7ba43745226fd65328475731526e0b20ea6eeafeb937eb01cdc2cdfcb859.json similarity index 68% rename from backend/.sqlx/query-d5661c7557cf3a8dee7cf799cd364d21d38edb827d2c08b0ca7d72311b78d574.json rename to backend/.sqlx/query-a32d7ba43745226fd65328475731526e0b20ea6eeafeb937eb01cdc2cdfcb859.json index 907b140fdd..e30ebbe98b 100644 --- a/backend/.sqlx/query-d5661c7557cf3a8dee7cf799cd364d21d38edb827d2c08b0ca7d72311b78d574.json +++ b/backend/.sqlx/query-a32d7ba43745226fd65328475731526e0b20ea6eeafeb937eb01cdc2cdfcb859.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false ORDER BY created_at DESC LIMIT 1", + "query": "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false AND archived = false ORDER BY created_at DESC LIMIT 1", "describe": { "columns": [ { @@ -19,5 +19,5 @@ false ] }, - "hash": "d5661c7557cf3a8dee7cf799cd364d21d38edb827d2c08b0ca7d72311b78d574" + "hash": "a32d7ba43745226fd65328475731526e0b20ea6eeafeb937eb01cdc2cdfcb859" } diff --git a/backend/.sqlx/query-a37c2c4d5656d4b44433de84c454046f7586e36b7bd6a4679d70c359d4aacfcf.json b/backend/.sqlx/query-a37c2c4d5656d4b44433de84c454046f7586e36b7bd6a4679d70c359d4aacfcf.json new file mode 100644 index 0000000000..5661c59faf --- /dev/null +++ b/backend/.sqlx/query-a37c2c4d5656d4b44433de84c454046f7586e36b7bd6a4679d70c359d4aacfcf.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT COUNT(*) FROM usr WHERE is_service_account = true", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "count", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null + ] + }, + "hash": "a37c2c4d5656d4b44433de84c454046f7586e36b7bd6a4679d70c359d4aacfcf" +} diff --git a/backend/.sqlx/query-a37cfc632dd37cf37c06743239b5ebc784e5da5ee25d47af187a75220d8fded7.json b/backend/.sqlx/query-a37cfc632dd37cf37c06743239b5ebc784e5da5ee25d47af187a75220d8fded7.json deleted file mode 100644 index c993120dae..0000000000 --- a/backend/.sqlx/query-a37cfc632dd37cf37c06743239b5ebc784e5da5ee25d47af187a75220d8fded7.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n UPDATE kafka_trigger\n SET\n kafka_resource_path = $1,\n group_id = $2,\n topics = $3,\n filters = $4,\n auto_offset_reset = $5,\n auto_commit = $6,\n script_path = $7,\n path = $8,\n is_flow = $9,\n edited_by = $10,\n permissioned_as = $11,\n edited_at = now(),\n server_id = NULL,\n error = NULL,\n error_handler_path = $14,\n error_handler_args = $15,\n retry = $16\n WHERE\n workspace_id = $12 AND path = $13\n ", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "VarcharArray", - "JsonbArray", - "Varchar", - "Bool", - "Varchar", - "Varchar", - "Bool", - "Varchar", - "Varchar", - "Text", - "Text", - "Varchar", - "Jsonb", - "Jsonb" - ] - }, - "nullable": [] - }, - "hash": "a37cfc632dd37cf37c06743239b5ebc784e5da5ee25d47af187a75220d8fded7" -} diff --git a/backend/.sqlx/query-60118de85463098220b1c74f667b6fedb0f3f0040844c3774145e8f1f4c023ce.json b/backend/.sqlx/query-a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10.json similarity index 82% rename from backend/.sqlx/query-60118de85463098220b1c74f667b6fedb0f3f0040844c3774145e8f1f4c023ce.json rename to backend/.sqlx/query-a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10.json index 6d2382f494..dab14d9f1d 100644 --- a/backend/.sqlx/query-60118de85463098220b1c74f667b6fedb0f3f0040844c3774145e8f1f4c023ce.json +++ b/backend/.sqlx/query-a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "WITH active_users AS (SELECT distinct username as email FROM (SELECT username, timestamp, operation FROM audit_partitioned UNION ALL SELECT username, timestamp, operation FROM audit) AS a WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n authors as (SELECT distinct email FROM usr WHERE usr.operator IS false)\n SELECT email, email NOT IN (SELECT email FROM authors) as operator_only, login_type::text, verified, super_admin, devops, name, company, username, first_time_user, role_source\n FROM password\n WHERE email IN (SELECT email FROM active_users)\n ORDER BY super_admin DESC, devops DESC\n LIMIT $1 OFFSET $2", + "query": "WITH active_users AS (SELECT distinct username as email FROM (SELECT username, timestamp, operation FROM audit_partitioned UNION ALL SELECT username, timestamp, operation FROM audit) AS a WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n authors as (SELECT distinct email FROM usr WHERE usr.operator IS false)\n SELECT email, email NOT IN (SELECT email FROM authors) as operator_only, login_type::text, verified, super_admin, devops, name, company, username, first_time_user, role_source, disabled\n FROM password\n WHERE email IN (SELECT email FROM active_users)\n ORDER BY super_admin DESC, devops DESC\n LIMIT $1 OFFSET $2", "describe": { "columns": [ { @@ -57,6 +57,11 @@ "ordinal": 10, "name": "role_source", "type_info": "Varchar" + }, + { + "ordinal": 11, + "name": "disabled", + "type_info": "Bool" } ], "parameters": { @@ -76,8 +81,9 @@ true, true, false, + false, false ] }, - "hash": "60118de85463098220b1c74f667b6fedb0f3f0040844c3774145e8f1f4c023ce" + "hash": "a5fd115e7be5129d623543bbfa7b5b31f0efc6d8ef73f691009c73f833dcee10" } diff --git a/backend/.sqlx/query-a860dd9722f608184c4b1ef5e609b20cd61f9967a2012fc1c8fe352ee7596358.json b/backend/.sqlx/query-a860dd9722f608184c4b1ef5e609b20cd61f9967a2012fc1c8fe352ee7596358.json new file mode 100644 index 0000000000..539c029265 --- /dev/null +++ b/backend/.sqlx/query-a860dd9722f608184c4b1ef5e609b20cd61f9967a2012fc1c8fe352ee7596358.json @@ -0,0 +1,15 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO pip_resolution_cache (hash, lockfile, expiration) VALUES ($1, $2, now() + ('7 days')::interval) ON CONFLICT (hash) DO UPDATE SET lockfile = EXCLUDED.lockfile, expiration = EXCLUDED.expiration", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Text" + ] + }, + "nullable": [] + }, + "hash": "a860dd9722f608184c4b1ef5e609b20cd61f9967a2012fc1c8fe352ee7596358" +} diff --git a/backend/.sqlx/query-a9c3461ca3053f699c957f61780d1e889ad53dc5bf1669c24c0666c290656c00.json b/backend/.sqlx/query-a9c3461ca3053f699c957f61780d1e889ad53dc5bf1669c24c0666c290656c00.json new file mode 100644 index 0000000000..cb6ed918c4 --- /dev/null +++ b/backend/.sqlx/query-a9c3461ca3053f699c957f61780d1e889ad53dc5bf1669c24c0666c290656c00.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT AVG(pg_column_size(result))::bigint as \"avg_size\"\n FROM (\n SELECT result FROM v2_job_completed\n WHERE completed_at > now() - interval '30 days'\n AND result IS NOT NULL\n ORDER BY completed_at DESC\n LIMIT $1\n ) sub", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "avg_size", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [ + null + ] + }, + "hash": "a9c3461ca3053f699c957f61780d1e889ad53dc5bf1669c24c0666c290656c00" +} diff --git a/backend/.sqlx/query-add01e9e31d64e88b84c9505fe3de553031e581b1bb173413a9a3e3eb0817b43.json b/backend/.sqlx/query-add01e9e31d64e88b84c9505fe3de553031e581b1bb173413a9a3e3eb0817b43.json new file mode 100644 index 0000000000..2c00759a63 --- /dev/null +++ b/backend/.sqlx/query-add01e9e31d64e88b84c9505fe3de553031e581b1bb173413a9a3e3eb0817b43.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr_to_group (workspace_id, usr, group_) VALUES ($1, $2, $3)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "add01e9e31d64e88b84c9505fe3de553031e581b1bb173413a9a3e3eb0817b43" +} diff --git a/backend/.sqlx/query-a115d8ea786907561afdbbc07d11dc715d80b00c0e79b61b0057a3ae3886a85e.json b/backend/.sqlx/query-b0775af41a9b54cce040bf37cae770e63dab12a1383a0855d105c061e4e4ca48.json similarity index 67% rename from backend/.sqlx/query-a115d8ea786907561afdbbc07d11dc715d80b00c0e79b61b0057a3ae3886a85e.json rename to backend/.sqlx/query-b0775af41a9b54cce040bf37cae770e63dab12a1383a0855d105c061e4e4ca48.json index 660c855622..4109c7deaf 100644 --- a/backend/.sqlx/query-a115d8ea786907561afdbbc07d11dc715d80b00c0e79b61b0057a3ae3886a85e.json +++ b/backend/.sqlx/query-b0775af41a9b54cce040bf37cae770e63dab12a1383a0855d105c061e4e4ca48.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n SELECT\n nt.external_id,\n nt.workspace_id,\n nt.service_name AS \"service_name!: ServiceName\",\n nt.script_path,\n nt.is_flow,\n nt.webhook_token_hash,\n nt.service_config,\n nt.error,\n nt.created_at,\n nt.updated_at\n FROM\n native_trigger nt\n WHERE\n nt.workspace_id = $1 AND\n nt.service_name = $2 AND\n ($5::text IS NULL OR nt.script_path = $5) AND\n ($6::bool IS NULL OR nt.is_flow = $6) AND\n (\n (nt.is_flow = false AND EXISTS (\n SELECT 1 FROM script s\n WHERE s.workspace_id = nt.workspace_id\n AND s.path = nt.script_path\n ))\n OR\n (nt.is_flow = true AND EXISTS (\n SELECT 1 FROM flow f\n WHERE f.workspace_id = nt.workspace_id\n AND f.path = nt.script_path\n ))\n )\n LIMIT $3\n OFFSET $4\n ", + "query": "\n SELECT\n nt.external_id,\n nt.workspace_id,\n nt.service_name AS \"service_name!: ServiceName\",\n nt.script_path,\n nt.is_flow,\n nt.webhook_token_hash,\n nt.service_config,\n nt.error,\n nt.created_at,\n nt.updated_at,\n nt.summary\n FROM\n native_trigger nt\n WHERE\n nt.workspace_id = $1 AND\n nt.service_name = $2 AND\n ($5::text IS NULL OR nt.script_path = $5) AND\n ($6::bool IS NULL OR nt.is_flow = $6) AND\n (\n (nt.is_flow = false AND EXISTS (\n SELECT 1 FROM script s\n WHERE s.workspace_id = nt.workspace_id\n AND s.path = nt.script_path\n ))\n OR\n (nt.is_flow = true AND EXISTS (\n SELECT 1 FROM flow f\n WHERE f.workspace_id = nt.workspace_id\n AND f.path = nt.script_path\n ))\n )\n LIMIT $3\n OFFSET $4\n ", "describe": { "columns": [ { @@ -62,6 +62,11 @@ "ordinal": 9, "name": "updated_at", "type_info": "Timestamptz" + }, + { + "ordinal": 10, + "name": "summary", + "type_info": "Varchar" } ], "parameters": { @@ -94,8 +99,9 @@ true, true, false, - false + false, + true ] }, - "hash": "a115d8ea786907561afdbbc07d11dc715d80b00c0e79b61b0057a3ae3886a85e" + "hash": "b0775af41a9b54cce040bf37cae770e63dab12a1383a0855d105c061e4e4ca48" } diff --git a/backend/.sqlx/query-b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c.json b/backend/.sqlx/query-b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c.json deleted file mode 100644 index 5fb12bed16..0000000000 --- a/backend/.sqlx/query-b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c.json +++ /dev/null @@ -1,16 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "UPDATE v2_job_status SET flow_status = (\n SELECT jsonb_set(\n flow_status,\n ARRAY['modules', (idx - 1)::text],\n $2::jsonb\n )\n FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n LIMIT 1\n ) WHERE id = $1 AND (\n SELECT COUNT(*) FROM jsonb_array_elements(flow_status->'modules')\n WITH ORDINALITY arr(elem, idx)\n WHERE elem->>'id' = $3\n ) > 0", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Uuid", - "Jsonb", - "Text" - ] - }, - "nullable": [] - }, - "hash": "b1979a8249557d29e9055fde06191688f3ed0efd3a43e81f4ea296255248092c" -} diff --git a/backend/.sqlx/query-b38bd869477a729279cac3ccd4825191fb49e17e5f7e7297c0c819f52b486f49.json b/backend/.sqlx/query-b38bd869477a729279cac3ccd4825191fb49e17e5f7e7297c0c819f52b486f49.json new file mode 100644 index 0000000000..955fee77ad --- /dev/null +++ b/backend/.sqlx/query-b38bd869477a729279cac3ccd4825191fb49e17e5f7e7297c0c819f52b486f49.json @@ -0,0 +1,18 @@ +{ + "db_name": "PostgreSQL", + "query": "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Bool", + "Bool", + "Jsonb", + "Text", + "Text" + ] + }, + "nullable": [] + }, + "hash": "b38bd869477a729279cac3ccd4825191fb49e17e5f7e7297c0c819f52b486f49" +} diff --git a/backend/.sqlx/query-b760be4a0a80853073a061f7c9ebc2d411294d57b07d54d15d178db3c6ee2a30.json b/backend/.sqlx/query-b760be4a0a80853073a061f7c9ebc2d411294d57b07d54d15d178db3c6ee2a30.json new file mode 100644 index 0000000000..a01296f654 --- /dev/null +++ b/backend/.sqlx/query-b760be4a0a80853073a061f7c9ebc2d411294d57b07d54d15d178db3c6ee2a30.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT MIN(completed_at) as oldest, COUNT(*) as total FROM v2_job_completed", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "oldest", + "type_info": "Timestamptz" + }, + { + "ordinal": 1, + "name": "total", + "type_info": "Int8" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "b760be4a0a80853073a061f7c9ebc2d411294d57b07d54d15d178db3c6ee2a30" +} diff --git a/backend/.sqlx/query-bae31609123da68d16bea8e0f1c4624403b6f97e13f13f056501fe2f4efb0f06.json b/backend/.sqlx/query-bae31609123da68d16bea8e0f1c4624403b6f97e13f13f056501fe2f4efb0f06.json new file mode 100644 index 0000000000..bc0cc1a797 --- /dev/null +++ b/backend/.sqlx/query-bae31609123da68d16bea8e0f1c4624403b6f97e13f13f056501fe2f4efb0f06.json @@ -0,0 +1,14 @@ +{ + "db_name": "PostgreSQL", + "query": "DELETE FROM trashbin WHERE workspace_id = $1", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [] + }, + "hash": "bae31609123da68d16bea8e0f1c4624403b6f97e13f13f056501fe2f4efb0f06" +} diff --git a/backend/.sqlx/query-40a8bf6a5a42c275d73221bc5f386f2e18cb911352551d0a34bf1933e558674e.json b/backend/.sqlx/query-bf224f6441c36187f1402f9f01bfe15bb9edfa1dc9052f8a829e486b7334d708.json similarity index 66% rename from backend/.sqlx/query-40a8bf6a5a42c275d73221bc5f386f2e18cb911352551d0a34bf1933e558674e.json rename to backend/.sqlx/query-bf224f6441c36187f1402f9f01bfe15bb9edfa1dc9052f8a829e486b7334d708.json index 706fa0c9ea..b40a643d50 100644 --- a/backend/.sqlx/query-40a8bf6a5a42c275d73221bc5f386f2e18cb911352551d0a34bf1933e558674e.json +++ b/backend/.sqlx/query-bf224f6441c36187f1402f9f01bfe15bb9edfa1dc9052f8a829e486b7334d708.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "\n UPDATE native_trigger\n SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, error = NULL, updated_at = NOW()\n WHERE\n workspace_id = $5\n AND service_name = $6\n AND external_id = $7\n ", + "query": "\n UPDATE native_trigger\n SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, summary = $8, error = NULL, updated_at = NOW()\n WHERE\n workspace_id = $5\n AND service_name = $6\n AND external_id = $7\n ", "describe": { "columns": [], "parameters": { @@ -21,10 +21,11 @@ } } }, - "Text" + "Text", + "Varchar" ] }, "nullable": [] }, - "hash": "40a8bf6a5a42c275d73221bc5f386f2e18cb911352551d0a34bf1933e558674e" + "hash": "bf224f6441c36187f1402f9f01bfe15bb9edfa1dc9052f8a829e486b7334d708" } diff --git a/backend/.sqlx/query-c1fd495abb4353b46361ec94fd4ae8d224457171b1b73fe145d28e67f1fe03af.json b/backend/.sqlx/query-c1fd495abb4353b46361ec94fd4ae8d224457171b1b73fe145d28e67f1fe03af.json new file mode 100644 index 0000000000..d25ae0613a --- /dev/null +++ b/backend/.sqlx/query-c1fd495abb4353b46361ec94fd4ae8d224457171b1b73fe145d28e67f1fe03af.json @@ -0,0 +1,23 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2)", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "exists", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text", + "Text" + ] + }, + "nullable": [ + null + ] + }, + "hash": "c1fd495abb4353b46361ec94fd4ae8d224457171b1b73fe145d28e67f1fe03af" +} diff --git a/backend/.sqlx/query-c2347460b73ae9d3167031c263032e97ebefb46be9e58bd3da9067748075311b.json b/backend/.sqlx/query-c2347460b73ae9d3167031c263032e97ebefb46be9e58bd3da9067748075311b.json deleted file mode 100644 index 7d7842d7f4..0000000000 --- a/backend/.sqlx/query-c2347460b73ae9d3167031c263032e97ebefb46be9e58bd3da9067748075311b.json +++ /dev/null @@ -1,35 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n WITH dk AS (\n INSERT INTO debounce_key (job_id, key)\n VALUES ($1, $2)\n ON CONFLICT (key)\n DO UPDATE SET\n previous_job_id = debounce_key.job_id,\n job_id = EXCLUDED.job_id,\n debounced_times = debounce_key.debounced_times + 1\n RETURNING\n debounced_times,\n first_started_at,\n previous_job_id AS job_id_to_debounce\n ), _batch AS (\n INSERT INTO v2_job_debounce_batch (id, debounce_batch)\n SELECT\n $1,\n COALESCE(\n (SELECT debounce_batch FROM v2_job_debounce_batch WHERE id = dk.job_id_to_debounce LIMIT 1),\n nextval('debounce_batch_seq')\n )\n FROM dk\n )\n SELECT debounced_times, first_started_at, job_id_to_debounce FROM dk\n ", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "debounced_times", - "type_info": "Int4" - }, - { - "ordinal": 1, - "name": "first_started_at", - "type_info": "Timestamptz" - }, - { - "ordinal": 2, - "name": "job_id_to_debounce", - "type_info": "Uuid" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Varchar" - ] - }, - "nullable": [ - false, - false, - true - ] - }, - "hash": "c2347460b73ae9d3167031c263032e97ebefb46be9e58bd3da9067748075311b" -} diff --git a/backend/.sqlx/query-cf2a6ad6471a40b6298775cda9300aeecdd75503bed59d80cd62091d1642d1ec.json b/backend/.sqlx/query-c73e98e5a937f44724a96ee1b74d31fa71a7be3b8ba3dec9f59f54a6c4030462.json similarity index 51% rename from backend/.sqlx/query-cf2a6ad6471a40b6298775cda9300aeecdd75503bed59d80cd62091d1642d1ec.json rename to backend/.sqlx/query-c73e98e5a937f44724a96ee1b74d31fa71a7be3b8ba3dec9f59f54a6c4030462.json index 4e17af3ce2..06b2c50058 100644 --- a/backend/.sqlx/query-cf2a6ad6471a40b6298775cda9300aeecdd75503bed59d80cd62091d1642d1ec.json +++ b/backend/.sqlx/query-c73e98e5a937f44724a96ee1b74d31fa71a7be3b8ba3dec9f59f54a6c4030462.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg \n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by s.created_at DESC LIMIT 1", + "query": "SELECT s.hash as hash, dm.deployment_msg as deployment_msg, s.created_at as created_at\n FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash\n WHERE s.workspace_id = $1 AND s.path = $2\n ORDER by s.created_at DESC LIMIT 1", "describe": { "columns": [ { @@ -12,6 +12,11 @@ "ordinal": 1, "name": "deployment_msg", "type_info": "Text" + }, + { + "ordinal": 2, + "name": "created_at", + "type_info": "Timestamptz" } ], "parameters": { @@ -22,8 +27,9 @@ }, "nullable": [ false, - true + true, + false ] }, - "hash": "cf2a6ad6471a40b6298775cda9300aeecdd75503bed59d80cd62091d1642d1ec" + "hash": "c73e98e5a937f44724a96ee1b74d31fa71a7be3b8ba3dec9f59f54a6c4030462" } diff --git a/backend/.sqlx/query-c7aed7fe3b6774477d403bc3e7fcbce7cdbdd1feb553718cbde60bb8ccff4733.json b/backend/.sqlx/query-c7aed7fe3b6774477d403bc3e7fcbce7cdbdd1feb553718cbde60bb8ccff4733.json deleted file mode 100644 index 772b86a11c..0000000000 --- a/backend/.sqlx/query-c7aed7fe3b6774477d403bc3e7fcbce7cdbdd1feb553718cbde60bb8ccff4733.json +++ /dev/null @@ -1,29 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "\n UPDATE\n websocket_trigger\n SET\n url = $1,\n script_path = $2,\n path = $3,\n is_flow = $4,\n filters = $5,\n initial_messages = $6,\n url_runnable_args = $7,\n edited_by = $8,\n permissioned_as = $9,\n can_return_message = $10,\n can_return_error_result = $11,\n edited_at = now(),\n server_id = NULL,\n error = NULL,\n error_handler_path = $14,\n error_handler_args = $15,\n retry = $16\n WHERE\n workspace_id = $12 AND path = $13\n ", - "describe": { - "columns": [], - "parameters": { - "Left": [ - "Varchar", - "Varchar", - "Varchar", - "Bool", - "JsonbArray", - "JsonbArray", - "Jsonb", - "Varchar", - "Varchar", - "Bool", - "Bool", - "Text", - "Text", - "Varchar", - "Jsonb", - "Jsonb" - ] - }, - "nullable": [] - }, - "hash": "c7aed7fe3b6774477d403bc3e7fcbce7cdbdd1feb553718cbde60bb8ccff4733" -} diff --git a/backend/.sqlx/query-cb3862634f18160207ee2621ddfca43f00456a27fda32583846497116f92f96c.json b/backend/.sqlx/query-cb3862634f18160207ee2621ddfca43f00456a27fda32583846497116f92f96c.json deleted file mode 100644 index 8ccf623719..0000000000 --- a/backend/.sqlx/query-cb3862634f18160207ee2621ddfca43f00456a27fda32583846497116f92f96c.json +++ /dev/null @@ -1,38 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "WITH active_users as (SELECT distinct username as email FROM audit WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')),\n active_authors as (SELECT distinct email FROM usr WHERE usr.operator IS false AND email IN (SELECT email FROM active_users)),\n active_authors_agg as (SELECT array_agg(email) as authors FROM active_authors),\n active_ops_agg as (SELECT array_agg(email) as operators from active_users WHERE email NOT IN (SELECT email FROM active_authors))\n SELECT active_authors_agg.authors, active_ops_agg.operators, array_length(active_authors_agg.authors, 1) as author_count, array_length(active_ops_agg.operators, 1) as operator_count FROM active_authors_agg, active_ops_agg", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "authors", - "type_info": "VarcharArray" - }, - { - "ordinal": 1, - "name": "operators", - "type_info": "VarcharArray" - }, - { - "ordinal": 2, - "name": "author_count", - "type_info": "Int4" - }, - { - "ordinal": 3, - "name": "operator_count", - "type_info": "Int4" - } - ], - "parameters": { - "Left": [] - }, - "nullable": [ - null, - null, - null, - null - ] - }, - "hash": "cb3862634f18160207ee2621ddfca43f00456a27fda32583846497116f92f96c" -} diff --git a/backend/.sqlx/query-ccc49a2a6e11f874825365de758bdc0e1934d67d3f2b14047d434b77d370af21.json b/backend/.sqlx/query-ccc49a2a6e11f874825365de758bdc0e1934d67d3f2b14047d434b77d370af21.json new file mode 100644 index 0000000000..a9348dfa8b --- /dev/null +++ b/backend/.sqlx/query-ccc49a2a6e11f874825365de758bdc0e1934d67d3f2b14047d434b77d370af21.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT super_admin FROM password WHERE email = $1 AND disabled = false", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "super_admin", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "ccc49a2a6e11f874825365de758bdc0e1934d67d3f2b14047d434b77d370af21" +} diff --git a/backend/.sqlx/query-d43a4ff78e48580815fb912c98639a08d45596a9f11a2dcf5b1e0d135844ecda.json b/backend/.sqlx/query-d43a4ff78e48580815fb912c98639a08d45596a9f11a2dcf5b1e0d135844ecda.json new file mode 100644 index 0000000000..1cfd1e268c --- /dev/null +++ b/backend/.sqlx/query-d43a4ff78e48580815fb912c98639a08d45596a9f11a2dcf5b1e0d135844ecda.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT value FROM global_settings WHERE name = 'plain_emails_telemetry'", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "value", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + false + ] + }, + "hash": "d43a4ff78e48580815fb912c98639a08d45596a9f11a2dcf5b1e0d135844ecda" +} diff --git a/backend/.sqlx/query-638d3c2ba1198dce5b5b0e47df59a92ff8011e19fbefcc3960d6f0fe167e55b6.json b/backend/.sqlx/query-daa1a6bf3d4a1001da88301932a7ac9019767074158e0c027988e5b0d51a3656.json similarity index 59% rename from backend/.sqlx/query-638d3c2ba1198dce5b5b0e47df59a92ff8011e19fbefcc3960d6f0fe167e55b6.json rename to backend/.sqlx/query-daa1a6bf3d4a1001da88301932a7ac9019767074158e0c027988e5b0d51a3656.json index d37bc73dad..5b18e66ed4 100644 --- a/backend/.sqlx/query-638d3c2ba1198dce5b5b0e47df59a92ff8011e19fbefcc3960d6f0fe167e55b6.json +++ b/backend/.sqlx/query-daa1a6bf3d4a1001da88301932a7ac9019767074158e0c027988e5b0d51a3656.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO password (email, login_type, verified, username, name) VALUES ($1, 'saml', true, $2, $3) ON CONFLICT DO NOTHING", + "query": "INSERT INTO password (email, login_type, verified, username, name) VALUES ($1, 'saml', true, $2, $3) ON CONFLICT (email) DO UPDATE SET disabled = false", "describe": { "columns": [], "parameters": { @@ -12,5 +12,5 @@ }, "nullable": [] }, - "hash": "638d3c2ba1198dce5b5b0e47df59a92ff8011e19fbefcc3960d6f0fe167e55b6" + "hash": "daa1a6bf3d4a1001da88301932a7ac9019767074158e0c027988e5b0d51a3656" } diff --git a/backend/.sqlx/query-dbc5924bca3aa0b32e296b73f8a967bed68332caf526216597f10ffa5fa951c7.json b/backend/.sqlx/query-dbc5924bca3aa0b32e296b73f8a967bed68332caf526216597f10ffa5fa951c7.json new file mode 100644 index 0000000000..33eb1fbf7e --- /dev/null +++ b/backend/.sqlx/query-dbc5924bca3aa0b32e296b73f8a967bed68332caf526216597f10ffa5fa951c7.json @@ -0,0 +1,46 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT\n c.id as \"id!\",\n c.workspace_id as \"workspace_id!\",\n j.runnable_path as \"runnable_path\",\n pg_column_size(c.result) as \"result_size_bytes!\",\n c.completed_at as \"completed_at!\"\n FROM (\n SELECT id, workspace_id, result, completed_at\n FROM v2_job_completed\n WHERE completed_at > now() - interval '30 days'\n AND result IS NOT NULL\n ORDER BY completed_at DESC\n LIMIT $1\n ) c\n LEFT JOIN v2_job j ON j.id = c.id\n WHERE pg_column_size(c.result) > 1024\n ORDER BY pg_column_size(c.result) DESC\n LIMIT 10", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "id!", + "type_info": "Uuid" + }, + { + "ordinal": 1, + "name": "workspace_id!", + "type_info": "Varchar" + }, + { + "ordinal": 2, + "name": "runnable_path", + "type_info": "Varchar" + }, + { + "ordinal": 3, + "name": "result_size_bytes!", + "type_info": "Int4" + }, + { + "ordinal": 4, + "name": "completed_at!", + "type_info": "Timestamptz" + } + ], + "parameters": { + "Left": [ + "Int8" + ] + }, + "nullable": [ + false, + false, + true, + null, + false + ] + }, + "hash": "dbc5924bca3aa0b32e296b73f8a967bed68332caf526216597f10ffa5fa951c7" +} diff --git a/backend/.sqlx/query-e3d4f89ce36337af15d237b543eaca47771b480ff194884f9c947dcaf71d6cf9.json b/backend/.sqlx/query-e3d4f89ce36337af15d237b543eaca47771b480ff194884f9c947dcaf71d6cf9.json new file mode 100644 index 0000000000..8ff6f2e89c --- /dev/null +++ b/backend/.sqlx/query-e3d4f89ce36337af15d237b543eaca47771b480ff194884f9c947dcaf71d6cf9.json @@ -0,0 +1,30 @@ +{ + "db_name": "PostgreSQL", + "query": "\n UPDATE\n websocket_trigger\n SET\n url = $1,\n script_path = $2,\n path = $3,\n is_flow = $4,\n filters = $5,\n filter_logic = $6,\n initial_messages = $7,\n url_runnable_args = $8,\n edited_by = $9,\n permissioned_as = $10,\n can_return_message = $11,\n can_return_error_result = $12,\n edited_at = now(),\n server_id = NULL,\n error = NULL,\n error_handler_path = $15,\n error_handler_args = $16,\n retry = $17\n WHERE\n workspace_id = $13 AND path = $14\n ", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Bool", + "JsonbArray", + "Varchar", + "JsonbArray", + "Jsonb", + "Varchar", + "Varchar", + "Bool", + "Bool", + "Text", + "Text", + "Varchar", + "Jsonb", + "Jsonb" + ] + }, + "nullable": [] + }, + "hash": "e3d4f89ce36337af15d237b543eaca47771b480ff194884f9c947dcaf71d6cf9" +} diff --git a/backend/.sqlx/query-e58ef252b0d2b81e9cd76f394a396abefd791906ada29dd5a7a9148157635ca5.json b/backend/.sqlx/query-e58ef252b0d2b81e9cd76f394a396abefd791906ada29dd5a7a9148157635ca5.json new file mode 100644 index 0000000000..855c628c1f --- /dev/null +++ b/backend/.sqlx/query-e58ef252b0d2b81e9cd76f394a396abefd791906ada29dd5a7a9148157635ca5.json @@ -0,0 +1,26 @@ +{ + "db_name": "PostgreSQL", + "query": "\n SELECT auto_invite->'instance_groups_roles' as instance_groups_roles,\n auto_invite->'instance_groups' as instance_groups_json\n FROM workspace_settings WHERE workspace_id = 'ws-multi-group'\n ", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "instance_groups_roles", + "type_info": "Jsonb" + }, + { + "ordinal": 1, + "name": "instance_groups_json", + "type_info": "Jsonb" + } + ], + "parameters": { + "Left": [] + }, + "nullable": [ + null, + null + ] + }, + "hash": "e58ef252b0d2b81e9cd76f394a396abefd791906ada29dd5a7a9148157635ca5" +} diff --git a/backend/.sqlx/query-e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2.json b/backend/.sqlx/query-e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2.json index cdeb30f672..7be961c050 100644 --- a/backend/.sqlx/query-e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2.json +++ b/backend/.sqlx/query-e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2.json @@ -47,6 +47,11 @@ "ordinal": 8, "name": "added_via", "type_info": "Jsonb" + }, + { + "ordinal": 9, + "name": "is_service_account", + "type_info": "Bool" } ], "parameters": { @@ -63,7 +68,8 @@ false, false, true, - true + true, + false ] }, "hash": "e5fb3531f8bc7ef1f7484524f8c3bc9c48f71a44827ba0d01ac5588dc31082a2" diff --git a/backend/.sqlx/query-ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2.json b/backend/.sqlx/query-ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2.json deleted file mode 100644 index c43a2bcd30..0000000000 --- a/backend/.sqlx/query-ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2.json +++ /dev/null @@ -1,23 +0,0 @@ -{ - "db_name": "PostgreSQL", - "query": "SELECT EXISTS(\n SELECT 1 FROM v2_job\n WHERE parent_job = $1 AND id != $2\n AND id IN (SELECT id FROM v2_job_queue)\n ) as has", - "describe": { - "columns": [ - { - "ordinal": 0, - "name": "has", - "type_info": "Bool" - } - ], - "parameters": { - "Left": [ - "Uuid", - "Uuid" - ] - }, - "nullable": [ - null - ] - }, - "hash": "ecf67b08d327c351909b7ba80218903bec93ef79a71053c00227e17c6f0415a2" -} diff --git a/backend/.sqlx/query-65c59e224e460351c2f88261f8b1b1e7ce2bb160270b59c0f359b7952453b2b9.json b/backend/.sqlx/query-f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493.json similarity index 85% rename from backend/.sqlx/query-65c59e224e460351c2f88261f8b1b1e7ce2bb160270b59c0f359b7952453b2b9.json rename to backend/.sqlx/query-f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493.json index 2ccf7bfdc8..c1d113ee27 100644 --- a/backend/.sqlx/query-65c59e224e460351c2f88261f8b1b1e7ce2bb160270b59c0f359b7952453b2b9.json +++ b/backend/.sqlx/query-f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source FROM password WHERE email = $1", + "query": "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password WHERE email = $1", "describe": { "columns": [ { @@ -57,6 +57,11 @@ "ordinal": 10, "name": "role_source", "type_info": "Varchar" + }, + { + "ordinal": 11, + "name": "disabled", + "type_info": "Bool" } ], "parameters": { @@ -75,8 +80,9 @@ true, null, false, + false, false ] }, - "hash": "65c59e224e460351c2f88261f8b1b1e7ce2bb160270b59c0f359b7952453b2b9" + "hash": "f0c9c54740cc1c0c2a6fa4e79d4d504b7b5cb7a39538ab9abeb44f781c711493" } diff --git a/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json b/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json new file mode 100644 index 0000000000..a0d7f223d0 --- /dev/null +++ b/backend/.sqlx/query-f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6.json @@ -0,0 +1,104 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT DISTINCT ON (path) path, language AS \"language: ScriptLang\", content FROM script\n WHERE workspace_id = $1\n AND archived = false\n AND dedicated_worker = true\n AND language = ANY($2::SCRIPT_LANG[])\n ORDER BY path, created_at DESC", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "path", + "type_info": "Varchar" + }, + { + "ordinal": 1, + "name": "language: ScriptLang", + "type_info": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby" + ] + } + } + } + }, + { + "ordinal": 2, + "name": "content", + "type_info": "Text" + } + ], + "parameters": { + "Left": [ + "Text", + { + "Custom": { + "name": "script_lang[]", + "kind": { + "Array": { + "Custom": { + "name": "script_lang", + "kind": { + "Enum": [ + "python3", + "deno", + "go", + "bash", + "postgresql", + "nativets", + "bun", + "mysql", + "bigquery", + "snowflake", + "graphql", + "powershell", + "mssql", + "php", + "bunnative", + "rust", + "ansible", + "csharp", + "oracledb", + "nu", + "java", + "duckdb", + "ruby" + ] + } + } + } + } + } + } + ] + }, + "nullable": [ + false, + false, + false + ] + }, + "hash": "f2fa27ed5020aa9c085176b25466be1bceb79c82e7b5542f17b23b6d70cc02d6" +} diff --git a/backend/.sqlx/query-f4ad2cf2438c2ae31e388517d09a2c1a2f63ab88cdbc79ffad96c6f9ffb5764b.json b/backend/.sqlx/query-f4ad2cf2438c2ae31e388517d09a2c1a2f63ab88cdbc79ffad96c6f9ffb5764b.json new file mode 100644 index 0000000000..24d9cd9517 --- /dev/null +++ b/backend/.sqlx/query-f4ad2cf2438c2ae31e388517d09a2c1a2f63ab88cdbc79ffad96c6f9ffb5764b.json @@ -0,0 +1,20 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO token\n (token_hash, token_prefix, token, email, label, expiration, super_admin, owner)\n VALUES ($1, $2, $3, $4, $5, $6, false, $7)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Varchar", + "Timestamptz", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f4ad2cf2438c2ae31e388517d09a2c1a2f63ab88cdbc79ffad96c6f9ffb5764b" +} diff --git a/backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json b/backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json new file mode 100644 index 0000000000..117a8bdc1b --- /dev/null +++ b/backend/.sqlx/query-f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853.json @@ -0,0 +1,16 @@ +{ + "db_name": "PostgreSQL", + "query": "INSERT INTO usr\n (workspace_id, email, username, is_admin, operator, is_service_account)\n VALUES ($1, $2, $3, false, true, true)", + "describe": { + "columns": [], + "parameters": { + "Left": [ + "Varchar", + "Varchar", + "Varchar" + ] + }, + "nullable": [] + }, + "hash": "f8654d5f50a80d862edbf57355502a9bd039d16f7dfb11e22d16ff9090456853" +} diff --git a/backend/.sqlx/query-fc6c6310ae8ac5eb351d7e2af1678447d0aa3d143e94e49924ff7ac8b7abf924.json b/backend/.sqlx/query-fc6c6310ae8ac5eb351d7e2af1678447d0aa3d143e94e49924ff7ac8b7abf924.json new file mode 100644 index 0000000000..c85c557a90 --- /dev/null +++ b/backend/.sqlx/query-fc6c6310ae8ac5eb351d7e2af1678447d0aa3d143e94e49924ff7ac8b7abf924.json @@ -0,0 +1,22 @@ +{ + "db_name": "PostgreSQL", + "query": "SELECT disabled FROM password WHERE email = $1", + "describe": { + "columns": [ + { + "ordinal": 0, + "name": "disabled", + "type_info": "Bool" + } + ], + "parameters": { + "Left": [ + "Text" + ] + }, + "nullable": [ + false + ] + }, + "hash": "fc6c6310ae8ac5eb351d7e2af1678447d0aa3d143e94e49924ff7ac8b7abf924" +} diff --git a/backend/.sqlx/query-d05f20431cd08f737bfbf904efedfdf104e3d77b0725c5355305d19f67359e90.json b/backend/.sqlx/query-fd4c5391107af34a3bf9b83b0c3f7d5ee9490240a627b20a1037444845e39c5f.json similarity index 50% rename from backend/.sqlx/query-d05f20431cd08f737bfbf904efedfdf104e3d77b0725c5355305d19f67359e90.json rename to backend/.sqlx/query-fd4c5391107af34a3bf9b83b0c3f7d5ee9490240a627b20a1037444845e39c5f.json index e778c17bf6..50cf586c79 100644 --- a/backend/.sqlx/query-d05f20431cd08f737bfbf904efedfdf104e3d77b0725c5355305d19f67359e90.json +++ b/backend/.sqlx/query-fd4c5391107af34a3bf9b83b0c3f7d5ee9490240a627b20a1037444845e39c5f.json @@ -1,6 +1,6 @@ { "db_name": "PostgreSQL", - "query": "INSERT INTO token\n (token_hash, token_prefix, token, label, super_admin, email)\n VALUES ($1, $2, $3, $4, $5, $6)", + "query": "INSERT INTO token\n (token_hash, token_prefix, token, label, super_admin, email)\n VALUES ($1, $2, $3, $4, $5, $6)", "describe": { "columns": [], "parameters": { @@ -15,5 +15,5 @@ }, "nullable": [] }, - "hash": "d05f20431cd08f737bfbf904efedfdf104e3d77b0725c5355305d19f67359e90" + "hash": "fd4c5391107af34a3bf9b83b0c3f7d5ee9490240a627b20a1037444845e39c5f" } diff --git a/backend/Cargo.lock b/backend/Cargo.lock index e40478cc4b..8e7df014a2 100644 --- a/backend/Cargo.lock +++ b/backend/Cargo.lock @@ -39,7 +39,7 @@ version = "0.5.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "d122413f284cf2d62fb1b7db97e02edb8cda96d769b16e443a4f6195e35662b0" dependencies = [ - "crypto-common", + "crypto-common 0.1.7", "generic-array", ] @@ -65,7 +65,7 @@ checksum = "9e8b47f52ea9bae42228d07ec09eb676433d7c4ed1ebdf0f1d1c29ed446f1ab8" dependencies = [ "cfg-if", "cipher 0.3.0", - "cpufeatures", + "cpufeatures 0.2.17", "opaque-debug", ] @@ -77,7 +77,7 @@ checksum = "ac1f845298e95f983ff1944b728ae08b8cebab80d684f0a832ed0fc74dfa27e2" dependencies = [ "cfg-if", "cipher 0.4.4", - "cpufeatures", + "cpufeatures 0.2.17", ] [[package]] @@ -234,9 +234,9 @@ dependencies = [ [[package]] name = "arc-swap" -version = "1.8.2" +version = "1.9.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f9f3647c145568cec02c42054e07bdf9a5a698e15b466fb2341bfc393cd24aa5" +checksum = "a07d1f37ff60921c83bdfc7407723bdefe89b44b98a9b772f225c8f9d67141a6" dependencies = [ "rustversion", ] @@ -249,7 +249,7 @@ checksum = "3c3610892ee6e0cbce8ae2700349fcf8f98adb0dbfbee85aec3c9179d29cc072" dependencies = [ "base64ct", "blake2", - "cpufeatures", + "cpufeatures 0.2.17", "password-hash", ] @@ -405,7 +405,7 @@ dependencies = [ "arrow-data", "arrow-schema", "flatbuffers", - "lz4_flex", + "lz4_flex 0.11.6", ] [[package]] @@ -421,7 +421,7 @@ dependencies = [ "arrow-schema", "chrono", "half", - "indexmap 2.11.1", + "indexmap 2.12.0", "lexical-core", "memchr", "num", @@ -602,7 +602,7 @@ dependencies = [ "futures-core", "libc", "portable-atomic", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "tokio", "tokio-stream", "xattr", @@ -870,9 +870,9 @@ dependencies = [ [[package]] name = "aws-lc-sys" -version = "0.39.0" +version = "0.39.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1fa7e52a4c5c547c741610a2c6f123f3881e409b714cd27e6798ef020c514f0a" +checksum = "83a25cf98105baa966497416dbd42565ce3a8cf8dbfd59803ec9ad46f3126399" dependencies = [ "cc", "cmake", @@ -1113,7 +1113,7 @@ dependencies = [ "bytes", "form_urlencoded", "hex", - "hmac", + "hmac 0.12.1", "http 0.2.12", "http 1.4.0", "percent-encoding", @@ -1363,32 +1363,23 @@ checksum = "edca88bc138befd0323b20752846e6587272d3b03b0343c8ea28a6f819e6e71f" dependencies = [ "async-trait", "axum-core 0.4.5", - "axum-macros", "bytes", "futures-util", "http 1.4.0", "http-body 1.0.1", "http-body-util", - "hyper 1.8.1", - "hyper-util", "itoa", "matchit 0.7.3", "memchr", "mime", - "multer", "percent-encoding", "pin-project-lite", "rustversion", "serde", - "serde_json", - "serde_path_to_error", - "serde_urlencoded", "sync_wrapper", - "tokio", "tower 0.5.3", "tower-layer", "tower-service", - "tracing", ] [[package]] @@ -1398,6 +1389,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "021e862c184ae977658b36c4500f7feac3221ca5da43e3f25bd04ab6c79a29b5" dependencies = [ "axum-core 0.5.6", + "axum-macros", "bytes", "form_urlencoded", "futures-util", @@ -1410,6 +1402,7 @@ dependencies = [ "matchit 0.8.4", "memchr", "mime", + "multer", "percent-encoding", "pin-project-lite", "rustversion", @@ -1443,7 +1436,6 @@ dependencies = [ "sync_wrapper", "tower-layer", "tower-service", - "tracing", ] [[package]] @@ -1467,9 +1459,9 @@ dependencies = [ [[package]] name = "axum-macros" -version = "0.4.2" +version = "0.5.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "57d123550fa8d071b7255cb0cc04dc302baa6c8c4a79f55701552684d8399bce" +checksum = "604fde5e028fea851ce1d8570bbdc034bec850d157f7569d10f347d06808c05c" dependencies = [ "proc-macro2", "quote", @@ -1594,29 +1586,6 @@ dependencies = [ "serde", ] -[[package]] -name = "bindgen" -version = "0.69.5" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "271383c67ccabffb7381723dea0672a673f292304fcb45c01cc648c7a8d58088" -dependencies = [ - "bitflags 2.9.4", - "cexpr", - "clang-sys", - "itertools 0.12.1", - "lazy_static", - "lazycell", - "log", - "prettyplease", - "proc-macro2", - "quote", - "regex", - "rustc-hash 1.1.0", - "shlex", - "syn 2.0.117", - "which 4.4.2", -] - [[package]] name = "bindgen" version = "0.70.1" @@ -1652,7 +1621,7 @@ dependencies = [ "proc-macro2", "quote", "regex", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "shlex", "syn 2.0.117", ] @@ -1667,10 +1636,12 @@ dependencies = [ "cexpr", "clang-sys", "itertools 0.13.0", + "log", + "prettyplease", "proc-macro2", "quote", "regex", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "shlex", "syn 2.0.117", ] @@ -1761,7 +1732,7 @@ dependencies = [ "cc", "cfg-if", "constant_time_eq 0.4.2", - "cpufeatures", + "cpufeatures 0.2.17", ] [[package]] @@ -1789,6 +1760,15 @@ dependencies = [ "generic-array", ] +[[package]] +name = "block-buffer" +version = "0.12.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "cdd35008169921d80bc60d3d0ab416eecb028c4cd653352907921d95084790be" +dependencies = [ + "hybrid-array", +] + [[package]] name = "block-modes" version = "0.8.1" @@ -2124,7 +2104,7 @@ dependencies = [ "num-traits", "num_cpus", "rand 0.9.0", - "rand_distr 0.5.1", + "rand_distr", "rayon", "safetensors", "thiserror 2.0.18", @@ -2209,9 +2189,9 @@ dependencies = [ [[package]] name = "cc" -version = "1.2.57" +version = "1.2.58" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7a0dd1ca384932ff3641c8718a02769f1698e7563dc6974ffd03346116310423" +checksum = "e1e928d4b69e3077709075a938a05ffbedfa53a84c8f766efbf8220bb1ff60e1" dependencies = [ "find-msvc-tools", "jobserver", @@ -2258,6 +2238,17 @@ version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724" +[[package]] +name = "chacha20" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "rand_core 0.10.0", +] + [[package]] name = "chrono" version = "0.4.44" @@ -2307,7 +2298,7 @@ version = "0.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad" dependencies = [ - "crypto-common", + "crypto-common 0.1.7", "inout", ] @@ -2373,13 +2364,19 @@ dependencies = [ [[package]] name = "cmake" -version = "0.1.57" +version = "0.1.58" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "75443c44cd6b379beb8c5b45d85d0773baf31cce901fe7bb252f4eff3008ef7d" +checksum = "c0f78a02292a74a88ac736019ab962ece0bc380e3f977bf72e376c5d78ff0678" dependencies = [ "cc", ] +[[package]] +name = "cmov" +version = "0.5.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "de0758edba32d61d1fd9f4d69491b47604b91ee2f7e6b33de7e54ca4ebe55dc3" + [[package]] name = "codespan-reporting" version = "0.11.1" @@ -2477,6 +2474,12 @@ version = "0.9.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2459377285ad874054d797f3ccebf984978aa39129f6eafde5cdc8315b612f8" +[[package]] +name = "const-oid" +version = "0.10.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "a6ef517f0926dd24a1582492c791b6a4818a4d94e789a334894aa15b0d12f55c" + [[package]] name = "const-random" version = "0.1.18" @@ -2536,12 +2539,6 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3d52eff69cd5e647efe296129160853a42795992097e8af39800e1060caeea9b" -[[package]] -name = "convert_case" -version = "0.4.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6245d59a3e82a7fc217c5828a6692dbc6dfb63a0c8c90495621f7b9d79704a0e" - [[package]] name = "convert_case" version = "0.6.0" @@ -2566,16 +2563,6 @@ version = "5.0.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "147be55d677052dabc6b22252d5dd0fd4c29c8c27aa4f2fbef0f94aa003b406f" -[[package]] -name = "cookie" -version = "0.17.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7efb37c3e1ccb1ff97164ad95ac1606e8ccd35b3fa0a7d99a304c7f4a428cc24" -dependencies = [ - "time", - "version_check", -] - [[package]] name = "cookie" version = "0.18.1" @@ -2633,6 +2620,15 @@ dependencies = [ "libc", ] +[[package]] +name = "cpufeatures" +version = "0.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201" +dependencies = [ + "libc", +] + [[package]] name = "crc" version = "3.4.0" @@ -2668,13 +2664,14 @@ dependencies = [ [[package]] name = "cron" -version = "0.15.0" +version = "0.16.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5877d3fbf742507b66bc2a1945106bd30dd8504019d596901ddd012a4dd01740" +checksum = "089df96cf6a25253b4b6b6744d86f91150a3d4df546f31a95def47976b8cba97" dependencies = [ "chrono", "once_cell", - "winnow 0.6.26", + "phf 0.11.3", + "winnow 0.7.15", ] [[package]] @@ -2767,6 +2764,15 @@ dependencies = [ "typenum", ] +[[package]] +name = "crypto-common" +version = "0.2.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "77727bb15fa921304124b128af125e7e3b968275d1b108b379190264f4423710" +dependencies = [ + "hybrid-array", +] + [[package]] name = "csv" version = "1.3.1" @@ -2797,6 +2803,15 @@ dependencies = [ "cipher 0.4.4", ] +[[package]] +name = "ctutils" +version = "0.4.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1005a6d4446f5120ef475ad3d2af2b30c49c2c9c6904258e3bb30219bebed5e4" +dependencies = [ + "cmov", +] + [[package]] name = "curve25519-dalek" version = "4.1.3" @@ -2804,7 +2819,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "97fb8b7c4503de7d6ae7b42ab72a5a59857b4c937ec27a3d4539dba95b5ab2be" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "curve25519-dalek-derive", "digest 0.10.7", "fiat-crypto 0.2.9", @@ -3164,7 +3179,7 @@ dependencies = [ "base64 0.22.1", "half", "hashbrown 0.14.5", - "indexmap 2.11.1", + "indexmap 2.12.0", "libc", "log", "object_store", @@ -3343,7 +3358,7 @@ dependencies = [ "datafusion-functions-aggregate-common", "datafusion-functions-window-common", "datafusion-physical-expr-common", - "indexmap 2.11.1", + "indexmap 2.12.0", "paste", "recursive", "serde_json", @@ -3358,7 +3373,7 @@ checksum = "422ac9cf3b22bbbae8cdf8ceb33039107fde1b5492693168f13bd566b1bcc839" dependencies = [ "arrow", "datafusion-common", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "paste", ] @@ -3512,7 +3527,7 @@ dependencies = [ "datafusion-common", "datafusion-expr", "datafusion-physical-expr", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "log", "recursive", @@ -3535,7 +3550,7 @@ dependencies = [ "datafusion-physical-expr-common", "half", "hashbrown 0.14.5", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "log", "paste", @@ -3597,7 +3612,7 @@ dependencies = [ "futures", "half", "hashbrown 0.14.5", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "log", "parking_lot", @@ -3639,13 +3654,19 @@ dependencies = [ "bigdecimal", "datafusion-common", "datafusion-expr", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "recursive", "regex", "sqlparser 0.55.0", ] +[[package]] +name = "datasketches" +version = "0.2.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c286de4e81ea2590afc24d754e0f83810c566f50a1388fa75ebd57928c0d9745" + [[package]] name = "debug-helper" version = "0.3.13" @@ -3751,7 +3772,7 @@ dependencies = [ "deno_media_type", "deno_path_util", "http 1.4.0", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "once_cell", "parking_lot", @@ -3792,7 +3813,7 @@ dependencies = [ "glob", "ignore", "import_map", - "indexmap 2.11.1", + "indexmap 2.12.0", "jsonc-parser", "log", "percent-encoding", @@ -3833,7 +3854,7 @@ dependencies = [ "deno_path_util", "deno_unsync", "futures", - "indexmap 2.11.1", + "indexmap 2.12.0", "libc", "memoffset", "parking_lot", @@ -3885,7 +3906,7 @@ dependencies = [ "aes-kw", "base64 0.21.7", "cbc", - "const-oid", + "const-oid 0.9.6", "ctr", "curve25519-dalek", "deno_core", @@ -4214,7 +4235,7 @@ dependencies = [ "brotli 6.0.0", "bytes", "cbc", - "const-oid", + "const-oid 0.9.6", "ctr", "data-encoding", "deno_core", @@ -4244,7 +4265,7 @@ dependencies = [ "hyper 1.8.1", "hyper-util", "idna", - "indexmap 2.11.1", + "indexmap 2.12.0", "ipnetwork", "k256", "lazy-regex", @@ -4320,7 +4341,7 @@ version = "0.212.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c2d328067139909aa81522a5d90f119368b541fbddd73ab630e4d9f777865f0d" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "proc-macro-rules", "proc-macro2", "quote", @@ -4364,7 +4385,7 @@ dependencies = [ "deno_error", "deno_path_util", "deno_semver", - "indexmap 2.11.1", + "indexmap 2.12.0", "serde", "serde_json", "sys_traits", @@ -4572,9 +4593,9 @@ dependencies = [ "log", "once_cell", "opentelemetry 0.27.1", - "opentelemetry-http", - "opentelemetry-otlp", - "opentelemetry-semantic-conventions", + "opentelemetry-http 0.27.0", + "opentelemetry-otlp 0.27.0", + "opentelemetry-semantic-conventions 0.27.0", "opentelemetry_sdk 0.27.1", "pin-project", "serde", @@ -4801,7 +4822,7 @@ version = "0.7.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e7c1832837b905bbfb5101e07cc24c8deddf52f93225eee6ead5f4d63d53ddcb" dependencies = [ - "const-oid", + "const-oid 0.9.6", "der_derive", "pem-rfc7468", "zeroize", @@ -4848,12 +4869,12 @@ dependencies = [ [[package]] name = "deranged" -version = "0.5.3" +version = "0.5.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d630bccd429a5bb5a64b5e94f693bfc48c9f8566418fda4c494cc94f911f87cc" +checksum = "7cd812cc2bc1d69d4764bd80df88b4317eaef9e773c75226407d9bc0876b211c" dependencies = [ "powerfmt", - "serde", + "serde_core", ] [[package]] @@ -4862,7 +4883,16 @@ version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8d67778784b508018359cbc8696edb3db78160bab2c2a28ba7f56ef6932997f8" dependencies = [ - "derive_builder_macro", + "derive_builder_macro 0.12.0", +] + +[[package]] +name = "derive_builder" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "507dfb09ea8b7fa618fcf76e953f4f5e192547945816d5358edffe39f6f94947" +dependencies = [ + "derive_builder_macro 0.20.2", ] [[package]] @@ -4877,26 +4907,35 @@ dependencies = [ "syn 1.0.109", ] +[[package]] +name = "derive_builder_core" +version = "0.20.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "2d5bcf7b024d6835cfb3d473887cd966994907effbe9227e8c8219824d06c4e8" +dependencies = [ + "darling 0.20.11", + "proc-macro2", + "quote", + "syn 2.0.117", +] + [[package]] name = "derive_builder_macro" version = "0.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "ebcda35c7a396850a55ffeac740804b40ffec779b98fffbb1738f4033f0ee79e" dependencies = [ - "derive_builder_core", + "derive_builder_core 0.12.0", "syn 1.0.109", ] [[package]] -name = "derive_more" -version = "0.99.20" +name = "derive_builder_macro" +version = "0.20.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "6edb4b64a43d977b8e99788fe3a04d483834fba1215a7e02caa415b626497f7f" +checksum = "ab63b0e2bf4d5928aff72e83a7dace85d7bba5fe12dcc3c5a572d78caffd3f3c" dependencies = [ - "convert_case 0.4.0", - "proc-macro2", - "quote", - "rustc_version 0.4.1", + "derive_builder_core 0.20.2", "syn 2.0.117", ] @@ -4927,6 +4966,7 @@ dependencies = [ "proc-macro2", "quote", "syn 2.0.117", + "unicode-xid", ] [[package]] @@ -4968,11 +5008,23 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292" dependencies = [ "block-buffer 0.10.4", - "const-oid", - "crypto-common", + "const-oid 0.9.6", + "crypto-common 0.1.7", "subtle", ] +[[package]] +name = "digest" +version = "0.11.2" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "4850db49bf08e663084f7fb5c87d202ef91a3907271aff24a94eb97ff039153c" +dependencies = [ + "block-buffer 0.12.0", + "const-oid 0.10.2", + "crypto-common 0.2.1", + "ctutils", +] + [[package]] name = "dirs" version = "4.0.0" @@ -5422,11 +5474,12 @@ checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" [[package]] name = "erased-serde" -version = "0.4.6" +version = "0.4.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e004d887f51fcb9fef17317a2f3525c887d8aa3f4f50fed920816a688284a5b7" +checksum = "d2add8a07dd6a8d93ff627029c51de145e12686fbc36ecb298ac22e74cf02dec" dependencies = [ "serde", + "serde_core", "typeid", ] @@ -5696,7 +5749,7 @@ dependencies = [ "half", "num-traits", "rand 0.9.0", - "rand_distr 0.5.1", + "rand_distr", ] [[package]] @@ -6197,6 +6250,7 @@ dependencies = [ "cfg-if", "libc", "r-efi 6.0.0", + "rand_core 0.10.0", "wasip2", "wasip3", ] @@ -6333,7 +6387,7 @@ dependencies = [ "thiserror 1.0.69", "tokio", "tokio-retry2", - "tonic", + "tonic 0.12.3", "tower 0.4.13", "tracing", ] @@ -6346,7 +6400,7 @@ checksum = "886aa8ec755382a1fdf4651f6e6ec01f2f3bf49f2cb0f068b9a74cafd574a715" dependencies = [ "prost", "prost-types", - "tonic", + "tonic 0.12.3", ] [[package]] @@ -6471,7 +6525,7 @@ dependencies = [ "futures-sink", "futures-util", "http 0.2.12", - "indexmap 2.11.1", + "indexmap 2.12.0", "slab", "tokio", "tokio-util", @@ -6490,7 +6544,7 @@ dependencies = [ "futures-core", "futures-sink", "http 1.4.0", - "indexmap 2.11.1", + "indexmap 2.12.0", "slab", "tokio", "tokio-util", @@ -6508,7 +6562,7 @@ dependencies = [ "crunchy", "num-traits", "rand 0.9.0", - "rand_distr 0.5.1", + "rand_distr", "zerocopy", ] @@ -6750,7 +6804,7 @@ version = "0.12.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7b5f8eb2ad728638ea2c7d47a21db23b7b58a72ed6a38256b8a1849f15fbbdf7" dependencies = [ - "hmac", + "hmac 0.12.1", ] [[package]] @@ -6762,6 +6816,15 @@ dependencies = [ "digest 0.10.7", ] +[[package]] +name = "hmac" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6303bc9732ae41b04cb554b844a762b4115a61bfaa81e3e83050991eeb56863f" +dependencies = [ + "digest 0.11.2", +] + [[package]] name = "home" version = "0.5.12" @@ -6904,6 +6967,15 @@ version = "2.3.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "135b12329e5e3ce057a9f972339ea52bc954fe1e9358ef27f95e89716fbc5424" +[[package]] +name = "hybrid-array" +version = "0.4.9" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "1a79f2aff40c18ab8615ddc5caa9eb5b96314aef18fe5823090f204ad988e813" +dependencies = [ + "typenum", +] + [[package]] name = "hyper" version = "0.14.32" @@ -7127,15 +7199,6 @@ dependencies = [ "tower-service", ] -[[package]] -name = "hyperloglogplus" -version = "0.4.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "621debdf94dcac33e50475fdd76d34d5ea9c0362a834b9db08c3024696c1fbe3" -dependencies = [ - "serde", -] - [[package]] name = "iana-time-zone" version = "0.1.65" @@ -7317,7 +7380,7 @@ checksum = "1215d4d92511fbbdaea50e750e91f2429598ef817f02b579158e92803b52c00a" dependencies = [ "boxed_error", "deno_error", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "percent-encoding", "serde", @@ -7339,13 +7402,14 @@ dependencies = [ [[package]] name = "indexmap" -version = "2.11.1" +version = "2.12.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "206a8042aec68fa4a62e8d3f7aa4ceb508177d9324faf261e1959e495b7a1921" +checksum = "6717a8d2a5a929a1a2eb43a12812498ed141a0bcfb7e8f7844fbdbe4303bba9f" dependencies = [ "equivalent", - "hashbrown 0.15.5", + "hashbrown 0.16.0", "serde", + "serde_core", ] [[package]] @@ -7399,9 +7463,9 @@ checksum = "8bb03732005da905c88227371639bf1ad885cc712789c011c31c5fb3ab3ccf02" [[package]] name = "inventory" -version = "0.3.22" +version = "0.3.23" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "009ae045c87e7082cb72dab0ccd01ae075dd00141ddc108f43a0ea150a9e7227" +checksum = "8062b737e5389949f477d4760a2ebbff0c366f97798f2419b8d8f366363d3342" dependencies = [ "rustversion", ] @@ -7419,14 +7483,15 @@ dependencies = [ [[package]] name = "ipconfig" -version = "0.3.2" +version = "0.3.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b58db92f96b720de98181bbbe63c831e87005ab460c1bf306eb2622b4707997f" +checksum = "4d40460c0ce33d6ce4b0630ad68ff63d6661961c48b6dba35e5a4d81cfb48222" dependencies = [ - "socket2 0.5.10", + "socket2 0.6.3", "widestring", - "windows-sys 0.48.0", - "winreg", + "windows-registry", + "windows-result 0.4.1", + "windows-sys 0.61.2", ] [[package]] @@ -7446,9 +7511,9 @@ dependencies = [ [[package]] name = "iri-string" -version = "0.7.10" +version = "0.7.12" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c91338f0783edbd6195decb37bae672fd3b165faffb89bf7b9e6942f8b1a731a" +checksum = "25e659a4bb38e810ebc252e53b5814ff908a8c58c2a9ce2fae1bbec24cbf4e20" dependencies = [ "memchr", "serde", @@ -7496,15 +7561,6 @@ dependencies = [ "either", ] -[[package]] -name = "itertools" -version = "0.12.1" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ba291022dbbd398a455acf126c1e341954079855bc60dfdda641363bd6922569" -dependencies = [ - "either", -] - [[package]] name = "itertools" version = "0.13.0" @@ -7538,7 +7594,7 @@ dependencies = [ "cesu8", "cfg-if", "combine", - "jni-sys", + "jni-sys 0.3.1", "log", "thiserror 1.0.69", "walkdir", @@ -7547,9 +7603,31 @@ dependencies = [ [[package]] name = "jni-sys" -version = "0.3.0" +version = "0.3.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8eaf4bc02d17cbdd7ff4c7438cafcdf7fb9a4613313ad11b4f8fefe7d3fa0130" +checksum = "41a652e1f9b6e0275df1f15b32661cf0d4b78d4d87ddec5e0c3c20f097433258" +dependencies = [ + "jni-sys 0.4.1", +] + +[[package]] +name = "jni-sys" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "c6377a88cb3910bee9b0fa88d4f42e1d2da8e79915598f65fb0c7ee14c878af2" +dependencies = [ + "jni-sys-macros", +] + +[[package]] +name = "jni-sys-macros" +version = "0.4.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "38c0b942f458fe50cdac086d2f946512305e5631e720728f2a61aabcd47a6264" +dependencies = [ + "quote", + "syn 2.0.117", +] [[package]] name = "jobserver" @@ -7637,7 +7715,7 @@ checksum = "5a87cc7a48537badeae96744432de36f4be2b4a34a05a5ef32e9dd8a1c169dde" dependencies = [ "base64 0.22.1", "js-sys", - "pem 3.0.5", + "pem 3.0.6", "ring 0.17.14", "serde", "serde_json", @@ -7686,7 +7764,7 @@ version = "0.1.6" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "cb26cec98cce3a3d96cbb7bced3c4b16e3d13f27ec56dbd62cbc8f39cfb9d653" dependencies = [ - "cpufeatures", + "cpufeatures 0.2.17", ] [[package]] @@ -7695,7 +7773,7 @@ version = "0.4.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4ee7893dab2e44ae5f9d0173f26ff4aa327c10b01b06a72b52dd9405b628640d" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", ] [[package]] @@ -7786,7 +7864,7 @@ dependencies = [ "jsonpath-rust", "k8s-openapi", "kube-core", - "pem 3.0.5", + "pem 3.0.6", "rustls 0.23.35", "secrecy", "serde", @@ -7898,12 +7976,6 @@ dependencies = [ "spin 0.9.8", ] -[[package]] -name = "lazycell" -version = "1.3.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "830d08ce1d1d941e6b30645f1a0eb5643013d835ce3779a5fc208261dbe10f55" - [[package]] name = "leb128fmt" version = "0.1.0" @@ -8059,9 +8131,9 @@ dependencies = [ [[package]] name = "libredox" -version = "0.1.14" +version = "0.1.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1744e39d1d6a9948f4f388969627434e31128196de472883b39f148769bfe30a" +checksum = "7ddbf48fd451246b1f8c2610bd3b4ac0cc6e149d89832867093ab69a17194f08" dependencies = [ "bitflags 2.9.4", "libc", @@ -8185,11 +8257,11 @@ dependencies = [ [[package]] name = "lru" -version = "0.14.0" +version = "0.16.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9f8cc7106155f10bdf99a6f379688f543ad6596a415375b36a59a054ceda1198" +checksum = "a1dc47f592c06f33f8e3aea9591776ec7c9f9e4124778ff8a3c3b87159f7e593" dependencies = [ - "hashbrown 0.15.5", + "hashbrown 0.16.0", ] [[package]] @@ -8216,6 +8288,12 @@ dependencies = [ "twox-hash 2.1.2", ] +[[package]] +name = "lz4_flex" +version = "0.13.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "db9a0d582c2874f68138a16ce1867e0ffde6c0bb0a0df85e1f36d04146db488a" + [[package]] name = "lzma-sys" version = "0.1.20" @@ -8308,9 +8386,9 @@ dependencies = [ [[package]] name = "malachite" -version = "0.4.18" +version = "0.4.22" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4a6ecab92657eb234bfe98abd0b17920772c6b14ce69256950142e2eb36d000b" +checksum = "2fbdf9cb251732db30a7200ebb6ae5d22fe8e11397364416617d2c2cf0c51cb5" dependencies = [ "malachite-base", "malachite-nz", @@ -8331,11 +8409,11 @@ dependencies = [ [[package]] name = "malachite-bigint" -version = "0.2.0" +version = "0.2.3" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "17703a19c80bbdd0b7919f0f104f3b0597f7de4fc4e90a477c15366a5ba03faa" +checksum = "d149aaa2965d70381709d9df4c7ee1fc0de1c614a4efc2ee356f5e43d68749f8" dependencies = [ - "derive_more 0.99.20", + "derive_more 1.0.0", "malachite", "num-integer", "num-traits", @@ -8427,6 +8505,16 @@ dependencies = [ "digest 0.10.7", ] +[[package]] +name = "md-5" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "69b6441f590336821bb897fb28fc622898ccceb1d6cea3fde5ea86b090c4de98" +dependencies = [ + "cfg-if", + "digest 0.11.2", +] + [[package]] name = "md4" version = "0.10.2" @@ -8599,9 +8687,9 @@ dependencies = [ [[package]] name = "mio" -version = "1.1.1" +version = "1.2.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a69bcab0ad47271a0234d9422b131806bf3968021e5dc9328caf2d4cd58557fc" +checksum = "50b7e5b27aa02a74bac8c3f23f448f8d87ff11f92d3aac1a6ed369ee08cc56c1" dependencies = [ "libc", "wasi 0.11.1+wasi-snapshot-preview1", @@ -8610,9 +8698,9 @@ dependencies = [ [[package]] name = "moka" -version = "0.12.14" +version = "0.12.15" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "85f8024e1c8e71c778968af91d43700ce1d11b219d127d79fb2934153b82b42b" +checksum = "957228ad12042ee839f93c8f257b62b4c0ab5eaae1d4fa60de53b27c9d7c5046" dependencies = [ "async-lock", "crossbeam-channel", @@ -8636,19 +8724,20 @@ checksum = "b52c1b33ff98142aecea13138bd399b68aa7ab5d9546c300988c345004001eea" [[package]] name = "monostate" -version = "0.1.14" +version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "aafe1be9d0c75642e3e50fedc7ecadf1ef1cbce6eb66462153fc44245343fbee" +checksum = "3341a273f6c9d5bef1908f17b7267bbab0e95c9bf69a0d4dcf8e9e1b2c76ef67" dependencies = [ "monostate-impl", "serde", + "serde_core", ] [[package]] name = "monostate-impl" -version = "0.1.14" +version = "0.1.18" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "c402a4092d5e204f32c9e155431046831fa712637043c58cb73bc6bc6c9663b5" +checksum = "e4db6d5580af57bf992f59068d4ea26fd518574ff48d7639b255a36f9de6e7e9" dependencies = [ "proc-macro2", "quote", @@ -8704,9 +8793,9 @@ dependencies = [ [[package]] name = "mysql_async" -version = "0.36.1" +version = "0.36.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "277ce2f2459b2af4cc6d0a0b7892381f80800832f57c533f03e2845f4ea331ea" +checksum = "d1d9585dc9058886ff3a1f48a23024dd1d054264dee7c5ae0e4bd640c953bee5" dependencies = [ "bytes", "crossbeam-queue", @@ -8715,10 +8804,10 @@ dependencies = [ "futures-sink", "futures-util", "keyed_priority_queue", - "lru 0.14.0", + "lru 0.16.3", "mysql_common", "native-tls", - "pem 3.0.5", + "pem 3.0.6", "percent-encoding", "rand 0.9.0", "serde", @@ -8771,7 +8860,7 @@ dependencies = [ "bitflags 2.9.4", "codespan-reporting", "hexf-parse", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "num-traits", "rustc-hash 1.1.0", @@ -8826,7 +8915,7 @@ version = "0.5.0+25.2.9519653" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "8c196769dd60fd4f363e11d948139556a344e79d451aeb2fa2fd040738ef7691" dependencies = [ - "jni-sys", + "jni-sys 0.3.1", ] [[package]] @@ -9062,7 +9151,7 @@ dependencies = [ "dirs-sys 0.4.1", "fancy-regex 0.14.0", "heck 0.5.0", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "lru 0.12.5", "miette", @@ -9181,9 +9270,9 @@ dependencies = [ [[package]] name = "num-conv" -version = "0.1.0" +version = "0.2.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "51d515d32fb182ee37cda2ccdcb92950d6a3c2893aa280e540671c2cd0f3b1d9" +checksum = "c6673768db2d862beb9b39a78fdcb1a69439615d5794a1be50caa9bc92c81967" [[package]] name = "num-format" @@ -9332,7 +9421,7 @@ dependencies = [ "md-5 0.10.6", "parking_lot", "percent-encoding", - "quick-xml 0.37.5", + "quick-xml", "rand 0.9.0", "reqwest 0.12.28", "ring 0.17.14", @@ -9438,7 +9527,7 @@ dependencies = [ "chrono", "dyn-clone", "ed25519-dalek", - "hmac", + "hmac 0.12.1", "http 1.4.0", "itertools 0.10.5", "log", @@ -9535,9 +9624,9 @@ dependencies = [ [[package]] name = "opentelemetry" -version = "0.29.1" +version = "0.30.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "9e87237e2775f74896f9ad219d26a2081751187eb7c9f5c58dde20a23b95d16c" +checksum = "aaf416e4cb72756655126f7dd7bb0af49c674f4c1b9903e80c009e0c37e552e6" dependencies = [ "futures-core", "futures-sink", @@ -9549,11 +9638,11 @@ dependencies = [ [[package]] name = "opentelemetry-appender-tracing" -version = "0.27.0" +version = "0.30.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "ab5feffc321035ad94088a7e5333abb4d84a8726e54a802e736ce9dd7237e85b" +checksum = "e68f63eca5fad47e570e00e893094fc17be959c80c79a7d6ec1abdd5ae6ffc16" dependencies = [ - "opentelemetry 0.27.1", + "opentelemetry 0.30.0", "tracing", "tracing-core", "tracing-subscriber", @@ -9571,6 +9660,19 @@ dependencies = [ "opentelemetry 0.27.1", ] +[[package]] +name = "opentelemetry-http" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "50f6639e842a97dbea8886e3439710ae463120091e2e064518ba8e716e6ac36d" +dependencies = [ + "async-trait", + "bytes", + "http 1.4.0", + "opentelemetry 0.30.0", + "reqwest 0.12.28", +] + [[package]] name = "opentelemetry-otlp" version = "0.27.0" @@ -9581,14 +9683,33 @@ dependencies = [ "futures-core", "http 1.4.0", "opentelemetry 0.27.1", - "opentelemetry-http", + "opentelemetry-http 0.27.0", "opentelemetry-proto 0.27.0", "opentelemetry_sdk 0.27.1", "prost", "serde_json", "thiserror 1.0.69", "tokio", - "tonic", + "tonic 0.12.3", + "tracing", +] + +[[package]] +name = "opentelemetry-otlp" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "dbee664a43e07615731afc539ca60c6d9f1a9425e25ca09c57bc36c87c55852b" +dependencies = [ + "http 1.4.0", + "opentelemetry 0.30.0", + "opentelemetry-http 0.30.0", + "opentelemetry-proto 0.30.0", + "opentelemetry_sdk 0.30.0", + "prost", + "reqwest 0.12.28", + "thiserror 2.0.18", + "tokio", + "tonic 0.13.1", "tracing", ] @@ -9603,23 +9724,22 @@ dependencies = [ "opentelemetry_sdk 0.27.1", "prost", "serde", - "tonic", + "tonic 0.12.3", ] [[package]] name = "opentelemetry-proto" -version = "0.29.0" +version = "0.30.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8c40da242381435e18570d5b9d50aca2a4f4f4d8e146231adb4e7768023309b3" +checksum = "2e046fd7660710fe5a05e8748e70d9058dc15c94ba914e7c4faa7c728f0e8ddc" dependencies = [ "base64 0.22.1", "hex", - "opentelemetry 0.29.1", - "opentelemetry_sdk 0.29.0", + "opentelemetry 0.30.0", + "opentelemetry_sdk 0.30.0", "prost", "serde", - "tonic", - "tracing", + "tonic 0.13.1", ] [[package]] @@ -9628,6 +9748,12 @@ version = "0.27.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bc1b6902ff63b32ef6c489e8048c5e253e2e4a803ea3ea7e783914536eb15c52" +[[package]] +name = "opentelemetry-semantic-conventions" +version = "0.30.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "83d059a296a47436748557a353c5e6c5705b9470ef6c95cfc52c21a8814ddac2" + [[package]] name = "opentelemetry_sdk" version = "0.27.1" @@ -9644,26 +9770,25 @@ dependencies = [ "rand 0.8.5", "serde_json", "thiserror 1.0.69", - "tokio", - "tokio-stream", "tracing", ] [[package]] name = "opentelemetry_sdk" -version = "0.29.0" +version = "0.30.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "afdefb21d1d47394abc1ba6c57363ab141be19e27cc70d0e422b7f303e4d290b" +checksum = "11f644aa9e5e31d11896e024305d7e3c98a88884d9f8919dbf37a9991bc47a4b" dependencies = [ "futures-channel", "futures-executor", "futures-util", - "glob", - "opentelemetry 0.29.1", + "opentelemetry 0.30.0", "percent-encoding", "rand 0.9.0", "serde_json", "thiserror 2.0.18", + "tokio", + "tokio-stream", ] [[package]] @@ -9708,6 +9833,15 @@ dependencies = [ "num-traits", ] +[[package]] +name = "ordered-float" +version = "5.3.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b7d950ca161dc355eaf28f82b11345ed76c6e1f6eb1f4f4479e0323b9e2fbd0e" +dependencies = [ + "num-traits", +] + [[package]] name = "os_pipe" version = "1.1.5" @@ -9733,7 +9867,7 @@ checksum = "1a80800c0488c3a21695ea981a54918fbb37abf04f4d0720c453632255e2ff0e" [[package]] name = "ownedbytes" version = "0.9.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "stable_deref_trait", ] @@ -9845,7 +9979,7 @@ dependencies = [ "futures", "half", "hashbrown 0.15.5", - "lz4_flex", + "lz4_flex 0.11.6", "num", "num-bigint", "object_store", @@ -9901,7 +10035,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8ed6a7761f76e3b9f92dfb0a60a6a6477c61024b775147ff0973a02653abaf2" dependencies = [ "digest 0.10.7", - "hmac", + "hmac 0.12.1", ] [[package]] @@ -9915,12 +10049,12 @@ dependencies = [ [[package]] name = "pem" -version = "3.0.5" +version = "3.0.6" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "38af38e8470ac9dee3ce1bae1af9c1671fffc44ddfd8bd1d0a3445bf349a8ef3" +checksum = "1d30c53c26bc5b31a98cd02d20f25a7c8567146caf63ed593a9d87b2775291be" dependencies = [ "base64 0.22.1", - "serde", + "serde_core", ] [[package]] @@ -10000,7 +10134,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3672b37090dbd86368a4145bc067582552b29c27377cad4e0a306c97f9bd7772" dependencies = [ "fixedbitset", - "indexmap 2.11.1", + "indexmap 2.12.0", ] [[package]] @@ -10196,7 +10330,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "9d1fe60d06143b2430aa532c94cfe9e29783047f06c0d7fd359a9a51b729fa25" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "opaque-debug", "universal-hash", ] @@ -10239,7 +10373,7 @@ dependencies = [ "byteorder", "bytes", "fallible-iterator 0.2.0", - "hmac", + "hmac 0.12.1", "md-5 0.10.6", "memchr", "rand 0.8.5", @@ -10249,19 +10383,19 @@ dependencies = [ [[package]] name = "postgres-protocol" -version = "0.6.10" +version = "0.6.11" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "3ee9dd5fe15055d2b6806f4736aa0c9637217074e224bbec46d4041b91bb9491" +checksum = "56201207dac53e2f38e848e31b4b91616a6bb6e0c7205b77718994a7f49e70fc" dependencies = [ "base64 0.22.1", "byteorder", "bytes", "fallible-iterator 0.2.0", - "hmac", - "md-5 0.10.6", + "hmac 0.13.0", + "md-5 0.11.0", "memchr", - "rand 0.9.0", - "sha2 0.10.9", + "rand 0.10.0", + "sha2 0.11.0", "stringprep", ] @@ -10286,7 +10420,7 @@ dependencies = [ "bytes", "chrono", "fallible-iterator 0.2.0", - "postgres-protocol 0.6.10", + "postgres-protocol 0.6.11", "serde", "serde_json", "uuid", @@ -10435,7 +10569,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a3ef4f2f0422f23a82ec9f628ea2acd12871c81a9362b02c43c1aa86acfc3ba1" dependencies = [ "futures", - "indexmap 2.11.1", + "indexmap 2.12.0", "nix 0.30.1", "tokio", "tracing", @@ -10620,16 +10754,6 @@ dependencies = [ "thiserror 1.0.69", ] -[[package]] -name = "quick-xml" -version = "0.30.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "eff6510e86862b57b210fd8cbe8ed3f0d7d600b9c2863cd4549a2e033c66e956" -dependencies = [ - "memchr", - "serde", -] - [[package]] name = "quick-xml" version = "0.37.5" @@ -10663,7 +10787,7 @@ dependencies = [ "pin-project-lite", "quinn-proto", "quinn-udp", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "rustls 0.23.35", "socket2 0.6.3", "thiserror 2.0.18", @@ -10684,7 +10808,7 @@ dependencies = [ "lru-slab", "rand 0.9.0", "ring 0.17.14", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "rustls 0.23.35", "rustls-pki-types", "slab", @@ -10780,6 +10904,17 @@ dependencies = [ "zerocopy", ] +[[package]] +name = "rand" +version = "0.10.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8" +dependencies = [ + "chacha20", + "getrandom 0.4.2", + "rand_core 0.10.0", +] + [[package]] name = "rand_chacha" version = "0.2.2" @@ -10838,14 +10973,10 @@ dependencies = [ ] [[package]] -name = "rand_distr" -version = "0.4.3" +name = "rand_core" +version = "0.10.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "32cb0b9bc82b0a0876c2dd994a7e7a2683d3e7390ca40e6886785ef0c7e3ee31" -dependencies = [ - "num-traits", - "rand 0.8.5", -] +checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba" [[package]] name = "rand_distr" @@ -10924,7 +11055,7 @@ version = "0.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "75e669e5202259b5314d1ea5397316ad400819437857b90861765f24c4cf80a2" dependencies = [ - "pem 3.0.5", + "pem 3.0.6", "ring 0.17.14", "rustls-pki-types", "time", @@ -11119,6 +11250,7 @@ dependencies = [ "base64 0.22.1", "bytes", "encoding_rs", + "futures-channel", "futures-core", "futures-util", "h2 0.4.13", @@ -11260,7 +11392,7 @@ version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f8dd2a808d456c4a54e300a23e9f5a67e122c3024119acbfd73e3bf664491cb2" dependencies = [ - "hmac", + "hmac 0.12.1", "subtle", ] @@ -11422,7 +11554,7 @@ dependencies = [ "convert_case 0.10.0", "fnv", "ident_case", - "indexmap 2.11.1", + "indexmap 2.12.0", "proc-macro-crate", "proc-macro2", "quote", @@ -11445,7 +11577,7 @@ version = "0.9.10" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "b8573f03f5883dcaebdfcf4725caa1ecb9c15b2ef50c43a07b816e06799bb12d" dependencies = [ - "const-oid", + "const-oid 0.9.6", "digest 0.10.7", "num-bigint-dig", "num-integer", @@ -11540,9 +11672,9 @@ dependencies = [ [[package]] name = "rust_decimal" -version = "1.40.0" +version = "1.41.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "61f703d19852dbf87cbc513643fa81428361eb6940f1ac14fd58155d295a3eb0" +checksum = "2ce901f9a19d251159075a4c37af514c3b8ef99c22e02dd8c19161cf397ee94a" dependencies = [ "arrayvec", "borsh", @@ -11553,6 +11685,7 @@ dependencies = [ "rkyv", "serde", "serde_json", + "wasm-bindgen", ] [[package]] @@ -11569,9 +11702,9 @@ checksum = "08d43f7aa6b08d49f382cde6a7982047c3426db949b1424bc4b7ec9ae12c6ce2" [[package]] name = "rustc-hash" -version = "2.1.1" +version = "2.1.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "357703d41365b4b27c590e3ed91eabb1b663f07c4c084095e60cbed4362dff0d" +checksum = "94300abf3f1ae2e2b8ffb7b58043de3d399c73fa6f4b73826402a5c457614dbe" [[package]] name = "rustc_version" @@ -11663,7 +11796,7 @@ dependencies = [ "once_cell", "ring 0.17.14", "rustls-pki-types", - "rustls-webpki 0.103.9", + "rustls-webpki 0.103.10", "subtle", "zeroize", ] @@ -11747,7 +11880,7 @@ dependencies = [ "rustls 0.23.35", "rustls-native-certs 0.8.3", "rustls-platform-verifier-android", - "rustls-webpki 0.103.9", + "rustls-webpki 0.103.10", "security-framework 3.6.0", "security-framework-sys", "webpki-root-certs 1.0.6", @@ -11795,9 +11928,9 @@ dependencies = [ [[package]] name = "rustls-webpki" -version = "0.103.9" +version = "0.103.10" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d7df23109aa6c1567d1c575b9952556388da57401e4ace1d15f79eedad0d8f53" +checksum = "df33b2b81ac578cabaf06b89b0631153a3f416b0a886e8a7a1707fb51abbd1ef" dependencies = [ "aws-lc-rs", "ring 0.17.14", @@ -11934,15 +12067,14 @@ dependencies = [ [[package]] name = "samael" -version = "0.0.14" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "b75583aad4a51c50fc0af69c230d18078c9d5a69a98d0f6013d01053acf744f4" +version = "0.0.20" +source = "git+https://github.com/njaremko/samael?rev=f879f1942ec1b34b6d3027ce7e4724ad95d15dfa#f879f1942ec1b34b6d3027ce7e4724ad95d15dfa" dependencies = [ - "base64 0.21.7", - "bindgen 0.69.5", + "base64 0.22.1", + "bindgen 0.72.1", "chrono", "data-encoding", - "derive_builder", + "derive_builder 0.20.2", "flate2", "lazy_static", "libc", @@ -11951,10 +12083,10 @@ dependencies = [ "openssl-probe 0.1.6", "openssl-sys", "pkg-config", - "quick-xml 0.30.0", - "rand 0.8.5", + "quick-xml", + "rand 0.9.0", "serde", - "thiserror 1.0.69", + "thiserror 2.0.18", "url", "uuid", ] @@ -12186,10 +12318,11 @@ checksum = "1bc711410fbe7399f390ca1c3b60ad0f53f80e95c5eb935e52268a0e2cd49acc" [[package]] name = "serde" -version = "1.0.219" +version = "1.0.220" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5f0e2c6ed6606019b4e29e69dbaba95b11854410e5347d525002456dbbb786b6" +checksum = "ceecad4c782e936ac90ecfd6b56532322e3262b14320abf30ce89a92ffdbfe22" dependencies = [ + "serde_core", "serde_derive", ] @@ -12211,7 +12344,7 @@ version = "0.7.0" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "f3a1a3341211875ef120e117ea7fd5228530ae7e7036a779fdc9117be6b3282c" dependencies = [ - "ordered-float", + "ordered-float 2.10.1", "serde", ] @@ -12228,18 +12361,28 @@ dependencies = [ [[package]] name = "serde_bytes" -version = "0.11.17" +version = "0.11.19" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "8437fd221bde2d4ca316d61b90e337e9e702b3820b87d63caa9ba6c02bd06d96" +checksum = "a5d440709e79d88e51ac01c4b72fc6cb7314017bb7da9eeff678aa94c10e3ea8" dependencies = [ "serde", + "serde_core", +] + +[[package]] +name = "serde_core" +version = "1.0.220" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "ddba47394f3b862d6ff6efdbd26ca4673e3566a307880a0ffb98f274bbe0ec32" +dependencies = [ + "serde_derive", ] [[package]] name = "serde_derive" -version = "1.0.219" +version = "1.0.220" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "5b0276cf7f2c73365f7157c8123c21cd9a50fbbd844757af28ca1f5925fc2a00" +checksum = "60e1f3b1761e96def5ec6d04a6e7421c0404fa3cf5c0155f1e2848fae3d8cc08" dependencies = [ "proc-macro2", "quote", @@ -12259,15 +12402,16 @@ dependencies = [ [[package]] name = "serde_json" -version = "1.0.143" +version = "1.0.149" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "d401abef1d108fbd9cbaebc3e46611f4b1021f714a0597a71f41ee463f5f4a5a" +checksum = "83fc039473c5595ace860d8c4fafa220ff474b3fc6bfdb4293327f1a37e94d86" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "itoa", "memchr", - "ryu", "serde", + "serde_core", + "zmij", ] [[package]] @@ -12281,12 +12425,13 @@ dependencies = [ [[package]] name = "serde_path_to_error" -version = "0.1.17" +version = "0.1.20" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "59fab13f937fa393d08645bf3a84bdfe86e296747b506ada67bb15f10f218b2a" +checksum = "10a9ff822e371bb5403e391ecd83e182e0e77ba7f6fe0160b795797109d1b457" dependencies = [ "itoa", "serde", + "serde_core", ] [[package]] @@ -12354,7 +12499,7 @@ dependencies = [ "chrono", "hex", "indexmap 1.9.3", - "indexmap 2.11.1", + "indexmap 2.12.0", "schemars 0.9.0", "schemars 1.2.1", "serde", @@ -12382,7 +12527,7 @@ version = "0.9.34+deprecated" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "itoa", "ryu", "serde", @@ -12395,7 +12540,7 @@ version = "0.0.12" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "59e2dd588bf1597a252c3b920e0143eb99b0f76e4e082f4c92ce34fbc9e71ddd" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "itoa", "libyml", "memchr", @@ -12421,7 +12566,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "e3bf829a2d51ab4a5ddf1352d8470c140cadc8301b2ae1789db023f01cedd6ba" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest 0.10.7", ] @@ -12433,7 +12578,7 @@ checksum = "4d58a1e1bf39749807d89cf2d98ac2dfa0ff1cb3faa38fbb64dd88ac8013d800" dependencies = [ "block-buffer 0.9.0", "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest 0.9.0", "opaque-debug", ] @@ -12445,10 +12590,21 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283" dependencies = [ "cfg-if", - "cpufeatures", + "cpufeatures 0.2.17", "digest 0.10.7", ] +[[package]] +name = "sha2" +version = "0.11.0" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "446ba717509524cb3f22f17ecc096f10f4822d76ab5c0b9822c5f9c284e825f4" +dependencies = [ + "cfg-if", + "cpufeatures 0.3.0", + "digest 0.11.2", +] + [[package]] name = "sha3" version = "0.10.8" @@ -12546,9 +12702,9 @@ dependencies = [ [[package]] name = "simd-adler32" -version = "0.3.8" +version = "0.3.9" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "e320a6c5ad31d271ad523dcf3ad13e2767ad8b1cb8f047f75a8aeaf8da139da2" +checksum = "703d5c7ef118737c72f1af64ad2f6f8c5e1921f818cdcb97b8fe6fc69bf66214" [[package]] name = "simd-json" @@ -12573,9 +12729,9 @@ checksum = "e3a9fe34e3e7a50316060351f37187a3f546bce95496156754b601a5fa71b76e" [[package]] name = "simple_asn1" -version = "0.6.3" +version = "0.6.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "297f631f50729c8c99b84667867963997ec0b50f32b2a7dbcab828ef0541e8bb" +checksum = "0d585997b0ac10be3c5ee635f1bab02d512760d14b7c468801ac8a01d9ae5f1d" dependencies = [ "num-bigint", "num-traits", @@ -12603,9 +12759,9 @@ checksum = "1b6709c7b6754dca1311b3c73e79fcce40dd414c782c66d88e8823030093b02b" [[package]] name = "sketches-ddsketch" -version = "0.3.1" +version = "0.4.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0c6f73aeb92d671e0cc4dca167e59b2deb6387c375391bc99ee743f326994a2b" +checksum = "05e40b6cf54d988dc1a2223531b969c9a9e30906ad90ef64890c27b4bfbb46ea" dependencies = [ "serde", ] @@ -12727,7 +12883,7 @@ dependencies = [ "data-encoding", "debugid", "if_chain", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "serde", "serde_json", "unicode-id-start", @@ -12862,7 +13018,7 @@ dependencies = [ "futures-util", "hashbrown 0.15.5", "hashlink 0.10.0", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "memchr", "once_cell", @@ -12943,7 +13099,7 @@ dependencies = [ "generic-array", "hex", "hkdf", - "hmac", + "hmac 0.12.1", "itoa", "log", "md-5 0.10.6", @@ -12984,7 +13140,7 @@ dependencies = [ "futures-util", "hex", "hkdf", - "hmac", + "hmac 0.12.1", "home", "itoa", "log", @@ -13262,7 +13418,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "4740e53eaf68b101203c1df0937d5161a29f3c13bceed0836ddfe245b72dd000" dependencies = [ "anyhow", - "indexmap 2.11.1", + "indexmap 2.12.0", "serde", "serde_json", "swc_cached", @@ -13374,7 +13530,7 @@ checksum = "65f21494e75d0bd8ef42010b47cabab9caaed8f2207570e809f6f4eb51a710d1" dependencies = [ "better_scoped_tls", "bitflags 2.9.4", - "indexmap 2.11.1", + "indexmap 2.12.0", "once_cell", "phf 0.11.3", "rustc-hash 1.1.0", @@ -13443,7 +13599,7 @@ checksum = "76c76d8b9792ce51401d38da0fa62158d61f6d80d16d68fe5b03ce4bf5fba383" dependencies = [ "base64 0.21.7", "dashmap 5.5.3", - "indexmap 2.11.1", + "indexmap 2.12.0", "once_cell", "serde", "sha1", @@ -13483,7 +13639,7 @@ version = "0.134.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "029eec7dd485923a75b5a45befd04510288870250270292fc2c1b3a9e7547408" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "num_cpus", "once_cell", "rustc-hash 1.1.0", @@ -13700,8 +13856,8 @@ checksum = "7b2093cf4c8eb1e67749a6762251bc9cd836b6fc171623bd0a9d324d37af2417" [[package]] name = "tantivy" -version = "0.24.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.26.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "aho-corasick", "arc-swap", @@ -13712,17 +13868,17 @@ dependencies = [ "census", "crc32fast", "crossbeam-channel", + "datasketches", "downcast-rs", "fastdivide", "fnv", "fs4", "htmlescape", - "hyperloglogplus", "itertools 0.14.0", "levenshtein_automata", "log", - "lru 0.12.5", - "lz4_flex", + "lru 0.16.3", + "lz4_flex 0.13.0", "measure_time", "memmap2 0.9.10", "once_cell", @@ -13730,7 +13886,7 @@ dependencies = [ "rayon", "regex", "rust-stemmers", - "rustc-hash 2.1.1", + "rustc-hash 2.1.2", "serde", "serde_json", "sketches-ddsketch", @@ -13745,22 +13901,23 @@ dependencies = [ "tempfile", "thiserror 2.0.18", "time", + "typetag", "uuid", "winapi", ] [[package]] name = "tantivy-bitpacker" -version = "0.8.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.9.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "bitpacking", ] [[package]] name = "tantivy-columnar" -version = "0.5.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.6.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "downcast-rs", "fastdivide", @@ -13774,8 +13931,8 @@ dependencies = [ [[package]] name = "tantivy-common" -version = "0.9.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.10.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "async-trait", "byteorder", @@ -13797,18 +13954,20 @@ dependencies = [ [[package]] name = "tantivy-query-grammar" -version = "0.24.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.25.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ + "fnv", "nom 7.1.3", + "ordered-float 5.3.0", "serde", "serde_json", ] [[package]] name = "tantivy-sstable" -version = "0.5.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.6.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "futures-util", "itertools 0.14.0", @@ -13820,18 +13979,17 @@ dependencies = [ [[package]] name = "tantivy-stacker" -version = "0.5.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.6.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "murmurhash32", - "rand_distr 0.4.3", "tantivy-common", ] [[package]] name = "tantivy-tokenizer-api" -version = "0.5.0" -source = "git+https://github.com/windmill-labs/tantivy?rev=6a24621231202ccd77bec90d8787e2281fb94e4e#6a24621231202ccd77bec90d8787e2281fb94e4e" +version = "0.6.0" +source = "git+https://github.com/windmill-labs/tantivy?rev=6ae7c70bc603b8e69e27f3240e08bd00a93fb12c#6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" dependencies = [ "serde", ] @@ -13877,12 +14035,12 @@ dependencies = [ [[package]] name = "terminal_size" -version = "0.4.3" +version = "0.4.4" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "60b8cb979cb11c32ce1603f8137b22262a9d131aaa5c37b5678025f22b8becd0" +checksum = "230a1b821ccbd75b185820a1f1ff7b14d21da1e442e22c0863ea5f08771a8874" dependencies = [ "rustix 1.1.4", - "windows-sys 0.60.2", + "windows-sys 0.61.2", ] [[package]] @@ -13961,7 +14119,7 @@ checksum = "7e54bc85fc7faa8bc175c4bab5b92ba8d9a3ce893d0e9f42cc455c8ab16a9e09" dependencies = [ "byteorder", "integer-encoding", - "ordered-float", + "ordered-float 2.10.1", ] [[package]] @@ -14038,30 +14196,30 @@ dependencies = [ [[package]] name = "time" -version = "0.3.44" +version = "0.3.47" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "91e7d9e3bb61134e77bde20dd4825b97c010155709965fedf0f49bb138e52a9d" +checksum = "743bd48c283afc0388f9b8827b976905fb217ad9e647fae3a379a9283c4def2c" dependencies = [ "deranged", "itoa", "num-conv", "powerfmt", - "serde", + "serde_core", "time-core", "time-macros", ] [[package]] name = "time-core" -version = "0.1.6" +version = "0.1.8" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "40868e7c1d2f0b8d73e4a8c7f0ff63af4f6d19be117e90bd73eb1d62cf831c6b" +checksum = "7694e1cfe791f8d31026952abf09c69ca6f6fa4e1a1229e18988f06a04a12dca" [[package]] name = "time-macros" -version = "0.2.24" +version = "0.2.27" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "30cfb0125f12d9c277f35663a0a33f8c30190f4e4574868a330595412d34ebf3" +checksum = "2e70e4c5a0e0a8a4823ad65dfe1a6930e4f4d756dcd9dd7939022b5e8c501215" dependencies = [ "num-conv", "time-core", @@ -14124,7 +14282,7 @@ checksum = "d9be88c795d8b9f9c4002b3a8f26a6d0876103a6f523b32ea3bac52d8560c17c" dependencies = [ "aho-corasick", "clap", - "derive_builder", + "derive_builder 0.12.0", "esaxx-rs", "getrandom 0.2.17", "indicatif", @@ -14159,7 +14317,7 @@ dependencies = [ "bytes", "io-uring", "libc", - "mio 1.1.1", + "mio 1.2.0", "parking_lot", "pin-project-lite", "signal-hook-registry", @@ -14268,7 +14426,7 @@ dependencies = [ "percent-encoding", "phf 0.11.3", "pin-project-lite", - "postgres-protocol 0.6.10", + "postgres-protocol 0.6.11", "postgres-types 0.2.9", "rand 0.9.0", "socket2 0.5.10", @@ -14447,7 +14605,7 @@ version = "0.19.15" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "1b5bb770da30e5cbfde35a2d7b9b8a2c4b8ef89548a7a6aeab5c9a576e3e7421" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "serde", "serde_spanned", "toml_datetime 0.6.11", @@ -14460,7 +14618,7 @@ version = "0.23.4" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "7211ff1b8f0d3adae1663b7da9ffe396eabe1ca25f0b0bee42b0da29a9ddce93" dependencies = [ - "indexmap 2.11.1", + "indexmap 2.12.0", "toml_datetime 0.7.0", "toml_parser", "winnow 0.7.15", @@ -14468,11 +14626,11 @@ dependencies = [ [[package]] name = "toml_parser" -version = "1.0.10+spec-1.1.0" +version = "1.1.0+spec-1.1.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "7df25b4befd31c4816df190124375d5a20c6b6921e2cad937316de3fccd63420" +checksum = "2334f11ee363607eb04df9b8fc8a13ca1715a72ba8662a26ac285c98aabb4011" dependencies = [ - "winnow 1.0.0", + "winnow 1.0.1", ] [[package]] @@ -14497,7 +14655,6 @@ dependencies = [ "percent-encoding", "pin-project", "prost", - "rustls-native-certs 0.8.3", "rustls-pemfile 2.2.0", "socket2 0.5.10", "tokio", @@ -14510,6 +14667,37 @@ dependencies = [ "webpki-roots 0.26.11", ] +[[package]] +name = "tonic" +version = "0.13.1" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "7e581ba15a835f4d9ea06c55ab1bd4dce26fc53752c69a04aac00703bfb49ba9" +dependencies = [ + "async-trait", + "axum 0.8.4", + "base64 0.22.1", + "bytes", + "h2 0.4.13", + "http 1.4.0", + "http-body 1.0.1", + "http-body-util", + "hyper 1.8.1", + "hyper-timeout", + "hyper-util", + "percent-encoding", + "pin-project", + "prost", + "rustls-native-certs 0.8.3", + "socket2 0.5.10", + "tokio", + "tokio-rustls 0.26.4", + "tokio-stream", + "tower 0.5.3", + "tower-layer", + "tower-service", + "tracing", +] + [[package]] name = "tower" version = "0.4.13" @@ -14538,7 +14726,9 @@ checksum = "ebe5ef63511595f1344e2d5cfa636d973292adc0eec1f0ad45fae9f0851ab1d4" dependencies = [ "futures-core", "futures-util", + "indexmap 2.12.0", "pin-project-lite", + "slab", "sync_wrapper", "tokio", "tokio-util", @@ -14549,13 +14739,12 @@ dependencies = [ [[package]] name = "tower-cookies" -version = "0.10.0" +version = "0.11.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "4fd0118512cf0b3768f7fcccf0bef1ae41d68f2b45edc1e77432b36c97c56c6d" +checksum = "151b5a3e3c45df17466454bb74e9ecedecc955269bdedbf4d150dfa393b55a36" dependencies = [ - "async-trait", - "axum-core 0.4.5", - "cookie 0.18.1", + "axum-core 0.5.6", + "cookie", "futures-util", "http 1.4.0", "parking_lot", @@ -14670,14 +14859,14 @@ dependencies = [ [[package]] name = "tracing-opentelemetry" -version = "0.28.0" +version = "0.31.0" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "97a971f6058498b5c0f1affa23e7ea202057a7301dbff68e968b2d578bcbd053" +checksum = "ddcf5959f39507d0d04d6413119c04f33b623f4f951ebcbdddddfad2d0623a9c" dependencies = [ "js-sys", "once_cell", - "opentelemetry 0.27.1", - "opentelemetry_sdk 0.27.1", + "opentelemetry 0.30.0", + "opentelemetry_sdk 0.30.0", "smallvec", "tracing", "tracing-core", @@ -15040,9 +15229,9 @@ checksum = "7df058c713841ad818f1dc5d3fd88063241cc61f49f5fbea4b951e8cf5a8d71d" [[package]] name = "unicode-segmentation" -version = "1.12.0" +version = "1.13.2" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "f6ccf251212114b54433ec949fd6a7841275f9ada20dddd2f29e9ceea4501493" +checksum = "9629274872b2bfaf8d66f5f15725007f635594914870f65218920345aa11aa8c" [[package]] name = "unicode-width" @@ -15096,7 +15285,7 @@ version = "0.5.1" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "fc1de2c688dc15305988b563c3854064043356019f97a4b46276fe734c4f07ea" dependencies = [ - "crypto-common", + "crypto-common 0.1.7", "subtle", ] @@ -15242,7 +15431,7 @@ checksum = "97599c400fc79925922b58303e98fcb8fa88f573379a08ddb652e72cbd2e70f6" dependencies = [ "bitflags 2.9.4", "encoding_rs", - "indexmap 2.11.1", + "indexmap 2.12.0", "num-bigint", "serde", "thiserror 1.0.69", @@ -15358,6 +15547,7 @@ dependencies = [ "cfg-if", "once_cell", "rustversion", + "serde", "wasm-bindgen-macro", "wasm-bindgen-shared", ] @@ -15462,7 +15652,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909" dependencies = [ "anyhow", - "indexmap 2.11.1", + "indexmap 2.12.0", "wasm-encoder", "wasmparser", ] @@ -15498,7 +15688,7 @@ checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe" dependencies = [ "bitflags 2.9.4", "hashbrown 0.15.5", - "indexmap 2.11.1", + "indexmap 2.12.0", "semver 1.0.27", ] @@ -15584,7 +15774,7 @@ dependencies = [ "cfg_aliases 0.1.1", "codespan-reporting", "document-features", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "naga", "once_cell", @@ -15742,14 +15932,14 @@ dependencies = [ [[package]] name = "windmill" -version = "1.662.0" +version = "1.670.0" dependencies = [ "anyhow", "async-nats", "aws-config", "aws-credential-types", "aws-sdk-sqs", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "chrono", "constant_time_eq 0.3.1", @@ -15759,6 +15949,8 @@ dependencies = [ "git-version", "lazy_static", "once_cell", + "opentelemetry 0.30.0", + "opentelemetry_sdk 0.30.0", "prometheus", "rand 0.9.0", "rdkafka", @@ -15818,9 +16010,9 @@ dependencies = [ [[package]] name = "windmill-alerting" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "serde", "serde_json", @@ -15831,7 +16023,7 @@ dependencies = [ [[package]] name = "windmill-api" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "argon2", @@ -15847,14 +16039,14 @@ dependencies = [ "aws-sdk-config", "aws-sigv4", "aws-smithy-types", - "axum 0.7.9", + "axum 0.8.4", "base32", "base64 0.22.1", "bytes", "chrono", "chrono-tz", "const_format", - "cookie 0.17.0", + "cookie", "cron", "dashmap 6.1.0", "datafusion", @@ -15864,10 +16056,10 @@ dependencies = [ "futures", "git-version", "hex", - "hmac", + "hmac 0.12.1", "http 1.4.0", "hyper 1.8.1", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "jsonwebtoken 8.3.0", "lazy_static", @@ -15972,9 +16164,9 @@ dependencies = [ [[package]] name = "windmill-api-agent-workers" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "http 1.4.0", "hyper 1.8.1", @@ -15995,9 +16187,9 @@ dependencies = [ [[package]] name = "windmill-api-assets" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "serde", "serde_json", @@ -16008,10 +16200,10 @@ dependencies = [ [[package]] name = "windmill-api-auth" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "axum 0.8.4", "chrono", "http 1.4.0", "itertools 0.14.0", @@ -16034,7 +16226,7 @@ dependencies = [ [[package]] name = "windmill-api-client" -version = "1.662.0" +version = "1.669.1" dependencies = [ "reqwest 0.12.28", "serde", @@ -16044,9 +16236,9 @@ dependencies = [ [[package]] name = "windmill-api-configs" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "itertools 0.14.0", "serde", @@ -16061,9 +16253,9 @@ dependencies = [ [[package]] name = "windmill-api-debug" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "chrono", "ed25519-dalek", @@ -16084,10 +16276,10 @@ dependencies = [ [[package]] name = "windmill-api-embeddings" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "axum 0.8.4", "candle-core", "candle-nn", "candle-transformers", @@ -16107,9 +16299,9 @@ dependencies = [ [[package]] name = "windmill-api-flow-conversations" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "serde", "sql-builder", @@ -16123,9 +16315,9 @@ dependencies = [ [[package]] name = "windmill-api-flows" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "hyper 1.8.1", "serde", @@ -16143,9 +16335,9 @@ dependencies = [ [[package]] name = "windmill-api-groups" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "lazy_static", "regex", @@ -16163,9 +16355,9 @@ dependencies = [ [[package]] name = "windmill-api-inputs" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "serde", "serde_json", @@ -16177,17 +16369,20 @@ dependencies = [ [[package]] name = "windmill-api-integration-tests" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-nats", "aws-config", "aws-credential-types", "aws-sdk-sqs", + "axum 0.8.4", "base64 0.22.1", + "futures", "rand 0.9.0", "rdkafka", "reqwest 0.13.1", + "rmcp", "rumqttc", "serde", "serde_json", @@ -16205,10 +16400,10 @@ dependencies = [ [[package]] name = "windmill-api-jobs" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "chrono", "http 1.4.0", @@ -16230,9 +16425,9 @@ dependencies = [ [[package]] name = "windmill-api-npm-proxy" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "flate2", "reqwest 0.13.1", "serde", @@ -16248,18 +16443,19 @@ dependencies = [ [[package]] name = "windmill-api-openapi" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "axum 0.8.4", "http 1.4.0", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "lazy_static", "serde", "serde_json", "serde_yml", "sqlx", + "tracing", "url", "windmill-api-auth", "windmill-common", @@ -16269,9 +16465,9 @@ dependencies = [ [[package]] name = "windmill-api-schedule" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "chrono-tz", "serde", @@ -16289,9 +16485,9 @@ dependencies = [ [[package]] name = "windmill-api-scripts" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "futures", "http 1.4.0", @@ -16319,10 +16515,10 @@ dependencies = [ [[package]] name = "windmill-api-settings" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "bytes", "chrono", @@ -16346,7 +16542,7 @@ dependencies = [ [[package]] name = "windmill-api-sse" -version = "1.662.0" +version = "1.669.1" dependencies = [ "lazy_static", "serde", @@ -16358,10 +16554,10 @@ dependencies = [ [[package]] name = "windmill-api-users" -version = "1.662.0" +version = "1.669.1" dependencies = [ "argon2", - "axum 0.7.9", + "axum 0.8.4", "chrono", "http 1.4.0", "hyper 1.8.1", @@ -16373,6 +16569,7 @@ dependencies = [ "tokio", "tower-cookies", "tracing", + "url", "windmill-api-auth", "windmill-audit", "windmill-common", @@ -16381,9 +16578,9 @@ dependencies = [ [[package]] name = "windmill-api-workers" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "serde", "serde_json", @@ -16395,9 +16592,9 @@ dependencies = [ [[package]] name = "windmill-api-workspaces" -version = "1.662.0" +version = "1.669.1" dependencies = [ - "axum 0.7.9", + "axum 0.8.4", "chrono", "hex", "http 1.4.0", @@ -16420,13 +16617,14 @@ dependencies = [ "windmill-common", "windmill-dep-map", "windmill-git-sync", + "windmill-object-store", "windmill-queue", "windmill-types", ] [[package]] name = "windmill-audit" -version = "1.662.0" +version = "1.669.1" dependencies = [ "chrono", "lazy_static", @@ -16440,10 +16638,10 @@ dependencies = [ [[package]] name = "windmill-autoscaling" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "axum 0.8.4", "k8s-openapi", "kube", "serde", @@ -16459,9 +16657,10 @@ dependencies = [ [[package]] name = "windmill-common" -version = "1.662.0" +version = "1.669.1" dependencies = [ "aes-gcm", + "aho-corasick", "anyhow", "async-recursion", "async-stream", @@ -16473,7 +16672,7 @@ dependencies = [ "aws-sdk-sts", "aws-smithy-types", "aws-smithy-types-convert", - "axum 0.7.9", + "axum 0.8.4", "backon", "base64 0.22.1", "bitflags 2.9.4", @@ -16485,6 +16684,7 @@ dependencies = [ "crc", "cron", "croner", + "dashmap 6.1.0", "datafusion", "equivalent", "futures", @@ -16493,9 +16693,9 @@ dependencies = [ "git-version", "globset", "hex", - "hmac", + "hmac 0.12.1", "hyper 1.8.1", - "indexmap 2.11.1", + "indexmap 2.12.0", "itertools 0.14.0", "jsonwebtoken 8.3.0", "lazy_static", @@ -16504,11 +16704,11 @@ dependencies = [ "native-tls", "once_cell", "openidconnect", - "opentelemetry 0.27.1", + "opentelemetry 0.30.0", "opentelemetry-appender-tracing", - "opentelemetry-otlp", - "opentelemetry-semantic-conventions", - "opentelemetry_sdk 0.27.1", + "opentelemetry-otlp 0.30.0", + "opentelemetry-semantic-conventions 0.30.0", + "opentelemetry_sdk 0.30.0", "pep440_rs", "phf 0.11.3", "pin-project-lite", @@ -16541,7 +16741,7 @@ dependencies = [ "tokio-postgres 0.7.13", "tokio-stream", "tokio-util", - "tonic", + "tonic 0.13.1", "tracing", "tracing-appender", "tracing-opentelemetry", @@ -16559,7 +16759,7 @@ dependencies = [ [[package]] name = "windmill-dep-map" -version = "1.662.0" +version = "1.669.1" dependencies = [ "chrono", "itertools 0.14.0", @@ -16578,7 +16778,7 @@ dependencies = [ [[package]] name = "windmill-git-sync" -version = "1.662.0" +version = "1.669.1" dependencies = [ "regex", "serde", @@ -16593,7 +16793,7 @@ dependencies = [ [[package]] name = "windmill-indexer" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "astral-tokio-tar", @@ -16617,7 +16817,7 @@ dependencies = [ [[package]] name = "windmill-jseval" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "futures", @@ -16634,7 +16834,7 @@ dependencies = [ [[package]] name = "windmill-macros" -version = "1.662.0" +version = "1.669.1" dependencies = [ "itertools 0.14.0", "lazy_static", @@ -16650,7 +16850,7 @@ dependencies = [ [[package]] name = "windmill-mcp" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", @@ -16671,15 +16871,15 @@ dependencies = [ [[package]] name = "windmill-native-triggers" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "backon", "base64 0.22.1", "chrono", - "hmac", + "hmac 0.12.1", "http 1.4.0", "itertools 0.14.0", "lazy_static", @@ -16702,15 +16902,15 @@ dependencies = [ [[package]] name = "windmill-oauth" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-oauth2", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "chrono", "hex", - "hmac", + "hmac 0.12.1", "itertools 0.14.0", "lazy_static", "reqwest 0.12.28", @@ -16726,7 +16926,7 @@ dependencies = [ [[package]] name = "windmill-object-store" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-stream", @@ -16735,7 +16935,7 @@ dependencies = [ "aws-credential-types", "aws-sdk-sts", "aws-smithy-types-convert", - "axum 0.7.9", + "axum 0.8.4", "bytes", "chrono", "datafusion", @@ -16760,7 +16960,7 @@ dependencies = [ [[package]] name = "windmill-operator" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "futures", @@ -16778,7 +16978,7 @@ dependencies = [ [[package]] name = "windmill-parser" -version = "1.662.0" +version = "1.669.1" dependencies = [ "convert_case 0.6.0", "serde", @@ -16787,7 +16987,7 @@ dependencies = [ [[package]] name = "windmill-parser-bash" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "lazy_static", @@ -16799,7 +16999,7 @@ dependencies = [ [[package]] name = "windmill-parser-csharp" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "serde_json", @@ -16811,7 +17011,7 @@ dependencies = [ [[package]] name = "windmill-parser-go" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "gosyn", @@ -16823,7 +17023,7 @@ dependencies = [ [[package]] name = "windmill-parser-graphql" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "lazy_static", @@ -16835,7 +17035,7 @@ dependencies = [ [[package]] name = "windmill-parser-java" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "serde_json", @@ -16847,7 +17047,7 @@ dependencies = [ [[package]] name = "windmill-parser-nu" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "nu-parser", @@ -16858,7 +17058,7 @@ dependencies = [ [[package]] name = "windmill-parser-php" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "itertools 0.14.0", @@ -16869,7 +17069,7 @@ dependencies = [ [[package]] name = "windmill-parser-py" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "itertools 0.14.0", @@ -16881,7 +17081,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-asset" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "rustpython-ast", @@ -16892,7 +17092,7 @@ dependencies = [ [[package]] name = "windmill-parser-py-imports" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-recursion", @@ -16914,7 +17114,7 @@ dependencies = [ [[package]] name = "windmill-parser-ruby" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "lazy_static", @@ -16928,7 +17128,7 @@ dependencies = [ [[package]] name = "windmill-parser-rust" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "convert_case 0.6.0", @@ -16945,7 +17145,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "lazy_static", @@ -16958,7 +17158,7 @@ dependencies = [ [[package]] name = "windmill-parser-sql-asset" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "serde", @@ -16970,7 +17170,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "lazy_static", @@ -16988,7 +17188,7 @@ dependencies = [ [[package]] name = "windmill-parser-ts-asset" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "serde-wasm-bindgen", @@ -17004,7 +17204,7 @@ dependencies = [ [[package]] name = "windmill-parser-wac" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "rustpython-ast", @@ -17020,7 +17220,7 @@ dependencies = [ [[package]] name = "windmill-parser-yaml" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "serde", @@ -17031,11 +17231,11 @@ dependencies = [ [[package]] name = "windmill-queue" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-recursion", - "axum 0.7.9", + "axum 0.8.4", "backon", "chrono", "chrono-tz", @@ -17044,7 +17244,7 @@ dependencies = [ "futures", "futures-core", "hex", - "hmac", + "hmac 0.12.1", "itertools 0.14.0", "lazy_static", "once_cell", @@ -17068,7 +17268,7 @@ dependencies = [ [[package]] name = "windmill-runtime-nativets" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "const_format", @@ -17106,7 +17306,7 @@ dependencies = [ [[package]] name = "windmill-sql-datatype-parser-wasm" -version = "1.662.0" +version = "1.669.1" dependencies = [ "getrandom 0.3.4", "wasm-bindgen", @@ -17117,11 +17317,11 @@ dependencies = [ [[package]] name = "windmill-store" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-recursion", - "axum 0.7.9", + "axum 0.8.4", "chrono", "futures", "http 1.4.0", @@ -17146,10 +17346,11 @@ dependencies = [ [[package]] name = "windmill-test-utils" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", - "axum 0.7.9", + "async-trait", + "axum 0.8.4", "chrono", "futures", "serde", @@ -17169,11 +17370,11 @@ dependencies = [ [[package]] name = "windmill-trigger" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "chrono", "http 1.4.0", "hyper 1.8.1", @@ -17202,11 +17403,11 @@ dependencies = [ [[package]] name = "windmill-trigger-email" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "lazy_static", "regex", @@ -17222,11 +17423,11 @@ dependencies = [ [[package]] name = "windmill-trigger-gcp" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "bytes", "chrono", @@ -17244,7 +17445,7 @@ dependencies = [ "thiserror 2.0.18", "tokio", "tokio-util", - "tonic", + "tonic 0.13.1", "tower-http", "tracing", "windmill-api-auth", @@ -17256,16 +17457,17 @@ dependencies = [ [[package]] name = "windmill-trigger-http" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", + "chrono", "constant_time_eq 0.3.1", "futures", "hex", - "hmac", + "hmac 0.12.1", "http 1.4.0", "hyper 1.8.1", "itertools 0.14.0", @@ -17291,11 +17493,11 @@ dependencies = [ [[package]] name = "windmill-trigger-kafka" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "itertools 0.14.0", "rdkafka", @@ -17314,11 +17516,11 @@ dependencies = [ [[package]] name = "windmill-trigger-mqtt" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "bytes", "itertools 0.14.0", @@ -17338,12 +17540,12 @@ dependencies = [ [[package]] name = "windmill-trigger-nats" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-nats", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "base64 0.22.1", "itertools 0.14.0", "nkeys", @@ -17362,11 +17564,11 @@ dependencies = [ [[package]] name = "windmill-trigger-postgres" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "byteorder", "bytes", "chrono", @@ -17397,7 +17599,7 @@ dependencies = [ [[package]] name = "windmill-trigger-sqs" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", @@ -17406,7 +17608,7 @@ dependencies = [ "aws-sdk-sqs", "aws-sdk-sts", "aws-smithy-types", - "axum 0.7.9", + "axum 0.8.4", "backon", "chrono", "itertools 0.14.0", @@ -17425,11 +17627,11 @@ dependencies = [ [[package]] name = "windmill-trigger-websocket" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-trait", - "axum 0.7.9", + "axum 0.8.4", "futures", "http 1.4.0", "itertools 0.14.0", @@ -17448,7 +17650,7 @@ dependencies = [ [[package]] name = "windmill-types" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "bitflags 2.9.4", @@ -17467,7 +17669,7 @@ dependencies = [ [[package]] name = "windmill-worker" -version = "1.662.0" +version = "1.669.1" dependencies = [ "anyhow", "async-once-cell", @@ -17478,7 +17680,7 @@ dependencies = [ "aws-credential-types", "aws-sdk-bedrockruntime", "aws-smithy-types", - "axum 0.7.9", + "axum 0.8.4", "backon", "base64 0.22.1", "bit-vec 0.6.3", @@ -17495,6 +17697,7 @@ dependencies = [ "gcp_auth", "git-version", "hex", + "hmac 0.12.1", "hudsucker", "hyper-http-proxy", "hyper-tls", @@ -17510,10 +17713,10 @@ dependencies = [ "native-tls", "nix 0.27.1", "once_cell", - "opentelemetry 0.27.1", - "opentelemetry-proto 0.29.0", + "opentelemetry 0.30.0", + "opentelemetry-proto 0.30.0", "oracle", - "pem 3.0.5", + "pem 3.0.6", "pep440_rs", "postgres-native-tls 0.5.1", "process-wrap", @@ -17569,12 +17772,13 @@ dependencies = [ "windmill-runtime-nativets", "windmill-types", "windmill-worker-volumes", + "x509-parser 0.16.0", "yaml-rust", ] [[package]] name = "windmill-worker-volumes" -version = "1.662.0" +version = "1.669.1" dependencies = [ "bytes", "futures", @@ -18163,15 +18367,6 @@ dependencies = [ "memchr", ] -[[package]] -name = "winnow" -version = "0.6.26" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "1e90edd2ac1aa278a5c4599b1d89cf03074b610800f866d4026dc199d7929a28" -dependencies = [ - "memchr", -] - [[package]] name = "winnow" version = "0.7.15" @@ -18183,19 +18378,9 @@ dependencies = [ [[package]] name = "winnow" -version = "1.0.0" +version = "1.0.1" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "a90e88e4667264a994d34e6d1ab2d26d398dcdca8b7f52bec8668957517fc7d8" - -[[package]] -name = "winreg" -version = "0.50.0" -source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "524e57b2c537c0f9b1e69f1965311ec12182b4122e45035b1508cd24d2adadb1" -dependencies = [ - "cfg-if", - "windows-sys 0.48.0", -] +checksum = "09dac053f1cd375980747450bfc7250c264eaae0583872e845c0c7cd578872b5" [[package]] name = "winsafe" @@ -18231,7 +18416,7 @@ checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21" dependencies = [ "anyhow", "heck 0.5.0", - "indexmap 2.11.1", + "indexmap 2.12.0", "prettyplease", "syn 2.0.117", "wasm-metadata", @@ -18262,7 +18447,7 @@ checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2" dependencies = [ "anyhow", "bitflags 2.9.4", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "serde", "serde_derive", @@ -18281,7 +18466,7 @@ checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736" dependencies = [ "anyhow", "id-arena", - "indexmap 2.11.1", + "indexmap 2.12.0", "log", "semver 1.0.27", "serde", @@ -18463,18 +18648,18 @@ dependencies = [ [[package]] name = "zerocopy" -version = "0.8.47" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "efbb2a062be311f2ba113ce66f697a4dc589f85e78a4aea276200804cea0ed87" +checksum = "eed437bf9d6692032087e337407a86f04cd8d6a16a37199ed57949d415bd68e9" dependencies = [ "zerocopy-derive", ] [[package]] name = "zerocopy-derive" -version = "0.8.47" +version = "0.8.48" source = "registry+https://github.com/rust-lang/crates.io-index" -checksum = "0e8bc7269b54418e7aeeef514aa68f8690b8c0489a06b0136e5f57c4c5ccab89" +checksum = "70e3cd084b1788766f53af483dd21f93881ff30d7320490ec3ef7526d203bad4" dependencies = [ "proc-macro2", "quote", @@ -18562,7 +18747,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "c42e33efc22a0650c311c2ef19115ce232583abbe80850bc8b66509ebef02de0" dependencies = [ "crc32fast", - "indexmap 2.11.1", + "indexmap 2.12.0", "memchr", "typed-path", ] @@ -18573,6 +18758,12 @@ version = "0.6.3" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "3be3d40e40a133f9c916ee3f9f4fa2d9d63435b5fbe1bfc6d9dae0aa0ada1513" +[[package]] +name = "zmij" +version = "1.0.21" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "b8848ee67ecc8aedbaf3e4122217aff892639231befc6a1b58d29fff4c2cabaa" + [[package]] name = "zstd" version = "0.13.3" diff --git a/backend/Cargo.toml b/backend/Cargo.toml index 744518e703..4c244e2cf8 100644 --- a/backend/Cargo.toml +++ b/backend/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "windmill" -version = "1.662.0" +version = "1.670.0" authors.workspace = true edition.workspace = true @@ -79,10 +79,10 @@ members = [ "./windmill-test-utils", "./windmill-api-integration-tests", ] -exclude = ["./windmill-duckdb-ffi-internal"] +exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"] [workspace.package] -version = "1.662.0" +version = "1.670.0" authors = ["Ruben Fiszel "] edition = "2021" @@ -260,6 +260,8 @@ windmill-dep-map.workspace = true windmill-test-utils.workspace = true windmill-worker-volumes.workspace = true windmill-types.workspace = true +opentelemetry = { workspace = true } +opentelemetry_sdk = { workspace = true } windmill-trigger.workspace = true windmill-trigger-websocket.workspace = true windmill-trigger-postgres.workspace = true @@ -362,7 +364,7 @@ reqwest-middleware = { version = "^0", features = ["json"] } bitflags = "2.9.4" memchr = "2.7.4" -axum = { version = "^0.7", features = ["multipart", "macros"] } +axum = { version = "^0.8", features = ["multipart", "macros"] } headers = "^0" hyper = { version = "^1", features = ["full"] } hyper-tls = "^0.6" @@ -371,9 +373,9 @@ tokio = { version = "=1.46.1", features = ["full", "tracing", "time"] } tokio-stream = { version = "0.1.17" } tower = "^0" tower-http = { version = "^0.6", features = ["trace", "cors", "catch-panic"] } -tower-cookies = "^0.10" +tower-cookies = "^0.11" #stuck because of swc for now -serde = "=1.0.219" +serde = "=1.0.220" serde_json = { version = "^1", features = ["preserve_order", "raw_value"] } serde_yml = "0.0.12" uuid = { version = "^1", features = ["serde", "v4", "js"] } @@ -386,7 +388,7 @@ tracing = "^0" tracing-subscriber = { version = "^0", features = ["env-filter", "json"] } tracing-appender = "^0" prometheus = { version = "^0", default-features = false } -cookie = { version = "0.17.0" } +cookie = { version = "0.18.0" } phf = { version = "0.11", features = ["macros"] } rust-embed = { version = "^6", features = ["interpolate-folder-path"] } mime_guess = "^2" @@ -415,6 +417,7 @@ time = "^0" serde_urlencoded = "^0" astral-tokio-tar = "^0.5.6" tempfile = "^3" +x509-parser = "^0.16" tokio-util = { version = "=0.7.17", features = ["io"] } json-pointer = "^0" itertools = "^0.14.0" @@ -510,7 +513,7 @@ native-tls = ">=0.2, <0.2.17" # samael will break compilation on MacOS. Use this fork instead to make it work # samael = { git="https://github.com/njaremko/samael", rev="464d015e3ae393e4b5dd00b4d6baa1b617de0dd6", features = ["xmlsec"] } libxml = { version = "=0.3.3" } -samael = { version="0.0.14", features = ["xmlsec"] } +samael = { git="https://github.com/njaremko/samael", rev="f879f1942ec1b34b6d3027ce7e4724ad95d15dfa", features = ["xmlsec"] } gcp_auth = "0.9.0" rust_decimal = { version = "^1", features = ["db-postgres", "serde-float"]} jsonwebtoken = "8.3.0" @@ -566,18 +569,18 @@ flate2 = "^1" http = "^1" async-stream = "^0" -opentelemetry = "0.27.0" -tracing-opentelemetry = "0.28.0" -opentelemetry_sdk = { version = "0.27.1", features = ["rt-tokio"] } -opentelemetry-otlp = { version = "0.27.0", features = ["grpc-tonic", "tls"] } -opentelemetry-appender-tracing = "0.27.0" -opentelemetry-semantic-conventions = { version = "0.27.0", features = ["semconv_experimental"] } -opentelemetry-proto = { version = "0.29.0", features = ["with-serde", "gen-tonic"] } +opentelemetry = "0.30.0" +tracing-opentelemetry = "0.31.0" +opentelemetry_sdk = { version = "0.30.0", features = ["rt-tokio", "testing"] } +opentelemetry-otlp = { version = "0.30.0", features = ["grpc-tonic", "tls"] } +opentelemetry-appender-tracing = "0.30.0" +opentelemetry-semantic-conventions = { version = "0.30.0", features = ["semconv_experimental"] } +opentelemetry-proto = { version = "0.30.0", features = ["with-serde", "gen-tonic"] } prost = "0.13" bollard = "0.18.1" -tonic = { version = "=0.12.3", features = ["tls-native-roots"] } +tonic = { version = "^0.13", features = ["tls-native-roots"] } byteorder = "1.5.0" tikv-jemallocator = { version = "0.5" } @@ -587,7 +590,7 @@ tikv-jemalloc-ctl = { version = "^0.5" } triomphe = "^0" pin-project-lite = "^0" -tantivy = { git="https://github.com/windmill-labs/tantivy", rev="6a24621231202ccd77bec90d8787e2281fb94e4e" } +tantivy = { git="https://github.com/windmill-labs/tantivy", rev="6ae7c70bc603b8e69e27f3240e08bd00a93fb12c" } backon = "1.3.0" diff --git a/backend/ee-repo-ref.txt b/backend/ee-repo-ref.txt index 30331b64b5..07aa65d1f5 100644 --- a/backend/ee-repo-ref.txt +++ b/backend/ee-repo-ref.txt @@ -1 +1 @@ -c04f3851c03758662e4936ff4b6e71bc56dbae7e \ No newline at end of file +e08a87450627bef9013498e40ee93a47bedda7ee diff --git a/backend/migrations/20260323000000_native_trigger_summary.down.sql b/backend/migrations/20260323000000_native_trigger_summary.down.sql new file mode 100644 index 0000000000..7c7e63101d --- /dev/null +++ b/backend/migrations/20260323000000_native_trigger_summary.down.sql @@ -0,0 +1 @@ +ALTER TABLE native_trigger DROP COLUMN IF EXISTS summary; diff --git a/backend/migrations/20260323000000_native_trigger_summary.up.sql b/backend/migrations/20260323000000_native_trigger_summary.up.sql new file mode 100644 index 0000000000..7989a6fcd3 --- /dev/null +++ b/backend/migrations/20260323000000_native_trigger_summary.up.sql @@ -0,0 +1 @@ +ALTER TABLE native_trigger ADD COLUMN summary VARCHAR(1000); diff --git a/backend/migrations/20260324000000_scim_deactivated_user.down.sql b/backend/migrations/20260324000000_scim_deactivated_user.down.sql new file mode 100644 index 0000000000..1af6e77c62 --- /dev/null +++ b/backend/migrations/20260324000000_scim_deactivated_user.down.sql @@ -0,0 +1 @@ +ALTER TABLE password DROP COLUMN IF EXISTS disabled; diff --git a/backend/migrations/20260324000000_scim_deactivated_user.up.sql b/backend/migrations/20260324000000_scim_deactivated_user.up.sql new file mode 100644 index 0000000000..7410481bdd --- /dev/null +++ b/backend/migrations/20260324000000_scim_deactivated_user.up.sql @@ -0,0 +1 @@ +ALTER TABLE password ADD COLUMN disabled BOOLEAN NOT NULL DEFAULT false; diff --git a/backend/migrations/20260325000000_app_bundles_grants.down.sql b/backend/migrations/20260325000000_app_bundles_grants.down.sql new file mode 100644 index 0000000000..92e34705fe --- /dev/null +++ b/backend/migrations/20260325000000_app_bundles_grants.down.sql @@ -0,0 +1,3 @@ +-- Revoke grants for app_bundles table +REVOKE ALL ON app_bundles FROM windmill_user; +REVOKE ALL ON app_bundles FROM windmill_admin; diff --git a/backend/migrations/20260325000000_app_bundles_grants.up.sql b/backend/migrations/20260325000000_app_bundles_grants.up.sql new file mode 100644 index 0000000000..2ad56563db --- /dev/null +++ b/backend/migrations/20260325000000_app_bundles_grants.up.sql @@ -0,0 +1,3 @@ +-- Add grants for app_bundles table +GRANT ALL ON app_bundles TO windmill_user; +GRANT ALL ON app_bundles TO windmill_admin; diff --git a/backend/migrations/20260326000000_trashbin.down.sql b/backend/migrations/20260326000000_trashbin.down.sql new file mode 100644 index 0000000000..11dca69162 --- /dev/null +++ b/backend/migrations/20260326000000_trashbin.down.sql @@ -0,0 +1 @@ +DROP TABLE IF EXISTS trashbin; diff --git a/backend/migrations/20260326000000_trashbin.up.sql b/backend/migrations/20260326000000_trashbin.up.sql new file mode 100644 index 0000000000..a5bfd74294 --- /dev/null +++ b/backend/migrations/20260326000000_trashbin.up.sql @@ -0,0 +1,16 @@ +CREATE TABLE trashbin ( + id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY, + workspace_id VARCHAR(50) NOT NULL REFERENCES workspace(id) ON DELETE CASCADE, + item_kind VARCHAR(50) NOT NULL, + item_path TEXT NOT NULL, + item_data JSONB NOT NULL, + deleted_by VARCHAR(255) NOT NULL, + deleted_at TIMESTAMPTZ NOT NULL DEFAULT now(), + expires_at TIMESTAMPTZ NOT NULL DEFAULT now() + INTERVAL '3 days' +); + +CREATE INDEX idx_trashbin_expires_at ON trashbin(expires_at); +CREATE INDEX idx_trashbin_workspace_kind ON trashbin(workspace_id, item_kind); + +GRANT ALL ON trashbin TO windmill_user; +GRANT ALL ON trashbin TO windmill_admin; diff --git a/backend/migrations/20260326100000_account_scopes.down.sql b/backend/migrations/20260326100000_account_scopes.down.sql new file mode 100644 index 0000000000..ffff82fdd7 --- /dev/null +++ b/backend/migrations/20260326100000_account_scopes.down.sql @@ -0,0 +1 @@ +ALTER TABLE account DROP COLUMN IF EXISTS scopes; diff --git a/backend/migrations/20260326100000_account_scopes.up.sql b/backend/migrations/20260326100000_account_scopes.up.sql new file mode 100644 index 0000000000..7cb30c29a9 --- /dev/null +++ b/backend/migrations/20260326100000_account_scopes.up.sql @@ -0,0 +1 @@ +ALTER TABLE account ADD COLUMN scopes TEXT[]; diff --git a/backend/migrations/20260326200000_service_accounts.down.sql b/backend/migrations/20260326200000_service_accounts.down.sql new file mode 100644 index 0000000000..88a50692ab --- /dev/null +++ b/backend/migrations/20260326200000_service_accounts.down.sql @@ -0,0 +1 @@ +ALTER TABLE usr DROP COLUMN is_service_account; diff --git a/backend/migrations/20260326200000_service_accounts.up.sql b/backend/migrations/20260326200000_service_accounts.up.sql new file mode 100644 index 0000000000..b9e96d0baa --- /dev/null +++ b/backend/migrations/20260326200000_service_accounts.up.sql @@ -0,0 +1 @@ +ALTER TABLE usr ADD COLUMN IF NOT EXISTS is_service_account BOOLEAN NOT NULL DEFAULT FALSE; diff --git a/backend/migrations/20260327000000_email_varchar_255.down.sql b/backend/migrations/20260327000000_email_varchar_255.down.sql new file mode 100644 index 0000000000..b5ff4d22c2 --- /dev/null +++ b/backend/migrations/20260327000000_email_varchar_255.down.sql @@ -0,0 +1,2 @@ +ALTER TABLE magic_link ALTER COLUMN email TYPE VARCHAR(50); +ALTER TABLE schedule ALTER COLUMN email TYPE VARCHAR(50); diff --git a/backend/migrations/20260327000000_email_varchar_255.up.sql b/backend/migrations/20260327000000_email_varchar_255.up.sql new file mode 100644 index 0000000000..95adb957b3 --- /dev/null +++ b/backend/migrations/20260327000000_email_varchar_255.up.sql @@ -0,0 +1,2 @@ +ALTER TABLE magic_link ALTER COLUMN email TYPE VARCHAR(255); +ALTER TABLE schedule ALTER COLUMN email TYPE VARCHAR(255); diff --git a/backend/migrations/20260328000000_trigger_filter_logic.down.sql b/backend/migrations/20260328000000_trigger_filter_logic.down.sql new file mode 100644 index 0000000000..f6beac9bff --- /dev/null +++ b/backend/migrations/20260328000000_trigger_filter_logic.down.sql @@ -0,0 +1,2 @@ +ALTER TABLE kafka_trigger DROP COLUMN filter_logic; +ALTER TABLE websocket_trigger DROP COLUMN filter_logic; diff --git a/backend/migrations/20260328000000_trigger_filter_logic.up.sql b/backend/migrations/20260328000000_trigger_filter_logic.up.sql new file mode 100644 index 0000000000..ecb99bf396 --- /dev/null +++ b/backend/migrations/20260328000000_trigger_filter_logic.up.sql @@ -0,0 +1,2 @@ +ALTER TABLE kafka_trigger ADD COLUMN filter_logic VARCHAR(3) NOT NULL DEFAULT 'and'; +ALTER TABLE websocket_trigger ADD COLUMN filter_logic VARCHAR(3) NOT NULL DEFAULT 'and'; diff --git a/backend/parsers/windmill-parser-wac/src/dag.rs b/backend/parsers/windmill-parser-wac/src/dag.rs index 662f5a06b6..f70bd6ac53 100644 --- a/backend/parsers/windmill-parser-wac/src/dag.rs +++ b/backend/parsers/windmill-parser-wac/src/dag.rs @@ -27,11 +27,15 @@ pub struct DagNode { #[serde(tag = "type")] pub enum DagNodeType { Step { name: String, script: String }, + InlineStep { name: String }, + Sleep { seconds: String }, + WaitForApproval, Branch { condition_source: String }, ParallelStart, ParallelEnd, LoopStart { iter_source: String }, LoopEnd, + Merge, Return, } diff --git a/backend/parsers/windmill-parser-wac/src/python.rs b/backend/parsers/windmill-parser-wac/src/python.rs index 74f0117376..f91067b711 100644 --- a/backend/parsers/windmill-parser-wac/src/python.rs +++ b/backend/parsers/windmill-parser-wac/src/python.rs @@ -37,7 +37,8 @@ impl LineIndex { /// Maps task function name → optional external path (from `@task(path="...")`) type TaskFunctions = HashMap>; -/// First pass: scan top-level `@task async def foo(...)` declarations. +/// First pass: scan top-level `@task async def foo(...)` declarations +/// and `foo = task_script("path")` / `foo = task_flow("path")` assignments. fn collect_task_functions(stmts: &[Stmt]) -> TaskFunctions { let mut tasks = HashMap::new(); for stmt in stmts { @@ -61,6 +62,30 @@ fn collect_task_functions(stmts: &[Stmt]) -> TaskFunctions { } } } + // foo = task_script("path") or foo = task_flow("path") + if let Stmt::Assign(assign) = stmt { + if let Expr::Call(call) = assign.value.as_ref() { + if let Expr::Name(ExprName { id, .. }) = call.func.as_ref() { + if id.as_str() == "task_script" || id.as_str() == "task_flow" { + // Extract the path from the first positional argument + let path = call.args.first().and_then(|arg| { + if let Expr::Constant(c) = arg { + if let rustpython_parser::ast::Constant::Str(s) = &c.value { + return Some(s.to_string()); + } + } + None + }); + // Extract variable name from target + if let Some(Expr::Name(ExprName { id: var_name, .. })) = + assign.targets.first() + { + tasks.insert(var_name.to_string(), path); + } + } + } + } + } } tasks } @@ -88,8 +113,6 @@ struct WacWalker { node_counter: usize, line_index: LineIndex, task_functions: TaskFunctions, - in_try: bool, - in_while: bool, in_nested_func: bool, in_comprehension: bool, } @@ -103,8 +126,6 @@ impl WacWalker { node_counter: 0, line_index: LineIndex::new(source), task_functions, - in_try: false, - in_while: false, in_nested_func: false, in_comprehension: false, } @@ -292,6 +313,9 @@ impl WacWalker { if self.is_task_fn_call(expr) { return true; } + if Self::is_sdk_call(expr) { + return true; + } match expr { Expr::Await(ExprAwait { value, .. }) => self.expr_contains_step(value), Expr::Call(call) => { @@ -307,6 +331,17 @@ impl WacWalker { } } + /// Check if expr is a call to a known SDK function (step, sleep, wait_for_approval) + fn is_sdk_call(expr: &Expr) -> bool { + if let Expr::Call(call) = expr { + if let Expr::Name(ExprName { id, .. }) = call.func.as_ref() { + let name = id.as_str(); + return name == "step" || name == "sleep" || name == "wait_for_approval"; + } + } + false + } + /// Walk a list of statements, returning (first_node_id, last_node_id) fn walk_body(&mut self, body: &[Stmt]) -> Option<(String, String)> { let mut first_id: Option = None; @@ -353,13 +388,17 @@ impl WacWalker { } fn walk_expr_stmt(&mut self, expr: &Expr) -> Option<(String, String)> { - // await task_fn(...) + // await task_fn(...) / await step(...) / await sleep(...) / await wait_for_approval(...) if let Expr::Await(ExprAwait { value, .. }) = expr { // await task_fn(...) if let Expr::Call(call) = value.as_ref() { if self.is_task_fn_call(&Expr::Call(call.clone())) { return self.emit_step(call, expr); } + // Check for SDK-level calls: step(), sleep(), wait_for_approval() + if let Some(result) = self.try_emit_sdk_call(call, expr) { + return Some(result); + } } // await asyncio.gather(task_fn(...), task_fn(...), ...) if Self::is_asyncio_gather_call(value) { @@ -378,17 +417,69 @@ impl WacWalker { None } + /// Try to emit a node for SDK-level calls: step(), sleep(), wait_for_approval() + fn try_emit_sdk_call(&mut self, call: &ExprCall, expr: &Expr) -> Option<(String, String)> { + let callee_name = match call.func.as_ref() { + Expr::Name(ExprName { id, .. }) => Some(id.as_str()), + _ => None, + }?; + + let line = self.line_of_expr(expr); + + match callee_name { + "step" => { + // step("name", fn) — extract the name from the first string argument + let name = call + .args + .first() + .and_then(|arg| { + if let Expr::Constant(c) = arg { + if let rustpython_parser::ast::Constant::Str(s) = &c.value { + return Some(s.to_string()); + } + } + None + }) + .unwrap_or_else(|| "step".to_string()); + let id = self.next_id(); + let node_id = self.add_node(DagNode { + id: id.clone(), + node_type: DagNodeType::InlineStep { name: name.clone() }, + label: name, + line, + }); + Some((node_id.clone(), node_id)) + } + "sleep" => { + let seconds = call + .args + .first() + .map(|arg| Self::expr_to_source(arg)) + .unwrap_or_else(|| "?".to_string()); + let id = self.next_id(); + let node_id = self.add_node(DagNode { + id: id.clone(), + node_type: DagNodeType::Sleep { seconds: seconds.clone() }, + label: format!("sleep({seconds})"), + line, + }); + Some((node_id.clone(), node_id)) + } + "wait_for_approval" => { + let id = self.next_id(); + let node_id = self.add_node(DagNode { + id: id.clone(), + node_type: DagNodeType::WaitForApproval, + label: "wait_for_approval".to_string(), + line, + }); + Some((node_id.clone(), node_id)) + } + _ => None, + } + } + fn emit_step(&mut self, call: &ExprCall, expr: &Expr) -> Option<(String, String)> { - if self.in_try { - self.errors - .push(validation::error_step_in_try(self.line_of_expr(expr))); - return None; - } - if self.in_while { - self.errors - .push(validation::error_step_in_while(self.line_of_expr(expr))); - return None; - } if self.in_nested_func { self.errors.push(validation::error_step_in_nested_function( self.line_of_expr(expr), @@ -416,17 +507,6 @@ impl WacWalker { } fn emit_parallel(&mut self, gather_call: &ExprCall, expr: &Expr) -> Option<(String, String)> { - if self.in_try { - self.errors - .push(validation::error_step_in_try(self.line_of_expr(expr))); - return None; - } - if self.in_while { - self.errors - .push(validation::error_step_in_while(self.line_of_expr(expr))); - return None; - } - let line = self.line_of_expr(expr); let start_id = self.next_id(); let start_node_id = self.add_node(DagNode { @@ -491,8 +571,6 @@ impl WacWalker { line, }); - let merge_id = format!("{branch_id}_merge"); - let mut last_ids = Vec::new(); if let Some((true_first, true_last)) = self.walk_body(&if_stmt.body) { @@ -514,7 +592,17 @@ impl WacWalker { if last_ids.len() == 1 { Some((branch_node_id, last_ids.into_iter().next().unwrap())) } else { - Some((branch_node_id, merge_id)) + let merge_id = format!("{branch_id}_merge"); + let merge_node_id = self.add_node(DagNode { + id: merge_id, + node_type: DagNodeType::Merge, + label: "merge".to_string(), + line, + }); + for last in last_ids { + self.add_edge(&last, &merge_node_id, None); + } + Some((branch_node_id, merge_node_id)) } } @@ -552,11 +640,36 @@ impl WacWalker { } fn walk_while(&mut self, while_stmt: &StmtWhile) -> Option<(String, String)> { - if self.body_contains_step(&while_stmt.body) { - let line = self.line_index.line_of(while_stmt.range.start().to_usize()); - self.errors.push(validation::error_step_in_while(line)); + if !self.body_contains_step(&while_stmt.body) { + return None; } - None + + let line = self.line_index.line_of(while_stmt.range.start().to_usize()); + let condition = Self::expr_to_source(&while_stmt.test); + + let start_id = self.next_id(); + let start_node_id = self.add_node(DagNode { + id: start_id.clone(), + node_type: DagNodeType::LoopStart { iter_source: condition }, + label: "while".to_string(), + line, + }); + + if let Some((body_first, body_last)) = self.walk_body(&while_stmt.body) { + self.add_edge(&start_node_id, &body_first, None); + self.add_edge(&body_last, &start_node_id, Some("next".to_string())); + } + + let end_id = self.next_id(); + let end_node_id = self.add_node(DagNode { + id: end_id.clone(), + node_type: DagNodeType::LoopEnd, + label: "end while".to_string(), + line, + }); + self.add_edge(&start_node_id, &end_node_id, Some("done".to_string())); + + Some((start_node_id, end_node_id)) } fn walk_try(&mut self, try_stmt: &StmtTry) -> Option<(String, String)> { @@ -569,11 +682,17 @@ impl WacWalker { } }); - if has_steps { - let line = self.line_index.line_of(try_stmt.range.start().to_usize()); - self.errors.push(validation::error_step_in_try(line)); + if !has_steps { + return None; } - None + + let line = self.line_index.line_of(try_stmt.range.start().to_usize()); + self.emit_try_catch_branch( + &try_stmt.body, + &try_stmt.handlers, + &try_stmt.finalbody, + line, + ) } fn walk_try_star(&mut self, try_stmt: &StmtTryStar) -> Option<(String, String)> { @@ -586,11 +705,81 @@ impl WacWalker { } }); - if has_steps { - let line = self.line_index.line_of(try_stmt.range.start().to_usize()); - self.errors.push(validation::error_step_in_try(line)); + if !has_steps { + return None; } - None + + let line = self.line_index.line_of(try_stmt.range.start().to_usize()); + self.emit_try_catch_branch( + &try_stmt.body, + &try_stmt.handlers, + &try_stmt.finalbody, + line, + ) + } + + fn emit_try_catch_branch( + &mut self, + try_body: &[Stmt], + handlers: &[rustpython_parser::ast::ExceptHandler], + finally_body: &[Stmt], + line: usize, + ) -> Option<(String, String)> { + let branch_id = self.next_id(); + let branch_node_id = self.add_node(DagNode { + id: branch_id.clone(), + node_type: DagNodeType::Branch { condition_source: "try/except".to_string() }, + label: "try".to_string(), + line, + }); + + let mut last_ids = Vec::new(); + + // Try body + if let Some((try_first, try_last)) = self.walk_body(try_body) { + self.add_edge(&branch_node_id, &try_first, Some("try".to_string())); + last_ids.push(try_last); + } else { + last_ids.push(branch_node_id.clone()); + } + + // Except handlers + for handler in handlers { + match handler { + rustpython_parser::ast::ExceptHandler::ExceptHandler(eh) => { + if let Some((catch_first, catch_last)) = self.walk_body(&eh.body) { + self.add_edge(&branch_node_id, &catch_first, Some("except".to_string())); + last_ids.push(catch_last); + } + } + } + } + + // Finally body — sequential after merge + let merge_last = if last_ids.len() == 1 { + last_ids.into_iter().next().unwrap() + } else { + let merge_id = format!("{branch_id}_merge"); + let merge_node_id = self.add_node(DagNode { + id: merge_id, + node_type: DagNodeType::Merge, + label: "merge".to_string(), + line, + }); + for last in last_ids { + self.add_edge(&last, &merge_node_id, None); + } + merge_node_id + }; + + if !finally_body.is_empty() { + if let Some((finally_first, finally_last)) = self.walk_body(finally_body) { + self.add_edge(&merge_last, &finally_first, None); + return Some((branch_node_id, finally_last)); + } + } + + Some((branch_node_id, merge_last)) } fn walk_return(&mut self, ret: &StmtReturn) -> Option<(String, String)> { diff --git a/backend/parsers/windmill-parser-wac/src/typescript.rs b/backend/parsers/windmill-parser-wac/src/typescript.rs index bd777749ea..87fa7cef01 100644 --- a/backend/parsers/windmill-parser-wac/src/typescript.rs +++ b/backend/parsers/windmill-parser-wac/src/typescript.rs @@ -51,22 +51,29 @@ fn extract_var_name(pat: &Pat) -> Option { } } -/// Check if expr is `task(async fn)` or `task("path", async fn)`. -/// Returns Some(optional_path) if it is a task() call. +/// Check if expr is `task(async fn)`, `task("path", async fn)`, +/// `taskScript("path")`, or `taskFlow("path")`. +/// Returns Some(optional_path) if it is a task/taskScript/taskFlow call. fn extract_task_call_info(expr: &Expr) -> Option> { if let Expr::Call(call) = expr { if let Callee::Expr(callee) = &call.callee { if let Expr::Ident(ident) = callee.as_ref() { - if ident.sym.as_ref() == "task" { + let name = ident.sym.as_ref(); + if name == "task" { // task("f/path", async fn) or task(async fn) if call.args.len() == 2 { - // task("f/path", async fn) let path = extract_string_lit(&call.args[0].expr); return Some(path); } else if call.args.len() == 1 { - // task(async fn) return Some(None); } + } else if name == "taskScript" || name == "taskFlow" { + // taskScript("./helper.ts") or taskFlow("f/my_flow") + if let Some(first_arg) = call.args.first() { + let path = extract_string_lit(&first_arg.expr); + return Some(path); + } + return Some(None); } } } @@ -81,8 +88,6 @@ struct TsWacWalker { node_counter: usize, cm: Lrc, task_functions: TaskFunctions, - in_try: bool, - in_while: bool, in_nested_func: bool, } @@ -95,8 +100,6 @@ impl TsWacWalker { node_counter: 0, cm, task_functions, - in_try: false, - in_while: false, in_nested_func: false, } } @@ -224,6 +227,9 @@ impl TsWacWalker { if self.is_task_call(expr) { return true; } + if Self::is_sdk_call(expr) { + return true; + } match expr { Expr::Await(await_expr) => self.expr_contains_step(&await_expr.arg), Expr::Call(call) => { @@ -237,6 +243,19 @@ impl TsWacWalker { } } + /// Check if expr is a call to a known SDK function (step, sleep, waitForApproval) + fn is_sdk_call(expr: &Expr) -> bool { + if let Expr::Call(call) = expr { + if let Callee::Expr(callee) = &call.callee { + if let Expr::Ident(ident) = callee.as_ref() { + let name = ident.sym.as_ref(); + return name == "step" || name == "sleep" || name == "waitForApproval"; + } + } + } + false + } + fn walk_body(&mut self, stmts: &[Stmt]) -> Option<(String, String)> { let mut first_id: Option = None; let mut prev_id: Option = None; @@ -294,12 +313,16 @@ impl TsWacWalker { } fn walk_expr_stmt(&mut self, expr: &Expr) -> Option<(String, String)> { - // await task_fn(...) + // await task_fn(...) / await step(...) / await sleep(...) / await waitForApproval(...) if let Expr::Await(await_expr) = expr { if let Expr::Call(call) = await_expr.arg.as_ref() { if self.is_task_call(&Expr::Call(call.clone())) { return self.emit_step(call, expr); } + // Check for SDK-level calls: step(), sleep(), waitForApproval() + if let Some(result) = self.try_emit_sdk_call(call, expr) { + return Some(result); + } } // await Promise.all([task_fn(...), ...]) if Self::is_promise_all(&await_expr.arg) { @@ -318,17 +341,70 @@ impl TsWacWalker { None } + /// Try to emit a node for SDK-level calls: step(), sleep(), waitForApproval() + fn try_emit_sdk_call(&mut self, call: &CallExpr, expr: &Expr) -> Option<(String, String)> { + let callee_name = match &call.callee { + Callee::Expr(callee) => match callee.as_ref() { + Expr::Ident(ident) => Some(ident.sym.as_ref().to_string()), + _ => None, + }, + _ => None, + }?; + + let line = self.span_line(expr.span()); + + match callee_name.as_str() { + "step" => { + // step("name", fn) — extract the name from the first string argument + let name = call + .args + .first() + .and_then(|a| extract_string_lit(&a.expr)) + .unwrap_or_else(|| "step".to_string()); + let id = self.next_id(); + let node_id = self.add_node(DagNode { + id: id.clone(), + node_type: DagNodeType::InlineStep { name: name.clone() }, + label: name, + line, + }); + Some((node_id.clone(), node_id)) + } + "sleep" => { + // sleep(N) — extract the duration from the first argument + let seconds = call + .args + .first() + .map(|a| { + self.cm + .span_to_snippet(a.expr.span()) + .unwrap_or_else(|_| "?".to_string()) + }) + .unwrap_or_else(|| "?".to_string()); + let id = self.next_id(); + let node_id = self.add_node(DagNode { + id: id.clone(), + node_type: DagNodeType::Sleep { seconds: seconds.clone() }, + label: format!("sleep({seconds})"), + line, + }); + Some((node_id.clone(), node_id)) + } + "waitForApproval" => { + let id = self.next_id(); + let node_id = self.add_node(DagNode { + id: id.clone(), + node_type: DagNodeType::WaitForApproval, + label: "waitForApproval".to_string(), + line, + }); + Some((node_id.clone(), node_id)) + } + _ => None, + } + } + fn emit_step(&mut self, call: &CallExpr, expr: &Expr) -> Option<(String, String)> { - if self.in_try { - self.errors - .push(validation::error_step_in_catch(self.span_line(expr.span()))); - return None; - } - if self.in_while { - self.errors - .push(validation::error_step_in_while(self.span_line(expr.span()))); - return None; - } if self.in_nested_func { self.errors.push(validation::error_step_in_nested_function( self.span_line(expr.span()), @@ -350,17 +426,6 @@ impl TsWacWalker { } fn emit_parallel(&mut self, promise_call: &CallExpr, expr: &Expr) -> Option<(String, String)> { - if self.in_try { - self.errors - .push(validation::error_step_in_catch(self.span_line(expr.span()))); - return None; - } - if self.in_while { - self.errors - .push(validation::error_step_in_while(self.span_line(expr.span()))); - return None; - } - let line = self.span_line(expr.span()); let start_id = self.next_id(); let start_node_id = self.add_node(DagNode { @@ -457,7 +522,16 @@ impl TsWacWalker { Some((branch_node_id, last_ids.into_iter().next().unwrap())) } else { let merge_id = format!("{branch_id}_merge"); - Some((branch_node_id, merge_id)) + let merge_node_id = self.add_node(DagNode { + id: merge_id, + node_type: DagNodeType::Merge, + label: "merge".to_string(), + line, + }); + for last in last_ids { + self.add_edge(&last, &merge_node_id, None); + } + Some((branch_node_id, merge_node_id)) } } @@ -473,7 +547,7 @@ impl TsWacWalker { return None; } let iter_source = self.expr_to_source(&for_in.right); - self.walk_loop_body_with_iter(&for_in.body, for_in.span, &iter_source) + self.walk_loop_body_with_iter(&for_in.body, for_in.span, &iter_source, "for") } fn walk_for_of(&mut self, for_of: &ForOfStmt) -> Option<(String, String)> { @@ -481,7 +555,7 @@ impl TsWacWalker { return None; } let iter_source = self.expr_to_source(&for_of.right); - self.walk_loop_body_with_iter(&for_of.body, for_of.span, &iter_source) + self.walk_loop_body_with_iter(&for_of.body, for_of.span, &iter_source, "for") } fn walk_loop_body( @@ -490,7 +564,7 @@ impl TsWacWalker { span: swc_common::Span, _label: &str, ) -> Option<(String, String)> { - self.walk_loop_body_with_iter(body, span, "...") + self.walk_loop_body_with_iter(body, span, "...", "for") } fn walk_loop_body_with_iter( @@ -498,13 +572,14 @@ impl TsWacWalker { body: &Stmt, span: swc_common::Span, iter_source: &str, + loop_label: &str, ) -> Option<(String, String)> { let line = self.span_line(span); let start_id = self.next_id(); let start_node_id = self.add_node(DagNode { id: start_id.clone(), node_type: DagNodeType::LoopStart { iter_source: iter_source.to_string() }, - label: "for".to_string(), + label: loop_label.to_string(), line, }); @@ -526,12 +601,11 @@ impl TsWacWalker { } fn walk_while(&mut self, while_stmt: &WhileStmt) -> Option<(String, String)> { - if self.stmt_contains_step(&while_stmt.body) { - self.errors.push(validation::error_step_in_while( - self.span_line(while_stmt.span), - )); + if !self.stmt_contains_step(&while_stmt.body) { + return None; } - None + let condition = self.expr_to_source(&while_stmt.test); + self.walk_loop_body_with_iter(&while_stmt.body, while_stmt.span, &condition, "while") } fn walk_try(&mut self, try_stmt: &TryStmt) -> Option<(String, String)> { @@ -545,12 +619,62 @@ impl TsWacWalker { .as_ref() .map_or(false, |f| self.body_contains_step(&f.stmts)); - if has_steps { - self.errors.push(validation::error_step_in_catch( - self.span_line(try_stmt.span), - )); + if !has_steps { + return None; } - None + + let line = self.span_line(try_stmt.span); + let branch_id = self.next_id(); + let branch_node_id = self.add_node(DagNode { + id: branch_id.clone(), + node_type: DagNodeType::Branch { condition_source: "try/catch".to_string() }, + label: "try".to_string(), + line, + }); + + let mut last_ids = Vec::new(); + + // Try body + if let Some((try_first, try_last)) = self.walk_body(&try_stmt.block.stmts) { + self.add_edge(&branch_node_id, &try_first, Some("try".to_string())); + last_ids.push(try_last); + } else { + last_ids.push(branch_node_id.clone()); + } + + // Catch body + if let Some(handler) = &try_stmt.handler { + if let Some((catch_first, catch_last)) = self.walk_body(&handler.body.stmts) { + self.add_edge(&branch_node_id, &catch_first, Some("catch".to_string())); + last_ids.push(catch_last); + } + } + + // Finally body — sequential after merge + let merge_last = if last_ids.len() == 1 { + last_ids.into_iter().next().unwrap() + } else { + let merge_id = format!("{branch_id}_merge"); + let merge_node_id = self.add_node(DagNode { + id: merge_id, + node_type: DagNodeType::Merge, + label: "merge".to_string(), + line, + }); + for last in last_ids { + self.add_edge(&last, &merge_node_id, None); + } + merge_node_id + }; + + if let Some(finalizer) = &try_stmt.finalizer { + if let Some((finally_first, finally_last)) = self.walk_body(&finalizer.stmts) { + self.add_edge(&merge_last, &finally_first, None); + return Some((branch_node_id, finally_last)); + } + } + + Some((branch_node_id, merge_last)) } fn walk_return(&mut self, ret: &ReturnStmt) -> Option<(String, String)> { @@ -632,6 +756,19 @@ pub fn parse_ts_workflow(code: &str) -> Result> { } } } + // export const main = workflow(async (...) => { ... }) + if let ModuleItem::ModuleDecl(ModuleDecl::ExportDecl(export)) = item { + if let Decl::Var(var_decl) = &export.decl { + for decl in &var_decl.decls { + if let Some(init) = &decl.init { + if let Some(result) = find_workflow_call(init, &cm) { + workflow_body = Some(result); + break; + } + } + } + } + } } let (stmts, params) = workflow_body.ok_or_else(|| { diff --git a/backend/parsers/windmill-parser-wac/src/validation.rs b/backend/parsers/windmill-parser-wac/src/validation.rs index e3a57c6811..a76d2b1c17 100644 --- a/backend/parsers/windmill-parser-wac/src/validation.rs +++ b/backend/parsers/windmill-parser-wac/src/validation.rs @@ -12,23 +12,6 @@ impl std::fmt::Display for CompileError { } } -pub fn error_step_in_try(line: usize) -> CompileError { - CompileError { - message: - "Task calls inside try/except are not allowed. Steps have built-in error handling." - .to_string(), - line, - } -} - -pub fn error_step_in_while(line: usize) -> CompileError { - CompileError { - message: "Task calls inside while loops are not allowed. Use for loops instead." - .to_string(), - line, - } -} - pub fn error_step_in_nested_function(line: usize) -> CompileError { CompileError { message: "Task calls inside nested functions, closures, or lambdas are not allowed." @@ -53,12 +36,3 @@ pub fn error_missing_await(line: usize) -> CompileError { line, } } - -pub fn error_step_in_catch(line: usize) -> CompileError { - CompileError { - message: - "Task calls inside catch blocks are not allowed. Steps have built-in error handling." - .to_string(), - line, - } -} diff --git a/backend/parsers/windmill-parser-wac/tests/python_tests.rs b/backend/parsers/windmill-parser-wac/tests/python_tests.rs index 59f0b59f5c..cf117da9e7 100644 --- a/backend/parsers/windmill-parser-wac/tests/python_tests.rs +++ b/backend/parsers/windmill-parser-wac/tests/python_tests.rs @@ -147,7 +147,7 @@ async def my_etl(items: list): } #[test] -fn test_reject_step_in_try() { +fn test_step_in_try_except() { let code = r#" import asyncio from wmill import workflow, task @@ -155,39 +155,52 @@ from wmill import workflow, task @task async def extract_data(): ... +@task +async def handle_error(): ... + @workflow async def my_etl(): try: await extract_data() except Exception: - pass + await handle_error() "#; - let result = parse_python_workflow(code); - assert!(result.is_err()); - let errors = result.unwrap_err(); - assert!(errors[0].message.contains("try/except")); + let dag = parse_python_workflow(code).expect("should parse try/except"); + // Branch(try/except), extract_data, handle_error, merge = 4 + assert_eq!(dag.nodes.len(), 4); + assert!(matches!(dag.nodes[0].node_type, DagNodeType::Branch { .. })); + assert_eq!(dag.nodes[0].label, "try"); + assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[2].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[3].node_type, DagNodeType::Merge)); } #[test] -fn test_reject_step_in_while() { +fn test_step_in_while() { let code = r#" import asyncio from wmill import workflow, task @task -async def extract_data(): ... +async def poll_status(): ... @workflow async def my_etl(): while True: - await extract_data() + await poll_status() "#; - let result = parse_python_workflow(code); - assert!(result.is_err()); - let errors = result.unwrap_err(); - assert!(errors[0].message.contains("while")); + let dag = parse_python_workflow(code).expect("should parse while loop"); + // LoopStart, poll_status, LoopEnd = 3 + assert_eq!(dag.nodes.len(), 3); + assert!(matches!( + dag.nodes[0].node_type, + DagNodeType::LoopStart { .. } + )); + assert_eq!(dag.nodes[0].label, "while"); + assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[2].node_type, DagNodeType::LoopEnd)); } #[test] @@ -264,3 +277,89 @@ async def my_wf(x: int): _ => panic!("expected Step node"), } } + +#[test] +fn test_task_script_and_task_flow_py() { + let code = r#" +from wmill import workflow, task, task_script, task_flow + +helper = task_script("./helper.py") +pipeline = task_flow("f/etl/pipeline") + +@task() +async def process(x: str) -> str: + return f"processed: {x}" + +@workflow +async def main(x: str): + a = await process(x=x) + b = await helper(a=a) + c = await pipeline(b=b) + return {"a": a, "b": b, "c": c} +"#; + + let dag = parse_python_workflow(code).expect("should parse"); + assert_eq!(dag.nodes.len(), 4); // 3 steps + 1 return + + match &dag.nodes[1].node_type { + DagNodeType::Step { name, script } => { + assert_eq!(name, "helper"); + assert_eq!(script, "./helper.py"); + } + _ => panic!("expected Step node for task_script"), + } + + match &dag.nodes[2].node_type { + DagNodeType::Step { name, script } => { + assert_eq!(name, "pipeline"); + assert_eq!(script, "f/etl/pipeline"); + } + _ => panic!("expected Step node for task_flow"), + } +} + +#[test] +fn test_full_template_with_sdk_calls_py() { + let code = r#" +from wmill import workflow, task, task_script, step, sleep, wait_for_approval, get_resume_urls + +helper = task_script("./helper.py") + +@task() +async def process(x: str) -> str: + return f"processed: {x}" + +@workflow +async def main(x: str): + a = await process(x=x) + b = await helper(a=a) + urls = await step("get_urls", lambda: get_resume_urls()) + await sleep(1) + approval = await wait_for_approval(timeout=3600) + return {"processed": a, "helper_result": b, "approval": approval} +"#; + + let dag = parse_python_workflow(code).expect("should parse"); + // process, helper, step("get_urls"), sleep(1), wait_for_approval, return = 6 + assert_eq!(dag.nodes.len(), 6); + + match &dag.nodes[2].node_type { + DagNodeType::InlineStep { name } => { + assert_eq!(name, "get_urls"); + } + _ => panic!("expected InlineStep node, got {:?}", dag.nodes[2].node_type), + } + + match &dag.nodes[3].node_type { + DagNodeType::Sleep { seconds } => { + assert_eq!(seconds, "1"); + } + _ => panic!("expected Sleep node, got {:?}", dag.nodes[3].node_type), + } + + assert!(matches!( + dag.nodes[4].node_type, + DagNodeType::WaitForApproval + )); + assert!(matches!(dag.nodes[5].node_type, DagNodeType::Return)); +} diff --git a/backend/parsers/windmill-parser-wac/tests/ts_tests.rs b/backend/parsers/windmill-parser-wac/tests/ts_tests.rs index 949f326b90..bb006c6744 100644 --- a/backend/parsers/windmill-parser-wac/tests/ts_tests.rs +++ b/backend/parsers/windmill-parser-wac/tests/ts_tests.rs @@ -129,45 +129,56 @@ export default workflow(async (items: string[]) => { } #[test] -fn test_reject_step_in_try_catch() { +fn test_step_in_try_catch() { let code = r#" import { workflow, task } from "windmill-client"; const extract_data = task(async () => {}); +const handle_error = task(async (e: any) => {}); export default workflow(async () => { try { await extract_data(); } catch (e) { - console.log(e); + await handle_error(e); } }); "#; - let result = parse_ts_workflow(code); - assert!(result.is_err()); - let errors = result.unwrap_err(); - assert!(errors[0].message.contains("catch")); + let dag = parse_ts_workflow(code).expect("should parse try/catch"); + // Branch(try/catch), extract_data, handle_error, merge = 4 + assert_eq!(dag.nodes.len(), 4); + assert!(matches!(dag.nodes[0].node_type, DagNodeType::Branch { .. })); + assert_eq!(dag.nodes[0].label, "try"); + assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[2].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[3].node_type, DagNodeType::Merge)); } #[test] -fn test_reject_step_in_while_ts() { +fn test_step_in_while_ts() { let code = r#" import { workflow, task } from "windmill-client"; -const extract_data = task(async () => {}); +const poll_status = task(async () => {}); export default workflow(async () => { while (true) { - await extract_data(); + await poll_status(); } }); "#; - let result = parse_ts_workflow(code); - assert!(result.is_err()); - let errors = result.unwrap_err(); - assert!(errors[0].message.contains("while")); + let dag = parse_ts_workflow(code).expect("should parse while loop"); + // LoopStart, poll_status, LoopEnd = 3 + assert_eq!(dag.nodes.len(), 3); + assert!(matches!( + dag.nodes[0].node_type, + DagNodeType::LoopStart { .. } + )); + assert_eq!(dag.nodes[0].label, "while"); + assert!(matches!(dag.nodes[1].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[2].node_type, DagNodeType::LoopEnd)); } #[test] @@ -243,3 +254,158 @@ export default workflow(async (x: number) => { _ => panic!("expected Step node"), } } + +#[test] +fn test_task_script_and_task_flow() { + let code = r#" +import { workflow, task, taskScript, taskFlow } from "windmill-client"; + +const helper = taskScript("./helper.ts"); +const pipeline = taskFlow("f/etl/pipeline"); +const process = task(async (x: string) => {}); + +export default workflow(async (x: string) => { + const a = await process(x); + const b = await helper({ a }); + const c = await pipeline({ b }); + return { a, b, c }; +}); +"#; + + let dag = parse_ts_workflow(code).expect("should parse"); + assert_eq!(dag.nodes.len(), 4); // 3 steps + 1 return + + match &dag.nodes[0].node_type { + DagNodeType::Step { name, script } => { + assert_eq!(name, "process"); + assert_eq!(script, "process"); + } + _ => panic!("expected Step node"), + } + + match &dag.nodes[1].node_type { + DagNodeType::Step { name, script } => { + assert_eq!(name, "helper"); + assert_eq!(script, "./helper.ts"); + } + _ => panic!("expected Step node for taskScript"), + } + + match &dag.nodes[2].node_type { + DagNodeType::Step { name, script } => { + assert_eq!(name, "pipeline"); + assert_eq!(script, "f/etl/pipeline"); + } + _ => panic!("expected Step node for taskFlow"), + } +} + +#[test] +fn test_full_template_with_sdk_calls() { + let code = r#" +import { task, taskScript, step, sleep, waitForApproval, getResumeUrls, workflow } from "windmill-client"; + +const helper = taskScript("./helper.ts"); +const process = task(async (x: string): Promise => { + return `processed: ${x}`; +}); + +export const main = workflow(async (x: string) => { + const a = await process(x); + const b = await helper({ a }); + const urls = await step("get_urls", () => getResumeUrls()); + await sleep(1); + const approval = await waitForApproval({ timeout: 3600 }); + return { processed: a, helper_result: b, approval }; +}); +"#; + + let dag = parse_ts_workflow(code).expect("should parse"); + // process, helper, step("get_urls"), sleep(1), waitForApproval, return = 6 + assert_eq!(dag.nodes.len(), 6); + assert_eq!(dag.edges.len(), 5); + + assert!(matches!(dag.nodes[0].node_type, DagNodeType::Step { .. })); + + match &dag.nodes[1].node_type { + DagNodeType::Step { name, script } => { + assert_eq!(name, "helper"); + assert_eq!(script, "./helper.ts"); + } + _ => panic!("expected Step node"), + } + + match &dag.nodes[2].node_type { + DagNodeType::InlineStep { name } => { + assert_eq!(name, "get_urls"); + } + _ => panic!("expected InlineStep node, got {:?}", dag.nodes[2].node_type), + } + + match &dag.nodes[3].node_type { + DagNodeType::Sleep { seconds } => { + assert_eq!(seconds, "1"); + } + _ => panic!("expected Sleep node, got {:?}", dag.nodes[3].node_type), + } + + assert!(matches!( + dag.nodes[4].node_type, + DagNodeType::WaitForApproval + )); + assert!(matches!(dag.nodes[5].node_type, DagNodeType::Return)); +} + +#[test] +fn test_complex_mixed_workflow() { + let code = r#" +import { workflow, task, step, sleep } from "windmill-client"; + +const validate = task(async (data: any) => {}); +const process_csv = task(async (data: any) => {}); +const process_json = task(async (data: any) => {}); +const enrich = task(async (item: any) => {}); +const store = task(async (data: any) => {}); + +export default workflow(async (data: any) => { + const validated = await validate(data); + if (validated.format === "csv") { + const parsed = await process_csv(validated); + for (const row of parsed.rows) { + await enrich(row); + } + } else { + await process_json(validated); + } + await sleep(5); + const ts = await step("timestamp", () => new Date().toISOString()); + await store(validated); + return { done: true }; +}); +"#; + + let dag = parse_ts_workflow(code).expect("should parse"); + + // validate, Branch, process_csv, LoopStart, enrich, LoopEnd, process_json, + // merge, sleep(5), step("timestamp"), store, return = 12 + assert_eq!(dag.nodes.len(), 12); + + assert!(matches!(dag.nodes[0].node_type, DagNodeType::Step { .. })); + assert!(matches!(dag.nodes[1].node_type, DagNodeType::Branch { .. })); + assert!(matches!(dag.nodes[2].node_type, DagNodeType::Step { .. })); // process_csv + assert!(matches!( + dag.nodes[3].node_type, + DagNodeType::LoopStart { .. } + )); + assert!(matches!(dag.nodes[4].node_type, DagNodeType::Step { .. })); // enrich + assert!(matches!(dag.nodes[5].node_type, DagNodeType::LoopEnd)); + assert!(matches!(dag.nodes[6].node_type, DagNodeType::Step { .. })); // process_json + assert!(matches!(dag.nodes[7].node_type, DagNodeType::Merge)); + assert!(matches!(dag.nodes[8].node_type, DagNodeType::Sleep { .. })); + assert!(matches!( + dag.nodes[9].node_type, + DagNodeType::InlineStep { .. } + )); // timestamp + assert!(matches!(dag.nodes[10].node_type, DagNodeType::Step { .. })); // store + assert!(matches!(dag.nodes[11].node_type, DagNodeType::Return)); +} diff --git a/backend/parsers/windmill-parser-wasm/publish-pkgs.sh b/backend/parsers/windmill-parser-wasm/publish-pkgs.sh index 3ac0ceef18..42ae54f683 100755 --- a/backend/parsers/windmill-parser-wasm/publish-pkgs.sh +++ b/backend/parsers/windmill-parser-wasm/publish-pkgs.sh @@ -39,3 +39,6 @@ popd pushd "pkg-py-imports" && npm publish ${args} popd + +pushd "pkg-wac" && npm publish ${args} +popd diff --git a/backend/src/main.rs b/backend/src/main.rs index 17567167a3..2ac71aba57 100644 --- a/backend/src/main.rs +++ b/backend/src/main.rs @@ -36,24 +36,26 @@ use windmill_common::ee_oss::{ use windmill_common::{ agent_workers::AgentConfig, + ai_cache::bump_instance_ai_config_revision, global_settings::{ - APP_WORKSPACED_ROUTE_SETTING, AUDIT_LOG_RETENTION_DAYS_SETTING, BASE_URL_SETTING, - BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING, + AI_CONFIG_SETTING, APP_WORKSPACED_ROUTE_SETTING, AUDIT_LOG_RETENTION_DAYS_SETTING, + BASE_URL_SETTING, BUNFIG_INSTALL_SCOPES_SETTING, CRITICAL_ALERTS_ON_DB_OVERSIZE_SETTING, CRITICAL_ALERTS_ON_TOKEN_EXPIRY_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING, CRITICAL_ERROR_CHANNELS_SETTING, CUSTOM_TAGS_SETTING, DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING, EMAIL_DOMAIN_SETTING, ENV_SETTINGS, EXPOSE_DEBUG_METRICS_SETTING, EXPOSE_METRICS_SETTING, EXTRA_PIP_INDEX_URL_SETTING, - HUB_API_SECRET_SETTING, HUB_BASE_URL_SETTING, INDEXER_SETTING, - INSTANCE_EVENTS_WEBHOOK_SETTING, INSTANCE_PYTHON_VERSION_SETTING, + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, HUB_API_SECRET_SETTING, HUB_BASE_URL_SETTING, + INDEXER_SETTING, INSTANCE_EVENTS_WEBHOOK_SETTING, INSTANCE_PYTHON_VERSION_SETTING, JOB_DEFAULT_TIMEOUT_SECS_SETTING, JOB_ISOLATION_SETTING, JWT_SECRET_SETTING, KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, MAVEN_REPOS_SETTING, MAVEN_SETTINGS_XML_SETTING, MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NO_DEFAULT_MAVEN_SETTING, NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OAUTH_SETTING, OTEL_SETTING, OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING, - POWERSHELL_REPO_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING, - REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING, - RUBY_REPOS_SETTING, SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, SMTP_SETTING, TEAMS_SETTING, - TIMEOUT_WAIT_RESULT_SETTING, UV_INDEX_STRATEGY_SETTING, WORKSPACE_REGISTRIES_SETTING, + POWERSHELL_REPO_URL_SETTING, PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING, + REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RESTART_COORDINATION_SETTING, + RETENTION_PERIOD_SECS_SETTING, RUBY_REPOS_SETTING, SAML_METADATA_SETTING, + SCIM_TOKEN_SETTING, SMTP_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING, + UV_INDEX_STRATEGY_SETTING, WORKSPACE_REGISTRIES_SETTING, }, scripts::ScriptLang, stats_oss::schedule_stats, @@ -66,7 +68,7 @@ use windmill_common::{ is_native_mode_from_env, reload_custom_tags_setting, Connection, HUB_CACHE_DIR, HUB_RT_CACHE_DIR, NATIVE_MODE_RESOLVED, TMP_LOGS_DIR, WINDMILL_DIR, WORKER_GROUP, }, - KillpillSender, DEFAULT_HUB_BASE_URL, METRICS_ENABLED, + KillpillSender, DEFAULT_HUB_BASE_URL, INSTANCE_NAME, METRICS_ENABLED, }; #[cfg(feature = "enterprise")] @@ -97,12 +99,13 @@ use windmill_worker::{ }; use crate::monitor::{ - initial_load, load_keep_job_dir, load_metrics_debug_enabled, load_require_preexisting_user, - load_tag_per_workspace_enabled, load_tag_per_workspace_workspaces, monitor_db, - reload_app_workspaced_route_setting, reload_audit_log_retention_days_setting, - reload_base_url_setting, reload_bunfig_install_scopes_setting, - reload_critical_alert_mute_ui_setting, reload_critical_alerts_on_token_expiry_setting, - reload_critical_error_channels_setting, reload_extra_pip_index_url_setting, + initial_load, load_keep_job_dir, load_metrics_debug_enabled, load_preview_tags_override, + load_require_preexisting_user, load_tag_per_workspace_enabled, + load_tag_per_workspace_workspaces, monitor_db, reload_app_workspaced_route_setting, + reload_audit_log_retention_days_setting, reload_base_url_setting, + reload_bunfig_install_scopes_setting, reload_critical_alert_mute_ui_setting, + reload_critical_alerts_on_token_expiry_setting, reload_critical_error_channels_setting, + reload_extra_pip_index_url_setting, reload_http_route_workspaced_route_setting, reload_hub_api_secret_setting, reload_hub_base_url_setting, reload_instance_events_webhook_setting, reload_job_default_timeout_setting, reload_job_isolation_setting, reload_jwt_secret_setting, reload_license_key, @@ -1098,6 +1101,9 @@ Windmill Community Edition {GIT_VERSION} } let addr = SocketAddr::from((server_bind_address, port)); + let listener = tokio::net::TcpListener::bind(addr) + .await + .context("binding main windmill server")?; let (base_internal_tx, base_internal_rx) = tokio::sync::oneshot::channel::(); @@ -1231,7 +1237,7 @@ Windmill Community Edition {GIT_VERSION} db.clone(), index_reader, log_index_reader, - addr, + listener, server_killpill_rx, base_internal_tx, server_mode, @@ -1737,6 +1743,11 @@ async fn process_notify_event( ); } } + PREVIEW_TAGS_OVERRIDE_SETTING => { + if let Err(e) = load_preview_tags_override(db).await { + tracing::error!("Error loading preview tags override: {e:#}"); + } + } SMTP_SETTING => { reload_smtp_config(db).await; } @@ -1787,7 +1798,8 @@ async fn process_notify_event( reload_otel_tracing_proxy_setting(conn).await; if worker_mode { tracing::info!("OTEL tracing proxy setting changed, restarting worker"); - send_delayed_killpill(tx, 4, "OTEL tracing proxy setting change").await; + spawn_graceful_killpill(tx, db, 10, "OTEL tracing proxy setting change") + .await; } } REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING => { @@ -1795,12 +1807,12 @@ async fn process_notify_event( } EXPOSE_METRICS_SETTING => { tracing::info!("Metrics setting changed, restarting"); - send_delayed_killpill(tx, 40, "metrics setting change").await; + spawn_graceful_killpill(tx, db, 10, "metrics setting change").await; } EMAIL_DOMAIN_SETTING => { tracing::info!("Email domain setting changed"); if server_mode { - send_delayed_killpill(tx, 4, "email domain setting change").await; + spawn_graceful_killpill(tx, db, 10, "email domain setting change").await; } } EXPOSE_DEBUG_METRICS_SETTING => { @@ -1813,21 +1825,42 @@ async fn process_notify_event( tracing::error!(error = %e, "Could not reload app workspaced route setting"); } } + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING => { + if let Err(e) = reload_http_route_workspaced_route_setting(db).await { + tracing::error!(error = %e, "Could not reload http route workspaced route setting"); + } + #[cfg(feature = "http_trigger")] + match windmill_api::triggers::http::refresh_routers(db).await { + Ok((true, _)) => { + tracing::info!( + "Refreshed HTTP routers (http workspaced route setting change)" + ); + } + Err(err) => { + tracing::error!("Error refreshing HTTP routers (http workspaced route setting change): {err:#}"); + } + _ => {} + } + } + AI_CONFIG_SETTING => { + tracing::info!("AI config setting changed, bumping instance AI cache revision"); + bump_instance_ai_config_revision(); + } OTEL_SETTING => { tracing::info!("OTEL setting changed, restarting"); - send_delayed_killpill(tx, 4, "OTEL setting change").await; + spawn_graceful_killpill(tx, db, 10, "OTEL setting change").await; } REQUEST_SIZE_LIMIT_SETTING => { if server_mode { tracing::info!("Request limit size change detected, killing server expecting to be restarted"); - send_delayed_killpill(tx, 4, "request size limit change").await; + spawn_graceful_killpill(tx, db, 10, "request size limit change").await; } } SAML_METADATA_SETTING => { tracing::info!( "SAML metadata change detected, killing server expecting to be restarted" ); - send_delayed_killpill(tx, 0, "SAML metadata change").await; + spawn_graceful_killpill(tx, db, 10, "SAML metadata change").await; } HUB_BASE_URL_SETTING => { if let Err(e) = reload_hub_base_url_setting(conn, server_mode).await { @@ -1876,6 +1909,24 @@ async fn process_notify_event( .unwrap_or(false); tracing::info!("Workspace telemetry setting changed: enabled={}", enabled); } + RESTART_COORDINATION_SETTING => { + // Internal coordination key for staggered restarts, no action needed + } + "plain_emails_telemetry" => { + let enabled = sqlx::query_scalar!( + "SELECT value FROM global_settings WHERE name = 'plain_emails_telemetry'" + ) + .fetch_optional(db) + .await + .ok() + .flatten() + .and_then(|v| v.as_bool()) + .unwrap_or(false); + tracing::info!( + "Plain emails telemetry setting changed: enabled={}", + enabled + ); + } _ => { tracing::info!("Unrecognized Global Setting Change Payload: {:?}", payload); } @@ -2017,14 +2068,145 @@ pub async fn run_workers( Ok(()) } -async fn send_delayed_killpill(tx: &KillpillSender, mut max_delay_secs: u64, context: &str) { - if max_delay_secs == 0 { - max_delay_secs = 1; - } - // Random delay to avoid all servers/workers shutting down simultaneously - let rd_delay = rand::rng().random_range(0..max_delay_secs); - tracing::info!("Scheduling {context} shutdown in {rd_delay}s"); - tokio::time::sleep(Duration::from_secs(rd_delay)).await; +/// Schedule a graceful restart with DB-coordinated staggering. +/// +/// Uses a PostgreSQL advisory lock to serialize restart scheduling across server instances. +/// Each instance records its planned restart time in the `_restart_coordination` global setting; +/// subsequent instances read existing schedules and shift their restart to maintain at least +/// `safety_margin_secs` between consecutive restarts (must exceed the server startup time). +/// +/// Every server waits at least `DRAIN_DELAY_SECS` to let in-flight requests complete. +/// Each subsequent server waits an additional `safety_margin_secs` after the previous one, +/// guaranteeing zero downtime overlap. +/// +/// The DB coordination is done synchronously (fast, ~ms) to reserve our restart slot, +/// then the sleep+kill is spawned in the background so the notification handler is not blocked. +/// +/// Falls back to drain-only delay if DB coordination fails. +async fn spawn_graceful_killpill( + tx: &KillpillSender, + db: &Pool, + safety_margin_secs: u64, + context: &str, +) { + // Minimum delay before any restart to let in-flight requests drain + const DRAIN_DELAY_SECS: u64 = 3; - tx.send(); + let delay = match coordinate_restart_delay(db, safety_margin_secs, DRAIN_DELAY_SECS).await { + Ok(d) => d, + Err(e) => { + tracing::warn!( + "Failed to coordinate restart for {context}: {e:#}, \ + falling back to drain delay of {DRAIN_DELAY_SECS}s" + ); + DRAIN_DELAY_SECS + } + }; + + tracing::info!("Scheduling {context} graceful shutdown in {delay}s"); + let tx = tx.clone(); + tokio::spawn(async move { + tokio::time::sleep(Duration::from_secs(delay)).await; + tx.send(); + }); +} + +/// Coordinate a restart delay with other instances via the DB. +/// +/// Returns the delay (in seconds from now) at which this instance should restart. +/// The first server gets `drain_delay_secs` (to let in-flight requests complete). +/// Each subsequent server is spaced `safety_margin_secs` after the latest scheduled restart. +async fn coordinate_restart_delay( + db: &Pool, + safety_margin_secs: u64, + drain_delay_secs: u64, +) -> anyhow::Result { + const RESTART_LOCK_ID: i64 = 737_483_920; + // Stale threshold: ignore coordination entries older than this + const STALE_THRESHOLD_SECS: i64 = 120; + + let now = chrono::Utc::now(); + + let mut tx = db.begin().await.context("begin restart coordination tx")?; + + // Serialize access across all instances + sqlx::query("SELECT pg_advisory_xact_lock($1)") + .bind(RESTART_LOCK_ID) + .execute(&mut *tx) + .await + .context("acquire restart coordination lock")?; + + // Read existing coordination record + let existing: Option = + sqlx::query_scalar("SELECT value FROM global_settings WHERE name = $1") + .bind(RESTART_COORDINATION_SETTING) + .fetch_optional(&mut *tx) + .await + .context("read restart coordination")?; + + // Parse existing scheduled restarts, filtering out stale entries + // Each entry is (instance_name, restart_at) + let mut scheduled: Vec<(String, chrono::DateTime)> = Vec::new(); + if let Some(val) = &existing { + if let Some(arr) = val.get("restarts").and_then(|v| v.as_array()) { + for entry in arr { + let instance = entry + .get("instance") + .and_then(|v| v.as_str()) + .unwrap_or("unknown") + .to_string(); + if let Some(ts_str) = entry.get("restart_at").and_then(|v| v.as_str()) { + if let Ok(dt) = chrono::DateTime::parse_from_rfc3339(ts_str) { + let dt = dt.with_timezone(&chrono::Utc); + let stale_cutoff = now - chrono::Duration::seconds(STALE_THRESHOLD_SECS); + if dt > stale_cutoff { + scheduled.push((instance, dt)); + } + } + } + } + } + } + + // Find the latest scheduled restart + let latest = scheduled.iter().map(|(_, dt)| *dt).max(); + let earliest_allowed = now + chrono::Duration::seconds(drain_delay_secs as i64); + + // Our restart time: drain_delay from now, or safety_margin after the latest existing restart + let our_restart = match latest { + Some(last) => { + let after_last = last + chrono::Duration::seconds(safety_margin_secs as i64); + // Use whichever is later: drain delay or staggered position + earliest_allowed.max(after_last) + } + None => earliest_allowed, + }; + + // Record our restart time (deduplicate: remove any prior entry for this instance) + scheduled.retain(|(inst, _)| inst != &*INSTANCE_NAME); + scheduled.push((INSTANCE_NAME.clone(), our_restart)); + let new_value = serde_json::json!({ + "restarts": scheduled.iter().map(|(inst, dt)| { + serde_json::json!({ + "instance": inst, + "restart_at": dt.to_rfc3339() + }) + }).collect::>() + }); + + sqlx::query( + "INSERT INTO global_settings (name, value, updated_at) \ + VALUES ($1, $2, now()) \ + ON CONFLICT (name) DO UPDATE SET value = $2, updated_at = now()", + ) + .bind(RESTART_COORDINATION_SETTING) + .bind(&new_value) + .execute(&mut *tx) + .await + .context("write restart coordination")?; + + tx.commit().await.context("commit restart coordination")?; + + let delay = (our_restart - now).num_seconds().max(0) as u64; + Ok(delay) } diff --git a/backend/src/monitor.rs b/backend/src/monitor.rs index 082789ef4f..5e87f3d161 100644 --- a/backend/src/monitor.rs +++ b/backend/src/monitor.rs @@ -62,7 +62,7 @@ use windmill_common::{ KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NPMRC_SETTING, NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OTEL_SETTING, OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING, - POWERSHELL_REPO_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING, + POWERSHELL_REPO_URL_SETTING, PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING, REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING, SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, TIMEOUT_WAIT_RESULT_SETTING, UV_INDEX_STRATEGY_SETTING, @@ -79,8 +79,8 @@ use windmill_common::{ load_periodic_bash_script_interval_from_env, load_whitelist_env_vars_from_env, load_worker_config, reload_custom_tags_setting, store_pull_query, store_suspended_pull_query, Connection, WorkerConfig, DEFAULT_TAGS_PER_WORKSPACE, - DEFAULT_TAGS_WORKSPACES, INDEXER_CONFIG, SCRIPT_TOKEN_EXPIRY, SMTP_CONFIG, WINDMILL_DIR, - WORKER_CONFIG, WORKER_GROUP, + DEFAULT_TAGS_WORKSPACES, INDEXER_CONFIG, PREVIEW_TAGS_OVERRIDE, SCRIPT_TOKEN_EXPIRY, + SMTP_CONFIG, WINDMILL_DIR, WORKER_CONFIG, WORKER_GROUP, }, KillpillSender, AUDIT_LOG_RETENTION_DAYS, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE, CRITICAL_ALERTS_ON_TOKEN_EXPIRY, CRITICAL_ALERT_MUTE_UI_ENABLED, CRITICAL_ERROR_CHANNELS, DB, @@ -88,7 +88,13 @@ use windmill_common::{ MONITOR_LOGS_ON_OBJECT_STORE, OTEL_LOGS_ENABLED, OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED, SERVICE_LOG_RETENTION_SECS, }; -use windmill_common::{client::AuthedClient, global_settings::APP_WORKSPACED_ROUTE_SETTING}; +use windmill_common::{ + client::AuthedClient, + global_settings::{ + APP_WORKSPACED_ROUTE_SETTING, HTTP_ROUTE_WORKSPACED_ROUTE, + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, + }, +}; #[cfg(feature = "parquet")] use windmill_object_store::reload_object_store_setting; use windmill_queue::{cancel_job, get_queued_job_v2, SameWorkerPayload}; @@ -163,6 +169,8 @@ lazy_static::lazy_static! { static ref QUEUE_COUNT_TAGS: Arc>> = Arc::new(RwLock::new(Vec::new())); static ref QUEUE_RUNNING_COUNT_TAGS: Arc>> = Arc::new(RwLock::new(Vec::new())); + static ref OTEL_QUEUE_COUNT_TAGS: Arc>> = Arc::new(RwLock::new(Vec::new())); + static ref OTEL_QUEUE_RUNNING_COUNT_TAGS: Arc>> = Arc::new(RwLock::new(Vec::new())); static ref DISABLE_CONCURRENCY_LIMIT: bool = std::env::var("DISABLE_CONCURRENCY_LIMIT").is_ok_and(|s| s == "true"); //legacy typo @@ -227,6 +235,10 @@ pub async fn initial_load( if let Err(e) = load_tag_per_workspace_workspaces(db).await { tracing::error!("Error loading default tag per workpsace workspaces: {e:#}"); } + + if let Err(e) = load_preview_tags_override(db).await { + tracing::error!("Error loading preview tags override: {e:#}"); + } } if server_mode { @@ -296,6 +308,10 @@ pub async fn initial_load( if let Err(e) = reload_app_workspaced_route_setting(db).await { tracing::error!("Error reloading app workspaced route: {:?}", e) } + + if let Err(e) = reload_http_route_workspaced_route_setting(db).await { + tracing::error!("Error reloading http route workspaced route: {:?}", e) + } } #[cfg(feature = "parquet")] @@ -487,6 +503,16 @@ pub async fn load_tag_per_workspace_workspaces(db: &DB) -> error::Result<()> { Ok(()) } +pub async fn load_preview_tags_override(db: &DB) -> error::Result<()> { + let value = load_value_from_global_settings(db, PREVIEW_TAGS_OVERRIDE_SETTING).await; + + match value { + Ok(Some(serde_json::Value::Bool(t))) => PREVIEW_TAGS_OVERRIDE.store(t, Ordering::Relaxed), + _ => (), + }; + Ok(()) +} + pub async fn reload_critical_alert_mute_ui_setting(conn: &Connection) -> error::Result<()> { if let Ok(Some(serde_json::Value::Bool(t))) = load_value_from_global_settings_with_conn(conn, CRITICAL_ALERT_MUTE_UI_SETTING, true).await @@ -1168,6 +1194,15 @@ pub async fn delete_expired_items(db: &DB) -> () { tracing::error!("Error deleting custom concurrency key: {:?}", e); } } + + match windmill_common::trashbin::delete_expired_trash(db).await { + Ok(count) => { + if count > 0 { + tracing::info!("deleted {} expired trash items", count); + } + } + Err(e) => tracing::error!("Error deleting expired trash items: {}", e.to_string()), + } } pub async fn check_expiring_tokens(db: &DB) { @@ -2353,8 +2388,20 @@ pub async fn expose_queue_metrics(db: &Pool) { } } + let otel_enabled = OTEL_METRICS_ENABLED.load(Ordering::Relaxed); + + if otel_enabled { + for q in OTEL_QUEUE_COUNT_TAGS.read().await.iter() { + if queue_counts.get(q).is_none() { + otel_set_queue_count(q, 0); + } + } + } + #[allow(unused_mut)] let mut tags_to_watch = vec![]; + #[allow(unused_mut)] + let mut otel_tags_to_watch = vec![]; for q in queue_counts { let count = q.1; let tag = q.0; @@ -2366,6 +2413,9 @@ pub async fn expose_queue_metrics(db: &Pool) { tags_to_watch.push(tag.to_string()); } + if otel_enabled { + otel_tags_to_watch.push(tag.to_string()); + } otel_set_queue_count(&tag, count as i64); // save queue_count and delay metrics per tag @@ -2400,9 +2450,13 @@ pub async fn expose_queue_metrics(db: &Pool) { let mut w = QUEUE_COUNT_TAGS.write().await; *w = tags_to_watch; } + if otel_enabled { + let mut w = OTEL_QUEUE_COUNT_TAGS.write().await; + *w = otel_tags_to_watch; + } // Single DB query for running counts, shared by Prometheus and OTel - let otel_running = OTEL_METRICS_ENABLED.load(Ordering::Relaxed); + let otel_running = otel_enabled; #[cfg(feature = "prometheus")] let need_running_counts = metrics_enabled || otel_running; #[cfg(not(feature = "prometheus"))] @@ -2420,8 +2474,18 @@ pub async fn expose_queue_metrics(db: &Pool) { } } + if otel_running { + for q in OTEL_QUEUE_RUNNING_COUNT_TAGS.read().await.iter() { + if queue_running_counts.get(q).is_none() { + otel_set_queue_running_count(q, 0); + } + } + } + #[allow(unused_mut, unused_variables)] let mut running_tags_to_watch: Vec = vec![]; + #[allow(unused_mut, unused_variables)] + let mut otel_running_tags_to_watch: Vec = vec![]; for (tag, count) in &queue_running_counts { #[cfg(feature = "prometheus")] if metrics_enabled { @@ -2432,6 +2496,7 @@ pub async fn expose_queue_metrics(db: &Pool) { if otel_running { otel_set_queue_running_count(tag, *count as i64); + otel_running_tags_to_watch.push(tag.to_string()); } } @@ -2440,6 +2505,10 @@ pub async fn expose_queue_metrics(db: &Pool) { let mut w = QUEUE_RUNNING_COUNT_TAGS.write().await; *w = running_tags_to_watch; } + if otel_running { + let mut w = OTEL_QUEUE_RUNNING_COUNT_TAGS.write().await; + *w = otel_running_tags_to_watch; + } } } @@ -3390,6 +3459,39 @@ pub async fn reload_app_workspaced_route_setting(conn: &DB) -> error::Result<()> Ok(()) } +pub async fn reload_http_route_workspaced_route_setting(conn: &DB) -> error::Result<()> { + let http_route_workspaced_route = + load_value_from_global_settings(conn, HTTP_ROUTE_WORKSPACED_ROUTE_SETTING).await?; + + let ws_route = match http_route_workspaced_route { + Some(serde_json::Value::Bool(ws_route)) => ws_route, + None => false, + _ => { + tracing::error!( + "Expected {} to be a boolean got: {:?}. Defaulting to false", + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, + http_route_workspaced_route + ); + false + } + }; + + let mut l = HTTP_ROUTE_WORKSPACED_ROUTE.write().await; + + if *l != ws_route { + *l = ws_route; + drop(l); + // Bump the HTTP trigger version so the route cache is rebuilt with + // the updated workspaced_route behavior on the next request. + sqlx::query!("SELECT nextval('http_trigger_version_seq')") + .fetch_one(conn) + .await?; + } else { + *l = ws_route; + } + Ok(()) +} + pub async fn reload_critical_alerts_on_db_oversize(conn: &DB) -> error::Result<()> { #[derive(Deserialize)] struct DBOversize { diff --git a/backend/summarized_schema.txt b/backend/summarized_schema.txt index ce0905ac08..481ff43ea7 100644 --- a/backend/summarized_schema.txt +++ b/backend/summarized_schema.txt @@ -33,7 +33,7 @@ workspace_key_kind: cloud ## Tables _sqlx_migrations: version(bigint), description(text), installed_on(ts), success(bool), checksum(bytes), execution_time(bigint) -account: workspace_id(char), id(int), expires_at(ts), refresh_token(char), client(char), refresh_error(text), grant_type(char), cc_client_id(char), cc_client_secret(char), cc_token_url(char), mcp_server_url(text) +account: workspace_id(char), id(int), expires_at(ts), refresh_token(char), client(char), refresh_error(text), grant_type(char), cc_client_id(char), cc_client_secret(char), cc_token_url(char), mcp_server_url(text), scopes(text[]) FK: (workspace_id) -> workspace(id) agent_token_blacklist: token(char), expires_at(ts), blacklisted_at(ts), blacklisted_by(char) ai_agent_memory: workspace_id(char), conversation_id(uuid), step_id(char), messages(jsonb), created_at(ts), updated_at(ts) @@ -151,6 +151,9 @@ script: workspace_id(char), hash(bigint), path(char), parent_hashes(bigint[]), s skip_workspace_diff_tally: workspace_id(char), added_at(ts) sqs_trigger: path(char), queue_url(char), aws_resource_path(char), message_attributes(text[]), script_path(char), is_flow(bool), workspace_id(char), edited_by(char), email(char), edited_at(ts), extra_perms(jsonb), error(text), server_id(char), last_server_ping(ts), aws_auth_resource_type(aws_auth_resource_type), error_handler_path(char), error_handler_args(jsonb), retry(jsonb), mode(trigger_mode) FK: (workspace_id) -> workspace(id) +trashbin: id(bigint), workspace_id(char), item_kind(char), item_path(char), item_data(jsonb), deleted_by(char), deleted_at(ts), expires_at(ts) + FK: (workspace_id) -> workspace(id) + INDEX: idx_trashbin_expires_at (expires_at), idx_trashbin_workspace_kind (workspace_id, item_kind) token: token_hash(char), token_prefix(char), token(char), label(char), expiration(ts), workspace_id(char), owner(char), email(char), super_admin(bool), created_at(ts), last_used_at(ts), scopes(text[]), job(uuid) FK: (workspace_id) -> workspace(id) token_expiry_notification: token_hash(char), expiration(ts) diff --git a/backend/tests/bun_jobs.rs b/backend/tests/bun_jobs.rs index 2edfef8989..7183cf6e10 100644 --- a/backend/tests/bun_jobs.rs +++ b/backend/tests/bun_jobs.rs @@ -891,25 +891,34 @@ mod dedicated_worker_protocol { use std::process::{Command, Stdio}; use windmill_test_utils::{parse_dedicated_worker_line, DedicatedWorkerResult}; use windmill_worker::{ - build_loader, generate_dedicated_worker_wrapper, LoaderMode, BUN_DEDICATED_WORKER_ARGS, - BUN_PATH, NODE_BIN_PATH, + build_loader, compute_ts_codegen, generate_multi_script_wrapper, LoaderMode, TsScriptEntry, + BUN_DEDICATED_WORKER_ARGS, BUN_PATH, NODE_BIN_PATH, }; + const TEST_SCRIPT_PATH: &str = "f/test/script"; + /// Creates test worker files and optionally bundles for Node.js (like production) /// Returns the path to the wrapper file to execute fn create_test_worker_files( dir: &std::path::Path, script: &str, - arg_names: &[&str], bundle_for_node: bool, ) -> std::path::PathBuf { let dir_str = dir.to_str().unwrap(); + // Write main.ts at root (like production single-script) std::fs::write(dir.join("main.ts"), script).unwrap(); + let codegen = compute_ts_codegen(script); + let ext = if bundle_for_node { "js" } else { "ts" }; + let scripts = [TsScriptEntry { + import_name: "main", + original_path: TEST_SCRIPT_PATH, + codegen: &codegen, + }]; + let wrapper = generate_multi_script_wrapper(&scripts, ext); + if bundle_for_node { - // For Node.js: bundle to JavaScript first (like production's build_loader with LoaderMode::Node) - let wrapper = generate_dedicated_worker_wrapper(arg_names, "./main.js", None, None); - std::fs::write(dir.join("wrapper.mjs"), wrapper).unwrap(); + std::fs::write(dir.join("wrapper.mjs"), &wrapper).unwrap(); // Use the exact same build_loader function as production tokio::runtime::Runtime::new() @@ -919,7 +928,7 @@ mod dedicated_worker_protocol { "http://localhost:8000", "test_token", "test-workspace", - "f/test/script", + TEST_SCRIPT_PATH, LoaderMode::Node, &None, )) @@ -945,10 +954,8 @@ mod dedicated_worker_protocol { std::fs::rename(&bundled_path, &output_path).unwrap(); output_path } else { - // For Bun: use TypeScript directly (like production) - let wrapper = generate_dedicated_worker_wrapper(arg_names, "./main.ts", None, None); let wrapper_path = dir.join("wrapper.mjs"); - std::fs::write(&wrapper_path, wrapper).unwrap(); + std::fs::write(&wrapper_path, &wrapper).unwrap(); wrapper_path } } @@ -957,14 +964,12 @@ mod dedicated_worker_protocol { fn run_worker_test( runtime: &str, script: &str, - arg_names: &[&str], jobs: Vec, ) -> Vec> { let temp_dir = tempfile::tempdir().unwrap(); // Create files and get the wrapper path (bundled for node, raw for bun) - let wrapper_path = - create_test_worker_files(temp_dir.path(), script, arg_names, runtime == "node"); + let wrapper_path = create_test_worker_files(temp_dir.path(), script, runtime == "node"); let wrapper_str = wrapper_path.to_str().unwrap(); // Build args matching production behavior @@ -1008,7 +1013,8 @@ mod dedicated_worker_protocol { let mut results = Vec::new(); for job_args in jobs { - writeln!(stdin, "{}", job_args.to_string()).unwrap(); + // Protocol: exec:: + writeln!(stdin, "exec:{}:{}", TEST_SCRIPT_PATH, job_args.to_string()).unwrap(); stdin.flush().unwrap(); let mut response = String::new(); @@ -1043,12 +1049,7 @@ export function main(x: number, y: number): number { return x + y; } "#; - let results = run_worker_test( - "node", - script, - &["x", "y"], - vec![serde_json::json!({"x": 5, "y": 3})], - ); + let results = run_worker_test("node", script, vec![serde_json::json!({"x": 5, "y": 3})]); assert_eq!(results.len(), 1); assert_eq!(results[0], Ok(serde_json::json!(8))); @@ -1062,7 +1063,7 @@ export function main(n: number): number { } "#; let jobs: Vec = (1..=5).map(|i| serde_json::json!({"n": i})).collect(); - let results = run_worker_test("node", script, &["n"], jobs); + let results = run_worker_test("node", script, jobs); assert_eq!(results.len(), 5); for (i, result) in results.iter().enumerate() { @@ -1081,7 +1082,6 @@ export function main(msg: string): never { let results = run_worker_test( "node", script, - &["msg"], vec![serde_json::json!({"msg": "test error"})], ); @@ -1099,12 +1099,7 @@ export function main(x: number, y: number): number { return x + y; } "#; - let results = run_worker_test( - "bun", - script, - &["x", "y"], - vec![serde_json::json!({"x": 5, "y": 3})], - ); + let results = run_worker_test("bun", script, vec![serde_json::json!({"x": 5, "y": 3})]); assert_eq!(results.len(), 1); assert_eq!(results[0], Ok(serde_json::json!(8))); @@ -1118,7 +1113,7 @@ export function main(n: number): number { } "#; let jobs: Vec = (1..=5).map(|i| serde_json::json!({"n": i})).collect(); - let results = run_worker_test("bun", script, &["n"], jobs); + let results = run_worker_test("bun", script, jobs); assert_eq!(results.len(), 5); for (i, result) in results.iter().enumerate() { @@ -1137,7 +1132,6 @@ export function main(msg: string): never { let results = run_worker_test( "bun", script, - &["msg"], vec![serde_json::json!({"msg": "test error"})], ); @@ -1145,6 +1139,721 @@ export function main(msg: string): never { assert!(results[0].is_err()); assert_eq!(results[0], Err("test error".to_string())); } + + // ==================== Multi-Script (Runner Group) Tests ==================== + + /// Job to send to a specific script in a multi-script wrapper + struct MultiScriptJob { + script_path: String, + args: serde_json::Value, + } + + /// Creates a multi-script wrapper with multiple scripts as flat files, returns the wrapper path + fn create_multi_script_worker_files( + dir: &std::path::Path, + scripts: &[(&str, &str)], // (original_path, script_content) + ) -> std::path::PathBuf { + let mut entries_data = Vec::new(); + for (path, content) in scripts { + let safe_name = format!("_wm_{}", path.replace('/', "__")); + std::fs::write(dir.join(format!("{safe_name}.ts")), content).unwrap(); + entries_data.push((safe_name, path.to_string(), compute_ts_codegen(content))); + } + + let entries: Vec> = entries_data + .iter() + .map(|(safe, path, cg)| TsScriptEntry { + import_name: safe.as_str(), + original_path: path.as_str(), + codegen: cg, + }) + .collect(); + + let wrapper = generate_multi_script_wrapper(&entries, "ts"); + let wrapper_path = dir.join("wrapper.mjs"); + std::fs::write(&wrapper_path, &wrapper).unwrap(); + wrapper_path + } + + /// Helper to run a multi-script dedicated worker test + fn run_multi_script_worker_test( + scripts: &[(&str, &str)], + jobs: Vec, + ) -> Vec> { + let temp_dir = tempfile::tempdir().unwrap(); + let wrapper_path = create_multi_script_worker_files(temp_dir.path(), scripts); + let wrapper_str = wrapper_path.to_str().unwrap(); + + let mut cmd_args: Vec<&str> = BUN_DEDICATED_WORKER_ARGS.to_vec(); + cmd_args.push(wrapper_str); + + let mut child = Command::new(BUN_PATH.as_str()) + .args(cmd_args) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .current_dir(temp_dir.path()) + .spawn() + .expect("Failed to spawn worker process"); + + let mut stdin = child.stdin.take().unwrap(); + let stdout = child.stdout.take().unwrap(); + let mut reader = BufReader::new(stdout); + + // Wait for "start" signal + let mut start_line = String::new(); + reader.read_line(&mut start_line).unwrap(); + assert_eq!( + parse_dedicated_worker_line(start_line.trim()), + DedicatedWorkerResult::Start, + "Expected 'start', got: {}", + start_line.trim() + ); + + let mut results = Vec::new(); + + for job in &jobs { + writeln!(stdin, "exec:{}:{}", job.script_path, job.args.to_string()).unwrap(); + stdin.flush().unwrap(); + + let mut response = String::new(); + reader.read_line(&mut response).unwrap(); + + match parse_dedicated_worker_line(response.trim()) { + DedicatedWorkerResult::Success(value) => results.push(Ok(value)), + DedicatedWorkerResult::Error(err) => { + let msg = err["message"] + .as_str() + .unwrap_or("Unknown error") + .to_string(); + results.push(Err(msg)); + } + other => panic!("Unexpected response: {:?}", other), + } + } + + writeln!(stdin, "end").unwrap(); + stdin.flush().unwrap(); + let _ = child.wait().expect("Worker process failed to exit"); + + results + } + + #[test] + fn test_multi_script_routing_basic() { + let script_add = r#" +export function main(a: number, b: number): number { + return a + b; +} +"#; + let script_mul = r#" +export function main(x: number, y: number): number { + return x * y; +} +"#; + let results = run_multi_script_worker_test( + &[("f/math/add", script_add), ("f/math/mul", script_mul)], + vec![ + MultiScriptJob { + script_path: "f/math/add".to_string(), + args: serde_json::json!({"a": 3, "b": 4}), + }, + MultiScriptJob { + script_path: "f/math/mul".to_string(), + args: serde_json::json!({"x": 5, "y": 6}), + }, + // Route back to add + MultiScriptJob { + script_path: "f/math/add".to_string(), + args: serde_json::json!({"a": 10, "b": 20}), + }, + ], + ); + + assert_eq!(results.len(), 3); + assert_eq!(results[0], Ok(serde_json::json!(7))); // 3 + 4 + assert_eq!(results[1], Ok(serde_json::json!(30))); // 5 * 6 + assert_eq!(results[2], Ok(serde_json::json!(30))); // 10 + 20 + } + + #[test] + fn test_multi_script_interleaved_jobs() { + let script_upper = r#" +export function main(s: string): string { + return s.toUpperCase(); +} +"#; + let script_len = r#" +export function main(s: string): number { + return s.length; +} +"#; + let results = run_multi_script_worker_test( + &[("f/str/upper", script_upper), ("f/str/len", script_len)], + vec![ + MultiScriptJob { + script_path: "f/str/upper".to_string(), + args: serde_json::json!({"s": "hello"}), + }, + MultiScriptJob { + script_path: "f/str/len".to_string(), + args: serde_json::json!({"s": "hello"}), + }, + MultiScriptJob { + script_path: "f/str/upper".to_string(), + args: serde_json::json!({"s": "world"}), + }, + MultiScriptJob { + script_path: "f/str/len".to_string(), + args: serde_json::json!({"s": "ab"}), + }, + ], + ); + + assert_eq!(results.len(), 4); + assert_eq!(results[0], Ok(serde_json::json!("HELLO"))); + assert_eq!(results[1], Ok(serde_json::json!(5))); + assert_eq!(results[2], Ok(serde_json::json!("WORLD"))); + assert_eq!(results[3], Ok(serde_json::json!(2))); + } + + #[test] + fn test_multi_script_unknown_path_error() { + let script = r#" +export function main(x: number): number { + return x; +} +"#; + let results = run_multi_script_worker_test( + &[("f/known", script)], + vec![MultiScriptJob { + script_path: "f/unknown".to_string(), + args: serde_json::json!({"x": 1}), + }], + ); + + assert_eq!(results.len(), 1); + assert!(results[0].is_err()); + assert!(results[0] + .as_ref() + .unwrap_err() + .contains("Script not found")); + } + + #[test] + fn test_multi_script_error_doesnt_break_other_scripts() { + let script_ok = r#" +export function main(x: number): number { + return x * 2; +} +"#; + let script_err = r#" +export function main(msg: string): never { + throw new Error(msg); +} +"#; + let results = run_multi_script_worker_test( + &[("f/ok", script_ok), ("f/err", script_err)], + vec![ + MultiScriptJob { + script_path: "f/ok".to_string(), + args: serde_json::json!({"x": 5}), + }, + MultiScriptJob { + script_path: "f/err".to_string(), + args: serde_json::json!({"msg": "boom"}), + }, + // Should still work after error in other script + MultiScriptJob { + script_path: "f/ok".to_string(), + args: serde_json::json!({"x": 10}), + }, + ], + ); + + assert_eq!(results.len(), 3); + assert_eq!(results[0], Ok(serde_json::json!(10))); + assert!(results[1].is_err()); + assert_eq!(results[1], Err("boom".to_string())); + assert_eq!(results[2], Ok(serde_json::json!(20))); + } + + // ==================== exec_preprocess Tests ==================== + + /// Raw protocol command to send to a dedicated worker + enum ProtocolCmd { + Exec { path: String, args: serde_json::Value }, + ExecPreprocess { path: String, args: serde_json::Value }, + } + + /// Run a multi-script worker test with raw protocol commands, returning all protocol lines + fn run_raw_protocol_test( + scripts: &[(&str, &str)], + commands: Vec, + ) -> Vec { + let temp_dir = tempfile::tempdir().unwrap(); + let wrapper_path = create_multi_script_worker_files(temp_dir.path(), scripts); + let wrapper_str = wrapper_path.to_str().unwrap(); + + let mut cmd_args: Vec<&str> = BUN_DEDICATED_WORKER_ARGS.to_vec(); + cmd_args.push(wrapper_str); + + let mut child = Command::new(BUN_PATH.as_str()) + .args(cmd_args) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .current_dir(temp_dir.path()) + .spawn() + .expect("Failed to spawn worker process"); + + let mut stdin = child.stdin.take().unwrap(); + let stdout = child.stdout.take().unwrap(); + let mut reader = BufReader::new(stdout); + + let mut start_line = String::new(); + reader.read_line(&mut start_line).unwrap(); + assert_eq!( + parse_dedicated_worker_line(start_line.trim()), + DedicatedWorkerResult::Start, + ); + + let mut results = Vec::new(); + + for cmd in &commands { + let line = match cmd { + ProtocolCmd::Exec { path, args } => format!("exec:{}:{}", path, args), + ProtocolCmd::ExecPreprocess { path, args } => { + format!("exec_preprocess:{}:{}", path, args) + } + }; + writeln!(stdin, "{}", line).unwrap(); + stdin.flush().unwrap(); + + // exec_preprocess produces 2 response lines (preprocessed_args + success/error) + // exec produces 1 response line (success/error) + let expected_lines = match cmd { + ProtocolCmd::ExecPreprocess { .. } => 2, + ProtocolCmd::Exec { .. } => 1, + }; + + for _ in 0..expected_lines { + let mut response = String::new(); + reader.read_line(&mut response).unwrap(); + let parsed = parse_dedicated_worker_line(response.trim()); + // If it's an error, stop reading more lines for this command + if matches!(parsed, DedicatedWorkerResult::Error(_)) { + results.push(parsed); + break; + } + results.push(parsed); + } + } + + writeln!(stdin, "end").unwrap(); + stdin.flush().unwrap(); + let _ = child.wait().expect("Worker process failed to exit"); + + results + } + + #[test] + fn test_bun_exec_preprocess() { + let script = r#" +export function preprocessor(x: number) { + return { x: x * 10 }; +} +export function main(x: number): number { + return x + 1; +} +"#; + let results = run_raw_protocol_test( + &[("f/test/pre", script)], + vec![ProtocolCmd::ExecPreprocess { + path: "f/test/pre".to_string(), + args: serde_json::json!({"x": 5}), + }], + ); + // Should get preprocessed_args then success + assert_eq!(results.len(), 2); + assert_eq!( + results[0], + DedicatedWorkerResult::PreprocessedArgs(serde_json::json!({"x": 50})) + ); + // main(50) => 51 + assert_eq!( + results[1], + DedicatedWorkerResult::Success(serde_json::json!(51)) + ); + } + + #[test] + fn test_bun_exec_preprocess_missing_preprocessor() { + let script = r#" +export function main(x: number): number { + return x; +} +"#; + let results = run_raw_protocol_test( + &[("f/test/nopre", script)], + vec![ProtocolCmd::ExecPreprocess { + path: "f/test/nopre".to_string(), + args: serde_json::json!({"x": 5}), + }], + ); + assert_eq!(results.len(), 1); + assert!(matches!(results[0], DedicatedWorkerResult::Error(_))); + } + + #[test] + fn test_bun_exec_preprocess_then_exec() { + let script = r#" +export function preprocessor(x: number) { + return { x: x * 2 }; +} +export function main(x: number): number { + return x + 100; +} +"#; + let results = run_raw_protocol_test( + &[("f/test/mixed", script)], + vec![ + ProtocolCmd::ExecPreprocess { + path: "f/test/mixed".to_string(), + args: serde_json::json!({"x": 5}), + }, + ProtocolCmd::Exec { + path: "f/test/mixed".to_string(), + args: serde_json::json!({"x": 7}), + }, + ], + ); + // preprocess: preprocessor(5) => {"x":10}, main(10) => 110 + // exec: main(7) => 107 + assert_eq!(results.len(), 3); + assert_eq!( + results[0], + DedicatedWorkerResult::PreprocessedArgs(serde_json::json!({"x": 10})) + ); + assert_eq!( + results[1], + DedicatedWorkerResult::Success(serde_json::json!(110)) + ); + assert_eq!( + results[2], + DedicatedWorkerResult::Success(serde_json::json!(107)) + ); + } + + // ==================== Argument Transformation Tests ==================== + + #[test] + fn test_bun_date_arg_transformation() { + let script = r#" +export function main(d: Date): string { + return d instanceof Date ? d.toISOString() : typeof d; +} +"#; + let results = run_worker_test( + "bun", + script, + vec![serde_json::json!({"d": "2024-01-15T10:30:00.000Z"})], + ); + assert_eq!(results.len(), 1); + assert_eq!( + results[0], + Ok(serde_json::json!("2024-01-15T10:30:00.000Z")) + ); + } + + #[test] + fn test_bun_null_and_undefined_args() { + let script = r#" +export function main(x?: number): string { + return x === null ? "null" : x === undefined ? "undefined" : String(x); +} +"#; + let results = run_worker_test( + "bun", + script, + vec![ + serde_json::json!({"x": null}), + serde_json::json!({"x": 42}), + serde_json::json!({}), + ], + ); + assert_eq!(results.len(), 3); + assert_eq!(results[0], Ok(serde_json::json!("null"))); + assert_eq!(results[1], Ok(serde_json::json!("42"))); + // Missing arg should be undefined + assert_eq!(results[2], Ok(serde_json::json!("undefined"))); + } +} + +// ============================================================================ +// Deno Dedicated Worker Protocol Tests +// ============================================================================ + +mod dedicated_worker_protocol_deno { + use std::io::{BufRead, BufReader, Write}; + use std::process::{Command, Stdio}; + use windmill_test_utils::{parse_dedicated_worker_line, DedicatedWorkerResult}; + use windmill_worker::{generate_deno_dedicated_worker_wrapper, DENO_PATH}; + + const TEST_SCRIPT_PATH: &str = "f/test/script"; + + fn run_deno_worker_test( + script: &str, + jobs: Vec, + ) -> Vec> { + let temp_dir = tempfile::tempdir().unwrap(); + std::fs::write(temp_dir.path().join("main.ts"), script).unwrap(); + + let wrapper = generate_deno_dedicated_worker_wrapper(script).unwrap(); + std::fs::write(temp_dir.path().join("wrapper.ts"), &wrapper).unwrap(); + + let mut child = Command::new(DENO_PATH.as_str()) + .args([ + "run", + "--no-check", + "--unstable-unsafe-proto", + "--unstable-bare-node-builtins", + "-A", + "wrapper.ts", + ]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .current_dir(temp_dir.path()) + .spawn() + .expect("Failed to spawn deno process"); + + let mut stdin = child.stdin.take().unwrap(); + let stdout = child.stdout.take().unwrap(); + let mut reader = BufReader::new(stdout); + + // Wait for "start" — deno outputs 'start\n' via console.log which adds + // its own newline, producing double newlines. Skip empty lines. + loop { + let mut line = String::new(); + reader.read_line(&mut line).unwrap(); + if line.trim().is_empty() { + continue; + } + assert_eq!( + parse_dedicated_worker_line(line.trim()), + DedicatedWorkerResult::Start, + "Expected 'start', got: {}", + line.trim() + ); + break; + } + + let mut results = Vec::new(); + for job_args in jobs { + writeln!(stdin, "exec:{}:{}", TEST_SCRIPT_PATH, job_args.to_string()).unwrap(); + stdin.flush().unwrap(); + + loop { + let mut response = String::new(); + reader.read_line(&mut response).unwrap(); + let trimmed = response.trim(); + if trimmed.is_empty() { + continue; + } + match parse_dedicated_worker_line(trimmed) { + DedicatedWorkerResult::Success(value) => results.push(Ok(value)), + DedicatedWorkerResult::Error(err) => { + let msg = err["message"] + .as_str() + .unwrap_or("Unknown error") + .to_string(); + results.push(Err(msg)); + } + other => panic!("Unexpected response: {:?}", other), + } + break; + } + } + + writeln!(stdin, "end").unwrap(); + stdin.flush().unwrap(); + let _ = child.wait().expect("Worker process failed to exit"); + results + } + + #[test] + fn test_deno_dedicated_worker_simple() { + let script = r#" +export function main(x: number, y: number): number { + return x + y; +} +"#; + let results = run_deno_worker_test(script, vec![serde_json::json!({"x": 5, "y": 3})]); + assert_eq!(results.len(), 1); + assert_eq!(results[0], Ok(serde_json::json!(8))); + } + + #[test] + fn test_deno_dedicated_worker_multiple_jobs() { + let script = r#" +export function main(n: number): number { + return n * 2; +} +"#; + let jobs: Vec = (1..=5).map(|i| serde_json::json!({"n": i})).collect(); + let results = run_deno_worker_test(script, jobs); + assert_eq!(results.len(), 5); + for (i, result) in results.iter().enumerate() { + assert_eq!(*result, Ok(serde_json::json!(((i + 1) * 2) as i64))); + } + } + + #[test] + fn test_deno_dedicated_worker_error() { + let script = r#" +export function main(msg: string): never { + throw new Error(msg); +} +"#; + let results = run_deno_worker_test(script, vec![serde_json::json!({"msg": "test error"})]); + assert_eq!(results.len(), 1); + assert!(results[0].is_err()); + assert_eq!(results[0], Err("test error".to_string())); + } + + // ==================== exec_preprocess Tests ==================== + + /// Run a raw deno protocol test, reading all output lines per command + fn run_deno_raw_protocol_test( + script: &str, + commands: Vec<(&str, serde_json::Value)>, // ("exec" or "exec_preprocess", args) + ) -> Vec { + let temp_dir = tempfile::tempdir().unwrap(); + std::fs::write(temp_dir.path().join("main.ts"), script).unwrap(); + + let wrapper = generate_deno_dedicated_worker_wrapper(script).unwrap(); + std::fs::write(temp_dir.path().join("wrapper.ts"), &wrapper).unwrap(); + + let mut child = Command::new(DENO_PATH.as_str()) + .args([ + "run", + "--no-check", + "--unstable-unsafe-proto", + "--unstable-bare-node-builtins", + "-A", + "wrapper.ts", + ]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .current_dir(temp_dir.path()) + .spawn() + .expect("Failed to spawn deno process"); + + let mut stdin = child.stdin.take().unwrap(); + let stdout = child.stdout.take().unwrap(); + let mut reader = BufReader::new(stdout); + + // Wait for start, skip empty lines + loop { + let mut line = String::new(); + reader.read_line(&mut line).unwrap(); + if line.trim().is_empty() { + continue; + } + assert_eq!( + parse_dedicated_worker_line(line.trim()), + DedicatedWorkerResult::Start, + ); + break; + } + + let mut results = Vec::new(); + + for (cmd, args) in &commands { + writeln!(stdin, "{}:{}:{}", cmd, TEST_SCRIPT_PATH, args).unwrap(); + stdin.flush().unwrap(); + + let expected_lines = if *cmd == "exec_preprocess" { 2 } else { 1 }; + + for _ in 0..expected_lines { + loop { + let mut response = String::new(); + reader.read_line(&mut response).unwrap(); + if response.trim().is_empty() { + continue; + } + let parsed = parse_dedicated_worker_line(response.trim()); + if matches!(parsed, DedicatedWorkerResult::Error(_)) { + results.push(parsed); + break; + } + results.push(parsed); + break; + } + // If last result was an error, don't read more lines for this command + if matches!(results.last(), Some(DedicatedWorkerResult::Error(_))) { + break; + } + } + } + + writeln!(stdin, "end").unwrap(); + stdin.flush().unwrap(); + let _ = child.wait().expect("Worker process failed to exit"); + + results + } + + #[test] + fn test_deno_exec_preprocess() { + let script = r#" +export function preprocessor(x: number) { + return { x: x * 10 }; +} +export function main(x: number): number { + return x + 1; +} +"#; + let results = run_deno_raw_protocol_test( + script, + vec![("exec_preprocess", serde_json::json!({"x": 5}))], + ); + assert_eq!(results.len(), 2); + assert_eq!( + results[0], + DedicatedWorkerResult::PreprocessedArgs(serde_json::json!({"x": 50})) + ); + assert_eq!( + results[1], + DedicatedWorkerResult::Success(serde_json::json!(51)) + ); + } + + // Note: no "missing preprocessor" test for Deno because the wrapper only generates + // the exec_preprocess handler when the script actually has a preprocessor function. + // Without one, exec_preprocess messages are unrecognized (by design — Rust never sends them). + + // ==================== Argument Transformation Tests ==================== + + #[test] + fn test_deno_date_arg_transformation() { + let script = r#" +export function main(d: Date): string { + return d instanceof Date ? d.toISOString() : typeof d; +} +"#; + let results = run_deno_worker_test( + script, + vec![serde_json::json!({"d": "2024-01-15T10:30:00.000Z"})], + ); + assert_eq!(results.len(), 1); + assert_eq!( + results[0], + Ok(serde_json::json!("2024-01-15T10:30:00.000Z")) + ); + } } // ============================================================================ diff --git a/backend/tests/otel.rs b/backend/tests/otel.rs new file mode 100644 index 0000000000..dd56cbf425 --- /dev/null +++ b/backend/tests/otel.rs @@ -0,0 +1,504 @@ +//! E2E tests for OpenTelemetry integration. +//! +//! Verify that metrics are recorded with correct names/values/attributes and +//! spans are created with correct trace IDs, attributes, and status codes. +//! +//! Run with: cargo test --features enterprise,private,otel --test otel -- --test-threads=1 + +#![cfg(all(feature = "otel", feature = "enterprise"))] + +use std::sync::{atomic::Ordering, Arc}; + +use opentelemetry::global; +use opentelemetry::trace::TracerProvider as _; +use opentelemetry_sdk::{ + metrics::{InMemoryMetricExporter, PeriodicReader, SdkMeterProvider}, + trace::{InMemorySpanExporter, SdkTracerProvider, SimpleSpanProcessor}, +}; +use windmill_common::otel_ee::*; +use windmill_common::{OTEL_METRICS_ENABLED, OTEL_TRACING_ENABLED}; + +// ── Global test infrastructure ────────────────────────────────────────── + +struct OtelTestState { + metric_exporter: InMemoryMetricExporter, + span_exporter: InMemorySpanExporter, + meter_provider: SdkMeterProvider, +} + +static STATE: tokio::sync::OnceCell> = tokio::sync::OnceCell::const_new(); + +async fn ensure_setup() -> Arc { + STATE + .get_or_init(|| async { + // Metrics: InMemoryMetricExporter + PeriodicReader (needs async tokio context) + let metric_exporter = InMemoryMetricExporter::default(); + let reader = PeriodicReader::builder(metric_exporter.clone()).build(); + let meter_provider = SdkMeterProvider::builder().with_reader(reader).build(); + global::set_meter_provider(meter_provider.clone()); + OTEL_METRICS_ENABLED.store(true, Ordering::SeqCst); + + // Tracing: InMemorySpanExporter + SimpleSpanProcessor + let span_exporter = InMemorySpanExporter::default(); + let tracer_provider = SdkTracerProvider::builder() + .with_span_processor(SimpleSpanProcessor::new(span_exporter.clone())) + .build(); + let tracer = tracer_provider.tracer("windmill"); + *TRACER.write().unwrap() = Some(tracer); + OTEL_TRACING_ENABLED.store(true, Ordering::SeqCst); + + Arc::new(OtelTestState { metric_exporter, span_exporter, meter_provider }) + }) + .await + .clone() +} + +// ── Metric helper: flush + collect ────────────────────────────────────── + +fn flush_and_get_metrics( + state: &OtelTestState, +) -> Vec { + state.meter_provider.force_flush().expect("flush failed"); + state + .metric_exporter + .get_finished_metrics() + .expect("get_finished_metrics failed") +} + +fn find_metric<'a>( + all: &'a [opentelemetry_sdk::metrics::data::ResourceMetrics], + name: &str, +) -> Option<&'a opentelemetry_sdk::metrics::data::Metric> { + all.iter() + .flat_map(|rm| rm.scope_metrics()) + .flat_map(|sm| sm.metrics()) + .find(|m| m.name() == name) +} + +fn metric_names(all: &[opentelemetry_sdk::metrics::data::ResourceMetrics]) -> Vec { + all.iter() + .flat_map(|rm| rm.scope_metrics()) + .flat_map(|sm| sm.metrics()) + .map(|m| m.name().to_string()) + .collect() +} + +// ── Counter value helpers ─────────────────────────────────────────────── + +fn sum_u64_value(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option { + use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData}; + match metric.data() { + AggregatedMetrics::U64(MetricData::Sum(sum)) => { + Some(sum.data_points().map(|dp| dp.value()).sum()) + } + _ => None, + } +} + +fn gauge_i64_values( + metric: &opentelemetry_sdk::metrics::data::Metric, +) -> Vec<(Vec, i64)> { + use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData}; + match metric.data() { + AggregatedMetrics::I64(MetricData::Gauge(gauge)) => gauge + .data_points() + .map(|dp| (dp.attributes().cloned().collect(), dp.value())) + .collect(), + _ => panic!("expected I64 Gauge metric"), + } +} + +fn gauge_f64_value(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option { + use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData}; + match metric.data() { + AggregatedMetrics::F64(MetricData::Gauge(gauge)) => { + gauge.data_points().next().map(|dp| dp.value()) + } + _ => None, + } +} + +fn histogram_f64_count(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option { + use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData}; + match metric.data() { + AggregatedMetrics::F64(MetricData::Histogram(hist)) => { + Some(hist.data_points().map(|dp| dp.count()).sum()) + } + _ => None, + } +} + +fn histogram_f64_sum(metric: &opentelemetry_sdk::metrics::data::Metric) -> Option { + use opentelemetry_sdk::metrics::data::{AggregatedMetrics, MetricData}; + match metric.data() { + AggregatedMetrics::F64(MetricData::Histogram(hist)) => { + Some(hist.data_points().map(|dp| dp.sum()).sum()) + } + _ => None, + } +} + +// ═══════════════════════════════════════════════════════════════════════ +// METRICS E2E TEST +// +// All metric assertions live in one test function because the PeriodicReader's +// background task is tied to the tokio runtime that created it. Separate +// #[tokio::test] functions each get their own runtime, and the reader becomes +// disconnected after the first test's runtime is dropped. +// ═══════════════════════════════════════════════════════════════════════ + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_all_metrics_e2e() { + let state = ensure_setup().await; + + // ── Counters ──────────────────────────────────────────────────── + + otel_incr_queue_push_count(); + otel_incr_queue_push_count(); + otel_incr_queue_push_count(); + otel_incr_queue_delete_count(); + otel_incr_queue_pull_count(); + otel_incr_zombie_restart_count(7); + otel_incr_zombie_delete_count(3); + otel_incr_worker_execution_count("bun"); + otel_incr_worker_execution_count("bun"); + otel_incr_worker_execution_failed("go"); + otel_incr_worker_started(); + + // ── Gauges ────────────────────────────────────────────────────── + + otel_set_queue_count("python3", 42); + otel_set_queue_running_count("deno", 5); + otel_set_worker_busy("worker-test-1", 1); + otel_set_db_pool(5, 10, 20); + otel_set_health_db_latency(2.5); + otel_set_worker_uptime("w-uptime", 3600.0); + otel_set_health_status_phase("healthy"); + otel_set_health_db_unresponsive(true); + + // ── Histograms ────────────────────────────────────────────────── + + otel_record_worker_execution_duration("python3", 1.5); + otel_record_worker_execution_duration("python3", 2.5); + otel_record_worker_pull_duration("w1", true, 0.05); + otel_record_worker_pull_duration("w1", false, 0.01); + + // ── Flush and collect ─────────────────────────────────────────── + + let metrics = flush_and_get_metrics(&state); + let names = metric_names(&metrics); + + // ── Verify all 20 metric names are present ────────────────────── + + let expected = [ + "windmill.queue.push_count", + "windmill.queue.delete_count", + "windmill.queue.pull_count", + "windmill.queue.zombie_restart_count", + "windmill.queue.zombie_delete_count", + "windmill.queue.count", + "windmill.queue.running_count", + "windmill.worker.execution_count", + "windmill.worker.execution_duration", + "windmill.worker.busy", + "windmill.worker.pull_duration", + "windmill.worker.execution_failed", + "windmill.db.pool.active", + "windmill.db.pool.idle", + "windmill.db.pool.max", + "windmill.health.db_latency", + "windmill.worker.started", + "windmill.worker.uptime", + "windmill.health.status", + "windmill.health.db_unresponsive", + ]; + for name in expected { + assert!( + names.iter().any(|n| n == name), + "metric '{}' not found in {:?}", + name, + names + ); + } + + // ── Counter values ────────────────────────────────────────────── + + let m = find_metric(&metrics, "windmill.queue.push_count").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 3, "push_count should be >= 3"); + + let m = find_metric(&metrics, "windmill.queue.delete_count").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 1); + + let m = find_metric(&metrics, "windmill.queue.pull_count").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 1); + + let m = find_metric(&metrics, "windmill.queue.zombie_restart_count").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 7); + + let m = find_metric(&metrics, "windmill.queue.zombie_delete_count").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 3); + + let m = find_metric(&metrics, "windmill.worker.execution_count").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 2); + + let m = find_metric(&metrics, "windmill.worker.execution_failed").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 1); + + let m = find_metric(&metrics, "windmill.worker.started").unwrap(); + assert!(sum_u64_value(m).unwrap() >= 1); + + // ── Gauge values ──────────────────────────────────────────────── + + let m = find_metric(&metrics, "windmill.queue.count").unwrap(); + let values = gauge_i64_values(m); + let dp = values + .iter() + .find(|(attrs, _)| { + attrs + .iter() + .any(|kv| kv.key.as_str() == "tag" && kv.value.as_str() == "python3") + }) + .expect("queue.count data point with tag=python3 not found"); + assert_eq!(dp.1, 42); + + let m = find_metric(&metrics, "windmill.queue.running_count").unwrap(); + let values = gauge_i64_values(m); + let dp = values + .iter() + .find(|(attrs, _)| { + attrs + .iter() + .any(|kv| kv.key.as_str() == "tag" && kv.value.as_str() == "deno") + }) + .expect("running_count data point with tag=deno not found"); + assert_eq!(dp.1, 5); + + let m = find_metric(&metrics, "windmill.worker.busy").unwrap(); + let values = gauge_i64_values(m); + let dp = values + .iter() + .find(|(attrs, _)| { + attrs + .iter() + .any(|kv| kv.key.as_str() == "worker" && kv.value.as_str() == "worker-test-1") + }) + .expect("worker.busy data point with worker=worker-test-1 not found"); + assert_eq!(dp.1, 1); + + let m = find_metric(&metrics, "windmill.db.pool.active").unwrap(); + assert_eq!(gauge_i64_values(m)[0].1, 5); + let m = find_metric(&metrics, "windmill.db.pool.idle").unwrap(); + assert_eq!(gauge_i64_values(m)[0].1, 10); + let m = find_metric(&metrics, "windmill.db.pool.max").unwrap(); + assert_eq!(gauge_i64_values(m)[0].1, 20); + + let m = find_metric(&metrics, "windmill.health.db_latency").unwrap(); + assert!((gauge_f64_value(m).unwrap() - 2.5).abs() < f64::EPSILON); + + let m = find_metric(&metrics, "windmill.worker.uptime").unwrap(); + assert!((gauge_f64_value(m).unwrap() - 3600.0).abs() < f64::EPSILON); + + let m = find_metric(&metrics, "windmill.health.db_unresponsive").unwrap(); + assert_eq!(gauge_i64_values(m)[0].1, 1); + + // ── Health status phase (all 3 phases) ────────────────────────── + + let m = find_metric(&metrics, "windmill.health.status").unwrap(); + let values = gauge_i64_values(m); + let healthy = values + .iter() + .find(|(attrs, _)| { + attrs + .iter() + .any(|kv| kv.key.as_str() == "phase" && kv.value.as_str() == "healthy") + }) + .expect("phase=healthy"); + let degraded = values + .iter() + .find(|(attrs, _)| { + attrs + .iter() + .any(|kv| kv.key.as_str() == "phase" && kv.value.as_str() == "degraded") + }) + .expect("phase=degraded"); + let unhealthy = values + .iter() + .find(|(attrs, _)| { + attrs + .iter() + .any(|kv| kv.key.as_str() == "phase" && kv.value.as_str() == "unhealthy") + }) + .expect("phase=unhealthy"); + assert_eq!(healthy.1, 1); + assert_eq!(degraded.1, 0); + assert_eq!(unhealthy.1, 0); + + // ── Histogram values ──────────────────────────────────────────── + + let m = find_metric(&metrics, "windmill.worker.execution_duration").unwrap(); + assert!(histogram_f64_count(m).unwrap() >= 2); + assert!(histogram_f64_sum(m).unwrap() >= 4.0); + + let m = find_metric(&metrics, "windmill.worker.pull_duration").unwrap(); + assert!(histogram_f64_count(m).unwrap() >= 2); +} + +// ═══════════════════════════════════════════════════════════════════════ +// SPAN E2E TESTS +// ═══════════════════════════════════════════════════════════════════════ + +fn make_test_job(id: uuid::Uuid, parent: Option) -> windmill_queue::MiniPulledJob { + use windmill_types::jobs::JobKind; + let mut job = windmill_queue::MiniPulledJob::new_inline( + "test-workspace".to_string(), + None, + "test-user".to_string(), + "u/test-user".to_string(), + "test@example.com".to_string(), + Some("f/test/script".to_string()), + JobKind::Script, + None, + "deno".to_string(), + None, + ); + job.id = id; + job.parent_job = parent; + job.started_at = Some(chrono::Utc::now()); + job +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_root_job_span_created_on_success() { + let state = ensure_setup().await; + state.span_exporter.reset(); + + let job_id = uuid::Uuid::new_v4(); + let job = make_test_job(job_id, None); + windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true); + + let spans = state.span_exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|s| s.name == "full_job") + .expect("full_job span not found"); + + assert_eq!(span.status, opentelemetry::trace::Status::Ok,); + + // Verify attributes + let attrs: Vec<_> = span.attributes.iter().map(|kv| kv.key.as_str()).collect(); + assert!(attrs.contains(&"job_id"), "missing job_id attribute"); + assert!( + attrs.contains(&"workspace_id"), + "missing workspace_id attribute" + ); + assert!( + attrs.contains(&"script_path"), + "missing script_path attribute" + ); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_root_job_span_error_on_failure() { + let state = ensure_setup().await; + state.span_exporter.reset(); + + let job_id = uuid::Uuid::new_v4(); + let job = make_test_job(job_id, None); + windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, false); + + let spans = state.span_exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|s| s.name == "full_job") + .expect("full_job span not found"); + + match &span.status { + opentelemetry::trace::Status::Error { description } => { + assert_eq!(description.as_ref(), "Job failed"); + } + other => panic!("expected Error status, got {:?}", other), + } +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_root_job_trace_id_matches_uuid() { + let state = ensure_setup().await; + state.span_exporter.reset(); + + let job_id = uuid::Uuid::parse_str("550e8400-e29b-41d4-a716-446655440000").unwrap(); + let job = make_test_job(job_id, None); + windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true); + + let spans = state.span_exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|s| s.name == "full_job") + .expect("full_job span not found"); + + let expected_trace_id = + opentelemetry::trace::TraceId::from_bytes(job_id.as_u128().to_be_bytes()); + assert_eq!(span.span_context.trace_id(), expected_trace_id); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_root_job_span_id_matches_uuid() { + let state = ensure_setup().await; + state.span_exporter.reset(); + + let job_id = uuid::Uuid::parse_str("550e8400-e29b-41d4-a716-446655440000").unwrap(); + let job = make_test_job(job_id, None); + windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true); + + let spans = state.span_exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|s| s.name == "full_job") + .expect("full_job span not found"); + + let expected_span_id = + opentelemetry::trace::SpanId::from_bytes(job_id.as_u64_pair().1.to_be_bytes()); + assert_eq!(span.span_context.span_id(), expected_span_id); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_child_job_produces_no_span() { + let state = ensure_setup().await; + state.span_exporter.reset(); + + let parent_id = uuid::Uuid::new_v4(); + let job_id = uuid::Uuid::new_v4(); + let job = make_test_job(job_id, Some(parent_id)); + windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true); + + let spans = state.span_exporter.get_finished_spans().unwrap(); + let found = spans.iter().any(|s| s.name == "full_job"); + assert!(!found, "child job should not produce a span"); +} + +#[tokio::test(flavor = "multi_thread", worker_threads = 2)] +async fn test_root_job_span_attributes_values() { + let state = ensure_setup().await; + state.span_exporter.reset(); + + let job_id = uuid::Uuid::new_v4(); + let job = make_test_job(job_id, None); + windmill_worker::otel_ee::add_root_flow_job_to_otlp(&job, true); + + let spans = state.span_exporter.get_finished_spans().unwrap(); + let span = spans + .iter() + .find(|s| s.name == "full_job") + .expect("full_job span not found"); + + let get_attr = |key: &str| -> String { + span.attributes + .iter() + .find(|kv| kv.key.as_str() == key) + .map(|kv| kv.value.as_str().to_string()) + .unwrap_or_default() + }; + + assert_eq!(get_attr("job_id"), job_id.to_string()); + assert_eq!(get_attr("workspace_id"), "test-workspace"); + assert_eq!(get_attr("script_path"), "f/test/script"); +} diff --git a/backend/tests/python_jobs.rs b/backend/tests/python_jobs.rs index 2ffb85a418..f04e189315 100644 --- a/backend/tests/python_jobs.rs +++ b/backend/tests/python_jobs.rs @@ -1,9 +1,470 @@ use serde_json::json; +#[cfg(feature = "python")] use sqlx::postgres::Postgres; +#[cfg(feature = "python")] use sqlx::Pool; +#[cfg(feature = "python")] use windmill_common::scripts::ScriptLang; use windmill_test_utils::*; +// ============================================================================ +// Dedicated Worker Protocol Tests (Python) +// ============================================================================ + +#[cfg(feature = "python")] +mod dedicated_worker_protocol_python { + use std::io::{BufRead, BufReader, Write}; + use std::process::{Command, Stdio}; + use windmill_test_utils::{parse_dedicated_worker_line, DedicatedWorkerResult}; + use windmill_worker::{compute_py_codegen, generate_py_multi_script_wrapper, PyScriptEntry}; + + struct MultiScriptJob { + script_path: String, + args: serde_json::Value, + } + + /// Creates a multi-script Python wrapper, writes scripts to proper module paths + fn create_py_worker_files( + dir: &std::path::Path, + scripts: &[(&str, &str)], // (original_path, content) + ) -> std::path::PathBuf { + let mut codegens = Vec::new(); + for (path, content) in scripts { + let cg = compute_py_codegen(content, path); + let module_dir = dir.join(&cg.dirs); + std::fs::create_dir_all(&module_dir).unwrap(); + std::fs::write(module_dir.join(format!("{}.py", cg.module_name)), content).unwrap(); + codegens.push((path.to_string(), cg)); + } + + let entries: Vec> = codegens + .iter() + .map(|(path, cg)| PyScriptEntry { original_path: path.as_str(), codegen: cg }) + .collect(); + + let wrapper = generate_py_multi_script_wrapper(&entries, false, false); + let wrapper_path = dir.join("wrapper.py"); + std::fs::write(&wrapper_path, &wrapper).unwrap(); + wrapper_path + } + + fn run_py_multi_script_test( + scripts: &[(&str, &str)], + jobs: Vec, + ) -> Vec> { + let temp_dir = tempfile::tempdir().unwrap(); + create_py_worker_files(temp_dir.path(), scripts); + + let mut child = Command::new("python3") + .args(["-u", "-m", "wrapper"]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .current_dir(temp_dir.path()) + .spawn() + .expect("Failed to spawn python3 process"); + + let mut stdin = child.stdin.take().unwrap(); + let stdout = child.stdout.take().unwrap(); + let mut reader = BufReader::new(stdout); + + let mut start_line = String::new(); + reader.read_line(&mut start_line).unwrap(); + assert_eq!( + parse_dedicated_worker_line(start_line.trim()), + DedicatedWorkerResult::Start, + "Expected 'start', got: {}", + start_line.trim() + ); + + let mut results = Vec::new(); + for job in &jobs { + writeln!(stdin, "exec:{}:{}", job.script_path, job.args.to_string()).unwrap(); + stdin.flush().unwrap(); + + let mut response = String::new(); + reader.read_line(&mut response).unwrap(); + + match parse_dedicated_worker_line(response.trim()) { + DedicatedWorkerResult::Success(value) => results.push(Ok(value)), + DedicatedWorkerResult::Error(err) => { + let msg = err["message"] + .as_str() + .unwrap_or("Unknown error") + .to_string(); + results.push(Err(msg)); + } + other => panic!("Unexpected response: {:?}", other), + } + } + + writeln!(stdin, "end").unwrap(); + stdin.flush().unwrap(); + let _ = child.wait().expect("Worker process failed to exit"); + results + } + + fn run_py_single_script_test( + script_path: &str, + content: &str, + jobs: Vec, + ) -> Vec> { + run_py_multi_script_test( + &[(script_path, content)], + jobs.into_iter() + .map(|args| MultiScriptJob { script_path: script_path.to_string(), args }) + .collect(), + ) + } + + #[test] + fn test_python_dedicated_worker_simple() { + let results = run_py_single_script_test( + "f/test/add", + "def main(a: int, b: int):\n return a + b\n", + vec![serde_json::json!({"a": 3, "b": 4})], + ); + assert_eq!(results.len(), 1); + assert_eq!(results[0], Ok(serde_json::json!(7))); + } + + #[test] + fn test_python_dedicated_worker_multiple_jobs() { + let results = run_py_single_script_test( + "f/test/double", + "def main(n: int):\n return n * 2\n", + (1..=5).map(|i| serde_json::json!({"n": i})).collect(), + ); + assert_eq!(results.len(), 5); + for (i, result) in results.iter().enumerate() { + assert_eq!(*result, Ok(serde_json::json!(((i + 1) * 2) as i64))); + } + } + + #[test] + fn test_python_multi_script_routing() { + let results = run_py_multi_script_test( + &[ + ( + "f/math/add", + "def main(a: int, b: int):\n return a + b\n", + ), + ( + "f/math/mul", + "def main(x: int, y: int):\n return x * y\n", + ), + ], + vec![ + MultiScriptJob { + script_path: "f/math/add".to_string(), + args: serde_json::json!({"a": 3, "b": 4}), + }, + MultiScriptJob { + script_path: "f/math/mul".to_string(), + args: serde_json::json!({"x": 5, "y": 6}), + }, + MultiScriptJob { + script_path: "f/math/add".to_string(), + args: serde_json::json!({"a": 10, "b": 20}), + }, + ], + ); + assert_eq!(results.len(), 3); + assert_eq!(results[0], Ok(serde_json::json!(7))); + assert_eq!(results[1], Ok(serde_json::json!(30))); + assert_eq!(results[2], Ok(serde_json::json!(30))); + } + + #[test] + fn test_python_multi_script_error_isolation() { + let results = run_py_multi_script_test( + &[ + ("f/ok", "def main(x: int):\n return x * 2\n"), + ("f/err", "def main(msg: str):\n raise Exception(msg)\n"), + ], + vec![ + MultiScriptJob { + script_path: "f/ok".to_string(), + args: serde_json::json!({"x": 5}), + }, + MultiScriptJob { + script_path: "f/err".to_string(), + args: serde_json::json!({"msg": "boom"}), + }, + MultiScriptJob { + script_path: "f/ok".to_string(), + args: serde_json::json!({"x": 10}), + }, + ], + ); + assert_eq!(results.len(), 3); + assert_eq!(results[0], Ok(serde_json::json!(10))); + assert!(results[1].is_err()); + assert_eq!(results[1], Err("boom".to_string())); + assert_eq!(results[2], Ok(serde_json::json!(20))); + } + + #[test] + fn test_python_multi_script_unknown_path() { + let results = run_py_multi_script_test( + &[("f/known", "def main(x: int):\n return x\n")], + vec![MultiScriptJob { + script_path: "f/unknown".to_string(), + args: serde_json::json!({"x": 1}), + }], + ); + assert_eq!(results.len(), 1); + assert!(results[0].is_err()); + assert!(results[0] + .as_ref() + .unwrap_err() + .contains("Script not found")); + } + + // ==================== exec_preprocess Tests ==================== + + /// Raw protocol command for Python + enum ProtocolCmd { + Exec { path: String, args: serde_json::Value }, + ExecPreprocess { path: String, args: serde_json::Value }, + } + + /// Run a Python worker test with raw protocol commands + fn run_py_raw_protocol_test( + scripts: &[(&str, &str)], + commands: Vec, + ) -> Vec { + let temp_dir = tempfile::tempdir().unwrap(); + create_py_worker_files(temp_dir.path(), scripts); + + let mut child = Command::new("python3") + .args(["-u", "-m", "wrapper"]) + .stdin(Stdio::piped()) + .stdout(Stdio::piped()) + .stderr(Stdio::piped()) + .current_dir(temp_dir.path()) + .spawn() + .expect("Failed to spawn python3 process"); + + let mut stdin = child.stdin.take().unwrap(); + let stdout = child.stdout.take().unwrap(); + let mut reader = BufReader::new(stdout); + + let mut start_line = String::new(); + reader.read_line(&mut start_line).unwrap(); + assert_eq!( + parse_dedicated_worker_line(start_line.trim()), + DedicatedWorkerResult::Start, + ); + + let mut results = Vec::new(); + + for cmd in &commands { + let line = match cmd { + ProtocolCmd::Exec { path, args } => format!("exec:{}:{}", path, args), + ProtocolCmd::ExecPreprocess { path, args } => { + format!("exec_preprocess:{}:{}", path, args) + } + }; + writeln!(stdin, "{}", line).unwrap(); + stdin.flush().unwrap(); + + let expected_lines = match cmd { + ProtocolCmd::ExecPreprocess { .. } => 2, + ProtocolCmd::Exec { .. } => 1, + }; + + for _ in 0..expected_lines { + let mut response = String::new(); + reader.read_line(&mut response).unwrap(); + let parsed = parse_dedicated_worker_line(response.trim()); + if matches!(parsed, DedicatedWorkerResult::Error(_)) { + results.push(parsed); + break; + } + results.push(parsed); + } + } + + writeln!(stdin, "end").unwrap(); + stdin.flush().unwrap(); + let _ = child.wait().expect("Worker process failed to exit"); + + results + } + + #[test] + fn test_python_exec_preprocess() { + let script = r#" +def preprocessor(x: int): + return {"x": x * 10} + +def main(x: int): + return x + 1 +"#; + let results = run_py_raw_protocol_test( + &[("f/test/pre", script)], + vec![ProtocolCmd::ExecPreprocess { + path: "f/test/pre".to_string(), + args: serde_json::json!({"x": 5}), + }], + ); + assert_eq!(results.len(), 2); + assert_eq!( + results[0], + DedicatedWorkerResult::PreprocessedArgs(serde_json::json!({"x": 50})) + ); + // main(50) => 51 + assert_eq!( + results[1], + DedicatedWorkerResult::Success(serde_json::json!(51)) + ); + } + + #[test] + fn test_python_exec_preprocess_missing_preprocessor() { + let script = "def main(x: int):\n return x\n"; + let results = run_py_raw_protocol_test( + &[("f/test/nopre", script)], + vec![ProtocolCmd::ExecPreprocess { + path: "f/test/nopre".to_string(), + args: serde_json::json!({"x": 5}), + }], + ); + assert_eq!(results.len(), 1); + assert!(matches!(results[0], DedicatedWorkerResult::Error(_))); + } + + #[test] + fn test_python_exec_preprocess_then_exec() { + let script = r#" +def preprocessor(x: int): + return {"x": x * 2} + +def main(x: int): + return x + 100 +"#; + let results = run_py_raw_protocol_test( + &[("f/test/mixed", script)], + vec![ + ProtocolCmd::ExecPreprocess { + path: "f/test/mixed".to_string(), + args: serde_json::json!({"x": 5}), + }, + ProtocolCmd::Exec { + path: "f/test/mixed".to_string(), + args: serde_json::json!({"x": 7}), + }, + ], + ); + // preprocess: preprocessor(5) => {"x":10}, main(10) => 110 + // exec: main(7) => 107 + assert_eq!(results.len(), 3); + assert_eq!( + results[0], + DedicatedWorkerResult::PreprocessedArgs(serde_json::json!({"x": 10})) + ); + assert_eq!( + results[1], + DedicatedWorkerResult::Success(serde_json::json!(110)) + ); + assert_eq!( + results[2], + DedicatedWorkerResult::Success(serde_json::json!(107)) + ); + } + + // ==================== Argument Transformation Tests ==================== + + #[test] + fn test_python_datetime_arg_transformation() { + let script = r#" +from datetime import datetime + +def main(d: datetime): + return d.isoformat() +"#; + let results = run_py_single_script_test( + "f/test/dt", + script, + vec![serde_json::json!({"d": "2024-01-15T10:30:00+00:00"})], + ); + assert_eq!(results.len(), 1); + assert_eq!( + results[0], + Ok(serde_json::json!("2024-01-15T10:30:00+00:00")) + ); + } + + #[test] + fn test_python_bytes_arg_transformation() { + let script = r#" +def main(data: bytes): + return len(data) +"#; + // base64 of "hello" is "aGVsbG8=" + let results = run_py_single_script_test( + "f/test/bytes", + script, + vec![serde_json::json!({"data": "aGVsbG8="})], + ); + assert_eq!(results.len(), 1); + assert_eq!(results[0], Ok(serde_json::json!(5))); + } + + #[test] + fn test_python_kwargs_filtering() { + // Test that extra kwargs are filtered out and only declared args are passed + let script = "def main(a: int, b: int):\n return a + b\n"; + let results = run_py_single_script_test( + "f/test/kwargs", + script, + vec![serde_json::json!({"a": 1, "b": 2, "extra": 99})], + ); + assert_eq!(results.len(), 1); + assert_eq!(results[0], Ok(serde_json::json!(3))); + } + + #[test] + fn test_python_function_call_sentinel_removal() { + // Test that '' sentinel values are removed from args + let script = "def main(a: int, b: int = 10):\n return a + b\n"; + let results = run_py_single_script_test( + "f/test/sentinel", + script, + vec![serde_json::json!({"a": 5, "b": ""})], + ); + assert_eq!(results.len(), 1); + // b should be removed (sentinel), default 10 used + assert_eq!(results[0], Ok(serde_json::json!(15))); + } + + // ==================== Relative Import Tests ==================== + + #[test] + fn test_python_dedicated_worker_with_relative_import_detection() { + // Test that the wrapper includes 'import loader' when scripts have relative imports + let script_with_relative = "from f.helper import util\ndef main(x: int):\n return x\n"; + let cg = compute_py_codegen(script_with_relative, "f/test/rel"); + let entries = [PyScriptEntry { original_path: "f/test/rel", codegen: &cg }]; + let wrapper = generate_py_multi_script_wrapper(&entries, false, true); + assert!( + wrapper.contains("import loader"), + "wrapper should contain 'import loader' when any_relative_imports=true" + ); + + // Without relative imports + let script_no_relative = "def main(x: int):\n return x\n"; + let cg2 = compute_py_codegen(script_no_relative, "f/test/norel"); + let entries2 = [PyScriptEntry { original_path: "f/test/norel", codegen: &cg2 }]; + let wrapper2 = generate_py_multi_script_wrapper(&entries2, false, false); + assert!( + !wrapper2.contains("import loader"), + "wrapper should NOT contain 'import loader' when any_relative_imports=false" + ); + } +} + #[cfg(feature = "python")] #[sqlx::test(fixtures("base", "lockfile_python"))] async fn test_requirements_python(db: Pool) -> anyhow::Result<()> { diff --git a/backend/windmill-api-auth/src/auth.rs b/backend/windmill-api-auth/src/auth.rs index 8e3a4c7822..b5b04cb3e8 100644 --- a/backend/windmill-api-auth/src/auth.rs +++ b/backend/windmill-api-auth/src/auth.rs @@ -1,7 +1,6 @@ #[cfg(feature = "enterprise")] use crate::ee_oss::ExternalJwks; use axum::{ - async_trait, extract::{FromRequestParts, OriginalUri, Query}, Extension, Json, }; @@ -226,7 +225,15 @@ impl AuthCache { t_hash, w_id.as_ref(), ) - .map(|x| (x.owner, x.email, x.super_admin, x.scopes, x.label)) + .map(|x| { + ( + x.owner, + x.email, + x.super_admin, + x.scopes, + x.label, + ) + }) .fetch_optional(&self.db) .await .ok() @@ -235,7 +242,13 @@ impl AuthCache { if let Some(user) = user_o { let authed_o = { match user { - (Some(owner), Some(email), super_admin, _, label) if w_id.is_some() => { + ( + Some(owner), + Some(email), + super_admin, + _, + label, + ) if w_id.is_some() => { let username_override = username_override_from_label(label); if let Some((prefix, name)) = owner.split_once('/') { if prefix == "u" { @@ -451,7 +464,11 @@ pub(crate) async fn extract_token(parts: &mut Parts, state: &S) None => Extension::::from_request_parts(parts, state) .await .ok() - .and_then(|cookies| cookies.get(COOKIE_NAME).map(|c| c.value().to_owned())), + .and_then(|cookies| { + cookies + .get(COOKIE_NAME) + .map(|c| c.value_trimmed().to_owned()) + }), }; #[derive(Deserialize)] @@ -504,7 +521,6 @@ impl BruteForceCounter { } } -#[async_trait] impl FromRequestParts for Tokened where S: Send + Sync, @@ -535,7 +551,6 @@ where } } -#[async_trait] impl FromRequestParts for OptTokened where S: Send + Sync, diff --git a/backend/windmill-api-auth/src/lib.rs b/backend/windmill-api-auth/src/lib.rs index aceef77e01..4e230c019b 100644 --- a/backend/windmill-api-auth/src/lib.rs +++ b/backend/windmill-api-auth/src/lib.rs @@ -12,8 +12,7 @@ pub mod ee; pub mod ee_oss; pub mod scopes; -use axum::async_trait; -use axum::extract::FromRequestParts; +use axum::extract::{FromRequestParts, OptionalFromRequestParts}; use http::request::Parts; use windmill_audit::audit_oss::AuditAuthorable; @@ -345,7 +344,6 @@ pub async fn maybe_refresh_folders( // ------------ FromRequestParts impls (direct call to auth module) ------------ -#[async_trait] impl FromRequestParts for ApiAuthed where S: Send + Sync, @@ -361,7 +359,24 @@ where } } -#[async_trait] +impl OptionalFromRequestParts for ApiAuthed +where + S: Send + Sync, +{ + type Rejection = std::convert::Infallible; + + async fn from_request_parts( + parts: &mut Parts, + state: &S, + ) -> std::result::Result, Self::Rejection> { + Ok( + >::from_request_parts(parts, state) + .await + .ok(), + ) + } +} + impl FromRequestParts for OptJobAuthed where S: Send + Sync, @@ -397,7 +412,6 @@ fn empty_parts() -> Parts { #[derive(Clone, Debug)] pub struct OptAuthed(pub Option); -#[async_trait] impl FromRequestParts for OptAuthed where S: Send + Sync, @@ -408,7 +422,7 @@ where parts: &mut Parts, state: &S, ) -> std::result::Result { - ApiAuthed::from_request_parts(parts, state) + >::from_request_parts(parts, state) .await .map(|authed| Self(Some(authed))) .or_else(|_| Ok(Self(None))) diff --git a/backend/windmill-api-auth/src/scopes.rs b/backend/windmill-api-auth/src/scopes.rs index 26a2b3c74c..06ddbafcca 100644 --- a/backend/windmill-api-auth/src/scopes.rs +++ b/backend/windmill-api-auth/src/scopes.rs @@ -442,9 +442,22 @@ pub fn check_route_access( // Find the domain and kind for this route let (required_domain, required_kind, route_suffix) = extract_domain_from_route(route_path)?; - // Backward compatibility: MCP handlers expect unusual scope actions: all, favorites, hub. + // MCP scopes (mcp:all, mcp:favorites, mcp:hub:*, etc.) use a custom format + // that doesn't fit the standard domain:action model. Verify the token has at + // least one mcp: scope; MCP handlers do their own fine-grained checking. if required_domain == ScopeDomain::Mcp { - return Ok(()); + let is_scoped_token = token_scopes + .iter() + .any(|s| !s.starts_with("if_jobs:filter_tags:")); + if !is_scoped_token { + return Ok(()); + } + if token_scopes.iter().any(|s| s.starts_with("mcp:")) { + return Ok(()); + } + return Err(Error::NotAuthorized( + "Access denied. Required scope: mcp:*".to_string(), + )); } // tracing::error!("Checking route access {:?} {:?} {:?} {:?}", required_action, required_domain, required_kind, route_suffix); @@ -931,4 +944,50 @@ mod tests { ScopeDefinition::new("scripts", "read", None, Some(vec!["u/*".to_string()])); assert!(scope_specific_path.includes(&required_broad)); } + + #[test] + fn test_mcp_scope_bypass_blocked_without_mcp_scope() { + // A token with only jobs:read should NOT be able to access MCP endpoints + let scopes = vec!["jobs:read".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_err()); + assert!( + check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "POST").is_err() + ); + } + + #[test] + fn test_mcp_scope_allowed_with_mcp_scope() { + // A token with mcp:all should access MCP endpoints + let scopes = vec!["mcp:all".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok()); + + // mcp:favorites should also work + let scopes = vec!["mcp:favorites".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "POST").is_ok()); + + // mcp:scripts:path should also work + let scopes = vec!["mcp:scripts:u/admin/script1".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok()); + } + + #[test] + fn test_mcp_scope_filter_tags_only_treated_as_unrestricted() { + // Token with only filter_tags is not considered scoped — should be allowed + let scopes = vec!["if_jobs:filter_tags:tag1".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok()); + } + + #[test] + fn test_mcp_scope_mixed_scopes_without_mcp() { + // Token with multiple non-MCP scopes should be denied + let scopes = vec!["jobs:read".to_string(), "scripts:write".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_err()); + } + + #[test] + fn test_mcp_scope_mixed_scopes_with_mcp() { + // Token with MCP scope + other scopes should be allowed for MCP + let scopes = vec!["jobs:read".to_string(), "mcp:all".to_string()]; + assert!(check_route_access(&scopes, "/api/w/test_workspace/mcp/something", "GET").is_ok()); + } } diff --git a/backend/windmill-api-configs/src/lib.rs b/backend/windmill-api-configs/src/lib.rs index a05f2dbaa8..8485de5e95 100644 --- a/backend/windmill-api-configs/src/lib.rs +++ b/backend/windmill-api-configs/src/lib.rs @@ -28,11 +28,11 @@ use windmill_api_auth::{require_devops_role, ApiAuthed}; pub fn global_service() -> Router { Router::new() .route("/list_worker_groups", get(list_worker_groups)) - .route("/update/:name", post(update_config).delete(delete_config)) - .route("/get/:name", get(get_config)) + .route("/update/{name}", post(update_config).delete(delete_config)) + .route("/get/{name}", get(get_config)) .route("/list", get(list_configs)) .route( - "/list_autoscaling_events/:worker_group", + "/list_autoscaling_events/{worker_group}", get(list_autoscaling_events), ) .route( diff --git a/backend/windmill-api-debug/src/lib.rs b/backend/windmill-api-debug/src/lib.rs index 4f793e7da7..baece9e8e9 100644 --- a/backend/windmill-api-debug/src/lib.rs +++ b/backend/windmill-api-debug/src/lib.rs @@ -87,6 +87,7 @@ pub fn workspaced_service() -> Router { Router::new() .route("/sign", post(sign_debug_request)) .route("/sign_expression", post(sign_expression)) + .route("/sign_multiplayer", post(sign_multiplayer)) } /// JWKS response containing the public key for debug token verification @@ -416,3 +417,62 @@ async fn sign_expression( Ok(Json(SignedExpressionPayload { token })) } + +/// JWT claims for multiplayer session tokens +#[derive(Serialize, Deserialize)] +pub struct MultiplayerTokenClaims { + /// Workspace ID + pub workspace_id: String, + /// User email + pub email: String, + /// Issued at (Unix timestamp) + pub iat: i64, + /// Expiration (Unix timestamp) + pub exp: i64, + /// Token purpose (always "multiplayer") + pub purpose: String, +} + +#[derive(Serialize)] +pub struct SignedMultiplayerPayload { + pub token: String, +} + +/// Sign a multiplayer session request. +/// +/// Returns a JWT that the multiplayer server will verify using the public key from /api/debug/jwks. +async fn sign_multiplayer( + authed: ApiAuthed, + Path(w_id): Path, +) -> JsonResult { + let key_guard = DEBUG_SIGNING_KEY.read().await; + let signing_key = key_guard.as_ref().ok_or_else(|| { + windmill_common::error::Error::InternalErr("Debug signing key not initialized".to_string()) + })?; + + let now_ts = Utc::now().timestamp(); + let exp = now_ts + DEBUG_TOKEN_TTL_SECS; + + let claims = MultiplayerTokenClaims { + workspace_id: w_id, + email: authed.email, + iat: now_ts, + exp, + purpose: "multiplayer".to_string(), + }; + + let header = serde_json::json!({ + "alg": "EdDSA", + "typ": "JWT" + }); + let header_b64 = URL_SAFE_NO_PAD.encode(serde_json::to_string(&header).unwrap()); + let claims_b64 = URL_SAFE_NO_PAD.encode(serde_json::to_string(&claims).unwrap()); + let message = format!("{}.{}", header_b64, claims_b64); + + let signature = signing_key.sign(message.as_bytes()); + let signature_b64 = URL_SAFE_NO_PAD.encode(signature.to_bytes()); + + let token = format!("{}.{}", message, signature_b64); + + Ok(Json(SignedMultiplayerPayload { token })) +} diff --git a/backend/windmill-api-flow-conversations/src/lib.rs b/backend/windmill-api-flow-conversations/src/lib.rs index bc37c9863d..70c96d1405 100644 --- a/backend/windmill-api-flow-conversations/src/lib.rs +++ b/backend/windmill-api-flow-conversations/src/lib.rs @@ -21,8 +21,8 @@ use windmill_common::{ pub fn workspaced_service() -> Router { Router::new() .route("/list", get(list_conversations)) - .route("/delete/:conversation_id", delete(delete_conversation)) - .route("/:conversation_id/messages", get(list_messages)) + .route("/delete/{conversation_id}", delete(delete_conversation)) + .route("/{conversation_id}/messages", get(list_messages)) } #[derive(Serialize, FromRow, Debug)] diff --git a/backend/windmill-api-flows/src/flows.rs b/backend/windmill-api-flows/src/flows.rs index 41cacb4291..89fd9629b7 100644 --- a/backend/windmill-api-flows/src/flows.rs +++ b/backend/windmill-api-flows/src/flows.rs @@ -61,26 +61,26 @@ pub fn workspaced_service() -> Router { .route("/list", get(list_flows)) .route("/list_search", get(list_search_flows)) .route("/create", post(create_flow)) - .route("/update/*path", post(update_flow)) - .route("/archive/*path", post(archive_flow_by_path)) - .route("/delete/*path", delete(delete_flow_by_path)) - .route("/list_tokens/*path", get(list_tokens)) - .route("/get/*path", get(get_flow_by_path)) - .route("/deployment_status/p/*path", get(get_deployment_status)) - .route("/get/draft/*path", get(get_flow_by_path_w_draft)) - .route("/exists/*path", get(exists_flow_by_path)) + .route("/update/{*path}", post(update_flow)) + .route("/archive/{*path}", post(archive_flow_by_path)) + .route("/delete/{*path}", delete(delete_flow_by_path)) + .route("/list_tokens/{*path}", get(list_tokens)) + .route("/get/{*path}", get(get_flow_by_path)) + .route("/deployment_status/p/{*path}", get(get_deployment_status)) + .route("/get/draft/{*path}", get(get_flow_by_path_w_draft)) + .route("/exists/{*path}", get(exists_flow_by_path)) .route("/list_paths", get(list_paths)) - .route("/history/p/*path", get(get_flow_history)) - .route("/get_latest_version/*path", get(get_latest_version)) + .route("/history/p/{*path}", get(get_flow_history)) + .route("/get_latest_version/{*path}", get(get_latest_version)) .route( - "/list_paths_from_workspace_runnable/:runnable_kind/*path", + "/list_paths_from_workspace_runnable/{runnable_kind}/{*path}", get(list_paths_from_workspace_runnable), ) - .route("/history_update/v/:version", post(update_flow_history)) - .route("/get/v/:version", get(get_flow_version_by_id)) - .route("/get/v/:version/p/*path", get(get_flow_version)) + .route("/history_update/v/{version}", post(update_flow_history)) + .route("/get/v/{version}", get(get_flow_version_by_id)) + .route("/get/v/{version}/p/{*path}", get(get_flow_version)) .route( - "/toggle_workspace_error_handler/*path", + "/toggle_workspace_error_handler/{*path}", post(toggle_workspace_error_handler), ) } @@ -88,7 +88,7 @@ pub fn workspaced_service() -> Router { pub fn global_service() -> Router { Router::new() .route("/hub/list", get(list_hub_flows)) - .route("/hub/get/:id", get(get_hub_flow_by_id)) + .route("/hub/get/{id}", get(get_hub_flow_by_id)) } #[derive(Serialize, FromRow)] @@ -1657,6 +1657,38 @@ async fn delete_flow_by_path( } let mut tx = user_db.begin(&authed).await?; + // Capture all related data for trashbin before deleting (CASCADE will remove flow_version, flow_node) + let trash_flow: Option = + sqlx::query_scalar("SELECT to_jsonb(t) FROM flow t WHERE path = $1 AND workspace_id = $2") + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + + let trash_flow_versions: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM flow_version t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + + let trash_flow_nodes: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM flow_node t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + + let trash_drafts: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM draft t WHERE path = $1 AND workspace_id = $2 AND typ = 'flow'", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + sqlx::query!( "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'flow'", path, @@ -1673,6 +1705,28 @@ async fn delete_flow_by_path( .execute(&mut *tx) .await?; + if let Some(flow_data) = trash_flow { + let mut trash_data = serde_json::json!({"row": flow_data}); + if !trash_flow_versions.is_empty() { + trash_data["flow_versions"] = serde_json::Value::Array(trash_flow_versions); + } + if !trash_flow_nodes.is_empty() { + trash_data["flow_nodes"] = serde_json::Value::Array(trash_flow_nodes); + } + if !trash_drafts.is_empty() { + trash_data["drafts"] = serde_json::Value::Array(trash_drafts); + } + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "flow", + path, + trash_data, + &authed.username, + ) + .await?; + } + if !query.keep_captures.unwrap_or(false) { sqlx::query!( "DELETE FROM capture_config WHERE path = $1 AND workspace_id = $2 AND is_flow IS TRUE", diff --git a/backend/windmill-api-groups/src/folder_history.rs b/backend/windmill-api-groups/src/folder_history.rs index b11f328e3d..8ce1086dbf 100644 --- a/backend/windmill-api-groups/src/folder_history.rs +++ b/backend/windmill-api-groups/src/folder_history.rs @@ -22,7 +22,7 @@ use serde::Serialize; use sqlx::FromRow; pub fn workspaced_service() -> Router { - Router::new().route("/get/:name", get(get_folder_permission_history)) + Router::new().route("/get/{name}", get(get_folder_permission_history)) } #[derive(Serialize, FromRow)] diff --git a/backend/windmill-api-groups/src/folders.rs b/backend/windmill-api-groups/src/folders.rs index d7ea3590b5..ab1541adc8 100644 --- a/backend/windmill-api-groups/src/folders.rs +++ b/backend/windmill-api-groups/src/folders.rs @@ -18,7 +18,6 @@ use regex::Regex; use windmill_api_auth::{check_scopes, ApiAuthed, AuthCache, Tokened}; use windmill_audit::audit_oss::{audit_log, AuditAuthorable}; use windmill_audit::ActionKind; -use windmill_common::{error::Error, webhook::{WebhookMessage, WebhookShared}, workspaces::{check_user_against_rule, ProtectionRuleKind, RuleCheckResult}}; use windmill_common::DB; use windmill_common::{ db::UserDB, @@ -26,6 +25,11 @@ use windmill_common::{ users::username_to_permissioned_as, utils::{not_found_if_none, paginate, Pagination}, }; +use windmill_common::{ + error::Error, + webhook::{WebhookMessage, WebhookShared}, + workspaces::{check_user_against_rule, ProtectionRuleKind, RuleCheckResult}, +}; use serde::{Deserialize, Serialize}; use sqlx::{FromRow, Postgres, Transaction}; @@ -36,14 +40,14 @@ pub fn workspaced_service() -> Router { .route("/list", get(list_folders)) .route("/listnames", get(list_foldernames)) .route("/create", post(create_folder)) - .route("/get/:name", get(get_folder)) - .route("/exists/:name", get(exists_folder)) - .route("/update/:name", post(update_folder)) - .route("/getusage/:name", get(get_folder_usage)) - .route("/delete/:name", delete(delete_folder)) - .route("/addowner/:name", post(add_owner)) - .route("/removeowner/:name", post(remove_owner)) - .route("/is_owner/*path", get(is_owner_api)) + .route("/get/{name}", get(get_folder)) + .route("/exists/{name}", get(exists_folder)) + .route("/update/{name}", post(update_folder)) + .route("/getusage/{name}", get(get_folder_usage)) + .route("/delete/{name}", delete(delete_folder)) + .route("/addowner/{name}", post(add_owner)) + .route("/removeowner/{name}", post(remove_owner)) + .route("/is_owner/{*path}", get(is_owner_api)) } #[derive(FromRow, Serialize, Deserialize, Clone)] @@ -716,13 +720,14 @@ async fn add_owner( .await?; validate_owner(&owner)?; - sqlx::query(&format!( - "UPDATE folder SET extra_perms = jsonb_set(extra_perms, '{{\"{owner}\"}}', to_jsonb($1), \ - true) WHERE name = $2 AND workspace_id = $3 RETURNING extra_perms" - )) + sqlx::query( + "UPDATE folder SET extra_perms = jsonb_set(extra_perms, array[$4]::text[], to_jsonb($1), \ + true) WHERE name = $2 AND workspace_id = $3 RETURNING extra_perms", + ) .bind(true) .bind(&name) .bind(&w_id) + .bind(&owner) .fetch_optional(&mut *tx) .await?; @@ -787,14 +792,15 @@ async fn remove_owner( } if let Some(write) = write { - let old_write = sqlx::query_scalar::<_, Option>(&format!( - "UPDATE folder SET extra_perms = jsonb_set(extra_perms, '{{\"{owner}\"}}', to_jsonb($1), \ - true) FROM (SELECT (extra_perms->>'{owner}')::boolean as old_val FROM folder WHERE name = $2 AND workspace_id = $3) old \ + let old_write = sqlx::query_scalar::<_, Option>( + "UPDATE folder SET extra_perms = jsonb_set(extra_perms, array[$4]::text[], to_jsonb($1), \ + true) FROM (SELECT (extra_perms->>$4)::boolean as old_val FROM folder WHERE name = $2 AND workspace_id = $3) old \ WHERE name = $2 AND workspace_id = $3 RETURNING old.old_val" - )) + ) .bind(write) .bind(&name) .bind(&w_id) + .bind(&owner) .fetch_optional(&mut *tx) .await? .flatten(); diff --git a/backend/windmill-api-groups/src/granular_acls.rs b/backend/windmill-api-groups/src/granular_acls.rs index da9267419d..d7ea8418f9 100644 --- a/backend/windmill-api-groups/src/granular_acls.rs +++ b/backend/windmill-api-groups/src/granular_acls.rs @@ -48,9 +48,9 @@ const KINDS: [&str; 19] = [ pub fn workspaced_service() -> Router { Router::new() - .route("/get/*path", get(get_granular_acls)) - .route("/add/*path", post(add_granular_acl)) - .route("/remove/*path", post(remove_granular_acl)) + .route("/get/{*path}", get(get_granular_acls)) + .route("/add/{*path}", post(add_granular_acl)) + .route("/remove/{*path}", post(remove_granular_acl)) } #[derive(Serialize, Deserialize)] diff --git a/backend/windmill-api-groups/src/groups.rs b/backend/windmill-api-groups/src/groups.rs index 071c3a8649..af67f884c9 100644 --- a/backend/windmill-api-groups/src/groups.rs +++ b/backend/windmill-api-groups/src/groups.rs @@ -33,24 +33,24 @@ pub fn workspaced_service() -> Router { .route("/list", get(list_groups)) .route("/listnames", get(list_group_names)) .route("/create", post(create_group)) - .route("/get/:name", get(get_group)) - .route("/update/:name", post(update_group)) - .route("/delete/:name", delete(delete_group)) - .route("/adduser/:name", post(add_user)) - .route("/removeuser/:name", post(remove_user)) - .route("/is_owner/:name", get(is_owner)) + .route("/get/{name}", get(get_group)) + .route("/update/{name}", post(update_group)) + .route("/delete/{name}", delete(delete_group)) + .route("/adduser/{name}", post(add_user)) + .route("/removeuser/{name}", post(remove_user)) + .route("/is_owner/{name}", get(is_owner)) } pub fn global_service() -> Router { Router::new() .route("/list", get(list_igroups)) .route("/list_with_workspaces", get(list_igroups_with_workspaces)) - .route("/get/:name", get(get_igroup)) + .route("/get/{name}", get(get_igroup)) .route("/create", post(create_igroup)) - .route("/update/:name", post(update_igroup)) - .route("/delete/:name", delete(delete_igroup)) - .route("/adduser/:name", post(add_user_igroup)) - .route("/removeuser/:name", post(remove_user_igroup)) + .route("/update/{name}", post(update_igroup)) + .route("/delete/{name}", delete(delete_igroup)) + .route("/adduser/{name}", post(add_user_igroup)) + .route("/removeuser/{name}", post(remove_user_igroup)) .route("/export", get(export_igroups)) .route("/overwrite", post(overwrite_igroups)) } @@ -656,6 +656,7 @@ async fn delete_group( ) .execute(&mut *tx) .await?; + audit_log( &mut *tx, &authed, @@ -850,9 +851,21 @@ async fn add_user_igroup( #[cfg(all(feature = "private", feature = "enterprise"))] { use windmill_api_workspaces::workspaces_ee::auto_add_user; + use windmill_common::users::compute_highest_workspace_role; + + // Find all instance groups this user belongs to (includes the newly added group) + let user_igroups: Vec = sqlx::query_scalar!( + "SELECT igroup FROM email_to_igroup WHERE email = $1", + &email + ) + .fetch_all(&mut *tx) + .await?; + let workspaces = sqlx::query!( r#" - SELECT workspace_id, auto_invite->'instance_groups_roles' as instance_groups_roles + SELECT workspace_id, + auto_invite->'instance_groups_roles' as instance_groups_roles, + auto_invite->'instance_groups' as instance_groups_json FROM workspace_settings WHERE auto_invite->'instance_groups' ? $1 "#, @@ -860,34 +873,53 @@ async fn add_user_igroup( ) .fetch_all(&mut *tx) .await?; + for ws in workspaces { - let role = ws + let roles: std::collections::HashMap = ws .instance_groups_roles - .and_then(|r| r.get(&name).and_then(|v| v.as_str().map(String::from))) - .unwrap_or_else(|| "developer".to_string()); - let (is_admin, is_operator) = match role.as_str() { - "admin" => (true, false), - "operator" => (false, true), - _ => (false, false), - }; + .and_then(|r| serde_json::from_value(r).ok()) + .unwrap_or_default(); + + let ws_configured_groups: Vec = ws + .instance_groups_json + .and_then(|ig| serde_json::from_value(ig).ok()) + .unwrap_or_default(); + + let (best_group, is_admin, is_operator) = + compute_highest_workspace_role(&user_igroups, &ws_configured_groups, &roles); + + let instance_group_source = serde_json::json!({ + "source": "instance_group", + "group": &best_group + }); + + // auto_add_user creates the user if they don't exist (ON CONFLICT DO NOTHING). + // The operator flag here doesn't matter for the final state — the UPDATE below + // always sets the correct is_admin/operator based on the highest-precedence role. auto_add_user( &email, &ws.workspace_id, - &is_operator, + &false, &mut tx, &authed, - Some(serde_json::json!({"source": "instance_group", "group": &name})), + Some(instance_group_source.clone()), ) .await?; - if is_admin { - sqlx::query!( - "UPDATE usr SET is_admin = true WHERE workspace_id = $1 AND email = $2", - &ws.workspace_id, - &email - ) - .execute(&mut *tx) - .await?; - } + + // Set the correct role based on highest precedence across all groups. + // For new users, auto_add_user already stored added_via with source=instance_group, + // so this UPDATE will match. For existing instance_group users, it upgrades/corrects + // the role. Manually-added users (added_via is NULL or non-instance_group) are not affected. + sqlx::query!( + "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'", + is_admin, + is_operator, + &instance_group_source, + &ws.workspace_id, + &email + ) + .execute(&mut *tx) + .await?; } } diff --git a/backend/windmill-api-inputs/src/lib.rs b/backend/windmill-api-inputs/src/lib.rs index e253d02ea1..154e02e5a9 100644 --- a/backend/windmill-api-inputs/src/lib.rs +++ b/backend/windmill-api-inputs/src/lib.rs @@ -26,6 +26,7 @@ use windmill_common::{ jobs::JobKind, scripts::to_i64, utils::{not_found_if_none, paginate, Pagination}, + worker::CLOUD_HOSTED, }; pub fn workspaced_service() -> Router { Router::new() @@ -33,9 +34,9 @@ pub fn workspaced_service() -> Router { .route("/list", get(list_saved_inputs)) .route("/create", post(create_input)) .route("/update", post(update_input)) - .route("/delete/:id", post(delete_input)) + .route("/delete/{id}", post(delete_input)) .route( - "/:job_or_input_id/args", + "/{job_or_input_id}/args", get(get_args_from_history_or_saved_input), ) } @@ -134,11 +135,20 @@ async fn get_input_history( Query(g): Query, ) -> JsonResult> { let (per_page, offset) = paginate(pagination); + let per_page = if *CLOUD_HOSTED { + per_page.min(100) + } else { + per_page + }; let mut tx = user_db.begin(&authed).await?; let args_query = if let Some(args) = &g.args { - sql_builder::bind::Bind::bind(&"and v2_job.args @> ?", &args.replace("'", "''")) + if let Ok(v) = serde_json::from_str::(args) { + sql_builder::bind::Bind::bind(&"and v2_job.args @> ?", &v.to_string()) + } else { + "AND FALSE".to_string() + } } else { "".to_string() }; diff --git a/backend/windmill-api-integration-tests/Cargo.toml b/backend/windmill-api-integration-tests/Cargo.toml index 35dd64f995..54f836657c 100644 --- a/backend/windmill-api-integration-tests/Cargo.toml +++ b/backend/windmill-api-integration-tests/Cargo.toml @@ -13,7 +13,7 @@ default = [] private = ["windmill-test-utils/private", "dep:aws-config", "dep:aws-credential-types", "dep:aws-sdk-sqs", "windmill-git-sync/private"] enterprise = ["windmill-test-utils/enterprise", "dep:base64", "windmill-git-sync/enterprise"] deno_core = ["windmill-test-utils/deno_core"] -mcp = [] +mcp = ["windmill-test-utils/mcp", "dep:rmcp"] run_inline = ["dep:windmill-worker", "windmill-test-utils/run_inline", "windmill-test-utils/duckdb"] [dependencies] @@ -31,6 +31,7 @@ reqwest.workspace = true tokio.workspace = true anyhow.workspace = true uuid.workspace = true +futures.workspace = true rand.workspace = true rumqttc.workspace = true rdkafka.workspace = true @@ -39,3 +40,5 @@ aws-config = { workspace = true, optional = true } aws-credential-types = { workspace = true, optional = true } aws-sdk-sqs = { workspace = true, optional = true } base64 = { workspace = true, optional = true } +axum.workspace = true +rmcp = { workspace = true, optional = true } diff --git a/backend/windmill-api-integration-tests/tests/ai_routes.rs b/backend/windmill-api-integration-tests/tests/ai_routes.rs new file mode 100644 index 0000000000..543c3ed2a8 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/ai_routes.rs @@ -0,0 +1,106 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +/// Start a mock AI API that echoes back a valid chat completion response. +async fn start_mock_ai_api() -> u16 { + use axum::{routing::post, Json, Router}; + + let app = Router::new().fallback(post(|| async { + Json(json!({ + "id": "chatcmpl-test", + "object": "chat.completion", + "choices": [{"message": {"role": "assistant", "content": "hello"}}] + })) + })); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); + port +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_ai_proxy_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Start mock AI API + let mock_port = start_mock_ai_api().await; + let mock_url = format!("http://127.0.0.1:{mock_port}/v1"); + + // Create an openai resource pointing to the mock + let resp = authed( + client() + .post(format!( + "http://localhost:{port}/api/w/test-workspace/resources/create" + )) + .json(&json!({ + "path": "f/ai/openai_config", + "resource_type": "openai", + "value": { + "api_key": "test-key", + "base_url": mock_url + } + })), + ) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "create openai resource", + ); + + // Set ai_config on workspace_settings directly via SQL + sqlx::query( + "UPDATE workspace_settings SET ai_config = $1::jsonb WHERE workspace_id = 'test-workspace'", + ) + .bind(json!({ + "providers": { + "openai": { + "resource_path": "f/ai/openai_config", + "models": ["gpt-4"] + } + } + })) + .execute(&db) + .await?; + + // POST /w/{ws}/ai/proxy/chat/completions + let resp = authed( + client() + .post(format!( + "http://localhost:{port}/api/w/test-workspace/ai/proxy/chat/completions" + )) + .header("X-Provider", "openai") + .json(&json!({ + "model": "gpt-4", + "messages": [{"role": "user", "content": "hi"}] + })), + ) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /ai/proxy/chat/completions", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/audit.rs b/backend/windmill-api-integration-tests/tests/audit.rs new file mode 100644 index 0000000000..31994d8165 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/audit.rs @@ -0,0 +1,35 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_audit_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/audit"); + + // GET /list returns 200 (empty array) + let resp = authed(client().get(format!("{base}/list"))).send().await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /audit/list", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/capture_unauthed.rs b/backend/windmill-api-integration-tests/tests/capture_unauthed.rs new file mode 100644 index 0000000000..f30a6e31b3 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/capture_unauthed.rs @@ -0,0 +1,83 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_capture_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // POST /capture/set_config → 200 (authed) + let resp = authed( + client() + .post(format!( + "http://localhost:{port}/api/w/test-workspace/capture/set_config" + )) + .json(&json!({ + "trigger_kind": "webhook", + "path": "u/test-user/test_capture", + "is_flow": false + })), + ) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /capture/set_config"); + + // GET /capture/list/{...} → 200 (authed) + let resp = authed(client().get(format!( + "http://localhost:{port}/api/w/test-workspace/capture/list/script/u/test-user/test_capture" + ))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx( + status, + &body, + "GET /capture/list/script/u/test-user/test_capture", + ); + + // POST /capture/ping_config/{trigger_kind}/{runnable_kind}/{*path} → 200 + let resp = authed(client().post(format!( + "http://localhost:{port}/api/w/test-workspace/capture/ping_config/webhook/script/u/test-user/test_capture" + ))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /capture/ping_config", + ); + + // GET /capture/get_configs/{runnable_kind}/{*path} → 200 + let resp = authed(client().get(format!( + "http://localhost:{port}/api/w/test-workspace/capture/get_configs/script/u/test-user/test_capture" + ))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /capture/get_configs", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/concurrency_groups.rs b/backend/windmill-api-integration-tests/tests/concurrency_groups.rs new file mode 100644 index 0000000000..e4cf4ab8be --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/concurrency_groups.rs @@ -0,0 +1,48 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_concurrency_groups_2xx(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let resp = authed(client().get(format!( + "http://localhost:{port}/api/concurrency_groups/list" + ))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /api/concurrency_groups/list", + ); + + let resp = authed(client().get(format!( + "http://localhost:{port}/api/w/test-workspace/concurrency_groups/list_jobs" + ))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /api/w/test-workspace/concurrency_groups/list_jobs", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/favorites.rs b/backend/windmill-api-integration-tests/tests/favorites.rs new file mode 100644 index 0000000000..888425eef7 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/favorites.rs @@ -0,0 +1,72 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_favorites_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let ws = format!("http://localhost:{port}/api/w/test-workspace"); + + // Setup: create a script to favorite + let resp = authed(client().post(format!("{ws}/scripts/create"))) + .json(&json!({ + "path": "u/test-user/test_fav_script", + "summary": "test", + "description": "", + "content": "export function main() { return 1; }", + "language": "deno", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": {}, + "required": [] + } + })) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /scripts/create (setup)"); + + let fav_body = json!({ + "favorite_kind": "script", + "path": "u/test-user/test_fav_script" + }); + + // POST /favorites/star → 200 + let resp = authed(client().post(format!("{ws}/favorites/star"))) + .json(&fav_body) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /favorites/star"); + + // POST /favorites/unstar → 200 + let resp = authed(client().post(format!("{ws}/favorites/unstar"))) + .json(&fav_body) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /favorites/unstar"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/folder_history.rs b/backend/windmill-api-integration-tests/tests/folder_history.rs new file mode 100644 index 0000000000..065977f386 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/folder_history.rs @@ -0,0 +1,51 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_folder_history_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // Create a folder first + let resp = authed( + client() + .post(format!( + "http://localhost:{port}/api/w/test-workspace/folders/create" + )) + .json(&json!({"name": "test_hist_folder", "owners": ["u/test-user"]})), + ) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /folders/create"); + + // GET /folders_history/get/{folder} → 200 (empty array) + let resp = authed(client().get(format!( + "http://localhost:{port}/api/w/test-workspace/folders_history/get/test_hist_folder" + ))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /folders_history/get/test_hist_folder"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/granular_acls.rs b/backend/windmill-api-integration-tests/tests/granular_acls.rs new file mode 100644 index 0000000000..c9cbbf0cf4 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/granular_acls.rs @@ -0,0 +1,54 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_granular_acls_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/acls"); + + // GET /acls/get/group_/all → 200 + let resp = authed(client().get(format!("{base}/get/group_/all"))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /acls/get/group_/all"); + + // POST /acls/add/group_/all → 200 + let resp = authed(client().post(format!("{base}/add/group_/all"))) + .json(&json!({"owner": "u/test-user-2", "write": true})) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /acls/add/group_/all"); + + // POST /acls/remove/group_/all → 200 + let resp = authed(client().post(format!("{base}/remove/group_/all"))) + .json(&json!({"owner": "u/test-user-2"})) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /acls/remove/group_/all"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/group_history.rs b/backend/windmill-api-integration-tests/tests/group_history.rs new file mode 100644 index 0000000000..5a18901454 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/group_history.rs @@ -0,0 +1,32 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_group_history_2xx(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/groups_history"); + + let resp = authed(client().get(format!("{base}/get/all"))) + .send() + .await?; + assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get/all"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/health.rs b/backend/windmill-api-integration-tests/tests/health.rs new file mode 100644 index 0000000000..f624431510 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/health.rs @@ -0,0 +1,41 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_health_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/health"); + + // GET /health/status → 200 (no auth required) + let resp = client().get(format!("{base}/status")).send().await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /health/status"); + + // GET /health/detailed → 200 (authed) + let resp = authed(client().get(format!("{base}/detailed"))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /health/detailed"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/inputs.rs b/backend/windmill-api-integration-tests/tests/inputs.rs new file mode 100644 index 0000000000..6f806efdb5 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/inputs.rs @@ -0,0 +1,76 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_inputs_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/inputs"); + + // GET /history with fake runnable → 200 empty array + let resp = authed(client().get(format!( + "{base}/history?runnable_id=u/test-user/test&runnable_type=ScriptPath" + ))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /inputs/history"); + + // GET /list with fake runnable → 200 empty array + let resp = authed(client().get(format!( + "{base}/list?runnable_id=u/test-user/test&runnable_type=ScriptPath" + ))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /inputs/list"); + + // POST /create → 200, returns UUID + let resp = authed(client().post(format!( + "{base}/create?runnable_id=u/test-user/test&runnable_type=ScriptPath" + ))) + .json(&json!({"name": "test_input", "args": {}})) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /inputs/create"); + let input_id: String = serde_json::from_str(&body)?; + + // GET /{id}/args → 200 + let resp = authed(client().get(format!("{base}/{input_id}/args"))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /inputs/{id}/args"); + + // POST /delete/{id} → 200 + let resp = authed(client().post(format!("{base}/delete/{input_id}"))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /inputs/delete/{id}"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/job_metrics.rs b/backend/windmill-api-integration-tests/tests/job_metrics.rs new file mode 100644 index 0000000000..7408e4d3e3 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/job_metrics.rs @@ -0,0 +1,59 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +const FAKE_UUID: &str = "00000000-0000-0000-0000-000000000000"; + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_job_metrics_2xx(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/job_metrics"); + + let resp = authed(client().post(format!("{base}/get/{FAKE_UUID}"))) + .json(&json!({})) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /get/{id}", + ); + + let resp = authed(client().post(format!("{base}/set_progress/{FAKE_UUID}"))) + .json(&json!({"percent": 50})) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /set_progress/{id}", + ); + + let resp = authed(client().get(format!("{base}/get_progress/{FAKE_UUID}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /get_progress/{id}", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/jobs_authed.rs b/backend/windmill-api-integration-tests/tests/jobs_authed.rs new file mode 100644 index 0000000000..766c300d3d --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/jobs_authed.rs @@ -0,0 +1,309 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use uuid::Uuid; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +fn assert_route_reachable(status: u16, body: &str, endpoint: &str) { + assert!( + status != 404 || !body.is_empty(), + "Router-level 404 for {endpoint}", + ); +} + +async fn insert_completed_job(db: &Pool) -> Uuid { + let id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, args) + VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'script', 'deno', '{}'::jsonb)", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + sqlx::query( + "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, result, status) + VALUES ($1, 'test-workspace', 100, '42'::jsonb, 'success')", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + id +} + +#[allow(dead_code)] +async fn create_script(port: u16) -> String { + let base = format!("http://localhost:{port}/api/w/test-workspace/scripts"); + let resp = authed(client().post(format!("{base}/create"))) + .json(&json!({ + "path": "u/test-user/test_job_script", + "summary": "test", + "description": "", + "content": "export function main() { return 42; }", + "language": "deno", + "schema": { + "$schema": "https://json-schema.org/draft/2020-12/schema", + "type": "object", + "properties": {}, + "required": [] + } + })) + .send() + .await + .unwrap(); + assert!( + resp.status().is_success(), + "create script: {}", + resp.status() + ); + "u/test-user/test_job_script".to_string() +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_jobs_authed_list_and_count(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/jobs"); + + // --- List/count endpoints (2xx with empty results) --- + + let resp = authed(client().get(format!("{base}/list"))).send().await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/list", + ); + + let resp = authed(client().get(format!("{base}/queue/list"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/queue/list", + ); + + let resp = authed(client().get(format!("{base}/queue/count"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/queue/count", + ); + + let resp = authed(client().get(format!("{base}/completed/list"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/list", + ); + + let resp = authed(client().get(format!("{base}/completed/count"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/count", + ); + + // --- Global endpoints --- + + let resp = client() + .get(format!("http://localhost:{port}/api/jobs/db_clock")) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/db_clock", + ); + + let resp = authed(client().get(format!( + "http://localhost:{port}/api/jobs/completed/count_by_tag" + ))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/count_by_tag", + ); + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_jobs_authed_completed_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/jobs"); + + let job_id = insert_completed_job(&db).await; + + let resp = authed(client().get(format!("{base}/completed/get/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/get", + ); + + let resp = authed(client().get(format!("{base}/completed/get_result/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/get_result", + ); + + let resp = authed(client().get(format!("{base}/completed/get_result_maybe/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/get_result_maybe", + ); + + let resp = authed(client().get(format!("{base}/completed/get_timing/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/completed/get_timing", + ); + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_jobs_authed_run_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/jobs"); + + // Run preview — no pre-existing script needed + let resp = authed(client().post(format!("{base}/run/preview"))) + .json(&json!({ + "content": "export function main() { return 1; }", + "language": "deno", + "args": {} + })) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/run/preview", + ); + + // Run preview flow + let resp = authed(client().post(format!("{base}/run/preview_flow"))) + .json(&json!({ + "value": {"modules": []}, + "args": {} + })) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/run/preview_flow", + ); + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_jobs_authed_reachability(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/jobs"); + let fake = Uuid::nil(); + + // These need complex runtime but should hit the handler (not 404) + + let resp = authed(client().post(format!("{base}/flow/resume/{fake}"))) + .json(&json!({})) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/flow/resume", + ); + + let resp = authed(client().get(format!("{base}/job_signature/{fake}/1"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/job_signature", + ); + + let resp = authed(client().get(format!("{base}/resume_urls/{fake}/1"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/resume_urls", + ); + + let resp = authed(client().get(format!("{base}/result_by_id/{fake}/step1"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /jobs/result_by_id", + ); + + let resp = authed(client().post(format!("{base}/restart/f/{fake}"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/restart/f", + ); + + let resp = authed(client().post(format!("{base}/run/workflow_as_code/{fake}/main"))) + .json(&json!({})) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /jobs/run/workflow_as_code", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/jobs_unauthed.rs b/backend/windmill-api-integration-tests/tests/jobs_unauthed.rs new file mode 100644 index 0000000000..fa8b6cb66f --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/jobs_unauthed.rs @@ -0,0 +1,250 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use uuid::Uuid; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +/// Insert a minimal completed job directly into the database for testing. +async fn insert_completed_job(db: &Pool) -> Uuid { + let id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, args) + VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'script', 'deno', '{}'::jsonb)", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + + sqlx::query( + "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, result, status) + VALUES ($1, 'test-workspace', 100, '42'::jsonb, 'success')", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + + id +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_jobs_unauthed_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u"); + + let job_id = insert_completed_job(&db).await; + + // --- No-data endpoints --- + + let resp = authed(client().post(format!("{base}/queue/get_started_at_by_ids"))) + .json(&json!([])) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /queue/get_started_at_by_ids", + ); + + // --- Completed job endpoints (unauthed service, with auth header) --- + + let resp = authed(client().get(format!("{base}/get/{job_id}"))) + .send() + .await?; + assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get"); + + let resp = authed(client().get(format!("{base}/get_logs/{job_id}"))) + .send() + .await?; + assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get_logs"); + + let resp = authed(client().get(format!("{base}/get_completed_logs_tail/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /get_completed_logs_tail", + ); + + let resp = authed(client().get(format!("{base}/get_args/{job_id}"))) + .send() + .await?; + assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /get_args"); + + let resp = authed(client().get(format!("{base}/completed/get/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /completed/get", + ); + + let resp = authed(client().get(format!("{base}/completed/get_result/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /completed/get_result", + ); + + let resp = authed(client().get(format!("{base}/completed/get_result_maybe/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /completed/get_result_maybe", + ); + + let resp = authed(client().get(format!("{base}/completed/get_timing/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /completed/get_timing", + ); + + let resp = authed(client().get(format!("{base}/getupdate/{job_id}"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /getupdate", + ); + + Ok(()) +} + +const FAKE_UUID: &str = "00000000-0000-0000-0000-000000000000"; +const FAKE_SECRET: &str = "aabb"; + +/// Reachability tests for endpoints that need complex runtime. +/// These just verify the route matches (handler runs), not 2xx. +fn assert_route_reachable(status: u16, body: &str, endpoint: &str) { + assert!( + status != 404 || !body.is_empty(), + "Router-level 404 for {endpoint}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_jobs_unauthed_complex_reachability(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u"); + + let resp = authed(client().get(format!("{base}/resume/{FAKE_UUID}/1/{FAKE_SECRET}"))) + .send() + .await?; + assert_route_reachable(resp.status().as_u16(), &resp.text().await?, "GET /resume"); + + let resp = authed(client().post(format!("{base}/cancel/{FAKE_UUID}/1/{FAKE_SECRET}"))) + .send() + .await?; + assert_route_reachable(resp.status().as_u16(), &resp.text().await?, "POST /cancel"); + + let resp = authed(client().get(format!("{base}/get_flow/{FAKE_UUID}/1/{FAKE_SECRET}"))) + .send() + .await?; + assert_route_reachable(resp.status().as_u16(), &resp.text().await?, "GET /get_flow"); + + let resp = authed(client().post(format!("{base}/queue/cancel/{FAKE_UUID}"))) + .json(&serde_json::json!({"reason": "test"})) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /queue/cancel", + ); + + let resp = authed(client().post(format!("{base}/queue/force_cancel/{FAKE_UUID}"))) + .json(&serde_json::json!({"reason": "test"})) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /queue/force_cancel", + ); + + let resp = authed(client().post(format!("{base}/flow/resume_suspended/{FAKE_UUID}"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /flow/resume_suspended", + ); + + let resp = authed(client().get(format!("{base}/flow/approval_info/{FAKE_UUID}"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /flow/approval_info", + ); + + let resp = authed(client().get(format!("{base}/get_root_job_id/{FAKE_UUID}"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /get_root_job_id", + ); + + let resp = authed(client().get(format!("{base}/get_flow_debug_info/{FAKE_UUID}"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /get_flow_debug_info", + ); + + let resp = authed(client().get(format!("{base}/get_log_file/{FAKE_UUID}/test.txt"))) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "GET /get_log_file", + ); + + let resp = authed(client().post(format!("{base}/queue/cancel_persistent/u/test-user/fake"))) + .json(&serde_json::json!({"reason": "test"})) + .send() + .await?; + assert_route_reachable( + resp.status().as_u16(), + &resp.text().await?, + "POST /queue/cancel_persistent", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/native_triggers.rs b/backend/windmill-api-integration-tests/tests/native_triggers.rs index 519e1f5262..e0a2e51ab5 100644 --- a/backend/windmill-api-integration-tests/tests/native_triggers.rs +++ b/backend/windmill-api-integration-tests/tests/native_triggers.rs @@ -422,6 +422,7 @@ async fn test_delete_integration_full_cascade(db: Pool) -> anyhow::Res "ext-1", &trigger_config, json!({"triggerType": "drive"}), + None, ) .await?; @@ -511,6 +512,7 @@ async fn test_cleanup_preserves_triggers(db: Pool) -> anyhow::Result<( "ext-1", &trigger_config, json!({"triggerType": "drive"}), + None, ) .await?; diff --git a/backend/windmill-api-integration-tests/tests/npm_proxy.rs b/backend/windmill-api-integration-tests/tests/npm_proxy.rs new file mode 100644 index 0000000000..9f3e813807 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/npm_proxy.rs @@ -0,0 +1,85 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +/// Start a mock npm registry that returns valid JSON for any GET request. +async fn start_mock_registry() -> u16 { + use axum::{routing::get, Json, Router}; + + let app = Router::new().fallback(get(|| async { + Json(json!({ + "name": "test-package", + "versions": {"1.0.0": {"name": "test-package", "version": "1.0.0"}}, + "dist-tags": {"latest": "1.0.0"} + })) + })); + let listener = tokio::net::TcpListener::bind("127.0.0.1:0").await.unwrap(); + let port = listener.local_addr().unwrap().port(); + tokio::spawn(async move { axum::serve(listener, app).await.unwrap() }); + port +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_npm_proxy_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/npm_proxy"); + + // Start mock npm registry + let mock_port = start_mock_registry().await; + let mock_url = format!("http://127.0.0.1:{mock_port}"); + + // Configure the npm registry to point to our mock + let resp = authed( + client() + .post(format!( + "http://localhost:{port}/api/settings/global/npm_config_registry" + )) + .json(&json!({"value": mock_url})), + ) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /settings/global/npm_config_registry", + ); + + // GET /metadata/{package} + let resp = authed(client().get(format!("{base}/metadata/lodash"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /npm_proxy/metadata/lodash", + ); + + // GET /resolve/{package} + let resp = authed(client().get(format!("{base}/resolve/lodash"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /npm_proxy/resolve/lodash", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/raw_apps.rs b/backend/windmill-api-integration-tests/tests/raw_apps.rs new file mode 100644 index 0000000000..97f90709b1 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/raw_apps.rs @@ -0,0 +1,33 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_raw_apps_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/raw_apps"); + + // GET /raw_apps/list → 200 (empty array) + let resp = authed(client().get(format!("{base}/list"))).send().await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /raw_apps/list"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/resources.rs b/backend/windmill-api-integration-tests/tests/resources.rs index 22dab5d418..363217712f 100644 --- a/backend/windmill-api-integration-tests/tests/resources.rs +++ b/backend/windmill-api-integration-tests/tests/resources.rs @@ -1,5 +1,7 @@ use serde_json::json; use sqlx::{Pool, Postgres}; +#[cfg(feature = "mcp")] +use uuid::Uuid; use windmill_test_utils::*; @@ -518,3 +520,182 @@ async fn test_mcp_tools(db: Pool) -> anyhow::Result<()> { Ok(()) } + +#[cfg(feature = "mcp")] +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_mcp_endpoint_tools_list(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + let resp = authed(client().get(format!( + "http://localhost:{port}/api/mcp/w/test-workspace/list_tools" + ))) + .send() + .await?; + assert_eq!(resp.status(), 200); + + let tools: Vec = resp.json().await?; + + let tool_names: Vec<&str> = tools.iter().filter_map(|t| t["name"].as_str()).collect(); + + assert!( + tool_names.contains(&"getJob"), + "getJob not found in MCP endpoint tools: {tool_names:?}" + ); + assert!( + tool_names.contains(&"getJobLogs"), + "getJobLogs not found in MCP endpoint tools: {tool_names:?}" + ); + + // Verify getJob has the expected path and method + let get_job_tool = tools.iter().find(|t| t["name"] == "getJob").unwrap(); + assert_eq!(get_job_tool["path"], "/w/{workspace}/jobs_u/get/{id}"); + assert_eq!(get_job_tool["method"], "GET"); + + // Verify getJobLogs has the expected path and method + let get_job_logs_tool = tools.iter().find(|t| t["name"] == "getJobLogs").unwrap(); + assert_eq!( + get_job_logs_tool["path"], + "/w/{workspace}/jobs_u/get_logs/{id}" + ); + assert_eq!(get_job_logs_tool["method"], "GET"); + + Ok(()) +} + +#[cfg(feature = "mcp")] +async fn insert_completed_job_with_logs(db: &Pool) -> Uuid { + let id = Uuid::new_v4(); + sqlx::query( + "INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, args) + VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'script', 'deno', '{}'::jsonb)", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + + sqlx::query( + "INSERT INTO v2_job_completed (id, workspace_id, duration_ms, result, status) + VALUES ($1, 'test-workspace', 100, '42'::jsonb, 'success')", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + + sqlx::query( + "INSERT INTO job_logs (job_id, workspace_id, logs, log_offset) + VALUES ($1, 'test-workspace', 'hello world test log', 0)", + ) + .bind(id) + .execute(db) + .await + .unwrap(); + + id +} + +#[cfg(feature = "mcp")] +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_mcp_client_get_job_and_logs(db: Pool) -> anyhow::Result<()> { + use rmcp::model::{ + CallToolRequestParams, ClientCapabilities, ClientInfo, Implementation, + InitializeRequestParams, + }; + use rmcp::service::{RoleClient, RunningService}; + use rmcp::transport::streamable_http_client::{ + StreamableHttpClientTransport, StreamableHttpClientTransportConfig, + }; + use rmcp::ServiceExt; + + initialize_tracing().await; + set_jwt_secret().await; + let server = ApiServer::start_mcp(db.clone()).await?; + let port = server.addr.port(); + + let job_id = insert_completed_job_with_logs(&db).await; + + // Create a token with MCP scopes + sqlx::query( + "INSERT INTO token (token_hash, token_prefix, token, email, label, super_admin, scopes) + VALUES (encode(sha256('MCP_TOKEN'::bytea), 'hex'), 'MCP_TOK', 'MCP_TOKEN', 'test@windmill.dev', 'mcp token', true, ARRAY['mcp:all'])", + ) + .execute(&db) + .await?; + + // Connect as MCP client + let config = StreamableHttpClientTransportConfig::with_uri(format!( + "http://localhost:{port}/api/mcp/w/test-workspace/mcp" + )) + .auth_header("MCP_TOKEN"); + let transport = StreamableHttpClientTransport::from_config(config); + + let client_info = ClientInfo { + protocol_version: Default::default(), + capabilities: ClientCapabilities::default(), + client_info: Implementation { + name: "test-client".to_string(), + title: None, + version: "0.0.1".to_string(), + description: None, + website_url: None, + icons: None, + }, + meta: None, + }; + + let client: RunningService = + client_info.serve(transport).await?; + + // --- Test getJob --- + let result = client + .call_tool(CallToolRequestParams { + name: "getJob".into(), + arguments: Some(serde_json::from_value(json!({ "id": job_id.to_string() }))?), + task: None, + meta: None, + }) + .await?; + + let text = result + .content + .first() + .and_then(|c| c.raw.as_text()) + .expect("getJob should return text content"); + let job: serde_json::Value = serde_json::from_str(&text.text)?; + assert_eq!(job["id"], job_id.to_string()); + assert_eq!(job["workspace_id"], "test-workspace"); + assert_eq!(job["created_by"], "test-user"); + assert_eq!(job["job_kind"], "script"); + assert!( + job["success"].as_bool().unwrap_or(false), + "job should be successful: {job}" + ); + + // --- Test getJobLogs --- + let result = client + .call_tool(CallToolRequestParams { + name: "getJobLogs".into(), + arguments: Some(serde_json::from_value(json!({ "id": job_id.to_string() }))?), + task: None, + meta: None, + }) + .await?; + + let text = result + .content + .first() + .and_then(|c| c.raw.as_text()) + .expect("getJobLogs should return text content"); + // The logs endpoint returns text/plain, which gets wrapped as a JSON string by call_endpoint + let logs: String = serde_json::from_str(&text.text)?; + assert!( + logs.contains("hello world test log"), + "expected logs to contain test log, got: {logs}" + ); + + client.cancel().await?; + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/scripts.rs b/backend/windmill-api-integration-tests/tests/scripts.rs index 74fd9c8611..f5e78f880f 100644 --- a/backend/windmill-api-integration-tests/tests/scripts.rs +++ b/backend/windmill-api-integration-tests/tests/scripts.rs @@ -108,7 +108,10 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { let resp = authed_get(port, "raw/p", "u/test-user/test_script.ts").await; assert_eq!(resp.status(), 200); let body = resp.text().await?; - assert!(body.contains("return 42"), "expected script content, got: {body}"); + assert!( + body.contains("return 42"), + "expected script content, got: {body}" + ); // --- raw by hash (requires .ts suffix) --- let resp = authed_get(port, "raw/h", &format!("{hash}.ts")).await; @@ -131,12 +134,10 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { assert!(list.iter().any(|s| s["path"] == "u/test-user/test_script")); // list with path_start filter - let resp = authed(client().get(format!( - "{base}/list?path_start=u/test-user/another" - ))) - .send() - .await - .unwrap(); + let resp = authed(client().get(format!("{base}/list?path_start=u/test-user/another"))) + .send() + .await + .unwrap(); assert_eq!(resp.status(), 200); let list = resp.json::>().await?; assert_eq!(list.len(), 1); @@ -233,12 +234,7 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { .send() .await .unwrap(); - assert_eq!( - resp.status(), - 200, - "history_update: {}", - resp.text().await? - ); + assert_eq!(resp.status(), 200, "history_update: {}", resp.text().await?); // --- toggle_workspace_error_handler (EE-gated, expect 400 in OSS) --- let resp = authed(client().post(script_url( @@ -268,22 +264,13 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { .send() .await .unwrap(); - assert_eq!( - resp.status(), - 200, - "tokened_raw: {}", - resp.text().await? - ); + assert_eq!(resp.status(), 200, "tokened_raw: {}", resp.text().await?); // --- archive by path --- - let resp = authed(client().post(script_url( - port, - "archive/p", - "u/test-user/another_script", - ))) - .send() - .await - .unwrap(); + let resp = authed(client().post(script_url(port, "archive/p", "u/test-user/another_script"))) + .send() + .await + .unwrap(); assert_eq!(resp.status(), 200); // archived script should still be gettable @@ -333,12 +320,10 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { // ===== Hub endpoints (require external network, expect 500 or 200) ===== // --- hub/top --- - let resp = authed(client().get(format!( - "http://localhost:{port}/api/scripts/hub/top" - ))) - .send() - .await - .unwrap(); + let resp = authed(client().get(format!("http://localhost:{port}/api/scripts/hub/top"))) + .send() + .await + .unwrap(); assert!( resp.status() == 200 || resp.status() == 500, "hub/top: unexpected status {}", @@ -372,12 +357,10 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { ); // --- integrations hub/list --- - let resp = authed(client().get(format!( - "http://localhost:{port}/api/integrations/hub/list" - ))) - .send() - .await - .unwrap(); + let resp = authed(client().get(format!("http://localhost:{port}/api/integrations/hub/list"))) + .send() + .await + .unwrap(); assert!( resp.status() == 200 || resp.status() == 500, "integrations hub/list: unexpected status {}", @@ -386,3 +369,97 @@ async fn test_script_endpoints(db: Pool) -> anyhow::Result<()> { Ok(()) } + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_auto_parent_resolves_parent_hash(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/scripts"); + + // Create v1 + let resp = authed(client().post(format!("{base}/create"))) + .json(&new_script( + "u/test-user/auto_parent_test", + "v1", + "export async function main() { return 1; }", + )) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 201, "create v1: {}", resp.text().await?); + + // Get the hash of v1 + let resp = authed_get(port, "get/p", "u/test-user/auto_parent_test").await; + let body = resp.json::().await?; + let v1_hash = body["hash"].as_str().unwrap().to_string(); + + // Create v2 using auto_parent (no parent_hash provided) + let mut v2 = new_script( + "u/test-user/auto_parent_test", + "v2", + "export async function main() { return 2; }", + ); + v2["auto_parent"] = json!(true); + let resp = authed(client().post(format!("{base}/create"))) + .json(&v2) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 201, + "create v2 with auto_parent: {}", + resp.text().await? + ); + + // Get v2 and verify its parent_hash points to v1 + let resp = authed_get(port, "get/p", "u/test-user/auto_parent_test").await; + let body = resp.json::().await?; + assert_eq!(body["summary"], "v2"); + let v2_hash = body["hash"].as_str().unwrap().to_string(); + assert_ne!(v2_hash, v1_hash); + + // v2's parent_hashes should contain v1 + let parent_hashes = body["parent_hashes"].as_array().unwrap(); + assert!( + parent_hashes + .iter() + .any(|h| h.as_str() == Some(v1_hash.as_str())), + "v2 parent_hashes should contain v1 hash {v1_hash}, got: {parent_hashes:?}" + ); + + // Create v3 with auto_parent to confirm it chains correctly + let mut v3 = new_script( + "u/test-user/auto_parent_test", + "v3", + "export async function main() { return 3; }", + ); + v3["auto_parent"] = json!(true); + let resp = authed(client().post(format!("{base}/create"))) + .json(&v3) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 201, + "create v3 with auto_parent: {}", + resp.text().await? + ); + + let resp = authed_get(port, "get/p", "u/test-user/auto_parent_test").await; + let body = resp.json::().await?; + assert_eq!(body["summary"], "v3"); + + // v3's parent_hashes should contain v2 (and transitively v1) + let parent_hashes = body["parent_hashes"].as_array().unwrap(); + assert!( + parent_hashes + .iter() + .any(|h| h.as_str() == Some(v2_hash.as_str())), + "v3 parent_hashes should contain v2 hash {v2_hash}, got: {parent_hashes:?}" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/sensitive_log_masking.rs b/backend/windmill-api-integration-tests/tests/sensitive_log_masking.rs new file mode 100644 index 0000000000..d174f3759b --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/sensitive_log_masking.rs @@ -0,0 +1,465 @@ +//! Integration tests for sensitive log masking. +//! +//! A single comprehensive test that runs real bun scripts through real workers, +//! covering all masking scenarios: secret variables, non-secret variables, +//! multiple secrets, mid-string secrets, `$encrypted:` args, resources +//! referencing secret variables, and cross-job isolation. +//! +//! Run with: +//! cargo test -p windmill-api-integration-tests --test sensitive_log_masking -- --nocapture +//! +//! Requires: bun runtime, live database (migrations applied by sqlx::test). + +use futures::StreamExt; +use serde_json::json; +use sqlx::{Pool, Postgres}; +use uuid::Uuid; +use windmill_common::jobs::{JobPayload, RawCode}; +use windmill_common::scripts::ScriptLang; +use windmill_common::worker::to_raw_value; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +/// Helper: create a variable via the API. +async fn create_variable(port: u16, path: &str, value: &str, is_secret: bool) { + let base = format!("http://localhost:{port}/api/w/test-workspace/variables"); + let resp = authed(client().post(format!("{base}/create"))) + .json(&json!({ + "path": path, + "value": value, + "is_secret": is_secret, + "description": "test variable for log masking" + })) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 201, + "failed to create variable {path}: {}", + resp.text().await.unwrap_or_default() + ); +} + +/// Helper: create a resource via the API. +async fn create_resource(port: u16, path: &str, value: serde_json::Value) { + let base = format!("http://localhost:{port}/api/w/test-workspace/resources"); + let resp = authed(client().post(format!("{base}/create"))) + .json(&json!({ + "path": path, + "value": value, + "resource_type": "object", + "description": "test resource for log masking" + })) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 201, + "failed to create resource {path}: {}", + resp.text().await.unwrap_or_default() + ); +} + +/// Helper: fetch job logs from the job_logs table. +async fn get_job_logs(db: &Pool, job_id: Uuid) -> Option { + sqlx::query_scalar!( + r#"SELECT logs as "logs!" FROM job_logs WHERE job_id = $1"#, + job_id, + ) + .fetch_optional(db) + .await + .unwrap() +} + +/// Helper: push a bun preview job and return its UUID. +async fn push_bun_job(db: &Pool, code: String) -> Uuid { + RunJob::from(JobPayload::Code(RawCode { + hash: None, + content: code, + path: None, + language: ScriptLang::Bun, + lock: None, + cache_ttl: None, + cache_ignore_s3_path: None, + dedicated_worker: None, + concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default() + .into(), + debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(), + modules: None, + })) + .push(db) + .await +} + +/// Helper: push a bun preview job with encrypted args. +async fn push_bun_job_with_encrypted_arg( + db: &Pool, + code: String, + arg_name: &str, + plaintext_value: &str, +) -> Uuid { + // We need to know the job_id in advance to encrypt with the right key suffix. + let job_id = Uuid::new_v4(); + + // Encrypt the value the same way the frontend does: + // build_crypt_with_key_suffix(db, workspace, root_job_id) + let mc = windmill_common::variables::build_crypt_with_key_suffix( + db, + "test-workspace", + &job_id.to_string(), + ) + .await + .expect("build_crypt_with_key_suffix"); + + // Encrypt the JSON-serialized string value + let json_str = serde_json::to_string(plaintext_value).unwrap(); + let encrypted = windmill_common::variables::encrypt(&mc, &json_str); + let arg_value = format!("$encrypted:{encrypted}"); + + let mut args = std::collections::HashMap::new(); + args.insert(arg_name.to_string(), to_raw_value(&json!(arg_value))); + + RunJob::from(JobPayload::Code(RawCode { + hash: None, + content: code, + path: None, + language: ScriptLang::Bun, + lock: None, + cache_ttl: None, + cache_ignore_s3_path: None, + dedicated_worker: None, + concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default() + .into(), + debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(), + modules: None, + })) + .job_id(job_id) + .arg(arg_name, json!(arg_value)) + .push(db) + .await +} + +/// Comprehensive test covering all sensitive log masking scenarios in a single +/// test function to amortize server/worker startup cost. +/// +/// Scenarios covered (each as a separate job inside the same worker): +/// 1. Secret variable fetched and logged → masked +/// 2. Non-secret variable fetched and logged → NOT masked (no false positives) +/// 3. Two different secrets fetched and logged in the same job → both masked +/// 4. Secret embedded mid-string (e.g. "token=SECRET&user=bob") → masked +/// 5. Same secret logged 3 times → all occurrences masked +/// 6. `$encrypted:` password arg logged → masked +/// 7. Resource referencing a secret variable via `$var:` → secret masked when logged +/// 8. Cross-job isolation: job A's secret does NOT leak into job B's logs +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_sensitive_log_masking(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + + // === Setup: create variables and resources === + let secret1 = "alpha_secret_value_9x7k2m"; + let secret2 = "beta_secret_token_4j8n3p"; + let plain_val = "plain_visible_value_12345"; + let encrypted_password = "encrypted_pass_w0rd_zq5r"; + let resource_secret = "resource_db_password_h7t2"; + + create_variable(port, "u/test-user/secret_alpha", secret1, true).await; + create_variable(port, "u/test-user/secret_beta", secret2, true).await; + create_variable(port, "u/test-user/plain_var", plain_val, false).await; + // Secret variable that will be referenced by a resource via $var: + create_variable(port, "u/test-user/res_secret_var", resource_secret, true).await; + // Resource whose "password" field references the secret variable + create_resource( + port, + "u/test-user/db_with_secret", + json!({"host": "db.example.com", "password": "$var:u/test-user/res_secret_var"}), + ) + .await; + + let mut completed = listen_for_completed_jobs(&db).await; + let db2 = db.clone(); + in_test_worker( + db.clone(), + async move { + // ================================================================ + // Scenario 1: Secret variable fetched and console.logged → masked + // ================================================================ + let job1 = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const secret = await wmill.getVariable("u/test-user/secret_alpha"); + console.log("The secret value is: " + secret); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob1 = completed_job(job1, &db2).await; + assert!(cjob1.success, "scenario 1 job failed"); + let logs1 = get_job_logs(&db2, job1).await.expect("scenario 1: no logs"); + + assert!( + !logs1.contains(secret1), + "scenario 1: secret value leaked in logs\nLogs:\n{logs1}" + ); + assert!( + logs1.contains("The secret value is: alp*****k2m"), + "scenario 1: expected masked output with first 3 + last 3 chars\nLogs:\n{logs1}" + ); + assert!( + logs1.contains("[windmill] secret value was masked for security reasons, use string transformations to display full value"), + "scenario 1: expected security notice\nLogs:\n{logs1}" + ); + + // ================================================================ + // Scenario 2: Non-secret variable → NOT masked (no false positives) + // ================================================================ + let job2 = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const val = await wmill.getVariable("u/test-user/plain_var"); + console.log("The plain value is: " + val); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob2 = completed_job(job2, &db2).await; + assert!(cjob2.success, "scenario 2 job failed"); + let logs2 = get_job_logs(&db2, job2).await.expect("scenario 2: no logs"); + + assert!( + logs2.contains(plain_val), + "scenario 2: plain value should appear unmasked\nLogs:\n{logs2}" + ); + + // ================================================================ + // Scenario 3: Two different secrets fetched in the same job → both masked + // ================================================================ + let job3 = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const s1 = await wmill.getVariable("u/test-user/secret_alpha"); + const s2 = await wmill.getVariable("u/test-user/secret_beta"); + console.log("secret1=" + s1); + console.log("secret2=" + s2); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob3 = completed_job(job3, &db2).await; + assert!(cjob3.success, "scenario 3 job failed"); + let logs3 = get_job_logs(&db2, job3).await.expect("scenario 3: no logs"); + + assert!( + !logs3.contains(secret1), + "scenario 3: secret1 leaked\nLogs:\n{logs3}" + ); + assert!( + !logs3.contains(secret2), + "scenario 3: secret2 leaked\nLogs:\n{logs3}" + ); + assert!( + logs3.contains("secret1=alp*****k2m"), + "scenario 3: secret1 not masked\nLogs:\n{logs3}" + ); + assert!( + logs3.contains("secret2=bet*****n3p"), + "scenario 3: secret2 not masked\nLogs:\n{logs3}" + ); + + // ================================================================ + // Scenario 4: Secret embedded mid-string → masked in place + // ================================================================ + let job4 = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const secret = await wmill.getVariable("u/test-user/secret_alpha"); + console.log("token=" + secret + "&user=bob&format=json"); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob4 = completed_job(job4, &db2).await; + assert!(cjob4.success, "scenario 4 job failed"); + let logs4 = get_job_logs(&db2, job4).await.expect("scenario 4: no logs"); + + assert!( + !logs4.contains(secret1), + "scenario 4: secret leaked mid-string\nLogs:\n{logs4}" + ); + assert!( + logs4.contains("token=alp*****k2m&user=bob&format=json"), + "scenario 4: mid-string masking failed\nLogs:\n{logs4}" + ); + + // ================================================================ + // Scenario 5: Same secret logged 3 times → all occurrences masked + // ================================================================ + let job5 = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const secret = await wmill.getVariable("u/test-user/secret_beta"); + console.log("First: " + secret); + console.log("Second: " + secret); + console.log("Third: " + secret); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob5 = completed_job(job5, &db2).await; + assert!(cjob5.success, "scenario 5 job failed"); + let logs5 = get_job_logs(&db2, job5).await.expect("scenario 5: no logs"); + + assert!( + !logs5.contains(secret2), + "scenario 5: secret leaked\nLogs:\n{logs5}" + ); + let mask_count = logs5.matches("bet*****n3p").count(); + assert!( + mask_count >= 3, + "scenario 5: expected >= 3 masked occurrences, found {mask_count}\nLogs:\n{logs5}" + ); + // Security notice should appear only once even though masking happened 3 times + let notice_count = logs5.matches("[windmill] secret value was masked").count(); + assert_eq!( + notice_count, 1, + "scenario 5: security notice should appear exactly once, found {notice_count}\nLogs:\n{logs5}" + ); + + // ================================================================ + // Scenario 6: $encrypted: password arg → masked when logged + // ================================================================ + let job6 = push_bun_job_with_encrypted_arg( + &db2, + r#"export async function main(password: string) { + console.log("password is: " + password); + return "ok"; +}"# + .into(), + "password", + encrypted_password, + ) + .await; + completed.next().await; + let cjob6 = completed_job(job6, &db2).await; + assert!(cjob6.success, "scenario 6 job failed"); + let logs6 = get_job_logs(&db2, job6).await.expect("scenario 6: no logs"); + + assert!( + !logs6.contains(encrypted_password), + "scenario 6: encrypted password leaked\nLogs:\n{logs6}" + ); + assert!( + logs6.contains("password is: enc*****q5r"), + "scenario 6: encrypted password not masked\nLogs:\n{logs6}" + ); + + // ================================================================ + // Scenario 7: Resource with $var: referencing a secret → masked + // ================================================================ + let job7 = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const res = await wmill.getResource("u/test-user/db_with_secret"); + console.log("db password: " + res.password); + console.log("db host: " + res.host); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob7 = completed_job(job7, &db2).await; + assert!(cjob7.success, "scenario 7 job failed"); + let logs7 = get_job_logs(&db2, job7).await.expect("scenario 7: no logs"); + + assert!( + !logs7.contains(resource_secret), + "scenario 7: resource secret leaked\nLogs:\n{logs7}" + ); + assert!( + logs7.contains("db password: res*****7t2"), + "scenario 7: resource secret not masked\nLogs:\n{logs7}" + ); + // Non-secret field should remain visible + assert!( + logs7.contains("db host: db.example.com"), + "scenario 7: non-secret resource field should be visible\nLogs:\n{logs7}" + ); + + // ================================================================ + // Scenario 8: Cross-job isolation — job A fetches secret_alpha, + // then job B logs "alpha_secret_value_9x7k2m" as a + // literal string (not fetched as a secret). + // Job B should NOT mask it because the secret belongs + // to job A which already completed. + // ================================================================ + // Job A: fetch the secret (registers it) then completes + let job_a = push_bun_job( + &db2, + r#"import * as wmill from "windmill-client"; +export async function main() { + const s = await wmill.getVariable("u/test-user/secret_alpha"); + console.log("fetched secret"); + return "ok"; +}"# + .into(), + ) + .await; + completed.next().await; + let cjob_a = completed_job(job_a, &db2).await; + assert!(cjob_a.success, "scenario 8 job A failed"); + + // Job B: logs the same string as a hardcoded literal (NOT fetched as secret) + // Since job A already completed and unregistered, and job B never + // fetched the secret, it should NOT be masked. + let job_b_code = format!( + r#"export async function main() {{ + console.log("literal value: {secret1}"); + return "ok"; +}}"# + ); + let job_b = push_bun_job(&db2, job_b_code).await; + completed.next().await; + let cjob_b = completed_job(job_b, &db2).await; + assert!(cjob_b.success, "scenario 8 job B failed"); + let logs_b = get_job_logs(&db2, job_b) + .await + .expect("scenario 8 job B: no logs"); + + assert!( + logs_b.contains(secret1), + "scenario 8: job B should show the literal string unmasked (it never fetched a secret)\nLogs:\n{logs_b}" + ); + }, + port, + ) + .await; + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/service_logs.rs b/backend/windmill-api-integration-tests/tests/service_logs.rs new file mode 100644 index 0000000000..0c66916053 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/service_logs.rs @@ -0,0 +1,36 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_service_logs_2xx(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/service_logs"); + + let resp = authed(client().get(format!("{base}/list_files"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /list_files", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/settings.rs b/backend/windmill-api-integration-tests/tests/settings.rs new file mode 100644 index 0000000000..8f21ffb483 --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/settings.rs @@ -0,0 +1,116 @@ +use serde_json::json; +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_settings_2xx(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/settings"); + + let resp = authed(client().get(format!("{base}/envs"))).send().await?; + assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /envs"); + + let resp = authed(client().get(format!("{base}/global/hub_base_url"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /global/hub_base_url", + ); + + let resp = authed(client().post(format!("{base}/global/test_key"))) + .json(&json!({"value": "test"})) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /global/test_key", + ); + + let resp = authed(client().get(format!("{base}/instance_config"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /instance_config", + ); + + let resp = authed(client().get(format!("{base}/instance_config/yaml"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /instance_config/yaml", + ); + + let resp = authed(client().get(format!("{base}/latest_key_renewal_attempt"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /latest_key_renewal_attempt", + ); + + let resp = authed(client().post(format!("{base}/sync_cached_resource_types"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "POST /sync_cached_resource_types", + ); + + // --- Reachability only (need external services) --- + + let resp = authed( + client() + .post(format!("{base}/test_smtp")) + .json(&json!({"to": "test@test.com", "subject": "test", "content": "test"})), + ) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert!( + status != 404 || !body.is_empty(), + "Router-level 404 for POST /test_smtp" + ); + + let resp = authed( + client() + .post(format!("{base}/test_license_key")) + .json(&json!({"license_key": "fake"})), + ) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert!( + status != 404 || !body.is_empty(), + "Router-level 404 for POST /test_license_key" + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/trash.rs b/backend/windmill-api-integration-tests/tests/trash.rs new file mode 100644 index 0000000000..df32a5b3fc --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/trash.rs @@ -0,0 +1,41 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_trash_endpoints(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/trash"); + + // GET /trash/list → 200 (admin, empty array) + let resp = authed(client().get(format!("{base}/list"))).send().await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "GET /trash/list"); + + // POST /trash/empty → 200 (admin) + let resp = authed(client().post(format!("{base}/empty"))) + .send() + .await?; + let status = resp.status().as_u16(); + let body = resp.text().await?; + assert_2xx(status, &body, "POST /trash/empty"); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/workspace_dependencies_git_sync.rs b/backend/windmill-api-integration-tests/tests/workspace_dependencies_git_sync.rs index 72a102dfd8..9226d3f74a 100644 --- a/backend/windmill-api-integration-tests/tests/workspace_dependencies_git_sync.rs +++ b/backend/windmill-api-integration-tests/tests/workspace_dependencies_git_sync.rs @@ -14,6 +14,7 @@ use serde_json::json; use sqlx::{Pool, Postgres}; use std::time::Duration; +#[allow(unused_imports)] use windmill_test_utils::*; /// Row shape for querying deployment callback jobs from v2_job_queue @@ -27,6 +28,7 @@ struct DeploymentCallbackJob { } /// Poll for deployment callback jobs in the queue for a given script path +#[allow(dead_code)] async fn get_deployment_callback_jobs( db: &Pool, script_path: &str, @@ -63,6 +65,7 @@ async fn get_deployment_callback_jobs( } /// Configure git sync for the test workspace with workspace dependencies enabled +#[allow(dead_code)] async fn setup_git_sync_config(db: &Pool, sync_script_path: &str) -> anyhow::Result<()> { let git_sync_config = json!({ "include_type": ["workspacedependencies"], @@ -87,6 +90,7 @@ async fn setup_git_sync_config(db: &Pool, sync_script_path: &str) -> a } /// Create a git repository resource for testing +#[allow(dead_code)] async fn create_git_repo_resource(db: &Pool) -> anyhow::Result<()> { sqlx::query( r#" @@ -107,6 +111,7 @@ async fn create_git_repo_resource(db: &Pool) -> anyhow::Result<()> { } /// Create a dummy sync script for testing (with version >= 28103 for debouncing support) +#[allow(dead_code)] async fn create_sync_script(db: &Pool, path: &str) -> anyhow::Result { let hash: i64 = rand::random::().unsigned_abs() as i64; sqlx::query( @@ -126,6 +131,7 @@ async fn create_sync_script(db: &Pool, path: &str) -> anyhow::Result, name: &str) -> anyhow::Result<()> { sqlx::query( r#" diff --git a/backend/windmill-api-integration-tests/tests/workspace_deps.rs b/backend/windmill-api-integration-tests/tests/workspace_deps.rs new file mode 100644 index 0000000000..dcfe8877dd --- /dev/null +++ b/backend/windmill-api-integration-tests/tests/workspace_deps.rs @@ -0,0 +1,39 @@ +use sqlx::{Pool, Postgres}; +use windmill_test_utils::*; + +fn client() -> reqwest::Client { + reqwest::Client::new() +} + +fn authed(builder: reqwest::RequestBuilder) -> reqwest::RequestBuilder { + builder.header("Authorization", "Bearer SECRET_TOKEN") +} + +fn assert_2xx(status: u16, body: &str, endpoint: &str) { + assert!( + (200..300).contains(&status), + "{endpoint} returned {status}: {body}", + ); +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_workspace_deps_2xx(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/workspace_dependencies"); + + let resp = authed(client().get(format!("{base}/list"))).send().await?; + assert_2xx(resp.status().as_u16(), &resp.text().await?, "GET /list"); + + let resp = authed(client().get(format!("{base}/get_latest/python3"))) + .send() + .await?; + assert_2xx( + resp.status().as_u16(), + &resp.text().await?, + "GET /get_latest/python3", + ); + + Ok(()) +} diff --git a/backend/windmill-api-integration-tests/tests/workspaces.rs b/backend/windmill-api-integration-tests/tests/workspaces.rs index 58c07965dc..131cfbbae5 100644 --- a/backend/windmill-api-integration-tests/tests/workspaces.rs +++ b/backend/windmill-api-integration-tests/tests/workspaces.rs @@ -82,12 +82,10 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { assert_eq!(resp.status(), 200); // --- allowed_domain_auto_invite --- - let resp = authed(client().get(format!( - "{global_base}/allowed_domain_auto_invite" - ))) - .send() - .await - .unwrap(); + let resp = authed(client().get(format!("{global_base}/allowed_domain_auto_invite"))) + .send() + .await + .unwrap(); assert_eq!(resp.status(), 200); resp.json::().await?; @@ -213,12 +211,10 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { resp.json::>().await?; // --- get_dependents (empty, no dependencies exist) --- - let resp = authed(client().get(format!( - "{base}/get_dependents/u/test-user/nonexistent" - ))) - .send() - .await - .unwrap(); + let resp = authed(client().get(format!("{base}/get_dependents/u/test-user/nonexistent"))) + .send() + .await + .unwrap(); assert_eq!(resp.status(), 200); let dependents = resp.json::>().await?; assert!(dependents.is_empty()); @@ -425,13 +421,11 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { ); // --- edit_large_file_storage_config --- - let resp = authed(client().post(format!( - "{base}/edit_large_file_storage_config" - ))) - .json(&json!({"large_file_storage": null})) - .send() - .await - .unwrap(); + let resp = authed(client().post(format!("{base}/edit_large_file_storage_config"))) + .json(&json!({"large_file_storage": null})) + .send() + .await + .unwrap(); assert_eq!( resp.status(), 200, @@ -532,9 +526,7 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { .unwrap(); let invites = resp.json::>().await?; assert!( - invites - .iter() - .any(|i| i["email"] == "invited@example.com"), + invites.iter().any(|i| i["email"] == "invited@example.com"), "invite not found: {:?}", invites ); @@ -549,12 +541,7 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { .send() .await .unwrap(); - assert_eq!( - resp.status(), - 201, - "delete_invite: {}", - resp.text().await? - ); + assert_eq!(resp.status(), 201, "delete_invite: {}", resp.text().await?); // ===== Critical alerts (EE-gated, returns 404 in OSS) ===== @@ -612,64 +599,60 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { .unwrap(); assert_eq!(resp.status(), 200, "tarball: {}", resp.status()); - // ===== Fork operations (on the newly created workspace) ===== + // ===== Fork operations (EE-only: CE limits workspace count to 2) ===== + #[cfg(feature = "enterprise")] + { + let new_ws_base = format!("http://localhost:{port}/api/w/new-test-ws/workspaces"); + let resp = authed(client().post(format!("{new_ws_base}/create_fork"))) + .json(&json!({ + "id": "wm-fork-test-ws", + "name": "Forked Test Workspace" + })) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200, "create_fork: {}", resp.text().await?); - // --- create_fork (workspace-scoped, from new-test-ws) --- - let new_ws_base = format!("http://localhost:{port}/api/w/new-test-ws/workspaces"); - let resp = authed(client().post(format!("{new_ws_base}/create_fork"))) - .json(&json!({ - "id": "wm-fork-test-ws", - "name": "Forked Test Workspace" - })) - .send() - .await - .unwrap(); - assert_eq!( - resp.status(), - 200, - "create_fork: {}", - resp.text().await? - ); + // verify fork exists + let resp = authed(client().post(format!("{global_base}/exists"))) + .json(&json!({"id": "wm-fork-test-ws"})) + .send() + .await + .unwrap(); + assert_eq!(resp.json::().await?, true); - // verify fork exists - let resp = authed(client().post(format!("{global_base}/exists"))) - .json(&json!({"id": "wm-fork-test-ws"})) - .send() - .await - .unwrap(); - assert_eq!(resp.json::().await?, true); + // --- change_workspace_id --- + let fork_ws_base = format!("http://localhost:{port}/api/w/wm-fork-test-ws/workspaces"); + let resp = authed(client().post(format!("{fork_ws_base}/change_workspace_id"))) + .json(&json!({ + "new_id": "wm-fork-renamed", + "new_name": "Renamed Fork" + })) + .send() + .await + .unwrap(); + assert_eq!( + resp.status(), + 200, + "change_workspace_id: {}", + resp.text().await? + ); - // --- change_workspace_id --- - let fork_ws_base = format!("http://localhost:{port}/api/w/wm-fork-test-ws/workspaces"); - let resp = authed(client().post(format!("{fork_ws_base}/change_workspace_id"))) - .json(&json!({ - "new_id": "wm-fork-renamed", - "new_name": "Renamed Fork" - })) - .send() - .await - .unwrap(); - assert_eq!( - resp.status(), - 200, - "change_workspace_id: {}", - resp.text().await? - ); + // verify renamed workspace exists + let resp = authed(client().post(format!("{global_base}/exists"))) + .json(&json!({"id": "wm-fork-renamed"})) + .send() + .await + .unwrap(); + assert_eq!(resp.json::().await?, true); - // verify renamed workspace exists - let resp = authed(client().post(format!("{global_base}/exists"))) - .json(&json!({"id": "wm-fork-renamed"})) - .send() - .await - .unwrap(); - assert_eq!(resp.json::().await?, true); - - // clean up renamed fork - let resp = authed(client().delete(format!("{global_base}/delete/wm-fork-renamed"))) - .send() - .await - .unwrap(); - assert_eq!(resp.status(), 200); + // clean up renamed fork + let resp = authed(client().delete(format!("{global_base}/delete/wm-fork-renamed"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + } // --- archive workspace (on the newly created one, not our main test workspace) --- let new_ws_base = format!("http://localhost:{port}/api/w/new-test-ws/workspaces"); @@ -702,13 +685,140 @@ async fn test_workspace_endpoints(db: Pool) -> anyhow::Result<()> { assert_eq!(resp.json::().await?, false); // --- create_workspace_require_superadmin --- - let resp = authed(client().get(format!( - "{global_base}/create_workspace_require_superadmin" - ))) - .send() - .await - .unwrap(); + let resp = authed(client().get(format!("{global_base}/create_workspace_require_superadmin"))) + .send() + .await + .unwrap(); assert_eq!(resp.status(), 200); Ok(()) } + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_get_copilot_settings_state_reports_instance_ai_fallback_flags( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/workspaces"); + + let instance_ai_config = json!({ + "providers": { + "openai": { + "resource_path": "u/test-user/openai_instance", + "models": ["gpt-4o-mini"] + } + } + }); + let workspace_ai_config = json!({ + "providers": { + "anthropic": { + "resource_path": "u/test-user/anthropic_workspace", + "models": ["claude-3-5-haiku-latest"] + } + } + }); + + sqlx::query("UPDATE workspace_settings SET ai_config = NULL WHERE workspace_id = $1") + .bind("test-workspace") + .execute(&db) + .await?; + sqlx::query( + "INSERT INTO global_settings (name, value) VALUES ($1, $2) \ + ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value", + ) + .bind("ai_config") + .bind(instance_ai_config) + .execute(&db) + .await?; + + let resp = authed(client().get(format!("{base}/get_copilot_settings_state"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + let settings = resp.json::().await?; + assert_eq!(settings["has_instance_ai_config"], true); + assert_eq!(settings["uses_instance_ai_config"], true); + assert_eq!( + settings["instance_ai_summary"]["providers"][0]["provider"], + "openai" + ); + assert_eq!( + settings["instance_ai_summary"]["providers"][0]["models"][0], + "gpt-4o-mini" + ); + + sqlx::query("UPDATE workspace_settings SET ai_config = $1 WHERE workspace_id = $2") + .bind(workspace_ai_config) + .bind("test-workspace") + .execute(&db) + .await?; + + let resp = authed(client().get(format!("{base}/get_copilot_settings_state"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + let settings = resp.json::().await?; + assert_eq!(settings["has_instance_ai_config"], true); + assert_eq!(settings["uses_instance_ai_config"], false); + assert_eq!( + settings["instance_ai_summary"]["providers"][0]["provider"], + "openai" + ); + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_get_copilot_info_ignores_empty_instance_ai_row( + db: Pool, +) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/workspaces"); + + sqlx::query("UPDATE workspace_settings SET ai_config = NULL WHERE workspace_id = $1") + .bind("test-workspace") + .execute(&db) + .await?; + sqlx::query( + "INSERT INTO global_settings (name, value) VALUES ($1, $2) \ + ON CONFLICT (name) DO UPDATE SET value = EXCLUDED.value", + ) + .bind("ai_config") + .bind(json!({})) + .execute(&db) + .await?; + + let resp = authed(client().get(format!("{base}/get_copilot_info"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + let settings = resp.json::().await?; + assert!(settings["providers"].is_null()); + + Ok(()) +} + +#[sqlx::test(migrations = "../migrations", fixtures("base"))] +async fn test_get_imports(db: Pool) -> anyhow::Result<()> { + initialize_tracing().await; + let server = ApiServer::start(db.clone()).await?; + let port = server.addr.port(); + let base = format!("http://localhost:{port}/api/w/test-workspace/workspaces"); + + let resp = authed(client().get(format!("{base}/get_imports/u/test-user/nonexistent_script"))) + .send() + .await + .unwrap(); + assert_eq!(resp.status(), 200); + let imports = resp.json::>().await?; + assert!(imports.is_empty()); + + Ok(()) +} diff --git a/backend/windmill-api-jobs/src/concurrency_groups.rs b/backend/windmill-api-jobs/src/concurrency_groups.rs index 33f6045e49..f3ef3b14a8 100644 --- a/backend/windmill-api-jobs/src/concurrency_groups.rs +++ b/backend/windmill-api-jobs/src/concurrency_groups.rs @@ -25,8 +25,8 @@ use uuid::Uuid; pub fn global_service() -> Router { Router::new() .route("/list", get(list_concurrency_groups)) - .route("/prune/*concurrency_key", delete(prune_concurrency_group)) - .route("/:job_id/key", get(get_concurrency_key)) + .route("/prune/{*concurrency_key}", delete(prune_concurrency_group)) + .route("/{job_id}/key", get(get_concurrency_key)) } pub fn workspaced_service() -> Router { diff --git a/backend/windmill-api-jobs/src/job_metrics.rs b/backend/windmill-api-jobs/src/job_metrics.rs index ad316150c0..59cb23deff 100644 --- a/backend/windmill-api-jobs/src/job_metrics.rs +++ b/backend/windmill-api-jobs/src/job_metrics.rs @@ -22,13 +22,13 @@ pub fn workspaced_service() -> Router { .allow_origin(Any); Router::new() - .route("/get/:id", post(get_job_metrics).layer(cors.clone())) + .route("/get/{id}", post(get_job_metrics).layer(cors.clone())) .route( - "/set_progress/:id", + "/set_progress/{id}", post(set_job_progress).layer(cors.clone()), ) .route( - "/get_progress/:id", + "/get_progress/{id}", get(get_job_progress).layer(cors.clone()), ) } diff --git a/backend/windmill-api-jobs/src/query.rs b/backend/windmill-api-jobs/src/query.rs index d8128d90e7..0d7503b5bc 100644 --- a/backend/windmill-api-jobs/src/query.rs +++ b/backend/windmill-api-jobs/src/query.rs @@ -8,6 +8,7 @@ //! Query builders for filtering job lists (queue and completed). +use serde_json; use sql_builder::prelude::*; use sql_builder::SqlBuilder; use windmill_common::utils::{escape_ilike_pattern, paginate_without_limits, Pagination}; @@ -50,11 +51,11 @@ pub fn filter_list_queue_query( .values .iter() .map(|v| { - let p = v.replace("*", "%").replace("'", "''"); + let p = v.replace("*", "%"); if w.negated { - format!("v2_job_queue.worker NOT LIKE '{p}'") + format!("v2_job_queue.worker NOT LIKE {}", quote(&p)) } else { - format!("v2_job_queue.worker LIKE '{p}'") + format!("v2_job_queue.worker LIKE {}", quote(&p)) } }) .collect(); @@ -77,11 +78,11 @@ pub fn filter_list_queue_query( .values .iter() .map(|v| { - let e = v.replace("'", "''"); + let p = format!("{}%", v); if ps.negated { - format!("runnable_path NOT LIKE '{e}%'") + format!("runnable_path NOT LIKE {}", quote(&p)) } else { - format!("runnable_path LIKE '{e}%'") + format!("runnable_path LIKE {}", quote(&p)) } }) .collect(); @@ -123,11 +124,11 @@ pub fn filter_list_queue_query( .values .iter() .map(|v| { - let p = v.replace("*", "%").replace("'", "''"); + let p = v.replace("*", "%"); if t.negated { - format!("v2_job.tag NOT LIKE '{p}'") + format!("v2_job.tag NOT LIKE {}", quote(&p)) } else { - format!("v2_job.tag LIKE '{p}'") + format!("v2_job.tag LIKE {}", quote(&p)) } }) .collect(); @@ -200,7 +201,11 @@ pub fn filter_list_queue_query( } if let Some(args) = &lq.args { - sqlb.and_where("args @> ?".bind(&args.replace("'", "''"))); + if let Ok(v) = serde_json::from_str::(args) { + sqlb.and_where("args @> ?".bind(&v.to_string())); + } else { + sqlb.and_where("FALSE"); + } } if lq.scheduled_for_before_now.is_some_and(|x| x) { @@ -287,14 +292,14 @@ pub fn filter_list_completed_query( .values .iter() .map(|v| { - let p = v.replace("*", "%").replace("'", "''"); + let p = v.replace("*", "%"); if label.negated { format!( - "NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE '{p}')" + "NOT EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE {})", quote(&p) ) } else { format!( - "EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE '{p}')" + "EXISTS (SELECT 1 FROM jsonb_array_elements_text(result->'wm_labels') lbl WHERE jsonb_typeof(result->'wm_labels') = 'array' AND lbl LIKE {})", quote(&p) ) } }) @@ -308,14 +313,14 @@ pub fn filter_list_completed_query( let clauses: Vec<_> = label .values .iter() - .map(|v| format!("NOT (result->'wm_labels' ? '{}')", v.replace("'", "''"))) + .map(|v| format!("NOT (result->'wm_labels' ? {})", quote(v))) .collect(); sqlb.and_where(format!("({})", clauses.join(" AND "))); } else { let clauses: Vec<_> = label .values .iter() - .map(|v| format!("result->'wm_labels' ? '{}'", v.replace("'", "''"))) + .map(|v| format!("result->'wm_labels' ? {}", quote(v))) .collect(); sqlb.and_where("result ? 'wm_labels'"); sqlb.and_where(format!("({})", clauses.join(" OR "))); @@ -329,11 +334,11 @@ pub fn filter_list_completed_query( .values .iter() .map(|v| { - let p = v.replace("*", "%").replace("'", "''"); + let p = v.replace("*", "%"); if worker.negated { - format!("v2_job_completed.worker NOT LIKE '{p}'") + format!("v2_job_completed.worker NOT LIKE {}", quote(&p)) } else { - format!("v2_job_completed.worker LIKE '{p}'") + format!("v2_job_completed.worker LIKE {}", quote(&p)) } }) .collect(); @@ -366,11 +371,11 @@ pub fn filter_list_completed_query( .values .iter() .map(|v| { - let e = v.replace("'", "''"); + let p = format!("{}%", v); if ps.negated { - format!("runnable_path NOT LIKE '{e}%'") + format!("runnable_path NOT LIKE {}", quote(&p)) } else { - format!("runnable_path LIKE '{e}%'") + format!("runnable_path LIKE {}", quote(&p)) } }) .collect(); @@ -400,11 +405,11 @@ pub fn filter_list_completed_query( .values .iter() .map(|v| { - let p = v.replace("*", "%").replace("'", "''"); + let p = v.replace("*", "%"); if t.negated { - format!("v2_job.tag NOT LIKE '{p}'") + format!("v2_job.tag NOT LIKE {}", quote(&p)) } else { - format!("v2_job.tag LIKE '{p}'") + format!("v2_job.tag LIKE {}", quote(&p)) } }) .collect(); @@ -449,11 +454,7 @@ pub fn filter_list_completed_query( } if let Some(dt) = &lq.created_or_started_after { let ts = dt.to_rfc3339(); - sqlb.and_where(format!( - "(created_at >= '{}' OR started_at >= '{}')", - ts.replace("'", "''"), - ts.replace("'", "''") - )); + sqlb.and_where("(created_at >= ? OR started_at >= ?)".bind(&ts).bind(&ts)); } if let Some(dt) = &lq.created_before { @@ -503,11 +504,19 @@ pub fn filter_list_completed_query( } if let Some(args) = &lq.args { - sqlb.and_where("args @> ?".bind(&args.replace("'", "''"))); + if let Ok(v) = serde_json::from_str::(args) { + sqlb.and_where("args @> ?".bind(&v.to_string())); + } else { + sqlb.and_where("FALSE"); + } } if let Some(result) = &lq.result { - sqlb.and_where("result @> ?".bind(&result.replace("'", "''"))); + if let Ok(v) = serde_json::from_str::(result) { + sqlb.and_where("result @> ?".bind(&v.to_string())); + } else { + sqlb.and_where("FALSE"); + } } if lq.is_not_schedule.unwrap_or(false) { diff --git a/backend/windmill-api-jobs/src/types.rs b/backend/windmill-api-jobs/src/types.rs index c59a54bd1c..f040e6eeb7 100644 --- a/backend/windmill-api-jobs/src/types.rs +++ b/backend/windmill-api-jobs/src/types.rs @@ -511,18 +511,14 @@ pub struct ResumeUrls { pub struct QueryOrBody(pub Option); -#[axum::async_trait] -impl FromRequest for QueryOrBody +impl FromRequest for QueryOrBody where D: DeserializeOwned, S: Send + Sync, { type Rejection = Response; - async fn from_request( - req: Request, - state: &S, - ) -> std::result::Result { + async fn from_request(req: Request, state: &S) -> std::result::Result { return if req.method() == axum::http::Method::GET { let Query(InPayload { payload }) = Query::from_request(req, state) .await diff --git a/backend/windmill-api-npm-proxy/src/lib.rs b/backend/windmill-api-npm-proxy/src/lib.rs index 903c2fb33c..25a3dd22d8 100644 --- a/backend/windmill-api-npm-proxy/src/lib.rs +++ b/backend/windmill-api-npm-proxy/src/lib.rs @@ -119,10 +119,10 @@ struct FileEntry { pub fn workspaced_service() -> Router { Router::new() // Use wildcards for package names to support scoped packages like @scope/package - .route("/metadata/*package", get(get_package_metadata)) - .route("/resolve/*package", get(resolve_package_version)) - .route("/filetree/*package_version", get(get_package_filetree)) - .route("/file/*package_version_filepath", get(get_package_file)) + .route("/metadata/{*package}", get(get_package_metadata)) + .route("/resolve/{*package}", get(resolve_package_version)) + .route("/filetree/{*package_version}", get(get_package_filetree)) + .route("/file/{*package_version_filepath}", get(get_package_file)) .layer( CorsLayer::new() .allow_origin(Any) diff --git a/backend/windmill-api-schedule/src/lib.rs b/backend/windmill-api-schedule/src/lib.rs index a6224a7639..6e35b0f897 100644 --- a/backend/windmill-api-schedule/src/lib.rs +++ b/backend/windmill-api-schedule/src/lib.rs @@ -56,12 +56,12 @@ pub fn workspaced_service() -> Router { Router::new() .route("/list", get(list_schedule)) .route("/list_with_jobs", get(list_schedule_with_jobs)) - .route("/get/*path", get(get_schedule)) - .route("/exists/*path", get(exists_schedule)) + .route("/get/{*path}", get(get_schedule)) + .route("/exists/{*path}", get(exists_schedule)) .route("/create", post(create_schedule)) - .route("/update/*path", post(edit_schedule)) - .route("/delete/*path", delete(delete_schedule)) - .route("/setenabled/*path", post(set_enabled)) + .route("/update/{*path}", post(edit_schedule)) + .route("/delete/{*path}", delete(delete_schedule)) + .route("/setenabled/{*path}", post(set_enabled)) .route("/setdefaulthandler", post(set_default_error_handler)) // .route("/catchup/*path", post(do_catchup).get(list_catchup)) } @@ -658,7 +658,11 @@ async fn list_schedule( sqlb.and_where_eq("is_flow", "?".bind(&is_flow)); } if let Some(args) = &lsq.args { - sqlb.and_where("args @> ?".bind(&args.replace("'", "''"))); + if let Ok(v) = serde_json::from_str::(args) { + sqlb.and_where("args @> ?".bind(&v.to_string())); + } else { + sqlb.and_where("FALSE"); + } } if let Some(path_start) = &lsq.path_start { sqlb.and_where_like_left("path", path_start); @@ -963,6 +967,15 @@ async fn delete_schedule( ))); } + // Capture row for trashbin before deleting + let trash_data: Option = sqlx::query_scalar( + "SELECT jsonb_build_object('row', to_jsonb(t)) FROM schedule t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + let del = sqlx::query_scalar!( "DELETE FROM schedule WHERE path = $1 AND workspace_id = $2 RETURNING 1", path, @@ -979,6 +992,18 @@ async fn delete_schedule( ))); } + if let Some(data) = trash_data { + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "schedule", + path, + data, + &authed.username, + ) + .await?; + } + audit_log( &mut *tx, &authed, diff --git a/backend/windmill-api-scripts/src/scripts.rs b/backend/windmill-api-scripts/src/scripts.rs index 56de4d9892..a909fc0ffc 100644 --- a/backend/windmill-api-scripts/src/scripts.rs +++ b/backend/windmill-api-scripts/src/scripts.rs @@ -190,18 +190,18 @@ impl ScriptWDraft { pub fn global_service() -> Router { Router::new() .route("/hub/top", get(get_top_hub_scripts)) - .route("/hub/get/*path", get(get_hub_script_by_path)) - .route("/hub/get_full/*path", get(get_full_hub_script_by_path)) - .route("/hub/pick/*path", get(pick_hub_script_by_path)) + .route("/hub/get/{*path}", get(get_hub_script_by_path)) + .route("/hub/get_full/{*path}", get(get_full_hub_script_by_path)) + .route("/hub/pick/{*path}", get(pick_hub_script_by_path)) } pub fn global_unauthed_service() -> Router { Router::new() .route( - "/tokened_raw/:workspace/:token/*path", + "/tokened_raw/{workspace}/{token}/{*path}", get(get_tokened_raw_script_by_path), ) - .route("/empty_ts/*path", get(get_empty_ts_script_by_path)) + .route("/empty_ts/{*path}", get(get_empty_ts_script_by_path)) } pub fn workspaced_service() -> Router { @@ -210,35 +210,36 @@ pub fn workspaced_service() -> Router { .route("/list_search", get(list_search_scripts)) .route("/create", post(create_script)) .route("/create_snapshot", post(create_snapshot_script)) - .route("/archive/p/*path", post(archive_script_by_path)) - .route("/get/draft/*path", get(get_script_by_path_w_draft)) - .route("/get/p/*path", get(get_script_by_path)) - .route("/list_tokens/*path", get(list_tokens)) - .route("/raw/p/*path", get(raw_script_by_path)) - .route("/raw_unpinned/p/*path", get(raw_script_by_path_unpinned)) - .route("/exists/p/*path", get(exists_script_by_path)) - .route("/archive/h/:hash", post(archive_script_by_hash)) - .route("/delete/h/:hash", post(delete_script_by_hash)) - .route("/delete/p/*path", post(delete_script_by_path)) + .route("/archive/p/{*path}", post(archive_script_by_path)) + .route("/get/draft/{*path}", get(get_script_by_path_w_draft)) + .route("/get/p/{*path}", get(get_script_by_path)) + .route("/list_tokens/{*path}", get(list_tokens)) + .route("/raw/p/{*path}", get(raw_script_by_path)) + .route("/raw_unpinned/p/{*path}", get(raw_script_by_path_unpinned)) + .route("/exists/p/{*path}", get(exists_script_by_path)) + .route("/archive/h/{hash}", post(archive_script_by_hash)) + .route("/delete/h/{hash}", post(delete_script_by_hash)) + .route("/delete/p/{*path}", post(delete_script_by_path)) .route("/delete_bulk", delete(delete_scripts_bulk)) - .route("/get/h/:hash", get(get_script_by_hash)) - .route("/raw/h/:hash", get(raw_script_by_hash)) - .route("/deployment_status/h/:hash", get(get_deployment_status)) + .route("/get/h/{hash}", get(get_script_by_hash)) + .route("/raw/h/{hash}", get(raw_script_by_hash)) + .route("/deployment_status/h/{hash}", get(get_deployment_status)) .route("/list_paths", get(list_paths)) .route( - "/toggle_workspace_error_handler/p/*path", + "/toggle_workspace_error_handler/p/{*path}", post(toggle_workspace_error_handler), ) - .route("/history/p/*path", get(get_script_history)) - .route("/get_latest_version/*path", get(get_latest_version)) + .route("/history/p/{*path}", get(get_script_history)) + .route("/get_latest_version/{*path}", get(get_latest_version)) .route( - "/list_paths_from_workspace_runnable/*path", + "/list_paths_from_workspace_runnable/{*path}", get(list_paths_from_workspace_runnable), ) .route( - "/history_update/h/:hash/p/*path", + "/history_update/h/{hash}/p/{*path}", post(update_script_history), ) + .route("/list_dedicated_with_deps", get(list_dedicated_with_deps)) // Temporary raw script storage for CLI lock generation .route("/raw_temp/store", post(store_raw_script_temp)) .route("/raw_temp/diff", post(diff_raw_scripts_with_deployed)) @@ -604,7 +605,7 @@ impl HandleDeploymentMetadata { } async fn create_script_internal<'c>( - ns: NewScript, + mut ns: NewScript, w_id: String, authed: ApiAuthed, db: sqlx::Pool, @@ -674,6 +675,17 @@ async fn create_script_internal<'c>( .to_owned(), )); }; + // When auto_parent is set, serialize concurrent creates for the same (workspace, path) + // so the clashing_script query always sees the latest committed head. + if ns.auto_parent.unwrap_or(false) { + sqlx::query_scalar!( + "SELECT pg_advisory_xact_lock(hashtext($1 || '/' || $2))", + &w_id, + &ns.path + ) + .fetch_one(&mut *tx) + .await?; + } let clashing_script = sqlx::query_as::<_, Script>( "SELECT * FROM script WHERE path = $1 AND archived = false AND workspace_id = $2", ) @@ -686,6 +698,15 @@ async fn create_script_internal<'c>( perms: serde_json::Value, p_path: String, } + // When auto_parent is set, resolve parent_hash to the current head for this path + // within the transaction. The advisory lock above ensures the second concurrent + // request waits until the first commits, so this query sees the updated head. + if ns.auto_parent.unwrap_or(false) { + if let Some(ref cs) = clashing_script { + ns.parent_hash = Some(cs.hash.clone()); + } + } + let parent_hashes_and_perms: Option = match (&ns.parent_hash, clashing_script) { (None, None) => Ok(None), (None, Some(s)) if !s.draft_only.unwrap_or(false) => Err(Error::BadRequest(format!( @@ -1426,7 +1447,7 @@ async fn get_script_history( check_scopes(&authed, || format!("scripts:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; let query_result = sqlx::query!( - "SELECT s.hash as hash, dm.deployment_msg as deployment_msg + "SELECT s.hash as hash, dm.deployment_msg as deployment_msg, s.created_at as created_at FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash WHERE s.workspace_id = $1 AND s.path = $2 ORDER by s.created_at DESC", @@ -1442,6 +1463,7 @@ async fn get_script_history( .map(|row| ScriptHistory { script_hash: ScriptHash(row.hash), deployment_msg: row.deployment_msg, + created_at: Some(row.created_at), }) .collect(); return Ok(Json(result)); @@ -1456,7 +1478,7 @@ async fn get_latest_version( check_scopes(&authed, || format!("scripts:read:{}", path))?; let mut tx = user_db.begin(&authed).await?; let row_o = sqlx::query!( - "SELECT s.hash as hash, dm.deployment_msg as deployment_msg + "SELECT s.hash as hash, dm.deployment_msg as deployment_msg, s.created_at as created_at FROM script s LEFT JOIN deployment_metadata dm ON s.hash = dm.script_hash WHERE s.workspace_id = $1 AND s.path = $2 ORDER by s.created_at DESC LIMIT 1", @@ -1470,7 +1492,8 @@ async fn get_latest_version( if let Some(row) = row_o { let result = ScriptHistory { script_hash: ScriptHash(row.hash), - deployment_msg: row.deployment_msg, // + deployment_msg: row.deployment_msg, + created_at: Some(row.created_at), }; return Ok(Json(Some(result))); } else { @@ -2239,33 +2262,58 @@ async fn delete_script_by_path( .await? .unwrap_or(false); - let script = if !draft_only { + if !draft_only { require_admin(authed.is_admin, &authed.username)?; - sqlx::query_scalar!( - "DELETE FROM script WHERE path = $1 AND workspace_id = $2 RETURNING path", + } + + // Capture all script versions and drafts for trashbin before deleting + let trash_scripts: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM script t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + + let trash_drafts: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM draft t WHERE path = $1 AND workspace_id = $2 AND typ = 'script'", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + + let script = sqlx::query_scalar!( + "DELETE FROM script WHERE path = $1 AND workspace_id = $2 RETURNING path", + path, + w_id + ) + .fetch_one(&mut *tx) + .await + .map_err(|e| Error::internal_err(format!("deleting script by path {w_id}: {e:#}")))?; + + if !trash_scripts.is_empty() { + let mut trash_data = serde_json::json!({"scripts": trash_scripts}); + if !trash_drafts.is_empty() { + trash_data["drafts"] = serde_json::Value::Array(trash_drafts); + } + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "script", path, - w_id + trash_data, + &authed.username, ) - .fetch_one(&db) - .await - .map_err(|e| Error::internal_err(format!("deleting script by path {w_id}: {e:#}")))? - } else { - sqlx::query_scalar!( - "DELETE FROM script WHERE path = $1 AND workspace_id = $2 RETURNING path", - path, - w_id - ) - .fetch_one(&mut *tx) - .await - .map_err(|e| Error::internal_err(format!("deleting script by path {w_id}: {e:#}")))? - }; + .await?; + } sqlx::query!( "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'script'", path, w_id ) - .execute(&db) + .execute(&mut *tx) .await?; if !query.keep_captures.unwrap_or(false) { @@ -2274,7 +2322,7 @@ async fn delete_script_by_path( path, w_id ) - .execute(&db) + .execute(&mut *tx) .await?; sqlx::query!( @@ -2282,7 +2330,7 @@ async fn delete_script_by_path( path, w_id ) - .execute(&db) + .execute(&mut *tx) .await?; } @@ -2369,6 +2417,30 @@ async fn delete_scripts_bulk( let mut tx = db.begin().await?; + // Capture scripts for trashbin per path before bulk delete + for path in &request.paths { + let trash_scripts: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM script t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + + if !trash_scripts.is_empty() { + let trash_data = serde_json::json!({"scripts": trash_scripts}); + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "script", + path, + trash_data, + &authed.username, + ) + .await?; + } + } + let mut deleted_paths = sqlx::query_scalar!( "DELETE FROM script WHERE workspace_id = $1 AND path = ANY($2) RETURNING path", w_id, @@ -2480,6 +2552,62 @@ async fn guard_script_from_debounce_data(ns: &NewScript) -> Result<()> { } } +#[derive(Serialize)] +struct DedicatedScriptDeps { + path: String, + language: ScriptLang, + workspace_dep_names: Vec, +} + +async fn list_dedicated_with_deps( + authed: ApiAuthed, + Extension(user_db): Extension, + Path(w_id): Path, +) -> JsonResult> { + let mut tx = user_db.begin(&authed).await?; + + let rows = sqlx::query!( + "SELECT DISTINCT ON (path) path, language AS \"language: ScriptLang\", content FROM script + WHERE workspace_id = $1 + AND archived = false + AND dedicated_worker = true + AND language = ANY($2::SCRIPT_LANG[]) + ORDER BY path, created_at DESC", + &w_id, + &[ + ScriptLang::Python3, + ScriptLang::Bun, + ScriptLang::Bunnative, + ScriptLang::Deno, + ] as &[ScriptLang], + ) + .fetch_all(&mut *tx) + .await?; + + tx.commit().await?; + + let result = rows + .into_iter() + .map(|row| { + let dep_names = + windmill_common::scripts::extract_workspace_dependencies_annotated_refs( + &row.language, + &row.content, + &row.path, + ) + .map(|refs| refs.external) + .unwrap_or_default(); + DedicatedScriptDeps { + path: row.path, + language: row.language, + workspace_dep_names: dep_names, + } + }) + .collect(); + + Ok(Json(result)) +} + // ============================================================================ // Temporary Raw Script Storage for CLI Lock Generation // ============================================================================ @@ -2508,11 +2636,9 @@ async fn store_raw_script_temp( .await?; // Clean up old entries (1 week TTL) - sqlx::query!( - "DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'" - ) - .execute(&db) - .await?; + sqlx::query!("DELETE FROM raw_script_temp WHERE created_at < NOW() - INTERVAL '1 week'") + .execute(&db) + .await?; Ok(Json(hash)) } @@ -2560,7 +2686,7 @@ async fn diff_raw_scripts_with_deployed( FROM script s \ WHERE s.path = local.path AND s.workspace_id = $3 AND s.archived = false \ ORDER BY s.created_at DESC LIMIT 1 \ - ) deployed ON deployed.deployed_hash = local.hash" + ) deployed ON deployed.deployed_hash = local.hash", ) .bind(&paths) .bind(&hashes) @@ -2582,7 +2708,7 @@ async fn diff_raw_scripts_with_deployed( AND wd.language = $3::SCRIPT_LANG \ AND wd.name IS NOT DISTINCT FROM $4 \ AND encode(sha256(convert_to(wd.content, 'UTF8')), 'hex') = $5 \ - )" + )", ) .bind(&dep.path) .bind(&w_id) diff --git a/backend/windmill-api-settings/src/lib.rs b/backend/windmill-api-settings/src/lib.rs index b93ae0a38e..5594c24d2a 100644 --- a/backend/windmill-api-settings/src/lib.rs +++ b/backend/windmill-api-settings/src/lib.rs @@ -38,14 +38,15 @@ use windmill_common::ee_oss::{send_critical_alert, CriticalAlertKind, CriticalEr #[cfg(all(feature = "private", feature = "enterprise"))] use windmill_common::secret_backend::{SecretMigrationReport, VaultSettings}; use windmill_common::{ + ai_cache::bump_instance_ai_config_revision, email_oss::send_email_plain_text, error::{self, JsonResult, Result}, get_database_url, global_settings::{ - APP_WORKSPACED_ROUTE_SETTING, AUTOMATE_USERNAME_CREATION_SETTING, + AI_CONFIG_SETTING, APP_WORKSPACED_ROUTE_SETTING, AUTOMATE_USERNAME_CREATION_SETTING, CRITICAL_ALERT_MUTE_UI_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING, DISABLE_HUB_SETTING, - EMAIL_DOMAIN_SETTING, ENV_SETTINGS, HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING, - WS_BASE_URL_SETTING, + EMAIL_DOMAIN_SETTING, ENV_SETTINGS, HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, + HUB_ACCESSIBLE_URL_SETTING, HUB_BASE_URL_SETTING, WS_BASE_URL_SETTING, }, instance_config::{self, ApplyMode, InstanceConfig}, server::Smtp, @@ -57,7 +58,7 @@ pub fn global_service() -> Router { let r = Router::new() .route("/envs", get(get_local_settings)) .route( - "/global/:key", + "/global/{key}", post(set_global_setting).get(get_global_setting), ) .route("/list_global", get(list_global_settings)) @@ -79,7 +80,7 @@ pub fn global_service() -> Router { .route("/test_critical_channels", post(test_critical_channels)) .route("/critical_alerts", get(get_critical_alerts)) .route( - "/critical_alerts/:id/acknowledge", + "/critical_alerts/{id}/acknowledge", post(acknowledge_critical_alert), ) .route( @@ -91,7 +92,7 @@ pub fn global_service() -> Router { post(refresh_custom_instance_user_pwd), ) .route( - "/setup_custom_instance_pg_database/:name", + "/setup_custom_instance_pg_database/{name}", post(setup_custom_instance_pg_database), ) .route( @@ -288,6 +289,7 @@ pub async fn set_global_setting_internal( key: String, value: serde_json::Value, ) -> error::Result<()> { + let should_bump_instance_ai_revision = key == AI_CONFIG_SETTING; let value = if key == "retention_period_secs" { instance_config::clamp_retention_period(value) } else { @@ -329,6 +331,10 @@ pub async fn set_global_setting_internal( } }; + if should_bump_instance_ai_revision { + bump_instance_ai_config_revision(); + } + Ok(()) } @@ -422,6 +428,74 @@ async fn run_setting_pre_write_hook( } } } + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING => { + let serde_json::Value::Bool(workspaced_route) = value else { + return Err(error::Error::BadRequest(format!( + "{} setting expected to be boolean", + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING + ))); + }; + + if !*workspaced_route { + #[derive(Debug, Deserialize, Serialize)] + #[allow(unused)] + struct DuplicateRoute { + route_path: String, + workspace_id: String, + http_method: String, + } + let duplicate_routes = sqlx::query_as!( + DuplicateRoute, + r#" + SELECT + route_path, + workspace_id, + http_method::TEXT AS "http_method!" + FROM + http_trigger + WHERE + workspaced_route IS FALSE + AND route_path_key IN ( + SELECT + route_path_key + FROM + http_trigger + WHERE + workspaced_route IS FALSE + GROUP BY + route_path_key, http_method + HAVING COUNT(*) > 1 + ) + ORDER BY route_path_key + "# + ) + .fetch_all(db) + .await?; + + if !duplicate_routes.is_empty() { + tracing::error!( + "Cannot disable {} setting as duplicate http routes were found: {:?}", + HTTP_ROUTE_WORKSPACED_ROUTE_SETTING, + &duplicate_routes + ); + + #[derive(Serialize)] + struct ErrorResponse { + error: String, + details: Vec, + } + + let error_response = ErrorResponse { + error: "Duplicate HTTP route paths detected".to_string(), + details: duplicate_routes, + }; + + return Err(error::Error::JsonErr( + serde_json::to_value(error_response).unwrap(), + )); + } + } + } _ => {} } Ok(()) @@ -475,6 +549,10 @@ async fn set_instance_config( let current_map = current.global_settings.to_settings_map(); let settings_diff = instance_config::diff_global_settings(¤t_map, &desired_map, ApplyMode::Merge); + let ai_config_changed = settings_diff + .upserts + .iter() + .any(|(key, _)| key == AI_CONFIG_SETTING); for (key, value) in &settings_diff.upserts { run_setting_pre_write_hook(&db, key, value).await?; @@ -483,6 +561,10 @@ async fn set_instance_config( instance_config::apply_settings_diff(&db, &settings_diff) .await .map_err(|e| error::Error::internal_err(e.to_string()))?; + + if ai_config_changed { + bump_instance_ai_config_revision(); + } } if !desired.worker_configs.is_empty() { @@ -531,6 +613,7 @@ pub async fn get_global_setting( && key != DISABLE_HUB_SETTING && key != EMAIL_DOMAIN_SETTING && key != APP_WORKSPACED_ROUTE_SETTING + && key != HTTP_ROUTE_WORKSPACED_ROUTE_SETTING && key != WS_BASE_URL_SETTING { require_super_admin(&db, &authed.email).await?; @@ -1126,6 +1209,59 @@ struct CachedResourceType { description: Option, } +#[derive(serde::Deserialize)] +struct HubResourceTypeRaw { + id: i64, + name: String, + schema: Option, + app: String, + description: Option, +} + +async fn fetch_resource_types_from_hub() -> error::Result> { + let response = HTTP_CLIENT + .get(format!( + "{}/resource_types/list", + windmill_common::DEFAULT_HUB_BASE_URL + )) + .header("Accept", "application/json") + .send() + .await + .map_err(|e| error::Error::InternalErr(format!("Failed to fetch from hub: {}", e)))?; + + if !response.status().is_success() { + return Err(error::Error::InternalErr(format!( + "Hub returned status {}", + response.status() + ))); + } + + let raw_types: Vec = response + .json() + .await + .map_err(|e| error::Error::InternalErr(format!("Failed to parse hub response: {}", e)))?; + + Ok(raw_types + .into_iter() + .filter_map(|rt| { + let schema = match rt.schema { + Some(s) => match serde_json::from_str(&s) { + Ok(v) => Some(v), + Err(_) => return None, + }, + None => None, + }; + Some(CachedResourceType { + id: rt.id, + name: rt.name, + schema, + app: rt.app, + description: rt.description, + }) + }) + .collect()) +} + async fn sync_cached_resource_types( Extension(db): Extension, authed: ApiAuthed, @@ -1135,16 +1271,12 @@ async fn sync_cached_resource_types( use windmill_common::worker::HUB_RT_CACHE_DIR; let cache_path = format!("{}/resource_types.json", *HUB_RT_CACHE_DIR); - let content = tokio::fs::read_to_string(&cache_path).await.map_err(|e| { - error::Error::NotFound(format!( - "No cached resource types found at {}: {}", - cache_path, e - )) - })?; - - let cached_types: Vec = serde_json::from_str(&content).map_err(|e| { - error::Error::InternalErr(format!("Failed to parse cached resource types: {}", e)) - })?; + let cached_types = match tokio::fs::read_to_string(&cache_path).await { + Ok(content) => serde_json::from_str::>(&content).map_err(|e| { + error::Error::InternalErr(format!("Failed to parse cached resource types: {}", e)) + })?, + Err(_) => fetch_resource_types_from_hub().await?, + }; let mut synced_count = 0; diff --git a/backend/windmill-api-users/Cargo.toml b/backend/windmill-api-users/Cargo.toml index e720b37eb7..13ab8143d8 100644 --- a/backend/windmill-api-users/Cargo.toml +++ b/backend/windmill-api-users/Cargo.toml @@ -34,3 +34,4 @@ time.workspace = true tokio.workspace = true tower-cookies.workspace = true tracing.workspace = true +url.workspace = true diff --git a/backend/windmill-api-users/src/lib.rs b/backend/windmill-api-users/src/lib.rs index 913bd46b82..ee5369e616 100644 --- a/backend/windmill-api-users/src/lib.rs +++ b/backend/windmill-api-users/src/lib.rs @@ -1 +1,4 @@ pub mod users; +#[cfg(feature = "private")] +pub mod users_ee; +mod users_oss; diff --git a/backend/windmill-api-users/src/users.rs b/backend/windmill-api-users/src/users.rs index 8f7bec1398..6024c756fb 100644 --- a/backend/windmill-api-users/src/users.rs +++ b/backend/windmill-api-users/src/users.rs @@ -49,16 +49,16 @@ use windmill_common::users::truncate_token; use windmill_common::users::COOKIE_NAME; use windmill_common::utils::paginate; use windmill_common::worker::CLOUD_HOSTED; -use windmill_common::BASE_URL; use windmill_common::{ auth::{get_folders_for_user, get_groups_for_user}, db::UserDB, error::{self, Error, JsonResult, Result}, utils::{not_found_if_none, rd_string, require_admin, Pagination, StripPath}, }; +use windmill_common::{BASE_URL, HUB_BASE_URL}; use windmill_git_sync::handle_deployment_metadata; -const COOKIE_PATH: &str = "/"; +pub const COOKIE_PATH: &str = "/"; pub fn workspaced_service() -> Router { Router::new() @@ -66,32 +66,37 @@ pub fn workspaced_service() -> Router { .route("/list_usage", get(list_user_usage)) .route("/list_usernames", get(list_usernames)) .route("/exists", post(exists_username)) - .route("/get/:user", get(get_workspace_user)) - .route("/update/:user", post(update_workspace_user)) - .route("/delete/:user", delete(delete_workspace_user)) - .route("/convert_to_group/:user", post(convert_user_to_group)) - .route("/is_owner/*path", get(is_owner_of_path)) - .route("/whois/:username", get(whois)) + .route("/get/{user}", get(get_workspace_user)) + .route("/update/{user}", post(update_workspace_user)) + .route("/delete/{user}", delete(delete_workspace_user)) + .route("/convert_to_group/{user}", post(convert_user_to_group)) + .route("/is_owner/{*path}", get(is_owner_of_path)) + .route("/whois/{username}", get(whois)) .route("/whoami", get(whoami)) .route("/leave", post(leave_workspace)) - .route("/username_to_email/:username", get(username_to_email)) + .route("/username_to_email/{username}", get(username_to_email)) + .route( + "/impersonate_service_account", + post(impersonate_service_account), + ) + .route("/exit_impersonation", post(exit_impersonation)) } pub fn global_service() -> Router { Router::new() - .route("/exists/:email", get(exists_email)) + .route("/exists/{email}", get(exists_email)) .route("/email", get(get_email)) .route("/whoami", get(global_whoami)) .route("/list_invites", get(list_invites)) .route("/decline_invite", post(decline_invite)) .route("/accept_invite", post(accept_invite)) .route("/list_as_super_admin", get(list_users_as_super_admin)) - .route("/set_login_type/:user", post(set_login_type)) - .route("/update/:user", post(update_user)) - .route("/delete/:user", delete(delete_user)) - .route("/username_info/:user", get(get_instance_username_info)) + .route("/set_login_type/{user}", post(set_login_type)) + .route("/update/{user}", post(update_user)) + .route("/delete/{user}", delete(delete_user)) + .route("/username_info/{user}", get(get_instance_username_info)) .route("/tokens/create", post(create_token)) - .route("/tokens/delete/:token_prefix", delete(delete_token)) + .route("/tokens/delete/{token_prefix}", delete(delete_token)) .route("/tokens/list", get(list_tokens)) .route("/tokens/impersonate", post(impersonate)) .route("/usage", get(get_usage)) @@ -135,6 +140,7 @@ pub struct User { pub role: Option, #[serde(skip_serializing_if = "Option::is_none")] pub added_via: Option, + pub is_service_account: bool, } #[derive(Serialize)] @@ -157,6 +163,7 @@ pub struct GlobalUserInfo { operator_only: Option, first_time_user: bool, role_source: String, + disabled: bool, } #[derive(Serialize, Debug)] @@ -175,6 +182,7 @@ pub struct UserInfo { pub folders: Vec, pub folders_owners: Vec, pub name: Option, + pub is_service_account: bool, } #[derive(FromRow, Serialize)] @@ -213,6 +221,7 @@ pub struct EditUser { pub is_super_admin: Option, pub is_devops: Option, pub name: Option, + pub disabled: Option, } #[derive(Deserialize)] @@ -396,7 +405,7 @@ async fn list_users_as_super_admin( GlobalUserInfo, "WITH active_users AS (SELECT distinct username as email FROM (SELECT username, timestamp, operation FROM audit_partitioned UNION ALL SELECT username, timestamp, operation FROM audit) AS a WHERE timestamp > NOW() - INTERVAL '1 month' AND (operation = 'users.login' OR operation = 'oauth.login' OR operation = 'users.token.refresh')), authors as (SELECT distinct email FROM usr WHERE usr.operator IS false) - SELECT email, email NOT IN (SELECT email FROM authors) as operator_only, login_type::text, verified, super_admin, devops, name, company, username, first_time_user, role_source + SELECT email, email NOT IN (SELECT email FROM authors) as operator_only, login_type::text, verified, super_admin, devops, name, company, username, first_time_user, role_source, disabled FROM password WHERE email IN (SELECT email FROM active_users) ORDER BY super_admin DESC, devops DESC @@ -409,7 +418,7 @@ async fn list_users_as_super_admin( } else { sqlx::query_as!( GlobalUserInfo, - "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT \ + "SELECT email, login_type::text, verified, super_admin, devops, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password ORDER BY super_admin DESC, devops DESC, email LIMIT \ $1 OFFSET $2", per_page as i32, offset as i32 @@ -577,17 +586,50 @@ async fn logout( } tx.commit().await?; if let Some(rd) = rd { - Ok((StatusCode::TEMPORARY_REDIRECT, [(LOCATION, rd)]).into_response()) + if is_valid_logout_redirect(&rd).await { + Ok((StatusCode::TEMPORARY_REDIRECT, [(LOCATION, rd)]).into_response()) + } else { + tracing::warn!("Blocked logout redirect to non-whitelisted URL: {}", rd); + Ok((StatusCode::OK, "logged out successfully".to_string()).into_response()) + } } else { Ok((StatusCode::OK, "logged out successfully".to_string()).into_response()) } } +async fn is_valid_logout_redirect(rd: &str) -> bool { + // Allow relative paths (same-origin redirects) + if rd.starts_with('/') && !rd.starts_with("//") { + return true; + } + let parsed = match url::Url::parse(rd) { + Ok(u) => u, + Err(_) => return false, + }; + let host: &str = match parsed.host_str() { + Some(h) => h, + None => return false, + }; + if host == "windmill.dev" || host.ends_with(".windmill.dev") { + return true; + } + let hub_url = HUB_BASE_URL.read().await.clone(); + if let Ok(hub_parsed) = url::Url::parse(&hub_url) { + if let Some(hub_host) = hub_parsed.host_str() { + if host == hub_host { + return true; + } + } + } + false +} + async fn whoami( Extension(db): Extension, Path(w_id): Path, - ApiAuthed { username, email, is_admin, groups, folders, .. }: ApiAuthed, + authed: ApiAuthed, ) -> JsonResult { + let ApiAuthed { username, email, is_admin, groups, folders, .. } = authed; let user = get_user(&w_id, &username, &db).await?; if let Some(user) = user { Ok(Json(user)) @@ -614,6 +656,7 @@ async fn whoami( .into_iter() .filter_map(|x| if x.2 { Some(x.0) } else { None }) .collect(), + is_service_account: false, })) } } @@ -625,15 +668,15 @@ async fn global_whoami( ) -> JsonResult { let user = sqlx::query_as!( GlobalUserInfo, - "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source FROM password WHERE \ + "SELECT email, login_type::TEXT, super_admin, devops, verified, name, company, username, NULL::bool as operator_only, first_time_user, role_source, disabled FROM password WHERE \ email = $1", email ) - .fetch_one(&db) + .fetch_optional(&db) .await - .map_err(|e| Error::internal_err(format!("fetching global identity: {e:#}"))); + .map_err(|e| Error::internal_err(format!("fetching global identity: {e:#}")))?; - if let Ok(user) = user { + if let Some(user) = user { Ok(Json(user)) } else if std::env::var("SUPERADMIN_SECRET").ok() == Some(token) { Ok(Json(GlobalUserInfo { @@ -648,9 +691,24 @@ async fn global_whoami( operator_only: None, first_time_user: false, role_source: "manual".to_string(), + disabled: false, })) } else { - Err(user.unwrap_err()) + // Service accounts don't have a password row + Ok(Json(GlobalUserInfo { + email: email.clone(), + login_type: Some("service_account".to_string()), + super_admin: false, + devops: false, + verified: true, + name: None, + company: None, + username: None, + operator_only: Some(true), + first_time_user: false, + role_source: "service_account".to_string(), + disabled: false, + })) } } @@ -701,12 +759,13 @@ pub struct User2 { pub name: Option, #[serde(skip_serializing_if = "Option::is_none")] pub added_via: Option, + pub is_service_account: bool, } async fn get_user(w_id: &str, username: &str, db: &DB) -> Result> { let user = sqlx::query_as!( User2, - "SELECT usr.*, password.super_admin, password.name FROM usr LEFT JOIN password ON usr.email = password.email Where usr.username = $1 AND workspace_id = $2 + "SELECT usr.*, COALESCE(password.super_admin, false) as \"super_admin!\", password.name FROM usr LEFT JOIN password ON usr.email = password.email Where usr.username = $1 AND workspace_id = $2 ", username, w_id @@ -747,6 +806,7 @@ async fn get_user(w_id: &str, username: &str, db: &DB) -> Result, Extension(argon2): Extension>>, @@ -1677,8 +1757,10 @@ async fn login( return Ok("no_auth".to_string()); } - let mut tx = db.begin().await?; let email = email.to_lowercase(); + windmill_common::login_rate_limit::check_and_increment_login_attempt(&headers, &email)?; + + let mut tx = db.begin().await?; let audit_author = AuditAuthor { email: email.clone(), username: email.clone(), @@ -1687,7 +1769,7 @@ async fn login( }; let email_w_h: Option<(String, String, bool)> = sqlx::query_as( "SELECT email, password_hash, super_admin FROM password WHERE email = $1 AND login_type = \ - 'password'", + 'password' AND disabled = false", ) .bind(&email) .fetch_optional(&mut *tx) @@ -1710,6 +1792,7 @@ async fn login( None, ) .await?; + windmill_common::login_rate_limit::record_login_failure(&email); Err(Error::BadRequest("Invalid login".to_string())) } else { let token = create_session_token(&email, super_admin, &mut tx, cookies).await?; @@ -1746,6 +1829,7 @@ async fn login( None, ) .await?; + windmill_common::login_rate_limit::record_login_failure(&email); Err(Error::BadRequest("Invalid login".to_string())) } } @@ -1776,7 +1860,7 @@ async fn refresh_token( } let super_admin = sqlx::query_scalar!( - "SELECT super_admin FROM password WHERE email = $1", + "SELECT super_admin FROM password WHERE email = $1 AND disabled = false", &authed.email ) .fetch_optional(&mut *tx) @@ -1971,6 +2055,44 @@ async fn impersonate( Ok((StatusCode::CREATED, token)) } +#[derive(Deserialize)] +pub struct ImpersonateServiceAccountRequest { + pub username: String, +} + +async fn impersonate_service_account( + Extension(db): Extension, + authed: ApiAuthed, + cookies: Cookies, + Tokened { token: current_token }: Tokened, + Path(w_id): Path, + Json(req): Json, +) -> Result<(StatusCode, String)> { + crate::users_oss::impersonate_service_account(db, authed, cookies, current_token, w_id, req) + .await +} + +#[derive(Deserialize)] +struct ExitImpersonationRequest { + token: String, +} + +async fn exit_impersonation( + cookies: Cookies, + Json(req): Json, +) -> Result { + let mut cookie = tower_cookies::Cookie::new(COOKIE_NAME, req.token); + cookie.set_secure(IS_SECURE.read().await.clone()); + cookie.set_same_site(Some(tower_cookies::cookie::SameSite::Lax)); + cookie.set_http_only(true); + cookie.set_path(COOKIE_PATH); + if COOKIE_DOMAIN.is_some() { + cookie.set_domain(COOKIE_DOMAIN.clone().unwrap()); + } + cookies.add(cookie); + Ok("exited impersonation".to_string()) +} + #[derive(Deserialize)] struct ListTokenQuery { exclude_ephemeral: Option, diff --git a/backend/windmill-api-users/src/users_oss.rs b/backend/windmill-api-users/src/users_oss.rs new file mode 100644 index 0000000000..a42cce8405 --- /dev/null +++ b/backend/windmill-api-users/src/users_oss.rs @@ -0,0 +1,28 @@ +#[cfg(feature = "private")] +#[allow(unused)] +pub use crate::users_ee::*; + +#[cfg(not(feature = "private"))] +use crate::users::ImpersonateServiceAccountRequest; +#[cfg(not(feature = "private"))] +use http::StatusCode; +#[cfg(not(feature = "private"))] +use tower_cookies::Cookies; +#[cfg(not(feature = "private"))] +use windmill_api_auth::ApiAuthed; +#[cfg(not(feature = "private"))] +use windmill_common::DB; + +#[cfg(not(feature = "private"))] +pub async fn impersonate_service_account( + _db: DB, + _authed: ApiAuthed, + _cookies: Cookies, + _current_token: String, + _w_id: String, + _req: ImpersonateServiceAccountRequest, +) -> windmill_common::error::Result<(StatusCode, String)> { + Err(windmill_common::error::Error::BadRequest( + "Service accounts require Windmill Enterprise Edition".to_string(), + )) +} diff --git a/backend/windmill-api-workspaces/Cargo.toml b/backend/windmill-api-workspaces/Cargo.toml index a03bb3a490..86f0649c73 100644 --- a/backend/windmill-api-workspaces/Cargo.toml +++ b/backend/windmill-api-workspaces/Cargo.toml @@ -14,9 +14,11 @@ enterprise = ["windmill-common/enterprise"] private = ["windmill-common/private"] cloud = ["windmill-common/cloud"] no_auth = ["windmill-api-auth/no_auth"] +parquet = ["windmill-object-store/parquet"] [dependencies] windmill-common = { workspace = true, default-features = false } +windmill-object-store = { workspace = true, optional = true } windmill-types.workspace = true windmill-api-auth.workspace = true windmill-api-users.workspace = true diff --git a/backend/windmill-api-workspaces/src/workspaces.rs b/backend/windmill-api-workspaces/src/workspaces.rs index 17086cf3aa..f533008180 100644 --- a/backend/windmill-api-workspaces/src/workspaces.rs +++ b/backend/windmill-api-workspaces/src/workspaces.rs @@ -35,7 +35,6 @@ use windmill_common::variables::{ build_crypt, decrypt, encrypt, SECRET_SALT, WORKSPACE_CRYPT_CACHE, }; use windmill_common::worker::{to_raw_value, CLOUD_HOSTED}; -#[cfg(feature = "enterprise")] use windmill_common::workspaces::GitRepositorySettings; #[cfg(feature = "enterprise")] use windmill_common::workspaces::WorkspaceDeploymentUISettings; @@ -76,12 +75,18 @@ pub fn workspaced_service() -> Router { .route("/archive", post(archive_workspace)) .route("/invite_user", post(invite_user)) .route("/add_user", post(add_user)) + .route("/create_service_account", post(create_service_account)) .route("/delete_invite", post(delete_invite)) .route("/rebuild_dependency_map", post(rebuild_dependency_map)) .route("/get_dependency_map", get(get_dependency_map)) - .route("/get_dependents/*imported_path", get(get_dependents)) + .route("/get_dependents/{*imported_path}", get(get_dependents)) + .route("/get_imports/{*importer_path}", get(get_imports)) .route("/get_dependents_amounts", post(get_dependents_amounts)) .route("/get_settings", get(get_settings)) + .route( + "/get_copilot_settings_state", + get(get_copilot_settings_state), + ) .route("/get_deploy_to", get(get_deploy_to)) .route("/edit_slack_command", post(edit_slack_command)) .route( @@ -111,6 +116,7 @@ pub fn workspaced_service() -> Router { .route("/list_datatables", get(list_datatables)) .route("/list_datatable_schemas", get(list_datatable_schemas)) .route("/edit_datatable_config", post(edit_datatable_config)) + .route("/git_sync_enabled", get(get_git_sync_enabled)) .route("/edit_git_sync_config", post(edit_git_sync_config)) .route("/edit_git_sync_repository", post(edit_git_sync_repository)) .route( @@ -147,10 +153,10 @@ pub fn workspaced_service() -> Router { post(create_workspace_fork_branch), ) .route( - "/reset_diff_tally/:fork_workspace_id", + "/reset_diff_tally/{fork_workspace_id}", post(reset_workspace_diffs), ) - .route("/compare/:target_workspace_id", get(compare_workspaces)) + .route("/compare/{target_workspace_id}", get(compare_workspaces)) .route("/import_pg_database", post(import_pg_database)) .route("/export_pg_schema", post(export_pg_schema)) .route( @@ -160,7 +166,7 @@ pub fn workspaced_service() -> Router { .route("/protection_rules", get(list_protection_rules)) .route("/protection_rules", post(create_protection_rule)) .route( - "/protection_rules/:rule_name", + "/protection_rules/{rule_name}", post(update_protection_rule).delete(delete_protection_rule), ) .route("/log_chat", post(log_ai_chat)) @@ -177,9 +183,9 @@ pub fn global_service() -> Router { .route("/exists", post(exists_workspace)) .route("/exists_username", post(exists_username)) .route("/allowed_domain_auto_invite", get(is_allowed_auto_domain)) - .route("/unarchive/:workspace", post(unarchive_workspace)) + .route("/unarchive/{workspace}", post(unarchive_workspace)) .route( - "/delete/:workspace", + "/delete/{workspace}", delete(crate::workspaces_extra::delete_workspace), ) .route( @@ -263,6 +269,35 @@ pub struct WorkspaceSettings { pub public_app_execution_limit_per_minute: Option, } +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +pub struct CopilotSettingsState { + pub has_instance_ai_config: bool, + pub uses_instance_ai_config: bool, + #[serde(skip_serializing_if = "Option::is_none")] + pub instance_ai_summary: Option, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +pub struct InstanceAIProviderSummary { + pub provider: String, + pub models: Vec, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +pub struct InstanceAIModelSummary { + pub provider: String, + pub model: String, +} + +#[derive(Serialize, Deserialize, Debug, Clone, PartialEq)] +pub struct InstanceAISummary { + pub providers: Vec, + #[serde(skip_serializing_if = "Option::is_none")] + pub default_model: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub code_completion_model: Option, +} + /// #[derive(sqlx::Type, Serialize, Deserialize, Debug)] // #[sqlx(type_name = "WORKSPACE_KEY_KIND", rename_all = "lowercase")] // pub enum WorkspaceKeyKind { @@ -614,15 +649,104 @@ async fn get_settings( .await .map_err(|e| Error::internal_err(format!("getting settings: {e:#}")))?; + let mut settings = not_found_if_none(settings, "workspace settings", &w_id)?; tx.commit().await?; - let mut settings = not_found_if_none(settings, "workspace settings", &w_id)?; if !authed.is_admin { settings.slack_oauth_client_secret = None; } Ok(Json(settings)) } +async fn get_copilot_settings_state( + _authed: ApiAuthed, + Path(w_id): Path, + Extension(db): Extension, +) -> JsonResult { + let workspace_ai_config = sqlx::query_scalar!( + "SELECT ai_config FROM workspace_settings WHERE workspace_id = $1", + &w_id + ) + .fetch_optional(&db) + .await + .map_err(|e| Error::internal_err(format!("getting workspace ai settings: {e:#}")))?; + let workspace_ai_config = not_found_if_none(workspace_ai_config, "workspace settings", &w_id)?; + let instance_ai_config: Option = + sqlx::query_scalar("SELECT value FROM global_settings WHERE name = 'ai_config'") + .fetch_optional(&db) + .await + .map_err(|e| Error::internal_err(format!("getting instance ai settings: {e:#}")))?; + + Ok(Json(build_copilot_settings_state( + has_ai_providers(workspace_ai_config.as_ref()), + instance_ai_config.as_ref(), + ))) +} + +pub fn has_ai_providers(config: Option<&serde_json::Value>) -> bool { + config + .and_then(|value| value.get("providers")) + .and_then(|providers| providers.as_object()) + .map(|providers| !providers.is_empty()) + .unwrap_or(false) +} + +pub fn build_copilot_settings_state( + has_workspace_ai_config: bool, + instance_ai_config: Option<&serde_json::Value>, +) -> CopilotSettingsState { + let has_instance_ai_config = has_ai_providers(instance_ai_config); + CopilotSettingsState { + has_instance_ai_config, + uses_instance_ai_config: !has_workspace_ai_config && has_instance_ai_config, + instance_ai_summary: build_instance_ai_summary(instance_ai_config), + } +} + +pub fn build_instance_ai_summary(config: Option<&serde_json::Value>) -> Option { + let config = config?; + if !has_ai_providers(Some(config)) { + return None; + } + let providers = config.get("providers")?.as_object()?; + + let mut provider_summaries = providers + .iter() + .map(|(provider, provider_config)| InstanceAIProviderSummary { + provider: provider.clone(), + models: provider_config + .get("models") + .and_then(|models| models.as_array()) + .map(|models| { + models + .iter() + .filter_map(|model| model.as_str().map(ToOwned::to_owned)) + .collect::>() + }) + .unwrap_or_default(), + }) + .collect::>(); + + provider_summaries.sort_by(|left, right| left.provider.cmp(&right.provider)); + + Some(InstanceAISummary { + providers: provider_summaries, + default_model: extract_instance_ai_model_summary(config, "default_model"), + code_completion_model: extract_instance_ai_model_summary(config, "code_completion_model"), + }) +} + +fn extract_instance_ai_model_summary( + config: &serde_json::Value, + key: &str, +) -> Option { + let model_config = config.get(key)?.as_object()?; + Some(InstanceAIModelSummary { + provider: model_config.get("provider")?.as_str()?.to_owned(), + model: model_config.get("model")?.as_str()?.to_owned(), + }) +} + #[derive(Serialize)] struct DeployTo { deploy_to: Option, @@ -1016,6 +1140,12 @@ async fn edit_webhook( ) -> Result { require_admin(is_admin, &username)?; + if *CLOUD_HOSTED { + return Err(Error::BadRequest( + "Workspace webhooks are not available on cloud-hosted instances".to_string(), + )); + } + let mut tx = db.begin().await?; if let Some(webhook) = &ew.webhook { @@ -1994,24 +2124,20 @@ async fn edit_datatable_config( #[derive(Deserialize)] pub struct EditGitSyncConfig { - #[cfg(feature = "enterprise")] pub git_sync_settings: Option, } -#[cfg(feature = "enterprise")] #[derive(Deserialize, Debug)] pub struct EditGitSyncRepository { pub git_repo_resource_path: String, pub repository: GitRepositorySettings, } -#[cfg(feature = "enterprise")] #[derive(Deserialize, Debug)] pub struct DeleteGitSyncRepositoryRequest { pub git_repo_resource_path: String, } -#[cfg(feature = "enterprise")] fn validate_git_repo_resource_path(path: &str) -> Result<()> { // Resource paths should follow the pattern: $res:f// or $res:u// if path.is_empty() { @@ -2060,7 +2186,6 @@ fn validate_git_repo_resource_path(path: &str) -> Result<()> { Ok(()) } -#[cfg(feature = "enterprise")] fn cleanup_legacy_git_sync_settings_in_memory( git_sync_settings: &mut windmill_common::workspaces::WorkspaceGitSyncSettings, workspace_id: &str, @@ -2087,18 +2212,84 @@ fn cleanup_legacy_git_sync_settings_in_memory( } #[cfg(not(feature = "enterprise"))] -async fn edit_git_sync_config( - _authed: ApiAuthed, - Extension(_db): Extension, - Path(_w_id): Path, - Json(_new_config): Json, -) -> Result { - return Err(Error::BadRequest( - "Git sync is only available on Windmill Enterprise Edition".to_string(), - )); +const CE_GIT_SYNC_MAX_USERS: i64 = 2; + +#[cfg(feature = "enterprise")] +async fn check_git_sync_access(_db: &DB, _w_id: &str) -> Result<()> { + Ok(()) +} + +// Anchor the CE-only query for `cargo sqlx prepare` (which runs with --features enterprise) +#[cfg(feature = "enterprise")] +#[allow(dead_code)] +async fn _sqlx_anchor_ce_user_count(db: &DB, w_id: &str) { + let _ = sqlx::query_scalar!( + "SELECT COUNT(*) FROM usr WHERE workspace_id = $1 AND disabled = false", + w_id + ) + .fetch_one(db) + .await; +} + +#[cfg(not(feature = "enterprise"))] +async fn check_git_sync_access(db: &DB, w_id: &str) -> Result<()> { + let user_count: i64 = sqlx::query_scalar!( + "SELECT COUNT(*) FROM usr WHERE workspace_id = $1 AND disabled = false", + w_id + ) + .fetch_one(db) + .await? + .unwrap_or(0); + + if user_count > CE_GIT_SYNC_MAX_USERS { + return Err(Error::BadRequest(format!( + "Git sync is available for workspaces with up to {} members. \ + Upgrade to Windmill Enterprise Edition for unlimited workspace members.", + CE_GIT_SYNC_MAX_USERS + ))); + } + Ok(()) } #[cfg(feature = "enterprise")] +async fn get_git_sync_enabled( + _authed: ApiAuthed, + Extension(_db): Extension, + Path(_w_id): Path, +) -> JsonResult { + Ok(Json(serde_json::json!({ + "enabled": true, + "reason": "enterprise", + "max_repos": null, + "user_count": null, + "max_users": null, + }))) +} + +#[cfg(not(feature = "enterprise"))] +async fn get_git_sync_enabled( + _authed: ApiAuthed, + Extension(db): Extension, + Path(w_id): Path, +) -> JsonResult { + let user_count: i64 = sqlx::query_scalar!( + "SELECT COUNT(*) FROM usr WHERE workspace_id = $1 AND disabled = false", + &w_id + ) + .fetch_one(&db) + .await? + .unwrap_or(0); + + let enabled = user_count <= CE_GIT_SYNC_MAX_USERS; + Ok(Json(serde_json::json!({ + "enabled": enabled, + "reason": if enabled { Some("free_tier") } else { None::<&str> }, + "max_repos": if enabled { Some(1) } else { None:: }, + "user_count": user_count, + "max_users": CE_GIT_SYNC_MAX_USERS, + }))) +} + async fn edit_git_sync_config( authed: ApiAuthed, Extension(db): Extension, @@ -2107,6 +2298,7 @@ async fn edit_git_sync_config( Json(new_config): Json, ) -> Result { require_admin(is_admin, &username)?; + check_git_sync_access(&db, &w_id).await?; let mut tx = db.begin().await?; @@ -2163,19 +2355,6 @@ async fn edit_git_sync_config( Ok(format!("Edit git sync config for workspace {}", &w_id)) } -#[cfg(not(feature = "enterprise"))] -async fn edit_git_sync_repository( - _authed: ApiAuthed, - Extension(_db): Extension, - Path(_w_id): Path, - Json(_new_config): Json, -) -> Result { - return Err(Error::BadRequest( - "Git sync is only available on Windmill Enterprise Edition".to_string(), - )); -} - -#[cfg(feature = "enterprise")] async fn edit_git_sync_repository( authed: ApiAuthed, Extension(db): Extension, @@ -2184,10 +2363,19 @@ async fn edit_git_sync_repository( Json(new_config): Json, ) -> Result { require_admin(is_admin, &username)?; + check_git_sync_access(&db, &w_id).await?; // Validate the resource path format validate_git_repo_resource_path(&new_config.git_repo_resource_path)?; + // Promotion mode: EE only + #[cfg(not(feature = "enterprise"))] + if new_config.repository.use_individual_branch.unwrap_or(false) { + return Err(Error::BadRequest( + "Promotion mode is an Enterprise Edition feature".to_string(), + )); + } + let mut tx = db.begin().await?; // First, get the current git sync settings @@ -2209,6 +2397,20 @@ async fn edit_git_sync_repository( WorkspaceGitSyncSettings::default() }; + // Multi-repo: EE only + #[cfg(not(feature = "enterprise"))] + { + let is_new = !git_sync_settings + .repositories + .iter() + .any(|r| r.git_repo_resource_path == new_config.git_repo_resource_path); + if is_new && !git_sync_settings.repositories.is_empty() { + return Err(Error::BadRequest( + "Multiple git sync repositories is an Enterprise Edition feature".to_string(), + )); + } + } + // Audit log before we move the repository audit_log( &mut *tx, @@ -2292,19 +2494,6 @@ async fn edit_git_sync_repository( )) } -#[cfg(not(feature = "enterprise"))] -async fn delete_git_sync_repository( - _authed: ApiAuthed, - Extension(_db): Extension, - Path(_w_id): Path, - Json(_request): Json, -) -> Result { - return Err(Error::BadRequest( - "Git sync is only available on Windmill Enterprise Edition".to_string(), - )); -} - -#[cfg(feature = "enterprise")] async fn delete_git_sync_repository( authed: ApiAuthed, Extension(db): Extension, @@ -2314,7 +2503,7 @@ async fn delete_git_sync_repository( ) -> Result { require_admin(is_admin, &username)?; - // For deletion, only validate that path is not empty to allow cleanup of malformed entries + // No check_git_sync_access here — admins should always be able to delete/clean up repos if request.git_repo_resource_path.is_empty() { return Err(Error::BadRequest( "Resource path cannot be empty".to_string(), @@ -2632,22 +2821,29 @@ async fn edit_default_app( #[derive(Serialize)] struct WorkspaceDefaultApp { pub default_app_path: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub default_app_raw: Option, } async fn get_default_app( Extension(db): Extension, Path(w_id): Path, ) -> JsonResult { - let mut tx = db.begin().await?; - let default_app_path = sqlx::query_scalar!( - "SELECT default_app FROM workspace_settings WHERE workspace_id = $1", + let row = sqlx::query!( + "SELECT ws.default_app AS default_app_path, av.raw_app AS \"default_app_raw: Option\" + FROM workspace_settings ws + LEFT JOIN app ON app.path = ws.default_app AND app.workspace_id = ws.workspace_id + LEFT JOIN app_version av ON av.id = app.versions[array_upper(app.versions, 1)] + WHERE ws.workspace_id = $1", &w_id ) - .fetch_one(&mut *tx) + .fetch_one(&db) .await .map_err(|err| Error::internal_err(format!("getting default_app: {err}")))?; - tx.commit().await?; - Ok(Json(WorkspaceDefaultApp { default_app_path })) + Ok(Json(WorkspaceDefaultApp { + default_app_path: row.default_app_path, + default_app_raw: row.default_app_raw, + })) } async fn edit_error_handler( @@ -3819,6 +4015,11 @@ async fn clone_apps( .fetch_all(&mut **tx) .await?; + let mut cloned_from_db: std::collections::HashSet<(i64, String)> = HashSet::new(); + for bundle in &bundles { + cloned_from_db.insert((bundle.app_version_id, bundle.file_type.clone())); + } + for bundle in bundles { if let Some(&new_version_id) = version_id_mapping.get(&bundle.app_version_id) { sqlx::query!( @@ -3833,6 +4034,66 @@ async fn clone_apps( .await?; } } + + // Clone bundles from S3 for versions not found in DB + #[cfg(all(feature = "enterprise", feature = "parquet"))] + { + let object_store = windmill_object_store::get_object_store().await; + if let Some(os) = object_store { + for (&old_version_id, &new_version_id) in &version_id_mapping { + for file_type in &["js", "css"] { + if cloned_from_db.contains(&(old_version_id, file_type.to_string())) { + continue; + } + let src_path = format!( + "/app_bundles/{}/{}.{}", + source_workspace_id, old_version_id, file_type + ); + let get_result = os + .get(&windmill_object_store::object_store_reexports::Path::from( + src_path, + )) + .await; + match get_result { + Ok(result) => { + let data = result.bytes().await.map_err( + windmill_object_store::object_store_error_to_error, + )?; + let dst_path = format!( + "/app_bundles/{}/{}.{}", + target_workspace_id, new_version_id, file_type + ); + os.put( + &windmill_object_store::object_store_reexports::Path::from( + dst_path.clone(), + ), + data.into(), + ) + .await + .map_err( + windmill_object_store::object_store_error_to_error, + )?; + tracing::info!( + "Cloned app bundle from S3: {}.{} -> {}.{}", + old_version_id, + file_type, + new_version_id, + file_type + ); + } + Err(windmill_object_store::object_store_reexports::ObjectStoreError::NotFound { .. }) => { + // No bundle in S3 for this version/type, skip + } + Err(e) => { + return Err( + windmill_object_store::object_store_error_to_error(e), + ); + } + } + } + } + } + } } // Update app versions arrays @@ -4502,6 +4763,20 @@ If you do not have an account on {}, login with SSO or ask an admin to create an )) } +#[derive(Deserialize)] +pub struct NewServiceAccount { + pub username: String, +} + +async fn create_service_account( + authed: ApiAuthed, + Extension(db): Extension, + Path(w_id): Path, + Json(nu): Json, +) -> Result<(StatusCode, String)> { + crate::workspaces_oss::create_service_account(authed, db, w_id, nu).await +} + async fn delete_invite( ApiAuthed { username, is_admin, .. }: ApiAuthed, Extension(db): Extension, @@ -4627,6 +4902,30 @@ async fn get_dependents( Ok(Json(dependents)) } +async fn get_imports( + Extension(db): Extension, + Path((w_id, importer_path)): Path<(String, String)>, + _authed: ApiAuthed, +) -> JsonResult> { + tracing::debug!( + workspace_id = %w_id, + importer_path = %importer_path, + "API: Getting imports for importer path" + ); + + let imports = ScopedDependencyMap::get_imports(&importer_path, &w_id, &db).await?; + + tracing::debug!( + workspace_id = %w_id, + importer_path = %importer_path, + imports_count = imports.len(), + "API: Found imports: {:?}", + imports + ); + + Ok(Json(imports)) +} + #[derive(Serialize, Debug)] struct DependentsAmount { imported_path: String, diff --git a/backend/windmill-api-workspaces/src/workspaces_oss.rs b/backend/windmill-api-workspaces/src/workspaces_oss.rs index da46622c26..872061e554 100644 --- a/backend/windmill-api-workspaces/src/workspaces_oss.rs +++ b/backend/windmill-api-workspaces/src/workspaces_oss.rs @@ -3,7 +3,9 @@ pub use crate::workspaces_ee::*; #[cfg(not(feature = "private"))] -use crate::workspaces::EditAutoInvite; +use crate::workspaces::{EditAutoInvite, NewServiceAccount}; +#[cfg(not(feature = "private"))] +use http::StatusCode; #[cfg(not(feature = "private"))] use windmill_api_auth::ApiAuthed; #[cfg(not(feature = "private"))] @@ -20,3 +22,15 @@ pub async fn edit_auto_invite( "Not implemented on OSS".to_string(), )) } + +#[cfg(not(feature = "private"))] +pub async fn create_service_account( + _authed: ApiAuthed, + _db: DB, + _w_id: String, + _nu: NewServiceAccount, +) -> windmill_common::error::Result<(StatusCode, String)> { + Err(windmill_common::error::Error::BadRequest( + "Service accounts require Windmill Enterprise Edition".to_string(), + )) +} diff --git a/backend/windmill-api/Cargo.toml b/backend/windmill-api/Cargo.toml index 1b09f37861..3f015513e2 100644 --- a/backend/windmill-api/Cargo.toml +++ b/backend/windmill-api/Cargo.toml @@ -18,7 +18,7 @@ agent_worker_server = ["dep:windmill-worker", "dep:windmill-api-agent-workers"] enterprise_saml = ["dep:samael", "dep:libxml"] benchmark = [] embedding = ["windmill-api-embeddings/embedding"] -parquet = ["dep:datafusion", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker?/parquet", "windmill-api-users/parquet", "windmill-api-settings/parquet", "dep:aws-sigv4", "dep:aws-sdk-config"] +parquet = ["dep:datafusion", "windmill-common/parquet", "windmill-object-store/parquet", "windmill-worker?/parquet", "windmill-api-users/parquet", "windmill-api-settings/parquet", "windmill-api-workspaces/parquet", "dep:aws-sigv4", "dep:aws-sdk-config"] prometheus = ["windmill-common/prometheus", "windmill-queue/prometheus", "dep:prometheus", "windmill-worker?/prometheus"] openidconnect = ["dep:openidconnect", "windmill-common/openidconnect", "windmill-store/openidconnect"] tantivy = ["dep:windmill-indexer"] diff --git a/backend/windmill-api/openapi.yaml b/backend/windmill-api/openapi.yaml index e59a2a52b8..cdb0fd15f6 100644 --- a/backend/windmill-api/openapi.yaml +++ b/backend/windmill-api/openapi.yaml @@ -1,7 +1,7 @@ openapi: "3.0.3" info: - version: 1.662.0 + version: 1.670.0 title: Windmill API contact: @@ -588,6 +588,8 @@ paths: type: boolean name: type: string + disabled: + type: boolean responses: "200": description: user updated @@ -2144,6 +2146,87 @@ paths: schema: type: string + /w/{workspace}/workspaces/create_service_account: + post: + summary: create a service account + operationId: createServiceAccount + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + username: + type: string + required: + - username + responses: + "201": + description: service account created + content: + text/plain: + schema: + type: string + + /w/{workspace}/users/impersonate_service_account: + post: + summary: impersonate a service account + operationId: impersonateServiceAccount + tags: + - user + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + username: + type: string + required: + - username + responses: + "201": + description: impersonation token + content: + text/plain: + schema: + type: string + + /w/{workspace}/users/exit_impersonation: + post: + summary: exit service account impersonation + operationId: exitImpersonation + tags: + - user + parameters: + - $ref: "#/components/parameters/WorkspaceId" + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + token: + type: string + required: + - token + responses: + "200": + description: exited impersonation + content: + text/plain: + schema: + type: string + /w/{workspace}/workspaces/delete_invite: post: summary: delete user invite @@ -2745,6 +2828,30 @@ paths: items: $ref: "#/components/schemas/DependencyDependent" + /w/{workspace}/workspaces/get_imports/{importer_path}: + get: + summary: get script imports for an importer path + operationId: getImports + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: importer_path + in: path + required: true + schema: + type: string + description: The script path to get imports for + responses: + "200": + description: list of imported script paths + content: + application/json: + schema: + type: array + items: + type: string + /w/{workspace}/workspaces/get_dependents_amounts: post: summary: get dependents amounts for multiple imported paths @@ -3224,9 +3331,49 @@ paths: "200": description: status content: - text/plain: + application/json: schema: - type: string + type: object + properties: + effective_ai_config: + $ref: "#/components/schemas/AIConfig" + has_instance_ai_config: + type: boolean + uses_instance_ai_config: + type: boolean + instance_ai_summary: + $ref: "#/components/schemas/InstanceAISummary" + required: + - effective_ai_config + - has_instance_ai_config + - uses_instance_ai_config + + /w/{workspace}/workspaces/get_copilot_settings_state: + get: + summary: get copilot settings state + operationId: getCopilotSettingsState + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + + responses: + "200": + description: status + content: + application/json: + schema: + type: object + properties: + has_instance_ai_config: + type: boolean + uses_instance_ai_config: + type: boolean + instance_ai_summary: + $ref: "#/components/schemas/InstanceAISummary" + required: + - has_instance_ai_config + - uses_instance_ai_config /w/{workspace}/workspaces/get_copilot_info: get: @@ -3579,6 +3726,37 @@ paths: pk_constraint_name: type: string + /w/{workspace}/workspaces/git_sync_enabled: + get: + summary: Check if git sync is available for this workspace + operationId: getGitSyncEnabled + tags: + - workspace + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: Git sync availability status + content: + application/json: + schema: + type: object + properties: + enabled: + type: boolean + reason: + type: string + nullable: true + max_repos: + type: integer + nullable: true + user_count: + type: integer + nullable: true + max_users: + type: integer + nullable: true + /w/{workspace}/workspaces/edit_git_sync_config: post: summary: edit workspace git sync settings @@ -3851,6 +4029,8 @@ paths: properties: default_app_path: type: string + default_app_raw: + type: boolean /w/{workspace}/workspaces/usage: get: @@ -4948,6 +5128,11 @@ paths: mcp_server_url: type: string description: "MCP server URL for MCP OAuth token refresh" + scopes: + type: array + items: + type: string + description: "OAuth scopes to use for token refresh. Overrides instance-level scopes." required: - refresh_token - expires_in @@ -7056,6 +7241,60 @@ paths: schema: type: string + /w/{workspace}/scripts/list_dedicated_with_deps: + get: + summary: list dedicated worker scripts with workspace dependency annotations + operationId: listDedicatedWithDeps + tags: + - script + parameters: + - $ref: "#/components/parameters/WorkspaceId" + responses: + "200": + description: list of dedicated scripts with their workspace dependency names + content: + application/json: + schema: + type: array + items: + type: object + properties: + path: + type: string + language: + type: string + enum: + - python3 + - deno + - go + - bash + - powershell + - postgresql + - mysql + - bigquery + - snowflake + - mssql + - graphql + - nativets + - bun + - bunnative + - php + - rust + - ansible + - csharp + - oracledb + - duckdb + - java + - ruby + workspace_dep_names: + type: array + items: + type: string + required: + - path + - language + - workspace_dep_names + /w/{workspace}/scripts/raw/p/{path}: get: summary: raw script by path @@ -10548,6 +10787,7 @@ paths: get: summary: get job operationId: getJob + x-mcp-tool: true tags: - job parameters: @@ -10589,7 +10829,8 @@ paths: /w/{workspace}/jobs_u/get_logs/{id}: get: summary: get job logs - operationId: getJob logs + operationId: getJobLogs + x-mcp-tool: true tags: - job parameters: @@ -10760,6 +11001,10 @@ paths: in: query schema: type: boolean + - name: fast + in: query + schema: + type: boolean responses: "200": @@ -11237,6 +11482,129 @@ paths: "200": description: Interactive slack approval message sent successfully + /w/{workspace}/jobs_u/flow/resume_suspended/{job_id}: + post: + summary: resume or cancel a suspended flow/WAC job + description: > + Resume or cancel a suspended flow/WAC job. Uses approval rules to + determine authorization. Either a valid approval_token or an + authenticated session is required. + operationId: resumeSuspended + tags: + - job + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: job_id + in: path + required: true + schema: + type: string + format: uuid + requestBody: + required: true + content: + application/json: + schema: + type: object + properties: + payload: + description: payload to send to the resumed job + approval_token: + type: string + description: approval token for unauthenticated access + approved: + type: boolean + description: whether to approve (true) or cancel (false) the job + default: true + responses: + "201": + description: job resumed + content: + text/plain: + schema: + type: string + + /w/{workspace}/jobs_u/flow/approval_info/{job_id}: + get: + summary: get approval info for a suspended flow/WAC job + description: > + Get approval info for a suspended flow/WAC job. Returns form schema, + approval rules, and whether the current user can approve. Either a + valid token query parameter or an authenticated session is required. + operationId: getApprovalInfo + tags: + - job + parameters: + - $ref: "#/components/parameters/WorkspaceId" + - name: job_id + in: path + required: true + schema: + type: string + format: uuid + - name: token + in: query + required: false + schema: + type: string + description: approval token for unauthenticated access + responses: + "200": + description: approval info + content: + application/json: + schema: + type: object + required: + - flow_id + - can_approve + - user_auth_required + - approvers + properties: + flow_id: + type: string + format: uuid + form_schema: + description: form schema for the approval step + description: + description: description of the approval step + approval_conditions: + type: object + properties: + user_auth_required: + type: boolean + user_groups_required: + type: array + items: + type: string + self_approval_disabled: + type: boolean + required: + - user_auth_required + - user_groups_required + - self_approval_disabled + can_approve: + type: boolean + description: whether the current user/token holder can approve + user_auth_required: + type: boolean + description: whether user authentication is required to approve + hide_cancel: + type: boolean + description: whether to hide the cancel button in the UI + approvers: + type: array + items: + type: object + required: + - resume_id + - approver + properties: + resume_id: + type: integer + approver: + type: string + /w/{workspace}/jobs_u/resume/{id}/{resume_id}/{signature}: get: summary: resume a job for a suspended flow @@ -17453,6 +17821,27 @@ paths: description: count of log lines that matched the query per hostname type: object + /srch/index/storage/disk: + get: + summary: Get index disk storage sizes from the indexer. + operationId: getIndexDiskStorageSizes + tags: + - indexSearch + responses: + "200": + description: disk storage sizes for each index + content: + application/json: + schema: + type: object + properties: + job_index_disk_size_bytes: + type: integer + nullable: true + log_index_disk_size_bytes: + type: integer + nullable: true + /indexer/delete/{idx_name}: delete: summary: Clear an index and restart the indexer. @@ -18981,6 +19370,33 @@ components: minimum: 1 maximum: 2000000 + InstanceAIProviderSummary: + type: object + properties: + provider: + $ref: "#/components/schemas/AIProvider" + models: + type: array + items: + type: string + required: + - provider + - models + + InstanceAISummary: + type: object + properties: + providers: + type: array + items: + $ref: "#/components/schemas/InstanceAIProviderSummary" + default_model: + $ref: "#/components/schemas/AIProviderModel" + code_completion_model: + $ref: "#/components/schemas/AIProviderModel" + required: + - providers + Alert: type: object properties: @@ -19979,6 +20395,8 @@ components: nullable: true allOf: - $ref: "#/components/schemas/UserSource" + is_service_account: + type: boolean required: - email - username @@ -21590,6 +22008,13 @@ components: required: - key - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: "Logic to apply when evaluating filters. 'and' requires all filters to match, 'or' requires any filter to match." initial_messages: type: array nullable: true @@ -21651,6 +22076,13 @@ components: required: - key - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: "Logic to apply when evaluating filters. 'and' requires all filters to match, 'or' requires any filter to match." initial_messages: type: array nullable: true @@ -21719,6 +22151,13 @@ components: required: - key - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: "Logic to apply when evaluating filters. 'and' requires all filters to match, 'or' requires any filter to match." initial_messages: type: array nullable: true @@ -22595,6 +23034,13 @@ components: required: - key - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: "Logic to apply when evaluating filters. 'and' requires all filters to match, 'or' requires any filter to match." auto_offset_reset: type: string enum: @@ -22666,6 +23112,13 @@ components: required: - key - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: "Logic to apply when evaluating filters. 'and' requires all filters to match, 'or' requires any filter to match." auto_offset_reset: type: string enum: @@ -22729,6 +23182,13 @@ components: required: - key - value + filter_logic: + type: string + enum: + - and + - or + default: and + description: "Logic to apply when evaluating filters. 'and' requires all filters to match, 'or' requires any filter to match." auto_offset_reset: type: string enum: @@ -23360,6 +23820,8 @@ components: role_source: type: string enum: ["manual", "instance_group"] + disabled: + type: boolean required: - email @@ -23368,6 +23830,7 @@ components: - verified - first_time_user - role_source + - disabled Flow: allOf: @@ -24791,6 +25254,10 @@ components: type: string nullable: true description: Error message if the trigger is in an error state + summary: + type: string + nullable: true + description: Short summary to be displayed when listed required: - external_id - workspace_id @@ -24825,6 +25292,10 @@ components: type: string nullable: true description: Error message if the trigger is in an error state + summary: + type: string + nullable: true + description: Short summary to be displayed when listed external_data: type: object description: Configuration data from the external service @@ -24917,6 +25388,10 @@ components: type: object description: Service-specific configuration (e.g., event types, filters) additionalProperties: true + summary: + type: string + nullable: true + description: Short summary to be displayed when listed required: - script_path - is_flow diff --git a/backend/windmill-api/src/ai.rs b/backend/windmill-api/src/ai.rs index ed9cdb366c..16cfb0166c 100644 --- a/backend/windmill-api/src/ai.rs +++ b/backend/windmill-api/src/ai.rs @@ -16,6 +16,7 @@ use serde_json::{json, value::RawValue}; use std::collections::HashMap; use std::time::Duration; use windmill_audit::{audit_oss::audit_log, ActionKind}; +use windmill_common::ai_cache::current_instance_ai_config_revision; use windmill_common::ai_providers::{ empty_string_as_none, AIPlatform, AIProvider, ProviderConfig, ProviderModel, }; @@ -127,6 +128,10 @@ lazy_static::lazy_static! { }; } +pub(crate) fn invalidate_ai_request_cache_for_workspace(workspace_id: &str) { + AI_REQUEST_CACHE.retain(|(cached_workspace_id, _), _| cached_workspace_id != workspace_id); +} + #[derive(Deserialize, Debug)] struct AIOAuthResource { client_id: String, @@ -373,8 +378,7 @@ impl AIRequestConfig { let is_azure = provider.is_azure_openai(base_url); let is_anthropic = matches!(provider, AIProvider::Anthropic); - let is_anthropic_vertex = - is_anthropic && self.platform == AIPlatform::GoogleVertexAi; + let is_anthropic_vertex = is_anthropic && self.platform == AIPlatform::GoogleVertexAi; let is_anthropic_sdk = headers.get("X-Anthropic-SDK").is_some(); let is_google_ai = matches!(provider, AIProvider::GoogleAI); @@ -483,18 +487,27 @@ impl AIRequestConfig { pub struct ExpiringAIRequestConfig { config: AIRequestConfig, expires_at: std::time::Instant, + instance_ai_config_revision: Option, } impl ExpiringAIRequestConfig { - fn new(config: AIRequestConfig) -> Self { - Self { config, expires_at: std::time::Instant::now() + std::time::Duration::from_secs(60) } + fn new(config: AIRequestConfig, instance_ai_config_revision: Option) -> Self { + Self { + config, + expires_at: std::time::Instant::now() + std::time::Duration::from_secs(60), + instance_ai_config_revision, + } } + fn is_expired(&self) -> bool { self.expires_at < std::time::Instant::now() + || self + .instance_ai_config_revision + .is_some_and(|revision| revision != current_instance_ai_config_revision()) } } -#[derive(Serialize, Deserialize, Debug)] +#[derive(Serialize, Deserialize, Debug, Default)] pub struct AIConfig { #[serde(skip_serializing_if = "Option::is_none")] pub providers: Option>, @@ -508,6 +521,14 @@ pub struct AIConfig { pub max_tokens_per_model: Option>, } +impl AIConfig { + pub fn has_providers(&self) -> bool { + self.providers + .as_ref() + .is_some_and(|providers| !providers.is_empty()) + } +} + /// Anthropic API version for Google Vertex AI const ANTHROPIC_VERSION_VERTEX: &str = "vertex-2023-10-16"; @@ -586,11 +607,11 @@ fn transform_fim_to_chat_completions(body: &Bytes) -> Result<(Bytes, String)> { } pub fn global_service() -> Router { - Router::new().route("/proxy/*ai", post(global_proxy).get(global_proxy)) + Router::new().route("/proxy/{*ai}", post(global_proxy).get(global_proxy)) } pub fn workspaced_service() -> Router { - let router = Router::new().route("/proxy/*ai", post(proxy).get(proxy)); + let router = Router::new().route("/proxy/{*ai}", post(proxy).get(proxy)); #[cfg(feature = "bedrock")] let router = router.route("/check_bedrock_credentials", get(check_bedrock_credentials)); @@ -762,47 +783,76 @@ async fn proxy( request_cache.config } _ => { - let (resource_path, save_to_cache) = if let Some(resource_path) = forced_resource_path { - // forced resource path - (resource_path, false) - } else { - let ai_config = sqlx::query_scalar!( - "SELECT ai_config FROM workspace_settings WHERE workspace_id = $1", - &w_id - ) - .fetch_one(&db) - .await?; + let (resource_path, save_to_cache, resource_workspace, instance_ai_config_revision) = + if let Some(resource_path) = forced_resource_path { + // forced resource path + (resource_path, false, w_id.clone(), None) + } else { + let workspace_ai_config = sqlx::query_scalar!( + "SELECT ai_config FROM workspace_settings WHERE workspace_id = $1", + &w_id + ) + .fetch_one(&db) + .await?; - if ai_config.is_none() { - return Err(Error::internal_err( - "AI resource not configured".to_string(), - )); - } + let (ai_config_value, resource_workspace, instance_ai_config_revision) = { + let ws_has_config = workspace_ai_config + .as_ref() + .and_then(|v| serde_json::from_value::(v.clone()).ok()) + .is_some_and(|config| config.has_providers()); - let mut ai_config = serde_json::from_value::(ai_config.unwrap()) - .map_err(|e| Error::BadRequest(e.to_string()))?; + if ws_has_config { + (workspace_ai_config.unwrap(), w_id.clone(), None) + } else { + let instance_config = sqlx::query_scalar!( + "SELECT value FROM global_settings WHERE name = 'ai_config'" + ) + .fetch_optional(&db) + .await?; - let provider_config = ai_config - .providers - .as_mut() - .map(|providers| providers.remove(&provider)) - .flatten() - .ok_or_else(|| { - Error::BadRequest(format!("Provider {:?} not configured", provider)) - })?; + match instance_config { + Some(config) => ( + config, + "admins".to_string(), + Some(current_instance_ai_config_revision()), + ), + None => { + return Err(Error::internal_err( + "AI resource not configured".to_string(), + )); + } + } + } + }; - if provider_config.resource_path.is_empty() { - return Err(Error::BadRequest("Resource path is empty".to_string())); - } + let mut ai_config = serde_json::from_value::(ai_config_value) + .map_err(|e| Error::BadRequest(e.to_string()))?; - (provider_config.resource_path, true) - }; + let provider_config = ai_config + .providers + .as_mut() + .and_then(|providers| providers.remove(&provider)) + .ok_or_else(|| { + Error::BadRequest(format!("Provider {:?} not configured", provider)) + })?; - let resource= sqlx::query_scalar!( - "SELECT value as \"value: sqlx::types::Json>\" FROM resource WHERE path = $1 AND workspace_id = $2", - &resource_path, - &w_id + if provider_config.resource_path.is_empty() { + return Err(Error::BadRequest("Resource path is empty".to_string())); + } + + ( + provider_config.resource_path, + true, + resource_workspace, + instance_ai_config_revision, + ) + }; + + let resource = sqlx::query_scalar::<_, Option>>>( + "SELECT value FROM resource WHERE path = $1 AND workspace_id = $2", ) + .bind(&resource_path) + .bind(&resource_workspace) .fetch_optional(&db) .await? .ok_or_else(|| Error::NotFound(format!("Could not find the resource {}, update the resource path in the workspace settings", resource_path)))? @@ -811,11 +861,15 @@ async fn proxy( let resource = serde_json::from_str::(resource.0.get()) .map_err(|e| Error::BadRequest(e.to_string()))?; - let request_config = AIRequestConfig::new(&provider, &db, &w_id, resource).await?; + let request_config = + AIRequestConfig::new(&provider, &db, &resource_workspace, resource).await?; if save_to_cache { AI_REQUEST_CACHE.insert( (w_id.clone(), provider.clone()), - ExpiringAIRequestConfig::new(request_config.clone()), + ExpiringAIRequestConfig::new( + request_config.clone(), + instance_ai_config_revision, + ), ); } request_config @@ -858,9 +912,7 @@ async fn proxy( "chat/completions" => { crate::google::handle_google_ai_chat(&body, api_key, base_url, is_vertex).await } - "models" => { - crate::google::handle_google_ai_models(api_key, base_url, is_vertex).await - } + "models" => crate::google::handle_google_ai_models(api_key, base_url, is_vertex).await, _ => Err(Error::BadRequest(format!( "Unsupported Google AI path: {}", ai_path @@ -1005,3 +1057,76 @@ async fn proxy( }; Ok((status_code, headers, body)) } + +#[cfg(test)] +mod tests { + use super::*; + use std::sync::{LazyLock, Mutex}; + use windmill_common::ai_cache::bump_instance_ai_config_revision; + use windmill_common::ai_providers::AIPlatform; + + static TEST_LOCK: LazyLock> = LazyLock::new(|| Mutex::new(())); + + fn sample_request_config() -> AIRequestConfig { + AIRequestConfig { + base_url: "https://example.com".to_string(), + api_key: None, + access_token: None, + organization_id: None, + user: None, + region: None, + aws_access_key_id: None, + aws_secret_access_key: None, + aws_session_token: None, + platform: AIPlatform::Standard, + enable_1m_context: false, + custom_headers: HashMap::new(), + } + } + + #[test] + fn invalidates_all_cached_providers_for_workspace() { + let _guard = TEST_LOCK.lock().unwrap(); + AI_REQUEST_CACHE.clear(); + AI_REQUEST_CACHE.insert( + ("workspace-a".to_string(), AIProvider::OpenAI), + ExpiringAIRequestConfig::new(sample_request_config(), None), + ); + AI_REQUEST_CACHE.insert( + ("workspace-a".to_string(), AIProvider::Anthropic), + ExpiringAIRequestConfig::new(sample_request_config(), None), + ); + AI_REQUEST_CACHE.insert( + ("workspace-b".to_string(), AIProvider::OpenAI), + ExpiringAIRequestConfig::new(sample_request_config(), None), + ); + + invalidate_ai_request_cache_for_workspace("workspace-a"); + + assert!(AI_REQUEST_CACHE + .get(&("workspace-a".to_string(), AIProvider::OpenAI)) + .is_none()); + assert!(AI_REQUEST_CACHE + .get(&("workspace-a".to_string(), AIProvider::Anthropic)) + .is_none()); + assert!(AI_REQUEST_CACHE + .get(&("workspace-b".to_string(), AIProvider::OpenAI)) + .is_some()); + } + + #[test] + fn instance_backed_cache_entries_expire_when_revision_changes() { + let _guard = TEST_LOCK.lock().unwrap(); + AI_REQUEST_CACHE.clear(); + + let cached = ExpiringAIRequestConfig::new( + sample_request_config(), + Some(current_instance_ai_config_revision()), + ); + assert!(!cached.is_expired()); + + bump_instance_ai_config_revision(); + + assert!(cached.is_expired()); + } +} diff --git a/backend/windmill-api/src/approvals.rs b/backend/windmill-api/src/approvals.rs index fe9adb602e..e5b5f019eb 100644 --- a/backend/windmill-api/src/approvals.rs +++ b/backend/windmill-api/src/approvals.rs @@ -259,7 +259,7 @@ pub async fn get_approval_form_details( } }; - let flow_value = &flow_data.flow; + let flow_value = flow_data.value(); let flow_step_id = flow_step_id.unwrap_or(""); let module = flow_value.modules.iter().find(|m| m.id == flow_step_id); diff --git a/backend/windmill-api/src/apps.rs b/backend/windmill-api/src/apps.rs index 33a632ecbd..e9f262881a 100644 --- a/backend/windmill-api/src/apps.rs +++ b/backend/windmill-api/src/apps.rs @@ -79,52 +79,64 @@ use windmill_common::{jwt, oauth2::HmacSha256, variables::get_workspace_key}; #[cfg(feature = "parquet")] use windmill_types::s3::{S3Object, S3Permission}; -pub fn workspaced_service() -> Router { +pub fn workspaced_service(raw_app_body_limit: usize) -> Router { Router::new() .route("/list", get(list_apps)) .route("/list_search", get(list_search_apps)) - .route("/get/p/*path", get(get_app)) - .route("/get/lite/*path", get(get_app_lite)) - .route("/get/draft/*path", get(get_app_w_draft)) - .route("/secret_of/*path", get(get_secret_id)) + .route("/get/p/{*path}", get(get_app)) + .route("/get/lite/{*path}", get(get_app_lite)) + .route("/get/draft/{*path}", get(get_app_w_draft)) + .route("/secret_of/{*path}", get(get_secret_id)) .route( - "/secret_of_latest_version/*path", + "/secret_of_latest_version/{*path}", get(get_latest_version_secret_id), ) - .route("/get/v/*id", get(get_app_by_id)) - .route("/get_data/v/*id", get(get_raw_app_data)) - .route("/exists/*path", get(exists_app)) - .route("/update/*path", post(update_app)) - .route("/update_raw/*path", post(update_app_raw)) - .route("/delete/*path", delete(delete_app)) - .route("/create", post(create_app)) - .route("/create_raw", post(create_app_raw)) - .route("/history/p/*path", get(get_app_history)) - .route("/get_latest_version/*path", get(get_latest_version)) - .route("/history_update/a/:id/v/:version", post(update_app_history)) + .route("/get/v/{*id}", get(get_app_by_id)) + .route("/get_data/v/{*id}", get(get_raw_app_data)) + .route("/exists/{*path}", get(exists_app)) + .route("/update/{*path}", post(update_app)) .route( - "/list_paths_from_workspace_runnable/:runnable_kind/*path", + "/update_raw/{*path}", + post(update_app_raw).layer(axum::extract::DefaultBodyLimit::max(raw_app_body_limit)), + ) + .route("/delete/{*path}", delete(delete_app)) + .route("/create", post(create_app)) + .route( + "/create_raw", + post(create_app_raw).layer(axum::extract::DefaultBodyLimit::max(raw_app_body_limit)), + ) + .route("/history/p/{*path}", get(get_app_history)) + .route("/get_latest_version/{*path}", get(get_latest_version)) + .route( + "/history_update/a/{id}/v/{version}", + post(update_app_history), + ) + .route( + "/list_paths_from_workspace_runnable/{runnable_kind}/{*path}", get(list_paths_from_workspace_runnable), ) - .route("/custom_path_exists/*custom_path", get(custom_path_exists)) + .route( + "/custom_path_exists/{*custom_path}", + get(custom_path_exists), + ) .route("/sign_s3_objects", post(sign_s3_objects)) } pub fn unauthed_service() -> Router { Router::new() - .route("/execute_component/*path", post(execute_component)) - .route("/upload_s3_file/*path", post(upload_s3_file_from_app)) + .route("/execute_component/{*path}", post(execute_component)) + .route("/upload_s3_file/{*path}", post(upload_s3_file_from_app)) .route("/delete_s3_file", delete(delete_s3_file_from_app)) - .route("/download_s3_file/*path", get(download_s3_file_from_app)) - .route("/public_app/:secret", get(get_public_app_by_secret)) - .route("/public_resource/*path", get(get_public_resource)) - .route("/get_data/v/*id", get(get_raw_app_data)) + .route("/download_s3_file/{*path}", get(download_s3_file_from_app)) + .route("/public_app/{secret}", get(get_public_app_by_secret)) + .route("/public_resource/{*path}", get(get_public_resource)) + .route("/get_data/v/{*id}", get(get_raw_app_data)) } pub fn global_service() -> Router { Router::new() .route("/hub/list", get(list_hub_apps)) - .route("/hub/get/:id", get(get_hub_app_by_id)) - .route("/hub/get_raw/:id", get(get_hub_raw_app_by_id)) + .route("/hub/get/{id}", get(get_hub_app_by_id)) + .route("/hub/get_raw/{id}", get(get_hub_raw_app_by_id)) } #[derive(FromRow, Deserialize, Serialize)] @@ -440,17 +452,29 @@ async fn get_raw_app_data( #[cfg(all(feature = "enterprise", feature = "parquet"))] if let Some(os) = object_store { let path = format!("/app_bundles/{}/{}.{}", w_id, id, file_type); - let stream = os + match os .get(&windmill_object_store::object_store_reexports::Path::from( path, )) .await - .map_err(windmill_object_store::object_store_error_to_error)? - .bytes() - .await - .map_err(windmill_object_store::object_store_error_to_error)?; - tracing::info!("stream: {}", stream.len()); - body = Some(Body::from(stream)); + { + Ok(result) => { + let stream = result + .bytes() + .await + .map_err(windmill_object_store::object_store_error_to_error)?; + tracing::info!("stream: {}", stream.len()); + body = Some(Body::from(stream)); + } + Err(windmill_object_store::object_store_reexports::ObjectStoreError::NotFound { + .. + }) => { + // S3 key not found, fall through to DB lookup below + } + Err(e) => { + return Err(windmill_object_store::object_store_error_to_error(e)); + } + } } if body.is_none() { @@ -992,18 +1016,20 @@ macro_rules! process_app_multipart { let mut saved_app = None; let mut uploaded_js = false; + let request_size_limit_mb = *crate::REQUEST_SIZE_LIMIT.read().await / (1024 * 1024); + let raw_app_limit_mb = request_size_limit_mb * 5; let mut multipart = $multipart; while let Some(field) = multipart .next_field() .await - .map_err(|e| Error::BadRequest(format!("failed to read multipart field: {e}")))? + .map_err(|e| Error::BadRequest(format!("failed to read multipart field: {e}. Could be due to the request size limit for raw app bundles which is {raw_app_limit_mb}MB (adjustable in instance settings)")))? { let name = field .name() .ok_or_else(|| Error::BadRequest("multipart field missing name".to_string()))? .to_string(); let data = field.bytes().await.map_err(|e| { - Error::BadRequest(format!("failed to read multipart stream: {e}")) + Error::BadRequest(format!("failed to read multipart stream: {e}. Could be due to the request size limit for raw app bundles which is {raw_app_limit_mb}MB (adjustable in instance settings)")) })?; if name == "app" { let app = serde_json::from_slice(&data).map_err(to_anyhow)?; @@ -1439,6 +1465,30 @@ async fn delete_app( let mut tx = user_db.begin(&authed).await?; + // Capture all related data for trashbin before deleting (CASCADE will remove app_version, etc.) + let trash_app: Option = + sqlx::query_scalar("SELECT to_jsonb(t) FROM app t WHERE path = $1 AND workspace_id = $2") + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + + let trash_app_versions: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM app_version t WHERE app_id = (SELECT id FROM app WHERE path = $1 AND workspace_id = $2)", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + + let trash_drafts: Vec = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM draft t WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", + ) + .bind(path) + .bind(&w_id) + .fetch_all(&mut *tx) + .await?; + sqlx::query!( "DELETE FROM draft WHERE path = $1 AND workspace_id = $2 AND typ = 'app'", path, @@ -1455,6 +1505,25 @@ async fn delete_app( .execute(&mut *tx) .await?; + if let Some(app_data) = trash_app { + let mut trash_data = serde_json::json!({"row": app_data}); + if !trash_app_versions.is_empty() { + trash_data["app_versions"] = serde_json::Value::Array(trash_app_versions); + } + if !trash_drafts.is_empty() { + trash_data["drafts"] = serde_json::Value::Array(trash_drafts); + } + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "app", + path, + trash_data, + &authed.username, + ) + .await?; + } + audit_log( &mut *tx, &authed, diff --git a/backend/windmill-api/src/args.rs b/backend/windmill-api/src/args.rs index 2734be6740..a881fb989d 100644 --- a/backend/windmill-api/src/args.rs +++ b/backend/windmill-api/src/args.rs @@ -451,8 +451,7 @@ where } } -#[axum::async_trait] -impl FromRequest for RawWebhookArgs +impl FromRequest for RawWebhookArgs where S: Send + Sync, { diff --git a/backend/windmill-api/src/audit.rs b/backend/windmill-api/src/audit.rs index 336fd32881..7f81df849c 100644 --- a/backend/windmill-api/src/audit.rs +++ b/backend/windmill-api/src/audit.rs @@ -19,7 +19,7 @@ use crate::db::ApiAuthed; pub fn workspaced_service() -> Router { Router::new() .route("/list", get(list_audit)) - .route("/get/:id", get(get_audit)) + .route("/get/{id}", get(get_audit)) } async fn get_audit( diff --git a/backend/windmill-api/src/capture.rs b/backend/windmill-api/src/capture.rs index 277504b481..c8ddea37d2 100644 --- a/backend/windmill-api/src/capture.rs +++ b/backend/windmill-api/src/capture.rs @@ -93,22 +93,22 @@ pub fn workspaced_service() -> Router { Router::new() .route("/set_config", post(set_config)) .route( - "/ping_config/:trigger_kind/:runnable_kind/*path", + "/ping_config/{trigger_kind}/{runnable_kind}/{*path}", post(ping_config), ) - .route("/get_configs/:runnable_kind/*path", get(get_configs)) - .route("/list/:runnable_kind/*path", get(list_captures)) + .route("/get_configs/{runnable_kind}/{*path}", get(get_configs)) + .route("/list/{runnable_kind}/{*path}", get(list_captures)) .route( - "/move/:runnable_kind/*path", + "/move/{runnable_kind}/{*path}", post(move_captures_and_configs), ) - .route("/:id", delete(delete_capture)) - .route("/:id", get(get_capture)) + .route("/{id}", delete(delete_capture)) + .route("/{id}", get(get_capture)) } pub fn workspaced_unauthed_service() -> Router { let router = Router::new().route( - "/webhook/:runnable_kind/*path", + "/webhook/{runnable_kind}/{*path}", head(|| async {}).post(webhook_payload), ); @@ -118,12 +118,12 @@ pub fn workspaced_unauthed_service() -> Router { ))] { #[cfg(feature = "http_trigger")] - let router = router.route("/http/:runnable_kind/:path/*route_path", { + let router = router.route("/http/{runnable_kind}/{path}/{*route_path}", { head(|| async {}).fallback(http_payload) }); #[cfg(all(feature = "enterprise", feature = "gcp_trigger", feature = "private"))] - let router = router.route("/gcp/:runnable_kind/*path", post(gcp_payload)); + let router = router.route("/gcp/{runnable_kind}/{*path}", post(gcp_payload)); router } diff --git a/backend/windmill-api/src/db_health.rs b/backend/windmill-api/src/db_health.rs new file mode 100644 index 0000000000..5b881331c0 --- /dev/null +++ b/backend/windmill-api/src/db_health.rs @@ -0,0 +1,551 @@ +/* + * Author: Windmill Labs + * Copyright: Windmill Labs, Inc 2024 + * This file and its contents are licensed under the AGPLv3 License. + * Please see the included NOTICE for copyright information and + * LICENSE-AGPL for a copy of the license. + */ + +use axum::{extract::Query, routing::get, Extension, Json, Router}; +use serde::{Deserialize, Serialize}; + +use windmill_common::error::JsonResult; + +use crate::db::{ApiAuthed, DB}; +use crate::utils::require_super_admin; + +pub fn global_service() -> Router { + Router::new().route("/", get(get_db_health)) +} + +// --- Response types --- + +#[derive(Serialize)] +#[serde(rename_all = "lowercase")] +pub enum HealthLevel { + Green, + Yellow, + Red, +} + +#[derive(Serialize)] +pub struct DbHealthResponse { + pub database_size: DatabaseSizeInfo, + pub job_retention: JobRetentionInfo, + pub large_results: LargeResultsInfo, + pub connection_pool: ConnectionPoolInfo, + pub table_maintenance: Vec, + pub slow_queries: Option, + pub datatables: Vec, +} + +#[derive(Serialize)] +pub struct DatabaseSizeInfo { + pub total_size_bytes: i64, + pub total_size_pretty: String, + pub top_tables: Vec, +} + +#[derive(Serialize)] +pub struct TableSizeInfo { + pub table_name: String, + pub total_size_bytes: i64, + pub total_size_pretty: String, +} + +#[derive(Serialize)] +pub struct JobRetentionInfo { + pub oldest_completed_at: Option>, + pub total_completed_jobs: i64, + pub retention_period_secs: Option, + pub status: HealthLevel, + pub message: String, +} + +#[derive(Serialize)] +pub struct LargeResultsInfo { + pub top_large_results: Vec, + pub avg_result_size_bytes: Option, +} + +#[derive(Serialize)] +pub struct LargeResultRow { + pub id: uuid::Uuid, + pub workspace_id: String, + pub runnable_path: Option, + pub result_size_bytes: i64, + pub completed_at: chrono::DateTime, +} + +#[derive(Serialize)] +pub struct ConnectionPoolInfo { + pub pg_max_connections: i64, + pub pg_total_connections: i64, + pub pg_active_connections: i64, + pub pg_idle_connections: i64, + pub status: HealthLevel, + pub message: String, +} + +#[derive(Serialize)] +pub struct TableMaintenanceInfo { + pub table_name: String, + pub live_tuples: i64, + pub dead_tuples: i64, + pub dead_ratio: f64, + pub last_autovacuum: Option, + pub last_autoanalyze: Option, + pub status: HealthLevel, +} + +#[derive(Serialize)] +pub struct SlowQueriesInfo { + pub queries: Vec, + pub message: Option, +} + +#[derive(Serialize)] +pub struct SlowQueryRow { + pub query: String, + pub calls: i64, + pub total_exec_time_ms: f64, + pub mean_exec_time_ms: f64, +} + +#[derive(Serialize)] +pub struct DatatableInfo { + pub workspace_id: String, + pub name: String, + pub table_name: String, + pub size_bytes: i64, + pub size_pretty: String, + pub estimated_rows: f64, +} + +// --- Handler --- + +#[derive(Deserialize)] +struct DbHealthQuery { + /// Max number of recent completed jobs to scan for large results (default 10000) + scan_limit: Option, +} + +async fn get_db_health( + ApiAuthed { email, .. }: ApiAuthed, + Extension(db): Extension, + Query(query): Query, +) -> JsonResult { + require_super_admin(&db, &email).await?; + + let scan_limit = query.scan_limit.unwrap_or(10_000).clamp(1_000, 1_000_000); + + let ( + database_size, + job_retention, + large_results, + connection_pool, + table_maintenance, + slow_queries, + datatables, + ) = tokio::try_join!( + fetch_database_size(&db), + fetch_job_retention(&db), + fetch_large_results(&db, scan_limit), + fetch_connection_pool(&db), + fetch_table_maintenance(&db), + fetch_slow_queries(&db), + fetch_datatables(&db), + )?; + + Ok(Json(DbHealthResponse { + database_size, + job_retention, + large_results, + connection_pool, + table_maintenance, + slow_queries, + datatables, + })) +} + +// --- Diagnostic queries --- + +async fn fetch_database_size(db: &DB) -> windmill_common::error::Result { + let row = sqlx::query!( + "SELECT pg_database_size(current_database()) as size_bytes, pg_size_pretty(pg_database_size(current_database())) as size_pretty" + ) + .fetch_one(db) + .await?; + + let top_tables = sqlx::query_as!( + TableSizeInfo, + r#"SELECT + schemaname || '.' || relname as "table_name!", + pg_total_relation_size(relid) as "total_size_bytes!", + pg_size_pretty(pg_total_relation_size(relid)) as "total_size_pretty!" + FROM pg_catalog.pg_statio_user_tables + ORDER BY pg_total_relation_size(relid) DESC + LIMIT 15"# + ) + .fetch_all(db) + .await?; + + Ok(DatabaseSizeInfo { + total_size_bytes: row.size_bytes.unwrap_or(0), + total_size_pretty: row.size_pretty.unwrap_or_default(), + top_tables, + }) +} + +async fn fetch_job_retention(db: &DB) -> windmill_common::error::Result { + let job_row = + sqlx::query!("SELECT MIN(completed_at) as oldest, COUNT(*) as total FROM v2_job_completed") + .fetch_one(db) + .await?; + + let retention_row = + sqlx::query!("SELECT value FROM global_settings WHERE name = 'retention_period_secs'") + .fetch_optional(db) + .await?; + + let retention_period_secs: Option = + retention_row.map(|r| r.value).and_then(|v| v.as_i64()); + + let oldest = job_row.oldest; + let total = job_row.total.unwrap_or(0); + + let (status, message) = if let (Some(oldest_ts), Some(retention_secs)) = + (oldest, retention_period_secs) + { + let age_secs: i64 = (chrono::Utc::now() - oldest_ts).num_seconds(); + let ratio = if retention_secs > 0 { + age_secs as f64 / retention_secs as f64 + } else { + 0.0 + }; + if ratio <= 2.0 { + ( + HealthLevel::Green, + format!( + "Oldest job is {:.1}x the retention period. Cleanup is keeping up.", + ratio + ), + ) + } else if ratio <= 5.0 { + ( + HealthLevel::Yellow, + format!( + "Oldest job is {:.1}x the retention period. Cleanup may be falling behind.", + ratio + ), + ) + } else { + (HealthLevel::Red, format!("Oldest job is {:.1}x the retention period. Consider reducing retention or investigating cleanup.", ratio)) + } + } else if oldest.is_some() && retention_period_secs.is_none() { + ( + HealthLevel::Yellow, + "No retention_period_secs configured. Old jobs will accumulate.".to_string(), + ) + } else { + (HealthLevel::Green, "No completed jobs found.".to_string()) + }; + + Ok(JobRetentionInfo { + oldest_completed_at: oldest, + total_completed_jobs: total, + retention_period_secs, + status, + message, + }) +} + +async fn fetch_large_results( + db: &DB, + scan_limit: i64, +) -> windmill_common::error::Result { + let top_large_results = sqlx::query_as!( + LargeResultRow, + r#"SELECT + c.id as "id!", + c.workspace_id as "workspace_id!", + j.runnable_path as "runnable_path", + pg_column_size(c.result) as "result_size_bytes!", + c.completed_at as "completed_at!" + FROM ( + SELECT id, workspace_id, result, completed_at + FROM v2_job_completed + WHERE completed_at > now() - interval '30 days' + AND result IS NOT NULL + ORDER BY completed_at DESC + LIMIT $1 + ) c + LEFT JOIN v2_job j ON j.id = c.id + WHERE pg_column_size(c.result) > 1024 + ORDER BY pg_column_size(c.result) DESC + LIMIT 10"#, + scan_limit + ) + .fetch_all(db) + .await?; + + let avg_row = sqlx::query!( + r#"SELECT AVG(pg_column_size(result))::bigint as "avg_size" + FROM ( + SELECT result FROM v2_job_completed + WHERE completed_at > now() - interval '30 days' + AND result IS NOT NULL + ORDER BY completed_at DESC + LIMIT $1 + ) sub"#, + scan_limit + ) + .fetch_one(db) + .await?; + + Ok(LargeResultsInfo { top_large_results, avg_result_size_bytes: avg_row.avg_size }) +} + +async fn fetch_connection_pool(db: &DB) -> windmill_common::error::Result { + let max_row = sqlx::query_scalar!( + r#"SELECT setting::bigint as "max!" FROM pg_settings WHERE name = 'max_connections'"# + ) + .fetch_one(db) + .await?; + + let stats_row = sqlx::query!( + r#"SELECT + COUNT(*) as "total!", + COUNT(*) FILTER (WHERE state = 'active') as "active!", + COUNT(*) FILTER (WHERE state = 'idle') as "idle!" + FROM pg_stat_activity + WHERE backend_type = 'client backend'"# + ) + .fetch_one(db) + .await?; + + let pg_max = max_row; + let pg_total = stats_row.total; + let pg_active = stats_row.active; + let pg_idle = stats_row.idle; + + let utilization = if pg_max > 0 { + pg_total as f64 / pg_max as f64 + } else { + 0.0 + }; + + let (status, message) = if utilization < 0.8 { + ( + HealthLevel::Green, + format!( + "Connection utilization: {:.0}% ({}/{})", + utilization * 100.0, + pg_total, + pg_max + ), + ) + } else if utilization < 0.95 { + ( + HealthLevel::Yellow, + format!( + "Connection utilization is high: {:.0}% ({}/{}). Consider increasing max_connections.", + utilization * 100.0, + pg_total, + pg_max + ), + ) + } else { + ( + HealthLevel::Red, + format!( + "Connections near exhaustion: {:.0}% ({}/{}). Increase max_connections urgently.", + utilization * 100.0, + pg_total, + pg_max + ), + ) + }; + + Ok(ConnectionPoolInfo { + pg_max_connections: pg_max, + pg_total_connections: pg_total, + pg_active_connections: pg_active, + pg_idle_connections: pg_idle, + status, + message, + }) +} + +async fn fetch_table_maintenance( + db: &DB, +) -> windmill_common::error::Result> { + // Aggregate partitioned tables (e.g. audit_YYYYMMDD -> audit_partitioned) + // while keeping non-partitioned tables as-is + let rows = sqlx::query!( + r#"SELECT + table_name as "table_name!", + SUM(live_tuples)::bigint as "live_tuples!", + SUM(dead_tuples)::bigint as "dead_tuples!", + MAX(last_autovacuum) as "last_autovacuum", + MAX(last_autoanalyze) as "last_autoanalyze" + FROM ( + SELECT + CASE + WHEN i.inhparent IS NOT NULL THEN schemaname || '.' || p.relname + ELSE schemaname || '.' || s.relname + END as table_name, + COALESCE(n_live_tup, 0) as live_tuples, + COALESCE(n_dead_tup, 0) as dead_tuples, + last_autovacuum, + last_autoanalyze + FROM pg_stat_user_tables s + LEFT JOIN pg_class c ON c.relname = s.relname AND c.relnamespace = ( + SELECT oid FROM pg_namespace WHERE nspname = s.schemaname + ) + LEFT JOIN pg_inherits i ON i.inhrelid = c.oid + LEFT JOIN pg_class p ON p.oid = i.inhparent + ) sub + GROUP BY table_name + ORDER BY SUM(dead_tuples) DESC"# + ) + .fetch_all(db) + .await?; + + Ok(rows + .into_iter() + .map(|r| { + let total = r.live_tuples + r.dead_tuples; + let dead_ratio = if total > 0 { + r.dead_tuples as f64 / total as f64 + } else { + 0.0 + }; + let status = if dead_ratio < 0.1 { + HealthLevel::Green + } else if dead_ratio < 0.3 { + HealthLevel::Yellow + } else { + HealthLevel::Red + }; + TableMaintenanceInfo { + table_name: r.table_name, + live_tuples: r.live_tuples, + dead_tuples: r.dead_tuples, + dead_ratio, + last_autovacuum: r.last_autovacuum.map(|t| t.naive_utc()), + last_autoanalyze: r.last_autoanalyze.map(|t| t.naive_utc()), + status, + } + }) + .collect()) +} + +async fn fetch_slow_queries(db: &DB) -> windmill_common::error::Result> { + let ext_exists: bool = sqlx::query_scalar!( + r#"SELECT EXISTS(SELECT 1 FROM pg_extension WHERE extname = 'pg_stat_statements') as "exists!""# + ) + .fetch_one(db) + .await?; + + if !ext_exists { + return Ok(Some(SlowQueriesInfo { + queries: vec![], + message: Some( + "pg_stat_statements extension is not installed. Enable it for slow query insights." + .to_string(), + ), + })); + } + + // Use raw query since pg_stat_statements may not exist at compile time + let rows: Vec = sqlx::query_as::<_, (String, i64, f64, f64)>( + r#"SELECT + LEFT(query, 200), + calls::bigint, + total_exec_time::float8, + mean_exec_time::float8 + FROM pg_stat_statements + WHERE query NOT LIKE '%pg_stat_statements%' + ORDER BY mean_exec_time DESC + LIMIT 10"#, + ) + .fetch_all(db) + .await? + .into_iter() + .map( + |(query, calls, total_exec_time_ms, mean_exec_time_ms)| SlowQueryRow { + query, + calls, + total_exec_time_ms, + mean_exec_time_ms, + }, + ) + .collect(); + + Ok(Some(SlowQueriesInfo { queries: rows, message: None })) +} + +async fn fetch_datatables(db: &DB) -> windmill_common::error::Result> { + // Find instance-type datatables from workspace_settings + let rows = sqlx::query!( + r#"SELECT + ws.workspace_id as "workspace_id!", + dt.key as "name!", + dt.value->>'table_name' as "table_name" + FROM workspace_settings ws, + jsonb_each(ws.datatable) dt + WHERE dt.value->>'resource_type' = 'instance' + AND dt.value->>'table_name' IS NOT NULL"# + ) + .fetch_all(db) + .await?; + + let table_names: Vec = rows.iter().filter_map(|r| r.table_name.clone()).collect(); + + if table_names.is_empty() { + return Ok(vec![]); + } + + // Batch lookup: single query for all table sizes + let size_rows = sqlx::query!( + r#"SELECT + c.relname as "table_name!", + pg_total_relation_size(c.oid) as "size_bytes!", + pg_size_pretty(pg_total_relation_size(c.oid)) as "size_pretty!", + COALESCE(c.reltuples, 0) as "estimated_rows!" + FROM pg_class c + JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = 'public' AND c.relname = ANY($1)"#, + &table_names + ) + .fetch_all(db) + .await?; + + let size_map: std::collections::HashMap = size_rows + .into_iter() + .map(|s| (s.table_name.clone(), s)) + .collect(); + + let mut result = Vec::new(); + for row in rows { + let table_name = match &row.table_name { + Some(t) => t.clone(), + None => continue, + }; + if let Some(s) = size_map.get(&table_name) { + result.push(DatatableInfo { + workspace_id: row.workspace_id, + name: row.name, + table_name, + size_bytes: s.size_bytes, + size_pretty: s.size_pretty.clone(), + estimated_rows: s.estimated_rows as f64, + }); + } + } + + // Sort by size descending + result.sort_by(|a, b| b.size_bytes.cmp(&a.size_bytes)); + Ok(result) +} diff --git a/backend/windmill-api/src/drafts.rs b/backend/windmill-api/src/drafts.rs index 41e8d4709d..39a68d8f9a 100644 --- a/backend/windmill-api/src/drafts.rs +++ b/backend/windmill-api/src/drafts.rs @@ -23,7 +23,7 @@ use windmill_common::{db::UserDB, error::Result, utils::StripPath}; pub fn workspaced_service() -> Router { Router::new() .route("/create", post(create_draft)) - .route("/delete/:kind/*path", delete(delete_draft)) + .route("/delete/{kind}/{*path}", delete(delete_draft)) } #[derive(sqlx::Type, Serialize, Deserialize, Debug, PartialEq, Clone)] diff --git a/backend/windmill-api/src/flows.rs b/backend/windmill-api/src/flows.rs index c6a09a9e6a..15c9262967 100644 --- a/backend/windmill-api/src/flows.rs +++ b/backend/windmill-api/src/flows.rs @@ -21,7 +21,7 @@ use windmill_common::{error::JsonResult, utils::StripPath, DB}; /// that depends on windmill-api internals. pub fn workspaced_service() -> Router { windmill_api_flows::flows::workspaced_service() - .route("/get_triggers_count/*path", get(get_triggers_count)) + .route("/get_triggers_count/{*path}", get(get_triggers_count)) } async fn get_triggers_count( diff --git a/backend/windmill-api/src/google.rs b/backend/windmill-api/src/google.rs index dd2a47bca3..fb67f18931 100644 --- a/backend/windmill-api/src/google.rs +++ b/backend/windmill-api/src/google.rs @@ -111,17 +111,16 @@ pub async fn handle_google_ai_chat( let (contents, system_instruction) = openai_messages_to_gemini(&request.messages); - let generation_config = - if request.temperature.is_some() || request.max_tokens.is_some() { - Some(GeminiGenerationConfig { - temperature: request.temperature, - max_output_tokens: request.max_tokens, - response_mime_type: None, - response_schema: None, - }) - } else { - None - }; + let generation_config = if request.temperature.is_some() || request.max_tokens.is_some() { + Some(GeminiGenerationConfig { + temperature: request.temperature, + max_output_tokens: request.max_tokens, + response_mime_type: None, + response_schema: None, + }) + } else { + None + }; let gemini_tools = request.tools.as_ref().map(|tools| { let declarations: Vec = tools @@ -136,10 +135,7 @@ pub async fn handle_google_ai_chat( } }) .collect(); - vec![GeminiTool { - function_declarations: Some(declarations), - google_search: None, - }] + vec![GeminiTool { function_declarations: Some(declarations), google_search: None }] }); let gemini_request = GeminiTextRequest { @@ -184,9 +180,10 @@ async fn handle_streaming( .body(request_body); let request = set_auth(request, api_key, is_vertex); - let response = request.send().await.map_err(|e| { - Error::internal_err(format!("Failed to send request to Gemini API: {}", e)) - })?; + let response = request + .send() + .await + .map_err(|e| Error::internal_err(format!("Failed to send request to Gemini API: {}", e)))?; if let Err(e) = response.error_for_status_ref() { let status = e.status().map(|s| s.to_string()).unwrap_or_default(); @@ -273,9 +270,10 @@ pub async fn handle_google_ai_models( let request = HTTP_CLIENT.get(&endpoint); let request = set_auth(request, api_key, is_vertex); - let response = request.send().await.map_err(|e| { - Error::internal_err(format!("Failed to fetch Gemini models: {}", e)) - })?; + let response = request + .send() + .await + .map_err(|e| Error::internal_err(format!("Failed to fetch Gemini models: {}", e)))?; if let Err(e) = response.error_for_status_ref() { let status = e.status().map(|s| s.to_string()).unwrap_or_default(); @@ -327,9 +325,10 @@ async fn handle_non_streaming( .body(request_body); let request = set_auth(request, api_key, is_vertex); - let response = request.send().await.map_err(|e| { - Error::internal_err(format!("Failed to send request to Gemini API: {}", e)) - })?; + let response = request + .send() + .await + .map_err(|e| Error::internal_err(format!("Failed to send request to Gemini API: {}", e)))?; if let Err(e) = response.error_for_status_ref() { let status = e.status().map(|s| s.to_string()).unwrap_or_default(); @@ -337,9 +336,10 @@ async fn handle_non_streaming( return Err(Error::AIError(format!("{}: {}", status, body))); } - let body = response.bytes().await.map_err(|e| { - Error::internal_err(format!("Failed to read Gemini response body: {}", e)) - })?; + let body = response + .bytes() + .await + .map_err(|e| Error::internal_err(format!("Failed to read Gemini response body: {}", e)))?; let parsed = parse_gemini_response(&body)?; let openai_response = gemini_response_to_openai(&parsed, model); diff --git a/backend/windmill-api/src/group_history.rs b/backend/windmill-api/src/group_history.rs index 0c2c84038d..73162345bc 100644 --- a/backend/windmill-api/src/group_history.rs +++ b/backend/windmill-api/src/group_history.rs @@ -22,7 +22,7 @@ use serde::Serialize; use sqlx::FromRow; pub fn workspaced_service() -> Router { - Router::new().route("/get/:name", get(get_group_permission_history)) + Router::new().route("/get/{name}", get(get_group_permission_history)) } #[derive(Serialize, FromRow)] diff --git a/backend/windmill-api/src/health.rs b/backend/windmill-api/src/health.rs index c31b104429..fab9979c12 100644 --- a/backend/windmill-api/src/health.rs +++ b/backend/windmill-api/src/health.rs @@ -231,7 +231,7 @@ async fn check_database_with_latency(db: &DB) -> DatabaseCheckResult { DatabaseCheckResult { healthy, latency_ms } } -fn get_pool_stats(db: &DB) -> PoolStats { +pub(crate) fn get_pool_stats(db: &DB) -> PoolStats { PoolStats { size: db.size(), idle: db.num_idle() as u32, diff --git a/backend/windmill-api/src/jobs.rs b/backend/windmill-api/src/jobs.rs index 7f9a3f36db..8dcbd2a7f4 100644 --- a/backend/windmill-api/src/jobs.rs +++ b/backend/windmill-api/src/jobs.rs @@ -103,7 +103,7 @@ use windmill_common::{ cache, db::UserDB, error::{self, to_anyhow, Error}, - flow_status::{Approval, FlowStatus, FlowStatusModule}, + flow_status::{Approval, ApprovalConditions, FlowStatus, FlowStatusModule}, flows::{add_virtual_items_if_necessary, resolve_maybe_value, FlowValue}, jobs::{script_path_to_payload, CompletedJob, JobKind, JobPayload, QueuedJob, RawCode}, oauth2::HmacSha256, @@ -134,14 +134,14 @@ pub fn workspaced_service() -> Router { Router::new() .route( - "/run/f/*script_path", + "/run/f/{*script_path}", post(run_flow_by_path) .head(|| async { "" }) .layer(cors.clone()) .layer(ce_headers.clone()), ) .route( - "/run/fv/:version", + "/run/fv/{version}", post(run_flow_by_version) .head(|| async { "" }) .layer(cors.clone()) @@ -155,25 +155,25 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run/workflow_as_code/:job_id/:entrypoint", + "/run/workflow_as_code/{job_id}/{entrypoint}", post(run_workflow_as_code) .head(|| async { "" }) .layer(cors.clone()) .layer(ce_headers.clone()), ) .route( - "/restart/f/:job_id", + "/restart/f/{job_id}", post(restart_flow).head(|| async { "" }).layer(cors.clone()), ) .route( - "/run/p/*script_path", + "/run/p/{*script_path}", post(run_script_by_path) .head(|| async { "" }) .layer(cors.clone()) .layer(ce_headers.clone()), ) .route( - "/run_wait_result/p/*script_path", + "/run_wait_result/p/{*script_path}", post(run_wait_result_script_by_path) .get(run_wait_result_job_by_path_get) .head(|| async { "" }) @@ -181,14 +181,14 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run_wait_result/h/:hash", + "/run_wait_result/h/{hash}", post(run_wait_result_script_by_hash) .head(|| async { "" }) .layer(cors.clone()) .layer(ce_headers.clone()), ) .route( - "/run_wait_result/f/*script_path", + "/run_wait_result/f/{*script_path}", post(run_wait_result_flow_by_path) .get(run_wait_result_flow_by_path_get) .head(|| async { "" }) @@ -196,7 +196,7 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run_wait_result/fv/:version", + "/run_wait_result/fv/{version}", post(run_wait_result_flow_by_version) .get(run_wait_result_flow_by_version_get) .head(|| async { "" }) @@ -204,7 +204,7 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run_and_stream/f/*script_path", + "/run_and_stream/f/{*script_path}", get(stream_flow_by_path) .post(stream_flow_by_path) .head(|| async { "" }) @@ -212,7 +212,7 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run_and_stream/fv/:version", + "/run_and_stream/fv/{version}", get(stream_flow_by_version) .post(stream_flow_by_version) .head(|| async { "" }) @@ -220,7 +220,7 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run_and_stream/p/*script_path", + "/run_and_stream/p/{*script_path}", get(stream_script_by_path) .post(stream_script_by_path) .head(|| async { "" }) @@ -228,7 +228,7 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run_and_stream/h/:hash", + "/run_and_stream/h/{hash}", get(stream_script_by_hash) .post(stream_script_by_hash) .head(|| async { "" }) @@ -236,7 +236,7 @@ pub fn workspaced_service() -> Router { .layer(ce_headers.clone()), ) .route( - "/run/h/:hash", + "/run/h/{hash}", post(run_job_by_hash) .head(|| async { "" }) .layer(cors.clone()) @@ -245,10 +245,10 @@ pub fn workspaced_service() -> Router { .route("/run/preview", post(run_preview_script)) .route("/run_inline/preview", post(run_inline_preview_script)) .route( - "/run_inline/p/*script_path", + "/run_inline/p/{*script_path}", post(run_inline_script_by_path), ) - .route("/run_inline/h/:hash", post(run_inline_script_by_hash)) + .route("/run_inline/h/{hash}", post(run_inline_script_by_hash)) .route( "/run_wait_result/preview", post(run_wait_result_preview_script), @@ -257,7 +257,7 @@ pub fn workspaced_service() -> Router { "/run/preview_bundle", post(run_bundle_preview_script).layer(axum::extract::DefaultBodyLimit::disable()), ) - .route("/add_batch_jobs/:n", post(add_batch_jobs)) + .route("/add_batch_jobs/{n}", post(add_batch_jobs)) .route("/run/preview_flow", post(run_preview_flow_job)) .route( "/run_wait_result/preview_flow", @@ -280,8 +280,8 @@ pub fn workspaced_service() -> Router { ) .route("/queue/count", get(count_queue_jobs)) .route("/queue/list_filtered_uuids", get(list_filtered_uuids)) - .route("/queue/position/:timestamp", get(get_queue_position)) - .route("/queue/scheduled_for/:id", get(get_scheduled_for)) + .route("/queue/position/{timestamp}", get(get_queue_position)) + .route("/queue/scheduled_for/{id}", get(get_scheduled_for)) .route("/queue/cancel_selection", post(cancel_selection)) .route("/completed/count", get(count_completed_jobs)) .route("/completed/count_jobs", get(count_completed_jobs_detail)) @@ -299,49 +299,49 @@ pub fn workspaced_service() -> Router { ) .route("/delete", post(crate::jobs_export::delete_jobs)) .route( - "/completed/get/:id", + "/completed/get/{id}", get(get_completed_job).layer(cors.clone()), ) .route( - "/completed/get_result/:id", + "/completed/get_result/{id}", get(get_completed_job_result).layer(cors.clone()), ) .route( - "/completed/get_result_maybe/:id", + "/completed/get_result_maybe/{id}", get(get_completed_job_result_maybe).layer(cors.clone()), ) .route( - "/completed/get_timing/:id", + "/completed/get_timing/{id}", get(get_completed_job_timing).layer(cors.clone()), ) .route( - "/completed/delete/:id", + "/completed/delete/{id}", post(delete_completed_job).layer(cors.clone()), ) .route( - "/flow/resume/:id", + "/flow/resume/{id}", post(resume_suspended_flow_as_owner).layer(cors.clone()), ) .route( - "/job_signature/:job_id/:resume_id", + "/job_signature/{job_id}/{resume_id}", get(create_job_signature).layer(cors.clone()), ) .route( - "/flow/user_states/:job_id/:key", + "/flow/user_states/{job_id}/{key}", get(get_flow_user_state) .post(set_flow_user_state) .layer(cors.clone()), ) .route( - "/resume_urls/:job_id/:resume_id", + "/resume_urls/{job_id}/{resume_id}", get(get_resume_urls).layer(cors.clone()), ) .route( - "/result_by_id/:job_id/:node_id", + "/result_by_id/{job_id}/{node_id}", get(get_result_by_id).layer(cors.clone()), ) .route( - "/flow_env_by_flow_job_id/:flow_job_id/:var_name", + "/flow_env_by_flow_job_id/{flow_job_id}/{var_name}", get(get_flow_env_by_flow_job_id).layer(cors.clone()), ) .route("/run/dependencies", post(run_dependencies_job)) @@ -350,57 +350,59 @@ pub fn workspaced_service() -> Router { "/send_email_with_instance_smtp", post(send_email_with_instance_smtp), ) - .route("/get_otel_traces/:id", get(get_otel_traces)) + .route("/get_otel_traces/{id}", get(get_otel_traces)) } pub fn workspace_unauthed_service() -> Router { Router::new() .route( - "/resume/:job_id/:resume_id/:secret", + "/resume/{job_id}/{resume_id}/{secret}", get(resume_suspended_job), ) .route( - "/resume/:job_id/:resume_id/:secret", + "/resume/{job_id}/{resume_id}/{secret}", post(resume_suspended_job), ) .route( - "/cancel/:job_id/:resume_id/:secret", + "/cancel/{job_id}/{resume_id}/{secret}", get(cancel_suspended_job), ) .route( - "/cancel/:job_id/:resume_id/:secret", + "/cancel/{job_id}/{resume_id}/{secret}", post(cancel_suspended_job), ) .route( - "/get_flow/:job_id/:resume_id/:secret", + "/get_flow/{job_id}/{resume_id}/{secret}", get(get_suspended_job_flow), ) - .route("/get_root_job_id/:id", get(get_root_job)) - .route("/get/:id", get(get_job)) - .route("/get_logs/:id", get(get_job_logs)) + .route("/get_root_job_id/{id}", get(get_root_job)) + .route("/get/{id}", get(get_job)) + .route("/get_logs/{id}", get(get_job_logs)) .route( - "/get_completed_logs_tail/:id", + "/get_completed_logs_tail/{id}", get(get_completed_job_logs_tail), ) - .route("/get_args/:id", get(get_args)) + .route("/get_args/{id}", get(get_args)) .route("/queue/get_started_at_by_ids", post(get_started_at_by_ids)) - .route("/get_flow_debug_info/:id", get(get_flow_job_debug_info)) - .route("/completed/get/:id", get(get_completed_job)) - .route("/completed/get_result/:id", get(get_completed_job_result)) + .route("/get_flow_debug_info/{id}", get(get_flow_job_debug_info)) + .route("/completed/get/{id}", get(get_completed_job)) + .route("/completed/get_result/{id}", get(get_completed_job_result)) .route( - "/completed/get_result_maybe/:id", + "/completed/get_result_maybe/{id}", get(get_completed_job_result_maybe), ) - .route("/completed/get_timing/:id", get(get_completed_job_timing)) - .route("/getupdate/:id", get(get_job_update)) - .route("/getupdate_sse/:id", get(get_job_update_sse)) - .route("/get_log_file/*file_path", get(get_log_file)) - .route("/queue/cancel/:id", post(cancel_job_api)) + .route("/completed/get_timing/{id}", get(get_completed_job_timing)) + .route("/getupdate/{id}", get(get_job_update)) + .route("/getupdate_sse/{id}", get(get_job_update_sse)) + .route("/get_log_file/{*file_path}", get(get_log_file)) + .route("/queue/cancel/{id}", post(cancel_job_api)) .route( - "/queue/cancel_persistent/*script_path", + "/queue/cancel_persistent/{*script_path}", post(cancel_persistent_script_api), ) - .route("/queue/force_cancel/:id", post(force_cancel)) + .route("/queue/force_cancel/{id}", post(force_cancel)) + .route("/flow/resume_suspended/{job_id}", post(resume_suspended)) + .route("/flow/approval_info/{job_id}", get(get_approval_info)) } pub fn global_root_service() -> Router { @@ -1058,6 +1060,7 @@ impl<'a> GetQuery<'a> { Self { with_code: false, ..self } } + #[allow(dead_code)] fn without_flow(self) -> Self { Self { with_flow: false, ..self } } @@ -2027,10 +2030,7 @@ async fn count_completed_jobs_detail( if let Some(tags) = query.tags { sqlb.and_where_in( "v2_job.tag", - &tags - .split(",") - .map(|t| format!("'{}'", t)) - .collect::>(), + &tags.split(",").map(|t| quote(t)).collect::>(), ); } @@ -2179,12 +2179,24 @@ async fn list_jobs( pub async fn resume_suspended_flow_as_owner( authed: ApiAuthed, Extension(db): Extension, - Path((_w_id, flow_id)): Path<(String, Uuid)>, + Path((w_id, flow_id)): Path<(String, Uuid)>, QueryOrBody(value): QueryOrBody, ) -> error::Result { let mut tx = db.begin().await?; - let (flow, job_id) = get_suspended_flow_info(flow_id, &mut tx).await?; + let (flow, job_id, is_wac) = get_suspended_flow_info(flow_id, &mut tx).await?; + + // Verify the job belongs to this workspace + let job_workspace: Option = + sqlx::query_scalar("SELECT workspace_id FROM v2_job WHERE id = $1") + .bind(&flow.id) + .fetch_optional(&mut *tx) + .await?; + if job_workspace.as_deref() != Some(w_id.as_str()) { + return Err(Error::NotFound( + "Job not found in this workspace".to_string(), + )); + } let flow_path = flow.script_path.as_deref().unwrap_or_else(|| ""); require_owner_of_path(&authed, flow_path)?; @@ -2192,10 +2204,17 @@ pub async fn resume_suspended_flow_as_owner( // Check approval conditions (self-approval, required groups, etc.) if let Some(ref flow_status_value) = flow.flow_status { - if let Ok(flow_status) = serde_json::from_value::(flow_status_value.clone()) { - let trigger_email = flow.email.as_deref().unwrap_or(""); - conditionally_require_authed_user(Some(authed.clone()), flow_status, trigger_email)?; - } + let trigger_email = flow.email.as_deref().unwrap_or(""); + let ac = serde_json::from_value::(flow_status_value.clone()) + .ok() + .and_then(|fs| fs.approval_conditions) + .or_else(|| { + // WAC flows store approval_conditions directly in flow_status JSONB + flow_status_value + .get("approval_conditions") + .and_then(|v| serde_json::from_value::(v.clone()).ok()) + }); + conditionally_require_authed_user(Some(authed.clone()), ac, trigger_email)?; } let value = value.unwrap_or(serde_json::Value::Null); @@ -2211,12 +2230,454 @@ pub async fn resume_suspended_flow_as_owner( ) .await?; - resume_immediately_if_relevant(flow, job_id, &mut tx).await?; + if is_wac { + // WAC: directly decrement suspend counter + if flow.suspend > 0 { + sqlx::query!( + "UPDATE v2_job_queue SET suspend = GREATEST(suspend - 1, 0) WHERE id = $1", + flow.id, + ) + .execute(&mut *tx) + .await?; + } + } else { + resume_immediately_if_relevant(flow, job_id, &mut tx).await?; + } tx.commit().await?; Ok(StatusCode::CREATED) } +// --- New approval system endpoints --- + +use windmill_common::variables::generate_approval_token; + +/// Verify an approval token against the workspace key + job_id. +async fn validate_approval_token( + db: &DB, + token: &str, + job_id: Uuid, + workspace_id: &str, +) -> error::Result<()> { + let expected = generate_approval_token(workspace_id, job_id, db).await?; + if token != expected { + return Err(Error::NotAuthorized("Invalid approval token".to_string())); + } + Ok(()) +} + +#[derive(Deserialize)] +struct ResumeSuspendedBody { + payload: Option, + approval_token: Option, + approved: Option, +} + +async fn resume_suspended( + OptAuthed(opt_authed): OptAuthed, + Extension(db): Extension, + Path((w_id, job_id)): Path<(String, Uuid)>, + Json(body): Json, +) -> error::Result { + let approved = body.approved.unwrap_or(true); + let value = body.payload.unwrap_or(serde_json::Value::Null); + + // Determine if we have a valid authed user or token + let has_token = if let Some(ref token) = body.approval_token { + validate_approval_token(&db, token, job_id, &w_id) + .await + .is_ok() + } else { + false + }; + + if opt_authed.is_none() && !has_token { + return Err(Error::NotAuthorized( + "Must be logged in or provide a valid approval token".to_string(), + )); + } + + let mut tx = db.begin().await?; + + // Resolve the suspended flow (works for both WAC and classic flows) + let (flow, resume_job_id, is_wac) = get_suspended_flow_info(job_id, &mut tx).await?; + + // Verify the job belongs to this workspace + let job_workspace: Option = + sqlx::query_scalar("SELECT workspace_id FROM v2_job WHERE id = $1") + .bind(&flow.id) + .fetch_optional(&mut *tx) + .await?; + if job_workspace.as_deref() != Some(w_id.as_str()) { + return Err(Error::NotFound( + "Job not found in this workspace".to_string(), + )); + } + + // Check approval conditions + let approval_conditions = if is_wac { + flow.flow_status + .as_ref() + .and_then(|v| v.get("approval_conditions")) + .and_then(|v| serde_json::from_value::(v.clone()).ok()) + } else { + flow.flow_status + .as_ref() + .and_then(|v| serde_json::from_value::(v.clone()).ok()) + .and_then(|fs| fs.approval_conditions) + }; + + if let Some(ref ac) = approval_conditions { + if ac.user_auth_required && opt_authed.is_none() { + return Err(Error::NotAuthorized( + "This approval requires a logged-in user. Please sign in.".to_string(), + )); + } + } + + // If logged in, check authorization rules + if let Some(ref authed) = opt_authed { + let is_admin = authed.is_admin; + let is_owner = flow + .script_path + .as_deref() + .map(|p| require_owner_of_path(authed, p).is_ok()) + .unwrap_or(false); + + if !is_admin && !is_owner { + let trigger_email = flow.email.as_deref().unwrap_or(""); + conditionally_require_authed_user( + Some(authed.clone()), + approval_conditions.clone(), + trigger_email, + )?; + } + } else if !has_token { + return Err(Error::NotAuthorized( + "Must be logged in or provide a valid approval token".to_string(), + )); + } + + // Generate a unique resume_id + let resume_id: u32 = rand::random(); + + // Check for duplicate + let exists: bool = sqlx::query_scalar("SELECT EXISTS (SELECT 1 FROM resume_job WHERE id = $1)") + .bind(Uuid::from_u128(resume_job_id.as_u128() ^ resume_id as u128)) + .fetch_one(&mut *tx) + .await?; + + if exists { + return Err(Error::BadRequest("Resume request already sent".to_string())); + } + + let approver_value = opt_authed.as_ref().map(|a| a.username.clone()); + + insert_resume_job( + resume_id, + resume_job_id, + &flow, + value, + approver_value.clone(), + approved, + &mut tx, + ) + .await?; + + if !approved { + sqlx::query("UPDATE v2_job_queue SET suspend = 0 WHERE id = $1") + .bind(&flow.id) + .execute(&mut *tx) + .await?; + } else if is_wac { + if flow.suspend > 0 { + sqlx::query("UPDATE v2_job_queue SET suspend = GREATEST(suspend - 1, 0) WHERE id = $1") + .bind(&flow.id) + .execute(&mut *tx) + .await?; + } + } else { + resume_immediately_if_relevant(flow, resume_job_id, &mut tx).await?; + } + + let approver = approver_value.unwrap_or_else(|| "anonymous".to_string()); + let audit_author = if let Some(ref authed) = opt_authed { + AuditAuthor::from(authed) + } else { + AuditAuthor { + email: approver.clone(), + username: approver.clone(), + username_override: None, + token_prefix: None, + } + }; + + audit_log( + &mut *tx, + &audit_author, + "jobs.suspend_resume", + ActionKind::Update, + &w_id, + Some( + &serde_json::json!({ + "approved": approved, + "job_id": job_id, + "details": if approved { + format!("Approved by {}", &approver) + } else { + format!("Cancelled by {}", &approver) + } + }) + .to_string(), + ), + None, + ) + .await?; + + tx.commit().await?; + Ok(StatusCode::CREATED) +} + +#[derive(Deserialize)] +struct ApprovalInfoQuery { + token: Option, +} + +#[derive(Serialize)] +struct ApprovalInfo { + flow_id: Uuid, + #[serde(skip_serializing_if = "Option::is_none")] + form_schema: Option, + #[serde(skip_serializing_if = "Option::is_none")] + description: Option, + #[serde(skip_serializing_if = "Option::is_none")] + default_args: Option, + #[serde(skip_serializing_if = "Option::is_none")] + enums: Option, + #[serde(skip_serializing_if = "Option::is_none")] + approval_conditions: Option, + can_approve: bool, + user_auth_required: bool, + #[serde(skip_serializing_if = "Option::is_none")] + hide_cancel: Option, + approvers: Vec, +} + +async fn get_approval_info( + OptAuthed(opt_authed): OptAuthed, + Extension(db): Extension, + Path((w_id, job_id)): Path<(String, Uuid)>, + Query(query): Query, +) -> error::Result> { + // Validate access: either logged in or valid token + let has_token = if let Some(ref token) = query.token { + validate_approval_token(&db, token, job_id, &w_id) + .await + .is_ok() + } else { + false + }; + + if opt_authed.is_none() && !has_token { + return Err(Error::NotAuthorized( + "Must be logged in or provide a valid approval token".to_string(), + )); + } + + // Fetch job info + #[derive(sqlx::FromRow)] + struct ApprovalJobRow { + id: Uuid, + script_path: Option, + email: String, + flow_status: Option, + workflow_as_code_status: Option, + } + let row = sqlx::query_as::<_, ApprovalJobRow>( + "SELECT j.id, j.runnable_path as script_path, j.permissioned_as_email as email, + s.flow_status, s.workflow_as_code_status + FROM v2_job j + LEFT JOIN v2_job_status s ON s.id = j.id + WHERE j.id = $1 AND j.workspace_id = $2", + ) + .bind(&job_id) + .bind(&w_id) + .fetch_optional(&db) + .await? + .ok_or_else(|| Error::NotFound(format!("Job {job_id} not found")))?; + + let is_wac = row.workflow_as_code_status.is_some(); + + // Extract approval info based on WAC vs classic flow + let (form_schema, description, default_args, enums, approval_conditions, hide_cancel) = + if is_wac { + let approval_meta = row + .workflow_as_code_status + .as_ref() + .and_then(|v| v.get("_approval")); + let form = approval_meta.and_then(|m| m.get("form").cloned()); + let default_args = approval_meta.and_then(|m| m.get("default_args").cloned()); + let enums = approval_meta.and_then(|m| m.get("enums").cloned()); + let description = approval_meta.and_then(|m| m.get("description").cloned()); + let ac = row + .flow_status + .as_ref() + .and_then(|v| v.get("approval_conditions")) + .and_then(|v| serde_json::from_value::(v.clone()).ok()); + (form, description, default_args, enums, ac, None) + } else { + let fs = row + .flow_status + .as_ref() + .and_then(|v| serde_json::from_value::(v.clone()).ok()); + let ac = fs.as_ref().and_then(|s| s.approval_conditions.clone()); + + // For classic flows, form/description come from the flow definition and step result + let approval_step = fs.as_ref().map(|s| (s.step as usize).saturating_sub(1)); + + // Fetch flow definition to get suspend settings (form schema, hide_cancel). + // Try raw_flow on the job first, fall back to flow_version for deployed flows, + // then flow_node for graph-based branch/loop sub-flows. + let raw_flow: Option = { + let from_job: Option = sqlx::query_scalar( + "SELECT raw_flow FROM v2_job WHERE id = $1 AND workspace_id = $2", + ) + .bind(&job_id) + .bind(&w_id) + .fetch_optional(&db) + .await? + .flatten(); + + if let Some(v) = from_job { + serde_json::from_value(v).ok() + } else { + // Deployed flow: fetch from flow_version using runnable_id + let from_version: Option = sqlx::query_scalar( + "SELECT fv.value FROM v2_job j JOIN flow_version fv ON fv.id = j.runnable_id \ + WHERE j.id = $1 AND j.workspace_id = $2", + ) + .bind(&job_id) + .bind(&w_id) + .fetch_optional(&db) + .await? + .flatten(); + if let Some(v) = from_version { + serde_json::from_value(v).ok() + } else { + // FlowNode sub-flow (graph-based branch/loop): raw_flow is not stored + // in v2_job for newer versions, fetch from flow_node table + let from_node: Option = sqlx::query_scalar( + "SELECT fn.flow FROM v2_job j \ + JOIN flow_node fn ON fn.id = j.runnable_id \ + WHERE j.id = $1 AND j.workspace_id = $2", + ) + .bind(&job_id) + .bind(&w_id) + .fetch_optional(&db) + .await? + .flatten(); + from_node.and_then(|v| serde_json::from_value(v).ok()) + } + } + }; + + let suspend_module = raw_flow + .as_ref() + .and_then(|rf| approval_step.and_then(|s| rf.modules.get(s))); + let suspend_settings = suspend_module.and_then(|m| m.suspend.as_ref()); + + let form = suspend_settings + .and_then(|s| s.resume_form.as_ref()) + .map(|rf| serde_json::json!(rf)); + let hc = suspend_settings.map(|s| s.hide_cancel.unwrap_or(false)); + + // Fetch description, default_args, and enums from the step's completed job result + let step_job_id = fs + .as_ref() + .and_then(|s| approval_step.and_then(|step| s.modules.get(step))) + .and_then(|m| m.job()); + let (desc, default_args, enums) = if let Some(sjid) = step_job_id { + let result: Option = sqlx::query_scalar( + "SELECT result FROM v2_job_completed WHERE id = $1 AND workspace_id = $2", + ) + .bind(sjid) + .bind(&w_id) + .fetch_optional(&db) + .await? + .flatten(); + let desc = result.as_ref().and_then(|r| r.get("description").cloned()); + let da = result.as_ref().and_then(|r| r.get("default_args").cloned()); + let enums = result.as_ref().and_then(|r| r.get("enums").cloned()); + (desc, da, enums) + } else { + (None, None, None) + }; + + (form, desc, default_args, enums, ac, hc) + }; + + let user_auth_required = approval_conditions + .as_ref() + .map(|ac| ac.user_auth_required) + .unwrap_or(false); + + // Determine if current user can approve + let can_approve = if let Some(ref authed) = opt_authed { + if authed.is_admin { + true + } else { + let is_owner = row + .script_path + .as_deref() + .map(|p| require_owner_of_path(authed, p).is_ok()) + .unwrap_or(false); + if is_owner { + true + } else { + let trigger_email = row.email.as_str(); + conditionally_require_authed_user( + Some(authed.clone()), + approval_conditions.clone(), + trigger_email, + ) + .is_ok() + } + } + } else { + // Not logged in — can approve only if no auth required + !user_auth_required + }; + + // Get existing approvers + let approvers: Vec = sqlx::query_as::<_, (i32, Option)>( + "SELECT resume_id, approver FROM resume_job WHERE flow = $1", + ) + .bind(&job_id) + .fetch_all(&db) + .await? + .into_iter() + .map(|(rid, approver)| Approval { + resume_id: rid as u16, + approver: approver.unwrap_or_else(|| "anonymous".to_string()), + }) + .collect(); + + Ok(Json(ApprovalInfo { + flow_id: row.id, + form_schema, + description, + default_args, + enums, + approval_conditions, + can_approve, + user_auth_required, + hide_cancel, + approvers, + })) +} + +// --- End new approval system endpoints --- + pub async fn resume_suspended_job( authed: Option, opt_tokened: OptTokened, @@ -2258,26 +2719,8 @@ async fn resume_suspended_job_internal( // Get flow info - works for step-level, flow-level, and WAC approval let (flow_info, is_flow_level, is_wac) = get_flow_info_for_resume(job_id, &db).await?; - // For step-level resumes, verify user auth and flow status - // For flow-level resumes (pre-approvals), the flow might not be at a suspended step yet - // For WAC approvals, skip flow status checks (there is no flow) - if !is_flow_level && !is_wac { - let parent_flow = GetQuery::new() - .without_logs() - .without_code() - .without_flow() - .fetch(&db, &flow_info.id, &w_id) - .await?; - let flow_status = parent_flow - .flow_status() - .ok_or_else(|| anyhow::anyhow!("unable to find the flow status in the flow job"))?; - - let trigger_email = match &parent_flow { - Job::CompletedJob(job) => &job.email, - Job::QueuedJob(job) => &job.email, - }; - conditionally_require_authed_user(authed.clone(), flow_status, trigger_email)?; - } + // HMAC secret = full capability. Skip approval_conditions checks. + // Authorization rules are enforced by the new resume_suspended endpoint instead. let exists = sqlx::query_scalar!( r#" @@ -2543,7 +2986,7 @@ async fn get_flow_info_for_resume(job_id: Uuid, db: &DB) -> error::Result<(FlowI async fn get_suspended_flow_info<'c>( job_id: Uuid, tx: &mut Transaction<'c, Postgres>, -) -> error::Result<(FlowInfo, Uuid)> { +) -> error::Result<(FlowInfo, Uuid, bool)> { let flow = sqlx::query_as!( FlowInfo, r#" @@ -2556,7 +2999,9 @@ async fn get_suspended_flow_info<'c>( .fetch_optional(&mut **tx) .await? .ok_or_else(|| anyhow::anyhow!("parent flow job not found"))?; - let job_id = flow + + // Try to extract step job_id from FlowStatus modules (classic flow path) + let step_job_id = flow .flow_status .as_ref() .and_then(|v| serde_json::from_value::(v.clone()).ok()) @@ -2565,8 +3010,31 @@ async fn get_suspended_flow_info<'c>( _ => None, }); - if let Some(job_id) = job_id { - Ok((flow, job_id)) + if let Some(step_job_id) = step_job_id { + // Classic flow + Ok((flow, step_job_id, false)) + } else if flow.suspend > 0 { + // WAC approval: no FlowStatus modules, but the job is suspended + // The flow_status here comes from COALESCE(flow_status, workflow_as_code_status), + // so for WAC it may contain approval_conditions from flow_status column + // or the WAC checkpoint from workflow_as_code_status column. + // We need the approval_conditions which are in flow_status column. + // Re-fetch just flow_status (without COALESCE fallback) for the auth check. + let flow_status_only: Option = + sqlx::query_scalar("SELECT flow_status FROM v2_job_status WHERE id = $1") + .bind(&job_id) + .fetch_optional(&mut **tx) + .await? + .flatten(); + + let flow = FlowInfo { + id: flow.id, + flow_status: flow_status_only, + suspend: flow.suspend, + script_path: flow.script_path, + email: flow.email, + }; + Ok((flow, job_id, true)) } else { Err(anyhow::anyhow!("the flow is not in a suspended state anymore").into()) } @@ -2643,7 +3111,11 @@ pub async fn get_suspended_job_flow( Job::CompletedJob(job) => &job.email, Job::QueuedJob(job) => &job.email, }; - conditionally_require_authed_user(authed.clone(), flow_status.clone(), trigger_email)?; + conditionally_require_authed_user( + authed.clone(), + flow_status.approval_conditions.clone(), + trigger_email, + )?; let approvers_from_status = match flow_module_status { FlowStatusModule::Success { approvers, .. } => approvers.to_owned(), @@ -2684,16 +3156,25 @@ pub async fn get_suspended_job_flow( fn conditionally_require_authed_user( _authed: Option, - flow_status: FlowStatus, + approval_conditions_opt: Option, _trigger_email: &str, ) -> error::Result<()> { - let approval_conditions_opt = flow_status.approval_conditions; - if approval_conditions_opt.is_none() { return Ok(()); } let approval_conditions = approval_conditions_opt.unwrap(); + // Check self-approval independently of user_auth_required + if approval_conditions.self_approval_disabled { + if let Some(ref authed) = _authed { + if !authed.is_admin && authed.email.eq(_trigger_email) { + return Err(Error::PermissionDenied( + "Self-approval is disabled for this flow step".to_string(), + )); + } + } + } + if approval_conditions.user_auth_required { { #[cfg(not(feature = "enterprise"))] @@ -2711,13 +3192,6 @@ fn conditionally_require_authed_user( let authed = _authed.unwrap(); if !authed.is_admin { - if approval_conditions.self_approval_disabled && authed.email.eq(_trigger_email) - { - return Err(Error::PermissionDenied( - "Self-approval is disabled for this flow step".to_string(), - )); - } - if !approval_conditions.user_groups_required.is_empty() { #[cfg(feature = "enterprise")] { @@ -2863,11 +3337,18 @@ pub async fn get_resume_urls_internal( .map(|x| format!("?approver={}", encode(x))) .unwrap_or_else(String::new); + // Generate approval token for the new approval page URL. + // The token targets the parent flow/WAC job for proper resolution. + let approval_target_id = get_flow_id_for_job(&db, job_id) + .await + .unwrap_or(target_job_id); + let approval_token = generate_approval_token(&w_id, approval_target_id, &db).await?; + let base_url_str = BASE_URL.read().await.clone(); let base_url = base_url_str.as_str(); let res = ResumeUrls { approvalPage: format!( - "{base_url}/approve/{w_id}/{target_job_id}/{resume_id}/{signature}{approver_query}" + "{base_url}/approve/{w_id}/{approval_target_id}?token={approval_token}" ), cancel: build_resume_url( "cancel", diff --git a/backend/windmill-api/src/lib.rs b/backend/windmill-api/src/lib.rs index 565b01a4b4..fd356de67f 100644 --- a/backend/windmill-api/src/lib.rs +++ b/backend/windmill-api/src/lib.rs @@ -37,6 +37,7 @@ use axum::body::Body; use axum::extract::DefaultBodyLimit; use axum::http::HeaderValue; use axum::response::Response; +use axum::serve::ListenerExt; use axum::{middleware::from_extractor, routing::get, routing::post, Extension, Json, Router}; use db::DB; use tokio::task::JoinHandle; @@ -45,8 +46,8 @@ use windmill_common::global_settings::EMAIL_DOMAIN_SETTING; use windmill_common::worker::HUB_CACHE_DIR; use std::fs::DirBuilder; +use std::sync::Arc; use std::time::Duration; -use std::{net::SocketAddr, sync::Arc}; use tokio::sync::RwLock; use tower::ServiceBuilder; use tower_cookies::CookieManagerLayer; @@ -76,6 +77,7 @@ mod bedrock; mod capture; mod concurrency_groups; mod db; +mod db_health; mod google; mod drafts; @@ -165,6 +167,7 @@ pub mod teams_ee; mod teams_oss; mod token; mod tracing_init; +mod trash; pub mod triggers; mod users; #[cfg(feature = "private")] @@ -325,7 +328,7 @@ pub async fn run_server( db: DB, job_index_reader: Option, log_index_reader: Option, - addr: SocketAddr, + listener: tokio::net::TcpListener, mut killpill_rx: tokio::sync::broadcast::Receiver<()>, port_tx: tokio::sync::oneshot::Sender, server_mode: bool, @@ -376,6 +379,8 @@ pub async fn run_server( REQUEST_SIZE_LIMIT.read().await.clone(), )); + let request_size_limit = REQUEST_SIZE_LIMIT.read().await.clone(); + let cors = CorsLayer::new() .allow_methods([http::Method::GET, http::Method::POST, http::Method::DELETE]) .allow_headers([http::header::CONTENT_TYPE, http::header::AUTHORIZATION]) @@ -411,6 +416,9 @@ pub async fn run_server( auth_cache: auth_cache.clone(), base_internal_url: _base_internal_url.clone(), }); + let addr = listener + .local_addr() + .unwrap_or_else(|_| std::net::SocketAddr::from(([127, 0, 0, 1], 0))); if let Err(err) = smtp_server.start_listener_thread(addr).await { tracing::error!("Error starting SMTP server: {err:#}"); } @@ -451,9 +459,6 @@ pub async fn run_server( health::start_health_check_loop(db.clone(), killpill_rx.resubscribe()); } - let listener = tokio::net::TcpListener::bind(addr) - .await - .context("binding main windmill server")?; let port = listener.local_addr().map(|x| x.port()).unwrap_or(8000); let ip = listener .local_addr() @@ -526,11 +531,11 @@ pub async fn run_server( "/api", Router::new() .nest( - "/w/:workspace_id", + "/w/{workspace_id}", Router::new() // Reordered alphabetically .nest("/acls", granular_acls::workspaced_service()) - .nest("/apps", apps::workspaced_service()) + .nest("/apps", apps::workspaced_service(request_size_limit * 5)) .nest("/assets", windmill_api_assets::workspaced_service()) .nest("/audit", audit::workspaced_service()) .nest("/capture", capture::workspaced_service()) @@ -597,6 +602,7 @@ pub async fn run_server( .nest("/resources", resources::workspaced_service()) .nest("/schedules", windmill_api_schedule::workspaced_service()) .nest("/scripts", scripts::workspaced_service()) + .nest("/trash", trash::workspaced_service()) .nest( "/users", users::workspaced_service().layer(Extension(argon2.clone())), @@ -638,7 +644,8 @@ pub async fn run_server( .nest("/ai", ai::global_service()) .nest("/inkeep", inkeep_oss::global_service()) .nest("/indexer", indexer_oss::management_service()) - .nest("/mcp/w/:workspace_id/list_tools", mcp_list_tools_service) + .nest("/mcp/w/{workspace_id}/list_tools", mcp_list_tools_service) + .nest("/db_health", db_health::global_service()) .nest("/health/detailed", health::detailed_service()) .nest( "/saml", @@ -657,7 +664,7 @@ pub async fn run_server( .route_layer(from_extractor::()) // Workspace-scoped OAuth endpoints that don't require authentication // (authorize and token are called by MCP client before user is authenticated) - .nest("/w/:workspace_id/mcp/oauth/server", { + .nest("/w/{workspace_id}/mcp/oauth/server", { #[cfg(feature = "mcp")] { mcp::oauth_server::workspaced_unauthed_service() @@ -678,7 +685,7 @@ pub async fn run_server( }) .nest("/jobs", jobs::global_root_service()) .nest( - "/srch/w/:workspace_id/index", + "/srch/w/{workspace_id}/index", indexer_oss::workspaced_service(), ) .nest("/srch/index", indexer_oss::global_service()) @@ -708,19 +715,19 @@ pub async fn run_server( } }) .nest( - "/w/:workspace_id/apps_u", + "/w/{workspace_id}/apps_u", apps::unauthed_service() .layer(from_extractor::()) .layer(cors.clone()), ) .layer(from_extractor::()) - // Deprecated, here for backwards compatibility: user should use /mcp/w/:workspace_id/mcp instead + // Deprecated, here for backwards compatibility: user should use /mcp/w/{workspace_id}/mcp instead .nest( - "/mcp/w/:workspace_id/sse", + "/mcp/w/{workspace_id}/sse", mcp_router.clone().layer(cors.clone()), ) .nest( - "/mcp/w/:workspace_id/mcp", + "/mcp/w/{workspace_id}/mcp", mcp_router.clone().layer(cors.clone()), ) .nest("/mcp/gateway", gateway_mcp_router.layer(cors.clone())) @@ -743,7 +750,7 @@ pub async fn run_server( Router::new() } }) - .nest("/w/:workspace_id/agent_workers", { + .nest("/w/{workspace_id}/agent_workers", { #[cfg(feature = "agent_worker_server")] { agent_workers_router @@ -760,7 +767,7 @@ pub async fn run_server( } }) .nest( - "/w/:workspace_id/jobs_u", + "/w/{workspace_id}/jobs_u", jobs::workspace_unauthed_service().layer(cors.clone()), ) .route("/slack", post(slack_approvals::slack_app_callback_handler)) @@ -776,14 +783,14 @@ pub async fn run_server( } }) .route( - "/w/:workspace_id/jobs/slack_approval/:job_id", + "/w/{workspace_id}/jobs/slack_approval/{job_id}", get(slack_approvals::request_slack_approval), ) .route( - "/w/:workspace_id/jobs/teams_approval/:job_id", + "/w/{workspace_id}/jobs/teams_approval/{job_id}", get(teams_approvals_oss::request_teams_approval), ) - .nest("/w/:workspace_id/github_app", { + .nest("/w/{workspace_id}/github_app", { #[cfg(feature = "enterprise")] { git_sync_oss::workspaced_service() @@ -802,14 +809,14 @@ pub async fn run_server( Router::new() }) .nest( - "/w/:workspace_id/resources_u", + "/w/{workspace_id}/resources_u", public_service().layer(cors.clone()), ) .nest( - "/w/:workspace_id/capture_u", + "/w/{workspace_id}/capture_u", capture::workspaced_unauthed_service().layer(cors.clone()), ) - .nest("/w/:workspace_id/s3_proxy", { + .nest("/w/{workspace_id}/s3_proxy", { s3_proxy_oss::workspaced_unauthed_service() }) .nest( @@ -854,7 +861,7 @@ pub async fn run_server( Router::new() } }) - .nest("/gcp/w/:workspace_id", { + .nest("/gcp/w/{workspace_id}", { #[cfg(all( feature = "enterprise", feature = "gcp_trigger", @@ -881,10 +888,10 @@ pub async fn run_server( .route("/openapi.json", get(openapi_json)), ) // Clients must use workspace-scoped OAuth metadata at: - // /.well-known/oauth-authorization-server/api/w/:workspace_id/mcp/oauth/server + // /.well-known/oauth-authorization-server/api/w/{workspace_id}/mcp/oauth/server // This is discovered via /.well-known/oauth-protected-resource?workspace_id=... .route( - "/.well-known/oauth-authorization-server/api/w/:workspace_id/mcp/oauth/server", + "/.well-known/oauth-authorization-server/api/w/{workspace_id}/mcp/oauth/server", { #[cfg(feature = "mcp")] { @@ -896,9 +903,9 @@ pub async fn run_server( } }, ) - // RFC 9728 path-based discovery: /.well-known/oauth-protected-resource/api/mcp/w/:workspace_id/mcp + // RFC 9728 path-based discovery: /.well-known/oauth-protected-resource/api/mcp/w/{workspace_id}/mcp .route( - "/.well-known/oauth-protected-resource/api/mcp/w/:workspace_id/mcp", + "/.well-known/oauth-protected-resource/api/mcp/w/{workspace_id}/mcp", { #[cfg(feature = "mcp")] { @@ -974,7 +981,10 @@ pub async fn run_server( if let Some(name) = name.as_ref() { tracing::info!("server starting for name={name}"); } - let server = axum::serve(listener, app.into_make_service()).tcp_nodelay(!server_mode); + let listener = listener.tap_io(move |tcp_stream| { + let _ = tcp_stream.set_nodelay(!server_mode); + }); + let server = axum::serve(listener, app.into_make_service()); tracing::info!( instance = %*INSTANCE_NAME, diff --git a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs index 9425b6a01f..c9295ea8e9 100644 --- a/backend/windmill-api/src/mcp/auto_generated_endpoints.rs +++ b/backend/windmill-api/src/mcp/auto_generated_endpoints.rs @@ -221,6 +221,22 @@ pub fn all_tools() -> Vec { "type": "string", "description": "filter variables by path prefix" }, + "path": { + "type": "string", + "description": "exact path match filter" + }, + "description": { + "type": "string", + "description": "pattern match filter for description field (case-insensitive)" + }, + "value": { + "type": "string", + "description": "pattern match filter for non-secret variable values (case-insensitive)" + }, + "broad_filter": { + "type": "string", + "description": "broad search across multiple fields (case-insensitive substring match)" + }, "page": { "type": "integer", "description": "which page to return (start at 1, default 1)" @@ -405,6 +421,22 @@ pub fn all_tools() -> Vec { "path_start": { "type": "string", "description": "filter resources by path prefix" + }, + "path": { + "type": "string", + "description": "exact path match filter" + }, + "description": { + "type": "string", + "description": "pattern match filter for description field (case-insensitive)" + }, + "value": { + "type": "string", + "description": "JSONB subset match filter using base64 encoded JSON" + }, + "broad_filter": { + "type": "string", + "description": "broad search across multiple fields (case-insensitive substring match)" } }, "required": [] @@ -451,7 +483,7 @@ pub fn all_tools() -> Vec { }, "created_by": { "type": "string", - "description": "mask to filter exact matching user creator" + "description": "filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob')" }, "path_start": { "type": "string", @@ -562,7 +594,6 @@ pub fn all_tools() -> Vec { "required": [ "path", "summary", - "description", "content", "language" ] @@ -708,7 +739,7 @@ pub fn all_tools() -> Vec { }, "created_by": { "type": "string", - "description": "mask to filter exact matching user creator" + "description": "filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob')" }, "path_start": { "type": "string", @@ -1078,6 +1109,37 @@ pub fn all_tools() -> Vec { }, "lock": { "type": "string" + }, + "flow_path": { + "type": "string" + }, + "modules": { + "type": "object", + "nullable": true, + "description": "Additional script modules keyed by relative file path", + "additionalProperties": { + "type": "object", + "description": "An additional module file associated with a script", + "properties": { + "content": { + "type": "string", + "description": "The source code content of this module" + }, + "language": { + "type": "string", + "description": "Possible values: python3, deno, go, bash, powershell, postgresql, mysql, bigquery, snowflake, mssql, oracledb, graphql, nativets, bun, php, rust, ansible, csharp, nu, java, ruby, duckdb, bunnative" + }, + "lock": { + "type": "string", + "nullable": true, + "description": "Lock file content for this module's dependencies" + } + }, + "required": [ + "content", + "language" + ] + } } }, "required": [ @@ -1106,7 +1168,7 @@ pub fn all_tools() -> Vec { }, "created_by": { "type": "string", - "description": "mask to filter exact matching user creator" + "description": "filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob')" }, "parent_job": { "type": "string", @@ -1115,15 +1177,15 @@ pub fn all_tools() -> Vec { }, "worker": { "type": "string", - "description": "worker this job was ran on" + "description": "filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2')" }, "script_path_exact": { "type": "string", - "description": "mask to filter exact matching path" + "description": "filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2')" }, "script_path_start": { "type": "string", - "description": "mask to filter matching starting path" + "description": "filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2')" }, "schedule_path": { "type": "string", @@ -1131,11 +1193,11 @@ pub fn all_tools() -> Vec { }, "trigger_path": { "type": "string", - "description": "mask to filter by trigger path" + "description": "filter by trigger path. Supports comma-separated list (e.g. 'f/trigger1,f/trigger2') and negation by prefixing all values with '!' (e.g. '!f/trigger1,!f/trigger2')" }, "trigger_kind": { - "description": "trigger kind (schedule, http, websocket...). Possible values: webhook, default_email, email, schedule, http, websocket, postgres, kafka, nats, mqtt, sqs, gcp", - "type": "string" + "type": "string", + "description": "filter by trigger kind. Supports comma-separated list (e.g. 'schedule,webhook') and negation by prefixing all values with '!' (e.g. '!schedule,!webhook')" }, "script_hash": { "type": "string", @@ -1161,7 +1223,7 @@ pub fn all_tools() -> Vec { }, "job_kinds": { "type": "string", - "description": "filter on job kind (values 'preview', 'script', 'dependencies', 'flow') separated by," + "description": "filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies')" }, "suspended": { "type": "boolean", @@ -1185,7 +1247,7 @@ pub fn all_tools() -> Vec { }, "tag": { "type": "string", - "description": "filter on jobs with a given tag/worker group" + "description": "filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem')" }, "page": { "type": "integer", @@ -1223,15 +1285,15 @@ pub fn all_tools() -> Vec { "properties": { "created_by": { "type": "string", - "description": "mask to filter exact matching user creator" + "description": "filter by exact matching user creator. Supports comma-separated list (e.g. 'alice,bob') and negation by prefixing all values with '!' (e.g. '!alice,!bob')" }, "label": { "type": "string", - "description": "mask to filter exact matching job's label (job labels are completed jobs with as a result an object containing a string in the array at key 'wm_labels')" + "description": "filter by exact matching job label. Supports comma-separated list (e.g. 'deploy,release') and negation by prefixing all values with '!' (e.g. '!deploy,!release')" }, "worker": { "type": "string", - "description": "worker this job was ran on" + "description": "filter by worker this job ran on. Supports comma-separated list (e.g. 'worker-1,worker-2') and negation by prefixing all values with '!' (e.g. '!worker-1,!worker-2')" }, "parent_job": { "type": "string", @@ -1240,11 +1302,11 @@ pub fn all_tools() -> Vec { }, "script_path_exact": { "type": "string", - "description": "mask to filter exact matching path" + "description": "filter by exact matching script path. Supports comma-separated list (e.g. 'f/script1,f/script2') and negation by prefixing all values with '!' (e.g. '!f/script1,!f/script2')" }, "script_path_start": { "type": "string", - "description": "mask to filter matching starting path" + "description": "filter by script path prefix. Supports comma-separated list (e.g. 'f/folder1,f/folder2') and negation by prefixing all values with '!' (e.g. '!f/folder1,!f/folder2')" }, "schedule_path": { "type": "string", @@ -1304,7 +1366,7 @@ pub fn all_tools() -> Vec { }, "job_kinds": { "type": "string", - "description": "filter on job kind (values 'preview', 'script', 'dependencies', 'flow') separated by," + "description": "filter by job kind. Supports comma-separated list of values ('preview', 'script', 'dependencies', 'flow') and negation by prefixing all values with '!' (e.g. '!preview,!dependencies')" }, "suspended": { "type": "boolean", @@ -1316,7 +1378,7 @@ pub fn all_tools() -> Vec { }, "tag": { "type": "string", - "description": "filter on jobs with a given tag/worker group" + "description": "filter by tag/worker group. Supports comma-separated list (e.g. 'gpu,highmem') and negation by prefixing all values with '!' (e.g. '!gpu,!highmem')" }, "result": { "type": "string", @@ -1331,8 +1393,8 @@ pub fn all_tools() -> Vec { "description": "number of items to return for a given page (default 30, max 100)" }, "trigger_kind": { - "description": "trigger kind (schedule, http, websocket...). Possible values: webhook, default_email, email, schedule, http, websocket, postgres, kafka, nats, mqtt, sqs, gcp", - "type": "string" + "type": "string", + "description": "filter by trigger kind. Supports comma-separated list (e.g. 'schedule,webhook') and negation by prefixing all values with '!' (e.g. '!schedule,!webhook')" }, "is_skipped": { "type": "boolean", @@ -1357,6 +1419,77 @@ pub fn all_tools() -> Vec { "is_not_schedule": { "type": "boolean", "description": "is not a scheduled job" + }, + "broad_filter": { + "type": "string", + "description": "broad search across multiple fields (case-insensitive substring match on path, tag, schedule path, trigger kind, label)" + } + }, + "required": [] +})), + body_schema: None, + path_field_renames: None, + query_field_renames: None, + body_field_renames: None, + }, + EndpointTool { + name: Cow::Borrowed("getJob"), + description: Cow::Borrowed("get job"), + instructions: Cow::Borrowed(""), + path: Cow::Borrowed("/w/{workspace}/jobs_u/get/{id}"), + method: Cow::Borrowed("GET"), + path_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "id": { + "type": "string", + "format": "uuid" + } + }, + "required": [ + "id" + ] +})), + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "no_logs": { + "type": "boolean" + }, + "no_code": { + "type": "boolean" + } + }, + "required": [] +})), + body_schema: None, + path_field_renames: None, + query_field_renames: None, + body_field_renames: None, + }, + EndpointTool { + name: Cow::Borrowed("getJobLogs"), + description: Cow::Borrowed("get job logs"), + instructions: Cow::Borrowed(""), + path: Cow::Borrowed("/w/{workspace}/jobs_u/get_logs/{id}"), + method: Cow::Borrowed("GET"), + path_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "id": { + "type": "string", + "format": "uuid" + } + }, + "required": [ + "id" + ] +})), + query_params_schema: Some(serde_json::json!({ + "type": "object", + "properties": { + "remove_ansi_warnings": { + "type": "boolean" } }, "required": [] @@ -1411,14 +1544,17 @@ You should get the schema of the script or flow before creating the schedule to }, "on_failure": { "type": "string", + "nullable": true, "description": "Path to a script or flow to run when the scheduled job fails" }, "on_failure_times": { "type": "number", + "nullable": true, "description": "Number of consecutive failures before the on_failure handler is triggered (default 1)" }, "on_failure_exact": { "type": "boolean", + "nullable": true, "description": "If true, trigger on_failure handler only on exactly N failures, not on every failure after N" }, "on_failure_extra_args": { @@ -1428,10 +1564,12 @@ You should get the schema of the script or flow before creating the schedule to }, "on_recovery": { "type": "string", + "nullable": true, "description": "Path to a script or flow to run when the schedule recovers after failures" }, "on_recovery_times": { "type": "number", + "nullable": true, "description": "Number of consecutive successes before the on_recovery handler is triggered (default 1)" }, "on_recovery_extra_args": { @@ -1441,6 +1579,7 @@ You should get the schema of the script or flow before creating the schedule to }, "on_success": { "type": "string", + "nullable": true, "description": "Path to a script or flow to run after each successful execution" }, "on_success_extra_args": { @@ -1516,28 +1655,42 @@ You should get the schema of the script or flow before creating the schedule to }, "summary": { "type": "string", + "nullable": true, "description": "Short summary describing the purpose of this schedule" }, "description": { "type": "string", + "nullable": true, "description": "Detailed description of what this schedule does" }, "tag": { "type": "string", + "nullable": true, "description": "Worker tag to route jobs to specific worker groups" }, "paused_until": { "type": "string", + "nullable": true, "format": "date-time", "description": "ISO 8601 datetime until which the schedule is paused. Schedule resumes automatically after this time" }, "cron_version": { "type": "string", + "nullable": true, "description": "Cron parser version. Use 'v2' for extended syntax with additional features" }, "dynamic_skip": { "type": "string", + "nullable": true, "description": "Path to a script that validates scheduled datetimes. Receives scheduled_for datetime and returns boolean to skip (true) or run (false)" + }, + "permissioned_as": { + "type": "string", + "description": "The user or group this schedule runs as. Used during deployment to preserve the original schedule owner." + }, + "preserve_permissioned_as": { + "type": "boolean", + "description": "When true and the caller is a member of the 'wm_deployers' group, preserves the original permissioned_as value instead of overwriting it." } }, "required": [ @@ -1592,14 +1745,17 @@ You should get the schema of the script or flow before updating the schedule to }, "on_failure": { "type": "string", + "nullable": true, "description": "Path to a script or flow to run when the scheduled job fails" }, "on_failure_times": { "type": "number", + "nullable": true, "description": "Number of consecutive failures before the on_failure handler is triggered (default 1)" }, "on_failure_exact": { "type": "boolean", + "nullable": true, "description": "If true, trigger on_failure handler only on exactly N failures, not on every failure after N" }, "on_failure_extra_args": { @@ -1609,10 +1765,12 @@ You should get the schema of the script or flow before updating the schedule to }, "on_recovery": { "type": "string", + "nullable": true, "description": "Path to a script or flow to run when the schedule recovers after failures" }, "on_recovery_times": { "type": "number", + "nullable": true, "description": "Number of consecutive successes before the on_recovery handler is triggered (default 1)" }, "on_recovery_extra_args": { @@ -1622,6 +1780,7 @@ You should get the schema of the script or flow before updating the schedule to }, "on_success": { "type": "string", + "nullable": true, "description": "Path to a script or flow to run after each successful execution" }, "on_success_extra_args": { @@ -1697,28 +1856,44 @@ You should get the schema of the script or flow before updating the schedule to }, "summary": { "type": "string", + "nullable": true, "description": "Short summary describing the purpose of this schedule" }, "description": { "type": "string", + "nullable": true, "description": "Detailed description of what this schedule does" }, "tag": { "type": "string", + "nullable": true, "description": "Worker tag to route jobs to specific worker groups" }, "paused_until": { "type": "string", + "nullable": true, "format": "date-time", "description": "ISO 8601 datetime until which the schedule is paused. Schedule resumes automatically after this time" }, "cron_version": { "type": "string", + "nullable": true, "description": "Cron parser version. Use 'v2' for extended syntax with additional features" }, "dynamic_skip": { "type": "string", + "nullable": true, "description": "Path to a script that validates scheduled datetimes. Receives scheduled_for datetime and returns boolean to skip (true) or run (false)" + }, + "permissioned_as": { + "type": "string", + "nullable": true, + "description": "The user or group this schedule runs as (e.g., 'u/admin' or 'g/mygroup'). Only admins and wm_deployers can set this via preserve_permissioned_as." + }, + "preserve_permissioned_as": { + "type": "boolean", + "nullable": true, + "description": "If true and user is admin/wm_deployers, preserve the provided permissioned_as instead of using the deploying user's identity" } }, "required": [ @@ -1801,7 +1976,7 @@ You should get the schema of the script or flow before updating the schedule to }, "path": { "type": "string", - "description": "filter by path" + "description": "filter by path (script path)" }, "is_flow": { "type": "boolean", @@ -1810,6 +1985,22 @@ You should get the schema of the script or flow before updating the schedule to "path_start": { "type": "string", "description": "filter schedules by path prefix" + }, + "schedule_path": { + "type": "string", + "description": "exact match on the schedule's path" + }, + "description": { + "type": "string", + "description": "pattern match filter for description field (case-insensitive)" + }, + "summary": { + "type": "string", + "description": "pattern match filter for summary field (case-insensitive)" + }, + "broad_filter": { + "type": "string", + "description": "broad search across multiple fields (case-insensitive substring match)" } }, "required": [] diff --git a/backend/windmill-api/src/mcp/core.rs b/backend/windmill-api/src/mcp/core.rs index 7699853559..a20484e3eb 100644 --- a/backend/windmill-api/src/mcp/core.rs +++ b/backend/windmill-api/src/mcp/core.rs @@ -546,7 +546,7 @@ pub async fn setup_mcp_server( let service = StreamableHttpService::new(move || Ok(runner.clone()), session_manager, service_config); - let router = Router::new().nest_service("/", service); + let router = Router::new().route_service("/", service); Ok((router, cancellation_token)) } diff --git a/backend/windmill-api/src/raw_apps.rs b/backend/windmill-api/src/raw_apps.rs index e331aa1176..ed746b8770 100644 --- a/backend/windmill-api/src/raw_apps.rs +++ b/backend/windmill-api/src/raw_apps.rs @@ -27,7 +27,7 @@ use windmill_common::{ pub fn workspaced_service() -> Router { Router::new() .route("/list", get(list_apps)) - .route("/get_data/:version/*path", get(get_data)) + .route("/get_data/{version}/{*path}", get(get_data)) } #[derive(FromRow, Deserialize, Serialize)] diff --git a/backend/windmill-api/src/resources.rs b/backend/windmill-api/src/resources.rs index f4bcfc621b..ee26ce758e 100644 --- a/backend/windmill-api/src/resources.rs +++ b/backend/windmill-api/src/resources.rs @@ -10,7 +10,7 @@ pub fn workspaced_service() -> Router { #[cfg(feature = "mcp")] use crate::mcp_tools::get_mcp_tools; #[cfg(feature = "mcp")] - let router = router.route("/mcp_tools/*path", get(get_mcp_tools)); + let router = router.route("/mcp_tools/{*path}", get(get_mcp_tools)); router } diff --git a/backend/windmill-api/src/scim_oss.rs b/backend/windmill-api/src/scim_oss.rs index 5210411466..845c854960 100644 --- a/backend/windmill-api/src/scim_oss.rs +++ b/backend/windmill-api/src/scim_oss.rs @@ -11,9 +11,7 @@ pub use crate::scim_ee::*; */ #[cfg(not(feature = "private"))] -use axum::{middleware::Next, response::Response, routing::get, Router}; -#[cfg(not(feature = "private"))] -use hyper::Request; +use axum::{extract::Request, middleware::Next, response::Response, routing::get, Router}; #[cfg(not(feature = "private"))] pub fn global_service() -> Router { @@ -26,7 +24,7 @@ pub async fn ee() -> String { } #[cfg(not(feature = "private"))] -pub async fn has_scim_token(_request: Request, _next: Next) -> Response { +pub async fn has_scim_token(_request: Request, _next: Next) -> Response { //Not implemented in open-source version todo!() } diff --git a/backend/windmill-api/src/scripts.rs b/backend/windmill-api/src/scripts.rs index abd0b4201b..f9dd17fa83 100644 --- a/backend/windmill-api/src/scripts.rs +++ b/backend/windmill-api/src/scripts.rs @@ -21,7 +21,7 @@ use windmill_common::{error::JsonResult, utils::StripPath, DB}; /// that depends on windmill-api internals. pub fn workspaced_service() -> Router { windmill_api_scripts::scripts::workspaced_service() - .route("/get_triggers_count/*path", get(get_triggers_count)) + .route("/get_triggers_count/{*path}", get(get_triggers_count)) } async fn get_triggers_count( diff --git a/backend/windmill-api/src/service_logs.rs b/backend/windmill-api/src/service_logs.rs index c83bb21f2c..57131d823e 100644 --- a/backend/windmill-api/src/service_logs.rs +++ b/backend/windmill-api/src/service_logs.rs @@ -20,7 +20,7 @@ use crate::db::{ApiAuthed, DB}; pub fn global_service() -> Router { Router::new() .route("/list_files", get(list_files)) - .route("/get_log_file/*path", get(get_log_file)) + .route("/get_log_file/{*path}", get(get_log_file)) } use axum::extract::Path; @@ -97,6 +97,9 @@ async fn get_log_file( require_devops_role(&db, &email).await?; let path = path.to_path(); + if path.contains("..") { + return Err(Error::BadRequest("Invalid path".to_string())); + } #[cfg(feature = "parquet")] let s3_client = windmill_object_store::get_object_store().await; #[cfg(feature = "parquet")] diff --git a/backend/windmill-api/src/trash.rs b/backend/windmill-api/src/trash.rs new file mode 100644 index 0000000000..a4bc105238 --- /dev/null +++ b/backend/windmill-api/src/trash.rs @@ -0,0 +1,511 @@ +use axum::{ + extract::{Extension, Json, Path, Query}, + routing::{delete, get, post}, + Router, +}; +use serde::Deserialize; +use windmill_audit::{audit_oss::audit_log, ActionKind}; +use windmill_common::{ + db::UserDB, + error::{Error, Result}, + trashbin::{self, TrashItem, TrashItemWithData}, + utils::require_admin, +}; + +use crate::db::{ApiAuthed, DB}; + +pub fn workspaced_service() -> Router { + Router::new() + .route("/list", get(list_trash)) + .route("/get/{id}", get(get_trash_item)) + .route("/restore/{id}", post(restore_trash_item)) + .route("/delete/{id}", delete(permanently_delete_item)) + .route("/empty", post(empty_trash)) +} + +#[derive(Deserialize)] +struct ListTrashQuery { + item_kind: Option, + page: Option, + per_page: Option, +} + +async fn list_trash( + authed: ApiAuthed, + Extension(db): Extension, + Path(w_id): Path, + Query(query): Query, +) -> Result>> { + require_admin(authed.is_admin, &authed.username)?; + let items = trashbin::list_trash( + &db, + &w_id, + query.item_kind.as_deref(), + query.page, + query.per_page, + ) + .await?; + Ok(Json(items)) +} + +async fn get_trash_item( + authed: ApiAuthed, + Extension(db): Extension, + Path((w_id, id)): Path<(String, i64)>, +) -> Result> { + require_admin(authed.is_admin, &authed.username)?; + let item = trashbin::get_trash_item(&db, &w_id, id).await?; + Ok(Json(item)) +} + +async fn restore_trash_item( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, i64)>, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + let item = trashbin::get_trash_item(&db, &w_id, id).await?; + let mut tx = user_db.begin(&authed).await?; + + match item.item_kind.as_str() { + "script" => restore_script(&mut tx, &item).await?, + "flow" => restore_flow(&mut tx, &item).await?, + "app" => restore_app(&mut tx, &item).await?, + "schedule" => restore_schedule(&mut tx, &item).await?, + "variable" => restore_variable(&mut tx, &item).await?, + "resource" => restore_resource(&mut tx, &item).await?, + kind if kind.ends_with("_trigger") => restore_trigger(&mut tx, &item).await?, + _ => { + return Err(Error::BadRequest(format!( + "Unknown item kind: {}", + item.item_kind + ))) + } + } + + sqlx::query!("DELETE FROM trashbin WHERE id = $1", item.id) + .execute(&mut *tx) + .await?; + + audit_log( + &mut *tx, + &authed, + "trash.restore", + ActionKind::Create, + &w_id, + Some(&item.item_path), + None, + ) + .await?; + + tx.commit().await?; + Ok(format!("{} '{}' restored", item.item_kind, item.item_path)) +} + +async fn permanently_delete_item( + authed: ApiAuthed, + Extension(db): Extension, + Extension(user_db): Extension, + Path((w_id, id)): Path<(String, i64)>, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + let item = trashbin::get_trash_item(&db, &w_id, id).await?; + let mut tx = user_db.begin(&authed).await?; + trashbin::permanently_delete_item(&mut *tx, &w_id, id).await?; + audit_log( + &mut *tx, + &authed, + "trashbin.permanently_delete", + ActionKind::Delete, + &w_id, + Some(&item.item_path), + None, + ) + .await?; + tx.commit().await?; + Ok("permanently deleted".to_string()) +} + +async fn empty_trash( + authed: ApiAuthed, + Extension(_db): Extension, + Extension(user_db): Extension, + Path(w_id): Path, +) -> Result { + require_admin(authed.is_admin, &authed.username)?; + let mut tx = user_db.begin(&authed).await?; + let count = trashbin::empty_trash(&mut *tx, &w_id).await?; + audit_log( + &mut *tx, + &authed, + "trashbin.empty", + ActionKind::Delete, + &w_id, + None, + None, + ) + .await?; + tx.commit().await?; + Ok(format!("{} items permanently deleted", count)) +} + +// --- Restore functions per item kind --- + +async fn restore_script(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + // Check for path conflict + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM script WHERE path = $1 AND workspace_id = $2)", + &item.item_path, + &item.workspace_id, + ) + .fetch_one(&mut *tx) + .await? + .unwrap_or(false); + + if exists { + return Err(Error::BadRequest(format!( + "A script already exists at path '{}'", + item.item_path + ))); + } + + // Scripts are stored as an array (all versions for the path) + let scripts = data + .get("scripts") + .and_then(|v| v.as_array()) + .ok_or_else(|| Error::internal_err("Invalid trash data for script"))?; + + for script in scripts { + sqlx::query("INSERT INTO script SELECT * FROM jsonb_populate_record(null::script, $1)") + .bind(script) + .execute(&mut *tx) + .await + .map_err(|e| Error::internal_err(format!("restoring script: {e:#}")))?; + } + + // Restore drafts if present + if let Some(drafts) = data.get("drafts").and_then(|v| v.as_array()) { + for draft in drafts { + sqlx::query( + "INSERT INTO draft SELECT * FROM jsonb_populate_record(null::draft, $1) + ON CONFLICT DO NOTHING", + ) + .bind(draft) + .execute(&mut *tx) + .await?; + } + } + + Ok(()) +} + +async fn restore_flow(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM flow WHERE path = $1 AND workspace_id = $2)", + &item.item_path, + &item.workspace_id, + ) + .fetch_one(&mut *tx) + .await? + .unwrap_or(false); + + if exists { + return Err(Error::BadRequest(format!( + "A flow already exists at path '{}'", + item.item_path + ))); + } + + let row = data + .get("row") + .ok_or_else(|| Error::internal_err("Invalid trash data for flow"))?; + + sqlx::query("INSERT INTO flow SELECT * FROM jsonb_populate_record(null::flow, $1)") + .bind(row) + .execute(&mut *tx) + .await?; + + // Restore flow_versions + if let Some(versions) = data.get("flow_versions").and_then(|v| v.as_array()) { + for version in versions { + sqlx::query( + "INSERT INTO flow_version SELECT * FROM jsonb_populate_record(null::flow_version, $1) + ON CONFLICT DO NOTHING", + ) + .bind(version) + .execute(&mut *tx) + .await?; + } + } + + // Restore flow_nodes + if let Some(nodes) = data.get("flow_nodes").and_then(|v| v.as_array()) { + for node in nodes { + sqlx::query( + "INSERT INTO flow_node SELECT * FROM jsonb_populate_record(null::flow_node, $1) + ON CONFLICT DO NOTHING", + ) + .bind(node) + .execute(&mut *tx) + .await?; + } + } + + // Restore drafts + if let Some(drafts) = data.get("drafts").and_then(|v| v.as_array()) { + for draft in drafts { + sqlx::query( + "INSERT INTO draft SELECT * FROM jsonb_populate_record(null::draft, $1) + ON CONFLICT DO NOTHING", + ) + .bind(draft) + .execute(&mut *tx) + .await?; + } + } + + Ok(()) +} + +async fn restore_app(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM app WHERE path = $1 AND workspace_id = $2)", + &item.item_path, + &item.workspace_id, + ) + .fetch_one(&mut *tx) + .await? + .unwrap_or(false); + + if exists { + return Err(Error::BadRequest(format!( + "An app already exists at path '{}'", + item.item_path + ))); + } + + let row = data + .get("row") + .ok_or_else(|| Error::internal_err("Invalid trash data for app"))?; + + // Insert app first (app_version has FK to app.id) + sqlx::query("INSERT INTO app SELECT * FROM jsonb_populate_record(null::app, $1)") + .bind(row) + .execute(&mut *tx) + .await + .map_err(|e| Error::internal_err(format!("restoring app row: {e:#}")))?; + + // Then restore app_versions + if let Some(versions) = data.get("app_versions").and_then(|v| v.as_array()) { + for version in versions { + sqlx::query( + "INSERT INTO app_version SELECT * FROM jsonb_populate_record(null::app_version, $1) + ON CONFLICT DO NOTHING", + ) + .bind(version) + .execute(&mut *tx) + .await + .map_err(|e| Error::internal_err(format!("restoring app_version: {e:#}")))?; + } + } + + // Restore drafts + if let Some(drafts) = data.get("drafts").and_then(|v| v.as_array()) { + for draft in drafts { + sqlx::query( + "INSERT INTO draft SELECT * FROM jsonb_populate_record(null::draft, $1) + ON CONFLICT DO NOTHING", + ) + .bind(draft) + .execute(&mut *tx) + .await?; + } + } + + Ok(()) +} + +async fn restore_schedule(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM schedule WHERE path = $1 AND workspace_id = $2)", + &item.item_path, + &item.workspace_id, + ) + .fetch_one(&mut *tx) + .await? + .unwrap_or(false); + + if exists { + return Err(Error::BadRequest(format!( + "A schedule already exists at path '{}'", + item.item_path + ))); + } + + let row = data + .get("row") + .ok_or_else(|| Error::internal_err("Invalid trash data for schedule"))?; + + sqlx::query("INSERT INTO schedule SELECT * FROM jsonb_populate_record(null::schedule, $1)") + .bind(row) + .execute(&mut *tx) + .await?; + + Ok(()) +} + +async fn restore_variable(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM variable WHERE path = $1 AND workspace_id = $2)", + &item.item_path, + &item.workspace_id, + ) + .fetch_one(&mut *tx) + .await? + .unwrap_or(false); + + if exists { + return Err(Error::BadRequest(format!( + "A variable already exists at path '{}'", + item.item_path + ))); + } + + let row = data + .get("row") + .ok_or_else(|| Error::internal_err("Invalid trash data for variable"))?; + + sqlx::query("INSERT INTO variable SELECT * FROM jsonb_populate_record(null::variable, $1)") + .bind(row) + .execute(&mut *tx) + .await?; + + // Restore linked resource if present + if let Some(linked_resource) = data.get("linked_resource") { + if !linked_resource.is_null() { + sqlx::query( + "INSERT INTO resource SELECT * FROM jsonb_populate_record(null::resource, $1) + ON CONFLICT DO NOTHING", + ) + .bind(linked_resource) + .execute(&mut *tx) + .await?; + } + } + + Ok(()) +} + +async fn restore_resource(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + let exists = sqlx::query_scalar!( + "SELECT EXISTS(SELECT 1 FROM resource WHERE path = $1 AND workspace_id = $2)", + &item.item_path, + &item.workspace_id, + ) + .fetch_one(&mut *tx) + .await? + .unwrap_or(false); + + if exists { + return Err(Error::BadRequest(format!( + "A resource already exists at path '{}'", + item.item_path + ))); + } + + let row = data + .get("row") + .ok_or_else(|| Error::internal_err("Invalid trash data for resource"))?; + + sqlx::query("INSERT INTO resource SELECT * FROM jsonb_populate_record(null::resource, $1)") + .bind(row) + .execute(&mut *tx) + .await?; + + // Restore linked variables if present + if let Some(linked_vars) = data.get("linked_variables").and_then(|v| v.as_array()) { + for var in linked_vars { + sqlx::query( + "INSERT INTO variable SELECT * FROM jsonb_populate_record(null::variable, $1) + ON CONFLICT DO NOTHING", + ) + .bind(var) + .execute(&mut *tx) + .await?; + } + } + + Ok(()) +} + +async fn restore_trigger(tx: &mut sqlx::PgConnection, item: &TrashItemWithData) -> Result<()> { + let data = &item.item_data; + + let table_name = data + .get("table_name") + .and_then(|v| v.as_str()) + .ok_or_else(|| Error::internal_err("Invalid trash data for trigger: missing table_name"))?; + + // Validate table name to prevent SQL injection + let valid_tables = [ + "http_trigger", + "websocket_trigger", + "kafka_trigger", + "nats_trigger", + "postgres_trigger", + "mqtt_trigger", + "sqs_trigger", + "gcp_trigger", + "email_trigger", + ]; + + if !valid_tables.contains(&table_name) { + return Err(Error::BadRequest(format!( + "Invalid trigger table: {}", + table_name + ))); + } + + let exists: bool = sqlx::query_scalar(&format!( + "SELECT EXISTS(SELECT 1 FROM {} WHERE path = $1 AND workspace_id = $2)", + table_name + )) + .bind(&item.item_path) + .bind(&item.workspace_id) + .fetch_one(&mut *tx) + .await?; + + if exists { + return Err(Error::BadRequest(format!( + "A trigger already exists at path '{}'", + item.item_path + ))); + } + + let row = data + .get("row") + .ok_or_else(|| Error::internal_err("Invalid trash data for trigger"))?; + + sqlx::query(&format!( + "INSERT INTO {} SELECT * FROM jsonb_populate_record(null::{}, $1)", + table_name, table_name + )) + .bind(row) + .execute(&mut *tx) + .await?; + + Ok(()) +} diff --git a/backend/windmill-api/src/triggers/handler.rs b/backend/windmill-api/src/triggers/handler.rs index 78715e2661..f798b4d706 100644 --- a/backend/windmill-api/src/triggers/handler.rs +++ b/backend/windmill-api/src/triggers/handler.rs @@ -108,11 +108,11 @@ pub fn generate_trigger_routers() -> Router { router = router .route( - "/trigger/:trigger_kind/resume_suspended_trigger_jobs/*trigger_path", + "/trigger/{trigger_kind}/resume_suspended_trigger_jobs/{*trigger_path}", post(resume_suspended_trigger_jobs), ) .route( - "/trigger/:trigger_kind/cancel_suspended_trigger_jobs/*trigger_path", + "/trigger/{trigger_kind}/cancel_suspended_trigger_jobs/{*trigger_path}", post(cancel_suspended_trigger_jobs), ); } diff --git a/backend/windmill-api/src/triggers/http/handler.rs b/backend/windmill-api/src/triggers/http/handler.rs index 8af10bda18..ccde55bab1 100644 --- a/backend/windmill-api/src/triggers/http/handler.rs +++ b/backend/windmill-api/src/triggers/http/handler.rs @@ -95,7 +95,7 @@ async fn conditional_cors_middleware( pub fn http_route_trigger_handler() -> Router { Router::new() .route( - "/*path", + "/{*path}", get(route_job) .post(route_job) .delete(route_job) diff --git a/backend/windmill-api/src/triggers/http/http_trigger_args.rs b/backend/windmill-api/src/triggers/http/http_trigger_args.rs index 7df1e9f200..f8460f42e2 100644 --- a/backend/windmill-api/src/triggers/http/http_trigger_args.rs +++ b/backend/windmill-api/src/triggers/http/http_trigger_args.rs @@ -25,8 +25,7 @@ use crate::{ pub struct RawHttpTriggerArgs(pub RawWebhookArgs); -#[axum::async_trait] -impl FromRequest for RawHttpTriggerArgs +impl FromRequest for RawHttpTriggerArgs where S: Send + Sync, { diff --git a/backend/windmill-api/src/users.rs b/backend/windmill-api/src/users.rs index 498e93e61a..aec6080bfd 100644 --- a/backend/windmill-api/src/users.rs +++ b/backend/windmill-api/src/users.rs @@ -35,9 +35,9 @@ use windmill_common::{ pub fn global_service() -> Router { windmill_api_users::users::global_service() .route("/setpassword", post(set_password)) - .route("/set_password_of/:user", post(set_password_of_user)) + .route("/set_password_of/{user}", post(set_password_of_user)) .route("/create", post(create_user)) - .route("/rename/:user", post(rename_user)) + .route("/rename/{user}", post(rename_user)) .route("/onboarding", post(submit_onboarding_data)) } diff --git a/backend/windmill-api/src/workspace_dependencies.rs b/backend/windmill-api/src/workspace_dependencies.rs index e5c9377b01..194cc1f0d5 100644 --- a/backend/windmill-api/src/workspace_dependencies.rs +++ b/backend/windmill-api/src/workspace_dependencies.rs @@ -24,9 +24,9 @@ pub fn workspaced_service() -> Router { Router::new() .route("/create", post(create)) .route("/list", get(list)) - .route("/archive/:language", post(archive)) - .route("/get_latest/:language", get(get_latest)) - .route("/delete/:language", post(delete)) + .route("/archive/{language}", post(archive)) + .route("/get_latest/{language}", get(get_latest)) + .route("/delete/{language}", post(delete)) } #[axum::debug_handler] diff --git a/backend/windmill-api/src/workspaces.rs b/backend/windmill-api/src/workspaces.rs index 50c3913b47..6d644a7946 100644 --- a/backend/windmill-api/src/workspaces.rs +++ b/backend/windmill-api/src/workspaces.rs @@ -8,8 +8,9 @@ // Re-export everything from windmill-api-workspaces pub use windmill_api_workspaces::workspaces::*; +use windmill_api_workspaces::workspaces::{build_copilot_settings_state, InstanceAISummary}; -use crate::ai::{AIConfig, AI_REQUEST_CACHE}; +use crate::ai::{invalidate_ai_request_cache_for_workspace, AIConfig}; use crate::db::ApiAuthed; use crate::teams_oss::{ connect_teams, edit_teams_command, run_teams_message_test_job, @@ -24,7 +25,7 @@ use axum::{ use windmill_audit::audit_oss::audit_log; use windmill_audit::ActionKind; use windmill_common::{ - error::{Error, JsonResult, Result}, + error::{Error, JsonResult}, utils::require_admin, DB, }; @@ -34,6 +35,9 @@ use windmill_git_sync::{handle_deployment_metadata, DeployedObject}; use axum::extract::Query; #[cfg(feature = "enterprise")] use serde::Deserialize; +use serde::Serialize; +#[cfg(feature = "enterprise")] +use windmill_common::error::Result; #[cfg(feature = "enterprise")] use windmill_common::utils::require_admin_or_devops; @@ -59,7 +63,7 @@ pub fn workspaced_service() -> Router { .route("/get_copilot_info", get(get_copilot_info)) .route("/critical_alerts", get(get_critical_alerts)) .route( - "/critical_alerts/:id/acknowledge", + "/critical_alerts/{id}/acknowledge", post(acknowledge_critical_alert), ) .route( @@ -83,7 +87,7 @@ async fn edit_copilot_config( Path(w_id): Path, ApiAuthed { is_admin, username, .. }: ApiAuthed, Json(ai_config): Json, -) -> Result { +) -> JsonResult { require_admin(is_admin, &username)?; if let Some(ref custom_prompts) = ai_config.custom_prompts { @@ -109,11 +113,7 @@ async fn edit_copilot_config( .execute(&mut *tx) .await?; - if let Some(ref providers) = ai_config.providers { - for provider in providers.keys() { - AI_REQUEST_CACHE.remove(&(w_id.clone(), provider.clone())); - } - } + invalidate_ai_request_cache_for_workspace(&w_id); audit_log( &mut *tx, @@ -139,37 +139,66 @@ async fn edit_copilot_config( ) .await?; - Ok(format!("Edit copilot config for workspace {}", &w_id)) + let workspace_has_config = ai_config.has_providers(); + let instance_ai_config = + sqlx::query_scalar!("SELECT value FROM global_settings WHERE name = 'ai_config'") + .fetch_optional(&db) + .await?; + let settings_state = + build_copilot_settings_state(workspace_has_config, instance_ai_config.as_ref()); + let effective_ai_config = if workspace_has_config { + ai_config + } else if let Some(instance_ai_config) = instance_ai_config { + serde_json::from_value::(instance_ai_config).unwrap_or_default() + } else { + AIConfig::default() + }; + + Ok(Json(EditCopilotConfigResponse { + effective_ai_config, + has_instance_ai_config: settings_state.has_instance_ai_config, + uses_instance_ai_config: settings_state.uses_instance_ai_config, + instance_ai_summary: settings_state.instance_ai_summary, + })) +} + +#[derive(Serialize)] +struct EditCopilotConfigResponse { + effective_ai_config: AIConfig, + has_instance_ai_config: bool, + uses_instance_ai_config: bool, + #[serde(skip_serializing_if = "Option::is_none")] + instance_ai_summary: Option, } async fn get_copilot_info( Extension(db): Extension, Path(w_id): Path, ) -> JsonResult { - let mut tx = db.begin().await?; - let copilot_info = sqlx::query_scalar!( + let workspace_ai_config = sqlx::query_scalar!( "SELECT ai_config as \"ai_config: sqlx::types::Json\" FROM workspace_settings WHERE workspace_id = $1", &w_id ) - .fetch_one(&mut *tx) + .fetch_one(&db) .await .map_err(|e| { Error::internal_err(format!( "getting ai config: {e:#}" )) })?; - tx.commit().await?; - if let Some(sqlx::types::Json(copilot_info)) = copilot_info { - Ok(Json(copilot_info)) + if let Some(workspace_ai_config) = workspace_ai_config.filter(|c| c.0.has_providers()) { + Ok(Json(workspace_ai_config.0)) + } else if let Some(instance_config) = + sqlx::query_scalar!("SELECT value FROM global_settings WHERE name = 'ai_config'") + .fetch_optional(&db) + .await? + { + Ok(Json( + serde_json::from_value::(instance_config).unwrap_or_default(), + )) } else { - Ok(Json(AIConfig { - providers: None, - default_model: None, - code_completion_model: None, - custom_prompts: None, - max_tokens_per_model: None, - })) + Ok(Json(AIConfig::default())) } } diff --git a/backend/windmill-api/src/workspaces_export.rs b/backend/windmill-api/src/workspaces_export.rs index d685d4c4cc..6c6c783ba7 100644 --- a/backend/windmill-api/src/workspaces_export.rs +++ b/backend/windmill-api/src/workspaces_export.rs @@ -388,8 +388,6 @@ pub(crate) async fn tarball_workspace( settings_version, }): Query, ) -> Result<([(HeaderName, String); 2], impl IntoResponse)> { - // require_admin(authed.is_admin, &authed.username)?; - tracing::info!( "tarball_workspace called for workspace {}: include_workspace_dependencies={:?}, skip_variables={:?}, skip_resources={:?}", w_id, @@ -1078,6 +1076,8 @@ pub(crate) async fn tarball_workspace( } if include_key.unwrap_or(false) { + require_admin(authed.is_admin, &authed.username)?; + let key = sqlx::query_scalar!( "SELECT key FROM workspace_key WHERE workspace_id = $1", &w_id diff --git a/backend/windmill-common/Cargo.toml b/backend/windmill-common/Cargo.toml index a0aa27342d..fcf69c13ee 100644 --- a/backend/windmill-common/Cargo.toml +++ b/backend/windmill-common/Cargo.toml @@ -56,6 +56,7 @@ tokio-util.workspace = true datafusion = { workspace = true, optional = true} reqwest = { workspace = true } tracing-subscriber = { workspace = true } +aho-corasick = "1" lazy_static.workspace = true tracing-appender.workspace = true gethostname.workspace = true @@ -116,6 +117,7 @@ pin-project-lite.workspace = true futures.workspace = true tempfile.workspace = true globset.workspace = true +dashmap.workspace = true opentelemetry-semantic-conventions = { workspace = true, optional = true } opentelemetry-otlp = { workspace = true, optional = true } diff --git a/backend/windmill-common/src/ai_bedrock.rs b/backend/windmill-common/src/ai_bedrock.rs index ea817a1e41..9a7c5baeeb 100644 --- a/backend/windmill-common/src/ai_bedrock.rs +++ b/backend/windmill-common/src/ai_bedrock.rs @@ -11,8 +11,9 @@ use aws_config::BehaviorVersion; use aws_credential_types::provider::token::ProvideToken; use aws_credential_types::provider::ProvideCredentials; use aws_sdk_bedrockruntime::types::{ - ContentBlock, ConversationRole, ConverseStreamOutput, ImageBlock, ImageFormat, ImageSource, - InferenceConfiguration, Message, SystemContentBlock, Tool, ToolInputSchema, ToolSpecification, + ContentBlock, ConversationRole, ConverseStreamOutput, DocumentBlock, DocumentFormat, + DocumentSource, ImageBlock, ImageFormat, ImageSource, InferenceConfiguration, Message, + SystemContentBlock, Tool, ToolInputSchema, ToolSpecification, }; use aws_sdk_bedrockruntime::Client as BedrockRuntimeClient; use serde::{Deserialize, Serialize}; @@ -356,13 +357,12 @@ pub fn content_to_text(content: &OpenAIContent) -> String { } } -/// Parse image data URL and extract format and base64 data -fn parse_image_data_url(url: &str) -> Result<(ImageFormat, Vec), Error> { +/// Parse a data URL and extract MIME type and decoded bytes. +fn parse_data_url_bytes(url: &str) -> Result<(String, Vec), Error> { if !url.starts_with("data:") { - return Err(Error::internal_err("Image URL must be a data URL")); + return Err(Error::internal_err("URL must be a data URL")); } - // Parse data:image/png;base64, let base64_start = url .find("base64,") .ok_or_else(|| Error::internal_err("Invalid data URL format"))?; @@ -372,30 +372,51 @@ fn parse_image_data_url(url: &str) -> Result<(ImageFormat, Vec), Error> { .split(';') .next() .and_then(|s| s.strip_prefix("data:")) - .unwrap_or("image/png"); + .unwrap_or("application/octet-stream"); + + let bytes = base64::Engine::decode(&base64::engine::general_purpose::STANDARD, base64_data) + .map_err(|e| Error::internal_err(format!("Failed to decode base64 data: {}", e)))?; + + Ok((mime_type.to_string(), bytes)) +} + +/// Parse an image data URL and extract ImageFormat and decoded bytes. +fn parse_image_data_url(url: &str) -> Result<(ImageFormat, Vec), Error> { + let (mime_type, bytes) = parse_data_url_bytes(url)?; - // Extract format from MIME type (e.g., "image/png" -> "png") let format_str = mime_type .rsplit_once('/') .map(|(_, format)| format) .unwrap_or("png"); - // Map to ImageFormat enum let format = match format_str { "png" => ImageFormat::Png, "jpeg" | "jpg" => ImageFormat::Jpeg, "gif" => ImageFormat::Gif, "webp" => ImageFormat::Webp, - _ => ImageFormat::Png, // Default to PNG + _ => ImageFormat::Png, }; - // Decode base64 - let bytes = base64::Engine::decode(&base64::engine::general_purpose::STANDARD, base64_data) - .map_err(|e| Error::internal_err(format!("Failed to decode base64 image: {}", e)))?; - Ok((format, bytes)) } +/// Map a MIME type to a Bedrock DocumentFormat. +fn mime_to_document_format(mime_type: &str) -> DocumentFormat { + match mime_type { + "application/pdf" => DocumentFormat::Pdf, + "text/csv" => DocumentFormat::Csv, + "text/html" => DocumentFormat::Html, + "text/plain" => DocumentFormat::Txt, + "application/vnd.openxmlformats-officedocument.wordprocessingml.document" => { + DocumentFormat::Docx + } + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" => { + DocumentFormat::Xlsx + } + _ => DocumentFormat::Pdf, + } +} + /// Convert a ContentPart to Bedrock ContentBlock fn content_part_to_block(part: &ContentPart) -> Result, Error> { match part { @@ -418,8 +439,21 @@ fn content_part_to_block(part: &ContentPart) -> Result, Err Ok(Some(ContentBlock::Image(image_block))) } + ContentPart::File { file } => { + let (mime_type, bytes) = parse_data_url_bytes(&file.file_data)?; + let doc_source = DocumentSource::Bytes(bytes.into()); + let doc_block = DocumentBlock::builder() + .format(mime_to_document_format(&mime_type)) + .name(file.filename.replace('.', "_")) + .source(doc_source) + .build() + .map_err(|e| { + Error::internal_err(format!("Failed to build document block: {}", e)) + })?; + Ok(Some(ContentBlock::Document(doc_block))) + } ContentPart::S3Object { .. } => { - // S3Objects should be converted to ImageUrl before calling this function + // S3Objects should be converted before calling this function Ok(None) } } diff --git a/backend/windmill-common/src/ai_cache.rs b/backend/windmill-common/src/ai_cache.rs new file mode 100644 index 0000000000..11bcd42254 --- /dev/null +++ b/backend/windmill-common/src/ai_cache.rs @@ -0,0 +1,11 @@ +use std::sync::atomic::{AtomicU64, Ordering}; + +static INSTANCE_AI_CONFIG_REVISION: AtomicU64 = AtomicU64::new(0); + +pub fn current_instance_ai_config_revision() -> u64 { + INSTANCE_AI_CONFIG_REVISION.load(Ordering::SeqCst) +} + +pub fn bump_instance_ai_config_revision() -> u64 { + INSTANCE_AI_CONFIG_REVISION.fetch_add(1, Ordering::SeqCst) + 1 +} diff --git a/backend/windmill-common/src/ai_google.rs b/backend/windmill-common/src/ai_google.rs index ccf34685e5..b76d976a4c 100644 --- a/backend/windmill-common/src/ai_google.rs +++ b/backend/windmill-common/src/ai_google.rs @@ -9,7 +9,10 @@ use serde::{Deserialize, Serialize}; -use crate::ai_types::{ContentPart, ExtraContent, GoogleExtraContent, OpenAIContent, OpenAIMessage, ToolDef, UrlCitation}; +use crate::ai_types::{ + ContentPart, ExtraContent, GoogleExtraContent, OpenAIContent, OpenAIMessage, ToolDef, + UrlCitation, +}; use crate::error::Error; // ============================================================================ @@ -87,7 +90,10 @@ pub struct GeminiTextRequest { /// Tool definition — function declarations and/or Google Search grounding. #[derive(Serialize)] pub struct GeminiTool { - #[serde(rename = "functionDeclarations", skip_serializing_if = "Option::is_none")] + #[serde( + rename = "functionDeclarations", + skip_serializing_if = "Option::is_none" + )] pub function_declarations: Option>, #[serde(rename = "googleSearch", skip_serializing_if = "Option::is_none")] pub google_search: Option, @@ -115,7 +121,10 @@ pub struct GeminiToolConfig { #[derive(Serialize)] pub struct GeminiFunctionCallingConfig { pub mode: String, - #[serde(rename = "allowedFunctionNames", skip_serializing_if = "Option::is_none")] + #[serde( + rename = "allowedFunctionNames", + skip_serializing_if = "Option::is_none" + )] pub allowed_function_names: Option>, } @@ -341,7 +350,12 @@ pub fn convert_content_to_gemini_parts(content: &OpenAIContent) -> Vec { - parse_data_url(&image_url.url).map(|(mime_type, data)| { + parse_data_url(&image_url.url).map(|(mime_type, data)| GeminiPart::InlineData { + inline_data: GeminiInlineData { mime_type, data }, + }) + } + ContentPart::File { file } => { + parse_data_url(&file.file_data).map(|(mime_type, data)| { GeminiPart::InlineData { inline_data: GeminiInlineData { mime_type, data }, } @@ -372,15 +386,12 @@ pub fn openai_messages_to_gemini( if let Some(content) = &msg.content { let parts = convert_content_to_gemini_parts(content); if !parts.is_empty() { - system_instruction = - Some(GeminiContentMessage { role: None, parts }); + system_instruction = Some(GeminiContentMessage { role: None, parts }); } } } "tool" => { - if let (Some(tool_call_id), Some(content)) = - (&msg.tool_call_id, &msg.content) - { + if let (Some(tool_call_id), Some(content)) = (&msg.tool_call_id, &msg.content) { let func_name = find_gemini_function_name(messages, tool_call_id); let response_text = match content { OpenAIContent::Text(text) => text.clone(), @@ -435,10 +446,8 @@ pub fn openai_messages_to_gemini( } if !parts.is_empty() { - contents.push(GeminiContentMessage { - role: Some(gemini_role.to_string()), - parts, - }); + contents + .push(GeminiContentMessage { role: Some(gemini_role.to_string()), parts }); } } } @@ -469,10 +478,8 @@ pub fn openai_tools_to_gemini( .collect(); if !declarations.is_empty() { - gemini_tools.push(GeminiTool { - function_declarations: Some(declarations), - google_search: None, - }); + gemini_tools + .push(GeminiTool { function_declarations: Some(declarations), google_search: None }); } if has_websearch { diff --git a/backend/windmill-common/src/ai_types.rs b/backend/windmill-common/src/ai_types.rs index 707fab0c2a..bb690fe986 100644 --- a/backend/windmill-common/src/ai_types.rs +++ b/backend/windmill-common/src/ai_types.rs @@ -33,6 +33,11 @@ pub enum ContentPart { ImageUrl { image_url: ImageUrlData, }, + /// File content block for OpenAI Chat Completions format (PDFs, etc.) + #[serde(rename = "file")] + File { + file: FileData, + }, #[serde(rename = "s3_object")] S3Object { s3_object: S3Object, @@ -44,6 +49,25 @@ pub struct ImageUrlData { pub url: String, // data:image/png;base64,... or https://... } +#[derive(Serialize, Deserialize, Clone, Debug)] +pub struct FileData { + pub filename: String, + pub file_data: String, // data:application/pdf;base64,... +} + +/// Check if a MIME type represents a document (as opposed to an image). +pub fn is_document_mime(mime_type: &str) -> bool { + matches!( + mime_type, + "application/pdf" + | "text/csv" + | "text/html" + | "text/plain" + | "application/vnd.openxmlformats-officedocument.wordprocessingml.document" + | "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + ) +} + #[derive(Serialize, Deserialize, Clone, Debug)] #[serde(untagged)] pub enum OpenAIContent { diff --git a/backend/windmill-common/src/cache.rs b/backend/windmill-common/src/cache.rs index eae1648657..e39fd6ea43 100644 --- a/backend/windmill-common/src/cache.rs +++ b/backend/windmill-common/src/cache.rs @@ -15,7 +15,6 @@ use crate::{ scripts::{ScriptHash, ScriptLang, ScriptModule}, }; use anyhow::anyhow; -use serde_json::value::to_raw_value; #[cfg(feature = "scoped_cache")] use std::thread::ThreadId; @@ -42,15 +41,14 @@ pub use quick_cache::sync::Cache; #[cfg(not(feature = "scoped_cache"))] lazy_static! { /// Cache directory for windmill server/worker(s). - /// 1. If `XDG_CACHE_HOME` is set, use `"${XDG_CACHE_HOME}/windmill"`. - /// 2. If `HOME` is set, use `"${HOME}/.cache/windmill"`. - /// 3. Otherwise, use `"{std::env::temp_dir()}/windmill/cache"`. + /// Lives under `WINDMILL_DIR` (default `/tmp/windmill`) as `cache_db/`. + /// If `WINDMILL_CACHE_PREFIX` (or `WEBMUX_BRANCH`) is set, uses `cache_db/{prefix}/`. pub static ref CACHE_PATH: PathBuf = { - std::env::var("XDG_CACHE_HOME") - .map(PathBuf::from) - .or_else(|_| std::env::var("HOME").map(|home| PathBuf::from(home).join(".cache"))) - .map(|cache| cache.join("windmill")) - .unwrap_or_else(|_| std::env::temp_dir().join("windmill/cache")) + let base = PathBuf::from(&*crate::worker::WINDMILL_DIR).join("cache_db"); + let prefix = std::env::var("WINDMILL_CACHE_PREFIX") + .or_else(|_| std::env::var("WEBMUX_BRANCH")) + .unwrap_or_default(); + if prefix.is_empty() { base } else { base.join(prefix) } }; } @@ -103,7 +101,7 @@ pub trait Import: Sized { } /// A type that can be exported to [`Storage`]. -pub trait Export: Clone { +pub trait Export: Sized { /// The untrusted type that can be imported from [`Storage`]. type Untrusted: Import; @@ -122,7 +120,9 @@ pub struct FsBackedCache { root: Root, } -impl> FsBackedCache { +impl> + FsBackedCache +{ /// Create a new file-system backed cache with `items_capacity` capacity. /// The cache will be stored in the `root` directory. pub fn new(root: Root, items_capacity: usize) -> Self { @@ -265,7 +265,11 @@ pub mod future { /// assert_eq!(result.unwrap(), 42u64); /// }; /// ``` - fn cached, Root: AsRef>( + fn cached< + Key: Eq + Hash + Item + Clone, + Val: Export + Clone, + Root: AsRef, + >( self, cache: &FsBackedCache, key: Key, @@ -278,60 +282,57 @@ pub mod future { } /// Flow data: i.e. a cached `raw_flow`. -/// Contains the original json raw value and a pre-parsed [`FlowValue`]. -#[derive(Debug, Clone)] +/// Contains the original json raw value; [`FlowValue`] is parsed lazily on first access. pub struct FlowData { pub raw_flow: Box, - pub flow: FlowValue, + flow: std::sync::OnceLock, +} + +impl std::fmt::Debug for FlowData { + fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result { + f.debug_struct("FlowData") + .field("raw_flow", &"") + .field("flow", &self.flow.get().map(|_| "")) + .finish() + } } #[derive(Debug, Clone, Deserialize, Serialize)] -pub struct FlowNotes { +pub struct FlowExtras { pub notes: Option>, + pub groups: Option>, } impl FlowData { - pub fn notes(&self) -> Option { - serde_json::from_str::(self.raw_flow.get()) + pub fn extras(&self) -> Option { + serde_json::from_str::(self.raw_flow.get()) .map_err(|e| { - tracing::error!("Failed to parse notes into FlowNotes: {}", e); - error::Error::internal_err(format!("Failed to parse notes into FlowNotes: {}", e)) + tracing::error!("Failed to parse flow extras: {}", e); + error::Error::internal_err(format!("Failed to parse flow extras: {}", e)) }) .ok() } } -/// !!!Shouldn't be used. Reverted optimization for ai agent steps.!!! -#[derive(Deserialize)] -struct RevertedFlowNodeFlow { - value: FlowValue, -} - impl FlowData { pub fn from_raw(raw_flow: Box) -> error::Result { - match serde_json::from_str::(raw_flow.get()) { - Ok(flow) => Ok(FlowData { raw_flow, flow }), - _ => { - // fallback for compatibility with bad version 1.560.0 - // TODO: remove this in a future version. Reverted optimization for ai agent steps. - let flow_node_flow = serde_json::from_str::(raw_flow.get()) - .map_err(|e| { - error::Error::internal_err(format!( - "Failed to parse as RevertedFlowNodeFlow: {}", - e - )) - })?; - let raw_flow = to_raw_value(&flow_node_flow.value)?; - Ok(FlowData { raw_flow, flow: flow_node_flow.value }) - } - } + let val = serde_json::from_str::(raw_flow.get()).map_err(|e| { + error::Error::internal_err(format!("Failed to parse flow value: {}", e)) + })?; + let flow = std::sync::OnceLock::new(); + let _ = flow.set(val); + Ok(FlowData { raw_flow, flow }) } + /// Return the parsed [`FlowValue`]. Already parsed from `from_raw`. pub fn value(&self) -> &FlowValue { - &self.flow + self.flow.get_or_init(|| { + serde_json::from_str::(self.raw_flow.get()) + .expect("FlowData raw_flow was validated at construction") + }) } } -#[derive(Debug, Clone)] +#[derive(Debug)] pub struct ScriptData { pub lock: Option, pub code: String, @@ -422,9 +423,15 @@ impl From for RawNode { } } -#[derive(Debug, Clone)] +#[derive(Debug)] struct Entry(Arc); +impl Clone for Entry { + fn clone(&self) -> Self { + Entry(Arc::clone(&self.0)) + } +} + #[derive(Debug, Clone)] struct ScriptFull { pub data: Arc, @@ -1012,13 +1019,10 @@ pub mod raw_script_temp { /// Load content from cache, falling back to DB. pub fn load(hash: String, db: &DB) -> impl Future> + '_ { CACHE.get_or_insert_async(hash.clone(), async move { - sqlx::query_scalar!( - "SELECT content FROM raw_script_temp WHERE hash = $1", - &hash - ) - .fetch_optional(db) - .await? - .ok_or_else(|| error::Error::NotFound(format!("raw_script_temp hash: {}", hash))) + sqlx::query_scalar!("SELECT content FROM raw_script_temp WHERE hash = $1", &hash) + .fetch_optional(db) + .await? + .ok_or_else(|| error::Error::NotFound(format!("raw_script_temp hash: {}", hash))) }) } } @@ -1227,3 +1231,75 @@ const _: () = { } } }; + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn flow_data_extras_preserves_notes_and_groups() { + let raw = serde_json::value::to_raw_value(&json!({ + "modules": [], + "notes": [{"id": "n1", "text": "hello", "color": "blue", "type": "group", + "contained_node_ids": ["a", "b"], "locked": false}], + "groups": [{"start_id": "a", "end_id": "b", "summary": "grp", "color": "green"}] + })) + .unwrap(); + + let data = FlowData::from_raw(raw).unwrap(); + + // FlowValue ignores notes/groups + assert!(data.value().modules.is_empty()); + + // But extras() recovers them from the raw JSON + let extras = data.extras().expect("extras should parse"); + let notes: serde_json::Value = + serde_json::from_str(extras.notes.expect("notes present").get()).unwrap(); + assert_eq!(notes.as_array().unwrap().len(), 1); + assert_eq!(notes[0]["id"], "n1"); + assert_eq!(notes[0]["color"], "blue"); + + let groups: serde_json::Value = + serde_json::from_str(extras.groups.expect("groups present").get()).unwrap(); + assert_eq!(groups.as_array().unwrap().len(), 1); + assert_eq!(groups[0]["start_id"], "a"); + } + + #[test] + fn flow_data_extras_returns_none_when_missing() { + let raw = serde_json::value::to_raw_value(&json!({"modules": []})).unwrap(); + let data = FlowData::from_raw(raw).unwrap(); + + let extras = data + .extras() + .expect("extras should parse even without notes/groups"); + assert!(extras.notes.is_none()); + assert!(extras.groups.is_none()); + } + + #[test] + fn flow_data_extras_lost_after_flow_value_roundtrip() { + // Demonstrates the bug: serializing through FlowValue drops notes/groups. + // This is the root cause of #8641. + let raw = serde_json::value::to_raw_value(&json!({ + "modules": [], + "notes": [{"id": "n1", "text": "t", "color": "blue", "type": "free"}] + })) + .unwrap(); + + let data = FlowData::from_raw(raw).unwrap(); + + // Re-serialize through FlowValue (what RunFlowDependenciesRequest does) + let stripped = serde_json::to_string(data.value()).unwrap(); + let stripped_raw = RawValue::from_string(stripped).unwrap(); + let data2 = FlowData::from_raw(stripped_raw).unwrap(); + + // Notes are gone after the FlowValue round-trip + let extras = data2.extras().expect("extras should parse"); + assert!( + extras.notes.is_none(), + "notes lost after FlowValue round-trip" + ); + } +} diff --git a/backend/windmill-common/src/ee_oss.rs b/backend/windmill-common/src/ee_oss.rs index 51b1efd2e2..93d0061ade 100644 --- a/backend/windmill-common/src/ee_oss.rs +++ b/backend/windmill-common/src/ee_oss.rs @@ -23,6 +23,7 @@ lazy_static::lazy_static! { } #[cfg(not(feature = "private"))] +#[derive(PartialEq, Eq)] pub enum LicensePlan { Community, Pro, diff --git a/backend/windmill-common/src/flows.rs b/backend/windmill-common/src/flows.rs index 9c10bf62ce..a0923ccabf 100644 --- a/backend/windmill-common/src/flows.rs +++ b/backend/windmill-common/src/flows.rs @@ -8,16 +8,110 @@ pub use windmill_types::flows::*; +use anyhow::Context; +use serde::Deserialize; +use serde::Serialize; use sqlx::types::Json; use sqlx::types::JsonRawValue; use crate::{ - cache, + cache::{self, FlowExtras}, db::DB, - error::Error, + error::{to_anyhow, Error}, + utils::{http_get_from_hub, StripPath}, worker::{to_raw_value, Connection}, + DEFAULT_HUB_BASE_URL, HUB_BASE_URL, PRIVATE_HUB_MIN_VERSION, }; +#[derive(Deserialize)] +pub struct HubFlow { + pub value: FlowValue, +} + +#[derive(Deserialize)] +struct HubFlowResponse { + flow: HubFlow, +} + +fn extract_hub_flow_id_from_path(path: &str) -> Result { + let hub_flow_path = path.strip_prefix("hub/flows/").ok_or_else(|| { + Error::BadRequest(format!( + "expected hub flow path to start with hub/flows/ (got {path})" + )) + })?; + + let flow_id = hub_flow_path + .split('/') + .next() + .filter(|segment| !segment.is_empty()) + .ok_or_else(|| { + Error::BadRequest(format!( + "expected hub flow path to include a numeric id after hub/flows/ (got {path})" + )) + })?; + + let flow_id = flow_id.parse::().map_err(|_| { + Error::BadRequest(format!( + "expected hub flow path to include a numeric id after hub/flows/ (got {path})" + )) + })?; + + if flow_id <= 0 { + return Err(Error::BadRequest(format!( + "expected hub flow path to include a positive numeric id after hub/flows/ (got {path})" + ))); + } + + Ok(flow_id) +} + +pub async fn get_full_hub_flow_by_path( + path: StripPath, + http_client: &reqwest::Client, + db: Option<&DB>, +) -> crate::error::Result { + let path = path.to_path(); + let flow_id = extract_hub_flow_id_from_path(&path)?; + let hub_base_url = HUB_BASE_URL.read().await.clone(); + let hub_url = format!("{hub_base_url}/flows/{flow_id}/json"); + + let response = match http_get_from_hub(http_client, &hub_url, false, None, db) + .await? + .error_for_status() + .map_err(to_anyhow) + { + Ok(response) => response, + Err(_) if hub_base_url != DEFAULT_HUB_BASE_URL && flow_id < PRIVATE_HUB_MIN_VERSION => + { + tracing::info!("Not found on private hub, fallback to default hub for hub flow {path}"); + let fallback_url = format!("{DEFAULT_HUB_BASE_URL}/flows/{flow_id}/json"); + http_get_from_hub(http_client, &fallback_url, false, None, db) + .await? + .error_for_status() + .map_err(to_anyhow)? + } + Err(err) => return Err(err.into()), + }; + + Ok(response + .json::() + .await + .context(format!("Decoding hub response for flow at path {path}"))? + .flow) +} + +/// Serialize-only wrapper that combines resolved FlowValue with display-only extras. +/// flatten + RawValue is fine for serialization (only deserialization breaks). +#[derive(Serialize)] +struct FlowValueWithExtras<'a> { + #[serde(flatten)] + flow: &'a FlowValue, + #[serde(skip_serializing_if = "Option::is_none")] + notes: Option<&'a Box>, + #[serde(skip_serializing_if = "Option::is_none")] + groups: Option<&'a Box>, +} + /// Resolve the value of a flow if any. pub async fn resolve_maybe_value( e: &sqlx::PgPool, @@ -32,24 +126,36 @@ pub async fn resolve_maybe_value( let Some(value) = value_mut(&mut container) else { return Ok(Some(container)); }; - resolve_value(e, workspace_id, &mut value.0, with_code).await?; + resolve_value_for_api(e, workspace_id, &mut value.0, with_code).await?; Ok(Some(container)) } /// Resolve modules recursively. -async fn resolve_value( +/// Stashes display-only fields (notes, groups) before the FlowValue round-trip +/// and re-injects them after, since FlowValue doesn't carry them. +async fn resolve_value_for_api( e: &sqlx::PgPool, workspace_id: &str, value: &mut Box, with_code: bool, ) -> Result<(), Error> { + let extras = serde_json::from_str::(value.get()) + .map_err(|e| tracing::warn!("Failed to parse flow extras: {e}")) + .ok(); + let mut val = serde_json::from_str::(value.get()).map_err(|err| { Error::internal_err(format!("resolve: Failed to parse flow value: {}", err)) })?; for module in &mut val.modules { resolve_module(e, workspace_id, &mut module.value, with_code).await?; } - *value = to_raw_value(&val); + + let extras = extras.unwrap_or(FlowExtras { notes: None, groups: None }); + *value = to_raw_value(&FlowValueWithExtras { + flow: &val, + notes: extras.notes.as_ref(), + groups: extras.groups.as_ref(), + }); Ok(()) } @@ -162,3 +268,77 @@ pub async fn resolve_modules( } Ok(()) } + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn flow_value_with_extras_serializes_notes_and_groups() { + let input = json!({ + "modules": [], + "notes": [{"id": "n1", "text": "hello", "color": "yellow", "type": "free"}], + "groups": [{"start_id": "a", "end_id": "b", "summary": "grp"}] + }); + let input_str = serde_json::to_string(&input).unwrap(); + + // Parse FlowValue (drops notes/groups) and FlowExtras (captures them) + let val: FlowValue = serde_json::from_str(&input_str).unwrap(); + let extras: FlowExtras = serde_json::from_str(&input_str).unwrap(); + + // Serialize via FlowValueWithExtras — should include both + let combined = FlowValueWithExtras { + flow: &val, + notes: extras.notes.as_ref(), + groups: extras.groups.as_ref(), + }; + let output: serde_json::Value = + serde_json::from_str(&serde_json::to_string(&combined).unwrap()).unwrap(); + + assert_eq!(output["notes"], input["notes"]); + assert_eq!(output["groups"], input["groups"]); + assert!(output["modules"].is_array()); + } + + #[test] + fn flow_value_with_extras_omits_none_extras() { + let val: FlowValue = serde_json::from_str(r#"{"modules":[]}"#).unwrap(); + let combined = FlowValueWithExtras { flow: &val, notes: None, groups: None }; + let output = serde_json::to_string(&combined).unwrap(); + assert!(!output.contains("notes")); + assert!(!output.contains("groups")); + } + + #[test] + fn extract_hub_flow_id_accepts_id_only_paths() { + assert_eq!(extract_hub_flow_id_from_path("hub/flows/76").unwrap(), 76); + } + + #[test] + fn extract_hub_flow_id_accepts_id_and_slug_paths() { + assert_eq!( + extract_hub_flow_id_from_path("hub/flows/76/send-message-to-company-ai-assistant") + .unwrap(), + 76 + ); + } + + #[test] + fn extract_hub_flow_id_rejects_non_numeric_ids() { + let err = extract_hub_flow_id_from_path("hub/flows/send_message").unwrap_err(); + assert!(matches!(err, Error::BadRequest(_))); + } + + #[test] + fn extract_hub_flow_id_rejects_missing_ids() { + let err = extract_hub_flow_id_from_path("hub/flows/").unwrap_err(); + assert!(matches!(err, Error::BadRequest(_))); + } + + #[test] + fn extract_hub_flow_id_rejects_zero_ids() { + let err = extract_hub_flow_id_from_path("hub/flows/0").unwrap_err(); + assert!(matches!(err, Error::BadRequest(_))); + } +} diff --git a/backend/windmill-common/src/global_settings.rs b/backend/windmill-common/src/global_settings.rs index 937bebf09c..62a4bdc700 100644 --- a/backend/windmill-common/src/global_settings.rs +++ b/backend/windmill-common/src/global_settings.rs @@ -1,9 +1,11 @@ pub const CUSTOM_TAGS_SETTING: &str = "custom_tags"; pub const DEFAULT_TAGS_PER_WORKSPACE_SETTING: &str = "default_tags_per_workspace"; pub const DEFAULT_TAGS_WORKSPACES_SETTING: &str = "default_tags_workspaces"; +pub const PREVIEW_TAGS_OVERRIDE_SETTING: &str = "preview_tags_override"; pub const BASE_URL_SETTING: &str = "base_url"; pub const WS_BASE_URL_SETTING: &str = "ws_base_url"; pub const OAUTH_SETTING: &str = "oauths"; +pub const AI_CONFIG_SETTING: &str = "ai_config"; pub const RETENTION_PERIOD_SECS_SETTING: &str = "retention_period_secs"; pub const AUDIT_LOG_RETENTION_DAYS_SETTING: &str = "audit_log_retention_days"; pub const MONITOR_LOGS_ON_OBJECT_STORE_SETTING: &str = "monitor_logs_on_s3"; @@ -57,11 +59,20 @@ pub const EMAIL_DOMAIN_SETTING: &str = "email_domain"; pub const OTEL_SETTING: &str = "otel"; pub const OTEL_TRACING_PROXY_SETTING: &str = "otel_tracing_proxy"; pub const APP_WORKSPACED_ROUTE_SETTING: &str = "app_workspaced_route"; +pub const HTTP_ROUTE_WORKSPACED_ROUTE_SETTING: &str = "http_route_workspaced_route"; pub const SECRET_BACKEND_SETTING: &str = "secret_backend"; pub const MIN_KEEP_ALIVE_VERSION_SETTING: &str = "min_keep_alive_version"; pub const GITHUB_ENTERPRISE_APP_SETTING: &str = "github_enterprise_app"; pub const INSTANCE_EVENTS_WEBHOOK_SETTING: &str = "instance_events_webhook"; pub const WORKSPACE_REGISTRIES_SETTING: &str = "workspace_registries"; +pub const RESTART_COORDINATION_SETTING: &str = "_restart_coordination"; + +use std::sync::Arc; +use tokio::sync::RwLock; + +lazy_static::lazy_static! { + pub static ref HTTP_ROUTE_WORKSPACED_ROUTE: Arc> = Arc::new(RwLock::new(false)); +} pub const ENV_SETTINGS: &[&str] = &[ "DISABLE_NSJAIL", diff --git a/backend/windmill-common/src/instance_config.rs b/backend/windmill-common/src/instance_config.rs index 6e791906f9..3023a5ab33 100644 --- a/backend/windmill-common/src/instance_config.rs +++ b/backend/windmill-common/src/instance_config.rs @@ -237,10 +237,14 @@ pub struct GlobalSettings { #[serde(skip_serializing_if = "Option::is_none")] pub app_workspaced_route: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub http_route_workspaced_route: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub no_default_maven: Option, #[serde(skip_serializing_if = "Option::is_none")] pub default_tags_per_workspace: Option, #[serde(skip_serializing_if = "Option::is_none")] + pub preview_tags_override: Option, + #[serde(skip_serializing_if = "Option::is_none")] pub disable_hub: Option, // String settings @@ -351,6 +355,13 @@ pub struct GlobalSettings { std::collections::HashMap>, >, + #[serde(skip_serializing_if = "Option::is_none")] + #[cfg_attr( + feature = "instance_config_schema", + schemars(schema_with = "opaque_json_schema") + )] + pub ai_config: Option, + /// Catch-all for settings not yet covered by typed fields. #[serde(flatten)] pub extra: BTreeMap, @@ -861,6 +872,7 @@ pub const HIDDEN_SETTINGS: &[&str] = &[ "uid", "min_keep_alive_version", "automate_username_creation", + "_restart_coordination", ]; /// Top-level settings whose entire value is sensitive and must be fully redacted in logs. diff --git a/backend/windmill-common/src/jobs.rs b/backend/windmill-common/src/jobs.rs index 583914c573..d5521288bd 100644 --- a/backend/windmill-common/src/jobs.rs +++ b/backend/windmill-common/src/jobs.rs @@ -14,6 +14,7 @@ use crate::{ client::AuthedClient, db::{AuthedRef, UserDbWithAuthed, DB}, error::{self, to_anyhow, Error}, + flows::get_full_hub_flow_by_path, get_latest_deployed_hash_for_path, get_latest_flow_version_info_for_path, scripts::{get_full_hub_script_by_path, ScriptHash, ScriptLang}, users::username_to_permissioned_as, @@ -154,15 +155,31 @@ pub async fn get_payload_tag_from_prefixed_path( .await? } else if path.starts_with("flow/") { let path = path.strip_prefix("flow/").unwrap().to_string(); - let FlowVersionInfo { dedicated_worker, tag, version, .. } = - get_latest_flow_version_info_for_path(None, &db, w_id, &path, true).await?; - ( - JobPayload::Flow { path, dedicated_worker, apply_preprocessor: false, version }, - tag, - None, - None, - None, - ) + if path.starts_with("hub/flows/") { + let hub_flow = + get_full_hub_flow_by_path(StripPath(path.clone()), &HTTP_CLIENT, Some(db)).await?; + ( + JobPayload::RawFlow { + value: hub_flow.value, + path: Some(path), + restarted_from: None, + }, + None, + None, + None, + None, + ) + } else { + let FlowVersionInfo { dedicated_worker, tag, version, .. } = + get_latest_flow_version_info_for_path(None, &db, w_id, &path, true).await?; + ( + JobPayload::Flow { path, dedicated_worker, apply_preprocessor: false, version }, + tag, + None, + None, + None, + ) + } } else { return Err(Error::BadRequest(format!( "path must start with script/ or flow/ (got {})", diff --git a/backend/windmill-common/src/lib.rs b/backend/windmill-common/src/lib.rs index ea871d6e25..886521b42c 100644 --- a/backend/windmill-common/src/lib.rs +++ b/backend/windmill-common/src/lib.rs @@ -29,6 +29,7 @@ use sqlx::{Acquire, Postgres}; pub mod agent_workers; #[cfg(feature = "bedrock")] pub mod ai_bedrock; +pub mod ai_cache; pub mod ai_google; pub mod ai_providers; pub mod ai_types; @@ -68,6 +69,7 @@ pub mod git_sync_ee; pub mod git_sync_oss; pub mod jobs; pub mod jwt; +pub mod login_rate_limit; pub mod more_serde; pub mod oauth2; #[cfg(all(feature = "enterprise", feature = "openidconnect", feature = "private"))] @@ -85,6 +87,7 @@ pub mod schedule; pub mod schema; pub mod scripts; pub mod secret_backend; +pub mod sensitive_log_masks; pub mod server; pub mod ssrf; #[cfg(feature = "private")] @@ -95,6 +98,7 @@ pub mod stream; pub mod teams_ee; pub mod teams_oss; pub mod tracing_init; +pub mod trashbin; pub mod triggers; pub mod usernames; pub mod users; @@ -403,6 +407,8 @@ pub struct PgDatabase { pub sslmode: Option, pub dbname: String, pub root_certificate_pem: Option, + pub use_iam_auth: Option, + pub region: Option, } // Wrapper enum to hold either Tls or NoTls connection @@ -510,6 +516,75 @@ impl PgDatabase { } } + #[cfg(all(feature = "enterprise", feature = "private"))] + pub async fn connect_with_iam( + &self, + ) -> Result<(tokio_postgres::Client, TokioPgConnection), error::Error> { + use native_tls::TlsConnector; + use postgres_native_tls::MakeTlsConnector; + + // Resolve region: resource field takes priority, then env var + let region = match self.region.as_deref() { + Some(r) => r.to_string(), + None => std::env::var("AWS_REGION").map_err(|_| { + error::Error::BadConfig( + "Region is required for IAM RDS auth. Set 'region' on the resource or AWS_REGION env var".to_string(), + ) + })?, + }; + + let port = self.port.unwrap_or(5432); + let user = self.user.as_deref().unwrap_or("postgres"); + + let token = db_iam_ee::generate_auth_token(®ion, &self.host, port as u64, user) + .await + .map_err(|e| { + error::Error::InternalErr(format!("IAM token generation failed: {e:#}")) + })?; + + // RDS IAM auth requires SSL + let mut connector = TlsConnector::builder(); + if let Some(root_certificate_pem) = &self.root_certificate_pem { + if !root_certificate_pem.is_empty() { + connector.add_root_certificate( + native_tls::Certificate::from_pem(root_certificate_pem.as_bytes()) + .map_err(|e| error::Error::BadConfig(format!("Invalid Certs: {e:#}")))?, + ); + } else { + connector.danger_accept_invalid_certs(true); + connector.danger_accept_invalid_hostnames(true); + } + } else { + tracing::warn!("IAM RDS auth without root certificate: TLS certificate verification is disabled. Consider providing root_certificate_pem for production use."); + connector + .danger_accept_invalid_certs(true) + .danger_accept_invalid_hostnames(true); + } + + tracing::info!("Creating new IAM RDS connection to {}", &self.host); + + // Use Config builder directly to pass the IAM token as the password. + // This avoids needing to URL-encode the token into a connection string. + let mut config = tokio_postgres::Config::new(); + config + .host(&self.host) + .port(port as u16) + .user(user) + .password(&token) + .dbname(&self.dbname) + .ssl_mode(tokio_postgres::config::SslMode::Require); + + let (client, connection) = tokio::time::timeout( + std::time::Duration::from_secs(20), + config.connect(MakeTlsConnector::new(connector.build().map_err(to_anyhow)?)), + ) + .await + .map_err(to_anyhow)? + .map_err(to_anyhow)?; + + Ok((client, TokioPgConnection::Tls(connection))) + } + pub fn parse_uri(url: &str) -> Result { let parsed_url = url::Url::parse(url) .map_err(|_| Error::BadConfig("Invalid PostgreSQL URL".to_string()))?; @@ -548,6 +623,8 @@ impl PgDatabase { dbname, sslmode, root_certificate_pem: None, + use_iam_auth: None, + region: None, }) } } diff --git a/backend/windmill-common/src/login_rate_limit.rs b/backend/windmill-common/src/login_rate_limit.rs new file mode 100644 index 0000000000..aede8df2e4 --- /dev/null +++ b/backend/windmill-common/src/login_rate_limit.rs @@ -0,0 +1,206 @@ +use chrono::Utc; +use dashmap::DashMap; +use hyper::StatusCode; +use std::sync::atomic::{AtomicI32, AtomicI64, AtomicU64, Ordering}; +use std::sync::LazyLock; + +use crate::error::{Error, Result}; +use crate::worker::CLOUD_HOSTED; + +const DEFAULT_PER_IP_LIMIT: i32 = 120; +const DEFAULT_PER_ACCOUNT_LIMIT: i32 = 30; +const DEFAULT_GLOBAL_LIMIT: i32 = 10000; +const EVICTION_INTERVAL: u64 = 256; + +struct RateLimitEntry { + count: i32, + minute_bucket: i64, +} + +static IP_RATE_LIMIT: LazyLock> = LazyLock::new(DashMap::new); +static ACCOUNT_RATE_LIMIT: LazyLock> = LazyLock::new(DashMap::new); + +static GLOBAL_COUNT: AtomicI32 = AtomicI32::new(0); +static GLOBAL_MINUTE: AtomicI64 = AtomicI64::new(0); + +static EVICTION_COUNTER: AtomicU64 = AtomicU64::new(0); + +static PER_IP_LIMIT: LazyLock = LazyLock::new(|| { + std::env::var("LOGIN_RATE_LIMIT_PER_IP") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_PER_IP_LIMIT) +}); + +static PER_IP_LIMIT_EXPLICIT: LazyLock = LazyLock::new(|| { + std::env::var("LOGIN_RATE_LIMIT_PER_IP") + .ok() + .and_then(|v| v.parse::().ok()) + .is_some() +}); + +static PER_ACCOUNT_LIMIT: LazyLock = LazyLock::new(|| { + std::env::var("LOGIN_RATE_LIMIT_PER_ACCOUNT") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_PER_ACCOUNT_LIMIT) +}); + +static PER_ACCOUNT_LIMIT_EXPLICIT: LazyLock = LazyLock::new(|| { + std::env::var("LOGIN_RATE_LIMIT_PER_ACCOUNT") + .ok() + .and_then(|v| v.parse::().ok()) + .is_some() +}); + +static GLOBAL_LIMIT: LazyLock = LazyLock::new(|| { + std::env::var("LOGIN_RATE_LIMIT_GLOBAL") + .ok() + .and_then(|v| v.parse().ok()) + .unwrap_or(DEFAULT_GLOBAL_LIMIT) +}); + +/// Extract client IP from proxy headers. Only meaningful when behind a trusted +/// reverse proxy (e.g. CLOUD_HOSTED). Returns `None` if no proxy header is present. +pub fn extract_client_ip(headers: &axum::http::HeaderMap) -> Option { + if let Some(real_ip) = headers.get("x-real-ip") { + if let Ok(ip) = real_ip.to_str() { + let trimmed = ip.trim(); + if !trimmed.is_empty() { + return Some(trimmed.to_string()); + } + } + } + + if let Some(forwarded_for) = headers.get("x-forwarded-for") { + if let Ok(ips) = forwarded_for.to_str() { + if let Some(first_ip) = ips.split(',').next() { + let trimmed = first_ip.trim(); + if !trimmed.is_empty() { + return Some(trimmed.to_string()); + } + } + } + } + + None +} + +fn maybe_evict(maps: &[&DashMap], current_minute: i64) { + let count = EVICTION_COUNTER.fetch_add(1, Ordering::Relaxed); + if count % EVICTION_INTERVAL == 0 { + for map in maps { + map.retain(|_, v| v.minute_bucket >= current_minute - 1); + } + } +} + +/// Atomically check the rate limit and increment the counter. Follows the +/// `public_app_rate_limit.rs` pattern — the DashMap entry lock is held across +/// both the check and the increment, preventing TOCTOU races. +fn check_and_increment( + map: &DashMap, + key: &str, + limit: i32, + current_minute: i64, +) -> Result<()> { + let mut entry = map + .entry(key.to_string()) + .or_insert(RateLimitEntry { count: 0, minute_bucket: current_minute }); + + if entry.minute_bucket != current_minute { + entry.count = 0; + entry.minute_bucket = current_minute; + } + + if entry.count >= limit { + return Err(Error::Generic( + StatusCode::TOO_MANY_REQUESTS, + "Too many login attempts. Please try again later.".to_string(), + )); + } + + entry.count += 1; + Ok(()) +} + +fn record_failure(map: &DashMap, key: &str) { + let current_minute = Utc::now().timestamp() / 60; + + let mut entry = map + .entry(key.to_string()) + .or_insert(RateLimitEntry { count: 0, minute_bucket: current_minute }); + + if entry.minute_bucket != current_minute { + entry.count = 1; + entry.minute_bucket = current_minute; + } else { + entry.count += 1; + } +} + +/// Called BEFORE authentication. Checks and increments global + per-IP counters. +/// The global counter counts all login attempts (not just failures), so it acts as +/// a general throttle on login traffic per server instance. +/// Per-IP is only active on CLOUD_HOSTED or when LOGIN_RATE_LIMIT_PER_IP is explicitly set. +pub fn check_and_increment_login_attempt( + headers: &axum::http::HeaderMap, + email: &str, +) -> Result<()> { + let current_minute = Utc::now().timestamp() / 60; + maybe_evict(&[&IP_RATE_LIMIT, &ACCOUNT_RATE_LIMIT], current_minute); + + // Global limit: always on, uses atomics (single key, no need for DashMap) + check_and_increment_global(current_minute)?; + + // Per-IP limit: CLOUD_HOSTED or explicit opt-in + if *CLOUD_HOSTED || *PER_IP_LIMIT_EXPLICIT { + if let Some(ip) = extract_client_ip(headers) { + check_and_increment(&IP_RATE_LIMIT, &ip, *PER_IP_LIMIT, current_minute)?; + } + } + + // Per-account check (read-only, does not increment — failures are recorded separately) + if *CLOUD_HOSTED || *PER_ACCOUNT_LIMIT_EXPLICIT { + let entry = ACCOUNT_RATE_LIMIT.get(email); + if let Some(entry) = entry { + if entry.minute_bucket == current_minute && entry.count >= *PER_ACCOUNT_LIMIT { + return Err(Error::Generic( + StatusCode::TOO_MANY_REQUESTS, + "Too many login attempts. Please try again later.".to_string(), + )); + } + } + } + + Ok(()) +} + +fn check_and_increment_global(current_minute: i64) -> Result<()> { + let stored_minute = GLOBAL_MINUTE.load(Ordering::Relaxed); + if stored_minute != current_minute { + // Minute rolled over — reset. Race here is benign: worst case two threads + // both reset, and we lose a few counts at the boundary. + GLOBAL_MINUTE.store(current_minute, Ordering::Relaxed); + GLOBAL_COUNT.store(1, Ordering::Relaxed); + return Ok(()); + } + + let count = GLOBAL_COUNT.fetch_add(1, Ordering::Relaxed); + if count >= *GLOBAL_LIMIT { + return Err(Error::Generic( + StatusCode::TOO_MANY_REQUESTS, + "Too many login attempts. Please try again later.".to_string(), + )); + } + + Ok(()) +} + +/// Called AFTER authentication failure. Records per-account failure. +/// Per-account is only active on CLOUD_HOSTED or when LOGIN_RATE_LIMIT_PER_ACCOUNT is explicitly set. +pub fn record_login_failure(email: &str) { + if *CLOUD_HOSTED || *PER_ACCOUNT_LIMIT_EXPLICIT { + record_failure(&ACCOUNT_RATE_LIMIT, email); + } +} diff --git a/backend/windmill-common/src/otel_oss.rs b/backend/windmill-common/src/otel_oss.rs index 3710464607..27c7101dc0 100644 --- a/backend/windmill-common/src/otel_oss.rs +++ b/backend/windmill-common/src/otel_oss.rs @@ -59,7 +59,7 @@ pub(crate) fn init_otlp_tracer( _mode: &Mode, _hostname: &str, _env: &str, -) -> Option { +) -> Option { None } diff --git a/backend/windmill-common/src/query_builders.rs b/backend/windmill-common/src/query_builders.rs index c52707823e..6d2c9519a5 100644 --- a/backend/windmill-common/src/query_builders.rs +++ b/backend/windmill-common/src/query_builders.rs @@ -39,7 +39,14 @@ fn deserialize_string_from_null<'de, D>(deserializer: D) -> Result, { - Option::::deserialize(deserializer).map(|v| v.unwrap_or_default()) + // DuckDB may return booleans for fields that other databases return as strings + let v = serde_json::Value::deserialize(deserializer)?; + match v { + serde_json::Value::Null => Ok(String::new()), + serde_json::Value::String(s) => Ok(s), + serde_json::Value::Bool(b) => Ok(b.to_string()), + other => Ok(other.to_string()), + } } fn deserialize_column_identity_from_null<'de, D>( @@ -49,15 +56,21 @@ where D: Deserializer<'de>, { // MySQL returns uppercase "YES"/"NO" while the enum expects title case. - let v = Option::::deserialize(deserializer)?; - match v.as_deref() { - None => Ok(ColumnIdentity::default()), - Some(s) => match s.to_lowercase().as_str() { + // DuckDB returns a boolean false instead of a string. + let v = serde_json::Value::deserialize(deserializer)?; + match v { + serde_json::Value::Null => Ok(ColumnIdentity::default()), + serde_json::Value::Bool(_) => Ok(ColumnIdentity::No), + serde_json::Value::String(s) => match s.to_lowercase().as_str() { "no" => Ok(ColumnIdentity::No), "yes" | "always" => Ok(ColumnIdentity::Always), "by default" => Ok(ColumnIdentity::ByDefault), _ => Ok(ColumnIdentity::No), }, + _ => Err(serde::de::Error::custom(format!( + "expected string, bool, or null for isidentity, got {}", + v + ))), } } @@ -2369,7 +2382,7 @@ fn make_load_table_metadata_query( COLUMN_DEFAULT as DefaultValue, false as IsPrimaryKey, false as IsIdentity, - IS_NULLABLE as IsNullable, + CASE WHEN IS_NULLABLE = true THEN 'YES' ELSE 'NO' END as IsNullable, false as IsEnum, TABLE_NAME as table_name FROM information_schema.columns c diff --git a/backend/windmill-common/src/scripts.rs b/backend/windmill-common/src/scripts.rs index d3035b1b49..50790fe65b 100644 --- a/backend/windmill-common/src/scripts.rs +++ b/backend/windmill-common/src/scripts.rs @@ -428,6 +428,7 @@ pub async fn clone_script<'c>( preserve_on_behalf_of: None, assets: s.assets, modules: s.modules, + auto_parent: None, }; let new_hash = hash_script(&ns); diff --git a/backend/windmill-common/src/sensitive_log_masks.rs b/backend/windmill-common/src/sensitive_log_masks.rs new file mode 100644 index 0000000000..b6f6262b77 --- /dev/null +++ b/backend/windmill-common/src/sensitive_log_masks.rs @@ -0,0 +1,167 @@ +//! In-memory store for masking sensitive values (secrets, password args) in job logs. +//! +//! Workers run an embedded server in the same process, so we use global state to track: +//! - Which jobs are currently running +//! - Which secret values each job should mask in its stdout +//! +//! When a secret is fetched via `get_value_internal` (embedded server handler), we don't know +//! which job triggered the request (auth is user-based, not job-based), so we register the +//! secret for ALL currently running jobs on this worker process. + +use std::borrow::Cow; +use std::collections::{HashMap, HashSet}; +use std::sync::RwLock; +use uuid::Uuid; + +/// Minimum length for a secret to be registered for masking. +/// Short strings (e.g. "true", "1234") would cause too many false positives. +const MIN_SECRET_LENGTH: usize = 8; + +const MASKED_NOTICE: &str = + "[windmill] secret value was masked for security reasons, use string transformations to display full value"; + +lazy_static::lazy_static! { + /// Map of job_id -> set of secret values that should be masked in that job's logs. + static ref SENSITIVE_MASKS: RwLock>> = + RwLock::new(HashMap::new()); + + /// Set of currently running job IDs on this worker process. + static ref RUNNING_JOBS: RwLock> = + RwLock::new(HashSet::new()); + +} + +/// A lock-free snapshot of secrets for a job, taken once per log batch. +/// Uses Aho-Corasick for O(m) multi-pattern matching in a single pass, +/// regardless of the number of secrets registered. +pub struct MaskSnapshot { + /// Aho-Corasick automaton for fast matching. + ac: aho_corasick::AhoCorasick, + /// Replacement strings, indexed to match the automaton's pattern order. + replacements: Vec, + /// Whether the security notice has already been appended for this snapshot. + /// Tracked locally to avoid a global write lock on every masked line. + notice_shown: std::cell::Cell, +} + +impl MaskSnapshot { + /// Mask all secrets in `text`. Returns `Cow::Borrowed` when no match (zero allocation). + /// The Aho-Corasick scan is O(text_len) regardless of how many secrets are registered. + pub fn mask<'a>(&self, text: &'a str) -> Cow<'a, str> { + if text.is_empty() { + return Cow::Borrowed(text); + } + + // Single-pass check + replace using the pre-built automaton + if !self.ac.is_match(text) { + return Cow::Borrowed(text); + } + + let mut result = self.ac.replace_all(text, &self.replacements); + + // Append the notice only once per snapshot (i.e. per batch) + if !self.notice_shown.get() { + self.notice_shown.set(true); + result.push('\n'); + result.push_str(MASKED_NOTICE); + } + + Cow::Owned(result) + } +} + +/// Take a snapshot of the current secrets for a job. Returns `None` if no secrets +/// are registered (the caller can then skip masking entirely for the whole batch). +/// +/// Call this once per log batch in `write_lines`, not per line. +pub fn snapshot(job_id: &Uuid) -> Option { + let masks = SENSITIVE_MASKS.read().unwrap_or_else(|e| e.into_inner()); + let secrets = masks.get(job_id)?; + if secrets.is_empty() { + return None; + } + + // Sort longest-first so longer secrets are matched before shorter substrings + let mut sorted: Vec<&String> = secrets.iter().collect(); + sorted.sort_by(|a, b| b.len().cmp(&a.len())); + + let replacements: Vec = sorted + .iter() + .map(|s| { + let char_count = s.chars().count(); + if char_count > 20 { + let prefix: String = s.chars().take(3).collect(); + let suffix: String = s.chars().skip(char_count - 3).collect(); + format!("{}*****{}", prefix, suffix) + } else { + let first: String = s.chars().take(1).collect(); + let last: String = s.chars().skip(char_count - 1).collect(); + format!("{}*****{}", first, last) + } + }) + .collect(); + + let ac = aho_corasick::AhoCorasickBuilder::new() + .match_kind(aho_corasick::MatchKind::LeftmostLongest) + .build(sorted.iter().map(|s| s.as_str())) + .expect("failed to build aho-corasick automaton"); + + Some(MaskSnapshot { ac, replacements, notice_shown: std::cell::Cell::new(false) }) +} + +/// Register a job as currently running. Call this before `handle_queued_job`. +pub fn register_running_job(job_id: Uuid) { + { + let mut jobs = RUNNING_JOBS.write().unwrap_or_else(|e| e.into_inner()); + jobs.insert(job_id); + } + { + let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner()); + masks.entry(job_id).or_default(); + } +} + +/// Unregister a job when it completes. Removes both the running job entry and its mask set. +pub fn unregister_running_job(job_id: Uuid) { + { + let mut jobs = RUNNING_JOBS.write().unwrap_or_else(|e| e.into_inner()); + jobs.remove(&job_id); + } + { + let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner()); + masks.remove(&job_id); + } +} + +/// Register a secret value for ALL currently running jobs. +/// Used when a secret is fetched via the embedded server (we don't know which job triggered it). +pub fn register_secret_for_all_running_jobs(secret: &str) { + if secret.len() < MIN_SECRET_LENGTH { + return; + } + let jobs = RUNNING_JOBS.read().unwrap_or_else(|e| e.into_inner()); + if jobs.is_empty() { + return; + } + let job_ids: Vec = jobs.iter().copied().collect(); + drop(jobs); + + let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner()); + for job_id in job_ids { + if let Some(set) = masks.get_mut(&job_id) { + set.insert(secret.to_string()); + } + } +} + +/// Register a secret value for a specific job. +/// Used for `$encrypted:` args where we know the job ID. +pub fn register_secret_for_job(job_id: Uuid, secret: &str) { + if secret.len() < MIN_SECRET_LENGTH { + return; + } + let mut masks = SENSITIVE_MASKS.write().unwrap_or_else(|e| e.into_inner()); + if let Some(set) = masks.get_mut(&job_id) { + set.insert(secret.to_string()); + } +} diff --git a/backend/windmill-common/src/tracing_init.rs b/backend/windmill-common/src/tracing_init.rs index 3b009a36eb..1a2ca33518 100644 --- a/backend/windmill-common/src/tracing_init.rs +++ b/backend/windmill-common/src/tracing_init.rs @@ -273,3 +273,4 @@ where } } } + diff --git a/backend/windmill-common/src/trashbin.rs b/backend/windmill-common/src/trashbin.rs new file mode 100644 index 0000000000..6d88f12871 --- /dev/null +++ b/backend/windmill-common/src/trashbin.rs @@ -0,0 +1,155 @@ +use serde::Serialize; +use sqlx::PgConnection; + +use crate::error::Result; + +#[derive(Serialize, sqlx::FromRow)] +pub struct TrashItem { + pub id: i64, + pub workspace_id: String, + pub item_kind: String, + pub item_path: String, + pub deleted_by: String, + pub deleted_at: chrono::DateTime, + pub expires_at: chrono::DateTime, +} + +#[derive(Serialize, sqlx::FromRow)] +pub struct TrashItemWithData { + pub id: i64, + pub workspace_id: String, + pub item_kind: String, + pub item_path: String, + pub item_data: serde_json::Value, + pub deleted_by: String, + pub deleted_at: chrono::DateTime, + pub expires_at: chrono::DateTime, +} + +pub async fn move_to_trash( + tx: &mut PgConnection, + workspace_id: &str, + item_kind: &str, + item_path: &str, + item_data: serde_json::Value, + deleted_by: &str, +) -> Result { + let id = sqlx::query_scalar!( + "INSERT INTO trashbin (workspace_id, item_kind, item_path, item_data, deleted_by) + VALUES ($1, $2, $3, $4, $5) RETURNING id", + workspace_id, + item_kind, + item_path, + item_data, + deleted_by, + ) + .fetch_one(&mut *tx) + .await?; + + Ok(id) +} + +pub async fn list_trash<'e, E: sqlx::PgExecutor<'e>>( + db: E, + workspace_id: &str, + kind_filter: Option<&str>, + page: Option, + per_page: Option, +) -> Result> { + let per_page = per_page.unwrap_or(100).min(1000); + let offset = page.unwrap_or(0) * per_page; + + let items = if let Some(kind) = kind_filter { + sqlx::query_as!( + TrashItem, + "SELECT id, workspace_id, item_kind, item_path, deleted_by, deleted_at, expires_at + FROM trashbin + WHERE workspace_id = $1 AND item_kind = $2 + ORDER BY deleted_at DESC + LIMIT $3 OFFSET $4", + workspace_id, + kind, + per_page, + offset, + ) + .fetch_all(db) + .await? + } else { + sqlx::query_as!( + TrashItem, + "SELECT id, workspace_id, item_kind, item_path, deleted_by, deleted_at, expires_at + FROM trashbin + WHERE workspace_id = $1 + ORDER BY deleted_at DESC + LIMIT $2 OFFSET $3", + workspace_id, + per_page, + offset, + ) + .fetch_all(db) + .await? + }; + + Ok(items) +} + +pub async fn get_trash_item<'e, E: sqlx::PgExecutor<'e>>( + db: E, + workspace_id: &str, + id: i64, +) -> Result { + let item = sqlx::query_as!( + TrashItemWithData, + "SELECT id, workspace_id, item_kind, item_path, item_data, deleted_by, deleted_at, expires_at + FROM trashbin + WHERE workspace_id = $1 AND id = $2", + workspace_id, + id, + ) + .fetch_optional(db) + .await? + .ok_or_else(|| crate::error::Error::NotFound("Trash item not found".to_string()))?; + + Ok(item) +} + +pub async fn permanently_delete_item<'e, E: sqlx::PgExecutor<'e>>( + db: E, + workspace_id: &str, + id: i64, +) -> Result<()> { + let rows = sqlx::query!( + "DELETE FROM trashbin WHERE workspace_id = $1 AND id = $2", + workspace_id, + id, + ) + .execute(db) + .await? + .rows_affected(); + + if rows == 0 { + return Err(crate::error::Error::NotFound( + "Trash item not found".to_string(), + )); + } + + Ok(()) +} + +pub async fn empty_trash<'e, E: sqlx::PgExecutor<'e>>(db: E, workspace_id: &str) -> Result { + let rows = sqlx::query!("DELETE FROM trashbin WHERE workspace_id = $1", workspace_id,) + .execute(db) + .await? + .rows_affected(); + + Ok(rows as i64) +} + +pub async fn delete_expired_trash<'e, E: sqlx::PgExecutor<'e>>(db: E) -> Result { + let rows = sqlx::query!("DELETE FROM trashbin WHERE expires_at <= now()") + .execute(db) + .await? + .rows_affected(); + + Ok(rows as i64) +} diff --git a/backend/windmill-common/src/users.rs b/backend/windmill-common/src/users.rs index 453466ac7d..f91c21701a 100644 --- a/backend/windmill-common/src/users.rs +++ b/backend/windmill-common/src/users.rs @@ -81,6 +81,46 @@ pub async fn get_email_from_permissioned_as( } } +/// Compute the highest-precedence workspace role for a user across all their instance groups. +/// +/// Precedence: admin (3) > developer (2) > operator (1). +/// Returns `(best_group_name, is_admin, is_operator)`. +pub fn compute_highest_workspace_role( + user_igroups: &[String], + ws_configured_groups: &[String], + ws_roles: &std::collections::HashMap, +) -> (String, bool, bool) { + let mut best_group = String::new(); + let mut best_precedence = 0u8; + + for group in user_igroups { + if !ws_configured_groups.contains(group) { + continue; + } + let default_role = "developer".to_string(); + let role = ws_roles.get(group).unwrap_or(&default_role); + let precedence = match role.as_str() { + "admin" => 3u8, + "operator" => 1, + _ => 2, + }; + if precedence > best_precedence { + best_precedence = precedence; + best_group = group.clone(); + } + } + + let default_role = "developer".to_string(); + let best_role_str = ws_roles.get(&best_group).unwrap_or(&default_role); + let (is_admin, is_operator) = match best_role_str.as_str() { + "admin" => (true, false), + "operator" => (false, true), + _ => (false, false), + }; + + (best_group, is_admin, is_operator) +} + pub fn truncate_token(token: &str) -> String { if token.len() > 10 { let mut s = token[..10].to_owned(); @@ -105,4 +145,63 @@ mod tests { assert_eq!(username_to_permissioned_as("group-all"), "g/all"); assert_eq!(username_to_permissioned_as("group-my-team"), "g/my-team"); } + + #[test] + fn test_compute_highest_workspace_role_admin_wins() { + let user_groups = vec!["ops".to_string(), "admins".to_string()]; + let ws_groups = vec!["ops".to_string(), "admins".to_string()]; + let mut roles = std::collections::HashMap::new(); + roles.insert("ops".to_string(), "operator".to_string()); + roles.insert("admins".to_string(), "admin".to_string()); + + let (group, is_admin, is_operator) = + compute_highest_workspace_role(&user_groups, &ws_groups, &roles); + assert_eq!(group, "admins"); + assert!(is_admin); + assert!(!is_operator); + } + + #[test] + fn test_compute_highest_workspace_role_developer_over_operator() { + let user_groups = vec!["devs".to_string(), "ops".to_string()]; + let ws_groups = vec!["devs".to_string(), "ops".to_string()]; + let mut roles = std::collections::HashMap::new(); + roles.insert("devs".to_string(), "developer".to_string()); + roles.insert("ops".to_string(), "operator".to_string()); + + let (group, is_admin, is_operator) = + compute_highest_workspace_role(&user_groups, &ws_groups, &roles); + assert_eq!(group, "devs"); + assert!(!is_admin); + assert!(!is_operator); + } + + #[test] + fn test_compute_highest_workspace_role_skips_unconfigured_groups() { + let user_groups = vec!["admins".to_string(), "other".to_string()]; + let ws_groups = vec!["ops".to_string()]; // admins not configured for this workspace + let mut roles = std::collections::HashMap::new(); + roles.insert("admins".to_string(), "admin".to_string()); + roles.insert("ops".to_string(), "operator".to_string()); + + let (group, is_admin, is_operator) = + compute_highest_workspace_role(&user_groups, &ws_groups, &roles); + // No user groups match ws_configured_groups, so best_group stays empty + assert_eq!(group, ""); + assert!(!is_admin); + assert!(!is_operator); + } + + #[test] + fn test_compute_highest_workspace_role_defaults_to_developer() { + let user_groups = vec!["team".to_string()]; + let ws_groups = vec!["team".to_string()]; + let roles = std::collections::HashMap::new(); // no role configured → developer + + let (group, is_admin, is_operator) = + compute_highest_workspace_role(&user_groups, &ws_groups, &roles); + assert_eq!(group, "team"); + assert!(!is_admin); + assert!(!is_operator); + } } diff --git a/backend/windmill-common/src/variables.rs b/backend/windmill-common/src/variables.rs index 8129f39b3f..e7595f9d1b 100644 --- a/backend/windmill-common/src/variables.rs +++ b/backend/windmill-common/src/variables.rs @@ -140,6 +140,24 @@ pub async fn get_workspace_key(w_id: &str, db: &DB) -> crate::error::Result crate::error::Result { + use hmac::{Hmac, Mac}; + use sha2::Sha256; + let key = get_workspace_key(w_id, db).await?; + let mut mac = Hmac::::new_from_slice(key.as_bytes()) + .map_err(|e| crate::Error::internal_err(format!("HMAC key error: {e}")))?; + mac.update(job_id.as_bytes()); + mac.update(b"approval_token"); + Ok(hex::encode(mac.finalize().into_bytes())) +} + pub async fn get_secret_value_as_admin( db: &DB, w_id: &str, diff --git a/backend/windmill-common/src/webhook.rs b/backend/windmill-common/src/webhook.rs index caf8ec08fc..d695ba1f5a 100644 --- a/backend/windmill-common/src/webhook.rs +++ b/backend/windmill-common/src/webhook.rs @@ -233,6 +233,9 @@ impl WebhookShared { } pub fn send_message(&self, workspace_id: String, message: WebhookMessage) { + if *crate::worker::CLOUD_HOSTED { + return; + } let _ = self.channel.send(WebhookPayload::WorkspaceEvent( workspace_id.clone(), message, diff --git a/backend/windmill-common/src/worker.rs b/backend/windmill-common/src/worker.rs index 8b20d331c0..026f60da75 100644 --- a/backend/windmill-common/src/worker.rs +++ b/backend/windmill-common/src/worker.rs @@ -205,6 +205,7 @@ lazy_static::lazy_static! { pub static ref DEFAULT_TAGS_PER_WORKSPACE: AtomicBool = AtomicBool::new(false); pub static ref DEFAULT_TAGS_WORKSPACES: Arc>>> = Arc::new(RwLock::new(None)); + pub static ref PREVIEW_TAGS_OVERRIDE: AtomicBool = AtomicBool::new(false); pub static ref MAX_TIMEOUT: u64 = std::env::var("TIMEOUT") .ok() @@ -1531,6 +1532,16 @@ pub fn dedicated_worker_tag(workspace_id: &str, path: &str) -> String { ) } +/// Configuration for a runner group — a single long-lived subprocess +/// that can execute multiple scripts sharing the same workspace dependency. +/// Auto-detected from script content annotations at worker startup. +#[derive(Clone, PartialEq, Debug)] +pub struct RunnerGroupConfig { + pub workspace_id: String, + pub dep_name: String, + pub language: String, +} + pub async fn load_worker_config( db: &DB, killpill_tx: KillpillSender, @@ -1638,7 +1649,7 @@ pub async fn load_worker_config( let worker_tags = config .worker_tags .or_else(|| { - // Check for multiple dedicated workers first + // Check for multiple dedicated workers if let Some(ref dws) = dedicated_workers.as_ref() { let mut dedi_tags: Vec = dws .iter() diff --git a/backend/windmill-common/src/workspaces.rs b/backend/windmill-common/src/workspaces.rs index b7b4ca3b63..c7fde7728d 100644 --- a/backend/windmill-common/src/workspaces.rs +++ b/backend/windmill-common/src/workspaces.rs @@ -149,7 +149,7 @@ pub enum ObjectType { WorkspaceDependencies, } -pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28160/sync-script-to-git-repo-windmill"; +pub const LATEST_GIT_SYNC_SCRIPT_PATH: &str = "hub/28186/sync-script-to-git-repo-windmill"; #[derive(Serialize, Deserialize, Debug)] pub struct GitRepositorySettings { diff --git a/backend/windmill-common/tests/instance_group_auto_add.rs b/backend/windmill-common/tests/instance_group_auto_add.rs index e8a6c841c6..95b20cc842 100644 --- a/backend/windmill-common/tests/instance_group_auto_add.rs +++ b/backend/windmill-common/tests/instance_group_auto_add.rs @@ -28,10 +28,14 @@ mod tests { use serde_json::json; use sqlx::{Pool, Postgres}; + use windmill_common::users::compute_highest_workspace_role; /// Test that configuring instance groups for a workspace auto-adds existing group members #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_configure_instance_groups_adds_existing_members(db: Pool) { // Configure workspace to auto-add users from 'engineering' group with 'developer' role let groups = vec!["engineering".to_string()]; @@ -112,8 +116,14 @@ mod tests { "Alice should be in the workspace" ); let alice = alice_in_workspace.unwrap(); - assert!(!alice.is_admin, "Alice should not be admin (developer role)"); - assert!(!alice.operator, "Alice should not be operator (developer role)"); + assert!( + !alice.is_admin, + "Alice should not be admin (developer role)" + ); + assert!( + !alice.operator, + "Alice should not be operator (developer role)" + ); // Check added_via field let added_via = alice.added_via.expect("added_via should be set"); @@ -158,7 +168,10 @@ mod tests { /// Test role assignment based on instance group configuration #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_role_assignment_admin(db: Pool) { // Configure workspace with admins group having admin role let groups = vec!["admins".to_string()]; @@ -209,7 +222,10 @@ mod tests { /// Test role assignment for operator #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_role_assignment_operator(db: Pool) { // Configure workspace with sales group having operator role let groups = vec!["sales".to_string()]; @@ -260,7 +276,10 @@ mod tests { /// Test role precedence when user is in multiple instance groups #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_role_precedence_multiple_groups(db: Pool) { // Configure workspace with multiple groups: engineering (admin), sales (operator) // Bob is in both groups, should get admin role (highest precedence) @@ -306,7 +325,10 @@ mod tests { .await .expect("Failed to query user"); - assert!(bob.is_admin, "Bob should be admin (highest precedence role)"); + assert!( + bob.is_admin, + "Bob should be admin (highest precedence role)" + ); assert!(!bob.operator, "Bob should not be operator"); // Verify added_via tracks the primary group (engineering, the one with highest precedence) @@ -320,9 +342,305 @@ mod tests { println!("✓ Role precedence works correctly for users in multiple groups"); } + /// Test that adding a user to a second instance group upgrades their workspace role + /// if the new group has a higher-precedence role. + /// This is a regression test for the bug where only the newly-added group's role was used. + #[ignore = "requires database setup - run with --ignored flag"] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] + async fn test_role_upgrade_when_added_to_higher_group(db: Pool) { + // Configure workspace: engineering=operator, admins=admin + let groups = vec!["engineering".to_string(), "admins".to_string()]; + let roles = json!({"engineering": "operator", "admins": "admin"}); + + sqlx::query!( + r#" + UPDATE workspace_settings + SET auto_invite = jsonb_build_object( + 'instance_groups', $2::jsonb, + 'instance_groups_roles', $3::jsonb + ) + WHERE workspace_id = $1 + "#, + "ws-multi-group", + serde_json::to_value(&groups).unwrap(), + &roles, + ) + .execute(&db) + .await + .expect("Failed to update workspace settings"); + + // Step 1: Alice is added via engineering group (operator) + // (alice is already in engineering from fixture) + let added_via = json!({"source": "instance_group", "group": "engineering"}); + sqlx::query!( + "INSERT INTO usr (workspace_id, username, email, is_admin, operator, added_via) + VALUES ($1, 'alice', 'alice@example.com', false, true, $2)", + "ws-multi-group", + &added_via, + ) + .execute(&db) + .await + .expect("Failed to add user"); + + sqlx::query!( + "INSERT INTO usr_to_group (workspace_id, usr, group_) VALUES ($1, 'alice', 'all')", + "ws-multi-group", + ) + .execute(&db) + .await + .expect("Failed to add user to all group"); + + // Verify initial state: alice is operator + let alice = sqlx::query!( + "SELECT is_admin, operator FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'" + ) + .fetch_one(&db) + .await + .expect("Failed to query user"); + assert!(!alice.is_admin, "Alice should start as non-admin"); + assert!(alice.operator, "Alice should start as operator"); + + // Step 2: Alice is added to admins group + sqlx::query!( + "INSERT INTO email_to_igroup (email, igroup) VALUES ('alice@example.com', 'admins') ON CONFLICT DO NOTHING" + ) + .execute(&db) + .await + .expect("Failed to add to admins group"); + + // Step 3: Simulate the fixed logic — find all user's groups, compute highest role, update + let user_igroups: Vec = sqlx::query_scalar!( + "SELECT igroup FROM email_to_igroup WHERE email = 'alice@example.com'" + ) + .fetch_all(&db) + .await + .expect("Failed to fetch user groups"); + + let ws = sqlx::query!( + r#" + SELECT auto_invite->'instance_groups_roles' as instance_groups_roles, + auto_invite->'instance_groups' as instance_groups_json + FROM workspace_settings WHERE workspace_id = 'ws-multi-group' + "#, + ) + .fetch_one(&db) + .await + .expect("Failed to fetch workspace settings"); + + let ws_roles: std::collections::HashMap = ws + .instance_groups_roles + .and_then(|r| serde_json::from_value(r).ok()) + .unwrap_or_default(); + + let ws_configured_groups: Vec = ws + .instance_groups_json + .and_then(|ig| serde_json::from_value(ig).ok()) + .unwrap_or_default(); + + let (best_group, is_admin, is_operator) = + compute_highest_workspace_role(&user_igroups, &ws_configured_groups, &ws_roles); + + let instance_group_source = json!({ + "source": "instance_group", + "group": &best_group + }); + + sqlx::query!( + "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'", + is_admin, + is_operator, + &instance_group_source, + "ws-multi-group", + "alice@example.com" + ) + .execute(&db) + .await + .expect("Failed to update user role"); + + // Verify: alice should now be admin (highest precedence) + let alice = sqlx::query!( + "SELECT is_admin, operator, added_via FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'" + ) + .fetch_one(&db) + .await + .expect("Failed to query user"); + + assert!(alice.is_admin, "Alice should be upgraded to admin"); + assert!(!alice.operator, "Alice should no longer be operator"); + + let added_via = alice.added_via.expect("added_via should be set"); + assert_eq!( + added_via.get("group").and_then(|v| v.as_str()), + Some("admins"), + "added_via should track the admin group (highest precedence)" + ); + + println!("✓ Role is upgraded when user is added to a higher-precedence group"); + } + + /// Test that adding a user to a lower-precedence group does NOT downgrade their role + #[ignore = "requires database setup - run with --ignored flag"] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] + async fn test_no_role_downgrade_when_added_to_lower_group(db: Pool) { + // Configure workspace: engineering=admin, sales=operator + let groups = vec!["engineering".to_string(), "sales".to_string()]; + let roles = json!({"engineering": "admin", "sales": "operator"}); + + sqlx::query!( + r#" + UPDATE workspace_settings + SET auto_invite = jsonb_build_object( + 'instance_groups', $2::jsonb, + 'instance_groups_roles', $3::jsonb + ) + WHERE workspace_id = $1 + "#, + "ws-multi-group", + serde_json::to_value(&groups).unwrap(), + &roles, + ) + .execute(&db) + .await + .expect("Failed to update workspace settings"); + + // Alice starts as admin from engineering + let added_via = json!({"source": "instance_group", "group": "engineering"}); + sqlx::query!( + "INSERT INTO usr (workspace_id, username, email, is_admin, operator, added_via) + VALUES ($1, 'alice', 'alice@example.com', true, false, $2)", + "ws-multi-group", + &added_via, + ) + .execute(&db) + .await + .expect("Failed to add user"); + + // Now simulate adding alice to sales group (operator — lower precedence) + // The fixed code should keep her as admin + let user_igroups = vec!["engineering".to_string(), "sales".to_string()]; + let ws_configured_groups = vec!["engineering".to_string(), "sales".to_string()]; + let ws_roles: std::collections::HashMap = + serde_json::from_value(roles).unwrap(); + + let (best_group, is_admin, is_operator) = + compute_highest_workspace_role(&user_igroups, &ws_configured_groups, &ws_roles); + + let instance_group_source = json!({"source": "instance_group", "group": &best_group}); + sqlx::query!( + "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'", + is_admin, is_operator, &instance_group_source, "ws-multi-group", "alice@example.com" + ) + .execute(&db) + .await + .expect("Failed to update user role"); + + let alice = sqlx::query!( + "SELECT is_admin, operator, added_via FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'" + ) + .fetch_one(&db) + .await + .expect("Failed to query user"); + + assert!(alice.is_admin, "Alice should remain admin (not downgraded)"); + assert!(!alice.operator, "Alice should not become operator"); + assert_eq!( + alice + .added_via + .unwrap() + .get("group") + .and_then(|v| v.as_str()), + Some("engineering"), + "added_via should still track engineering (highest precedence)" + ); + + println!("✓ Role is NOT downgraded when user is added to a lower-precedence group"); + } + + /// Test that manually-added users are not affected by instance group role updates + #[ignore = "requires database setup - run with --ignored flag"] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] + async fn test_manual_users_not_affected_by_group_role_update(db: Pool) { + // Configure workspace + let groups = vec!["engineering".to_string()]; + let roles = json!({"engineering": "operator"}); + + sqlx::query!( + r#" + UPDATE workspace_settings + SET auto_invite = jsonb_build_object( + 'instance_groups', $2::jsonb, + 'instance_groups_roles', $3::jsonb + ) + WHERE workspace_id = $1 + "#, + "ws-multi-group", + serde_json::to_value(&groups).unwrap(), + &roles, + ) + .execute(&db) + .await + .expect("Failed to update workspace settings"); + + // Alice was manually added as admin (no added_via) + sqlx::query!( + "INSERT INTO usr (workspace_id, username, email, is_admin, operator) + VALUES ('ws-multi-group', 'alice', 'alice@example.com', true, false)", + ) + .execute(&db) + .await + .expect("Failed to add user"); + + // The UPDATE with added_via->>'source' = 'instance_group' filter should NOT match + let instance_group_source = json!({"source": "instance_group", "group": "engineering"}); + let result = sqlx::query!( + "UPDATE usr SET is_admin = $1, operator = $2, added_via = $3 WHERE workspace_id = $4 AND email = $5 AND added_via->>'source' = 'instance_group'", + false, true, &instance_group_source, "ws-multi-group", "alice@example.com" + ) + .execute(&db) + .await + .expect("Failed to execute update"); + + assert_eq!( + result.rows_affected(), + 0, + "UPDATE should not affect manually-added users" + ); + + let alice = sqlx::query!( + "SELECT is_admin, operator, added_via FROM usr WHERE workspace_id = 'ws-multi-group' AND email = 'alice@example.com'" + ) + .fetch_one(&db) + .await + .expect("Failed to query user"); + + assert!(alice.is_admin, "Manually-added admin should remain admin"); + assert!( + !alice.operator, + "Manually-added admin should not become operator" + ); + assert!( + alice.added_via.is_none(), + "added_via should remain NULL for manual users" + ); + + println!("✓ Manually-added users are not affected by instance group role updates"); + } + /// Test removing user from instance group removes them from workspace #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_remove_user_from_instance_group(db: Pool) { // First, add alice to the workspace via engineering group let added_via = json!({"source": "instance_group", "group": "engineering"}); @@ -416,7 +734,10 @@ mod tests { /// Test that users added via domain are not affected by instance group removal #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_domain_added_users_not_affected_by_group_removal(db: Pool) { // Add alice via domain (not instance group) let added_via = json!({"source": "domain", "domain": "example.com"}); @@ -469,7 +790,10 @@ mod tests { /// Test cleanup when instance group is removed from workspace configuration #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_cleanup_removed_instance_groups(db: Pool) { // First, add users via engineering group for (username, email) in &[("alice", "alice@example.com"), ("bob", "bob@example.com")] { @@ -513,12 +837,11 @@ mod tests { // This should trigger cleanup of users added via that group // Get all users in the engineering group - let group_users = sqlx::query_scalar!( - "SELECT email FROM email_to_igroup WHERE igroup = 'engineering'" - ) - .fetch_all(&db) - .await - .expect("Failed to get group users"); + let group_users = + sqlx::query_scalar!("SELECT email FROM email_to_igroup WHERE igroup = 'engineering'") + .fetch_all(&db) + .await + .expect("Failed to get group users"); // Remove users who were added via engineering group for email in group_users { @@ -588,7 +911,10 @@ mod tests { /// Test that users are not duplicated if already in workspace #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_no_duplicate_users(db: Pool) { // Add alice to workspace first (without instance group tracking) sqlx::query!( @@ -636,7 +962,10 @@ mod tests { /// Test workspace without auto-add configured is not affected #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_workspace_without_auto_add_not_affected(db: Pool) { // ws-no-auto-add has no instance_groups configured @@ -672,7 +1001,10 @@ mod tests { /// Test querying workspaces configured with a specific instance group #[ignore = "requires database setup - run with --ignored flag"] - #[sqlx::test(migrations = "../migrations", fixtures("base", "instance_group_auto_add"))] + #[sqlx::test( + migrations = "../migrations", + fixtures("base", "instance_group_auto_add") + )] async fn test_query_workspaces_with_instance_group(db: Pool) { // Configure ws-with-auto-add to use engineering group let groups = vec!["engineering".to_string()]; @@ -708,7 +1040,11 @@ mod tests { .await .expect("Failed to query workspaces"); - assert_eq!(workspaces.len(), 1, "Should find 1 workspace with engineering group"); + assert_eq!( + workspaces.len(), + 1, + "Should find 1 workspace with engineering group" + ); assert_eq!(workspaces[0].workspace_id, "ws-with-auto-add"); // Verify the role configuration is returned correctly diff --git a/backend/windmill-dep-map/src/scoped_dependency_map.rs b/backend/windmill-dep-map/src/scoped_dependency_map.rs index 987cc4d529..fcee6fe5f1 100644 --- a/backend/windmill-dep-map/src/scoped_dependency_map.rs +++ b/backend/windmill-dep-map/src/scoped_dependency_map.rs @@ -329,11 +329,12 @@ SELECT importer_node_id, imported_path, imported_lockfile_hash let mut tx = db.begin().await?; let mut to_process = vec![]; - let mut modules_to_check = flow_data.flow.modules.iter().collect::>(); - if let Some(failure_module) = flow_data.flow.failure_module.as_ref() { + let flow_value = flow_data.value(); + let mut modules_to_check = flow_value.modules.iter().collect::>(); + if let Some(failure_module) = flow_value.failure_module.as_ref() { modules_to_check.push(failure_module.as_ref()); } - if let Some(preprocessor_module) = flow_data.flow.preprocessor_module.as_ref() { + if let Some(preprocessor_module) = flow_value.preprocessor_module.as_ref() { modules_to_check.push(preprocessor_module.as_ref()); } @@ -444,7 +445,28 @@ SELECT importer_node_id, imported_path, imported_lockfile_hash } } - /// Get dependents of any imported path - returns scripts/flows/apps that depend on it + /// Get imports of a given importer path - returns paths that the importer depends on + pub async fn get_imports<'c>( + importer_path: &str, + workspace_id: &str, + e: impl PgExecutor<'c>, + ) -> Result> { + sqlx::query_scalar!( + r#" + SELECT DISTINCT imported_path as "imported_path!" + FROM dependency_map + WHERE workspace_id = $1 + AND importer_path = $2 + AND imported_path NOT LIKE 'dependencies/%' + "#, + workspace_id, + importer_path + ) + .fetch_all(e) + .await + .map_err(Error::from) + } + pub async fn get_dependents<'c>( imported_path: &str, workspace_id: &str, diff --git a/backend/windmill-dep-map/src/trigger_dependents.rs b/backend/windmill-dep-map/src/trigger_dependents.rs index d2e3fd56dd..1254e41ed1 100644 --- a/backend/windmill-dep-map/src/trigger_dependents.rs +++ b/backend/windmill-dep-map/src/trigger_dependents.rs @@ -61,14 +61,14 @@ pub async fn trigger_dependents_to_recompute_dependencies( ); let mut debouncing_settings = DebouncingSettings { - debounce_key: Some(format!("{w_id}:{importer_path}:dependency")), + debounce_key: Some(format!("{w_id}:{importer_path}:{importer_kind}:dependency")), debounce_delay_s: Some(5), ..Default::default() }; let job_payload = match importer_kind.as_str() { "script" => match sqlx::query_scalar!( - "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false ORDER BY created_at DESC LIMIT 1", + "SELECT hash FROM script WHERE path = $1 AND workspace_id = $2 AND deleted = false AND archived = false ORDER BY created_at DESC LIMIT 1", importer_path, w_id ) diff --git a/backend/windmill-git-sync/Cargo.toml b/backend/windmill-git-sync/Cargo.toml index 063e1ce54a..148746dcca 100644 --- a/backend/windmill-git-sync/Cargo.toml +++ b/backend/windmill-git-sync/Cargo.toml @@ -9,7 +9,7 @@ name = "windmill_git_sync" path = "./src/lib.rs" [features] -private = [] +private = ["windmill-common/private"] enterprise = ["windmill-queue/enterprise", "windmill-common/enterprise"] all_sqlx_features = ["enterprise"] default = [] diff --git a/backend/windmill-git-sync/src/lib.rs b/backend/windmill-git-sync/src/lib.rs index f9cecd46ce..dcbcd5bcb2 100644 --- a/backend/windmill-git-sync/src/lib.rs +++ b/backend/windmill-git-sync/src/lib.rs @@ -272,7 +272,10 @@ mod tests { path: "f/folder/script".to_string(), parent_path: Some("f/folder/old_script".to_string()), }; - assert_eq!(obj.get_parent_path(), Some("f/folder/old_script".to_string())); + assert_eq!( + obj.get_parent_path(), + Some("f/folder/old_script".to_string()) + ); } #[test] @@ -313,21 +316,13 @@ mod tests { #[test] fn test_get_kind_flow() { - let obj = DeployedObject::Flow { - path: "test".to_string(), - parent_path: None, - version: 1, - }; + let obj = DeployedObject::Flow { path: "test".to_string(), parent_path: None, version: 1 }; assert_eq!(obj.get_kind(), "flow"); } #[test] fn test_get_kind_app() { - let obj = DeployedObject::App { - path: "test".to_string(), - version: 1, - parent_path: None, - }; + let obj = DeployedObject::App { path: "test".to_string(), version: 1, parent_path: None }; assert_eq!(obj.get_kind(), "app"); } @@ -346,7 +341,8 @@ mod tests { "http_trigger" ); assert_eq!( - DeployedObject::WebsocketTrigger { path: "t".to_string(), parent_path: None }.get_kind(), + DeployedObject::WebsocketTrigger { path: "t".to_string(), parent_path: None } + .get_kind(), "websocket_trigger" ); assert_eq!( diff --git a/backend/windmill-native-triggers/src/handler.rs b/backend/windmill-native-triggers/src/handler.rs index b66a8ae3e0..60eb006a97 100644 --- a/backend/windmill-native-triggers/src/handler.rs +++ b/backend/windmill-native-triggers/src/handler.rs @@ -181,6 +181,7 @@ async fn create_native_trigger( &external_id, &config, service_config, + data.summary.as_deref(), ) .await?; @@ -304,6 +305,7 @@ async fn update_native_trigger_handler( &external_id, &config, service_config, + data.summary.as_deref(), ) .await?; @@ -511,13 +513,13 @@ pub fn service_routes(handler: T) -> Router { let standard_routes = Router::new() .route("/create", post(create_native_trigger::)) .route("/list", get(list_native_triggers_handler::)) - .route("/get/:external_id", get(get_native_trigger_handler::)) + .route("/get/{external_id}", get(get_native_trigger_handler::)) .route( - "/update/:external_id", + "/update/{external_id}", post(update_native_trigger_handler::), ) .route( - "/delete/:external_id", + "/delete/{external_id}", delete(delete_native_trigger_handler::), ); diff --git a/backend/windmill-native-triggers/src/lib.rs b/backend/windmill-native-triggers/src/lib.rs index 25234ff3f0..7853bb63e3 100644 --- a/backend/windmill-native-triggers/src/lib.rs +++ b/backend/windmill-native-triggers/src/lib.rs @@ -195,6 +195,7 @@ pub struct NativeTrigger { pub error: Option, pub created_at: DateTime, pub updated_at: DateTime, + pub summary: Option, } #[derive(Debug, Clone, Serialize, Deserialize)] @@ -209,6 +210,7 @@ pub struct NativeTriggerData { pub script_path: String, pub is_flow: bool, pub service_config: C, + pub summary: Option, } #[derive(Debug, Clone, FromRow, Serialize, Deserialize)] @@ -821,6 +823,7 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres> external_id: &str, config: &NativeTriggerConfig, service_config: C, + summary: Option<&str>, ) -> Result<()> { use windmill_common::auth::hash_token; @@ -835,12 +838,13 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres> script_path, is_flow, webhook_token_hash, - service_config + service_config, + summary ) VALUES ( - $1, $2, $3, $4, $5, $6, $7 + $1, $2, $3, $4, $5, $6, $7, $8 ) ON CONFLICT (external_id, workspace_id, service_name) - DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, error = NULL, updated_at = NOW() + DO UPDATE SET script_path = $4, is_flow = $5, webhook_token_hash = $6, service_config = $7, summary = $8, error = NULL, updated_at = NOW() "#, external_id, workspace_id, @@ -849,6 +853,7 @@ pub async fn store_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres> config.is_flow, webhook_token_hash, sqlx::types::Json(service_config) as _, + summary, ) .execute(db) .await?; @@ -863,6 +868,7 @@ pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres external_id: &str, config: &NativeTriggerConfig, service_config: Option<&RawValue>, + summary: Option<&str>, ) -> Result<()> { use windmill_common::auth::hash_token; @@ -871,7 +877,7 @@ pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres sqlx::query!( r#" UPDATE native_trigger - SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, error = NULL, updated_at = NOW() + SET script_path = $1, is_flow = $2, webhook_token_hash = $3, service_config = $4, summary = $8, error = NULL, updated_at = NOW() WHERE workspace_id = $5 AND service_name = $6 @@ -884,6 +890,7 @@ pub async fn update_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres workspace_id, service_name as ServiceName, external_id, + summary, ) .execute(db) .await?; @@ -934,7 +941,8 @@ pub async fn get_native_trigger<'c, E: sqlx::Executor<'c, Database = Postgres>>( service_config, error, created_at, - updated_at + updated_at, + summary FROM native_trigger WHERE @@ -972,7 +980,8 @@ pub async fn get_native_trigger_by_script<'c, E: sqlx::Executor<'c, Database = P service_config, error, created_at, - updated_at + updated_at, + summary FROM native_trigger WHERE @@ -1018,7 +1027,8 @@ pub async fn list_native_triggers<'c, E: sqlx::Executor<'c, Database = Postgres> nt.service_config, nt.error, nt.created_at, - nt.updated_at + nt.updated_at, + nt.summary FROM native_trigger nt WHERE diff --git a/backend/windmill-native-triggers/src/workspace_integrations.rs b/backend/windmill-native-triggers/src/workspace_integrations.rs index 87d40d5b05..9453f033ab 100644 --- a/backend/windmill-native-triggers/src/workspace_integrations.rs +++ b/backend/windmill-native-triggers/src/workspace_integrations.rs @@ -964,22 +964,22 @@ async fn generate_instance_connect_url( pub fn workspaced_service() -> Router { let router = Router::new() .route("/list", get(list_integrations)) - .route("/:service_name/exists", get(integration_exist)) - .route("/:service_name/create", post(create_workspace_integration)) + .route("/{service_name}/exists", get(integration_exist)) + .route("/{service_name}/create", post(create_workspace_integration)) .route( - "/:service_name/generate_connect_url", + "/{service_name}/generate_connect_url", post(generate_connect_url), ) .route( - "/:service_name/instance_sharing_available", + "/{service_name}/instance_sharing_available", get(check_instance_sharing_available), ) .route( - "/:service_name/generate_instance_connect_url", + "/{service_name}/generate_instance_connect_url", post(generate_instance_connect_url), ) - .route("/:service_name/delete", delete(delete_integration)) - .route("/:service_name/callback", post(oauth_callback)); + .route("/{service_name}/delete", delete(delete_integration)) + .route("/{service_name}/callback", post(oauth_callback)); Router::new().nest("/integrations", router) } diff --git a/backend/windmill-oauth/src/lib.rs b/backend/windmill-oauth/src/lib.rs index 26d807dac3..eff53be481 100644 --- a/backend/windmill-oauth/src/lib.rs +++ b/backend/windmill-oauth/src/lib.rs @@ -516,7 +516,7 @@ pub async fn exchange_code( }; let csrf_state = cookies .get(name) - .map(|x| x.value().to_string()) + .map(|x| x.value_trimmed().to_string()) .unwrap_or("".to_string()); if callback.state != csrf_state { return Err(error::Error::BadRequest("csrf did not match".to_string())); @@ -544,14 +544,22 @@ pub async fn exchange_token( grant_type: &str, oauth_client_info: Option<&ClientWithScopes>, http_client: &reqwest::Client, + scopes: Option<&[String]>, ) -> Result { let token_json = match grant_type { - "authorization_code" => client - .exchange_refresh_token(&RefreshToken::from(refresh_token)) - .with_client(http_client) - .execute::() - .await - .map_err(to_anyhow)?, + "authorization_code" | "" => { + let mut request = client.exchange_refresh_token(&RefreshToken::from(refresh_token)); + if let Some(scopes) = scopes { + if !scopes.is_empty() { + request = request.param("scope", scopes.join(" ")); + } + } + request + .with_client(http_client) + .execute::() + .await + .map_err(to_anyhow)? + } "client_credentials" => { let mut token_request = client.exchange_client_credentials(); @@ -569,12 +577,6 @@ pub async fn exchange_token( .await .map_err(to_anyhow)? } - "" | _ if grant_type.is_empty() => client - .exchange_refresh_token(&RefreshToken::from(refresh_token)) - .with_client(http_client) - .execute::() - .await - .map_err(to_anyhow)?, _ => { return Err(Error::BadRequest(format!( "Unsupported grant type: {}", @@ -599,6 +601,7 @@ pub struct OAuthAccountInfo { pub cc_client_id: Option, pub cc_client_secret: Option, pub cc_token_url: Option, + pub scopes: Option>, } /// Refresh an OAuth token and update the database. @@ -615,7 +618,7 @@ pub async fn refresh_token<'c>( ) -> error::Result { let account = sqlx::query_as!( OAuthAccountInfo, - "SELECT client, refresh_token, grant_type, cc_client_id, cc_client_secret, cc_token_url FROM account WHERE workspace_id = $1 AND id = $2", + "SELECT client, refresh_token, grant_type, cc_client_id, cc_client_secret, cc_token_url, scopes FROM account WHERE workspace_id = $1 AND id = $2", w_id, id, ) @@ -679,8 +682,15 @@ pub async fn refresh_token_for_account<'c>( oauth_client_info.client.to_owned() }; + // Account-level scopes override instance-level scopes + let effective_scopes = account + .scopes + .as_deref() + .filter(|s| !s.is_empty()) + .unwrap_or(&oauth_client_info.scopes); + if account.grant_type == "client_credentials" { - for scope in oauth_client_info.scopes.iter() { + for scope in effective_scopes.iter() { client.add_scope(scope); } } @@ -699,6 +709,7 @@ pub async fn refresh_token_for_account<'c>( &account.grant_type, Some(&oauth_client_info), http_client, + Some(effective_scopes), ) .await; diff --git a/backend/windmill-object-store/src/lib.rs b/backend/windmill-object-store/src/lib.rs index 4caf67f348..e8d33da2e2 100644 --- a/backend/windmill-object-store/src/lib.rs +++ b/backend/windmill-object-store/src/lib.rs @@ -5,13 +5,13 @@ use std::collections::HashMap; use quick_cache::sync::Cache; use windmill_common::error::{self}; +#[cfg(feature = "parquet")] +use async_trait::async_trait; #[cfg(feature = "parquet")] use aws_config::{default_provider::credentials::DefaultCredentialsChain, Region}; #[cfg(feature = "parquet")] use aws_sdk_sts::config::ProvideCredentials; #[cfg(feature = "parquet")] -use axum::async_trait; -#[cfg(feature = "parquet")] use bytes::Bytes; #[cfg(feature = "parquet")] use chrono::{DateTime, Utc}; diff --git a/backend/windmill-queue/src/jobs.rs b/backend/windmill-queue/src/jobs.rs index a6ba745c24..53c7460c4e 100644 --- a/backend/windmill-queue/src/jobs.rs +++ b/backend/windmill-queue/src/jobs.rs @@ -77,8 +77,8 @@ use windmill_common::{ users::{SUPERADMIN_NOTIFICATION_EMAIL, SUPERADMIN_SECRET_EMAIL}, utils::{not_found_if_none, report_critical_error, StripPath, WarnAfterExt}, worker::{ - to_raw_value, CLOUD_HOSTED, DISABLE_FLOW_SCRIPT, NO_LOGS, WORKER_PULL_QUERIES, - WORKER_SUSPENDED_PULL_QUERY, + to_raw_value, CLOUD_HOSTED, DISABLE_FLOW_SCRIPT, NO_LOGS, PREVIEW_TAGS_OVERRIDE, + WORKER_PULL_QUERIES, WORKER_SUSPENDED_PULL_QUERY, }, DB, METRICS_ENABLED, }; @@ -3302,7 +3302,10 @@ pub async fn pull( }; if let Some(job) = job.as_ref() { - if job.is_flow() || job.is_dependency() { + if (job.is_flow() || job.is_dependency()) + && !(job.kind.is_preview() + && PREVIEW_TAGS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed)) + { let per_workspace = per_workspace_tag(&job.workspace_id).await; let base_tag = if job.is_flow() { "flow".to_string() @@ -5493,25 +5496,35 @@ async fn push_inner<'c, 'd>( }; interpolated_tag.unwrap_or_else(|| { - language - .as_ref() - .map(|x| { - let tag_lang = if x == &ScriptLang::Bunnative { - if job_kind == JobKind::Dependencies { - ScriptLang::Bun.as_str() + if job_kind.is_preview() + && PREVIEW_TAGS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed) + { + if per_workspace { + format!("preview-{}", workspace_id) + } else { + "preview".to_string() + } + } else { + language + .as_ref() + .map(|x| { + let tag_lang = if x == &ScriptLang::Bunnative { + if job_kind == JobKind::Dependencies { + ScriptLang::Bun.as_str() + } else { + ScriptLang::Nativets.as_str() + } } else { - ScriptLang::Nativets.as_str() + x.as_str() + }; + if per_workspace { + format!("{}-{}", tag_lang, workspace_id) + } else { + tag_lang.to_string() } - } else { - x.as_str() - }; - if per_workspace { - format!("{}-{}", tag_lang, workspace_id) - } else { - tag_lang.to_string() - } - }) - .unwrap_or_else(default) + }) + .unwrap_or_else(default) + } }) }; diff --git a/backend/windmill-store/src/resources.rs b/backend/windmill-store/src/resources.rs index c90e185d7a..b17895b40a 100644 --- a/backend/windmill-store/src/resources.rs +++ b/backend/windmill-store/src/resources.rs @@ -7,6 +7,7 @@ */ use std::collections::HashMap; +use std::net::IpAddr; use windmill_api_auth::{ check_scopes, maybe_refresh_folders, require_owner_of_path, require_super_admin, ApiAuthed, @@ -55,26 +56,26 @@ pub fn workspaced_service() -> Router { Router::new() .route("/list", get(list_resources)) .route("/list_search", get(list_search_resources)) - .route("/list_names/:type", get(list_names)) - .route("/get/*path", get(get_resource)) - .route("/exists/*path", get(exists_resource)) - .route("/get_value/*path", get(get_resource_value)) + .route("/list_names/{type}", get(list_names)) + .route("/get/{*path}", get(get_resource)) + .route("/exists/{*path}", get(exists_resource)) + .route("/get_value/{*path}", get(get_resource_value)) .route( - "/get_value_interpolated/*path", + "/get_value_interpolated/{*path}", get(get_resource_value_interpolated), ) - .route("/update/*path", post(update_resource)) - .route("/update_value/*path", post(update_resource_value)) - .route("/delete/*path", delete(delete_resource)) + .route("/update/{*path}", post(update_resource)) + .route("/update_value/{*path}", post(update_resource_value)) + .route("/delete/{*path}", delete(delete_resource)) .route("/delete_bulk", delete(delete_resources_bulk)) .route("/create", post(create_resource)) - .route("/git_commit_hash/*path", get(get_git_commit_hash)) + .route("/git_commit_hash/{*path}", get(get_git_commit_hash)) .route("/type/list", get(list_resource_types)) .route("/type/listnames", get(list_resource_types_names)) - .route("/type/get/:name", get(get_resource_type)) - .route("/type/exists/:name", get(exists_resource_type)) - .route("/type/update/:name", post(update_resource_type)) - .route("/type/delete/:name", delete(delete_resource_type)) + .route("/type/get/{name}", get(get_resource_type)) + .route("/type/exists/{name}", get(exists_resource_type)) + .route("/type/update/{name}", post(update_resource_type)) + .route("/type/delete/{name}", delete(delete_resource_type)) .route( "/file_resource_type_to_file_ext_map", get(file_resource_ext_to_resource_type), @@ -83,7 +84,7 @@ pub fn workspaced_service() -> Router { } pub fn public_service() -> Router { - Router::new().route("/custom_component/:name", get(custom_component)) + Router::new().route("/custom_component/{name}", get(custom_component)) } #[derive(FromRow, Serialize, Deserialize)] @@ -305,7 +306,11 @@ async fn list_resources( } if let Some(value) = &lq.value { - sqlb.and_where("resource.value @> ?".bind(&value.replace("'", "''"))); + if let Ok(v) = serde_json::from_str::(value) { + sqlb.and_where("resource.value @> ?".bind(&v.to_string())); + } else { + sqlb.and_where("FALSE"); + } } if let Some(broad_filter) = &lq.broad_filter { @@ -578,6 +583,16 @@ pub async fn transform_json_value( .await?; Ok(Value::String(v)) } + Value::String(y) if y.starts_with("$jsonvar:") => { + let path = y.strip_prefix("$jsonvar:").unwrap(); + + let v = + crate::variables::get_value_internal(&db_with_opt_authed, workspace, path, false) + .await?; + serde_json::from_str::(&v).map_err(|e| { + Error::internal_err(format!("Failed to parse $jsonvar value as JSON: {e}")) + }) + } Value::String(y) if y.starts_with("$res:") => { let path = y.strip_prefix("$res:").unwrap(); if path.split("/").count() < 2 { @@ -886,6 +901,15 @@ async fn delete_resource( } let mut tx = user_db.begin(&authed).await?; + // Capture resource data for trashbin before deleting + let trash_resource: Option = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM resource t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + // Fetch the resource value before deleting, so we can find linked $var: references let resource_value: Option> = sqlx::query_scalar("SELECT value FROM resource WHERE path = $1 AND workspace_id = $2") @@ -894,6 +918,32 @@ async fn delete_resource( .fetch_optional(&mut *tx) .await?; + // Collect all $var: paths referenced in the resource value + let mut linked_var_paths: Vec = Vec::new(); + if let Some(Some(ref value)) = resource_value { + collect_var_refs(value, &mut linked_var_paths); + } + + // Capture linked variables for trashbin before deleting them + let trash_linked_vars: Vec = if linked_var_paths.is_empty() { + Vec::new() + } else { + let placeholders: Vec = linked_var_paths + .iter() + .enumerate() + .map(|(i, _)| format!("${}", i + 2)) + .collect(); + let query = format!( + "SELECT to_jsonb(t) FROM variable t WHERE workspace_id = $1 AND path IN ({})", + placeholders.join(", ") + ); + let mut q = sqlx::query_scalar::<_, serde_json::Value>(&query).bind(&w_id); + for var_path in &linked_var_paths { + q = q.bind(var_path); + } + q.fetch_all(&mut *tx).await? + }; + let deleted_path = sqlx::query_scalar!( "DELETE FROM resource WHERE path = $1 AND workspace_id = $2 RETURNING path", path, @@ -903,12 +953,6 @@ async fn delete_resource( .await?; not_found_if_none(deleted_path, "Resource", &path)?; - // Collect all $var: paths referenced in the resource value - let mut linked_var_paths: Vec = Vec::new(); - if let Some(Some(value)) = resource_value { - collect_var_refs(&value, &mut linked_var_paths); - } - // Delete linked variables that are actually referenced in the resource value let deleted_linked_variables: Vec = if linked_var_paths.is_empty() { Vec::new() @@ -928,6 +972,23 @@ async fn delete_resource( } q.fetch_all(&mut *tx).await? }; + + if let Some(res_data) = trash_resource { + let mut trash_data = serde_json::json!({"row": res_data}); + if !trash_linked_vars.is_empty() { + trash_data["linked_variables"] = serde_json::Value::Array(trash_linked_vars); + } + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "resource", + path, + trash_data, + &authed.username, + ) + .await?; + } + audit_log( &mut *tx, &authed, @@ -1034,6 +1095,30 @@ async fn delete_resources_bulk( let mut tx = user_db.begin(&authed).await?; + // Capture resources for trashbin per path before bulk delete + for path in &request.paths { + let trash_resource: Option = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM resource t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + + if let Some(res_data) = trash_resource { + let trash_data = serde_json::json!({"row": res_data}); + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "resource", + path, + trash_data, + &authed.username, + ) + .await?; + } + } + let deleted_paths = sqlx::query_scalar!( "DELETE FROM resource WHERE path = ANY($1) AND workspace_id = $2 RETURNING path", &request.paths, @@ -1709,9 +1794,74 @@ struct GitRepositoryResource { branch: Option, } -/// Validates a git URL to prevent git option injection attacks. -/// Git URLs starting with '-' could be interpreted as command-line options. -fn validate_git_url(url: &str) -> Result<()> { +/// Checks whether an IP address belongs to a private, loopback, link-local, or +/// otherwise reserved range that should not be reachable from git operations. +fn is_private_or_reserved_ip(ip: &IpAddr) -> bool { + match ip { + IpAddr::V4(v4) => { + v4.is_loopback() + || v4.is_private() + || v4.is_link_local() + || v4.is_unspecified() + || v4.is_broadcast() + // 100.64.0.0/10 (Carrier-grade NAT / CGNAT) + || (v4.octets()[0] == 100 && (v4.octets()[1] & 0xC0) == 64) + } + IpAddr::V6(v6) => { + v6.is_loopback() + || v6.is_unspecified() + // IPv4-mapped IPv6 (::ffff:x.x.x.x) — check the inner v4 + || v6.to_ipv4_mapped().map_or(false, |v4| { + is_private_or_reserved_ip(&IpAddr::V4(v4)) + }) + } + } +} + +/// Extracts the hostname from a git URL. +/// +/// Handles standard URLs (`https://host/path`, `ssh://user@host/path`) and +/// SCP-style (`user@host:path`). +fn extract_host_from_git_url(url: &str) -> Option { + if let Some(after_scheme) = url.split("://").nth(1) { + // Standard URL with scheme + let host_part = match after_scheme.find('@') { + Some(pos) => &after_scheme[pos + 1..], + None => after_scheme, + }; + // Handle IPv6 in brackets: [::1] + if host_part.starts_with('[') { + let end = host_part.find(']')?; + let host = &host_part[1..end]; + return if host.is_empty() { + None + } else { + Some(host.to_lowercase()) + }; + } + let host_port = host_part.split('/').next()?; + let host = host_port.rsplit_once(':').map_or(host_port, |(h, _)| h); + if host.is_empty() { + return None; + } + return Some(host.to_lowercase()); + } + + // SCP-style: user@host:path + if let Some(at_pos) = url.find('@') { + let after_at = &url[at_pos + 1..]; + let host = after_at.split(':').next()?; + if host.is_empty() { + return None; + } + return Some(host.to_lowercase()); + } + + None +} + +/// Validates a git URL to prevent option injection, SSRF, and local file read. +async fn validate_git_url(url: &str) -> Result<()> { let url = url.trim(); if url.is_empty() { return Err(Error::BadRequest("Git URL cannot be empty".to_string())); @@ -1721,12 +1871,59 @@ fn validate_git_url(url: &str) -> Result<()> { "Git URL cannot start with '-' (potential option injection)".to_string(), )); } - // Block other potentially dangerous patterns if url.contains('\0') || url.contains('\n') || url.contains('\r') { return Err(Error::BadRequest( "Git URL contains invalid characters".to_string(), )); } + + let lower = url.to_lowercase(); + + // Allowlist of URL formats — blocks file://, ftp://, local paths, etc. + let has_valid_scheme = lower.starts_with("https://") + || lower.starts_with("http://") + || lower.starts_with("git://") + || lower.starts_with("ssh://"); + + // SCP-style: user@host:path (no scheme, has @ before :) + let is_scp_style = !url.contains("://") && url.contains('@') && url.contains(':'); + + if !has_valid_scheme && !is_scp_style { + return Err(Error::BadRequest( + "Git URL must use https://, http://, git://, ssh://, or user@host:path format" + .to_string(), + )); + } + + let host = extract_host_from_git_url(url) + .ok_or_else(|| Error::BadRequest("Could not parse hostname from git URL".to_string()))?; + + if host == "localhost" || host.ends_with(".local") || host == "[::1]" { + return Err(Error::BadRequest( + "Git URLs targeting localhost or local network are not allowed".to_string(), + )); + } + + // Check literal IP addresses + if let Ok(ip) = host.parse::() { + if is_private_or_reserved_ip(&ip) { + return Err(Error::BadRequest( + "Git URLs targeting private or reserved IP addresses are not allowed".to_string(), + )); + } + } else { + // Hostname — resolve via DNS and reject if any address is private + if let Ok(addrs) = tokio::net::lookup_host(format!("{}:443", host)).await { + for addr in addrs { + if is_private_or_reserved_ip(&addr.ip()) { + return Err(Error::BadRequest( + "Git URL hostname resolves to a private or reserved IP address".to_string(), + )); + } + } + } + } + Ok(()) } @@ -1918,8 +2115,8 @@ async fn get_repo_latest_commit_hash( git_resource: &GitRepositoryResource, git_ssh_command: Option, ) -> Result { - // Validate URL and branch to prevent option injection attacks - validate_git_url(&git_resource.url)?; + // Validate URL and branch to prevent option injection and SSRF attacks + validate_git_url(&git_resource.url).await?; let ref_spec = git_resource .branch @@ -2083,4 +2280,149 @@ mod tests { assert!(result.is_err()); } + + #[test] + fn test_extract_host_from_git_url() { + // Standard HTTPS + assert_eq!( + extract_host_from_git_url("https://github.com/user/repo.git"), + Some("github.com".to_string()) + ); + // HTTPS with port + assert_eq!( + extract_host_from_git_url("https://git.example.com:8443/repo.git"), + Some("git.example.com".to_string()) + ); + // SSH with scheme + assert_eq!( + extract_host_from_git_url("ssh://git@github.com/user/repo.git"), + Some("github.com".to_string()) + ); + // SCP-style + assert_eq!( + extract_host_from_git_url("git@github.com:user/repo.git"), + Some("github.com".to_string()) + ); + // Git protocol + assert_eq!( + extract_host_from_git_url("git://example.com/repo.git"), + Some("example.com".to_string()) + ); + // IPv6 in brackets + assert_eq!( + extract_host_from_git_url("http://[::1]:8080/repo.git"), + Some("::1".to_string()) + ); + // No host extractable + assert_eq!(extract_host_from_git_url("/local/path"), None); + assert_eq!( + extract_host_from_git_url("file:///etc/passwd"), + Some("".to_string()).filter(|s| !s.is_empty()) + ); + } + + #[test] + fn test_is_private_or_reserved_ip() { + use std::net::IpAddr; + // Loopback + assert!(is_private_or_reserved_ip( + &"127.0.0.1".parse::().unwrap() + )); + assert!(is_private_or_reserved_ip( + &"127.0.0.2".parse::().unwrap() + )); + // Private ranges + assert!(is_private_or_reserved_ip( + &"10.0.0.1".parse::().unwrap() + )); + assert!(is_private_or_reserved_ip( + &"172.16.0.1".parse::().unwrap() + )); + assert!(is_private_or_reserved_ip( + &"192.168.1.1".parse::().unwrap() + )); + // Link-local / cloud metadata + assert!(is_private_or_reserved_ip( + &"169.254.169.254".parse::().unwrap() + )); + // CGNAT + assert!(is_private_or_reserved_ip( + &"100.64.0.1".parse::().unwrap() + )); + // Unspecified + assert!(is_private_or_reserved_ip( + &"0.0.0.0".parse::().unwrap() + )); + // IPv6 loopback + assert!(is_private_or_reserved_ip(&"::1".parse::().unwrap())); + // IPv4-mapped IPv6 + assert!(is_private_or_reserved_ip( + &"::ffff:127.0.0.1".parse::().unwrap() + )); + // Public IPs should pass + assert!(!is_private_or_reserved_ip( + &"8.8.8.8".parse::().unwrap() + )); + assert!(!is_private_or_reserved_ip( + &"140.82.121.4".parse::().unwrap() + )); + } + + #[tokio::test] + async fn test_validate_git_url_blocks_file_scheme() { + let result = validate_git_url("file:///etc/passwd").await; + assert!(result.is_err()); + assert!(result.unwrap_err().to_string().contains("https://")); + } + + #[tokio::test] + async fn test_validate_git_url_blocks_private_ips() { + assert!(validate_git_url("http://127.0.0.1/repo.git").await.is_err()); + assert!(validate_git_url("http://169.254.169.254/latest/meta-data/") + .await + .is_err()); + assert!(validate_git_url("http://10.0.0.1/repo.git").await.is_err()); + assert!(validate_git_url("http://172.16.0.1/repo.git") + .await + .is_err()); + assert!(validate_git_url("http://192.168.1.1/repo.git") + .await + .is_err()); + assert!(validate_git_url("git://0.0.0.0/repo.git").await.is_err()); + } + + #[tokio::test] + async fn test_validate_git_url_blocks_localhost() { + assert!(validate_git_url("http://localhost/repo.git").await.is_err()); + assert!(validate_git_url("http://myhost.local/repo.git") + .await + .is_err()); + } + + #[tokio::test] + async fn test_validate_git_url_blocks_local_paths() { + assert!(validate_git_url("/etc/passwd").await.is_err()); + assert!(validate_git_url("../relative/path").await.is_err()); + assert!(validate_git_url("./local/repo").await.is_err()); + } + + #[tokio::test] + async fn test_validate_git_url_allows_valid_urls() { + // These should succeed (host resolution may fail but validation passes) + assert!(validate_git_url("https://github.com/user/repo.git") + .await + .is_ok()); + assert!(validate_git_url("git@github.com:user/repo.git") + .await + .is_ok()); + assert!(validate_git_url("ssh://git@github.com/user/repo.git") + .await + .is_ok()); + } + + #[tokio::test] + async fn test_validate_git_url_blocks_option_injection() { + assert!(validate_git_url("-evil").await.is_err()); + assert!(validate_git_url("--upload-pack=evil").await.is_err()); + } } diff --git a/backend/windmill-store/src/variables.rs b/backend/windmill-store/src/variables.rs index 544f7d6c0a..c893f61cb0 100644 --- a/backend/windmill-store/src/variables.rs +++ b/backend/windmill-store/src/variables.rs @@ -54,11 +54,11 @@ pub fn workspaced_service() -> Router { Router::new() .route("/list", get(list_variables)) .route("/list_contextual", get(list_contextual_variables)) - .route("/get/*path", get(get_variable)) - .route("/get_value/*path", get(get_value)) - .route("/exists/*path", get(exists_variable)) - .route("/update/*path", post(update_variable)) - .route("/delete/*path", delete(delete_variable)) + .route("/get/{*path}", get(get_variable)) + .route("/get_value/{*path}", get(get_value)) + .route("/exists/{*path}", get(exists_variable)) + .route("/update/{*path}", post(update_variable)) + .route("/delete/{*path}", delete(delete_variable)) .route("/delete_bulk", delete(delete_variables_bulk)) .route("/create", post(create_variable)) .route("/encrypt", post(encrypt_value)) @@ -133,21 +133,12 @@ async fn list_variables( ]) .left() .join("account") - .on(&format!( - "variable.account = account.id AND account.workspace_id = '{}'", - w_id - )) + .on("variable.account = account.id AND account.workspace_id = ?".bind(&w_id)) .left() .join("resource") - .on(&format!( - "resource.path = variable.path AND resource.workspace_id = '{}'", - w_id - )) + .on("resource.path = variable.path AND resource.workspace_id = ?".bind(&w_id)) .and_where("variable.workspace_id = ?".bind(&w_id)) - .and_where(&format!( - "variable.path NOT LIKE 'u/' || '{}' || '/secret_arg/%'", - authed.username - )) + .and_where("variable.path NOT LIKE 'u/' || ? || '/secret_arg/%'".bind(&authed.username)) .order_by("path", false) .limit(per_page) .offset(offset) @@ -529,6 +520,23 @@ async fn delete_variable( let mut tx = user_db.begin(&authed).await?; + // Capture data for trashbin before deleting + let trash_var: Option = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM variable t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + + let trash_linked_resource: Option = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM resource t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + sqlx::query!( "DELETE FROM variable WHERE path = $1 AND workspace_id = $2", path, @@ -543,6 +551,23 @@ async fn delete_variable( ) .fetch_optional(&mut *tx) .await?; + + if let Some(var_data) = trash_var { + let mut trash_data = serde_json::json!({"row": var_data}); + if let Some(linked) = trash_linked_resource { + trash_data["linked_resource"] = linked; + } + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "variable", + path, + trash_data, + &authed.username, + ) + .await?; + } + audit_log( &mut *tx, &authed, @@ -642,6 +667,41 @@ async fn delete_variables_bulk( let mut tx = user_db.begin(&authed).await?; + // Capture variables for trashbin per path before bulk delete + for path in &request.paths { + let trash_var: Option = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM variable t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + + if let Some(var_data) = trash_var { + let trash_linked: Option = sqlx::query_scalar( + "SELECT to_jsonb(t) FROM resource t WHERE path = $1 AND workspace_id = $2", + ) + .bind(path) + .bind(&w_id) + .fetch_optional(&mut *tx) + .await?; + + let mut trash_data = serde_json::json!({"row": var_data}); + if let Some(linked) = trash_linked { + trash_data["linked_resource"] = linked; + } + windmill_common::trashbin::move_to_trash( + &mut *tx, + &w_id, + "variable", + path, + trash_data, + &authed.username, + ) + .await?; + } + } + let deleted_paths = sqlx::query_scalar!( "DELETE FROM variable WHERE path = ANY($1) AND workspace_id = $2 RETURNING path", &request.paths, @@ -1055,6 +1115,10 @@ pub async fn get_value_internal<'a>( variable.value }; + if variable.is_secret && !r.is_empty() { + windmill_common::sensitive_log_masks::register_secret_for_all_running_jobs(&r); + } + // Cache the result when explicitly allowed and caching appropriate if allow_cache { cache_variable(&w_id, &path, db_with_opt_authed.email(), r.clone()); diff --git a/backend/windmill-test-utils/Cargo.toml b/backend/windmill-test-utils/Cargo.toml index d1729a7511..a221e53911 100644 --- a/backend/windmill-test-utils/Cargo.toml +++ b/backend/windmill-test-utils/Cargo.toml @@ -14,6 +14,7 @@ private = ["windmill-api/private"] enterprise = ["windmill-api/enterprise"] python = ["windmill-common/python"] deno_core = ["dep:windmill-runtime-nativets"] +mcp = ["windmill-api/mcp"] agent_worker_server = ["dep:windmill-api-agent-workers"] run_inline = ["windmill-api/run_inline"] duckdb = ["windmill-worker/duckdb"] @@ -35,5 +36,6 @@ tokio.workspace = true uuid.workspace = true chrono.workspace = true axum.workspace = true +async-trait.workspace = true anyhow.workspace = true tracing.workspace = true diff --git a/backend/windmill-test-utils/src/lib.rs b/backend/windmill-test-utils/src/lib.rs index 8e83eae1da..153bdeeb7c 100644 --- a/backend/windmill-test-utils/src/lib.rs +++ b/backend/windmill-test-utils/src/lib.rs @@ -81,30 +81,38 @@ pub struct ApiServer { impl ApiServer { pub async fn start(db: Pool) -> anyhow::Result { - Self::start_inner(db, false).await + Self::start_inner(db, false, false).await } pub async fn start_agent_mode(db: Pool) -> anyhow::Result { - Self::start_inner(db, true).await + Self::start_inner(db, true, false).await } /// Start the API server with server_mode=true so trigger listeners are active. /// Alias for `start_agent_mode` with a clearer name for trigger e2e tests. pub async fn start_with_listeners(db: Pool) -> anyhow::Result { - Self::start_inner(db, true).await + Self::start_inner(db, true, false).await } - async fn start_inner(db: Pool, agent_mode: bool) -> anyhow::Result { + /// Start the API server with mcp_mode=true so MCP routes are active. + pub async fn start_mcp(db: Pool) -> anyhow::Result { + Self::start_inner(db, false, true).await + } + + async fn start_inner( + db: Pool, + server_mode: bool, + mcp_mode: bool, + ) -> anyhow::Result { let (tx, rx) = tokio::sync::broadcast::channel::<()>(1); - let sock = tokio::net::TcpListener::bind("127.0.0.1:0") + let listener = tokio::net::TcpListener::bind("127.0.0.1:0") .await .map_err(|e| anyhow::anyhow!("failed to bind TCP listener: {}", e))?; - let addr = sock + let addr = listener .local_addr() .map_err(|e| anyhow::anyhow!("failed to get local address: {}", e))?; - drop(sock); let (port_tx, _port_rx) = tokio::sync::oneshot::channel::(); let name = next_worker_name(); tracing::info!("starting api server for name={name}"); @@ -112,11 +120,11 @@ impl ApiServer { db.clone(), None, None, - addr, + listener, rx, port_tx, - agent_mode, - false, + server_mode, + mcp_mode, format!("http://localhost:{}", addr.port()), Some(name.clone()), )); @@ -479,7 +487,7 @@ pub async fn completed_job(uuid: Uuid, db: &Pool) -> CompletedJob { .unwrap() } -#[axum::async_trait(?Send)] +#[async_trait::async_trait(?Send)] pub trait StreamFind: futures::Stream + Unpin + Sized { async fn find(self, item: &Self::Item) -> Option where @@ -899,6 +907,8 @@ pub enum DedicatedWorkerResult { Start, /// Worker returned a successful result Success(serde_json::Value), + /// Worker returned preprocessed args (from exec_preprocess) + PreprocessedArgs(serde_json::Value), /// Worker returned an error result Error(serde_json::Value), /// Line is not a protocol message (e.g., logs) @@ -908,6 +918,7 @@ pub enum DedicatedWorkerResult { /// Parse a line from dedicated worker stdout according to the protocol: /// - "start" -> Ready signal /// - "wm_res[success]:JSON" -> Success with result +/// - "wm_res[preprocessed_args]:JSON" -> Preprocessed args /// - "wm_res[error]:JSON" -> Error with details /// - anything else -> Other (logs) pub fn parse_dedicated_worker_line(line: &str) -> DedicatedWorkerResult { @@ -922,6 +933,13 @@ pub fn parse_dedicated_worker_line(line: &str) -> DedicatedWorkerResult { } } + if let Some(json_str) = line.strip_prefix("wm_res[preprocessed_args]:") { + match serde_json::from_str(json_str) { + Ok(value) => return DedicatedWorkerResult::PreprocessedArgs(value), + Err(_) => return DedicatedWorkerResult::Other(line.to_string()), + } + } + if let Some(json_str) = line.strip_prefix("wm_res[error]:") { match serde_json::from_str(json_str) { Ok(value) => return DedicatedWorkerResult::Error(value), diff --git a/backend/windmill-trigger-email/src/handler_oss.rs b/backend/windmill-trigger-email/src/handler_oss.rs index 9579bee274..b6cac94cc2 100644 --- a/backend/windmill-trigger-email/src/handler_oss.rs +++ b/backend/windmill-trigger-email/src/handler_oss.rs @@ -8,7 +8,7 @@ pub use super::handler_ee::*; #[cfg(not(feature = "private"))] use { super::EmailTrigger, - axum::async_trait, + async_trait::async_trait, sqlx::PgConnection, windmill_api_auth::ApiAuthed, windmill_common::{ diff --git a/backend/windmill-trigger-gcp/src/handler_oss.rs b/backend/windmill-trigger-gcp/src/handler_oss.rs index b259c87834..5cf0f17c02 100644 --- a/backend/windmill-trigger-gcp/src/handler_oss.rs +++ b/backend/windmill-trigger-gcp/src/handler_oss.rs @@ -5,7 +5,7 @@ pub use super::handler_ee::*; #[cfg(not(feature = "private"))] use { super::GcpTrigger, - axum::async_trait, + async_trait::async_trait, sqlx::PgConnection, windmill_api_auth::ApiAuthed, windmill_common::{ diff --git a/backend/windmill-trigger-http/Cargo.toml b/backend/windmill-trigger-http/Cargo.toml index 865b82a682..9e81e1d0dd 100644 --- a/backend/windmill-trigger-http/Cargo.toml +++ b/backend/windmill-trigger-http/Cargo.toml @@ -45,3 +45,4 @@ itertools.workspace = true thiserror.workspace = true anyhow.workspace = true hex.workspace = true +chrono.workspace = true diff --git a/backend/windmill-trigger-http/src/handler.rs b/backend/windmill-trigger-http/src/handler.rs index c7ca7431fd..46e278ca5b 100644 --- a/backend/windmill-trigger-http/src/handler.rs +++ b/backend/windmill-trigger-http/src/handler.rs @@ -2,12 +2,14 @@ use super::{ validate_authentication_method, HttpConfig, HttpConfigRequest, HttpMethod, HttpTrigger, RouteExists, ROUTE_PATH_KEY_RE, VALID_ROUTE_PATH_RE, }; -use axum::{async_trait, extract::Path, routing::post, Extension, Json, Router}; +use async_trait::async_trait; +use axum::{extract::Path, routing::post, Extension, Json, Router}; use http::StatusCode; use sqlx::PgConnection; use std::collections::HashSet; use windmill_api_auth::ApiAuthed; use windmill_audit::{audit_oss::audit_log, ActionKind}; +use windmill_common::global_settings::HTTP_ROUTE_WORKSPACED_ROUTE; use windmill_common::{ db::UserDB, error::{Error, Result}, @@ -61,11 +63,12 @@ pub async fn route_path_key_exists( .await? .unwrap_or(false) } else { - let route_path_key = match workspaced_route { - Some(true) => { - std::borrow::Cow::Owned(format!("{}/{}", w_id, route_path_key.trim_matches('/'))) - } - _ => std::borrow::Cow::Borrowed(route_path_key), + let http_route_workspaced = *HTTP_ROUTE_WORKSPACED_ROUTE.read().await; + let effective_workspaced = workspaced_route.unwrap_or(false) || http_route_workspaced; + let route_path_key = if effective_workspaced { + std::borrow::Cow::Owned(format!("{}/{}", w_id, route_path_key.trim_matches('/'))) + } else { + std::borrow::Cow::Borrowed(route_path_key) }; sqlx::query_scalar!( @@ -146,6 +149,10 @@ pub async fn insert_new_trigger_into_db( ) -> Result<()> { require_admin(authed.is_admin, &authed.username)?; + let http_route_workspaced = *HTTP_ROUTE_WORKSPACED_ROUTE.read().await; + let effective_workspaced = + trigger.config.workspaced_route.unwrap_or(false) || http_route_workspaced; + let request_type = trigger.config.request_type; let resolved_edited_by = trigger.base.resolve_edited_by(authed); let resolved_permissioned_as = trigger.base.resolve_permissioned_as(authed); @@ -186,7 +193,7 @@ pub async fn insert_new_trigger_into_db( trigger.base.path, trigger.config.route_path, route_path_key, - trigger.config.workspaced_route.unwrap_or(false), + effective_workspaced, trigger.config.authentication_resource_path, trigger.config.wrap_body.unwrap_or(false), trigger.config.raw_string.unwrap_or(false), @@ -445,6 +452,10 @@ impl TriggerCrud for HttpTrigger { let route_path_key = check_if_route_exist(db, &trigger.config, workspace_id, Some(path)).await?; + let http_route_workspaced = *HTTP_ROUTE_WORKSPACED_ROUTE.read().await; + let effective_workspaced = + trigger.config.workspaced_route.unwrap_or(false) || http_route_workspaced; + let request_type = trigger.config.request_type; sqlx::query!( @@ -481,7 +492,7 @@ impl TriggerCrud for HttpTrigger { "#, route_path, &route_path_key, - trigger.config.workspaced_route, + Some(effective_workspaced), trigger.config.wrap_body, trigger.config.raw_string, trigger.config.authentication_resource_path, diff --git a/backend/windmill-trigger-http/src/http_trigger_auth.rs b/backend/windmill-trigger-http/src/http_trigger_auth.rs index 987d0ac811..19766cdbc2 100644 --- a/backend/windmill-trigger-http/src/http_trigger_auth.rs +++ b/backend/windmill-trigger-http/src/http_trigger_auth.rs @@ -12,6 +12,23 @@ use sha1::Sha1; use sha2::{Sha256, Sha512}; use std::{borrow::Cow, collections::HashMap}; +const MAX_TIMESTAMP_AGE_SECS: i64 = 300; // 5 minutes + +fn validate_unix_timestamp(timestamp_str: &str) -> Result<(), AuthenticationError> { + let ts: i64 = timestamp_str + .parse() + .map_err(|_| AuthenticationError::InvalidTimestamp)?; + let now = chrono::Utc::now().timestamp(); + let diff = now - ts; + if diff > MAX_TIMESTAMP_AGE_SECS { + return Err(AuthenticationError::TimestampTooOldError); + } + if diff < -MAX_TIMESTAMP_AGE_SECS { + return Err(AuthenticationError::FutureTimestampError); + } + Ok(()) +} + pub type HmacSha256 = Hmac; pub type HmacSha512 = Hmac; pub type HmacSha1 = Hmac; @@ -82,6 +99,11 @@ mod slack { SignatureAuthenticationDetails::new(HmacAlgorithm::Sha256, Encoding::Hex), )) } + + fn validate_timestamp(&self, headers: &HeaderMap) -> Result<(), AuthenticationError> { + let ts = headers.try_get_webhook_header("X-Slack-Request-Timestamp")?; + validate_unix_timestamp(ts) + } } } @@ -126,6 +148,13 @@ mod stripe { SignatureAuthenticationDetails::new(HmacAlgorithm::Sha256, Encoding::Hex), )) } + + fn validate_timestamp(&self, headers: &HeaderMap) -> Result<(), AuthenticationError> { + let sig_header = headers.try_get_webhook_header("STRIPE-SIGNATURE")?; + let sig = parse_signature(sig_header, (",", "=")); + let ts = *sig.get("t").ok_or(AuthenticationError::InvalidTimestamp)?; + validate_unix_timestamp(ts) + } } } @@ -170,6 +199,13 @@ mod tiktok { SignatureAuthenticationDetails::new(HmacAlgorithm::Sha256, Encoding::Hex), )) } + + fn validate_timestamp(&self, headers: &HeaderMap) -> Result<(), AuthenticationError> { + let sig_header = headers.try_get_webhook_header("TikTok-Signature")?; + let sig = parse_signature(sig_header, (",", "=")); + let ts = *sig.get("t").ok_or(AuthenticationError::InvalidTimestamp)?; + validate_unix_timestamp(ts) + } } } @@ -244,6 +280,23 @@ mod twitch { Ok(Some(response.into_response())) } + + fn validate_timestamp(&self, headers: &HeaderMap) -> Result<(), AuthenticationError> { + let ts_str = headers.try_get_webhook_header("Twitch-Eventsub-Message-Timestamp")?; + let ts: chrono::DateTime = chrono::DateTime::parse_from_rfc3339(ts_str) + .map_err(|_| AuthenticationError::InvalidTimestamp)? + .into(); + let now = chrono::Utc::now(); + let diff = (now - ts).num_seconds(); + // Twitch recommends 10 minutes tolerance + if diff > 600 { + return Err(AuthenticationError::TimestampTooOldError); + } + if diff < -600 { + return Err(AuthenticationError::FutureTimestampError); + } + Ok(()) + } } } @@ -321,6 +374,11 @@ mod zoom { SignatureAuthenticationDetails::new(HmacAlgorithm::Sha256, Encoding::Hex), )) } + + fn validate_timestamp(&self, headers: &HeaderMap) -> Result<(), AuthenticationError> { + let ts = headers.try_get_webhook_header("x-zm-request-timestamp")?; + validate_unix_timestamp(ts) + } } } @@ -397,6 +455,10 @@ pub trait WebhookHandler { headers: &'header HeaderMap, raw_payload: &'payload str, ) -> Result, AuthenticationError>; + + fn validate_timestamp(&self, _headers: &HeaderMap) -> Result<(), AuthenticationError> { + Ok(()) + } } #[derive(Clone, Copy, Debug, Serialize, Deserialize)] @@ -592,6 +654,10 @@ impl AuthenticationMethod { return Ok(Some(challenge_response)); } + if let Some(handler) = handler { + handler.validate_timestamp(headers)?; + } + let authentication_data = match handler { Some(handler) => handler.get_hmac_authentication_data(headers, raw_payload)?, None => { @@ -621,7 +687,7 @@ impl AuthenticationMethod { let api_key_to_cmp = headers .try_get_webhook_header(&api_key_header) .map_err(|_| AuthenticationError::InvalidApiKey)?; - if api_key_to_cmp != api_key_secret { + if !constant_time_eq(api_key_to_cmp.as_bytes(), api_key_secret.as_bytes()) { return Err(AuthenticationError::InvalidApiKey); } } @@ -654,8 +720,11 @@ impl AuthenticationMethod { return Err(AuthenticationError::UnauthorizedBasicHttpAuth); } - if credentials.get(0).unwrap() != username - || credentials.get(1).unwrap() != password + if !constant_time_eq(credentials.get(0).unwrap().as_bytes(), username.as_bytes()) + || !constant_time_eq( + credentials.get(1).unwrap().as_bytes(), + password.as_bytes(), + ) { return Err(AuthenticationError::UnauthorizedBasicHttpAuth); } @@ -667,7 +736,6 @@ impl AuthenticationMethod { } #[derive(thiserror::Error, Debug)] -#[allow(unused)] pub enum AuthenticationError { #[error("failed to parse timestamp")] InvalidTimestamp, @@ -1207,11 +1275,15 @@ mod tests { headers } + fn current_timestamp() -> String { + chrono::Utc::now().timestamp().to_string() + } + #[test] fn test_slack_authenticate_valid() { let secret = "slack_signing_secret"; let payload = "token=xxx&command=%2Ftest".to_string(); - let timestamp = "1531420618"; + let timestamp = ¤t_timestamp(); let headers = slack_headers(secret, &payload, timestamp); let method = AuthenticationMethod::Signature(SignatureAuthentication { @@ -1225,7 +1297,7 @@ mod tests { } #[test] - fn test_slack_authenticate_wrong_timestamp() { + fn test_slack_authenticate_stale_timestamp_rejected() { let secret = "slack_secret"; let payload = "data".to_string(); let headers = slack_headers(secret, &payload, "1000000000"); @@ -1235,10 +1307,10 @@ mod tests { secret_key: secret.to_string(), authentication_config: None, }); - // Constructed with timestamp "1000000000" but that's valid - it just needs to match - assert!(method - .authenticate_http_request(&headers, Some(&payload)) - .is_ok()); + assert!(matches!( + method.authenticate_http_request(&headers, Some(&payload)), + Err(AuthenticationError::TimestampTooOldError) + )); } // --- Stripe webhook end-to-end --- @@ -1259,7 +1331,7 @@ mod tests { fn test_stripe_authenticate_valid() { let secret = "whsec_stripe_secret"; let payload = r#"{"id":"evt_123"}"#.to_string(); - let timestamp = "1614556800"; + let timestamp = ¤t_timestamp(); let headers = stripe_headers(secret, &payload, timestamp); let method = AuthenticationMethod::Signature(SignatureAuthentication { @@ -1305,7 +1377,7 @@ mod tests { fn test_tiktok_authenticate_valid() { let secret = "tiktok_secret"; let payload = r#"{"event":"video.upload"}"#.to_string(); - let timestamp = "1700000000"; + let timestamp = ¤t_timestamp(); let headers = tiktok_headers(secret, &payload, timestamp); let method = AuthenticationMethod::Signature(SignatureAuthentication { @@ -1350,17 +1422,16 @@ mod tests { headers } + fn current_rfc3339_timestamp() -> String { + chrono::Utc::now().to_rfc3339() + } + #[test] fn test_twitch_authenticate_valid_notification() { let secret = "twitch_secret"; let payload = r#"{"subscription":{},"event":{"user_id":"123"}}"#.to_string(); - let headers = twitch_headers( - secret, - &payload, - "msg-123", - "2024-01-01T00:00:00Z", - "notification", - ); + let ts = current_rfc3339_timestamp(); + let headers = twitch_headers(secret, &payload, "msg-123", &ts, "notification"); let method = AuthenticationMethod::Signature(SignatureAuthentication { signature_provider: WebhookType::Twitch, @@ -1376,11 +1447,12 @@ mod tests { fn test_twitch_challenge_response() { let secret = "twitch_secret"; let payload = r#"{"challenge":"test_challenge_string","subscription":{"id":"sub-123"}}"#; + let ts = current_rfc3339_timestamp(); let headers = twitch_headers( secret, payload, "msg-456", - "2024-01-01T00:00:00Z", + &ts, "webhook_callback_verification", ); @@ -1396,13 +1468,8 @@ mod tests { fn test_twitch_non_challenge_returns_none() { let secret = "twitch_secret"; let payload = r#"{"subscription":{},"event":{}}"#; - let headers = twitch_headers( - secret, - payload, - "msg-789", - "2024-01-01T00:00:00Z", - "notification", - ); + let ts = current_rfc3339_timestamp(); + let headers = twitch_headers(secret, payload, "msg-789", &ts, "notification"); let handler = WebhookType::Twitch.get_webhook_handler().unwrap(); let config_data = SignatureConfigData { secret_key: secret }; @@ -1434,7 +1501,7 @@ mod tests { fn test_zoom_authenticate_valid() { let secret = "zoom_secret"; let payload = r#"{"event":"meeting.started"}"#.to_string(); - let timestamp = "1700000000"; + let timestamp = ¤t_timestamp(); let headers = zoom_headers(secret, &payload, timestamp); let method = AuthenticationMethod::Signature(SignatureAuthentication { @@ -1789,4 +1856,84 @@ mod tests { let response = AuthenticationError::InvalidTimestamp.into_response(); assert_eq!(response.status(), StatusCode::BAD_REQUEST); } + + // --- validate_unix_timestamp --- + + #[test] + fn test_validate_unix_timestamp_current() { + let now = chrono::Utc::now().timestamp().to_string(); + assert!(validate_unix_timestamp(&now).is_ok()); + } + + #[test] + fn test_validate_unix_timestamp_recent() { + let ts = (chrono::Utc::now().timestamp() - 60).to_string(); + assert!(validate_unix_timestamp(&ts).is_ok()); + } + + #[test] + fn test_validate_unix_timestamp_too_old() { + let ts = (chrono::Utc::now().timestamp() - 600).to_string(); + assert!(matches!( + validate_unix_timestamp(&ts), + Err(AuthenticationError::TimestampTooOldError) + )); + } + + #[test] + fn test_validate_unix_timestamp_future() { + let ts = (chrono::Utc::now().timestamp() + 600).to_string(); + assert!(matches!( + validate_unix_timestamp(&ts), + Err(AuthenticationError::FutureTimestampError) + )); + } + + #[test] + fn test_validate_unix_timestamp_invalid() { + assert!(matches!( + validate_unix_timestamp("not-a-number"), + Err(AuthenticationError::InvalidTimestamp) + )); + } + + // --- Slack timestamp validation --- + + #[test] + fn test_slack_validate_timestamp_current() { + let handler = slack::Slack; + let mut headers = HeaderMap::new(); + let now = chrono::Utc::now().timestamp().to_string(); + headers.insert("X-Slack-Request-Timestamp", now.parse().unwrap()); + assert!(handler.validate_timestamp(&headers).is_ok()); + } + + #[test] + fn test_slack_validate_timestamp_stale() { + let handler = slack::Slack; + let mut headers = HeaderMap::new(); + let old = (chrono::Utc::now().timestamp() - 600).to_string(); + headers.insert("X-Slack-Request-Timestamp", old.parse().unwrap()); + assert!(handler.validate_timestamp(&headers).is_err()); + } + + // --- Twitch timestamp validation (ISO 8601) --- + + #[test] + fn test_twitch_validate_timestamp_current() { + let handler = twitch::Twitch; + let mut headers = HeaderMap::new(); + let now = chrono::Utc::now().to_rfc3339(); + headers.insert("Twitch-Eventsub-Message-Timestamp", now.parse().unwrap()); + assert!(handler.validate_timestamp(&headers).is_ok()); + } + + #[test] + fn test_twitch_validate_timestamp_stale() { + let handler = twitch::Twitch; + let mut headers = HeaderMap::new(); + let old = (chrono::Utc::now() - chrono::TimeDelta::seconds(1200)).to_rfc3339(); + headers.insert("Twitch-Eventsub-Message-Timestamp", old.parse().unwrap()); + assert!(handler.validate_timestamp(&headers).is_err()); + } } diff --git a/backend/windmill-trigger-http/src/lib.rs b/backend/windmill-trigger-http/src/lib.rs index f5fd563ab3..f851dfc12e 100644 --- a/backend/windmill-trigger-http/src/lib.rs +++ b/backend/windmill-trigger-http/src/lib.rs @@ -7,6 +7,7 @@ use tokio::sync::{RwLock, RwLockReadGuard}; use windmill_common::{ error::{Error, Result}, flows::Retry, + global_settings::HTTP_ROUTE_WORKSPACED_ROUTE, utils::ExpiringCacheEntry, worker::CLOUD_HOSTED, DB, @@ -273,13 +274,15 @@ pub async fn refresh_routers(db: &DB) -> Result<(bool, RwLockReadGuard<'_, Route .await?; let mut router = matchit::Router::new(); + let http_route_workspaced = *HTTP_ROUTE_WORKSPACED_ROUTE.read().await; for trigger in triggers { - let full_path = if trigger.workspaced_route || *CLOUD_HOSTED { - format!("/{}/{}", trigger.workspace_id, trigger.route_path) - } else { - format!("/{}", trigger.route_path) - }; + let full_path = + if trigger.workspaced_route || *CLOUD_HOSTED || http_route_workspaced { + format!("/{}/{}", trigger.workspace_id, trigger.route_path) + } else { + format!("/{}", trigger.route_path) + }; if trigger.is_static_website { router diff --git a/backend/windmill-trigger-kafka/src/handler_oss.rs b/backend/windmill-trigger-kafka/src/handler_oss.rs index ace4b87d0a..2e1cb8bfb4 100644 --- a/backend/windmill-trigger-kafka/src/handler_oss.rs +++ b/backend/windmill-trigger-kafka/src/handler_oss.rs @@ -8,7 +8,7 @@ pub use super::handler_ee::*; #[cfg(not(feature = "private"))] use { super::KafkaTrigger, - axum::async_trait, + async_trait::async_trait, sqlx::PgConnection, windmill_api_auth::ApiAuthed, windmill_common::{ diff --git a/backend/windmill-trigger-mqtt/src/handler.rs b/backend/windmill-trigger-mqtt/src/handler.rs index 6edcfbabc3..49fb701241 100644 --- a/backend/windmill-trigger-mqtt/src/handler.rs +++ b/backend/windmill-trigger-mqtt/src/handler.rs @@ -1,4 +1,4 @@ -use axum::async_trait; +use async_trait::async_trait; use itertools::Itertools; use sqlx::{types::Json as SqlxJson, PgConnection}; use windmill_api_auth::ApiAuthed; diff --git a/backend/windmill-trigger-nats/src/handler_oss.rs b/backend/windmill-trigger-nats/src/handler_oss.rs index b00d973621..f322335cb8 100644 --- a/backend/windmill-trigger-nats/src/handler_oss.rs +++ b/backend/windmill-trigger-nats/src/handler_oss.rs @@ -8,7 +8,7 @@ use windmill_trigger::TriggerData; #[cfg(not(feature = "private"))] use { super::NatsTrigger, - axum::async_trait, + async_trait::async_trait, sqlx::PgConnection, windmill_api_auth::ApiAuthed, windmill_common::{ diff --git a/backend/windmill-trigger-postgres/src/handler.rs b/backend/windmill-trigger-postgres/src/handler.rs index cbb149af52..dc2f4776fd 100644 --- a/backend/windmill-trigger-postgres/src/handler.rs +++ b/backend/windmill-trigger-postgres/src/handler.rs @@ -1,7 +1,7 @@ use std::collections::HashMap; +use async_trait::async_trait; use axum::{ - async_trait, extract::Path, routing::{delete, get, post}, Extension, Json, Router, @@ -282,10 +282,10 @@ impl TriggerCrud for PostgresTrigger { fn additional_routes(&self) -> Router { Router::new() - .route("/get_template_script/:id", get(get_template_script)) + .route("/get_template_script/{id}", get(get_template_script)) .route("/create_template_script", post(create_template_script)) .route( - "/is_valid_postgres_configuration/*path", + "/is_valid_postgres_configuration/{*path}", get(is_database_in_logical_level), ) .nest("/publication", publication_service()) @@ -296,25 +296,31 @@ impl TriggerCrud for PostgresTrigger { fn publication_service() -> Router { Router::new() - .route("/get/:publication_name/*path", get(get_publication_info)) - .route("/create/:publication_name/*path", post(create_publication)) - .route("/update/:publication_name/*path", post(alter_publication)) + .route("/get/{publication_name}/{*path}", get(get_publication_info)) .route( - "/delete/:publication_name/*path", + "/create/{publication_name}/{*path}", + post(create_publication), + ) + .route( + "/update/{publication_name}/{*path}", + post(alter_publication), + ) + .route( + "/delete/{publication_name}/{*path}", delete(delete_publication), ) - .route("/list/*path", get(list_database_publication)) + .route("/list/{*path}", get(list_database_publication)) } fn slot_service() -> Router { Router::new() - .route("/list/*path", get(list_slot_name)) - .route("/create/*path", post(create_slot)) - .route("/delete/*path", delete(drop_slot_name)) + .route("/list/{*path}", get(list_slot_name)) + .route("/create/{*path}", post(create_slot)) + .route("/delete/{*path}", delete(drop_slot_name)) } fn postgres_service() -> Router { - Router::new().route("/version/*path", get(get_postgres_version)) + Router::new().route("/version/{*path}", get(get_postgres_version)) } async fn check_if_logical_replication_slot_exist( diff --git a/backend/windmill-trigger-sqs/src/handler_oss.rs b/backend/windmill-trigger-sqs/src/handler_oss.rs index 90396e9994..fc72159e24 100644 --- a/backend/windmill-trigger-sqs/src/handler_oss.rs +++ b/backend/windmill-trigger-sqs/src/handler_oss.rs @@ -5,7 +5,7 @@ pub use super::handler_ee::*; #[cfg(not(feature = "private"))] use { super::SqsTrigger, - axum::async_trait, + async_trait::async_trait, sqlx::PgConnection, windmill_api_auth::ApiAuthed, windmill_common::{ diff --git a/backend/windmill-trigger-websocket/src/handler.rs b/backend/windmill-trigger-websocket/src/handler.rs index fd0080950d..223868e96e 100644 --- a/backend/windmill-trigger-websocket/src/handler.rs +++ b/backend/windmill-trigger-websocket/src/handler.rs @@ -1,6 +1,6 @@ use std::borrow::Cow; -use axum::async_trait; +use async_trait::async_trait; use itertools::Itertools; use serde_json::value::RawValue; use sqlx::{types::Json as SqlxJson, PgConnection}; @@ -36,6 +36,7 @@ impl TriggerCrud for WebsocketTrigger { const ADDITIONAL_SELECT_FIELDS: &[&'static str] = &[ "url", "filters", + "filter_logic", "initial_messages", "url_runnable_args", "can_return_message", @@ -103,6 +104,7 @@ impl TriggerCrud for WebsocketTrigger { is_flow, mode, filters, + filter_logic, initial_messages, url_runnable_args, edited_by, @@ -114,7 +116,7 @@ impl TriggerCrud for WebsocketTrigger { error_handler_args, retry ) VALUES ( - $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, now(), $14, $15, $16 + $1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, now(), $15, $16, $17 ) "#, w_id, @@ -124,6 +126,7 @@ impl TriggerCrud for WebsocketTrigger { trigger.base.is_flow, trigger.base.mode() as _, &filters as _, + trigger.config.filter_logic, &initial_messages as _, trigger .config @@ -178,26 +181,28 @@ impl TriggerCrud for WebsocketTrigger { path = $3, is_flow = $4, filters = $5, - initial_messages = $6, - url_runnable_args = $7, - edited_by = $8, - permissioned_as = $9, - can_return_message = $10, - can_return_error_result = $11, + filter_logic = $6, + initial_messages = $7, + url_runnable_args = $8, + edited_by = $9, + permissioned_as = $10, + can_return_message = $11, + can_return_error_result = $12, edited_at = now(), server_id = NULL, error = NULL, - error_handler_path = $14, - error_handler_args = $15, - retry = $16 + error_handler_path = $15, + error_handler_args = $16, + retry = $17 WHERE - workspace_id = $12 AND path = $13 + workspace_id = $13 AND path = $14 ", trigger.config.url, trigger.base.script_path, trigger.base.path, trigger.base.is_flow, filters.as_slice() as &[SqlxJson>], + trigger.config.filter_logic, initial_messages.as_slice() as &[SqlxJson>], trigger .config diff --git a/backend/windmill-trigger-websocket/src/lib.rs b/backend/windmill-trigger-websocket/src/lib.rs index ca754d8349..4fe96bb7c2 100644 --- a/backend/windmill-trigger-websocket/src/lib.rs +++ b/backend/windmill-trigger-websocket/src/lib.rs @@ -1,12 +1,9 @@ use std::collections::HashMap; -use windmill_api_auth::ApiAuthed; -use windmill_trigger::trigger_helpers::{ - trigger_runnable_and_wait_for_raw_result_with_error_ctx, TriggerJobArgs, -}; use serde::{Deserialize, Serialize}; use serde_json::value::RawValue; use sqlx::{types::Json as SqlxJson, FromRow}; +use windmill_api_auth::ApiAuthed; use windmill_common::{ error::{Error, Result}, jobs::JobTriggerKind, @@ -15,6 +12,9 @@ use windmill_common::{ DB, }; use windmill_queue::PushArgsOwned; +use windmill_trigger::trigger_helpers::{ + trigger_runnable_and_wait_for_raw_result_with_error_ctx, TriggerJobArgs, +}; pub mod handler; pub mod listener; @@ -30,11 +30,17 @@ impl TriggerJobArgs for WebsocketTrigger { } } +fn default_filter_logic() -> String { + "and".to_string() +} + #[derive(Debug, Clone, FromRow, Serialize, Deserialize)] pub struct WebsocketConfig { pub url: String, #[serde(default)] pub filters: Vec>>, + #[serde(default = "default_filter_logic")] + pub filter_logic: String, #[serde(skip_serializing_if = "Option::is_none")] pub initial_messages: Option>>>, #[serde(skip_serializing_if = "Option::is_none")] @@ -49,6 +55,8 @@ pub struct WebsocketConfig { pub struct WebsocketConfigRequest { url: String, filters: Vec, + #[serde(default = "default_filter_logic")] + filter_logic: String, initial_messages: Option>, url_runnable_args: Option, can_return_message: bool, diff --git a/backend/windmill-trigger-websocket/src/listener.rs b/backend/windmill-trigger-websocket/src/listener.rs index 8aab61d9c6..205f2fe501 100644 --- a/backend/windmill-trigger-websocket/src/listener.rs +++ b/backend/windmill-trigger-websocket/src/listener.rs @@ -18,7 +18,7 @@ use windmill_common::{ DB, }; use windmill_queue::PushArgsOwned; -use windmill_trigger::filter::{is_value_superset, Filter, JsonFilter}; +use windmill_trigger::filter::{check_filters, Filter}; use windmill_trigger::listener::ListeningTrigger; use windmill_trigger::trigger_helpers::{ trigger_runnable, trigger_runnable_and_wait_for_raw_result, @@ -267,26 +267,8 @@ impl Listener for WebsocketTrigger { match msg { tokio_tungstenite::tungstenite::Message::Text(text) => { tracing::debug!("Received text message from WebSocket {}: {}", url, text); - let mut should_handle = true; - for filter in &filters { - match filter { - Filter::JsonFilter(JsonFilter { key, value }) => { - let mut deserializer = serde_json::Deserializer::from_str(text.as_str()); - should_handle = match is_value_superset(&mut deserializer, key, &value) { - Ok(filter_match) => { - filter_match - }, - Err(err) => { - tracing::warn!("Error deserializing filter for WebSocket {}: {:?}", url, err); - false - } - }; - } - } - if !should_handle { - break; - } - } + let use_or = listening_trigger.trigger_config.filter_logic == "or"; + let should_handle = check_filters(&text, &filters, use_or); if should_handle { let trigger_info = HashMap::from([ ("url".to_string(), to_raw_value(&listening_trigger.trigger_config.url)), diff --git a/backend/windmill-trigger/src/filter.rs b/backend/windmill-trigger/src/filter.rs index a1dd59f848..3c9c857058 100644 --- a/backend/windmill-trigger/src/filter.rs +++ b/backend/windmill-trigger/src/filter.rs @@ -80,6 +80,27 @@ where deserializer.deserialize_map(SupersetVisitor { key, value_to_check }) } +pub fn check_filters(text: &str, filters: &[Filter], use_or_logic: bool) -> bool { + if filters.is_empty() { + return true; + } + + let check = |filter: &Filter| -> bool { + match filter { + Filter::JsonFilter(JsonFilter { key, value }) => { + let mut deserializer = serde_json::Deserializer::from_str(text); + is_value_superset(&mut deserializer, key, value).unwrap_or(false) + } + } + }; + + if use_or_logic { + filters.iter().any(check) + } else { + filters.iter().all(check) + } +} + #[cfg(test)] mod tests { use super::*; diff --git a/backend/windmill-trigger/src/handler.rs b/backend/windmill-trigger/src/handler.rs index 1f89be9419..16f095a903 100644 --- a/backend/windmill-trigger/src/handler.rs +++ b/backend/windmill-trigger/src/handler.rs @@ -374,11 +374,11 @@ pub fn trigger_routes() -> Router { let mut router = Router::new() .route("/create", post(create_trigger::)) .route("/list", get(list_triggers::)) - .route("/get/*path", get(get_trigger::)) - .route("/update/*path", post(update_trigger::)) - .route("/delete/*path", delete(delete_trigger::)) - .route("/exists/*path", get(exists_trigger::)) - .route("/setmode/*path", post(set_trigger_mode::)); + .route("/get/{*path}", get(get_trigger::)) + .route("/update/{*path}", post(update_trigger::)) + .route("/delete/{*path}", delete(delete_trigger::)) + .route("/exists/{*path}", get(exists_trigger::)) + .route("/setmode/{*path}", post(set_trigger_mode::)); if T::SUPPORTS_TEST_CONNECTION { router = router.route("/test", post(test_connection::)); @@ -610,6 +610,17 @@ async fn delete_trigger( })?; let mut tx = user_db.begin(&authed).await?; + + // Capture trigger data for trashbin before deleting + let trash_data: Option = sqlx::query_scalar(&format!( + "SELECT jsonb_build_object('row', to_jsonb(t), 'table_name', '{table}') FROM {table} t WHERE path = $1 AND workspace_id = $2", + table = T::TABLE_NAME + )) + .bind(path) + .bind(&workspace_id) + .fetch_optional(&mut *tx) + .await?; + let deleted = handler .delete_by_path(&mut *tx, &workspace_id, path) .await?; @@ -621,6 +632,19 @@ async fn delete_trigger( ))); } + if let Some(data) = trash_data { + let item_kind = format!("{}_trigger", T::TRIGGER_TYPE); + windmill_common::trashbin::move_to_trash( + &mut *tx, + &workspace_id, + &item_kind, + path, + data, + &authed.username, + ) + .await?; + } + audit_log( &mut *tx, &authed, diff --git a/backend/windmill-types/src/flows.rs b/backend/windmill-types/src/flows.rs index 699e71b105..70199dbc70 100644 --- a/backend/windmill-types/src/flows.rs +++ b/backend/windmill-types/src/flows.rs @@ -1123,3 +1123,40 @@ pub fn add_virtual_items_if_necessary(modules: &mut Vec) { }); } } + +#[cfg(test)] +mod tests { + use super::*; + use serde_json::json; + + #[test] + fn flow_value_ignores_notes_and_groups() { + // FlowValue should parse successfully even when notes/groups are present — + // it just ignores them (they're not its fields). + let input = json!({ + "modules": [], + "notes": [{"id": "n1", "text": "hello", "color": "yellow", "type": "free"}], + "groups": [{"start_id": "a", "end_id": "b", "summary": "grp"}] + }); + let val: FlowValue = serde_json::from_value(input).unwrap(); + assert_eq!(val.modules.len(), 0); + + // Round-trip through FlowValue drops notes/groups (by design) + let output = serde_json::to_string(&val).unwrap(); + assert!(!output.contains("notes")); + assert!(!output.contains("groups")); + } + + #[test] + fn flow_value_parses_modules() { + let input = json!({ + "modules": [{ + "id": "a", + "value": {"type": "identity"} + }], + "notes": [{"id": "n1", "text": "t", "color": "blue", "type": "free"}] + }); + let val: FlowValue = serde_json::from_value(input).unwrap(); + assert_eq!(val.modules.len(), 1); + } +} diff --git a/backend/windmill-types/src/jobs.rs b/backend/windmill-types/src/jobs.rs index 28c3242695..99a5607508 100644 --- a/backend/windmill-types/src/jobs.rs +++ b/backend/windmill-types/src/jobs.rs @@ -118,6 +118,10 @@ impl JobKind { JobKind::FlowDependencies | JobKind::AppDependencies | JobKind::Dependencies ) } + + pub fn is_preview(&self) -> bool { + matches!(self, JobKind::Preview | JobKind::FlowPreview) + } } #[derive(sqlx::FromRow, Debug, Serialize, Clone)] diff --git a/backend/windmill-types/src/scripts.rs b/backend/windmill-types/src/scripts.rs index 5eca2be82c..e5a980eaba 100644 --- a/backend/windmill-types/src/scripts.rs +++ b/backend/windmill-types/src/scripts.rs @@ -450,6 +450,8 @@ pub struct ScriptHistory { pub script_hash: ScriptHash, #[serde(skip_serializing_if = "Option::is_none")] pub deployment_msg: Option, + #[serde(skip_serializing_if = "Option::is_none")] + pub created_at: Option>, } #[derive(Deserialize)] @@ -510,6 +512,8 @@ pub struct NewScript { pub assets: Option>, #[serde(skip_serializing_if = "Option::is_none")] pub modules: Option>, + #[serde(default)] + pub auto_parent: Option, } // IMPORTANT: update this Hash impl when adding fields to NewScript diff --git a/backend/windmill-worker/Cargo.toml b/backend/windmill-worker/Cargo.toml index ed82298038..8955041435 100644 --- a/backend/windmill-worker/Cargo.toml +++ b/backend/windmill-worker/Cargo.toml @@ -10,7 +10,7 @@ path = "src/lib.rs" [features] default = [] -private = ["windmill-worker-volumes/private", "windmill-queue/private"] +private = ["windmill-worker-volumes/private", "windmill-queue/private", "windmill-common/private"] mcp = ["dep:windmill-mcp"] prometheus = ["dep:prometheus", "windmill-common/prometheus"] enterprise = ["windmill-queue/enterprise", "windmill-git-sync/enterprise", "windmill-common/enterprise", "windmill-worker-volumes/enterprise", "dep:pem", "dep:tokio-util", "dep:opentelemetry-proto", "dep:prost", "dep:hudsucker", "dep:rcgen", "dep:hyper-http-proxy", "dep:hyper-tls", "dep:hyper-util"] @@ -110,6 +110,7 @@ gcp_auth = { workspace = true, optional = true } rust_decimal.workspace = true jsonwebtoken.workspace = true sha2.workspace = true +hmac.workspace = true pem = { workspace = true, optional = true } urlencoding.workspace = true nix.workspace = true @@ -145,6 +146,7 @@ rcgen = { workspace = true, optional = true } [dev-dependencies] tempfile.workspace = true +x509-parser.workspace = true [build-dependencies] libffi-sys = { workspace = true, optional = true } diff --git a/backend/windmill-worker/loader.bun.js b/backend/windmill-worker/loader.bun.js index f2a00de0cf..d03f19b6ba 100644 --- a/backend/windmill-worker/loader.bun.js +++ b/backend/windmill-worker/loader.bun.js @@ -21,7 +21,7 @@ const p = { // On Windows, normalize path to POSIX format to match args.path from Bun's resolver const cdirPosix = cdir.replace(/\\/g, "/").replace(/^[a-zA-Z]:/, ""); const filterResolve = new RegExp( - `^(?!\\.\/main\\.ts)(?!${cdir}\/main\\.ts)(?!${cdirPosix}\/main\\.ts)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$` + `^(?!\\.\/main\\.ts)(?!\\.\/_wm_)(?!${cdir}\/main\\.ts)(?!${cdir}\/_wm_)(?!${cdirPosix}\/main\\.ts)(?!${cdirPosix}\/_wm_)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$` ); let cdirNodeModules = `${cdir}/node_modules/`; diff --git a/backend/windmill-worker/loader.bun.windows.js b/backend/windmill-worker/loader.bun.windows.js index 7d14ffcbc3..e89b70c26c 100644 --- a/backend/windmill-worker/loader.bun.windows.js +++ b/backend/windmill-worker/loader.bun.windows.js @@ -27,7 +27,7 @@ const p = { const cdirFwd = cdir.replace(/\\/g, "/"); const cdirPosix = cdirFwd.replace(/^[a-zA-Z]:/, ""); const filterResolve = new RegExp( - `^(?!\\.\/main\\.ts)(?!${cdirFwd}\/main\\.ts)(?!${cdirPosix}\/main\\.ts)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$` + `^(?!\\.\/main\\.ts)(?!\\.\/_wm_)(?!${cdirFwd}\/main\\.ts)(?!${cdirFwd}\/_wm_)(?!${cdirPosix}\/main\\.ts)(?!${cdirPosix}\/_wm_)(?!(?:/private)?${cdirNoPrivate}\/wrapper\\.mjs).*\\.ts$` ); let cdirNodeModules = `${cdirFwd}/node_modules/`; diff --git a/backend/windmill-worker/src/ai/image_handler.rs b/backend/windmill-worker/src/ai/image_handler.rs index 63d8aeaec3..1e30f6bb32 100644 --- a/backend/windmill-worker/src/ai/image_handler.rs +++ b/backend/windmill-worker/src/ai/image_handler.rs @@ -70,6 +70,29 @@ pub async fn download_and_encode_s3_image( Ok((mime_type.to_string(), base64_data)) } +/// Convert an S3Object to the appropriate ContentPart based on MIME type. +pub async fn s3_object_to_content_part( + s3_object: &S3Object, + client: &AuthedClient, + workspace_id: &str, +) -> Result { + let (mime_type, file_bytes) = + download_and_encode_s3_image(s3_object, client, workspace_id).await?; + let data_url = format!("data:{};base64,{}", mime_type, file_bytes); + + if windmill_common::ai_types::is_document_mime(&mime_type) { + let filename = s3_object + .s3 + .rsplit('/') + .next() + .unwrap_or("document.pdf") + .to_string(); + Ok(ContentPart::File { file: FileData { filename, file_data: data_url } }) + } else { + Ok(ContentPart::ImageUrl { image_url: ImageUrlData { url: data_url } }) + } +} + /// Prepare messages for API by converting S3Objects to base64 ImageUrls pub async fn prepare_messages_for_api( messages: &[OpenAIMessage], @@ -92,15 +115,10 @@ pub async fn prepare_messages_for_api( for part in parts { match part { ContentPart::S3Object { s3_object } => { - // Convert S3Object to base64 image URL - let (mime_type, image_bytes) = - download_and_encode_s3_image(s3_object, client, workspace_id) - .await?; - prepared_content.push(ContentPart::ImageUrl { - image_url: ImageUrlData { - url: format!("data:{};base64,{}", mime_type, image_bytes), - }, - }); + prepared_content.push( + s3_object_to_content_part(s3_object, client, workspace_id) + .await?, + ); } other => { // Keep Text and ImageUrl as-is diff --git a/backend/windmill-worker/src/ai/providers/anthropic.rs b/backend/windmill-worker/src/ai/providers/anthropic.rs index 847c706d71..19f539a3d4 100644 --- a/backend/windmill-worker/src/ai/providers/anthropic.rs +++ b/backend/windmill-worker/src/ai/providers/anthropic.rs @@ -92,7 +92,9 @@ pub enum AnthropicRequestContent { cache_control: Option, }, #[serde(rename = "image")] - Image { source: AnthropicImageSource }, + Image { source: AnthropicBase64Source }, + #[serde(rename = "document")] + Document { source: AnthropicBase64Source }, #[serde(rename = "tool_use")] ToolUse { id: String, name: String, input: Box }, #[serde(rename = "tool_result")] @@ -104,9 +106,9 @@ pub enum AnthropicRequestContent { }, } -/// Image source for Anthropic API +/// Base64 source for Anthropic API (used by both Image and Document content blocks) #[derive(Serialize, Debug)] -pub struct AnthropicImageSource { +pub struct AnthropicBase64Source { pub r#type: String, pub media_type: String, pub data: String, @@ -270,10 +272,20 @@ fn convert_content_to_anthropic(content: &Option) -> Vec { - // Handle base64 images if let Some((media_type, data)) = parse_data_url(&image_url.url) { result.push(AnthropicRequestContent::Image { - source: AnthropicImageSource { + source: AnthropicBase64Source { + r#type: "base64".to_string(), + media_type, + data, + }, + }); + } + } + ContentPart::File { file } => { + if let Some((media_type, data)) = parse_data_url(&file.file_data) { + result.push(AnthropicRequestContent::Document { + source: AnthropicBase64Source { r#type: "base64".to_string(), media_type, data, diff --git a/backend/windmill-worker/src/ai/providers/google_ai.rs b/backend/windmill-worker/src/ai/providers/google_ai.rs index 8ce0b5b951..a490bae7bc 100644 --- a/backend/windmill-worker/src/ai/providers/google_ai.rs +++ b/backend/windmill-worker/src/ai/providers/google_ai.rs @@ -84,13 +84,13 @@ impl GoogleAIQueryBuilder { ); } - if let Some(images) = args.images { - for image in images.iter() { - if !image.s3.is_empty() { - let (mime_type, image_bytes) = - download_and_encode_s3_image(image, client, workspace_id).await?; + if let Some(attachments) = args.attachments { + for attachment in attachments.iter() { + if !attachment.s3.is_empty() { + let (mime_type, file_bytes) = + download_and_encode_s3_image(attachment, client, workspace_id).await?; parts.push(GeminiPart::InlineData { - inline_data: GeminiInlineData { mime_type, data: image_bytes }, + inline_data: GeminiInlineData { mime_type, data: file_bytes }, }); } } diff --git a/backend/windmill-worker/src/ai/providers/openai.rs b/backend/windmill-worker/src/ai/providers/openai.rs index 51bb6e3ec3..4b34a27174 100644 --- a/backend/windmill-worker/src/ai/providers/openai.rs +++ b/backend/windmill-worker/src/ai/providers/openai.rs @@ -4,7 +4,7 @@ use serde_json::value::RawValue; use windmill_common::{ai_providers::AIProvider, client::AuthedClient, error::Error}; use crate::ai::{ - image_handler::{download_and_encode_s3_image, prepare_messages_for_api}, + image_handler::{prepare_messages_for_api, s3_object_to_content_part}, query_builder::{BuildRequestArgs, ParsedResponse, QueryBuilder, StreamEventProcessor}, sse::{OpenAIResponsesSSEParser, SSEParser}, types::*, @@ -103,6 +103,8 @@ pub enum ImageGenerationContent { InputText { text: String }, #[serde(rename = "input_image")] InputImage { image_url: String }, + #[serde(rename = "input_file")] + InputFile { filename: String, file_data: String }, } /// Output content for assistant messages in Responses API @@ -240,6 +242,12 @@ fn convert_content_to_responses_format( image_url: image_url.url.clone(), }) } + ContentPart::File { file } => { + Some(ImageGenerationContent::InputFile { + filename: file.filename.clone(), + file_data: file.file_data.clone(), + }) + } // S3 objects should have been resolved earlier, but handle gracefully ContentPart::S3Object { .. } => None, }) @@ -421,15 +429,26 @@ impl OpenAIQueryBuilder { let mut content = vec![ImageGenerationContent::InputText { text: args.user_message.to_string() }]; - // Add images if provided - if let Some(images) = args.images { - for image in images.iter() { - if !image.s3.is_empty() { - let (mime_type, image_bytes) = - download_and_encode_s3_image(image, client, workspace_id).await?; - content.push(ImageGenerationContent::InputImage { - image_url: format!("data:{};base64,{}", mime_type, image_bytes), - }); + // Add attachments (images, PDFs, etc.) if provided + if let Some(attachments) = args.attachments { + for attachment in attachments.iter() { + if !attachment.s3.is_empty() { + let part = + s3_object_to_content_part(attachment, client, workspace_id).await?; + match part { + ContentPart::File { file } => { + content.push(ImageGenerationContent::InputFile { + filename: file.filename, + file_data: file.file_data, + }); + } + ContentPart::ImageUrl { image_url } => { + content.push(ImageGenerationContent::InputImage { + image_url: image_url.url, + }); + } + _ => {} + } } } } diff --git a/backend/windmill-worker/src/ai/query_builder.rs b/backend/windmill-worker/src/ai/query_builder.rs index c1d0d79a05..27fc424475 100644 --- a/backend/windmill-worker/src/ai/query_builder.rs +++ b/backend/windmill-worker/src/ai/query_builder.rs @@ -28,7 +28,7 @@ pub struct BuildRequestArgs<'a> { pub output_type: &'a OutputType, pub system_prompt: Option<&'a str>, pub user_message: &'a str, - pub images: Option<&'a [S3Object]>, + pub attachments: Option<&'a [S3Object]>, pub has_websearch: bool, } diff --git a/backend/windmill-worker/src/ai/types.rs b/backend/windmill-worker/src/ai/types.rs index 62a13d958a..85528ccec1 100644 --- a/backend/windmill-worker/src/ai/types.rs +++ b/backend/windmill-worker/src/ai/types.rs @@ -26,7 +26,8 @@ use windmill_types::s3::S3Object; // Re-export shared types from windmill_common pub use windmill_common::ai_providers::AIPlatform; pub use windmill_common::ai_types::{ - ContentPart, ImageUrlData, OpenAIContent, OpenAIMessage, ToolDef, ToolDefFunction, UrlCitation, + ContentPart, FileData, ImageUrlData, OpenAIContent, OpenAIMessage, ToolDef, ToolDefFunction, + UrlCitation, }; /// same as OpenAIMessage but with agent_action field included in the serialization @@ -96,7 +97,8 @@ struct AIAgentArgsRaw { max_completion_tokens: Option, output_schema: Option, output_type: Option, - user_images: Option>, + #[serde(alias = "user_images")] + user_attachments: Option>, streaming: Option, max_iterations: Option, memory: Option, @@ -116,7 +118,7 @@ pub struct AIAgentArgs { pub max_completion_tokens: Option, pub output_schema: Option, pub output_type: Option, - pub user_images: Option>, + pub user_attachments: Option>, pub streaming: Option, pub max_iterations: Option, pub memory: Option, @@ -148,7 +150,7 @@ impl From for AIAgentArgs { max_completion_tokens: raw.max_completion_tokens, output_schema: raw.output_schema, output_type: raw.output_type, - user_images: raw.user_images, + user_attachments: raw.user_attachments, streaming: raw.streaming, max_iterations: raw.max_iterations, memory, diff --git a/backend/windmill-worker/src/ai_executor.rs b/backend/windmill-worker/src/ai_executor.rs index 2c8def4802..594830cebd 100644 --- a/backend/windmill-worker/src/ai_executor.rs +++ b/backend/windmill-worker/src/ai_executor.rs @@ -714,24 +714,45 @@ pub async fn run_agent( } }; - // Add user message if provided and non-empty - if let Some(ref user_message) = args.user_message { - if !user_message.is_empty() { + // Add user message and attachments as a single user message + // (Bedrock requires a text block alongside document blocks in the same message) + { + let has_message = args + .user_message + .as_ref() + .map(|m| !m.is_empty()) + .unwrap_or(false); + let has_attachments = args + .user_attachments + .as_ref() + .map(|a| !a.is_empty()) + .unwrap_or(false); + + if has_message && has_attachments { + let mut parts = vec![ContentPart::Text { + text: args.user_message.clone().unwrap(), + }]; + for attachment in args.user_attachments.as_ref().unwrap() { + if !attachment.s3.is_empty() { + parts.push(ContentPart::S3Object { s3_object: attachment.clone() }); + } + } messages.push(OpenAIMessage { role: "user".to_string(), - content: Some(OpenAIContent::Text(user_message.clone())), + content: Some(OpenAIContent::Parts(parts)), ..Default::default() }); - } - } - - // Add user images if provided - if let Some(ref user_images) = args.user_images { - if !user_images.is_empty() { + } else if has_message { + messages.push(OpenAIMessage { + role: "user".to_string(), + content: Some(OpenAIContent::Text(args.user_message.clone().unwrap())), + ..Default::default() + }); + } else if has_attachments { let mut parts = vec![]; - for image in user_images.iter() { - if !image.s3.is_empty() { - parts.push(ContentPart::S3Object { s3_object: image.clone() }); + for attachment in args.user_attachments.as_ref().unwrap() { + if !attachment.s3.is_empty() { + parts.push(ContentPart::S3Object { s3_object: attachment.clone() }); } } messages.push(OpenAIMessage { @@ -882,7 +903,7 @@ pub async fn run_agent( output_type, system_prompt: args.system_prompt.as_deref(), user_message: args.user_message.as_deref().unwrap_or(""), - images: args.user_images.as_deref(), + attachments: args.user_attachments.as_deref(), has_websearch, }; diff --git a/backend/windmill-worker/src/bash_executor.rs b/backend/windmill-worker/src/bash_executor.rs index a9947faaf5..011a72fe15 100644 --- a/backend/windmill-worker/src/bash_executor.rs +++ b/backend/windmill-worker/src/bash_executor.rs @@ -180,6 +180,17 @@ exit $exit_status let _ = write_file(job_dir, "result.out", "")?; let _ = write_file(job_dir, "result2.out", "")?; + // Forward DOCKER_HOST to the bash script when in docker mode so the docker CLI + // connects to the right daemon (e.g. a dind sidecar instead of /var/run/docker.sock) + let docker_envs: Vec<(&str, String)> = if annotation.docker { + ["DOCKER_HOST", "DOCKER_TLS_VERIFY", "DOCKER_CERT_PATH"] + .iter() + .filter_map(|k| std::env::var(k).ok().map(|v| (*k, v))) + .collect() + } else { + vec![] + }; + // Check if this is a regular job (not init or periodic script) // Init/periodic scripts need full system access without isolation let is_regular_job = job @@ -225,6 +236,7 @@ exit $exit_status ) .env("PATH", PATH_ENV.as_str()) .env("BASE_INTERNAL_URL", base_internal_url) + .envs(docker_envs.iter().cloned()) .args(cmd_args) .stdout(Stdio::piped()) .stderr(Stdio::piped()); @@ -255,6 +267,7 @@ exit $exit_status .env("PATH", PATH_ENV.as_str()) .env("BASE_INTERNAL_URL", base_internal_url) .env("HOME", HOME_ENV.as_str()) + .envs(docker_envs.iter().cloned()) .stdin(Stdio::null()) .stdout(Stdio::piped()) .stderr(Stdio::piped()); @@ -337,6 +350,19 @@ async fn rm_container(client: &bollard::Docker, container_id: &str) { } } +#[cfg(feature = "dind")] +/// Connect to the Docker daemon, respecting DOCKER_HOST if set (e.g. for dind sidecar), +/// otherwise falling back to the default unix socket at /var/run/docker.sock. +fn connect_docker() -> Result { + if std::env::var("DOCKER_HOST").is_ok() { + // DOCKER_HOST is set — use it (e.g. tcp://dind:2375 for docker-in-docker) + bollard::Docker::connect_with_defaults() + } else { + // No DOCKER_HOST — use the unix socket (backward compatible default) + bollard::Docker::connect_with_unix_defaults() + } +} + #[cfg(feature = "dind")] async fn handle_docker_job( job_id: Uuid, @@ -351,7 +377,7 @@ async fn handle_docker_job( ) -> Result, Error> { use crate::job_logger::append_logs_with_compaction; - let client = bollard::Docker::connect_with_unix_defaults().map_err(to_anyhow)?; + let client = connect_docker().map_err(to_anyhow)?; let container_id = job_id.to_string(); let inspected = client.inspect_container(&container_id, None).await; @@ -396,7 +422,7 @@ async fn handle_docker_job( let workspace_id2 = workspace_id.to_string(); let mut killpill_rx = killpill_rx.resubscribe(); let logs = tokio::spawn(async move { - let client = bollard::Docker::connect_with_unix_defaults().map_err(to_anyhow); + let client = connect_docker().map_err(to_anyhow); if let Ok(client) = client { let mut log_stream = client.logs( &ncontainer_id, @@ -464,7 +490,7 @@ async fn handle_docker_job( } }); - let mem_client = bollard::Docker::connect_with_unix_defaults().map_err(to_anyhow); + let mem_client = connect_docker().map_err(to_anyhow); let ncontainer_id = container_id.clone(); let result = run_future_with_polling_update_job_poller( job_id, diff --git a/backend/windmill-worker/src/bun_executor.rs b/backend/windmill-worker/src/bun_executor.rs index fb07806fd6..433f96bcec 100644 --- a/backend/windmill-worker/src/bun_executor.rs +++ b/backend/windmill-worker/src/bun_executor.rs @@ -77,95 +77,236 @@ pub const EMPTY_FILE: &str = ""; /// Bun args for dedicated worker (without the script path) pub const BUN_DEDICATED_WORKER_ARGS: &[&str] = &["run", "-i", "--prefer-offline"]; -/// Generate the dedicated worker wrapper content. -/// - `arg_names`: The argument names for the main function (e.g., ["x", "y"]) -/// - `main_import`: The import path for the main module (e.g., "./main.ts") -/// - `date_conversions`: Optional date conversion statements for Datetime args -/// - `preprocessor_spread`: If the script has a preprocessor function, the comma-separated arg names for it -pub fn generate_dedicated_worker_wrapper( - arg_names: &[&str], - main_import: &str, - date_conversions: Option<&str>, - preprocessor_spread: Option<&str>, -) -> String { - let spread = arg_names.join(","); - let dates = date_conversions.unwrap_or(""); +/// Pre-computed codegen data for a TypeScript/Bun/Deno script. +/// Computed in Rust from the parsed signature, then baked into the wrapper template. +#[cfg(any(feature = "private", test))] +pub struct TsScriptCodegen { + pub spread: String, + pub date_conversions: String, + pub preprocessor_spread: Option, + pub preprocessor_date_conversions: Option, +} + +/// Parse a TS script and compute the codegen data (arg spread, date conversions, preprocessor). +/// This is the same logic that was used on main in `start_worker`. +#[cfg(any(feature = "private", test))] +pub fn compute_ts_codegen(content: &str) -> TsScriptCodegen { + let sig = + windmill_parser_ts::parse_deno_signature(content, true, false, None).unwrap_or_default(); + let arg_names: Vec<&str> = sig.args.iter().map(|a| a.name.as_str()).collect(); + let spread = arg_names.join(", "); + + let dates = sig + .args + .iter() + .filter(|a| matches!(a.typ, Typ::Datetime)) + .map(|a| { + format!( + "{name} = {name} ? new Date({name}) : undefined", + name = a.name + ) + }) + .join("\n "); + + let pre_sig = windmill_parser_ts::parse_deno_signature( + content, + true, + false, + Some("preprocessor".to_string()), + ) + .ok() + .filter(|s| !s.args.is_empty()); + + let preprocessor_spread = pre_sig + .as_ref() + .map(|s| s.args.iter().map(|a| a.name.as_str()).join(", ")); + let preprocessor_date_conversions = pre_sig.as_ref().map(|s| { + s.args + .iter() + .filter(|a| matches!(a.typ, Typ::Datetime)) + .map(|a| { + format!( + "{name} = {name} ? new Date({name}) : undefined", + name = a.name + ) + }) + .join("\n ") + }); + + TsScriptCodegen { + spread, + date_conversions: dates, + preprocessor_spread, + preprocessor_date_conversions, + } +} + +/// Script entry for the unified wrapper generator. +/// `import_name`: the file stem used in the import path (e.g., "main" → `./main.ts`, or "f__script" → `./f__script.ts`) +#[cfg(any(feature = "private", test))] +pub struct TsScriptEntry<'a> { + pub import_name: &'a str, + pub original_path: &'a str, + pub codegen: &'a TsScriptCodegen, +} + +/// Generate a wrapper for dedicated workers and runner groups. +/// All scripts are baked in at codegen time with static imports and inline arg handling. +/// Protocol: +/// exec:: -> wm_res[success]: | wm_res[error]: +/// exec_preprocess:: -> wm_res[preprocessed_args]: then wm_res[success]: | wm_res[error]: +/// end -> exit +#[cfg(any(feature = "private", test))] +pub fn generate_multi_script_wrapper(scripts: &[TsScriptEntry<'_>], ext: &str) -> String { let is_debug = std::env::var("RUST_LOG").is_ok_and(|x| x == "windmill=debug"); let print_lines = if is_debug { - r#"console.log(line);"# + r#"console.log("[debug] " + line);"# } else { "" }; - let preprocessor_logic = if let Some(pre_spread) = preprocessor_spread { - format!( + let imports: String = scripts + .iter() + .enumerate() + .map(|(i, e)| { + format!( + "import * as _s{i} from \"./{import_name}.{ext}\";", + import_name = e.import_name + ) + }) + .collect::>() + .join("\n"); + + // Generate per-script getArgs / getPreArgs functions + let mut functions = String::new(); + let mut registrations = String::new(); + + for (i, entry) in scripts.iter().enumerate() { + let cg = entry.codegen; + let spread = &cg.spread; + let dates = &cg.date_conversions; + + functions.push_str(&format!( r#" - if (rawLine.startsWith("preprocess:")) {{ - const preInput = rawLine.slice("preprocess:".length); - const parsedArgs = JSON.parse(preInput); - if (Main.preprocessor === undefined || typeof Main.preprocessor !== 'function') {{ - console.log("wm_res[error]:" + JSON.stringify({{ message: "preprocessor function is missing", name: "Error" }})); - continue; - }} - try {{ - function preArgsObjToArr({{ {pre_spread} }}) {{ - return [ {pre_spread} ]; - }} - const preprocessedArgs = await Main.preprocessor(...preArgsObjToArr(parsedArgs)); - console.log("wm_res[preprocessed_args]:" + JSON.stringify(preprocessedArgs ?? {{}}, (key, value) => typeof value === 'undefined' ? null : value)); - // Now call main with preprocessed args - const mainArgs = getArgs(JSON.stringify(preprocessedArgs ?? {{}})); - const res = await Main.main(...mainArgs); - console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value)); - }} catch (e) {{ - console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: rawLine }})); - }} - continue; - }}"# - ) - } else { - String::new() - }; +function getArgs_{i}(line) {{ + let {{ {spread} }} = JSON.parse(line); + {dates} + return [ {spread} ]; +}} +"# + )); + + let pre_fn = if let Some(ref pre_spread) = cg.preprocessor_spread { + let pre_dates = cg.preprocessor_date_conversions.as_deref().unwrap_or(""); + functions.push_str(&format!( + r#" +function getPreArgs_{i}(line) {{ + let {{ {pre_spread} }} = JSON.parse(line); + {pre_dates} + return [ {pre_spread} ]; +}} +"# + )); + format!("getPreArgs_{i}") + } else { + "null".to_string() + }; + + registrations.push_str(&format!( + "scripts.set(\"{path}\", {{ module: _s{i}, getArgs: getArgs_{i}, getPreArgs: {pre_fn} }});\n", + path = entry.original_path, + )); + } format!( r#" -import * as Main from "{main_import}"; +{imports} import * as Readline from "node:readline" BigInt.prototype.toJSON = function () {{ return this.toString(); }}; -console.log('start'); +const scripts = new Map(); +{functions} +{registrations} -function getArgs(line) {{ - let {{ {spread} }} = JSON.parse(line) - {dates} - return [ {spread} ]; -}} +console.log('start'); for await (const line of Readline.createInterface({{ input: process.stdin }})) {{ {print_lines} - const rawLine = line; - if (rawLine === "end") {{ + if (line === "end") {{ process.exit(0); }} - {preprocessor_logic} - try {{ - const args = getArgs(rawLine); - const res = await Main.main(...args); - console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value)); - }} catch (e) {{ - console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: rawLine }})); + + if (line.startsWith("exec_preprocess:")) {{ + const rest = line.slice("exec_preprocess:".length); + const colonIdx = rest.indexOf(":"); + if (colonIdx === -1) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "Malformed exec_preprocess command: missing colon separator", name: "Error" }})); + continue; + }} + const scriptPath = rest.slice(0, colonIdx); + const argsJson = rest.slice(colonIdx + 1); + + const entry = scripts.get(scriptPath); + if (!entry) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "Script not found: " + scriptPath, name: "Error" }})); + continue; + }} + + try {{ + if (!entry.getPreArgs) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "preprocessor function is missing", name: "Error" }})); + continue; + }} + const preArgs = entry.getPreArgs(argsJson); + const preprocessedArgs = await entry.module.preprocessor(...preArgs); + console.log("wm_res[preprocessed_args]:" + JSON.stringify(preprocessedArgs ?? {{}}, (key, value) => typeof value === 'undefined' ? null : value)); + const mainArgs = entry.getArgs(JSON.stringify(preprocessedArgs ?? {{}})); + const res = await entry.module.main(...mainArgs); + console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value)); + }} catch (e) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: argsJson }})); + }} + continue; }} + + if (line.startsWith("exec:")) {{ + const rest = line.slice("exec:".length); + const colonIdx = rest.indexOf(":"); + if (colonIdx === -1) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "Malformed exec command: missing colon separator", name: "Error" }})); + continue; + }} + const scriptPath = rest.slice(0, colonIdx); + const argsJson = rest.slice(colonIdx + 1); + + const entry = scripts.get(scriptPath); + if (!entry) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "Script not found: " + scriptPath, name: "Error" }})); + continue; + }} + + try {{ + const args = entry.getArgs(argsJson); + const res = await entry.module.main(...args); + console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value)); + }} catch (e) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: argsJson }})); + }} + continue; + }} + + console.error("Unknown command:", line); }} "# ) } /// Returns (package.json, bun.lock(b), is_empty, is_binary) -fn split_lockfile(lockfile: &str) -> (&str, Option<&str>, bool, bool) { +pub(crate) fn split_lockfile(lockfile: &str) -> (&str, Option<&str>, bool, bool) { if let Some(index) = lockfile.find(BUN_LOCK_SPLIT) { // Split using "\n//bun.lock\n" let (before, after_with_sep) = lockfile.split_at(index); @@ -951,7 +1092,12 @@ pub async fn prebundle_bun_script( } let origin = format!("{job_dir}/main.js"); - write_file(job_dir, "main.ts", &remove_pinned_imports(inner_content)?)?; + let mut content = remove_pinned_imports(inner_content)?; + if crate::wac_executor::is_wac_v2_ts(inner_content) { + content = crate::wac_executor::inject_wac_task_names(&content); + content = format!("export {{ WorkflowCtx, StepSuspend, setWorkflowCtx }} from \"windmill-client\";\n{content}"); + } + write_file(job_dir, "main.ts", &content)?; build_loader( job_dir, base_internal_url, @@ -1177,29 +1323,12 @@ pub async fn handle_bun_job( // Also handles: export const, let, var, and optional generic type parameters. // Skips calls that already have a string argument: `task("path", async ...` let inner_content = if is_wac_v2 { - use regex::Regex; - use std::borrow::Cow; - lazy_static::lazy_static! { - static ref TASK_RE: Regex = - Regex::new(r#"(?m)((?:export\s+)?(?:const|let|var)\s+)(\w+)(\s*=\s*task\s*(?:<[^>]*>)?\s*\(\s*)(async\b)"#).unwrap(); - } - let replaced = TASK_RE.replace_all(inner_content, r#"${1}${2}${3}"${2}", ${4}"#); - match replaced { - Cow::Borrowed(_) => inner_content.to_string(), - Cow::Owned(s) => s, - } + crate::wac_executor::inject_wac_task_names(inner_content) } else { inner_content.to_string() }; let inner_content = inner_content.as_str(); - // WAC v2 scripts can't use bundle caching because the wrapper imports - // windmill-client from node_modules, which isn't available in bundle mode - if is_wac_v2 && has_bundle_cache { - has_bundle_cache = false; - let _ = write_file(job_dir, "main.ts", inner_content)?; - } - let mut format = BundleFormat::Cjs; if has_bundle_cache { let target; @@ -1420,6 +1549,12 @@ pub async fn handle_bun_job( "./main.ts" }; + let wac_client_import = if has_bundle_cache { + "./main.js" + } else { + "windmill-client" + }; + let preprocessor = if let Some(pre_args) = pre_args { let pre_spread = pre_args.into_iter().map(|x| x.name).join(","); format!( @@ -1447,7 +1582,7 @@ pub async fn handle_bun_job( format!( r#" import * as Main from "{main_import}"; -import {{ WorkflowCtx, StepSuspend, setWorkflowCtx }} from "windmill-client"; +import {{ WorkflowCtx, StepSuspend, setWorkflowCtx }} from "{wac_client_import}"; import * as fs from "fs/promises"; @@ -1504,7 +1639,7 @@ async function run() {{ return {{ type: "inline_checkpoint", key: dispatch.key, result: dispatch.result ?? null, started_at: dispatch.started_at, duration_ms: dispatch.duration_ms }}; }} if (dispatch.mode === "approval") {{ - return {{ type: "approval", key: dispatch.key, timeout: dispatch.timeout, form: dispatch.form }}; + return {{ type: "approval", key: dispatch.key, timeout: dispatch.timeout, form: dispatch.form, self_approval_disabled: dispatch.self_approval_disabled }}; }} if (dispatch.mode === "sleep") {{ return {{ type: "sleep", key: dispatch.key, seconds: dispatch.seconds }}; @@ -1638,7 +1773,6 @@ try {{ && !annotation.nobundling && !*DISABLE_BUNDLING && !codebase.is_some() - && !is_wac_v2 && (maybe_lock.get_lock().is_some() || annotation.native); let write_loader_f = async { @@ -1703,6 +1837,17 @@ try {{ } } + // Prepend WAC re-exports to main.ts so the bundle includes WorkflowCtx etc. + if build_cache && is_wac_v2 { + let main_path = format!("{job_dir}/main.ts"); + let current = read_file_content(&main_path).await?; + write_file( + job_dir, + "main.ts", + &format!("export {{ WorkflowCtx, StepSuspend, setWorkflowCtx }} from \"windmill-client\";\n{current}"), + )?; + } + if !codebase.is_some() && !has_bundle_cache { if build_cache { generate_bun_bundle( @@ -1741,14 +1886,14 @@ try {{ } if !annotation.native { let ex_wrapper = read_file_content(&format!("{job_dir}/wrapper.mjs")).await?; - write_file( - job_dir, - "wrapper.mjs", - &ex_wrapper.replace( - "import * as Main from \"./main.ts\"", - "import * as Main from \"./main.js\"", - ), - )?; + let mut rewritten = ex_wrapper.replace( + "import * as Main from \"./main.ts\"", + "import * as Main from \"./main.js\"", + ); + if is_wac_v2 { + rewritten = rewritten.replace("from \"windmill-client\"", "from \"./main.js\""); + } + write_file(job_dir, "wrapper.mjs", &rewritten)?; write_file(job_dir, "package.json", r#"{ "type": "module" }"#)?; } fs::remove_file(format!("{job_dir}/main.ts"))?; @@ -2634,7 +2779,7 @@ pub async fn handle_wac_v2_output( job.id, num_steps ))) } - WacOutput::Approval { key, timeout, form } => { + WacOutput::Approval { key, timeout, form, self_approval_disabled } => { let db = match conn { Connection::Sql(db) => db, _ => { @@ -2676,11 +2821,91 @@ pub async fn handle_wac_v2_output( .await .map_err(|e| error::Error::internal_err(format!("Failed to save checkpoint: {e}")))?; + // Store approval_conditions in flow_status for resume endpoint auth checks + let sad = self_approval_disabled.unwrap_or(false); + if sad { + #[cfg(not(feature = "enterprise"))] + return Err(error::Error::ExecutionErr( + "Disabling self-approval is an enterprise only feature".to_string(), + )); + + #[cfg(feature = "enterprise")] + { + use windmill_common::flow_status::ApprovalConditions; + let approval_conditions = ApprovalConditions { + user_auth_required: true, + user_groups_required: vec![], + self_approval_disabled: true, + }; + sqlx::query( + "UPDATE v2_job_status SET flow_status = JSONB_SET( + COALESCE(flow_status, '{}'::jsonb), + '{approval_conditions}', + $2::jsonb + ) WHERE id = $1", + ) + .bind(&job.id) + .bind(&serde_json::json!(approval_conditions)) + .execute(&mut *tx) + .await + .map_err(|e| { + error::Error::internal_err(format!( + "Failed to save approval conditions: {e}" + )) + })?; + } + } + + // Generate resume URLs for the inline approval buttons. + // Use a hash of the step key as resume_id so each waitForApproval() + // in the same workflow gets a unique resume_job record. + let resume_id: u32 = { + use std::hash::{Hash, Hasher}; + let mut hasher = std::collections::hash_map::DefaultHasher::new(); + key.hash(&mut hasher); + (hasher.finish() & 0xFFFF_FFFF) as u32 + }; + // Generate stateless approval token using shared utility + let approval_token = + windmill_common::variables::generate_approval_token(&job.workspace_id, job.id, db) + .await?; + + let (resume_url, cancel_url, approval_page_url) = { + use hmac::{Hmac, Mac}; + use sha2::Sha256; + use windmill_common::variables::get_workspace_key; + + let wkey = get_workspace_key(&job.workspace_id, db).await?; + let mut mac = Hmac::::new_from_slice(wkey.as_bytes()) + .map_err(|e| error::Error::internal_err(format!("HMAC key error: {e}")))?; + mac.update(job.id.as_bytes()); + mac.update(resume_id.to_be_bytes().as_ref()); + let signature = hex::encode(mac.finalize().into_bytes()); + + let base_url = windmill_common::BASE_URL.read().await.clone(); + let w_id = &job.workspace_id; + let job_id = &job.id; + + let resume = format!( + "{base_url}/api/w/{w_id}/jobs_u/resume/{job_id}/{resume_id}/{signature}" + ); + let cancel = format!( + "{base_url}/api/w/{w_id}/jobs_u/cancel/{job_id}/{resume_id}/{signature}" + ); + let approval_page = + format!("{base_url}/approve/{w_id}/{job_id}?token={approval_token}"); + (resume, cancel, approval_page) + }; + // Store approval form metadata for the approval page endpoint let approval_meta = serde_json::json!({ "key": key, "form": form, "timeout": timeout_secs as u32, + "self_approval_disabled": sad, + "resume": resume_url, + "cancel": cancel_url, + "approvalPage": approval_page_url, }); sqlx::query( "UPDATE v2_job_status SET workflow_as_code_status = jsonb_set( @@ -2705,6 +2930,11 @@ pub async fn handle_wac_v2_output( "started_at": &now_str, "name": key, "approval": true, + "self_approval_disabled": sad, + "form": form, + "resume": &resume_url, + "cancel": &cancel_url, + "approvalPage": &approval_page_url, }); let step_timeline_key = format!("_step/{}", key); sqlx::query( @@ -3008,7 +3238,10 @@ pub fn build_nativets_env_code( "const process = {{ env: {{}} }};\nconst BASE_URL = '{base_internal_url}';\nconst BASE_INTERNAL_URL = '{base_internal_url}';\nprocess.env['BASE_URL'] = BASE_URL;process.env['BASE_INTERNAL_URL'] = BASE_INTERNAL_URL;\n{}", reserved_variables .iter() - .map(|(k, v)| format!("process.env['{}'] = '{}';", k, v)) + .map(|(k, v)| { + let escaped = v.replace('\\', "\\\\").replace('\'', "\\'").replace('\n', "\\n").replace('\r', "\\r"); + format!("process.env['{}'] = '{}';", k, escaped) + }) .collect::>() .join("\n") ) @@ -3499,53 +3732,18 @@ pub async fn start_worker( let main_code = remove_pinned_imports(inner_content)?; let _ = write_file(job_dir, "main.ts", &main_code)?; + let codegen = compute_ts_codegen(inner_content); + let wrapper_ext = if codebase.is_some() { "js" } else { "ts" }; { - // let mut start = Instant::now(); - let args = windmill_parser_ts::parse_deno_signature(inner_content, true, false, None)?.args; - let dates = args - .iter() - .filter_map(|x| { - if matches!(x.typ, Typ::Datetime) { - Some(x.name.clone()) - } else { - None - } - }) - .map(|x| return format!("{x} = {x} ? new Date({x}) : undefined")) - .join("\n"); - - let arg_names: Vec<&str> = args.iter().map(|x| x.name.as_str()).collect(); - - // Parse preprocessor signature if it exists - let pre_spread = windmill_parser_ts::parse_deno_signature( - inner_content, - true, - false, - Some("preprocessor".to_string()), - ) - .ok() - .filter(|sig| !sig.args.is_empty()) - .map(|sig| sig.args.into_iter().map(|x| x.name).join(",")); - - // logs.push_str(format!("infer args: {:?}\n", start.elapsed().as_micros()).as_str()); - // we cannot use Bun.read and Bun.write because it results in an EBADF error on cloud - - let main_import = if codebase.is_some() { - "./main.js" - } else { - "./main.ts" - }; - let dates_opt = if dates.is_empty() { - None - } else { - Some(dates.as_str()) - }; - let wrapper_content = generate_dedicated_worker_wrapper( - &arg_names, - main_import, - dates_opt, - pre_spread.as_deref(), - ); + let scripts = + [ + TsScriptEntry { + import_name: "main", + original_path: script_path, + codegen: &codegen, + }, + ]; + let wrapper_content = generate_multi_script_wrapper(&scripts, wrapper_ext); write_file(job_dir, "wrapper.mjs", &wrapper_content)?; } @@ -3744,4 +3942,48 @@ lockfile-content"#; assert!(!is_empty); assert!(!is_binary); } + + #[test] + fn test_compute_ts_codegen_basic_args() { + let code = r#"export function main(x: string, y: number) { return x; }"#; + let cg = compute_ts_codegen(code); + assert_eq!(cg.spread, "x, y"); + assert!(cg.date_conversions.is_empty()); + assert!(cg.preprocessor_spread.is_none()); + } + + #[test] + fn test_compute_ts_codegen_with_datetime() { + let code = r#"export function main(name: string, created_at: Date, count: number) { return name; }"#; + let cg = compute_ts_codegen(code); + assert_eq!(cg.spread, "name, created_at, count"); + assert!(cg.date_conversions.contains("created_at")); + assert!(cg.date_conversions.contains("new Date")); + } + + #[test] + fn test_compute_ts_codegen_with_preprocessor() { + let code = r#" +export function main(x: string, ts: Date) { return x; } +export function preprocessor(input: string, when: Date) { return { x: input, ts: when }; } +"#; + let cg = compute_ts_codegen(code); + assert_eq!(cg.spread, "x, ts"); + assert!(cg.date_conversions.contains("ts")); + assert_eq!(cg.preprocessor_spread.as_deref(), Some("input, when")); + assert!(cg + .preprocessor_date_conversions + .as_ref() + .unwrap() + .contains("when")); + } + + #[test] + fn test_compute_ts_codegen_no_args() { + let code = r#"export function main() { return 42; }"#; + let cg = compute_ts_codegen(code); + assert!(cg.spread.is_empty()); + assert!(cg.date_conversions.is_empty()); + assert!(cg.preprocessor_spread.is_none()); + } } diff --git a/backend/windmill-worker/src/common.rs b/backend/windmill-worker/src/common.rs index 0c81c45286..d303771cc8 100644 --- a/backend/windmill-worker/src/common.rs +++ b/backend/windmill-worker/src/common.rs @@ -145,7 +145,7 @@ pub async fn write_file_binary(dir: &str, path: &str, content: &[u8]) -> error:: } lazy_static::lazy_static! { - static ref RE_RES_VAR: Regex = Regex::new(r#"\$(?:var|res|encrypted)\:"#).unwrap(); + static ref RE_RES_VAR: Regex = Regex::new(r#"\$(?:var|jsonvar|res|encrypted)\:"#).unwrap(); } pub async fn transform_json<'a>( @@ -255,6 +255,15 @@ pub async fn transform_json_value( Error::NotFound(format!("Variable {path} not found for `{name}`: {e:#}")) }) } + Value::String(y) if y.starts_with("$jsonvar:") => { + let path = y.strip_prefix("$jsonvar:").unwrap(); + let v = client.get_variable_value(path).await.map_err(|e| { + Error::NotFound(format!("Variable {path} not found for `{name}`: {e:#}")) + })?; + serde_json::from_str::(&v).map_err(|e| { + Error::internal_err(format!("Failed to parse $jsonvar value as JSON: {e}")) + }) + } Value::String(y) if y.starts_with("$res:") => { let path = y.strip_prefix("$res:").unwrap(); @@ -286,6 +295,18 @@ pub async fn transform_json_value( ) .await?; decrypt(&mc, encrypted.to_string()).and_then(|x| { + // Register the raw decrypted string for log masking. + // This covers both string values and their JSON representations + // (numbers, objects, etc.) that could appear in logs. + windmill_common::sensitive_log_masks::register_secret_for_job(job.id, &x); + if let serde_json::Value::String(ref s) = + serde_json::from_str::(&x).unwrap_or_default() + { + // Also register the inner string value (without JSON quotes) + windmill_common::sensitive_log_masks::register_secret_for_job( + job.id, s, + ); + } serde_json::from_str(&x).map_err(|e| { Error::internal_err(format!( "Failed to decrypt '$encrypted:' value: {e}" diff --git a/backend/windmill-worker/src/deno_executor.rs b/backend/windmill-worker/src/deno_executor.rs index 46f42af3bf..3013d10689 100644 --- a/backend/windmill-worker/src/deno_executor.rs +++ b/backend/windmill-worker/src/deno_executor.rs @@ -27,6 +27,16 @@ use windmill_common::{ }; use windmill_parser::Typ; +pub const DENO_UNSTABLE_ARGS: &[&str] = &[ + "--unstable-unsafe-proto", + "--unstable-bare-node-builtins", + "--unstable-webgpu", + "--unstable-ffi", + "--unstable-fs", + "--unstable-worker-options", + "--unstable-http", +]; + lazy_static::lazy_static! { static ref DENO_FLAGS: Option> = std::env::var("DENO_FLAGS") @@ -172,22 +182,14 @@ pub async fn generate_deno_lock( let mut child_cmd = Command::new(DENO_PATH.as_str()); child_cmd .current_dir(job_dir) - .args(vec![ - "cache", - "--unstable-unsafe-proto", - "--unstable-bare-node-builtins", - "--unstable-webgpu", - "--unstable-ffi", - "--unstable-fs", - "--unstable-worker-options", - "--unstable-http", + .args(["cache"].iter().chain(DENO_UNSTABLE_ARGS).chain(&[ "--lock=lock.json", "--frozen=false", "--allow-import", "--import-map", - &import_map_path, + import_map_path.as_str(), "main.ts", - ]) + ])) .envs(deno_envs) .stdout(Stdio::piped()) .stderr(Stdio::piped()); @@ -442,13 +444,7 @@ try {{ args.push("--import-map"); args.push(&import_map_path); args.push(&reload); - args.push("--unstable-unsafe-proto"); - args.push("--unstable-bare-node-builtins"); - args.push("--unstable-webgpu"); - args.push("--unstable-ffi"); - args.push("--unstable-fs"); - args.push("--unstable-worker-options"); - args.push("--unstable-http"); + args.extend_from_slice(DENO_UNSTABLE_ARGS); if !*DISABLE_DENO_LOCK { if let Some(reqs) = requirements_o { @@ -547,7 +543,7 @@ try {{ read_result(job_dir, handle_result.result_stream).await } -async fn build_import_map( +pub(crate) async fn build_import_map( w_id: &str, script_path: &str, base_internal_url: &str, @@ -592,6 +588,132 @@ async fn build_import_map( #[cfg(feature = "private")] use crate::{dedicated_worker_oss::handle_dedicated_process, JobCompletedSender}; +/// Generate the dedicated worker wrapper for Deno. +/// Parses the script signature and bakes in arg destructuring, date conversions, +/// and preprocessor logic. Uses the `exec::` protocol. +#[cfg(any(feature = "private", test))] +pub fn generate_dedicated_worker_wrapper(inner_content: &str) -> Result { + let args = windmill_parser_ts::parse_deno_signature(inner_content, true, false, None)?.args; + let dates = args + .iter() + .filter_map(|x| { + if matches!(x.typ, Typ::Datetime) { + Some(x.name.clone()) + } else { + None + } + }) + .map(|x| format!("{x} = {x} ? new Date({x}) : undefined")) + .join("\n"); + + let spread = args.into_iter().map(|x| x.name).join(","); + + let pre_spread = windmill_parser_ts::parse_deno_signature( + inner_content, + true, + false, + Some("preprocessor".to_string()), + ) + .ok() + .filter(|sig| !sig.args.is_empty()) + .map(|sig| sig.args.into_iter().map(|x| x.name).join(",")); + + let preprocessor_import = if pre_spread.is_some() { + r#"import { preprocessor } from "./main.ts";"# + } else { + "" + }; + + let preprocessor_logic = if let Some(ref pre_spread) = pre_spread { + format!( + r#" + if (line.startsWith("exec_preprocess:")) {{ + const rest = line.slice("exec_preprocess:".length); + const colonIdx = rest.indexOf(":"); + if (colonIdx === -1) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "Malformed exec_preprocess command: missing colon separator", name: "Error" }}) + '\n'); + continue; + }} + const argsJson = rest.slice(colonIdx + 1); + const parsedArgs = JSON.parse(argsJson); + if (typeof preprocessor !== 'function') {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "preprocessor function is missing", name: "Error" }}) + '\n'); + continue; + }} + try {{ + function preArgsObjToArr({{ {pre_spread} }}: any) {{ + return [ {pre_spread} ]; + }} + const preprocessedArgs: any = await preprocessor(...preArgsObjToArr(parsedArgs)); + console.log("wm_res[preprocessed_args]:" + JSON.stringify(preprocessedArgs ?? {{}}, (key, value) => typeof value === 'undefined' ? null : value) + '\n'); + let {{ {spread} }} = preprocessedArgs ?? {{}}; + {dates} + let res: any = await main(...[ {spread} ]); + console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value) + '\n'); + }} catch (e) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: line }}) + '\n'); + }} + continue; + }}"# + ) + } else { + String::new() + }; + + Ok(format!( + r#" +import {{ main }} from "./main.ts"; +{preprocessor_import} + +BigInt.prototype.toJSON = function () {{ + return this.toString(); +}}; + +console.log('start\n'); + +const decoder = new TextDecoder(); +let _buffer = ""; +for await (const chunk of Deno.stdin.readable) {{ + _buffer += decoder.decode(chunk, {{ stream: true }}); + const _parts = _buffer.split("\n"); + _buffer = _parts.pop() ?? ""; + let exit = false; + for (const _part of _parts) {{ + const line = _part.trim(); + if (!line) continue; + if (line === "end") {{ + exit = true; + break; + }} + {preprocessor_logic} + if (line.startsWith("exec:")) {{ + const rest = line.slice("exec:".length); + const colonIdx = rest.indexOf(":"); + if (colonIdx === -1) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: "Malformed exec command: missing colon separator", name: "Error" }}) + '\n'); + continue; + }} + const argsJson = rest.slice(colonIdx + 1); + try {{ + let {{ {spread} }} = JSON.parse(argsJson) + {dates} + let res: any = await main(...[ {spread} ]); + console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value) + '\n'); + }} catch (e) {{ + console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: argsJson }}) + '\n'); + }} + continue; + }} + console.error("Unknown command:", line); + }} + if (exit) {{ + break; + }} +}} +"#, + )) +} + #[cfg(feature = "private")] use tokio::sync::mpsc::Receiver; #[cfg(feature = "private")] @@ -650,115 +772,19 @@ pub async fn start_worker( let context_envs = build_envs_map(context.to_vec()).await; { - // let mut start = Instant::now(); - let args = windmill_parser_ts::parse_deno_signature(inner_content, true, false, None)?.args; - let dates = args - .iter() - .filter_map(|x| { - if matches!(x.typ, Typ::Datetime) { - Some(x.name.clone()) - } else { - None - } - }) - .map(|x| return format!("{x} = {x} ? new Date({x}) : undefined")) - .join("\n"); - - let spread = args.into_iter().map(|x| x.name).join(","); - - // Parse preprocessor signature if it exists - let pre_spread = windmill_parser_ts::parse_deno_signature( - inner_content, - true, - false, - Some("preprocessor".to_string()), - ) - .ok() - .filter(|sig| !sig.args.is_empty()) - .map(|sig| sig.args.into_iter().map(|x| x.name).join(",")); - - let preprocessor_import = if pre_spread.is_some() { - r#"import { preprocessor } from "./main.ts";"# - } else { - "" - }; - - let preprocessor_logic = if let Some(ref pre_spread) = pre_spread { - format!( - r#" - if (line.startsWith("preprocess:")) {{ - const preInput = line.slice("preprocess:".length); - const parsedArgs = JSON.parse(preInput); - if (typeof preprocessor !== 'function') {{ - console.log("wm_res[error]:" + JSON.stringify({{ message: "preprocessor function is missing", name: "Error" }}) + '\n'); - continue; - }} - try {{ - function preArgsObjToArr({{ {pre_spread} }}: any) {{ - return [ {pre_spread} ]; - }} - const preprocessedArgs: any = await preprocessor(...preArgsObjToArr(parsedArgs)); - console.log("wm_res[preprocessed_args]:" + JSON.stringify(preprocessedArgs ?? {{}}, (key, value) => typeof value === 'undefined' ? null : value) + '\n'); - // Now call main with preprocessed args - let {{ {spread} }} = preprocessedArgs ?? {{}}; - {dates} - let res: any = await main(...[ {spread} ]); - console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value) + '\n'); - }} catch (e) {{ - console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: line }}) + '\n'); - }} - continue; - }}"# - ) - } else { - String::new() - }; - - // logs.push_str(format!("infer args: {:?}\n", start.elapsed().as_micros()).as_str()); - // we cannot use Bun.read and Bun.write because it results in an EBADF error on cloud - let wrapper_content: String = format!( - r#" -import {{ main }} from "./main.ts"; -{preprocessor_import} - -BigInt.prototype.toJSON = function () {{ - return this.toString(); -}}; - -{dates} - -console.log('start\n'); - -const decoder = new TextDecoder(); -for await (const chunk of Deno.stdin.readable) {{ - const lines = decoder.decode(chunk); - let exit = false; - for (const line of lines.trim().split("\n")) {{ - if (line === "end") {{ - exit = true; - break; - }} - {preprocessor_logic} - try {{ - let {{ {spread} }} = JSON.parse(line) - {dates} - let res: any = await main(...[ {spread} ]); - console.log("wm_res[success]:" + JSON.stringify(res ?? null, (key, value) => typeof value === 'undefined' ? null : value) + '\n'); - }} catch (e) {{ - console.log("wm_res[error]:" + JSON.stringify({{ message: e.message, name: e.name, stack: e.stack, line: line }}) + '\n'); - }} - }} - if (exit) {{ - break; - }} -}} -"#, - ); + let wrapper_content = generate_dedicated_worker_wrapper(inner_content)?; write_file(job_dir, "wrapper.ts", &wrapper_content)?; } build_import_map(w_id, script_path, base_internal_url, job_dir).await?; + let import_map = format!("{job_dir}/import_map.json"); + let reload = format!("--reload={base_internal_url}"); + let wrapper = format!("{job_dir}/wrapper.ts"); + let mut deno_args = vec!["run", "--no-check", "--import-map", &import_map, &reload]; + deno_args.extend_from_slice(DENO_UNSTABLE_ARGS); + deno_args.extend_from_slice(&["-A", &wrapper]); + handle_dedicated_process( &*DENO_PATH, job_dir, @@ -766,22 +792,7 @@ for await (const chunk of Deno.stdin.readable) {{ envs, context, common_deno_proc_envs, - vec![ - "run", - "--no-check", - "--import-map", - &format!("{job_dir}/import_map.json"), - &format!("--reload={base_internal_url}"), - "--unstable-unsafe-proto", - "--unstable-bare-node-builtins", - "--unstable-webgpu", - "--unstable-ffi", - "--unstable-fs", - "--unstable-worker-options", - "--unstable-http", - "-A", - &format!("{job_dir}/wrapper.ts"), - ], + deno_args, killpill_rx, job_completed_tx, token, diff --git a/backend/windmill-worker/src/handle_child.rs b/backend/windmill-worker/src/handle_child.rs index 46cc047bf0..2edaa4f3a1 100644 --- a/backend/windmill-worker/src/handle_child.rs +++ b/backend/windmill-worker/src/handle_child.rs @@ -432,12 +432,26 @@ pub async fn write_lines( let job_id = job_id.clone(); let mut nstream = String::new(); + // Snapshot secrets once per batch — no lock needed per line. + // Trade-off: secrets registered mid-batch (between snapshot and log line) + // won't be masked until the next batch. In practice the async HTTP round-trip + // to fetch a secret completes before the script's log line arrives. + let mask_snapshot = windmill_common::sensitive_log_masks::snapshot(&job_id); + while let Some(line) = read_lines.next().await { match line { Ok(line) => { if line.is_empty() { continue; } + let line = if let Some(ref snap) = mask_snapshot { + match snap.mask(&line) { + std::borrow::Cow::Owned(masked) => masked, + std::borrow::Cow::Borrowed(_) => line, + } + } else { + line + }; if *OTEL_JOB_LOGS { if let Some(otel_suffix) = line.strip_prefix(OTEL_PREFIX) { tracing::event!(tracing::Level::INFO, otel_suffix); diff --git a/backend/windmill-worker/src/lib.rs b/backend/windmill-worker/src/lib.rs index 2fe56f50b9..e97ec8ff9a 100644 --- a/backend/windmill-worker/src/lib.rs +++ b/backend/windmill-worker/src/lib.rs @@ -86,12 +86,23 @@ pub use otel_tracing_proxy_ee::{load_internal_otel_exporter, DENO_OTEL_INITIALIZ pub use worker::*; pub use bun_executor::{ - build_loader, compute_bundle_local_and_remote_path, generate_dedicated_worker_wrapper, - get_common_bun_proc_envs, install_bun_lockfile, prebundle_bun_script, prepare_job_dir, - LoaderMode, BUN_DEDICATED_WORKER_ARGS, RELATIVE_BUN_BUILDER, RELATIVE_BUN_LOADER, + build_loader, compute_bundle_local_and_remote_path, get_common_bun_proc_envs, + install_bun_lockfile, prebundle_bun_script, prepare_job_dir, LoaderMode, + BUN_DEDICATED_WORKER_ARGS, RELATIVE_BUN_BUILDER, RELATIVE_BUN_LOADER, }; -pub use deno_executor::generate_deno_lock; +#[cfg(any(feature = "private", test))] +pub use bun_executor::{ + compute_ts_codegen, generate_multi_script_wrapper, TsScriptCodegen, TsScriptEntry, +}; +#[cfg(any(feature = "private", test))] +pub use deno_executor::generate_dedicated_worker_wrapper as generate_deno_dedicated_worker_wrapper; +pub use deno_executor::{generate_deno_lock, DENO_UNSTABLE_ARGS}; pub use prepare_deps::run_prepare_deps_cli; +#[cfg(all(feature = "python", any(feature = "private", test)))] +pub use python_executor::{ + compute_py_codegen, generate_multi_script_wrapper as generate_py_multi_script_wrapper, + PyScriptCodegen, PyScriptEntry, +}; #[cfg(feature = "python")] pub use python_versions::PyV; diff --git a/backend/windmill-worker/src/pg_executor.rs b/backend/windmill-worker/src/pg_executor.rs index 5b309a611c..f769e1034d 100644 --- a/backend/windmill-worker/src/pg_executor.rs +++ b/backend/windmill-worker/src/pg_executor.rs @@ -285,7 +285,16 @@ pub async fn do_postgresql( annotations.result_collection }; - let database_string = database.to_uri(); + let use_iam_auth = database.use_iam_auth == Some(true); + + // Include use_iam_auth in cache key to distinguish IAM vs non-IAM connections to the same host. + // The cache key is static (doesn't include the token), which is correct because PostgreSQL + // connections remain valid after initial auth — fresh tokens are generated on cache miss. + let database_string = if use_iam_auth { + format!("{}?iam=true", database.to_uri()) + } else { + database.to_uri() + }; let database_string_clone = database_string.clone(); let mtex; @@ -309,7 +318,20 @@ pub async fn do_postgresql( ); (None, mtex) } else { - let (client, connection) = database.connect().await?; + let (client, connection) = if use_iam_auth { + #[cfg(all(feature = "enterprise", feature = "private"))] + { + database.connect_with_iam().await? + } + #[cfg(not(all(feature = "enterprise", feature = "private")))] + { + return Err(Error::ExecutionErr( + "IAM RDS authentication requires Windmill Enterprise Edition".to_string(), + )); + } + } else { + database.connect().await? + }; let handle = tokio::spawn(async move { if let Err(e) = connection.await { let mut mtex = CONNECTION_CACHE.lock().await; diff --git a/backend/windmill-worker/src/php_executor.rs b/backend/windmill-worker/src/php_executor.rs index 89d3bd5b62..d3d0fb36fb 100644 --- a/backend/windmill-worker/src/php_executor.rs +++ b/backend/windmill-worker/src/php_executor.rs @@ -84,8 +84,27 @@ pub async fn composer_install( ) -> Result { check_executor_binary_exists("php", PHP_PATH.as_str(), "php")?; - // When a lock file is available the dependency set is fully pinned, so we - // can cache the installed vendor/ directory and reuse it across executions. + // When no lock is provided (previews), try to reuse a previously resolved + // lockfile from the DB so we can hit the same vendor cache as deployed scripts. + let lock = if lock.is_none() && !*COMPOSER_VENDOR_CACHE_DISABLED { + let req_hash = format!("composer-{}", calculate_hash(&requirements)); + if let Some(db) = conn.as_sql() { + sqlx::query_scalar!( + "SELECT lockfile FROM pip_resolution_cache WHERE hash = $1", + req_hash + ) + .fetch_optional(db) + .await + .ok() + .flatten() + } else { + None + } + } else { + lock + }; + + // Cache the installed vendor/ directory keyed by requirements + lock content. // Set COMPOSER_VENDOR_CACHE_DISABLED=1 to opt out. let vendor_cache_hit = if !*COMPOSER_VENDOR_CACHE_DISABLED { if let Some(ref lock_content) = lock { @@ -171,6 +190,9 @@ pub async fn composer_install( ) .await?; + // lock was `None` means composer resolved deps from scratch (no lock from + // caller or DB). This is the only case where we should update the DB cache. + let freshly_resolved = lock.is_none(); let resolved_lock = match lock { Some(l) => l, None => { @@ -195,6 +217,27 @@ pub async fn composer_install( { tracing::warn!("Could not save composer vendor dir to cache: {e:?}"); } + + // Cache the resolved lockfile in the DB so future previews (which lack a + // lock file) can look it up by requirements hash and hit the same vendor + // cache. TTL of 7 days keeps previews reasonably fresh. + // Only write when composer resolved from scratch (no lock from caller or + // DB) to avoid endlessly refreshing the TTL on stale resolutions. + if freshly_resolved { + let req_hash = format!("composer-{}", calculate_hash(&requirements)); + if let Some(db) = conn.as_sql() { + if let Err(e) = sqlx::query!( + "INSERT INTO pip_resolution_cache (hash, lockfile, expiration) VALUES ($1, $2, now() + ('7 days')::interval) ON CONFLICT (hash) DO UPDATE SET lockfile = EXCLUDED.lockfile, expiration = EXCLUDED.expiration", + req_hash, + &resolved_lock + ) + .execute(db) + .await + { + tracing::warn!("Could not cache composer lockfile resolution: {e:?}"); + } + } + } } Ok(format!( diff --git a/backend/windmill-worker/src/python_executor.rs b/backend/windmill-worker/src/python_executor.rs index 3b6507a7e0..78d3d54e15 100644 --- a/backend/windmill-worker/src/python_executor.rs +++ b/backend/windmill-worker/src/python_executor.rs @@ -117,7 +117,12 @@ async fn handle_piptar_uploads(mut rx: tokio::sync::mpsc::UnboundedReceiver bool { + RELATIVE_IMPORT_REGEX.is_match(content) +} #[cfg(all(feature = "enterprise", feature = "parquet", unix))] use crate::global_cache::pull_from_tar; @@ -1084,6 +1089,320 @@ fn python_preprocessor_spread(sig: windmill_parser::MainArgSignature, indent: &s } } +/// Pre-computed codegen data for a Python script. +/// Computed in Rust from the parsed signature, then baked into the wrapper template. +#[cfg(any(feature = "private", test))] +pub struct PyScriptCodegen { + /// Python module directory dot notation (e.g., "f.my") + pub module_dir_dot: String, + /// Python module name / last path component (e.g., "script") + pub module_name: String, + /// Directory path for the module (e.g., "f/my") + pub dirs: String, + /// Inline Python code for type transforms (dates, bytes, etc.) + pub transforms: String, + /// Inline Python code for arg spread / filtering + pub spread: String, + /// Inline Python code for preprocessor arg spread (if applicable) + pub pre_spread: Option, +} + +/// Parse a Python script and compute the codegen data. +/// This reuses the same logic that was used on main in `prepare_wrapper`. +#[cfg(any(feature = "private", test))] +pub fn compute_py_codegen(content: &str, script_path: &str) -> PyScriptCodegen { + let dirs = compute_python_module_dir(script_path); + let last = script_path + .split("/") + .map(|x| { + if x.starts_with(|x: char| x.is_ascii_digit()) { + format!("_{}", x) + } else { + x.to_string() + } + }) + .last() + .unwrap() + .replace("-", "_") + .replace(" ", "_") + .to_lowercase(); + + let sig = windmill_parser_py::parse_python_signature(content, None, false).unwrap_or_default(); + let pre_sig = windmill_parser_py::parse_python_signature( + content, + Some("preprocessor".to_string()), + false, + ) + .ok() + .filter(|s| !s.args.is_empty()); + + let init_sig = pre_sig.as_ref().unwrap_or(&sig); + + let transforms = init_sig + .args + .iter() + .map(|x| match x.typ { + windmill_parser::Typ::Bytes => { + let name = &x.name; + format!( + "if \"{name}\" in kwargs and kwargs[\"{name}\"] is not None:\n \ + kwargs[\"{name}\"] = base64.b64decode(kwargs[\"{name}\"])\n", + ) + } + windmill_parser::Typ::Datetime => { + let name = &x.name; + format!( + "if \"{name}\" in kwargs and kwargs[\"{name}\"] is not None:\n \ + kwargs[\"{name}\"] = datetime.fromisoformat(kwargs[\"{name}\"])\n", + ) + } + windmill_parser::Typ::Date => { + let name = &x.name; + format!( + "if \"{name}\" in kwargs and kwargs[\"{name}\"] is not None:\n \ + try:\n \ + kwargs[\"{name}\"] = date.fromisoformat(kwargs[\"{name}\"])\n \ + except ValueError:\n \ + for _fmt in (\"%d-%m-%Y\", \"%m/%d/%Y\", \"%d/%m/%Y\", \"%Y/%m/%d\"):\n \ + try:\n \ + kwargs[\"{name}\"] = datetime.strptime(kwargs[\"{name}\"], _fmt).date()\n \ + break\n \ + except ValueError:\n \ + continue\n", + ) + } + _ => "".to_string(), + }) + .collect::>() + .join(""); + + let spread = if sig.star_kwargs { + "args = kwargs".to_string() + } else { + sig.args + .into_iter() + .map(|x| { + let name = &x.name; + if x.default.is_none() { + format!("args[\"{name}\"] = kwargs.get(\"{name}\")") + } else { + format!( + r#"args["{name}"] = kwargs.get("{name}") + if args["{name}"] is None: + del args["{name}"]"# + ) + } + }) + .join("\n ") + }; + + let pre_spread = pre_sig.map(|sig| python_preprocessor_spread(sig, " ")); + + let module_dir_dot = dirs.replace("/", ".").replace("-", "_"); + + PyScriptCodegen { module_dir_dot, module_name: last, dirs, transforms, spread, pre_spread } +} + +/// Script entry for the Python unified wrapper generator. +#[cfg(any(feature = "private", test))] +pub struct PyScriptEntry<'a> { + pub original_path: &'a str, + pub codegen: &'a PyScriptCodegen, +} + +/// Generate a wrapper for Python dedicated workers and runner groups. +/// All scripts are baked in at codegen time with proper Python imports and inline arg handling. +/// Protocol: +/// exec:: -> wm_res[success]: | wm_res[error]: +/// exec_preprocess:: -> wm_res[preprocessed_args]: then wm_res[success]: | wm_res[error]: +/// end -> exit +#[cfg(any(feature = "private", test))] +pub fn generate_multi_script_wrapper( + scripts: &[PyScriptEntry<'_>], + skip_result_postprocessing: bool, + any_relative_imports: bool, +) -> String { + let postprocessor = get_result_postprocessor(skip_result_postprocessing); + let res_to_json_body = python_res_to_json_body(postprocessor); + + let imports: String = scripts + .iter() + .enumerate() + .map(|(i, e)| { + format!( + "from {module_dir_dot} import {module_name} as _s{i}", + module_dir_dot = e.codegen.module_dir_dot, + module_name = e.codegen.module_name, + ) + }) + .collect::>() + .join("\n"); + + let mut functions = String::new(); + let mut registrations = String::new(); + + for (i, entry) in scripts.iter().enumerate() { + let cg = entry.codegen; + let indented_transforms = cg + .transforms + .split('\n') + .map(|line| { + if line.is_empty() { + String::new() + } else { + format!(" {}", line) + } + }) + .collect::>() + .join("\n"); + + functions.push_str(&format!( + r#" +def transform_{i}(kwargs): +{indented_transforms} + args = dict() + {spread} + for k, v in list(args.items()): + if v == '': + del args[k] + return args +"#, + spread = cg.spread + )); + + let pre_fn = if let Some(ref pre_spread) = cg.pre_spread { + functions.push_str(&format!( + r#" +def pre_transform_{i}(kwargs): + pre_args = dict() + {pre_spread} + for k, v in list(pre_args.items()): + if v == '': + del pre_args[k] + return pre_args +"#, + )); + format!("pre_transform_{i}") + } else { + "None".to_string() + }; + + registrations.push_str(&format!( + "scripts[\"{path}\"] = {{ 'mod': _s{i}, 'transform': transform_{i}, 'pre_transform': {pre_fn} }}\n", + path = entry.original_path, + )); + } + + let import_loader = if any_relative_imports { + "import loader" + } else { + "" + }; + + format!( + r#" +import json +import sys +import traceback +import re +import base64 +from datetime import datetime, date +{import_loader} + +{imports} + +scripts = {{}} + +def to_b_64(v: bytes): + b64 = base64.b64encode(v) + return b64.decode('ascii') + +replace_invalid_fields = re.compile(r'(?:\bNaN\b|\\u0000|Infinity|\-Infinity)') + +def res_to_json(res, typ): +{res_to_json_body} +{functions} +{registrations} + +sys.stdout.write('start\n') +sys.stdout.flush() + +for line in sys.stdin: + line = line.strip() + if line == 'end': + break + + if line.startswith('exec_preprocess:'): + try: + rest = line[len('exec_preprocess:'):] + colon_idx = rest.index(':') + script_path = rest[:colon_idx] + args_json = rest[colon_idx + 1:] + + entry = scripts.get(script_path) + if not entry: + err_json = json.dumps({{ "message": "Script not found: " + script_path, "name": "Error" }}, separators=(',', ':'), default=str).replace('\n', '') + sys.stdout.write("wm_res[error]:" + err_json + "\n") + sys.stdout.flush() + continue + + mod = entry['mod'] + if not hasattr(mod, 'preprocessor') or not callable(mod.preprocessor): + err_json = json.dumps({{"message": "preprocessor function is missing", "name": "Error"}}, separators=(',', ':'), default=str).replace('\n', '') + sys.stdout.write("wm_res[error]:" + err_json + "\n") + sys.stdout.flush() + continue + kwargs = json.loads(args_json, strict=False) + pre_args = entry['pre_transform'](kwargs) + preprocessed = mod.preprocessor(**pre_args) + preprocessed_json = json.dumps(preprocessed, separators=(',', ':'), default=str).replace('\n', '') + sys.stdout.write("wm_res[preprocessed_args]:" + preprocessed_json + "\n") + main_args = entry['transform'](preprocessed if preprocessed else {{}}) + res = mod.main(**main_args) + typ = type(res) + res_json = res_to_json(res, typ) + sys.stdout.write("wm_res[success]:" + res_json + "\n") + except BaseException as e: + exc_type, exc_value, exc_traceback = sys.exc_info() + tb = traceback.format_tb(exc_traceback) + err_json = json.dumps({{ "message": str(e), "name": e.__class__.__name__, "stack": '\n'.join(tb[1:]) }}, separators=(',', ':'), default=str).replace('\n', '') + sys.stdout.write("wm_res[error]:" + err_json + "\n") + sys.stdout.flush() + continue + + if line.startswith('exec:'): + try: + rest = line[len('exec:'):] + colon_idx = rest.index(':') + script_path = rest[:colon_idx] + args_json = rest[colon_idx + 1:] + + entry = scripts.get(script_path) + if not entry: + err_json = json.dumps({{ "message": "Script not found: " + script_path, "name": "Error" }}, separators=(',', ':'), default=str).replace('\n', '') + sys.stdout.write("wm_res[error]:" + err_json + "\n") + sys.stdout.flush() + continue + + kwargs = json.loads(args_json, strict=False) + args = entry['transform'](kwargs) + res = entry['mod'].main(**args) + typ = type(res) + res_json = res_to_json(res, typ) + sys.stdout.write("wm_res[success]:" + res_json + "\n") + except BaseException as e: + exc_type, exc_value, exc_traceback = sys.exc_info() + tb = traceback.format_tb(exc_traceback) + err_json = json.dumps({{ "message": str(e), "name": e.__class__.__name__, "stack": '\n'.join(tb[1:]) }}, separators=(',', ':'), default=str).replace('\n', '') + sys.stdout.write("wm_res[error]:" + err_json + "\n") + sys.stdout.flush() + continue + + sys.stderr.write("Unknown command: " + line + "\n") +"# + ) +} + async fn prepare_wrapper( job_dir: &str, job_flow_step_id: Option<&str>, @@ -1303,7 +1622,7 @@ async fn replace_pip_secret( } } -async fn handle_python_deps( +pub(crate) async fn handle_python_deps( job_dir: &str, requirements_o: Option<&String>, inner_content: &str, @@ -2419,6 +2738,22 @@ pub async fn start_worker( use crate::PyV; tracing::info!("script path: {}", script_path); + let codegen = compute_py_codegen(inner_content, script_path); + + // Write script to proper module path (e.g., f/my/script.py) + let module_dir = format!("{}/{}", job_dir, codegen.dirs); + tokio::fs::create_dir_all(&module_dir).await?; + write_file( + &module_dir, + &format!("{}.py", codegen.module_name), + inner_content, + )?; + + let any_relative_imports = RELATIVE_IMPORT_REGEX.is_match(inner_content); + if any_relative_imports { + let _ = write_file(job_dir, "loader.py", RELATIVE_PYTHON_LOADER)?; + } + let mut mem_peak: i32 = 0; let mut canceled_by: Option = None; let context = variables::get_reserved_variables( @@ -2463,122 +2798,12 @@ pub async fn start_worker( ) .await?; - let ( - import_loader, - import_base64, - import_datetime, - module_dir_dot, - _dirs, - last, - transforms, - spread, - _, - _, - ) = prepare_wrapper(job_dir, None, None, None, inner_content, script_path).await?; - - // Parse preprocessor signature if the script has one - let pre_spread = windmill_parser_py::parse_python_signature( - inner_content, - Some("preprocessor".to_string()), - false, - ) - .ok() - .filter(|sig| !sig.args.is_empty()) - .map(|sig| python_preprocessor_spread(sig, " ")); - { - let postprocessor = get_result_postprocessor(annotations.skip_result_postprocessing); - let indented_transforms = transforms - .lines() - .map(|x| format!(" {}", x)) - .collect::>() - .join("\n"); - - let preprocessor_logic = if let Some(ref pre_spread) = pre_spread { - format!( - r#" - if line.startswith('preprocess:'): - pre_input = line[len('preprocess:'):] - kwargs = json.loads(pre_input, strict=False) - if not hasattr(inner_script, 'preprocessor') or not callable(inner_script.preprocessor): - err_json = json.dumps({{"message": "preprocessor function is missing", "name": "Error"}}, separators=(',', ':'), default=str).replace('\n', '') - sys.stdout.write("wm_res[error]:" + err_json + "\n") - sys.stdout.flush() - continue - try: - pre_args = {{}} - {pre_spread} - for k, v in list(pre_args.items()): - if v == '': - del pre_args[k] - preprocessed_kwargs = inner_script.preprocessor(**pre_args) - preprocessed_json = json.dumps(preprocessed_kwargs, separators=(',', ':'), default=str).replace('\n', '') - sys.stdout.write("wm_res[preprocessed_args]:" + preprocessed_json + "\n") - transform_and_run(preprocessed_kwargs) - except BaseException as e: - exc_type, exc_value, exc_traceback = sys.exc_info() - tb = traceback.format_tb(exc_traceback) - err_json = json.dumps({{ "message": str(e), "name": e.__class__.__name__, "stack": '\n'.join(tb[1:]) }}, separators=(',', ':'), default=str).replace('\n', '') - sys.stdout.write("wm_res[error]:" + err_json + "\n") - sys.stdout.flush() - continue -"# - ) - } else { - String::new() - }; - - let res_to_json_body = python_res_to_json_body(postprocessor); - let wrapper_content: String = format!( - r#" -import json -{import_loader} -{import_base64} -{import_datetime} -import traceback -import sys -from {module_dir_dot} import {last} as inner_script -import re - - -def to_b_64(v: bytes): - import base64 - b64 = base64.b64encode(v) - return b64.decode('ascii') - -def res_to_json(res, typ): -{res_to_json_body} - -def transform_and_run(kwargs): - args = {{}} -{indented_transforms} - {spread} - for k, v in list(args.items()): - if v == '': - del args[k] - res = inner_script.main(**args) - typ = type(res) - res_json = res_to_json(res, typ) - sys.stdout.write("wm_res[success]:" + res_json + "\n") - -replace_invalid_fields = re.compile(r'(?:\bNaN\b|\\u0000|Infinity|\-Infinity)') -sys.stdout.write('start\n') - -for line in sys.stdin: - if line == 'end\n': - break - line = line.strip() - {preprocessor_logic} - kwargs = json.loads(line, strict=False) - try: - transform_and_run(kwargs) - except BaseException as e: - exc_type, exc_value, exc_traceback = sys.exc_info() - tb = traceback.format_tb(exc_traceback) - err_json = json.dumps({{ "message": str(e), "name": e.__class__.__name__, "stack": '\n'.join(tb[1:]) }}, separators=(',', ':'), default=str).replace('\n', '') - sys.stdout.write("wm_res[error]:" + err_json + "\n") - sys.stdout.flush() -"#, + let scripts = [PyScriptEntry { original_path: script_path, codegen: &codegen }]; + let wrapper_content = generate_multi_script_wrapper( + &scripts, + annotations.skip_result_postprocessing, + any_relative_imports, ); write_file(job_dir, "wrapper.py", &wrapper_content)?; } @@ -2632,6 +2857,7 @@ for line in sys.stdin: &mut None, ) .await?; + handle_dedicated_process( &python_path, job_dir, @@ -2704,4 +2930,43 @@ mod tests { // @ is replaced with . assert_eq!(compute_python_module_dir("u/@admin/script"), "u/.admin"); } + + #[test] + fn test_compute_py_codegen_basic_args() { + let code = "def main(x: str, y: int):\n return x\n"; + let cg = compute_py_codegen(code, "f/test/script"); + assert!(cg.spread.contains("args[\"x\"]")); + assert!(cg.spread.contains("args[\"y\"]")); + assert!(cg.transforms.is_empty()); + assert!(cg.pre_spread.is_none()); + assert_eq!(cg.module_name, "script"); + } + + #[test] + fn test_compute_py_codegen_with_datetime_and_bytes() { + let code = "import datetime\n\ndef main(name: str, created_at: datetime.datetime, file: bytes):\n return name\n"; + let cg = compute_py_codegen(code, "f/my/handler"); + assert!(cg.transforms.contains("datetime.fromisoformat")); + assert!(cg.transforms.contains("base64.b64decode")); + assert!(cg.spread.contains("args[\"name\"]")); + assert_eq!(cg.module_dir_dot, "f.my"); + assert_eq!(cg.module_name, "handler"); + } + + #[test] + fn test_compute_py_codegen_star_kwargs() { + let code = "def main(**kwargs):\n return kwargs\n"; + let cg = compute_py_codegen(code, "f/test/star"); + assert_eq!(cg.spread, "args = kwargs"); + } + + #[test] + fn test_compute_py_codegen_with_preprocessor() { + let code = "import datetime\n\ndef main(x: str, ts: datetime.datetime):\n return x\n\ndef preprocessor(input: str, when: datetime.datetime):\n return {\"x\": input, \"ts\": when}\n"; + let cg = compute_py_codegen(code, "f/test/pre"); + assert!(cg.spread.contains("args[\"x\"]")); + assert!(cg.pre_spread.is_some()); + let pre = cg.pre_spread.as_ref().unwrap(); + assert!(pre.contains("pre_args[\"input\"]")); + } } diff --git a/backend/windmill-worker/src/wac_executor.rs b/backend/windmill-worker/src/wac_executor.rs index 208012ccce..28de226102 100644 --- a/backend/windmill-worker/src/wac_executor.rs +++ b/backend/windmill-worker/src/wac_executor.rs @@ -59,7 +59,13 @@ pub enum WacOutput { /// No child job is dispatched — the parent suspends directly and resumes /// when a user hits the resume/cancel endpoint. #[serde(rename = "approval")] - Approval { key: String, timeout: Option, form: Option }, + Approval { + key: String, + timeout: Option, + form: Option, + #[serde(default)] + self_approval_disabled: Option, + }, /// Server-side sleep — suspend the workflow for a duration without holding a worker. #[serde(rename = "sleep")] Sleep { key: String, seconds: u32 }, @@ -306,15 +312,13 @@ pub async fn prepare_checkpoint_for_resume( } /// Detect WAC v2 patterns in TypeScript/Bun code. -/// Checks for `import ... from "windmill-client"` containing workflow/task, +/// Checks for `import ... from "windmill-client"` containing workflow, /// skipping comment lines. Handles both single-line and multi-line imports. pub fn is_wac_v2_ts(code: &str) -> bool { let mut has_wac_import = false; let mut has_workflow = false; - let mut has_task = false; let mut in_import_block = false; let mut import_block_has_workflow = false; - let mut import_block_has_task = false; for line in code.lines() { let trimmed = line.trim(); if trimmed.starts_with("//") { @@ -328,34 +332,24 @@ pub fn is_wac_v2_ts(code: &str) -> bool { if trimmed.contains("workflow") { has_workflow = true; } - if trimmed.contains("task") { - has_task = true; - } in_import_block = false; } // Start of multi-line import: import { else if trimmed.starts_with("import") && trimmed.contains("{") && !trimmed.contains("}") { in_import_block = true; import_block_has_workflow = trimmed.contains("workflow"); - import_block_has_task = trimmed.contains("task"); } // Inside multi-line import block else if in_import_block { if trimmed.contains("workflow") { import_block_has_workflow = true; } - if trimmed.contains("task") { - import_block_has_task = true; - } // End of multi-line import: } from "windmill-client" if trimmed.contains("windmill-client") { has_wac_import = true; if import_block_has_workflow { has_workflow = true; } - if import_block_has_task { - has_task = true; - } in_import_block = false; } // End of import block but not windmill-client @@ -367,7 +361,24 @@ pub fn is_wac_v2_ts(code: &str) -> bool { has_workflow = true; } } - has_wac_import && has_workflow && has_task + has_wac_import && has_workflow +} + +/// Inject the variable name as the first argument to `task()` calls in WAC v2 scripts. +/// `const double = task(async ...` → `const double = task("double", async ...` +/// Skips calls that already have a string argument. +pub fn inject_wac_task_names(content: &str) -> String { + use regex::Regex; + use std::borrow::Cow; + lazy_static::lazy_static! { + static ref TASK_RE: Regex = + Regex::new(r#"(?m)((?:export\s+)?(?:const|let|var)\s+)(\w+)(\s*=\s*task\s*(?:<[^>]*>)?\s*\(\s*)(async\b)"#).unwrap(); + } + let replaced = TASK_RE.replace_all(content, r#"${1}${2}${3}"${2}", ${4}"#); + match replaced { + Cow::Borrowed(_) => content.to_string(), + Cow::Owned(s) => s, + } } /// Detect WAC v2 patterns in Python code. diff --git a/backend/windmill-worker/src/worker.rs b/backend/windmill-worker/src/worker.rs index 03e97a6ac1..8aeb3a9e5e 100644 --- a/backend/windmill-worker/src/worker.rs +++ b/backend/windmill-worker/src/worker.rs @@ -965,6 +965,7 @@ async fn get_otel_tracing_proxy_envs( TRACING_PROXY_CA_CERT_PATH.to_string(), ), ("CURL_CA_BUNDLE", TRACING_PROXY_CA_CERT_PATH.to_string()), + ("GIT_SSL_CAINFO", TRACING_PROXY_CA_CERT_PATH.to_string()), ("DENO_CERT", TRACING_PROXY_CA_CERT_PATH.to_string()), ]) } @@ -2744,6 +2745,8 @@ pub async fn run_worker( let arc_job = Arc::new(job); + windmill_common::sensitive_log_masks::register_running_job(arc_job.id); + let span = create_span_with_name(&arc_job, &worker_name, Some(hostname), "job"); let job_result = handle_queued_job( @@ -2843,6 +2846,8 @@ pub async fn run_worker( _ => {} } + windmill_common::sensitive_log_masks::unregister_running_job(job_id); + #[cfg(feature = "prometheus")] if let Some(duration) = _timer.map(|x| x.stop_and_record()) { register_metric( @@ -4493,7 +4498,10 @@ pub async fn run_language_executor( "const process = {{ env: {{}} }};\nconst BASE_URL = '{base_internal_url}';\nconst BASE_INTERNAL_URL = '{base_internal_url}';\nprocess.env['BASE_URL'] = BASE_URL;process.env['BASE_INTERNAL_URL'] = BASE_INTERNAL_URL;\n{}", reserved_variables .iter() - .map(|(k, v)| format!("const {} = '{}';\nprocess.env['{}'] = '{}';\n", k, v, k, v)) + .map(|(k, v)| { + let escaped = v.replace('\\', "\\\\").replace('\'', "\\'").replace('\n', "\\n").replace('\r', "\\r"); + format!("const {} = '{}';\nprocess.env['{}'] = '{}';\n", k, escaped, k, escaped) + }) .collect::>() .join("\n")); @@ -5404,3 +5412,4 @@ pub fn get_worker_internal_server_inline_utils( )), } } + diff --git a/backend/windmill-worker/src/worker_lockfiles.rs b/backend/windmill-worker/src/worker_lockfiles.rs index 93b48cf818..5ea5b74d64 100644 --- a/backend/windmill-worker/src/worker_lockfiles.rs +++ b/backend/windmill-worker/src/worker_lockfiles.rs @@ -427,13 +427,13 @@ pub async fn handle_flow_dependency_job( // `JobKind::FlowDependencies` job store either: // - A saved flow version `id` in the `script_hash` column. // - Preview raw flow in the `queue` or `job` table. - let (mut flow, notes) = match job.runnable_id { + let (mut flow, extras) = match job.runnable_id { Some(ScriptHash(id)) => { let flow = cache::flow::fetch_version(db, id).await?; - (flow.value().clone(), flow.notes()) + (flow.value().clone(), flow.extras()) } _ => match preview_data { - Some(RawData::Flow(data)) => (data.value().clone(), data.notes()), + Some(RawData::Flow(data)) => (data.value().clone(), data.extras()), _ => return Err(Error::internal_err("expected script hash")), }, }; @@ -528,18 +528,22 @@ pub async fn handle_flow_dependency_job( } #[derive(Debug, Clone, Serialize)] - struct FlowValueWithNotes<'a> { + struct FlowValueWithExtras<'a> { #[serde(flatten)] value: &'a FlowValue, #[serde(skip_serializing_if = "Option::is_none")] - notes: Option>, // TODO: Make this a Vec + notes: Option>, + + #[serde(skip_serializing_if = "Option::is_none")] + groups: Option>, } let new_flow_value = Json( - serde_json::value::to_raw_value(&FlowValueWithNotes { + serde_json::value::to_raw_value(&FlowValueWithExtras { value: &flow, - notes: notes.and_then(|n| n.notes).map(|n| n.into()), + notes: extras.as_ref().and_then(|e| e.notes.clone()), + groups: extras.as_ref().and_then(|e| e.groups.clone()), }) .map_err(to_anyhow)?, ); @@ -598,11 +602,19 @@ pub async fn handle_flow_dependency_job( ) .await?; + let value_lite_with_extras = Json( + serde_json::value::to_raw_value(&FlowValueWithExtras { + value: &value_lite, + notes: extras.as_ref().and_then(|e| e.notes.clone()), + groups: extras.as_ref().and_then(|e| e.groups.clone()), + }) + .map_err(to_anyhow)?, + ); sqlx::query!( "INSERT INTO flow_version_lite (id, value) VALUES ($1, $2) ON CONFLICT (id) DO UPDATE SET value = EXCLUDED.value", version, - Json(value_lite) as Json, + &value_lite_with_extras as &Json>, ) .execute(&mut *tx) .await?; diff --git a/benchmarks/Dockerfile b/benchmarks/Dockerfile index c8f3fe83d5..7655e5f2d2 100644 --- a/benchmarks/Dockerfile +++ b/benchmarks/Dockerfile @@ -1,14 +1,20 @@ -FROM denoland/deno:alpine-1.26.2 +FROM denoland/deno:alpine-2.1.4 WORKDIR /app USER deno +ADD ./lib.ts . +ADD ./action.ts . ADD ./main.ts . -RUN deno cache --unstable main.ts +RUN deno cache main.ts ADD ./worker.ts . -RUN deno cache --unstable worker.ts +RUN deno cache worker.ts ADD ./scraper.ts . -RUN deno cache --unstable scraper.ts +RUN deno cache scraper.ts +ADD ./benchmark_oneoff.ts . +RUN deno cache benchmark_oneoff.ts +ADD ./benchmark_suite.ts . +RUN deno cache benchmark_suite.ts -ENTRYPOINT [ "/tini", "--", "docker-entrypoint.sh", "run", "--unstable", "-A", "main.ts" ] \ No newline at end of file +ENTRYPOINT [ "/tini", "--", "docker-entrypoint.sh", "run", "-A", "main.ts" ] diff --git a/benchmarks/README.md b/benchmarks/README.md index c358471626..fc758c006f 100644 --- a/benchmarks/README.md +++ b/benchmarks/README.md @@ -1,85 +1,71 @@ # Benchmarks -This folder includes a small deno/ts utility to benchmark execution of jobs & -flows. +Deno/TS benchmark suite for measuring Windmill job and flow execution throughput. -## Installation +## Quick Start -Install the `wmill` CLI tool using -`deno install --unstable -A https://deno.land/x/wmillbench/main.ts`. +```bash +# Install Deno +curl -fsSL https://deno.land/install.sh | sh -Update to the latest version using `wmillbench upgrade`. +# Run a single benchmark +deno run -A benchmark_oneoff.ts --kind noop --jobs 10000 -To build a local version, you can just run: -``` -deno install -A main.ts +# Run the full suite +deno run -A benchmark_suite.ts -c suite_config.json + +# Run WAC v2 benchmarks (workflow-as-code vs flow comparison) +deno run -A benchmark_suite.ts -c suite_wac.json ``` -## Quickstart +## Benchmark Kinds -Have your instance expose prometheus metrics (METRICS_ADDR=true). +### Script benchmarks +- `noop` — Empty jobs (measures pure scheduling overhead) +- `deno`, `bun`, `python`, `go`, `bash` — Language runtimes +- `nativets` — BunNative (no isolation) +- `dedicated`, `dedicated_nativets` — Dedicated worker mode -Then +### Flow benchmarks +- `2steps` — 2-step flow (deno + identity) +- `bigscriptinflow` — Flow with large raw bash script +- `flow_seq_2_bun` — 2 sequential bun steps +- `flow_par_2_bun` — 2 parallel bun steps (branchall) +- `flow_seq_3_bun` — 3 sequential bun steps +- `flow:` — Custom flow by path +- `script:` — Custom script by path -``` -wmillbench -e admin@windmill.dev -p changeme --host YOUR_HOST +### WAC v2 benchmarks (workflow-as-code) +- `wac_seq_2` — 2 sequential tasks +- `wac_par_2` — 2 parallel tasks (Promise.all) +- `wac_seq_3` — 3 sequential tasks +- `wac_inline_2` — 2 inline steps (no child jobs) + +## Suite Configs + +| File | Description | +|------|-------------| +| `suite_config.json` | Main benchmark suite (noop, languages, flows) | +| `suite_dedicated.json` | Dedicated worker benchmarks | +| `suite_dedicated_nativets.json` | Dedicated NativeTS benchmarks | +| `suite_wac.json` | WAC v2 vs flow comparison benchmarks | + +## Interactive Benchmark Tool + +```bash +deno run -A main.ts -e admin@windmill.dev -p changeme --host http://localhost:8000 ``` -## Usage +Options: `--workers`, `--seconds`, `--maximum-throughput`, `--use-flows`, `--script-pattern`, `--export-json`, `--export-csv` -Usage: wmillbench +## Graph Generation -Description: - -Run Benchmark to measure throughput of windmill. - -Options: - --h, --help - Show this help. --V, --version - Show the version number for this program. ---host - The windmill host to benchmark. (Default: "http://127.0.0.1:8000/") ---workers - The number of workers to run at once. (Default: 1) --s, --seconds - How long to run the benchmark for (in seconds). (Default: 30) --e, --email - The email to use to login. --p, --password - The password to use to login. --t, --token - The token to use when talking to the API server. Preferred over manual login. --w, --workspace - The workspace to spawn scripts from. (Default: "starter") --m, --metrics - The url to scrape metrics from. (Default: "http://localhost:8001/metrics") ---export-json - If set, exports will be into a JSON file. ---export-csv - If set, exports will be into a csv file. ---export-histograms [histograms...] - Mark metrics (without label) that are reported as histograms to export. ---export-simple [simple...] - Mark metrics (without label) that are reported as simple values. ---maximum-throughput - Maximum number of jobs/flows to start in one second. (Default: Infinity) ---use-flows - Run flows instead of jobs. ---histogram-buckets [buckets...] - Define what buckets to collect from histograms. (Default: [ "+Inf", "10", "5", "2.5", "2.5", "1", "0.5", "0.25", "0.1", "0.05", "0.025", "0.01", "0.005" ]) - -Environment variables: - -WM_TOKEN - The token to use when talking to the API server. Preferred -over manual login. WM_WORKSPACE - The workspace to spawn scripts -from. - - - -This will run a simple benchmark against localhost (the default admin email + -password are set above), all execution is done in the "bench" workspace (as set -via `--workspace`). - -Metrics are exported to JSON will only include mean & stdev, histograms get one -entry for each bucket. CSV will include a full list of all values scraped. - -## NOOP jobs benchmark - -A specific benchmark creating a set of NOOP jobs all at once in windmill is also available. -in `benchmarks_noop.ts` - -You can build it locally with: -``` -deno install -A benchmarks_noop.ts -``` -and then -``` -benchmarks_noop -e admin@windmill.dev -p changeme --host YOUR_HOST +```bash +deno run -A benchmark_graphs.ts -c graphs_config.json ``` -By default it creates 10000 jobs in Windmill in a single batch, but this is parametrizable. \ No newline at end of file +Generates SVG graphs from `*_benchmark.json` data files. + +## CI + +The GitHub Actions workflow (`.github/workflows/benchmark.yml`) runs hourly with 1/4/8 worker configurations plus WAC benchmarks. Results are committed to the `benchmarks` branch. diff --git a/benchmarks/benchmark_graphs.ts b/benchmarks/benchmark_graphs.ts index 23b3b763f7..bfaad7dbb6 100644 --- a/benchmarks/benchmark_graphs.ts +++ b/benchmarks/benchmark_graphs.ts @@ -3,32 +3,20 @@ import { UpgradeCommand } from "https://deno.land/x/cliffy@v0.25.7/command/upgra import { DenoLandProvider } from "https://deno.land/x/cliffy@v0.25.7/command/upgrade/mod.ts"; import { drawGraph, drawGraphMulti } from "./graph.ts"; -import { VERSION } from "./lib.ts"; +import { VERSION, loadJsonConfig } from "./lib.ts"; -type GraphsConfig = [ - { - graph_title: string; - benchmarks: { - kind: string; - workers: number; - label: string; - }[]; - jobs: number; - } -]; +type GraphsConfig = { + graph_title: string; + benchmarks: { + kind: string; + workers: number; + label: string; + }[]; +}[]; async function main({ configPath }: { configPath: string }) { - async function getConfig(configPath: string): Promise { - if (configPath.startsWith("http")) { - const response = await fetch(configPath); - return await response.json(); - } else { - return JSON.parse(await Deno.readTextFile(configPath)); - } - } - try { - const config = await getConfig(configPath); + const config = await loadJsonConfig(configPath); for (const graphConfig of config || []) { const data: { @@ -81,7 +69,7 @@ async function main({ configPath }: { configPath: string }) { } await new Command() - .name("wmillbenchsuite") + .name("wmillbenchgraphs") .description("Create and save graphs from benchmark data.") .version(VERSION) .option("-c --config-path ", "The path of the config file", { diff --git a/benchmarks/benchmark_oneoff.ts b/benchmarks/benchmark_oneoff.ts index 0cd4c3483f..5f075d2034 100644 --- a/benchmarks/benchmark_oneoff.ts +++ b/benchmarks/benchmark_oneoff.ts @@ -10,7 +10,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -import { VERSION, createBenchScript, getFlowPayload, login } from "./lib.ts"; +import { VERSION, createBenchScript, createWacBenchScript, getFlowPayload, login, WAC_KINDS, STEPS_PER_WORKFLOW } from "./lib.ts"; async function verifyOutputs(uuids: string[], workspace: string) { console.log("Verifying outputs"); @@ -38,6 +38,8 @@ async function verifyOutputs(uuids: string[], workspace: string) { } export const NON_TEST_TAGS = ["deno", "python", "go", "bash", "dedicated", "bun", "nativets", "dedicated_nativets", "flow"] + +const FLOW_COMPARISON_KINDS = ["flow_seq_2_bun", "flow_par_2_bun", "flow_seq_3_bun"]; export async function main({ host, email, @@ -151,6 +153,8 @@ export async function main({ ) ) { await createBenchScript(kind, workspace); + } else if (WAC_KINDS.includes(kind)) { + await createWacBenchScript(kind, workspace); } @@ -173,6 +177,20 @@ export async function main({ kind: "script", path: "f/benchmarks/" + kind, }); + } else if (WAC_KINDS.includes(kind)) { + // WAC v2 scripts are deployed as bun scripts, run via script path + nStepsFlow = STEPS_PER_WORKFLOW[kind] ?? 0; + body = JSON.stringify({ + kind: "script", + path: "f/benchmarks/" + kind, + }); + } else if (FLOW_COMPARISON_KINDS.includes(kind)) { + nStepsFlow = STEPS_PER_WORKFLOW[kind] ?? 0; + const payload = getFlowPayload(kind); + body = JSON.stringify({ + kind: "flow", + flow_value: payload.value, + }); } else if (["2steps", "bigscriptinflow"].includes(kind)) { nStepsFlow = kind == "2steps" ? 2 : 1; const payload = getFlowPayload(kind); @@ -182,7 +200,7 @@ export async function main({ }); } else if (kind.startsWith("flow:")) { console.log("Detected custom flow "); - let flow_path = kind.substr(5); + let flow_path = kind.substring(5); nStepsFlow = await getFlowStepCount(config.workspace_id, flow_path); console.log(`Total steps of flow including sub-flows: ${nStepsFlow}`); body = JSON.stringify({ @@ -193,7 +211,7 @@ export async function main({ console.log("Detected custom script"); body = JSON.stringify({ kind: "script", - path: kind.substr(7), + path: kind.substring(7), }); } else if (kind == "bigrawscript") { noVerify = true; @@ -281,6 +299,9 @@ export async function main({ let lastElapsed = 0; let lastCompletedJobs = 0; + // Timeout: 10 minutes for the polling loop to prevent hanging forever + // (e.g. if WAC suspend/resume fails or jobs get stuck) + const POLL_TIMEOUT_MS = 10 * 60 * 1000; let didStart = false; while (completedJobs < jobsSent) { const loopStart = Date.now(); @@ -292,6 +313,10 @@ export async function main({ } } else { const elapsed = start ? Date.now() - start : 0; + if (elapsed > POLL_TIMEOUT_MS) { + console.error(`\nTimeout: benchmark did not complete within ${POLL_TIMEOUT_MS / 1000}s (${completedJobs}/${jobsSent} completed)`); + break; + } completedJobs = await getCompletedJobsCount(NON_TEST_TAGS); if (nStepsFlow > 0) { completedJobs = Math.floor(completedJobs / (nStepsFlow + 1)); @@ -338,7 +363,9 @@ export async function main({ kind !== "nativets" && kind !== "dedicated_nativets" && !kind.startsWith("flow:") && - !kind.startsWith("script:") + !kind.startsWith("script:") && + !WAC_KINDS.includes(kind) && + !FLOW_COMPARISON_KINDS.includes(kind) ) { await verifyOutputs(uuids, config.workspace_id); } @@ -387,7 +414,7 @@ if (import.meta.main) { ) .option( "--kind ", - "Specifiy the benchmark kind among: deno, identity, python, go, bash, dedicated, bun, noop, 2steps, nativets, dedicated_nativets", + "Specifiy the benchmark kind among: deno, identity, python, go, bash, dedicated, bun, noop, 2steps, nativets, dedicated_nativets, wac_seq_2, wac_par_2, wac_seq_3, wac_inline_2, flow_seq_2_bun, flow_par_2_bun, flow_seq_3_bun", { required: true, } diff --git a/benchmarks/benchmark_suite.ts b/benchmarks/benchmark_suite.ts index f4840dda05..caba48aa72 100644 --- a/benchmarks/benchmark_suite.ts +++ b/benchmarks/benchmark_suite.ts @@ -4,7 +4,7 @@ import { DenoLandProvider } from "https://deno.land/x/cliffy@v0.25.7/command/upg import { main as runBenchmark } from "./benchmark_oneoff.ts"; -import { VERSION } from "./lib.ts"; +import { VERSION, loadJsonConfig } from "./lib.ts"; type Config = { kind: string; @@ -50,21 +50,12 @@ async function main({ workers: number; factor?: number; }) { - async function getConfig(configPath: string): Promise { - if (configPath.startsWith("http")) { - const response = await fetch(configPath); - return await response.json(); - } else { - return JSON.parse(await Deno.readTextFile(configPath)); - } - } - if (!Deno.args.includes("--no-warm-up")) { await warmUp(host, email, password, token, workspace); } try { - const config = await getConfig(configPath); + const config = await loadJsonConfig(configPath); for (const benchmark of config) { try { console.log( diff --git a/benchmarks/graphs_config.json b/benchmarks/graphs_config.json index 174990ecb2..8e37695d66 100644 --- a/benchmarks/graphs_config.json +++ b/benchmarks/graphs_config.json @@ -223,5 +223,75 @@ "label": "noop" } ] + }, + { + "graph_title": "WAC v2 sequential vs flow sequential (2 steps, bun)", + "benchmarks": [ + { + "kind": "wac_seq_2", + "workers": 1, + "label": "WAC v2 sequential" + }, + { + "kind": "flow_seq_2_bun", + "workers": 1, + "label": "Flow sequential" + } + ] + }, + { + "graph_title": "WAC v2 parallel vs flow parallel (2 steps, bun)", + "benchmarks": [ + { + "kind": "wac_par_2", + "workers": 1, + "label": "WAC v2 parallel" + }, + { + "kind": "flow_par_2_bun", + "workers": 1, + "label": "Flow parallel" + } + ] + }, + { + "graph_title": "WAC v2 sequential vs flow sequential (3 steps, bun)", + "benchmarks": [ + { + "kind": "wac_seq_3", + "workers": 1, + "label": "WAC v2 sequential" + }, + { + "kind": "flow_seq_3_bun", + "workers": 1, + "label": "Flow sequential" + } + ] + }, + { + "graph_title": "WAC v2 patterns comparison", + "benchmarks": [ + { + "kind": "wac_seq_2", + "workers": 1, + "label": "sequential 2-task" + }, + { + "kind": "wac_par_2", + "workers": 1, + "label": "parallel 2-task" + }, + { + "kind": "wac_seq_3", + "workers": 1, + "label": "sequential 3-task" + }, + { + "kind": "wac_inline_2", + "workers": 1, + "label": "inline 2-step" + } + ] } ] \ No newline at end of file diff --git a/benchmarks/lib.ts b/benchmarks/lib.ts index b235377979..36f6bebac6 100644 --- a/benchmarks/lib.ts +++ b/benchmarks/lib.ts @@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts"; import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts"; import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts"; -export const VERSION = "v1.662.0"; +export const VERSION = "v1.670.0"; export async function login(email: string, password: string): Promise { return await windmill.UserService.login({ @@ -132,6 +132,119 @@ export async function createBenchScript( } } +// WAC v2 benchmark script content patterns +const WAC_SCRIPTS: Record = { + wac_seq_2: [ + 'import { task, workflow } from "windmill-client";', + "const step_a = task(async () => { return 1; });", + "const step_b = task(async () => { return 2; });", + "export const main = workflow(async () => {", + " const a = await step_a();", + " const b = await step_b();", + " return { a, b };", + "});", + ].join("\n"), + + wac_par_2: [ + 'import { task, workflow } from "windmill-client";', + "const step_a = task(async () => { return 1; });", + "const step_b = task(async () => { return 2; });", + "export const main = workflow(async () => {", + " const [a, b] = await Promise.all([step_a(), step_b()]);", + " return { a, b };", + "});", + ].join("\n"), + + wac_seq_3: [ + 'import { task, workflow } from "windmill-client";', + "const step_a = task(async () => { return 1; });", + "const step_b = task(async () => { return 2; });", + "const step_c = task(async () => { return 3; });", + "export const main = workflow(async () => {", + " const a = await step_a();", + " const b = await step_b();", + " const c = await step_c();", + " return { a, b, c };", + "});", + ].join("\n"), + + wac_inline_2: [ + 'import { step, workflow } from "windmill-client";', + "export const main = workflow(async () => {", + ' const a = await step("a", () => 1);', + ' const b = await step("b", () => 2);', + " return { a, b };", + "});", + ].join("\n"), +}; + +export const WAC_KINDS = Object.keys(WAC_SCRIPTS); + +// Number of child jobs created per workflow instance (used to compute throughput) +// For task(): each task creates a child job. For step(): no child job. +// Total completed jobs per workflow = nSteps + 1 (children + parent) +export const STEPS_PER_WORKFLOW: Record = { + wac_seq_2: 2, + wac_par_2: 2, + wac_seq_3: 3, + wac_inline_2: 0, // inline steps don't create child jobs + flow_seq_2_bun: 2, + flow_par_2_bun: 2, + flow_seq_3_bun: 3, +}; + +export async function createWacBenchScript( + wacPattern: string, + workspace: string, +) { + const scriptContent = WAC_SCRIPTS[wacPattern]; + if (!scriptContent) { + throw new Error("Unknown WAC pattern: " + wacPattern); + } + + const path = `f/benchmarks/${wacPattern}`; + const exists = await windmill.ScriptService.existsScriptByPath({ + workspace, + path, + }); + + if (exists) { + await windmill.ScriptService.deleteScriptByPath({ + workspace, + path, + }); + } + + const hash = await windmill.ScriptService.createScript({ + workspace, + requestBody: { + path, + content: scriptContent, + summary: wacPattern + " WAC v2 benchmark", + description: "", + language: "bun" as api.NewScript.language, + schema: { + $schema: "https://json-schema.org/draft/2020-12/schema", + properties: {}, + required: [], + type: "object", + }, + }, + }); + + await waitForDeployment(workspace, hash); + console.log("Created WAC v2 benchmark script at path", path); +} + +export async function loadJsonConfig(configPath: string): Promise { + if (configPath.startsWith("http")) { + const response = await fetch(configPath); + return await response.json(); + } else { + return JSON.parse(await Deno.readTextFile(configPath)); + } +} + export const getFlowPayload = (flowPattern: string): api.FlowPreview => { if (flowPattern == "branchone") { return { @@ -260,6 +373,113 @@ export const getFlowPayload = (flowPattern: string): api.FlowPreview => { ], }, }; + } else if (flowPattern == "flow_seq_2_bun") { + return { + path: "flow_seq_2_bun", + args: {}, + value: { + modules: [ + { + id: "a", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 1; }", + }, + }, + { + id: "b", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 2; }", + }, + }, + ], + }, + }; + } else if (flowPattern == "flow_par_2_bun") { + return { + path: "flow_par_2_bun", + args: {}, + value: { + modules: [ + { + id: "a", + value: { + type: "branchall", + parallel: true, + branches: [ + { + modules: [ + { + id: "b", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 1; }", + }, + }, + ], + }, + { + modules: [ + { + id: "c", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 2; }", + }, + }, + ], + }, + ], + }, + }, + ], + }, + }; + } else if (flowPattern == "flow_seq_3_bun") { + return { + path: "flow_seq_3_bun", + args: {}, + value: { + modules: [ + { + id: "a", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 1; }", + }, + }, + { + id: "b", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 2; }", + }, + }, + { + id: "c", + value: { + input_transforms: {}, + language: "bun" as api.RawScript.language, + type: "rawscript", + content: "export function main() { return 3; }", + }, + }, + ], + }, + }; } else { return { path: "2steps", diff --git a/benchmarks/main.ts b/benchmarks/main.ts index 4d8d144cd4..1f681a9c03 100644 --- a/benchmarks/main.ts +++ b/benchmarks/main.ts @@ -264,24 +264,6 @@ export async function main({ ); const shutdown_start = Date.now(); - // let zombie_jobs = 0; - // let incorrect_results = 0; - // workers.forEach((worker, i) => { - // const l = (evt: MessageEvent) => { - // if (evt.data.type === "zombie_jobs") { - // zombie_jobs += evt.data.zombie_jobs; - // incorrect_results += evt.data.incorrect_results; - // worker.removeEventListener("message", l); - // workers = workers.filter((w) => w != worker); - // jobsSent[i] = evt.data.jobs_sent; - // worker.terminate(); - // } - // }; - // worker.addEventListener("message", l); - // worker.postMessage( - // Number.isSafeInteger(zombieTimeout) ? zombieTimeout : 90000 - // ); - // }); workers.forEach((worker, i) => { const l = (evt: MessageEvent) => { if (evt.data.type === "done") { @@ -327,8 +309,6 @@ export async function main({ console.log("time (s + tts):", time); console.log("throughput /s (jobs/time):", sum / time); - // console.log("zombie jobs: ", zombie_jobs); - // console.log("incorrect results: ", incorrect_results); console.log( "queue length:", ( diff --git a/benchmarks/suite_wac.json b/benchmarks/suite_wac.json new file mode 100644 index 0000000000..e677952761 --- /dev/null +++ b/benchmarks/suite_wac.json @@ -0,0 +1,30 @@ +[ + { + "kind": "wac_seq_2", + "jobs": 250 + }, + { + "kind": "wac_par_2", + "jobs": 250 + }, + { + "kind": "wac_seq_3", + "jobs": 200 + }, + { + "kind": "wac_inline_2", + "jobs": 500 + }, + { + "kind": "flow_seq_2_bun", + "jobs": 250 + }, + { + "kind": "flow_par_2_bun", + "jobs": 250 + }, + { + "kind": "flow_seq_3_bun", + "jobs": 200 + } +] diff --git a/benchmarks/worker.ts b/benchmarks/worker.ts index cd56d0fb45..ae06fa9779 100644 --- a/benchmarks/worker.ts +++ b/benchmarks/worker.ts @@ -139,96 +139,6 @@ while (cont) { clearInterval(updateStatusInterval); -// const end_time = Date.now() + complete_timeout; - -// let incorrect_results = 0; -// const enc = (s: string) => new TextEncoder().encode(s); - -// let last_queue_length = await getQueueCount(); -// console.log(`waiting for ${last_queue_length} jobs to complete...`); - -// while ( -// outstanding.length > 0 && -// last_queue_length > 0 && -// Date.now() < end_time -// ) { -// try { -// if (!config.hideProgress) { -// await Deno.stdout.write( -// enc( -// "\rwaiting for jobs to complete: outstanding " + -// outstanding.length + -// " - queue" + -// last_queue_length + -// "\n" -// ) -// ); -// } -// last_queue_length = await getQueueCount(); - -// const uuid = outstanding.shift()!; - -// let r: Job; -// try { -// r = await windmill.JobService.getJob({ -// workspace: config.workspace_id, -// id: uuid, -// }); -// } catch (e) { -// console.log("job not found: " + uuid + " " + e.message); -// continue; -// } -// if (r.type == "QueuedJob") { -// outstanding.push(uuid); - -// if (!config.hideProgress) { -// await Deno.stdout.write( -// enc(`uuid: ${uuid}, queue length: ${last_queue_length}\r`) -// ); -// } -// } else { -// r = r as api.CompletedJob; -// try { -// if ( -// ![ -// "httpversion", -// "identity", -// "httpslow", -// "noop", -// "dedicated", -// ].includes(config.scriptPattern) && -// r.result != uuid -// ) { -// console.log( -// "job did not return correct UUID: " + -// r.result + -// " != " + -// uuid + -// "job: \n" + -// JSON.stringify(r, null, 2) -// ); -// incorrect_results++; -// } else { -// // console.log(r.result); -// } -// } catch (e) { -// console.log("error during wait: ", e); -// outstanding.push(uuid); -// } -// } -// } catch (e) { -// console.log("error while waiting for outstanding jobs, sleeing: ", e); -// await sleep(0.5); -// } -// } - -// self.postMessage({ -// type: "zombie_jobs", -// zombie_jobs: outstanding.length, -// incorrect_results, -// jobs_sent: total_spawned, -// }); - self.postMessage({ type: "done", jobs_sent: total_spawned, diff --git a/cli/TESTING.md b/cli/TESTING.md index 9928e75266..542baab368 100644 --- a/cli/TESTING.md +++ b/cli/TESTING.md @@ -3,57 +3,57 @@ ## Running Tests ```bash -# Run all tests -deno test -A --no-check test/ +# Run unit tests only (fast — no backend, no database, no cargo build) +bun run test:unit + +# Run all tests (unit + integration — requires PostgreSQL + cargo) +DATABASE_URL=postgres://postgres:changeme@localhost:5432 bun run test # Run specific test files -deno test -A --no-check test/gitsync_settings_features.test.ts -deno test -A --no-check test/init_no_git_sync.test.ts -deno test -A --no-check test/multi_instance_workspace.test.ts -deno test -A --no-check test/override_settings_behavior.test.ts -deno test -A --no-check test/sync_config_resolution.test.ts -deno test -A --no-check test/workspace_conflicts.test.ts - -# Run with specific test patterns -deno test -A --no-check test/ --filter "workspace" -deno test -A --no-check test/ --filter "sync" +bun test test/sync_pull_push.test.ts +bun test test/workspace_conflicts_unit.test.ts ``` -## Test Files +## Test Categories -- **`gitsync_settings_features.test.ts`** - Git sync settings functionality -- **`init_no_git_sync.test.ts`** - Init without git sync -- **`multi_instance_workspace.test.ts`** - Multi-instance workspace handling -- **`override_settings_behavior.test.ts`** - Settings override behavior -- **`sync_config_resolution.test.ts`** - Sync configuration resolution -- **`workspace_conflicts.test.ts`** - Workspace conflict detection +### Unit tests (`*_unit.test.ts`) -## Docker Requirements +Pure local tests — no backend, no database. Uses `bunfig.unit.toml` (no preload). + +Examples: `git_unit`, `lint_command_unit`, `tar_creation_unit`, `workspace_conflicts_unit` + +### Integration tests + +Require a running backend and PostgreSQL. The `setup.ts` preload builds the backend +binary and starts a shared backend instance. + +Examples: `sync_pull_push`, `dev_server`, `standalone_commands` + +## Environment Variables + +| Variable | Purpose | Default | +|----------|---------|---------| +| `DATABASE_URL` | PostgreSQL connection string (without database name) | `postgres://postgres:changeme@localhost:5432` | +| `TEST_BACKEND` | `cargo` or `docker` | `cargo` | +| `CI_MINIMAL_FEATURES` | `true` for CI mode (zip-only features) | unset | +| `EE_LICENSE_KEY` | Enterprise license for EE feature tests | unset | +| `TEST_FEATURES` | Additional cargo features (comma-separated) | unset | +| `TEST_CLI_RUNTIME` | `node` to test npm package | unset | +| `UNIT_ONLY` | `1` to skip backend setup in preload (used by `test:unit`) | unset | +| `VERBOSE` | `1` for backend process output | unset | + +## Cleanup + +Stale test databases (`windmill_test_*`) and orphaned backend processes from +previous crashed runs are automatically cleaned up when starting a new test run. + +To manually check for leftovers: ```bash -# Ensure Docker is running -docker --version -docker-compose --version +# Check for stale test databases +psql postgres://postgres:changeme@localhost:5432/postgres -c \ + "SELECT datname FROM pg_database WHERE datname LIKE 'windmill_test_%';" -# Ensure EE license key is available -echo $EE_LICENSE_KEY +# Check for orphaned backend processes +ps aux | grep "target/debug/windmill" | grep -v grep ``` - -## Debugging Failed Tests - -```bash -# Run with verbose output -deno test -A --no-check test/ --reporter=verbose - -# Check container status -docker ps - -# View backend logs -docker logs test-test_windmill_server-1 - -# Manual container management -cd test -docker compose -f docker-compose.test.yml up -d -docker compose -f docker-compose.test.yml down -docker compose -f docker-compose.test.yml down -v -``` \ No newline at end of file diff --git a/cli/bootstrap/flow_bootstrap.ts b/cli/bootstrap/flow_bootstrap.ts index 8bae373b17..3a71051505 100644 --- a/cli/bootstrap/flow_bootstrap.ts +++ b/cli/bootstrap/flow_bootstrap.ts @@ -13,7 +13,6 @@ export interface FlowDefinition { properties: { [name: string]: SchemaProperty}, required: string[] } - ws_error_handler_muted: false } export function defaultFlowDefinition(): FlowDefinition { @@ -30,6 +29,5 @@ export function defaultFlowDefinition(): FlowDefinition { properties: {}, required: [] }, - ws_error_handler_muted: false, } } diff --git a/cli/generate-schema.ts b/cli/generate-schema.ts new file mode 100644 index 0000000000..a5925968d6 --- /dev/null +++ b/cli/generate-schema.ts @@ -0,0 +1,16 @@ +#!/usr/bin/env npx tsx +/** + * Regenerate cli/wmill.schema.json from CONFIG_REFERENCE. + * + * Run after adding or modifying config options in src/commands/init/template.ts: + * npx tsx generate-schema.ts + */ +import { writeFileSync } from "node:fs"; +import { join, dirname } from "node:path"; +import { fileURLToPath } from "node:url"; +import { generateJsonSchema } from "./src/commands/init/template.ts"; + +const dir = dirname(fileURLToPath(import.meta.url)); +const out = join(dir, "wmill.schema.json"); +writeFileSync(out, JSON.stringify(generateJsonSchema(), null, 2) + "\n"); +console.log(`Wrote ${out}`); diff --git a/cli/package-lock.json b/cli/package-lock.json index 0e86b9d2b7..ae46c240c9 100644 --- a/cli/package-lock.json +++ b/cli/package-lock.json @@ -25,10 +25,11 @@ "windmill-parser-wasm-nu": "*", "windmill-parser-wasm-php": "*", "windmill-parser-wasm-py": "*", + "windmill-parser-wasm-py-imports": "*", "windmill-parser-wasm-regex": "*", "windmill-parser-wasm-ruby": "*", "windmill-parser-wasm-rust": "*", - "windmill-parser-wasm-ts": "*", + "windmill-parser-wasm-ts": "^1.659.1", "windmill-parser-wasm-yaml": "*", "windmill-yaml-validator": "1.1.1", "ws": "8.18.0", @@ -1414,6 +1415,11 @@ "resolved": "https://registry.npmjs.org/windmill-parser-wasm-py/-/windmill-parser-wasm-py-1.628.3.tgz", "integrity": "sha512-TlluqknZpg8cZ+A3m6JFLPseY2PpKtDsxdj26fAnCUzKPtse8TxQR+n0dwC80rfW5TwdWSulvNGRDgcNuf7CTw==" }, + "node_modules/windmill-parser-wasm-py-imports": { + "version": "1.659.1", + "resolved": "https://registry.npmjs.org/windmill-parser-wasm-py-imports/-/windmill-parser-wasm-py-imports-1.659.1.tgz", + "integrity": "sha512-nfnf04WBRf8f/mNIwdvggYOgz3erxrFGjKqULYBH+bKFMlKA6V7eB19m6CXOBkq9rjTp0ZFG+rgsR+Us7JEkyQ==" + }, "node_modules/windmill-parser-wasm-regex": { "version": "1.639.0", "resolved": "https://registry.npmjs.org/windmill-parser-wasm-regex/-/windmill-parser-wasm-regex-1.639.0.tgz", @@ -1430,9 +1436,9 @@ "integrity": "sha512-9yGLYZX2Hn9TdTqGY/5Fp50ftzgUsrfBkSK9vJkKJd5Amyg+yXLBGzd8pz6Org+4uxMenz/16wpsgijvo6uhhQ==" }, "node_modules/windmill-parser-wasm-ts": { - "version": "1.647.1", - "resolved": "https://registry.npmjs.org/windmill-parser-wasm-ts/-/windmill-parser-wasm-ts-1.647.1.tgz", - "integrity": "sha512-64iSAUMU5W/WtePqE1vtDvglDqtkiZVndyieYBVDX0nl7UuovS+wPgH/P3TEoKbR+FwAPacki0CX3DsEzZ/Yxw==" + "version": "1.659.1", + "resolved": "https://registry.npmjs.org/windmill-parser-wasm-ts/-/windmill-parser-wasm-ts-1.659.1.tgz", + "integrity": "sha512-EmXMzOmazC5r29UZh+1TVF9g/N2X51pqK11qDL6xWGeWTIIonhfOZ5nWdGvKQMDUR650fGxehImZzW2v9hNy+w==" }, "node_modules/windmill-parser-wasm-yaml": { "version": "1.593.0", diff --git a/cli/package.json b/cli/package.json index e44a215631..105915a720 100644 --- a/cli/package.json +++ b/cli/package.json @@ -9,6 +9,7 @@ "dev": "bun run src/main.ts", "build": "./build.sh", "test": "bun test test/", + "test:unit": "UNIT_ONLY=1 bun test test/*_unit*", "check": "bunx tsc --noEmit", "gen-client": "./gen_wm_client.sh && ./windmill-utils-internal/gen_wm_client.sh" }, diff --git a/cli/src/commands/app/app.ts b/cli/src/commands/app/app.ts index f048aac616..13851e5f8c 100644 --- a/cli/src/commands/app/app.ts +++ b/cli/src/commands/app/app.ts @@ -5,6 +5,7 @@ import { Table } from "@cliffy/table"; import { colors } from "@cliffy/ansi/colors"; import * as log from "../../core/log.ts"; import { sep as SEP } from "node:path"; +import { stat } from "node:fs/promises"; import * as windmillUtils from "@windmill-labs/shared-utils"; import { yamlParseFile } from "../../utils/yaml.ts"; import * as wmill from "../../../gen/services.gen.ts"; @@ -241,8 +242,26 @@ async function push(opts: GlobalOptions, filePath: string, remotePath: string) { const workspace = await resolveWorkspace(opts); await requireLogin(opts); - await pushApp(workspace.workspaceId, remotePath, filePath); - log.info(colors.bold.underline.green("App pushed")); + // Detect raw apps by checking for raw_app.yaml or __raw_app/.raw_app suffix + const normalizedPath = filePath.endsWith(SEP) ? filePath.slice(0, -1) : filePath; + const isRawApp = normalizedPath.endsWith("__raw_app") || normalizedPath.endsWith(".raw_app"); + let hasRawAppYaml = false; + if (!isRawApp) { + try { + const rawAppPath = (filePath.endsWith(SEP) ? filePath : filePath + SEP) + "raw_app.yaml"; + await stat(rawAppPath); + hasRawAppYaml = true; + } catch { /* not a raw app */ } + } + + if (isRawApp || hasRawAppYaml) { + const { pushRawApp } = await import("./raw_apps.ts"); + await pushRawApp(workspace.workspaceId, remotePath, filePath); + log.info(colors.bold.underline.green("Raw app pushed")); + } else { + await pushApp(workspace.workspaceId, remotePath, filePath); + log.info(colors.bold.underline.green("App pushed")); + } } const command = new Command() diff --git a/cli/src/commands/app/bundle.ts b/cli/src/commands/app/bundle.ts index 998546d364..8d15488491 100644 --- a/cli/src/commands/app/bundle.ts +++ b/cli/src/commands/app/bundle.ts @@ -166,8 +166,10 @@ export async function createBundle( // Dynamically import esbuild const esbuild = await import("esbuild"); - // Detect frameworks to determine default entry point - const frameworks = detectFrameworks(process.cwd()); + // Detect frameworks to determine default entry point. + // Use the entryPoint's directory if provided, otherwise fall back to cwd. + const appDir = options.entryPoint ? path.dirname(options.entryPoint) : process.cwd(); + const frameworks = detectFrameworks(appDir); const defaultEntry = (frameworks.svelte || frameworks.vue) ? "index.ts" : "index.tsx"; const entryPoint = options.entryPoint ?? defaultEntry; @@ -184,7 +186,6 @@ export async function createBundle( } // Ensure node_modules exists in the app directory - const appDir = path.dirname(entryPoint) || process.cwd(); await ensureNodeModules(appDir); // Load framework-specific plugins (svelte, vue) based on package.json diff --git a/cli/src/commands/app/lint.ts b/cli/src/commands/app/lint.ts index 12014cc7d5..9356b5dd3c 100644 --- a/cli/src/commands/app/lint.ts +++ b/cli/src/commands/app/lint.ts @@ -6,7 +6,7 @@ import { colors } from "@cliffy/ansi/colors"; import * as log from "../../core/log.ts"; import { yamlParseFile } from "../../utils/yaml.ts"; import { GlobalOptions } from "../../types.ts"; -import { createBundle } from "./bundle.ts"; +import { createBundle, detectFrameworks } from "./bundle.ts"; import { APP_BACKEND_FOLDER } from "./app_metadata.ts"; import { loadRunnablesFromBackend } from "./raw_apps.ts"; import { @@ -113,7 +113,11 @@ async function validateBuild( log.info(colors.blue("🔨 Testing build...")); // Try to create a bundle - this will validate that all dependencies are in place + const frameworks = detectFrameworks(appDir); + const entryFile = frameworks.svelte || frameworks.vue ? "index.ts" : "index.tsx"; + const entryPoint = path.join(appDir, entryFile); await createBundle({ + entryPoint, production: true, minify: false, }); diff --git a/cli/src/commands/audit/audit.ts b/cli/src/commands/audit/audit.ts new file mode 100644 index 0000000000..e0a85e89b6 --- /dev/null +++ b/cli/src/commands/audit/audit.ts @@ -0,0 +1,120 @@ +import { GlobalOptions } from "../../types.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { resolveWorkspace } from "../../core/context.ts"; +import { Command } from "@cliffy/command"; +import { Table } from "@cliffy/table"; +import { colors } from "@cliffy/ansi/colors"; +import * as log from "../../core/log.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; +import * as wmill from "../../../gen/services.gen.ts"; +import { formatTimestamp } from "../../utils/utils.ts"; + +async function list( + opts: GlobalOptions & { + json?: boolean; + username?: string; + operation?: string; + actionKind?: string; + before?: string; + after?: string; + limit?: number; + } +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const logs = await wmill.listAuditLogs({ + workspace: workspace.workspaceId, + username: opts.username, + operation: opts.operation, + actionKind: opts.actionKind as any, + before: opts.before, + after: opts.after, + perPage: opts.limit ?? 30, + }); + + if (opts.json) { + console.log(JSON.stringify(logs)); + } else { + if (logs.length === 0) { + log.info("No audit logs found."); + return; + } + if (logs.every((l) => l.operation === "redacted")) { + log.info(colors.yellow( + "Audit log details are not available on the Community Edition.\n" + + "Upgrade to the Enterprise Edition for full audit logging with operation details." + )); + return; + } + new Table() + .header(["ID", "Timestamp", "Username", "Operation", "Action", "Resource"]) + .padding(2) + .border(true) + .body( + logs.map((l) => [ + String(l.id), + formatTimestamp(l.timestamp), + l.username, + l.operation, + l.action_kind, + l.resource ?? "-", + ]) + ) + .render(); + } +} + +async function get( + opts: GlobalOptions & { json?: boolean }, + id: string +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const auditLog = await wmill.getAuditLog({ + workspace: workspace.workspaceId, + id: parseInt(id, 10), + }); + + if (opts.json) { + console.log(JSON.stringify(auditLog)); + } else { + console.log(colors.bold("ID:") + " " + auditLog.id); + console.log(colors.bold("Timestamp:") + " " + formatTimestamp(auditLog.timestamp)); + console.log(colors.bold("Username:") + " " + auditLog.username); + console.log(colors.bold("Operation:") + " " + auditLog.operation); + console.log(colors.bold("Action Kind:") + " " + auditLog.action_kind); + console.log(colors.bold("Resource:") + " " + (auditLog.resource ?? "-")); + if (auditLog.parameters && Object.keys(auditLog.parameters).length > 0) { + console.log(colors.bold("Parameters:")); + console.log(JSON.stringify(auditLog.parameters, null, 2)); + } + } +} + +const auditListOptions = (cmd: Command) => + cmd + .option("--json", "Output as JSON (for piping to jq)") + .option("--username ", "Filter by username") + .option("--operation ", "Filter by operation (exact or prefix)") + .option("--action-kind ", "Filter by action kind (Create, Update, Delete, Execute)") + .option("--before ", "Filter events before this timestamp") + .option("--after ", "Filter events after this timestamp") + .option("--limit ", "Number of entries to return (default 30, max 100)"); + +const command = auditListOptions(new Command() + .description("View audit logs (requires admin)")) + .action(list as any) + .command("list", auditListOptions(new Command().description("List audit log entries"))) + .action(list as any) + .command("get", "Get a specific audit log entry") + .arguments("") + .option("--json", "Output as JSON (for piping to jq)") + .action(get as any); + +export default command; diff --git a/cli/src/commands/config/config.ts b/cli/src/commands/config/config.ts new file mode 100644 index 0000000000..500e65a113 --- /dev/null +++ b/cli/src/commands/config/config.ts @@ -0,0 +1,26 @@ +import { Command } from "@cliffy/command"; +import * as log from "../../core/log.ts"; +import { + formatConfigReference, + formatConfigReferenceJson, +} from "../init/template.ts"; + +interface ConfigOptions { + json?: boolean; +} + +async function configAction(opts: ConfigOptions) { + if (opts.json) { + console.log(formatConfigReferenceJson()); + } else { + log.info(formatConfigReference()); + } +} + +const command = new Command() + .name("config") + .description("Show all available wmill.yaml configuration options") + .option("--json", "Output as JSON for programmatic consumption") + .action(configAction as any); + +export default command; diff --git a/cli/src/commands/dependencies/dependencies.ts b/cli/src/commands/dependencies/dependencies.ts index 9cbcdf86df..cde4fe256c 100644 --- a/cli/src/commands/dependencies/dependencies.ts +++ b/cli/src/commands/dependencies/dependencies.ts @@ -43,70 +43,66 @@ export async function pushWorkspaceDependencies( _befObj: any, newDependenciesContent: string, ): Promise { - try { - const res = workspaceDependenciesPathToLanguageAndFilename(path); - if (!res) { - throw new Error(`Unknown workspace dependencies file format: ${path}`); - } - - const { language, name } = res; - - const displayName = name - ? `named dependencies "${name}"` - : `workspace default dependencies`; - - // Fetch remote workspace dependencies and compare content directly - try { - const remoteDeps = await wmill.getLatestWorkspaceDependencies({ - workspace, - language, - name, - }); - - if (remoteDeps && remoteDeps.content === newDependenciesContent) { - log.info( - colors.green( - `${displayName} for ${language} are up-to-date, skipping push`, - ), - ); - return; - } - } catch (e: any) { - // If 404 or not found, the dependency doesn't exist remotely yet - proceed with push - if (e.status !== 404 && !e.message?.includes("not found")) { - throw e; - } - } - - log.info( - colors.yellow( - `Pushing ${ - name ? "named" : "workspace default" - } dependencies for ${language}...`, - ), + const res = workspaceDependenciesPathToLanguageAndFilename(path); + if (!res) { + throw new Error( + `Unknown workspace dependencies file format: ${path}. ` + + `Valid files: package.json, requirements.in, composer.json, go.mod, modules.json` ); + } - await wmill.createWorkspaceDependencies({ + const { language, name } = res; + + const displayName = name + ? `named dependencies "${name}"` + : `workspace default dependencies`; + + // Fetch remote workspace dependencies and compare content directly + try { + const remoteDeps = await wmill.getLatestWorkspaceDependencies({ workspace, - requestBody: { - name, - content: newDependenciesContent, - language, - workspace_id: workspace, - // Description is not supported in cli, it will use old description - description: undefined, - }, + language, + name, }); - log.info( - colors.green(`Successfully pushed ${displayName} for ${language}`), - ); - } catch (error: any) { - log.error( - colors.red(`Failed to push workspace dependencies: ${error.message}`), - ); - throw error; + if (remoteDeps && remoteDeps.content === newDependenciesContent) { + log.info( + colors.green( + `${displayName} for ${language} are up-to-date, skipping push`, + ), + ); + return; + } + } catch (e: any) { + // If 404 or not found, the dependency doesn't exist remotely yet - proceed with push + if (e.status !== 404 && !e.message?.includes("not found")) { + throw e; + } } + + log.info( + colors.yellow( + `Pushing ${ + name ? "named" : "workspace default" + } dependencies for ${language}...`, + ), + ); + + await wmill.createWorkspaceDependencies({ + workspace, + requestBody: { + name, + content: newDependenciesContent, + language, + workspace_id: workspace, + // Description is not supported in cli, it will use old description + description: undefined, + }, + }); + + log.info( + colors.green(`Successfully pushed ${displayName} for ${language}`), + ); } export default command; diff --git a/cli/src/commands/dev/dev.ts b/cli/src/commands/dev/dev.ts index 41b6b8fd9c..5352270f15 100644 --- a/cli/src/commands/dev/dev.ts +++ b/cli/src/commands/dev/dev.ts @@ -236,7 +236,7 @@ async function dev(opts: GlobalOptions & SyncOptions) { } const command = new Command() - .description("Launch a dev server that will spawn a webserver with HMR") + .description("Launch a dev server that watches for local file changes and auto-pushes them to the remote workspace. Provides live reload for scripts and flows during development.") .option( "--includes ", "Filter paths givena glob pattern or path" diff --git a/cli/src/commands/docs/docs.ts b/cli/src/commands/docs/docs.ts index 288faf335d..d86d4ca367 100644 --- a/cli/src/commands/docs/docs.ts +++ b/cli/src/commands/docs/docs.ts @@ -106,7 +106,7 @@ async function docs( const command = new Command() .name("docs") - .description("Search Windmill documentation. Requires Enterprise Edition.") + .description("Search Windmill documentation.") .arguments("") .option("--json", "Output results as JSON.") .action(docs as any); diff --git a/cli/src/commands/flow/flow.ts b/cli/src/commands/flow/flow.ts index 58d0777f90..2ddaaa36c7 100644 --- a/cli/src/commands/flow/flow.ts +++ b/cli/src/commands/flow/flow.ts @@ -7,9 +7,11 @@ import * as log from "../../core/log.ts"; import { sep as SEP } from "node:path"; import { stringify as yamlStringify } from "yaml"; import { yamlParseFile } from "../../utils/yaml.ts"; +import { validateRequiredArgs } from "../../utils/utils.ts"; import * as wmill from "../../../gen/services.gen.ts"; import { readFile } from "node:fs/promises"; import { mkdirSync, writeFileSync } from "node:fs"; +import { buildFolderPath, getMetadataFileName, loadNonDottedPathsSetting } from "../../utils/resource_folders.ts"; import { requireLogin } from "../../core/auth.ts"; import { resolveWorkspace, validatePath } from "../../core/context.ts"; @@ -152,18 +154,27 @@ export async function pushFlow( const localFlow = (await yamlParseFile(localPath + "flow.yaml")) as FlowFile; const fileReader = async (path: string) => await readFile(localPath + path, "utf-8"); + const missingFiles: string[] = []; await replaceInlineScripts( localFlow.value.modules, fileReader, log, localPath, - SEP + SEP, + undefined, + missingFiles ); if (localFlow.value.failure_module) { - await replaceInlineScripts([localFlow.value.failure_module], fileReader, log, localPath, SEP); + await replaceInlineScripts([localFlow.value.failure_module], fileReader, log, localPath, SEP, undefined, missingFiles); } if (localFlow.value.preprocessor_module) { - await replaceInlineScripts([localFlow.value.preprocessor_module], fileReader, log, localPath, SEP); + await replaceInlineScripts([localFlow.value.preprocessor_module], fileReader, log, localPath, SEP, undefined, missingFiles); + } + if (missingFiles.length > 0) { + log.warn(colors.yellow( + `Warning: missing inline script file(s): ${missingFiles.join(", ")}. ` + + `The flow will be pushed with unresolved !inline references.` + )); } if (flow) { @@ -203,20 +214,21 @@ export async function pushFlow( type Options = GlobalOptions; -async function push(opts: Options, filePath: string, remotePath: string) { +async function push(opts: Options & { message?: string }, filePath: string, remotePath: string) { if (!validatePath(remotePath)) { return; } const workspace = await resolveWorkspace(opts); await requireLogin(opts); - await pushFlow(workspace.workspaceId, remotePath, filePath); + await pushFlow(workspace.workspaceId, remotePath, filePath, opts.message); log.info(colors.bold.underline.green("Flow pushed")); } async function list( opts: GlobalOptions & { showArchived?: boolean; includeDraftOnly?: boolean; json?: boolean } ) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -250,6 +262,7 @@ async function list( } } async function get(opts: GlobalOptions & { json?: boolean }, path: string) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); const f = await wmill.getFlowByPath({ @@ -264,6 +277,31 @@ async function get(opts: GlobalOptions & { json?: boolean }, path: string) { console.log(colors.bold("Description:") + " " + (f.description ?? "")); console.log(colors.bold("Edited by:") + " " + (f.edited_by ?? "")); console.log(colors.bold("Edited at:") + " " + (f.edited_at ?? "")); + // API response type doesn't include flow value/modules — cast needed to access them + const modules = (f as any).value?.modules; + if (modules && Array.isArray(modules) && modules.length > 0) { + console.log(colors.bold("Steps:")); + function printModules(mods: any[], indent: string = " ") { + for (const mod of mods) { + const type = mod.value?.type ?? "unknown"; + const detail = mod.value?.language ?? mod.value?.path ?? ""; + console.log(`${indent}${mod.id}: ${type}${detail ? " (" + detail + ")" : ""}`); + if (type === "branchall" || type === "branchone") { + for (const branch of mod.value?.branches ?? []) { + console.log(`${indent} Branch: ${branch.summary || "(default)"}`); + if (branch.modules) printModules(branch.modules, indent + " "); + } + if (type === "branchone" && mod.value?.default) { + console.log(`${indent} Default:`); + printModules(mod.value.default, indent + " "); + } + } else if (type === "forloopflow" || type === "whileloopflow") { + if (mod.value?.modules) printModules(mod.value.modules, indent + " "); + } + } + } + printModules(modules); + } } } @@ -274,54 +312,177 @@ async function run( }, path: string ) { + if (opts.silent) { + log.setSilent(true); + } const workspace = await resolveWorkspace(opts); await requireLogin(opts); const input = opts.data ? await resolve(opts.data) : {}; + // Validate required args against schema when no data provided + if (!opts.data) { + try { + const flow = await wmill.getFlowByPath({ + workspace: workspace.workspaceId, + path, + }); + validateRequiredArgs(flow.schema as Record); + } catch (e: any) { + if (e.message?.startsWith("Missing required")) throw e; + log.warn(`Could not fetch schema to validate args: ${e.message}`); + } + } + const id = await wmill.runFlowByPath({ workspace: workspace.workspaceId, path, requestBody: input, }); + // Build step label map from raw_flow if available + const stepLabels = new Map(); + try { + const initialJob = await wmill.getJob({ + workspace: workspace.workspaceId, + id, + }); + const rawFlow = (initialJob as any).raw_flow; + if (rawFlow?.modules) { + for (const mod of rawFlow.modules) { + if (mod.id) { + const label = mod.summary ? `${mod.id}: ${mod.summary}` : mod.id; + stepLabels.set(mod.id, label); + } + } + } + } catch { + // Best-effort — fall back to module IDs + } + let i = 0; + let lastStatus = ""; while (true) { const jobInfo = await wmill.getJob({ workspace: workspace.workspaceId, id, }); - if (jobInfo.flow_status!.modules.length <= i) { + + // Check if flow has completed (success or failure) + const isCompleted = (jobInfo as any).type === "CompletedJob"; + const flowStatus = jobInfo.flow_status!; + + if (flowStatus.modules.length <= i) { break; } - const module = jobInfo.flow_status!.modules[i]; + const module = flowStatus.modules[i]; - if (module.job) { - if (!opts.silent) { - log.info("====== Job " + (i + 1) + " ======"); + // If a module has failed, track its job (to show error logs), then break + if (module.type === "Failure") { + if (module.job && !opts.silent) { + const label = stepLabels.get(module.id!) ?? `Step ${i + 1}`; + log.info("====== " + label + " ======"); await track_job(workspace.workspaceId, module.job); } + break; + } + + if (module.job) { + const label = stepLabels.get(module.id!) ?? `Step ${i + 1}`; + const isForLoop = (module as any).flow_jobs !== undefined; + + if (isForLoop) { + // For-loop: track iterations as they appear, re-polling until module completes + let trackedIterations = 0; + let forLoopFailed = false; + while (true) { + const refreshed = await wmill.getJob({ + workspace: workspace.workspaceId, + id, + }); + const refreshedModule = refreshed.flow_status!.modules[i]; + const flowJobs = ((refreshedModule as any).flow_jobs as string[] | undefined) ?? []; + + // Track any new iterations + while (trackedIterations < flowJobs.length) { + if (!opts.silent) { + log.info(`====== ${label} (iteration ${trackedIterations}) ======`); + await track_job(workspace.workspaceId, flowJobs[trackedIterations]); + } + trackedIterations++; + } + + if (refreshedModule.type === "Success" || refreshedModule.type === "Failure") { + forLoopFailed = refreshedModule.type === "Failure"; + break; + } + await new Promise((resolve) => setTimeout(resolve, 200)); + } + if (forLoopFailed) break; + } else { + if (!opts.silent) { + log.info("====== " + label + " ======"); + await track_job(workspace.workspaceId, module.job); + } + } } else { - if (!opts.silent) { - log.info(module.type); + // Module not started yet — deduplicate status messages + const status = String(module.type); + if (!opts.silent && status !== lastStatus) { + log.info(colors.dim(status)); + lastStatus = status; } await new Promise((resolve, _) => setTimeout(() => resolve(undefined), 100) ); + + // If flow already completed while we were waiting, break out + if (isCompleted) break; + continue; } + lastStatus = ""; i++; } - if (!opts.silent) { - log.info(colors.green.underline.bold("Flow ran to completion")); - log.info("\n"); + // Wait for flow completion with retry (handles race when --silent skips module tracking) + const MAX_RETRIES = 600; // ~60 seconds at 100ms intervals + let retries = 0; + while (retries < MAX_RETRIES) { + try { + const jobInfo = await wmill.getCompletedJob({ + workspace: workspace.workspaceId, + id, + }); + + if (!opts.silent) { + if (jobInfo.success === false) { + log.info(colors.red.underline.bold("Flow failed")); + } else { + log.info(colors.green.underline.bold("Flow ran to completion")); + } + log.info("\n"); + } + + if (jobInfo.success === false) { + process.exitCode = 1; + } + + if (opts.silent) { + console.log(JSON.stringify(jobInfo.result ?? {})); + } else { + log.info(JSON.stringify(jobInfo.result ?? {}, null, 2)); + } + + break; + } catch { + retries++; + await new Promise((resolve) => setTimeout(resolve, 100)); + } + } + if (retries >= MAX_RETRIES) { + throw new Error(`Timed out waiting for flow ${id} to complete`); } - const jobInfo = await wmill.getCompletedJob({ - workspace: workspace.workspaceId, - id, - }); - log.info(JSON.stringify(jobInfo.result ?? {}, null, 2)); } async function preview( @@ -332,6 +493,9 @@ async function preview( } & SyncOptions, flowPath: string ) { + if (opts.silent) { + log.setSilent(true); + } const useLocalPathScripts = !opts.remote; if (useLocalPathScripts) { opts = await mergeConfigWithConfigFile(opts); @@ -340,14 +504,16 @@ async function preview( await requireLogin(opts); const codebases = useLocalPathScripts ? listSyncCodebases(opts) : []; - // Normalize path - ensure it's a directory path to a .flow folder - if (!flowPath.endsWith(".flow") && !flowPath.endsWith(".flow" + SEP)) { + // Normalize path - ensure it's a directory path to a .flow or __flow folder + const isFlowDir = flowPath.endsWith(".flow") || flowPath.endsWith(".flow" + SEP) + || flowPath.endsWith("__flow") || flowPath.endsWith("__flow" + SEP); + if (!isFlowDir) { // Check if it's a flow.yaml file if (flowPath.endsWith("flow.yaml") || flowPath.endsWith("flow.json")) { flowPath = flowPath.substring(0, flowPath.lastIndexOf(SEP)); } else { throw new Error( - "Flow path must be a .flow directory or a flow.yaml file" + "Flow path must be a .flow/__flow directory or a flow.yaml file" ); } } @@ -421,13 +587,23 @@ async function preview( }); } catch (e: any) { if (e.body) { - log.error(`Flow preview failed: ${JSON.stringify(e.body)}`); + // If a failure_module ran, the body contains its result — not an error + if (e.body.result !== undefined) { + if (opts.silent) { + console.log(JSON.stringify(e.body.result)); + } else { + log.info(colors.yellow.bold("Flow failed, error handler result:")); + log.info(JSON.stringify(e.body.result, null, 2)); + } + process.exitCode = 1; + return; + } } throw e; } if (opts.silent) { - console.log(JSON.stringify(result, null, 2)); + console.log(JSON.stringify(result)); } else { log.info(colors.bold.underline.green("Flow preview completed")); log.info(JSON.stringify(result, null, 2)); @@ -516,7 +692,7 @@ export async function generateLocks( } } -export function bootstrap( +export async function bootstrap( opts: GlobalOptions & { summary: string; description: string }, flowPath: string ) { @@ -524,8 +700,10 @@ export function bootstrap( return; } - const flowDirFullPath = `${flowPath}.flow`; - mkdirSync(flowDirFullPath, { recursive: false }); + await loadNonDottedPathsSetting(); + + const flowDirFullPath = buildFolderPath(flowPath, "flow"); + mkdirSync(flowDirFullPath, { recursive: true }); const newFlowDefinition = defaultFlowDefinition(); if (opts.summary !== undefined) { @@ -539,10 +717,76 @@ export function bootstrap( newFlowDefinition as Record ); - const flowYamlPath = `${flowDirFullPath}/flow.yaml`; + const metadataFile = getMetadataFileName("flow", "yaml"); + const flowYamlPath = `${flowDirFullPath}/${metadataFile}`; writeFileSync(flowYamlPath, newFlowDefinitionYaml, { flag: "wx", encoding: "utf-8" }); } +async function history( + opts: GlobalOptions & { json?: boolean }, + flowPath: string +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const versions = await wmill.getFlowHistory({ + workspace: workspace.workspaceId, + path: flowPath, + }); + + if (opts.json) { + console.log(JSON.stringify(versions)); + } else { + if (versions.length === 0) { + log.info("No version history found for " + flowPath); + return; + } + new Table() + .header(["Version", "Created At", "Deployment Message"]) + .padding(2) + .border(true) + .body( + versions.map((v) => [ + String(v.id), + new Date(v.created_at).toISOString().replace("T", " ").substring(0, 19), + v.deployment_msg ?? "-", + ]) + ) + .render(); + } +} + +async function showVersion( + opts: GlobalOptions & { json?: boolean }, + flowPath: string, + version: string +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const flow = await wmill.getFlowVersion({ + workspace: workspace.workspaceId, + path: flowPath, + version: parseInt(version, 10), + }); + + if (opts.json) { + console.log(JSON.stringify(flow)); + } else { + console.log(colors.bold("Path:") + " " + flow.path); + console.log(colors.bold("Summary:") + " " + (flow.summary ?? "-")); + console.log(colors.bold("Description:") + " " + (flow.description ?? "-")); + console.log(colors.bold("Schema:")); + console.log(JSON.stringify(flow.schema, null, 2)); + console.log(colors.bold("Value:")); + console.log(JSON.stringify(flow.value, null, 2)); + } +} + const command = new Command() .description("flow related commands") .option("--show-archived", "Enable archived flows in output") @@ -561,6 +805,7 @@ const command = new Command() "push a local flow spec. This overrides any remote versions." ) .arguments(" ") + .option("--message ", "Deployment message") .action(push as any) .command("run", "run a flow by path.") .arguments("") @@ -616,6 +861,14 @@ const command = new Command() .arguments("") .option("--summary ", "flow summary") .option("--description ", "flow description") - .action(bootstrap as any); + .action(bootstrap as any) + .command("history", "Show version history for a flow") + .arguments("") + .option("--json", "Output as JSON (for piping to jq)") + .action(history as any) + .command("show-version", "Show a specific version of a flow") + .arguments(" ") + .option("--json", "Output as JSON (for piping to jq)") + .action(showVersion as any); export default command; diff --git a/cli/src/commands/flow/flow_metadata.ts b/cli/src/commands/flow/flow_metadata.ts index 1f5d86b8ea..57a9289fa1 100644 --- a/cli/src/commands/flow/flow_metadata.ts +++ b/cli/src/commands/flow/flow_metadata.ts @@ -18,12 +18,12 @@ import { filterWorkspaceDependenciesForScripts, } from "../../utils/metadata.ts"; import { ScriptLanguage } from "../../utils/script_common.ts"; -import { extractInlineScripts as extractInlineScriptsForFlows } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; +import { extractInlineScripts as extractInlineScriptsForFlows, extractCurrentMapping } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; import { newPathAssigner } from "../../../windmill-utils-internal/src/path-utils/path-assigner.ts"; import { generateHash, getHeaders, writeIfChanged } from "../../utils/utils.ts"; import { exts } from "../script/script.ts"; -import { FSFSElement } from "../sync/sync.ts"; +import { FSFSElement, yamlOptions } from "../sync/sync.ts"; import { Workspace } from "../workspace/workspace.ts"; import { FlowFile } from "./flow.ts"; import { FlowValue } from "../../../gen/types.gen.ts"; @@ -188,6 +188,17 @@ export async function generateFlowLockInternal( log.info(`Recomputing locks of ${changedScripts.join(", ")} in ${folder}`); } const fileReader = async (path: string) => await readFile(folder + SEP + path, "utf-8"); + + // Capture existing module-ID-to-file-path mapping before replaceInlineScripts + // overwrites the !inline references with actual file content. This preserves + // the original filenames when re-extracting inline scripts after lock generation. + const currentMapping = extractCurrentMapping( + flowValue.value.modules, + {}, + flowValue.value.failure_module, + flowValue.value.preprocessor_module, + ); + // In tree mode, use the tree's staleness info (which includes transitive dependency changes) // to determine which scripts need relocking, instead of only content-changed ones. const locksToRemove = (tree && !legacyBehaviour) @@ -215,6 +226,12 @@ export async function generateFlowLockInternal( //removeChangedLocks const tempScriptRefs = tree?.getTempScriptRefs(folderNormalized); + + // Preserve notes and groups — the backend round-trips through FlowValue + // which doesn't include these fields, so they'd be lost (#8641). + const savedNotes = flowValue.value.notes; + const savedGroups = flowValue.value.groups; + flowValue.value = await updateFlow( workspace, flowValue.value, @@ -223,21 +240,25 @@ export async function generateFlowLockInternal( tempScriptRefs ); + // Restore notes and groups that the backend stripped + if (savedNotes !== undefined) flowValue.value.notes = savedNotes; + if (savedGroups !== undefined) flowValue.value.groups = savedGroups; + const lockAssigner = newPathAssigner(opts.defaultTs ?? "bun", { skipInlineScriptSuffix: getNonDottedPaths(), }); const inlineScripts = extractInlineScriptsForFlows( flowValue.value.modules, - {}, + currentMapping, SEP, opts.defaultTs, lockAssigner ); if (flowValue.value.failure_module) { - inlineScripts.push(...extractInlineScriptsForFlows([flowValue.value.failure_module], {}, SEP, opts.defaultTs, lockAssigner)); + inlineScripts.push(...extractInlineScriptsForFlows([flowValue.value.failure_module], currentMapping, SEP, opts.defaultTs, lockAssigner)); } if (flowValue.value.preprocessor_module) { - inlineScripts.push(...extractInlineScriptsForFlows([flowValue.value.preprocessor_module], {}, SEP, opts.defaultTs, lockAssigner)); + inlineScripts.push(...extractInlineScriptsForFlows([flowValue.value.preprocessor_module], currentMapping, SEP, opts.defaultTs, lockAssigner)); } inlineScripts.forEach((s) => { writeIfChanged(process.cwd() + SEP + folder + SEP + s.path, s.content); @@ -246,7 +267,7 @@ export async function generateFlowLockInternal( // Overwrite `flow.yaml` with the new lockfile references writeIfChanged( process.cwd() + SEP + folder + SEP + "flow.yaml", - yamlStringify(flowValue as Record) + yamlStringify(flowValue as Record, yamlOptions) ); } diff --git a/cli/src/commands/folder/folder.ts b/cli/src/commands/folder/folder.ts index eb5a51c4f8..bd298e6f88 100644 --- a/cli/src/commands/folder/folder.ts +++ b/cli/src/commands/folder/folder.ts @@ -22,6 +22,7 @@ export interface FolderFile { } async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); diff --git a/cli/src/commands/generate-metadata/generate-metadata.ts b/cli/src/commands/generate-metadata/generate-metadata.ts index d273f4b631..7abd155301 100644 --- a/cli/src/commands/generate-metadata/generate-metadata.ts +++ b/cli/src/commands/generate-metadata/generate-metadata.ts @@ -355,71 +355,102 @@ async function generateMetadata( return colors.dim(colors.white(`[${n}/${total}]`.padEnd(maxWidth, " "))); }; + const errors: { path: string; error: string }[] = []; + // Process scripts for (const item of scripts) { current++; log.info(`${formatProgress(current)} script ${item.path}`); - await generateScriptMetadataInternal( - item.path, // originalPath with extension - workspace, - opts, - false, // dryRun - true, // noStaleMessage - mismatchedWorkspaceDeps, - codebases, - false, - false, // legacyBehaviour - tree - ); + try { + await generateScriptMetadataInternal( + item.path, // originalPath with extension + workspace, + opts, + false, // dryRun + true, // noStaleMessage + mismatchedWorkspaceDeps, + codebases, + false, + false, // legacyBehaviour + tree + ); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + errors.push({ path: item.path, error: msg }); + log.error(` Failed: ${msg}`); + } } // Process flows for (const item of flows) { current++; - const result = await generateFlowLockInternal( - item.folder.replaceAll("/", SEP), - false, // dryRun - workspace, - opts, - false, - true, // noStaleMessage - false, // legacyBehaviour - tree - ); - const flowResult = result as FlowLocksResult | undefined; - const scriptsInfo = flowResult?.updatedScripts?.length - ? colors.dim(colors.white(`: ${flowResult.updatedScripts.join(", ")}`)) - : ""; - log.info(`${formatProgress(current)} flow ${item.path}${scriptsInfo}`); + try { + const result = await generateFlowLockInternal( + item.folder.replaceAll("/", SEP), + false, // dryRun + workspace, + opts, + false, + true, // noStaleMessage + false, // legacyBehaviour + tree + ); + const flowResult = result as FlowLocksResult | undefined; + const scriptsInfo = flowResult?.updatedScripts?.length + ? colors.dim(colors.white(`: ${flowResult.updatedScripts.join(", ")}`)) + : ""; + log.info(`${formatProgress(current)} flow ${item.path}${scriptsInfo}`); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + errors.push({ path: item.path, error: msg }); + log.info(`${formatProgress(current)} flow ${item.path}`); + log.error(` Failed: ${msg}`); + } } // Process apps for (const item of apps) { current++; - const result = await generateAppLocksInternal( - item.folder.replaceAll("/", SEP), - item.isRawApp!, // rawApp - false, // dryRun - workspace, - opts, - false, - true, // noStaleMessage - false, // legacyBehaviour - tree - ); - const appResult = result as AppLocksResult | undefined; - const scriptsInfo = appResult?.updatedScripts?.length - ? colors.dim(colors.white(`: ${appResult.updatedScripts.join(", ")}`)) - : ""; - log.info(`${formatProgress(current)} app ${item.path}${scriptsInfo}`); + try { + const result = await generateAppLocksInternal( + item.folder.replaceAll("/", SEP), + item.isRawApp!, // rawApp + false, // dryRun + workspace, + opts, + false, + true, // noStaleMessage + false, // legacyBehaviour + tree + ); + const appResult = result as AppLocksResult | undefined; + const scriptsInfo = appResult?.updatedScripts?.length + ? colors.dim(colors.white(`: ${appResult.updatedScripts.join(", ")}`)) + : ""; + log.info(`${formatProgress(current)} app ${item.path}${scriptsInfo}`); + } catch (e) { + const msg = e instanceof Error ? e.message : String(e); + errors.push({ path: item.path, error: msg }); + log.info(`${formatProgress(current)} app ${item.path}`); + log.error(` Failed: ${msg}`); + } } // Persist all stale workspace dep hashes (not just filtered — deps are global, not folder-scoped) const allStaleDeps = staleItems.filter((i) => i.type === "dependencies"); await tree.persistDepsHashes(allStaleDeps.map((d) => d.path)); + const succeeded = total - errors.length; log.info(""); - log.info(`Done. Updated ${colors.bold(String(total))} item(s).`); + if (errors.length > 0) { + log.info(`Done. Updated ${colors.bold(String(succeeded))}/${total} item(s). ${colors.red(String(errors.length) + " failed")}:`); + for (const { path, error } of errors) { + log.error(` ${path}: ${error}`); + } + process.exitCode = 1; + } else { + log.info(`Done. Updated ${colors.bold(String(total))} item(s).`); + } } const command = new Command() diff --git a/cli/src/commands/group/group.ts b/cli/src/commands/group/group.ts new file mode 100644 index 0000000000..5d06a7df1f --- /dev/null +++ b/cli/src/commands/group/group.ts @@ -0,0 +1,157 @@ +import { GlobalOptions } from "../../types.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { resolveWorkspace } from "../../core/context.ts"; +import { Command } from "@cliffy/command"; +import { Table } from "@cliffy/table"; +import { colors } from "@cliffy/ansi/colors"; +import * as log from "../../core/log.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; +import * as wmill from "../../../gen/services.gen.ts"; + +async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const groups = await wmill.listGroups({ + workspace: workspace.workspaceId, + }); + + if (opts.json) { + console.log(JSON.stringify(groups)); + } else { + if (groups.length === 0) { + log.info("No groups found."); + return; + } + new Table() + .header(["Name", "Summary", "Members"]) + .padding(2) + .border(true) + .body( + groups.map((g) => [ + g.name, + g.summary ?? "-", + String(g.members?.length ?? 0), + ]) + ) + .render(); + } +} + +async function get( + opts: GlobalOptions & { json?: boolean }, + name: string +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const group = await wmill.getGroup({ + workspace: workspace.workspaceId, + name, + }); + + if (opts.json) { + console.log(JSON.stringify(group)); + } else { + console.log(colors.bold("Name:") + " " + group.name); + console.log(colors.bold("Summary:") + " " + (group.summary ?? "-")); + console.log( + colors.bold("Members:") + + " " + + (group.members && group.members.length > 0 + ? group.members.join(", ") + : "(none)") + ); + } +} + +async function create( + opts: GlobalOptions & { summary?: string }, + name: string +) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.createGroup({ + workspace: workspace.workspaceId, + requestBody: { + name, + summary: opts.summary, + }, + }); + + log.info(colors.green(`Group '${name}' created.`)); +} + +async function deleteGroup(opts: GlobalOptions, name: string) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.deleteGroup({ + workspace: workspace.workspaceId, + name, + }); + + log.info(colors.green(`Group '${name}' deleted.`)); +} + +async function addUser(opts: GlobalOptions, name: string, username: string) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.addUserToGroup({ + workspace: workspace.workspaceId, + name, + requestBody: { username }, + }); + + log.info(colors.green(`User '${username}' added to group '${name}'.`)); +} + +async function removeUser(opts: GlobalOptions, name: string, username: string) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.removeUserToGroup({ + workspace: workspace.workspaceId, + name, + requestBody: { username }, + }); + + log.info(colors.green(`User '${username}' removed from group '${name}'.`)); +} + +const command = new Command() + .description("Manage workspace groups") + .option("--json", "Output as JSON (for piping to jq)") + .action(list as any) + .command("list", "List all groups in the workspace") + .option("--json", "Output as JSON (for piping to jq)") + .action(list as any) + .command("get", "Get group details and members") + .arguments("") + .option("--json", "Output as JSON (for piping to jq)") + .action(get as any) + .command("create", "Create a new group") + .arguments("") + .option("--summary ", "Group summary/description") + .action(create as any) + .command("delete", "Delete a group") + .arguments("") + .action(deleteGroup as any) + .command("add-user", "Add a user to a group") + .arguments(" ") + .action(addUser as any) + .command("remove-user", "Remove a user from a group") + .arguments(" ") + .action(removeUser as any); + +export default command; diff --git a/cli/src/commands/init/init.ts b/cli/src/commands/init/init.ts index 5883967b77..db4950575b 100644 --- a/cli/src/commands/init/init.ts +++ b/cli/src/commands/init/init.ts @@ -3,13 +3,14 @@ import { colors } from "@cliffy/ansi/colors"; import { Command } from "@cliffy/command"; import { Confirm } from "@cliffy/prompt/confirm"; import * as log from "../../core/log.ts"; -import { stringify as yamlStringify } from "yaml"; +import { type BranchBinding } from "./template.ts"; import { GlobalOptions } from "../../types.ts"; import { readLockfile } from "../../utils/metadata.ts"; import { getActiveWorkspaceOrFallback } from "../workspace/workspace.ts"; import { generateRTNamespace } from "../resource-type/resource-type.ts"; import { SKILLS, SKILL_CONTENT, SCHEMAS, SCHEMA_MAPPINGS } from "../../guidance/skills.ts"; import { generateAgentsMdContent } from "../../guidance/core.ts"; +import { generateCommentedTemplate } from "./template.ts"; /** * Format a YAML schema for inclusion in skill markdown files. @@ -42,61 +43,37 @@ export interface InitOptions { */ async function initAction(opts: InitOptions) { if (await stat("wmill.yaml").catch(() => null)) { - log.error(colors.red("wmill.yaml already exists")); + log.info("wmill.yaml already exists, skipping config generation"); } else { - // Import DEFAULT_SYNC_OPTIONS from conf.ts - const { DEFAULT_SYNC_OPTIONS } = await import("../../core/conf.ts"); - - // Create initial config with defaults - const initialConfig = { ...DEFAULT_SYNC_OPTIONS } as any; - - // Add branch structure + // Detect current git branch for template const { isGitRepository, getCurrentGitBranch } = await import( "../../utils/git.ts" ); + let branchName: string | undefined; + let binding: BranchBinding | undefined; if (isGitRepository()) { - const currentBranch = getCurrentGitBranch(); - if (currentBranch) { - initialConfig.gitBranches = { - [currentBranch]: { overrides: {} }, - }; - } else { - initialConfig.gitBranches = {}; - } - } else { - initialConfig.gitBranches = {}; + branchName = getCurrentGitBranch() ?? undefined; } - initialConfig.nonDottedPaths = true; - await writeFile("wmill.yaml", yamlStringify(initialConfig), "utf-8"); - log.info(colors.green("wmill.yaml created with default settings")); - - // Create lock file - await readLockfile(); - - // Offer to bind workspace profile to current branch - if (isGitRepository()) { + // Determine workspace binding before writing the template + if (isGitRepository() && branchName) { const activeWorkspace = await getActiveWorkspaceOrFallback( opts as GlobalOptions ); - const currentBranch = getCurrentGitBranch(); - if (activeWorkspace && currentBranch) { - // Determine binding behavior based on flags + if (activeWorkspace) { const shouldBind = opts.bindProfile === true; const shouldPrompt = opts.bindProfile === undefined && !!process.stdin.isTTY && !opts.useDefault; - const shouldSkip = opts.bindProfile != true && - (opts.useDefault || !!!process.stdin.isTTY); + (opts.useDefault || !process.stdin.isTTY); if (!shouldSkip) { - // Show workspace info if we're binding or prompting if (shouldBind || shouldPrompt) { log.info( - colors.yellow(`\nCurrent Git branch: ${colors.bold(currentBranch)}`) + colors.yellow(`\nCurrent Git branch: ${colors.bold(branchName)}`) ); log.info( colors.yellow( @@ -118,37 +95,31 @@ async function initAction(opts: InitOptions) { default: true, }))) ) { - // Update the config with workspace binding - const currentConfig = await import("../../core/conf.ts").then((m) => - m.readConfigFile() - ); - if (!currentConfig.gitBranches) { - currentConfig.gitBranches = {}; - } - if (!currentConfig.gitBranches[currentBranch]) { - currentConfig.gitBranches[currentBranch] = { overrides: {} }; - } - log.info( - `binding branch ${currentBranch} to workspace ${activeWorkspace.name} on ${activeWorkspace.remote}` - ); - currentConfig.gitBranches[currentBranch].baseUrl = - activeWorkspace.remote; - currentConfig.gitBranches[currentBranch].workspaceId = - activeWorkspace.workspaceId; - - await writeFile("wmill.yaml", yamlStringify(currentConfig), "utf-8"); - - log.info( - colors.green( - `✓ Bound branch '${currentBranch}' to workspace '${activeWorkspace.name}'` - ) + `binding branch ${branchName} to workspace ${activeWorkspace.name} on ${activeWorkspace.remote}` ); + binding = { + baseUrl: activeWorkspace.remote, + workspaceId: activeWorkspace.workspaceId, + }; } } } } + await writeFile("wmill.yaml", generateCommentedTemplate(branchName, binding), "utf-8"); + log.info(colors.green("wmill.yaml created with default settings")); + if (binding) { + log.info( + colors.green( + `✓ Bound branch '${branchName}' to workspace` + ) + ); + } + + // Create lock file + await readLockfile(); + // Check for backend git-sync settings unless --use-default is specified if (!opts.useDefault) { try { diff --git a/cli/src/commands/init/template.ts b/cli/src/commands/init/template.ts new file mode 100644 index 0000000000..0684b7ca14 --- /dev/null +++ b/cli/src/commands/init/template.ts @@ -0,0 +1,395 @@ +/** + * Configuration option descriptor — each entry IS a JSON Schema property + * with extra metadata for template rendering and reference table display. + * + * To generate the JSON Schema: iterate entries, strip NON_SCHEMA_KEYS, done. + * Sub-fields of complex types (codebases items, gitBranches branch config) + * are defined inline in the parent's schema — no duplicate entries needed. + * The reference table auto-expands nested schemas into rows. + * + * Adding a new option: + * 1. Add an entry to CONFIG_REFERENCE with JSON Schema type fields + description + * 2. Add template rendering hints (section, commented, templateValue, etc.) + * 3. `wmill init` (YAML template), `wmill config` (table), and wmill.schema.json all update automatically + */ +export interface ConfigOption { + // --- JSON Schema fields (kept when generating schema) --- + type: string; + description: string; + enum?: string[]; + items?: Record; + properties?: Record; + additionalProperties?: Record | boolean; + required?: string[]; + + // --- Non-schema metadata (stripped when generating schema) --- + name: string; + default: string; + + // --- Template rendering hints (also stripped) --- + section?: string; + sectionNote?: string; + commented?: boolean; + templateValue?: string; + example?: string; + inlineComment?: string; + groupNote?: string; +} + +/** Keys to strip from ConfigOption entries when generating JSON Schema. */ +const NON_SCHEMA_KEYS = new Set([ + "name", "default", + "section", "sectionNote", "commented", "templateValue", + "example", "inlineComment", "groupNote", +]); + +// Reusable sub-schemas for nested types +const SPECIFIC_ITEMS_SCHEMA = { + type: "object", + description: "Sync only specific items", + properties: { + variables: { type: "array", items: { type: "string" }, description: "Specific variable paths to sync" }, + resources: { type: "array", items: { type: "string" }, description: "Specific resource paths to sync" }, + triggers: { type: "array", items: { type: "string" }, description: "Specific trigger paths to sync" }, + folders: { type: "array", items: { type: "string" }, description: "Specific folder paths to sync" }, + settings: { type: "boolean", description: "Whether to sync settings" }, + }, + additionalProperties: false, +} as const; + +const BRANCH_CONFIG_SCHEMA = { + type: "object", + properties: { + baseUrl: { type: "string", description: "Windmill instance URL for this branch" }, + workspaceId: { type: "string", description: "Workspace ID to sync with for this branch" }, + overrides: { type: "object", description: "Override any top-level sync option for this branch" }, + promotionOverrides: { type: "object", description: "Overrides applied when using --promotion flag" }, + specificItems: SPECIFIC_ITEMS_SCHEMA, + }, + additionalProperties: false, +} as const; + +/** + * All wmill.yaml configuration options — single source of truth. + * Each entry is a JSON Schema property with extra metadata. + */ +export const CONFIG_REFERENCE: ConfigOption[] = [ + // ── Core ────────────────────────────────────────────────────────────── + { name: "defaultTs", type: "string", enum: ["bun", "deno"], default: "bun", description: "Default TypeScript runtime for new scripts" }, + { name: "includes", type: "array", items: { type: "string" }, default: '["f/**"]', description: "Glob patterns for files to include in sync", + templateValue: '\n - "f/**"' }, + { name: "extraIncludes", type: "array", items: { type: "string" }, default: "[]", description: "Additional glob patterns merged with includes (useful in branch overrides)", + commented: true }, + { name: "excludes", type: "array", items: { type: "string" }, default: "[]", description: "Glob patterns for files to exclude from sync" }, + + // ── What to sync ────────────────────────────────────────────────────── + { name: "skipVariables", type: "boolean", default: "false", description: "Skip syncing variables", + section: "What to sync", sectionNote: '"skip" options default to false (synced), "include" options default to false (not synced)' }, + { name: "skipResources", type: "boolean", default: "false", description: "Skip syncing resources" }, + { name: "skipResourceTypes", type: "boolean", default: "false", description: "Skip syncing resource types" }, + { name: "skipSecrets", type: "boolean", default: "true", description: "Skip syncing secrets (true by default for security)", + inlineComment: "true by default — secrets are not synced for security" }, + { name: "skipScripts", type: "boolean", default: "false", description: "Skip syncing scripts" }, + { name: "skipFlows", type: "boolean", default: "false", description: "Skip syncing flows" }, + { name: "skipApps", type: "boolean", default: "false", description: "Skip syncing apps" }, + { name: "skipFolders", type: "boolean", default: "false", description: "Skip syncing folders" }, + { name: "skipWorkspaceDependencies", type: "boolean", default: "false", description: "Skip syncing workspace dependencies" }, + + { name: "includeSchedules", type: "boolean", default: "false", description: "Include schedules in sync", + commented: true, templateValue: "true", groupNote: "Uncomment to include these (excluded by default):" }, + { name: "includeTriggers", type: "boolean", default: "false", description: "Include triggers (http, websocket, kafka, etc.) in sync", + commented: true, templateValue: "true" }, + { name: "includeUsers", type: "boolean", default: "false", description: "Include workspace users in sync", + commented: true, templateValue: "true" }, + { name: "includeGroups", type: "boolean", default: "false", description: "Include workspace groups in sync", + commented: true, templateValue: "true" }, + { name: "includeSettings", type: "boolean", default: "false", description: "Include workspace settings in sync", + commented: true, templateValue: "true" }, + { name: "includeKey", type: "boolean", default: "false", description: "Include encryption key in sync", + commented: true, templateValue: "true" }, + + // ── Sync behavior ───────────────────────────────────────────────────── + { name: "parallel", type: "integer", default: "(unset)", description: "Number of parallel operations during sync", + section: "Sync behavior", commented: true, templateValue: "4" }, + { name: "locksRequired", type: "boolean", default: "false", description: "Require lock files for all scripts", + commented: true, templateValue: "true" }, + { name: "lint", type: "boolean", default: "false", description: "Run linting before push", + commented: true, templateValue: "true" }, + { name: "plainSecrets", type: "boolean", default: "false", description: "Handle secrets as plain text (not recommended)", + commented: true }, + { name: "message", type: "string", default: "(unset)", description: "Default commit message for sync operations", + commented: true, templateValue: '"my commit message"' }, + { name: "promotion", type: "string", default: "(unset)", description: "Branch name to use promotion overrides from during sync", + commented: true, templateValue: "staging" }, + { name: "skipBranchValidation", type: "boolean", default: "false", description: "Skip validation that current git branch matches a configured branch", + commented: true }, + { name: "nonDottedPaths", type: "boolean", default: "true", description: "Use __flow/__app/__raw_app suffixes instead of .flow/.app/.raw_app" }, + + // ── Codebase bundling ───────────────────────────────────────────────── + { name: "codebases", type: "array", default: "[]", description: "Codebase bundling configurations for shared libraries", + items: { + type: "object", + properties: { + relative_path: { type: "string", description: "Path to the codebase directory" }, + includes: { type: "array", items: { type: "string" }, description: "Glob patterns for files to include in bundle" }, + excludes: { type: "array", items: { type: "string" }, description: "Glob patterns for files to exclude from bundle" }, + format: { type: "string", enum: ["cjs", "esm"], description: "Bundle output format" }, + external: { type: "array", items: { type: "string" }, description: "Dependencies to leave unbundled (externals)" }, + assets: { type: "array", items: { type: "object", properties: { from: { type: "string" }, to: { type: "string" } }, required: ["from", "to"] }, description: "Static files to copy into the bundle" }, + customBundler: { type: "string", description: "Path to a custom bundler script (replaces esbuild)" }, + inject: { type: "array", items: { type: "string" }, description: "Files to inject into every entry point" }, + define: { type: "object", additionalProperties: { type: "string" }, description: "Compile-time constant definitions" }, + banner: { type: "object", additionalProperties: { type: "string" }, description: "Text to prepend to output files by type" }, + loader: { type: "object", additionalProperties: { type: "string" }, description: "esbuild loader overrides by extension" }, + }, + required: ["relative_path"], + additionalProperties: false, + }, + section: "Codebase bundling (shared libraries)", + sectionNote: "Bundle TypeScript/JavaScript codebases that scripts import from.\nEach entry is bundled and uploaded so scripts can import shared code.", + example: [ + "# codebases:", + '# - relative_path: ./shared # path to the codebase', + '# includes: ["**/*.ts"] # files to include in bundle', + '# excludes: ["node_modules/**"] # files to exclude', + '# format: esm # bundle format: "cjs" or "esm"', + '# external: ["pg", "axios"] # dependencies to leave unbundled', + "# assets: # static files to copy into bundle", + "# - from: ./static", + "# to: ./dist", + "# # customBundler: ./build.ts # custom bundler script (replaces esbuild)", + '# # inject: ["./polyfills.ts"] # files to inject into every entry point', + "# # define: # compile-time constants", + "# # API_URL: '\"https://api.example.com\"'", + "# # banner: # text prepended to output files", + '# # js: "/* bundled by windmill */"', + "# # loader: # esbuild loader overrides", + '# # ".png": "dataurl"', + ].join("\n"), + }, + + // ── Git branches ────────────────────────────────────────────────────── + { name: "gitBranches", type: "object", default: "{}", description: "Map git branches to workspaces and per-branch sync overrides", + properties: { commonSpecificItems: SPECIFIC_ITEMS_SCHEMA }, + additionalProperties: BRANCH_CONFIG_SCHEMA, + section: "Git branch / environment bindings", + sectionNote: "Map git branches to Windmill workspaces and override settings per branch.\nUse \"environments\" as an alias if you prefer environment-based terminology.", + templateValue: "\n {{BRANCH}}:\n overrides: {}", + example: [ + "{{BASEURL_LINE}}", + "{{WORKSPACE_ID_LINE}}", + " # promotionOverrides: # overrides applied during --promotion", + " # skipSecrets: false", + " # specificItems: # only sync these specific items", + ' # variables: ["f/my_folder/my_var"]', + ' # resources: ["f/my_folder/my_res"]', + ' # triggers: ["f/my_folder/my_trigger"]', + ' # folders: ["my_folder"]', + " # settings: true", + "", + " # Example: staging branch bound to a different workspace", + " # staging:", + " # baseUrl: https://staging.windmill.dev", + " # workspaceId: staging-workspace", + " # overrides:", + " # skipSecrets: false", + " # includeSchedules: true", + "", + " # Items shared across ALL branches", + " # commonSpecificItems:", + ' # variables: ["f/shared/api_key"]', + ' # resources: ["f/shared/db_conn"]', + ' # folders: ["shared"]', + ].join("\n"), + }, + + { name: "environments", type: "object", default: "{}", description: "Alias for gitBranches — use if you prefer environment-based terminology", + properties: { commonSpecificItems: SPECIFIC_ITEMS_SCHEMA }, + additionalProperties: BRANCH_CONFIG_SCHEMA, + commented: true }, +]; + +// ─── Template generator ───────────────────────────────────────────────────── + +export interface BranchBinding { + baseUrl: string; + workspaceId: string; +} + +/** Quote a string for use as a YAML key if it contains special characters. */ +function yamlKey(s: string): string { + if ( + /^[a-zA-Z0-9_/.@-]+$/.test(s) && + !/^(true|false|yes|no|on|off|null|~)$/i.test(s) && + !/^\d+(\.\d+)?$/.test(s) + ) { + return s; + } + return `"${s.replace(/\\/g, "\\\\").replace(/"/g, '\\"')}"`; +} + +export function generateCommentedTemplate(branchName?: string, binding?: BranchBinding): string { + const branch = yamlKey(branchName ?? "main"); + const lines: string[] = [ + "# yaml-language-server: $schema=wmill.schema.json", + "# wmill.yaml — Windmill CLI configuration", + '# Full reference: run "wmill config"', + "", + ]; + + for (const opt of CONFIG_REFERENCE) { + if (opt.section) { + const ruler = "-".repeat(Math.max(0, 65 - opt.section.length)); + lines.push(`# --- ${opt.section} ${ruler}`); + if (opt.sectionNote) { + for (const noteLine of opt.sectionNote.split("\n")) { + lines.push(`# ${noteLine}`); + } + } + lines.push(""); + } + + if (opt.groupNote) { + lines.push(`# ${opt.groupNote}`); + } + + const value = opt.templateValue ?? opt.default; + const resolvedValue = value.replace("{{BRANCH}}", branch); + + if (opt.commented) { + lines.push(`# ${opt.description}`); + lines.push(`# ${opt.name}: ${resolvedValue}`); + } else { + lines.push(`# ${opt.description}`); + if (opt.inlineComment) { + const base = `${opt.name}: ${resolvedValue}`; + const pad = " ".repeat(Math.max(1, 32 - base.length)); + lines.push(`${base}${pad}# ${opt.inlineComment}`); + } else { + lines.push(`${opt.name}: ${resolvedValue}`); + } + } + + if (opt.example) { + let resolvedExample = opt.example.replace(/\{\{BRANCH\}\}/g, branch); + if (binding) { + resolvedExample = resolvedExample + .replace("{{BASEURL_LINE}}", ` baseUrl: ${binding.baseUrl}`) + .replace("{{WORKSPACE_ID_LINE}}", ` workspaceId: ${binding.workspaceId}`); + } else { + resolvedExample = resolvedExample + .replace("{{BASEURL_LINE}}", " # baseUrl: https://app.windmill.dev # Windmill instance URL for this branch") + .replace("{{WORKSPACE_ID_LINE}}", " # workspaceId: my-workspace # workspace to sync with"); + } + for (const exLine of resolvedExample.split("\n")) { + lines.push(exLine); + } + } + + lines.push(""); + } + + return lines.join("\n"); +} + +// ─── Reference formatters ─────────────────────────────────────────────────── + +/** Recursively expand a schema's properties into flat reference rows. */ +function expandSchema( + prefix: string, + schema: Record, + rows: { name: string; description: string; default: string }[] +): void { + if (schema.properties) { + for (const [key, prop] of Object.entries(schema.properties) as [string, Record][]) { + const name = prefix ? `${prefix}.${key}` : key; + rows.push({ name, description: prop.description ?? "", default: "" }); + // Recurse into nested object properties (e.g., specificItems) + if (prop.properties && prop.type === "object") { + expandSchema(name, prop, rows); + } + } + } +} + +export function formatConfigReference(): string { + const nameWidth = 48; + const descWidth = 70; + + const header = [ + "OPTION".padEnd(nameWidth), + "DESCRIPTION".padEnd(descWidth), + "DEFAULT", + ].join(" "); + + const separator = "-".repeat(header.length + 10); + + const allRows: { name: string; description: string; default: string }[] = []; + for (const opt of CONFIG_REFERENCE) { + allRows.push({ name: opt.name, description: opt.description, default: opt.default }); + + // Auto-expand array item properties (e.g., codebases[].*) + if (opt.items?.properties) { + expandSchema(`${opt.name}[]`, opt.items, allRows); + } + // Auto-expand additionalProperties (e.g., gitBranches..*) + if (opt.additionalProperties && typeof opt.additionalProperties === "object" && opt.additionalProperties.properties) { + expandSchema(`${opt.name}.`, opt.additionalProperties as Record, allRows); + } + // Auto-expand named properties (e.g., gitBranches.commonSpecificItems) + if (opt.properties) { + expandSchema(opt.name, opt, allRows); + } + } + + const rows = allRows.map((r) => + [r.name.padEnd(nameWidth), r.description.padEnd(descWidth), r.default].join(" ") + ); + + return [ + "wmill.yaml — Configuration Reference", + "", + "Full documentation: https://www.windmill.dev/docs/advanced/cli", + "", + separator, + header, + separator, + ...rows, + separator, + "", + 'Run "wmill init" to generate a wmill.yaml with commented examples.', + ].join("\n"); +} + +export function formatConfigReferenceJson(): string { + const clean = CONFIG_REFERENCE.map((opt) => ({ + name: opt.name, type: opt.type, default: opt.default, description: opt.description, + })); + return JSON.stringify(clean, null, 2); +} + +// ─── JSON Schema generator ────────────────────────────────────────────────── + +/** + * Generate a JSON Schema for wmill.yaml by stripping non-schema keys from CONFIG_REFERENCE. + */ +export function generateJsonSchema(): Record { + const properties: Record = {}; + for (const opt of CONFIG_REFERENCE) { + const entry: Record = {}; + for (const [k, v] of Object.entries(opt)) { + if (!NON_SCHEMA_KEYS.has(k) && k !== "name") { + entry[k] = v; + } + } + properties[opt.name] = entry; + } + return { + $schema: "http://json-schema.org/draft-07/schema#", + title: "wmill.yaml", + description: "Windmill CLI configuration file. Full reference: wmill config", + type: "object", + properties, + additionalProperties: false, + }; +} diff --git a/cli/src/commands/instance/instance.ts b/cli/src/commands/instance/instance.ts index 6b22d49b27..d95fe7269b 100644 --- a/cli/src/commands/instance/instance.ts +++ b/cli/src/commands/instance/instance.ts @@ -219,6 +219,22 @@ export async function pickInstance( prefix: opts.prefix ?? "custom", }; } + // Try to use the active workspace profile's remote as a fallback + if (instances.length < 1) { + try { + const ws = await getActiveWorkspace({}); + if (ws?.remote && ws?.token) { + const remote = ws.remote.endsWith("/") ? ws.remote.slice(0, -1) : ws.remote; + setClient(ws.token, remote); + return { + name: ws.name, + remote: ws.remote, + token: ws.token, + prefix: ws.name, + }; + } + } catch { /* ignore */ } + } if (!allowNew && instances.length < 1) { throw new Error("No instance found, please add one first"); } @@ -648,9 +664,27 @@ export async function getActiveInstance(opts: { } } -async function getConfig(opts: InstanceSyncOptions & { outputFile?: string }) { +async function getConfig(opts: InstanceSyncOptions & { outputFile?: string; showSecrets?: boolean }) { await pickInstance(opts, false); - const config = await wmill.getInstanceConfig(); + const config = await wmill.getInstanceConfig() as any; + + // In interactive mode, mask secrets by default and prompt + const hasSecrets = config?.global_settings?.license_key || config?.global_settings?.jwt_secret; + let showSecrets = opts.showSecrets ?? false; + if (!showSecrets && hasSecrets && process.stdout.isTTY && !opts.outputFile) { + log.warn("Config contains sensitive fields (license_key, jwt_secret). They are masked by default."); + log.warn("Use --show-secrets to include them, or press Y to show them now."); + showSecrets = await Confirm.prompt({ message: "Show secrets?", default: false }); + } else if (!process.stdout.isTTY || opts.outputFile) { + // Non-interactive or writing to file: always include secrets + showSecrets = true; + } + + if (!showSecrets && config?.global_settings) { + if (config.global_settings.license_key) config.global_settings.license_key = "***"; + if (config.global_settings.jwt_secret) config.global_settings.jwt_secret = "***"; + } + const yaml = yamlStringify(config as Record); if (opts.outputFile) { await writeFile(opts.outputFile, yaml, "utf-8"); @@ -786,6 +820,7 @@ const command = new Command() .command("get-config") .description("Dump the current instance config (global settings + worker configs) as YAML") .option("-o, --output-file ", "Write YAML to a file instead of stdout") + .option("--show-secrets", "Include sensitive fields (license key, JWT secret) without prompting") .option( "--instance ", "Name of the instance, override the active instance", diff --git a/cli/src/commands/job/job.ts b/cli/src/commands/job/job.ts new file mode 100644 index 0000000000..f542355fb7 --- /dev/null +++ b/cli/src/commands/job/job.ts @@ -0,0 +1,400 @@ +import { GlobalOptions } from "../../types.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { resolveWorkspace } from "../../core/context.ts"; +import { Command } from "@cliffy/command"; +import { Table } from "@cliffy/table"; +import { colors } from "@cliffy/ansi/colors"; +import * as log from "../../core/log.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; +import * as wmill from "../../../gen/services.gen.ts"; +import { formatTimestamp } from "../../utils/utils.ts"; + +function formatDuration(ms: number): string { + if (ms < 1000) return `${ms}ms`; + const seconds = Math.floor(ms / 1000); + if (seconds < 60) return `${seconds}s`; + const minutes = Math.floor(seconds / 60); + const remainingSeconds = seconds % 60; + if (minutes < 60) return `${minutes}m${remainingSeconds}s`; + const hours = Math.floor(minutes / 60); + const remainingMinutes = minutes % 60; + return `${hours}h${remainingMinutes}m`; +} + +function getJobStatus(job: any): string { + if (job.type === "QueuedJob") { + if (job.canceled) return colors.red("canceled"); + if (job.running) return colors.blue("running"); + return colors.yellow("queued"); + } + // CompletedJob + if (job.canceled) return colors.red("canceled"); + if (job.success) return colors.green("success"); + return colors.red("failure"); +} + +function getJobStatusPlain(job: any): string { + if (job.type === "QueuedJob") { + if (job.canceled) return "canceled"; + if (job.running) return "running"; + return "queued"; + } + if (job.canceled) return "canceled"; + if (job.success) return "success"; + return "failure"; +} + +async function list( + opts: GlobalOptions & { + json?: boolean; + scriptPath?: string; + createdBy?: string; + running?: boolean; + success?: boolean; + failed?: boolean; + limit?: number; + jobKinds?: string; + label?: string; + all?: boolean; + parent?: string; + isFlowStep?: boolean; + } +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + // --failed is a convenience alias for --success false + let successFilter = opts.success; + if (opts.failed) successFilter = false; + + // When --all or --parent is used, include flow sub-job kinds too + const showSubJobs = opts.all || opts.parent; + const defaultJobKinds = showSubJobs + ? "script,flow,singlestepflow,flowscript,flowdependencies" + : "script,flow,singlestepflow"; + + const limit = Math.min(opts.limit ?? 30, 100); + const allJobs = await wmill.listJobs({ + workspace: workspace.workspaceId, + scriptPathExact: opts.scriptPath, + createdBy: opts.createdBy, + running: opts.running, + success: successFilter, + perPage: limit, + jobKinds: opts.jobKinds ?? defaultJobKinds, + label: opts.label, + hasNullParent: showSubJobs ? undefined : true, + parentJob: opts.parent, + isFlowStep: opts.isFlowStep, + }); + // API may return more than perPage — enforce limit client-side + const jobs = allJobs.slice(0, limit); + + if (opts.json) { + console.log(JSON.stringify(jobs)); + } else { + if (jobs.length === 0) { + log.info("No jobs found."); + return; + } + new Table() + .header(["ID", "Status", "Script/Flow", "Created By", "Duration", "Created At"]) + .padding(2) + .border(true) + .body( + jobs.map((j: any) => [ + j.id, + getJobStatus(j), + j.script_path ?? j.raw_code?.substring(0, 30) ?? "-", + j.created_by ?? j.email ?? "-", + j.duration_ms != null ? formatDuration(j.duration_ms) : (j.running ? "running" : "-"), + j.created_at ? formatTimestamp(j.created_at) : "-", + ]) + ) + .render(); + log.info(`\nShowing ${jobs.length} job(s). Use --limit to show more.`); + } +} + +function getModuleStatusIcon(type: string, success?: boolean): string { + switch (type) { + case "Success": return colors.green("✓"); + case "Failure": return colors.red("✗"); + case "InProgress": return colors.blue("▶"); + case "WaitingForPriorSteps": return colors.dim("○"); + case "WaitingForEvents": return colors.yellow("⏳"); + default: return colors.dim("·"); + } +} + +function formatFlowSteps( + flowStatus: any, + rawFlow: any, +) { + const modules = flowStatus?.modules ?? []; + const rawModules = rawFlow?.modules ?? []; + + // Build summary map from raw_flow + const summaryMap = new Map(); + for (const mod of rawModules) { + if (mod.id && mod.summary) { + summaryMap.set(mod.id, mod.summary); + } + } + + console.log(colors.bold("\nSteps:")); + for (const mod of modules) { + const icon = getModuleStatusIcon(mod.type); + const summary = summaryMap.get(mod.id) ?? ""; + const label = summary ? `${mod.id}: ${summary}` : mod.id; + const jobId = mod.job ? colors.dim(mod.job) : ""; + const flowJobsDuration = mod.flow_jobs_duration; + + // For-loop modules: show parent line + iteration sub-lines + const flowJobs = mod.flow_jobs as string[] | undefined; + if (flowJobs && flowJobs.length > 0) { + // Total duration for the for-loop + const totalMs = flowJobsDuration?.duration_ms + ? (flowJobsDuration.duration_ms as number[]).reduce((a: number, b: number) => a + b, 0) + : undefined; + const durationStr = totalMs != null ? colors.dim(formatDuration(totalMs)) : ""; + console.log(` ${icon} ${label} ${durationStr}`); + + const flowJobsSuccess = (mod.flow_jobs_success ?? []) as boolean[]; + const durationMs = (flowJobsDuration?.duration_ms ?? []) as number[]; + for (let iter = 0; iter < flowJobs.length; iter++) { + const iterSuccess = flowJobsSuccess[iter]; + const iterIcon = iterSuccess === true ? colors.green("✓") + : iterSuccess === false ? colors.red("✗") + : colors.dim("·"); + const iterDur = durationMs[iter] != null ? colors.dim(formatDuration(durationMs[iter])) : ""; + const iterJobId = colors.dim(flowJobs[iter]); + console.log(` ${iterIcon} iteration ${iter} ${iterJobId} ${iterDur}`); + } + } else { + // Regular step + const durationStr = mod.duration_ms != null + ? colors.dim(formatDuration(mod.duration_ms)) + : ""; + console.log(` ${icon} ${label} ${jobId} ${durationStr}`); + } + } + + // Show hint for diving into step logs + const hasJobs = modules.some((m: any) => m.job); + if (hasJobs) { + console.log(colors.dim("\nUse 'wmill job logs ' for step logs")); + } +} + +async function get( + opts: GlobalOptions & { json?: boolean }, + id: string +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const job = await wmill.getJob({ + workspace: workspace.workspaceId, + id, + }); + + if (opts.json) { + console.log(JSON.stringify(job)); + } else { + const j = job as any; + console.log(colors.bold("ID:") + " " + j.id); + console.log(colors.bold("Status:") + " " + getJobStatusPlain(j)); + console.log(colors.bold("Kind:") + " " + j.job_kind); + console.log(colors.bold("Script Path:") + " " + (j.script_path ?? "-")); + console.log(colors.bold("Created By:") + " " + (j.created_by ?? "-")); + console.log(colors.bold("Created At:") + " " + (j.created_at ? formatTimestamp(j.created_at) : "-")); + if (j.started_at) { + console.log(colors.bold("Started At:") + " " + formatTimestamp(j.started_at)); + } + if (j.duration_ms != null) { + console.log(colors.bold("Duration:") + " " + formatDuration(j.duration_ms)); + } + if (j.schedule_path) { + console.log(colors.bold("Schedule:") + " " + j.schedule_path); + } + + // Flow: show hierarchical step status + const isFlow = j.job_kind === "flow" || j.job_kind === "flowpreview"; + if (isFlow && j.flow_status) { + formatFlowSteps(j.flow_status, j.raw_flow); + } + + if (j.result !== undefined) { + console.log(colors.bold("\nResult:")); + console.log(JSON.stringify(j.result, null, 2)); + } + } +} + +async function result( + opts: GlobalOptions, + id: string +) { + log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const jobResult = await wmill.getCompletedJobResult({ + workspace: workspace.workspaceId, + id, + }); + + console.log(JSON.stringify(jobResult)); +} + +async function logs( + opts: GlobalOptions, + id: string +) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + // Check if this is a flow job — if so, aggregate all step logs + try { + const job = await wmill.getJob({ + workspace: workspace.workspaceId, + id, + }); + const j = job as any; + const jobKind = j.job_kind; + if ((jobKind === "flow" || jobKind === "flowpreview") && j.flow_status?.modules) { + const modules = j.flow_status.modules; + const rawModules = j.raw_flow?.modules ?? []; + const summaryMap = new Map(); + for (const mod of rawModules) { + if (mod.id && mod.summary) summaryMap.set(mod.id, mod.summary); + } + + // Strip the "to remove ansi colors" hint that appears in each step's logs + const stripHint = (text: string) => + text.replace(/^to remove ansi colors.*\n?/gm, ""); + + let hasLogs = false; + for (const mod of modules) { + const summary = summaryMap.get(mod.id) ?? ""; + const label = summary ? `${mod.id}: ${summary}` : mod.id; + + // For-loop modules: get logs for each iteration + const flowJobs = mod.flow_jobs as string[] | undefined; + if (flowJobs && flowJobs.length > 0) { + for (let iter = 0; iter < flowJobs.length; iter++) { + try { + const stepLogs = await wmill.getJobLogs({ + workspace: workspace.workspaceId, + id: flowJobs[iter], + }); + if (stepLogs) { + console.log(colors.bold.cyan(`\n====== ${label} (iteration ${iter}) ======`)); + console.log(stripHint(stepLogs)); + hasLogs = true; + } + } catch { /* step may not exist yet */ } + } + } else if (mod.job) { + // Regular step + try { + const stepLogs = await wmill.getJobLogs({ + workspace: workspace.workspaceId, + id: mod.job, + }); + if (stepLogs) { + console.log(colors.bold.cyan(`\n====== ${label} ======`)); + console.log(stripHint(stepLogs)); + hasLogs = true; + } + } catch { /* step may not exist yet */ } + } + } + + if (!hasLogs) { + log.info("No logs available for this flow's steps."); + } + return; + } + } catch { + // If we can't get the job info, proceed with trying to get logs anyway + } + + const jobLogs = await wmill.getJobLogs({ + workspace: workspace.workspaceId, + id, + }); + + if (jobLogs == null || jobLogs === "") { + log.info("No logs available for this job."); + } else { + // Strip the hint if the API already includes it, then print it once to stderr + const stripped = jobLogs.replace(/^to remove ansi colors.*\n?/gm, ""); + console.error("to remove ansi colors, use: | sed 's/\\x1B\\[[0-9;]\\{1,\\}[A-Za-z]//g'"); + console.log(stripped); + } +} + +async function cancel( + opts: GlobalOptions & { reason?: string }, + id: string +) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.cancelQueuedJob({ + workspace: workspace.workspaceId, + id, + requestBody: { + reason: opts.reason ?? "Canceled via CLI", + }, + }); + + log.info(colors.green(`Job ${id} canceled.`)); +} + +// Shared list options to avoid repetition between default action and list subcommand +const listOptions = (cmd: Command) => + cmd + .option("--json", "Output as JSON (for piping to jq)") + .option("--script-path ", "Filter by exact script/flow path") + .option("--created-by ", "Filter by creator username") + .option("--running", "Show only running jobs") + .option("--failed", "Show only failed jobs") + .option("--success ", "Filter by success status (true/false)") + .option("--limit ", "Number of jobs to return (default 30, max 100)") + .option("--job-kinds ", "Filter by job kinds (default: script,flow,singlestepflow)") + .option("--label ", "Filter by job label") + .option("--all", "Include sub-jobs (flow steps). By default only top-level jobs are shown") + .option("--parent ", "Filter by parent job ID (show sub-jobs of a specific flow)") + .option("--is-flow-step", "Show only flow step jobs"); + +const command = listOptions(new Command() + .description("Manage jobs (list, inspect, cancel)")) + .action(list as any) + .command("list", listOptions(new Command().description("List recent jobs"))) + .action(list as any) + .command("get", "Get job details. For flows: shows step tree with sub-job IDs") + .arguments("") + .option("--json", "Output as JSON (for piping to jq)") + .action(get as any) + .command("result", "Get the result of a completed job (machine-friendly)") + .arguments("") + .action(result as any) + .command("logs", "Get job logs. For flows: aggregates all step logs") + .arguments("") + .action(logs as any) + .command("cancel", "Cancel a running or queued job") + .arguments("") + .option("--reason ", "Reason for cancellation") + .action(cancel as any); + +export default command; diff --git a/cli/src/commands/lint/lint.ts b/cli/src/commands/lint/lint.ts index 62a491eee1..c726bff818 100644 --- a/cli/src/commands/lint/lint.ts +++ b/cli/src/commands/lint/lint.ts @@ -625,7 +625,13 @@ export async function runLint( throw new Error(`Path is not a directory: ${targetDirectory}`); } - const ignore = await ignoreF(mergedOpts); + // When the user specifies a subdirectory (that doesn't contain wmill.yaml), + // skip include/exclude filters since they're relative to the project root. + const isSubdirectory = explicitTargetDirectory && + !(await stat(path.join(targetDirectory, "wmill.yaml")).catch(() => null)); + const ignore = isSubdirectory + ? (_p: string, _isDir: boolean) => false + : await ignoreF(mergedOpts); const root = await FSFSElement(targetDirectory, [], false); const validator = new WindmillYamlValidator(); @@ -640,9 +646,10 @@ export async function runLint( if (entry.isDirectory || entry.ignored) { continue; } - scannedFiles += 1; const normalizedPath = normalizePath(entry.path); + + scannedFiles += 1; if (!YAML_FILE_REGEX.test(normalizedPath)) { continue; } @@ -742,7 +749,11 @@ export function printReport(report: LintReport, jsonOutput: boolean) { } } -async function lint(opts: LintOptions, directory?: string) { +async function lint(opts: LintOptions & { watch?: boolean }, directory?: string) { + if (opts.watch) { + await lintWatch(opts, directory); + return; + } try { const report = await runLint(opts, directory); printReport(report, !!opts.json); @@ -770,6 +781,37 @@ async function lint(opts: LintOptions, directory?: string) { } } +async function lintWatch(opts: LintOptions, directory?: string) { + const { watch } = await import("node:fs"); + const targetDir = directory ? path.resolve(process.cwd(), directory) : process.cwd(); + + log.info(colors.blue(`Watching ${targetDir} for changes... (Ctrl+C to stop)`)); + + async function runAndReport() { + try { + const report = await runLint(opts, directory); + // Clear screen for readability + process.stdout.write("\x1Bc"); + log.info(colors.gray(`[${new Date().toLocaleTimeString()}] Lint results:\n`)); + printReport(report, false); + } catch (error) { + log.error(error instanceof Error ? error.message : String(error)); + } + } + + await runAndReport(); + + let debounce: ReturnType | null = null; + watch(targetDir, { recursive: true }, (_event, filename) => { + if (!filename || !filename.toString().endsWith(".yaml") && !filename.toString().endsWith(".yml")) return; + if (debounce) clearTimeout(debounce); + debounce = setTimeout(runAndReport, 300); + }); + + // Keep the process alive + await new Promise(() => {}); +} + const command = new Command() .description( "Validate Windmill flow, schedule, and trigger YAML files in a directory", @@ -781,6 +823,7 @@ const command = new Command() "--locks-required", "Fail if scripts or flow inline scripts that need locks have no locks", ) + .option("-w, --watch", "Watch for file changes and re-lint automatically") .action(lint as any); export default command; diff --git a/cli/src/commands/resource-type/resource-type.ts b/cli/src/commands/resource-type/resource-type.ts index 0a5afe8dcb..96c8429eb4 100644 --- a/cli/src/commands/resource-type/resource-type.ts +++ b/cli/src/commands/resource-type/resource-type.ts @@ -88,6 +88,7 @@ async function push(opts: PushOptions, filePath: string, name: string) { } async function list(opts: GlobalOptions & { schema?: boolean; json?: boolean }) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); const res = await wmill.listResourceType({ @@ -96,6 +97,10 @@ async function list(opts: GlobalOptions & { schema?: boolean; json?: boolean }) if (opts.json) { console.log(JSON.stringify(res)); + } else if (res.length === 0) { + log.info("No custom resource types found in this workspace."); + log.info("Built-in types like 'postgresql', 'slack', 'mysql', etc. are available from the Windmill Hub."); + return; } else if (opts.schema) { new Table() .header(["Workspace", "Name", "Schema"]) diff --git a/cli/src/commands/resource/resource.ts b/cli/src/commands/resource/resource.ts index 332c8f3311..2f8ff47a06 100644 --- a/cli/src/commands/resource/resource.ts +++ b/cli/src/commands/resource/resource.ts @@ -1,4 +1,4 @@ -import { stat, writeFile, readdir, readFile } from "node:fs/promises"; +import { mkdir, stat, writeFile, readdir, readFile } from "node:fs/promises"; import { stringify as yamlStringify } from "yaml"; import nodePath from "node:path"; @@ -156,6 +156,7 @@ async function push(opts: PushOptions, filePath: string, remotePath: string) { } async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); let page = 0; @@ -203,6 +204,7 @@ async function newResource(opts: GlobalOptions, path: string) { resource_type: "", description: "", }; + await mkdir(nodePath.dirname(filePath), { recursive: true }); await writeFile(filePath, yamlStringify(template as Record), { flag: "wx", encoding: "utf-8", @@ -211,6 +213,7 @@ async function newResource(opts: GlobalOptions, path: string) { } async function get(opts: GlobalOptions & { json?: boolean }, path: string) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); const r = await wmill.getResource({ diff --git a/cli/src/commands/schedule/schedule.ts b/cli/src/commands/schedule/schedule.ts index c8582c5315..db57cabc64 100644 --- a/cli/src/commands/schedule/schedule.ts +++ b/cli/src/commands/schedule/schedule.ts @@ -1,4 +1,5 @@ -import { stat, writeFile } from "node:fs/promises"; +import { mkdir, stat, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; import { stringify as yamlStringify } from "yaml"; import { Command } from "@cliffy/command"; @@ -8,6 +9,7 @@ import * as log from "../../core/log.ts"; import { sep as SEP } from "node:path"; import { requireLogin } from "../../core/auth.ts"; import { resolveWorkspace, validatePath } from "../../core/context.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; import * as wmill from "../../../gen/services.gen.ts"; import { @@ -29,6 +31,7 @@ export interface ScheduleFile { } async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -60,7 +63,7 @@ async function newSchedule(opts: GlobalOptions, path: string) { if (e.message?.startsWith("File already exists")) throw e; } const template: ScheduleFile = { - schedule: "0 */6 * * *", + schedule: "0 0 */6 * * *", on_failure: "", script_path: "", args: {}, @@ -68,6 +71,7 @@ async function newSchedule(opts: GlobalOptions, path: string) { is_flow: false, enabled: false, }; + await mkdir(dirname(filePath), { recursive: true }); await writeFile(filePath, yamlStringify(template as Record), { flag: "wx", encoding: "utf-8", @@ -76,6 +80,7 @@ async function newSchedule(opts: GlobalOptions, path: string) { } async function get(opts: GlobalOptions & { json?: boolean }, path: string) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); const s = await wmill.getSchedule({ @@ -162,6 +167,34 @@ export async function pushSchedule( } } +async function enable(opts: GlobalOptions, path: string) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.setScheduleEnabled({ + workspace: workspace.workspaceId, + path, + requestBody: { enabled: true }, + }); + + log.info(colors.green(`Schedule ${path} enabled.`)); +} + +async function disable(opts: GlobalOptions, path: string) { + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + await wmill.setScheduleEnabled({ + workspace: workspace.workspaceId, + path, + requestBody: { enabled: false }, + }); + + log.info(colors.yellow(`Schedule ${path} disabled.`)); +} + async function push(opts: GlobalOptions, filePath: string, remotePath: string) { const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -205,6 +238,12 @@ const command = new Command() "push a local schedule spec. This overrides any remote versions." ) .arguments(" ") - .action(push as any); + .action(push as any) + .command("enable", "Enable a schedule") + .arguments("") + .action(enable as any) + .command("disable", "Disable a schedule") + .arguments("") + .action(disable as any); export default command; diff --git a/cli/src/commands/script/script.ts b/cli/src/commands/script/script.ts index 14f3155ff7..1d2b562497 100644 --- a/cli/src/commands/script/script.ts +++ b/cli/src/commands/script/script.ts @@ -1,7 +1,7 @@ import { GlobalOptions } from "../../types.ts"; import { requireLogin } from "../../core/auth.ts"; import { resolveWorkspace, validatePath } from "../../core/context.ts"; -import { readFile, writeFile, stat } from "node:fs/promises"; +import { readFile, writeFile, stat, mkdir } from "node:fs/promises"; import { Buffer } from "node:buffer"; import { colors } from "@cliffy/ansi/colors"; import { Command } from "@cliffy/command"; @@ -23,10 +23,13 @@ import { import { Workspace } from "../workspace/workspace.ts"; import { + checkifMetadataUptodate, generateScriptMetadataInternal, getRawWorkspaceDependencies, parseMetadataFile, + readLockfile, } from "../../utils/metadata.ts"; +import { generateHash, validateRequiredArgs } from "../../utils/utils.ts"; import { WorkspaceDependenciesLanguage, ScriptLanguage, @@ -101,7 +104,7 @@ export function isFlowInlineScriptPath(filePath: string): boolean { return isFlowInlineScriptPathInternal(filePath); } -type PushOptions = GlobalOptions; +type PushOptions = GlobalOptions & { message?: string }; async function push(opts: PushOptions, filePath: string) { opts = await mergeConfigWithConfigFile(opts); const workspace = await resolveWorkspace(opts); @@ -122,13 +125,35 @@ async function push(opts: PushOptions, filePath: string) { } await requireLogin(opts); + + // Warn about metadata state before pushing + try { + const content = await readFile(filePath, "utf-8"); + const remotePath = removeExtensionToPath(filePath).replaceAll(SEP, "/"); + const contentHash = await generateHash(content + remotePath); + const conf = await readLockfile(); + const hasLockEntry = conf.locks && (conf.locks[remotePath] !== undefined || conf.locks[`${remotePath}.ts`] !== undefined); + if (!hasLockEntry) { + log.warn(colors.yellow( + `No metadata generated yet for ${filePath}. Run 'wmill generate-metadata' to generate schema and lock.` + )); + } else if (!(await checkifMetadataUptodate(remotePath, contentHash, conf))) { + log.warn(colors.yellow( + `Metadata for ${filePath} appears stale (content changed since last 'wmill generate-metadata').\n` + + `The schema and lock may not match the current code. Consider running 'wmill generate-metadata' first.` + )); + } + } catch { + // Don't block push if check fails + } + const codebases = await listSyncCodebases(opts as SyncOptions); await handleFile( filePath, workspace, [], - undefined, + opts.message, opts, await getRawWorkspaceDependencies(true), codebases @@ -494,6 +519,7 @@ export async function handleFile( const body = { ...requestBodyCommon, parent_hash: remote.hash, + auto_parent: true, }; const execTime = await createScript( bundleContent, @@ -857,6 +883,7 @@ async function list( json?: boolean; } ) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -919,41 +946,93 @@ async function run( }, path: string ) { + if (opts.silent) { + log.setSilent(true); + } const workspace = await resolveWorkspace(opts); await requireLogin(opts); const input = opts.data ? await resolve(opts.data) : {}; - const id = await wmill.runScriptByPath({ - workspace: workspace.workspaceId, - path, - requestBody: input, - }); + + // Validate required args against schema when no data provided + if (!opts.data) { + try { + const script = await wmill.getScriptByPath({ + workspace: workspace.workspaceId, + path, + }); + validateRequiredArgs(script.schema as Record); + } catch (e: any) { + if (e.message?.startsWith("Missing required")) throw e; + log.warn(`Could not fetch schema to validate args: ${e.message}`); + } + } + + let id: string; + try { + id = await wmill.runScriptByPath({ + workspace: workspace.workspaceId, + path, + requestBody: input, + }); + } catch (e: any) { + if (e?.status === 404) { + // Script might exist but have a lock/deployment error — check before giving up + try { + const script = await wmill.getScriptByPath({ + workspace: workspace.workspaceId, + path, + }); + if (script.lock_error_logs) { + throw new Error( + `Script '${path}' has a deployment error and cannot be run:\n${script.lock_error_logs}` + ); + } + } catch (lookupErr: any) { + if (lookupErr?.message?.includes("deployment error")) throw lookupErr; + // Re-throw non-404 lookup errors (e.g. auth/network issues) + if (lookupErr?.status && lookupErr.status !== 404) throw lookupErr; + } + throw new Error( + `Script '${path}' not found. Run 'wmill script list' to see available scripts.` + ); + } + throw e; + } if (!opts.silent) { await track_job(workspace.workspaceId, id); } - while (true) { + const MAX_RETRIES = 600; // ~60 seconds at 100ms intervals + let retries = 0; + while (retries < MAX_RETRIES) { try { - const result = - ( - await wmill.getCompletedJob({ - workspace: workspace.workspaceId, - id, - }) - ).result ?? {}; + const completedJob = await wmill.getCompletedJob({ + workspace: workspace.workspaceId, + id, + }); + if (completedJob.success === false) { + process.exitCode = 1; + } + + const result = completedJob.result ?? {}; if (opts.silent) { - console.log(result); + console.log(JSON.stringify(result)); } else { log.info(JSON.stringify(result, null, 2)); } break; } catch { + retries++; await new Promise((resolve) => setTimeout(resolve, 100)); } } + if (retries >= MAX_RETRIES) { + throw new Error(`Timed out waiting for job ${id} to complete`); + } } export async function track_job(workspace: string, id: string) { @@ -1050,6 +1129,7 @@ async function show(opts: GlobalOptions, path: string) { } async function get(opts: GlobalOptions & { json?: boolean }, path: string) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); const s = await wmill.getScriptByPath({ @@ -1069,24 +1149,33 @@ async function get(opts: GlobalOptions & { json?: boolean }, path: string) { } } +const languageAliases: Record = { + python: "python3", +}; + async function bootstrap( opts: GlobalOptions & { summary: string; description: string }, scriptPath: string, - language: ScriptLanguage + language: ScriptLanguage | string ) { if (!validatePath(scriptPath)) { return; } - const scriptInitialCode = scriptBootstrapCode[language]; + const resolvedLanguage = (languageAliases[language] ?? language) as ScriptLanguage; + + const scriptInitialCode = scriptBootstrapCode[resolvedLanguage]; if (scriptInitialCode === undefined) { - throw new Error("Language unknown"); + const validLanguages = Object.keys(scriptBootstrapCode).sort().join(", "); + throw new Error( + `Unknown language '${language}'. Valid languages: ${validLanguages}` + ); } const config = await readConfigFile(); const extension = filePathExtensionFromContentType( - language, + resolvedLanguage, config.defaultTs ); const scriptCodeFileFullPath = scriptPath + extension; @@ -1118,6 +1207,9 @@ async function bootstrap( yamlOptions ); + const parentDir = path.dirname(scriptCodeFileFullPath); + await mkdir(parentDir, { recursive: true }); + await writeFile(scriptCodeFileFullPath, scriptInitialCode, { flag: 'wx', encoding: 'utf-8', }); @@ -1252,6 +1344,9 @@ async function preview( } & SyncOptions, filePath: string ) { + if (opts.silent) { + log.setSilent(true); + } opts = await mergeConfigWithConfigFile(opts); const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -1455,13 +1550,50 @@ async function preview( } } +async function history( + opts: GlobalOptions & { json?: boolean }, + scriptPath: string +) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + const workspace = await resolveWorkspace(opts); + await requireLogin(opts); + + const versions = await wmill.getScriptHistoryByPath({ + workspace: workspace.workspaceId, + path: scriptPath, + }); + + if (opts.json) { + console.log(JSON.stringify(versions)); + } else { + if (versions.length === 0) { + log.info("No version history found for " + scriptPath); + return; + } + new Table() + .header(["#", "Hash", "Created At", "Deployment Message"]) + .padding(2) + .border(true) + .body( + versions.map((v, i) => [ + String(versions.length - i), + v.script_hash, + v.created_at ? new Date(v.created_at).toLocaleString() : "-", + v.deployment_msg ?? "-", + ]) + ) + .render(); + } +} + const command = new Command() .description("script related commands") - .option("--show-archived", "Enable archived scripts in output") + .option("--show-archived", "Show archived scripts instead of active ones") .option("--json", "Output as JSON (for piping to jq)") .action(list as any) .command("list", "list all scripts") - .option("--show-archived", "Enable archived scripts in output") + .option("--show-archived", "Show archived scripts instead of active ones") .option("--json", "Output as JSON (for piping to jq)") .action(list as any) .command( @@ -1469,6 +1601,7 @@ const command = new Command() "push a local script spec. This overrides any remote versions. Use the script file (.ts, .js, .py, .sh)" ) .arguments("") + .option("--message ", "Deployment message") .action(push as any) .command("get", "get a script's details") .arguments("") @@ -1529,6 +1662,13 @@ const command = new Command() "-e --excludes ", "Comma separated patterns to specify which file to NOT take into account." ) - .action(generateMetadata as any); + .action(generateMetadata as any) + .command( + "history", + "show version history for a script" + ) + .arguments("") + .option("--json", "Output as JSON (for piping to jq)") + .action(history as any); export default command; diff --git a/cli/src/commands/sync/pull.ts b/cli/src/commands/sync/pull.ts index 826c1de090..0b4da237e1 100644 --- a/cli/src/commands/sync/pull.ts +++ b/cli/src/commands/sync/pull.ts @@ -3,9 +3,72 @@ import { colors } from "@cliffy/ansi/colors"; import { Command } from "@cliffy/command"; import * as log from "../../core/log.ts"; import JSZip from "jszip"; +import { extract } from "tar-stream"; +import { Readable } from "node:stream"; import { Workspace } from "../workspace/workspace.ts"; import { getHeaders } from "../../utils/utils.ts"; +/** + * Adapter that wraps tar entries in a JSZip-compatible interface + * so ZipFSElement in sync.ts can consume it without changes. + */ +class TarAsZip { + files: Record }> = {}; + + constructor(entries: Map) { + for (const [name, entry] of entries) { + const content = entry.content; + this.files[name] = { + dir: entry.isDir, + name, + async(_type: "text") { + return content; + }, + }; + } + } + + /** Return a filtered view containing only entries under the given prefix, with relative paths. */ + folder(prefix: string): TarAsZip | null { + const normalized = prefix.endsWith("/") ? prefix : prefix + "/"; + const sub = new TarAsZip(new Map()); + for (const [name, file] of Object.entries(this.files)) { + if (name.startsWith(normalized)) { + const relative = name.slice(normalized.length); + if (relative) { + sub.files[relative] = { ...file, name: relative }; + } + } + } + return Object.keys(sub.files).length > 0 ? sub : null; + } +} + +async function parseTarResponse(response: Response): Promise { + const buffer = Buffer.from(await response.arrayBuffer()); + const entries = new Map(); + const ex = extract(); + + return new Promise((resolve, reject) => { + ex.on("entry", (header, stream, next) => { + const chunks: Buffer[] = []; + stream.on("data", (chunk: Buffer) => chunks.push(chunk)); + stream.on("end", () => { + entries.set(header.name, { + content: Buffer.concat(chunks).toString("utf-8"), + isDir: header.type === "directory", + }); + next(); + }); + stream.on("error", reject); + stream.resume(); + }); + ex.on("finish", () => resolve(new TarAsZip(entries))); + ex.on("error", reject); + Readable.from(buffer).pipe(ex); + }); +} + export async function downloadZip( workspace: Workspace, plainSecrets: boolean | undefined, @@ -21,7 +84,7 @@ export async function downloadZip( includeKey?: boolean, skipWorkspaceDependencies?: boolean, defaultTs?: "bun" | "deno" -): Promise { +): Promise { const requestHeaders = new Headers(); requestHeaders.set("Authorization", "Bearer " + workspace.token); requestHeaders.set("Content-Type", "application/octet-stream"); @@ -34,38 +97,51 @@ export async function downloadZip( } const includeWorkspaceDependenciesValue = !(skipWorkspaceDependencies ?? false); - const url = workspace.remote + - "api/w/" + - workspace.workspaceId + - `/workspaces/tarball?archive_type=zip&plain_secret=${plainSecrets ?? false + const baseParams = `&plain_secret=${plainSecrets ?? false }&skip_variables=${skipVariables ?? false}&skip_resources=${skipResources ?? false }&skip_secrets=${skipSecrets ?? false}&include_schedules=${includeSchedules ?? false }&include_triggers=${includeTriggers ?? false}&include_users=${includeUsers ?? false }&include_groups=${includeGroups ?? false}&include_settings=${includeSettings ?? false }&include_key=${includeKey ?? false}&include_workspace_dependencies=${includeWorkspaceDependenciesValue}&default_ts=${defaultTs ?? "bun"}&skip_resource_types=${skipResourceTypes ?? false}&settings_version=v2`; - const zipResponse = await fetch(url, { - headers: requestHeaders, - method: "GET", - } - ); + const baseUrl = workspace.remote + "api/w/" + workspace.workspaceId + "/workspaces/tarball?"; - if (!zipResponse.ok) { - const body = await zipResponse.text(); - if (zipResponse.status === 404 || body.includes("no rows returned")) { - log.info(colors.red(`Workspace '${workspace.workspaceId}' not found on ${workspace.remote}. Please check your --workspace and try again.`)); - } else { - log.info(colors.red(`Failed to request tarball from API: ${zipResponse.status} ${zipResponse.statusText}`)); - if (body) { - log.info(colors.red(body)); - } - } - return process.exit(1); - } else { - log.debug(`Downloaded zip/tarball successfully`); + // Try zip first (standard format), fall back to tar if zip is not supported + const zipUrl = baseUrl + "archive_type=zip" + baseParams; + const zipResponse = await fetch(zipUrl, { headers: requestHeaders, method: "GET" }); + + if (zipResponse.ok) { + log.debug("Downloaded zip archive successfully"); + const blob = await zipResponse.blob(); + return await JSZip.loadAsync((await blob.arrayBuffer()) as any); } - const blob = await zipResponse.blob(); - return await JSZip.loadAsync((await blob.arrayBuffer()) as any); + + const body = await zipResponse.text(); + + // If zip format is not supported (backend compiled without zip feature), try tar + if (zipResponse.status === 400 && body.includes("Invalid Archive Type")) { + log.debug("Zip archive not supported by backend, falling back to tar"); + const tarUrl = baseUrl + "archive_type=tar" + baseParams; + const tarResponse = await fetch(tarUrl, { headers: requestHeaders, method: "GET" }); + + if (tarResponse.ok) { + log.debug("Downloaded tar archive successfully"); + return await parseTarResponse(tarResponse); + } + + const tarBody = await tarResponse.text(); + log.info(colors.red(`Failed to request tarball from API: ${tarResponse.status} ${tarResponse.statusText}`)); + if (tarBody) log.info(colors.red(tarBody)); + return process.exit(1); + } + + if (zipResponse.status === 404 || body.includes("no rows returned")) { + log.info(colors.red(`Workspace '${workspace.workspaceId}' not found on ${workspace.remote}. Please check your --workspace and try again.`)); + } else { + log.info(colors.red(`Failed to request tarball from API: ${zipResponse.status} ${zipResponse.statusText}`)); + if (body) log.info(colors.red(body)); + } + return process.exit(1); } function stub(_opts: GlobalOptions & { override: boolean }, _dir: string) { diff --git a/cli/src/commands/sync/sync.ts b/cli/src/commands/sync/sync.ts index d62c39cba9..c595af1130 100644 --- a/cli/src/commands/sync/sync.ts +++ b/cli/src/commands/sync/sync.ts @@ -76,7 +76,7 @@ import { newRawAppPathAssigner, PathAssigner, } from "../../../windmill-utils-internal/src/path-utils/path-assigner.ts"; -import { extractInlineScripts as extractInlineScriptsForFlows } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; +import { extractInlineScripts as extractInlineScriptsForFlows, extractCurrentMapping } from "../../../windmill-utils-internal/src/inline-scripts/extractor.ts"; import { generateFlowLockInternal } from "../flow/flow_metadata.ts"; import { isExecutionModeAnonymous } from "../app/app.ts"; import { @@ -93,6 +93,8 @@ import { isAppMetadataFile, isRawAppMetadataFile, isRawAppFolderMetadataFile, + isAppFolderMetadataFile, + isFlowFolderMetadataFile, getDeleteSuffix, transformJsonPathToDir, getFolderSuffix, @@ -636,9 +638,16 @@ function ZipFSElement( let inlineScripts; try { const assigner = newPathAssigner(defaultTs, { skipInlineScriptSuffix: getNonDottedPaths() }); - inlineScripts = extractInlineScriptsForFlows( + // Preserve original !inline filenames from the flow to avoid phantom renames + const inlineMapping = extractCurrentMapping( flow.value.modules as any, {}, + flow.value.failure_module, + flow.value.preprocessor_module, + ); + inlineScripts = extractInlineScriptsForFlows( + flow.value.modules as any, + inlineMapping, SEP, defaultTs, assigner, @@ -647,7 +656,7 @@ function ZipFSElement( if (flow.value.failure_module) { inlineScripts.push(...extractInlineScriptsForFlows( [flow.value.failure_module], - {}, + inlineMapping, SEP, defaultTs, assigner, @@ -657,7 +666,7 @@ function ZipFSElement( if (flow.value.preprocessor_module) { inlineScripts.push(...extractInlineScriptsForFlows( [flow.value.preprocessor_module], - {}, + inlineMapping, SEP, defaultTs, assigner, @@ -1516,6 +1525,10 @@ async function compareDynFSElement( continue; } if (k.startsWith("dependencies/")) { + if (!workspaceDependenciesPathToLanguageAndFilename(k)) { + log.warn(`Skipping unrecognized workspace dependencies file: ${k}`); + continue; + } log.info(`Adding workspace dependencies file: ${k}`); } changes.push({ name: "added", path: k, content: v }); @@ -1985,9 +1998,15 @@ export async function pull( opts: GlobalOptions & SyncOptions & { repository?: string; promotion?: string; branch?: string }, ) { + if ((opts as any).jsonOutput) log.setSilent(true); const originalCliOpts = { ...opts }; opts = await mergeConfigWithConfigFile(opts); + // --include-secrets overrides skipSecrets from wmill.yaml + if ((originalCliOpts as any).includeSecrets) { + opts.skipSecrets = false; + } + // Validate branch configuration early (skipped when --branch is used) try { await validateBranchConfiguration(opts, opts.branch); @@ -2476,12 +2495,18 @@ function removeSuffix(str: string, suffix: string) { export async function push( opts: GlobalOptions & SyncOptions & { repository?: string; branch?: string }, ) { + if ((opts as any).jsonOutput) log.setSilent(true); // Save original CLI options before merging with config file const originalCliOpts = { ...opts }; // Load configuration from wmill.yaml and merge with CLI options opts = await mergeConfigWithConfigFile(opts); + // --include-secrets overrides skipSecrets from wmill.yaml + if ((originalCliOpts as any).includeSecrets) { + opts.skipSecrets = false; + } + // Validate branch configuration early (skipped when --branch is used) try { await validateBranchConfiguration(opts, opts.branch); @@ -2615,6 +2640,7 @@ export async function push( const tracker: ChangeTracker = await buildTracker(changes); + const autoRegenerate = !!(opts as any).autoMetadata; const staleScripts: string[] = []; const staleFlows: string[] = []; const staleApps: string[] = []; @@ -2624,7 +2650,7 @@ export async function push( change, workspace, opts, - true, + !autoRegenerate, // dryRun=false when --auto is set true, rawWorkspaceDependencies, codebases, @@ -2637,11 +2663,19 @@ export async function push( if (staleScripts.length > 0) { log.info(""); - log.warn( - "Stale scripts metadata found, you may want to update them using 'wmill script generate-metadata' before pushing:", - ); + if (autoRegenerate) { + log.info("Auto-regenerated metadata for stale scripts:"); + } else { + log.warn( + "Stale scripts metadata found, you may want to update them using 'wmill script generate-metadata' before pushing:", + ); + } for (const stale of staleScripts) { - log.warn(stale); + if (autoRegenerate) { + log.info(` ${stale}`); + } else { + log.warn(stale); + } } log.info(""); @@ -2650,7 +2684,7 @@ export async function push( for (const change of tracker.flows) { const stale = await generateFlowLockInternal( change, - true, + !autoRegenerate, // dryRun=false when --auto is set workspace, opts, false, @@ -2662,11 +2696,19 @@ export async function push( } if (staleFlows.length > 0) { - log.warn( - "Stale flows locks found, you may want to update them using 'wmill flow generate-locks' before pushing:", - ); + if (autoRegenerate) { + log.info("Auto-regenerated locks for stale flows:"); + } else { + log.warn( + "Stale flows locks found, you may want to update them using 'wmill flow generate-locks' before pushing:", + ); + } for (const stale of staleFlows) { - log.warn(stale); + if (autoRegenerate) { + log.info(` ${stale}`); + } else { + log.warn(stale); + } } log.info(""); } @@ -2675,7 +2717,7 @@ export async function push( const stale = await generateAppLocksInternal( change, false, - true, + !autoRegenerate, workspace, opts, true, @@ -2690,7 +2732,7 @@ export async function push( const stale = await generateAppLocksInternal( change, true, - true, + !autoRegenerate, workspace, opts, true, @@ -2702,15 +2744,46 @@ export async function push( } if (staleApps.length > 0) { - log.warn( - "Stale apps locks found, you may want to update them using 'wmill app generate-locks' before pushing:", - ); + if (autoRegenerate) { + log.info("Auto-regenerated locks for stale apps:"); + } else { + log.warn( + "Stale apps locks found, you may want to update them using 'wmill app generate-locks' before pushing:", + ); + } for (const stale of staleApps) { - log.warn(stale); + if (autoRegenerate) { + log.info(` ${stale}`); + } else { + log.warn(stale); + } } log.info(""); } + // Warn about local files for skipped types. Walks the in-memory DynFSElement tree + // (not a fresh disk scan), but does re-traverse it. Acceptable cost for a one-time check. + { + const skippedWarnings: string[] = []; + let scheduleCount = 0; + let triggerCount = 0; + for await (const entry of readDirRecursiveWithIgnore(() => false, local)) { + if (entry.isDirectory) continue; + if (!opts.includeSchedules && entry.path.endsWith(".schedule.yaml")) scheduleCount++; + if (!opts.includeTriggers && entry.path.endsWith("_trigger.yaml")) triggerCount++; + } + if (scheduleCount > 0) { + skippedWarnings.push(`Skipping ${scheduleCount} schedule file(s). Use --include-schedules or set includeSchedules: true in wmill.yaml`); + } + if (triggerCount > 0) { + skippedWarnings.push(`Skipping ${triggerCount} trigger file(s). Use --include-triggers or set includeTriggers: true in wmill.yaml`); + } + for (const warning of skippedWarnings) { + log.warn(warning); + } + if (skippedWarnings.length > 0) log.info(""); + } + await fetchRemoteVersion(workspace); log.info( @@ -3160,16 +3233,88 @@ export async function push( }); break; case "flow": - await wmill.deleteFlowByPath({ - workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("flow", "json")), - }); + if (isFlowFolderMetadataFile(target)) { + // Metadata file deleted — delete the entire flow + await wmill.deleteFlowByPath({ + workspace: workspaceId, + path: removeSuffix(target, getDeleteSuffix("flow", "json")), + }); + } else { + // Inline script file deleted within flow folder + const flowFolder = extractFolderPath(target, "flow"); + let flowFolderExists = false; + if (flowFolder) { + try { + await stat(flowFolder); + flowFolderExists = true; + } catch { + // folder doesn't exist + } + } + if (flowFolderExists) { + // Re-push the entire flow so the backend gets the updated definition + await pushObj( + workspaceId, + target, + undefined, + undefined, + opts.plainSecrets ?? false, + alreadySynced, + opts.message, + ); + } else { + // Flow folder doesn't exist locally — delete on server + const remotePath = extractResourceName(target, "flow"); + if (remotePath) { + await wmill.deleteFlowByPath({ + workspace: workspaceId, + path: remotePath, + }); + } + } + } break; case "app": - await wmill.deleteApp({ - workspace: workspaceId, - path: removeSuffix(target, getDeleteSuffix("app", "json")), - }); + if (isAppFolderMetadataFile(target)) { + // Metadata file deleted — delete the entire app + await wmill.deleteApp({ + workspace: workspaceId, + path: removeSuffix(target, getDeleteSuffix("app", "json")), + }); + } else { + // Inline script file deleted within app folder + const appFolder = extractFolderPath(target, "app"); + let appFolderExists = false; + if (appFolder) { + try { + await stat(appFolder); + appFolderExists = true; + } catch { + // folder doesn't exist + } + } + if (appFolderExists) { + // Re-push the entire app so the backend gets the updated definition + await pushObj( + workspaceId, + target, + undefined, + undefined, + opts.plainSecrets ?? false, + alreadySynced, + opts.message, + ); + } else { + // App folder doesn't exist locally — delete on server + const remotePath = extractResourceName(target, "app"); + if (remotePath) { + await wmill.deleteApp({ + workspace: workspaceId, + path: remotePath, + }); + } + } + } break; case "raw_app": if (isRawAppFolderMetadataFile(target)) { @@ -3448,6 +3593,7 @@ const command = new Command() .option("--json", "Use JSON instead of YAML") .option("--skip-variables", "Skip syncing variables (including secrets)") .option("--skip-secrets", "Skip syncing only secrets variables") + .option("--include-secrets", "Include secrets in sync (overrides skipSecrets in wmill.yaml)") .option("--skip-resources", "Skip syncing resources") .option("--skip-resource-types", "Skip syncing resource types") .option("--skip-scripts", "Skip syncing scripts") @@ -3503,6 +3649,7 @@ const command = new Command() .option("--json", "Use JSON instead of YAML") .option("--skip-variables", "Skip syncing variables (including secrets)") .option("--skip-secrets", "Skip syncing only secrets variables") + .option("--include-secrets", "Include secrets in sync (overrides skipSecrets in wmill.yaml)") .option("--skip-resources", "Skip syncing resources") .option("--skip-resource-types", "Skip syncing resource types") .option("--skip-scripts", "Skip syncing scripts") @@ -3552,6 +3699,7 @@ const command = new Command() "--locks-required", "Fail if scripts or flow inline scripts that need locks have no locks", ) + .option("--auto-metadata", "Automatically regenerate stale metadata (locks and schemas) before pushing") .action(push as any); export default command; diff --git a/cli/src/commands/token/token.ts b/cli/src/commands/token/token.ts new file mode 100644 index 0000000000..d0c8cda76d --- /dev/null +++ b/cli/src/commands/token/token.ts @@ -0,0 +1,87 @@ +import { GlobalOptions } from "../../types.ts"; +import { requireLogin } from "../../core/auth.ts"; +import { Command } from "@cliffy/command"; +import { Table } from "@cliffy/table"; +import { colors } from "@cliffy/ansi/colors"; +import * as log from "../../core/log.ts"; +import { mergeConfigWithConfigFile } from "../../core/conf.ts"; +import * as wmill from "../../../gen/services.gen.ts"; +import { formatTimestamp } from "../../utils/utils.ts"; + +async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); + opts = await mergeConfigWithConfigFile(opts); + await requireLogin(opts); + + const tokens = await wmill.listTokens({ + excludeEphemeral: true, + }); + + if (opts.json) { + console.log(JSON.stringify(tokens)); + } else { + if (tokens.length === 0) { + log.info("No tokens found."); + return; + } + new Table() + .header(["Prefix", "Label", "Created At", "Last Used", "Expiration"]) + .padding(2) + .border(true) + .body( + tokens.map((t) => [ + t.token_prefix, + t.label ?? "-", + formatTimestamp(t.created_at), + formatTimestamp(t.last_used_at), + t.expiration ? formatTimestamp(t.expiration) : "never", + ]) + ) + .render(); + } +} + +async function create( + opts: GlobalOptions & { + label?: string; + expiration?: string; + } +) { + opts = await mergeConfigWithConfigFile(opts); + await requireLogin(opts); + + const token = await wmill.createToken({ + requestBody: { + label: opts.label, + expiration: opts.expiration, + }, + }); + + console.log(token); +} + +async function deleteToken(opts: GlobalOptions, tokenPrefix: string) { + opts = await mergeConfigWithConfigFile(opts); + await requireLogin(opts); + + await wmill.deleteToken({ tokenPrefix }); + + log.info(colors.green(`Token with prefix '${tokenPrefix}' deleted.`)); +} + +const command = new Command() + .description("Manage API tokens") + .option("--json", "Output as JSON (for piping to jq)") + .action(list as any) + .command("list", "List API tokens") + .option("--json", "Output as JSON (for piping to jq)") + .action(list as any) + .command("create", "Create a new API token") + .option("--label ", "Token label") + .option("--expiration ", "Token expiration (ISO 8601 timestamp)") + .action(create as any) + .command("delete", "Delete a token by its prefix") + .arguments("") + .action(deleteToken as any); + +export default command; diff --git a/cli/src/commands/trigger/trigger.ts b/cli/src/commands/trigger/trigger.ts index be645a83fa..e8a310a102 100644 --- a/cli/src/commands/trigger/trigger.ts +++ b/cli/src/commands/trigger/trigger.ts @@ -1,4 +1,5 @@ -import { stat, writeFile } from "node:fs/promises"; +import { mkdir, stat, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; import { stringify as yamlStringify } from "yaml"; import * as wmill from "../../../gen/services.gen.ts"; @@ -231,6 +232,7 @@ export async function pushNativeTrigger( is_flow: result.is_flow, service_config: result.service_config, error: result.error, + summary: result.summary, }; log.debug(`Native trigger ${serviceName}/${externalId} exists on remote`); } catch { @@ -243,6 +245,7 @@ export async function pushNativeTrigger( script_path: localTrigger.script_path, is_flow: localTrigger.is_flow, service_config: localTrigger.service_config, + summary: localTrigger.summary, }; if (remoteTrigger) { @@ -251,11 +254,13 @@ export async function pushNativeTrigger( script_path: localTrigger.script_path, is_flow: localTrigger.is_flow, service_config: localTrigger.service_config, + summary: localTrigger.summary, }; const remoteCompare = { script_path: remoteTrigger.script_path, is_flow: remoteTrigger.is_flow, service_config: remoteTrigger.service_config, + summary: remoteTrigger.summary, }; if (isSuperset(localCompare, remoteCompare)) { @@ -304,11 +309,20 @@ const triggerTemplates: Record> = { http_method: "get", is_async: false, requires_auth: true, + request_type: "sync", + authentication_method: "none", + is_static_website: false, + workspaced_route: false, + wrap_body: false, + raw_string: false, }, websocket: { script_path: "", is_flow: false, url: "", + filters: [], + can_return_message: false, + can_return_error_result: false, enabled: false, }, kafka: { @@ -317,6 +331,7 @@ const triggerTemplates: Record> = { kafka_resource_path: "", group_id: "", topics: [], + filters: [], enabled: false, }, nats: { @@ -324,6 +339,7 @@ const triggerTemplates: Record> = { is_flow: false, nats_resource_path: "", subjects: [], + use_jetstream: false, enabled: false, }, postgres: { @@ -338,28 +354,31 @@ const triggerTemplates: Record> = { script_path: "", is_flow: false, mqtt_resource_path: "", - topics: [], - subscribe_qos: 0, + subscribe_topics: [], enabled: false, }, sqs: { script_path: "", is_flow: false, - sqs_resource_path: "", queue_url: "", + aws_resource_path: "", + aws_auth_resource_type: "credentials", enabled: false, }, gcp: { script_path: "", is_flow: false, gcp_resource_path: "", - subscription_id: "", topic_id: "", + subscription_id: "", + delivery_type: "pull", + subscription_mode: "create_update", enabled: false, }, email: { script_path: "", is_flow: false, + local_part: "", enabled: false, }, }; @@ -383,6 +402,7 @@ async function newTrigger(opts: GlobalOptions & { kind: string }, path: string) if (e.message?.startsWith("File already exists")) throw e; } const template = triggerTemplates[kind]; + await mkdir(dirname(filePath), { recursive: true }); await writeFile(filePath, yamlStringify(template), { flag: "wx", encoding: "utf-8", @@ -390,7 +410,29 @@ async function newTrigger(opts: GlobalOptions & { kind: string }, path: string) log.info(colors.green(`Created ${filePath}`)); } +const TRIGGER_SKIP_FIELDS = new Set(["workspace_id", "extra_perms", "edited_by", "edited_at"]); + +function printTriggerDetails(trigger: any, kind: string) { + console.log(colors.bold("Path:") + " " + trigger.path); + console.log(colors.bold("Kind:") + " " + kind); + console.log(colors.bold("Enabled:") + " " + (trigger.enabled ?? trigger.mode ?? "-")); + console.log(colors.bold("Script Path:") + " " + (trigger.script_path ?? "")); + console.log(colors.bold("Is Flow:") + " " + (trigger.is_flow ? "true" : "false")); + // Show all other non-internal fields + for (const [key, value] of Object.entries(trigger)) { + if (["path", "enabled", "mode", "script_path", "is_flow"].includes(key)) continue; + if (TRIGGER_SKIP_FIELDS.has(key)) continue; + if (value === undefined || value === null || value === "") continue; + const display = Array.isArray(value) ? (value.length > 0 ? JSON.stringify(value) : "[]") : + typeof value === "object" ? JSON.stringify(value) : String(value); + if (display === "[]" || display === "{}") continue; + const label = key.replace(/_/g, " ").replace(/\b\w/g, c => c.toUpperCase()); + console.log(colors.bold(label + ":") + " " + display); + } +} + async function get(opts: GlobalOptions & { json?: boolean; kind?: string }, path: string) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -402,11 +444,7 @@ async function get(opts: GlobalOptions & { json?: boolean; kind?: string }, path if (opts.json) { console.log(JSON.stringify(trigger)); } else { - console.log(colors.bold("Path:") + " " + (trigger as any).path); - console.log(colors.bold("Kind:") + " " + opts.kind); - console.log(colors.bold("Enabled:") + " " + ((trigger as any).enabled ?? "-")); - console.log(colors.bold("Script Path:") + " " + ((trigger as any).script_path ?? "")); - console.log(colors.bold("Is Flow:") + " " + ((trigger as any).is_flow ? "true" : "false")); + printTriggerDetails(trigger as any, opts.kind); } return; } @@ -431,11 +469,7 @@ async function get(opts: GlobalOptions & { json?: boolean; kind?: string }, path if (opts.json) { console.log(JSON.stringify(trigger)); } else { - console.log(colors.bold("Path:") + " " + trigger.path); - console.log(colors.bold("Kind:") + " " + kind); - console.log(colors.bold("Enabled:") + " " + (trigger.enabled ?? "-")); - console.log(colors.bold("Script Path:") + " " + (trigger.script_path ?? "")); - console.log(colors.bold("Is Flow:") + " " + (trigger.is_flow ? "true" : "false")); + printTriggerDetails(trigger, kind); } return; } @@ -457,6 +491,7 @@ async function listOrEmpty(fn: () => Promise): Promise { } async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); diff --git a/cli/src/commands/user/user.ts b/cli/src/commands/user/user.ts index 207958ecce..b5ed0e3196 100644 --- a/cli/src/commands/user/user.ts +++ b/cli/src/commands/user/user.ts @@ -530,7 +530,7 @@ const command = new Command() .command("remove", "Delete a user") .arguments("") .action(remove as any) - .command("create-token") + .command("create-token", "Create a new API token for the authenticated user") .option( "--email ", "Specify credentials to use for authentication. This will not be stored. It will only be used to exchange for a token with the API server, which will not be stored either.", diff --git a/cli/src/commands/variable/variable.ts b/cli/src/commands/variable/variable.ts index 21b7a69eba..d902b8f79f 100644 --- a/cli/src/commands/variable/variable.ts +++ b/cli/src/commands/variable/variable.ts @@ -1,4 +1,5 @@ -import { stat, writeFile } from "node:fs/promises"; +import { mkdir, stat, writeFile } from "node:fs/promises"; +import { dirname } from "node:path"; import { stringify as yamlStringify } from "yaml"; import { requireLogin } from "../../core/auth.ts"; @@ -20,6 +21,7 @@ import * as wmill from "../../../gen/services.gen.ts"; import { ListableVariable } from "../../../gen/types.gen.ts"; async function list(opts: GlobalOptions & { json?: boolean }) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); @@ -62,6 +64,7 @@ async function newVariable(opts: GlobalOptions, path: string) { is_secret: false, description: "", }; + await mkdir(dirname(filePath), { recursive: true }); await writeFile(filePath, yamlStringify(template as Record), { flag: "wx", encoding: "utf-8", @@ -70,6 +73,7 @@ async function newVariable(opts: GlobalOptions, path: string) { } async function get(opts: GlobalOptions & { json?: boolean }, path: string) { + if (opts.json) log.setSilent(true); const workspace = await resolveWorkspace(opts); await requireLogin(opts); const v = await wmill.getVariable({ @@ -214,10 +218,10 @@ async function add( undefined, { value, - is_secret: !opts.public && !opts.plainSecrets, + is_secret: !opts.public, description: "", }, - opts.plainSecrets ?? false + true // value from CLI is always plaintext — tell API not to treat it as pre-encrypted ); log.info(colors.bold.underline.green(`Variable ${remotePath} pushed`)); } diff --git a/cli/src/commands/workspace/fork.ts b/cli/src/commands/workspace/fork.ts index 619f29fa2c..92bb38b799 100644 --- a/cli/src/commands/workspace/fork.ts +++ b/cli/src/commands/workspace/fork.ts @@ -129,10 +129,16 @@ async function createWorkspaceFork( const newBranchName = `${WM_FORK_PREFIX}/${clonedBranchName}/${workspaceId}` log.info(`Created forked workspace ${trueWorkspaceId}. To start contributing to your fork, create and push edits to the branch \`${newBranchName}\` by using the command: - + \t`+colors.white(`git checkout -b ${newBranchName}`) + ` - -When doing operations on the forked workspace, it will use the remote setup in gitBranches for the branch it was forked from.`); + +When doing operations on the forked workspace, it will use the remote setup in gitBranches for the branch it was forked from. + +To merge changes back to the parent workspace, you can: + - Use the Merge UI from the forked workspace home page + - Deploy individual items via the Deploy to staging/prod UI + - Use git: ` + colors.white(`git checkout ${clonedBranchName} && git merge ${newBranchName} && wmill sync push`) + ` + See: https://www.windmill.dev/docs/advanced/workspace_forks`); } async function deleteWorkspaceFork( @@ -141,54 +147,69 @@ async function deleteWorkspaceFork( }, name: string, ) { + let forkWorkspaceId: string; + let token: string; + let remote: string; + let hasLocalProfile = false; + + // Try local profile first (existing behavior) const orgWorkspaces = await allWorkspaces(opts.configDir); - const idxOf = orgWorkspaces.findIndex((x) => x.name === name) ; - if (idxOf === -1) { - log.info( - colors.red.bold(`! Workspace profile ${name} does not exist locally`) - ); - log.info("available workspace profiles:"); - await list(opts); - return; - } + const idxOf = orgWorkspaces.findIndex((x) => x.name === name); - const workspace = orgWorkspaces[idxOf]; - - if (!workspace.workspaceId.startsWith(WM_FORK_PREFIX)) { + if (idxOf !== -1) { + const workspace = orgWorkspaces[idxOf]; + if (!workspace.workspaceId.startsWith(WM_FORK_PREFIX)) { throw new Error( `You can only delete forked workspaces where the workspace id starts with \`${WM_FORK_PREFIX}.\` Failed while attempting to delete \`${workspace.workspaceId}\``, ); + } + forkWorkspaceId = workspace.workspaceId; + token = workspace.token; + remote = workspace.remote; + hasLocalProfile = true; + } else { + // Fallback: resolve parent workspace from branch config and construct fork ID + const parentWorkspace = await tryResolveBranchWorkspace(opts); + if (!parentWorkspace) { + throw new Error( + "Could not resolve parent workspace. Make sure you are in a git repo with gitBranches configured in wmill.yaml, or create a local workspace profile for the fork.", + ); + } + forkWorkspaceId = name.startsWith(`${WM_FORK_PREFIX}-`) ? name : `${WM_FORK_PREFIX}-${name}`; + token = parentWorkspace.token; + remote = parentWorkspace.remote; } if (!opts.yes) { - const { Select } = await import("@cliffy/prompt/select"); - const choice = await Select.prompt({ - message: `Are you sure you want to delete the forked workspace with id: \`${workspace.workspaceId}\`? This action will delete the workspace `, - options: [ - { name: "Yes", value: "confirm" }, - { name: "No", value: "cancel" }, - ], - }); + const { Select } = await import("@cliffy/prompt/select"); + const choice = await Select.prompt({ + message: `Are you sure you want to delete the forked workspace \`${forkWorkspaceId}\`?`, + options: [ + { name: "Yes", value: "confirm" }, + { name: "No", value: "cancel" }, + ], + }); - if (choice === "cancel") { - log.info("Operation cancelled"); - return; - } + if (choice === "cancel") { + log.info("Operation cancelled"); + return; + } } - const remote = workspace.remote setClient( - workspace.token, + token, remote.endsWith("/") ? remote.substring(0, remote.length - 1) : remote ); const result = await wmill.deleteWorkspace({ - workspace: workspace.workspaceId + workspace: forkWorkspaceId }); log.info( - colors.green(`✅ Forked workspace '${workspace.workspaceId}' deleted successfully!\n${result}`), + colors.green(`✅ Forked workspace '${forkWorkspaceId}' deleted successfully!\n${result}`), ); - await removeWorkspace(name, false, opts); + if (hasLocalProfile) { + await removeWorkspace(name, false, opts); + } } export { createWorkspaceFork, deleteWorkspaceFork }; diff --git a/cli/src/commands/workspace/workspace.ts b/cli/src/commands/workspace/workspace.ts index d82290e743..1b5d287069 100644 --- a/cli/src/commands/workspace/workspace.ts +++ b/cli/src/commands/workspace/workspace.ts @@ -253,8 +253,12 @@ export async function add( "On that instance and with those credentials, the workspaces that you can access are:" ); const workspaces = await wmill.listWorkspaces(); - for (const workspace of workspaces) { - log.info(`- ${workspace.id} (name: ${workspace.name})`); + if (workspaces.length === 0) { + log.info(" (none)"); + } else { + for (const workspace of workspaces) { + log.info(`- ${workspace.id} (name: ${workspace.name})`); + } } process.exit(1); } @@ -411,31 +415,94 @@ async function whoami(_opts: GlobalOptions) { const whoamiInfo = await wmill.globalWhoami(); log.info(JSON.stringify(whoamiInfo, null, 2)); const activeName = await getActiveWorkspaceName(_opts); - log.info("Active: " + colors.green.bold(activeName || "none")); + const { getCurrentGitBranch, getOriginalBranchForWorkspaceForks } = await import("../../utils/git.ts"); + const branch = getCurrentGitBranch(); + const originalBranch = branch ? getOriginalBranchForWorkspaceForks(branch) : null; + if (originalBranch) { + const { resolveWorkspace } = await import("../../core/context.ts"); + try { + const ws = await resolveWorkspace(_opts); + log.info("Active: " + colors.green.bold(ws.workspaceId) + ` (fork of ${activeName || "unknown"})`); + } catch { + log.info("Active: " + colors.green.bold(activeName || "none") + " (fork branch)"); + } + } else { + log.info("Active: " + colors.green.bold(activeName || "none")); + } } async function listRemote(_opts: GlobalOptions) { - const { resolveWorkspace } = await import("../../core/context.ts"); - const workspace = await resolveWorkspace(_opts); - await requireLogin(_opts); + let remote: string; + + if (_opts.baseUrl && _opts.token && !_opts.workspace) { + // Allow listing workspaces with just --base-url and --token (no --workspace needed) + const { setClient } = await import("../../core/client.ts"); + remote = new URL(_opts.baseUrl).toString(); + setClient(_opts.token, remote.replace(/\/$/, "")); + } else { + const { resolveWorkspace } = await import("../../core/context.ts"); + const workspace = await resolveWorkspace(_opts); + await requireLogin(_opts); + remote = workspace.remote; + } + const userWorkspaces = await wmill.listUserWorkspaces(); + const hasForks = userWorkspaces.workspaces.some((x) => x.parent_workspace_id); + const headers = hasForks + ? ["id", "name", "username", "fork of", "disabled"] + : ["id", "name", "username", "disabled"]; + new Table() - .header(["id", "name", "username", "disabled"]) + .header(headers) .padding(2) .border(true) .body( - userWorkspaces.workspaces.map((x) => [ + userWorkspaces.workspaces.map((x) => { + const row = [ + x.id, + x.name, + x.username, + ]; + if (hasForks) row.push(x.parent_workspace_id ?? "-"); + row.push(x.disabled ? colors.red("true") : "false"); + return row; + }) + ) + .render(); + + log.info(`Remote: ${colors.bold(remote)}`); + log.info(`Logged in as: ${colors.green.bold(userWorkspaces.email)}`); +} + +async function listForks(_opts: GlobalOptions) { + const { resolveWorkspace } = await import("../../core/context.ts"); + const workspace = await resolveWorkspace(_opts); + await requireLogin(_opts); + + const userWorkspaces = await wmill.listUserWorkspaces(); + const forks = userWorkspaces.workspaces.filter((w) => w.parent_workspace_id); + + if (forks.length === 0) { + log.info("No forked workspaces found."); + return; + } + + new Table() + .header(["id", "name", "fork of", "username"]) + .padding(2) + .border(true) + .body( + forks.map((x) => [ x.id, x.name, + x.parent_workspace_id ?? "", x.username, - x.disabled ? colors.red("true") : "false", ]) ) .render(); log.info(`Remote: ${colors.bold(workspace.remote)}`); - log.info(`Logged in as: ${colors.green.bold(userWorkspaces.email)}`); } export async function getActiveWorkspaceOrFallback(opts: GlobalOptions) { @@ -566,8 +633,11 @@ const command = new Command() .command("list-remote") .description("List workspaces on the remote server that you have access to") .action(listRemote as any) + .command("list-forks") + .description("List forked workspaces on the remote server") + .action(listForks as any) .command("bind") - .description("Bind the current Git branch to the active workspace") + .description("Bind the current Git branch to the active workspace. This adds the branch to gitBranches in wmill.yaml so sync operations use the correct workspace for each branch.") .option("--branch, --env ", "Specify branch/environment (defaults to current)") .action((opts) => bind(opts as any, true)) .command("unbind") diff --git a/cli/src/core/conf.ts b/cli/src/core/conf.ts index e72aa14414..826b207323 100644 --- a/cli/src/core/conf.ts +++ b/cli/src/core/conf.ts @@ -57,6 +57,7 @@ export interface SyncOptions { variables?: string[]; resources?: string[]; triggers?: string[]; + schedules?: string[]; folders?: string[]; settings?: boolean; }; @@ -70,6 +71,7 @@ export interface SyncOptions { variables?: string[]; resources?: string[]; triggers?: string[]; + schedules?: string[]; folders?: string[]; settings?: boolean; }; @@ -83,6 +85,7 @@ export interface SyncOptions { variables?: string[]; resources?: string[]; triggers?: string[]; + schedules?: string[]; folders?: string[]; settings?: boolean; }; @@ -96,6 +99,7 @@ export interface SyncOptions { variables?: string[]; resources?: string[]; triggers?: string[]; + schedules?: string[]; folders?: string[]; settings?: boolean; }; @@ -191,15 +195,18 @@ export function getWmillYamlPath(): string | null { return findWmillYaml(); } -export async function readConfigFile(): Promise { +export async function readConfigFile(opts?: { warnIfMissing?: boolean }): Promise { + const warnIfMissing = opts?.warnIfMissing ?? true; try { // First, try to find wmill.yaml recursively const wmillYamlPath = findWmillYaml(); if (!wmillYamlPath) { - log.warn( - "No wmill.yaml found. Use 'wmill init' to bootstrap it." - ); + if (warnIfMissing) { + log.warn( + "No wmill.yaml found. Use 'wmill init' to bootstrap it." + ); + } return {}; } diff --git a/cli/src/core/context.ts b/cli/src/core/context.ts index 2b7a2266e2..b65cdcab49 100644 --- a/cli/src/core/context.ts +++ b/cli/src/core/context.ts @@ -262,8 +262,8 @@ export async function tryResolveBranchWorkspace( } } - // Read wmill.yaml to check for branch workspace configuration - const config = await readConfigFile(); + // Read wmill.yaml to check for branch workspace configuration (silent — just probing) + const config = await readConfigFile({ warnIfMissing: false }); const branchConfig = config.gitBranches?.[currentBranch]; // Check if branch has workspace configuration @@ -366,7 +366,7 @@ export async function tryResolveBranchWorkspace( selectedProfile.name = `${selectedProfile.name}/${workspaceIdIfForked}`; selectedProfile.workspaceId = workspaceIdIfForked; log.info( - `Inferred workspace id \`${workspaceId}\` from branch name because this is a workspace fork branch (\`${rawBranch}\`). ` + `Using fork workspace \`${workspaceIdIfForked}\` (parent: \`${workspaceId}\`) from branch \`${rawBranch}\`` ); } @@ -458,15 +458,16 @@ export async function resolveWorkspace( const branch = branchOverride ?? getCurrentGitBranch(); // Try explicit workspace flag first (should override branch-based resolution). Unless it's a - // forked workspace, that we detect through the branch name (only when not using branchOverride) + // forked workspace, that we detect through the branch name (only when not using branchOverride + // and --workspace was not explicitly provided) const res = await tryResolveWorkspace(opts); if (!res.isError) { const workspace = (res as { isError: false; value: Workspace }).value; - if (branchOverride || !branch || !branch.startsWith(WM_FORK_PREFIX)) { + if (branchOverride || opts.workspace || !branch || !branch.startsWith(WM_FORK_PREFIX)) { return workspace; } else { log.info( - `Found an active workspace \`${workspace.name}\` but the branch name indicates this is a forked workspace. Ignoring active workspace and trying to resolve the correct workspace from the branch name \`${branch}\`` + `Found an active workspace \`${workspace.name}\` but the branch name indicates this is a forked workspace. Ignoring active workspace and trying to resolve the correct workspace from the branch name \`${branch}\`. Use --workspace to override.` ); } } else if (opts.workspace) { @@ -549,7 +550,7 @@ export async function resolveWorkspace( } // If everything failed, show error - log.info(colors.red.bold("No workspace given and no default set.")); + log.info(colors.red.bold("No workspace given and no default set. Run 'wmill workspace add' to configure one.")); return process.exit(-1); } diff --git a/cli/src/core/log.ts b/cli/src/core/log.ts index d7bed9a0d4..034e13e3e1 100644 --- a/cli/src/core/log.ts +++ b/cli/src/core/log.ts @@ -1,4 +1,5 @@ let logLevel: "DEBUG" | "INFO" | "WARN" | "ERROR" = "INFO"; +let silentMode = false; const levels = { DEBUG: 0, INFO: 1, WARN: 2, ERROR: 3 }; @@ -6,19 +7,25 @@ export function setup(level: "DEBUG" | "INFO" | "WARN" | "ERROR") { logLevel = level; } +export function setSilent(silent: boolean) { + silentMode = silent; +} + export function debug(msg: unknown) { if (levels[logLevel] <= levels.DEBUG) console.log(`\x1b[90m${String(msg)}\x1b[39m`); } export function info(msg: unknown) { + if (silentMode) return; console.log(`\x1b[34m${String(msg)}\x1b[39m`); } export function warn(msg: unknown) { + if (silentMode) return; console.log(`\x1b[33m${String(msg)}\x1b[39m`); } export function error(msg: unknown) { - console.log(`\x1b[31m${String(msg)}\x1b[39m`); + console.error(`\x1b[31m${String(msg)}\x1b[39m`); } diff --git a/cli/src/core/specific_items.ts b/cli/src/core/specific_items.ts index cfd806d9c6..dca403e905 100644 --- a/cli/src/core/specific_items.ts +++ b/cli/src/core/specific_items.ts @@ -8,6 +8,7 @@ export interface SpecificItemsConfig { variables?: string[]; resources?: string[]; triggers?: string[]; + schedules?: string[]; folders?: string[]; settings?: boolean; } @@ -17,6 +18,7 @@ function getBranchSpecificTypes() { return { variable: '.variable.yaml', resource: '.resource.yaml', + schedule: '.schedule.yaml', // Generate trigger patterns from the list ...Object.fromEntries( TRIGGER_TYPES.map(t => [`${t}_trigger`, `.${t}_trigger.yaml`]) @@ -31,6 +33,13 @@ function isTriggerFile(path: string): boolean { return TRIGGER_TYPES.some(type => path.endsWith(`.${type}_trigger.yaml`)); } +/** + * Check if a path is a schedule file + */ +function isScheduleFile(path: string): boolean { + return path.endsWith('.schedule.yaml'); +} + /** * Extract the file type suffix from a path */ @@ -53,7 +62,7 @@ function getFileTypeSuffix(path: string): string | null { * Build regex pattern for all supported yaml file types */ function buildYamlTypePattern(): string { - const basicTypes = ['variable', 'resource']; + const basicTypes = ['variable', 'resource', 'schedule']; const triggerTypes = TRIGGER_TYPES.map(t => `${t}_trigger`); return `((${basicTypes.join('|')})|(${triggerTypes.join('|')}))`; } @@ -100,6 +109,9 @@ export function getSpecificItemsForCurrentBranch(config: SyncOptions, branchOver if (commonItems?.triggers) { merged.triggers = [...commonItems.triggers]; } + if (commonItems?.schedules) { + merged.schedules = [...commonItems.schedules]; + } if (commonItems?.folders) { merged.folders = [...commonItems.folders]; } @@ -117,6 +129,9 @@ export function getSpecificItemsForCurrentBranch(config: SyncOptions, branchOver if (branchItems?.triggers) { merged.triggers = [...(merged.triggers || []), ...branchItems.triggers]; } + if (branchItems?.schedules) { + merged.schedules = [...(merged.schedules || []), ...branchItems.schedules]; + } if (branchItems?.folders) { merged.folders = [...(merged.folders || []), ...branchItems.folders]; } @@ -157,6 +172,10 @@ export function isItemTypeConfigured(path: string, specificItems: SpecificItemsC return specificItems.triggers !== undefined; } + if (isScheduleFile(path)) { + return specificItems.schedules !== undefined; + } + if (path.endsWith('/folder.meta.yaml')) { return specificItems.folders !== undefined; } @@ -194,6 +213,11 @@ export function isSpecificItem(path: string, specificItems: SpecificItemsConfig return specificItems.triggers ? matchesPatterns(path, specificItems.triggers) : false; } + // Check for schedule files + if (isScheduleFile(path)) { + return specificItems.schedules ? matchesPatterns(path, specificItems.schedules) : false; + } + // Check for folder meta files if (path.endsWith('/folder.meta.yaml')) { if (specificItems.folders) { diff --git a/cli/src/guidance/skills.ts b/cli/src/guidance/skills.ts index 45279e631f..053e491122 100644 --- a/cli/src/guidance/skills.ts +++ b/cli/src/guidance/skills.ts @@ -740,7 +740,7 @@ workflow(fn: (...args: any[]) => Promise): void * await step("notify", () => sendEmail(urls.approvalPage)); * const { value, approver } = await waitForApproval({ timeout: 3600 }); */ -waitForApproval(options?: { timeout?: number; form?: object; }): PromiseLike<{ value: any; approver: string; approved: boolean }> +waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }> /** * Process items in parallel with optional concurrency control. @@ -1403,7 +1403,7 @@ workflow(fn: (...args: any[]) => Promise): void * await step("notify", () => sendEmail(urls.approvalPage)); * const { value, approver } = await waitForApproval({ timeout: 3600 }); */ -waitForApproval(options?: { timeout?: number; form?: object; }): PromiseLike<{ value: any; approver: string; approved: boolean }> +waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }> /** * Process items in parallel with optional concurrency control. @@ -2129,7 +2129,7 @@ workflow(fn: (...args: any[]) => Promise): void * await step("notify", () => sendEmail(urls.approvalPage)); * const { value, approver } = await waitForApproval({ timeout: 3600 }); */ -waitForApproval(options?: { timeout?: number; form?: object; }): PromiseLike<{ value: any; approver: string; approved: boolean }> +waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }> /** * Process items in parallel with optional concurrency control. @@ -3069,7 +3069,7 @@ workflow(fn: (...args: any[]) => Promise): void * await step("notify", () => sendEmail(urls.approvalPage)); * const { value, approver } = await waitForApproval({ timeout: 3600 }); */ -waitForApproval(options?: { timeout?: number; form?: object; }): PromiseLike<{ value: any; approver: string; approved: boolean }> +waitForApproval(options?: { timeout?: number; form?: object; selfApproval?: boolean; }): PromiseLike<{ value: any; approver: string; approved: boolean }> /** * Process items in parallel with optional concurrency control. @@ -4078,12 +4078,17 @@ async def sleep(seconds: int) # # Returns a dict with \`\`value\`\` (form data), \`\`approver\`\`, and \`\`approved\`\`. # +# Args: +# timeout: Approval timeout in seconds (default 1800). +# form: Optional form schema for the approval page. +# self_approval: Whether the user who triggered the flow can approve it (default True). +# # Example:: # # urls = await step("urls", lambda: get_resume_urls()) # await step("notify", lambda: send_email(urls["approvalPage"])) # result = await wait_for_approval(timeout=3600) -async def wait_for_approval(timeout: int = 1800, form: dict | None = None) -> dict +async def wait_for_approval(timeout: int = 1800, form: dict | None = None, self_approval: bool = True) -> dict # Process items in parallel with optional concurrency control. # @@ -4354,7 +4359,7 @@ Reference a specific resource using \`$res:\` prefix: ## OpenFlow Schema -{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_use":{"type":"boolean","description":"If true, this step's result is deleted after use to save memory"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_images":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of image references for vision-capable models.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'images/photo.jpg' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, +{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \\"yellow\\", \\"#ffff00\\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution \\u2014 modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_use":{"type":"boolean","description":"If true, this step's result is deleted after use to save memory"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticProviderTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind","mapping":{"off":"#/components/schemas/MemoryOff","auto":"#/components/schemas/MemoryAuto","manual":"#/components/schemas/MemoryManual"}}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type","mapping":{"static":"#/components/schemas/StaticMemoryTransform","javascript":"#/components/schemas/JavascriptTransform","ai":"#/components/schemas/AiTransform"}}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type","mapping":{"rawscript":"#/components/schemas/RawScript","script":"#/components/schemas/PathScript","flow":"#/components/schemas/PathFlow","forloopflow":"#/components/schemas/ForloopFlow","whileloopflow":"#/components/schemas/WhileloopFlow","branchone":"#/components/schemas/BranchOne","branchall":"#/components/schemas/BranchAll","identity":"#/components/schemas/Identity","aiagent":"#/components/schemas/AiAgent"}}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}],"discriminator":{"propertyName":"tool_type","mapping":{"flowmodule":"#/components/schemas/FlowModuleTool","mcp":"#/components/schemas/McpToolValue","websearch":"#/components/schemas/WebsearchToolValue"}}},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\\nValid values: 'text' (default) - plain text response, 'image' - image generation\\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\\nFormat: Array<{ bucket: string, key: string }> - S3 object references\\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\\nRange: 0.0 to 2.0 (provider-dependent)\\n- 0.0 = deterministic, focused responses\\n- 0.7 = balanced (common default)\\n- 1.0+ = more creative/random\\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}`, "raw-app": `--- name: raw-app description: MUST use when creating raw apps. @@ -4602,18 +4607,9 @@ Tell the user they can run these commands (do NOT run them yourself): | \`wmill app dev\` | Start dev server with live reload | | \`wmill app generate-agents\` | Refresh AGENTS.md and DATATABLES.md | | \`wmill app generate-locks\` | Generate lock files for backend runnables | -| \`wmill sync push --extra-includes "f//.raw_app/**" --yes\` | Deploy this specific raw app to Windmill (never do a blanket \`wmill sync push\`) | +| \`wmill sync push\` | Deploy app to Windmill | | \`wmill sync pull\` | Pull latest from Windmill | -## Svelte 5 Event Handling - -When building Svelte 5 raw apps, be aware of event delegation: - -- The Svelte runtime version in \`node_modules/svelte\` **must match** the compiler version used by \`wmill sync push\`. If you get \`$.delegated is undefined\` errors at runtime, run \`npm install svelte@latest\` in the raw app folder and re-push. -- \`onclick\` on \`
\`, \`\`, and other non-interactive elements uses Svelte's event delegation system. If the runtime doesn't support it, you'll get errors. -- \`onclick\` on \` {/snippet} {#snippet content()} -
+
Add a new user - Email - + {#if isServiceAccount} + Username + + {:else} + Email + - {#if !automateUsernameCreation} - Username - + {#if !automateUsernameCreation} + Username + + {/if} {/if} Role @@ -112,6 +135,13 @@ tooltip="An admin has full control over a specific Windmill workspace, including the ability to manage users, edit entities, and control permissions within the workspace." {item} /> + {/snippet} diff --git a/frontend/src/lib/components/ApiConnectForm.svelte b/frontend/src/lib/components/ApiConnectForm.svelte index 636a20f290..7d10db1d48 100644 --- a/frontend/src/lib/components/ApiConnectForm.svelte +++ b/frontend/src/lib/components/ApiConnectForm.svelte @@ -16,6 +16,7 @@ import BedrockCredentialsCheck from './BedrockCredentialsCheck.svelte' import { isCloudHosted } from '$lib/cloud' import ResourceGen from './copilot/ResourceGen.svelte' + import SyncResourceTypes from './SyncResourceTypes.svelte' interface Props { resourceType: string @@ -25,6 +26,7 @@ isValid?: boolean linkedSecretCandidates?: string[] | undefined description?: string | undefined + onSynced?: () => void } let { @@ -34,7 +36,8 @@ linkedSecrets = $bindable([]), isValid = $bindable(true), linkedSecretCandidates = undefined, - description = $bindable(undefined) + description = $bindable(undefined), + onSynced = undefined }: Props = $props() let schema = $state(emptySchema()) @@ -227,6 +230,7 @@ >No corresponding resource type found in your workspace for {resourceType}. Define the value in JSON directly

+ {/if} {#if notFound || viewJsonSchema} {#if !emptyString(error)} {#snippet actions()}
diff --git a/frontend/src/lib/components/AppConnectInner.svelte b/frontend/src/lib/components/AppConnectInner.svelte index d7b37fcdd9..68698a05db 100644 --- a/frontend/src/lib/components/AppConnectInner.svelte +++ b/frontend/src/lib/components/AppConnectInner.svelte @@ -1,15 +1,14 @@ {#if !express} @@ -718,20 +715,16 @@ {/each} {/if}
- {#if $superadmin} - - {#if hubRtSync.status === 'error'} - - Error syncing resource types : {JSON.stringify(hubRtSync.error)} - - {/if} - {/if} +
+ { + connectsManual = undefined + await loadResourceTypes() + connects = undefined + await loadConnects() + }} + /> +
{:else if step == 2 && manual}
+ {#if resourceTypeNotFound} +
+

+ Resource type '{resourceType}' not found in your workspace +

+ +
+ {/if} {#key resourceTypeInfo} {/key}
diff --git a/frontend/src/lib/components/ArgInfo.svelte b/frontend/src/lib/components/ArgInfo.svelte index c99f994728..a1cfdf6276 100644 --- a/frontend/src/lib/components/ArgInfo.svelte +++ b/frontend/src/lib/components/ArgInfo.svelte @@ -74,7 +74,15 @@ +{:else if isString(value) && value.startsWith('$jsonvar:')} + diff --git a/frontend/src/lib/components/ArgInput.svelte b/frontend/src/lib/components/ArgInput.svelte index 197275fbc1..56d0d4b0fa 100644 --- a/frontend/src/lib/components/ArgInput.svelte +++ b/frontend/src/lib/components/ArgInput.svelte @@ -495,6 +495,8 @@ let { debounced, clearDebounce } = debounce(() => compareValues(value), 50) let inputCat = $derived(computeInputCat(type, format, itemsType?.type, enum_, contentEncoding)) + let isNonStringSecret = $derived((password || extra?.['password'] == true) && type === 'object') + let displayJsonToggleHeader = $derived( displayHeader && inputCat === 'list' && @@ -558,6 +560,12 @@ class="text-accent underline font-normal" onclick={() => variableEditor?.editVariable?.(value.slice(5))}>{value.slice(5)} + {:else if value && typeof value == 'string' && value?.startsWith('$jsonvar:')} + Linked to variable {/if}
{/if} @@ -1436,12 +1444,13 @@ {:else} {/if} {:else} - + {/if} {:else} {#key extra?.['minRows']} @@ -1487,6 +1496,18 @@ {@render actions?.()}
+ {#if isNonStringSecret} + {#if typeof value === 'string' && value.startsWith('$jsonvar:')} +
+ Sensitive — stored as secret: {value.slice('$jsonvar:'.length)} +
+ {:else} +
Sensitive — will be stored as secret on submit
+ {/if} + {/if} + {#if !compact || (error && error != '')}
{#if disabled || error === ''} diff --git a/frontend/src/lib/components/DedicatedWorkersSelector.svelte b/frontend/src/lib/components/DedicatedWorkersSelector.svelte index 1ed9d36a7c..94086982fb 100644 --- a/frontend/src/lib/components/DedicatedWorkersSelector.svelte +++ b/frontend/src/lib/components/DedicatedWorkersSelector.svelte @@ -1,6 +1,22 @@ +{#snippet depBadge(dep: string)} + {#if existingDeps.has(dep)} + + {dep} + + + {:else} + + Workspace dependency '{dep}' not found. Create it in workspace settings to enable shared + runners. + + + + {dep} + + {/if} +{/snippet} + +{#snippet tagRow(tag: string, info: SelectedTagInfo | undefined)} +
+
+ {#if info?.type === 'flow' && info.runners && info.runners.length > 0} + + {:else} +
+ {/if} +
+ {#if info} + {#if info.type === 'flow'} + + {:else} + + {/if} + {info.path} + ({info.workspace}) + + {#if !tagRunnerGroup.has(tag)} + {#if info.workspaceDeps} + {#each info.workspaceDeps as dep} + {@render depBadge(dep)} + {/each} + {/if} + {#if info.type === 'flow' && info.runners} + + {info.runners.length} runner{info.runners.length !== 1 ? 's' : ''} + + {:else if info.language} + {info.language} + {/if} + {/if} + {:else} + {tag} + {/if} +
+ {#if !disabled} + + {/if} +
+ + {#if info?.type === 'flow' && info.expanded && info.runners} +
+ {#each info.runners as runner (runner.stepId)} +
+ {runner.stepId} + {#if runner.stepSummary} + {runner.stepSummary} + {/if} + + {runner.isInline ? runner.language : runner.scriptPath} + +
+ {/each} +
+ {/if} +
+{/snippet} +
- {#if selectedTags.length > 0}
-
- {#each selectedTags as tag (tag)} - {@const info = selectedTagsInfo.get(tag)} +
+ + {#each runnerGroups as group (`${group.depName}:${group.language}`)}
-
- {#if info?.type === 'flow' && info.runners && info.runners.length > 0} - - {:else} -
- {/if} -
-
- {#if info} - {#if info.type === 'flow'} - - {:else} - - {/if} - {info.path} - ({info.workspace}) - {#if info.type === 'flow' && info.runners} - - {info.runners.length} runner{info.runners.length !== 1 ? 's' : ''} - - {:else if info.type === 'script'} - 1 runner - {/if} - {:else} - {tag} - {/if} -
-
- {#if !disabled} - - {/if} +
+ + Shared runner + + {@render depBadge(group.depName)} + {group.language}
+
+ {#each group.tags as tag (tag)} + {@const info = selectedTagsInfo.get(tag)} + {@render tagRow(tag, info)} + {/each} +
+
+ {/each} - {#if info?.type === 'flow' && info.expanded && info.runners} -
- {#each info.runners as runner (runner.stepId)} -
- {runner.stepId} - {#if runner.stepSummary} - {runner.stepSummary} - {/if} - - {runner.isInline ? runner.language : runner.scriptPath} - -
- {/each} -
+ + {#each standaloneTags as tag (tag)} + {@const info = selectedTagsInfo.get(tag)} +
+ {#if info?.type === 'flow'} + + {:else} + + {/if} + {info?.path ?? tag} + ({info?.workspace ?? ''}) + + {#if info?.workspaceDeps} + {#each info.workspaceDeps as dep} + {@render depBadge(dep)} + {/each} + {/if} + {#if info?.type === 'flow' && info.runners} + + {info.runners.length} runner{info.runners.length !== 1 ? 's' : ''} + + {:else if info?.language} + {info.language} + {/if} + {#if !disabled} + {/if}
{/each} @@ -585,15 +824,18 @@ {/if}
- {runnable.displayName} + {runnable.displayName} {#if runnable.type === 'flow' && runnable.runners} {runnable.runners.length} {/if} - + {#if runnable.workspaceDeps} + {#each runnable.workspaceDeps as dep} + {@render depBadge(dep)} + {/each} + {/if} + {runnable.type === 'flow' ? 'flow' : runnable.language} diff --git a/frontend/src/lib/components/DefaultTagsInner.svelte b/frontend/src/lib/components/DefaultTagsInner.svelte index ab2d403ec8..e57afd9659 100644 --- a/frontend/src/lib/components/DefaultTagsInner.svelte +++ b/frontend/src/lib/components/DefaultTagsInner.svelte @@ -4,7 +4,11 @@ import { SettingService, WorkerService, WorkspaceService } from '$lib/gen' import { sendUserToast } from '$lib/toast' import { enterpriseLicense, superadmin } from '$lib/stores' - import { DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING } from '$lib/consts' + import { + DEFAULT_TAGS_PER_WORKSPACE_SETTING, + DEFAULT_TAGS_WORKSPACES_SETTING, + PREVIEW_TAGS_OVERRIDE_SETTING + } from '$lib/consts' import Toggle from './Toggle.svelte' import MultiSelect from './select/MultiSelect.svelte' import { safeSelectItems } from './select/utils.svelte' @@ -22,16 +26,19 @@ let defaultTags = $state(undefined) let limitToWorkspaces = $state(false) + let previewTagsOverride = $state(false) // Change detection let originalDefaultTagPerWorkspace = $state(defaultTagPerWorkspace) let originalDefaultTagWorkspaces = $state(defaultTagWorkspaces) + let originalPreviewTagsOverride = $state(false) // Detect changes let hasChanges = $derived( originalDefaultTagPerWorkspace !== defaultTagPerWorkspace || JSON.stringify($state.snapshot(originalDefaultTagWorkspaces)?.sort() || []) !== - JSON.stringify($state.snapshot(defaultTagWorkspaces)?.sort() || []) + JSON.stringify($state.snapshot(defaultTagWorkspaces)?.sort() || []) || + originalPreviewTagsOverride !== previewTagsOverride ) let workspaces: string[] = $state([]) @@ -47,6 +54,11 @@ key: DEFAULT_TAGS_WORKSPACES_SETTING })) as any) ?? [] limitToWorkspaces = defaultTagWorkspaces ? defaultTagWorkspaces.length > 0 : false + previewTagsOverride = + ((await SettingService.getGlobal({ + key: PREVIEW_TAGS_OVERRIDE_SETTING + })) as any) ?? false + originalPreviewTagsOverride = previewTagsOverride } catch (err) { sendUserToast(`Could not load default tags: ${err}`, true) } @@ -68,10 +80,17 @@ : undefined } }) + await SettingService.setGlobal({ + key: PREVIEW_TAGS_OVERRIDE_SETTING, + requestBody: { + value: previewTagsOverride + } + }) // Update original state after save originalDefaultTagPerWorkspace = defaultTagPerWorkspace originalDefaultTagWorkspaces = [...(defaultTagWorkspaces || [])] + originalPreviewTagsOverride = previewTagsOverride loadDefaultTags() sendUserToast('Saved') @@ -146,6 +165,18 @@ /> {/if} {/if} +
+ +
@@ -168,6 +199,17 @@
{/each} + {#if previewTagsOverride} +
+
+ preview +
+
→
+
+ {defaultTagPerWorkspace ? 'preview-$workspace' : 'preview'} +
+
+ {/if}
{/if} diff --git a/frontend/src/lib/components/DeployWorkspace.svelte b/frontend/src/lib/components/DeployWorkspace.svelte index 9f1fd71146..3645bd587b 100644 --- a/frontend/src/lib/components/DeployWorkspace.svelte +++ b/frontend/src/lib/components/DeployWorkspace.svelte @@ -262,13 +262,25 @@ return getTriggerDependency(additionalInformation.triggers.kind, path, $workspaceStore!) } throw new Error('Missing trigger information') + } else if (kind == 'script') { + const imports = await WorkspaceService.getImports({ + workspace: $workspaceStore!, + importerPath: path + }) + return imports.map((importedPath) => ({ kind: 'script' as Kind, path: importedPath })) } return [] } let toProcess = [{ kind, path }] + let processedSet = new Set() let processed: { kind: Kind; path: string }[] = [] while (toProcess.length > 0) { const { kind, path } = toProcess.pop()! + const key = `${kind}:${path}` + if (processedSet.has(key)) { + continue + } + processedSet.add(key) toProcess.push(...(await rec(kind, path))) processed.push({ kind, path }) } diff --git a/frontend/src/lib/components/Dev.svelte b/frontend/src/lib/components/Dev.svelte index 79f3be91f1..d9ec19a211 100644 --- a/frontend/src/lib/components/Dev.svelte +++ b/frontend/src/lib/components/Dev.svelte @@ -34,6 +34,7 @@ import type { FlowEditorContext, FlowInput, FlowInputEditorState } from './flows/types' import { SelectionManager } from './graph/selectionUtils.svelte' import { NoteEditor, setNoteEditorContext } from './graph/noteEditor.svelte' + import { GroupEditor, setGroupEditorContext } from './graph/groupEditor.svelte' import { dfs } from './flows/dfs' import { loadSchemaFromModule } from './flows/flowInfers' import { CornerDownLeft, Play } from 'lucide-svelte' @@ -561,6 +562,11 @@ }) setNoteEditorContext(noteEditor) + // Set up GroupEditor context for group editing capabilities + const groupEditor = new GroupEditor(flowStore) + let canCreateGroup = $state({ val: false }) + setGroupEditorContext(groupEditor, canCreateGroup) + let lastSent: OpenFlow | undefined = undefined function updateFlow(flow: OpenFlow) { if (lockChanges) { diff --git a/frontend/src/lib/components/EditableSchemaForm.svelte b/frontend/src/lib/components/EditableSchemaForm.svelte index 989ce6a2fe..85b8abf753 100644 --- a/frontend/src/lib/components/EditableSchemaForm.svelte +++ b/frontend/src/lib/components/EditableSchemaForm.svelte @@ -51,6 +51,7 @@ noPreview?: boolean jsonEnabled?: boolean isAppInput?: boolean + showSensitiveToggle?: boolean displayWebhookWarning?: boolean onlyMaskPassword?: boolean editTab: @@ -95,6 +96,7 @@ noPreview = false, jsonEnabled = true, isAppInput = false, + showSensitiveToggle = false, displayWebhookWarning = false, onlyMaskPassword = false, editTab, @@ -297,7 +299,9 @@ } const editTabDefaultSize = untrack(() => noPreview) ? 100 : 50 - editPanelSize = untrack(() => editTab) ? (untrack(() => editPanelInitialSize) ?? editTabDefaultSize) : 0 + editPanelSize = untrack(() => editTab) + ? (untrack(() => editPanelInitialSize) ?? editTabDefaultSize) + : 0 let inputPanelSize = $state(100 - editPanelSize) let editPanelSizeSmooth = tweened(editPanelSize, { duration: 150 @@ -677,6 +681,7 @@ bind:order={schema.properties[argName].order} {isFlowInput} {isAppInput} + {showSensitiveToggle} > {#snippet typeeditor()} {#if isFlowInput || isAppInput} diff --git a/frontend/src/lib/components/EditorBar.svelte b/frontend/src/lib/components/EditorBar.svelte index 6ff3eeaee2..e11731fe63 100644 --- a/frontend/src/lib/components/EditorBar.svelte +++ b/frontend/src/lib/components/EditorBar.svelte @@ -87,6 +87,8 @@ | 'powershell' | 'bunnative' | 'claudesandbox' + | 'wac_python' + | 'wac_typescript' collabMode?: boolean collabLive?: boolean collabUsers?: { name: string }[] @@ -746,7 +748,7 @@ JsonNode ${windmillPathToCamelCaseName(path)} = JsonNode.Parse(await client.GetS startIcon={{ icon: Settings }} target="_blank" variant="accent" - href="{base}/workspace_settings?tab=windmill_lfs" + href="{base}/workspace_settings?tab=ducklake" > Go to settings diff --git a/frontend/src/lib/components/FirstStepInputs.svelte b/frontend/src/lib/components/FirstStepInputs.svelte index 30399e20f7..cf936b64f0 100644 --- a/frontend/src/lib/components/FirstStepInputs.svelte +++ b/frontend/src/lib/components/FirstStepInputs.svelte @@ -85,7 +85,6 @@ > flowStore.val.value.modules }) diff --git a/frontend/src/lib/components/FlowGraphDiffViewer.svelte b/frontend/src/lib/components/FlowGraphDiffViewer.svelte index f66a582eb5..62e5b208c2 100644 --- a/frontend/src/lib/components/FlowGraphDiffViewer.svelte +++ b/frontend/src/lib/components/FlowGraphDiffViewer.svelte @@ -140,6 +140,7 @@ - import type { FlowModule, FlowValue } from '$lib/gen' + import type { FlowModule, FlowValue, TriggersCount } from '$lib/gen' + import type { TriggerContext } from '$lib/components/triggers' + import { Triggers } from '$lib/components/triggers/triggers.svelte' - import { createEventDispatcher } from 'svelte' + import { createEventDispatcher, hasContext, setContext } from 'svelte' + import { writable } from 'svelte/store' import { twMerge } from 'tailwind-merge' import FlowGraphViewerStep from './FlowGraphViewerStep.svelte' @@ -9,24 +12,26 @@ import { dfs } from './flows/dfs' import { workspaceStore } from '$lib/stores' - interface Props { flow: { - summary: string - description?: string - value: FlowValue - schema?: any - path?: string - }; - overflowAuto?: boolean; - noSide?: boolean; - download?: boolean; - noGraph?: boolean; - triggerNode?: boolean; - stepDetail?: FlowModule | string | undefined; - workspace?: string | undefined; - minHeight?: number; - noBorder?: boolean; + summary: string + description?: string + value: FlowValue + schema?: any + path?: string + } + overflowAuto?: boolean + noSide?: boolean + download?: boolean + noGraph?: boolean + triggerNode?: boolean + stepDetail?: FlowModule | string | undefined + workspace?: string | undefined + minHeight?: number + noBorder?: boolean + hideDefaultInputs?: boolean + provideTriggerContext?: boolean + fillAvailableHeight?: boolean } let { @@ -39,16 +44,33 @@ stepDetail = $bindable(undefined), workspace = $workspaceStore, minHeight = 400, - noBorder = false - }: Props = $props(); + noBorder = false, + hideDefaultInputs = false, + provideTriggerContext = false, + fillAvailableHeight = false + }: Props = $props() + + let availableHeight = $state(0) + + if (provideTriggerContext && !hasContext('TriggerContext')) { + const triggersCount = writable(undefined) + setContext('TriggerContext', { + triggersCount, + simplifiedPoll: writable(false), + showCaptureHint: writable(undefined), + triggersState: new Triggers() + }) + } const dispatch = createEventDispatcher() -
+
{#if !noGraph}
@@ -58,12 +80,13 @@ cache={flow.value.cache_ttl !== undefined} path={flow?.path} {download} - {minHeight} + minHeight={fillAvailableHeight ? Math.max(minHeight, availableHeight) : minHeight} {workspace} modules={flow?.value?.modules} failureModule={flow?.value?.failure_module} preprocessorModule={flow?.value?.preprocessor_module} notes={flow?.value?.notes} + groups={flow?.value?.groups} onSelect={(nodeId) => { if (nodeId === 'Trigger') { dispatch('triggerDetail') @@ -81,14 +104,16 @@ />
{/if} - {#if !noSide} + {#if !noSide && !(hideDefaultInputs && stepDetail == undefined)} {/if}
diff --git a/frontend/src/lib/components/FlowGraphViewerStep.svelte b/frontend/src/lib/components/FlowGraphViewerStep.svelte index 8c560abef0..5ca73d7d1f 100644 --- a/frontend/src/lib/components/FlowGraphViewerStep.svelte +++ b/frontend/src/lib/components/FlowGraphViewerStep.svelte @@ -23,9 +23,10 @@ schema?: any | undefined stepDetail?: FlowModule | string | undefined jobScriptHash?: string | undefined + hideDefaultInputs?: boolean } - let { schema = undefined, stepDetail = undefined, jobScriptHash = undefined }: Props = $props() + let { schema = undefined, stepDetail = undefined, jobScriptHash = undefined, hideDefaultInputs = false }: Props = $props() let codeViewer: Drawer | undefined = $state() @@ -50,17 +51,10 @@
- {#if stepDetail.value.path.startsWith('hub/')} -
-

Code

- -
- {/if} +
+

Code

+ +
{:else if stepDetail.value.type == 'rawscript'}

Step inputs

@@ -92,10 +86,10 @@
{#if stepDetail == undefined}
-

+

Click on a step to see its details

- {#if schema} + {#if schema && !hideDefaultInputs}

Flow Inputs

{/if} @@ -217,27 +211,16 @@
{/if} - {#if stepDetail.value.path.startsWith('hub/')} -
-
-

Code

- -
- -
- {:else} - - {/if} +
+

Code

+ +
+ {:else if stepDetail.value.type == 'aiagent'}

Step inputs

diff --git a/frontend/src/lib/components/FlowMetadata.svelte b/frontend/src/lib/components/FlowMetadata.svelte index 3d4be84545..022fa874a2 100644 --- a/frontend/src/lib/components/FlowMetadata.svelte +++ b/frontend/src/lib/components/FlowMetadata.svelte @@ -2,6 +2,7 @@ import { type Job } from '$lib/gen' import { base } from '$lib/base' import JobStatus from '$lib/components/JobStatus.svelte' + import { flowPathToHref } from '$lib/scripts' import { displayDate, truncateRev } from '$lib/utils' import ScheduleEditor from '$lib/components/triggers/schedules/ScheduleEditor.svelte' import TimeAgo from './TimeAgo.svelte' @@ -94,7 +95,9 @@ {#if (job && job.job_kind == 'flow') || job?.job_kind == 'script'} {@const stem = `${job?.job_kind}s`} {@const isScript = job?.job_kind === 'script'} - {@const viewHref = `${base}/${stem}/get/${isScript ? job?.script_hash : job?.script_path}`} + {@const viewHref = isScript + ? `${base}/${stem}/get/${job?.script_hash}` + : flowPathToHref(job?.script_path ?? '')}
{#if isScript} diff --git a/frontend/src/lib/components/FlowPreviewContent.svelte b/frontend/src/lib/components/FlowPreviewContent.svelte index 1b8b126ca4..01ae4c53fa 100644 --- a/frontend/src/lib/components/FlowPreviewContent.svelte +++ b/frontend/src/lib/components/FlowPreviewContent.svelte @@ -11,6 +11,7 @@ import { createEventDispatcher, getContext, untrack } from 'svelte' import type { FlowEditorContext } from './flows/types' import { runFlowPreview } from './flows/utils.svelte' + import { processSecretArgs } from './secretArgUtils' import SchemaForm from './SchemaForm.svelte' import SchemaFormWithArgPicker from './SchemaFormWithArgPicker.svelte' import FlowStatusViewer from '../components/FlowStatusViewer.svelte' @@ -171,6 +172,7 @@ lastPreviewFlow = JSON.stringify(flowStore.val) flowProgressBar?.reset() const newFlow = extractFlow(previewMode) + args = await processSecretArgs(args, flowStore.val.schema as any) newJobId = await runFlowPreview(args, newFlow, $pathStore, restartedFrom, conversationId) jobId = newJobId isRunning = true @@ -437,7 +439,8 @@ startIcon={{ icon: isRunning ? RefreshCw : Play }} size="sm" btnClasses="w-full max-w-lg" - on:click={() => recordingMode ? recordAndTest() : runPreview(previewArgs.val, undefined)} + on:click={() => + recordingMode ? recordAndTest() : runPreview(previewArgs.val, undefined)} id="flow-editor-test-flow-drawer" shortCut={{ Icon: CornerDownLeft }} > @@ -675,6 +678,8 @@ {render} {customUi} showLogsWithResult + notes={flowStore.val.value.notes} + groups={flowStore.val.value.groups} /> {:else if loadingHistory}
diff --git a/frontend/src/lib/components/FlowStatusViewer.svelte b/frontend/src/lib/components/FlowStatusViewer.svelte index 950eb7354f..4b4e08c1b9 100644 --- a/frontend/src/lib/components/FlowStatusViewer.svelte +++ b/frontend/src/lib/components/FlowStatusViewer.svelte @@ -5,7 +5,7 @@ import type { DurationStatus, FlowStatusViewerContext, GraphModuleState } from './graph' import { isOwner as loadIsOwner, type StateStore } from '$lib/utils' import { userStore, workspaceStore } from '$lib/stores' - import type { CompletedJob, Job } from '$lib/gen' + import type { CompletedJob, FlowNote, FlowValue, Job } from '$lib/gen' interface Props { jobId: string @@ -34,6 +34,8 @@ onJobsLoaded?: ({ job, force }: { job: Job; force: boolean }) => void onDone?: ({ job }: { job: CompletedJob }) => void showLogsWithResult?: boolean + notes?: FlowNote[] + groups?: FlowValue['groups'] } let { @@ -42,6 +44,7 @@ workspaceId = undefined, flowState = $bindable({}), selectedJobStep = $bindable(undefined), + hideFlowResult = false, hideTimeline = false, hideDownloadInGraph = false, hideNodeDefinition = false, @@ -59,7 +62,9 @@ onStart, onJobsLoaded, onDone, - showLogsWithResult = false + showLogsWithResult = false, + notes: notesProp = undefined, + groups: groupsProp = undefined }: Props = $props() let lastJobId: string = untrack(() => jobId) @@ -171,5 +176,8 @@ } }} {showLogsWithResult} + {hideFlowResult} + notes={notesProp} + groups={groupsProp} /> {/key} diff --git a/frontend/src/lib/components/FlowStatusViewerInner.svelte b/frontend/src/lib/components/FlowStatusViewerInner.svelte index c75b4c7d3d..d35ae53c47 100644 --- a/frontend/src/lib/components/FlowStatusViewerInner.svelte +++ b/frontend/src/lib/components/FlowStatusViewerInner.svelte @@ -10,7 +10,9 @@ type FlowModule, ResourceService, type CompletedJob, - type WorkflowStatus + type WorkflowStatus, + type FlowNote, + type FlowValue } from '$lib/gen' import { workspaceStore } from '$lib/stores' import { base } from '$lib/base' @@ -134,6 +136,9 @@ } showLogsWithResult?: boolean showJobDetailHeader?: boolean + hideFlowResult?: boolean + notes?: FlowNote[] + groups?: FlowValue['groups'] } let { @@ -173,7 +178,10 @@ onDone = undefined, toolCallStore, showLogsWithResult = false, - showJobDetailHeader = false + showJobDetailHeader = false, + hideFlowResult = false, + notes: notesProp = undefined, + groups: groupsProp = undefined }: Props = $props() let getTopModuleStates = $derived(topModuleStates ?? localModuleStates) @@ -236,7 +244,9 @@ }) let jobResults: any[] = $state( - untrack(() => flowJobIds)?.flowJobs?.map((x, id) => `iter #${id + 1} not loaded by frontend yet`) ?? [] + untrack(() => flowJobIds)?.flowJobs?.map( + (x, id) => `iter #${id + 1} not loaded by frontend yet` + ) ?? [] ) function asWorkflowStatus(x: any): Record { @@ -255,7 +265,7 @@ let retry_selected = $state('') let timeout: number | undefined = undefined - let expandedSubflows: Record = $state({}) + let expandedSubflows: Record = $state({}) let selectionManager = new SelectionManager() @@ -684,10 +694,7 @@ } }) .catch((e) => { - console.error( - `Could not load inner module duration status for job ${mod.job}`, - e - ) + console.error(`Could not load inner module duration status for job ${mod.job}`, e) }) } } else { @@ -1154,7 +1161,7 @@ function allModulesForTimeline( modules: FlowModule[], - expandedSubflows: Record + expandedSubflows: Record ): FlowModuleForTimeline[] { const ids = dfs(modules, (x) => ({ id: x.id, type: x.value.type }) as FlowModuleForTimeline, { skipToolNodes: true @@ -1166,7 +1173,7 @@ ): FlowModuleForTimeline[] { return ids.concat( ids.flatMap(({ id }) => { - let fms = expandedSubflows[id] + let fms = expandedSubflows[id]?.modules let oid = id.split(':').pop() if (!oid) { return [] @@ -1351,7 +1358,7 @@ />
{/if} - {:else if render} + {:else if render && !hideFlowResult}
{#if showLogsWithResult && job} @@ -1901,7 +1908,8 @@ earlyStop={job.raw_flow?.skip_expr !== undefined} cache={job.raw_flow?.cache_ttl !== undefined} modules={job.raw_flow?.modules ?? []} - notes={job.raw_flow?.notes ?? []} + notes={notesProp ?? job.raw_flow?.notes ?? []} + groups={groupsProp ?? job.raw_flow?.groups} failureModule={job.raw_flow?.failure_module} preprocessorModule={job.raw_flow?.preprocessor_module} allowSimplifiedPoll={false} @@ -1994,7 +2002,9 @@ {#if job.args} {:else} @@ -2064,20 +2074,25 @@
Inputs
{/if} {#if node.workflow_as_code_status}
-
Workflow timeline
+
Workflow timeline
{/if} @@ -2128,7 +2143,7 @@ likely did not run yet

{/if} - {:else}

Select a node to see its details here

{/if}
@@ -2144,7 +2159,7 @@ {#if node?.job_id} {:else} -
Select a node with a job to see HTTP request traces
{/if} diff --git a/frontend/src/lib/components/FlowStatusWaitingForEvents.svelte b/frontend/src/lib/components/FlowStatusWaitingForEvents.svelte index 60f30b1adc..195a1d6fad 100644 --- a/frontend/src/lib/components/FlowStatusWaitingForEvents.svelte +++ b/frontend/src/lib/components/FlowStatusWaitingForEvents.svelte @@ -18,11 +18,9 @@ light?: boolean } - let { isOwner, workspaceId, job, light = false }: Props = $props() + let { isOwner: _isOwner, workspaceId, job, light = false }: Props = $props() let default_payload: object = $state({}) - let resumeUrl: string | undefined = $state(undefined) - let cancelUrl: string | undefined = $state(undefined) let description: any = $state(undefined) let hide_cancel = $state(false) @@ -49,8 +47,6 @@ defaultValues = JSON.parse(JSON.stringify(args)) default_payload = args - resumeUrl = job_result?.['resume'] - cancelUrl = job_result?.['cancel'] hide_cancel = job?.raw_flow?.modules?.[approvalStep]?.suspend?.hide_cancel ?? false schema = mergeSchema( job?.raw_flow?.modules?.[approvalStep]?.suspend?.resume_form?.schema ?? {}, @@ -61,61 +57,19 @@ let loading = $state(false) async function continu(approve: boolean) { loading = true - if ((resumeUrl && approve) || (cancelUrl && !approve)) { - let split = (approve ? resumeUrl : cancelUrl)!.split('/') - let signatureUrl = split.pop() ?? '' - const regex = /([^?]+)(?:\?[^=]+=(\w+))?/ - - const matches = signatureUrl.match(regex) - - const signature = matches?.[1] - if (!signature) { - sendUserToast(`Could not parse signature: ${signatureUrl}`, true) - return - } - const approver = matches?.[2] || undefined - - let resumeId = -1 - let parsedResumeId = split.pop() ?? '' - try { - resumeId = new Number(parsedResumeId).valueOf() - } catch (e) { - console.error(`Could not parse resume id: ${parsedResumeId}`) - } - let jobId = split.pop() ?? '' - if (approve) { - await JobService.resumeSuspendedJobPost({ - workspace: workspaceId ?? $workspaceStore ?? '', - id: jobId, - requestBody: default_payload as any, - resumeId, - signature, - approver - }) - } else { - await JobService.cancelSuspendedJobPost({ - workspace: workspaceId ?? $workspaceStore ?? '', - id: jobId, - resumeId, - signature, - approver, - requestBody: {} - }) - } - } else { - if (approve) { - await JobService.resumeSuspendedFlowAsOwner({ - workspace: workspaceId ?? $workspaceStore ?? '', - id: job?.id ?? '', - requestBody: default_payload as any - }) - } else { - await JobService.cancelQueuedJob({ - workspace: workspaceId ?? $workspaceStore ?? '', - id: job?.id ?? '', - requestBody: {} - }) - } + try { + await JobService.resumeSuspended({ + workspace: workspaceId ?? $workspaceStore ?? '', + jobId: job?.id ?? '', + requestBody: { + payload: approve ? (default_payload as any) : undefined, + approved: approve + } + }) + } catch (e: any) { + sendUserToast(e?.body ?? e?.message ?? 'Failed', true) + } finally { + loading = false } } let approvalStep = $derived((job?.flow_status?.step ?? 1) - 1) @@ -130,51 +84,41 @@
{/if}
- {#if isOwner || resumeUrl} -
- {#if !hide_cancel} -
-
- {/if} +
+ {#if !hide_cancel}
- +
- - {#if job?.raw_flow?.modules?.[approvalStep]?.suspend?.resume_form?.schema} -
- -
- - The payload is optional, it is passed to the following step through the `resume` - variable - - {/if} + {/if} +
+
- {:else} - You cannot resume the flow yourself without receiving the resume secret since you are not an - owner of {job.script_path} and the approval step did not contain the resume url at key `resume` - {/if} + + {#if job?.raw_flow?.modules?.[approvalStep]?.suspend?.resume_form?.schema} +
+ +
+ + The payload is optional, it is passed to the following step through the `resume` variable + + {/if} +
diff --git a/frontend/src/lib/components/FlowTimeline.svelte b/frontend/src/lib/components/FlowTimeline.svelte index 3fbb69503a..9bff159a03 100644 --- a/frontend/src/lib/components/FlowTimeline.svelte +++ b/frontend/src/lib/components/FlowTimeline.svelte @@ -81,36 +81,31 @@ }} /> {#if items} -
-
-
{min ? displayDate(new Date(min), true) : ''}
{#if max && min} - {/if}
{max ? displayDate(new Date(max), true) : ''}{#if !max && min}{#if now} - {msToSec(now - min, 1)}s - {/if}{/if}
-
-
-
-
-
Waiting for executor/Suspend
-
+
+
+
+ {min ? displayDate(new Date(min), true) : ''} +
+
+
+
+ Wait
- -
-
Execution
-
+
+
+ Execution
+ {#if max && min} + {msToSec(max - min, 1)}s + {/if} + {#if !max && min}{#if now} + {msToSec(now - min, 1)}s + {/if}{/if}
{#if selfWaitTime} -
- root: +
+ root: x.created_at && x.started_at)} -
-
-
+
+
{k.startsWith('subflow:') ? k.substring(8) : k} {#if localModuleStates[k]?.selectedForloop && (typ == 'forloopflow' || typ == 'whileloopflow')} @@ -141,70 +136,67 @@ {/if}
-
- {#if subItems?.length > 1} -
- {subItems?.length} jobs -
- {/if} - {#if min && total} - subItems?.[index]?.id} - > - {#snippet item({ index, style })} - {@const b = subItems?.[index]} - {#if b?.created_at} - - {@const waitingLen = b?.created_at - ? b.started_at - ? b.started_at - b?.created_at - : b.duration_ms - ? 0 - : now - b?.created_at - : 0} -
+ {#if subItems?.length > 1} + + {subItems?.length} jobs + + {/if} +
+
+ {#if min && total} + subItems?.[index]?.id} + > + {#snippet item({ index, style })} + {@const b = subItems?.[index]} + {#if b?.created_at} + {@const waitingLen = b?.created_at + ? b.started_at + ? b.started_at - b?.created_at + : b.duration_ms + ? 0 + : now - b?.created_at + : 0} +
+ + {#if b.started_at} - {#if b.started_at} - - {/if} -
- {:else} -
-
- -
-
- {/if} - {/snippet} - - {/if}
+ {/if} +
+ {:else} +
+ {/if} + {/snippet} + + {/if} +
{/each}
+ {:else} {/if} diff --git a/frontend/src/lib/components/FlowViewer.svelte b/frontend/src/lib/components/FlowViewer.svelte index 2355ddf68d..1e321584e3 100644 --- a/frontend/src/lib/components/FlowViewer.svelte +++ b/frontend/src/lib/components/FlowViewer.svelte @@ -20,7 +20,7 @@ schema?: any } - type TabValue = 'ui' | 'raw' | 'schema' | 'diff' + export type TabValue = 'ui' | 'raw' | 'schema' | 'diff' interface Props { flow: { @@ -33,10 +33,16 @@ noSide?: boolean noGraph?: boolean initTab?: TabValue + selectedTab?: TabValue + hideTabs?: boolean noSummary?: boolean + noInput?: boolean + hideDefaultInputs?: boolean + showStepHint?: boolean noGraphDownload?: boolean availableVersions?: Array<{ id: number; deployment_msg?: string }> selectedVersionId?: number + graphContent?: import('svelte').Snippet } let { @@ -46,9 +52,15 @@ noGraph = false, availableVersions = undefined, initTab = undefined, + selectedTab = $bindable(), + hideTabs = false, noSummary = false, + noInput = false, + hideDefaultInputs = false, + showStepHint = false, noGraphDownload = false, - selectedVersionId = undefined + selectedVersionId = undefined, + graphContent = undefined }: Props = $props() let open: { [id: number]: boolean } = {} @@ -59,7 +71,10 @@ let previousVersionId: number | undefined = $state(undefined) let previousFlow: PreviousFlow | undefined = $state(undefined) - let tab: TabValue = $state(untrack(() => initTab) ?? 'diff') + const tabControlledExternally = selectedTab !== undefined + if (!tabControlledExternally) { + selectedTab = initTab ?? 'diff' + } let previousFlowCache: Record = {} @@ -90,16 +105,16 @@ }) $effect.pre(() => { - if (initTab) { + if (initTab || tabControlledExternally) { return } if (availableVersions && availableVersions.length > 0) { - tab = 'diff' + selectedTab = 'diff' } else { if (noGraph) { - tab = 'schema' + selectedTab = 'schema' } else { - tab = 'ui' + selectedTab = 'ui' } } }) @@ -127,7 +142,7 @@ - + {#if availableVersions && availableVersions.length > 0} {/if} @@ -167,23 +182,38 @@ {/if} -
- {#if !noSummary} -

{flow.summary}

-
{flow.description ?? ''}
- {/if} + {#if graphContent} + {@render graphContent()} + {:else} +
+ {#if showStepHint} +

Click on a step to see its details

+ {/if} + {#if !noSummary} +

{flow.summary}

+
{flow.description ?? ''}
+ {/if} -

- Flow Input -

- {#if flow.schema && flow.schema.properties && Object.keys(flow.schema.properties).length > 0 && flow.schema} - - {:else} -
No inputs
- {/if} + {#if !noInput} +

+ Flow Input +

+ {#if flow.schema && flow.schema.properties && Object.keys(flow.schema.properties).length > 0 && flow.schema} + + {:else} +
No inputs
+ {/if} + {/if} - -
+ +
+ {/if} diff --git a/frontend/src/lib/components/InstanceSettings.svelte b/frontend/src/lib/components/InstanceSettings.svelte index 18514b3836..d558910aad 100644 --- a/frontend/src/lib/components/InstanceSettings.svelte +++ b/frontend/src/lib/components/InstanceSettings.svelte @@ -21,6 +21,7 @@ import SettingsFooter from './workspaceSettings/SettingsFooter.svelte' import SettingsPageHeader from './settings/SettingsPageHeader.svelte' import WorkspaceRegistries from './instanceSettings/WorkspaceRegistries.svelte' + import DbHealth from './instanceSettings/DbHealth.svelte' interface Props { tab?: string @@ -1052,6 +1053,12 @@ title="GitHub Enterprise App" description="Configure a self-managed GitHub App for GitHub Enterprise Server git sync." /> + {:else if category == 'DB Health'} + + {:else if category == 'Auth/OAuth/SAML'} { if (noLogs != lastNoLogs) { lastNoLogs = noLogs - if (!noLogs) { + if (!noLogs && !getActiveReplay()) { currentEventSource?.onerror?.(new Event(noLogsChangeRestartEvent)) const lastJobId = lastCompletedJobId if (lastJobId && (job || lastCallbacks?.loadExtraLogs)) { @@ -255,7 +255,7 @@ } } export async function getLogs() { - if (job) { + if (job && !getActiveReplay()) { refreshLogOffset() const getUpdate = await JobService.getJobUpdates({ workspace: workspace!, diff --git a/frontend/src/lib/components/Login.svelte b/frontend/src/lib/components/Login.svelte index 1e375cb53d..c1a2a7ebdb 100644 --- a/frontend/src/lib/components/Login.svelte +++ b/frontend/src/lib/components/Login.svelte @@ -164,7 +164,8 @@ workspace: $workspaceStore! }) if (!emptyString(defaultApp.default_app_path)) { - goto(`/apps/get/${defaultApp.default_app_path}`) + const prefix = defaultApp.default_app_raw ? '/apps_raw/get' : '/apps/get' + goto(`${prefix}/${defaultApp.default_app_path}`) } else { goto(rd ?? '/') } diff --git a/frontend/src/lib/components/OktaSetting.svelte b/frontend/src/lib/components/OktaSetting.svelte index 8f19ce5666..495ed5794c 100644 --- a/frontend/src/lib/components/OktaSetting.svelte +++ b/frontend/src/lib/components/OktaSetting.svelte @@ -18,17 +18,18 @@ function changeDomain(domain, custom) { if (value) { let baseUrl = custom ? `https://${domain}` : `https://${domain}.okta.com` + let authPath = custom ? '/v1' : '/oauth2/v1' value = { ...value, login_config: { - auth_url: `${baseUrl}/oauth2/v1/authorize`, - token_url: `${baseUrl}/oauth2/v1/token`, - userinfo_url: `${baseUrl}/oauth2/v1/userinfo`, + auth_url: `${baseUrl}${authPath}/authorize`, + token_url: `${baseUrl}${authPath}/token`, + userinfo_url: `${baseUrl}${authPath}/userinfo`, scopes: ['openid', 'profile', 'email'] }, connect_config: { - auth_url: `${baseUrl}/oauth2/v1/authorize`, - token_url: `${baseUrl}/oauth2/v1/token`, + auth_url: `${baseUrl}${authPath}/authorize`, + token_url: `${baseUrl}${authPath}/token`, scopes: ['openid', 'profile', 'email'] } } diff --git a/frontend/src/lib/components/Password.svelte b/frontend/src/lib/components/Password.svelte index 673f6416db..12c374bf70 100644 --- a/frontend/src/lib/components/Password.svelte +++ b/frontend/src/lib/components/Password.svelte @@ -1,5 +1,6 @@
-
+
- onBlur?.(e), - onkeydown: (e) => { - onKeyDown?.(e) - bubble('keydown')(e) - }, - type: hideValue ? 'password' : 'text' - }} - class="pr-8" - /> + {#if isMultiline} + onBlur?.(e), + onkeydown: (e) => { + onKeyDown?.(e) + bubble('keydown')(e) + }, + style: hideValue ? '-webkit-text-security: disc' : '' + }} + class="pr-8" + unifiedHeight={false} + /> + {:else} + onBlur?.(e), + onkeydown: (e) => { + if (e.key === 'Enter') { + e.preventDefault() + insertAndSwitchToMultiline(e.currentTarget as HTMLInputElement, '\n') + return + } + onKeyDown?.(e) + bubble('keydown')(e) + }, + onpaste: (e) => { + const text = e.clipboardData?.getData('text') + if (text?.includes('\n')) { + e.preventDefault() + insertAndSwitchToMultiline(e.currentTarget as HTMLInputElement, text) + } + }, + type: hideValue ? 'password' : 'text' + }} + class="pr-8" + /> + {/if}
{#if red}
This field is required
diff --git a/frontend/src/lib/components/PasswordArgInput.svelte b/frontend/src/lib/components/PasswordArgInput.svelte index 8e7d548c1b..60474e53cb 100644 --- a/frontend/src/lib/components/PasswordArgInput.svelte +++ b/frontend/src/lib/components/PasswordArgInput.svelte @@ -9,9 +9,10 @@ interface Props { value?: string | undefined disabled: boolean + minRows?: number } - let { value = $bindable(undefined), disabled }: Props = $props() + let { value = $bindable(undefined), disabled, minRows }: Props = $props() let path = $state('') let password = $state( @@ -95,5 +96,5 @@
{:else} - + {/if} diff --git a/frontend/src/lib/components/ResourceEditor.svelte b/frontend/src/lib/components/ResourceEditor.svelte index 70aaf6a2ee..416068bf8b 100644 --- a/frontend/src/lib/components/ResourceEditor.svelte +++ b/frontend/src/lib/components/ResourceEditor.svelte @@ -25,6 +25,7 @@ import Button from './common/button/Button.svelte' import { clearJsonSchemaResourceCache } from './schema/jsonSchemaResource.svelte' import ResourceGen from './copilot/ResourceGen.svelte' + import SyncResourceTypes from './SyncResourceTypes.svelte' interface Props { canSave?: boolean @@ -33,6 +34,7 @@ hidePath?: boolean onChange?: (args: { path: string; args: Record; description: string }) => void defaultValues?: Record | undefined + workspace?: string | undefined } let { @@ -41,9 +43,12 @@ path = $bindable(''), hidePath = false, onChange, - defaultValues = undefined + defaultValues = undefined, + workspace = undefined }: Props = $props() + let effectiveWorkspace = $derived(workspace ?? $workspaceStore!) + let isValid = $state(true) let jsonError = $state('') let can_write = $state(true) @@ -68,13 +73,13 @@ let rawCode: string | undefined = $state(undefined) async function initEdit() { - resourceToEdit = await ResourceService.getResource({ workspace: $workspaceStore!, path }) + resourceToEdit = await ResourceService.getResource({ workspace: effectiveWorkspace, path }) description = resourceToEdit!.description ?? '' resource_type = resourceToEdit!.resource_type args = resourceToEdit?.value ?? ({} as any) loadResourceType() can_write = - resourceToEdit.workspace_id == $workspaceStore && + resourceToEdit.workspace_id == effectiveWorkspace && canWrite(path, resourceToEdit.extra_perms ?? {}, $userStore) linkedVars = Object.entries(args) .filter(([_, v]) => typeof v == 'string' && v == `$var:${initialPath}`) @@ -92,12 +97,12 @@ export async function editResource(): Promise { if (resourceToEdit) { await ResourceService.updateResource({ - workspace: $workspaceStore!, + workspace: effectiveWorkspace, path: resourceToEdit.path, requestBody: { path, value: args, description } }) if (resourceToEdit.resource_type === 'json_schema') { - clearJsonSchemaResourceCache(resourceToEdit.path, $workspaceStore!) + clearJsonSchemaResourceCache(resourceToEdit.path, effectiveWorkspace) } sendUserToast(`Updated resource at ${path}`) dispatch('refresh', path) @@ -108,7 +113,7 @@ export async function createResource(): Promise { await ResourceService.createResource({ - workspace: $workspaceStore!, + workspace: effectiveWorkspace, requestBody: { path, value: args, description, resource_type: resource_type! } }) sendUserToast(`Updated resource at ${path}`) @@ -119,7 +124,7 @@ if (resource_type) { try { const resourceType = await ResourceService.getResourceType({ - workspace: $workspaceStore!, + workspace: effectiveWorkspace, path: resource_type }) @@ -343,10 +348,13 @@ {:else} {#if !viewJsonSchema} -

- No corresponding resource type found in your workspace for {resource_type}. Define the - value in JSON directly -

+
+

+ Resource type '{resource_type}' not found in your workspace +

+ +

Define the value in JSON directly

+
{/if} {#if !emptyString(jsonError)} - + void excludedValues?: string[] datatableAsPgResource?: boolean + workspace?: string | undefined + disableChatOffset?: boolean } let { @@ -47,9 +49,13 @@ class: className = '', onClear = undefined, excludedValues = undefined, - datatableAsPgResource = false + datatableAsPgResource = false, + workspace = undefined, + disableChatOffset = false }: Props = $props() + let effectiveWorkspace = $derived(workspace ?? $workspaceStore!) + if (initialValue && value == undefined) { value = initialValue } @@ -104,7 +110,7 @@ const resources = await Promise.all( resourceTypesToQuery.map((rt) => ResourceService.listResource({ - workspace: $workspaceStore!, + workspace: effectiveWorkspace, resourceType: rt }) ) @@ -121,9 +127,11 @@ if (datatableAsPgResource && resourceType === 'postgresql') { try { - const datatables = (await WorkspaceService.listDataTables({ - workspace: $workspaceStore! - })).map(d => d.name) + const datatables = ( + await WorkspaceService.listDataTables({ + workspace: effectiveWorkspace + }) + ).map((d) => d.name) for (const dt of datatables) { nc.push({ value: `datatable://${dt}`, @@ -155,7 +163,7 @@ let previousResourceType = untrack(() => resourceType) $effect(() => { - $workspaceStore && resourceType + effectiveWorkspace && resourceType untrack(() => { if (previousResourceType != resourceType) { previousResourceType = resourceType @@ -167,7 +175,7 @@ $effect(() => { excludedValues - if ($workspaceStore && resourceType && !disabled) { + if (effectiveWorkspace && resourceType && !disabled) { untrack(() => loadResources(resourceType)) } }) @@ -186,9 +194,13 @@ }} bind:this={appConnect} {expressOAuthSetup} + {workspace} + {disableChatOffset} /> { await loadResources(resourceType) if (e.detail) { diff --git a/frontend/src/lib/components/RunForm.svelte b/frontend/src/lib/components/RunForm.svelte index da0c87a15f..6c2ac09da0 100644 --- a/frontend/src/lib/components/RunForm.svelte +++ b/frontend/src/lib/components/RunForm.svelte @@ -3,7 +3,8 @@ computeSharableHash as computeSharableHash, defaultIfEmptyString, emptyString, - truncateHash + truncateHash, + sendUserToast } from '$lib/utils' import type { Schema } from '$lib/common' @@ -21,6 +22,7 @@ import { triggerableByAI } from '$lib/actions/triggerableByAI.svelte' import InputSelectedBadge from './schema/InputSelectedBadge.svelte' import { untrack } from 'svelte' + import { processSecretArgs } from './secretArgUtils' let reloadArgs = $state(0) let jsonEditor: JsonInputs | undefined = $state(undefined) @@ -33,8 +35,20 @@ reloadArgs++ } - export function run() { - runAction(scheduledForStr, args ?? {}, invisible_to_owner, overrideTag) + export async function run(overrideScheduledForStr?: string | undefined | null) { + let processedArgs: Record + try { + processedArgs = await processSecretArgs(args ?? {}, runnable?.schema) + } catch (e) { + sendUserToast('Failed to process sensitive args: ' + e, true) + return + } + runAction( + overrideScheduledForStr === null ? undefined : (overrideScheduledForStr ?? scheduledForStr), + processedArgs, + invisible_to_owner, + overrideTag + ) } interface Props { @@ -276,7 +290,7 @@ unifiedSize="md" btnClasses="!inline-flex" disabled={!isValid && !jsonView} - on:click={() => runAction(scheduledForStr, args ?? {}, invisible_to_owner, overrideTag)} + on:click={() => run()} shortCut={{ Icon: CornerDownLeft, hide: !viewKeybinding }} > {scheduledForStr ? 'Schedule to run later' : buttonText} @@ -315,7 +329,7 @@ btnClasses="!px-6 !py-1 w-full" variant="accent" disabled={!isValid && !jsonView} - on:click={() => runAction(undefined, args ?? {}, invisible_to_owner, overrideTag)} + on:click={() => run(null)} shortCut={{ Icon: CornerDownLeft, hide: !viewKeybinding }} > {buttonText} diff --git a/frontend/src/lib/components/ScriptBuilder.svelte b/frontend/src/lib/components/ScriptBuilder.svelte index 024fe0e907..16ebfa8851 100644 --- a/frontend/src/lib/components/ScriptBuilder.svelte +++ b/frontend/src/lib/components/ScriptBuilder.svelte @@ -178,7 +178,7 @@ let confirmCallback: () => void = $state(() => {}) // What happens when user clicks `override` in warning let open: boolean = $state(false) // Is confirmation modal open let args: Record = $state(untrack(() => initialArgs)) // Test args input - let selectedInputTab: 'main' | 'preprocessor' = $state('main') + let selectedInputTab: 'main' | 'preprocessor' | 'diagram' = $state('main') let hasPreprocessor = $state(false) let preserveOnBehalfOf = $state(false) diff --git a/frontend/src/lib/components/ScriptEditor.svelte b/frontend/src/lib/components/ScriptEditor.svelte index 6a4fd088e0..66101fc892 100644 --- a/frontend/src/lib/components/ScriptEditor.svelte +++ b/frontend/src/lib/components/ScriptEditor.svelte @@ -1,5 +1,6 @@ - + diff --git a/frontend/src/lib/components/StringTypeNarrowing.svelte b/frontend/src/lib/components/StringTypeNarrowing.svelte index 7d84b4c4fc..8ecedf039e 100644 --- a/frontend/src/lib/components/StringTypeNarrowing.svelte +++ b/frontend/src/lib/components/StringTypeNarrowing.svelte @@ -52,7 +52,8 @@ computeKind(enum_, contentEncoding, pattern, format) ) - const allowKindChange = untrack(() => overrideAllowKindChange) || untrack(() => originalType) === 'string' + const allowKindChange = + untrack(() => overrideAllowKindChange) || untrack(() => originalType) === 'string' let patternStr: string = $state(pattern ?? '') let resource: string | undefined = $state() @@ -383,7 +384,7 @@ options={{ right: 'Is Password/Sensitive', rightTooltip: - 'The value will be stored as an ephemeral secret variable in the user space of the caller of the job, only viewable by him.' + 'The value will be stored as an ephemeral secret variable in the user space of the caller of the job, only viewable by that user.' }} checked={password} on:change={(e) => { diff --git a/frontend/src/lib/components/SuperadminSettings.svelte b/frontend/src/lib/components/SuperadminSettings.svelte index b9dbb7c6d6..316a6237cd 100644 --- a/frontend/src/lib/components/SuperadminSettings.svelte +++ b/frontend/src/lib/components/SuperadminSettings.svelte @@ -146,6 +146,7 @@ bind:this={innerComponent} closeDrawer={handleClose} showHeaderInfo={false} + {disableChatOffset} bind:yamlMode bind:hasUnsavedChanges bind:hasAnyInvalid diff --git a/frontend/src/lib/components/SuperadminSettingsInner.svelte b/frontend/src/lib/components/SuperadminSettingsInner.svelte index 964414cafa..683db57724 100644 --- a/frontend/src/lib/components/SuperadminSettingsInner.svelte +++ b/frontend/src/lib/components/SuperadminSettingsInner.svelte @@ -17,7 +17,7 @@ import ToggleButtonGroup from './common/toggleButton-v2/ToggleButtonGroup.svelte' import ToggleButton from './common/toggleButton-v2/ToggleButton.svelte' import { userStore, workspaceStore } from '$lib/stores' - import { ExternalLink, Pencil, UserMinus, UserPlus } from 'lucide-svelte' + import { Ban, CheckCircle2, ExternalLink, Pencil, UserMinus, UserPlus } from 'lucide-svelte' import DropdownV2 from './DropdownV2.svelte' import Popover from './meltComponents/Popover.svelte' import ConfirmationModal from './common/confirmationModal/ConfirmationModal.svelte' @@ -39,12 +39,14 @@ import TextInput from './text_input/TextInput.svelte' import SettingsPageHeader from './settings/SettingsPageHeader.svelte' import SettingsSearchInput from './instanceSettings/SettingsSearchInput.svelte' + import InstanceAISettings from './instanceSettings/InstanceAISettings.svelte' let filter = $state('') let { closeDrawer, showHeaderInfo = true, + disableChatOffset = false, yamlMode = $bindable(false), hasUnsavedChanges = $bindable(false), hasAnyInvalid = $bindable(false) @@ -65,6 +67,8 @@ let filteredUsers: GlobalUserInfo[] = $state([]) let deleteConfirmedCallback: (() => void) | undefined = $state(undefined) let deleteUserEmail: string = $state('') + let disableConfirmedCallback: (() => void) | undefined = $state(undefined) + let disableUserEmail: string = $state('') let editWrappers: Record = $state({}) let activeOnly = $state(false) @@ -234,7 +238,9 @@
- {#if tab === 'users' && !yamlMode} + {#if tab === 'ai' && !yamlMode} + + {:else if tab === 'users' && !yamlMode}
{#if !automateUsernameCreation && !isCloudHosted()}
@@ -289,9 +295,9 @@ /> @@ -343,13 +349,25 @@ {#if filteredUsers && users} - {#each filteredUsers.slice(0, nbDisplayed) as { email, super_admin, devops, login_type, name, username, operator_only, role_source }, i (email)} - - {email} + {#each filteredUsers.slice(0, nbDisplayed) as { email, super_admin, devops, login_type, name, username, operator_only, role_source, disabled }, i (email)} + + +
+ {email} + {#if disabled} + Disabled + {/if} +
+ {#if automateUsernameCreation} {#if username} @@ -510,6 +528,39 @@ if (btn instanceof HTMLElement) btn.click() } }, + { + displayName: disabled ? 'Enable' : 'Disable', + icon: disabled ? CheckCircle2 : Ban, + action: () => { + if (!disabled) { + disableUserEmail = email + disableConfirmedCallback = async () => { + try { + await UserService.globalUserUpdate({ + email, + requestBody: { disabled: true } + }) + sendUserToast('User disabled') + listUsers(activeOnly) + } catch (e) { + sendUserToast('Failed to disable user', true) + } + } + } else { + UserService.globalUserUpdate({ + email, + requestBody: { disabled: false } + }) + .then(() => { + sendUserToast('User enabled') + listUsers(activeOnly) + }) + .catch(() => { + sendUserToast('Failed to enable user', true) + }) + } + } + }, { displayName: 'Remove', icon: UserMinus, @@ -574,6 +625,33 @@ }} >
- Are you sure you want to remove {deleteUserEmail}? + Are you sure you want to remove {deleteUserEmail}? They will be removed from all + workspaces and instance groups, and all their sessions and tokens will be revoked. This action + is irreversible. Their workspace content (scripts, flows, apps) will not be deleted. +
+ + { + disableConfirmedCallback = undefined + listUsers(activeOnly) + }} + on:confirmed={() => { + if (disableConfirmedCallback) { + disableConfirmedCallback() + } + disableConfirmedCallback = undefined + }} +> +
+ Are you sure you want to disable {disableUserEmail}? All their active sessions and + tokens will be revoked immediately. They will be unable to log in until re-enabled. Their + workspace memberships and content will be preserved.
diff --git a/frontend/src/lib/components/SyncResourceTypes.svelte b/frontend/src/lib/components/SyncResourceTypes.svelte new file mode 100644 index 0000000000..dd343e43f6 --- /dev/null +++ b/frontend/src/lib/components/SyncResourceTypes.svelte @@ -0,0 +1,41 @@ + + +{#if $superadmin} + + {#if hubRtSync.status === 'error'} + + Error syncing resource types: {hubRtSync.error?.message ?? JSON.stringify(hubRtSync.error)} + + {/if} +{/if} diff --git a/frontend/src/lib/components/TimelineBar.svelte b/frontend/src/lib/components/TimelineBar.svelte index e4fb511450..5d1f53e961 100644 --- a/frontend/src/lib/components/TimelineBar.svelte +++ b/frontend/src/lib/components/TimelineBar.svelte @@ -14,6 +14,7 @@ running: boolean concat?: boolean gray?: boolean + spacerClass?: string } let { @@ -25,25 +26,26 @@ id, running, concat = false, - gray = false + gray = false, + spacerClass = '' }: Props = $props() {#if min && started_at != undefined} {#if !concat} -
+
{/if} {#snippet text()} 0} {@const narrow = len / total < 0.09} - {@const endPos = started_at != undefined && min != undefined ? (started_at - min + len) / total : 1} + {@const endPos = + started_at != undefined && min != undefined ? (started_at - min + len) / total : 1} {@const nearStart = endPos < 0.15} - {#if len}{msToSec(len, 1)}s{/if} {/if} diff --git a/frontend/src/lib/components/WorkflowTimeline.svelte b/frontend/src/lib/components/WorkflowTimeline.svelte index 3dea7279d6..2ea5ebba50 100644 --- a/frontend/src/lib/components/WorkflowTimeline.svelte +++ b/frontend/src/lib/components/WorkflowTimeline.svelte @@ -1,6 +1,6 @@ {#if flow_status} @@ -167,22 +216,74 @@ sleep ({(v as any).sleep_duration_s}s)
{:else if isApproval} -
-
- - - {v.name ?? stepKey(k)} - - {#if !isDone} - - - waiting + {@const selfApprovalDisabled = (v as any).self_approval_disabled === true} + {@const formSchema = (v as any).form?.schema ?? (v as any).form} + {@const hasForm = + formSchema && typeof formSchema === 'object' && Object.keys(formSchema).length > 0} + {@const canApprove = !isDone && jobId} +
+
+
+ + + {v.name ?? stepKey(k)} - {:else} - {msToSec(v.duration_ms ?? 0)}s + {#if !isDone} + + + waiting + + {#if canApprove} +
+ + +
+ {/if} + {:else} + {msToSec(v.duration_ms ?? 0)}s + {/if} +
+ {#if canApprove && selfApprovalDisabled && $userStore?.is_admin} +
+ Self-approval is disabled but allowed because you are an admin/owner +
+ {/if} + {#if canApprove && hasForm} +
+ {#if emptyString($enterpriseLicense)} + + {:else} + + {/if} +
{/if}
{:else} @@ -275,13 +376,13 @@ {@const result = stepResults[stepKey(k)]} {#if isDone && result !== undefined}
-
Result
+
Result
{:else} -
Step completed (no result)
+
Step completed (no result)
{/if} {:else if loadingJobs[k] && !childJobs[k]}
@@ -293,7 +394,7 @@ {#if job.logs || isRunning}
-
Logs
+
Logs
{#if isDone && job.result !== undefined}
-
Result
+
Result
diff --git a/frontend/src/lib/components/apps/editor/inlineScriptsPanel/InlineScriptRunnableByPath.svelte b/frontend/src/lib/components/apps/editor/inlineScriptsPanel/InlineScriptRunnableByPath.svelte index 5d6ddf7b93..6c991ea52a 100644 --- a/frontend/src/lib/components/apps/editor/inlineScriptsPanel/InlineScriptRunnableByPath.svelte +++ b/frontend/src/lib/components/apps/editor/inlineScriptsPanel/InlineScriptRunnableByPath.svelte @@ -4,9 +4,11 @@ const bubble = createBubbler() import { Button, Drawer, DrawerContent } from '$lib/components/common' import { base } from '$lib/base' + import FlowGraphViewer from '$lib/components/FlowGraphViewer.svelte' + import Skeleton from '$lib/components/common/skeleton/Skeleton.svelte' import FlowModuleScript from '$lib/components/flows/content/FlowModuleScript.svelte' import FlowPathViewer from '$lib/components/flows/content/FlowPathViewer.svelte' - import { emptySchema, sendUserToast } from '$lib/utils' + import { emptySchema, getHubFlowIdFromPath, isHubFlowPath, sendUserToast } from '$lib/utils' import { getContext, tick, untrack } from 'svelte' import type { ConnectedAppInput, @@ -31,7 +33,8 @@ import Popover from '$lib/components/meltComponents/Popover.svelte' import ScriptEditorDrawer from '$lib/components/flows/content/ScriptEditorDrawer.svelte' import FlowEditorDrawer from '$lib/components/flows/content/FlowEditorDrawer.svelte' - import { ScriptService } from '$lib/gen' + import { FlowService, ScriptService, type OpenFlow } from '$lib/gen' + import { replaceScriptPlaceholderWithItsValues } from '$lib/hub' interface Props { runnable: RunnableByPath @@ -43,6 +46,7 @@ isLoading?: boolean onRun?: any onCancel?: any + hubFlowPreview?: OpenFlow | undefined } let { @@ -52,14 +56,17 @@ rawApps = false, isLoading = false, onRun = async () => {}, - onCancel = async () => {} + onCancel = async () => {}, + hubFlowPreview = $bindable(undefined) }: Props = $props() const viewerContext = getContext('AppViewerContext') let drawerFlowViewer: Drawer | undefined = $state(undefined) let flowPath: string = $state('') + let drawerShowsHubFlow = $state(false) let notFound = $state(false) + let hubFlowId = $derived(getHubFlowIdFromPath(runnable.path)) // Key to force re-mounting of viewer components (bypasses FlowModuleScript cache) let refreshKey = $state(0) @@ -70,6 +77,7 @@ const dispatch = createEventDispatcher() async function refreshScript(runnable: RunnableByPath) { + hubFlowPreview = undefined try { let { schema } = await getScriptByPath(runnable.path) if (!deepEqual(runnable.schema, schema)) { @@ -86,7 +94,39 @@ } async function refreshFlow(runnable: RunnableByPath) { + hubFlowPreview = undefined try { + const hubFlowId = getHubFlowIdFromPath(runnable.path) + if (hubFlowId !== undefined) { + const hub = await FlowService.getHubFlowById({ id: hubFlowId }) + const flow = hub.flow ? structuredClone(hub.flow) : undefined + if (flow?.value.preprocessor_module?.value.type === 'rawscript') { + flow.value.preprocessor_module.value.content = replaceScriptPlaceholderWithItsValues( + String(hubFlowId), + flow.value.preprocessor_module.value.content + ) + } + + if (!flow) { + notFound = true + return + } + + hubFlowPreview = flow + const schema = + flow.schema && typeof flow.schema === 'object' && Object.keys(flow.schema).length > 0 + ? (flow.schema as any) + : emptySchema() + if (!deepEqual(runnable.schema, schema)) { + runnable.schema = schema + if (!runnable.schema.order) { + runnable.schema.order = Object.keys(runnable.schema.properties ?? {}) + } + fields = computeFields(schema, false, fields ?? {}) + } + return + } + const { schema } = (await loadSchema($workspaceStore ?? '', runnable.path, 'flow')) ?? emptySchema() if (!deepEqual(runnable.schema, schema)) { @@ -158,6 +198,8 @@ refreshScript(runnable) } else if (runnable.runType == 'flow') { refreshFlow(runnable) + } else { + hubFlowPreview = undefined } lastRunnable = runnable } @@ -170,8 +212,34 @@ - - + { + flowPath = '' + drawerShowsHubFlow = false + drawerFlowViewer?.closeDrawer() + }} + > + {#if drawerShowsHubFlow} +
+ {#if hubFlowPreview} + + {:else if notFound} +
Hub flow not found at {flowPath}
+ {:else} +
+ +
+ {/if} +
+ {:else if flowPath} + + {/if} @@ -210,7 +278,7 @@ size="xs" startIcon={{ icon: RefreshCw }} on:click={async () => { - sendUserToast('Getting latest script version at that path') + sendUserToast('Getting latest runnable version at that path') // Increment refreshKey to force re-mounting of viewer components (bypasses cache) refreshKey++ lastRunnable = undefined @@ -238,31 +306,45 @@ startIcon={{ icon: Eye }} on:click={() => { flowPath = runnable.path + drawerShowsHubFlow = isHubFlowPath(runnable.path) drawerFlowViewer?.openDrawer() }} > Expand - - + {#if hubFlowId} + + {:else} + + + {/if} {:else} + Cache + {/snippet} {#snippet content()} - Since this is a reference to a workspace {runnable.runType}, set the cache in the {runnable.runType} - settings directly by editing it. The cache will be shared by any app or flow that uses this - {runnable.runType}. + {#if runnable.runType == 'flow' && isHubFlowPath(runnable.path)} + Since this is a reference to a hub flow, cache settings are managed from the flow after + you fork it into your workspace. + {:else} + Since this is a reference to a workspace {runnable.runType}, set the cache in the + {runnable.runType} settings directly by editing it. The cache will be shared by any app or + flow that uses this {runnable.runType}. + {/if} {/snippet} @@ -325,18 +414,37 @@ class="!text-xs !rounded-xs" />
-
+
{#key `${viewerContext?.stateId ? get(viewerContext.stateId) : 0}-${refreshKey}`} {#if notFound} -
{runnable.runType} not found at {runnable.path} in workspace {$workspaceStore}
+
+ {#if runnable.runType == 'flow' && isHubFlowPath(runnable.path)} + Hub flow not found at {runnable.path} + {:else} + {runnable.runType} not found at {runnable.path} in workspace {$workspaceStore} + {/if} +
{:else if runnable.runType == 'script' || runnable.runType == 'hubscript'}
{:else if runnable.runType == 'flow'} - + {#if isHubFlowPath(runnable.path)} + {#if hubFlowPreview} +
+ +
+ {:else} + + {/if} + {:else} + + {/if} {:else} Unrecognized runType {runnable.runType} {/if} diff --git a/frontend/src/lib/components/assets/AssetButtons.svelte b/frontend/src/lib/components/assets/AssetButtons.svelte index 62bd981609..8e8c851f42 100644 --- a/frontend/src/lib/components/assets/AssetButtons.svelte +++ b/frontend/src/lib/components/assets/AssetButtons.svelte @@ -43,32 +43,28 @@ {#if (asset.kind === 'resource' && resourceDataCacheValue === undefined) || ducklakeNotFound || datatableNotFound} {#snippet trigger()} - - - {:else if datatableNotFound} - - {:else if asset.kind === 'resource' && resourceDataCacheValue === undefined} - - {/if} - - {/snippet} + Not found + {#if ducklakeNotFound} + + {:else if datatableNotFound} + + {:else if asset.kind === 'resource' && resourceDataCacheValue === undefined} + + {/if} + {/snippet} {:else if assetCanBeExplored(asset, { resource_type: resourceDataCacheValue })} -{#snippet rightBadge(text: string | undefined, tooltip?: string)} - {#if text} +{#snippet rightBadge(badgeText: string | undefined, tooltip?: string)} + {#if badgeText}
- {text} + {badgeText}
{#snippet text()} - - {#if tooltip} - {tooltip} - {/if} - - {/snippet} + {#if tooltip} + {tooltip} + {/if} + {/snippet} {/if} {/snippet} diff --git a/frontend/src/lib/components/common/confirmationModal/ConfirmationModal.svelte b/frontend/src/lib/components/common/confirmationModal/ConfirmationModal.svelte index a49c28cf94..9dee3b6e87 100644 --- a/frontend/src/lib/components/common/confirmationModal/ConfirmationModal.svelte +++ b/frontend/src/lib/components/common/confirmationModal/ConfirmationModal.svelte @@ -15,6 +15,7 @@ type?: 'danger' | 'reload' showIcon?: boolean id?: string + trashbin?: boolean children?: Snippet onConfirmed?: () => void | Promise onCanceled?: () => void @@ -29,6 +30,7 @@ type: _type, showIcon = true, id, + trashbin = false, children, onConfirmed, onCanceled @@ -120,6 +122,12 @@
{@render children?.()}
+ {#if trashbin} +

This item will be moved to the trashbin and can be restored by a workspace admin + within 3 days.

+ {/if}
diff --git a/frontend/src/lib/components/common/fileUpload/FileUpload.svelte b/frontend/src/lib/components/common/fileUpload/FileUpload.svelte index f0ce068ea7..dfe5c158cc 100644 --- a/frontend/src/lib/components/common/fileUpload/FileUpload.svelte +++ b/frontend/src/lib/components/common/fileUpload/FileUpload.svelte @@ -6,6 +6,7 @@ import { sendUserToast } from '$lib/toast' import { workspaceStore } from '$lib/stores' import { AppService, HelpersService } from '$lib/gen' + import { OpenAPI } from '$lib/gen/core/OpenAPI' import { writable, type Writable } from 'svelte/store' import { Ban, CheckCheck, FileWarning, Files, RefreshCcw, Trash, XIcon } from 'lucide-svelte' import { twMerge } from 'tailwind-merge' @@ -92,7 +93,7 @@ let initialS3 = $derived( Array.isArray(initialValue) - ? initialValue?.map((v) => v.s3) + ? initialValue?.filter((v) => v != null).map((v) => v.s3) : initialValue?.s3 ? [initialValue?.s3] : undefined @@ -112,7 +113,7 @@ if (!$fileUploads.find((fileUpload) => fileUpload.path === s3)) { let initialFileUploads = initialValue ? Array.isArray(initialValue) - ? initialValue.map(transform) + ? initialValue.filter((v) => v != null).map(transform) : [transform(initialValue)] : [] $fileUploads = [...$fileUploads, ...initialFileUploads] @@ -334,6 +335,9 @@ true ) xhr?.setRequestHeader('Content-Type', 'application/octet-stream') + if (OpenAPI.TOKEN) { + xhr?.setRequestHeader('Authorization', `Bearer ${OpenAPI.TOKEN}`) + } xhr?.send(fileToUpload) })) as any diff --git a/frontend/src/lib/components/copilot/TestAIKey.svelte b/frontend/src/lib/components/copilot/TestAIKey.svelte index 575159aef8..1f0f97c4ef 100644 --- a/frontend/src/lib/components/copilot/TestAIKey.svelte +++ b/frontend/src/lib/components/copilot/TestAIKey.svelte @@ -7,6 +7,7 @@ interface Props { disabled?: boolean apiKey?: string | undefined + workspace?: string | undefined resourcePath?: string | undefined aiProvider: AIProvider model: string @@ -15,6 +16,7 @@ let { disabled = false, apiKey = undefined, + workspace = undefined, resourcePath = undefined, aiProvider, model @@ -38,6 +40,7 @@ await testKey({ apiKey, + workspace, resourcePath, messages: [ { diff --git a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts index 00a1a11fdf..140bbf4f14 100644 --- a/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts +++ b/frontend/src/lib/components/copilot/chat/AIChatManager.svelte.ts @@ -23,8 +23,7 @@ import { } from './shared' import type { ChatCompletionMessageParam, - ChatCompletionSystemMessageParam, - ChatCompletionUserMessageParam + ChatCompletionSystemMessageParam } from 'openai/resources/chat/completions.mjs' import { prepareInlineChatSystemPrompt, @@ -37,7 +36,7 @@ import { loadApiTools } from './api/apiTools' import { prepareScriptUserMessage } from './script/core' import { prepareNavigatorUserMessage } from './navigator/core' import { sendUserToast } from '$lib/toast' -import { getCompletion, getModelContextWindow, parseOpenAICompletion } from '../lib' +import { getModelContextWindow, workspaceAIClients } from '../lib' import { dfs } from '$lib/components/flows/previousResults' import { getStringError } from './utils' import type { FlowModuleState, FlowState } from '$lib/components/flows/flowState' @@ -56,8 +55,7 @@ import type { import type { Selection } from 'monaco-editor' import type AIChatInput from './AIChatInput.svelte' import { prepareApiSystemMessage, prepareApiUserMessage } from './api/core' -import { getAnthropicCompletion, parseAnthropicCompletion } from './anthropic' -import { getOpenAIResponsesCompletion, parseOpenAIResponsesCompletion } from './openai-responses' +import { runChatLoop } from './chatLoop' import type { ReviewChangesOpts } from './monaco-adapter' import { getCurrentModel, tryGetCurrentModel, getCombinedCustomPrompt } from '$lib/aiStore' @@ -413,130 +411,63 @@ class AIChatManager { systemMessage?: ChatCompletionSystemMessageParam }) => { try { - let addedMessages: ChatCompletionMessageParam[] = [] - while (true) { - const systemMessage = systemMessageOverride ?? this.systemMessage - const helpers = this.helpers - const tools = this.tools - for (const tool of tools) { - if (tool.setSchema) { - await tool.setSchema(helpers) - } - } - - let pendingPrompt = this.pendingPrompt - let pendingUserMessage: ChatCompletionUserMessageParam | undefined = undefined - if (pendingPrompt) { + // Use JS getters so runChatLoop re-reads tools/helpers/systemMessage/modelProvider + // on each iteration. This is critical for changeModeTool (Navigator → Script/Flow) + // which reassigns this.tools, this.helpers, this.systemMessage mid-loop. + const self = this + const result = await runChatLoop({ + messages, + get systemMessage() { + return systemMessageOverride ?? self.systemMessage + }, + get tools() { + return self.tools + }, + get helpers() { + return self.helpers + }, + abortController, + callbacks, + get modelProvider() { + return getCurrentModel() + }, + clients: { + openai: workspaceAIClients.getOpenaiClient(), + anthropic: workspaceAIClients.getAnthropicClient() + }, + workspace: get(workspaceStore) ?? '', + skipResponsesApi: this.skipResponsesApi, + onSkipResponsesApi: () => { + this.skipResponsesApi = true + }, + getPendingUserMessage: () => { + const pendingPrompt = this.pendingPrompt + if (!pendingPrompt) return undefined + this.pendingPrompt = '' if (this.mode === AIMode.SCRIPT) { - pendingUserMessage = prepareScriptUserMessage( + return prepareScriptUserMessage( pendingPrompt, this.contextManager.getSelectedContext() ) } else if (this.mode === AIMode.FLOW) { - pendingUserMessage = prepareFlowUserMessage( + return prepareFlowUserMessage( pendingPrompt, this.flowAiChatHelpers!.getFlowAndSelectedId() ) } else if (this.mode === AIMode.NAVIGATOR) { - pendingUserMessage = prepareNavigatorUserMessage(pendingPrompt) + return prepareNavigatorUserMessage(pendingPrompt) } - this.pendingPrompt = '' - } - - const model = getCurrentModel() - const isOpenAI = model.provider === 'openai' || model.provider === 'azure_openai' - const isAnthropic = model.provider === 'anthropic' - - const messageParams = [ - systemMessage, - ...messages, - ...(pendingUserMessage ? [pendingUserMessage] : []) - ] - const toolDefs = tools.map((t) => t.def) - - // For OpenAI/Azure, try Responses API first, fallback to Completions API - if (isOpenAI) { - let useCompletionsApi = this.skipResponsesApi - if (!this.skipResponsesApi) { - try { - const completion = await getOpenAIResponsesCompletion( - messageParams, - abortController, - toolDefs - ) - const continueCompletion = await parseOpenAIResponsesCompletion( - completion, - callbacks, - messages, - addedMessages, - tools, - helpers - ) - if (!continueCompletion) { - break - } - } catch (err) { - console.warn('OpenAI Responses API failed, falling back to Completions API:', err) - // If the error indicates Responses API is not available in this region, skip it for future requests - const errorMessage = err instanceof Error ? err.message : String(err) - if (errorMessage.includes('Responses API is not enabled')) { - this.skipResponsesApi = true - } - useCompletionsApi = true - } - } - - // Use Completions API if Responses API is not available or failed - if (useCompletionsApi) { - const completion = await getCompletion(messageParams, abortController, toolDefs, { - forceCompletions: true - }) - const continueCompletion = await parseOpenAICompletion( - completion, - callbacks, - messages, - addedMessages, - tools, - helpers - ) - if (!continueCompletion) { - break - } - } - } else if (isAnthropic) { - const completion = await getAnthropicCompletion(messageParams, abortController, toolDefs) - if (completion) { - const continueCompletion = await parseAnthropicCompletion( - completion, - callbacks, - messages, - addedMessages, - tools, - helpers, - abortController - ) - if (!continueCompletion) { - break - } - } - } else { - const completion = await getCompletion(messageParams, abortController, toolDefs) - if (completion) { - const continueCompletion = await parseOpenAICompletion( - completion, - callbacks, - messages, - addedMessages, - tools, - helpers - ) - if (!continueCompletion) { - break + return undefined + }, + onBeforeIteration: async (tools) => { + for (const tool of tools) { + if (tool.setSchema) { + await tool.setSchema(this.helpers) } } } - } - return addedMessages + }) + return result.addedMessages } catch (err) { console.log('chatRequest error', err) console.error('chatRequest error', err) diff --git a/frontend/src/lib/components/copilot/chat/__tests__/app/appChat.eval.test.ts b/frontend/src/lib/components/copilot/chat/__tests__/app/appChat.eval.test.ts index 5183377caf..a42ee1f099 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/app/appChat.eval.test.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/app/appChat.eval.test.ts @@ -6,44 +6,77 @@ import { loadAppFixtureForEval } from './appFixtureLoader' import { dirname, join } from 'path' // @ts-ignore - Node.js url import { fileURLToPath } from 'url' +import type { AIProvider } from '$lib/gen/types.gen' -// Get API key from environment - tests will be skipped if not set +// Get API keys from environment - tests will be skipped if none are set // @ts-ignore -const OPENROUTER_API_KEY = process.env.OPENROUTER_API_KEY +const OPENAI_API_KEY = process.env.OPENAI_API_KEY +// @ts-ignore +const ANTHROPIC_API_KEY = process.env.ANTHROPIC_API_KEY -// Skip all tests if no API key is provided -const describeWithApiKey = OPENROUTER_API_KEY ? describe : describe.skip +const hasAnyKey = OPENAI_API_KEY || ANTHROPIC_API_KEY +const describeWithApiKey = hasAnyKey ? describe : describe.skip // Get __dirname equivalent for ES modules const __filename = fileURLToPath(import.meta.url) const __dirname = dirname(__filename) -const MODELS = ['google/gemini-2.5-flash', 'anthropic/claude-haiku-4.5', 'openai/gpt-4o'] +// Build model variants based on available keys +interface ModelVariant { + model: string + provider: AIProvider + apiKey: string +} + +const MODEL_VARIANTS: ModelVariant[] = [ + ...(OPENAI_API_KEY + ? [{ model: 'gpt-4o', provider: 'openai' as AIProvider, apiKey: OPENAI_API_KEY }] + : []), + ...(ANTHROPIC_API_KEY + ? [ + { + model: 'claude-haiku-4-5-20241022', + provider: 'anthropic' as AIProvider, + apiKey: ANTHROPIC_API_KEY + } + ] + : []) +] + const VARIANTS = [ - ...MODELS.map((model) => ({ + ...MODEL_VARIANTS.map((mv) => ({ ...BASELINE_VARIANT, - model, - name: `baseline-${model.replace('/', '-')}` + model: mv.model, + name: `baseline-${mv.provider}-${mv.model}`, + _provider: mv.provider, + _apiKey: mv.apiKey })), - ...MODELS.map((model) => ({ + ...MODEL_VARIANTS.map((mv) => ({ ...STREAMLINED_VARIANT, - model, - name: `streamlined-${model.replace('/', '-')}` + model: mv.model, + name: `streamlined-${mv.provider}-${mv.model}`, + _provider: mv.provider, + _apiKey: mv.apiKey })) ] describeWithApiKey('App Chat LLM Evaluation', () => { const TEST_TIMEOUT = 120_000 - if (!OPENROUTER_API_KEY) { - console.warn('OPENROUTER_API_KEY is not set, skipping tests') + if (!hasAnyKey) { + console.warn('No API keys set (OPENAI_API_KEY or ANTHROPIC_API_KEY), skipping tests') } it( 'test1: creates a simple counter app', async () => { const USER_PROMPT = `Create a counter app with increment/decrement buttons` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!) - // Write results to files + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + undefined, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) console.log(`App files: ${appPaths.join(', ')}`) @@ -56,17 +89,21 @@ describeWithApiKey('App Chat LLM Evaluation', () => { it( 'test2: modifies existing counter app to add reset button', async () => { - // Load initial app from fixture folder const { initialFrontend, initialBackend } = await loadAppFixtureForEval( join(__dirname, 'initial', 'test1_counter_app') ) const USER_PROMPT = `Add a reset button that sets the counter back to 0` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialFrontend, - initialBackend - }) - // Write results to files + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialFrontend, + initialBackend + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) console.log(`App files: ${appPaths.join(', ')}`) @@ -86,10 +123,16 @@ describeWithApiKey('App Chat LLM Evaluation', () => { ) const USER_PROMPT = `Add a quantity selector (+ and - buttons) to each cart item so users can adjust quantities without removing and re-adding items` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialFrontend, - initialBackend - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialFrontend, + initialBackend + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -108,10 +151,16 @@ describeWithApiKey('App Chat LLM Evaluation', () => { ) const USER_PROMPT = `Add a discount code input field in the cart. When the code "SAVE10" is entered, apply a 10% discount to the total` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialFrontend, - initialBackend - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialFrontend, + initialBackend + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -132,10 +181,16 @@ describeWithApiKey('App Chat LLM Evaluation', () => { ) const USER_PROMPT = `Add a search bar in the toolbar that filters files and folders by name as the user types` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialFrontend, - initialBackend - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialFrontend, + initialBackend + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -154,10 +209,16 @@ describeWithApiKey('App Chat LLM Evaluation', () => { ) const USER_PROMPT = `Show file size (formatted as KB/MB) and modified date in the file list for each item` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialFrontend, - initialBackend - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialFrontend, + initialBackend + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -176,10 +237,16 @@ describeWithApiKey('App Chat LLM Evaluation', () => { ) const USER_PROMPT = `Add a "Select All" checkbox in the file list header and individual checkboxes for each file. Add a "Delete Selected" button that appears when items are selected` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialFrontend, - initialBackend - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialFrontend, + initialBackend + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -196,7 +263,13 @@ describeWithApiKey('App Chat LLM Evaluation', () => { 'test8: create quiz app from scratch', async () => { const USER_PROMPT = `Create a multiple choice quiz app with 5 questions about general knowledge. Show one question at a time with 4 answer options. Track the score and show results at the end with percentage correct.` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + undefined, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -211,7 +284,13 @@ describeWithApiKey('App Chat LLM Evaluation', () => { 'test9: create recipe book from scratch', async () => { const USER_PROMPT = `Create a recipe book app where users can add recipes with a name, ingredients list, and instructions. Include a search bar to filter recipes by name and the ability to delete recipes.` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + undefined, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, appPaths } = await writeAppComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) diff --git a/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalComparison.ts b/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalComparison.ts index 456299c142..e6c795d445 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalComparison.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalComparison.ts @@ -1,4 +1,4 @@ -import OpenAI from 'openai' +import Anthropic from '@anthropic-ai/sdk' import type { AppFiles, BackendRunnable } from '../../app/core' import { BASE_EVALUATOR_RESPONSE_FORMAT } from '../shared' import type { EvaluationResult } from '../shared' @@ -71,12 +71,7 @@ ${BASE_EVALUATOR_RESPONSE_FORMAT}` /** * Evaluates how well a generated app fulfills the user's request, considering any initial app state. - * This evaluator does not require an expected reference app - it evaluates based on the request alone. - * - * @param userPrompt The original user request - * @param generatedApp The app generated by the AI - * @param initialApp Optional initial app state (what the app looked like before AI changes) - * @returns Evaluation result with score, statement, and missing requirements + * Uses Anthropic API directly. */ export async function evaluateAppGeneration( userPrompt: string, @@ -84,9 +79,17 @@ export async function evaluateAppGeneration( initialApp?: InitialApp ): Promise { // @ts-ignore - const apiKey = process.env.OPENROUTER_API_KEY + const apiKey = process.env.ANTHROPIC_API_KEY + if (!apiKey) { + return { + success: false, + resemblanceScore: 0, + statement: 'No API key available for evaluation', + error: 'ANTHROPIC_API_KEY not set' + } + } - const client = new OpenAI({ baseURL: 'https://openrouter.ai/api/v1', apiKey }) + const client = new Anthropic({ apiKey }) let userMessage = `## User's Original Request ${userPrompt} @@ -117,16 +120,18 @@ Please evaluate how well the generated app: 2. ${initialApp ? 'Makes appropriate modifications to the initial app state' : 'Implements a complete and correct new app'}` try { - const response = await client.chat.completions.create({ - model: 'anthropic/claude-sonnet-4.5', + const response = await client.messages.create({ + model: 'claude-sonnet-4-5-20250514', + max_tokens: 2048, + system: APP_GENERATION_EVALUATOR_SYSTEM_PROMPT, messages: [ - { role: 'system', content: APP_GENERATION_EVALUATOR_SYSTEM_PROMPT }, { role: 'user', content: userMessage } ], temperature: 0 }) - const content = response.choices[0]?.message?.content + const textBlock = response.content.find((block) => block.type === 'text') + const content = textBlock?.text if (!content) { return { success: false, diff --git a/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalRunner.ts b/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalRunner.ts index 3f0da73c92..2e6a491bce 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalRunner.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/app/appEvalRunner.ts @@ -14,6 +14,7 @@ import { type VariantDefaults } from '../shared' import { writeAppComparisonResultsToFolders } from './appResultsWriter' +import type { AIProvider } from '$lib/gen/types.gen' // Re-export for convenience export type { InitialApp } from './appEvalComparison' @@ -38,6 +39,8 @@ export interface AppEvalOptions { variant?: VariantConfig /** Whether to evaluate the generated app with LLM. Default: true. Set to false to skip evaluation. */ evaluateWithLLM?: boolean + /** AI provider (inferred from model name if omitted) */ + provider?: AIProvider } /** @@ -49,12 +52,11 @@ const appDefaults: VariantDefaults = { } /** - * Runs an app chat evaluation with real OpenAI API calls. - * Executes tool calls using the actual app tools from core.ts or variant-configured tools. + * Runs an app chat evaluation using the shared chat loop (same code path as production). */ export async function runAppEval( userPrompt: string, - openaiApiKey: string, + apiKey: string, options?: AppEvalOptions ): Promise { const { helpers, getFiles } = createAppEvalHelpers( @@ -69,7 +71,7 @@ export async function runAppEval( appDefaults, options?.customSystemPrompt ) - const { toolDefs, tools } = resolveTools(options?.variant, appDefaults) + const { tools } = resolveTools(options?.variant, appDefaults) const model = resolveModel(options?.variant, options?.model) // Build user message @@ -80,15 +82,15 @@ export async function runAppEval( userPrompt, systemMessage, userMessage, - toolDefs, tools, helpers, - apiKey: openaiApiKey, + apiKey, getOutput: getFiles, options: { maxIterations: options?.maxIterations, model, - workspace: 'test-workspace' + workspace: 'test-workspace', + provider: options?.provider } }) @@ -114,21 +116,32 @@ export async function runAppEval( } } +/** + * Per-variant provider override. + */ +export interface VariantProviderOverride { + provider: AIProvider + apiKey: string +} + /** * Runs the same prompt against multiple variants sequentially for comparison. - * Returns results in the same order as the input variants. + * Accepts optional per-variant provider/apiKey overrides. */ export async function runVariantComparison( userPrompt: string, variants: VariantConfig[], - openaiApiKey: string, - baseOptions?: Omit + defaultApiKey: string, + baseOptions?: Omit, + providerOverrides?: VariantProviderOverride[] ): Promise { const results: AppEvalResult[] = await Promise.all( - variants.map(async (variant) => { - return await runAppEval(userPrompt, openaiApiKey, { + variants.map(async (variant, i) => { + const override = providerOverrides?.[i] + return await runAppEval(userPrompt, override?.apiKey ?? defaultApiKey, { ...baseOptions, - variant + variant, + provider: override?.provider ?? baseOptions?.provider }) }) ) diff --git a/frontend/src/lib/components/copilot/chat/__tests__/flow/flowChat.eval.test.ts b/frontend/src/lib/components/copilot/chat/__tests__/flow/flowChat.eval.test.ts index 8210ea50fb..de9b8e5f43 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/flow/flowChat.eval.test.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/flow/flowChat.eval.test.ts @@ -22,35 +22,60 @@ import initialTest6 from './initial/test6_initial.json' // @ts-ignore - JSON import import initialTest7 from './initial/test7_initial.json' import type { FlowModule } from '$lib/gen' +import type { AIProvider } from '$lib/gen/types.gen' -// Get API key from environment - tests will be skipped if not set +// Get API keys from environment - tests will be skipped if none are set // @ts-ignore -// const OPENAI_API_KEY = process.env.OPENAI_API_KEY -const OPENROUTER_API_KEY = process.env.OPENROUTER_API_KEY +const OPENAI_API_KEY = process.env.OPENAI_API_KEY +// @ts-ignore +const ANTHROPIC_API_KEY = process.env.ANTHROPIC_API_KEY -// Skip all tests if no API key is provided -// const describeWithApiKey = OPENAI_API_KEY ? describe : describe.skip -const describeWithApiKey = OPENROUTER_API_KEY ? describe : describe.skip +const hasAnyKey = OPENAI_API_KEY || ANTHROPIC_API_KEY +const describeWithApiKey = hasAnyKey ? describe : describe.skip -const MODELS = ['google/gemini-2.5-flash', 'anthropic/claude-haiku-4.5', 'openai/gpt-4o'] +// Build model variants based on available keys +interface ModelVariant { + model: string + provider: AIProvider + apiKey: string +} + +const MODEL_VARIANTS: ModelVariant[] = [ + ...(OPENAI_API_KEY + ? [{ model: 'gpt-4o', provider: 'openai' as AIProvider, apiKey: OPENAI_API_KEY }] + : []), + ...(ANTHROPIC_API_KEY + ? [ + { + model: 'claude-haiku-4-5-20241022', + provider: 'anthropic' as AIProvider, + apiKey: ANTHROPIC_API_KEY + } + ] + : []) +] const VARIANTS = [ - ...MODELS.map((model) => ({ + ...MODEL_VARIANTS.map((mv) => ({ ...BASELINE_VARIANT, - model, - name: `baseline-${model.replace('/', '-')}` + model: mv.model, + name: `baseline-${mv.provider}-${mv.model}`, + _provider: mv.provider, + _apiKey: mv.apiKey })), - ...MODELS.map((model) => ({ + ...MODEL_VARIANTS.map((mv) => ({ ...MINIMAL_SINGLE_TOOL_VARIANT, - model, - name: `minimal-single-tool-${model.replace('/', '-')}` + model: mv.model, + name: `minimal-single-tool-${mv.provider}-${mv.model}`, + _provider: mv.provider, + _apiKey: mv.apiKey })) ] describeWithApiKey('Flow Chat LLM Evaluation', () => { const TEST_TIMEOUT = 120_000 - if (!OPENROUTER_API_KEY) { - console.warn('OPENROUTER_API_KEY is not set, skipping tests') + if (!hasAnyKey) { + console.warn('No API keys set (OPENAI_API_KEY or ANTHROPIC_API_KEY), skipping tests') } it( @@ -65,9 +90,15 @@ STEP 3: Loop on all users STEP 4: Do branches based on user's role, do different action based on that. Roles are admin, user, moderator STEP 5: Return action taken for each user ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - expectedFlow: expectedTest1 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + expectedFlow: expectedTest1 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) // Write results to files const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) @@ -112,9 +143,15 @@ STEP 5: Branch based on inventory - if all items available, create shipment reco STEP 6: Send confirmation (mock email to customer_email) STEP 7: Return final order summary with status ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - expectedFlow: expectedTest2 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + expectedFlow: expectedTest2 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -161,9 +198,15 @@ STEP 5: Branch based on quality score: - If score < 70: Store in quarantine and send alert STEP 6: Return processing report with statistics (total records, quality score, destination) ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - expectedFlow: expectedTest3 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + expectedFlow: expectedTest3 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -210,9 +253,15 @@ STEP 3: Use an AI agent to handle the customer query. The agent should have acce STEP 4: Log the interaction to audit trail (customer_id, query, response summary) STEP 5: Return the agent's response and any actions taken ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - expectedFlow: expectedTest4 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + expectedFlow: expectedTest4 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -256,11 +305,17 @@ Modify this existing flow to add error handling: - If validation passes, return the data for the next step - Update save_results to handle the validation result appropriately ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialModules: initialTest5.value.modules as FlowModule[], - initialSchema: initialTest5.schema, - expectedFlow: expectedTest5 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialModules: initialTest5.value.modules as FlowModule[], + initialSchema: initialTest5.schema, + expectedFlow: expectedTest5 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -302,11 +357,17 @@ Modify the order processing loop to handle different order types: - Move the original process_order step to the default branch for unknown order types - Each branch step should return the orderId, shipping cost, and shipping type ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialModules: initialTest6.value.modules as FlowModule[], - initialSchema: initialTest6.schema, - expectedFlow: expectedTest6 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialModules: initialTest6.value.modules as FlowModule[], + initialSchema: initialTest6.schema, + expectedFlow: expectedTest6 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) @@ -348,11 +409,17 @@ Refactor this flow for better performance by parallelizing the enrichment steps: - The combine_data step should check if any enrichment used a fallback value and set a hasFallbacks flag - Keep get_item as the first step and return_result as the last step unchanged ` - const results = await runVariantComparison(USER_PROMPT, VARIANTS, OPENROUTER_API_KEY!, { - initialModules: initialTest7.value.modules as FlowModule[], - initialSchema: initialTest7.schema, - expectedFlow: expectedTest7 as ExpectedFlow - }) + const results = await runVariantComparison( + USER_PROMPT, + VARIANTS, + VARIANTS[0]._apiKey, + { + initialModules: initialTest7.value.modules as FlowModule[], + initialSchema: initialTest7.schema, + expectedFlow: expectedTest7 as ExpectedFlow + }, + VARIANTS.map((v) => ({ provider: v._provider, apiKey: v._apiKey })) + ) const { summaryPath, flowPaths } = await writeFlowComparisonResults(USER_PROMPT, results) console.log(`\nResults written to: ${summaryPath}`) diff --git a/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalComparison.ts b/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalComparison.ts index f55979bb40..4c2b41d577 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalComparison.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalComparison.ts @@ -59,14 +59,10 @@ export async function evaluateFlowComparison( expectedFlow: ExpectedFlow, userPrompt: string ): Promise { - // @ts-ignore - const apiKey = process.env.OPENROUTER_API_KEY - return evaluateWithLLM({ userPrompt, generatedOutput: generatedFlow, expectedOutput: expectedFlow, - evaluatorSystemPrompt: FLOW_EVALUATOR_SYSTEM_PROMPT, - apiKey + evaluatorSystemPrompt: FLOW_EVALUATOR_SYSTEM_PROMPT }) } diff --git a/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalRunner.ts b/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalRunner.ts index 3f27143c69..f3c976950d 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalRunner.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/flow/flowEvalRunner.ts @@ -1,4 +1,5 @@ import type { FlowModule } from '$lib/gen' +import type { AIProvider } from '$lib/gen/types.gen' import type { ExtendedOpenFlow } from '$lib/components/flows/types' import { flowTools, prepareFlowSystemMessage, prepareFlowUserMessage, type FlowAIChatHelpers } from '../../flow/core' import { createFlowEvalHelpers } from './flowEvalHelpers' @@ -38,6 +39,8 @@ export interface FlowEvalOptions { maxIterations?: number variant?: VariantConfig expectedFlow?: ExpectedFlow + /** AI provider (inferred from model name if omitted) */ + provider?: AIProvider } /** @@ -49,12 +52,11 @@ const flowDefaults: VariantDefaults = { } /** - * Runs a flow chat evaluation with real OpenAI API calls. - * Executes tool calls using the actual flowTools from core.ts or variant-configured tools. + * Runs a flow chat evaluation using the shared chat loop (same code path as production). */ export async function runFlowEval( userPrompt: string, - openaiApiKey: string, + apiKey: string, options?: FlowEvalOptions ): Promise { const { helpers, getFlow } = createFlowEvalHelpers( @@ -65,7 +67,7 @@ export async function runFlowEval( // Resolve variant configuration const variantName = options?.variant?.name ?? 'baseline' const systemMessage = resolveSystemPrompt(options?.variant, flowDefaults, options?.customSystemPrompt) - const { toolDefs, tools } = resolveTools(options?.variant, flowDefaults) + const { tools } = resolveTools(options?.variant, flowDefaults) const model = resolveModel(options?.variant, options?.model) // Build user message @@ -76,15 +78,15 @@ export async function runFlowEval( userPrompt, systemMessage, userMessage, - toolDefs, tools, helpers, - apiKey: openaiApiKey, + apiKey, getOutput: getFlow, options: { maxIterations: options?.maxIterations, model, - workspace: 'test-workspace' + workspace: 'test-workspace', + provider: options?.provider } }) @@ -111,21 +113,32 @@ export async function runFlowEval( } } +/** + * Per-variant provider override. + */ +export interface VariantProviderOverride { + provider: AIProvider + apiKey: string +} + /** * Runs the same prompt against multiple variants sequentially for comparison. - * Returns results in the same order as the input variants. + * Accepts optional per-variant provider/apiKey overrides. */ export async function runVariantComparison( userPrompt: string, variants: VariantConfig[], - openaiApiKey: string, - baseOptions?: Omit + defaultApiKey: string, + baseOptions?: Omit, + providerOverrides?: VariantProviderOverride[] ): Promise { const results: FlowEvalResult[] = await Promise.all( - variants.map(async (variant) => { - return await runFlowEval(userPrompt, openaiApiKey, { + variants.map(async (variant, i) => { + const override = providerOverrides?.[i] + return await runFlowEval(userPrompt, override?.apiKey ?? defaultApiKey, { ...baseOptions, - variant + variant, + provider: override?.provider ?? baseOptions?.provider }) }) ) diff --git a/frontend/src/lib/components/copilot/chat/__tests__/shared/baseEvalRunner.ts b/frontend/src/lib/components/copilot/chat/__tests__/shared/baseEvalRunner.ts index b9b7820568..f46acb9108 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/shared/baseEvalRunner.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/shared/baseEvalRunner.ts @@ -1,8 +1,14 @@ -import OpenAI, { APIError } from 'openai' -import type { ChatCompletionMessageParam, ChatCompletionSystemMessageParam } from 'openai/resources/chat/completions.mjs' -import type { ChatCompletionTool } from 'openai/resources/chat/completions.mjs' +import OpenAI from 'openai' +import Anthropic from '@anthropic-ai/sdk' +import type { + ChatCompletionMessageParam, + ChatCompletionSystemMessageParam +} from 'openai/resources/chat/completions.mjs' +import type { AIProvider, AIProviderModel } from '$lib/gen/types.gen' import type { TokenUsage, ToolCallDetail, EvalRunnerOptions } from './types' import type { Tool } from './baseVariants' +import { runChatLoop, type ChatClients } from '../../chatLoop' +import type { Tool as ProductionTool, ToolCallbacks } from '../../shared' /** * Result from a single eval run (before domain-specific evaluation). @@ -29,13 +35,13 @@ export interface RunEvalParams { systemMessage: ChatCompletionSystemMessageParam /** User message for the LLM */ userMessage: ChatCompletionMessageParam - /** Tool definitions for the LLM API */ - toolDefs: ChatCompletionTool[] + /** Tool definitions for the LLM API (unused — derived from tools) */ + toolDefs?: unknown /** Full tool implementations for execution */ tools: Tool[] /** Domain-specific helpers for tool execution */ helpers: THelpers - /** API key for OpenRouter */ + /** API key for the provider */ apiKey: string /** Function to get the current output state */ getOutput: () => TOutput @@ -44,10 +50,37 @@ export interface RunEvalParams { } /** - * Runs a generic evaluation with real LLM API calls. - * Executes tool calls in a loop until the LLM stops calling tools. - * - * This is the core execution loop shared across all chat eval tests. + * Creates SDK clients for the given provider. + */ +function createEvalClients(provider: AIProvider, apiKey: string): ChatClients { + if (provider === 'anthropic') { + return { + openai: new OpenAI({ apiKey: 'unused' }), + anthropic: new Anthropic({ apiKey }) + } + } + return { + openai: new OpenAI({ apiKey }), + anthropic: new Anthropic({ apiKey: 'unused' }) + } +} + +/** + * Resolves model string to AIProviderModel. + */ +function resolveModelProvider( + model: string, + provider?: AIProvider +): AIProviderModel { + if (provider) return { provider, model } + if (model.startsWith('claude')) return { provider: 'anthropic', model } + if (model.startsWith('gpt') || model.startsWith('o')) return { provider: 'openai', model } + return { provider: 'openai', model } +} + +/** + * Runs a generic evaluation using the shared chat loop (same code path as production). + * Uses streaming via real provider SDKs instead of OpenRouter non-streaming. */ export async function runEval( params: RunEvalParams @@ -55,7 +88,6 @@ export async function runEval( const { systemMessage, userMessage, - toolDefs, tools, helpers, apiKey, @@ -63,134 +95,82 @@ export async function runEval( options } = params - const client = new OpenAI({ baseURL: 'https://openrouter.ai/api/v1', apiKey }) const model = options?.model ?? 'gpt-4o' const maxIterations = options?.maxIterations ?? 20 const workspace = options?.workspace ?? 'test-workspace' + const provider = options?.provider - const messages: ChatCompletionMessageParam[] = [systemMessage, userMessage] - const totalTokens: TokenUsage = { prompt: 0, completion: 0, total: 0 } + const modelProvider = resolveModelProvider(model, provider) + const clients = createEvalClients(modelProvider.provider, apiKey) + + const messages: ChatCompletionMessageParam[] = [userMessage] let toolCallsCount = 0 const toolsCalled: string[] = [] const toolCallDetails: ToolCallDetail[] = [] - let iterations = 0 - // No-op tool callbacks for eval - const toolCallbacks = { + // Wrap tools to intercept fn calls for tracking. + // Cast to ProductionTool since the eval Tool has a narrower toolCallbacks type + // but the actual callbacks passed at runtime will satisfy both interfaces. + const wrappedTools = tools.map((tool) => ({ + ...tool, + fn: async (p: any) => { + toolCallsCount++ + toolsCalled.push(tool.def.function.name) + try { + const args = + typeof p.args === 'string' ? JSON.parse(p.args) : p.args + toolCallDetails.push({ name: tool.def.function.name, arguments: args }) + } catch { + toolCallDetails.push({ + name: tool.def.function.name, + arguments: p.args + }) + } + return tool.fn(p) + } + })) as ProductionTool[] + + // No-op callbacks for eval + const callbacks: ToolCallbacks & { + onNewToken: (token: string) => void + onMessageEnd: () => void + } = { setToolStatus: () => {}, - removeToolStatus: () => {} + removeToolStatus: () => {}, + onNewToken: () => {}, + onMessageEnd: () => {} } + const abortController = new AbortController() + try { - // Tool resolution loop - while (iterations < maxIterations) { - iterations++ - - const response = await client.chat.completions.create({ - model, - messages, - tools: toolDefs, - temperature: 0 - }) - - // Track token usage - if (response.usage) { - totalTokens.prompt += response.usage.prompt_tokens - totalTokens.completion += response.usage.completion_tokens - totalTokens.total += response.usage.total_tokens - } - - if (!response.choices.length) { - throw new Error('No response from API') - } - - const choice = response.choices[0] - const assistantMessage = choice.message - - // Add assistant message to history - messages.push(assistantMessage) - - // If no tool calls, we're done - if (!assistantMessage.tool_calls?.length) { - break - } - - // Execute each tool call - for (const toolCall of assistantMessage.tool_calls) { - toolCallsCount++ - - // Type guard: only handle function tool calls - if (toolCall.type !== 'function') { - messages.push({ - role: 'tool', - tool_call_id: toolCall.id, - content: `Unsupported tool type: ${toolCall.type}` - }) - continue - } - - toolsCalled.push(toolCall.function.name) - - const tool = tools.find((t) => t.def.function.name === toolCall.function.name) - if (!tool) { - messages.push({ - role: 'tool', - tool_call_id: toolCall.id, - content: `Unknown tool: ${toolCall.function.name}` - }) - continue - } - - try { - const args = JSON.parse(toolCall.function.arguments) - toolCallDetails.push({ name: toolCall.function.name, arguments: args }) - const result = await tool.fn({ - args, - workspace, - helpers, - toolCallbacks, - toolId: toolCall.id - }) - messages.push({ - role: 'tool', - tool_call_id: toolCall.id, - content: result - }) - } catch (err) { - const errorMessage = err instanceof Error ? err.message : String(err) - messages.push({ - role: 'tool', - tool_call_id: toolCall.id, - content: `Error: ${errorMessage}` - }) - } - } - } + const result = await runChatLoop({ + messages, + systemMessage, + tools: wrappedTools, + helpers, + abortController, + callbacks, + modelProvider, + clients, + workspace, + maxIterations, + skipResponsesApi: modelProvider.provider !== 'openai' && modelProvider.provider !== 'azure_openai' + }) return { success: true, output: getOutput(), - tokenUsage: totalTokens, + tokenUsage: { prompt: 0, completion: 0, total: 0 }, toolCallsCount, toolsCalled, toolCallDetails, - iterations, + iterations: Math.max(1, result.addedMessages.filter((m) => m.role === 'assistant').length), messages } } catch (err) { - // Build detailed error message let errorMessage: string - if (err instanceof APIError) { - const details: string[] = [`${err.status} ${err.message}`] - if (err.code) details.push(`Code: ${err.code}`) - if (err.type) details.push(`Type: ${err.type}`) - if (err.param) details.push(`Param: ${err.param}`) - if (err.requestID) details.push(`Request ID: ${err.requestID}`) - if (err.error && typeof err.error === 'object') { - details.push(`Response: ${JSON.stringify(err.error, null, 2)}`) - } - errorMessage = details.join('\n') - } else if (err instanceof Error) { + if (err instanceof Error) { errorMessage = err.stack ?? err.message } else { errorMessage = String(err) @@ -200,11 +180,11 @@ export async function runEval( success: false, output: getOutput(), error: errorMessage, - tokenUsage: totalTokens, + tokenUsage: { prompt: 0, completion: 0, total: 0 }, toolCallsCount, toolsCalled, toolCallDetails, - iterations, + iterations: 0, messages } } diff --git a/frontend/src/lib/components/copilot/chat/__tests__/shared/baseLLMEvaluator.ts b/frontend/src/lib/components/copilot/chat/__tests__/shared/baseLLMEvaluator.ts index 63c17828f4..bd7bd06d44 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/shared/baseLLMEvaluator.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/shared/baseLLMEvaluator.ts @@ -1,4 +1,4 @@ -import OpenAI from 'openai' +import Anthropic from '@anthropic-ai/sdk' import type { EvaluationResult } from './types' /** @@ -13,9 +13,9 @@ export interface EvaluateParams { expectedOutput: unknown /** Domain-specific system prompt for the evaluator */ evaluatorSystemPrompt: string - /** API key for OpenRouter */ - apiKey: string - /** Model to use for evaluation (default: 'anthropic/claude-sonnet-4.5') */ + /** Anthropic API key for evaluation */ + apiKey?: string + /** Model to use for evaluation (default: 'claude-sonnet-4-5-20250514') */ model?: string } @@ -41,10 +41,7 @@ Score guidelines: /** * Evaluates how well a generated output matches an expected output using an LLM. - * Returns a resemblance score (0-100), a qualitative statement, and any missing requirements. - * - * @param params Evaluation parameters including prompts, outputs, and API configuration - * @returns Evaluation result with score, statement, and missing requirements + * Uses Anthropic API directly instead of OpenRouter. */ export async function evaluateWithLLM(params: EvaluateParams): Promise { const { @@ -53,10 +50,21 @@ export async function evaluateWithLLM(params: EvaluateParams): Promise block.type === 'text') + const content = textBlock?.text if (!content) { return { success: false, @@ -98,7 +108,6 @@ Please evaluate how well the generated output: // Parse JSON response - handle potential markdown code blocks let jsonContent = content.trim() if (jsonContent.startsWith('```')) { - // Remove markdown code block wrapper jsonContent = jsonContent.replace(/^```(?:json)?\n?/, '').replace(/\n?```$/, '') } diff --git a/frontend/src/lib/components/copilot/chat/__tests__/shared/types.ts b/frontend/src/lib/components/copilot/chat/__tests__/shared/types.ts index 021e776440..61f7f1fd1f 100644 --- a/frontend/src/lib/components/copilot/chat/__tests__/shared/types.ts +++ b/frontend/src/lib/components/copilot/chat/__tests__/shared/types.ts @@ -1,4 +1,5 @@ import type { ChatCompletionMessageParam } from 'openai/resources/chat/completions.mjs' +import type { AIProvider } from '$lib/gen/types.gen' /** * Token usage tracking for LLM calls. @@ -83,6 +84,8 @@ export interface EvalRunnerOptions { model?: string /** Workspace ID for tool calls */ workspace?: string + /** AI provider (inferred from model name if omitted) */ + provider?: AIProvider } /** diff --git a/frontend/src/lib/components/copilot/chat/anthropic.ts b/frontend/src/lib/components/copilot/chat/anthropic.ts index 03d0f363a0..ac45c175a6 100644 --- a/frontend/src/lib/components/copilot/chat/anthropic.ts +++ b/frontend/src/lib/components/copilot/chat/anthropic.ts @@ -1,4 +1,5 @@ import { OpenAI } from 'openai' +import Anthropic from '@anthropic-ai/sdk' import type { ChatCompletionMessageParam, ChatCompletionMessageFunctionToolCall @@ -13,19 +14,28 @@ import type { RawMessageStreamEvent } from '@anthropic-ai/sdk/resources' import type { MessageStream } from '@anthropic-ai/sdk/lib/MessageStream' +import type { AIProviderModel } from '$lib/gen' import { getProviderAndCompletionConfig, workspaceAIClients } from '../lib' import { processToolCall, type Tool, type ToolCallbacks } from './shared' export async function getAnthropicCompletion( messages: ChatCompletionMessageParam[], abortController: AbortController, - tools?: OpenAI.Chat.Completions.ChatCompletionFunctionTool[] + tools?: OpenAI.Chat.Completions.ChatCompletionFunctionTool[], + options?: { + forceModelProvider?: AIProviderModel + anthropicClient?: Anthropic + } ): Promise { - const { provider, config } = getProviderAndCompletionConfig({ messages, stream: true }) + const { provider, config } = getProviderAndCompletionConfig({ + messages, + stream: true, + forceModelProvider: options?.forceModelProvider + }) const { system, messages: anthropicMessages } = convertOpenAIToAnthropicMessages(messages) const anthropicTools = convertOpenAIToolsToAnthropic(tools) - const anthropicClient = workspaceAIClients.getAnthropicClient() + const client = options?.anthropicClient ?? workspaceAIClients.getAnthropicClient() const anthropicParams = { model: config.model, @@ -36,7 +46,7 @@ export async function getAnthropicCompletion( ...(typeof config.temperature === 'number' && { temperature: config.temperature }) } - const stream = anthropicClient.messages.stream(anthropicParams, { + const stream = client.messages.stream(anthropicParams, { signal: abortController.signal, headers: { 'X-Provider': provider, @@ -58,7 +68,8 @@ export async function parseAnthropicCompletion( addedMessages: ChatCompletionMessageParam[], tools: Tool[], helpers: any, - abortController?: AbortController + abortController?: AbortController, + options?: { workspace?: string } ): Promise { let toolCallsToProcess: ChatCompletionMessageFunctionToolCall[] = [] let error = null @@ -209,7 +220,8 @@ export async function parseAnthropicCompletion( tools, toolCall, helpers, - toolCallbacks: callbacks + toolCallbacks: callbacks, + workspace: options?.workspace }) messages.push(messageToAdd) addedMessages.push(messageToAdd) diff --git a/frontend/src/lib/components/copilot/chat/app/core.ts b/frontend/src/lib/components/copilot/chat/app/core.ts index 8fc88d4d16..c6a64a3e5a 100644 --- a/frontend/src/lib/components/copilot/chat/app/core.ts +++ b/frontend/src/lib/components/copilot/chat/app/core.ts @@ -10,6 +10,7 @@ import { createGetRunnableDetailsTool, type Tool } from '../shared' +import { getDatatableSdkReference } from '$system_prompts' import { aiChatManager } from '../AIChatManager.svelte' import type { ContextElement, @@ -842,38 +843,30 @@ For inline scripts, the code must have a \`main\` function as its entrypoint. Backend runnables should only perform **data operations** (SELECT, INSERT, UPDATE, DELETE) on **existing tables**. Never use CREATE TABLE, DROP TABLE, or ALTER TABLE inside runnables. -**TypeScript (Bun)**: +**TypeScript (Bun) example**: \`\`\`typescript import * as wmill from 'windmill-client'; export async function main(user_id: string) { const sql = ${datatableCall}; - - // Safe string interpolation (parameterized query) const user = await sql\`SELECT * FROM ${schemaPrefix}users WHERE id = \${user_id}\`.fetchOne(); return user; } \`\`\` -**Python**: +**Python example**: \`\`\`python import wmill def main(user_id: str): db = ${datatableCall} - - # Use positional arguments ($1, $2, etc.) user = db.query('SELECT * FROM ${schemaPrefix}users WHERE id = $1', user_id).fetch_one() return user \`\`\` -### Common Operations (for use in backend runnables) +### Datatable Client API Reference -- **Fetch all**: \`sql\`SELECT * FROM ${schemaPrefix}table\`.fetch()\` or \`db.query('SELECT * FROM ${schemaPrefix}table').fetch()\` -- **Fetch one**: \`.fetchOne()\` or \`.fetch_one()\` -- **Insert**: \`sql\`INSERT INTO ${schemaPrefix}table (col) VALUES (\${value})\`\` -- **Update**: \`sql\`UPDATE ${schemaPrefix}table SET col = \${value} WHERE id = \${id}\`\` -- **Delete**: \`sql\`DELETE FROM ${schemaPrefix}table WHERE id = \${id}\`\` +${getDatatableSdkReference()} ### Schema Modifications (DDL) - Use exec_datatable_sql tool ONLY diff --git a/frontend/src/lib/components/copilot/chat/chatLoop.ts b/frontend/src/lib/components/copilot/chat/chatLoop.ts new file mode 100644 index 0000000000..4b239e4a05 --- /dev/null +++ b/frontend/src/lib/components/copilot/chat/chatLoop.ts @@ -0,0 +1,211 @@ +import OpenAI from 'openai' +import Anthropic from '@anthropic-ai/sdk' +import type { + ChatCompletionMessageParam, + ChatCompletionSystemMessageParam, + ChatCompletionUserMessageParam +} from 'openai/resources/chat/completions.mjs' +import type { AIProviderModel } from '$lib/gen' +import { getCompletion, parseOpenAICompletion } from '../lib' +import { getAnthropicCompletion, parseAnthropicCompletion } from './anthropic' +import { + getOpenAIResponsesCompletion, + parseOpenAIResponsesCompletion +} from './openai-responses' +import type { Tool, ToolCallbacks } from './shared' + +export interface ChatClients { + openai: OpenAI + anthropic: Anthropic +} + +export interface ChatLoopConfig { + messages: ChatCompletionMessageParam[] + /** + * System message, tools, helpers, and modelProvider are re-read from this config + * on every iteration. Callers can use JS getters to provide dynamic values + * (e.g. AIChatManager uses getters so mode changes mid-loop take effect). + */ + systemMessage: ChatCompletionSystemMessageParam + tools: Tool[] + helpers: any + abortController: AbortController + callbacks: ToolCallbacks & { + onNewToken: (token: string) => void + onMessageEnd: () => void + } + modelProvider: AIProviderModel + clients: ChatClients + workspace: string + /** Maximum iterations for the loop. undefined = unlimited (production). */ + maxIterations?: number + skipResponsesApi?: boolean + onSkipResponsesApi?: () => void + /** Return a pending user message to inject between iterations, or undefined. */ + getPendingUserMessage?: () => ChatCompletionUserMessageParam | undefined + /** Called before each iteration (e.g. to refresh tool schemas). */ + onBeforeIteration?: (tools: Tool[], helpers: any) => Promise +} + +export interface ChatLoopResult { + addedMessages: ChatCompletionMessageParam[] +} + +export async function runChatLoop(config: ChatLoopConfig): Promise { + const { + messages, + abortController, + callbacks, + clients, + workspace, + maxIterations, + onSkipResponsesApi, + getPendingUserMessage, + onBeforeIteration + } = config + let skipResponsesApi = config.skipResponsesApi ?? false + + const addedMessages: ChatCompletionMessageParam[] = [] + let iterations = 0 + + while (true) { + if (maxIterations !== undefined && iterations >= maxIterations) { + break + } + iterations++ + + // Re-read these from config each iteration so that mode changes + // (e.g. changeModeTool in Navigator) take effect immediately. + // Callers can use JS getter properties to provide dynamic values. + const tools = config.tools + const helpers = config.helpers + const systemMessage = config.systemMessage + const modelProvider = config.modelProvider + + if (onBeforeIteration) { + await onBeforeIteration(tools, helpers) + } + + const pendingUserMessage = getPendingUserMessage?.() + + const isOpenAI = + modelProvider.provider === 'openai' || modelProvider.provider === 'azure_openai' + const isAnthropic = modelProvider.provider === 'anthropic' + + const messageParams = [ + systemMessage, + ...messages, + ...(pendingUserMessage ? [pendingUserMessage] : []) + ] + const toolDefs = tools.map((t) => t.def) + const parseOptions = { workspace } + + if (isOpenAI) { + let useCompletionsApi = skipResponsesApi + if (!skipResponsesApi) { + try { + const completion = await getOpenAIResponsesCompletion( + messageParams, + abortController, + toolDefs, + { + forceModelProvider: modelProvider, + openaiClient: clients.openai + } + ) + const continueCompletion = await parseOpenAIResponsesCompletion( + completion, + callbacks, + messages, + addedMessages, + tools, + helpers, + parseOptions + ) + if (!continueCompletion) { + break + } + } catch (err) { + console.warn( + 'OpenAI Responses API failed, falling back to Completions API:', + err + ) + const errorMessage = err instanceof Error ? err.message : String(err) + if (errorMessage.includes('Responses API is not enabled')) { + skipResponsesApi = true + onSkipResponsesApi?.() + } + useCompletionsApi = true + } + } + + if (useCompletionsApi) { + const completion = await getCompletion(messageParams, abortController, toolDefs, { + forceCompletions: true, + forceModelProvider: modelProvider, + openaiClient: clients.openai + }) + const continueCompletion = await parseOpenAICompletion( + completion, + callbacks, + messages, + addedMessages, + tools, + helpers, + undefined, + parseOptions + ) + if (!continueCompletion) { + break + } + } + } else if (isAnthropic) { + const completion = await getAnthropicCompletion( + messageParams, + abortController, + toolDefs, + { + forceModelProvider: modelProvider, + anthropicClient: clients.anthropic + } + ) + if (completion) { + const continueCompletion = await parseAnthropicCompletion( + completion, + callbacks, + messages, + addedMessages, + tools, + helpers, + abortController, + parseOptions + ) + if (!continueCompletion) { + break + } + } + } else { + const completion = await getCompletion(messageParams, abortController, toolDefs, { + forceModelProvider: modelProvider, + openaiClient: clients.openai + }) + if (completion) { + const continueCompletion = await parseOpenAICompletion( + completion, + callbacks, + messages, + addedMessages, + tools, + helpers, + undefined, + parseOptions + ) + if (!continueCompletion) { + break + } + } + } + } + + return { addedMessages } +} diff --git a/frontend/src/lib/components/copilot/chat/flow/openFlow.json b/frontend/src/lib/components/copilot/chat/flow/openFlow.json index 01dc802a69..308b35e416 100644 --- a/frontend/src/lib/components/copilot/chat/flow/openFlow.json +++ b/frontend/src/lib/components/copilot/chat/flow/openFlow.json @@ -1 +1 @@ -{"openapi":"3.0.3","info":{"version":"1.624.0","title":"OpenFlow Spec","contact":{"name":"Ruben Fiszel","email":"ruben@windmill.dev","url":"https://windmill.dev"},"license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html"}},"paths":{},"externalDocs":{"description":"documentation portal","url":"https://windmill.dev"},"components":{"schemas":{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"number","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"number","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"number","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"flow_env":{"type":"object","description":"Environment variables available to all steps","additionalProperties":{"type":"string"}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","description":"Custom error message shown when stopping"}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_use":{"type":"boolean","description":"If true, this step's result is deleted after use to save memory"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind"}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type"}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable"]},"access_type":{"type":"string","description":"Access level for this asset","enum":["r","w","rw"]},"alt_access_type":{"type":"string","description":"Alternative access level","enum":["r","w","rw"]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}]},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_images":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of image references for vision-capable models.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'images/photo.jpg' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}}} \ No newline at end of file +{"openapi":"3.0.3","info":{"version":"1.664.0","title":"OpenFlow Spec","contact":{"name":"Ruben Fiszel","email":"ruben@windmill.dev","url":"https://windmill.dev"},"license":{"name":"Apache 2.0","url":"https://www.apache.org/licenses/LICENSE-2.0.html"}},"paths":{},"externalDocs":{"description":"documentation portal","url":"https://windmill.dev"},"components":{"schemas":{"OpenFlow":{"type":"object","description":"Top-level flow definition containing metadata, configuration, and the flow structure","properties":{"summary":{"type":"string","description":"Short description of what this flow does"},"description":{"type":"string","description":"Detailed documentation for this flow"},"value":{"$ref":"#/components/schemas/FlowValue"},"schema":{"type":"object","description":"JSON Schema for flow inputs. Use this to define input parameters, their types, defaults, and validation. For resource inputs, set type to 'object' and format to 'resource-' (e.g., 'resource-stripe')"},"on_behalf_of_email":{"type":"string","description":"The flow will be run with the permissions of the user with this email."}},"required":["summary","value"]},"FlowValue":{"type":"object","description":"The flow structure containing modules and optional preprocessor/failure handlers","properties":{"modules":{"type":"array","description":"Array of steps that execute in sequence. Each step can be a script, subflow, loop, or branch","items":{"$ref":"#/components/schemas/FlowModule"}},"failure_module":{"description":"Special module that executes when the flow fails. Receives error object with message, name, stack, and step_id. Must have id 'failure'. Only supports script/rawscript types","$ref":"#/components/schemas/FlowModule"},"preprocessor_module":{"description":"Special module that runs before the first step on external triggers. Must have id 'preprocessor'. Only supports script/rawscript types. Cannot reference other step results","$ref":"#/components/schemas/FlowModule"},"same_worker":{"type":"boolean","description":"If true, all steps run on the same worker for better performance"},"concurrent_limit":{"type":"number","description":"Maximum number of concurrent executions of this flow"},"concurrency_key":{"type":"string","description":"Expression to group concurrent executions (e.g., by user ID)"},"concurrency_time_window_s":{"type":"number","description":"Time window in seconds for concurrent_limit"},"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce flow executions"},"debounce_key":{"type":"string","description":"Expression to group debounced executions"},"debounce_args_to_accumulate":{"type":"array","description":"Arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds that a job can be debounced"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of times a job can be debounced"},"skip_expr":{"type":"string","description":"JavaScript expression to conditionally skip the entire flow"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for flow results"},"cache_ignore_s3_path":{"type":"boolean"},"flow_env":{"type":"object","description":"Environment variables available to all steps. Values can be strings, JSON values, or special references: '$var:path' (workspace variable) or '$res:path' (resource).","additionalProperties":{}},"priority":{"type":"number","description":"Execution priority (higher numbers run first)"},"early_return":{"type":"string","description":"JavaScript expression to return early from the flow"},"chat_input_enabled":{"type":"boolean","description":"Whether this flow accepts chat-style input"},"notes":{"type":"array","description":"Sticky notes attached to the flow","items":{"$ref":"#/components/schemas/FlowNote"}},"groups":{"type":"array","description":"Semantic groups of modules for organizational purposes","items":{"$ref":"#/components/schemas/FlowGroup"}}},"required":["modules"]},"Retry":{"type":"object","description":"Retry configuration for failed module executions","properties":{"constant":{"type":"object","description":"Retry with constant delay between attempts","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"seconds":{"type":"integer","description":"Seconds to wait between retries"}}},"exponential":{"type":"object","description":"Retry with exponential backoff (delay doubles each time)","properties":{"attempts":{"type":"integer","description":"Number of retry attempts"},"multiplier":{"type":"integer","description":"Multiplier for exponential backoff"},"seconds":{"type":"integer","minimum":1,"description":"Initial delay in seconds"},"random_factor":{"type":"integer","minimum":0,"maximum":100,"description":"Random jitter percentage (0-100) to avoid thundering herd"}}},"retry_if":{"$ref":"#/components/schemas/RetryIf"}}},"FlowNote":{"type":"object","description":"A sticky note attached to a flow for documentation and annotation","properties":{"id":{"type":"string","description":"Unique identifier for the note"},"text":{"type":"string","description":"Content of the note"},"position":{"type":"object","description":"Position of the note in the flow editor","properties":{"x":{"type":"number","description":"X coordinate"},"y":{"type":"number","description":"Y coordinate"}},"required":["x","y"]},"size":{"type":"object","description":"Size of the note in the flow editor","properties":{"width":{"type":"number","description":"Width in pixels"},"height":{"type":"number","description":"Height in pixels"}},"required":["width","height"]},"color":{"type":"string","description":"Color of the note (e.g., \"yellow\", \"#ffff00\")"},"type":{"type":"string","enum":["free","group"],"description":"Type of note - 'free' for standalone notes, 'group' for notes that group other nodes"},"locked":{"type":"boolean","default":false,"description":"Whether the note is locked and cannot be edited or moved"},"contained_node_ids":{"type":"array","items":{"type":"string"},"description":"For group notes, the IDs of nodes contained within this group"}},"required":["id","text","color","type"]},"FlowGroup":{"type":"object","description":"A semantic group of flow modules for organizational purposes. Does not affect execution — modules remain in their original position in the flow. Groups provide naming and collapsibility in the editor. Members are computed dynamically from all nodes on paths between start_id and end_id.","properties":{"summary":{"type":"string","description":"Display name for this group"},"note":{"type":"string","description":"Markdown note shown below the group header"},"autocollapse":{"type":"boolean","default":false,"description":"If true, this group is collapsed by default in the flow editor. UI hint only."},"start_id":{"type":"string","description":"ID of the first flow module in this group (topological entry point)"},"end_id":{"type":"string","description":"ID of the last flow module in this group (topological exit point)"},"color":{"type":"string","description":"Color for the group in the flow editor"}},"required":["start_id","end_id"]},"RetryIf":{"type":"object","description":"Conditional retry based on error or result","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables"}},"required":["expr"]},"StopAfterIf":{"type":"object","description":"Early termination condition for a module","properties":{"skip_if_stopped":{"type":"boolean","description":"If true, following steps are skipped when this condition triggers"},"expr":{"type":"string","description":"JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"},"error_message":{"type":"string","nullable":true,"description":"Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised."}},"required":["expr"]},"FlowModule":{"type":"object","description":"A single step in a flow. Can be a script, subflow, loop, or branch","properties":{"id":{"type":"string","description":"Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"},"value":{"$ref":"#/components/schemas/FlowModuleValue"},"stop_after_if":{"description":"Early termination condition evaluated after this step completes","$ref":"#/components/schemas/StopAfterIf"},"stop_after_all_iters_if":{"description":"For loops only - early termination condition evaluated after all iterations complete","$ref":"#/components/schemas/StopAfterIf"},"skip_if":{"type":"object","description":"Conditionally skip this step based on previous results or flow inputs","properties":{"expr":{"type":"string","description":"JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'"}},"required":["expr"]},"sleep":{"description":"Delay before executing this step (in seconds or as expression)","$ref":"#/components/schemas/InputTransform"},"cache_ttl":{"type":"number","description":"Cache duration in seconds for this step's results"},"cache_ignore_s3_path":{"type":"boolean"},"timeout":{"description":"Maximum execution time in seconds (static value or expression)","$ref":"#/components/schemas/InputTransform"},"delete_after_use":{"type":"boolean","description":"If true, this step's result is deleted after use to save memory"},"summary":{"type":"string","description":"Short description of what this step does"},"mock":{"type":"object","description":"Mock configuration for testing without executing the actual step","properties":{"enabled":{"type":"boolean","description":"If true, return mock value instead of executing"},"return_value":{"description":"Value to return when mocked"}}},"suspend":{"type":"object","description":"Configuration for approval/resume steps that wait for user input","properties":{"required_events":{"type":"integer","description":"Number of approvals required before continuing"},"timeout":{"type":"integer","description":"Timeout in seconds before auto-continuing or canceling"},"resume_form":{"type":"object","description":"Form schema for collecting input when resuming","properties":{"schema":{"type":"object","description":"JSON Schema for the resume form"}}},"user_auth_required":{"type":"boolean","description":"If true, only authenticated users can approve"},"user_groups_required":{"description":"Expression or list of groups that can approve","$ref":"#/components/schemas/InputTransform"},"self_approval_disabled":{"type":"boolean","description":"If true, the user who started the flow cannot approve"},"hide_cancel":{"type":"boolean","description":"If true, hide the cancel button on the approval form"},"continue_on_disapprove_timeout":{"type":"boolean","description":"If true, continue flow on timeout instead of canceling"}}},"priority":{"type":"number","description":"Execution priority for this step (higher numbers run first)"},"continue_on_error":{"type":"boolean","description":"If true, flow continues even if this step fails"},"retry":{"description":"Retry configuration if this step fails","$ref":"#/components/schemas/Retry"},"debouncing":{"description":"Debounce configuration for this step (EE only)","type":"object","properties":{"debounce_delay_s":{"type":"integer","description":"Delay in seconds to debounce this step's executions across flow runs"},"debounce_key":{"type":"string","description":"Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-"},"debounce_args_to_accumulate":{"type":"array","description":"Array-type arguments to accumulate across debounced executions","items":{"type":"string"}},"max_total_debouncing_time":{"type":"integer","description":"Maximum total time in seconds before forced execution"},"max_total_debounces_amount":{"type":"integer","description":"Maximum number of debounces before forced execution"}}}},"required":["value","id"]},"InputTransform":{"description":"Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs","oneOf":[{"$ref":"#/components/schemas/StaticTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"StaticTransform":{"type":"object","description":"Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'","properties":{"value":{"description":"The static value. For resources, use format '$res:path/to/resource'"},"type":{"type":"string","enum":["static"]}},"required":["type"]},"JavascriptTransform":{"type":"object","description":"JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value","properties":{"expr":{"type":"string","description":"JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"},"type":{"type":"string","enum":["javascript"]}},"required":["expr","type"]},"AiTransform":{"type":"object","description":"Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.","properties":{"type":{"type":"string","enum":["ai"]}},"required":["type"]},"AIProviderKind":{"type":"string","description":"Supported AI provider types","enum":["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]},"ProviderConfig":{"type":"object","description":"Complete AI provider configuration with resource reference and model selection","properties":{"kind":{"$ref":"#/components/schemas/AIProviderKind"},"resource":{"type":"string","description":"Resource reference in format '$res:{resource_path}' pointing to provider credentials"},"model":{"type":"string","description":"Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')"}},"required":["kind","resource","model"]},"StaticProviderTransform":{"type":"object","description":"Static provider configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/ProviderConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"ProviderTransform":{"description":"Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticProviderTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"MemoryOff":{"type":"object","description":"No conversation memory/context","properties":{"kind":{"type":"string","enum":["off"]}},"required":["kind"]},"MemoryAuto":{"type":"object","description":"Automatic context management","properties":{"kind":{"type":"string","enum":["auto"]},"context_length":{"type":"integer","description":"Maximum number of messages to retain in context"},"memory_id":{"type":"string","description":"Identifier for persistent memory across agent invocations"}},"required":["kind"]},"MemoryMessage":{"type":"object","description":"A single message in conversation history","properties":{"role":{"type":"string","enum":["user","assistant","system"]},"content":{"type":"string"}},"required":["role","content"]},"MemoryManual":{"type":"object","description":"Explicit message history","properties":{"kind":{"type":"string","enum":["manual"]},"messages":{"type":"array","items":{"$ref":"#/components/schemas/MemoryMessage"}}},"required":["kind","messages"]},"MemoryConfig":{"description":"Conversation memory configuration","oneOf":[{"$ref":"#/components/schemas/MemoryOff"},{"$ref":"#/components/schemas/MemoryAuto"},{"$ref":"#/components/schemas/MemoryManual"}],"discriminator":{"propertyName":"kind"}},"StaticMemoryTransform":{"type":"object","description":"Static memory configuration passed directly to the AI agent","properties":{"value":{"$ref":"#/components/schemas/MemoryConfig"},"type":{"type":"string","enum":["static"]}},"required":["type","value"]},"MemoryTransform":{"description":"Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined","oneOf":[{"$ref":"#/components/schemas/StaticMemoryTransform"},{"$ref":"#/components/schemas/JavascriptTransform"},{"$ref":"#/components/schemas/AiTransform"}],"discriminator":{"propertyName":"type"}},"FlowModuleValue":{"description":"The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type","oneOf":[{"$ref":"#/components/schemas/RawScript"},{"$ref":"#/components/schemas/PathScript"},{"$ref":"#/components/schemas/PathFlow"},{"$ref":"#/components/schemas/ForloopFlow"},{"$ref":"#/components/schemas/WhileloopFlow"},{"$ref":"#/components/schemas/BranchOne"},{"$ref":"#/components/schemas/BranchAll"},{"$ref":"#/components/schemas/Identity"},{"$ref":"#/components/schemas/AiAgent"}],"discriminator":{"propertyName":"type"}},"RawScript":{"type":"object","description":"Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"content":{"type":"string","description":"The script source code. Should export a 'main' function"},"language":{"type":"string","description":"Programming language for this script","enum":["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","duckdb"]},"path":{"type":"string","description":"Optional path for saving this script"},"lock":{"type":"string","description":"Lock file content for dependencies"},"type":{"type":"string","enum":["rawscript"]},"tag":{"type":"string","description":"Worker group tag for execution routing"},"concurrent_limit":{"type":"number","description":"Maximum concurrent executions of this script"},"concurrency_time_window_s":{"type":"number","description":"Time window for concurrent_limit"},"custom_concurrency_key":{"type":"string","description":"Custom key for grouping concurrent executions"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"},"assets":{"type":"array","description":"External resources this script accesses (S3 objects, resources, etc.)","items":{"type":"object","required":["path","kind"],"properties":{"path":{"type":"string","description":"Path to the asset"},"kind":{"type":"string","description":"Type of asset","enum":["s3object","resource","ducklake","datatable","volume"]},"access_type":{"type":"string","nullable":true,"description":"Access level for this asset","enum":["r","w","rw",null]},"alt_access_type":{"type":"string","nullable":true,"description":"Alternative access level","enum":["r","w","rw",null]}}}}},"required":["type","content","language","input_transforms"]},"PathScript":{"type":"object","description":"Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the script in the workspace (e.g., 'f/scripts/send_email')"},"hash":{"type":"string","description":"Optional specific version hash of the script to use"},"type":{"type":"string","enum":["script"]},"tag_override":{"type":"string","description":"Override the script's default worker group tag"},"is_trigger":{"type":"boolean","description":"If true, this script is a trigger that can start the flow"}},"required":["type","path","input_transforms"]},"PathFlow":{"type":"object","description":"Reference to an existing flow by path. Use this to call another flow as a subflow","properties":{"input_transforms":{"type":"object","description":"Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments","additionalProperties":{"$ref":"#/components/schemas/InputTransform"}},"path":{"type":"string","description":"Path to the flow in the workspace (e.g., 'f/flows/process_user')"},"type":{"type":"string","enum":["flow"]}},"required":["type","path","input_transforms"]},"ForloopFlow":{"type":"object","description":"Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations","properties":{"modules":{"type":"array","description":"Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'","items":{"$ref":"#/components/schemas/FlowModule"}},"iterator":{"description":"JavaScript expression that returns an array to iterate over. Can reference 'results.step_id' or 'flow_input'","$ref":"#/components/schemas/InputTransform"},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["forloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true. Limits resource usage. Can be static number or expression","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","iterator","skip_failures","type"]},"WhileloopFlow":{"type":"object","description":"Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination","properties":{"modules":{"type":"array","description":"Steps to execute in each iteration. Use stop_after_if to control when the loop ends","items":{"$ref":"#/components/schemas/FlowModule"}},"skip_failures":{"type":"boolean","description":"If true, iteration failures don't stop the loop. Failed iterations return null"},"type":{"type":"string","enum":["whileloopflow"]},"parallel":{"type":"boolean","description":"If true, iterations run concurrently (use with caution in while loops)"},"parallelism":{"description":"Maximum number of concurrent iterations when parallel=true","$ref":"#/components/schemas/InputTransform"},"squash":{"type":"boolean"}},"required":["modules","skip_failures","type"]},"BranchOne":{"type":"object","description":"Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes","properties":{"branches":{"type":"array","description":"Array of branches to evaluate in order. The first branch with expr evaluating to true executes","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch condition"},"expr":{"type":"string","description":"JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"},"modules":{"type":"array","description":"Steps to execute if this branch's expr is true","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules","expr"]}},"default":{"type":"array","description":"Steps to execute if no branch expressions match","items":{"$ref":"#/components/schemas/FlowModule"}},"type":{"type":"string","enum":["branchone"]}},"required":["branches","default","type"]},"BranchAll":{"type":"object","description":"Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently","properties":{"branches":{"type":"array","description":"Array of branches that all execute (either in parallel or sequentially)","items":{"type":"object","properties":{"summary":{"type":"string","description":"Short description of this branch's purpose"},"skip_failure":{"type":"boolean","description":"If true, failure in this branch doesn't fail the entire flow"},"modules":{"type":"array","description":"Steps to execute in this branch","items":{"$ref":"#/components/schemas/FlowModule"}}},"required":["modules"]}},"type":{"type":"string","enum":["branchall"]},"parallel":{"type":"boolean","description":"If true, all branches execute concurrently. If false, they execute sequentially"}},"required":["branches","type"]},"AgentTool":{"type":"object","description":"A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool","properties":{"id":{"type":"string","description":"Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"},"summary":{"type":"string","description":"Short description of what this tool does (shown to the AI)"},"value":{"$ref":"#/components/schemas/ToolValue"}},"required":["id","value"]},"ToolValue":{"description":"The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference","oneOf":[{"$ref":"#/components/schemas/FlowModuleTool"},{"$ref":"#/components/schemas/McpToolValue"},{"$ref":"#/components/schemas/WebsearchToolValue"}]},"FlowModuleTool":{"description":"A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module","allOf":[{"type":"object","properties":{"tool_type":{"type":"string","enum":["flowmodule"]}},"required":["tool_type"]},{"$ref":"#/components/schemas/FlowModuleValue"}]},"WebsearchToolValue":{"type":"object","description":"A tool implemented as a websearch tool. The AI can call this like any other websearch tool","properties":{"tool_type":{"type":"string","enum":["websearch"]}},"required":["tool_type"]},"McpToolValue":{"type":"object","description":"Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers","properties":{"tool_type":{"type":"string","enum":["mcp"]},"resource_path":{"type":"string","description":"Path to the MCP resource/server configuration"},"include_tools":{"type":"array","description":"Whitelist of specific tools to include from this MCP server","items":{"type":"string"}},"exclude_tools":{"type":"array","description":"Blacklist of tools to exclude from this MCP server","items":{"type":"string"}}},"required":["tool_type","resource_path"]},"AiAgent":{"type":"object","description":"AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task","properties":{"input_transforms":{"type":"object","description":"Input parameters for the AI agent mapped to their values","properties":{"provider":{"$ref":"#/components/schemas/ProviderTransform"},"output_type":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"},"user_message":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."},"system_prompt":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"System instructions that guide the AI's behavior, persona, and response style. Optional."},"streaming":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n"},"memory":{"$ref":"#/components/schemas/MemoryTransform"},"output_schema":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n"},"user_attachments":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n"},"max_completion_tokens":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n"},"temperature":{"allOf":[{"$ref":"#/components/schemas/InputTransform"}],"description":"Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n"}},"required":["provider","user_message","output_type"]},"tools":{"type":"array","description":"Array of tools the agent can use. The agent decides which tools to call based on the task","items":{"$ref":"#/components/schemas/AgentTool"}},"type":{"type":"string","enum":["aiagent"]},"parallel":{"type":"boolean","description":"If true, the agent can execute multiple tool calls in parallel"}},"required":["tools","type","input_transforms"]},"Identity":{"type":"object","description":"Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder","properties":{"type":{"type":"string","enum":["identity"]},"flow":{"type":"boolean","description":"If true, marks this as a flow identity (special handling)"}},"required":["type"]},"FlowStatus":{"type":"object","properties":{"step":{"type":"integer"},"modules":{"type":"array","items":{"$ref":"#/components/schemas/FlowStatusModule"}},"user_states":{"additionalProperties":true},"preprocessor_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"}]},"failure_module":{"allOf":[{"$ref":"#/components/schemas/FlowStatusModule"},{"type":"object","properties":{"parent_module":{"type":"string"}}}]},"retry":{"type":"object","properties":{"fail_count":{"type":"integer"},"failed_jobs":{"type":"array","items":{"type":"string","format":"uuid"}}}}},"required":["step","modules","failure_module"]},"FlowStatusModule":{"type":"object","properties":{"type":{"type":"string","enum":["WaitingForPriorSteps","WaitingForEvents","WaitingForExecutor","InProgress","Success","Failure"]},"id":{"type":"string"},"job":{"type":"string","format":"uuid"},"count":{"type":"integer"},"progress":{"type":"integer"},"iterator":{"type":"object","properties":{"index":{"type":"integer"},"itered":{"type":"array","items":{}},"itered_len":{"type":"integer"},"args":{}}},"flow_jobs":{"type":"array","items":{"type":"string"}},"flow_jobs_success":{"type":"array","items":{"type":"boolean"}},"flow_jobs_duration":{"type":"object","properties":{"started_at":{"type":"array","items":{"type":"string"}},"duration_ms":{"type":"array","items":{"type":"integer"}}}},"branch_chosen":{"type":"object","properties":{"type":{"type":"string","enum":["branch","default"]},"branch":{"type":"integer"}},"required":["type"]},"branchall":{"type":"object","properties":{"branch":{"type":"integer"},"len":{"type":"integer"}},"required":["branch","len"]},"approvers":{"type":"array","items":{"type":"object","properties":{"resume_id":{"type":"integer"},"approver":{"type":"string"}},"required":["resume_id","approver"]}},"failed_retries":{"type":"array","items":{"type":"string","format":"uuid"}},"skipped":{"type":"boolean"},"agent_actions":{"type":"array","items":{"type":"object","oneOf":[{"type":"object","properties":{"job_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"type":{"type":"string","enum":["tool_call"]},"module_id":{"type":"string"}},"required":["job_id","function_name","type","module_id"]},{"type":"object","properties":{"call_id":{"type":"string","format":"uuid"},"function_name":{"type":"string"},"resource_path":{"type":"string"},"type":{"type":"string","enum":["mcp_tool_call"]},"arguments":{"type":"object"}},"required":["call_id","function_name","resource_path","type"]},{"type":"object","properties":{"type":{"type":"string","enum":["web_search"]}},"required":["type"]},{"type":"object","properties":{"type":{"type":"string","enum":["message"]}},"required":["content","type"]}]}},"agent_actions_success":{"type":"array","items":{"type":"boolean"}}},"required":["type"]}}}} \ No newline at end of file diff --git a/frontend/src/lib/components/copilot/chat/flow/openFlowZod.ts b/frontend/src/lib/components/copilot/chat/flow/openFlowZod.ts index 9782c77d41..b7a7cc13f6 100644 --- a/frontend/src/lib/components/copilot/chat/flow/openFlowZod.ts +++ b/frontend/src/lib/components/copilot/chat/flow/openFlowZod.ts @@ -1,6 +1,6 @@ import { z } from "zod" -export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable"]).describe("Type of asset"), "access_type": z.enum(["r","w","rw"]).describe("Access level for this asset").optional(), "alt_access_type": z.enum(["r","w","rw"]).describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_images": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of image references for vision-capable models.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'images/photo.jpg' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { +export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.lazy(() => flowModuleValueSchema), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch")).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { const schemas = [z.intersection(z.object({ "tool_type": z.literal("flowmodule") }), z.lazy(() => flowModuleValueSchema)).describe("A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module"), z.object({ "tool_type": z.literal("mcp"), "resource_path": z.string().describe("Path to the MCP resource/server configuration"), "include_tools": z.array(z.string()).describe("Whitelist of specific tools to include from this MCP server").optional(), "exclude_tools": z.array(z.string()).describe("Blacklist of tools to exclude from this MCP server").optional() }).describe("Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers"), z.object({ "tool_type": z.literal("websearch") }).describe("A tool implemented as a websearch tool. The AI can call this like any other websearch tool")]; const errors = schemas.reduce( (errors, schema) => @@ -20,7 +20,7 @@ export const flowModuleValueSchema = z.discriminatedUnion("type", [z.object({ "i }).describe("The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference") }).describe("A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool")).describe("Array of tools the agent can use. The agent decides which tools to call based on the task"), "type": z.literal("aiagent"), "parallel": z.boolean().describe("If true, the agent can execute multiple tool calls in parallel").optional() }).describe("AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task")]).describe("The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type") -export const flowModuleSchema = z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable"]).describe("Type of asset"), "access_type": z.enum(["r","w","rw"]).describe("Access level for this asset").optional(), "alt_access_type": z.enum(["r","w","rw"]).describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_images": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of image references for vision-capable models.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'images/photo.jpg' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { +export const flowModuleSchema = z.object({ "id": z.string().describe("Unique identifier for this step. Used to reference results via 'results.step_id'. Must be a valid identifier (alphanumeric, underscore, hyphen)"), "value": z.discriminatedUnion("type", [z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "content": z.string().describe("The script source code. Should export a 'main' function"), "language": z.enum(["deno","bun","python3","go","bash","powershell","postgresql","mysql","bigquery","snowflake","mssql","oracledb","graphql","nativets","php","rust","ansible","csharp","nu","java","ruby","duckdb"]).describe("Programming language for this script"), "path": z.string().describe("Optional path for saving this script").optional(), "lock": z.string().describe("Lock file content for dependencies").optional(), "type": z.literal("rawscript"), "tag": z.string().describe("Worker group tag for execution routing").optional(), "concurrent_limit": z.number().describe("Maximum concurrent executions of this script").optional(), "concurrency_time_window_s": z.number().describe("Time window for concurrent_limit").optional(), "custom_concurrency_key": z.string().describe("Custom key for grouping concurrent executions").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional(), "assets": z.array(z.object({ "path": z.string().describe("Path to the asset"), "kind": z.enum(["s3object","resource","ducklake","datatable","volume"]).describe("Type of asset"), "access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Access level for this asset").optional(), "alt_access_type": z.union([z.literal("r"), z.literal("w"), z.literal("rw"), z.literal(null)]).nullable().describe("Alternative access level").optional() })).describe("External resources this script accesses (S3 objects, resources, etc.)").optional() }).describe("Inline script with code defined directly in the flow. Use 'bun' as default language if unspecified. The script receives arguments from input_transforms"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the script's input arguments"), "path": z.string().describe("Path to the script in the workspace (e.g., 'f/scripts/send_email')"), "hash": z.string().describe("Optional specific version hash of the script to use").optional(), "type": z.literal("script"), "tag_override": z.string().describe("Override the script's default worker group tag").optional(), "is_trigger": z.boolean().describe("If true, this script is a trigger that can start the flow").optional() }).describe("Reference to an existing script by path. Use this when calling a previously saved script instead of writing inline code"), z.object({ "input_transforms": z.record(z.string(), z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs")).describe("Map of parameter names to their values (static or JavaScript expressions). These become the subflow's input arguments"), "path": z.string().describe("Path to the flow in the workspace (e.g., 'f/flows/process_user')"), "type": z.literal("flow") }).describe("Reference to an existing flow by path. Use this to call another flow as a subflow"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute for each iteration. These can reference the iteration value via 'flow_input.iter.value'"), "iterator": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("forloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (faster for I/O-bound operations). Use with parallelism to control concurrency").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules in a loop over an iterator. Inside the loop, use 'flow_input.iter.value' to access the current iteration value, and 'flow_input.iter.index' for the index. Supports parallel execution for better performance on I/O-bound operations"), z.object({ "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in each iteration. Use stop_after_if to control when the loop ends"), "skip_failures": z.boolean().describe("If true, iteration failures don't stop the loop. Failed iterations return null"), "type": z.literal("whileloopflow"), "parallel": z.boolean().describe("If true, iterations run concurrently (use with caution in while loops)").optional(), "parallelism": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "squash": z.boolean().optional() }).describe("Executes nested modules repeatedly while a condition is true. The loop checks the condition after each iteration. Use stop_after_if on modules to control loop termination"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch condition").optional(), "expr": z.string().describe("JavaScript expression that returns boolean. Can use 'results.step_id' or 'flow_input'. First true expr wins"), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if this branch's expr is true") })).describe("Array of branches to evaluate in order. The first branch with expr evaluating to true executes"), "default": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute if no branch expressions match"), "type": z.literal("branchone") }).describe("Conditional branching where only the first matching branch executes. Branches are evaluated in order, and the first one with a true expression runs. If no branches match, the default branch executes"), z.object({ "branches": z.array(z.object({ "summary": z.string().describe("Short description of this branch's purpose").optional(), "skip_failure": z.boolean().describe("If true, failure in this branch doesn't fail the entire flow").optional(), "modules": z.array(z.lazy(() => flowModuleSchema)).describe("Steps to execute in this branch") })).describe("Array of branches that all execute (either in parallel or sequentially)"), "type": z.literal("branchall"), "parallel": z.boolean().describe("If true, all branches execute concurrently. If false, they execute sequentially").optional() }).describe("Parallel branching where all branches execute simultaneously. Unlike BranchOne, all branches run regardless of conditions. Useful for executing independent tasks concurrently"), z.object({ "type": z.literal("identity"), "flow": z.boolean().describe("If true, marks this as a flow identity (special handling)").optional() }).describe("Pass-through module that returns its input unchanged. Useful for flow structure or as a placeholder"), z.object({ "input_transforms": z.object({ "provider": z.discriminatedUnion("type", [z.object({ "value": z.object({ "kind": z.enum(["openai","azure_openai","anthropic","mistral","deepseek","googleai","groq","openrouter","togetherai","customai","aws_bedrock"]).describe("Supported AI provider types"), "resource": z.string().describe("Resource reference in format '$res:{resource_path}' pointing to provider credentials"), "model": z.string().describe("Model identifier (e.g., 'gpt-4', 'claude-3-opus-20240229', 'gemini-pro')") }).describe("Complete AI provider configuration with resource reference and model selection"), "type": z.literal("static") }).describe("Static provider configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Provider configuration - can be static (ProviderConfig), JavaScript expression, or AI-determined"), "output_type": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Output format type.\nValid values: 'text' (default) - plain text response, 'image' - image generation\n"), "user_message": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("The user's prompt/message to the AI agent. Supports variable interpolation with flow.input syntax."), "system_prompt": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("System instructions that guide the AI's behavior, persona, and response style. Optional.").optional(), "streaming": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Boolean. If true, stream the AI response incrementally.\nStreaming events include: token_delta, tool_call, tool_call_arguments, tool_execution, tool_result\n").optional(), "memory": z.discriminatedUnion("type", [z.object({ "value": z.discriminatedUnion("kind", [z.object({ "kind": z.literal("off") }).describe("No conversation memory/context"), z.object({ "kind": z.literal("auto"), "context_length": z.number().int().describe("Maximum number of messages to retain in context").optional(), "memory_id": z.string().describe("Identifier for persistent memory across agent invocations").optional() }).describe("Automatic context management"), z.object({ "kind": z.literal("manual"), "messages": z.array(z.object({ "role": z.enum(["user","assistant","system"]), "content": z.string() }).describe("A single message in conversation history")) }).describe("Explicit message history")]).describe("Conversation memory configuration"), "type": z.literal("static") }).describe("Static memory configuration passed directly to the AI agent"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Memory configuration - can be static (MemoryConfig), JavaScript expression, or AI-determined").optional(), "output_schema": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("JSON Schema object defining structured output format. Used when you need the AI to return data in a specific shape.\nSupports standard JSON Schema properties: type, properties, required, items, enum, pattern, minLength, maxLength, minimum, maximum, etc.\nExample: { type: 'object', properties: { name: { type: 'string' }, age: { type: 'integer' } }, required: ['name'] }\n").optional(), "user_attachments": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Array of file references (images or PDFs) for the AI agent.\nFormat: Array<{ bucket: string, key: string }> - S3 object references\nExample: [{ bucket: 'my-bucket', key: 'documents/report.pdf' }]\n").optional(), "max_completion_tokens": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Integer. Maximum number of tokens the AI will generate in its response.\nRange: 1 to 4,294,967,295. Typical values: 256-4096 for most use cases.\n").optional(), "temperature": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").describe("Float. Controls randomness/creativity of responses.\nRange: 0.0 to 2.0 (provider-dependent)\n- 0.0 = deterministic, focused responses\n- 0.7 = balanced (common default)\n- 1.0+ = more creative/random\n").optional() }).describe("Input parameters for the AI agent mapped to their values"), "tools": z.array(z.object({ "id": z.string().describe("Unique identifier for this tool. Cannot contain spaces - use underscores instead (e.g., 'get_user_data' not 'get user data')"), "summary": z.string().describe("Short description of what this tool does (shown to the AI)").optional(), "value": z.any().superRefine((x, ctx) => { const schemas = [z.intersection(z.object({ "tool_type": z.literal("flowmodule") }), z.lazy(() => flowModuleValueSchema)).describe("A tool implemented as a flow module (script, flow, etc.). The AI can call this like any other flow module"), z.object({ "tool_type": z.literal("mcp"), "resource_path": z.string().describe("Path to the MCP resource/server configuration"), "include_tools": z.array(z.string()).describe("Whitelist of specific tools to include from this MCP server").optional(), "exclude_tools": z.array(z.string()).describe("Blacklist of tools to exclude from this MCP server").optional() }).describe("Reference to an external MCP (Model Context Protocol) tool. The AI can call tools from MCP servers"), z.object({ "tool_type": z.literal("websearch") }).describe("A tool implemented as a websearch tool. The AI can call this like any other websearch tool")]; const errors = schemas.reduce( (errors, schema) => @@ -37,7 +37,7 @@ export const flowModuleSchema = z.object({ "id": z.string().describe("Unique ide message: "Invalid input: Should pass single schema", }); } - }).describe("The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference") }).describe("A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool")).describe("Array of tools the agent can use. The agent decides which tools to call based on the task"), "type": z.literal("aiagent"), "parallel": z.boolean().describe("If true, the agent can execute multiple tool calls in parallel").optional() }).describe("AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task")]).describe("The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type"), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().describe("Custom error message shown when stopping").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch") + }).describe("The implementation of a tool. Can be a flow module (script/flow) or an MCP tool reference") }).describe("A tool available to an AI agent. Can be a flow module or an external MCP (Model Context Protocol) tool")).describe("Array of tools the agent can use. The agent decides which tools to call based on the task"), "type": z.literal("aiagent"), "parallel": z.boolean().describe("If true, the agent can execute multiple tool calls in parallel").optional() }).describe("AI agent step that can use tools to accomplish tasks. The agent receives inputs and can call any of its configured tools to complete the task")]).describe("The actual implementation of a flow step. Can be a script (inline or referenced), subflow, loop, branch, or special module type"), "stop_after_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "stop_after_all_iters_if": z.object({ "skip_if_stopped": z.boolean().describe("If true, following steps are skipped when this condition triggers").optional(), "expr": z.string().describe("JavaScript expression evaluated after the module runs. Can use 'result' (step's result) or 'flow_input'. Return true to stop"), "error_message": z.string().nullable().describe("Custom error message when stopping with an error. Mutually exclusive with skip_if_stopped. If set to a non-empty string, the flow stops with this error. If empty string, a default error message is used. If null or omitted, no error is raised.").optional() }).describe("Early termination condition for a module").optional(), "skip_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to skip. Can use 'flow_input' or 'results.'") }).describe("Conditionally skip this step based on previous results or flow inputs").optional(), "sleep": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "cache_ttl": z.number().describe("Cache duration in seconds for this step's results").optional(), "cache_ignore_s3_path": z.boolean().optional(), "timeout": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "delete_after_use": z.boolean().describe("If true, this step's result is deleted after use to save memory").optional(), "summary": z.string().describe("Short description of what this step does").optional(), "mock": z.object({ "enabled": z.boolean().describe("If true, return mock value instead of executing").optional(), "return_value": z.any().describe("Value to return when mocked").optional() }).describe("Mock configuration for testing without executing the actual step").optional(), "suspend": z.object({ "required_events": z.number().int().describe("Number of approvals required before continuing").optional(), "timeout": z.number().int().describe("Timeout in seconds before auto-continuing or canceling").optional(), "resume_form": z.object({ "schema": z.record(z.string(), z.any()).describe("JSON Schema for the resume form").optional() }).describe("Form schema for collecting input when resuming").optional(), "user_auth_required": z.boolean().describe("If true, only authenticated users can approve").optional(), "user_groups_required": z.discriminatedUnion("type", [z.object({ "value": z.any().describe("The static value. For resources, use format '$res:path/to/resource'").optional(), "type": z.literal("static") }).describe("Static value passed directly to the step. Use for hardcoded values or resource references like '$res:path/to/resource'"), z.object({ "expr": z.string().describe("JavaScript expression returning the value. Available variables - results (object with all previous step results), flow_input (flow inputs), flow_input.iter (in loops)"), "type": z.literal("javascript") }).describe("JavaScript expression evaluated at runtime. Can reference previous step results via 'results.step_id' or flow inputs via 'flow_input.property'. Inside loops, use 'flow_input.iter.value' for the current iteration value"), z.object({ "type": z.literal("ai") }).describe("Value resolved by the AI runtime for this input. The AI engine decides how to satisfy the parameter.")]).describe("Maps input parameters for a step. Can be a static value or a JavaScript expression that references previous results or flow inputs").optional(), "self_approval_disabled": z.boolean().describe("If true, the user who started the flow cannot approve").optional(), "hide_cancel": z.boolean().describe("If true, hide the cancel button on the approval form").optional(), "continue_on_disapprove_timeout": z.boolean().describe("If true, continue flow on timeout instead of canceling").optional() }).describe("Configuration for approval/resume steps that wait for user input").optional(), "priority": z.number().describe("Execution priority for this step (higher numbers run first)").optional(), "continue_on_error": z.boolean().describe("If true, flow continues even if this step fails").optional(), "retry": z.object({ "constant": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "seconds": z.number().int().describe("Seconds to wait between retries").optional() }).describe("Retry with constant delay between attempts").optional(), "exponential": z.object({ "attempts": z.number().int().describe("Number of retry attempts").optional(), "multiplier": z.number().int().describe("Multiplier for exponential backoff").optional(), "seconds": z.number().int().gte(1).describe("Initial delay in seconds").optional(), "random_factor": z.number().int().gte(0).lte(100).describe("Random jitter percentage (0-100) to avoid thundering herd").optional() }).describe("Retry with exponential backoff (delay doubles each time)").optional(), "retry_if": z.object({ "expr": z.string().describe("JavaScript expression that returns true to retry. Has access to 'result' and 'error' variables") }).describe("Conditional retry based on error or result").optional() }).describe("Retry configuration for failed module executions").optional(), "debouncing": z.object({ "debounce_delay_s": z.number().int().describe("Delay in seconds to debounce this step's executions across flow runs").optional(), "debounce_key": z.string().describe("Expression to group debounced executions. Supports $workspace and $args[name]. Default: $workspace/flow/-").optional(), "debounce_args_to_accumulate": z.array(z.string()).describe("Array-type arguments to accumulate across debounced executions").optional(), "max_total_debouncing_time": z.number().int().describe("Maximum total time in seconds before forced execution").optional(), "max_total_debounces_amount": z.number().int().describe("Maximum number of debounces before forced execution").optional() }).describe("Debounce configuration for this step (EE only)").optional() }).describe("A single step in a flow. Can be a script, subflow, loop, or branch") export const flowModulesSchema = z.array(flowModuleSchema) diff --git a/frontend/src/lib/components/copilot/chat/openai-responses.ts b/frontend/src/lib/components/copilot/chat/openai-responses.ts index d7e5ad5c96..56364e1401 100644 --- a/frontend/src/lib/components/copilot/chat/openai-responses.ts +++ b/frontend/src/lib/components/copilot/chat/openai-responses.ts @@ -5,10 +5,14 @@ import type { ChatCompletionCreateParams } from 'openai/resources/index.mjs' import type { ResponseErrorEvent } from 'openai/resources/responses/responses.mjs' -import { getProviderAndCompletionConfig, workspaceAIClients } from '../lib' +import { + createOpenAIProxyClient, + getAiProxyBaseURL, + getProviderAndCompletionConfig, + workspaceAIClients +} from '../lib' import { processToolCall, type Tool, type ToolCallbacks } from './shared' import type { ResponseStream } from 'openai/lib/responses/ResponseStream.mjs' -import { OpenAPI } from '$lib/gen' import type { AIProviderModel } from '$lib/gen' // Conversion utilities for Responses API @@ -121,15 +125,24 @@ function convertCompletionConfigToResponsesConfig( export async function getOpenAIResponsesCompletion( messages: ChatCompletionMessageParam[], abortController: AbortController, - tools?: OpenAI.Chat.Completions.ChatCompletionTool[] + tools?: OpenAI.Chat.Completions.ChatCompletionTool[], + options?: { + forceModelProvider?: AIProviderModel + openaiClient?: OpenAI + } ) { - const { provider, config } = getProviderAndCompletionConfig({ messages, stream: true, tools }) + const { provider, config } = getProviderAndCompletionConfig({ + messages, + stream: true, + tools, + forceModelProvider: options?.forceModelProvider + }) const { instructions, input } = convertMessagesToResponsesInput(messages) const responsesConfig = convertCompletionConfigToResponsesConfig(config) - const openaiClient = workspaceAIClients.getOpenaiClient() + const client = options?.openaiClient ?? workspaceAIClients.getOpenaiClient() - const runner = openaiClient.responses.stream( + const runner = client.responses.stream( { ...responsesConfig, input, @@ -204,7 +217,8 @@ export async function parseOpenAIResponsesCompletion( messages: ChatCompletionMessageParam[], addedMessages: ChatCompletionMessageParam[], tools: Tool[], - helpers: any + helpers: any, + options?: { workspace?: string } ): Promise { let toolCallsToProcess: ChatCompletionMessageFunctionToolCall[] = [] let error: OpenAIError | ResponseErrorEvent | null = null @@ -338,7 +352,8 @@ export async function parseOpenAIResponsesCompletion( tools, toolCall, helpers, - toolCallbacks: callbacks + toolCallbacks: callbacks, + workspace: options?.workspace }) messages.push(messageToAdd) addedMessages.push(messageToAdd) @@ -354,6 +369,7 @@ export async function getNonStreamingOpenAIResponsesCompletion( abortController: AbortController, testOptions?: { apiKey?: string + workspace?: string resourcePath?: string forceModelProvider: AIProviderModel } @@ -390,15 +406,10 @@ export async function getNonStreamingOpenAIResponsesCompletion( } const openaiClient = testOptions?.apiKey - ? new OpenAI({ - baseURL: `${location.origin}${OpenAPI.BASE}/ai/proxy`, - apiKey: 'fake-key', - defaultHeaders: { - Authorization: '' // a non empty string will be unable to access Windmill backend proxy - }, - dangerouslyAllowBrowser: true - }) - : workspaceAIClients.getOpenaiClient() + ? createOpenAIProxyClient(getAiProxyBaseURL()) + : testOptions?.workspace + ? workspaceAIClients.createOpenaiClient(testOptions.workspace) + : workspaceAIClients.getOpenaiClient() const response = await openaiClient.responses.create( { diff --git a/frontend/src/lib/components/copilot/chat/shared.ts b/frontend/src/lib/components/copilot/chat/shared.ts index 4a95912b47..20e488d923 100644 --- a/frontend/src/lib/components/copilot/chat/shared.ts +++ b/frontend/src/lib/components/copilot/chat/shared.ts @@ -417,12 +417,14 @@ export async function processToolCall({ tools, toolCall, helpers, - toolCallbacks + toolCallbacks, + workspace }: { tools: Tool[] toolCall: ChatCompletionMessageFunctionToolCall helpers: T toolCallbacks: ToolCallbacks + workspace?: string }): Promise { try { const args = JSON.parse(toolCall.function.arguments || '{}') @@ -472,7 +474,7 @@ export async function processToolCall({ tools, functionName: toolCall.function.name, args, - workspace: get(workspaceStore) ?? '', + workspace: workspace ?? get(workspaceStore) ?? '', helpers, toolCallbacks, toolId: toolCall.id diff --git a/frontend/src/lib/components/copilot/lib.ts b/frontend/src/lib/components/copilot/lib.ts index f9afd9bb2c..d8149086d4 100644 --- a/frontend/src/lib/components/copilot/lib.ts +++ b/frontend/src/lib/components/copilot/lib.ts @@ -67,7 +67,14 @@ export const AI_PROVIDERS: Record = { }, googleai: { label: 'Google AI', - defaultModels: ['gemini-2.5-flash', 'gemini-2.5-pro', 'gemini-2.5-flash-lite', 'gemini-3-flash', 'gemini-3.1-pro', 'gemini-3.1-flash-lite'] + defaultModels: [ + 'gemini-2.5-flash', + 'gemini-2.5-pro', + 'gemini-2.5-flash-lite', + 'gemini-3-flash', + 'gemini-3.1-pro', + 'gemini-3.1-flash-lite' + ] }, groq: { label: 'Groq', @@ -289,7 +296,12 @@ function getModelSpecificConfig( ) { const defaultMaxTokens = getModelMaxTokens(modelProvider.provider, modelProvider.model) const modelKey = `${modelProvider.provider}:${modelProvider.model}` - const customMaxTokensStore = get(copilotInfo)?.maxTokensPerModel + let customMaxTokensStore: Record | undefined + try { + customMaxTokensStore = get(copilotInfo)?.maxTokensPerModel + } catch { + // copilotInfo store may not be initialized in vitest + } const maxTokens = customMaxTokensStore?.[modelKey] ?? defaultMaxTokens if ( (modelProvider.provider === 'openai' || modelProvider.provider === 'azure_openai') && @@ -364,38 +376,46 @@ export const PROVIDER_COMPLETION_CONFIG_MAP: Record> { - const { provider, config } = getProviderAndCompletionConfig({ messages, stream: true, tools }) + const { provider, config } = getProviderAndCompletionConfig({ + messages, + stream: true, + tools, + forceModelProvider: options?.forceModelProvider + }) // Use Responses API for OpenAI and Azure OpenAI if ((provider === 'openai' || provider === 'azure_openai') && !options?.forceCompletions) { @@ -876,8 +903,8 @@ export async function getCompletion( } // Use Completions API for other providers - const openaiClient = workspaceAIClients.getOpenaiClient() - const completion = openaiClient.chat.completions.create(config, { + const client = options?.openaiClient ?? workspaceAIClients.getOpenaiClient() + const completion = client.chat.completions.create(config, { signal: abortController.signal, headers: { 'X-Provider': provider @@ -906,7 +933,8 @@ export async function parseOpenAICompletion( addedMessages: ChatCompletionMessageParam[], tools: Tool[], helpers: any, - _abortController?: AbortController // unused, for signature compatibility with parseAnthropicCompletion + _abortController?: AbortController, // unused, for signature compatibility with parseAnthropicCompletion + options?: { workspace?: string } ): Promise { const finalToolCalls: Record = {} let malformedFunctionCallError = false @@ -1045,7 +1073,8 @@ export async function parseOpenAICompletion( tools, toolCall, helpers, - toolCallbacks: callbacks + toolCallbacks: callbacks, + workspace: options?.workspace }) messages.push(messageToAdd) addedMessages.push(messageToAdd) diff --git a/frontend/src/lib/components/custom_ui.ts b/frontend/src/lib/components/custom_ui.ts index 536471f74d..5f017ee950 100644 --- a/frontend/src/lib/components/custom_ui.ts +++ b/frontend/src/lib/components/custom_ui.ts @@ -42,6 +42,8 @@ export type FlowBuilderWhitelabelCustomUi = { tagLabel?: string aiAgent?: boolean aiSandbox?: boolean + suggestIntegration?: boolean + suggestScript?: boolean } export type DisplayResultUi = { diff --git a/frontend/src/lib/components/debug/debugUtils.ts b/frontend/src/lib/components/debug/debugUtils.ts index ec9c12cfb4..d58fb23983 100644 --- a/frontend/src/lib/components/debug/debugUtils.ts +++ b/frontend/src/lib/components/debug/debugUtils.ts @@ -9,9 +9,7 @@ import { VariableService, UserService } from '$lib/gen' * Fetch contextual variables (WM_WORKSPACE, WM_TOKEN, etc.) for the debugger. * Creates a fresh short-lived token for the debug session. */ -export async function fetchContextualVariables( - workspace: string -): Promise> { +export async function fetchContextualVariables(workspace: string): Promise> { if (!workspace) { return {} } @@ -77,6 +75,30 @@ export async function signDebugRequest( return await response.json() } +/** + * Sign a multiplayer session request. Returns a JWT token that the + * multiplayer server will verify before accepting the WebSocket connection. + */ +export async function signMultiplayerRequest(workspace: string): Promise { + if (!workspace) { + throw new Error('No workspace selected') + } + + const response = await fetch(`/api/w/${workspace}/debug/sign_multiplayer`, { + method: 'POST', + headers: { 'Content-Type': 'application/json' }, + body: JSON.stringify({}) + }) + + if (!response.ok) { + const errorText = await response.text() + throw new Error(errorText || 'Failed to sign multiplayer session') + } + + const data: { token: string } = await response.json() + return data.token +} + /** * Get a user-friendly error message for debug errors */ diff --git a/frontend/src/lib/components/debug/index.ts b/frontend/src/lib/components/debug/index.ts index 50eb614858..c7397cea74 100644 --- a/frontend/src/lib/components/debug/index.ts +++ b/frontend/src/lib/components/debug/index.ts @@ -61,6 +61,7 @@ export { export { fetchContextualVariables, signDebugRequest, + signMultiplayerRequest, getDebugErrorMessage, isDebuggableLanguage, getDebugFileExtension diff --git a/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte b/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte index 3d9a533f48..471790d539 100644 --- a/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte +++ b/frontend/src/lib/components/flows/content/FlowEditorPanel.svelte @@ -112,6 +112,7 @@ onDeleteSelected={() => flowModuleSchemaMap?.deleteMultiple(resolvedModuleIds)} onDuplicateSelected={() => flowModuleSchemaMap?.duplicateMultiple(resolvedModuleIds)} onMoveSelected={() => flowModuleSchemaMap?.moveMultiple(resolvedModuleIds)} + onCreateGroup={() => flowModuleSchemaMap?.createGroup(selectionManager.selectedIds)} {canMoveSelected} resolvedCount={resolvedModuleIds.length} /> diff --git a/frontend/src/lib/components/flows/content/FlowInput.svelte b/frontend/src/lib/components/flows/content/FlowInput.svelte index a9eab8074c..5c7002f147 100644 --- a/frontend/src/lib/components/flows/content/FlowInput.svelte +++ b/frontend/src/lib/components/flows/content/FlowInput.svelte @@ -686,6 +686,7 @@ bind:schema={flowStore.val.schema} hiddenArgs={['user_message']} isFlowInput + showSensitiveToggle editTab={chatInputsEditTab ? 'inputEditor' : undefined} showDynOpt bind:dynCode @@ -741,6 +742,7 @@ bind:this={editableSchemaForm} bind:schema={flowStore.val.schema} isFlowInput + showSensitiveToggle on:delete={(e) => { addPropertyV2?.handleDeleteArgument([e.detail]) }} diff --git a/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte b/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte index 0dd5635816..db199c745a 100644 --- a/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte +++ b/frontend/src/lib/components/flows/content/FlowInputsQuick.svelte @@ -308,7 +308,7 @@ bind:selectedFilter={selected} resourceType /> - {#if !selected} + {#if !selected && customUi?.suggestIntegration != false}
{#if flow} - + {:else} {/if} diff --git a/frontend/src/lib/components/flows/content/FlowSelectionPanel.svelte b/frontend/src/lib/components/flows/content/FlowSelectionPanel.svelte index 7efb44b82d..1b54d83928 100644 --- a/frontend/src/lib/components/flows/content/FlowSelectionPanel.svelte +++ b/frontend/src/lib/components/flows/content/FlowSelectionPanel.svelte @@ -3,8 +3,8 @@ import type { SelectionManager } from '$lib/components/graph/selectionUtils.svelte' import { Button } from '$lib/components/common' import DropdownV2 from '$lib/components/DropdownV2.svelte' - import { getNoteEditorContext } from '$lib/components/graph/noteEditor.svelte' - import { StickyNote, Move, Copy, Trash2 } from 'lucide-svelte' + import { getGroupEditorContext } from '$lib/components/graph/groupEditor.svelte' + import { Group, Move, Copy, Trash2 } from 'lucide-svelte' import type { Item } from '$lib/utils' interface Props { @@ -13,6 +13,7 @@ onDeleteSelected?: () => void onDuplicateSelected?: () => void onMoveSelected?: () => void + onCreateGroup?: () => void canMoveSelected?: boolean resolvedCount?: number } @@ -22,18 +23,14 @@ onDeleteSelected, onDuplicateSelected, onMoveSelected, + onCreateGroup, canMoveSelected = false, resolvedCount = 0 }: Props = $props() - const noteEditorContext = getNoteEditorContext() + const groupEditorContext = getGroupEditorContext() - function addGroupNote() { - if (selectionManager.selectedIds.length > 0 && noteEditorContext?.noteEditor) { - // Create the group note - noteEditorContext.noteEditor.createGroupNote(selectionManager.selectedIds) - } - } + let canCreateGroup = $derived(groupEditorContext?.canCreateGroup.val ?? false) let menuItems: Item[] = $derived([ { @@ -60,11 +57,11 @@ {#snippet action()}
{#if resolvedCount > 0} diff --git a/frontend/src/lib/components/flows/content/SuspendDrawer.svelte b/frontend/src/lib/components/flows/content/SuspendDrawer.svelte index ee22349cac..3ac2afe001 100644 --- a/frontend/src/lib/components/flows/content/SuspendDrawer.svelte +++ b/frontend/src/lib/components/flows/content/SuspendDrawer.svelte @@ -39,27 +39,9 @@ render a cancel button, providing the operator with an option to cancel the step. e.g: - {#snippet content()} - - - ): Record { + // Migrate user_images → user_attachments + if ('user_images' in inputTransforms && !('user_attachments' in inputTransforms)) { + inputTransforms.user_attachments = inputTransforms.user_images + delete inputTransforms.user_images + } + // Check if this has the legacy format if ('messages_context_length' in inputTransforms && !('memory' in inputTransforms)) { const legacyValue = inputTransforms.messages_context_length diff --git a/frontend/src/lib/components/flows/header/FlowYamlEditor.svelte b/frontend/src/lib/components/flows/header/FlowYamlEditor.svelte index b0504a8055..51d1f07d3f 100644 --- a/frontend/src/lib/components/flows/header/FlowYamlEditor.svelte +++ b/frontend/src/lib/components/flows/header/FlowYamlEditor.svelte @@ -31,9 +31,22 @@ editor?.setCode(code) } + function validateGroups(groups: { start_id: string; end_id: string }[] | undefined) { + if (!groups) return + const seen = new Set() + for (const g of groups) { + const key = `${g.start_id}:${g.end_id}` + if (seen.has(key)) { + throw new Error(`Duplicate group: '${g.start_id}' → '${g.end_id}'`) + } + seen.add(key) + } + } + function apply() { try { const parsed = YAML.parse(code) + validateGroups(parsed.value?.groups) if (parsed.summary && typeof parsed.summary === 'string') { flowStore.val.summary = parsed.summary } @@ -59,7 +72,7 @@ initialCode = code sendUserToast('Changes applied') } catch (e) { - ;(sendUserToast('Error parsing yaml: ' + e), true) + sendUserToast('Error parsing yaml: ' + e, true) } } @@ -69,8 +82,12 @@ drawer?.toggleDrawer()}> {#snippet actions()} - - + + {/snippet} {#if flowStore.val} diff --git a/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte b/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte index 8e0250cec8..e6a548b39a 100644 --- a/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte +++ b/frontend/src/lib/components/flows/map/FlowModuleSchemaItem.svelte @@ -192,10 +192,10 @@ !!id && !!$flowPropPickerConfig && !!pickableIds && Object.keys(pickableIds).includes(id) ) - let isDragging = $derived(!!moveManager?.dragging) + let isMoving = $derived(!!moveManager?.dragging || !!moveManager?.movingModuleId) const outputPickerVisible = $derived( - editMode && (isConnectingCandidate || alwaysShowOutputPicker) && !!id && !isDragging + editMode && (isConnectingCandidate || alwaysShowOutputPicker) && !!id && !isMoving ) const icon_render = $derived(icon) @@ -214,7 +214,7 @@ flowStore?.val?.value.failure_module )} - (editId = false)}> + (editId = false)}>
{#snippet icon()} @@ -484,11 +482,10 @@ {/if}
- {#if deletable && !isDragging} + {#if deletable && !isMoving} {#if maximizeSubflow !== undefined} {@render buttonMaximizeSubflow?.()} {/if} - {#if (id && Object.values($flowInputsStore?.[id]?.flowStepWarnings || {}).length > 0) || Boolean(warningMessage)} - {#if editMode && enableTestRun && flowJob?.type !== 'QueuedJob' && !isDragging} + {#if editMode && enableTestRun && flowJob?.type !== 'QueuedJob' && !isMoving}
(hover = false)} > {#if !isMultiSelected && (hover || selected || testRunDropdownOpen) && outputPickerVisible} -
+
{#if !testIsLoading}
{/each} + + 0} + on:confirmed={() => { + affectedGroupsAction?.() + affectedGroupsPending = [] + affectedGroupsAction = undefined + affectedGroupsCancel = undefined + }} + on:canceled={() => { + affectedGroupsCancel?.() + affectedGroupsPending = [] + affectedGroupsAction = undefined + affectedGroupsCancel = undefined + }} + > + {#if affectedGroupsPending.length === 1} + {@const group = affectedGroupsPending[0]} +

The group{group.summary ? ` "${group.summary}"` : ''} will be removed (empty or duplicate). + Are you sure you want to {affectedGroupsActionLabel} the step?

+ {:else} +

The following groups will be removed (empty or duplicate):

+
    + {#each affectedGroupsPending as group} +
  • {group.summary || `${group.start_id} → ${group.end_id}`}
    • + {/each} +
+

Are you sure you want to {affectedGroupsActionLabel} the step?

+ {/if} +
{ dependents = getDependentComponents(id, flowStore.val) - const cb = () => { - push(history, flowStore.val) - if (id === 'preprocessor') { + + if (id === 'preprocessor') { + const cb = () => { + push(history, flowStore.val) selectionManager.selectId('Input') flowStore.val.value.preprocessor_module = undefined - } else { - selectNextId(id) - removeAtId(flowStore.val.value.modules, id) + refreshStateStore(flowStore) + onDelete?.(id) + delete flowStateStore.val[id] } + if (Object.keys(dependents).length > 0) { + deleteCallback = cb + } else { + cb() + } + return + } + + const dsOpts = { displayState: groupDisplayState } + const { emptiedGroups, duplicateGroups, commit } = proxy.prepareMutation((tree) => { + const found = findInStructure(tree, id) + if (found) found.parentChildren.splice(found.index, 1) + }, dsOpts) + + const affectedGroups = [...emptiedGroups, ...duplicateGroups] + + const cb = () => { + push(history, flowStore.val) + selectNextId(id) + commit({ removeDuplicates: duplicateGroups.length > 0 }) refreshStateStore(flowStore) onDelete?.(id) delete flowStateStore.val[id] } - if (Object.keys(dependents).length > 0) { - deleteCallback = cb + const proceed = () => { + if (Object.keys(dependents).length > 0) { + deleteCallback = cb + } else { + cb() + } + } + + if (affectedGroups.length > 0) { + affectedGroupsPending = affectedGroups + affectedGroupsActionLabel = 'delete' + affectedGroupsAction = proceed } else { - cb() + proceed() } }} onInsert={async (detail) => { - { - let originalModules - let targetModules - if ( - detail.sourceId == 'Input' || - detail.targetId == 'Result' || - detail.kind == 'trigger' - ) { - targetModules = flowStore.val.value.modules + if (!flowStore.val.value.modules || !Array.isArray(flowStore.val.value.modules)) return + await tick() + + // --- MOVE --- + if (moveManager.movingModuleId) { + const movedIds = moveManager.movingIds ?? [moveManager.movingModuleId] + const movingId = moveManager.movingModuleId + + let mutated = false + const moveOpts = { displayState: groupDisplayState } + const { emptiedGroups, duplicateGroups, commit } = proxy.prepareMutation((tree) => { + let originalModules: FlowStructureNode[] | undefined + let targetModules: FlowStructureNode[] | undefined + + if (detail.sourceId == 'Input' || detail.targetId == 'Result') { + targetModules = tree + } + dfsStructure(tree, (node, parentArray) => { + if (matchStructureNode(node, movingId)) originalModules = parentArray + if (detail.branch && matchStructureNode(node, detail.branch.rootId)) { + targetModules = node.branches[detail.branch.branch]?.children + } else if ( + matchStructureNode(node, detail.sourceId ?? '') || + matchStructureNode(node, detail.targetId ?? '') + ) { + targetModules = parentArray + } + }) + + if (!originalModules || !targetModules) return + + if (movedIds.length > 1) { + const firstIndex = originalModules.findIndex((m) => + matchStructureNode(m, movedIds[0]) + ) + if (firstIndex < 0) return + const removedModules = originalModules.splice(firstIndex, movedIds.length) + let insertIndex = detail.index + if (originalModules === targetModules && firstIndex < detail.index) { + insertIndex -= movedIds.length + } + targetModules.splice(insertIndex, 0, ...removedModules) + } else { + const indexToRemove = originalModules.findIndex((m) => + matchStructureNode(m, movingId) + ) + if (indexToRemove < 0) return + const [removed] = originalModules.splice(indexToRemove, 1) + let insertIndex = detail.index + if (originalModules === targetModules && indexToRemove < detail.index) + insertIndex -= 1 + targetModules.splice(insertIndex, 0, removed) + } + mutated = true + }, moveOpts) + + if (!mutated) { + moveManager.clearMoving() + return } - dfs(flowStore.val.value.modules, (mod, modules, branches) => { - if (mod.id == moveManager.movingModuleId) { - originalModules = modules - } - if (detail.branch) { - if (mod.id == detail.branch.rootId) { - targetModules = branches[detail.branch.branch] - } - } else if (mod.id == detail.sourceId || mod.id == detail.targetId) { - targetModules = modules - } else if (mod.id == detail.agentId && mod.value.type === 'aiagent') { - targetModules = mod.value.tools - } - }) - if (flowStore.val.value.modules && Array.isArray(flowStore.val.value.modules)) { - await tick() - if (moveManager.movingModuleId) { - push(history, flowStore.val) - if (!originalModules || !targetModules) { - moveManager.clearMoving() - return - } - if (moveManager.movingIds && moveManager.movingIds.length > 1) { - // Multi-move: splice out all moving modules from their parent, insert at target - const firstIndex = originalModules.findIndex( - (m) => m.id === moveManager.movingIds?.[0] - ) - const removedModules = originalModules.splice( - firstIndex, - moveManager.movingIds.length - ) - let insertIndex = detail.index - if (originalModules === targetModules && firstIndex < detail.index) { - insertIndex -= moveManager.movingIds.length - } - targetModules.splice(insertIndex, 0, ...removedModules) - selectionManager.selectByIds(removedModules.map((m) => m.id)) - } else { - let indexToRemove = originalModules.findIndex((m) => moveManager.movingModuleId == m.id) - let [removedModule] = originalModules.splice(indexToRemove, 1) - // When moving within the same array, removal shifts subsequent indices down by 1 - let insertIndex = detail.index - if (originalModules === targetModules && indexToRemove < detail.index) { - insertIndex -= 1 - } - targetModules.splice(insertIndex, 0, removedModule) - selectionManager.selectId(removedModule.id) - } - moveManager.clearMoving() + const affectedGroups = [...emptiedGroups, ...duplicateGroups] + + const doMove = () => { + push(history, flowStore.val) + commit({ removeDuplicates: duplicateGroups.length > 0 }) + if (movedIds.length > 1) { + selectionManager.selectByIds(movedIds) } else { - if (detail.isPreprocessor) { - await insertNewPreprocessorModule( - flowStore, - flowStateStore, - detail.inlineScript, - detail.script - ) - selectionManager.selectId('preprocessor') - - if (detail.inlineScript?.instructions) { - dispatch('generateStep', { - moduleId: 'preprocessor', - lang: detail.inlineScript?.language, - instructions: detail.inlineScript?.instructions - }) - } - } else { - const index = (detail.agentId ? targetModules?.length : detail.index) ?? 0 - const toolKind: SpecialToolKind | 'flowmoduleTool' | undefined = detail.agentId - ? (SPECIAL_TOOL_KINDS as readonly string[]).includes(detail.kind) - ? (detail.kind as SpecialToolKind) - : 'flowmoduleTool' - : undefined - - await insertNewModuleAtIndex( - targetModules, - index, - detail.kind, - detail.script, - detail.flow, - detail.inlineScript, - toolKind - ) - const id = targetModules[index].id - selectionManager.selectId(id) - - if (detail.inlineScript?.instructions) { - dispatch('generateStep', { - moduleId: id, - lang: detail.inlineScript?.language, - instructions: detail.inlineScript?.instructions - }) - } - if (detail.kind == 'trigger') { - await insertNewModuleAtIndex( - targetModules, - index + 1, - 'forloop', - undefined, - undefined, - undefined - ) - setExpr(targetModules[index + 1], `results.${id}`) - setScheduledPollSchedule(triggersState, triggersCount) - } - - if (detail.flow?.path) { - loadLastJob(detail.flow.path, id) - } else if (detail.script?.path) { - loadLastJob(detail.script?.path, id) - } - } - } - - if (['branchone', 'branchall'].includes(detail.kind)) { - await addBranch(targetModules[detail.index ?? 0].id) + selectionManager.selectId(movingId) } + moveManager.clearMoving() refreshStateStore(flowStore) dispatch('change') } + + if (affectedGroups.length > 0) { + affectedGroupsPending = affectedGroups + affectedGroupsActionLabel = 'move' + affectedGroupsAction = doMove + affectedGroupsCancel = () => moveManager.clearMoving() + } else { + doMove() + } + return } + + // --- INSERT --- + if (detail.isPreprocessor) { + await insertNewPreprocessorModule( + flowStore, + flowStateStore, + detail.inlineScript, + detail.script + ) + selectionManager.selectId('preprocessor') + if (detail.inlineScript?.instructions) { + dispatch('generateStep', { + moduleId: 'preprocessor', + lang: detail.inlineScript?.language, + instructions: detail.inlineScript?.instructions + }) + } + refreshStateStore(flowStore) + dispatch('change') + return + } + + push(history, flowStore.val) + + const isAgentInsert = !!detail.agentId + const toolKind: SpecialToolKind | 'flowmoduleTool' | undefined = isAgentInsert + ? (SPECIAL_TOOL_KINDS as readonly string[]).includes(detail.kind) + ? (detail.kind as SpecialToolKind) + : 'flowmoduleTool' + : undefined + + // Agent tool inserts operate on the FlowModule's tools array directly + if (isAgentInsert) { + const agentMod = getAllModules(flowStore.val.value.modules).find( + (m) => m.id === detail.agentId + ) + if (agentMod && (agentMod.value as any).tools) { + const tools = (agentMod.value as any).tools as AgentTool[] + await insertNewModuleAtIndex( + tools, + tools.length, + detail.kind as InsertKind, + detail.script, + detail.flow ? { path: detail.flow.path, summary: detail.flow.summary } : undefined, + detail.inlineScript, + toolKind + ) + const id = tools[tools.length - 1].id + selectionManager.selectId(id) + } + refreshStateStore(flowStore) + dispatch('change') + return + } + + // Regular module insert: create the module, then insert a leaf node via tree mutation + const module = await createNewModule( + detail.kind as InsertKind, + detail.script, + detail.flow ? { path: detail.flow.path, summary: detail.flow.summary } : undefined, + detail.inlineScript + ) + const index = detail.index ?? 0 + const extraModules: FlowModule[] = [module] + + // For trigger inserts, also create the forloop module + let loopModule: FlowModule | undefined + if (detail.kind == 'trigger') { + loopModule = await createNewModule('forloop') + setExpr(loopModule, `results.${module.id}`) + extraModules.push(loopModule) + } + + proxy.applyTreeMutation( + (tree) => { + // Find target array in the snapshot + let targetArray: FlowStructureNode[] | undefined + if ( + detail.sourceId == 'Input' || + detail.targetId == 'Result' || + detail.kind == 'trigger' + ) { + targetArray = tree + } + dfsStructure(tree, (node, parentArray) => { + if (detail.branch && matchStructureNode(node, detail.branch.rootId)) { + targetArray = node.branches[detail.branch.branch]?.children + } else if ( + matchStructureNode(node, detail.sourceId ?? '') || + matchStructureNode(node, detail.targetId ?? '') + ) { + targetArray = parentArray + } + }) + if (!targetArray) targetArray = tree + + // Insert the structure node (correct kind for containers like branchone/branchall) + targetArray.splice(index, 0, moduleToStructureNode(module)) + + // For trigger: also insert the forloop node after it + if (loopModule) { + targetArray.splice(index + 1, 0, moduleToStructureNode(loopModule)) + } + }, + { extraModules, displayState: groupDisplayState } + ) + + selectionManager.selectId(module.id) + + if (detail.inlineScript?.instructions) { + dispatch('generateStep', { + moduleId: module.id, + lang: detail.inlineScript?.language, + instructions: detail.inlineScript?.instructions + }) + } + if (detail.kind == 'trigger') { + setScheduledPollSchedule(triggersState, triggersCount) + } + if (detail.flow?.path) { + loadLastJob(detail.flow.path, module.id) + } else if (detail.script?.path) { + loadLastJob(detail.script?.path, module.id) + } + + if (['branchone', 'branchall'].includes(detail.kind)) { + await addBranch(module.id) + } + refreshStateStore(flowStore) + dispatch('change') }} onNewBranch={async (id) => { if (id) { @@ -761,6 +973,17 @@ mod.id = newId } }) + const groups = flowStore.val.value.groups + if (groups) { + for (const group of groups) { + if (group.start_id === id) { + group.start_id = newId + } + if (group.end_id === id) { + group.end_id = newId + } + } + } flowStateStore.val[newId] = flowStateStore.val[id] delete flowStateStore.val[id] refreshStateStore(flowStore) diff --git a/frontend/src/lib/components/flows/map/VirtualItemWrapper.svelte b/frontend/src/lib/components/flows/map/VirtualItemWrapper.svelte index 798116b47d..49cbb56cc6 100644 --- a/frontend/src/lib/components/flows/map/VirtualItemWrapper.svelte +++ b/frontend/src/lib/components/flows/map/VirtualItemWrapper.svelte @@ -46,7 +46,7 @@
@@ -909,7 +1064,7 @@ bind:this={flowContainer} > {#if graph?.error} -
+
{graph.error} @@ -1008,6 +1163,12 @@ /> {/if} + + @@ -1065,7 +1226,7 @@ try { localStorage.setItem( 'svelvet', - encodeState({ modules, failureModule, preprocessorModule, notes }) + encodeState({ modules, failureModule, preprocessorModule, notes, groups }) ) } catch (e) { console.error('error interacting with local storage', e) diff --git a/frontend/src/lib/components/graph/GroupActionBar.svelte b/frontend/src/lib/components/graph/GroupActionBar.svelte new file mode 100644 index 0000000000..1ae30395b5 --- /dev/null +++ b/frontend/src/lib/components/graph/GroupActionBar.svelte @@ -0,0 +1,158 @@ + + +
+ {#if moveManager && moveModuleId} + moveManager.toggleMoving(moveModuleId!)} + /> + {/if} + {#if note == null} + + {/if} + + {#snippet buttonReplacement()} + + {/snippet} + {#snippet menu()} +
+ +
+
+ {#each Object.values(NoteColor) as c (c)} + + {/each} +
+
+ + +
+ onUpdateAutocollapse(e.detail)} + /> +
+ +
+ + + + + {#if onDeleteGroup} +
+ + + + {/if} +
+ {/snippet} + +
diff --git a/frontend/src/lib/components/graph/GroupHeader.svelte b/frontend/src/lib/components/graph/GroupHeader.svelte new file mode 100644 index 0000000000..474d71c1a4 --- /dev/null +++ b/frontend/src/lib/components/graph/GroupHeader.svelte @@ -0,0 +1,116 @@ + + + + +
{}))} + title={collapsed ? 'Expand group' : 'Collapse group'} +> +
+ +
+
+ {#if editingSummary} +
+ +
+ {:else} + {})) : undefined} + >{summary || PLACEHOLDER} + {/if} +
+
+ + diff --git a/frontend/src/lib/components/graph/GroupHeaderBlock.svelte b/frontend/src/lib/components/graph/GroupHeaderBlock.svelte new file mode 100644 index 0000000000..089c0642a4 --- /dev/null +++ b/frontend/src/lib/components/graph/GroupHeaderBlock.svelte @@ -0,0 +1,76 @@ + + + +
(hovered = true)} + onmouseleave={() => (hovered = false)} +> + graphContext?.groupDisplayState?.toggleRuntimeCollapse(groupId)} + onSummaryUpdate={(text) => groupEditorContext?.groupEditor.updateSummary(groupId, text)} + /> + {#if showNotes && note != null} + graphContext?.groupDisplayState?.setNoteHeight(groupId, h)} + onNoteUpdate={(text) => groupEditorContext?.groupEditor.updateNote(groupId, text)} + /> + {/if} + {#if editMode} + (menuOpen = open)} + onAddNote={() => groupEditorContext?.groupEditor.addNote(groupId)} + onRemoveNote={() => groupEditorContext?.groupEditor.removeNote(groupId)} + onUpdateColor={(c) => groupEditorContext?.groupEditor.updateColor(groupId, c)} + onUpdateAutocollapse={(v) => groupEditorContext?.groupEditor.updateAutocollapse(groupId, v)} + onDeleteGroup={() => groupEditorContext?.groupEditor.deleteGroup(groupId)} + /> + {/if} +
diff --git a/frontend/src/lib/components/graph/GroupModuleIcons.svelte b/frontend/src/lib/components/graph/GroupModuleIcons.svelte new file mode 100644 index 0000000000..3df1aecfaa --- /dev/null +++ b/frontend/src/lib/components/graph/GroupModuleIcons.svelte @@ -0,0 +1,182 @@ + + +
+ {#each displayModules as mod (mod.id)} + {@const selected = selectionManager.isNodeSelected(mod.id)} + {@const nodeState = flowModuleStates?.[mod.id]?.type} + {@const colorClasses = getNodeColorClasses(nodeState, selected)} + + {#snippet children()} + + +
selectModule(mod)} + > +
+ +
+ {mod.id} +
+ {/snippet} + {#snippet text()} + {mod.id}: {moduleLabel(mod)} + {/snippet} + + {/each} + {#if overflowModules.length > 0} + {@const overflowColorClasses = getNodeColorClasses(overflowAggregateState, false)} + + {#snippet buttonReplacement()} +
+ +{overflowModules.length} +
+ {/snippet} + {#snippet menu()} +
+ {#each overflowModules as mod (mod.id)} + {@const nodeState = flowModuleStates?.[mod.id]?.type} + {@const colorClasses = getNodeColorClasses(nodeState, false)} + {@const selected = selectionManager.isNodeSelected(mod.id)} + + +
selectModule(mod)} + > +
+ +
+ {moduleLabel(mod)} + {mod.id} +
+ {/each} +
+ {/snippet} + + {/if} +
diff --git a/frontend/src/lib/components/graph/GroupNodeCard.svelte b/frontend/src/lib/components/graph/GroupNodeCard.svelte new file mode 100644 index 0000000000..dd69fcf0d0 --- /dev/null +++ b/frontend/src/lib/components/graph/GroupNodeCard.svelte @@ -0,0 +1,165 @@ + + +
+
+
+ {#if modules && modules.length > 0} + + {:else} + + {/if} +
+ {#if editingSummary} + + {:else} + + + {})) : undefined} + >{summary || 'Group'} + {/if} +
+
+ {#if stepCount != null} + + + {stepCount} node{stepCount !== 1 ? 's' : ''} + {/if} +
+ + {#if showNote} +
+ onHeightChange?.(h)} + onNoteUpdate={(text) => onNoteUpdate?.(text)} + /> +
+ {/if} +
diff --git a/frontend/src/lib/components/graph/GroupNoteArea.svelte b/frontend/src/lib/components/graph/GroupNoteArea.svelte new file mode 100644 index 0000000000..1b4562e8d0 --- /dev/null +++ b/frontend/src/lib/components/graph/GroupNoteArea.svelte @@ -0,0 +1,151 @@ + + + +
+
+ {#if editing} +
+ + +
+ + {:else if note} + +
{}) : undefined} + > + +
+ {:else} + +
{}) : undefined} + > + Double click to edit the note +
+ {/if} +
+
diff --git a/frontend/src/lib/components/graph/GroupOverlay.svelte b/frontend/src/lib/components/graph/GroupOverlay.svelte new file mode 100644 index 0000000000..c15095bfb4 --- /dev/null +++ b/frontend/src/lib/components/graph/GroupOverlay.svelte @@ -0,0 +1,90 @@ + + +{#each groups as group (groupKey(group))} + {@const bounds = groupBoundsMap[groupKey(group)]} + {#if bounds} + +
+ + {/if} +{/each} diff --git a/frontend/src/lib/components/graph/MiniFlowGraph.svelte b/frontend/src/lib/components/graph/MiniFlowGraph.svelte index fdb73745df..ab0c5042e8 100644 --- a/frontend/src/lib/components/graph/MiniFlowGraph.svelte +++ b/frontend/src/lib/components/graph/MiniFlowGraph.svelte @@ -1,6 +1,12 @@ - -{#if noteEditorContext?.noteEditor && selectedNodeIds.length > 1} - - {@render children()} - -{/if} diff --git a/frontend/src/lib/components/graph/NoteColorPicker.svelte b/frontend/src/lib/components/graph/NoteColorPicker.svelte index d17adb9317..ff3a1a276c 100644 --- a/frontend/src/lib/components/graph/NoteColorPicker.svelte +++ b/frontend/src/lib/components/graph/NoteColorPicker.svelte @@ -10,7 +10,11 @@ isOpen?: boolean } - let { selectedColor, onColorChange, isOpen = $bindable(false) }: Props = $props() + let { + selectedColor, + onColorChange, + isOpen = $bindable(false) + }: Props = $props() import { ViewportPortal, type Node } from '@xyflow/svelte' import { calculateNodesBoundsWithOffset } from './util' - import { StickyNote, Move, Copy, Trash2, EllipsisVertical } from 'lucide-svelte' + import { Move, Copy, Trash2, EllipsisVertical, Group } from 'lucide-svelte' import { Button } from '../common' import DropdownV2 from '../DropdownV2.svelte' - import { getNoteEditorContext } from './noteEditor.svelte' + import { getGroupEditorContext } from './groupEditor.svelte' import { getGraphContext } from './graphContext' import MoveHandleButton from './MoveHandleButton.svelte' import { tick } from 'svelte' @@ -36,18 +36,19 @@ let resolvedCount = $derived(resolvedModuleIds.length) - // Get NoteEditor context for group note creation - const noteEditorContext = getNoteEditorContext() + // Get GroupEditor context for group creation + const groupEditorContext = getGroupEditorContext() // Get Graph context for clearFlowSelection function and moveManager const graphContext = getGraphContext() const moveManager = graphContext?.moveManager - function handleAddGroupNote() { - if (selectedNodes.length > 0 && noteEditorContext?.noteEditor && graphContext) { - // Create the group note first - noteEditorContext.noteEditor.createGroupNote(selectedNodes) + let canCreateGroup = $derived(groupEditorContext?.canCreateGroup.val ?? false) + + function handleAddGroup() { + if (selectedNodes.length > 0 && groupEditorContext?.groupEditor && graphContext) { + const flowNodes = graphContext.getFlowNodes?.() ?? [] + groupEditorContext.groupEditor.createGroup(selectedNodes, flowNodes) - // Wait for next tick to ensure DOM updates tick().then(() => { graphContext?.clearFlowSelection?.() graphContext?.selectionManager.clearSelection() @@ -74,13 +75,13 @@ shortcut: isMac() ? '⌫' : 'Del', action: () => onDeleteSelected?.() }, - ...(noteEditorContext?.noteEditor + ...(groupEditorContext?.groupEditor ? [ { - displayName: 'Add note', - icon: StickyNote, - separatorTop: true, - action: handleAddGroupNote + displayName: 'Create group', + icon: Group, + action: handleAddGroup, + disabled: !canCreateGroup } ] : []) diff --git a/frontend/src/lib/components/graph/WacDiagram.svelte b/frontend/src/lib/components/graph/WacDiagram.svelte new file mode 100644 index 0000000000..ed7f2d22cc --- /dev/null +++ b/frontend/src/lib/components/graph/WacDiagram.svelte @@ -0,0 +1,109 @@ + + +
+ {#if errors.length > 0} +
+ {#each errors as error (error.line)} +
+ + + {#if error.line > 0} + L{error.line}: + {/if} + {error.message} + +
+ {/each} +
+ {:else if empty} +
+ + No workflow diagram +
+ {:else} + + + + + + {/if} +
diff --git a/frontend/src/lib/components/graph/compoundLayout.ts b/frontend/src/lib/components/graph/compoundLayout.ts index 4bf3ddebda..c47aa62584 100644 --- a/frontend/src/lib/components/graph/compoundLayout.ts +++ b/frontend/src/lib/components/graph/compoundLayout.ts @@ -1,5 +1,6 @@ import { sugiyama, dagStratify, coordCenter, decrossTwoLayer, decrossOpt } from 'd3-dag' import { NODE } from './util' +import { GROUP_HEADER_HEIGHT } from './groupEditor.svelte' type LayoutNode = { id: string @@ -14,7 +15,7 @@ type LayoutConstants = { } type CompoundGroup = { - type: 'branch' | 'loop' + type: 'branch' | 'loop' | 'group' headId: string endId: string branches: { @@ -27,9 +28,12 @@ type LayoutResult = { positions: Map bbox: { width: number; height: number } contentMinX: number + groupDimensions?: Map } const LOOP_INDENT = 25 +export const GROUP_PADDING = 16 +export const GROUP_TOP_PADDING = 32 /** * Detect compound groups from a flat list of node IDs. @@ -83,6 +87,18 @@ function detectGroups( endId: id, branches: [{ labelId: `${baseId}-start`, innerIds }] }) + } else if (baseId.startsWith('group:')) { + // Group pattern: group:{groupId} head + group:{groupId}-end + // Body is everything reachable from head to end + const innerIds = findInnerIds(baseId, id, nodeIds, childrenMap) + if (innerIds.length > 0) { + groups.push({ + type: 'group', + headId: baseId, + endId: id, + branches: [{ labelId: innerIds[0], innerIds: innerIds.slice(1) }] + }) + } } } @@ -230,6 +246,29 @@ function runSugiyama( * 5. Run sugiyama on the simplified graph * 6. Expand wrapper positions back to absolute positions */ +/** + * Build nodeSizes map for sugiyama from nodeExtraSpace. + * Each node's effective height = top + NODE.height + bottom. + */ +function buildNodeSizes( + nodeIds: string[], + constants: LayoutConstants, + nodeExtraSpace?: Map +): Map | undefined { + if (!nodeExtraSpace || nodeExtraSpace.size === 0) return undefined + const sizes = new Map() + for (const id of nodeIds) { + const extra = nodeExtraSpace.get(id) + if (extra && (extra.top > 0 || extra.bottom > 0 || extra.left > 0 || extra.right > 0)) { + sizes.set(id, { + width: constants.nodeWidth + extra.left + extra.right, + height: constants.nodeHeight + extra.top + extra.bottom + }) + } + } + return sizes.size > 0 ? sizes : undefined +} + const MAX_RECURSION_DEPTH = 50 function layoutLevel( @@ -237,7 +276,8 @@ function layoutLevel( allNodes: Map, constants: LayoutConstants, childrenMap: Map, - depth: number = 0 + depth: number = 0, + nodeExtraSpace?: Map ): LayoutResult { const positions = new Map() const nodeIdSet = new Set(nodeIds) @@ -256,8 +296,15 @@ function layoutLevel( const n = allNodes.get(id)! return { id, parentIds: (n.parentIds ?? []).filter((pid) => nodeIdSet.has(pid)) } }) - const result = runSugiyama(flatNodes, constants) + const extraSizes = buildNodeSizes( + flatNodes.map((n) => n.id), + constants, + nodeExtraSpace + ) + const result = runSugiyama(flatNodes, constants, extraSizes) for (const [id, pos] of result.positions) { + const extra = nodeExtraSpace?.get(id) + if (extra) pos.y += extra.top positions.set(id, pos) } return { positions, bbox: { width: result.width, height: result.height }, contentMinX: 0 } @@ -322,7 +369,14 @@ function layoutLevel( const branchNodeIds = [branch.labelId, ...branch.innerIds] // Find sub-groups within this branch - const result = layoutLevel(branchNodeIds, allNodes, constants, childrenMap, depth + 1) + const result = layoutLevel( + branchNodeIds, + allNodes, + constants, + childrenMap, + depth + 1, + nodeExtraSpace + ) branchLayouts.push({ labelId: branch.labelId, @@ -349,6 +403,16 @@ function layoutLevel( maxBranchHeight = Math.max(0, ...branchLayouts.map((bl) => bl.bbox.height)) // head row + branch content + end row wrapperHeight = rowHeight + maxBranchHeight + rowHeight + } else if (group.type === 'group') { + // Group: body is centered with padding on all sides + const bodyWidth = branchLayouts[0]?.bbox.width ?? constants.nodeWidth + const bodyHeight = branchLayouts[0]?.bbox.height ?? 0 + wrapperWidth = Math.max(bodyWidth + GROUP_PADDING * 2, constants.nodeWidth) + maxBranchHeight = bodyHeight + const headExtra = nodeExtraSpace?.get(group.headId) + const groupHeadRow = GROUP_HEADER_HEIGHT + (headExtra?.bottom ?? 0) + GROUP_TOP_PADDING + // head row + body + bottom padding + wrapperHeight = groupHeadRow + bodyHeight + GROUP_PADDING } else { // Loop: body is indented const bodyWidth = branchLayouts[0]?.bbox.width ?? constants.nodeWidth @@ -395,13 +459,30 @@ function layoutLevel( } // Step 5: Run sugiyama on flattened nodes - const sugResult = runSugiyama(flatNodes, constants, wrapperSizes) + // Merge wrapperSizes with nodeExtraSpace-derived sizes for non-group nodes + const extraSizes = buildNodeSizes( + flatNodes.map((n) => n.id), + constants, + nodeExtraSpace + ) + const mergedSizes = new Map() + if (extraSizes) { + for (const [id, size] of extraSizes) mergedSizes.set(id, size) + } + for (const [id, size] of wrapperSizes) mergedSizes.set(id, size) + const sugResult = runSugiyama( + flatNodes, + constants, + mergedSizes.size > 0 ? mergedSizes : undefined + ) // Step 6: Resolve absolute positions // First, set positions for regular (non-group) nodes + // Apply per-node y-offset from nodeExtraSpace so decorations above have room for (const [nid, pos] of sugResult.positions) { if (groupByHeadId.has(nid)) continue // Handle groups separately - positions.set(nid, { x: pos.x, y: pos.y }) + const extra = nodeExtraSpace?.get(nid) + positions.set(nid, { x: pos.x, y: pos.y + (extra?.top ?? 0) }) } // Now expand group wrappers into absolute positions @@ -411,9 +492,15 @@ function layoutLevel( const rowHeight = constants.nodeHeight + constants.gapV const isBranch = gl.group.type === 'branch' + const isGroup = gl.group.type === 'group' // Position the head node at the top-center of the wrapper - positions.set(headId, { x: wrapperPos.x, y: wrapperPos.y }) + // Apply extra top padding so decorations above the head node have room + const headExtra = nodeExtraSpace?.get(headId) + positions.set(headId, { + x: wrapperPos.x, + y: wrapperPos.y + (headExtra?.top ?? 0) + }) if (isBranch) { // Reuse cached branchWidths and totalWidth @@ -441,6 +528,26 @@ function layoutLevel( x: wrapperPos.x, y: wrapperPos.y + rowHeight + maxBranchHeight + constants.gapV }) + } else if (isGroup) { + // Group: body is centered within wrapper (no x offset) + const headExtra = nodeExtraSpace?.get(gl.group.headId) + const groupHeadRow = GROUP_HEADER_HEIGHT + (headExtra?.bottom ?? 0) + GROUP_TOP_PADDING + const bl = gl.branchLayouts[0] + if (bl) { + for (const [innerNodeId, innerPos] of bl.result.positions) { + positions.set(innerNodeId, { + x: wrapperPos.x + innerPos.x, + y: wrapperPos.y + groupHeadRow + innerPos.y + }) + } + } + + // Position end node below body + const bodyHeight = bl?.bbox.height ?? 0 + positions.set(gl.group.endId, { + x: wrapperPos.x, + y: wrapperPos.y + groupHeadRow + bodyHeight + GROUP_PADDING + }) } else { // Loop: position start, body, and end const bl = gl.branchLayouts[0] @@ -463,16 +570,34 @@ function layoutLevel( } } + // Collect group dimensions from this level and child layouts + const groupDimensions = new Map() + for (const [headId, gl] of groupLayouts) { + groupDimensions.set(headId, { width: gl.wrapperWidth, height: gl.wrapperHeight }) + // Propagate child groupDimensions from recursive branch layouts + for (const bl of gl.branchLayouts) { + if (bl.result.groupDimensions) { + for (const [childId, dims] of bl.result.groupDimensions) { + groupDimensions.set(childId, dims) + } + } + } + } + // Compute overall bbox (nodes + group wrapper extents) let minX = Infinity let maxX = -Infinity let minY = Infinity let maxY = -Infinity - for (const pos of positions.values()) { - minX = Math.min(minX, pos.x - constants.nodeWidth / 2) - maxX = Math.max(maxX, pos.x + constants.nodeWidth / 2) - minY = Math.min(minY, pos.y) - maxY = Math.max(maxY, pos.y + constants.nodeHeight) + for (const [nid, pos] of positions) { + // Group end nodes are zero-height markers — skip them + if (nid.startsWith('group:') && nid.endsWith('-end')) continue + const extra = nodeExtraSpace?.get(nid) + minX = Math.min(minX, pos.x - constants.nodeWidth / 2 - (extra?.left ?? 0)) + maxX = Math.max(maxX, pos.x + constants.nodeWidth / 2 + (extra?.right ?? 0)) + // Account for top decoration space above the node + minY = Math.min(minY, pos.y - (extra?.top ?? 0)) + maxY = Math.max(maxY, pos.y + constants.nodeHeight + (extra?.bottom ?? 0)) } // Account for group wrapper extents in bbox (e.g. LOOP_INDENT makes wrappers wider than nodes) for (const [headId, gl] of groupLayouts) { @@ -492,7 +617,12 @@ function layoutLevel( width: Math.max(bboxWidth, constants.nodeWidth), height: Math.max(bboxHeight, 0) } - return { positions, bbox: finalBbox, contentMinX } + return { + positions, + bbox: finalBbox, + contentMinX, + groupDimensions: groupDimensions.size > 0 ? groupDimensions : undefined + } } /** @@ -500,10 +630,16 @@ function layoutLevel( * * Takes the flat list of nodes and edges from graphBuilder and produces * absolute positions that account for compound structure (branches, loops). + * + * nodeExtraSpace: per-node top/bottom/left/right padding that should be allocated in layout. + * After layout, each node's y is shifted down by its top padding so decorations + * (assets, AI tools, group headers) have room above. Left/right padding widens the + * column allocated to the node so neighbors are pushed further away. */ export function compoundLayout( nodes: { id: string; parentIds?: string[] }[], - constants?: Partial + constants?: Partial, + nodeExtraSpace?: Map ): LayoutResult { const c: LayoutConstants = { nodeWidth: constants?.nodeWidth ?? NODE.width, @@ -528,7 +664,7 @@ export function compoundLayout( } const nodeIds = nodes.map((n) => n.id) - const result = layoutLevel(nodeIds, allNodes, c, childrenMap) + const result = layoutLevel(nodeIds, allNodes, c, childrenMap, 0, nodeExtraSpace) // Shift positions so minX=0 (left-aligned). // FlowGraphV2 centers with: xCenter = viewport/2 - bbox.width/2 diff --git a/frontend/src/lib/components/graph/flowStructure.test.ts b/frontend/src/lib/components/graph/flowStructure.test.ts new file mode 100644 index 0000000000..fce5071e80 --- /dev/null +++ b/frontend/src/lib/components/graph/flowStructure.test.ts @@ -0,0 +1,245 @@ +import { describe, it, expect, vi } from 'vitest' + +// Mock modules that transitively import CSS/Monaco +vi.mock('monaco-editor', () => ({})) +vi.mock('@xyflow/svelte', () => ({})) +vi.mock('./renderers/nodes/AssetNode.svelte', () => ({ + assetDisplaysAsOutputInFlowGraph: () => false +})) +vi.mock('../modulesTest.svelte', () => ({})) + +import type { GraphGroup } from './groupEditor.svelte' +import type { FlowModule } from '$lib/gen' +import { + buildStructureTree, + flattenStructureIds, + deriveGroupsFromStructure, + collectLeafIds, + findInStructure +} from './flowStructure' + +function makeModule(id: string): FlowModule { + return { + id, + value: { type: 'rawscript', content: '', language: 'python3' } as any + } as FlowModule +} + +function makeBranchAll(id: string, branchInnerIds: string[][]): FlowModule { + return { + id, + value: { + type: 'branchall', + branches: branchInnerIds.map((ids) => ({ modules: ids.map((iid) => makeModule(iid)) })) + } as any + } as FlowModule +} + +function makeForloop(id: string, innerIds: string[]): FlowModule { + return { + id, + value: { + type: 'forloopflow', + modules: innerIds.map((iid) => makeModule(iid)), + iterator: { type: 'javascript', expr: '' } + } as any + } as FlowModule +} + +function makeGroup( + id: string, + start_id: string, + end_id: string, + moduleIds: string[] = [] +): GraphGroup { + return { id, start_id, end_id, moduleIds } +} + +describe('buildStructureTree', () => { + const modules = [makeModule('a'), makeModule('b'), makeModule('c')] + + it('builds structure tree for a valid group', () => { + const groups = [makeGroup('g1', 'a', 'b', ['a', 'b'])] + const result = buildStructureTree(modules, groups) + // Should have a group node + the remaining leaf 'c' + expect(result).toHaveLength(2) + expect(result[0].kind).toBe('group') + expect(result[0].id).toBe('g1') + expect(result[0].branches[0].children).toHaveLength(2) + expect(result[1].kind).toBe('leaf') + expect(result[1].id).toBe('c') + }) + + it('throws on duplicate group IDs', () => { + const groups = [makeGroup('g1', 'a', 'a', ['a']), makeGroup('g1', 'b', 'c', ['b', 'c'])] + expect(() => buildStructureTree(modules, groups)).toThrow(/duplicate group id.*g1/i) + }) + + it('throws on inverted range (start_id after end_id)', () => { + const groups = [makeGroup('g1', 'c', 'a', ['a', 'b', 'c'])] + expect(() => buildStructureTree(modules, groups)).toThrow(/inverted range/i) + }) + + it('throws on partially overlapping groups', () => { + const groups = [makeGroup('g1', 'a', 'b', ['a', 'b']), makeGroup('g2', 'b', 'c', ['b', 'c'])] + expect(() => buildStructureTree(modules, groups)).toThrow(/overlap without nesting/i) + }) + + it('throws when group start_id is a virtual node (Input)', () => { + const groups = [makeGroup('g1', 'Input', 'b', ['a', 'b'])] + expect(() => buildStructureTree(modules, groups)).toThrow(/virtual node/i) + }) + + it('throws when group end_id is a virtual node (Result)', () => { + const groups = [makeGroup('g1', 'a', 'Result', ['a', 'b', 'c'])] + expect(() => buildStructureTree(modules, groups)).toThrow(/virtual node/i) + }) + + it('throws when group references Trigger', () => { + const groups = [makeGroup('g1', 'Trigger', 'c', ['a', 'b', 'c'])] + expect(() => buildStructureTree(modules, groups)).toThrow(/virtual node/i) + }) + + it('allows fully nested groups', () => { + const mods = [makeModule('a'), makeModule('b'), makeModule('c'), makeModule('d')] + const groups = [ + makeGroup('outer', 'a', 'd', ['a', 'b', 'c', 'd']), + makeGroup('inner', 'b', 'c', ['b', 'c']) + ] + const result = buildStructureTree(mods, groups) + expect(result).toHaveLength(1) // outer group contains everything + expect(result[0].kind).toBe('group') + // Inner group should be nested + const outerChildren = result[0].branches[0].children + expect(outerChildren).toHaveLength(3) // a, inner-group, d + expect(outerChildren[1].kind).toBe('group') + expect(outerChildren[1].id).toBe('inner') + }) + + it('handles empty modules', () => { + const result = buildStructureTree([], []) + expect(result).toHaveLength(0) + }) + + it('handles container modules (forloop)', () => { + const mods = [makeForloop('loop', ['x', 'y']), makeModule('c')] + const result = buildStructureTree(mods, []) + expect(result).toHaveLength(2) + expect(result[0].kind).toBe('forloopflow') + expect(result[0].branches).toHaveLength(1) + expect(result[0].branches[0].children).toHaveLength(2) + expect(result[0].branches[0].children[0].id).toBe('x') + }) + + it('handles groups inside containers', () => { + const mods = [makeForloop('loop', ['x', 'y', 'z'])] + const groups = [makeGroup('g1', 'x', 'y', ['x', 'y'])] + const result = buildStructureTree(mods, groups) + expect(result).toHaveLength(1) + expect(result[0].kind).toBe('forloopflow') + const innerChildren = result[0].branches[0].children + expect(innerChildren).toHaveLength(2) // group + z + expect(innerChildren[0].kind).toBe('group') + expect(innerChildren[0].id).toBe('g1') + }) + + it('throws when group spans parallel branches (branchall)', () => { + const mods = [ + makeModule('a'), + makeBranchAll('ba', [ + ['x', 'y'], + ['p', 'q'] + ]), + makeModule('c') + ] + const groups = [makeGroup('g1', 'x', 'q', ['x', 'q'])] + expect(() => buildStructureTree(mods, groups)).toThrow(/could not be resolved/) + }) +}) + +describe('flattenStructureIds', () => { + it('flattens a simple tree', () => { + const modules = [makeModule('a'), makeModule('b'), makeModule('c')] + const groups = [makeGroup('g1', 'a', 'b', ['a', 'b'])] + const tree = buildStructureTree(modules, groups) + const ids = flattenStructureIds(tree) + expect(ids).toEqual(['a', 'b', 'c']) + }) + + it('flattens nested groups', () => { + const mods = [makeModule('a'), makeModule('b'), makeModule('c'), makeModule('d')] + const groups = [ + makeGroup('outer', 'a', 'd', ['a', 'b', 'c', 'd']), + makeGroup('inner', 'b', 'c', ['b', 'c']) + ] + const tree = buildStructureTree(mods, groups) + const ids = flattenStructureIds(tree) + expect(ids).toEqual(['a', 'b', 'c', 'd']) + }) +}) + +describe('deriveGroupsFromStructure', () => { + it('derives group definitions with correct start/end', () => { + const modules = [makeModule('a'), makeModule('b'), makeModule('c')] + const groups = [makeGroup('g1', 'a', 'b', ['a', 'b'])] + const tree = buildStructureTree(modules, groups) + const derived = deriveGroupsFromStructure(tree) + expect(derived).toHaveLength(1) + expect(derived[0].start_id).toBe('a') + expect(derived[0].end_id).toBe('b') + }) + + it('derives nested groups', () => { + const mods = [makeModule('a'), makeModule('b'), makeModule('c'), makeModule('d')] + const groups = [ + makeGroup('outer', 'a', 'd', ['a', 'b', 'c', 'd']), + makeGroup('inner', 'b', 'c', ['b', 'c']) + ] + const tree = buildStructureTree(mods, groups) + const derived = deriveGroupsFromStructure(tree) + expect(derived).toHaveLength(2) + expect(derived[0].start_id).toBe('a') + expect(derived[0].end_id).toBe('d') + expect(derived[1].start_id).toBe('b') + expect(derived[1].end_id).toBe('c') + }) +}) + +describe('findInStructure', () => { + it('finds a leaf node', () => { + const modules = [makeModule('a'), makeModule('b')] + const tree = buildStructureTree(modules, []) + const found = findInStructure(tree, 'b') + expect(found).toBeDefined() + expect(found!.index).toBe(1) + }) + + it('finds a node inside a group', () => { + const modules = [makeModule('a'), makeModule('b'), makeModule('c')] + const groups = [makeGroup('g1', 'a', 'b', ['a', 'b'])] + const tree = buildStructureTree(modules, groups) + const found = findInStructure(tree, 'b') + expect(found).toBeDefined() + expect(found!.index).toBe(1) + // parentChildren should be the group's branch children + expect(found!.parentChildren).toHaveLength(2) + }) + + it('finds a group node by group id', () => { + const modules = [makeModule('a'), makeModule('b')] + const groups = [makeGroup('g1', 'a', 'b', ['a', 'b'])] + const tree = buildStructureTree(modules, groups) + const found = findInStructure(tree, 'g1') + expect(found).toBeDefined() + expect(found!.index).toBe(0) + }) +}) + +describe('collectLeafIds', () => { + it('collects all leaf module IDs including inside containers', () => { + const mods = [makeForloop('loop', ['x', 'y']), makeModule('c')] + const tree = buildStructureTree(mods, []) + const ids = collectLeafIds(tree) + expect(ids).toEqual(['loop', 'x', 'y', 'c']) + }) +}) diff --git a/frontend/src/lib/components/graph/flowStructure.ts b/frontend/src/lib/components/graph/flowStructure.ts new file mode 100644 index 0000000000..e2a725bc85 --- /dev/null +++ b/frontend/src/lib/components/graph/flowStructure.ts @@ -0,0 +1,498 @@ +import type { FlowModule } from '$lib/gen' + +import type { FlowGroup, GraphGroup } from './groupEditor.svelte' +import { getContainerInnerArrays } from './groupEditor.svelte' +import { VIRTUAL_NODE_IDS } from './groupDetectionUtils' + +// --------------------------------------------------------------------------- +// Types +// --------------------------------------------------------------------------- + +export type ContainerKind = 'forloopflow' | 'whileloopflow' | 'branchone' | 'branchall' + +export type StructureBranch = { + label?: string + children: FlowStructureNode[] +} + +export type FlowStructureNode = { + /** FlowModule.id for modules, groupKey(g) for groups */ + id: string + kind: 'leaf' | 'group' | ContainerKind + /** Only present when kind === 'group' */ + group?: FlowGroup + /** Only present when kind === 'group' — flat module IDs for step count */ + moduleIds?: string[] + /** Child branches. leaf=[], group=[{children}], container=[{children}, ...] */ + branches: StructureBranch[] +} + +// --------------------------------------------------------------------------- +// Type guards +// --------------------------------------------------------------------------- +// Building the structure tree +// --------------------------------------------------------------------------- + +export function buildStructureTree( + modules: FlowModule[], + groups: GraphGroup[] +): FlowStructureNode[] { + const { items, consumed } = buildStructureTreeRecurse(modules, groups) + const unconsumed = groups.filter((g) => !consumed.has(g.id)) + if (unconsumed.length > 0) { + throw new Error( + `Group(s) ${unconsumed.map((g) => `'${g.id}'`).join(', ')} could not be resolved: ` + + `their start/end nodes do not belong to the same branch` + ) + } + return items +} + +export function moduleToStructureNode(mod: FlowModule): FlowStructureNode { + const innerArrays = getContainerInnerArrays(mod) + if (innerArrays.length === 0) { + return { id: mod.id, kind: 'leaf', branches: [] } + } + + const kind = (mod.value as any).type as ContainerKind + const branches: StructureBranch[] = innerArrays.map(({ get, label }) => ({ + label, + children: [] // filled later by recursion + })) + + return { id: mod.id, kind, branches } +} + +function buildStructureTreeRecurse( + modules: FlowModule[], + groups: GraphGroup[] +): { items: FlowStructureNode[]; consumed: Set } { + if (modules.length === 0) { + return { items: [], consumed: new Set() } + } + + const indexMap = new Map() + for (let i = 0; i < modules.length; i++) { + indexMap.set(modules[i].id, i) + } + + // Reject duplicate group IDs + const seenGroupIds = new Set() + for (const g of groups) { + if (seenGroupIds.has(g.id)) { + throw new Error(`Duplicate group id: '${g.id}'`) + } + seenGroupIds.add(g.id) + } + + // Reject groups referencing virtual nodes + for (const g of groups) { + if (VIRTUAL_NODE_IDS.has(g.start_id) || VIRTUAL_NODE_IDS.has(g.end_id)) { + throw new Error( + `Group '${g.id}' references virtual node: groups cannot include Input, Result, or Trigger` + ) + } + } + + // Partition: groups for this level vs rest + const levelGroups: GraphGroup[] = [] + const otherGroups: GraphGroup[] = [] + for (const g of groups) { + if (indexMap.has(g.start_id) && indexMap.has(g.end_id)) { + const s = indexMap.get(g.start_id)! + const e = indexMap.get(g.end_id)! + if (s > e) { + throw new Error( + `Group '${g.id}' has inverted range: start_id='${g.start_id}' (index ${s}) > end_id='${g.end_id}' (index ${e})` + ) + } + levelGroups.push(g) + } else { + otherGroups.push(g) + } + } + + // Validate no partial overlaps + for (let i = 0; i < levelGroups.length; i++) { + for (let j = i + 1; j < levelGroups.length; j++) { + const a = levelGroups[i] + const b = levelGroups[j] + const aStart = indexMap.get(a.start_id)! + const aEnd = indexMap.get(a.end_id)! + const bStart = indexMap.get(b.start_id)! + const bEnd = indexMap.get(b.end_id)! + + if (aEnd < bStart || bEnd < aStart) continue + if (aStart <= bStart && bEnd <= aEnd) continue + if (bStart <= aStart && aEnd <= bEnd) continue + + throw new Error(`Groups '${a.id}' and '${b.id}' overlap without nesting`) + } + } + + // Build grouped structure for this level + function build( + startIdx: number, + endIdx: number, + availableGroups: GraphGroup[] + ): FlowStructureNode[] { + const result: FlowStructureNode[] = [] + let i = startIdx + while (i <= endIdx) { + const candidates = availableGroups.filter((g) => { + const gStart = indexMap.get(g.start_id)! + const gEnd = indexMap.get(g.end_id)! + return gStart === i && gEnd <= endIdx + }) + candidates.sort((a, b) => { + const spanA = indexMap.get(a.end_id)! - indexMap.get(a.start_id)! + const spanB = indexMap.get(b.end_id)! - indexMap.get(b.start_id)! + return spanB - spanA + }) + + const group = candidates[0] + if (group) { + const gEnd = indexMap.get(group.end_id)! + const remaining = availableGroups.filter((g) => g.id !== group.id) + const innerNodes = build(i, gEnd, remaining) + + const moduleIds: string[] = [] + for (let k = i; k <= gEnd; k++) { + moduleIds.push(modules[k].id) + } + + result.push({ + id: group.id, + kind: 'group', + group: { + summary: group.summary, + note: group.note, + color: group.color, + autocollapse: group.autocollapse, + start_id: group.start_id, + end_id: group.end_id + }, + moduleIds, + branches: [{ children: innerNodes }] + }) + i = gEnd + 1 + } else { + result.push(moduleToStructureNode(modules[i])) + i++ + } + } + return result + } + + const result = build(0, modules.length - 1, levelGroups) + + // Recurse into containers with remaining unconsumed groups + const consumed = new Set(levelGroups.map((g) => g.id)) + let remaining = otherGroups + + function recurseIntoContainers(items: FlowStructureNode[]): void { + for (const item of items) { + if (item.kind === 'group') { + recurseIntoContainers(item.branches[0].children) + continue + } + if (item.branches.length === 0) continue + + // This is a container module — get inner FlowModule arrays and recurse + const modIdx = indexMap.get(item.id) + if (modIdx === undefined) continue + const mod = modules[modIdx] + + const innerArrays = getContainerInnerArrays(mod) + for (let bi = 0; bi < innerArrays.length; bi++) { + const inner = buildStructureTreeRecurse(innerArrays[bi].get(), remaining) + item.branches[bi] = { + label: item.branches[bi]?.label, + children: inner.items + } + for (const id of inner.consumed) consumed.add(id) + remaining = remaining.filter((g) => !inner.consumed.has(g.id)) + } + } + } + recurseIntoContainers(result) + + return { items: result, consumed } +} + +// --------------------------------------------------------------------------- +// Traversal utilities +// --------------------------------------------------------------------------- + +/** Generic DFS over the structure tree */ +export function dfsStructure( + nodes: FlowStructureNode[], + fn: (node: FlowStructureNode, parentArray: FlowStructureNode[]) => void +): void { + for (const node of nodes) { + fn(node, nodes) + for (const branch of node.branches) { + dfsStructure(branch.children, fn) + } + } +} + +/** Flatten to ordered module IDs (groups are transparent) */ +export function flattenStructureIds(nodes: FlowStructureNode[]): string[] { + const ids: string[] = [] + for (const node of nodes) { + if (node.kind === 'group') { + ids.push(...flattenStructureIds(node.branches[0].children)) + } else { + ids.push(node.id) + } + } + return ids +} + +/** Collect leaf module IDs recursively (including inside containers) */ +export function collectLeafIds(nodes: FlowStructureNode[]): string[] { + const ids: string[] = [] + for (const node of nodes) { + if (node.kind === 'group') { + ids.push(...collectLeafIds(node.branches[0].children)) + } else { + ids.push(node.id) + for (const branch of node.branches) { + ids.push(...collectLeafIds(branch.children)) + } + } + } + return ids +} + +// --------------------------------------------------------------------------- +// Finding nodes in the tree +// --------------------------------------------------------------------------- + +export type FindResult = { parentChildren: FlowStructureNode[]; index: number } + +export function findInStructure(nodes: FlowStructureNode[], id: string): FindResult | undefined { + for (let i = 0; i < nodes.length; i++) { + const node = nodes[i] + if (node.id === id) return { parentChildren: nodes, index: i } + for (const branch of node.branches) { + const found = findInStructure(branch.children, id) + if (found) return found + } + } + return undefined +} + +/** + * Match a structure node against a graph node ID. + * Handles group head/end IDs (group:X, group:X-end) and collapsed-group:X. + */ +export function matchStructureNode(node: FlowStructureNode, nodeId: string): boolean { + if (node.id === nodeId) return true + if (node.kind === 'group') { + return ( + nodeId === `group:${node.id}` || + nodeId === `group:${node.id}-end` || + nodeId === `collapsed-group:${node.id}` + ) + } + return false +} + +/** + * Find insert index using graph node IDs (handles group:X-end etc.). + * Returns the index OF the matched item (insert before it). + * For group-end nodes, returns index AFTER the group (insert after it). + */ +export function findInsertIndexByNodeId(items: FlowStructureNode[], targetNodeId: string): number { + // group-end: insert after the group + if (targetNodeId.startsWith('group:') && targetNodeId.endsWith('-end')) { + const groupId = targetNodeId.slice('group:'.length, -'-end'.length) + const idx = items.findIndex((n) => n.kind === 'group' && n.id === groupId) + return idx >= 0 ? idx + 1 : items.length + } + // Everything else: insert at the matched item's position + for (let i = 0; i < items.length; i++) { + if (matchStructureNode(items[i], targetNodeId)) return i + } + return items.length +} + +// --------------------------------------------------------------------------- +// Deriving groups from the structure tree +// --------------------------------------------------------------------------- + +export function deriveGroupsFromStructure(nodes: FlowStructureNode[]): FlowGroup[] { + const groups: FlowGroup[] = [] + for (const node of nodes) { + if (node.kind === 'group' && node.group) { + const flatIds = flattenStructureIds(node.branches[0].children) + if (flatIds.length === 0) { + console.warn(`deriveGroupsFromStructure: skipping empty group "${node.id}"`) + continue + } + groups.push({ + ...node.group, + start_id: flatIds[0], + end_id: flatIds[flatIds.length - 1] + }) + // Recurse for nested groups + groups.push(...deriveGroupsFromStructure(node.branches[0].children)) + } else { + for (const branch of node.branches) { + groups.push(...deriveGroupsFromStructure(branch.children)) + } + } + } + return groups +} + +// --------------------------------------------------------------------------- +// Syncing structure back to FlowModule[] +// --------------------------------------------------------------------------- + +/** + * Reconstruct a FlowModule[] from the structure tree, looking up originals + * from moduleMap and patching container inner arrays to match the tree ordering. + */ +export function applyStructureToModules( + nodes: FlowStructureNode[], + moduleMap: Map +): FlowModule[] { + const result: FlowModule[] = [] + for (const node of nodes) { + if (node.kind === 'group') { + // Groups are transparent — splice their children into this level + result.push(...applyStructureToModules(node.branches[0].children, moduleMap)) + } else { + const mod = moduleMap.get(node.id) + if (!mod) continue + + // Patch container inner arrays + if (node.branches.length > 0) { + const innerArrays = getContainerInnerArrays(mod) + for (let bi = 0; bi < innerArrays.length && bi < node.branches.length; bi++) { + innerArrays[bi].set(applyStructureToModules(node.branches[bi].children, moduleMap)) + } + } + + result.push(mod) + } + } + return result +} + +// --------------------------------------------------------------------------- +// Empty groups cleanup +// --------------------------------------------------------------------------- + +/** + * Walk the tree, remove group nodes that have no leaf modules, and return + * the removed groups. Mutates the input array in-place. + * Recurses depth-first so inner groups are cleaned before checking outer ones. + */ +export function removeEmptyGroups(nodes: FlowStructureNode[]): FlowGroup[] { + const removed: FlowGroup[] = [] + for (let i = nodes.length - 1; i >= 0; i--) { + const node = nodes[i] + if (node.kind === 'group' && node.group) { + // Recurse first — inner groups may become empty too + removed.push(...removeEmptyGroups(node.branches[0].children)) + if (flattenStructureIds(node.branches[0].children).length === 0) { + removed.push(node.group) + nodes.splice(i, 1) + } + } else { + for (const branch of node.branches) { + removed.push(...removeEmptyGroups(branch.children)) + } + } + } + return removed +} + +/** Walk the structure tree to compute nesting depth for each group (O(n)). */ +export function computeGroupDepths(tree: FlowStructureNode[]): Record { + const depths: Record = {} + function walk(nodes: FlowStructureNode[], groupDepth: number): void { + for (const node of nodes) { + if (node.kind === 'group') { + depths[node.id] = groupDepth + for (const branch of node.branches) { + walk(branch.children, groupDepth + 1) + } + } else { + for (const branch of node.branches) { + walk(branch.children, groupDepth) + } + } + } + } + walk(tree, 0) + return depths +} + +/** + * Find duplicate groups in the structure tree (same start_id:end_id after mutation). + * Returns the groups that should be removed (keeps the first, removes subsequent duplicates). + */ +export function findDuplicateGroups(nodes: FlowStructureNode[]): FlowGroup[] { + const duplicates: FlowGroup[] = [] + const seen = new Set() + + function walk(items: FlowStructureNode[]): void { + for (const node of items) { + if (node.kind === 'group' && node.group) { + const flatIds = flattenStructureIds(node.branches[0].children) + if (flatIds.length > 0) { + const key = `${flatIds[0]}:${flatIds[flatIds.length - 1]}` + if (seen.has(key)) { + duplicates.push(node.group) + } else { + seen.add(key) + } + } + walk(node.branches[0].children) + } else { + for (const branch of node.branches) { + walk(branch.children) + } + } + } + } + walk(nodes) + return duplicates +} + +/** Remove duplicate groups from the structure tree (keeps first occurrence). */ +export function removeDuplicateGroups(nodes: FlowStructureNode[]): FlowGroup[] { + const removed: FlowGroup[] = [] + const seen = new Set() + + function walk(items: FlowStructureNode[]): void { + for (let i = items.length - 1; i >= 0; i--) { + const node = items[i] + if (node.kind === 'group' && node.group) { + walk(node.branches[0].children) + const flatIds = flattenStructureIds(node.branches[0].children) + if (flatIds.length > 0) { + const key = `${flatIds[0]}:${flatIds[flatIds.length - 1]}` + if (seen.has(key)) { + // Replace group node with its children (ungroup) + removed.push(node.group) + items.splice(i, 1, ...node.branches[0].children) + } else { + seen.add(key) + } + } + } else { + for (const branch of node.branches) { + walk(branch.children) + } + } + } + } + walk(nodes) + return removed +} diff --git a/frontend/src/lib/components/graph/graphBuilder.svelte.ts b/frontend/src/lib/components/graph/graphBuilder.svelte.ts index 192b5c0c34..4fd7641096 100644 --- a/frontend/src/lib/components/graph/graphBuilder.svelte.ts +++ b/frontend/src/lib/components/graph/graphBuilder.svelte.ts @@ -8,6 +8,14 @@ import { getFlowModuleAssets, type AssetWithAltAccessType } from '../assets/lib' import { assetDisplaysAsOutputInFlowGraph } from './renderers/nodes/AssetNode.svelte' import type { ModulesTestStates, ModuleTestState } from '../modulesTest.svelte' import type { ModuleActionInfo } from '$lib/components/flows/flowDiff' +import { + type FlowStructureNode, + collectLeafIds, + findInsertIndexByNodeId, + buildStructureTree +} from './flowStructure' +import { groupKey, type FlowGroup } from './groupEditor.svelte' +import { computeGroupModuleIds } from './groupDetectionUtils' export type InsertKind = | 'script' @@ -62,6 +70,7 @@ export type GraphEventHandlers = { simplifyFlow: (b: boolean) => void expandSubflow: (id: string, path: string) => void minimizeSubflow: (id: string) => void + expandGroup: (groupId: string) => void updateMock: (detail: { mock: FlowModule['mock']; id: string }) => void testUpTo: (id: string) => void editInput: (moduleId: string, key: string) => void @@ -111,6 +120,9 @@ export type FlowNode = | AssetsOverflowedN | AiToolN | NewAiToolN + | CollapsedGroupN + | GroupHeadN + | GroupEndN export type InputN = { type: 'input2' @@ -316,6 +328,48 @@ export type NewAiToolN = { } } +export type CollapsedGroupN = { + type: 'collapsedGroup' + data: { + groupId: string + summary: string | undefined + note: string | undefined + color: string | undefined + autocollapse: boolean | undefined + stepCount: number + modules: FlowModule[] + flowModuleStates: Record | undefined + flowJob: Job | undefined + isOwner: boolean + suspendStatus: Record + showNotes: boolean + editMode: boolean + eventHandlers: GraphEventHandlers + } +} + +export type GroupHeadN = { + type: 'groupHead' + data: { + groupId: string + summary: string | undefined + note: string | undefined + color: string | undefined + autocollapse: boolean | undefined + editMode: boolean + showNotes: boolean + eventHandlers: GraphEventHandlers + wrapperWidth?: number + } +} + +export type GroupEndN = { + type: 'groupEnd' + data: { + groupId: string + } +} + export function topologicalSort( nodes: { id: string; parentIds?: string[] }[] ): { id: string; parentIds?: string[] }[] { @@ -336,22 +390,8 @@ export function topologicalSort( return result.reverse() } -// input2: InputNode, -// module: ModuleNode, -// branchAllStart: BranchAllStart, -// branchAllEnd: BranchAllEndNode, -// forLoopEnd: ForLoopEndNode, -// forLoopStart: ForLoopStartNode, -// result: ResultNode, -// whileLoopStart: ForLoopStartNode, -// whileLoopEnd: ForLoopEndNode, -// branchOneStart: BranchOneStart, -// branchOneEnd: BranchOneEndNode, -// subflowBound: SubflowBound, -// noBranch: NoBranchNode, -// trigger: TriggersNode - export function graphBuilder( + structureTree: FlowStructureNode[], modules: FlowModule[] | undefined, extra: { disableAi: boolean @@ -383,11 +423,9 @@ export function graphBuilder( selectedId: string | undefined, simplifiableFlow: SimplifiableFlow | undefined, flowPathForTriggerNode: string | undefined, - expandedSubflows: Record - // triggerProps?: { - // path?: string - // flowIsSimplifiable?: boolean - // } + expandedSubflows: Record, + showNotes: boolean, + collapsedGroupIds: Set ): { nodes: { [key: string]: NodeLayout } edges: Edge[] @@ -403,7 +441,13 @@ export function graphBuilder( const nodes: NodeLayout[] = [] const edges: Edge[] = [] - function addNode(module: FlowModule) { + // Lookup map from module ID to the original reactive FlowModule objects. + const moduleMap = new Map() + for (const m of getAllModules(modules, failureModule)) { + moduleMap.set(m.id, m) + } + + function addNode(module: FlowModule, extraData?: Record) { const duplicated = nodes.find((n) => n.id === module.id) if (duplicated) { console.log('Duplicated node detected: ', module, duplicated) @@ -424,7 +468,8 @@ export function graphBuilder( isOwner: extra.isOwner, flowJob: extra.flowJob, assets: getFlowModuleAssets(module, extra.additionalAssetsMap), - moduleAction: extra.moduleActions?.[module.id] + moduleAction: extra.moduleActions?.[module.id], + ...extraData }, type: 'module', selectable: true @@ -483,14 +528,20 @@ export function graphBuilder( customId?: string type?: string subModules?: FlowModule[] + currentItems?: FlowStructureNode[] disableMoveIds?: string[] } ) { parents[targetId] = [...(parents[targetId] ?? []), sourceId] - const mods = options?.subModules ?? modules - - let index = mods?.findIndex((m) => m.id === targetId) ?? -1 + let index: number + if (options?.currentItems) { + index = findInsertIndexByNodeId(options.currentItems, targetId) + } else { + const mods = options?.subModules ?? modules + const found = mods?.findIndex((m) => m.id === targetId) ?? -1 + index = found >= 0 ? found : (mods?.length ?? 0) + } const visited = new Set() const recStack = new Set() @@ -514,8 +565,7 @@ export function graphBuilder( simplifiedTriggerView: simplifiableFlow?.simplifiedFlow, disableMoveIds: options?.disableMoveIds, enableTrigger: sourceId === 'Input', - // If the index is -1, it means that the target module is not in the modules array, so we set it to the length of the array - index: index >= 0 ? index : (mods?.length ?? 0), + index, ...extra, insertable: extra.insertable && !options?.disableInsert && prefix == undefined, shouldOffsetInsertBtnDueToAssetNode: nodeIdsWithOutputAssets.has(sourceId) @@ -591,7 +641,7 @@ export function graphBuilder( } function processModules( - modules: FlowModule[], + items: FlowStructureNode[], branch: { rootId: string; branch: number } | undefined, beforeNode: NodeLayout, nextNode: NodeLayout | undefined, @@ -600,31 +650,166 @@ export function graphBuilder( disableMoveIds: string[] = [], parentIndex?: string ) { + // For subflow prefix rewriting, clone modules into moduleMap with prefixed IDs + // (avoid mutating reactive originals which would trigger state_unsafe_mutation in $derived) if (prefix != undefined) { - modules.forEach((m) => { - if (!m['oid']) { - m['oid'] = m.id + items.forEach((item) => { + if (item.kind === 'group') return + const m = moduleMap.get(item.id) + if (m) { + const oid = m['oid'] ?? m.id + const newId = 'subflow:' + prefix + oid + const clone = { ...m, id: newId, oid } as FlowModule & { oid: string } + clone['oid'] = oid + moduleMap.set(newId, clone) + item.id = newId } - m.id = 'subflow:' + prefix + m['oid'] }) } let previousId: string | undefined = undefined - if (modules.length === 0) { + if (items.length === 0) { if (nextNode) { addEdge(beforeNode.id, nextNode.id, branch, prefix, { - subModules: modules, + currentItems: items, disableMoveIds }) } } else { - modules.forEach((module, index) => { + items.forEach((item, index) => { + // --- Group items --- + if (item.kind === 'group') { + const g = item.group! + const gId = item.id + + if (collapsedGroupIds.has(gId)) { + // Collapsed group: single node + const nodeId = `collapsed-group:${gId}` + const leafIds = collectLeafIds(item.branches[0].children) + nodes.push({ + id: nodeId, + data: { + groupId: gId, + summary: g.summary, + note: g.note, + color: g.color, + autocollapse: g.autocollapse, + stepCount: item.moduleIds?.length ?? 0, + modules: leafIds + .map((id) => moduleMap.get(id)) + .filter((m): m is FlowModule => !!m), + flowModuleStates: extra.flowModuleStates, + flowJob: extra.flowJob, + isOwner: extra.isOwner, + suspendStatus: extra.suspendStatus, + showNotes, + editMode: prefix == undefined && extra.editMode, + eventHandlers + }, + type: 'collapsedGroup', + selectable: false + }) + + // Wire: previous → collapsedGroup + if (index > 0 && previousId) { + addEdge(previousId, nodeId, branch, prefix, { + currentItems: items, + disableMoveIds + }) + } + + previousId = nodeId + } else { + // Expanded group: head → recurse → end + const headId = `group:${gId}` + const endId = `group:${gId}-end` + const localDisableMoveIds = [...disableMoveIds, headId] + + const headNode: NodeLayout = { + id: headId, + data: { + groupId: gId, + summary: g.summary, + note: g.note, + color: g.color, + autocollapse: g.autocollapse, + editMode: prefix == undefined && extra.editMode, + showNotes, + eventHandlers + }, + type: 'groupHead', + selectable: false + } + + const endNode: NodeLayout = { + id: endId, + data: { + groupId: gId + }, + type: 'groupEnd', + selectable: false + } + + nodes.push(headNode) + nodes.push(endNode) + + // Wire: previous → headNode + if (index > 0 && previousId) { + addEdge(previousId, headId, branch, prefix, { + currentItems: items, + disableMoveIds + }) + } + + // Recurse inner modules + processModules( + item.branches[0].children, + { rootId: headId, branch: 0 }, + headNode, + endNode, + simplifiedTriggerView, + prefix, + localDisableMoveIds, + parentIndex + ) + + previousId = endId + } + + // Shared first/last edge wiring for groups + if (index === 0) { + addEdge( + beforeNode.id, + collapsedGroupIds.has(gId) ? `collapsed-group:${gId}` : `group:${gId}`, + undefined, + prefix, + { + currentItems: items, + disableMoveIds, + disableInsert: simplifiedTriggerView + } + ) + } + + if (index === items.length - 1 && previousId && nextNode) { + addEdge(previousId, nextNode.id, branch, prefix, { + currentItems: items, + disableMoveIds + }) + } + + return + } + + // --- Regular FlowModule items --- + const module = moduleMap.get(item.id) + if (!module) return const localDisableMoveIds = [...disableMoveIds, module.id] - // Add the edge between the previous node and the current one + // Inter-module edge: connect previous → current (expanded subflows handle their own) if (index > 0 && previousId && expandedSubflows[module.id] == undefined) { addEdge(previousId, module.id, branch, prefix, { - subModules: modules, + currentItems: items, disableMoveIds }) } @@ -700,7 +885,7 @@ export function graphBuilder( ) processModules( - branch.modules, + item.branches[branchIndex]?.children ?? [], { rootId: module.id, branch: branchIndex }, startNode, endNode, @@ -722,7 +907,7 @@ export function graphBuilder( id: `${module.id}-start`, data: { id: module.id, - module: module, + module: moduleMap.get(module.id) ?? module, simplifiedTriggerView, eventHandlers: eventHandlers, editMode: extra.editMode, @@ -759,7 +944,7 @@ export function graphBuilder( const selectedIterIndex = extra.flowModuleStates?.[module.id]?.selectedForloopIndex processModules( - module.value.modules, + item.branches[0]?.children ?? [], { rootId: module.id, branch: 0 }, startNode, endNode, @@ -798,7 +983,7 @@ export function graphBuilder( const selectedIterIndex = extra.flowModuleStates?.[module.id]?.selectedForloopIndex processModules( - module.value.modules, + item.branches[0]?.children ?? [], { rootId: module.id, branch: 0 }, startNode, endNode, @@ -825,21 +1010,6 @@ export function graphBuilder( } nodes.push(endNode) - // // Add default branch - // const defaultBranch: NodeLayout = { - // id: `${module.id}-default`, - // data: { - // offset: 0, - // label: 'Default', - // id: module.id, - // branchIndex: -1, - // eventHandlers: eventHandlers, - // branchOne: true, - // ...extra - // }, - // type: 'noBranch' - // } - const defaultBranch: NodeLayout = { id: `${module.id}-branch-default`, data: { @@ -863,7 +1033,7 @@ export function graphBuilder( }) processModules( - module.value.default, + item.branches[0]?.children ?? [], { rootId: module.id, branch: 0 }, defaultBranch, endNode, @@ -899,7 +1069,7 @@ export function graphBuilder( }) processModules( - branch.modules, + item.branches[branchIndex + 1]?.children ?? [], { rootId: module.id, branch: branchIndex + 1 }, startNode, endNode, @@ -912,9 +1082,9 @@ export function graphBuilder( previousId = endNode.id } else { - let expanded = expandedSubflows[module.id] - if (expanded) { - expanded = $state.snapshot(expanded) + const expandedData = expandedSubflows[module.id] + if (expandedData) { + const expandedMods = $state.snapshot(expandedData.modules) as FlowModule[] const startId = `${module.id}` const idWithoutPrefix = module.id.startsWith('subflow:') ? module.id.substring(8) @@ -936,12 +1106,12 @@ export function graphBuilder( if (previousId) { addEdge(previousId, startNode.id, branch, prefix, { - subModules: modules, + currentItems: items, disableMoveIds }) } else { addEdge(beforeNode.id, startNode.id, undefined, prefix, { - subModules: modules, + currentItems: items, disableMoveIds }) } @@ -962,8 +1132,20 @@ export function graphBuilder( nodes.push(endNode) + // Register expanded subflow modules so prefix rewriting finds + // the inner modules (not the parent flow's modules with same IDs) + for (const em of getAllModules(expandedMods)) { + moduleMap.set(em.id, em) + } + + const expandedGroups = (expandedData.groups ?? []).map((g) => ({ + ...g, + id: groupKey(g), + moduleIds: computeGroupModuleIds(g.start_id, g.end_id, getAllModules(expandedMods)) + })) + processModules( - expanded, + buildStructureTree(expandedMods, expandedGroups), undefined, startNode, endNode, @@ -981,15 +1163,15 @@ export function graphBuilder( if (index === 0 && expandedSubflows[module.id] == undefined) { addEdge(beforeNode.id, module.id, undefined, prefix, { - subModules: modules, + currentItems: items, disableMoveIds, disableInsert: simplifiedTriggerView }) } - if (index === modules.length - 1 && previousId && nextNode) { + if (index === items.length - 1 && previousId && nextNode) { addEdge(previousId, nextNode.id, branch, prefix, { - subModules: modules, + currentItems: items, disableMoveIds }) } @@ -997,10 +1179,12 @@ export function graphBuilder( } } + const topLevelItems = structureTree + if (simplifiableFlow?.simplifiedFlow === true && triggerNode) { - processModules(modules, undefined, triggerNode, undefined, true, undefined) + processModules(topLevelItems, undefined, triggerNode, undefined, true, undefined) } else { - processModules(modules, undefined, inputNode, resultNode, false, undefined) + processModules(topLevelItems, undefined, inputNode, resultNode, false, undefined) } if (failureModule) { diff --git a/frontend/src/lib/components/graph/graphContext.ts b/frontend/src/lib/components/graph/graphContext.ts index 8540982366..a6245769cf 100644 --- a/frontend/src/lib/components/graph/graphContext.ts +++ b/frontend/src/lib/components/graph/graphContext.ts @@ -4,6 +4,7 @@ import type { NoteManager } from './noteManager.svelte' import type { MoveManager } from './moveManager.svelte' import type { Writable } from 'svelte/store' import type { FlowDiffManager } from '../flows/flowDiffManager.svelte' +import type { GroupDisplayState } from './groupEditor.svelte' export type GraphContext = { selectionManager: SelectionManager @@ -14,6 +15,9 @@ export type GraphContext = { clearFlowSelection?: () => void yOffset?: number diffManager: FlowDiffManager + /** Current flow nodes for group validation (set by FlowGraphV2) */ + getFlowNodes?: () => { id: string; parentIds?: string[] }[] + groupDisplayState?: GroupDisplayState } const graphContextKey = 'FlowGraphContext' diff --git a/frontend/src/lib/components/graph/groupDetectionUtils.ts b/frontend/src/lib/components/graph/groupDetectionUtils.ts index e2c81c4d98..8dc6e3d8c2 100644 --- a/frontend/src/lib/components/graph/groupDetectionUtils.ts +++ b/frontend/src/lib/components/graph/groupDetectionUtils.ts @@ -1,7 +1,127 @@ +import { topologicalSort } from './graphBuilder.svelte' + +/** Node IDs synthesized by graphBuilder that are not real FlowModules */ +export const VIRTUAL_NODE_IDS = new Set(['Input', 'Result', 'Trigger']) + type FlowNode = { id: string; parentIds?: string[] } /** - * Use a simple algorithm to complete a group and split it into connected components + * Compute the set of module IDs that belong to a group defined by start_id and end_id. + * Uses the flattened module list (from getAllModules) and slices between start and end. + * Used for collapsed group icons, step count, and moduleToCollapsedGroup mapping. + */ +export function computeGroupModuleIds( + startId: string, + endId: string, + allModules: { id: string }[] +): string[] { + if (startId === endId) { + return allModules.some((m) => m.id === startId) ? [startId] : [] + } + + const startIdx = allModules.findIndex((m) => m.id === startId) + const endIdx = allModules.findIndex((m) => m.id === endId) + + if (startIdx === -1 || endIdx === -1 || startIdx > endIdx) { + if (startIdx > endIdx) { + console.warn( + `computeGroupModuleIds: inverted range for group ${startId}→${endId} (${startIdx} > ${endIdx})` + ) + } + return [] + } + + return allModules.slice(startIdx, endIdx + 1).map((m) => m.id) +} + +/** + * Check whether a set of selected node IDs can form a valid group. + * Normalizes marker IDs (branch/forloop) to parent module IDs, + * then uses topologicalSort to derive start and end boundaries. + */ +export function canFormValidGroup( + selectedIds: string[], + flowNodes: FlowNode[], + excludeIds?: Set +): { valid: true; startId: string; endId: string } | { valid: false } { + if (selectedIds.length === 0) return { valid: false } + + // Normalize marker IDs to parent module IDs. + // -start (forloop head) → parent ID. -end/-branch-* → skip if parent covered, else reject. + const rawSet = new Set(selectedIds) + const normalizedIds: string[] = [] + + for (const id of selectedIds) { + const parentId = id.replace(/-(end|start|branch-.*)$/, '') + if (parentId === id) { + normalizedIds.push(id) + continue + } + if (id.endsWith('-start')) { + normalizedIds.push(parentId) + continue + } + // -end or -branch-*: parent must be covered (directly or via -start) + if (!rawSet.has(parentId) && !rawSet.has(`${parentId}-start`)) { + return { valid: false } + } + } + + if (normalizedIds.length === 0) return { valid: false } + const normalizedSet = new Set(normalizedIds) + + // Topo sort full graph, filter to normalized selection. + // Include raw matches plus all markers (-start, -end, -branch-*) whose parent is selected. + const sorted = topologicalSort(flowNodes) + const selectedSorted = sorted.filter((n) => { + if (normalizedSet.has(n.id)) return true + const parentId = n.id.replace(/-(end|start|branch-.*)$/, '') + return parentId !== n.id && normalizedSet.has(parentId) + }) + + if (selectedSorted.length === 0) return { valid: false } + + // Reject virtual or excluded nodes + if (selectedSorted.some((n) => VIRTUAL_NODE_IDS.has(n.id) || excludeIds?.has(n.id))) { + return { valid: false } + } + + // Topo order is bottom-first: first = bottom (end), last = top (start). + // Use raw IDs for BFS traversal, normalize for the returned group boundaries. + const rawStartId = selectedSorted[selectedSorted.length - 1].id + const rawEndId = selectedSorted[0].id + const startId = rawStartId.replace(/-(end|start|branch-.*)$/, '') + const endId = rawEndId.replace(/-(end|start|branch-.*)$/, '') + + // Verify all selected nodes lie between start and end in the DAG. + // BFS backward from rawEndId to rawStartId to collect reachable nodes. + // Normalize collected IDs so container markers map to their parent module. + const between = new Set() + const queue = [rawEndId] + const visited = new Set() + const parentMap = new Map(flowNodes.map((n) => [n.id, n.parentIds ?? []])) + while (queue.length > 0) { + const cur = queue.shift()! + if (visited.has(cur)) continue + visited.add(cur) + const normalized = cur.replace(/-(end|start|branch-.*)$/, '') + between.add(cur) + between.add(normalized) + if (cur === rawStartId) continue + for (const p of parentMap.get(cur) ?? []) { + queue.push(p) + } + } + if (!normalizedIds.every((id) => between.has(id))) { + return { valid: false } + } + + return { valid: true, startId, endId } +} + +/** + * Legacy utility: complete a group and split it into connected components. + * Still used by NoteEditor for FlowNote group notes (contained_node_ids). */ export function completeAndSplitGroup(groupNodes: string[], flowNodes: FlowNode[]): string[][] { if (groupNodes.length <= 1) { diff --git a/frontend/src/lib/components/graph/groupEditor.svelte.ts b/frontend/src/lib/components/graph/groupEditor.svelte.ts new file mode 100644 index 0000000000..8405390b96 --- /dev/null +++ b/frontend/src/lib/components/graph/groupEditor.svelte.ts @@ -0,0 +1,325 @@ +import type { FlowModule } from '$lib/gen' +import type { StateStore } from '$lib/utils' +import type { ExtendedOpenFlow } from '../flows/types' + +import { canFormValidGroup } from './groupDetectionUtils' +import type { NoteColor } from './noteColors' +import { DEFAULT_GROUP_NOTE_COLOR, getNextAvailableColor } from './noteColors' +import { getContext, setContext } from 'svelte' + +/** + * Type for a flow group (matches the generated type from OpenAPI). + * Members are computed dynamically from all nodes on paths between start_id and end_id. + */ +export type FlowGroup = { + summary?: string + note?: string + autocollapse?: boolean + start_id: string + end_id: string + color?: string +} + +/** Derive a stable key from a group's boundaries. Used as ephemeral ID for graph nodes, runtime state, etc. */ +export function groupKey(g: { start_id: string; end_id: string }): string { + return `${g.start_id}:${g.end_id}` +} + +/** + * Display state for flow groups inside the graph. + * Handles runtime collapse state and note height tracking. + * Similar to NoteManager — instantiated inside FlowGraphV2. + */ +export class GroupDisplayState { + #getGroups: () => FlowGroup[] + #runtimeCollapsedIds = $state>(new Set()) + #runtimeInitialized = $state(false) + #noteHeights = $state>({}) + renderCount = $state(0) + + constructor(getGroups: () => FlowGroup[]) { + this.#getGroups = getGroups + } + + /** Initialize runtime state from autocollapse. Safe to call from event handlers. */ + private ensureRuntimeInitialized(): void { + if (this.#runtimeInitialized) return + const groups = this.#getGroups() + this.#runtimeCollapsedIds = new Set( + groups.filter((g) => g.autocollapse).map((g) => groupKey(g)) + ) + this.#runtimeInitialized = true + } + + /** Check if a group is currently collapsed (runtime). Safe to call from $derived. */ + isRuntimeCollapsed(groupId: string): boolean { + if (!this.#runtimeInitialized) { + return this.#getGroups().find((g) => groupKey(g) === groupId)?.autocollapse ?? false + } + return this.#runtimeCollapsedIds.has(groupId) + } + + /** Toggle runtime collapse (Minimize2 button) */ + toggleRuntimeCollapse(groupId: string): void { + this.ensureRuntimeInitialized() + const next = new Set(this.#runtimeCollapsedIds) + if (next.has(groupId)) next.delete(groupId) + else next.add(groupId) + this.#runtimeCollapsedIds = next + this.render() + } + + /** Expand a group at runtime (CollapsedGroupNode click) */ + expandGroup(groupId: string): void { + this.ensureRuntimeInitialized() + const next = new Set(this.#runtimeCollapsedIds) + next.delete(groupId) + this.#runtimeCollapsedIds = next + this.render() + } + + /** Set note height for a group (used for layout spacing) */ + setNoteHeight(groupId: string, height: number): void { + if (this.#noteHeights[groupId] !== height) { + this.#noteHeights[groupId] = height + this.render() + } + } + + /** Get all note heights */ + getNoteHeights(): Record { + return this.#noteHeights + } + + /** Bump render counter to trigger re-layout */ + render(): void { + this.renderCount++ + } + + /** Remap runtime state when a group's boundaries (and thus its key) change */ + remapGroupKey(oldKey: string, newKey: string): void { + if (this.#runtimeCollapsedIds.has(oldKey)) { + const next = new Set(this.#runtimeCollapsedIds) + next.delete(oldKey) + next.add(newKey) + this.#runtimeCollapsedIds = next + } + if (oldKey in this.#noteHeights) { + this.#noteHeights[newKey] = this.#noteHeights[oldKey] + delete this.#noteHeights[oldKey] + } + } + + /** Get currently collapsed groups for graph builder. Safe to call from $derived. */ + getCollapsedGroups(): FlowGroup[] { + if (!this.#runtimeInitialized) { + return this.#getGroups().filter((g) => g.autocollapse) + } + return this.#getGroups().filter((g) => this.#runtimeCollapsedIds.has(groupKey(g))) + } +} + +/** + * Utility class for editing flow groups via direct flowStore mutations. + * Follows the same pattern as NoteEditor. + */ +export class GroupEditor { + private flowStore: StateStore + + constructor(flowStore: StateStore) { + this.flowStore = flowStore + } + + getGroups(): FlowGroup[] { + return this.flowStore.val.value?.groups || [] + } + + private setGroups(groups: FlowGroup[]): void { + if (this.flowStore.val.value) { + this.flowStore.val.value.groups = groups + } + } + + /** IDs that cannot be part of a group (preprocessor, failure module) */ + getExcludeIds(): Set { + const excludeIds = new Set() + const pp = this.flowStore.val.value?.preprocessor_module?.id + if (pp) excludeIds.add(pp) + const fm = this.flowStore.val.value?.failure_module?.id + if (fm) excludeIds.add(fm) + return excludeIds + } + + /** Check whether the given selection can form a valid group */ + canCreateGroup( + selectedIds: string[], + flowNodes: { id: string; parentIds?: string[] }[] + ): boolean { + const result = canFormValidGroup(selectedIds, flowNodes, this.getExcludeIds()) + if (!result.valid) return false + // Reject if a group with the same boundaries already exists + return !this.getGroups().some((g) => g.start_id === result.startId && g.end_id === result.endId) + } + + /** + * Create a new group from selected node IDs. + * Uses canFormValidGroup to determine start_id and end_id. + * Returns the generated group ID. + */ + createGroup( + moduleIds: string[], + flowNodes: { id: string; parentIds?: string[] }[] + ): string | undefined { + // Filter subflow node IDs (same logic as NoteEditor.createGroupNote) + let filteredIds = [...moduleIds] + const subflowIds: string[] = [] + for (const id of moduleIds) { + if (id.startsWith('subflow:')) { + const match = id.match(/^subflow:([^:]+)/) + if (match) { + subflowIds.push(match[1]) + } + } + } + if (subflowIds.length > 0) { + filteredIds = filteredIds.filter((id) => !subflowIds.includes(id)) + filteredIds = [...filteredIds, ...subflowIds] + } + + const result = canFormValidGroup(filteredIds, flowNodes, this.getExcludeIds()) + if (!result.valid) return undefined + + const groups = this.getGroups() + + // Reject duplicate: a group with the same boundaries already exists + if (groups.some((g) => g.start_id === result.startId && g.end_id === result.endId)) { + return undefined + } + const usedColors = new Set() + for (const group of groups) { + if (group.color) { + usedColors.add(group.color as NoteColor) + } + } + const color = usedColors.size > 0 ? getNextAvailableColor(usedColors) : DEFAULT_GROUP_NOTE_COLOR + + const newGroup: FlowGroup = { + start_id: result.startId, + end_id: result.endId, + color + } + this.setGroups([...groups, newGroup]) + return groupKey(newGroup) + } + + deleteGroup(groupId: string): void { + const groups = this.getGroups() + this.setGroups(groups.filter((g) => groupKey(g) !== groupId)) + } + + updateColor(groupId: string, color: NoteColor): void { + const groups = this.getGroups() + this.setGroups(groups.map((g) => (groupKey(g) === groupId ? { ...g, color } : g))) + } + + updateSummary(groupId: string, summary: string): void { + const groups = this.getGroups() + this.setGroups(groups.map((g) => (groupKey(g) === groupId ? { ...g, summary } : g))) + } + + updateNote(groupId: string, note: string | undefined): void { + const groups = this.getGroups() + this.setGroups(groups.map((g) => (groupKey(g) === groupId ? { ...g, note } : g))) + } + + /** Add a note to a group (sets note to empty string to trigger the placeholder UI) */ + addNote(groupId: string): void { + this.updateNote(groupId, '') + } + + /** Remove a note from a group */ + removeNote(groupId: string): void { + this.updateNote(groupId, undefined) + } + + updateAutocollapse(groupId: string, autocollapse: boolean): void { + const groups = this.getGroups() + this.setGroups(groups.map((g) => (groupKey(g) === groupId ? { ...g, autocollapse } : g))) + } +} + +export type GroupEditorContext = { + groupEditor: GroupEditor + canCreateGroup: StateStore +} + +const CONTEXT_KEY = 'GroupEditorContext' + +export function setGroupEditorContext( + groupEditor: GroupEditor, + canCreateGroup: StateStore +): void { + setContext(CONTEXT_KEY, { groupEditor, canCreateGroup }) +} + +export function getGroupEditorContext(): GroupEditorContext | undefined { + return getContext(CONTEXT_KEY) +} + +/** Height of the group header bar */ +export const GROUP_HEADER_HEIGHT = 22 + +/** Extra margin between the header and the first node */ +export const GROUP_TOP_MARGIN = 30 + +export type GraphGroup = FlowGroup & { + id: string + moduleIds: string[] +} + +export type ContainerInnerArray = { + get: () => FlowModule[] + set: (v: any) => void + label?: string +} + +/** Get inner arrays from a container FlowModule with direct get/set accessors. */ +export function getContainerInnerArrays(mod: FlowModule): ContainerInnerArray[] { + const val = mod.value as any + if (val.type === 'forloopflow' || val.type === 'whileloopflow') { + return [ + { + get: () => val.modules, + set: (v) => { + val.modules = v + } + } + ] + } else if (val.type === 'branchone') { + return [ + { + get: () => val.default, + set: (v) => { + val.default = v + }, + label: 'Default' + }, + ...val.branches.map((b: any, i: number) => ({ + get: () => b.modules, + set: (v: any) => { + b.modules = v + }, + label: b.summary || `Branch ${i + 1}` + })) + ] + } else if (val.type === 'branchall') { + return val.branches.map((b: any, i: number) => ({ + get: () => b.modules, + set: (v: any) => { + b.modules = v + }, + label: b.summary || `Branch ${i + 1}` + })) + } + return [] +} diff --git a/frontend/src/lib/components/graph/groupedModulesProxy.svelte.ts b/frontend/src/lib/components/graph/groupedModulesProxy.svelte.ts new file mode 100644 index 0000000000..6965baf531 --- /dev/null +++ b/frontend/src/lib/components/graph/groupedModulesProxy.svelte.ts @@ -0,0 +1,181 @@ +import { untrack } from 'svelte' +import type { FlowModule } from '$lib/gen' +import { type FlowGroup, type GraphGroup, groupKey } from './groupEditor.svelte' +import type { StateStore } from '$lib/utils' +import { getAllModules } from '../flows/flowExplorer' +import { computeGroupModuleIds } from './groupDetectionUtils' +import { stateSnapshot } from '$lib/svelte5Utils.svelte' +import { + buildStructureTree, + deriveGroupsFromStructure, + applyStructureToModules, + removeEmptyGroups, + findDuplicateGroups, + removeDuplicateGroups, + flattenStructureIds, + type FlowStructureNode +} from './flowStructure' + +export type ExtendedOpenFlow = { + value: { + modules: FlowModule[] + groups?: FlowGroup[] + [key: string]: any + } + [key: string]: any +} + +/** + * Reactive read-only view of the flow structure tree. + * The tree is always derived from flowStore (single source of truth). + * Mutations go through prepareMutation: snapshot → mutate → clean empty groups → commit. + */ +export class GroupedModulesProxy { + #items = $state([]) + #error = $state(undefined) + #flowStore: StateStore + + constructor(flowStore: StateStore) { + this.#flowStore = flowStore + this.rebuild() + + // Rebuild tree whenever store changes (undo/load/mutation) + $effect(() => { + void flowStore.val.value.modules + void flowStore.val.value.groups + untrack(() => this.rebuild()) + }) + } + + /** Reactive access to the structure tree (read-only view) */ + get items(): FlowStructureNode[] { + return this.#items + } + + /** Reactive access to build errors */ + get error(): unknown { + return this.#error + } + + /** + * Prepare a structural mutation without writing to the store yet. + * Returns the list of groups that became empty (already removed from the snapshot) + * and a `commit` function that writes the result to the store. + * + * If no groups were emptied, the caller can commit immediately. + * If groups were emptied, the caller should show a confirmation modal + * and call commit() only on user confirmation. + */ + prepareMutation( + mutate: (tree: FlowStructureNode[]) => void, + opts?: { + extraModules?: FlowModule[] + displayState?: import('./groupEditor.svelte').GroupDisplayState + } + ): { + emptiedGroups: FlowGroup[] + duplicateGroups: FlowGroup[] + commit: (commitOpts?: { removeDuplicates?: boolean }) => void + } { + const snapshot = $state.snapshot(this.#items) as FlowStructureNode[] + mutate(snapshot) + + // Clean up empty groups and collect which ones were removed + const emptiedGroups = removeEmptyGroups(snapshot) + // Detect groups that became duplicates after the mutation + const duplicateGroups = findDuplicateGroups(snapshot) + + const commit = (commitOpts?: { removeDuplicates?: boolean }) => { + if (commitOpts?.removeDuplicates && duplicateGroups.length > 0) { + removeDuplicateGroups(snapshot) + } + + // Remap runtime state for groups whose boundaries shifted + if (opts?.displayState) { + this.#remapChangedGroupKeys(snapshot, opts.displayState) + } + + // Build moduleMap lazily at commit time so it reflects the latest store state + const moduleMap = new Map() + for (const m of getAllModules(this.#flowStore.val.value.modules)) { + moduleMap.set(m.id, m) + } + if (opts?.extraModules) { + for (const m of opts.extraModules) { + moduleMap.set(m.id, m) + } + } + this.#flowStore.val.value.modules = applyStructureToModules(snapshot, moduleMap) + this.#flowStore.val.value.groups = deriveGroupsFromStructure(snapshot) + } + + return { emptiedGroups, duplicateGroups, commit } + } + + /** + * Convenience: prepare + auto-commit. Only use for mutations that cannot + * empty groups (e.g. inserts). Throws if groups are unexpectedly emptied. + * For mutations that may empty groups, use prepareMutation() directly. + */ + applyTreeMutation( + mutate: (tree: FlowStructureNode[]) => void, + opts?: { + extraModules?: FlowModule[] + displayState?: import('./groupEditor.svelte').GroupDisplayState + } + ): void { + const { emptiedGroups, duplicateGroups, commit } = this.prepareMutation(mutate, opts) + if (emptiedGroups.length > 0) { + console.error('applyTreeMutation: unexpected empty groups', emptiedGroups) + } + if (duplicateGroups.length > 0) { + console.error('applyTreeMutation: unexpected duplicate groups', duplicateGroups) + } + commit() + } + + /** Remap runtime state for group nodes whose boundaries shifted after a mutation. */ + #remapChangedGroupKeys( + snapshot: FlowStructureNode[], + displayState: import('./groupEditor.svelte').GroupDisplayState + ): void { + const walk = (nodes: FlowStructureNode[]) => { + for (const node of nodes) { + if (node.kind === 'group') { + const oldKey = node.id + const flatIds = flattenStructureIds(node.branches[0].children) + const newKey = flatIds.length > 0 ? `${flatIds[0]}:${flatIds[flatIds.length - 1]}` : null + if (newKey && oldKey !== newKey) { + displayState.remapGroupKey(oldKey, newKey) + } + walk(node.branches[0].children) + } else { + for (const branch of node.branches) { + walk(branch.children) + } + } + } + } + walk(snapshot) + } + + /** Rebuild from flowStore */ + private rebuild(): void { + const modules = stateSnapshot(this.#flowStore.val.value.modules) as FlowModule[] + const allGroups = this.#flowStore.val.value.groups ?? [] + const allModules = getAllModules(modules) + const graphGroups: GraphGroup[] = allGroups.map((g) => ({ + ...g, + id: groupKey(g), + moduleIds: computeGroupModuleIds(g.start_id, g.end_id, allModules) + })) + try { + this.#items = buildStructureTree(modules, graphGroups) + this.#error = undefined + } catch (e) { + // Intentionally preserve last-known-good #items so the graph + // can still render while the error is surfaced to the user. + this.#error = e + } + } +} diff --git a/frontend/src/lib/components/graph/moveManager.svelte.ts b/frontend/src/lib/components/graph/moveManager.svelte.ts index 605c4b849a..1539cc0802 100644 --- a/frontend/src/lib/components/graph/moveManager.svelte.ts +++ b/frontend/src/lib/components/graph/moveManager.svelte.ts @@ -204,9 +204,6 @@ export class MoveManager { for (const [edgeId, zone] of this.#registeredDropZones) { if (zone.disableMoveIds.includes(draggedId)) continue - // Skip edges adjacent to the dragged node (no-op move) - if (zone.sourceId === draggedId || zone.targetId === draggedId) continue - const dx = Math.abs(flowPos.x - zone.centerX) const dy = Math.abs(flowPos.y - zone.centerY) diff --git a/frontend/src/lib/components/graph/nodeExtraSpace.ts b/frontend/src/lib/components/graph/nodeExtraSpace.ts new file mode 100644 index 0000000000..f51309a60b --- /dev/null +++ b/frontend/src/lib/components/graph/nodeExtraSpace.ts @@ -0,0 +1,153 @@ +import type { FlowNote } from '../../gen' +import type { AssetWithAltAccessType } from '../assets/lib' +import { + assetDisplaysAsInputInFlowGraph, + assetDisplaysAsOutputInFlowGraph, + NODE_WITH_READ_ASSET_Y_OFFSET, + NODE_WITH_WRITE_ASSET_Y_OFFSET +} from './renderers/nodes/AssetNode.svelte' +import { + AI_TOOL_BASE_OFFSET, + AI_TOOL_ROW_OFFSET, + BELOW_ADDITIONAL_OFFSET +} from './renderers/nodes/AIToolNode.svelte' +import { topologicalSort } from './graphBuilder.svelte' +import { GROUP_HEADER_HEIGHT } from './groupEditor.svelte' +import type { GroupDisplayState } from './groupEditor.svelte' +import type { GraphModuleState } from '.' + +type NodeDep = { + id: string + parentIds?: string[] + data?: { assets?: AssetWithAltAccessType[]; module?: any } +} + +type ExtraSpace = { top: number; bottom: number; left: number; right: number } + +const MAX_TOOLS_PER_ROW = 2 + +/** + * Pre-compute extra top/bottom space each node needs for decorations + * (assets, AI tools, group headers, group notes). + */ +export function computeNodeExtraSpace( + graphNodes: NodeDep[], + opts: { + showAssets: boolean + showNotes: boolean + notes: FlowNote[] | undefined + noteTextHeights: Record + groupDisplayState: GroupDisplayState + insertable: boolean + flowModuleStates: Record | undefined + } +): Map | undefined { + const extraSpace = new Map() + + // 1. Assets + if (opts.showAssets) { + for (const node of graphNodes) { + const assets = node.data?.assets ?? [] + if (!assets.length) continue + const hasRead = assets.some(assetDisplaysAsInputInFlowGraph) + const hasWrite = assets.some(assetDisplaysAsOutputInFlowGraph) + if (hasRead || hasWrite) { + const prev = extraSpace.get(node.id) ?? { top: 0, bottom: 0, left: 0, right: 0 } + extraSpace.set(node.id, { + ...prev, + top: prev.top + (hasRead ? NODE_WITH_READ_ASSET_Y_OFFSET : 0), + bottom: prev.bottom + (hasWrite ? NODE_WITH_WRITE_ASSET_Y_OFFSET : 0) + }) + } + } + } + + // 2. AI tools + for (const node of graphNodes) { + const mod = node.data?.module + if (!mod || mod.value?.type !== 'aiagent') continue + + const agentActions = !opts.insertable && opts.flowModuleStates?.[node.id]?.agent_actions + + if (agentActions) { + // Execution mode: tools below + const totalRows = Math.ceil(agentActions.length / MAX_TOOLS_PER_ROW) + const space = AI_TOOL_BASE_OFFSET + AI_TOOL_ROW_OFFSET * totalRows + BELOW_ADDITIONAL_OFFSET + const prev = extraSpace.get(node.id) ?? { top: 0, bottom: 0, left: 0, right: 0 } + extraSpace.set(node.id, { ...prev, bottom: prev.bottom + space }) + } else { + // Edit mode: tools above + const tools = mod.value.tools ?? [] + const totalRows = Math.ceil(tools.length / MAX_TOOLS_PER_ROW) + (opts.insertable ? 1 : 0) + const space = AI_TOOL_BASE_OFFSET + AI_TOOL_ROW_OFFSET * totalRows + const prev = extraSpace.get(node.id) ?? { top: 0, bottom: 0, left: 0, right: 0 } + extraSpace.set(node.id, { ...prev, top: prev.top + space }) + } + } + + // Topological sort (reversed: top-of-graph first) — shared by group notes and group headers + const sortedNodes = topologicalSort(graphNodes).reverse() + + // 3. Group notes (text above topmost node in each group note) + if (opts.showNotes) { + const groupNotes = (opts.notes ?? []).filter((n) => n.type === 'group') + if (groupNotes.length > 0) { + for (const groupNote of groupNotes) { + if (!groupNote.contained_node_ids?.length) continue + const topmostNodeId = sortedNodes.find((node) => + groupNote.contained_node_ids?.includes(node.id) + )?.id + if (topmostNodeId) { + const textHeight = opts.noteTextHeights[groupNote.id] || 60 + const spacing = textHeight + 16 // padding + const prev = extraSpace.get(topmostNodeId) ?? { + top: 0, + bottom: 0, + left: 0, + right: 0 + } + extraSpace.set(topmostNodeId, { + ...prev, + top: Math.max(prev.top, spacing + prev.top) + }) + } + } + } + } + + // 4. Collapsed group nodes are taller than regular nodes (header + module icons) + for (const node of graphNodes) { + if (node.id.startsWith('collapsed-group:')) { + const prev = extraSpace.get(node.id) ?? { top: 0, bottom: 0, left: 0, right: 0 } + extraSpace.set(node.id, { + ...prev, + bottom: prev.bottom + GROUP_HEADER_HEIGHT + }) + } + } + + // 5. Group nodes (expanded heads and collapsed) with notes need extra height + if (opts.showNotes) { + const noteHeights = opts.groupDisplayState.getNoteHeights() + for (const node of graphNodes) { + let groupId: string | undefined + if (node.id.startsWith('group:') && !node.id.endsWith('-end')) { + groupId = node.id.slice('group:'.length) + } else if (node.id.startsWith('collapsed-group:')) { + groupId = node.id.slice('collapsed-group:'.length) + } + if (groupId) { + const noteHeight = noteHeights[groupId] + if (noteHeight && noteHeight > 0) { + const prev = extraSpace.get(node.id) ?? { top: 0, bottom: 0, left: 0, right: 0 } + extraSpace.set(node.id, { + ...prev, + bottom: prev.bottom + noteHeight + }) + } + } + } + } + + return extraSpace.size > 0 ? extraSpace : undefined +} diff --git a/frontend/src/lib/components/graph/noteColors.ts b/frontend/src/lib/components/graph/noteColors.ts index f9024adf7e..2a82ed1f40 100644 --- a/frontend/src/lib/components/graph/noteColors.ts +++ b/frontend/src/lib/components/graph/noteColors.ts @@ -14,6 +14,7 @@ export enum NoteColor { export interface NoteColorConfig { background: string + backgroundLight: string outline: string outlineHover: string text: string @@ -24,70 +25,80 @@ export interface NoteColorConfig { export const NOTE_COLORS: Record = { [NoteColor.YELLOW]: { background: 'bg-yellow-200 dark:bg-yellow-900', - outline: 'outline-yellow-300 dark:outline-yellow-600', + backgroundLight: 'bg-yellow-400/5 dark:bg-yellow-600/5', + outline: 'outline-yellow-200 dark:outline-yellow-900', outlineHover: 'outline-yellow-300/60 dark:outline-yellow-600/60', text: 'text-yellow-900 dark:text-yellow-100', hover: 'hover:bg-yellow-200 dark:hover:bg-yellow-800' }, [NoteColor.BLUE]: { background: 'bg-blue-100 dark:bg-blue-950', - outline: 'outline-blue-300 dark:outline-blue-600', + backgroundLight: 'bg-blue-400/5 dark:bg-blue-600/5', + outline: 'outline-blue-100 dark:outline-blue-950', outlineHover: 'outline-blue-300/60 dark:outline-blue-600/60', text: 'text-blue-900 dark:text-blue-100', hover: 'hover:bg-blue-200 dark:hover:bg-blue-800' }, [NoteColor.GREEN]: { background: 'bg-green-200 dark:bg-green-900', - outline: 'outline-green-300 dark:outline-green-600', + backgroundLight: 'bg-green-400/5 dark:bg-green-600/5', + outline: 'outline-green-200 dark:outline-green-900', outlineHover: 'outline-green-300/60 dark:outline-green-600/60', text: 'text-green-900 dark:text-green-100', hover: 'hover:bg-green-200 dark:hover:bg-green-800' }, [NoteColor.PURPLE]: { background: 'bg-purple-200 dark:bg-purple-900', - outline: 'outline-purple-300 dark:outline-purple-600', + backgroundLight: 'bg-purple-400/5 dark:bg-purple-600/5', + outline: 'outline-purple-200 dark:outline-purple-900', outlineHover: 'outline-purple-300/60 dark:outline-purple-600/60', text: 'text-purple-900 dark:text-purple-100', hover: 'hover:bg-purple-200 dark:hover:bg-purple-800' }, [NoteColor.PINK]: { background: 'bg-pink-200 dark:bg-pink-900', - outline: 'outline-pink-300 dark:outline-pink-600', + backgroundLight: 'bg-pink-400/5 dark:bg-pink-600/5', + outline: 'outline-pink-200 dark:outline-pink-900', outlineHover: 'outline-pink-300/60 dark:outline-pink-600/60', text: 'text-pink-900 dark:text-pink-100', hover: 'hover:bg-pink-200 dark:hover:bg-pink-800' }, [NoteColor.ORANGE]: { background: 'bg-orange-200 dark:bg-orange-900', - outline: 'outline-orange-300 dark:outline-orange-600', + backgroundLight: 'bg-orange-400/5 dark:bg-orange-600/5', + outline: 'outline-orange-200 dark:outline-orange-900', outlineHover: 'outline-orange-300/60 dark:outline-orange-600/60', text: 'text-orange-900 dark:text-orange-100', hover: 'hover:bg-orange-200 dark:hover:bg-orange-800' }, [NoteColor.RED]: { background: 'bg-red-200 dark:bg-red-900', - outline: 'outline-red-300 dark:outline-red-600', + backgroundLight: 'bg-red-400/5 dark:bg-red-600/5', + outline: 'outline-red-200 dark:outline-red-900', outlineHover: 'outline-red-300/60 dark:outline-red-600/60', text: 'text-red-900 dark:text-red-100', hover: 'hover:bg-red-200 dark:hover:bg-red-800' }, [NoteColor.CYAN]: { background: 'bg-cyan-200 dark:bg-cyan-900', - outline: 'outline-cyan-300 dark:outline-cyan-600', + backgroundLight: 'bg-cyan-400/5 dark:bg-cyan-600/5', + outline: 'outline-cyan-200 dark:outline-cyan-900', outlineHover: 'outline-cyan-300/60 dark:outline-cyan-600/60', text: 'text-cyan-900 dark:text-cyan-100', hover: 'hover:bg-cyan-200 dark:hover:bg-cyan-800' }, [NoteColor.LIME]: { background: 'bg-lime-200 dark:bg-lime-900', - outline: 'outline-lime-300 dark:outline-lime-600', + backgroundLight: 'bg-lime-400/5 dark:bg-lime-600/5', + outline: 'outline-lime-200 dark:outline-lime-900', outlineHover: 'outline-lime-300/60 dark:outline-lime-600/60', text: 'text-lime-900 dark:text-lime-100', hover: 'hover:bg-lime-200 dark:hover:bg-lime-800' }, [NoteColor.GRAY]: { background: 'bg-gray-200 dark:bg-gray-800', - outline: 'outline-gray-300 dark:outline-gray-600', + backgroundLight: 'bg-gray-400/5 dark:bg-gray-600/5', + outline: 'outline-gray-200 dark:outline-gray-800', outlineHover: 'outline-gray-300/60 dark:outline-gray-600/60', text: 'text-gray-900 dark:text-gray-100', hover: 'hover:bg-gray-200 dark:hover:bg-gray-700' diff --git a/frontend/src/lib/components/graph/noteEditor.svelte.ts b/frontend/src/lib/components/graph/noteEditor.svelte.ts index e59e3be4f5..50c2df7bfa 100644 --- a/frontend/src/lib/components/graph/noteEditor.svelte.ts +++ b/frontend/src/lib/components/graph/noteEditor.svelte.ts @@ -219,7 +219,10 @@ export class NoteEditor { /** * Clean up group notes using DAG path completion */ - cleanupGroupNotes(flowNodes: { id: string; parentIds?: string[] }[]): void { + cleanupGroupNotes( + flowNodes: { id: string; parentIds?: string[] }[], + collapsedModuleIds?: Set + ): void { if (!this.isAvailable()) { return } @@ -231,6 +234,13 @@ export class NoteEditor { let hasChanges = false const nodeSet = new Set(flowNodes.map((n) => n.id)) + // Include collapsed module IDs as valid — they are hidden but still exist + if (collapsedModuleIds) { + for (const id of collapsedModuleIds) { + nodeSet.add(id) + } + } + // Step 1: Clean invalid nodes from existing group notes for (const note of groupNotes) { const originalIds = note.contained_node_ids || [] @@ -249,6 +259,12 @@ export class NoteEditor { const originalNodes = note.contained_node_ids || [] if (originalNodes.length === 0) continue + // Skip path completion for notes that reference collapsed modules, + // since the DAG is incomplete when groups are collapsed + if (collapsedModuleIds && originalNodes.some((id) => collapsedModuleIds.has(id))) { + continue + } + // Use the DAG path completion and splitting algorithm const completedGroups = completeAndSplitGroup(originalNodes, flowNodes) diff --git a/frontend/src/lib/components/graph/noteUtils.svelte.ts b/frontend/src/lib/components/graph/noteUtils.svelte.ts index 3350e07ce8..577b421a9e 100644 --- a/frontend/src/lib/components/graph/noteUtils.svelte.ts +++ b/frontend/src/lib/components/graph/noteUtils.svelte.ts @@ -19,20 +19,6 @@ export type NodeDep = { export type NoteComputeResult = { noteNodes: (Node & NodeLayout)[] - newNodePositions: Record -} - -export type AIToolSpacingInfo = { - toolNodes: (Node & NodeLayout)[] - toolEdges: any[] - newNodePositions: Record -} - -export interface GroupNoteBounds { - x: number - y: number - width: number - height: number } let computeNoteNodesCache: @@ -263,7 +249,8 @@ export function computeNoteNodes( noteTextHeights: Record, onTextHeightChange: (noteId: string, height: number) => void, editMode: boolean = false, - noteEditorContext: NoteEditorContext | undefined + noteEditorContext: NoteEditorContext | undefined, + collapsedModuleIds?: Set ): NoteComputeResult { // Check cache first if ( @@ -277,20 +264,15 @@ export function computeNoteNodes( if (editMode) { if (noteEditorContext?.noteEditor?.isAvailable()) { - noteEditorContext.noteEditor.cleanupGroupNotes(nodes) + noteEditorContext.noteEditor.cleanupGroupNotes(nodes, collapsedModuleIds) } } const allNoteNodes: (Node & NodeLayout)[] = [] - // Build a map of Y positions that need extra spacing for group notes - const yPosMap: Record = {} // Y position -> spacing needed - - // Group notes that need spacing + // Find topmost node per group note for layout calculation const groupNotes = notes.filter((n) => n.type === 'group') - const topMostNodesMap: Record = {} - const sortedNodes = topologicalSort(nodes).reverse() for (const groupNote of groupNotes) { @@ -298,59 +280,33 @@ export function computeNoteNodes( const topmostNodeId = sortedNodes.find((node) => groupNote.contained_node_ids?.includes(node.id) )?.id - const topmostNode = nodes.find((node) => node.id === topmostNodeId) - if (topmostNode) { - const textHeight = noteTextHeights[groupNote.id] || 60 - const spacing = textHeight + 16 // padding - // Mark this Y position as needing spacing - yPosMap[topmostNode.position.y] = Math.max(yPosMap[topmostNode.position.y] || 0, spacing) - topMostNodesMap[groupNote.id] = topmostNode.id + if (topmostNodeId) { + topMostNodesMap[groupNote.id] = topmostNodeId } } } - // Calculate new positions for nodes (offset by group notes) - const sortedNewNodes = nodes - .map((n) => ({ position: { ...n.position }, id: n.id })) - .sort((a, b) => a.position.y - b.position.y) - - let currentYOffset = 0 - let prevYPos = NaN - - for (const node of sortedNewNodes) { - if (node.position.y !== prevYPos) { - // Add spacing for group notes at this Y level - if (yPosMap[node.position.y]) { - currentYOffset += yPosMap[node.position.y] - } - prevYPos = node.position.y - } - node.position.y += currentYOffset - } - - // Create note nodes AFTER calculating adjusted node positions - // For group notes, we need to use the adjusted node positions - const adjustedNodes = sortedNewNodes.map((n) => { - const origNode = nodes.find((orig) => orig.id === n.id) - return { - ...n, - data: origNode?.data, - type: origNode?.type - } - }) - // Calculate all z-indexes at once using hierarchy information const noteZIndexes = calculateAllNoteZIndexes(notes, nodes) for (const note of notes) { const isGroupNote = note.type === 'group' + + // Skip group notes whose contained nodes are all inside collapsed groups + if (isGroupNote && collapsedModuleIds?.size) { + const ids = note.contained_node_ids ?? [] + if (ids.length > 0 && ids.every((id) => collapsedModuleIds.has(id))) { + continue + } + } + const zIndex = noteZIndexes[note.id] - // Calculate position and size using adjusted node positions for group notes + // Calculate position and size using node positions for group notes const { position, size } = isGroupNote ? calculateGroupNoteLayout( note, - adjustedNodes, + nodes, noteTextHeights[note.id] || 60, topMostNodesMap[note.id] ) @@ -375,13 +331,8 @@ export function computeNoteNodes( allNoteNodes.push(noteNode) } - const newNodePositions: Record = Object.fromEntries( - sortedNewNodes.map((n) => [n.id, n.position]) - ) - const result: NoteComputeResult = { - noteNodes: allNoteNodes, - newNodePositions + noteNodes: allNoteNodes } // Cache the result diff --git a/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte b/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte index 8f1ca0017f..5fbf084374 100644 --- a/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte +++ b/frontend/src/lib/components/graph/renderers/edges/BaseEdge.svelte @@ -12,11 +12,14 @@ import type { GraphModuleState } from '../../model' import InsertModuleButton from '$lib/components/flows/map/InsertModuleButton.svelte' import { getGraphContext } from '../../graphContext' + import { GROUP_TOP_PADDING } from '$lib/components/graph/compoundLayout' const { useDataflow, showAssets, moveManager } = getGraphContext() let { id, + source, + target, sourceX, sourceY, sourcePosition, @@ -45,6 +48,13 @@ } } = $props() + // Derive group boundary from source/target node IDs + let groupBoundary: 'top' | 'bottom' | undefined = $derived.by(() => { + if (source.startsWith('group:') && !source.endsWith('-end')) return 'top' + if (target.startsWith('group:') && target.endsWith('-end')) return 'bottom' + return undefined + }) + let [edgePath] = $derived( getBezierPath({ sourceX, @@ -75,9 +85,15 @@ ) let centerY = $derived( - sourceY + - 32 + - (data.shouldOffsetInsertBtnDueToAssetNode && $showAssets ? NODE_WITH_WRITE_ASSET_Y_OFFSET : 0) + groupBoundary === 'bottom' + ? targetY + : groupBoundary === 'top' + ? sourceY + GROUP_TOP_PADDING / 2 + : sourceY + + 32 + + (data.shouldOffsetInsertBtnDueToAssetNode && $showAssets + ? NODE_WITH_WRITE_ASSET_Y_OFFSET + : 0) ) let isDragging = $derived(!!moveManager?.dragging) @@ -87,13 +103,13 @@ data?.insertable && draggedId !== undefined && !data.disableMoveIds?.includes(draggedId) && - data.sourceId !== draggedId && - data.targetId !== draggedId + source !== draggedId && + target !== draggedId ) - let isNearestDrop = $derived(isValidDropTarget && moveManager?.nearestDropZone?.edgeId === id) - let isAdjacentToDragged = $derived( - isDragging && (data?.sourceId === draggedId || data?.targetId === draggedId) + let isNearestDrop = $derived( + isValidDropTarget && moveManager?.nearestDropZone?.edgeId === id ? true : false ) + let isAdjacentToDragged = $derived(isDragging && (source === draggedId || target === draggedId)) // Register this edge's drop zone position with the drag manager so proximity // detection uses the actual xyflow-computed position rather than re-deriving it. @@ -161,7 +177,7 @@ {@render dropTargetIndicator(isNearestDrop)}
- {:else if data?.insertable && !$useDataflow && !moveManager?.movingModuleId && !isDragging} + {:else if data?.insertable && !groupBoundary && !$useDataflow && !moveManager?.movingModuleId && !isDragging}
- {#if !(moveManager.movingIds ?? [moveManager.movingModuleId]).some((id) => data.disableMoveIds?.includes(id))} + {#if !(moveManager.movingIds ?? [moveManager.movingModuleId]).some( (id) => data.disableMoveIds?.includes(id) )} - {/if} -
+
+ {/if} {/snippet} diff --git a/frontend/src/lib/components/graph/renderers/nodes/WacControlNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/WacControlNode.svelte new file mode 100644 index 0000000000..0da9269d7d --- /dev/null +++ b/frontend/src/lib/components/graph/renderers/nodes/WacControlNode.svelte @@ -0,0 +1,57 @@ + + +
+
+
+
{displayLabel}
+
+
+
+ + + diff --git a/frontend/src/lib/components/graph/renderers/nodes/WacStepNode.svelte b/frontend/src/lib/components/graph/renderers/nodes/WacStepNode.svelte new file mode 100644 index 0000000000..18e77ac946 --- /dev/null +++ b/frontend/src/lib/components/graph/renderers/nodes/WacStepNode.svelte @@ -0,0 +1,39 @@ + + +
+
+
+
+
{label}
+
+ {#if hasExternalPath} + + {script} + + {:else if isInline} + + inline + + {/if} +
+
+
+ + + diff --git a/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte b/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte index 2ffa6a8d49..ccc5643981 100644 --- a/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte +++ b/frontend/src/lib/components/graph/renderers/triggers/TriggersWrapper.svelte @@ -53,7 +53,7 @@
+ + {#if loading} + This may take a few seconds... + {/if} +
+ + {#if error} +
+ {error} +
+ {/if} + + {#if data} + +
+ + {#if expandedSections.database_size} +
+

+ Total database size: {data.database_size.total_size_pretty} +

+
+ + + + + + + + + {#each data.database_size.top_tables as t} + + + + + {/each} + +
TableSize
{t.table_name}{t.total_size_pretty}
+
+
+ {/if} +
+ + +
+ + {#if expandedSections.job_retention} +
+

+ Total completed jobs: {formatNumber(data.job_retention.total_completed_jobs)} +

+

+ Oldest job: {formatDate(data.job_retention.oldest_completed_at)} +

+

+ Retention period: + {data.job_retention.retention_period_secs + ? formatNumber(data.job_retention.retention_period_secs) + 's' + : 'Not configured'} + +

+

+ {data.job_retention.message} +

+
+ {/if} +
+ + +
+ + {#if expandedSections.large_results} +
+ {#if data.large_results.top_large_results.length === 0} +

No job results larger than 1 KB found in the scanned jobs.

+ {:else} +
+ + + + + + + + + + + + {#each data.large_results.top_large_results as r} + + + + + + + + {/each} + +
Job IDWorkspaceScriptResult SizeCompleted
{r.id.substring(0, 8)}...{r.workspace_id}{r.runnable_path ?? '-'}{formatBytes(r.result_size_bytes)}{formatDate(r.completed_at)}
+
+ {/if} +
+ {/if} +
+ + +
+ + {#if expandedSections.connection_pool} +
+

+ Total connections: {data.connection_pool.pg_total_connections} / Max: + {data.connection_pool.pg_max_connections} +

+

+ Active: {data.connection_pool.pg_active_connections} + / Idle: {data.connection_pool.pg_idle_connections} +

+

+ {data.connection_pool.message} +

+
+ {/if} +
+ + +
+ + {#if expandedSections.table_maintenance} +
+
+ + + + + + + + + + + + + + {#each data.table_maintenance as t} + + + + + + + + + + {/each} + +
TableLive TuplesDead TuplesDead %Last VacuumLast AnalyzeStatus
{t.table_name}{formatNumber(t.live_tuples)}{formatNumber(t.dead_tuples)}{(t.dead_ratio * 100).toFixed(1)}%{formatDate(t.last_autovacuum)}{formatDate(t.last_autoanalyze)} + + {t.status} + +
+
+
+ {/if} +
+ + +
+ + {#if expandedSections.slow_queries} +
+ {#if data.slow_queries == null} +

Slow query data not available.

+ {:else if data.slow_queries.message} +

{data.slow_queries.message}

+ {:else if data.slow_queries.queries.length === 0} +

No slow queries found.

+ {:else} +
+ + + + + + + + + + + {#each data.slow_queries.queries as q} + + + + + + + {/each} + +
QueryCallsTotal TimeMean Time
{q.query}{formatNumber(q.calls)}{formatMs(q.total_exec_time_ms)}{formatMs(q.mean_exec_time_ms)}
+
+ {/if} +
+ {/if} +
+ + +
+ + {#if expandedSections.datatables} +
+ {#if data.datatables.length === 0} +

No instance-stored datatables found.

+ {:else} +
+ + + + + + + + + + + + {#each data.datatables as dt} + + + + + + + + {/each} + +
WorkspaceNameTableSizeEst. Rows
{dt.workspace_id}{dt.name}{dt.table_name}{dt.size_pretty}{formatNumber(Math.round(dt.estimated_rows))}
+
+ {/if} +
+ {/if} +
+ {:else if !loading} +

+ Click "Run Diagnostics" to analyze your database health. The queries are read-only and + lightweight. +

+ {/if} +
diff --git a/frontend/src/lib/components/instanceSettings/GhesAppSettings.svelte b/frontend/src/lib/components/instanceSettings/GhesAppSettings.svelte index 6ded7d82f2..3d6c314812 100644 --- a/frontend/src/lib/components/instanceSettings/GhesAppSettings.svelte +++ b/frontend/src/lib/components/instanceSettings/GhesAppSettings.svelte @@ -22,7 +22,7 @@
{ $values['github_enterprise_app'] = { @@ -45,14 +45,14 @@

1. On your GitHub instance, go to - Settings → Developer settings → GitHub Apps → New GitHub AppSettings → Developer settings → GitHub Apps → New GitHub App.

2. Fill in the required fields:

  • - GitHub App name: e.g. windmill-sync (this becomes the app - slug) + GitHub App name: e.g. windmill-sync (this becomes the app slug)
  • Homepage URL: your Windmill instance URL @@ -80,13 +80,12 @@ App ID and Client ID.

    - 6. Scroll down and click Generate a private key. Save the - downloaded .pem file — paste its contents into the Private Key field below. + 6. Scroll down and click Generate a private key. Save + the downloaded .pem file — paste its contents into the Private Key field below.

    7. The App Slug is the URL-friendly name shown in the - app's URL (e.g. github.com/apps/windmill-sync). + app's URL (e.g. github.com/apps/windmill-sync).

    8. The Base URL is your GitHub instance root (e.g. diff --git a/frontend/src/lib/components/instanceSettings/IndexerMemorySettings.svelte b/frontend/src/lib/components/instanceSettings/IndexerMemorySettings.svelte index 52bc3cd9bc..eb37ac8a0c 100644 --- a/frontend/src/lib/components/instanceSettings/IndexerMemorySettings.svelte +++ b/frontend/src/lib/components/instanceSettings/IndexerMemorySettings.svelte @@ -2,7 +2,7 @@ import { Button } from '$lib/components/common' import ConfirmationModal from '../common/confirmationModal/ConfirmationModal.svelte' import { IndexSearchService } from '$lib/gen' - import type { GetIndexerStatusResponse } from '$lib/gen' + import type { GetIndexerStatusResponse, GetIndexDiskStorageSizesResponse } from '$lib/gen' import { sendUserToast } from '$lib/toast' import { displaySize } from '$lib/utils' import Tooltip from '../Tooltip.svelte' @@ -24,6 +24,7 @@ let clearServiceLogsIndexModalOpen = $state(false) let status: GetIndexerStatusResponse | undefined = $state(undefined) + let diskSizes: GetIndexDiskStorageSizesResponse | undefined = $state(undefined) let statusLoading = $state(true) let statusError = $state(false) @@ -41,9 +42,15 @@ statusLoading = true statusError = false try { - status = await IndexSearchService.getIndexerStatus() + const [statusRes, diskRes] = await Promise.all([ + IndexSearchService.getIndexerStatus(), + IndexSearchService.getIndexDiskStorageSizes().catch(() => undefined) + ]) + status = statusRes + diskSizes = diskRes } catch (e) { status = undefined + diskSizes = undefined statusError = true } finally { statusLoading = false @@ -139,7 +146,11 @@ : 'bg-red-500'}" > {label}: - + {entry?.is_alive ? 'Running' : 'Stopped'} {#if entry?.last_locked_at} @@ -161,21 +172,21 @@

    Jobs index: - {#if status.job_indexer?.storage?.disk_size_bytes != null} - Disk: {displaySize(status.job_indexer.storage.disk_size_bytes) ?? 'N/A'} + {#if diskSizes?.job_index_disk_size_bytes != null} + Disk: {displaySize(diskSizes.job_index_disk_size_bytes) ?? 'N/A'} {/if} {#if status.job_indexer?.storage?.s3_size_bytes != null} - {#if status.job_indexer?.storage?.disk_size_bytes != null}·{/if} + {#if diskSizes?.job_index_disk_size_bytes != null}·{/if} S3: {displaySize(status.job_indexer.storage.s3_size_bytes) ?? 'N/A'} {/if} Service logs index: - {#if status.log_indexer?.storage?.disk_size_bytes != null} - Disk: {displaySize(status.log_indexer.storage.disk_size_bytes) ?? 'N/A'} + {#if diskSizes?.log_index_disk_size_bytes != null} + Disk: {displaySize(diskSizes.log_index_disk_size_bytes) ?? 'N/A'} {/if} {#if status.log_indexer?.storage?.s3_size_bytes != null} - {#if status.log_indexer?.storage?.disk_size_bytes != null}·{/if} + {#if diskSizes?.log_index_disk_size_bytes != null}·{/if} S3: {displaySize(status.log_indexer.storage.s3_size_bytes) ?? 'N/A'} {/if} diff --git a/frontend/src/lib/components/instanceSettings/InstanceAISettings.svelte b/frontend/src/lib/components/instanceSettings/InstanceAISettings.svelte new file mode 100644 index 0000000000..91b08eb46e --- /dev/null +++ b/frontend/src/lib/components/instanceSettings/InstanceAISettings.svelte @@ -0,0 +1,149 @@ + + +{#if loaded} + {#if showHubSync} +
    +
    +

    Resource types

    +

    + AI providers require their resource types. Sync from the Hub if they are missing. +

    +
    + +
    + {#if hubSyncStatus === 'success'} +
    + + {hubSyncMessage} + +
    + {:else if hubSyncStatus === 'error'} +
    + + {hubSyncMessage} + +
    + {/if} + {/if} + + +{/if} diff --git a/frontend/src/lib/components/mcp/McpScopeSelector.svelte b/frontend/src/lib/components/mcp/McpScopeSelector.svelte index 9fe525dd5c..3d019ff539 100644 --- a/frontend/src/lib/components/mcp/McpScopeSelector.svelte +++ b/frontend/src/lib/components/mcp/McpScopeSelector.svelte @@ -5,9 +5,8 @@ import Popover from '$lib/components/Popover.svelte' import MultiSelect from '$lib/components/select/MultiSelect.svelte' import { safeSelectItems } from '$lib/components/select/utils.svelte' - import FolderPicker from '$lib/components/FolderPicker.svelte' import TextInput from '$lib/components/text_input/TextInput.svelte' - import { FlowService, IntegrationService, ScriptService } from '$lib/gen' + import { FlowService, FolderService, IntegrationService, ScriptService } from '$lib/gen' import { mcpEndpointTools } from '$lib/mcpEndpointTools' import InfoIcon from 'lucide-svelte/icons/info' import { SvelteMap } from 'svelte/reactivity' @@ -20,7 +19,10 @@ let { workspaceId, scope = $bindable() }: Props = $props() let selectedMode = $state<'favorites' | 'all' | 'folder' | 'custom'>('favorites') - let selectedFolder = $state('') + let selectedFolders = $state([]) + let allFolders = $state([]) + let loadingFolders = $state(false) + let folderNamesCache = new Map() let selectedScripts = $state([]) let selectedFlows = $state([]) let selectedEndpoints = $state([]) @@ -70,8 +72,10 @@ scopeParts.push(`mcp:endpoints:${selectedEndpoints.join(',')}`) } } else if (selectedMode === 'folder') { - const folderPath = `f/${selectedFolder}/*` - scopeParts = [`mcp:scripts:${folderPath}`, `mcp:flows:${folderPath}`, `mcp:endpoints:*`] + const folderPaths = selectedFolders.map((f) => `f/${f}/*`).join(',') + if (selectedFolders.length > 0) { + scopeParts = [`mcp:scripts:${folderPaths}`, `mcp:flows:${folderPaths}`, `mcp:endpoints:*`] + } } else { scopeParts = [`mcp:${selectedMode}`] } @@ -91,13 +95,35 @@ } }) - // Clear folder when not in folder mode + // Clear folders when not in folder mode, load folder names when entering folder mode $effect(() => { - if (selectedMode !== 'folder') { - selectedFolder = '' + if (selectedMode === 'folder' && workspaceId) { + loadFolderNames(workspaceId) + } else { + selectedFolders = [] } }) + async function loadFolderNames(workspace: string) { + if (folderNamesCache.has(workspace)) { + allFolders = folderNamesCache.get(workspace)! + return + } + try { + loadingFolders = true + const excludedFolders = ['app_groups', 'app_custom', 'app_themes'] + const names = ( + await FolderService.listFolderNames({ workspace }) + ).filter((x) => !excludedFolders.includes(x)) + folderNamesCache.set(workspace, names) + allFolders = names + } catch { + allFolders = [] + } finally { + loadingFolders = false + } + } + // Load hub apps on mount async function getAllApps() { if (allApps.length > 0) return @@ -192,11 +218,42 @@ // Load runnables based on mode $effect(() => { if (workspaceId) { - const folderParam = selectedFolder.length > 0 ? selectedFolder : undefined - getScriptsAndFlows(selectedMode === 'favorites', workspaceId, folderParam) + if (selectedMode === 'folder') { + if (selectedFolders.length > 0) { + loadRunnablesForFolders(workspaceId, selectedFolders) + } else { + includedRunnables = [] + } + } else { + getScriptsAndFlows(selectedMode === 'favorites', workspaceId, undefined) + } } }) + async function getCachedRunnables(workspace: string, folder: string): Promise { + const cacheKey = `${workspace}-false-${folder}` + if (runnablesCache.has(cacheKey)) { + return runnablesCache.get(cacheKey) || [] + } + const [scripts, flows] = await Promise.all([ + getScripts(false, workspace, folder), + getFlows(false, workspace, folder) + ]) + const combined = [...scripts, ...flows] + runnablesCache.set(cacheKey, combined) + return combined + } + + async function loadRunnablesForFolders(workspace: string, folders: string[]) { + try { + loadingRunnables = true + const results = await Promise.all(folders.map((f) => getCachedRunnables(workspace, f))) + includedRunnables = [...new Set(results.flat())] + } finally { + loadingRunnables = false + } + } + // Load all scripts/flows for custom mode $effect(() => { if (selectedMode === 'custom' && workspaceId) { @@ -209,7 +266,7 @@ ? 'Create your first scripts or flows to make them available via MCP.' : selectedMode === 'favorites' ? `You do not have any favorite scripts or flows. You can favorite some scripts and flows to include them, or change the scope to "All scripts/flows" to include all your scripts and flows.` - : `You do not have any scripts or flows in the selected folder.` + : `You do not have any scripts or flows in the selected folder(s).` ) function selectAllScripts() { @@ -252,8 +309,8 @@ - Select Folder - + Select Folders + {#if loadingFolders} +
    Loading folders...
    + {:else} + + {/if}
    {/if} @@ -389,7 +454,7 @@
{/if} - {:else if selectedMode !== 'folder' || selectedFolder.length > 0} + {:else if selectedMode !== 'folder' || selectedFolders.length > 0} {#if loadingRunnables}
fork(e.detail)} on:delete {id} diff --git a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts index b90f1d3985..ec3e089745 100644 --- a/frontend/src/lib/components/raw_apps/rawAppPolicy.ts +++ b/frontend/src/lib/components/raw_apps/rawAppPolicy.ts @@ -1,18 +1,24 @@ import type { Policy, ScriptLang } from '$lib/gen' import { collectStaticFields, hash, type TriggerableV2 } from '../apps/editor/commonAppUtils' -import { isRunnableByName, isRunnableByPath, type InlineScript, type RunnableWithFields } from '../apps/inputType' +import { + isRunnableByName, + isRunnableByPath, + type InlineScript, + type RunnableWithFields +} from '../apps/inputType' export async function updateRawAppPolicy( runnables: Record, currentPolicy: Policy | undefined ): Promise { - const triggerables_v2 = Object.fromEntries( - (await Promise.all( + const entries = ( + await Promise.all( Object.entries(runnables).map(async ([id, runnable]) => { return await processRunnable(id, runnable, runnable?.fields ?? {}) }) - )) as [string, TriggerableV2][] - ) + ) + ).filter((entry): entry is [string, TriggerableV2] => entry != null) + const triggerables_v2 = Object.fromEntries(entries) return { ...currentPolicy, triggerables_v2 diff --git a/frontend/src/lib/components/recording/FlowRecordingReplay.svelte b/frontend/src/lib/components/recording/FlowRecordingReplay.svelte index c5d94a7b95..afd2cee94c 100644 --- a/frontend/src/lib/components/recording/FlowRecordingReplay.svelte +++ b/frontend/src/lib/components/recording/FlowRecordingReplay.svelte @@ -2,7 +2,8 @@ import type { Job } from '$lib/gen' import { workspaceStore } from '$lib/stores' import FlowStatusViewer from '$lib/components/FlowStatusViewer.svelte' - import FlowViewer from '$lib/components/FlowViewer.svelte' + import FlowViewer, { type TabValue } from '$lib/components/FlowViewer.svelte' + import FlowGraphViewer from '$lib/components/FlowGraphViewer.svelte' import FlowProgressBar from '$lib/components/flows/FlowProgressBar.svelte' import FlowExecutionStatus from '$lib/components/runs/FlowExecutionStatus.svelte' import { setActiveReplay } from './flowRecording.svelte' @@ -13,21 +14,37 @@ import { InfoIcon, LogOut, Play, Square } from 'lucide-svelte' import { onDestroy } from 'svelte' - interface Props { - recording: FlowRecording - } - - let { recording }: Props = $props() - type ReplayState = 'loaded' | 'playing' - let replayState: ReplayState = $state('loaded') + interface Props { + recording: FlowRecording + selectedTab?: TabValue + replayState?: ReplayState + hideControls?: boolean + hideTabs?: boolean + } + + let { + recording, + selectedTab = $bindable(), + replayState = $bindable(), + hideControls = false, + hideTabs = false + }: Props = $props() + + if (selectedTab === undefined) { + selectedTab = 'ui' + } + if (replayState === undefined) { + replayState = 'loaded' + } + let rootJobId: string | undefined = $state(undefined) let rootInitialJob: Job | undefined = $state(undefined) let job: Job | undefined = $state(undefined) let done = $derived((job as any)?.type === 'CompletedJob') - function stop() { + export function stop() { setActiveReplay(undefined) job = undefined initRecording() @@ -36,10 +53,7 @@ function findRootJobId(data: FlowRecording): string | undefined { for (const [id, recorded] of Object.entries(data.jobs)) { const j = recorded.initial_job - if ( - (j.job_kind === 'flow' || j.job_kind === 'flowpreview') && - !j.parent_job - ) { + if ((j.job_kind === 'flow' || j.job_kind === 'flowpreview') && !j.parent_job) { return id } } @@ -81,17 +95,19 @@ for (const mod of fs.modules) { const durations = mod.flow_jobs_duration if (durations?.started_at) { - durations.started_at = durations.started_at.map( - (d: string) => offsetDate(d) ?? d - ) + durations.started_at = durations.started_at.map((d: string) => offsetDate(d) ?? d) } } } for (const recorded of Object.values(data.jobs)) { offsetJobTimestamps(recorded.initial_job) + if (recorded.initial_job?.flow_status) offsetFlowStatus(recorded.initial_job.flow_status) for (const event of recorded.events) { - if (event.data?.job) offsetJobTimestamps(event.data.job) + if (event.data?.job) { + offsetJobTimestamps(event.data.job) + if (event.data.job.flow_status) offsetFlowStatus(event.data.job.flow_status) + } if (event.data?.flow_status) offsetFlowStatus(event.data.flow_status) } } @@ -141,22 +157,27 @@ // Push the root's completed event to fire after all sub-job events let completedIdx = -1 for (let i = rootEvents.length - 1; i >= 0; i--) { - if (rootEvents[i].data.completed) { completedIdx = i; break } + if (rootEvents[i].data.completed) { + completedIdx = i + break + } } if (completedIdx >= 0 && rootEvents[completedIdx].t < maxSubJobT) { rootEvents[completedIdx].t = maxSubJobT + 50 } } - function startReplay() { + export function startReplay() { + if (!rootJobId) return // JSON round-trip to unwrap reactive proxies and strip non-cloneable properties const snapshot = JSON.parse(JSON.stringify(recording)) as FlowRecording - fixEventOrdering(snapshot, rootJobId!) - rebaseTimestamps(snapshot, rootJobId!) + fixEventOrdering(snapshot, rootJobId) + rebaseTimestamps(snapshot, rootJobId) setActiveReplay(snapshot) - rootInitialJob = buildInitialJob(snapshot, rootJobId!) + rootInitialJob = buildInitialJob(snapshot, rootJobId) job = undefined replayState = 'playing' + selectedTab = 'ui' } onDestroy(() => { @@ -173,52 +194,81 @@

-{:else if replayState === 'loaded'} +{:else}
-
-
-

{recording.flow_path}

- - - {#snippet text()} - - Recorded {new Date(recording.recorded_at).toLocaleString()} — - {(recording.total_duration_ms / 1000).toFixed(1)}s - - {/snippet} - + {#if !hideControls} +
+
+

+ {replayState === 'playing' ? 'Replaying: ' : ''}{recording.flow_path} +

+ + + {#snippet text()} + + Recorded {new Date(recording.recorded_at).toLocaleString()} — + {(recording.total_duration_ms / 1000).toFixed(1)}s + + {/snippet} + +
+ {#if replayState === 'loaded'} + + {:else} + + {/if}
- -
- -
-{:else if replayState === 'playing' && rootJobId} -
-
-

Replaying: {recording.flow_path}

- -
- - {#if job} - {/if} - + + + {#snippet graphContent()} + {#if replayState === 'playing' && rootJobId} +
+ + {#if job} + + {/if} + +
+ {:else} +
+

Click on a step to see its details

+ +
+ {/if} + {/snippet} +
{/if} diff --git a/frontend/src/lib/components/recording/ScriptRecordingReplay.svelte b/frontend/src/lib/components/recording/ScriptRecordingReplay.svelte index 8c3d28a71a..0327044b3b 100644 --- a/frontend/src/lib/components/recording/ScriptRecordingReplay.svelte +++ b/frontend/src/lib/components/recording/ScriptRecordingReplay.svelte @@ -12,6 +12,7 @@ import { json as jsonLang } from 'svelte-highlight/languages' import HighlightTheme from '$lib/components/HighlightTheme.svelte' import JobArgs from '$lib/components/JobArgs.svelte' + import SchemaForm from '$lib/components/SchemaForm.svelte' import DisplayResult from '$lib/components/DisplayResult.svelte' import LogViewer from '$lib/components/LogViewer.svelte' import { ClipboardCopy, InfoIcon, LogOut, Play, Square } from 'lucide-svelte' @@ -19,15 +20,26 @@ import { onDestroy, tick } from 'svelte' import JobLoader from '$lib/components/JobLoader.svelte' - interface Props { - recording: ScriptRecording - } - - let { recording }: Props = $props() + export type ScriptTabValue = 'parameters' | 'code' | 'args' | 'schema' | 'result' type ReplayState = 'loaded' | 'playing' - let replayState: ReplayState = $state('loaded') + interface Props { + recording: ScriptRecording + selectedTab?: ScriptTabValue + replayState?: ReplayState + hideControls?: boolean + hideTabs?: boolean + } + + let { + recording, + selectedTab = $bindable(), + replayState = $bindable(), + hideControls = false, + hideTabs = false + }: Props = $props() + let jobId: string | undefined = $state(undefined) let job: Job | undefined = $state(undefined) let jobLoader: JobLoader | undefined = $state(undefined) @@ -35,7 +47,18 @@ let scriptRecordingStore = createScriptRecording() - function stop() { + let schema = $derived(recording.schema) + + if (selectedTab === undefined) { + if (schema && recording.args) selectedTab = 'parameters' + else if (recording.args && Object.keys(recording.args).length > 0) selectedTab = 'args' + else selectedTab = 'code' + } + if (replayState === undefined) { + replayState = 'loaded' + } + + export function stop() { setActiveReplay(undefined) job = undefined replayState = 'loaded' @@ -85,13 +108,14 @@ initRecording() - async function startReplay() { + export async function startReplay() { const snapshot = JSON.parse(JSON.stringify(recording)) as ScriptRecording rebaseTimestamps(snapshot) const replayData = scriptRecordingStore.toReplayData(snapshot) setActiveReplay(replayData) job = undefined replayState = 'playing' + selectedTab = 'result' await tick() if (jobLoader && jobId) { jobLoader.watchJob(jobId) @@ -101,8 +125,6 @@ onDestroy(() => { setActiveReplay(undefined) }) - - let schema = $derived(recording.schema) @@ -115,48 +137,141 @@

-{:else if replayState === 'loaded'} -
-
-
-

{recording.script_path || 'Untitled script'}

- {recording.language} - - - {#snippet text()} - - Recorded {new Date(recording.recorded_at).toLocaleString()} — - {(recording.total_duration_ms / 1000).toFixed(1)}s - - {/snippet} - +{:else} +
+ {#if !hideControls} +
+
+

+ {replayState === 'playing' ? 'Replaying: ' : ''}{recording.script_path || + 'Untitled script'} +

+ + {recording.language} + + + + {#snippet text()} + + Recorded {new Date(recording.recorded_at).toLocaleString()} — + {(recording.total_duration_ms / 1000).toFixed(1)}s + + {/snippet} + +
+ {#if replayState === 'loaded'} + + {:else} + + {/if}
- -
- - {#if recording.args && Object.keys(recording.args).length > 0} - {/if} - - + {#if replayState === 'playing'} + + {/if} + + + {#if replayState === 'playing'} + + {/if} + {#if schema && recording.args} + + {/if} + {#if recording.args && Object.keys(recording.args).length > 0} + + {/if} + {#if !schema || !recording.args} + + {/if} {#if schema} {/if} {#snippet content()} + + {#if replayState === 'playing' && jobId} +
+
+

Result

+
+ {#if job !== undefined && job.type === 'CompletedJob' && job.result !== undefined} + + {:else if done} +
+ No output available +
+ {:else} +
+ Waiting for result... +
+ {/if} +
+
+
+

Logs

+
+ +
+
+
+ {/if} + + + {#if schema && recording.args} +
+
+ +
+
+ +
+
+ {/if} + + + {#if recording.args && Object.keys(recording.args).length > 0} +
+ +
+ {/if} + -
+
@@ -180,46 +295,4 @@ {/snippet}
-{:else if replayState === 'playing' && jobId} -
-
-

Replaying: {recording.script_path || 'Untitled script'}

- -
- - - {#if done && job} -
-

Result

-
- {#if job.type === 'CompletedJob' && job.result !== undefined} - - {:else} -
No result available
- {/if} -
-
- {/if} - -
- -
-
{/if} diff --git a/frontend/src/lib/components/runs/JobDetailFieldConfig.ts b/frontend/src/lib/components/runs/JobDetailFieldConfig.ts index 23a7df12cd..50edf9e7cf 100644 --- a/frontend/src/lib/components/runs/JobDetailFieldConfig.ts +++ b/frontend/src/lib/components/runs/JobDetailFieldConfig.ts @@ -1,6 +1,7 @@ import type { Job } from '$lib/gen' import { triggerIconMap } from '$lib/components/triggers/utils' import { formatMemory } from '$lib/utils' +import { flowPathToHref } from '$lib/scripts' import { Calendar, Bot } from 'lucide-svelte' import BarsStaggered from '$lib/components/icons/BarsStaggered.svelte' @@ -240,11 +241,12 @@ export const fieldConfigs: Record = { field: 'script_path', label: 'Path', getValue: (job) => job.script_path || null, - getHref: (job, workspaceId) => { + getHref: (job, _workspaceId) => { if (!job.script_path) return null - const stem = job.job_kind === 'script' ? 'scripts' : 'flows' const isScript = job.job_kind === 'script' - return `/${stem}/get/${isScript ? job.script_hash : job.script_path}` + return isScript + ? `/scripts/get/${job.script_hash}` + : flowPathToHref(job.script_path) } }, diff --git a/frontend/src/lib/components/runs/JobDetailHeader.svelte b/frontend/src/lib/components/runs/JobDetailHeader.svelte index 7ca973dd00..b0941ddcad 100644 --- a/frontend/src/lib/components/runs/JobDetailHeader.svelte +++ b/frontend/src/lib/components/runs/JobDetailHeader.svelte @@ -14,6 +14,7 @@ import Button from '$lib/components/common/button/Button.svelte' import DropdownV2 from '$lib/components/DropdownV2.svelte' import { getRelevantFields, getTriggerInfo, type FieldConfig } from './JobDetailFieldConfig' + import { flowPathToHref } from '$lib/scripts' import { slide } from 'svelte/transition' import { twMerge } from 'tailwind-merge' @@ -385,10 +386,12 @@
- {#if job.script_path && (job.job_kind === 'script' || job.job_kind === 'flow' || job.job_kind === 'singlestepflow')} + {#if job.script_path && (job.job_kind === 'script' || job.job_kind === 'flow' || job.job_kind === 'singlestepflow' || job.job_kind === 'flowpreview')} {@const stem = job.job_kind === 'script' ? 'scripts' : 'flows'} {@const isScript = job.job_kind === 'script'} - {@const viewHref = `${base}/${stem}/get/${isScript ? job?.script_hash : job?.script_path}`} + {@const viewHref = isScript + ? `${base}/${stem}/get/${job?.script_hash}` + : flowPathToHref(job?.script_path ?? '')} import Tooltip from '$lib/components/meltComponents/Tooltip.svelte' import PreprocessedArgsDisplay from '$lib/components/runs/PreprocessedArgsDisplay.svelte' - import { truncateHash } from '$lib/utils' + import { getJobKindDisplayLabel, truncateHash } from '$lib/utils' import { base } from '$lib/base' import { truncateRev } from '$lib/utils' import { workspaceStore } from '$lib/stores' @@ -46,7 +46,7 @@ {/if} {#if job && 'job_kind' in job}
- Job kind: {job.job_kind} + Job kind: {getJobKindDisplayLabel(job.job_kind, job.script_path)}
{/if} {#if job && job.flow_status && job.job_kind === 'script'} diff --git a/frontend/src/lib/components/runs/RunRow.svelte b/frontend/src/lib/components/runs/RunRow.svelte index 3a14c71111..d4b1ac8403 100644 --- a/frontend/src/lib/components/runs/RunRow.svelte +++ b/frontend/src/lib/components/runs/RunRow.svelte @@ -9,6 +9,7 @@ isScriptPreview, msToReadableTime, isFlowPreview, + getJobKindDisplayLabel, getJobKindIcon } from '$lib/utils' import { Button } from '../common' @@ -155,12 +156,12 @@ {/if}
- {#snippet text()} - - {#if job && job.job_kind} - {job.job_kind} - {/if} - {#if job && job.is_flow_step && job.parent_job} + {#snippet text()} + + {#if job && job.job_kind} + {getJobKindDisplayLabel(job.job_kind, job.script_path)} + {/if} + {#if job && job.is_flow_step && job.parent_job}
Step of flow {truncateRev(job.parent_job, 10)} diff --git a/frontend/src/lib/components/schema/EditableSchemaWrapper.svelte b/frontend/src/lib/components/schema/EditableSchemaWrapper.svelte index 27f19928f1..65a865f1d9 100644 --- a/frontend/src/lib/components/schema/EditableSchemaWrapper.svelte +++ b/frontend/src/lib/components/schema/EditableSchemaWrapper.svelte @@ -20,6 +20,7 @@ fullHeight = true, formatExtension = $bindable(undefined), isFileset = $bindable(undefined), + showSensitiveToggle = false, customUi }: EditableSchemaWrapperProps = $props() @@ -113,6 +114,7 @@ bind:this={editableSchemaForm} bind:schema isFlowInput + {showSensitiveToggle} on:delete={(e) => { addPropertyComponent?.handleDeleteArgument([e.detail]) }} @@ -162,9 +164,9 @@ {:else if formatExtension && formatExtension !== ''} - The .{formatExtension} extension will be used to - infer the format when displaying the content and this is also how the resource will appear - when pulling via the CLI. + The .{formatExtension} extension will be used to infer + the format when displaying the content and this is also how the resource will appear when pulling + via the CLI.
{/if} @@ -175,10 +177,7 @@ path and contains text content. In the CLI, filesets are stored as directories. {/if} - switchResourceMode(mode)} -> + switchResourceMode(mode)}> {#snippet children({ item })} diff --git a/frontend/src/lib/components/schema/PropertyEditor.svelte b/frontend/src/lib/components/schema/PropertyEditor.svelte index 1dcf1a817d..3cd0ec5507 100644 --- a/frontend/src/lib/components/schema/PropertyEditor.svelte +++ b/frontend/src/lib/components/schema/PropertyEditor.svelte @@ -7,6 +7,7 @@ import NumberTypeNarrowing from '../NumberTypeNarrowing.svelte' import StringTypeNarrowing from '../StringTypeNarrowing.svelte' import Tooltip from '../Tooltip.svelte' + import Toggle from '../Toggle.svelte' import EditableSchemaForm from '../EditableSchemaForm.svelte' import { deepEqual } from 'fast-equals' @@ -35,6 +36,7 @@ nonEmpty?: boolean | undefined isFlowInput?: boolean isAppInput?: boolean + showSensitiveToggle?: boolean order?: string[] | undefined itemsType?: | { @@ -66,6 +68,7 @@ properties = $bindable(), isFlowInput = false, isAppInput = false, + showSensitiveToggle = false, order = $bindable(), itemsType = $bindable(undefined), typeeditor, @@ -290,5 +293,25 @@ {/if} {@render children?.()} + + {#if type == 'object' && showSensitiveToggle} + { + if (e.detail) { + extra['password'] = true + } else { + extra['password'] = undefined + } + dispatch('change') + }} + /> + {/if}
diff --git a/frontend/src/lib/components/schema/editable_schema_wrapper.ts b/frontend/src/lib/components/schema/editable_schema_wrapper.ts index 1ef4426613..bcc659b27e 100644 --- a/frontend/src/lib/components/schema/editable_schema_wrapper.ts +++ b/frontend/src/lib/components/schema/editable_schema_wrapper.ts @@ -7,6 +7,7 @@ export type EditableSchemaWrapperProps = { fullHeight?: boolean formatExtension?: string | undefined isFileset?: boolean | undefined + showSensitiveToggle?: boolean customUi?: { noAddPopover?: boolean } diff --git a/frontend/src/lib/components/scriptEditor/LogPanel.svelte b/frontend/src/lib/components/scriptEditor/LogPanel.svelte index 271e2a765c..82845c4a88 100644 --- a/frontend/src/lib/components/scriptEditor/LogPanel.svelte +++ b/frontend/src/lib/components/scriptEditor/LogPanel.svelte @@ -158,6 +158,7 @@ result={previewJob?.result} success={previewJob?.success !== false} autoExpandResult + jobId={previewJob?.id} />
{:else} diff --git a/frontend/src/lib/components/secretArgUtils.ts b/frontend/src/lib/components/secretArgUtils.ts new file mode 100644 index 0000000000..82aee1f53d --- /dev/null +++ b/frontend/src/lib/components/secretArgUtils.ts @@ -0,0 +1,49 @@ +import type { Schema } from '$lib/common' +import { VariableService } from '$lib/gen' +import { get } from 'svelte/store' +import { userStore, workspaceStore } from '$lib/stores' +import { generateRandomString } from '$lib/utils' + +/** + * Process args before job submission: for non-string fields marked as password/sensitive, + * create ephemeral secret variables and replace values with $jsonvar:path references. + * String password fields are already handled by PasswordArgInput (uses $var:). + */ +export async function processSecretArgs( + args: Record, + schema: Schema | undefined +): Promise> { + if (!schema?.properties) return args + + const workspace = get(workspaceStore) + const user = get(userStore) + if (!workspace || !user) return args + + const username = (user.username ?? user.email)?.split('@')[0] + if (!username) return args + const userPrefix = `u/${username}/secret_arg/` + + const result = { ...args } + + for (const [key, prop] of Object.entries(schema.properties)) { + if (!prop.password) continue + if (prop.type !== 'object') continue // only object types; strings handled by PasswordArgInput + if (result[key] == null || result[key] === undefined) continue + if (typeof result[key] === 'string' && result[key].startsWith('$jsonvar:')) continue // already processed + + const path = userPrefix + generateRandomString(12) + await VariableService.createVariable({ + workspace, + requestBody: { + value: JSON.stringify(result[key]), + is_secret: true, + path, + description: 'Ephemeral secret variable', + expires_at: new Date(Date.now() + 1000 * 60 * 60 * 24 * 7).toISOString() + } + }) + result[key] = '$jsonvar:' + path + } + + return result +} diff --git a/frontend/src/lib/components/select/SelectDropdown.svelte b/frontend/src/lib/components/select/SelectDropdown.svelte index 62a990bb46..d9fc2dcf70 100644 --- a/frontend/src/lib/components/select/SelectDropdown.svelte +++ b/frontend/src/lib/components/select/SelectDropdown.svelte @@ -117,7 +117,7 @@ ulClass )} > - {#each processedItems ?? [] as item, itemIndex (item.value)} + {#each processedItems ?? [] as item, itemIndex} {#if (item.__select_group && itemIndex === 0) || processedItems?.[itemIndex - 1]?.__select_group !== item.__select_group}
  • void onReset: () => void hasChanges: boolean - isWorkspaceSettings?: boolean + scope?: 'user' | 'workspace' | 'instance' } let { @@ -22,7 +22,7 @@ onSave, onReset, hasChanges, - isWorkspaceSettings = false + scope = 'user' }: Props = $props() const placeholders: Record = { @@ -63,9 +63,12 @@
    - {#if isWorkspaceSettings} + {#if scope === 'workspace'} Customize the system prompts for each AI mode. These prompts apply to all workspace members. + {:else if scope === 'instance'} + Customize the system prompts for each AI mode. These prompts apply to workspaces using + instance AI defaults. {:else} Customize the system prompts for each AI mode. These prompts are stored locally in your browser and apply in addition to workspace-level prompts. diff --git a/frontend/src/lib/components/settings/Trashbin.svelte b/frontend/src/lib/components/settings/Trashbin.svelte new file mode 100644 index 0000000000..3b9b4f5f29 --- /dev/null +++ b/frontend/src/lib/components/settings/Trashbin.svelte @@ -0,0 +1,249 @@ + + +
    + + +
    + +{#if items === undefined} + +{:else if items.length === 0} +
    + +

    Trashbin is empty

    +

    No recently deleted items.

    +
    +{:else} + + + + Type + Path + Deleted by + Deleted + Expires + Actions + + + {#each items as item (item.id)} + {@const Icon = getKindIcon(item.item_kind)} + + +
    + + {getKindLabel(item.item_kind)} +
    +     + + {item.item_path} + + + {item.deleted_by} + + + {timeAgo(item.deleted_at)} + + + {timeRemaining(item.expires_at)} + + +
    + + +
    +     +     + {/each} +     +{/if} + + { + deleteConfirmedCallback = undefined + }} + onConfirmed={() => { + if (deleteConfirmedCallback) { + deleteConfirmedCallback() + } + deleteConfirmedCallback = undefined + }} +> +

    This item will be permanently deleted. This action cannot be undone.

    +     + + { + emptyConfirmOpen = false + }} + onConfirmed={() => { + emptyAll() + emptyConfirmOpen = false + }} +> +

    All items in the trashbin will be permanently deleted. This action cannot be undone.

    +     diff --git a/frontend/src/lib/components/settings/WorkspaceUserSettings.svelte b/frontend/src/lib/components/settings/WorkspaceUserSettings.svelte index 03b57f6d9a..8b245c7c70 100644 --- a/frontend/src/lib/components/settings/WorkspaceUserSettings.svelte +++ b/frontend/src/lib/components/settings/WorkspaceUserSettings.svelte @@ -14,9 +14,15 @@ import Tooltip from '$lib/components/Tooltip.svelte' import type { CancelablePromise, User, UserUsage } from '$lib/gen' import { UserService, WorkspaceService, GroupService, type WorkspaceInvite } from '$lib/gen' - import { userStore, workspaceStore, superadmin, globalEmailInvite } from '$lib/stores' + import { + userStore, + workspaceStore, + superadmin, + globalEmailInvite, + enterpriseLicense + } from '$lib/stores' import { sendUserToast } from '$lib/toast' - import { Loader2, Mails, Search, Plus, UserMinus, X } from 'lucide-svelte' + import { Loader2, Mails, Search, Plus, UserMinus, X, Bot, LogIn } from 'lucide-svelte' import Select from '$lib/components/select/Select.svelte' import SearchItems from '../SearchItems.svelte' import Cell from '../table/Cell.svelte' @@ -45,6 +51,8 @@ let selectedNewInstanceGroup: string | undefined = $state(undefined) let selectedNewRole: string | undefined = $state('developer') + // Service account creation + // Available groups for dropdowns - filter out already configured groups let availableGroupItems = $derived( instanceGroups @@ -488,12 +496,14 @@ {#snippet children({ item })} {/if} -     {truncate(email, 20)} - {truncate(username, 30)} + + {#if user.is_service_account} + + + {email} + + {:else} +     {email} + {/if} + + {username} {#if hasNonManualUsers}
    @@ -796,14 +825,21 @@ {/if} {#if usage?.[email] != undefined}{usage?.[email]}{:else}{#if usage != undefined}{usage[email] ?? 0}{:else}{/if}
    - {#if added_via?.source === 'instance_group'} + {#if user.is_service_account} +
    + + Operator + + Service accounts are always operators. +
    + {:else if added_via?.source === 'instance_group'}
    {is_admin ? 'Admin' : operator ? 'Operator' : 'Developer'} @@ -840,6 +876,7 @@ {#snippet children({ item })}
    + {#if user.is_service_account && $userStore?.is_admin} + + {/if} {#snippet removeUserButton(disabled: boolean)}
    diff --git a/frontend/src/lib/components/triggers/CaptureWrapper.svelte b/frontend/src/lib/components/triggers/CaptureWrapper.svelte index a6908c6957..0c763987a9 100644 --- a/frontend/src/lib/components/triggers/CaptureWrapper.svelte +++ b/frontend/src/lib/components/triggers/CaptureWrapper.svelte @@ -260,7 +260,7 @@ {hasPreprocessor} {isFlow} {captureLoading} - {triggerDeployed} + groupId={args?.group_id} on:applyArgs on:updateSchema on:addPreprocessor diff --git a/frontend/src/lib/components/triggers/DeleteTriggerButton.svelte b/frontend/src/lib/components/triggers/DeleteTriggerButton.svelte index b7fed3d6d0..26b1933de7 100644 --- a/frontend/src/lib/components/triggers/DeleteTriggerButton.svelte +++ b/frontend/src/lib/components/triggers/DeleteTriggerButton.svelte @@ -21,6 +21,7 @@ title={`Are you sure you want to delete this ${trigger?.isDraft ? 'draft' : 'deployed'} trigger ?`} confirmationText="Delete" open={confirmationModalOpen} + trashbin on:canceled={() => { confirmationModalOpen = false }} diff --git a/frontend/src/lib/components/triggers/PermissionedAsLine.svelte b/frontend/src/lib/components/triggers/PermissionedAsLine.svelte new file mode 100644 index 0000000000..23723b7833 --- /dev/null +++ b/frontend/src/lib/components/triggers/PermissionedAsLine.svelte @@ -0,0 +1,84 @@ + + +{#if permissionedAs && $workspaceStore} +
    + Permissioned as + {#if canPreserve} + + {#if willChange} + + will change from {permissionedAs} on save + {/if} + {:else} + {permissionedAs} + {#if willChange} + + will change to {effectivePermissionedAs} on save + {/if} + {/if} +
    +{/if} diff --git a/frontend/src/lib/components/triggers/TriggerEditorToolbar.svelte b/frontend/src/lib/components/triggers/TriggerEditorToolbar.svelte index a09491e0ba..52c68aeed9 100644 --- a/frontend/src/lib/components/triggers/TriggerEditorToolbar.svelte +++ b/frontend/src/lib/components/triggers/TriggerEditorToolbar.svelte @@ -118,7 +118,7 @@ {trigger?.isDraft ? 'Deploy' : 'Update'} {#snippet text()} - + {#if !isDeployed} Deploy the runnable to enable trigger creation {:else if cloudDisabled} @@ -127,7 +127,7 @@ Enter a valid config to {trigger?.isDraft ? 'deploy' : 'update'} the trigger {/if} - {/snippet} + {/snippet} {/if}
    diff --git a/frontend/src/lib/components/triggers/TriggerFilters.svelte b/frontend/src/lib/components/triggers/TriggerFilters.svelte index e2657f6cbd..cb7f198346 100644 --- a/frontend/src/lib/components/triggers/TriggerFilters.svelte +++ b/frontend/src/lib/components/triggers/TriggerFilters.svelte @@ -1,23 +1,45 @@

    - Filters will limit the execution of the trigger to only messages that match all criteria.
    + {description}
    The JSON filter checks if the value at the key is equal or a superset of the filter value.

    + {#if filters.length > 0} +
    + - -