Compare commits

..
Author SHA1 Message Date
wendrul ba775edef2 Merge remote-tracking branch 'origin/main' into folder-deploy
# Conflicts:
#	frontend/src/lib/utils_workspace_deploy.ts
2026-04-02 15:39:36 +02:00
wendrul 91f0a564b9 fix: deployment UIs for folders 2026-04-02 15:33:12 +02:00
Diego Imbert c3a1c26be1 nit: revert ee.rs in substitute_ee_code.sh (#8672) 2026-04-02 10:16:28 +00:00
Ruben FiszelandClaude Opus 4.5 c87a6a0f2c fix: support branch-specific folder.meta.yaml in missing-meta check (#8661)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-02 10:05:02 +00:00
hugocasaandClaude Opus 4.6 350ffdce29 fix: pre-fix trigger edited_by for superadmins not in workspace (#8669)
Add a migration that runs just before 20260318000000 (add_permissioned_as).
For each trigger table, if the email column still exists, update edited_by
to the trigger's email when the user is not in the workspace but is a
superadmin. This ensures the subsequent permissioned_as migration stores
the raw email instead of an invalid u/{username} reference.

If 20260318000000 was already applied, the migration is a no-op (email
column is gone, guarded by information_schema check).

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 08:39:44 +00:00
centdixandClaude Opus 4.5 28c073056c fix: correct raw app flow inputs (#8667)
* fix: correct raw app flow inputs

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: remove raw app legacy migration

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-02 01:57:56 +00:00
Ruben FiszelandClaude Opus 4.6 c86846ac19 rate limit token creation on CLOUD_HOSTED (10/min per user) (#8664)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-02 01:52:02 +00:00
Ruben FiszelandClaude Opus 4.6 7ab0ea581d fix: strip f/ prefix from folder paths when deploying from workspace forks (#8662)
* fix: strip f/ prefix from folder paths when deploying from workspace forks

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: extract folderName helper for f/ prefix stripping

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 21:06:19 +00:00
Ruben FiszelandClaude Opus 4.6 bcce627387 fix: validate rd redirect on login with same rules as logout (#8655)
* fix: validate rd redirect on login with same rules as logout

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: sanitize rd at source in login callback to prevent leaking to goto

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: validate rd redirect in Login component for fresh login flow

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-01 19:44:22 +00:00
Ruben Fiszelandrubenfiszel 175af8032f chore(main): release 1.672.0 (#8654)
* chore(main): release 1.672.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-04-01 12:05:44 -04:00
Diego Imbert 1784bed4ac Fix WAC RunForm layout (#8658) 2026-04-01 12:01:29 -04:00
a46aa641f9 feat: add R language support (#8263)
* feat: add R language support

Add R as a new supported scripting language in Windmill, following the
same pattern used for Ruby. Includes:

- Backend: ScriptLang::Rlang enum variant, DB migration, tree-sitter-r
  parser crate with tests, WASM parser binding, R executor with NSJail
  sandboxing, job dispatch and signature parsing
- Frontend: language picker, R icon, syntax highlighting, editor bar
  insertions (Sys.getenv, get_variable, get_resource), schema inference,
  init code template, BETA badge
- CLI: .r extension mapping, sync support, bootstrap template

R scripts use `main <- function(...)` syntax, jsonlite for JSON
serialization, and system curl for the Windmill client helper.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add R package resolution and installation

Parse library()/require() calls from R scripts to extract dependencies.
Resolve versions from CRAN, cache lockfiles in pip_resolution_cache,
and install packages to a shared R library cache. The run step sets
R_LIBS_USER so installed packages are available to the script.

- Parser: parse_r_requirements() extracts package names from AST
- Executor: resolve() generates lockfile, install() installs from CRAN
- Worker lockfiles: wire up R resolve for dependency jobs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add nsjail sandboxing for R resolve and install phases

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix R get_variable/get_resource and add sandbox annotation + e2e tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix R arg inference with JS fallback parser and get_variable/get_resource

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix flake

* nsjail

* nits

* fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits

- Suppress renv verbose output during resolve and install (controlled by #verbose annotation)
- Filter renv from install list (already loaded, causes noisy restart message)
- Log compact "resolved N packages" instead of full renv.lock JSON
- Add R (r-base, r-cran-renv) to DockerfileFull and DockerfileFullEe
- Use disable_rl for nsjail install config (R compiles from source)
- Reduce default concurrency from 20 to 5
- Add rlang to openflow.openapi.yaml
- Fix MainArgSignature (no_main_func -> auto_kind) after main merge

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* final

* fix: remove accidental R install from multiplayer Dockerfile

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: remove R from Windows build and DockerfileExtra

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: rename R migration to avoid timestamp collision with trigger_filter_logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

* fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add clear error when Rscript binary is missing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: fix type errors in R fallback parser, use format! in wrap(), add R system prompts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pyranota <pyra@duck.com>
2026-04-01 06:11:37 +00:00
Alexander PetricandClaude Opus 4.5 7069202190 fix: approval page freeze, stale state, and missing approval link (#8653)
* fix: prevent browser freeze when approval form number field has no default value

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: disable approval buttons and keep polling after approve/deny action

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: restore approval page link and prevent double resume in flow viewer

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: guard against NaN fallback in Range and reset actionTaken on new approval step

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix approval page url

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-04-01 05:25:22 +02:00
Ruben Fiszelandrubenfiszel df7a8eebcf chore(main): release 1.671.0 (#8650)
* chore(main): release 1.671.0

* Apply automatic changes

---------

Co-authored-by: rubenfiszel <275584+rubenfiszel@users.noreply.github.com>
2026-03-31 21:26:34 +00:00
centdix 2862c1cf56 add codex PR review workflow (#8626)
* feat: add codex PR review workflow

* refactor: simplify codex PR review comments

* chore: use ubicloud for codex review

* fix: harden codex review workflow

* chore: use chatgpt auth for codex review
2026-03-31 19:21:39 +00:00
centdixandClaude Opus 4.6 d67223de9b chore: use fully qualified tmux pane targets in webmux systemPrompt (#8651)
* fix: use fully qualified tmux pane targets in webmux systemPrompt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: anchor tmux pane targets to $TMUX_PANE for stability across window switches

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 19:16:38 +00:00
Ruben FiszelandClaude Opus 4.6 da8886be85 feat: add configurable preview job tag override in default tags settings (#8649)
* feat: add configurable preview job tag override in default tags settings

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: skip re-tagging for FlowPreview jobs when preview override is active

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 18:59:23 +00:00
centdixandClaude Opus 4.5 040a199685 feat: support hub flows in raw app runnables (#8627)
* feat: support hub flows in raw app runnables

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: support hub flow previews in app ui

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: move trigger context into flow graph viewer

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use script viewer for hub flow steps

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: stretch raw app flow previews to pane height

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: improve hub flow run links

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: stabilize hub flow preview drawer

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: align hub flow id validation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* style: fix runnable panel indentation

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-31 18:26:56 +00:00
Alexander PetricandClaude Opus 4.6 6c3c971af5 feat: improve CLI flow log streaming and job inspection (#8644)
* fix: improve CLI flow log streaming, sub-job listing, and failure handling

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add hierarchical flow status in job get and aggregated flow logs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: remove duplicate ansi color hint in job logs output

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: update cli-commands skill with new job/flow features

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test: add integration tests for flow job inspection and log aggregation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: remove internal friction discovery doc from branch

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: trim cli-commands skill to reduce context bloat

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* docs: update job command descriptions and regenerate skills.ts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: commit auto-generated files from system_prompts

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address review comments on flow streaming and test assertions

- Move for-loop waiting logic outside --silent guard (Cubic #2)
- Break outer loop when for-loop module fails (Cubic #3)
- Strengthen test assertion: toContain("a") -> toContain("a: Generate data") (Cubic #1)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: generator regex truncating descriptions with parentheses

The .command() regex used [^)]+ for the second arg, stopping at the
first ')' inside description strings like "(machine-friendly)".
Now matches quoted strings properly before falling back.

Fixes 6 truncated descriptions across job, flow, and script commands.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-31 18:22:18 +00:00
Ruben FiszelandClaude Opus 4.6 852c59efbb fix: return default_args/enums in approval info and fix subflow resume buttons (#8648)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 18:16:47 +00:00
Ruben Fiszel 89d1acda24 chore(main): release 1.670.0 (#8625)
* chore(main): release 1.670.0

* update
2026-03-31 16:01:05 +00:00
Ruben FiszelandClaude Opus 4.6 12ea7e7423 fix: resolve missing form schema for nested suspend steps in FlowNode sub-flows (#8643)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 15:36:46 +00:00
Ruben FiszelandClaude Opus 4.6 375fb66abe feat: support sensitive/secret fields for non-string types (#8635)
* feat: support sensitive/secret fields for non-string types

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: restrict sensitive toggle to object type, move after showExpr

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: show sensitive toggle in PropertyEditor at bottom, after children

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: gate sensitive toggle with showSensitiveToggle prop

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: process secret args in flow test and script test paths

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: inline SecretArgInput into ArgInput, delete component

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address CI review feedback

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: pass showSensitiveToggle to flow input schema editors

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: use explicit prop syntax to satisfy svelte-check

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: narrow try/catch to only processSecretArgs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 15:13:23 +00:00
Ruben FiszelandClaude Opus 4.6 52a04d210f fix: preserve flow notes/groups and field ordering in generate-metadata (#8641) (#8642)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-31 13:48:22 +00:00
143 changed files with 5423 additions and 906 deletions
+23
View File
@@ -0,0 +1,23 @@
You are reviewing a GitHub pull request for this repository.
Review policy:
- Read `CLAUDE.md` before reviewing code.
- Only report issues you are confident are real and introduced by this pull request.
- Focus on bugs, security problems, and clear `CLAUDE.md` violations.
- Do not report style nits, speculative concerns, pre-existing issues, or problems that a normal linter/typechecker would obviously catch.
- Keep the review high signal. If there is no clear issue, return no findings.
Repository context:
- Read `./.github/codex/pr-review-context.md` for the PR metadata and the exact diff commands to use.
- Review only the changes introduced by this PR.
- Read additional files only when the diff is not enough to validate a finding.
- Do not modify any files.
Output requirements:
- Return a GitHub PR comment in markdown, not JSON.
- Start with `## Codex Review`.
- Give a short overall summary first.
- If you found high-signal issues, list them in a short numbered list with file paths and line numbers when you know them confidently.
- If you found no high-signal issues, say that explicitly.
- End with a `### Reproduction instructions` section containing a short descriptive paragraph for a tester explaining how to navigate the app to observe the change. Do not make it a numbered list. If the diff is not enough to infer this safely, say that plainly.
- Prefer at most 10 findings.
+145
View File
@@ -0,0 +1,145 @@
name: Codex Auto Review
on:
pull_request:
types: [ready_for_review, opened]
concurrency:
group: codex-review-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
codex-review:
runs-on: ubicloud-standard-2
timeout-minutes: 30
if: github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false
permissions:
contents: read
issues: write
steps:
- name: Check Codex configuration
id: codex_config
env:
CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
run: |
if [ -n "$CODEX_AUTH_JSON" ]; then
echo "enabled=true" >> "$GITHUB_OUTPUT"
else
echo "enabled=false" >> "$GITHUB_OUTPUT"
echo "CODEX_AUTH_JSON is not configured; skipping Codex review."
fi
- name: Checkout repository
if: steps.codex_config.outputs.enabled == 'true'
uses: actions/checkout@v5
with:
ref: refs/pull/${{ github.event.pull_request.number }}/merge
fetch-depth: 1
- name: Set up Node.js
if: steps.codex_config.outputs.enabled == 'true'
uses: actions/setup-node@v4
with:
node-version: 22
- name: Install Codex CLI
if: steps.codex_config.outputs.enabled == 'true'
run: npm install --global @openai/codex@0.117.0
- name: Configure file-backed Codex auth
if: steps.codex_config.outputs.enabled == 'true'
env:
CODEX_AUTH_JSON: ${{ secrets.CODEX_AUTH_JSON }}
run: |
CODEX_HOME="$HOME/.codex"
echo "CODEX_HOME=$CODEX_HOME" >> "$GITHUB_ENV"
mkdir -p "$CODEX_HOME"
chmod 700 "$CODEX_HOME"
cat > "$CODEX_HOME/config.toml" <<'EOF'
cli_auth_credentials_store = "file"
EOF
printf '%s' "$CODEX_AUTH_JSON" > "$CODEX_HOME/auth.json"
chmod 600 "$CODEX_HOME/auth.json"
node -e 'JSON.parse(require("fs").readFileSync(process.argv[1], "utf8"))' "$CODEX_HOME/auth.json"
- name: Pre-fetch base and head refs for the PR
if: steps.codex_config.outputs.enabled == 'true'
env:
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
PR_NUMBER: ${{ github.event.pull_request.number }}
run: |
git fetch --no-tags origin \
"$PR_BASE_REF" \
"+refs/pull/$PR_NUMBER/head"
- name: Write Codex review context
if: steps.codex_config.outputs.enabled == 'true'
env:
PR_REPOSITORY: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
PR_BASE_SHA: ${{ github.event.pull_request.base.sha }}
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
PR_TITLE: ${{ github.event.pull_request.title }}
PR_BODY: ${{ github.event.pull_request.body || '' }}
run: |
mkdir -p .github/codex
node <<'NODE'
const fs = require('fs');
const lines = [
`Repository: ${process.env.PR_REPOSITORY}`,
`PR number: ${process.env.PR_NUMBER}`,
`Base SHA: ${process.env.PR_BASE_SHA}`,
`Head SHA: ${process.env.PR_HEAD_SHA}`,
'',
'PR title:',
process.env.PR_TITLE || '(empty)',
'',
'PR body:',
process.env.PR_BODY || '(empty)',
'',
'Changed commits command:',
`git log --oneline ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
'',
'Changed files command:',
`git diff --stat ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`,
'',
'Full review diff command:',
`git diff --unified=0 ${process.env.PR_BASE_SHA}...${process.env.PR_HEAD_SHA}`
];
fs.writeFileSync('.github/codex/pr-review-context.md', `${lines.join('\n')}\n`);
NODE
- name: Run Codex review
if: steps.codex_config.outputs.enabled == 'true'
run: |
codex exec \
-C "$GITHUB_WORKSPACE" \
-m gpt-5.4 \
-c 'model_reasoning_effort="xhigh"' \
-s read-only \
-o codex-final-message.md \
- < .github/codex/pr-review.prompt.md
- name: Post Codex review comment
if: steps.codex_config.outputs.enabled == 'true'
uses: actions/github-script@v7
with:
github-token: ${{ github.token }}
script: |
const fs = require('fs');
const path = `${process.env.GITHUB_WORKSPACE}/codex-final-message.md`;
if (!fs.existsSync(path)) {
core.info('Codex did not produce a final message; skipping PR comment.');
return;
}
const body = fs.readFileSync(path, 'utf8').trim();
if (!body) {
core.info('Codex final message was empty; skipping PR comment.');
return;
}
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
body,
});
+2 -2
View File
@@ -43,7 +43,7 @@ profiles:
- Pane 0: this pane (claude agent)
- Pane 1: backend (cargo watch -x run)
- Pane 2: frontend (npm run dev)
To check logs, use: \`tmux capture-pane -t .1 -p -S -50\` (backend) or \`tmux capture-pane -t .2 -p -S -50\` (frontend).
To check logs, use: \`tmux capture-pane -t $(tmux display-message -t "$TMUX_PANE" -p '#{session_name}:#{window_name}').1 -p -S -50\` (backend) or \`tmux capture-pane -t $(tmux display-message -t "$TMUX_PANE" -p '#{session_name}:#{window_name}').2 -p -S -50\` (frontend).
For this window specifically, backend is running on: ${BACKEND_PORT} and frontend is running on: ${FRONTEND_PORT}.
To connect to the database, use this connection string: ${DATABASE_URL}
Because we are running backend with cargo watch, to verify your changes, just check the logs in the backend pane. No need for cargo check.
@@ -72,7 +72,7 @@ profiles:
Pane layout (current window):
- Pane 0: this pane (claude agent)
- Pane 1: frontend (npm run dev)
To check logs, use: \`tmux capture-pane -t .1 -p -S -50\` (frontend).
To check logs, use: \`tmux capture-pane -t $(tmux display-message -t "$TMUX_PANE" -p '#{session_name}:#{window_name}').1 -p -S -50\` (frontend).
On this window specifically, frontend is running on: ${FRONTEND_PORT}.
To connect to the database, use this connection string: ${DATABASE_URL}
Because we are running frontend with npm run dev, to verify your changes, just check the logs in the frontend pane. No need for npm run build.
+47
View File
@@ -1,5 +1,52 @@
# Changelog
## [1.672.0](https://github.com/windmill-labs/windmill/compare/v1.671.0...v1.672.0) (2026-04-01)
### Features
* add R language support ([#8263](https://github.com/windmill-labs/windmill/issues/8263)) ([a46aa64](https://github.com/windmill-labs/windmill/commit/a46aa641f9d72809c52a0eb11a877a0f2d587c32))
### Bug Fixes
* approval page freeze, stale state, and missing approval link ([#8653](https://github.com/windmill-labs/windmill/issues/8653)) ([7069202](https://github.com/windmill-labs/windmill/commit/70692021909443b86ed61fa621fe49f28742fb54))
## [1.671.0](https://github.com/windmill-labs/windmill/compare/v1.670.0...v1.671.0) (2026-03-31)
### Features
* add configurable preview job tag override in default tags settings ([#8649](https://github.com/windmill-labs/windmill/issues/8649)) ([da8886b](https://github.com/windmill-labs/windmill/commit/da8886be8575dd925b6d24c55ab379bc6984c5f8))
* improve CLI flow log streaming and job inspection ([#8644](https://github.com/windmill-labs/windmill/issues/8644)) ([6c3c971](https://github.com/windmill-labs/windmill/commit/6c3c971af5aa1362632ee0deeddf91b8bc47c853))
* support hub flows in raw app runnables ([#8627](https://github.com/windmill-labs/windmill/issues/8627)) ([040a199](https://github.com/windmill-labs/windmill/commit/040a199685cea5c99c944bacb5584a381d6ec829))
### Bug Fixes
* return default_args/enums in approval info and fix subflow resume buttons ([#8648](https://github.com/windmill-labs/windmill/issues/8648)) ([852c59e](https://github.com/windmill-labs/windmill/commit/852c59efbb04510e5e6f99919707effcf6769a2f))
## [1.670.0](https://github.com/windmill-labs/windmill/compare/v1.669.1...v1.670.0) (2026-03-31)
### Features
* add OR logic support to kafka/websocket trigger filters ([#8580](https://github.com/windmill-labs/windmill/issues/8580)) ([3876902](https://github.com/windmill-labs/windmill/commit/3876902a7be798fd5ef208bc5756b28fb55e569e))
* expose getJob and getJobLogs as MCP tools ([#8632](https://github.com/windmill-labs/windmill/issues/8632)) ([cd8edcd](https://github.com/windmill-labs/windmill/commit/cd8edcd94f2bf44c3e771000cb0bbad08accc0e7))
* support multiline secrets in resource password fields ([#8637](https://github.com/windmill-labs/windmill/issues/8637)) ([26050f9](https://github.com/windmill-labs/windmill/commit/26050f96c34f14826298760174a45f3559d3266c))
* support sensitive/secret fields for non-string types ([#8635](https://github.com/windmill-labs/windmill/issues/8635)) ([375fb66](https://github.com/windmill-labs/windmill/commit/375fb66abe2d1861b53dc2b36d2cf0e2eb82c3a8))
### Bug Fixes
* cap input history per_page to 100 on cloud ([#8624](https://github.com/windmill-labs/windmill/issues/8624)) ([8e973c8](https://github.com/windmill-labs/windmill/commit/8e973c892d768be2da2e6b4b7af9e40b62333052))
* compute highest workspace role across all instance groups ([#8633](https://github.com/windmill-labs/windmill/issues/8633)) ([92b9ac7](https://github.com/windmill-labs/windmill/commit/92b9ac72c5fc9a5085fcb2e9d835ccbb53bcd4b0))
* Ducklake UI Nits ([#8628](https://github.com/windmill-labs/windmill/issues/8628)) ([ef1757f](https://github.com/windmill-labs/windmill/commit/ef1757f5d747e513d201eb6fa48918dba8248abe))
* preserve flow notes/groups and field ordering in generate-metadata ([#8641](https://github.com/windmill-labs/windmill/issues/8641)) ([#8642](https://github.com/windmill-labs/windmill/issues/8642)) ([52a04d2](https://github.com/windmill-labs/windmill/commit/52a04d210f476f4598007f67770bc6520b045950))
* remove timeout on python client httpx to prevent ducklake query timeouts ([#8636](https://github.com/windmill-labs/windmill/issues/8636)) ([c5fccd2](https://github.com/windmill-labs/windmill/commit/c5fccd2f69ad8a6e46c514cf89b9aa21b380e6fe))
* resolve missing form schema for nested suspend steps in FlowNode sub-flows ([#8643](https://github.com/windmill-labs/windmill/issues/8643)) ([12ea7e7](https://github.com/windmill-labs/windmill/commit/12ea7e74237560a9dfc99b6bc1338e3343b57640))
* smarter secret masking based on secret length ([#8629](https://github.com/windmill-labs/windmill/issues/8629)) ([bfc2aef](https://github.com/windmill-labs/windmill/commit/bfc2aefdb8ab92b7284de7f9e485a5504502d944))
## [1.669.1](https://github.com/windmill-labs/windmill/compare/v1.669.0...v1.669.1) (2026-03-30)
+148 -125
View File
File diff suppressed because it is too large Load Diff
+9 -3
View File
@@ -1,6 +1,6 @@
[package]
name = "windmill"
version = "1.669.1"
version = "1.672.0"
authors.workspace = true
edition.workspace = true
@@ -66,10 +66,13 @@ members = [
"./parsers/windmill-parser-nu",
"./parsers/windmill-parser-java",
"./parsers/windmill-parser-ruby",
"./parsers/windmill-parser-r",
"./parsers/windmill-parser-bash",
"./parsers/windmill-parser-py",
"./parsers/windmill-parser-py-asset",
"./parsers/windmill-parser-py-imports",
# Uncomment to build wasm parsers:
# "./parsers/windmill-parser-wasm",
"./parsers/windmill-parser-wac",
"./parsers/windmill-parser-sql",
"./parsers/windmill-parser-sql-asset",
@@ -82,7 +85,7 @@ members = [
exclude = ["./windmill-duckdb-ffi-internal", "./parsers/windmill-parser-wasm"]
[workspace.package]
version = "1.669.1"
version = "1.672.0"
authors = ["Ruben Fiszel <ruben@windmill.dev>"]
edition = "2021"
@@ -163,7 +166,8 @@ csharp = ["windmill-worker/csharp"]
nu = ["windmill-worker/nu"]
java = ["windmill-worker/java"]
ruby = ["windmill-worker/ruby"]
all_languages = ["python", "deno_core", "rust", "mysql", "oracledb", "duckdb", "mssql-kerberos", "bigquery", "csharp", "nu", "php", "java", "ruby"]
rlang = ["windmill-worker/rlang"]
all_languages = ["python", "deno_core", "rust", "mysql", "oracledb", "duckdb", "mssql-kerberos", "bigquery", "csharp", "nu", "php", "java", "ruby", "rlang"]
# For windows we have another set of languages enabled
all_languages_windows = ["python", "deno_core", "rust", "mysql", "oracledb", "duckdb", "mssql-winauth", "bigquery", "csharp", "nu", "php", "java"]
# Edition meta-features: shared groups
@@ -347,6 +351,7 @@ windmill-parser-yaml = { path = "./parsers/windmill-parser-yaml" }
windmill-parser-csharp = { path = "./parsers/windmill-parser-csharp" }
windmill-parser-java = { path = "./parsers/windmill-parser-java" }
windmill-parser-ruby = { path = "./parsers/windmill-parser-ruby" }
windmill-parser-r = { path = "./parsers/windmill-parser-r" }
windmill-parser-nu = { path = "./parsers/windmill-parser-nu" }
windmill-parser-bash = { path = "./parsers/windmill-parser-bash" }
windmill-parser-sql = { path = "./parsers/windmill-parser-sql" }
@@ -613,6 +618,7 @@ tree-sitter = { version = "0.23.0", features = [] }
tree-sitter-c-sharp = "0.23.0"
tree-sitter-java = "0.23.0"
tree-sitter-ruby = "0.23.0"
tree-sitter-r = "1.2.0"
oracle = { version = "0.6.3", features = ["chrono"] }
rumqttc = { version = "0.24.0", features = ["use-native-tls"]}
strum = { version = "0.27", features = ["derive"] }
@@ -0,0 +1 @@
-- No-op: this migration is a data fixup and cannot be reversed.
@@ -0,0 +1,48 @@
-- Pre-fix: before permissioned_as migration drops the email column, update edited_by
-- for triggers where the user (edited_by) is not in the workspace but is a superadmin.
-- This ensures the subsequent 20260318000000 migration stores the raw email as permissioned_as
-- (via the `edited_by LIKE '%@%'` branch).
-- For instances that already applied 20260318000000, this is a no-op (email column is gone);
-- the 20260401000000 migration handles those as a fallback.
DO $$
DECLARE
trigger_table TEXT;
has_email BOOLEAN;
BEGIN
FOREACH trigger_table IN ARRAY ARRAY[
'http_trigger',
'websocket_trigger',
'postgres_trigger',
'mqtt_trigger',
'kafka_trigger',
'nats_trigger',
'sqs_trigger',
'gcp_trigger',
'email_trigger'
]
LOOP
SELECT EXISTS (
SELECT 1 FROM information_schema.columns
WHERE table_name = trigger_table AND column_name = 'email'
) INTO has_email;
IF has_email THEN
EXECUTE format($q$
UPDATE %I t
SET edited_by = t.email
WHERE NOT EXISTS (
SELECT 1 FROM usr u
WHERE u.username = t.edited_by
AND u.workspace_id = t.workspace_id
)
AND EXISTS (
SELECT 1 FROM password p
WHERE p.email = t.email
AND p.super_admin = true
)
$q$, trigger_table);
END IF;
END LOOP;
END;
$$;
@@ -0,0 +1,2 @@
ALTER TYPE SCRIPT_LANG ADD VALUE IF NOT EXISTS 'rlang';
UPDATE config SET config = jsonb_set(config, '{worker_tags}', config->'worker_tags' || '["rlang"]'::jsonb) WHERE name = 'worker__default' AND config @> '{"worker_tags": ["deno", "python3", "go", "bash", "powershell", "dependency", "flow", "hub", "other", "bun", "php", "rust", "ansible", "csharp", "nu", "java", "duckdb", "ruby"]}'::jsonb AND NOT config->'worker_tags' @> '"rlang"'::jsonb;
@@ -0,0 +1,17 @@
[package]
name = "windmill-parser-r"
version.workspace = true
edition.workspace = true
authors.workspace = true
[lib]
name = "windmill_parser_r"
path = "./src/lib.rs"
[dependencies]
windmill-parser.workspace = true
tree-sitter.workspace = true
tree-sitter-r.workspace = true
anyhow.workspace = true
wasm-bindgen.workspace = true
serde_json.workspace = true
@@ -0,0 +1,363 @@
#![cfg_attr(target_arch = "wasm32", feature(c_variadic))]
#[cfg(target_arch = "wasm32")]
pub mod wasm_libc;
use anyhow::anyhow;
use serde_json::Value;
use tree_sitter::Node;
use tree_sitter::Range;
use windmill_parser::json_to_typ;
use windmill_parser::Arg;
use windmill_parser::MainArgSignature;
pub fn parse_r_sig_meta(code: &str) -> anyhow::Result<MainArgSignature> {
let mut parser = tree_sitter::Parser::new();
let language = tree_sitter_r::LANGUAGE;
parser
.set_language(&language.into())
.map_err(|e| anyhow!("Error setting R as language: {e}"))?;
let tree = parser
.parse(code, None)
.ok_or(anyhow!("Failed to parse code"))?;
let root_node = tree.root_node();
let args = find_main_signature(root_node, code)?;
let main_sig = MainArgSignature {
star_args: false,
star_kwargs: false,
args: args.unwrap_or_default(),
has_preprocessor: None,
auto_kind: None,
};
Ok(main_sig)
}
pub fn parse_r_signature(code: &str) -> anyhow::Result<MainArgSignature> {
Ok(parse_r_sig_meta(code)?)
}
/// Extract package names from `library(...)` and `require(...)` calls in R code.
/// Returns a newline-separated list of package names.
pub fn parse_r_requirements(code: &str) -> anyhow::Result<String> {
let mut parser = tree_sitter::Parser::new();
let language = tree_sitter_r::LANGUAGE;
parser
.set_language(&language.into())
.map_err(|e| anyhow!("Error setting R as language: {e}"))?;
let tree = parser
.parse(code, None)
.ok_or(anyhow!("Failed to parse code"))?;
let root_node = tree.root_node();
let mut packages = vec![];
find_library_calls(root_node, code, &mut packages);
// Deduplicate and exclude base packages
packages.sort();
packages.dedup();
packages.retain(|p| !is_base_package(p));
Ok(packages.join("\n"))
}
fn find_library_calls(node: Node, code: &str, packages: &mut Vec<String>) {
let mut cursor = node.walk();
for child in node.children(&mut cursor) {
if child.kind() == "call" {
// call node: child 0 is the function name, child 1 is arguments
if let (Some(func_node), Some(args_node)) = (child.child(0), child.child(1)) {
let func_name = func_node.utf8_text(code.as_bytes()).unwrap_or("");
if func_name == "library" || func_name == "require" {
// AST: arguments → ( + argument → identifier/string + )
if args_node.kind() == "arguments" {
let mut args_cursor = args_node.walk();
for arg in args_node.children(&mut args_cursor) {
if arg.kind() == "argument" {
// The argument node wraps the actual value
if let Some(value_node) = arg.child(0) {
let pkg = value_node
.utf8_text(code.as_bytes())
.unwrap_or("")
.trim_matches('"')
.trim_matches('\'');
if !pkg.is_empty() {
packages.push(pkg.to_string());
}
}
break; // only first arg
}
}
}
}
}
}
// Recurse into children to find nested library() calls
find_library_calls(child, code, packages);
}
}
fn is_base_package(pkg: &str) -> bool {
matches!(
pkg,
"base"
| "compiler"
| "datasets"
| "grDevices"
| "graphics"
| "grid"
| "methods"
| "parallel"
| "splines"
| "stats"
| "stats4"
| "tcltk"
| "tools"
| "utils"
)
}
/// Find the main function signature in R code.
/// R function definitions look like: `main <- function(x, y = 10) { ... }`
/// In the tree-sitter-r AST, this is a `binary_operator` node with:
/// - child 0: identifier "main"
/// - child 1: "<-" or "="
/// - child 2: function_definition node
fn find_main_signature<'a>(root_node: Node<'a>, code: &str) -> anyhow::Result<Option<Vec<Arg>>> {
let mut cursor = root_node.walk();
for x in root_node.children(&mut cursor) {
if x.kind() == "binary_operator" {
let child_count = x.child_count();
if child_count < 3 {
continue;
}
// First child should be identifier "main"
let ident_node = x.child(0).unwrap();
if ident_node.kind() != "identifier" {
continue;
}
let ident = ident_node.utf8_text(code.as_bytes()).unwrap_or("");
if ident != "main" {
continue;
}
// Second child should be "<-" or "="
let op_node = x.child(1).unwrap();
let op = op_node.utf8_text(code.as_bytes()).unwrap_or("");
if op != "<-" && op != "=" {
continue;
}
// Third child should be the function_definition
let func_node = x.child(2).unwrap();
if func_node.kind() != "function_definition" {
continue;
}
return Ok(Some(parse_function_params(func_node, code)?));
}
}
Ok(None)
}
/// Parse parameters from a function_definition node.
/// function_definition has children: "function", parameters, body
/// Each parameter node has:
/// - 1 child (identifier) for positional args
/// - 3 children (identifier, "=", value) for default args
fn parse_function_params(func_node: Node, code: &str) -> anyhow::Result<Vec<Arg>> {
let mut args = vec![];
let mut func_cursor = func_node.walk();
for child in func_node.children(&mut func_cursor) {
if child.kind() == "parameters" {
let mut param_cursor = child.walk();
for param in child.children(&mut param_cursor) {
if param.kind() != "parameter" {
continue;
}
let param_child_count = param.child_count();
if param_child_count == 1 {
// Simple parameter: just identifier
let ident_node = param.child(0).unwrap();
let name = ident_node.utf8_text(code.as_bytes())?;
args.push(Arg { name: name.to_owned(), ..Default::default() });
} else if param_child_count >= 3 {
// Default parameter: identifier = value
let ident_node = param.child(0).unwrap();
let value_node = param.child(2).unwrap();
let name = ident_node.utf8_text(code.as_bytes())?;
let Range { start_byte, end_byte, .. } = value_node.range();
let raw = &code[start_byte..end_byte];
// Convert R literals to JSON
let unparsed = raw
.replace("NULL", "null")
.replace("TRUE", "true")
.replace("FALSE", "false");
match serde_json::from_str::<Value>(&unparsed) {
Ok(default) => {
args.push(Arg {
name: name.to_owned(),
typ: json_to_typ(&default, true),
default: Some(default),
has_default: true,
..Default::default()
});
}
Err(_) => {
args.push(Arg {
name: name.to_owned(),
has_default: true,
..Default::default()
});
}
}
}
}
}
}
Ok(args)
}
#[cfg(test)]
mod test {
use serde_json::json;
use windmill_parser::Typ;
use super::parse_r_sig_meta as parse;
#[test]
fn test_parse_r_no_main() {
let code = r#"
not_main <- function() {}
helper <- function(x) { x + 1 }
"#;
let sig = parse(code).unwrap();
assert_eq!(
sig,
windmill_parser::MainArgSignature { auto_kind: None, ..Default::default() }
);
}
#[test]
fn test_parse_r_no_args() {
let code = r#"
main <- function() {
return(42)
}
"#;
let sig = parse(code).unwrap();
assert_eq!(
sig,
windmill_parser::MainArgSignature { auto_kind: None, ..Default::default() }
);
}
#[test]
fn test_parse_r_positional_args() {
let code = r#"main <- function(a, b, c) { a + b + c }"#;
let sig = parse(code).unwrap();
assert_eq!(
sig,
windmill_parser::MainArgSignature {
args: vec![
windmill_parser::Arg { name: "a".into(), ..Default::default() },
windmill_parser::Arg { name: "b".into(), ..Default::default() },
windmill_parser::Arg { name: "c".into(), ..Default::default() },
],
auto_kind: None,
..Default::default()
}
);
}
#[test]
fn test_parse_r_default_args() {
let code = r#"main <- function(a = 10, b = "hey", c = FALSE) { }"#;
let sig = parse(code).unwrap();
assert_eq!(sig.args.len(), 3);
assert_eq!(sig.args[0].name, "a");
assert_eq!(sig.args[0].default, Some(json!(10)));
assert_eq!(sig.args[0].typ, Typ::Int);
assert_eq!(sig.args[1].name, "b");
assert_eq!(sig.args[1].default, Some(json!("hey")));
assert_eq!(sig.args[1].typ, Typ::Str(None));
assert_eq!(sig.args[2].name, "c");
assert_eq!(sig.args[2].default, Some(json!(false)));
assert_eq!(sig.args[2].typ, Typ::Bool);
}
#[test]
fn test_parse_r_equals_assignment() {
let code = r#"main = function(x, y = 5) { x + y }"#;
let sig = parse(code).unwrap();
assert_eq!(sig.args.len(), 2);
assert_eq!(sig.args[0].name, "x");
assert_eq!(sig.args[1].name, "y");
assert_eq!(sig.args[1].default, Some(json!(5)));
}
#[test]
fn test_parse_r_null_default() {
let code = r#"main <- function(x = NULL) { x }"#;
let sig = parse(code).unwrap();
assert_eq!(sig.args.len(), 1);
assert_eq!(sig.args[0].name, "x");
assert_eq!(sig.args[0].default, Some(json!(null)));
}
#[test]
fn test_parse_r_requirements() {
use super::parse_r_requirements;
let code = r#"
library(dplyr)
library(ggplot2)
require(tidyr)
library(stats)
main <- function(x) {
library(stringr)
x
}
"#;
let reqs = parse_r_requirements(code).unwrap();
let pkgs: Vec<&str> = reqs.lines().collect();
assert!(pkgs.contains(&"dplyr"));
assert!(pkgs.contains(&"ggplot2"));
assert!(pkgs.contains(&"tidyr"));
assert!(pkgs.contains(&"stringr"));
assert!(!pkgs.contains(&"stats")); // base package excluded
}
#[test]
fn test_parse_r_requirements_string_args() {
use super::parse_r_requirements;
let code = r#"
library("data.table")
require("jsonlite")
main <- function() { }
"#;
let reqs = parse_r_requirements(code).unwrap();
let pkgs: Vec<&str> = reqs.lines().collect();
assert!(pkgs.contains(&"data.table"));
assert!(pkgs.contains(&"jsonlite"));
}
#[test]
fn test_parse_r_requirements_no_deps() {
use super::parse_r_requirements;
let code = r#"main <- function(x) { x + 1 }"#;
let reqs = parse_r_requirements(code).unwrap();
assert!(reqs.is_empty());
}
}
@@ -0,0 +1,293 @@
use std::collections::BTreeMap;
use std::sync::{Mutex, OnceLock};
use std::{
alloc::{self, Layout},
ffi::{c_char, c_int, c_void},
mem::align_of,
ptr,
};
use wasm_bindgen::prelude::*;
/* -------------------------------- stdlib.h -------------------------------- */
#[no_mangle]
pub unsafe extern "C" fn abort() {
panic!("Aborted from C");
}
macro_rules! console_log {
($($t:tt)*) => (unsafe { log(&format_args!($($t)*).to_string()) })
}
#[wasm_bindgen]
extern "C" {
#[wasm_bindgen(js_namespace = console)]
fn log(a: &str);
}
#[no_mangle]
pub unsafe extern "C" fn malloc(size: usize) -> *mut c_void {
if size == 0 {
return ptr::null_mut();
}
let (layout, offset_to_data) = layout_for_size_prepended(size);
let buf = alloc::alloc(layout);
store_layout(buf, layout, offset_to_data)
}
#[no_mangle]
pub unsafe extern "C" fn calloc(count: usize, size: usize) -> *mut c_void {
if count == 0 || size == 0 {
return ptr::null_mut();
}
let (layout, offset_to_data) = layout_for_size_prepended(size * count);
let buf = alloc::alloc_zeroed(layout);
store_layout(buf, layout, offset_to_data)
}
#[no_mangle]
pub unsafe extern "C" fn realloc(buf: *mut c_void, new_size: usize) -> *mut c_void {
if buf.is_null() {
malloc(new_size)
} else if new_size == 0 {
free(buf);
ptr::null_mut()
} else {
let (old_buf, old_layout) = retrieve_layout(buf);
let (new_layout, offset_to_data) = layout_for_size_prepended(new_size);
let new_buf = alloc::realloc(old_buf, old_layout, new_layout.size());
store_layout(new_buf, new_layout, offset_to_data)
}
}
#[no_mangle]
pub unsafe extern "C" fn free(buf: *mut c_void) {
if buf.is_null() {
return;
}
let (buf, layout) = retrieve_layout(buf);
alloc::dealloc(buf, layout);
}
// In all these allocations, we store the layout before the data for later retrieval.
// This is because we need to know the layout when deallocating the memory.
// Here are some helper methods for that:
/// Given a pointer to the data, retrieve the layout and the pointer to the layout.
unsafe fn retrieve_layout(buf: *mut c_void) -> (*mut u8, Layout) {
let (_, layout_offset) = Layout::new::<Layout>()
.extend(Layout::from_size_align(0, align_of::<*const u8>() * 2).unwrap())
.unwrap();
let buf = (buf as *mut u8).offset(-(layout_offset as isize));
let layout = *(buf as *mut Layout);
(buf, layout)
}
/// Calculate a layout for a given size with space for storing a layout at the start.
/// Returns the layout and the offset to the data.
fn layout_for_size_prepended(size: usize) -> (Layout, usize) {
Layout::new::<Layout>()
.extend(Layout::from_size_align(size, align_of::<*const u8>() * 2).unwrap())
.unwrap()
}
/// Store a layout in the pointer, returning a pointer to where the data should be stored.
unsafe fn store_layout(buf: *mut u8, layout: Layout, offset_to_data: usize) -> *mut c_void {
*(buf as *mut Layout) = layout;
(buf as *mut u8).offset(offset_to_data as isize) as *mut c_void
}
/* -------------------------------- string.h -------------------------------- */
#[no_mangle]
pub unsafe extern "C" fn strncmp(ptr1: *const c_void, ptr2: *const c_void, n: usize) -> c_int {
let s1 = std::slice::from_raw_parts(ptr1 as *const u8, n);
let s2 = std::slice::from_raw_parts(ptr2 as *const u8, n);
for (a, b) in s1.iter().zip(s2.iter()) {
if *a != *b || *a == 0 {
return (*a as i32) - (*b as i32);
}
}
0
}
// Implementation by AI:
pub type size_t = usize;
use std::slice;
#[no_mangle]
pub unsafe extern "C" fn memchr(haystack: *const c_void, needle: c_int, len: usize) -> *mut c_void {
if haystack.is_null() || len == 0 {
return ptr::null_mut(); // Return null if the input pointer is null or length is zero
}
let needle_byte = needle as u8; // Convert needle to a byte
// Create a pointer to the start of the haystack
let mut current = haystack as *const u8;
// Iterate through the memory block
for _ in 0..len {
if *current == needle_byte {
return current as *mut c_void; // Return the pointer to the found byte
}
current = current.add(1); // Move to the next byte
}
ptr::null_mut() // Return null if the byte was not found
}
#[no_mangle]
pub unsafe extern "C" fn strchr(mut s: *const c_char, c: c_int) -> *mut c_char {
if s.is_null() {
return std::ptr::null_mut(); // Return null if the input string is null
}
let target = c as u8 as char; // Convert c to a char
let mut current = s;
// Iterate through the string until we find the character or reach the end
while *current != 0 {
if *current as u8 as char == target {
return current as *mut c_char; // Return the pointer to the found character
}
current = current.add(1); // Move to the next character
}
std::ptr::null_mut() // Return null if the character was not found
}
// End of AI implemetation
/* -------------------------------- wctype.h -------------------------------- */
#[no_mangle]
pub unsafe extern "C" fn iswspace(c: c_int) -> bool {
char::from_u32(c as u32).map_or(false, |c| c.is_whitespace())
}
#[no_mangle]
pub unsafe extern "C" fn iswalnum(c: c_int) -> bool {
char::from_u32(c as u32).map_or(false, |c| c.is_alphanumeric())
}
// Implementation by AI:
pub type wint_t = u32;
#[no_mangle]
pub extern "C" fn iswdigit(wc: wint_t) -> c_int {
// Check if the character is a digit ('0' to '9')
if wc >= '0' as wint_t && wc <= '9' as wint_t {
return 1; // Return true (1)
}
0 // Return false (0)
}
#[no_mangle]
pub extern "C" fn iswupper(wc: wint_t) -> c_int {
// Check if the character is an uppercase letter ('A' to 'Z')
if wc >= 'A' as wint_t && wc <= 'Z' as wint_t {
return 1; // Return true (1)
}
0 // Return false (0)
}
#[no_mangle]
pub extern "C" fn iswalpha(wc: wint_t) -> c_int {
// Check if the character is an alphabetic character ('A' to 'Z' or 'a' to 'z')
if (wc >= 'A' as wint_t && wc <= 'Z' as wint_t) || (wc >= 'a' as wint_t && wc <= 'z' as wint_t)
{
return 1; // Return true (1)
}
0 // Return false (0)
}
#[no_mangle]
pub extern "C" fn iswlower(wc: wint_t) -> c_int {
// Check if the character is a lowercase letter ('a' to 'z')
if wc >= 'a' as wint_t && wc <= 'z' as wint_t {
return 1; // Return true (1)
}
0 // Return false (0)
}
// End of AI implemetation
/* --------------------------------- time.h --------------------------------- */
#[no_mangle]
pub unsafe extern "C" fn clock() -> u64 {
panic!("clock is not supported");
}
/* --------------------------------- ctype.h -------------------------------- */
#[no_mangle]
pub unsafe extern "C" fn isprint(c: c_int) -> bool {
c >= 32 && c <= 126
}
/* --------------------------------- stdio.h -------------------------------- */
#[no_mangle]
pub unsafe extern "C" fn fprintf(_file: *mut c_void, _format: *const c_void, _args: ...) -> c_int {
panic!("fprintf is not supported");
}
#[no_mangle]
pub unsafe extern "C" fn fputs(_s: *const c_void, _file: *mut c_void) -> c_int {
panic!("fputs is not supported");
}
#[no_mangle]
pub unsafe extern "C" fn fputc(_c: c_int, _file: *mut c_void) -> c_int {
panic!("fputc is not supported");
}
#[no_mangle]
pub unsafe extern "C" fn fdopen(_fd: c_int, _mode: *const c_void) -> *mut c_void {
panic!("fdopen is not supported");
}
#[no_mangle]
pub unsafe extern "C" fn fclose(_file: *mut c_void) -> c_int {
panic!("fclose is not supported");
}
#[no_mangle]
pub unsafe extern "C" fn fwrite(
_ptr: *const c_void,
_size: usize,
_nmemb: usize,
_stream: *mut c_void,
) -> usize {
panic!("fwrite is not supported");
}
#[no_mangle]
pub unsafe extern "C" fn vsnprintf(
_buf: *mut c_char,
_size: usize,
_format: *const c_char,
_args: ...
) -> c_int {
panic!("vsnprintf is not supported");
}
#[no_mangle]
pub extern "C" fn clock_gettime(ptr: usize, new_size: usize) {
panic!("clock_gettime is not supported");
}
// int snprintf( char* restrict buffer, size_t bufsz, const char* restrict format, ... );
#[no_mangle]
pub extern "C" fn snprintf() {
panic!("snprintf is not supported");
}
#[no_mangle]
pub extern "C" fn __assert_fail(_: *const i32, _: *const i32, _: *const i32, _: *const i32) {
panic!("oh no");
}
@@ -38,6 +38,7 @@ csharp-parser = [ "dep:windmill-parser-csharp"]
nu-parser = [ "dep:windmill-parser-nu"]
java-parser = [ "dep:windmill-parser-java"]
ruby-parser = [ "dep:windmill-parser-ruby"]
r-parser = [ "dep:windmill-parser-r"]
wac-parser = [ "dep:windmill-parser-wac"]
asset-parser = [ "dep:windmill-parser-ts-asset", "dep:windmill-parser-py-asset", "dep:windmill-parser-sql-asset"]
py-imports-parser = [ "dep:windmill-parser-py-imports"]
@@ -58,6 +59,7 @@ windmill-parser-csharp = { workspace = true, optional = true }
windmill-parser-nu = { workspace = true, optional = true }
windmill-parser-java = { workspace = true, optional = true }
windmill-parser-ruby = { workspace = true, optional = true }
windmill-parser-r = { workspace = true, optional = true }
windmill-parser-wac = { workspace = true, optional = true }
windmill-parser-ts-asset = { workspace = true, optional = true }
windmill-parser-py-asset = { workspace = true, optional = true }
@@ -55,6 +55,11 @@ const targets = [
desc: "Ruby",
features: "ruby-parser",
env: "tree-sitter",
}, {
ident: "r",
desc: "R",
features: "r-parser",
env: "tree-sitter",
},
{
ident: "wac",
@@ -198,6 +198,12 @@ pub fn parse_ruby(code: &str) -> String {
wrap_sig(windmill_parser_ruby::parse_ruby_signature(code))
}
#[cfg(feature = "r-parser")]
#[wasm_bindgen]
pub fn parse_r(code: &str) -> String {
wrap_sig(windmill_parser_r::parse_r_signature(code))
}
#[cfg(feature = "asset-parser")]
#[wasm_bindgen]
pub fn parse_assets_sql(code: &str) -> String {
@@ -1,5 +1,7 @@
#pragma once
#include <stdint.h>
void *memcpy(void *dest, const void *src, unsigned long n);
void *memmove(void *dest, const void *src, unsigned long n);
void *memset(void *s, int c, unsigned long n);
+19 -12
View File
@@ -51,7 +51,7 @@ use windmill_common::{
MONITOR_LOGS_ON_OBJECT_STORE_SETTING, NO_DEFAULT_MAVEN_SETTING,
NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OAUTH_SETTING, OTEL_SETTING,
OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING,
POWERSHELL_REPO_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
POWERSHELL_REPO_URL_SETTING, PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING,
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RESTART_COORDINATION_SETTING,
RETENTION_PERIOD_SECS_SETTING, RUBY_REPOS_SETTING, SAML_METADATA_SETTING,
SCIM_TOKEN_SETTING, SMTP_SETTING, TEAMS_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
@@ -95,20 +95,21 @@ use windmill_worker::{
BUN_BUNDLE_CACHE_DIR, BUN_CACHE_DIR, CSHARP_CACHE_DIR, DENO_CACHE_DIR, DENO_CACHE_DIR_DEPS,
DENO_CACHE_DIR_NPM, GO_BIN_CACHE_DIR, GO_CACHE_DIR, JAVA_CACHE_DIR, NU_CACHE_DIR,
POWERSHELL_CACHE_DIR, PY310_CACHE_DIR, PY311_CACHE_DIR, PY312_CACHE_DIR, PY313_CACHE_DIR,
RUBY_CACHE_DIR, RUST_CACHE_DIR, TAR_JAVA_CACHE_DIR, UV_CACHE_DIR,
RUBY_CACHE_DIR, RUST_CACHE_DIR, R_CACHE_DIR, TAR_JAVA_CACHE_DIR, UV_CACHE_DIR,
};
use crate::monitor::{
initial_load, load_keep_job_dir, load_metrics_debug_enabled, load_require_preexisting_user,
load_tag_per_workspace_enabled, load_tag_per_workspace_workspaces, monitor_db,
reload_app_workspaced_route_setting, reload_audit_log_retention_days_setting,
reload_base_url_setting, reload_bunfig_install_scopes_setting,
reload_critical_alert_mute_ui_setting, reload_critical_alerts_on_token_expiry_setting,
reload_critical_error_channels_setting, reload_extra_pip_index_url_setting,
reload_http_route_workspaced_route_setting, reload_hub_api_secret_setting,
reload_hub_base_url_setting, reload_instance_events_webhook_setting,
reload_job_default_timeout_setting, reload_job_isolation_setting, reload_jwt_secret_setting,
reload_license_key, reload_npm_config_registry_setting, reload_otel_tracing_proxy_setting,
initial_load, load_keep_job_dir, load_metrics_debug_enabled, load_preview_tags_override,
load_require_preexisting_user, load_tag_per_workspace_enabled,
load_tag_per_workspace_workspaces, monitor_db, reload_app_workspaced_route_setting,
reload_audit_log_retention_days_setting, reload_base_url_setting,
reload_bunfig_install_scopes_setting, reload_critical_alert_mute_ui_setting,
reload_critical_alerts_on_token_expiry_setting, reload_critical_error_channels_setting,
reload_extra_pip_index_url_setting, reload_http_route_workspaced_route_setting,
reload_hub_api_secret_setting, reload_hub_base_url_setting,
reload_instance_events_webhook_setting, reload_job_default_timeout_setting,
reload_job_isolation_setting, reload_jwt_secret_setting, reload_license_key,
reload_npm_config_registry_setting, reload_otel_tracing_proxy_setting,
reload_pip_index_url_setting, reload_retention_period_setting, reload_scim_token_setting,
reload_smtp_config, reload_uv_index_strategy_setting, reload_worker_config, MonitorIteration,
};
@@ -1742,6 +1743,11 @@ async fn process_notify_event(
);
}
}
PREVIEW_TAGS_OVERRIDE_SETTING => {
if let Err(e) = load_preview_tags_override(db).await {
tracing::error!("Error loading preview tags override: {e:#}");
}
}
SMTP_SETTING => {
reload_smtp_config(db).await;
}
@@ -2005,6 +2011,7 @@ pub async fn run_workers(
&*POWERSHELL_CACHE_DIR,
&*JAVA_CACHE_DIR,
&*RUBY_CACHE_DIR,
&*R_CACHE_DIR,
&*TAR_JAVA_CACHE_DIR, // for related places search: ADD_NEW_LANG
] {
DirBuilder::new()
+17 -3
View File
@@ -62,7 +62,7 @@ use windmill_common::{
KEEP_JOB_DIR_SETTING, LICENSE_KEY_SETTING, MONITOR_LOGS_ON_OBJECT_STORE_SETTING,
NPMRC_SETTING, NPM_CONFIG_REGISTRY_SETTING, NUGET_CONFIG_SETTING, OTEL_SETTING,
OTEL_TRACING_PROXY_SETTING, PIP_INDEX_URL_SETTING, POWERSHELL_REPO_PAT_SETTING,
POWERSHELL_REPO_URL_SETTING, REQUEST_SIZE_LIMIT_SETTING,
POWERSHELL_REPO_URL_SETTING, PREVIEW_TAGS_OVERRIDE_SETTING, REQUEST_SIZE_LIMIT_SETTING,
REQUIRE_PREEXISTING_USER_FOR_OAUTH_SETTING, RETENTION_PERIOD_SECS_SETTING,
SAML_METADATA_SETTING, SCIM_TOKEN_SETTING, TIMEOUT_WAIT_RESULT_SETTING,
UV_INDEX_STRATEGY_SETTING,
@@ -79,8 +79,8 @@ use windmill_common::{
load_periodic_bash_script_interval_from_env, load_whitelist_env_vars_from_env,
load_worker_config, reload_custom_tags_setting, store_pull_query,
store_suspended_pull_query, Connection, WorkerConfig, DEFAULT_TAGS_PER_WORKSPACE,
DEFAULT_TAGS_WORKSPACES, INDEXER_CONFIG, SCRIPT_TOKEN_EXPIRY, SMTP_CONFIG, WINDMILL_DIR,
WORKER_CONFIG, WORKER_GROUP,
DEFAULT_TAGS_WORKSPACES, INDEXER_CONFIG, PREVIEW_TAGS_OVERRIDE, SCRIPT_TOKEN_EXPIRY,
SMTP_CONFIG, WINDMILL_DIR, WORKER_CONFIG, WORKER_GROUP,
},
KillpillSender, AUDIT_LOG_RETENTION_DAYS, BASE_URL, CRITICAL_ALERTS_ON_DB_OVERSIZE,
CRITICAL_ALERTS_ON_TOKEN_EXPIRY, CRITICAL_ALERT_MUTE_UI_ENABLED, CRITICAL_ERROR_CHANNELS, DB,
@@ -235,6 +235,10 @@ pub async fn initial_load(
if let Err(e) = load_tag_per_workspace_workspaces(db).await {
tracing::error!("Error loading default tag per workpsace workspaces: {e:#}");
}
if let Err(e) = load_preview_tags_override(db).await {
tracing::error!("Error loading preview tags override: {e:#}");
}
}
if server_mode {
@@ -499,6 +503,16 @@ pub async fn load_tag_per_workspace_workspaces(db: &DB) -> error::Result<()> {
Ok(())
}
pub async fn load_preview_tags_override(db: &DB) -> error::Result<()> {
let value = load_value_from_global_settings(db, PREVIEW_TAGS_OVERRIDE_SETTING).await;
match value {
Ok(Some(serde_json::Value::Bool(t))) => PREVIEW_TAGS_OVERRIDE.store(t, Ordering::Relaxed),
_ => (),
};
Ok(())
}
pub async fn reload_critical_alert_mute_ui_setting(conn: &Connection) -> error::Result<()> {
if let Ok(Some(serde_json::Value::Bool(t))) =
load_value_from_global_settings_with_conn(conn, CRITICAL_ALERT_MUTE_UI_SETTING, true).await
+1 -1
View File
@@ -84,7 +84,7 @@ fi
if [ "$REVERT" == "YES" ]; then
backend_dirpath="${root_dirpath}/backend/"
for ce_file in $(find "${root_dirpath}/backend" -name "*_ee.rs"); do
for ce_file in $(find "${root_dirpath}/backend" \( -name "*_ee.rs" -o -name "ee.rs" \)); do
if [ -L "${ce_file}" ]; then
rm "${ce_file}"
echo "Deleted symlink '${ce_file}'"
+1 -1
View File
@@ -26,7 +26,7 @@ native_trigger_service: nextcloud
request_type: sync, async, sync_sse
runnable_type: ScriptHash, ScriptPath, FlowPath
script_kind: script, trigger, failure, command, approval, preprocessor
script_lang: python3, deno, go, bash, postgresql, nativets, bun, mysql, bigquery, snowflake, graphql, powershell, mssql, php, bunnative, rust, ansible, csharp, oracledb, nu, java, duckdb, ruby
script_lang: python3, deno, go, bash, postgresql, nativets, bun, mysql, bigquery, snowflake, graphql, powershell, mssql, php, bunnative, rust, ansible, csharp, oracledb, nu, java, duckdb, ruby, rlang
trigger_kind: webhook, http, websocket, kafka, email, nats, postgres, sqs, mqtt, gcp, default_email, nextcloud
trigger_mode: enabled, disabled, suspended
workspace_key_kind: cloud
+109
View File
@@ -1081,6 +1081,115 @@ echo "$result"
Ok(())
}
#[cfg(feature = "rlang")]
#[sqlx::test(fixtures("base"))]
async fn test_r_job(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let content = r#"
main <- function(msg) {
return(paste("hello", msg))
}
"#
.to_owned();
let result = RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
language: ScriptLang::Rlang,
cache_ttl: None,
cache_ignore_s3_path: None,
dedicated_worker: None,
concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default()
.into(),
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
}))
.arg("msg", json!("world"))
.run_until_complete(&db, false, port)
.await
.json_result()
.unwrap();
assert_eq!(result, json!("hello world"));
Ok(())
}
#[cfg(feature = "rlang")]
#[sqlx::test(fixtures("base", "wmill_cli_test"))]
async fn test_r_get_variable(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let content = r#"
main <- function() {
return(get_variable("u/test-user/test_var"))
}
"#
.to_owned();
let result = RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
language: ScriptLang::Rlang,
cache_ttl: None,
cache_ignore_s3_path: None,
dedicated_worker: None,
concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default()
.into(),
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
}))
.run_until_complete(&db, false, port)
.await
.json_result()
.unwrap();
assert_eq!(result, json!("hello from variable"));
Ok(())
}
#[cfg(feature = "rlang")]
#[sqlx::test(fixtures("base", "wmill_cli_test"))]
async fn test_r_get_resource(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let content = r#"
main <- function() {
return(get_resource("u/test-user/test_res"))
}
"#
.to_owned();
let result = RunJob::from(JobPayload::Code(RawCode {
hash: None,
content,
path: None,
lock: None,
language: ScriptLang::Rlang,
cache_ttl: None,
cache_ignore_s3_path: None,
dedicated_worker: None,
concurrency_settings: windmill_common::runnable_settings::ConcurrencySettings::default()
.into(),
debouncing_settings: windmill_common::runnable_settings::DebouncingSettings::default(),
}))
.run_until_complete(&db, false, port)
.await
.json_result()
.unwrap();
assert_eq!(result, json!({"host": "localhost", "port": 5432}));
Ok(())
}
#[cfg(feature = "nu")]
#[sqlx::test(fixtures("base"))]
async fn test_nu_job(db: Pool<Postgres>) -> anyhow::Result<()> {
@@ -1,7 +1,6 @@
use serde_json::json;
use sqlx::{Pool, Postgres};
use uuid::Uuid;
use windmill_common::variables::generate_approval_token;
use windmill_test_utils::*;
fn client() -> reqwest::Client {
@@ -249,245 +248,3 @@ async fn test_jobs_unauthed_complex_reachability(db: Pool<Postgres>) -> anyhow::
Ok(())
}
/// Build a minimal FlowValue JSON with a suspend step (with resume_form) followed by an identity
/// step. The suspend step is at index 0, the identity step at index 1. After the suspend step
/// completes, `flow_status.step` = 1 and `approval_step = 0` points to the form.
fn flow_value_with_suspend_form() -> serde_json::Value {
json!({
"modules": [
{
"id": "a",
"value": {"type": "identity"},
"suspend": {
"required_events": 1,
"resume_form": {
"schema": {
"properties": {
"reason": {"type": "string", "description": "Approval reason"}
},
"order": ["reason"]
}
}
}
},
{
"id": "b",
"value": {"type": "identity"}
}
],
"same_worker": false
})
}
/// Build the flow_status JSON for a flow suspended at step 1 (step 0 completed with suspend).
fn flow_status_suspended_at_step_1(step_job_id: Uuid) -> serde_json::Value {
json!({
"step": 1,
"modules": [
{"type": "Success", "id": "a", "job": step_job_id, "skipped": false},
{"type": "WaitingForEvents", "id": "b", "count": 1, "job": step_job_id}
],
"failure_module": {
"parent_module": null,
"type": "WaitingForPriorSteps",
"id": "failure"
},
"cleanup_module": {"flow_jobs_to_clean": []}
})
}
/// Insert a v2_job_queue + v2_job_status pair (v2_job_status has FK to v2_job_queue).
async fn insert_queue_and_status(
db: &Pool<Postgres>,
flow_id: Uuid,
flow_status: &serde_json::Value,
) {
sqlx::query(
"INSERT INTO v2_job_queue (id, workspace_id, scheduled_for)
VALUES ($1, 'test-workspace', now())",
)
.bind(flow_id)
.execute(db)
.await
.unwrap();
sqlx::query(
"INSERT INTO v2_job_status (id, flow_status)
VALUES ($1, $2)",
)
.bind(flow_id)
.bind(flow_status)
.execute(db)
.await
.unwrap();
}
/// Insert a flow job with raw_flow stored in v2_job (RawFlow / FlowPreview path).
async fn insert_suspended_flow_with_raw_flow(
db: &Pool<Postgres>,
raw_flow: &serde_json::Value,
) -> Uuid {
let flow_id = Uuid::new_v4();
let step_job_id = Uuid::new_v4();
let flow_status = flow_status_suspended_at_step_1(step_job_id);
sqlx::query(
"INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, raw_flow)
VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'flowpreview', 'flow', $2)",
)
.bind(flow_id)
.bind(raw_flow)
.execute(db)
.await
.unwrap();
insert_queue_and_status(db, flow_id, &flow_status).await;
flow_id
}
/// Insert a flow job WITHOUT raw_flow but with a matching flow_node entry (FlowNode path).
async fn insert_suspended_flow_node(db: &Pool<Postgres>, flow_value: &serde_json::Value) -> Uuid {
let flow_id = Uuid::new_v4();
let step_job_id = Uuid::new_v4();
let flow_status = flow_status_suspended_at_step_1(step_job_id);
// Insert the flow_node entry first to get its id
let node_id: i64 = sqlx::query_scalar(
"INSERT INTO flow_node (workspace_id, hash, path, flow)
VALUES ('test-workspace', 12345, 'f/test/flow', $1)
RETURNING id",
)
.bind(flow_value)
.fetch_one(db)
.await
.unwrap();
// Insert the job with kind=flownode and runnable_id pointing to the flow_node
sqlx::query(
"INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, runnable_id)
VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'flownode', 'flow', $2)",
)
.bind(flow_id)
.bind(node_id)
.execute(db)
.await
.unwrap();
insert_queue_and_status(db, flow_id, &flow_status).await;
flow_id
}
/// Insert a flow job WITHOUT raw_flow but with runnable_id pointing to flow_node,
/// and NO flow_node entry — simulates the broken state before the fix.
async fn insert_suspended_flow_node_without_node_entry(db: &Pool<Postgres>) -> Uuid {
let flow_id = Uuid::new_v4();
let step_job_id = Uuid::new_v4();
let flow_status = flow_status_suspended_at_step_1(step_job_id);
// Use a non-existent runnable_id — simulates FlowNode with raw_flow=NULL and no fallback
sqlx::query(
"INSERT INTO v2_job (id, workspace_id, created_by, permissioned_as, kind, tag, runnable_id)
VALUES ($1, 'test-workspace', 'test-user', 'u/test-user', 'flownode', 'flow', 99999999)",
)
.bind(flow_id)
.execute(db)
.await
.unwrap();
insert_queue_and_status(db, flow_id, &flow_status).await;
flow_id
}
async fn get_approval_info_response(
port: u16,
db: &Pool<Postgres>,
job_id: Uuid,
) -> serde_json::Value {
let token = generate_approval_token("test-workspace", job_id, db)
.await
.unwrap();
let base = format!("http://localhost:{port}/api/w/test-workspace/jobs_u");
let resp = client()
.get(format!("{base}/flow/approval_info/{job_id}?token={token}"))
.send()
.await
.unwrap();
let status = resp.status().as_u16();
let body = resp.text().await.unwrap();
assert!(
(200..300).contains(&status),
"approval_info returned {status}: {body}",
);
serde_json::from_str(&body).unwrap()
}
/// Test: approval_info returns form_schema for a top-level suspend (raw_flow stored in v2_job).
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_approval_info_form_schema_from_raw_flow(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let flow_value = flow_value_with_suspend_form();
let flow_id = insert_suspended_flow_with_raw_flow(&db, &flow_value).await;
let info = get_approval_info_response(port, &db, flow_id).await;
assert!(
info.get("form_schema").is_some(),
"form_schema should be present for raw_flow path, got: {info}",
);
Ok(())
}
/// Test: approval_info returns form_schema for a FlowNode sub-flow (graph-based branch/loop).
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_approval_info_form_schema_from_flow_node(db: Pool<Postgres>) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
// We need a flow path to exist for the flow_node FK
sqlx::query(
"INSERT INTO flow (workspace_id, path, summary, description, versions, value, edited_by, edited_at, schema)
VALUES ('test-workspace', 'f/test/flow', '', '', '{}', '{}'::jsonb, 'test-user', now(), '{}'::jsonb)",
)
.execute(&db)
.await?;
let flow_value = flow_value_with_suspend_form();
let flow_id = insert_suspended_flow_node(&db, &flow_value).await;
let info = get_approval_info_response(port, &db, flow_id).await;
assert!(
info.get("form_schema").is_some(),
"form_schema should be present for flow_node path, got: {info}",
);
Ok(())
}
/// Test: approval_info returns no form_schema when FlowNode has no matching entry
/// (simulates the pre-fix behavior where flow_node fallback didn't exist).
#[sqlx::test(migrations = "../migrations", fixtures("base"))]
async fn test_approval_info_no_form_when_flow_node_missing(
db: Pool<Postgres>,
) -> anyhow::Result<()> {
initialize_tracing().await;
let server = ApiServer::start(db.clone()).await?;
let port = server.addr.port();
let flow_id = insert_suspended_flow_node_without_node_entry(&db).await;
let info = get_approval_info_response(port, &db, flow_id).await;
assert!(
info.get("form_schema").is_none(),
"form_schema should be absent when no flow definition found, got: {info}",
);
Ok(())
}
@@ -820,6 +820,7 @@ async fn create_script_internal<'c>(
|| ns.language == ScriptLang::Php
|| ns.language == ScriptLang::Java
|| ns.language == ScriptLang::Ruby
|| ns.language == ScriptLang::Rlang
// for related places search: ADD_NEW_LANG
) {
Some(String::new())
+1
View File
@@ -21,6 +21,7 @@ windmill-api-auth.workspace = true
windmill-audit.workspace = true
windmill-git-sync.workspace = true
dashmap.workspace = true
argon2.workspace = true
axum.workspace = true
chrono.workspace = true
+40
View File
@@ -12,6 +12,7 @@ use sqlx::{Postgres, Transaction};
use std::sync::atomic::AtomicBool;
use std::sync::Arc;
use std::sync::LazyLock;
use std::time::Duration;
use windmill_api_auth::ApiAuthed;
@@ -60,6 +61,43 @@ use windmill_git_sync::handle_deployment_metadata;
pub const COOKIE_PATH: &str = "/";
const TOKEN_CREATE_LIMIT_PER_MINUTE: i32 = 10;
struct TokenRateLimitEntry {
count: i32,
minute_bucket: i64,
}
static TOKEN_CREATE_RATE_LIMIT: LazyLock<dashmap::DashMap<String, TokenRateLimitEntry>> =
LazyLock::new(dashmap::DashMap::new);
fn check_token_create_rate_limit(username: &str) -> Result<()> {
if !*CLOUD_HOSTED {
return Ok(());
}
let current_minute = chrono::Utc::now().timestamp() / 60;
let mut entry = TOKEN_CREATE_RATE_LIMIT
.entry(username.to_string())
.or_insert(TokenRateLimitEntry { count: 0, minute_bucket: current_minute });
if entry.minute_bucket != current_minute {
entry.count = 0;
entry.minute_bucket = current_minute;
}
if entry.count >= TOKEN_CREATE_LIMIT_PER_MINUTE {
return Err(Error::Generic(
StatusCode::TOO_MANY_REQUESTS,
"Too many token creation requests. Please try again later.".to_string(),
));
}
entry.count += 1;
Ok(())
}
pub fn workspaced_service() -> Router {
Router::new()
.route("/list", get(list_users))
@@ -1975,6 +2013,8 @@ async fn create_token(
authed: ApiAuthed,
Json(token_config): Json<NewToken>,
) -> Result<(StatusCode, String)> {
check_token_create_rate_limit(&authed.username)?;
let mut tx = db.begin().await?;
let token = create_token_internal(&mut *tx, &db, &authed, token_config).await?;
+2 -1
View File
@@ -1,7 +1,7 @@
openapi: "3.0.3"
info:
version: 1.669.1
version: 1.672.0
title: Windmill API
contact:
@@ -20705,6 +20705,7 @@ components:
nu,
java,
ruby,
rlang,
duckdb,
bunnative,
# for related places search: ADD_NEW_LANG
+93 -82
View File
@@ -2451,6 +2451,10 @@ struct ApprovalInfo {
#[serde(skip_serializing_if = "Option::is_none")]
description: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
default_args: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
enums: Option<serde_json::Value>,
#[serde(skip_serializing_if = "Option::is_none")]
approval_conditions: Option<ApprovalConditions>,
can_approve: bool,
user_auth_required: bool,
@@ -2505,62 +2509,51 @@ async fn get_approval_info(
let is_wac = row.workflow_as_code_status.is_some();
// Extract approval info based on WAC vs classic flow
let (form_schema, description, approval_conditions, hide_cancel) = if is_wac {
let approval_meta = row
.workflow_as_code_status
.as_ref()
.and_then(|v| v.get("_approval"));
let form = approval_meta.and_then(|m| m.get("form").cloned());
let ac = row
.flow_status
.as_ref()
.and_then(|v| v.get("approval_conditions"))
.and_then(|v| serde_json::from_value::<ApprovalConditions>(v.clone()).ok());
(form, None, ac, None)
} else {
let fs = row
.flow_status
.as_ref()
.and_then(|v| serde_json::from_value::<FlowStatus>(v.clone()).ok());
let ac = fs.as_ref().and_then(|s| s.approval_conditions.clone());
let (form_schema, description, default_args, enums, approval_conditions, hide_cancel) =
if is_wac {
let approval_meta = row
.workflow_as_code_status
.as_ref()
.and_then(|v| v.get("_approval"));
let form = approval_meta.and_then(|m| m.get("form").cloned());
let default_args = approval_meta.and_then(|m| m.get("default_args").cloned());
let enums = approval_meta.and_then(|m| m.get("enums").cloned());
let description = approval_meta.and_then(|m| m.get("description").cloned());
let ac = row
.flow_status
.as_ref()
.and_then(|v| v.get("approval_conditions"))
.and_then(|v| serde_json::from_value::<ApprovalConditions>(v.clone()).ok());
(form, description, default_args, enums, ac, None)
} else {
let fs = row
.flow_status
.as_ref()
.and_then(|v| serde_json::from_value::<FlowStatus>(v.clone()).ok());
let ac = fs.as_ref().and_then(|s| s.approval_conditions.clone());
// For classic flows, form/description come from the flow definition and step result
let approval_step = fs.as_ref().map(|s| (s.step as usize).saturating_sub(1));
// For classic flows, form/description come from the flow definition and step result
let approval_step = fs.as_ref().map(|s| (s.step as usize).saturating_sub(1));
// Fetch flow definition to get suspend settings (form schema, hide_cancel).
// Try raw_flow on the job first, fall back to flow_version for deployed flows,
// then flow_node for graph-based branch/loop sub-flows.
let raw_flow: Option<FlowValue> = {
let from_job: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT raw_flow FROM v2_job WHERE id = $1 AND workspace_id = $2",
)
.bind(&job_id)
.bind(&w_id)
.fetch_optional(&db)
.await?
.flatten();
if let Some(v) = from_job {
serde_json::from_value(v).ok()
} else {
// Deployed flow: fetch from flow_version using runnable_id
let from_version: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT fv.value FROM v2_job j JOIN flow_version fv ON fv.id = j.runnable_id \
WHERE j.id = $1 AND j.workspace_id = $2",
// Fetch flow definition to get suspend settings (form schema, hide_cancel).
// Try raw_flow on the job first, fall back to flow_version for deployed flows,
// then flow_node for graph-based branch/loop sub-flows.
let raw_flow: Option<FlowValue> = {
let from_job: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT raw_flow FROM v2_job WHERE id = $1 AND workspace_id = $2",
)
.bind(&job_id)
.bind(&w_id)
.fetch_optional(&db)
.await?
.flatten();
if let Some(v) = from_version {
if let Some(v) = from_job {
serde_json::from_value(v).ok()
} else {
// FlowNode sub-flow (graph-based branch/loop): raw_flow is not stored
// in v2_job for newer versions, fetch from flow_node table
let from_node: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT fn.flow FROM v2_job j \
JOIN flow_node fn ON fn.id = j.runnable_id \
// Deployed flow: fetch from flow_version using runnable_id
let from_version: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT fv.value FROM v2_job j JOIN flow_version fv ON fv.id = j.runnable_id \
WHERE j.id = $1 AND j.workspace_id = $2",
)
.bind(&job_id)
@@ -2568,45 +2561,61 @@ async fn get_approval_info(
.fetch_optional(&db)
.await?
.flatten();
from_node.and_then(|v| serde_json::from_value(v).ok())
if let Some(v) = from_version {
serde_json::from_value(v).ok()
} else {
// FlowNode sub-flow (graph-based branch/loop): raw_flow is not stored
// in v2_job for newer versions, fetch from flow_node table
let from_node: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT fn.flow FROM v2_job j \
JOIN flow_node fn ON fn.id = j.runnable_id \
WHERE j.id = $1 AND j.workspace_id = $2",
)
.bind(&job_id)
.bind(&w_id)
.fetch_optional(&db)
.await?
.flatten();
from_node.and_then(|v| serde_json::from_value(v).ok())
}
}
}
};
let suspend_module = raw_flow
.as_ref()
.and_then(|rf| approval_step.and_then(|s| rf.modules.get(s)));
let suspend_settings = suspend_module.and_then(|m| m.suspend.as_ref());
let form = suspend_settings
.and_then(|s| s.resume_form.as_ref())
.map(|rf| serde_json::json!(rf));
let hc = suspend_settings.map(|s| s.hide_cancel.unwrap_or(false));
// Fetch description, default_args, and enums from the step's completed job result
let step_job_id = fs
.as_ref()
.and_then(|s| approval_step.and_then(|step| s.modules.get(step)))
.and_then(|m| m.job());
let (desc, default_args, enums) = if let Some(sjid) = step_job_id {
let result: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT result FROM v2_job_completed WHERE id = $1 AND workspace_id = $2",
)
.bind(sjid)
.bind(&w_id)
.fetch_optional(&db)
.await?
.flatten();
let desc = result.as_ref().and_then(|r| r.get("description").cloned());
let da = result.as_ref().and_then(|r| r.get("default_args").cloned());
let enums = result.as_ref().and_then(|r| r.get("enums").cloned());
(desc, da, enums)
} else {
(None, None, None)
};
(form, desc, default_args, enums, ac, hc)
};
let suspend_module = raw_flow
.as_ref()
.and_then(|rf| approval_step.and_then(|s| rf.modules.get(s)));
let suspend_settings = suspend_module.and_then(|m| m.suspend.as_ref());
let form = suspend_settings
.and_then(|s| s.resume_form.as_ref())
.map(|rf| serde_json::json!(rf));
let hc = suspend_settings.map(|s| s.hide_cancel.unwrap_or(false));
// Fetch description and default_args from the step's completed job result
let step_job_id = fs
.as_ref()
.and_then(|s| approval_step.and_then(|step| s.modules.get(step)))
.and_then(|m| m.job());
let (desc, _default_args) = if let Some(sjid) = step_job_id {
let result: Option<serde_json::Value> = sqlx::query_scalar(
"SELECT result FROM v2_job_completed WHERE id = $1 AND workspace_id = $2",
)
.bind(sjid)
.bind(&w_id)
.fetch_optional(&db)
.await?
.flatten();
let desc = result.as_ref().and_then(|r| r.get("description").cloned());
let da = result.as_ref().and_then(|r| r.get("default_args").cloned());
(desc, da)
} else {
(None, None)
};
(form, desc, ac, hc)
};
let user_auth_required = approval_conditions
.as_ref()
.map(|ac| ac.user_auth_required)
@@ -2657,6 +2666,8 @@ async fn get_approval_info(
flow_id: row.id,
form_schema,
description,
default_args,
enums,
approval_conditions,
can_approve,
user_auth_required,
@@ -484,6 +484,7 @@ pub(crate) async fn tarball_workspace(
ScriptLang::OracleDB => "odb.sql",
ScriptLang::Java => "java",
ScriptLang::Ruby => "rb",
ScriptLang::Rlang => "r",
// for related places search: ADD_NEW_LANG
};
archive
+72
View File
@@ -1231,3 +1231,75 @@ const _: () = {
}
}
};
#[cfg(test)]
mod tests {
use super::*;
use serde_json::json;
#[test]
fn flow_data_extras_preserves_notes_and_groups() {
let raw = serde_json::value::to_raw_value(&json!({
"modules": [],
"notes": [{"id": "n1", "text": "hello", "color": "blue", "type": "group",
"contained_node_ids": ["a", "b"], "locked": false}],
"groups": [{"start_id": "a", "end_id": "b", "summary": "grp", "color": "green"}]
}))
.unwrap();
let data = FlowData::from_raw(raw).unwrap();
// FlowValue ignores notes/groups
assert!(data.value().modules.is_empty());
// But extras() recovers them from the raw JSON
let extras = data.extras().expect("extras should parse");
let notes: serde_json::Value =
serde_json::from_str(extras.notes.expect("notes present").get()).unwrap();
assert_eq!(notes.as_array().unwrap().len(), 1);
assert_eq!(notes[0]["id"], "n1");
assert_eq!(notes[0]["color"], "blue");
let groups: serde_json::Value =
serde_json::from_str(extras.groups.expect("groups present").get()).unwrap();
assert_eq!(groups.as_array().unwrap().len(), 1);
assert_eq!(groups[0]["start_id"], "a");
}
#[test]
fn flow_data_extras_returns_none_when_missing() {
let raw = serde_json::value::to_raw_value(&json!({"modules": []})).unwrap();
let data = FlowData::from_raw(raw).unwrap();
let extras = data
.extras()
.expect("extras should parse even without notes/groups");
assert!(extras.notes.is_none());
assert!(extras.groups.is_none());
}
#[test]
fn flow_data_extras_lost_after_flow_value_roundtrip() {
// Demonstrates the bug: serializing through FlowValue drops notes/groups.
// This is the root cause of #8641.
let raw = serde_json::value::to_raw_value(&json!({
"modules": [],
"notes": [{"id": "n1", "text": "t", "color": "blue", "type": "free"}]
}))
.unwrap();
let data = FlowData::from_raw(raw).unwrap();
// Re-serialize through FlowValue (what RunFlowDependenciesRequest does)
let stripped = serde_json::to_string(data.value()).unwrap();
let stripped_raw = RawValue::from_string(stripped).unwrap();
let data2 = FlowData::from_raw(stripped_raw).unwrap();
// Notes are gone after the FlowValue round-trip
let extras = data2.extras().expect("extras should parse");
assert!(
extras.notes.is_none(),
"notes lost after FlowValue round-trip"
);
}
}
+114 -1
View File
@@ -8,6 +8,8 @@
pub use windmill_types::flows::*;
use anyhow::Context;
use serde::Deserialize;
use serde::Serialize;
use sqlx::types::Json;
use sqlx::types::JsonRawValue;
@@ -15,10 +17,89 @@ use sqlx::types::JsonRawValue;
use crate::{
cache::{self, FlowExtras},
db::DB,
error::Error,
error::{to_anyhow, Error},
utils::{http_get_from_hub, StripPath},
worker::{to_raw_value, Connection},
DEFAULT_HUB_BASE_URL, HUB_BASE_URL, PRIVATE_HUB_MIN_VERSION,
};
#[derive(Deserialize)]
pub struct HubFlow {
pub value: FlowValue,
}
#[derive(Deserialize)]
struct HubFlowResponse {
flow: HubFlow,
}
fn extract_hub_flow_id_from_path(path: &str) -> Result<i32, Error> {
let hub_flow_path = path.strip_prefix("hub/flows/").ok_or_else(|| {
Error::BadRequest(format!(
"expected hub flow path to start with hub/flows/ (got {path})"
))
})?;
let flow_id = hub_flow_path
.split('/')
.next()
.filter(|segment| !segment.is_empty())
.ok_or_else(|| {
Error::BadRequest(format!(
"expected hub flow path to include a numeric id after hub/flows/ (got {path})"
))
})?;
let flow_id = flow_id.parse::<i32>().map_err(|_| {
Error::BadRequest(format!(
"expected hub flow path to include a numeric id after hub/flows/ (got {path})"
))
})?;
if flow_id <= 0 {
return Err(Error::BadRequest(format!(
"expected hub flow path to include a positive numeric id after hub/flows/ (got {path})"
)));
}
Ok(flow_id)
}
pub async fn get_full_hub_flow_by_path(
path: StripPath,
http_client: &reqwest::Client,
db: Option<&DB>,
) -> crate::error::Result<HubFlow> {
let path = path.to_path();
let flow_id = extract_hub_flow_id_from_path(&path)?;
let hub_base_url = HUB_BASE_URL.read().await.clone();
let hub_url = format!("{hub_base_url}/flows/{flow_id}/json");
let response = match http_get_from_hub(http_client, &hub_url, false, None, db)
.await?
.error_for_status()
.map_err(to_anyhow)
{
Ok(response) => response,
Err(_) if hub_base_url != DEFAULT_HUB_BASE_URL && flow_id < PRIVATE_HUB_MIN_VERSION =>
{
tracing::info!("Not found on private hub, fallback to default hub for hub flow {path}");
let fallback_url = format!("{DEFAULT_HUB_BASE_URL}/flows/{flow_id}/json");
http_get_from_hub(http_client, &fallback_url, false, None, db)
.await?
.error_for_status()
.map_err(to_anyhow)?
}
Err(err) => return Err(err.into()),
};
Ok(response
.json::<HubFlowResponse>()
.await
.context(format!("Decoding hub response for flow at path {path}"))?
.flow)
}
/// Serialize-only wrapper that combines resolved FlowValue with display-only extras.
/// flatten + RawValue is fine for serialization (only deserialization breaks).
#[derive(Serialize)]
@@ -228,4 +309,36 @@ mod tests {
assert!(!output.contains("notes"));
assert!(!output.contains("groups"));
}
#[test]
fn extract_hub_flow_id_accepts_id_only_paths() {
assert_eq!(extract_hub_flow_id_from_path("hub/flows/76").unwrap(), 76);
}
#[test]
fn extract_hub_flow_id_accepts_id_and_slug_paths() {
assert_eq!(
extract_hub_flow_id_from_path("hub/flows/76/send-message-to-company-ai-assistant")
.unwrap(),
76
);
}
#[test]
fn extract_hub_flow_id_rejects_non_numeric_ids() {
let err = extract_hub_flow_id_from_path("hub/flows/send_message").unwrap_err();
assert!(matches!(err, Error::BadRequest(_)));
}
#[test]
fn extract_hub_flow_id_rejects_missing_ids() {
let err = extract_hub_flow_id_from_path("hub/flows/").unwrap_err();
assert!(matches!(err, Error::BadRequest(_)));
}
#[test]
fn extract_hub_flow_id_rejects_zero_ids() {
let err = extract_hub_flow_id_from_path("hub/flows/0").unwrap_err();
assert!(matches!(err, Error::BadRequest(_)));
}
}
@@ -1,6 +1,7 @@
pub const CUSTOM_TAGS_SETTING: &str = "custom_tags";
pub const DEFAULT_TAGS_PER_WORKSPACE_SETTING: &str = "default_tags_per_workspace";
pub const DEFAULT_TAGS_WORKSPACES_SETTING: &str = "default_tags_workspaces";
pub const PREVIEW_TAGS_OVERRIDE_SETTING: &str = "preview_tags_override";
pub const BASE_URL_SETTING: &str = "base_url";
pub const WS_BASE_URL_SETTING: &str = "ws_base_url";
pub const OAUTH_SETTING: &str = "oauths";
@@ -99,6 +100,7 @@ pub const ENV_SETTINGS: &[&str] = &[
"BUNDLE_PATH",
"GEM_PATH",
"RUBY_CONCURRENT_DOWNLOADS",
"RSCRIPT_PATH",
// for related places search: ADD_NEW_LANG
"GOPRIVATE",
"GOPROXY",
@@ -243,6 +243,8 @@ pub struct GlobalSettings {
#[serde(skip_serializing_if = "Option::is_none")]
pub default_tags_per_workspace: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub preview_tags_override: Option<bool>,
#[serde(skip_serializing_if = "Option::is_none")]
pub disable_hub: Option<bool>,
// String settings
@@ -595,6 +597,7 @@ pub enum ScriptLang {
Nu,
Java,
Ruby,
Rlang,
}
// ---------------------------------------------------------------------------
+26 -9
View File
@@ -14,6 +14,7 @@ use crate::{
client::AuthedClient,
db::{AuthedRef, UserDbWithAuthed, DB},
error::{self, to_anyhow, Error},
flows::get_full_hub_flow_by_path,
get_latest_deployed_hash_for_path, get_latest_flow_version_info_for_path,
scripts::{get_full_hub_script_by_path, ScriptHash, ScriptLang},
users::username_to_permissioned_as,
@@ -154,15 +155,31 @@ pub async fn get_payload_tag_from_prefixed_path(
.await?
} else if path.starts_with("flow/") {
let path = path.strip_prefix("flow/").unwrap().to_string();
let FlowVersionInfo { dedicated_worker, tag, version, .. } =
get_latest_flow_version_info_for_path(None, &db, w_id, &path, true).await?;
(
JobPayload::Flow { path, dedicated_worker, apply_preprocessor: false, version },
tag,
None,
None,
None,
)
if path.starts_with("hub/flows/") {
let hub_flow =
get_full_hub_flow_by_path(StripPath(path.clone()), &HTTP_CLIENT, Some(db)).await?;
(
JobPayload::RawFlow {
value: hub_flow.value,
path: Some(path),
restarted_from: None,
},
None,
None,
None,
None,
)
} else {
let FlowVersionInfo { dedicated_worker, tag, version, .. } =
get_latest_flow_version_info_for_path(None, &db, w_id, &path, true).await?;
(
JobPayload::Flow { path, dedicated_worker, apply_preprocessor: false, version },
tag,
None,
None,
None,
)
}
} else {
return Err(Error::BadRequest(format!(
"path must start with script/ or flow/ (got {})",
+9
View File
@@ -184,6 +184,7 @@ lazy_static::lazy_static! {
"nu".to_string(),
"java".to_string(),
"ruby".to_string(),
"rlang".to_string(),
"duckdb".to_string(),
// for related places search: ADD_NEW_LANG
"dependency".to_string(),
@@ -205,6 +206,7 @@ lazy_static::lazy_static! {
pub static ref DEFAULT_TAGS_PER_WORKSPACE: AtomicBool = AtomicBool::new(false);
pub static ref DEFAULT_TAGS_WORKSPACES: Arc<RwLock<Option<Vec<String>>>> = Arc::new(RwLock::new(None));
pub static ref PREVIEW_TAGS_OVERRIDE: AtomicBool = AtomicBool::new(false);
pub static ref MAX_TIMEOUT: u64 = std::env::var("TIMEOUT")
.ok()
@@ -727,6 +729,13 @@ pub struct RubyAnnotations {
pub verbose: bool,
}
#[annotations("#")]
pub struct RlangAnnotations {
pub renv_verbose: bool,
pub renv_install_verbose: bool,
pub sandbox: bool,
}
#[annotations("#")]
pub struct PythonAnnotations {
pub no_cache: bool,
+33 -20
View File
@@ -77,8 +77,8 @@ use windmill_common::{
users::{SUPERADMIN_NOTIFICATION_EMAIL, SUPERADMIN_SECRET_EMAIL},
utils::{not_found_if_none, report_critical_error, StripPath, WarnAfterExt},
worker::{
to_raw_value, CLOUD_HOSTED, DISABLE_FLOW_SCRIPT, NO_LOGS, WORKER_PULL_QUERIES,
WORKER_SUSPENDED_PULL_QUERY,
to_raw_value, CLOUD_HOSTED, DISABLE_FLOW_SCRIPT, NO_LOGS, PREVIEW_TAGS_OVERRIDE,
WORKER_PULL_QUERIES, WORKER_SUSPENDED_PULL_QUERY,
},
DB, METRICS_ENABLED,
};
@@ -3302,7 +3302,10 @@ pub async fn pull(
};
if let Some(job) = job.as_ref() {
if job.is_flow() || job.is_dependency() {
if (job.is_flow() || job.is_dependency())
&& !(job.kind.is_preview()
&& PREVIEW_TAGS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed))
{
let per_workspace = per_workspace_tag(&job.workspace_id).await;
let base_tag = if job.is_flow() {
"flow".to_string()
@@ -5493,25 +5496,35 @@ async fn push_inner<'c, 'd>(
};
interpolated_tag.unwrap_or_else(|| {
language
.as_ref()
.map(|x| {
let tag_lang = if x == &ScriptLang::Bunnative {
if job_kind == JobKind::Dependencies {
ScriptLang::Bun.as_str()
if job_kind.is_preview()
&& PREVIEW_TAGS_OVERRIDE.load(std::sync::atomic::Ordering::Relaxed)
{
if per_workspace {
format!("preview-{}", workspace_id)
} else {
"preview".to_string()
}
} else {
language
.as_ref()
.map(|x| {
let tag_lang = if x == &ScriptLang::Bunnative {
if job_kind == JobKind::Dependencies {
ScriptLang::Bun.as_str()
} else {
ScriptLang::Nativets.as_str()
}
} else {
ScriptLang::Nativets.as_str()
x.as_str()
};
if per_workspace {
format!("{}-{}", tag_lang, workspace_id)
} else {
tag_lang.to_string()
}
} else {
x.as_str()
};
if per_workspace {
format!("{}-{}", tag_lang, workspace_id)
} else {
tag_lang.to_string()
}
})
.unwrap_or_else(default)
})
.unwrap_or_else(default)
}
})
};
@@ -3000,6 +3000,7 @@ var $RawScript = {
"nativets",
"duckdb",
"ruby",
"rlang",
// for related places search: ADD_NEW_LANG
],
},
+10
View File
@@ -575,6 +575,16 @@ pub async fn transform_json_value(
.await?;
Ok(Value::String(v))
}
Value::String(y) if y.starts_with("$jsonvar:") => {
let path = y.strip_prefix("$jsonvar:").unwrap();
let v =
crate::variables::get_value_internal(&db_with_opt_authed, workspace, path, false)
.await?;
serde_json::from_str::<Value>(&v).map_err(|e| {
Error::internal_err(format!("Failed to parse $jsonvar value as JSON: {e}"))
})
}
Value::String(y) if y.starts_with("$res:") => {
let path = y.strip_prefix("$res:").unwrap();
if path.split("/").count() < 2 {
+4
View File
@@ -118,6 +118,10 @@ impl JobKind {
JobKind::FlowDependencies | JobKind::AppDependencies | JobKind::Dependencies
)
}
pub fn is_preview(&self) -> bool {
matches!(self, JobKind::Preview | JobKind::FlowPreview)
}
}
#[derive(sqlx::FromRow, Debug, Serialize, Clone)]
+4 -1
View File
@@ -65,6 +65,7 @@ pub enum ScriptLang {
Nu,
Java,
Ruby,
Rlang,
// for related places search: ADD_NEW_LANG
}
@@ -94,6 +95,7 @@ impl ScriptLang {
ScriptLang::Nu => "nu",
ScriptLang::Java => "java",
ScriptLang::Ruby => "ruby",
ScriptLang::Rlang => "rlang",
// for related places search: ADD_NEW_LANG
}
}
@@ -132,7 +134,7 @@ impl ScriptLang {
use ScriptLang::*;
match self {
Nativets | Bun | Bunnative | Deno | Go | Php | CSharp | Java => "//",
Python3 | Bash | Powershell | Graphql | Ansible | Nu | Ruby => "#",
Python3 | Bash | Powershell | Graphql | Ansible | Nu | Ruby | Rlang => "#",
Postgresql | Mysql | Bigquery | Snowflake | Mssql | OracleDB | DuckDb => "--",
Rust => "//!",
// for related places search: ADD_NEW_LANG
@@ -167,6 +169,7 @@ impl FromStr for ScriptLang {
"nu" => ScriptLang::Nu,
"java" => ScriptLang::Java,
"ruby" => ScriptLang::Ruby,
"rlang" => ScriptLang::Rlang,
// for related places search: ADD_NEW_LANG
language => return Err(anyhow::anyhow!("{} is currently not supported", language)),
};
+2
View File
@@ -36,6 +36,7 @@ rust = ["dep:windmill-parser-rust"]
nu = ["dep:windmill-parser-nu"]
java = ["dep:windmill-parser-java"]
ruby = ["dep:windmill-parser-ruby"]
rlang = ["dep:windmill-parser-r"]
duckdb = ["dep:libloading"]
quickjs = ["windmill-jseval/quickjs"]
bedrock = ["dep:aws-sdk-bedrockruntime", "windmill-common/bedrock", "dep:aws-config", "dep:aws-credential-types", "dep:aws-smithy-types"]
@@ -60,6 +61,7 @@ windmill-parser-csharp = { workspace = true, optional = true }
windmill-parser-nu = { workspace = true, optional = true }
windmill-parser-java = { workspace = true, optional = true }
windmill-parser-ruby = { workspace = true, optional = true }
windmill-parser-r = { workspace = true, optional = true }
windmill-parser-py = { workspace = true, optional = true }
windmill-parser-yaml.workspace = true
windmill-parser-py-imports = { workspace = true, optional = true }
@@ -0,0 +1,100 @@
name: "r install"
mode: ONCE
hostname: "r"
log_level: ERROR
time_limit: 900
disable_rl: true
envar: "HOME=/tmp"
envar: "R_INSTALL_TAR=/usr/bin/tar --no-same-owner"
cwd: "/tmp"
clone_newnet: false
clone_newuser: {CLONE_NEWUSER}
skip_setsid: true
keep_caps: true
keep_env: true
mount_proc: true
mount {
src: "/bin"
dst: "/bin"
is_bind: true
mandatory: false
}
mount {
src: "/lib"
dst: "/lib"
is_bind: true
mandatory: false
}
mount {
src: "/lib64"
dst: "/lib64"
is_bind: true
mandatory: false
}
mount {
src: "/usr"
dst: "/usr"
is_bind: true
mandatory: false
}
mount {
src: "/etc"
dst: "/etc"
is_bind: true
}
mount {
src: "/dev/null"
dst: "/dev/null"
is_bind: true
rw: true
}
mount {
src: "{JOB_DIR}"
dst: "/tmp"
is_bind: true
mandatory: false
rw: true
}
mount {
src: "{PKG_DIR}"
dst: "/install"
is_bind: true
rw: true
}
mount {
src: "/sys/devices/system/cpu"
dst: "/sys/devices/system/cpu"
is_bind: true
mandatory: false
}
mount {
src: "/dev/urandom"
dst: "/dev/urandom"
is_bind: true
}
mount {
src: "{TRACING_PROXY_CA_CERT_PATH}"
dst: "{TRACING_PROXY_CA_CERT_PATH}"
is_bind: true
mandatory: false
}
#{DEV}
@@ -0,0 +1,125 @@
name: "r run script"
mode: ONCE
hostname: "r"
log_level: ERROR
disable_rl: true
cwd: "/tmp"
clone_newnet: false
clone_newuser: {CLONE_NEWUSER}
skip_setsid: true
keep_caps: false
keep_env: true
# mount_proc: true
mount {
src: "/bin"
dst: "/bin"
is_bind: true
mandatory: false
}
mount {
src: "/lib"
dst: "/lib"
is_bind: true
mandatory: false
}
mount {
src: "/lib64"
dst: "/lib64"
is_bind: true
mandatory: false
}
mount {
src: "/usr"
dst: "/usr"
is_bind: true
mandatory: false
}
mount {
src: "/dev/null"
dst: "/dev/null"
is_bind: true
rw: true
}
mount {
dst: "/tmp"
fstype: "tmpfs"
rw: true
options: "size=500000000"
}
mount {
src: "{JOB_DIR}/main.r"
dst: "/tmp/main.r"
is_bind: true
mandatory: false
}
mount {
src: "{JOB_DIR}/args.json"
dst: "/tmp/args.json"
is_bind: true
}
mount {
src: "{JOB_DIR}/result.json"
dst: "/tmp/result.json"
rw: true
is_bind: true
}
mount {
src: "{R_CACHE_DIR}"
dst: "{R_CACHE_DIR}"
is_bind: true
mandatory: false
}
mount {
src: "/etc"
dst: "/etc"
is_bind: true
}
mount {
src: "/sys/devices/system/cpu"
dst: "/sys/devices/system/cpu"
is_bind: true
mandatory: false
}
mount {
src: "/dev/random"
dst: "/dev/random"
is_bind: true
}
mount {
src: "/dev/urandom"
dst: "/dev/urandom"
is_bind: true
}
iface_no_lo: true
{SHARED_MOUNT}
mount {
src: "{TRACING_PROXY_CA_CERT_PATH}"
dst: "{TRACING_PROXY_CA_CERT_PATH}"
is_bind: true
mandatory: false
}
#{DEV}
+10 -1
View File
@@ -145,7 +145,7 @@ pub async fn write_file_binary(dir: &str, path: &str, content: &[u8]) -> error::
}
lazy_static::lazy_static! {
static ref RE_RES_VAR: Regex = Regex::new(r#"\$(?:var|res|encrypted)\:"#).unwrap();
static ref RE_RES_VAR: Regex = Regex::new(r#"\$(?:var|jsonvar|res|encrypted)\:"#).unwrap();
}
pub async fn transform_json<'a>(
@@ -255,6 +255,15 @@ pub async fn transform_json_value(
Error::NotFound(format!("Variable {path} not found for `{name}`: {e:#}"))
})
}
Value::String(y) if y.starts_with("$jsonvar:") => {
let path = y.strip_prefix("$jsonvar:").unwrap();
let v = client.get_variable_value(path).await.map_err(|e| {
Error::NotFound(format!("Variable {path} not found for `{name}`: {e:#}"))
})?;
serde_json::from_str::<serde_json::Value>(&v).map_err(|e| {
Error::internal_err(format!("Failed to parse $jsonvar value as JSON: {e}"))
})
}
Value::String(y) if y.starts_with("$res:") => {
let path = y.strip_prefix("$res:").unwrap();
+3
View File
@@ -17,6 +17,9 @@ mod java_executor;
#[cfg(feature = "ruby")]
mod ruby_executor;
#[cfg(feature = "rlang")]
mod r_executor;
mod ai;
mod ai_executor;
mod bun_executor;
+715
View File
@@ -0,0 +1,715 @@
use std::{collections::HashMap, process::Stdio};
use itertools::Itertools;
use tokio::{
fs::{self, File},
io::{AsyncReadExt, AsyncWriteExt},
process::Command,
};
use uuid::Uuid;
use windmill_common::{
client::AuthedClient,
error::Error,
utils::calculate_hash,
worker::{write_file, Connection, RlangAnnotations},
};
use windmill_parser::Arg;
use windmill_parser_r::{parse_r_requirements, parse_r_signature};
use windmill_queue::{append_logs, CanceledBy, MiniPulledJob};
use crate::{
common::{
build_command_with_isolation, create_args_and_out_file, get_reserved_variables,
read_result, start_child_process, OccupancyMetrics, DEV_CONF_NSJAIL,
},
get_proxy_envs_for_lang,
handle_child::{self},
is_sandboxing_enabled,
universal_pkg_installer::{
par_install_language_dependencies_seq, DependencyGraph, InstallDeps, RequiredDependency,
},
DISABLE_NUSER, NSJAIL_AVAILABLE, NSJAIL_PATH, PATH_ENV, PROXY_ENVS, R_CACHE_DIR,
TRACING_PROXY_CA_CERT_PATH,
};
use windmill_common::scripts::ScriptLang;
lazy_static::lazy_static! {
static ref RSCRIPT_PATH: String = std::env::var("RSCRIPT_PATH").unwrap_or_else(|_| "/usr/bin/Rscript".to_string());
static ref R_CONCURRENT_DOWNLOADS: usize = std::env::var("R_CONCURRENT_DOWNLOADS").ok().map(|flag| flag.parse().unwrap_or(5)).unwrap_or(5);
static ref R_PROXY_ENVS: Vec<(String, String)> = {
PROXY_ENVS
.clone()
.into_iter()
.map(|(k, v)| (k.to_lowercase(), v))
.collect()
};
}
const NSJAIL_CONFIG_RUN_R_CONTENT: &str = include_str!("../nsjail/run.r.config.proto");
const NSJAIL_CONFIG_INSTALL_R_CONTENT: &str = include_str!("../nsjail/install.r.config.proto");
#[allow(dead_code)]
pub(crate) struct JobHandlerInput<'a> {
pub base_internal_url: &'a str,
pub canceled_by: &'a mut Option<CanceledBy>,
pub client: &'a AuthedClient,
pub parent_runnable_path: Option<String>,
pub conn: &'a Connection,
pub envs: HashMap<String, String>,
pub inner_content: &'a str,
pub job: &'a MiniPulledJob,
pub job_dir: &'a str,
pub mem_peak: &'a mut i32,
pub occupancy_metrics: &'a mut OccupancyMetrics,
pub requirements_o: Option<&'a String>,
pub shared_mount: &'a str,
pub worker_name: &'a str,
}
pub async fn handle_r_job<'a>(
mut args: JobHandlerInput<'a>,
) -> Result<Box<sqlx::types::JsonRawValue>, Error> {
let annotation = RlangAnnotations::parse(args.inner_content);
if !std::path::Path::new(RSCRIPT_PATH.as_str()).exists() {
return Err(Error::ExecutionErr(format!(
"Rscript binary not found at '{}'. R is only available in the windmill-full (CE) or windmill-ee-full (EE) Docker images.",
*RSCRIPT_PATH
)));
}
if annotation.sandbox && NSJAIL_AVAILABLE.is_none() {
return Err(Error::ExecutionErr(
"Script has #sandbox annotation but nsjail is not available on this worker. \
Please ensure nsjail is installed or remove the #sandbox annotation."
.to_string(),
));
}
// --- Prepare ---
{
prepare(&args).await?;
}
// --- Resolve lockfile ---
let lockfile = resolve(
&args.job.id,
args.inner_content,
args.mem_peak,
args.canceled_by,
args.job_dir,
args.conn,
args.worker_name,
&args.job.workspace_id,
annotation.renv_verbose,
)
.await?;
// --- Install ---
let lib_path = if !lockfile.is_empty() {
Some(
install(
&mut args,
&lockfile,
annotation.renv_verbose,
annotation.renv_install_verbose,
)
.await?,
)
} else {
None
};
// --- Execute ---
{
run(&mut args, lib_path.as_deref(), annotation.sandbox).await?;
}
// --- Retrieve results ---
{
read_result(&args.job_dir, None).await
}
}
pub async fn prepare<'a>(
JobHandlerInput { job, conn, job_dir, inner_content, client, .. }: &JobHandlerInput<'a>,
) -> Result<(), Error> {
create_args_and_out_file(&client, job, job_dir, conn).await?;
File::create(format!("{}/main.r", job_dir))
.await?
.write_all(&wrap(inner_content)?.into_bytes())
.await?;
// Create windmill client library for R
let wm_lib_path = format!("{}/r_libs", *R_CACHE_DIR);
fs::create_dir_all(&wm_lib_path).await?;
{
File::create(format!("{}/windmill.r", &wm_lib_path))
.await?
.write_all(
r##"
# Windmill mini client methods for R
# Uses base R url() + readLines() to avoid requiring any extra R packages
.wm_fetch_raw <- function(url) {
token <- Sys.getenv("WM_TOKEN")
con <- url(url, headers = c(Authorization = paste("Bearer", token)))
on.exit(close(con))
paste(readLines(con, warn = FALSE), collapse = "\n")
}
get_variable <- function(path) {
base_url <- Sys.getenv("BASE_INTERNAL_URL")
workspace <- Sys.getenv("WM_WORKSPACE")
url <- paste0(base_url, "/api/w/", workspace, "/variables/get_value/", path)
jsonlite::fromJSON(.wm_fetch_raw(url))
}
get_resource <- function(path) {
base_url <- Sys.getenv("BASE_INTERNAL_URL")
workspace <- Sys.getenv("WM_WORKSPACE")
url <- paste0(base_url, "/api/w/", workspace, "/resources/get_value_interpolated/", path)
jsonlite::fromJSON(.wm_fetch_raw(url))
}
"##
.as_bytes(),
)
.await?;
}
Ok(())
}
pub async fn resolve<'a>(
job_id: &Uuid,
inner_content: &str,
mem_peak: &mut i32,
canceled_by: &mut Option<CanceledBy>,
job_dir: &str,
conn: &Connection,
worker_name: &str,
w_id: &str,
verbose: bool,
) -> Result<String, Error> {
let mut packages = parse_r_requirements(inner_content)?;
// jsonlite is always needed by the wrapper for JSON arg parsing and result serialization
let has_jsonlite = packages.lines().any(|l| l.trim() == "jsonlite");
if !has_jsonlite {
if packages.is_empty() {
packages = "jsonlite".to_string();
} else {
packages.push_str("\njsonlite");
}
}
// Check cache
let req_hash = format!("r-{}", calculate_hash(&packages));
if let Some(db) = conn.as_sql() {
if let Some(cached) = sqlx::query_scalar!(
"SELECT lockfile FROM pip_resolution_cache WHERE hash = $1",
req_hash
)
.fetch_optional(db)
.await?
{
return Ok(cached);
}
}
append_logs(
job_id,
w_id,
format!("\n--- RESOLVING R PACKAGES ---\n"),
conn,
)
.await;
// main.r is already written by prepare() and contains the library() calls.
// renv will scan it to detect dependencies.
// Disable renv's own package cache — Windmill manages its own install cache.
let resolve_script = format!(
r#"options(
repos = c(CRAN = "https://cloud.r-project.org"),
renv.verbose = {verbose_r},
renv.config.cache.enabled = FALSE,
renv.config.restart.enabled = FALSE,
renv.config.synchronized.check = FALSE
)
renv::consent(provided = TRUE)
suppressMessages(renv::init(bare = TRUE, restart = FALSE))
suppressMessages(renv::install(prompt = FALSE))
suppressMessages(renv::snapshot(type = "implicit", prompt = FALSE))
"#,
verbose_r = if verbose { "TRUE" } else { "FALSE" },
);
let mut file = File::create(format!("{}/resolve.r", job_dir)).await?;
file.write_all(resolve_script.as_bytes()).await?;
let child = {
let renv_root = format!("{}/renv", *R_CACHE_DIR);
let rscript_executable = if cfg!(windows) {
"Rscript.exe"
} else {
RSCRIPT_PATH.as_str()
};
let mut cmd = Command::new(rscript_executable);
cmd.current_dir(job_dir)
.env("PATH", PATH_ENV.as_str())
.env("RENV_PATHS_ROOT", &renv_root)
.arg("resolve.r")
.stdout(Stdio::piped())
.stderr(Stdio::piped());
start_child_process(cmd, rscript_executable, false).await?
};
handle_child::handle_child(
job_id,
conn,
mem_peak,
canceled_by,
child,
false,
worker_name,
w_id,
"r resolve",
None,
false,
&mut None,
None,
None,
)
.await?;
let lock_path = format!("{}/renv.lock", job_dir);
let mut lock_file = File::open(&lock_path).await?;
let mut lock = String::new();
lock_file.read_to_string(&mut lock).await?;
// Cache the lockfile
if let Some(db) = conn.as_sql() {
sqlx::query!(
"INSERT INTO pip_resolution_cache (hash, lockfile, expiration) VALUES ($1, $2, now() + ('3 days')::interval) ON CONFLICT (hash) DO UPDATE SET lockfile = EXCLUDED.lockfile",
req_hash,
lock.clone(),
).fetch_optional(db).await?;
}
// Log a compact summary instead of the entire renv.lock JSON
let pkg_count = serde_json::from_str::<serde_json::Value>(&lock)
.ok()
.and_then(|v| v.get("Packages")?.as_object().map(|o| o.len()))
.unwrap_or(0);
append_logs(
job_id,
w_id,
format!("resolved {} packages\n", pkg_count),
conn,
)
.await;
Ok(lock)
}
struct RenvPackage {
name: String,
version: String,
repo_url: String,
/// Package names from Imports + Depends fields
dependencies: Vec<String>,
}
/// Parse renv.lock JSON and extract package info including dependency edges.
fn parse_renv_lock(lockfile: &str) -> Result<Vec<RenvPackage>, Error> {
let lock: serde_json::Value = serde_json::from_str(lockfile)
.map_err(|e| Error::ExecutionErr(format!("Failed to parse renv.lock: {}", e)))?;
// Build repo name -> URL map from R.Repositories
let mut repo_urls: HashMap<String, String> = HashMap::new();
if let Some(repos) = lock
.get("R")
.and_then(|r| r.get("Repositories"))
.and_then(|r| r.as_array())
{
for repo in repos {
if let (Some(name), Some(url)) = (
repo.get("Name").and_then(|v| v.as_str()),
repo.get("URL").and_then(|v| v.as_str()),
) {
repo_urls.insert(name.to_string(), url.to_string());
}
}
}
let packages = lock
.get("Packages")
.and_then(|p| p.as_object())
.ok_or_else(|| Error::ExecutionErr("renv.lock missing Packages field".to_string()))?;
let mut result = vec![];
for (_name, pkg) in packages {
let pkg_name = pkg
.get("Package")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
let version = pkg
.get("Version")
.and_then(|v| v.as_str())
.unwrap_or_default()
.to_string();
let repo_name = pkg
.get("Repository")
.and_then(|v| v.as_str())
.unwrap_or("CRAN");
let repo_url = repo_urls
.get(repo_name)
.cloned()
.unwrap_or_else(|| "https://cloud.r-project.org".to_string());
let mut dependencies = vec![];
if let Some(imports) = pkg.get("Imports").and_then(|v| v.as_array()) {
for entry in imports {
if let Some(s) = entry.as_str() {
// Entries look like "cli (>= 3.6.2)" — take just the name
let name = s.split_whitespace().next().unwrap_or("");
if !name.is_empty() && name != "R" {
dependencies.push(name.to_string());
}
}
}
}
// Skip renv itself — it's already loaded and reinstalling it while
// loaded triggers a noisy "Restart your R session" message.
if !pkg_name.is_empty() && !version.is_empty() && pkg_name != "renv" {
result.push(RenvPackage { name: pkg_name, version, repo_url, dependencies });
}
}
Ok(result)
}
async fn install<'a>(
args: &mut JobHandlerInput<'a>,
lockfile: &str,
verbose: bool,
install_verbose: bool,
) -> Result<String, Error> {
let lib_path = format!("{}/r_site_library", *R_CACHE_DIR);
fs::create_dir_all(&lib_path).await?;
let packages = parse_renv_lock(lockfile)?;
if packages.is_empty() {
return Ok(lib_path);
}
#[derive(Clone, Debug)]
struct RPackagePayload {
pkg: String,
version: String,
#[allow(dead_code)]
repo_url: String,
}
// Build dependency graph for topological layering
let mut graph = DependencyGraph::new();
for renv_pkg in &packages {
let handle = format!("{}-{}", renv_pkg.name, renv_pkg.version);
// renv uses staged installation: it builds to a temp dir then rename()s onto
// the target. If the target is a bind mount point, rename fails with
// "target file already exists". We work around this by mounting the parent
// (wrapper) dir at /install so renv can freely create /install/{pkg}/ via rename.
let pkg_outer = format!("{}/{}_outer", lib_path, renv_pkg.name);
let path = format!("{}/{}", pkg_outer, renv_pkg.name);
graph.insert(
renv_pkg.name.clone(),
RequiredDependency {
path,
_s3_handle: handle,
display_name: format!("{} ({})", renv_pkg.name, renv_pkg.version),
custom_payload: RPackagePayload {
pkg: renv_pkg.name.clone(),
version: renv_pkg.version.clone(),
repo_url: renv_pkg.repo_url.clone(),
},
},
renv_pkg.dependencies.clone(),
);
}
let jailed = !cfg!(windows) && is_sandboxing_enabled();
let job_dir = args.job_dir.to_owned();
par_install_language_dependencies_seq(
InstallDeps::Layered(graph),
"r",
"Rscript",
false,
*R_CONCURRENT_DOWNLOADS,
move |dependency| {
let lib_path_c = lib_path.clone();
let job_dir = job_dir.clone();
let pkg_name = &dependency.custom_payload.pkg;
// pkg_outer is the wrapper dir mounted rw at /install inside nsjail.
// renv creates /install/{pkg}/ inside it via staged rename.
let pkg_outer = format!("{}/{}_outer", lib_path_c, pkg_name);
std::fs::create_dir_all(&pkg_outer)?;
let mut cmd = if jailed {
let nsjail_proto = format!("{}.install.config.proto", Uuid::new_v4());
let config_content = NSJAIL_CONFIG_INSTALL_R_CONTENT
.replace("{JOB_DIR}", &job_dir)
.replace("{PKG_DIR}", &pkg_outer)
.replace("{CLONE_NEWUSER}", &(!*DISABLE_NUSER).to_string())
.replace("{TRACING_PROXY_CA_CERT_PATH}", &*TRACING_PROXY_CA_CERT_PATH)
.replace("#{DEV}", DEV_CONF_NSJAIL);
let _ = write_file(
&job_dir,
&nsjail_proto,
&config_content,
)?;
let mut cmd = Command::new(NSJAIL_PATH.as_str());
cmd.args(vec![
"--config",
&nsjail_proto,
"--",
RSCRIPT_PATH.as_str(),
]);
cmd
} else {
Command::new(if cfg!(windows) {
"Rscript.exe"
} else {
RSCRIPT_PATH.as_str()
})
};
let verbose_r = if verbose { "TRUE" } else { "FALSE" };
let install_verbose_r = if install_verbose { "TRUE" } else { "FALSE" };
let install_lib = if jailed { "/install".to_string() } else { pkg_outer.clone() };
cmd.env_clear()
.current_dir(&job_dir)
.env("PATH", PATH_ENV.as_str())
.envs(R_PROXY_ENVS.clone());
cmd
.args(&[
"-e",
&format!(
r#"options(renv.verbose = {verbose_r}, renv.config.install.verbose = {install_verbose_r}, renv.config.restart.enabled = FALSE); renv::install("{pkg}@{version}", library = "{lib}", dependencies = FALSE)"#,
verbose_r = verbose_r,
install_verbose_r = install_verbose_r,
pkg = dependency.custom_payload.pkg,
version = dependency.custom_payload.version,
lib = install_lib,
),
// install.packages fallback (no version pinning):
// &format!(
// r#"install.packages("{pkg}", lib = "{lib}", repos = "{repo}", dependencies = FALSE, quiet = {quiet}, INSTALL_opts = "--no-test-load --no-lock")"#,
// pkg = dependency.custom_payload.pkg,
// lib = install_lib,
// repo = dependency.custom_payload.repo_url,
// quiet = quiet_flag,
// ),
])
.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
Ok(cmd)
},
None,
&args.job.id,
&args.job.workspace_id,
args.worker_name,
jailed,
args.conn,
)
.await?;
Ok(format!("{}/r_site_library", *R_CACHE_DIR))
}
/// Build R_LIBS_USER from lib_path by listing *_outer subdirs.
/// Each package wrapper dir ({pkg}_outer) is added so R finds {pkg}_outer/{pkg}/DESCRIPTION.
fn r_libs_user(lib_path: &str) -> String {
std::fs::read_dir(lib_path)
.into_iter()
.flatten()
.filter_map(|e| e.ok())
.filter(|e| {
e.file_type().map(|t| t.is_dir()).unwrap_or(false)
&& e.file_name().to_string_lossy().ends_with("_outer")
})
.map(|e| e.path().to_string_lossy().to_string())
.collect::<Vec<_>>()
.join(":")
}
async fn run<'a>(
JobHandlerInput {
occupancy_metrics,
mem_peak,
canceled_by,
worker_name,
job,
conn,
job_dir,
shared_mount,
client,
envs,
base_internal_url,
parent_runnable_path,
..
}: &mut JobHandlerInput<'a>,
lib_path: Option<&str>,
sandbox: bool,
) -> Result<(), Error> {
let reserved_variables =
get_reserved_variables(job, &client.token, conn, parent_runnable_path.clone()).await?;
let nsjail = !cfg!(windows) && (is_sandboxing_enabled() || sandbox);
let child = if nsjail {
append_logs(
&job.id,
&job.workspace_id,
"\n--- R CODE EXECUTION (nsjail) ---\n".to_string(),
conn,
)
.await;
write_file(
job_dir,
"run.config.proto",
&NSJAIL_CONFIG_RUN_R_CONTENT
.replace("{JOB_DIR}", job_dir)
.replace("{SHARED_MOUNT}", &shared_mount)
.replace("{R_CACHE_DIR}", &*R_CACHE_DIR)
.replace("{TRACING_PROXY_CA_CERT_PATH}", &*TRACING_PROXY_CA_CERT_PATH)
.replace("#{DEV}", DEV_CONF_NSJAIL)
.replace("{CLONE_NEWUSER}", &(!*DISABLE_NUSER).to_string()),
)?;
let mut cmd = Command::new(NSJAIL_PATH.as_str());
cmd.env_clear()
.current_dir(job_dir)
.env("PATH", PATH_ENV.as_str())
.env("BASE_INTERNAL_URL", base_internal_url)
.envs(envs)
.envs(reserved_variables)
.envs(R_PROXY_ENVS.clone())
.envs(
get_proxy_envs_for_lang(&ScriptLang::Rlang, &job.id, &job.workspace_id, conn)
.await?,
);
if let Some(lp) = lib_path {
cmd.env("R_LIBS_USER", r_libs_user(lp));
}
cmd.args(vec![
"--config",
"run.config.proto",
"--",
RSCRIPT_PATH.as_str(),
"main.r",
]);
cmd.stdout(Stdio::piped()).stderr(Stdio::piped());
start_child_process(cmd, NSJAIL_PATH.as_str(), false).await?
} else {
append_logs(
&job.id,
&job.workspace_id,
format!("\n--- R CODE EXECUTION ---\n"),
conn,
)
.await;
let rscript_executable = if cfg!(windows) {
"Rscript.exe"
} else {
RSCRIPT_PATH.as_str()
};
let args = vec!["main.r"];
let mut cmd = build_command_with_isolation(rscript_executable, &args);
cmd.env_clear()
.current_dir(job_dir.to_owned())
.env("PATH", PATH_ENV.as_str())
.env("BASE_INTERNAL_URL", base_internal_url)
.envs(reserved_variables)
.envs(R_PROXY_ENVS.clone())
.envs(
get_proxy_envs_for_lang(&ScriptLang::Rlang, &job.id, &job.workspace_id, conn)
.await?,
)
.envs(envs);
if let Some(lp) = lib_path {
cmd.env("R_LIBS_USER", r_libs_user(lp));
}
cmd.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
#[cfg(windows)]
{
cmd.env("SystemRoot", crate::SYSTEM_ROOT.as_str())
.env("USERPROFILE", crate::USERPROFILE_ENV.as_str())
.env(
"TMP",
std::env::var("TMP").unwrap_or_else(|_| String::from("/tmp")),
);
}
start_child_process(cmd, rscript_executable, false).await?
};
handle_child::handle_child(
&job.id,
conn,
mem_peak,
canceled_by,
child,
nsjail,
worker_name,
&job.workspace_id,
"r",
job.timeout,
false,
&mut Some(occupancy_metrics),
None,
None,
)
.await?;
Ok(())
}
fn wrap(inner_content: &str) -> Result<String, Error> {
let sig = parse_r_signature(inner_content)?;
let spread = sig
.args
.clone()
.into_iter()
.map(|Arg { name, .. }| format!("{name} = args${name}", name = name))
.collect_vec()
.join(", ");
let wm_lib_path = format!("{}/r_libs/windmill.r", *R_CACHE_DIR);
Ok(format!(
r#"source("{wm_lib_path}")
suppressPackageStartupMessages({{
{inner_content}
}})
library(jsonlite)
args <- fromJSON("args.json")
tryCatch({{
res <- main({spread})
write(toJSON(res, auto_unbox = TRUE, null = "null"), "result.json")
}}, error = function(e) {{
error_obj <- list(
name = class(e)[1],
message = conditionMessage(e),
stack = paste(capture.output(traceback()), collapse = "\n")
)
write(toJSON(error_obj, auto_unbox = TRUE), "result.json")
stop(e)
}})
"#,
wm_lib_path = wm_lib_path,
inner_content = inner_content,
spread = spread,
))
}
+3 -3
View File
@@ -29,7 +29,7 @@ use crate::{
get_proxy_envs_for_lang,
handle_child::{self},
is_sandboxing_enabled, read_ee_registry_url_list_with_workspace_override,
universal_pkg_installer::{par_install_language_dependencies_seq, RequiredDependency},
universal_pkg_installer::{par_install_language_dependencies_seq, InstallDeps, RequiredDependency},
DISABLE_NUSER, NSJAIL_PATH, PATH_ENV, PROXY_ENVS, RUBY_CACHE_DIR, RUBY_REPOS,
TRACING_PROXY_CA_CERT_PATH,
};
@@ -618,7 +618,7 @@ async fn install<'a>(
get_reserved_variables(job, &client.token, conn, parent_runnable_path.clone()).await?,
);
par_install_language_dependencies_seq(
deps.clone(),
InstallDeps::Flat(deps.clone()),
"ruby",
"gem",
false,
@@ -721,7 +721,7 @@ async fn install<'a>(
Ok(cmd)
},
// async move |_| Ok(()),
None,
&job.id,
&job.workspace_id,
worker_name,
@@ -1,3 +1,4 @@
use std::collections::{HashMap, HashSet, VecDeque};
use std::sync::Arc;
use anyhow::bail;
@@ -31,6 +32,153 @@ pub struct RequiredDependency<T: Clone + Send + Sync> {
pub custom_payload: T,
}
/// Generic dependency graph that produces topologically sorted layers via Kahn's algorithm.
/// Each layer's packages only depend on packages from earlier layers, enabling parallel install
/// per layer.
#[allow(dead_code)]
pub struct DependencyGraph<T: Clone + Send + Sync> {
nodes: HashMap<String, RequiredDependency<T>>,
deps: HashMap<String, HashSet<String>>,
}
#[allow(dead_code)]
impl<T: Clone + Send + Sync> DependencyGraph<T> {
pub fn new() -> Self {
Self { nodes: HashMap::new(), deps: HashMap::new() }
}
/// Insert a dependency and the names of packages it depends on.
/// References to packages not in the graph are silently ignored during layering.
pub fn insert(
&mut self,
key: impl Into<String>,
dep: RequiredDependency<T>,
depends_on: Vec<String>,
) {
let key = key.into();
self.nodes.insert(key.clone(), dep);
self.deps.insert(key, depends_on.into_iter().collect());
}
/// Render a dependency tree string. Each package appears once, nested under the first parent
/// that pulls it in. Only includes packages present in `filter` (if provided).
pub fn print_tree(&self, filter: Option<&HashSet<String>>) -> String {
// Find roots: packages nothing else in the graph depends on
let mut depended_on: HashSet<&str> = HashSet::new();
for dep_set in self.deps.values() {
for to in dep_set {
if self.nodes.contains_key(to) {
depended_on.insert(to.as_str());
}
}
}
let roots: Vec<&String> = self
.nodes
.keys()
.filter(|k| !depended_on.contains(k.as_str()))
.filter(|k| filter.map_or(true, |f| f.contains(*k)))
.sorted()
.collect();
let mut out = String::new();
let mut seen = HashSet::new();
for root in roots {
self.print_tree_node(root, 0, &mut seen, filter, &mut out);
}
out
}
fn print_tree_node(
&self,
key: &str,
depth: usize,
seen: &mut HashSet<String>,
filter: Option<&HashSet<String>>,
out: &mut String,
) {
if !seen.insert(key.to_string()) {
return;
}
if let Some(dep) = self.nodes.get(key) {
let indent = " ".repeat(depth);
out.push_str(&format!("{}- {}\n", indent, dep.display_name));
if let Some(children) = self.deps.get(key) {
for child in children.iter().sorted() {
if self.nodes.contains_key(child)
&& filter.map_or(true, |f| f.contains(child))
&& !seen.contains(child)
{
self.print_tree_node(child, depth + 1, seen, filter, out);
}
}
}
}
}
/// Produce topologically sorted layers.
pub fn layers(self) -> Vec<Vec<RequiredDependency<T>>> {
let mut in_degree: HashMap<String, usize> =
self.nodes.keys().map(|k| (k.clone(), 0)).collect();
let mut reverse: HashMap<String, Vec<String>> = HashMap::new();
for (from, dep_set) in &self.deps {
for to in dep_set {
if self.nodes.contains_key(to) {
*in_degree.entry(from.clone()).or_default() += 1;
reverse.entry(to.clone()).or_default().push(from.clone());
}
}
}
let mut queue: VecDeque<String> = in_degree
.iter()
.filter(|(_, &d)| d == 0)
.map(|(k, _)| k.clone())
.sorted()
.collect();
let mut result = vec![];
let mut nodes = self.nodes;
while !queue.is_empty() {
let mut layer = vec![];
let mut next = VecDeque::new();
for key in queue {
if let Some(dep) = nodes.remove(&key) {
layer.push(dep);
}
if let Some(dependents) = reverse.get(&key) {
for d in dependents {
if let Some(deg) = in_degree.get_mut(d) {
*deg -= 1;
if *deg == 0 {
next.push_back(d.clone());
}
}
}
}
}
if !layer.is_empty() {
result.push(layer);
}
queue = next.into_iter().sorted().collect();
}
result
}
}
#[allow(dead_code)]
pub enum InstallDeps<T: Clone + Send + Sync> {
/// Flat list of dependencies — installed in one parallel batch (existing behavior).
Flat(Vec<RequiredDependency<T>>),
/// Dependency graph — split into topological layers, each installed in parallel.
/// A `--- Layer N ---` separator is printed between layers.
Layered(DependencyGraph<T>),
}
#[allow(dead_code)]
pub enum InstallStrategy<T: Clone + Send + Sync> {
/// Will invoke callback to install single dependency
@@ -105,8 +253,10 @@ pub async fn par_install_language_dependencies_all_at_once<
.await;
}
let total_time = std::time::Instant::now();
let (missing, name_max_length) = filter_to_missing(deps, job_id, w_id, jailed, conn).await?;
if missing.is_empty() {
let (layers, name_max_length, total_missing) =
filter_to_missing(InstallDeps::Flat(deps), job_id, w_id, jailed, conn).await?;
let missing: Vec<RequiredDependency<T>> = layers.into_iter().flatten().collect();
if total_missing == 0 {
return Ok(());
}
let to_batch_install = Arc::new(RwLock::new(vec![]));
@@ -122,6 +272,9 @@ pub async fn par_install_language_dependencies_all_at_once<
conn,
_language_name,
_platform_agnostic,
None,
None,
None,
)
.await?;
let installation_res = process_handles(handles, w_id).await;
@@ -231,18 +384,26 @@ pub async fn par_install_language_dependencies_seq<
'a,
T: Clone + std::marker::Send + Sync + 'a + 'static,
>(
deps: Vec<RequiredDependency<T>>,
install_deps: InstallDeps<T>,
_language_name: &'a str,
installer_executable_name: &'a str,
_platform_agnostic: bool,
concurrent_downloads: usize,
callback: impl Fn(RequiredDependency<T>) -> Result<Command, error::Error> + Send + Sync + 'static,
post_install: Option<Arc<dyn Fn(&RequiredDependency<T>) -> anyhow::Result<()> + Send + Sync + 'static>>,
job_id: &'a Uuid,
w_id: &'a str,
worker_name: &'a str,
jailed: bool,
conn: &'a Connection,
) -> anyhow::Result<()> {
let total_time = std::time::Instant::now();
let (layers, name_max_length, total_missing) =
filter_to_missing(install_deps, job_id, w_id, jailed, conn).await?;
if total_missing == 0 {
return Ok(());
}
#[cfg(all(feature = "enterprise", feature = "parquet"))]
let is_not_pro = !matches!(
windmill_common::ee_oss::get_license_plan().await,
@@ -258,65 +419,133 @@ pub async fn par_install_language_dependencies_seq<
)
.await;
}
let total_time = std::time::Instant::now();
let (missing, name_max_length) = filter_to_missing(deps, job_id, w_id, jailed, conn).await?;
if missing.is_empty() {
return Ok(());
}
let handles = spawn_wrapped_installation_threads(
missing,
name_max_length,
InstallStrategy::Single(Arc::new(callback)),
installer_executable_name,
concurrent_downloads,
let is_layered = layers.len() > 1;
let callback = Arc::new(callback);
let mut offset = 0usize;
windmill_queue::append_logs(
job_id,
w_id,
worker_name,
if jailed {
format!(
"\nStarting isolated installation... ({} tasks in parallel)\n",
concurrent_downloads
)
} else {
format!(
"\nStarting installation... ({} tasks in parallel)\n",
concurrent_downloads
)
},
conn,
_language_name,
_platform_agnostic,
)
.await?;
.await;
for (i, layer_deps) in layers.into_iter().enumerate() {
if layer_deps.is_empty() {
continue;
}
if is_layered && offset > 0 {
windmill_queue::append_logs(
job_id,
w_id,
format!("\n\n--- Layer {} ---", i + 1),
conn,
)
.await;
}
let layer_size = layer_deps.len();
tracing::info!("Layer {}: spawning {} installs", i + 1, layer_size);
let handles = spawn_wrapped_installation_threads(
layer_deps,
name_max_length,
InstallStrategy::Single(callback.clone()),
installer_executable_name,
concurrent_downloads,
job_id,
w_id,
worker_name,
conn,
_language_name,
_platform_agnostic,
Some(offset),
Some(total_missing),
post_install.clone(),
)
.await?;
tracing::info!("Layer {}: all spawned, waiting for handles", i + 1);
process_handles(handles, w_id).await?;
tracing::info!("Layer {}: done", i + 1);
offset += layer_size;
}
let installation_res = process_handles(handles, w_id).await;
finish_installation(total_time, job_id, w_id, conn).await;
installation_res
Ok(())
}
type NameMaxLength = usize;
/// Returns (layers of missing deps, name_max_length, total_missing).
/// Prints the "To be installed" header once with all missing packages.
/// For `Layered`, prints a dependency tree; for `Flat`, prints a flat list.
async fn filter_to_missing<'a, T: Clone + std::marker::Send + Sync + 'a + 'static>(
mut deps: Vec<RequiredDependency<T>>,
install_deps: InstallDeps<T>,
job_id: &Uuid,
w_id: &str,
jailed: bool,
conn: &Connection,
) -> anyhow::Result<(Vec<RequiredDependency<T>>, NameMaxLength)> {
// Unique to flatten all same values
deps = deps.into_iter().unique_by(|rd| rd.path.clone()).collect();
// Total to install
let mut missing = vec![];
// Name max length
let mut name_ml = 0;
for rd in deps.into_iter() {
let display_name = rd.display_name.clone();
if rd.path.ends_with("/") {
anyhow::bail!("Internal error: path should not end with '/'")
) -> anyhow::Result<(Vec<Vec<RequiredDependency<T>>>, NameMaxLength, usize)> {
let (mut layers, tree_data) = match install_deps {
InstallDeps::Flat(deps) => (vec![deps], None),
InstallDeps::Layered(graph) => {
let deps_map = graph.deps.clone();
let nodes_display: HashMap<String, String> = graph
.nodes
.iter()
.map(|(k, v)| (k.clone(), v.display_name.clone()))
.collect();
let layers = graph.layers();
(layers, Some((deps_map, nodes_display)))
}
{
// Later will help us align text in log console
if display_name.len() > name_ml {
};
let mut name_ml = 0;
let mut missing_keys: HashSet<String> = HashSet::new();
let mut total_missing = 0;
for layer in layers.iter_mut() {
*layer = std::mem::take(layer)
.into_iter()
.unique_by(|rd| rd.path.clone())
.collect();
let mut missing = vec![];
for rd in std::mem::take(layer) {
if rd.path.ends_with("/") {
anyhow::bail!("Internal error: path should not end with '/'")
}
if rd.display_name.len() > name_ml {
name_ml = rd.display_name.len();
}
if tokio::fs::metadata(rd.path.clone() + ".valid.windmill")
.await
.is_err()
{
if let Some(key) = rd.path.rsplit('/').next() {
missing_keys.insert(key.to_string());
}
missing.push(rd);
}
}
// Will look like: /tmp/windmill/cache/lang/dependency.valid.windmill
if tokio::fs::metadata(rd.path.clone() + ".valid.windmill")
.await
.is_err()
{
missing.push(rd);
}
total_missing += missing.len();
*layer = missing;
}
if !missing.is_empty() {
if total_missing > 0 {
windmill_queue::append_logs(
job_id,
w_id,
@@ -328,15 +557,40 @@ async fn filter_to_missing<'a, T: Clone + std::marker::Send + Sync + 'a + 'stati
conn,
)
.await;
let to_log = missing
.iter()
.map(|rd| format!("- {}", &rd.display_name))
.join("\n")
+ "\n";
let to_log = if let Some((deps_map, nodes_display)) = tree_data {
let mut print_graph: DependencyGraph<()> = DependencyGraph::new();
for (key, display) in &nodes_display {
if missing_keys.contains(key) {
print_graph.insert(
key.clone(),
RequiredDependency {
path: String::new(),
_s3_handle: String::new(),
display_name: display.clone(),
custom_payload: (),
},
deps_map
.get(key)
.map(|s| s.iter().cloned().collect())
.unwrap_or_default(),
);
}
}
print_graph.print_tree(Some(&missing_keys))
} else {
layers
.iter()
.flat_map(|l| l.iter())
.map(|rd| format!("- {}", &rd.display_name))
.join("\n")
+ "\n"
};
windmill_queue::append_logs(job_id, w_id, to_log, conn).await;
}
Ok((missing, name_ml))
Ok((layers, name_ml, total_missing))
}
enum Action<T: Clone + Send + Sync> {
@@ -369,6 +623,9 @@ async fn spawn_wrapped_installation_threads<
conn: &Connection,
_language_name: &str,
_platform_agnostic: bool,
counter_offset: Option<usize>,
total_override: Option<usize>,
post_install: Option<Arc<dyn Fn(&RequiredDependency<T>) -> anyhow::Result<()> + Send + Sync + 'static>>,
) -> anyhow::Result<(
Vec<JoinHandle<anyhow::Result<TaskKiller>>>,
tokio::sync::broadcast::Sender<()>,
@@ -382,11 +639,11 @@ async fn spawn_wrapped_installation_threads<
job_id
);
let (mut handles, semaphore, total_to_install, counter_arc) = (
let total_to_install = total_override.unwrap_or(missing.len());
let (mut handles, semaphore, counter_arc) = (
vec![],
Arc::new(Semaphore::new(parallel_limit)),
missing.len(),
Arc::new(tokio::sync::Mutex::new(0)),
Arc::new(tokio::sync::Mutex::new(counter_offset.unwrap_or(0))),
);
// Pretty sensitive. Single drop will fail installation
@@ -426,6 +683,7 @@ async fn spawn_wrapped_installation_threads<
),
InstallStrategy::AllAtOnce(ref rw_lock) => Action::AddToBulk(Arc::clone(rw_lock)),
};
let post_install_c = post_install.clone();
let task_fut = try_install_one_detached(
dep,
installer_executable_name.to_owned(),
@@ -441,6 +699,7 @@ async fn spawn_wrapped_installation_threads<
_platform_agnostic,
permit,
TaskKiller(kill_tx),
post_install_c,
);
handles.push(tokio::spawn(async move {
tokio::select! {
@@ -513,6 +772,7 @@ async fn try_install_one_detached<'a, T: Clone + std::marker::Send + Sync + 'a +
// If dropped the entire installation fails and all installation threads are being stopped
// That's why we just pass it to return so it is not being dropped
kill_all_tasks: TaskKiller,
post_install: Option<Arc<dyn Fn(&RequiredDependency<T>) -> anyhow::Result<()> + Send + Sync + 'static>>,
) -> anyhow::Result<TaskKiller> {
let start = std::time::Instant::now();
@@ -607,6 +867,9 @@ async fn try_install_one_detached<'a, T: Clone + std::marker::Send + Sync + 'a +
&dep.display_name
));
} else {
if let Some(ref cb) = post_install {
cb(&dep)?;
}
mark_success(dep.path.clone(), &job_id, &w_id).await;
print_success(
false,
+44 -1
View File
@@ -164,6 +164,9 @@ use crate::java_executor::{handle_java_job, JobHandlerInput as JobHandlerInputJa
#[cfg(feature = "ruby")]
use crate::ruby_executor::{handle_ruby_job, JobHandlerInput as JobHandlerInputRuby};
#[cfg(feature = "rlang")]
use crate::r_executor::{handle_r_job, JobHandlerInput as JobHandlerInputRlang};
#[cfg(feature = "php")]
use crate::php_executor::handle_php_job;
@@ -230,6 +233,9 @@ lazy_static::lazy_static! {
// Ruby
pub static ref RUBY_CACHE_DIR: String = format!("{}ruby", *ROOT_CACHE_DIR);
// R
pub static ref R_CACHE_DIR: String = format!("{}rlang", *ROOT_CACHE_DIR);
// for related places search: ADD_NEW_LANG
pub static ref BUN_CACHE_DIR: String = format!("{}bun", *ROOT_CACHE_NOMOUNT_DIR);
pub static ref BUN_BUNDLE_CACHE_DIR: String = format!("{}bun", *ROOT_CACHE_DIR);
@@ -4602,7 +4608,8 @@ mount {{
| ScriptLang::Bash
| ScriptLang::Powershell
| ScriptLang::Ansible
| ScriptLang::Ruby => "#",
| ScriptLang::Ruby
| ScriptLang::Rlang => "#",
ScriptLang::Deno
| ScriptLang::Bun
| ScriptLang::Bunnative
@@ -5114,6 +5121,38 @@ mount {{
.await
}
}
ScriptLang::Rlang => {
#[cfg(not(feature = "rlang"))]
return Err(
anyhow::anyhow!("R is not available because the feature is not enabled").into(),
);
#[cfg(feature = "rlang")]
{
if run_inline {
return Err(Error::internal_err(
"Inline execution is not yet supported for this language".to_string(),
));
}
Box::pin(handle_r_job(JobHandlerInputRlang {
mem_peak,
canceled_by,
job,
conn,
client,
parent_runnable_path,
inner_content: &code,
job_dir,
requirements_o: lock.as_ref(),
shared_mount: &shared_mount,
base_internal_url,
worker_name,
envs,
occupancy_metrics,
}))
.await
}
}
// for related places search: ADD_NEW_LANG
_ => panic!("unreachable, language is not supported: {language:#?}"),
};
@@ -5247,6 +5286,10 @@ pub fn parse_sig_of_lang(
ScriptLang::Ruby => Some(windmill_parser_ruby::parse_ruby_signature(code)?),
#[cfg(not(feature = "ruby"))]
ScriptLang::Ruby => None,
#[cfg(feature = "rlang")]
ScriptLang::Rlang => Some(windmill_parser_r::parse_r_signature(code)?),
#[cfg(not(feature = "rlang"))]
ScriptLang::Rlang => None,
// for related places search: ADD_NEW_LANG
}
} else {
@@ -61,6 +61,8 @@ use crate::csharp_executor::generate_nuget_lockfile;
#[cfg(feature = "java")]
use crate::java_executor;
#[cfg(feature = "rlang")]
use crate::r_executor;
#[cfg(feature = "ruby")]
use crate::ruby_executor;
@@ -2763,6 +2765,21 @@ async fn capture_dependency_job(
)
.await?
}
#[cfg(feature = "rlang")]
ScriptLang::Rlang => {
r_executor::resolve(
job_id,
job_raw_code,
mem_peak,
canceled_by,
job_dir,
&Connection::Sql(db.clone()),
worker_name,
w_id,
false,
)
.await?
}
// for related places search: ADD_NEW_LANG
_ => "".to_owned(),
};
+1 -1
View File
@@ -2,7 +2,7 @@ import { sleep } from "https://deno.land/x/sleep@v1.2.1/mod.ts";
import * as windmill from "https://deno.land/x/windmill@v1.174.0/mod.ts";
import * as api from "https://deno.land/x/windmill@v1.174.0/windmill-api/index.ts";
export const VERSION = "v1.669.1";
export const VERSION = "v1.672.0";
export async function login(email: string, password: string): Promise<string> {
return await windmill.UserService.login({
+5
View File
@@ -134,6 +134,11 @@ public class Main {
def main a, b, c
puts a, b, c
end
`,
rlang: `
main <- function(x, name = "default") {
return(list(result = x, name = name))
}
`,
// for related places search: ADD_NEW_LANG
};
+89 -8
View File
@@ -340,31 +340,108 @@ async function run(
requestBody: input,
});
// Build step label map from raw_flow if available
const stepLabels = new Map<string, string>();
try {
const initialJob = await wmill.getJob({
workspace: workspace.workspaceId,
id,
});
const rawFlow = (initialJob as any).raw_flow;
if (rawFlow?.modules) {
for (const mod of rawFlow.modules) {
if (mod.id) {
const label = mod.summary ? `${mod.id}: ${mod.summary}` : mod.id;
stepLabels.set(mod.id, label);
}
}
}
} catch {
// Best-effort — fall back to module IDs
}
let i = 0;
let lastStatus = "";
while (true) {
const jobInfo = await wmill.getJob({
workspace: workspace.workspaceId,
id,
});
if (jobInfo.flow_status!.modules.length <= i) {
// Check if flow has completed (success or failure)
const isCompleted = (jobInfo as any).type === "CompletedJob";
const flowStatus = jobInfo.flow_status!;
if (flowStatus.modules.length <= i) {
break;
}
const module = jobInfo.flow_status!.modules[i];
const module = flowStatus.modules[i];
if (module.job) {
if (!opts.silent) {
log.info("====== Job " + (i + 1) + " ======");
// If a module has failed, track its job (to show error logs), then break
if (module.type === "Failure") {
if (module.job && !opts.silent) {
const label = stepLabels.get(module.id!) ?? `Step ${i + 1}`;
log.info("====== " + label + " ======");
await track_job(workspace.workspaceId, module.job);
}
break;
}
if (module.job) {
const label = stepLabels.get(module.id!) ?? `Step ${i + 1}`;
const isForLoop = (module as any).flow_jobs !== undefined;
if (isForLoop) {
// For-loop: track iterations as they appear, re-polling until module completes
let trackedIterations = 0;
let forLoopFailed = false;
while (true) {
const refreshed = await wmill.getJob({
workspace: workspace.workspaceId,
id,
});
const refreshedModule = refreshed.flow_status!.modules[i];
const flowJobs = ((refreshedModule as any).flow_jobs as string[] | undefined) ?? [];
// Track any new iterations
while (trackedIterations < flowJobs.length) {
if (!opts.silent) {
log.info(`====== ${label} (iteration ${trackedIterations}) ======`);
await track_job(workspace.workspaceId, flowJobs[trackedIterations]);
}
trackedIterations++;
}
if (refreshedModule.type === "Success" || refreshedModule.type === "Failure") {
forLoopFailed = refreshedModule.type === "Failure";
break;
}
await new Promise((resolve) => setTimeout(resolve, 200));
}
if (forLoopFailed) break;
} else {
if (!opts.silent) {
log.info("====== " + label + " ======");
await track_job(workspace.workspaceId, module.job);
}
}
} else {
if (!opts.silent) {
log.info(module.type);
// Module not started yet — deduplicate status messages
const status = String(module.type);
if (!opts.silent && status !== lastStatus) {
log.info(colors.dim(status));
lastStatus = status;
}
await new Promise((resolve, _) =>
setTimeout(() => resolve(undefined), 100)
);
// If flow already completed while we were waiting, break out
if (isCompleted) break;
continue;
}
lastStatus = "";
i++;
}
@@ -379,7 +456,11 @@ async function run(
});
if (!opts.silent) {
log.info(colors.green.underline.bold("Flow ran to completion"));
if (jobInfo.success === false) {
log.info(colors.red.underline.bold("Flow failed"));
} else {
log.info(colors.green.underline.bold("Flow ran to completion"));
}
log.info("\n");
}
+12 -2
View File
@@ -23,7 +23,7 @@ import { newPathAssigner } from "../../../windmill-utils-internal/src/path-utils
import { generateHash, getHeaders, writeIfChanged } from "../../utils/utils.ts";
import { exts } from "../script/script.ts";
import { FSFSElement } from "../sync/sync.ts";
import { FSFSElement, yamlOptions } from "../sync/sync.ts";
import { Workspace } from "../workspace/workspace.ts";
import { FlowFile } from "./flow.ts";
import { FlowValue } from "../../../gen/types.gen.ts";
@@ -226,6 +226,12 @@ export async function generateFlowLockInternal(
//removeChangedLocks
const tempScriptRefs = tree?.getTempScriptRefs(folderNormalized);
// Preserve notes and groups — the backend round-trips through FlowValue
// which doesn't include these fields, so they'd be lost (#8641).
const savedNotes = flowValue.value.notes;
const savedGroups = flowValue.value.groups;
flowValue.value = await updateFlow(
workspace,
flowValue.value,
@@ -234,6 +240,10 @@ export async function generateFlowLockInternal(
tempScriptRefs
);
// Restore notes and groups that the backend stripped
if (savedNotes !== undefined) flowValue.value.notes = savedNotes;
if (savedGroups !== undefined) flowValue.value.groups = savedGroups;
const lockAssigner = newPathAssigner(opts.defaultTs ?? "bun", {
skipInlineScriptSuffix: getNonDottedPaths(),
});
@@ -257,7 +267,7 @@ export async function generateFlowLockInternal(
// Overwrite `flow.yaml` with the new lockfile references
writeIfChanged(
process.cwd() + SEP + folder + SEP + "flow.yaml",
yamlStringify(flowValue as Record<string, any>)
yamlStringify(flowValue as Record<string, any>, yamlOptions)
);
}
+154 -14
View File
@@ -56,6 +56,8 @@ async function list(
jobKinds?: string;
label?: string;
all?: boolean;
parent?: string;
isFlowStep?: boolean;
}
) {
if (opts.json) log.setSilent(true);
@@ -67,6 +69,12 @@ async function list(
let successFilter = opts.success;
if (opts.failed) successFilter = false;
// When --all or --parent is used, include flow sub-job kinds too
const showSubJobs = opts.all || opts.parent;
const defaultJobKinds = showSubJobs
? "script,flow,singlestepflow,flowscript,flowdependencies"
: "script,flow,singlestepflow";
const limit = Math.min(opts.limit ?? 30, 100);
const allJobs = await wmill.listJobs({
workspace: workspace.workspaceId,
@@ -75,9 +83,11 @@ async function list(
running: opts.running,
success: successFilter,
perPage: limit,
jobKinds: opts.jobKinds ?? "script,flow,singlestepflow",
jobKinds: opts.jobKinds ?? defaultJobKinds,
label: opts.label,
hasNullParent: opts.all ? undefined : true,
hasNullParent: showSubJobs ? undefined : true,
parentJob: opts.parent,
isFlowStep: opts.isFlowStep,
});
// API may return more than perPage — enforce limit client-side
const jobs = allJobs.slice(0, limit);
@@ -108,6 +118,77 @@ async function list(
}
}
function getModuleStatusIcon(type: string, success?: boolean): string {
switch (type) {
case "Success": return colors.green("✓");
case "Failure": return colors.red("✗");
case "InProgress": return colors.blue("▶");
case "WaitingForPriorSteps": return colors.dim("○");
case "WaitingForEvents": return colors.yellow("⏳");
default: return colors.dim("·");
}
}
function formatFlowSteps(
flowStatus: any,
rawFlow: any,
) {
const modules = flowStatus?.modules ?? [];
const rawModules = rawFlow?.modules ?? [];
// Build summary map from raw_flow
const summaryMap = new Map<string, string>();
for (const mod of rawModules) {
if (mod.id && mod.summary) {
summaryMap.set(mod.id, mod.summary);
}
}
console.log(colors.bold("\nSteps:"));
for (const mod of modules) {
const icon = getModuleStatusIcon(mod.type);
const summary = summaryMap.get(mod.id) ?? "";
const label = summary ? `${mod.id}: ${summary}` : mod.id;
const jobId = mod.job ? colors.dim(mod.job) : "";
const flowJobsDuration = mod.flow_jobs_duration;
// For-loop modules: show parent line + iteration sub-lines
const flowJobs = mod.flow_jobs as string[] | undefined;
if (flowJobs && flowJobs.length > 0) {
// Total duration for the for-loop
const totalMs = flowJobsDuration?.duration_ms
? (flowJobsDuration.duration_ms as number[]).reduce((a: number, b: number) => a + b, 0)
: undefined;
const durationStr = totalMs != null ? colors.dim(formatDuration(totalMs)) : "";
console.log(` ${icon} ${label} ${durationStr}`);
const flowJobsSuccess = (mod.flow_jobs_success ?? []) as boolean[];
const durationMs = (flowJobsDuration?.duration_ms ?? []) as number[];
for (let iter = 0; iter < flowJobs.length; iter++) {
const iterSuccess = flowJobsSuccess[iter];
const iterIcon = iterSuccess === true ? colors.green("✓")
: iterSuccess === false ? colors.red("✗")
: colors.dim("·");
const iterDur = durationMs[iter] != null ? colors.dim(formatDuration(durationMs[iter])) : "";
const iterJobId = colors.dim(flowJobs[iter]);
console.log(` ${iterIcon} iteration ${iter} ${iterJobId} ${iterDur}`);
}
} else {
// Regular step
const durationStr = mod.duration_ms != null
? colors.dim(formatDuration(mod.duration_ms))
: "";
console.log(` ${icon} ${label} ${jobId} ${durationStr}`);
}
}
// Show hint for diving into step logs
const hasJobs = modules.some((m: any) => m.job);
if (hasJobs) {
console.log(colors.dim("\nUse 'wmill job logs <job-id>' for step logs"));
}
}
async function get(
opts: GlobalOptions & { json?: boolean },
id: string
@@ -141,8 +222,15 @@ async function get(
if (j.schedule_path) {
console.log(colors.bold("Schedule:") + " " + j.schedule_path);
}
// Flow: show hierarchical step status
const isFlow = j.job_kind === "flow" || j.job_kind === "flowpreview";
if (isFlow && j.flow_status) {
formatFlowSteps(j.flow_status, j.raw_flow);
}
if (j.result !== undefined) {
console.log(colors.bold("Result:"));
console.log(colors.bold("\nResult:"));
console.log(JSON.stringify(j.result, null, 2));
}
}
@@ -173,18 +261,66 @@ async function logs(
const workspace = await resolveWorkspace(opts);
await requireLogin(opts);
// Check if this is a flow job (flows don't have top-level logs)
// Check if this is a flow job — if so, aggregate all step logs
try {
const job = await wmill.getJob({
workspace: workspace.workspaceId,
id,
});
const jobKind = (job as any).job_kind; // job_kind not in generated types yet
if (jobKind === "flow" || jobKind === "flowpreview") {
log.info(colors.yellow(
"Flow jobs don't have direct logs. Each step runs as a separate job.\n" +
"Use 'wmill job list --all' to see sub-jobs, then 'wmill job logs <sub-job-id>' for individual step logs."
));
const j = job as any;
const jobKind = j.job_kind;
if ((jobKind === "flow" || jobKind === "flowpreview") && j.flow_status?.modules) {
const modules = j.flow_status.modules;
const rawModules = j.raw_flow?.modules ?? [];
const summaryMap = new Map<string, string>();
for (const mod of rawModules) {
if (mod.id && mod.summary) summaryMap.set(mod.id, mod.summary);
}
// Strip the "to remove ansi colors" hint that appears in each step's logs
const stripHint = (text: string) =>
text.replace(/^to remove ansi colors.*\n?/gm, "");
let hasLogs = false;
for (const mod of modules) {
const summary = summaryMap.get(mod.id) ?? "";
const label = summary ? `${mod.id}: ${summary}` : mod.id;
// For-loop modules: get logs for each iteration
const flowJobs = mod.flow_jobs as string[] | undefined;
if (flowJobs && flowJobs.length > 0) {
for (let iter = 0; iter < flowJobs.length; iter++) {
try {
const stepLogs = await wmill.getJobLogs({
workspace: workspace.workspaceId,
id: flowJobs[iter],
});
if (stepLogs) {
console.log(colors.bold.cyan(`\n====== ${label} (iteration ${iter}) ======`));
console.log(stripHint(stepLogs));
hasLogs = true;
}
} catch { /* step may not exist yet */ }
}
} else if (mod.job) {
// Regular step
try {
const stepLogs = await wmill.getJobLogs({
workspace: workspace.workspaceId,
id: mod.job,
});
if (stepLogs) {
console.log(colors.bold.cyan(`\n====== ${label} ======`));
console.log(stripHint(stepLogs));
hasLogs = true;
}
} catch { /* step may not exist yet */ }
}
}
if (!hasLogs) {
log.info("No logs available for this flow's steps.");
}
return;
}
} catch {
@@ -199,8 +335,10 @@ async function logs(
if (jobLogs == null || jobLogs === "") {
log.info("No logs available for this job.");
} else {
// Strip the hint if the API already includes it, then print it once to stderr
const stripped = jobLogs.replace(/^to remove ansi colors.*\n?/gm, "");
console.error("to remove ansi colors, use: | sed 's/\\x1B\\[[0-9;]\\{1,\\}[A-Za-z]//g'");
console.log(jobLogs);
console.log(stripped);
}
}
@@ -235,21 +373,23 @@ const listOptions = (cmd: Command) =>
.option("--limit <limit:number>", "Number of jobs to return (default 30, max 100)")
.option("--job-kinds <jobKinds:string>", "Filter by job kinds (default: script,flow,singlestepflow)")
.option("--label <label:string>", "Filter by job label")
.option("--all", "Include sub-jobs (flow steps). By default only top-level jobs are shown");
.option("--all", "Include sub-jobs (flow steps). By default only top-level jobs are shown")
.option("--parent <parent:string>", "Filter by parent job ID (show sub-jobs of a specific flow)")
.option("--is-flow-step", "Show only flow step jobs");
const command = listOptions(new Command()
.description("Manage jobs (list, inspect, cancel)"))
.action(list as any)
.command("list", listOptions(new Command().description("List recent jobs")))
.action(list as any)
.command("get", "Get job details and result")
.command("get", "Get job details. For flows: shows step tree with sub-job IDs")
.arguments("<id:string>")
.option("--json", "Output as JSON (for piping to jq)")
.action(get as any)
.command("result", "Get the result of a completed job (machine-friendly)")
.arguments("<id:string>")
.action(result as any)
.command("logs", "Get job logs")
.command("logs", "Get job logs. For flows: aggregates all step logs")
.arguments("<id:string>")
.action(logs as any)
.command("cancel", "Cancel a running or queued job")
+3
View File
@@ -832,6 +832,8 @@ export function filePathExtensionFromContentType(
return ".java";
} else if (language === "ruby") {
return ".rb";
} else if (language === "rlang") {
return ".r";
// for related places search: ADD_NEW_LANG
} else {
throw new Error("Invalid language: " + language);
@@ -863,6 +865,7 @@ export const exts = [
".playbook.yml",
".java",
".rb",
".r",
// for related places search: ADD_NEW_LANG
];
+27 -3
View File
@@ -1306,6 +1306,7 @@ export async function elementsToMap(
"nu",
"java",
"rb",
"r",
// for related places search: ADD_NEW_LANG
].includes(path.split(".").pop() ?? "")
) {
@@ -2800,9 +2801,32 @@ export async function push(
}
}
for (const folderName of folderNames) {
try {
await stat(path.join("f", folderName, "folder.meta.yaml"));
} catch {
const basePath = path.join("f", folderName, "folder.meta.yaml");
const branchPath = getBranchSpecificPath(
`f/${folderName}/folder.meta.yaml`,
specificItems,
opts.branch,
);
let found = false;
// Check branch-specific variant first (e.g. folder.dev.meta.yaml)
if (branchPath) {
try {
await stat(branchPath);
found = true;
} catch {
// fall through to base path check
}
}
// Then check base path
if (!found) {
try {
await stat(basePath);
found = true;
} catch {
// not found
}
}
if (!found) {
missingFolders.push(folderName);
}
}
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -78,7 +78,7 @@ export {
token,
};
export const VERSION = "1.669.1";
export const VERSION = "1.672.0";
// Re-exported from constants.ts to maintain backwards compatibility
export { WM_FORK_PREFIX } from "./core/constants.ts";
+1
View File
@@ -300,6 +300,7 @@ export function getTypeStrFromPath(
parsed.ext == ".nu" ||
parsed.ext == ".java" ||
parsed.ext == ".rb" ||
parsed.ext == ".r" ||
// for related places search: ADD_NEW_LANG
(parsed.ext == ".yml" && parsed.name.split(".").pop() == "playbook")
) {
+3
View File
@@ -868,6 +868,9 @@ export async function inferSchema(
} else if (language === "ruby") {
const { parse_ruby } = await loadParser("windmill-parser-wasm-ruby");
inferedSchema = JSON.parse(parse_ruby(content));
} else if (language === "rlang") {
const { parse_r } = await loadParser("windmill-parser-wasm-r");
inferedSchema = JSON.parse(parse_r(content));
// for related places search: ADD_NEW_LANG
} else {
throw new Error("Invalid language: " + language);
+3
View File
@@ -20,6 +20,7 @@ export type ScriptLanguage =
| "nu"
| "ansible"
| "ruby"
| "rlang"
| "java";
// for related places search: ADD_NEW_LANG
@@ -105,6 +106,8 @@ export function inferContentTypeFromFilePath(
return "java";
} else if (contentPath.endsWith(".rb")) {
return "ruby";
} else if (contentPath.endsWith(".r")) {
return "rlang";
// for related places search: ADD_NEW_LANG
} else {
throw new Error(
+219
View File
@@ -0,0 +1,219 @@
/**
* Unit tests for flow notes/groups preservation and YAML field ordering.
*
* Verifies that:
* - Notes and groups survive a round-trip through generate-metadata (#8641)
* - YAML output uses consistent field ordering via yamlOptions
*
* No backend required tests the YAML parse/stringify layer.
*/
import { expect, test, describe } from "bun:test";
import { yamlParseContent } from "../src/utils/yaml.ts";
import { stringify as yamlStringify } from "yaml";
import { yamlOptions } from "../src/commands/sync/sync.ts";
const FLOW_WITH_NOTES = `
summary: Sync item
description: ''
value:
modules:
- id: fetch
summary: Fetch product
value:
type: script
input_transforms:
connection:
type: static
value: some_resource
is_trigger: false
path: f/api/product_get
- id: map
summary: Map item
value:
type: script
input_transforms:
bc_item:
type: javascript
expr: flow_input.bc_item
is_trigger: false
path: f/mapping/item_to_product
notes:
- id: note-abc123
type: group
color: blue
contained_node_ids:
- fetch
- map
locked: false
text: These steps must run together
schema:
$schema: https://json-schema.org/draft/2020-12/schema
type: object
`;
const FLOW_WITH_GROUPS = `
summary: Test flow
description: ''
value:
modules:
- id: a
value:
type: identity
groups:
- summary: My group
start_id: a
end_id: a
color: green
schema:
type: object
`;
describe("flow notes preservation (#8641)", () => {
test("notes survive YAML round-trip with yamlOptions", () => {
const parsed = yamlParseContent("flow.yaml", FLOW_WITH_NOTES);
// Verify notes were parsed
expect(parsed.value.notes).toBeDefined();
expect(parsed.value.notes).toHaveLength(1);
expect(parsed.value.notes[0].id).toBe("note-abc123");
expect(parsed.value.notes[0].color).toBe("blue");
expect(parsed.value.notes[0].text).toBe("These steps must run together");
// Simulate the generate-metadata round-trip:
// 1. Backend returns a new value WITHOUT notes (like FlowValue does)
const backendResponse = { ...parsed.value };
delete backendResponse.notes;
// 2. CLI preserves notes (our fix)
const savedNotes = parsed.value.notes;
parsed.value = backendResponse;
if (savedNotes !== undefined) parsed.value.notes = savedNotes;
// 3. Serialize back to YAML
const output = yamlStringify(parsed, yamlOptions);
// 4. Re-parse and verify notes are intact
const reparsed = yamlParseContent("flow.yaml", output);
expect(reparsed.value.notes).toBeDefined();
expect(reparsed.value.notes).toHaveLength(1);
expect(reparsed.value.notes[0].id).toBe("note-abc123");
expect(reparsed.value.notes[0].color).toBe("blue");
expect(reparsed.value.notes[0].contained_node_ids).toEqual(["fetch", "map"]);
expect(reparsed.value.notes[0].text).toBe("These steps must run together");
});
test("groups survive YAML round-trip with yamlOptions", () => {
const parsed = yamlParseContent("flow.yaml", FLOW_WITH_GROUPS);
expect(parsed.value.groups).toBeDefined();
expect(parsed.value.groups).toHaveLength(1);
expect(parsed.value.groups[0].summary).toBe("My group");
// Simulate backend stripping groups
const backendResponse = { ...parsed.value };
delete backendResponse.groups;
const savedGroups = parsed.value.groups;
parsed.value = backendResponse;
if (savedGroups !== undefined) parsed.value.groups = savedGroups;
const output = yamlStringify(parsed, yamlOptions);
const reparsed = yamlParseContent("flow.yaml", output);
expect(reparsed.value.groups).toBeDefined();
expect(reparsed.value.groups).toHaveLength(1);
expect(reparsed.value.groups[0].summary).toBe("My group");
});
test("flow without notes or groups is unaffected", () => {
const yaml = `
summary: Simple flow
value:
modules:
- id: a
value:
type: identity
schema:
type: object
`;
const parsed = yamlParseContent("flow.yaml", yaml);
expect(parsed.value.notes).toBeUndefined();
expect(parsed.value.groups).toBeUndefined();
// Simulate the save/restore logic with undefined
const savedNotes = parsed.value.notes;
const savedGroups = parsed.value.groups;
// Replace value (simulating backend response)
parsed.value = { ...parsed.value };
if (savedNotes !== undefined) parsed.value.notes = savedNotes;
if (savedGroups !== undefined) parsed.value.groups = savedGroups;
const output = yamlStringify(parsed, yamlOptions);
const reparsed = yamlParseContent("flow.yaml", output);
expect(reparsed.value.notes).toBeUndefined();
expect(reparsed.value.groups).toBeUndefined();
});
});
describe("flow YAML field ordering", () => {
test("yamlOptions produces consistent field order for flow modules", () => {
// Simulate a flow value with fields in random order (like backend response)
const unordered = {
summary: "Test",
value: {
modules: [
{
value: { type: "script", path: "f/test", is_trigger: false, input_transforms: {} },
id: "step1",
summary: "Step 1",
},
],
},
schema: { type: "object" },
description: "",
};
const output = yamlStringify(unordered, yamlOptions);
// With yamlOptions, 'id' should come before 'summary' and 'value'
// because prioritizeName gives "id" → "aa", "summary" → "ad", "value" → "ah"
// Note: YAML sequence items start with "- id:" on the first key
const lines = output.split("\n");
const idLine = lines.findIndex((l) => /^\s*-?\s*id:/.test(l));
const summaryLine = lines.findIndex((l, i) => i > idLine && /^\s+summary:/.test(l));
expect(idLine).toBeGreaterThan(-1);
expect(summaryLine).toBeGreaterThan(idLine);
});
test("yamlOptions produces same output regardless of input key order", () => {
const order1 = {
summary: "Flow",
description: "",
value: { modules: [{ id: "a", summary: "S", value: { type: "identity" } }] },
schema: { type: "object" },
};
const order2 = {
schema: { type: "object" },
value: { modules: [{ value: { type: "identity" }, summary: "S", id: "a" }] },
description: "",
summary: "Flow",
};
const output1 = yamlStringify(order1, yamlOptions);
const output2 = yamlStringify(order2, yamlOptions);
expect(output1).toBe(output2);
});
test("notes field is preserved in correct position after modules", () => {
const parsed = yamlParseContent("flow.yaml", FLOW_WITH_NOTES);
const output = yamlStringify(parsed, yamlOptions);
// 'modules' should appear before 'notes' in the output
const modulesIdx = output.indexOf("modules:");
const notesIdx = output.indexOf("notes:");
expect(modulesIdx).toBeGreaterThan(-1);
expect(notesIdx).toBeGreaterThan(-1);
expect(modulesIdx).toBeLessThan(notesIdx);
});
});
+35
View File
@@ -397,4 +397,39 @@ describe("sync push missing folder detection", () => {
expect(output).not.toContain("Missing folder.meta.yaml");
});
});
test("no warning when branch-specific folder.meta.yaml exists", async () => {
await withIsolatedWorkspace(async ({ tempDir, runCLICommand }) => {
const uniqueId = Date.now();
const folderName = `branchmeta${uniqueId}`;
// wmill.yaml with branch-specific folders configured
await writeFile(
join(tempDir, "wmill.yaml"),
`defaultTs: bun\nincludes:\n - "**"\nexcludes: []\ngitBranches:\n dev:\n specificItems:\n folders:\n - "f/${folderName}"\n`,
"utf-8"
);
// Create folder with branch-specific meta only (no base folder.meta.yaml)
await mkdir(join(tempDir, "f", folderName), { recursive: true });
await writeFile(
join(tempDir, "f", folderName, "folder.dev.meta.yaml"),
`summary: ""\ndisplay_name: "${folderName}"\nowners: []\nextra_perms: {}\n`,
"utf-8"
);
await writeFile(
join(tempDir, "f", folderName, "test_script.ts"),
'export async function main() { return "hello"; }',
"utf-8"
);
const result = await runCLICommand(
["sync", "push", "--yes", "--branch", "dev", "--includes", `f/${folderName}/**`],
);
expect(result.code).toEqual(0);
const output = result.stdout + result.stderr;
expect(output).not.toContain("Missing folder.meta.yaml");
});
});
});
+131 -3
View File
@@ -9,6 +9,8 @@ import {
setupWorkspaceProfile,
createRemoteScript,
createRemoteFlow,
createRemoteMultiStepFlow,
createRemoteFailingFlow,
runRemoteScript,
runRemoteFlow,
waitForJob,
@@ -169,13 +171,13 @@ describe("job command", () => {
});
});
test("job logs for flow job shows helpful message", async () => {
test("job logs for flow job aggregates step logs", async () => {
await withTestBackend(async (backend, tempDir) => {
await setupWorkspaceProfile(backend);
const uniqueId = Date.now();
const flowPath = `f/test/flow_logs_${uniqueId}`;
await createRemoteFlow(backend, flowPath);
await createRemoteMultiStepFlow(backend, flowPath);
const jobId = await runRemoteFlow(backend, flowPath);
await waitForJob(backend, jobId);
@@ -185,7 +187,9 @@ describe("job command", () => {
);
expect(result.code).toEqual(0);
expect(result.stdout).toContain("Flow jobs don't have direct logs");
// Should show labeled step headers instead of "no direct logs"
expect(result.stdout).toContain("======");
expect(result.stdout).toContain("a: Generate data");
});
});
@@ -215,6 +219,130 @@ describe("job command", () => {
expect(output).toContain("cancel");
expect(output).toContain("--failed");
expect(output).toContain("--running");
expect(output).toContain("--parent");
expect(output).toContain("--is-flow-step");
});
});
test("job get for flow shows hierarchical step tree", async () => {
await withTestBackend(async (backend, tempDir) => {
await setupWorkspaceProfile(backend);
const uniqueId = Date.now();
const flowPath = `f/test/flow_get_${uniqueId}`;
await createRemoteMultiStepFlow(backend, flowPath);
const jobId = await runRemoteFlow(backend, flowPath);
await waitForJob(backend, jobId);
const result = await backend.runCLICommand(
["job", "get", jobId],
tempDir
);
expect(result.code).toEqual(0);
expect(result.stdout).toContain("Steps:");
// Should show step IDs from the flow definition
expect(result.stdout).toContain("a");
expect(result.stdout).toContain("b");
// Should show status icons (✓ for success)
expect(result.stdout).toContain("✓");
});
});
test("job get --json for flow includes flow_status", async () => {
await withTestBackend(async (backend, tempDir) => {
await setupWorkspaceProfile(backend);
const uniqueId = Date.now();
const flowPath = `f/test/flow_json_${uniqueId}`;
await createRemoteMultiStepFlow(backend, flowPath);
const jobId = await runRemoteFlow(backend, flowPath);
await waitForJob(backend, jobId);
const result = await backend.runCLICommand(
["job", "get", jobId, "--json"],
tempDir
);
expect(result.code).toEqual(0);
const parsed = JSON.parse(result.stdout);
expect(parsed.flow_status).toBeDefined();
expect(parsed.flow_status.modules).toBeDefined();
expect(parsed.flow_status.modules.length).toBe(2);
});
});
test("job list --parent shows sub-jobs of a flow", async () => {
await withTestBackend(async (backend, tempDir) => {
await setupWorkspaceProfile(backend);
const uniqueId = Date.now();
const flowPath = `f/test/flow_parent_${uniqueId}`;
await createRemoteMultiStepFlow(backend, flowPath);
const jobId = await runRemoteFlow(backend, flowPath);
await waitForJob(backend, jobId);
const result = await backend.runCLICommand(
["job", "list", "--json", "--parent", jobId],
tempDir
);
expect(result.code).toEqual(0);
const parsed = JSON.parse(result.stdout);
expect(Array.isArray(parsed)).toBe(true);
// A 2-step flow should have at least 2 sub-jobs
expect(parsed.length).toBeGreaterThanOrEqual(2);
// All sub-jobs should reference the parent flow
expect(parsed.every((j: any) => j.parent_job === jobId)).toBe(true);
});
});
test("job list --all includes sub-jobs", async () => {
await withTestBackend(async (backend, tempDir) => {
await setupWorkspaceProfile(backend);
const uniqueId = Date.now();
const flowPath = `f/test/flow_all_${uniqueId}`;
await createRemoteMultiStepFlow(backend, flowPath);
const jobId = await runRemoteFlow(backend, flowPath);
await waitForJob(backend, jobId);
const result = await backend.runCLICommand(
["job", "list", "--json", "--all"],
tempDir
);
expect(result.code).toEqual(0);
const parsed = JSON.parse(result.stdout);
// Should contain both the parent flow and its sub-jobs
const parentJob = parsed.find((j: any) => j.id === jobId);
const subJobs = parsed.filter((j: any) => j.parent_job === jobId);
expect(parentJob).toBeDefined();
expect(subJobs.length).toBeGreaterThanOrEqual(2);
});
});
test("job get for failed flow shows failure status", async () => {
await withTestBackend(async (backend, tempDir) => {
await setupWorkspaceProfile(backend);
const uniqueId = Date.now();
const flowPath = `f/test/flow_fail_${uniqueId}`;
await createRemoteFailingFlow(backend, flowPath);
const jobId = await runRemoteFlow(backend, flowPath);
await waitForJob(backend, jobId);
const result = await backend.runCLICommand(
["job", "get", jobId],
tempDir
);
expect(result.code).toEqual(0);
expect(result.stdout).toContain("failure");
expect(result.stdout).toContain("Steps:");
// Step a should succeed, step b should fail
expect(result.stdout).toContain("✓");
expect(result.stdout).toContain("✗");
});
});
});
+124
View File
@@ -170,6 +170,130 @@ export async function runRemoteFlow(
throw new Error(`Failed to run flow ${flowPath} after ${retries} retries`);
}
/**
* Create a multi-step flow with 2 steps (a prints, b returns result).
* Useful for testing hierarchical job get and aggregated logs.
*/
export async function createRemoteMultiStepFlow(
backend: TestBackend,
flowPath: string
): Promise<void> {
const parts = flowPath.split("/");
if (parts[0] === "f" && parts.length > 2) {
await ensureFolder(backend, parts[1]);
}
const resp = await backend.apiRequest!(
`/api/w/${backend.workspace}/flows/create`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
path: flowPath,
summary: "Multi-step test flow",
description: "A flow with two steps for testing",
value: {
modules: [
{
id: "a",
summary: "Generate data",
value: {
type: "rawscript",
content:
'export async function main() { console.log("step a running"); return { value: 42 }; }',
language: "bun",
input_transforms: {},
},
},
{
id: "b",
summary: "Process data",
value: {
type: "rawscript",
content:
'export async function main(data: any) { console.log("step b running"); return "done"; }',
language: "bun",
input_transforms: {
data: { type: "javascript", expr: "results.a" },
},
},
},
],
},
schema: {
$schema: "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: {},
required: [],
},
}),
}
);
expect(resp.status).toBeLessThan(300);
await resp.text();
}
/**
* Create a flow where step b throws an error.
* Useful for testing failure handling.
*/
export async function createRemoteFailingFlow(
backend: TestBackend,
flowPath: string
): Promise<void> {
const parts = flowPath.split("/");
if (parts[0] === "f" && parts.length > 2) {
await ensureFolder(backend, parts[1]);
}
const resp = await backend.apiRequest!(
`/api/w/${backend.workspace}/flows/create`,
{
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({
path: flowPath,
summary: "Failing test flow",
description: "A flow where step b fails",
value: {
modules: [
{
id: "a",
summary: "Succeeding step",
value: {
type: "rawscript",
content:
'export async function main() { return "ok"; }',
language: "bun",
input_transforms: {},
},
},
{
id: "b",
summary: "Failing step",
value: {
type: "rawscript",
content:
'export async function main() { throw new Error("simulated failure"); }',
language: "bun",
input_transforms: {},
},
},
],
},
schema: {
$schema: "https://json-schema.org/draft/2020-12/schema",
type: "object",
properties: {},
required: [],
},
}),
}
);
expect(resp.status).toBeLessThan(300);
await resp.text();
}
export async function createRemoteSchedule(
backend: TestBackend,
schedulePath: string,
@@ -35,6 +35,7 @@ export const LANGUAGE_EXTENSIONS: Record<SupportedLanguage, string> = {
duckdb: "duckdb.sql",
bunnative: "ts",
ruby: "rb",
rlang: "r",
// for related places search: ADD_NEW_LANG
};
+4
View File
@@ -27,6 +27,10 @@ RUN /usr/bin/java -jar /usr/bin/coursier about
# Ruby
RUN apt-get install -y ruby ruby-bundler
# R
RUN apt-get install -y r-base-dev \
&& Rscript -e 'install.packages("renv", lib="/usr/lib/R/library", repos="https://cloud.r-project.org")'
# Fix UV cache permissions for non-root user support (uid 1000, etc.)
# The uv tool install ansible command populates the UV cache with root-owned files
RUN chmod -R a+rw /tmp/windmill/cache/uv && \
+4
View File
@@ -51,6 +51,10 @@ RUN /usr/bin/java -jar /usr/bin/coursier about
# Ruby
RUN apt-get install -y ruby ruby-bundler
# R
RUN apt-get install -y r-base-dev \
&& Rscript -e 'install.packages("renv", lib="/usr/lib/R/library", repos="https://cloud.r-project.org")'
# iptables
RUN apt-get install -y iptables
+122 -55
View File
@@ -27,6 +27,16 @@
extensions = [ "rust-src" "rust-analyzer" "rustfmt" ];
};
patchedClang = pkgs.llvmPackages_18.clang.overrideAttrs (oldAttrs: {
postFixup = ''
# Copy the original postFixup logic but skip add-hardening.sh
${oldAttrs.postFixup or ""}
# Remove the line that substitutes add-hardening.sh
sed -i 's/.*source.*add-hardening\.sh.*//' $out/bin/clang
'';
});
# ---------------------------------------------------------------
# Native C/C++ dependencies (required to compile the backend)
# ---------------------------------------------------------------
@@ -72,14 +82,16 @@
version = "130.0.7";
target = stdenv.hostPlatform.rust.rustcTarget;
sha256 = {
x86_64-linux = "sha256-pkdsuU6bAkcIHEZUJOt5PXdzK424CEgTLXjLtQ80t10=";
x86_64-linux =
"sha256-pkdsuU6bAkcIHEZUJOt5PXdzK424CEgTLXjLtQ80t10=";
aarch64-linux = lib.fakeHash;
x86_64-darwin = lib.fakeHash;
aarch64-darwin = lib.fakeHash;
}.${system};
in pkgs.fetchurl {
name = "librusty_v8-${version}";
url = "https://github.com/denoland/rusty_v8/releases/download/v${version}/librusty_v8_release_${target}.a.gz";
url =
"https://github.com/denoland/rusty_v8/releases/download/v${version}/librusty_v8_release_${target}.a.gz";
inherit sha256;
};
@@ -87,15 +99,28 @@
# pkg-config search path for native libraries
# ---------------------------------------------------------------
pkgConfigPath = lib.makeSearchPath "lib/pkgconfig"
(with pkgs; [ openssl.dev libxml2.dev xmlsec.dev libxslt.dev cyrus_sasl.dev krb5.dev ]);
pkgConfigPath = lib.makeSearchPath "lib/pkgconfig" (with pkgs; [
openssl.dev
libxml2.dev
xmlsec.dev
libxslt.dev
cyrus_sasl.dev
krb5.dev
]);
# ---------------------------------------------------------------
# RPATH — embed Nix store library paths into compiled binaries
# ---------------------------------------------------------------
rpathLibs = lib.makeLibraryPath (with pkgs; [
openssl libffi cyrus_sasl krb5 libxml2 xmlsec libxslt stdenv.cc.cc.lib
openssl
libffi
cyrus_sasl
krb5
libxml2
xmlsec
libxslt
stdenv.cc.cc.lib
]);
# ---------------------------------------------------------------
@@ -113,11 +138,17 @@
(builtins.readFile "${stdenv.cc}/nix-support/libcxx-cxxflags")
"-idirafter ${pkgs.libiconv}/include"
] ++ lib.optionals stdenv.cc.isClang [
"-idirafter ${stdenv.cc.cc}/lib/clang/${lib.getVersion stdenv.cc.cc}/include"
"-idirafter ${stdenv.cc.cc}/lib/clang/${
lib.getVersion stdenv.cc.cc
}/include"
] ++ lib.optionals stdenv.cc.isGNU [
"-isystem ${stdenv.cc.cc}/include/c++/${lib.getVersion stdenv.cc.cc}"
"-isystem ${stdenv.cc.cc}/include/c++/${lib.getVersion stdenv.cc.cc}/${stdenv.hostPlatform.config}"
"-idirafter ${stdenv.cc.cc}/lib/gcc/${stdenv.hostPlatform.config}/${lib.getVersion stdenv.cc.cc}/include"
"-isystem ${stdenv.cc.cc}/include/c++/${
lib.getVersion stdenv.cc.cc
}/${stdenv.hostPlatform.config}"
"-idirafter ${stdenv.cc.cc}/lib/gcc/${stdenv.hostPlatform.config}/${
lib.getVersion stdenv.cc.cc
}/include"
]);
# ---------------------------------------------------------------
@@ -131,12 +162,16 @@
BINDGEN_EXTRA_CLANG_ARGS = bindgenClangArgs;
# Force clang 18 as cargo linker (stdenv may bring a newer clang that causes SIGSEGV with mold)
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER = "${pkgs.llvmPackages_18.clang}/bin/clang";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER = "${pkgs.llvmPackages_18.clang}/bin/clang";
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER =
"${pkgs.llvmPackages_18.clang}/bin/clang";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_LINKER =
"${pkgs.llvmPackages_18.clang}/bin/clang";
# Embed rpath so binaries find Nix store .so files at runtime
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS = "-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS = "-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_RUSTFLAGS =
"-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
CARGO_TARGET_AARCH64_UNKNOWN_LINUX_GNU_RUSTFLAGS =
"-C link-arg=-fuse-ld=mold -C link-arg=-Wl,-rpath,${rpathLibs}";
CARGO_HOST_RUSTFLAGS = "-C link-arg=-Wl,-rpath,${rpathLibs}";
# https://github.com/NixOS/nixpkgs/issues/370494 — jemalloc build fix
@@ -197,6 +232,10 @@
hash = "sha256-8E0WtDFc7RcqmftDigMyy1xXUkjgL4X4kpf7h1GdE48=";
};
rWithPackages = pkgs.rWrapper.override {
packages = with pkgs.rPackages; [ renv ];
};
extraRuntimes = with pkgs; [
dotnet-sdk_9
php
@@ -222,6 +261,7 @@
ANSIBLE_PLAYBOOK_PATH = "${pkgs.ansible}/bin/ansible-playbook";
ANSIBLE_GALAXY_PATH = "${pkgs.ansible}/bin/ansible-galaxy";
CARGO_SWEEP_PATH = "${pkgs.cargo-sweep}/bin/cargo-sweep";
RSCRIPT_PATH = "${rWithPackages}/bin/Rscript";
};
# ---------------------------------------------------------------
@@ -251,13 +291,23 @@
(pkgs.writeScriptBin "wm" ''
cd ./frontend
npm install
npm run ${if stdenv.isDarwin then "generate-backend-client-mac" else "generate-backend-client"}
npm run ${
if stdenv.isDarwin then
"generate-backend-client-mac"
else
"generate-backend-client"
}
npm run dev "$@"
'')
(pkgs.writeScriptBin "wm-build" ''
cd ./frontend
npm install
npm run ${if stdenv.isDarwin then "generate-backend-client-mac" else "generate-backend-client"}
npm run ${
if stdenv.isDarwin then
"generate-backend-client-mac"
else
"generate-backend-client"
}
npm run build "$@"
'')
(pkgs.writeScriptBin "wm-migrate" ''
@@ -322,22 +372,20 @@
# Shared inputs and settings for default + full shells
# ---------------------------------------------------------------
coreBuildInputs = nativeBuildDeps ++ commonRuntimes ++ [
rustStable
openapi-generator-cli
] ++ (with pkgs; [
nodejs
git
sqlx-cli
cargo-watch
jq
gnused
coreBuildInputs = nativeBuildDeps ++ commonRuntimes
++ [ rustStable openapi-generator-cli ] ++ (with pkgs; [
nodejs
git
sqlx-cli
cargo-watch
jq
gnused
# CLI tools (for AI agents and dev workflow)
gh
asciinema
mermaid-cli
]);
# CLI tools (for AI agents and dev workflow)
gh
asciinema
mermaid-cli
]);
# Playwright: use Nix-provided browsers (version-matched to playwright-driver)
# Mermaid/Puppeteer: point at Nix chromium (Puppeteer respects this env var)
@@ -380,16 +428,26 @@
sandboxEnv = pkgs.buildEnv {
name = "windmill-sandbox";
paths = coreBuildInputs ++ helperScriptsBase
++ [ playwrightWrapper sandboxEnvScript pkgConfigWrapper pkgs.chromium ];
paths = coreBuildInputs ++ helperScriptsBase ++ [
playwrightWrapper
sandboxEnvScript
pkgConfigWrapper
pkgs.chromium
];
};
sandboxFullEnv = pkgs.buildEnv {
name = "windmill-sandbox-full";
paths = coreBuildInputs ++ extraRuntimes
++ helperScriptsBase ++ helperScriptsFull
++ [ playwrightWrapper sandboxEnvScript pkgConfigWrapper pkgs.chromium
pkgs.cargo-sweep pkgs.xcaddy pkgs.nsjail ];
paths = coreBuildInputs ++ extraRuntimes ++ helperScriptsBase
++ helperScriptsFull ++ [
playwrightWrapper
sandboxEnvScript
pkgConfigWrapper
pkgs.chromium
pkgs.cargo-sweep
pkgs.xcaddy
pkgs.nsjail
];
};
in {
@@ -412,8 +470,8 @@
shellHook = devShellHook;
buildInputs = coreBuildInputs;
packages = helperScriptsBase ++ [ playwrightWrapper ];
});
packages = helperScriptsBase ++ [ playwrightWrapper ];
});
# =============================================================
# full — all language runtimes, k8s tooling, specialized scripts
@@ -428,27 +486,28 @@
pyright
openapi-python-client
# LSP / editor
svelte-language-server
taplo
# LSP / editor
svelte-language-server
taplo
# Extra dev tools
cargo-sweep
# Extra dev tools
cargo-sweep
# Kubernetes
minikube
kubectl
kubernetes-helm
conntrack-tools
cri-tools
# Kubernetes
minikube
kubectl
kubernetes-helm
conntrack-tools
cri-tools
# Extra
xcaddy
nsjail
]);
# Extra
xcaddy
nsjail
]);
packages = helperScriptsBase ++ helperScriptsFull ++ [ playwrightWrapper ];
});
packages = helperScriptsBase ++ helperScriptsFull
++ [ playwrightWrapper ];
});
# =============================================================
# wasm — WASM target compilation (nightly Rust)
@@ -458,15 +517,23 @@
devShells.wasm = pkgs.mkShell (buildEnvVars // {
hardeningDisable = [ "all" ];
# Explicitly set paths for headers and linker
# DO NOT REMOVE - if absent, breaks wasm builds on NixOS.
shellHook = ''
export CC=${patchedClang}/bin/clang
'';
buildInputs = nativeBuildDeps ++ (with pkgs; [
(rust-bin.nightly.latest.default.override {
extensions = [ "rust-src" "rust-analyzer" ];
targets = [ "wasm32-unknown-unknown" "wasm32-unknown-emscripten" ];
targets =
[ "wasm32-unknown-unknown" "wasm32-unknown-emscripten" ];
})
wasm-pack
deno
emscripten
nushell
nodejs
glibc_multi
]);
});
+9 -3
View File
@@ -1,12 +1,12 @@
{
"name": "windmill-components",
"version": "1.669.1",
"version": "1.672.0",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "windmill-components",
"version": "1.669.1",
"version": "1.672.0",
"hasInstallScript": true,
"license": "AGPL-3.0",
"dependencies": {
@@ -83,6 +83,7 @@
"windmill-parser-wasm-nu": "1.510.1",
"windmill-parser-wasm-php": "1.647.1",
"windmill-parser-wasm-py": "1.657.2",
"windmill-parser-wasm-r": "1.668.1",
"windmill-parser-wasm-regex": "1.653.0",
"windmill-parser-wasm-ruby": "1.526.1",
"windmill-parser-wasm-rust": "1.647.1",
@@ -164,7 +165,7 @@
},
"../backend/parsers/windmill-parser-wasm/pkg-ts": {
"name": "windmill-parser-wasm-ts",
"version": "1.623.1"
"version": "1.589.3"
},
"node_modules/@alloc/quick-lru": {
"version": "5.2.0",
@@ -13685,6 +13686,11 @@
"resolved": "https://registry.npmjs.org/windmill-parser-wasm-py/-/windmill-parser-wasm-py-1.657.2.tgz",
"integrity": "sha512-3CN2rziafgCWcZri812+CkzuaE3P3/7dXmV9lSDpK9ma6Esd4zkHRXUFSyRzQE/R7Fxj5mSmSNX6xTff8eX5mw=="
},
"node_modules/windmill-parser-wasm-r": {
"version": "1.668.1",
"resolved": "https://registry.npmjs.org/windmill-parser-wasm-r/-/windmill-parser-wasm-r-1.668.1.tgz",
"integrity": "sha512-5YNeUibxpNBvYrxCgQcz1PxGhTFx2CyEpg2udtIhq7bx0d4gF/KDZVupMeQmAObmrEtTSFGUWNRJ4zXSWNrSpQ=="
},
"node_modules/windmill-parser-wasm-regex": {
"version": "1.653.0",
"resolved": "https://registry.npmjs.org/windmill-parser-wasm-regex/-/windmill-parser-wasm-regex-1.653.0.tgz",
+2 -1
View File
@@ -1,6 +1,6 @@
{
"name": "windmill-components",
"version": "1.669.1",
"version": "1.672.0",
"scripts": {
"dev": "vite dev",
"build": "vite build",
@@ -156,6 +156,7 @@
"windmill-parser-wasm-nu": "1.510.1",
"windmill-parser-wasm-php": "1.647.1",
"windmill-parser-wasm-py": "1.657.2",
"windmill-parser-wasm-r": "1.668.1",
"windmill-parser-wasm-regex": "1.653.0",
"windmill-parser-wasm-ruby": "1.526.1",
"windmill-parser-wasm-rust": "1.647.1",
+9 -1
View File
@@ -74,7 +74,15 @@
<button
class="text-xs text-accent"
onclick={async () => {
await getVariable(value.substring('$res:'.length))
await getVariable(value.substring('$var:'.length))
jsonViewer?.toggleDrawer()
}}>{value}</button
>
{:else if isString(value) && value.startsWith('$jsonvar:')}
<button
class="text-xs text-accent"
onclick={async () => {
await getVariable(value.substring('$jsonvar:'.length))
jsonViewer?.toggleDrawer()
}}>{value}</button
>
@@ -495,6 +495,8 @@
let { debounced, clearDebounce } = debounce(() => compareValues(value), 50)
let inputCat = $derived(computeInputCat(type, format, itemsType?.type, enum_, contentEncoding))
let isNonStringSecret = $derived((password || extra?.['password'] == true) && type === 'object')
let displayJsonToggleHeader = $derived(
displayHeader &&
inputCat === 'list' &&
@@ -558,6 +560,12 @@
class="text-accent underline font-normal"
onclick={() => variableEditor?.editVariable?.(value.slice(5))}>{value.slice(5)}</button
>
{:else if value && typeof value == 'string' && value?.startsWith('$jsonvar:')}
Linked to variable <button
class="text-accent underline font-normal"
onclick={() => variableEditor?.editVariable?.(value.slice('$jsonvar:'.length))}
>{value.slice('$jsonvar:'.length)}</button
>
{/if}
</div>
{/if}
@@ -1488,6 +1496,18 @@
{@render actions?.()}
</div>
{#if isNonStringSecret}
{#if typeof value === 'string' && value.startsWith('$jsonvar:')}
<div class="text-2xs text-tertiary">
Sensitive — stored as secret: <code class="text-2xs">{value.slice('$jsonvar:'.length)}</code
>
</div>
{:else}
<div class="text-2xs text-tertiary italic">Sensitive — will be stored as secret on submit</div
>
{/if}
{/if}
{#if !compact || (error && error != '')}
<div class="text-right text-xs leading-3 text-red-600 dark:text-red-400 mb-2">
{#if disabled || error === ''}
@@ -144,7 +144,7 @@
const app = await AppService.getAppByPath({ workspace, path })
return app.summary
} else if (kind === 'folder') {
const folder = await FolderService.getFolder({ workspace, name: path.slice(2) })
const folder = await FolderService.getFolder({ workspace, name: path.replace(/^f\//, '') })
return folder.summary
}
} catch (error) {
@@ -361,7 +361,14 @@
const parent = parentWorkspaceId
const current = currentWorkspaceId
for (const itemKey of selectedItems) {
const sortedItems = [...selectedItems].sort((a, b) => {
const aIsFolder = a.startsWith('folder:')
const bIsFolder = b.startsWith('folder:')
if (aIsFolder && !bIsFolder) return -1
if (!aIsFolder && bIsFolder) return 1
return 0
})
for (const itemKey of sortedItems) {
const diff = selectableDiffs.find((d) => itemKey == getItemKey(d))
if (!diff) {
@@ -4,7 +4,11 @@
import { SettingService, WorkerService, WorkspaceService } from '$lib/gen'
import { sendUserToast } from '$lib/toast'
import { enterpriseLicense, superadmin } from '$lib/stores'
import { DEFAULT_TAGS_PER_WORKSPACE_SETTING, DEFAULT_TAGS_WORKSPACES_SETTING } from '$lib/consts'
import {
DEFAULT_TAGS_PER_WORKSPACE_SETTING,
DEFAULT_TAGS_WORKSPACES_SETTING,
PREVIEW_TAGS_OVERRIDE_SETTING
} from '$lib/consts'
import Toggle from './Toggle.svelte'
import MultiSelect from './select/MultiSelect.svelte'
import { safeSelectItems } from './select/utils.svelte'
@@ -22,16 +26,19 @@
let defaultTags = $state<string[] | undefined>(undefined)
let limitToWorkspaces = $state(false)
let previewTagsOverride = $state(false)
// Change detection
let originalDefaultTagPerWorkspace = $state<boolean | undefined>(defaultTagPerWorkspace)
let originalDefaultTagWorkspaces = $state<string[]>(defaultTagWorkspaces)
let originalPreviewTagsOverride = $state(false)
// Detect changes
let hasChanges = $derived(
originalDefaultTagPerWorkspace !== defaultTagPerWorkspace ||
JSON.stringify($state.snapshot(originalDefaultTagWorkspaces)?.sort() || []) !==
JSON.stringify($state.snapshot(defaultTagWorkspaces)?.sort() || [])
JSON.stringify($state.snapshot(defaultTagWorkspaces)?.sort() || []) ||
originalPreviewTagsOverride !== previewTagsOverride
)
let workspaces: string[] = $state([])
@@ -47,6 +54,11 @@
key: DEFAULT_TAGS_WORKSPACES_SETTING
})) as any) ?? []
limitToWorkspaces = defaultTagWorkspaces ? defaultTagWorkspaces.length > 0 : false
previewTagsOverride =
((await SettingService.getGlobal({
key: PREVIEW_TAGS_OVERRIDE_SETTING
})) as any) ?? false
originalPreviewTagsOverride = previewTagsOverride
} catch (err) {
sendUserToast(`Could not load default tags: ${err}`, true)
}
@@ -68,10 +80,17 @@
: undefined
}
})
await SettingService.setGlobal({
key: PREVIEW_TAGS_OVERRIDE_SETTING,
requestBody: {
value: previewTagsOverride
}
})
// Update original state after save
originalDefaultTagPerWorkspace = defaultTagPerWorkspace
originalDefaultTagWorkspaces = [...(defaultTagWorkspaces || [])]
originalPreviewTagsOverride = previewTagsOverride
loadDefaultTags()
sendUserToast('Saved')
@@ -146,6 +165,18 @@
/>
{/if}
{/if}
<div class="flex flex-col gap-1">
<Toggle
bind:checked={previewTagsOverride}
options={{
right: 'route preview jobs to dedicated preview tag',
rightTooltip:
'When enabled, preview jobs (script previews and flow previews) will be routed to the "preview" tag instead of their language-specific tag, allowing you to dedicate specific workers for previews.'
}}
class="w-fit"
disabled={!$enterpriseLicense}
/>
</div>
</div>
<div class="flex gap-2 items-center mb-1">
@@ -168,6 +199,17 @@
</div>
</div>
{/each}
{#if previewTagsOverride}
<div class="flex gap-2 items-center">
<div class="w-36">
<Badge color="transparent">preview</Badge>
</div>
<div class="w-6 flex justify-center text-secondary">&rightarrow;</div>
<div class="flex-1">
<Badge color="blue">{defaultTagPerWorkspace ? 'preview-$workspace' : 'preview'}</Badge>
</div>
</div>
{/if}
</div>
{/if}
</Section>
@@ -132,10 +132,14 @@
}
})
for (const dep of sortedSet) {
allAlreadyExists[computeStatusPath(dep.kind, dep.path)] = await checkAlreadyExists(
dep.kind,
dep.path
)
try {
allAlreadyExists[computeStatusPath(dep.kind, dep.path)] = await checkAlreadyExists(
dep.kind,
dep.path
)
} catch {
allAlreadyExists[computeStatusPath(dep.kind, dep.path)] = false
}
}
dependencies = sortedSet.map((x) => ({
...x,
@@ -51,6 +51,7 @@
noPreview?: boolean
jsonEnabled?: boolean
isAppInput?: boolean
showSensitiveToggle?: boolean
displayWebhookWarning?: boolean
onlyMaskPassword?: boolean
editTab:
@@ -95,6 +96,7 @@
noPreview = false,
jsonEnabled = true,
isAppInput = false,
showSensitiveToggle = false,
displayWebhookWarning = false,
onlyMaskPassword = false,
editTab,
@@ -297,7 +299,9 @@
}
const editTabDefaultSize = untrack(() => noPreview) ? 100 : 50
editPanelSize = untrack(() => editTab) ? (untrack(() => editPanelInitialSize) ?? editTabDefaultSize) : 0
editPanelSize = untrack(() => editTab)
? (untrack(() => editPanelInitialSize) ?? editTabDefaultSize)
: 0
let inputPanelSize = $state(100 - editPanelSize)
let editPanelSizeSmooth = tweened(editPanelSize, {
duration: 150
@@ -677,6 +681,7 @@
bind:order={schema.properties[argName].order}
{isFlowInput}
{isAppInput}
{showSensitiveToggle}
>
{#snippet typeeditor()}
{#if isFlowInput || isAppInput}
+11 -2
View File
@@ -156,6 +156,7 @@
'nu',
'java',
'ruby',
'rlang',
'postgresql',
'mysql',
'bigquery',
@@ -182,7 +183,8 @@
'csharp',
'nu',
'java',
'ruby'
'ruby',
'rlang'
// for related places search: ADD_NEW_LANG
].includes(lang ?? '')
)
@@ -202,7 +204,8 @@
'csharp',
'nu',
'java',
'ruby'
'ruby',
'rlang'
// for related places search: ADD_NEW_LANG
].includes(lang ?? '')
)
@@ -515,6 +518,8 @@
// for related places search: ADD_NEW_LANG
} else if (lang == 'ruby') {
editor.insertAtCursor(`ENV['${name}']`)
} else if (lang == 'rlang') {
editor.insertAtCursor(`Sys.getenv("${name}")`)
} else if (
['postgresql', 'mysql', 'bigquery', 'mssql', 'oracledb', 'snowflake', 'duckdb'].includes(
lang ?? ''
@@ -583,6 +588,8 @@ string ${windmillPathToCamelCaseName(path)} = await client.GetStringAsync(uri);
editor.insertAtBeginning("require 'windmill/mini'\n")
}
editor.insertAtCursor(`get_variable("${path}")`)
} else if (lang == 'rlang') {
editor.insertAtCursor(`get_variable("${path}")`)
}
sendUserToast(`${name} inserted at cursor`)
}}
@@ -662,6 +669,8 @@ JsonNode ${windmillPathToCamelCaseName(path)} = JsonNode.Parse(await client.GetS
editor.insertAtBeginning("require 'windmill/mini'\n")
}
editor.insertAtCursor(`get_resource("${path}")`)
} else if (lang == 'rlang') {
editor.insertAtCursor(`get_resource("${path}")`)
} else if (lang == 'duckdb') {
let t = { postgresql: 'postgres', mysql: 'mysql', bigquery: 'bigquery' }[resType]
if (!t) {
@@ -1,7 +1,10 @@
<script lang="ts">
import type { FlowModule, FlowValue } from '$lib/gen'
import type { FlowModule, FlowValue, TriggersCount } from '$lib/gen'
import type { TriggerContext } from '$lib/components/triggers'
import { Triggers } from '$lib/components/triggers/triggers.svelte'
import { createEventDispatcher } from 'svelte'
import { createEventDispatcher, hasContext, setContext } from 'svelte'
import { writable } from 'svelte/store'
import { twMerge } from 'tailwind-merge'
import FlowGraphViewerStep from './FlowGraphViewerStep.svelte'
@@ -27,6 +30,8 @@
minHeight?: number
noBorder?: boolean
hideDefaultInputs?: boolean
provideTriggerContext?: boolean
fillAvailableHeight?: boolean
}
let {
@@ -40,18 +45,32 @@
workspace = $workspaceStore,
minHeight = 400,
noBorder = false,
hideDefaultInputs = false
hideDefaultInputs = false,
provideTriggerContext = false,
fillAvailableHeight = false
}: Props = $props()
let availableHeight = $state(0)
if (provideTriggerContext && !hasContext('TriggerContext')) {
const triggersCount = writable<TriggersCount | undefined>(undefined)
setContext<TriggerContext>('TriggerContext', {
triggersCount,
simplifiedPoll: writable(false),
showCaptureHint: writable(undefined),
triggersState: new Triggers()
})
}
const dispatch = createEventDispatcher()
</script>
<div class="grid grid-cols-3 w-full h-full">
<div bind:clientHeight={availableHeight} class="grid grid-cols-3 w-full h-full min-h-0">
{#if !noGraph}
<div
class="{noSide || (hideDefaultInputs && stepDetail == undefined)
? 'col-span-3'
: 'sm:col-span-2 col-span-3'} w-full max-h-full"
: 'sm:col-span-2 col-span-3'} w-full h-full min-h-0 max-h-full"
class:overflow-auto={overflowAuto}
class:border={!noBorder}
>
@@ -61,7 +80,7 @@
cache={flow.value.cache_ttl !== undefined}
path={flow?.path}
{download}
{minHeight}
minHeight={fillAvailableHeight ? Math.max(minHeight, availableHeight) : minHeight}
{workspace}
modules={flow?.value?.modules}
failureModule={flow?.value?.failure_module}
@@ -88,7 +107,9 @@
{#if !noSide && !(hideDefaultInputs && stepDetail == undefined)}
<div
class={twMerge(
'relative w-full h-full min-h-[150px] max-h-[90vh] border-r border-b border-t p-2 pt-0 overflow-auto hidden sm:flex flex-col gap-4',
fillAvailableHeight
? 'relative w-full h-full min-h-0 border-r border-b border-t p-2 pt-0 overflow-auto hidden sm:flex flex-col gap-4'
: 'relative w-full h-full min-h-[150px] max-h-[90vh] border-r border-b border-t p-2 pt-0 overflow-auto hidden sm:flex flex-col gap-4',
noGraph ? 'border-0 w-max' : ''
)}
>
@@ -51,17 +51,10 @@
<InputTransformsViewer inputTransforms={stepDetail?.value?.input_transforms ?? {}} />
</div>
{#if stepDetail.value.path.startsWith('hub/')}
<div class="mt-6">
<h3 class="mb-1 mt-6 text-xs font-semibold text-emphasis">Code</h3>
<iframe
class="w-full h-full text-sm"
title="embedded script from hub"
frameborder="0"
src="{$hubBaseUrlStore}/embed/script/{stepDetail.value?.path?.substring(4)}"
></iframe>
</div>
{/if}
<div class="mt-6">
<h3 class="mb-1 mt-6 text-xs font-semibold text-emphasis">Code</h3>
<FlowModuleScript path={stepDetail.value.path} hash={jobScriptHash} />
</div>
{:else if stepDetail.value.type == 'rawscript'}
<div class="text-2xs mb-4 mt-2">
<h3 class="mb-1 text-xs font-semibold text-emphasis">Step inputs</h3>
@@ -218,27 +211,16 @@
<InputTransformsViewer inputTransforms={stepDetail?.value?.input_transforms ?? {}} />
</div>
{/if}
{#if stepDetail.value.path.startsWith('hub/')}
<div class="flex flex-col grow">
<div class="mb-1 mt-6 flex justify-between items-center">
<h3 class="font-semibold text-xs text-emphasis">Code</h3>
<Button
unifiedSize="sm"
variant="subtle"
onClick={codeViewer?.openDrawer}
startIcon={{ icon: Expand }}>Expand</Button
>
</div>
<iframe
class="w-full grow text-sm h-full"
title="embedded script from hub"
frameborder="0"
src="{$hubBaseUrlStore}/embed/script/{stepDetail.value?.path?.substring(4)}"
></iframe>
</div>
{:else}
<FlowModuleScript path={stepDetail.value.path} hash={jobScriptHash} />
{/if}
<div class="mb-1 mt-6 flex justify-between items-center">
<h3 class="font-semibold text-xs text-emphasis">Code</h3>
<Button
unifiedSize="sm"
variant="subtle"
onClick={codeViewer?.openDrawer}
startIcon={{ icon: Expand }}>Expand</Button
>
</div>
<FlowModuleScript path={stepDetail.value.path} hash={jobScriptHash} />
{:else if stepDetail.value.type == 'aiagent'}
<div class="text-xs">
<h3 class="mb-1 font-semibold mt-2 text-xs text-emphasis">Step inputs</h3>
@@ -2,6 +2,7 @@
import { type Job } from '$lib/gen'
import { base } from '$lib/base'
import JobStatus from '$lib/components/JobStatus.svelte'
import { flowPathToHref } from '$lib/scripts'
import { displayDate, truncateRev } from '$lib/utils'
import ScheduleEditor from '$lib/components/triggers/schedules/ScheduleEditor.svelte'
import TimeAgo from './TimeAgo.svelte'
@@ -94,7 +95,9 @@
{#if (job && job.job_kind == 'flow') || job?.job_kind == 'script'}
{@const stem = `${job?.job_kind}s`}
{@const isScript = job?.job_kind === 'script'}
{@const viewHref = `${base}/${stem}/get/${isScript ? job?.script_hash : job?.script_path}`}
{@const viewHref = isScript
? `${base}/${stem}/get/${job?.script_hash}`
: flowPathToHref(job?.script_path ?? '')}
<div class="flex flex-row gap-2 items-center">
{#if isScript}
<Code2 size={SMALL_ICON_SIZE} class="min-w-3.5" />
@@ -11,6 +11,7 @@
import { createEventDispatcher, getContext, untrack } from 'svelte'
import type { FlowEditorContext } from './flows/types'
import { runFlowPreview } from './flows/utils.svelte'
import { processSecretArgs } from './secretArgUtils'
import SchemaForm from './SchemaForm.svelte'
import SchemaFormWithArgPicker from './SchemaFormWithArgPicker.svelte'
import FlowStatusViewer from '../components/FlowStatusViewer.svelte'
@@ -171,6 +172,7 @@
lastPreviewFlow = JSON.stringify(flowStore.val)
flowProgressBar?.reset()
const newFlow = extractFlow(previewMode)
args = await processSecretArgs(args, flowStore.val.schema as any)
newJobId = await runFlowPreview(args, newFlow, $pathStore, restartedFrom, conversationId)
jobId = newJobId
isRunning = true
@@ -3,7 +3,7 @@
import { type Job, JobService } from '$lib/gen'
import { workspaceStore } from '$lib/stores'
import { sendUserToast } from '$lib/toast'
import { X } from 'lucide-svelte'
import { ExternalLink, X } from 'lucide-svelte'
import DisplayResult from './DisplayResult.svelte'
import Tooltip from './Tooltip.svelte'
import { Button } from './common'
@@ -23,6 +23,7 @@
let default_payload: object = $state({})
let description: any = $state(undefined)
let hide_cancel = $state(false)
let approvalPageUrl: string | undefined = $state(undefined)
let defaultValues = $state({})
@@ -47,6 +48,8 @@
defaultValues = JSON.parse(JSON.stringify(args))
default_payload = args
approvalPageUrl = job_result?.['approvalPage']
actionTaken = false
hide_cancel = job?.raw_flow?.modules?.[approvalStep]?.suspend?.hide_cancel ?? false
schema = mergeSchema(
job?.raw_flow?.modules?.[approvalStep]?.suspend?.resume_form?.schema ?? {},
@@ -55,6 +58,7 @@
}
let loading = $state(false)
let actionTaken = $state(false)
async function continu(approve: boolean) {
loading = true
try {
@@ -66,6 +70,7 @@
approved: approve
}
})
actionTaken = true
} catch (e: any) {
sendUserToast(e?.body ?? e?.message ?? 'Failed', true)
} finally {
@@ -84,7 +89,7 @@
<div class="mt-2"></div>
{/if}
<div>
<div class={twMerge('flex gap-2', light ? 'flex-col' : 'flex-row ')}>
<div class={twMerge('flex gap-2 items-center', light ? 'flex-col' : 'flex-row ')}>
{#if !hide_cancel}
<div>
<Button
@@ -92,7 +97,7 @@
iconOnly
startIcon={{ icon: X }}
variant="default"
disabled={loading}
disabled={loading || actionTaken}
destructive
unifiedSize="md"
on:click={() => continu(false)}
@@ -100,12 +105,28 @@
</div>
{/if}
<div>
<Button variant="accent" onClick={() => continu(true)} disabled={loading} unifiedSize="md">
<Button
variant="accent"
onClick={() => continu(true)}
disabled={loading || actionTaken}
unifiedSize="md"
>
Resume
<Tooltip class="text-white">Resume or approve this suspended step</Tooltip>
</Button>
</div>
{#if approvalPageUrl}
<a
href={approvalPageUrl}
target="_blank"
rel="noreferrer"
class="text-accent flex items-center gap-1 whitespace-nowrap"
>
Approval page <ExternalLink size={12} />
</a>
{/if}
{#if job?.raw_flow?.modules?.[approvalStep]?.suspend?.resume_form?.schema}
<div
class={twMerge(
@@ -14,6 +14,7 @@
import yaml from 'svelte-highlight/languages/yaml'
import java from 'svelte-highlight/languages/java'
import ruby from 'svelte-highlight/languages/ruby'
import r from 'svelte-highlight/languages/r'
import type { Script } from '$lib/gen'
import { Button } from './common'
import { copyToClipboard } from '$lib/utils'
@@ -91,6 +92,8 @@
return java
case 'ruby':
return ruby
case 'rlang':
return r
case 'json':
return json
// for related places search: ADD_NEW_LANG
+6 -1
View File
@@ -21,6 +21,7 @@
import Skeleton from './common/skeleton/Skeleton.svelte'
import Button from './common/button/Button.svelte'
import { sameTopDomainOrigin } from '$lib/cookies'
import { isValidLogoutRedirect } from '$lib/logoutRedirect'
interface Props {
rd?: string | undefined
@@ -134,7 +135,11 @@
async function redirectUser() {
if (rd?.startsWith('http')) {
window.location.href = rd
if (isValidLogoutRedirect(rd)) {
window.location.href = rd
return
}
goto('/')
return
}
if ($workspaceStore) {
+9 -3
View File
@@ -19,7 +19,7 @@
min = 0,
max = 100,
initialValue = 0,
value = $bindable(typeof initialValue === 'string' ? parseInt(initialValue) : initialValue),
value = $bindable(),
disabled = false,
defaultValue = undefined,
format = (v) => `${v}`,
@@ -36,8 +36,14 @@
}
run(() => {
if (value === null) {
value = 0
if (value === null || value === undefined || Number.isNaN(value)) {
const fallback =
initialValue !== undefined
? typeof initialValue === 'string'
? parseInt(initialValue)
: initialValue
: (min ?? 0)
value = Number.isNaN(fallback) ? (min ?? 0) : fallback
}
})
+19 -5
View File
@@ -3,7 +3,8 @@
computeSharableHash as computeSharableHash,
defaultIfEmptyString,
emptyString,
truncateHash
truncateHash,
sendUserToast
} from '$lib/utils'
import type { Schema } from '$lib/common'
@@ -21,6 +22,7 @@
import { triggerableByAI } from '$lib/actions/triggerableByAI.svelte'
import InputSelectedBadge from './schema/InputSelectedBadge.svelte'
import { untrack } from 'svelte'
import { processSecretArgs } from './secretArgUtils'
let reloadArgs = $state(0)
let jsonEditor: JsonInputs | undefined = $state(undefined)
@@ -33,8 +35,20 @@
reloadArgs++
}
export function run() {
runAction(scheduledForStr, args ?? {}, invisible_to_owner, overrideTag)
export async function run(overrideScheduledForStr?: string | undefined | null) {
let processedArgs: Record<string, any>
try {
processedArgs = await processSecretArgs(args ?? {}, runnable?.schema)
} catch (e) {
sendUserToast('Failed to process sensitive args: ' + e, true)
return
}
runAction(
overrideScheduledForStr === null ? undefined : (overrideScheduledForStr ?? scheduledForStr),
processedArgs,
invisible_to_owner,
overrideTag
)
}
interface Props {
@@ -276,7 +290,7 @@
unifiedSize="md"
btnClasses="!inline-flex"
disabled={!isValid && !jsonView}
on:click={() => runAction(scheduledForStr, args ?? {}, invisible_to_owner, overrideTag)}
on:click={() => run()}
shortCut={{ Icon: CornerDownLeft, hide: !viewKeybinding }}
>
{scheduledForStr ? 'Schedule to run later' : buttonText}
@@ -315,7 +329,7 @@
btnClasses="!px-6 !py-1 w-full"
variant="accent"
disabled={!isValid && !jsonView}
on:click={() => runAction(undefined, args ?? {}, invisible_to_owner, overrideTag)}
on:click={() => run(null)}
shortCut={{ Icon: CornerDownLeft, hide: !viewKeybinding }}
>
{buttonText}
@@ -1268,7 +1268,7 @@
} as ButtonType.Icon}
>
<span class="truncate">{label}</span>
{#if lang === 'ruby'}
{#if lang === 'rlang'}
<span class="text-primary !text-xs"> BETA </span>
{/if}
</Button>
@@ -1,5 +1,6 @@
<script lang="ts">
import { buildWsUrl } from '$lib/wsUrl'
import { processSecretArgs } from './secretArgUtils'
import type { Schema, SupportedLanguage } from '$lib/common'
import {
type CompletedJob,
@@ -645,12 +646,14 @@
const testCode = activeModuleTab !== null ? editorCode : code
const testLang = activeModuleTab !== null ? effectiveLang : lang
const testArgs =
const rawTestArgs =
activeModuleTab !== null
? testPanelArgs
: selectedTab === 'preprocessor' || kind === 'preprocessor'
? { _ENTRYPOINT_OVERRIDE: 'preprocessor', ...(args ?? {}) }
: (args ?? {})
const testSchema = activeModuleTab !== null ? testPanelSchema : schema
const testArgs = await processSecretArgs(rawTestArgs, testSchema)
//@ts-ignore
let job = await jobLoader.runPreview(
@@ -12,4 +12,4 @@
let { schema = $bindable(), customUi = undefined }: Props = $props()
</script>
<EditableSchemaForm bind:schema uiOnly {customUi} editTab="inputEditor" />
<EditableSchemaForm bind:schema uiOnly {customUi} editTab="inputEditor" showSensitiveToggle />
@@ -4,9 +4,11 @@
const bubble = createBubbler()
import { Button, Drawer, DrawerContent } from '$lib/components/common'
import { base } from '$lib/base'
import FlowGraphViewer from '$lib/components/FlowGraphViewer.svelte'
import Skeleton from '$lib/components/common/skeleton/Skeleton.svelte'
import FlowModuleScript from '$lib/components/flows/content/FlowModuleScript.svelte'
import FlowPathViewer from '$lib/components/flows/content/FlowPathViewer.svelte'
import { emptySchema, sendUserToast } from '$lib/utils'
import { emptySchema, getHubFlowIdFromPath, isHubFlowPath, sendUserToast } from '$lib/utils'
import { getContext, tick, untrack } from 'svelte'
import type {
ConnectedAppInput,
@@ -31,7 +33,8 @@
import Popover from '$lib/components/meltComponents/Popover.svelte'
import ScriptEditorDrawer from '$lib/components/flows/content/ScriptEditorDrawer.svelte'
import FlowEditorDrawer from '$lib/components/flows/content/FlowEditorDrawer.svelte'
import { ScriptService } from '$lib/gen'
import { FlowService, ScriptService, type OpenFlow } from '$lib/gen'
import { replaceScriptPlaceholderWithItsValues } from '$lib/hub'
interface Props {
runnable: RunnableByPath
@@ -43,6 +46,7 @@
isLoading?: boolean
onRun?: any
onCancel?: any
hubFlowPreview?: OpenFlow | undefined
}
let {
@@ -52,14 +56,17 @@
rawApps = false,
isLoading = false,
onRun = async () => {},
onCancel = async () => {}
onCancel = async () => {},
hubFlowPreview = $bindable(undefined)
}: Props = $props()
const viewerContext = getContext<AppViewerContext>('AppViewerContext')
let drawerFlowViewer: Drawer | undefined = $state(undefined)
let flowPath: string = $state('')
let drawerShowsHubFlow = $state(false)
let notFound = $state(false)
let hubFlowId = $derived(getHubFlowIdFromPath(runnable.path))
// Key to force re-mounting of viewer components (bypasses FlowModuleScript cache)
let refreshKey = $state(0)
@@ -70,6 +77,7 @@
const dispatch = createEventDispatcher()
async function refreshScript(runnable: RunnableByPath) {
hubFlowPreview = undefined
try {
let { schema } = await getScriptByPath(runnable.path)
if (!deepEqual(runnable.schema, schema)) {
@@ -86,7 +94,39 @@
}
async function refreshFlow(runnable: RunnableByPath) {
hubFlowPreview = undefined
try {
const hubFlowId = getHubFlowIdFromPath(runnable.path)
if (hubFlowId !== undefined) {
const hub = await FlowService.getHubFlowById({ id: hubFlowId })
const flow = hub.flow ? structuredClone(hub.flow) : undefined
if (flow?.value.preprocessor_module?.value.type === 'rawscript') {
flow.value.preprocessor_module.value.content = replaceScriptPlaceholderWithItsValues(
String(hubFlowId),
flow.value.preprocessor_module.value.content
)
}
if (!flow) {
notFound = true
return
}
hubFlowPreview = flow
const schema =
flow.schema && typeof flow.schema === 'object' && Object.keys(flow.schema).length > 0
? (flow.schema as any)
: emptySchema()
if (!deepEqual(runnable.schema, schema)) {
runnable.schema = schema
if (!runnable.schema.order) {
runnable.schema.order = Object.keys(runnable.schema.properties ?? {})
}
fields = computeFields(schema, false, fields ?? {})
}
return
}
const { schema } =
(await loadSchema($workspaceStore ?? '', runnable.path, 'flow')) ?? emptySchema()
if (!deepEqual(runnable.schema, schema)) {
@@ -158,6 +198,8 @@
refreshScript(runnable)
} else if (runnable.runType == 'flow') {
refreshFlow(runnable)
} else {
hubFlowPreview = undefined
}
lastRunnable = runnable
}
@@ -170,8 +212,34 @@
</script>
<Drawer bind:this={drawerFlowViewer} size="1200px">
<DrawerContent title="Flow {flowPath}" on:close={drawerFlowViewer.closeDrawer}>
<FlowPathViewer path={flowPath ?? ''} />
<DrawerContent
title="Flow {flowPath}"
on:close={() => {
flowPath = ''
drawerShowsHubFlow = false
drawerFlowViewer?.closeDrawer()
}}
>
{#if drawerShowsHubFlow}
<div class="flex flex-col flex-1 h-full min-h-0 overflow-auto">
{#if hubFlowPreview}
<FlowGraphViewer
triggerNode
provideTriggerContext
fillAvailableHeight
flow={{ ...hubFlowPreview, path: flowPath }}
/>
{:else if notFound}
<div class="p-4 text-red-400">Hub flow not found at {flowPath}</div>
{:else}
<div class="p-4">
<Skeleton layout={[[40]]} />
</div>
{/if}
</div>
{:else if flowPath}
<FlowPathViewer path={flowPath} fillAvailableHeight />
{/if}
</DrawerContent>
</Drawer>
@@ -210,7 +278,7 @@
size="xs"
startIcon={{ icon: RefreshCw }}
on:click={async () => {
sendUserToast('Getting latest script version at that path')
sendUserToast('Getting latest runnable version at that path')
// Increment refreshKey to force re-mounting of viewer components (bypasses cache)
refreshKey++
lastRunnable = undefined
@@ -238,31 +306,45 @@
startIcon={{ icon: Eye }}
on:click={() => {
flowPath = runnable.path
drawerShowsHubFlow = isHubFlowPath(runnable.path)
drawerFlowViewer?.openDrawer()
}}
>
Expand
</Button>
<Button
variant="default"
size="xs"
startIcon={{ icon: Pen }}
on:click={() => {
openFlowEditor(runnable.path)
}}
>
Edit
</Button>
<Button
variant="default"
size="xs"
startIcon={{ icon: Eye }}
endIcon={{ icon: ExternalLink }}
target="_blank"
href="{base}/flows/get/{runnable.path}?workspace={$workspaceStore}"
>
Details
</Button>
{#if hubFlowId}
<Button
variant="default"
size="xs"
startIcon={{ icon: GitFork }}
endIcon={{ icon: ExternalLink }}
target="_blank"
href="{base}/flows/add?hub={hubFlowId}"
>
Fork
</Button>
{:else}
<Button
variant="default"
size="xs"
startIcon={{ icon: Pen }}
on:click={() => {
openFlowEditor(runnable.path)
}}
>
Edit
</Button>
<Button
variant="default"
size="xs"
startIcon={{ icon: Eye }}
endIcon={{ icon: ExternalLink }}
target="_blank"
href="{base}/flows/get/{runnable.path}?workspace={$workspaceStore}"
>
Details
</Button>
{/if}
{:else}
<Button
size="xs"
@@ -308,13 +390,20 @@
nonCaptureEvent={true}
btnClasses={'bg-surface text-primay hover:bg-hover'}
variant="default"
size="xs">Cache</Button
size="xs"
>
Cache
</Button>
{/snippet}
{#snippet content()}
Since this is a reference to a workspace {runnable.runType}, set the cache in the {runnable.runType}
settings directly by editing it. The cache will be shared by any app or flow that uses this
{runnable.runType}.
{#if runnable.runType == 'flow' && isHubFlowPath(runnable.path)}
Since this is a reference to a hub flow, cache settings are managed from the flow after
you fork it into your workspace.
{:else}
Since this is a reference to a workspace {runnable.runType}, set the cache in the
{runnable.runType} settings directly by editing it. The cache will be shared by any app or
flow that uses this {runnable.runType}.
{/if}
{/snippet}
</Popover>
@@ -325,18 +414,37 @@
class="!text-xs !rounded-xs"
/>
</div>
<div class="w-full grow overflow-y-auto">
<div class="w-full grow min-h-0 overflow-y-auto">
{#key `${viewerContext?.stateId ? get(viewerContext.stateId) : 0}-${refreshKey}`}
{#if notFound}
<div class="text-red-400"
>{runnable.runType} not found at {runnable.path} in workspace {$workspaceStore}</div
>
<div class="text-red-400">
{#if runnable.runType == 'flow' && isHubFlowPath(runnable.path)}
Hub flow not found at {runnable.path}
{:else}
{runnable.runType} not found at {runnable.path} in workspace {$workspaceStore}
{/if}
</div>
{:else if runnable.runType == 'script' || runnable.runType == 'hubscript'}
<div class="border">
<FlowModuleScript path={runnable.path} />
</div>
{:else if runnable.runType == 'flow'}
<FlowPathViewer path={runnable.path} />
{#if isHubFlowPath(runnable.path)}
{#if hubFlowPreview}
<div class="flex flex-col flex-1 h-full min-h-0 overflow-auto">
<FlowGraphViewer
triggerNode
provideTriggerContext
fillAvailableHeight
flow={{ ...hubFlowPreview, path: runnable.path }}
/>
</div>
{:else}
<Skeleton layout={[[40]]} />
{/if}
{:else}
<FlowPathViewer path={runnable.path} fillAvailableHeight />
{/if}
{:else}
Unrecognized runType {runnable.runType}
{/if}
@@ -8,10 +8,10 @@
import WorkspaceFlowList from './WorkspaceFlowList.svelte'
import { createEventDispatcher, untrack } from 'svelte'
import type { Schema } from '$lib/common'
import { schemaToInputsSpec } from '$lib/components/apps/utils'
import { defaultIfEmptyString, emptySchema } from '$lib/utils'
import { emptySchema } from '$lib/utils'
import { loadSchema } from '$lib/infer'
import { workspaceStore } from '$lib/stores'
import { buildPathRunnableSelection } from './runnableSelectorUtils'
type TabType = 'hubscripts' | 'workspacescripts' | 'workspaceflows' | 'inlinescripts'
@@ -62,51 +62,44 @@
}
async function pickScript(path: string) {
const schema = await loadSchemaFromTriggerable(path, 'script')
const fields = schemaToInputsSpec(schema.schema, defaultUserInput)
const runnable = {
type: 'path',
const selection = buildPathRunnableSelection(
path,
runType: 'script',
schema: schema.schema,
name: defaultIfEmptyString(schema.summary, path)
} as const
'script',
await loadSchemaFromTriggerable(path, 'script'),
defaultUserInput,
rawApps
)
dispatch('pick', {
runnable,
fields
runnable: selection.runnable,
fields: selection.fields
})
}
async function pickFlow(path: string) {
const schema = await loadSchemaFromTriggerable(path, 'flow')
const fields = schemaToInputsSpec(schema.schema, defaultUserInput)
const runnable = {
type: 'path',
const selection = buildPathRunnableSelection(
path,
runType: 'flow',
schema,
name: defaultIfEmptyString(schema.summary, path)
} as const
'flow',
await loadSchemaFromTriggerable(path, 'flow'),
defaultUserInput,
rawApps
)
dispatch('pick', {
runnable,
fields
runnable: selection.runnable,
fields: selection.fields
})
}
async function pickHubScript(path: string) {
const schema = await loadSchemaFromTriggerable(path, 'hubscript')
const fields = schemaToInputsSpec(schema.schema, defaultUserInput)
const runnable = {
type: 'path',
const selection = buildPathRunnableSelection(
path,
runType: 'hubscript',
schema: schema.schema,
name: defaultIfEmptyString(schema.summary, path)
} as const
'hubscript',
await loadSchemaFromTriggerable(path, 'hubscript'),
defaultUserInput,
rawApps
)
dispatch('pick', {
runnable,
fields
runnable: selection.runnable,
fields: selection.fields
})
}
@@ -0,0 +1,53 @@
import { describe, expect, it } from 'vitest'
import type { LoadedRunnableSchema } from './runnableSelectorUtils'
import { buildPathRunnableSelection } from './runnableSelectorUtils'
const loadedSchema: LoadedRunnableSchema = {
summary: 'My flow',
schema: {
$schema: 'https://json-schema.org/draft/2020-12/schema',
type: 'object',
required: ['string_input'],
properties: {
string_input: {
type: 'string',
default: ''
}
}
}
}
describe('buildPathRunnableSelection', () => {
it('keeps the actual schema object and defaults raw-app fields to user mode', () => {
const selection = buildPathRunnableSelection(
'u/dev/my_flow',
'flow',
loadedSchema,
false,
true
)
expect(selection.runnable).toMatchObject({
type: 'path',
path: 'u/dev/my_flow',
runType: 'flow',
schema: loadedSchema.schema,
name: 'My flow'
})
expect(selection.fields.string_input.type).toBe('user')
expect(selection.fields.string_input.value).toBe('')
})
it('preserves static defaults for non-raw-app pickers', () => {
const selection = buildPathRunnableSelection(
'u/dev/my_flow',
'flow',
loadedSchema,
false,
false
)
expect(selection.fields.string_input.type).toBe('static')
})
})
@@ -0,0 +1,31 @@
import type { Schema } from '$lib/common'
import type { Runnable, StaticAppInput } from '$lib/components/apps/inputType'
import { schemaToInputsSpec } from '$lib/components/apps/utils'
import { defaultIfEmptyString } from '$lib/utils'
export type LoadedRunnableSchema = {
schema: Schema
summary: string | undefined
}
export function buildPathRunnableSelection(
path: string,
runType: 'script' | 'flow' | 'hubscript',
loadedSchema: LoadedRunnableSchema,
defaultUserInput: boolean,
rawApps: boolean
): {
runnable: Runnable
fields: Record<string, StaticAppInput>
} {
return {
runnable: {
type: 'path',
path,
runType,
schema: loadedSchema.schema,
name: defaultIfEmptyString(loadedSchema.summary, path)
},
fields: schemaToInputsSpec(loadedSchema.schema, defaultUserInput || rawApps)
}
}
@@ -25,6 +25,7 @@
import JavaIcon from '$lib/components/icons/JavaIcon.svelte'
import DuckDbIcon from '$lib/components/icons/DuckDbIcon.svelte'
import RubyIcon from '$lib/components/icons/RubyIcon.svelte'
import RIcon from '$lib/components/icons/RIcon.svelte'
import ClaudeIcon from '$lib/components/icons/ClaudeIcon.svelte'
interface Props {
@@ -72,6 +73,7 @@
nu: 'Nu',
java: 'Java',
ruby: 'Ruby',
rlang: 'R',
claudesandbox: 'Claude Sandbox'
// for related places search: ADD_NEW_LANG
}
@@ -107,6 +109,7 @@
nu: NuIcon,
java: JavaIcon,
ruby: RubyIcon,
rlang: RIcon,
duckdb: DuckDbIcon,
claudesandbox: TypeScriptIcon
// for related places search: ADD_NEW_LANG
@@ -686,6 +686,7 @@
bind:schema={flowStore.val.schema}
hiddenArgs={['user_message']}
isFlowInput
showSensitiveToggle
editTab={chatInputsEditTab ? 'inputEditor' : undefined}
showDynOpt
bind:dynCode
@@ -741,6 +742,7 @@
bind:this={editableSchemaForm}
bind:schema={flowStore.val.schema}
isFlowInput
showSensitiveToggle
on:delete={(e) => {
addPropertyV2?.handleDeleteArgument([e.detail])
}}
@@ -90,7 +90,9 @@
async function loadCode(path: string, hash: string | undefined) {
try {
notFound = false
const script = hash
const script = path.startsWith('hub/')
? await getScriptByPath(path!)
: hash
? await ScriptService.getScriptByHash({ workspace: $workspaceStore!, hash })
: await getScriptByPath(path!)
code = script.content
@@ -13,9 +13,10 @@
interface Props {
path: string;
noSide?: boolean;
fillAvailableHeight?: boolean;
}
let { path, noSide = false }: Props = $props();
let { path, noSide = false, fillAvailableHeight = false }: Props = $props();
let flow: Flow | undefined = $state(undefined)
@@ -41,7 +42,7 @@
<div class="flex flex-col flex-1 h-full overflow-auto">
{#if flow}
<FlowGraphViewer triggerNode={true} {noSide} {flow} />
<FlowGraphViewer triggerNode={true} {noSide} {flow} {fillAvailableHeight} />
{:else}
<Skeleton layout={[[40]]} />
{/if}
@@ -0,0 +1,37 @@
<script lang="ts">
interface Props {
height?: number
width?: number
}
let { height = 24, width = 24 }: Props = $props()
</script>
<svg
xmlns="http://www.w3.org/2000/svg"
{width}
{height}
viewBox="0 0 724 561"
preserveAspectRatio="xMidYMid"
>
<defs>
<linearGradient id="r-grad-1" x1="0" x2="1" y1="0" y2="1">
<stop offset="0" stop-color="rgb(203,206,208)" />
<stop offset="1" stop-color="rgb(132,131,139)" />
</linearGradient>
<linearGradient id="r-grad-2" x1="0" x2="1" y1="0" y2="1">
<stop offset="0" stop-color="rgb(39,109,195)" />
<stop offset="1" stop-color="rgb(22,92,170)" />
</linearGradient>
</defs>
<path
d="M361.453,485.937 C162.329,485.937 0.906,377.828 0.906,244.469 C0.906,111.109 162.329,3.000 361.453,3.000 C560.578,3.000 722.000,111.109 722.000,244.469 C722.000,377.828 560.578,485.937 361.453,485.937 ZM416.641,97.406 C265.289,97.406 142.594,171.314 142.594,262.484 C142.594,353.654 265.289,427.562 416.641,427.562 C567.992,427.562 679.687,377.033 679.687,262.484 C679.687,147.971 567.992,97.406 416.641,97.406 Z"
fill="url(#r-grad-1)"
fill-rule="evenodd"
/>
<path
d="M550.000,377.000 C550.000,377.000 571.822,383.585 584.500,390.000 C588.899,392.226 596.510,396.668 602.000,402.500 C607.378,408.212 610.000,414.000 610.000,414.000 L696.000,559.000 L557.000,559.062 L492.000,437.000 C492.000,437.000 478.690,414.131 470.500,407.500 C463.668,401.969 460.755,400.000 454.000,400.000 C449.298,400.000 420.974,400.000 420.974,400.000 L421.000,558.974 L298.000,559.026 L298.000,152.938 L545.000,152.938 C545.000,152.938 657.500,154.967 657.500,262.000 C657.500,369.033 550.000,377.000 550.000,377.000 ZM496.500,241.024 L422.037,240.976 L422.000,310.026 L496.500,310.002 C496.500,310.002 531.000,309.895 531.000,274.877 C531.000,239.155 496.500,241.024 496.500,241.024 Z"
fill="url(#r-grad-2)"
fill-rule="evenodd"
/>
</svg>

Some files were not shown because too many files have changed in this diff Show More