Commit Graph

17 Commits

Author SHA1 Message Date
Ruben Fiszel 9c85565221 fix: bypass OTEL MITM tracing proxy for git sync jobs (#8796)
Git sync runs as a DeploymentCallback job. When the OTEL MITM tracing
proxy is enabled, all HTTP/HTTPS traffic from the script is rerouted
through a local intercepting proxy that chains to the corporate upstream
proxy. Git's HTTPS to GitHub fails in this setup (TLS interception with
chained CONNECT tunneling is fragile, and git's CA env handling diverges
from what the proxy injects), so customers see "GitHub.com URL couldn't
be reached" until they disable OTEL.

Detect DeploymentCallback jobs in get_proxy_envs_for_lang and fall back
to the stock PROXY_ENVS so git talks to the corporate proxy directly,
unmodified. The git sync script is system code; we don't need HTTP spans
for it.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 16:32:19 +00:00
Ruben Fiszel a46aa641f9 feat: add R language support (#8263)
* feat: add R language support

Add R as a new supported scripting language in Windmill, following the
same pattern used for Ruby. Includes:

- Backend: ScriptLang::Rlang enum variant, DB migration, tree-sitter-r
  parser crate with tests, WASM parser binding, R executor with NSJail
  sandboxing, job dispatch and signature parsing
- Frontend: language picker, R icon, syntax highlighting, editor bar
  insertions (Sys.getenv, get_variable, get_resource), schema inference,
  init code template, BETA badge
- CLI: .r extension mapping, sync support, bootstrap template

R scripts use `main <- function(...)` syntax, jsonlite for JSON
serialization, and system curl for the Windmill client helper.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add R package resolution and installation

Parse library()/require() calls from R scripts to extract dependencies.
Resolve versions from CRAN, cache lockfiles in pip_resolution_cache,
and install packages to a shared R library cache. The run step sets
R_LIBS_USER so installed packages are available to the script.

- Parser: parse_r_requirements() extracts package names from AST
- Executor: resolve() generates lockfile, install() installs from CRAN
- Worker lockfiles: wire up R resolve for dependency jobs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: add nsjail sandboxing for R resolve and install phases

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix R get_variable/get_resource and add sandbox annotation + e2e tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: fix R arg inference with JS fallback parser and get_variable/get_resource

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix flake

* nsjail

* nits

* fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits

- Suppress renv verbose output during resolve and install (controlled by #verbose annotation)
- Filter renv from install list (already loaded, causes noisy restart message)
- Log compact "resolved N packages" instead of full renv.lock JSON
- Add R (r-base, r-cran-renv) to DockerfileFull and DockerfileFullEe
- Use disable_rl for nsjail install config (R compiles from source)
- Reduce default concurrency from 20 to 5
- Add rlang to openflow.openapi.yaml
- Fix MainArgSignature (no_main_func -> auto_kind) after main merge

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* final

* fix: remove accidental R install from multiplayer Dockerfile

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: remove R from Windows build and DockerfileExtra

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: rename R migration to avoid timestamp collision with trigger_filter_logic

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* all

* fix: R install improvements - suppress verbose output, flat lockfile logging, Dockerfile R support, rlimits

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: add clear error when Rscript binary is missing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: fix type errors in R fallback parser, use format! in wrap(), add R system prompts

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: pyranota <pyra@duck.com>
2026-04-01 06:11:37 +00:00
Ruben Fiszel 73fe45b6cb feat: workspace-specific registry overrides (#8406)
* feat: add workspace-specific registry overrides

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: move workspace registries to end of registries tab

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: workspace overrides use field selector instead of showing all fields

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* style: polish workspace registries UI to match design guidelines

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: show field selector directly and fix addField initialization logic

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: namespace pip_resolution_cache by workspace when registry overrides exist

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: namespace binary/bundle caches by workspace when registry overrides exist

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* perf: zero-cost cache suffix when no workspace overrides exist

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* feat: reload workspace_registries via notify events on setting change

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address PR review findings

- Fix discardCategory not reverting workspace_registries changes
- Fix get_no_default: convert to async fn with owned Uuid param
- Fix append_logs: use windmill_queue import already available
- Fix ruby URL parsing: support both comma and whitespace delimiters
- Add WorkspaceRegistryMap type alias to reduce inline type noise

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* all

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-17 12:49:25 +00:00
hugocasa 65a92d9899 fix: set nsjail time_limit from job timeout so configured defaults are respected (#8389)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 12:06:31 +00:00
Ruben Fiszel 8b1fe8f9de fix: gracefully handle uninitialized OTEL tracing proxy port (#8274)
* fix: gracefully handle uninitialized OTEL tracing proxy port

When OTEL tracing proxy is enabled but the MITM proxy port hasn't been
assigned yet (race condition at startup, or NUM_WORKERS > 1), fall back
to standard proxy envs instead of failing the job with
"OTEL tracing proxy port not initialized".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: log to job logs when OTEL tracing proxy is unavailable

When the OTEL tracing proxy is enabled but the port isn't initialized
(race at startup or NUM_WORKERS > 1), append a warning to the job logs
explaining why HTTP request tracing is unavailable for that job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 09:34:21 +00:00
Ruben Fiszel 424ca59dfe feat: make WINDMILL_DIR configurable via environment variable (#8215)
* fix: auto-heal corrupted python runtime cache on remote workers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "fix: auto-heal corrupted python runtime cache on remote workers"

This reverts commit 0ea013a554.

* feat: make WINDMILL_DIR configurable via environment variable

Allow users to configure the base directory for Windmill's tmp/cache files
via the WINDMILL_DIR env var (default: /tmp/windmill). This fixes Python
runtime cache corruption on RHEL systems where systemd-tmpfiles-clean
removes files from /tmp.

Converts TMP_DIR (renamed to WINDMILL_DIR) and all derived cache directory
constants from compile-time const &str (concatcp!) to runtime lazy_static
String values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref ERROR_DIR lazy_static for AsRef<Path> and Display traits

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref to branch name for CI compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref lazy_static constants in all executor files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: panic if WINDMILL_DIR has trailing slash

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: also reject trailing backslash in WINDMILL_DIR for Windows

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref GO_BIN_CACHE_DIR in test utils

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: replace remaining hardcoded /tmp/windmill paths and validate empty WINDMILL_DIR

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: nsjail powershell mount dst, Windows path assumptions, pwsh deref consistency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore Windows /tmp path translation in go and bun executors

The Windows path translation replaces /tmp with the Windows temp dir
(e.g. C:\tmp) before normalizing slashes. Without this, the default
WINDMILL_DIR=/tmp/windmill produces paths without a drive letter on
Windows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee-repo-ref to 6fd5a2ce908235a17975ad4dbdf0051cd89334f3

This commit updates the EE repository reference after PR #436 was merged in windmill-ee-private.

Previous ee-repo-ref: e8c03e16720833230ebd1878b4c63642ecc6c80f

New ee-repo-ref: 6fd5a2ce908235a17975ad4dbdf0051cd89334f3

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-03-04 08:53:25 +00:00
Ruben Fiszel 423e07376b feat: gate private registry configuration behind enterprise feature (#7953)
Add read_ee_registry<T> helper that returns None and emits a job log
in CE builds when a private registry is configured. Applied to all
language executors: pip, npm/bun, deno, cargo, go, maven, nuget,
powershell, and ruby. Maven settings.xml file write in monitor.rs
is also gated.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-15 14:02:14 +00:00
Ruben Fiszel 2646629194 feat: add force_sandboxing global setting and #sandbox bash annotation (#7816)
* feat: add force_sandboxing global setting and #sandbox bash annotation

Add a new global setting `force_sandboxing` to enable nsjail sandboxing
for all jobs, overriding the DISABLE_NSJAIL environment variable.
Also add a #sandbox annotation for bash scripts to enable sandboxing
on a per-script basis.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: decouple nsjail probing from DISABLE_NSJAIL and apply force_sandboxing to all executors

NSJAIL_AVAILABLE now always probes for the nsjail binary at startup
regardless of the DISABLE_NSJAIL policy flag, fixing #sandbox annotation
and force_sandboxing being unusable in default (DISABLE_NSJAIL=true)
configuration. All language executors now use is_sandboxing_enabled()
instead of checking DISABLE_NSJAIL directly, so force_sandboxing
applies to all job types, not just bash.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: clarify force_sandboxing and DISABLE_NSJAIL relationship

Neither setting overrides the other — sandboxing is enabled when either
force_sandboxing=true OR DISABLE_NSJAIL=false. Updated frontend
description and backend comments to reflect this.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* all

* revert typo

* all

* all

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-12 18:20:22 +00:00
Ruben Fiszel e1a815f6a0 refactor: extract windmill-dep-map crate for parallel api/worker compilation (#7846)
* refactor: extract windmill-dep-map crate for parallel api/worker compilation

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: resolve WebhookShared type mismatch and missing enterprise propagation

- Make windmill-api webhook_util re-export from windmill-common instead of
  duplicating types, fixing Extension<WebhookShared> mismatch between
  windmill-store and windmill-api
- Add windmill-api-jobs/enterprise to windmill-trigger enterprise feature
  so check_license_key_valid is available when trigger subcrates enable
  enterprise on windmill-trigger

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: stop trigger features from unconditionally enabling enterprise

Move enterprise propagation for all trigger subcrates from individual
trigger feature definitions to the enterprise feature itself, so
enterprise is only enabled when explicitly requested.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* refactor: remove unused pub use re-exports and disable CI cargo cache

- Remove unused re-exports from windmill-worker/src/lib.rs:
  trigger_dependents_to_recompute_dependencies, handle_job_error,
  and unused bun/otel items
- Fix callers to use direct module paths instead
- Add windmill-dep-map as dev-dependency for tests
- Disable cargo cache in backend-check CI (faster from-scratch builds)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore bun re-exports used by tests

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* chore: re-enable cargo cache for check_ee_full CI job

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-08 00:39:56 +00:00
Pyra 95df7b9a6a feat: otel REST tracing (#7571) 2026-01-20 13:38:02 +00:00
Pyra d30ef89a6c feat(python): restart installation OOM (#7507)
* use tini with unshare to preserve signals (e.g oom)

* fix ci

* ci as nsjail

* simplify

* fix flaky go integration test

* feat(python): restart installation OOM

Signed-off-by: pyranota <pyra@duck.com>

* follow ups

Signed-off-by: pyranota <pyra@duck.com>

* fix oom_score_adj

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: Alexander Petric <alex@windmill.dev>
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>
2026-01-08 06:30:15 +00:00
Alexander Petric 5aa251a2d2 feat(security): unshare pid of worker job process (#7106)
* feat(security): unbind pid for worker jobs to create extra process isolation

* review

* simplify

* cleanup + compose files

* re-add removed comments from nu executor

* simplify

* fail immediately

* updates

* update ping backend

* nsjail / unshare in workers page

* migrations

* frontend + sqlx

* frontend

* frontend

* fix error message

* undo example changes
2025-11-18 23:04:31 +00:00
Pyra 6052714833 fix(ruby): propagate error correctly (#7046)
Signed-off-by: pyranota <pyra@duck.com>
2025-11-04 16:01:49 +01:00
Ruben Fiszel aeb6829011 use excluded where relevant 2025-10-16 09:19:44 +00:00
hugocasa 993baf46bd feat(backend): flow streaming (#6520)
* feat(backend): flow streaming

* all streaming languages + sync api

* sqlx

* fix build

* UI and nits

* nit

* feat: stream last flow step

* sqlx

* nit

* use get for stream endpoints + add snippet in UI

* refactor

* nits

* Update backend/windmill-worker/src/common.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nits

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-09-18 15:49:35 +00:00
claude[bot] fcd58191d4 fix: force stdin to Stdio::null for all user code execution (#6575)
Set stdin to Stdio::null for all Commands that execute user code across all supported languages to prevent unwanted input consumption. This affects Python, Deno, Bash, PowerShell, Go, Rust, PHP, Ruby, Java, C#, Ansible, Nu, and Bun executors.

The dedicated worker handler was intentionally left unchanged as it requires stdin for inter-process communication.

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2025-09-10 16:14:25 +00:00
pyranota 11dd4118ce feat: add ruby support (#5939)
* implement single line pin

* make panic-safe

* use pin even if multiple modules imported withing single statement

* add repins and make imports respect pins

* keep all pins

* Allow multiple pins

* add comments + handle stuff more safely

* fix fully qualified imports

* remove ignore

* sort nested

* apply unique to output requirements list

* fix typo

* remove mut

* update sqlx

* sort imports

* sort imports

* fix formatter and format

* feat: add ruby support

* fix

* update sqlx

* add ruby to flake.nix

Signed-off-by: pyranota <pyra@duck.com>

* upgrade par_install_language_dependencies helper

rename fields + allow passing custom payload

Signed-off-by: pyranota <pyra@duck.com>

* change migrations

Signed-off-by: pyranota <pyra@duck.com>

* make full pipeline work!

Signed-off-by: pyranota <pyra@duck.com>

* do a bit fixing

* stage

Signed-off-by: pyranota <pyra@duck.com>

* add ruby in dockerfiles

Signed-off-by: pyranota <pyra@duck.com>

* implement auto-require + 'windmill/inline'

Signed-off-by: pyranota <pyra@duck.com>

* create 'windmill/inline' only if does not exist

Signed-off-by: pyranota <pyra@duck.com>

* add nsjail

Signed-off-by: pyranota <pyra@duck.com>

* update global settings

Signed-off-by: pyranota <pyra@duck.com>

* add ruby icon

* generate lockfile on deploy (but broken)

Signed-off-by: pyranota <pyra@duck.com>

* fix(frontend): display deployed script lockfile and other info

Signed-off-by: pyranota <pyra@duck.com>

* feat: add mini-wmill client

Methods:
- get_resource
- get_variable

Additionally add shortcuts on frontend

Signed-off-by: pyranota <pyra@duck.com>

* add nit warning

Signed-off-by: pyranota <pyra@duck.com>

* support private repos credentials

Signed-off-by: pyranota <pyra@duck.com>

* fix compilation error

Signed-off-by: pyranota <pyra@duck.com>

* found rustc bug + refactor universal installer

Signed-off-by: pyranota <pyra@duck.com>

* more refactor + fixes

Signed-off-by: pyranota <pyra@duck.com>

* bug fixing + polishing

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* change 'windmill/inline'

Signed-off-by: pyranota <pyra@duck.com>

* delegate env_clear to universal installer

Signed-off-by: pyranota <pyra@duck.com>

* update init script

Signed-off-by: pyranota <pyra@duck.com>

* add cli support

Signed-off-by: pyranota <pyra@duck.com>

* unhardcode frontend

Signed-off-by: pyranota <pyra@duck.com>

* fix cli

Signed-off-by: pyranota <pyra@duck.com>

* fix init script

Signed-off-by: pyranota <pyra@duck.com>

* make it work on ms windows

Signed-off-by: pyranota <pyra@duck.com>

* do not clean env in the end

Signed-off-by: pyranota <pyra@duck.com>

* fix windows (again)

Signed-off-by: pyranota <pyra@duck.com>

* include PROXY_ENVS (untested)

Signed-off-by: pyranota <pyra@duck.com>

* feat: self-signed certs + proxy support (tested)

Signed-off-by: pyranota <pyra@duck.com>

* flake: pin ruby to 3.4

* merge follow-ups

Signed-off-by: pyranota <pyra@duck.com>

* remove obsolete libc crate

Signed-off-by: pyranota <pyra@duck.com>

* cleanup parser

Signed-off-by: pyranota <pyra@duck.com>

* fix java ci

Signed-off-by: pyranota <pyra@duck.com>

* cleanup

Signed-off-by: pyranota <pyra@duck.com>

* fix nits

Signed-off-by: pyranota <pyra@duck.com>

* Update backend/windmill-worker/src/universal_pkg_installer.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Update backend/windmill-worker/src/ruby_executor.rs

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* add nsjail for lock stage

Signed-off-by: pyranota <pyra@duck.com>

* fix merge

Signed-off-by: pyranota <pyra@duck.com>

* Update common.rs

* regen windmill-api-client

Signed-off-by: pyranota <pyra@duck.com>

* fix tests

Signed-off-by: pyranota <pyra@duck.com>

* fix tests again

Signed-off-by: pyranota <pyra@duck.com>

* reference published ruby parser

Signed-off-by: pyranota <pyra@duck.com>

---------

Signed-off-by: pyranota <pyra@duck.com>
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
2025-08-15 13:03:32 +00:00