Commit Graph

71 Commits

Author SHA1 Message Date
Ruben Fiszel 6407d9ff5c fix(bash): normalize CRLF line endings before running scripts (#10131)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-15 12:46:26 +02:00
Ruben Fiszel afddfe8445 feat(worker): #ssh directive to run a bash script on a remote SSH host (#9479)
* feat(worker): #ssh directive to run a bash script on a remote SSH host

Add a first-class `#ssh <resource_path>` bash directive that reroutes a
normal bash script to run on a remote host reached over SSH (a
jump/utility node) instead of on the worker, with full parity: typed
positional args in, structured result out, live streamed logs,
cancellation, and remote exit-code propagation.

It mirrors the existing `# sandbox <image>` precedent: the directive is
parsed in handle_bash_job and reroutes to a specialized handler that
reuses handle_child for all execution plumbing.

- windmill-common: BashAnnotations::ssh_target() parser (+ unit test)
  and the ssh_execution_enabled instance setting (off by default)
- windmill-worker: reroute hook in bash_executor + ssh_executor_oss
  shim. OSS returns a clear "enterprise feature" error; the real
  handler lives in ssh_executor_ee.rs (private feature) and is gated by
  a valid enterprise license + the instance setting.
- examples/usecase/ssh-execution-wrapper: the ssh_target resource type,
  a userland wrapper (no-license fallback), and a README documenting
  both paths and the trade-offs vs agent workers.

EE companion: windmill-labs/windmill-ee-private (ee-repo-ref.txt bumped).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(worker): ssh host-key opt-in, 0600 key write, instance setting UI

* chore: update ee-repo-ref

* feat(worker): #ssh $arg form to take the ssh target from a job argument

* fix(worker): #ssh token must look like a target; $arg restricted to path strings

* fix(worker): tighten #ssh parser to exact directive; add -- ssh destination guard

* chore: update ee-repo-ref to d45b9a6cbe40f7fe5d322c850c50f64a6980e4f0

This commit updates the EE repository reference after PR #609 was merged in windmill-ee-private.

Previous ee-repo-ref: 2804f1aa8e74b3a7733aeb6f5044d5085193872a

New ee-repo-ref: d45b9a6cbe40f7fe5d322c850c50f64a6980e4f0

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-06-10 15:14:37 +00:00
Ruben Fiszel 1727271e19 feat: sandboxed daemonless container runtime via '# sandbox <image>' (#9453)
* feat: add sandboxed docker v2 runtime via '# docker <image>'

Run a container image as a subprogram of the job's own nsjail sandbox:
extract the image rootfs with podman (rootless) and run it chrooted inside the
job's nsjail, so the container inherits the job's confinement and is safe under
nsjail / for untrusted code. Selected by '# docker <image>'; a bare '# docker'
keeps the v1 (dind) path untouched.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: default to daemonless docker (drop dind from compose, allow docker on cloud)

docker-compose no longer ships the dind sidecar (v2 is daemonless: podman + nsjail
in the worker); removed the dind service, DOCKER_HOST env, depends_on and volume.
Removed the language-picker guard that blocked Docker scripts on the multi-tenant
platform, now that v2 makes docker safe to run sandboxed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat: select sandboxed container via # sandbox <image>; add pull policy + size guards

- Surface moved from '# docker <image>' to '# sandbox <image>' (groups under the
  sandbox annotation; '# docker' stays v1-only, '# sandbox' stays nsjail-bash).
- SANDBOX_IMAGE_PULL_POLICY (default 'newer') so moving tags don't go stale.
- SANDBOX_IMAGE_MAX_SIZE_MB rejects oversized images before extraction.
- SANDBOX_IMAGE_CACHE_MAX_MB best-effort LRU eviction of podman's image store.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): support # volume, honor nsjail tmp instance settings, v2 docker template

- Thread shared_mount into the sandbox container nsjail config so '# volume' mounts
  (and the same-worker /tmp/shared folder) apply inside the container.
- Use resolve_nsjail_tmp_mount_block for the container's /tmp so it honors the same
  nsjail_tmp_backing / nsjail_tmpfs_size_mb instance settings as other nsjail jobs.
- docker-compose comment + the editor's Docker template now use '# sandbox <image>'.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): make image size/cache/pull-policy UI instance settings

Convert SANDBOX_IMAGE_* from worker env vars to DB-backed instance settings
(sandbox_image_max_size_mb, sandbox_image_cache_max_mb, sandbox_image_pull_policy),
hot-reloaded via the same mechanism as nsjail_tmpfs_size_mb and configurable in
#superadmin-settings. No worker restart needed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* feat(sandbox): windmill-managed registry — default registry + private auth

Two new instance settings:
- sandbox_image_default_registry: prepended to unqualified image refs (alpine ->
  <registry>/alpine); fully-qualified refs untouched.
- sandbox_registry_auth: docker/podman auth.json blob written to a per-job authfile
  (0600, removed with the job) and passed to podman --authfile for private registries.
Both hot-reloaded and configurable in #superadmin-settings.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): protobuf-safe proto_str escaper, atomic 0600 authfile, registry tests

Addresses local-review P2s: proto_str now emits valid protobuf octal escapes for
control/non-ASCII bytes (not Rust \u{..} that nsjail would reject); the registry
authfile is created 0600 atomically (no world-readable window); add a
registry_qualified table test + a non-ASCII proto_str case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P0 — deliver image env via nsjail envar:, never the launcher process env

CI review (P0): the image's OCI Env (attacker-controlled keys+values) was applied to
the nsjail launcher process via .envs(), so a hostile image could set LD_PRELOAD/
LD_LIBRARY_PATH/LD_AUDIT on nsjail itself and execute code as the worker outside the
jail. Now the image env is rendered as proto-escaped 'envar:' directives (child-only)
and nsjail's process env carries only windmill-trusted keys (reserved vars + proxy).
Also: warn instead of silently bypassing the size guard on inspect failure; reset the
eviction guard via a Drop guard (no stuck flag on panic/early-return). +render_envars test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): P0 symlink-write escape via rootfs script; P1 redact registry-auth logging

CI review:
- P0 (Codex): the body was written into the image-controlled rootfs as
  .windmill_docker_main.sh via write_file (follows symlinks) — a hostile image could
  plant that path as a symlink to a host file and capture the worker's write before
  nsjail starts. Now the body is passed straight to 'sh -c <body> sh <args>'; no file
  is written into the rootfs at all.
- P1 (Codex): sandbox_registry_auth flowed through the generic setting loader which
  logs the value (raw auth.json credentials). Replaced with a secret-aware reload that
  loads directly and logs only a redacted 'configured=' message.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix(sandbox): redact sandbox_registry_auth in instance-settings write log too

The settings API also logs 'Set global setting <key> to <value>' via format_setting_value;
add sandbox_registry_auth to SENSITIVE_SETTINGS so the credential is redacted there as
well as on reload.

* fix(sandbox): don't silently disable cache eviction on podman images parse error

Re-review (cubic/Claude P2): serde_json::from_slice(...).unwrap_or_default() meant any
parse hiccup (e.g. podman omitting Size/Created via omitempty for a zero value, or
schema drift) silently degraded to an empty Vec and disabled eviction with no log.
Now Size/Created are #[serde(default)] (a missing omitempty key -> 0, not a whole-array
parse failure) and a real parse error warns + breaks instead of being swallowed.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-05 08:35:51 +00:00
Ruben Fiszel b656dc6cdc feat(nsjail): optional disk-backed /tmp via instance setting (#9272)
* feat(nsjail): optional disk-backed /tmp via instance setting

* test(nsjail): unit-test tmp mount resolver and narrow visibility

* refactor(nsjail): switch tmp backing to select + conditional UI

* ui(nsjail): make tmpfs the visible default in /tmp backing select

* fix(nsjail): refuse preexisting jail_tmp to block symlink escape

* fix(nsjail): allow jail_tmp reuse on sequential nsjail calls

Codex flagged that python/ruby/rust executors invoke nsjail twice per
job_dir (install then run). The previous resolver treated any preexisting
jail_tmp as hostile and silently fell back to tmpfs on the second call,
so disk-backed mode never reached the main script run for those langs.

Use symlink_metadata().is_dir() to distinguish a real directory left by
an earlier call in the same job_dir (safe to reuse) from a symlink or
other entity (still refused, as the codebase-tar escape requires).

Also loosen the frontend visibility predicate: only hide nsjail settings
when job_isolation is explicitly 'none' or 'unshare', so deployments
that enable nsjail via DISABLE_NSJAIL=false with no DB setting can
still see the controls.
2026-05-21 15:34:49 +00:00
Ruben Fiszel 9111f8908d feat(nsjail): make tmpfs size configurable via instance setting (#9261)
* feat(nsjail): make tmpfs size configurable via instance setting

Adds a new `nsjail_tmpfs_size_mb` instance setting that overrides the
size of the `/tmp` tmpfs mount inside the nsjail sandbox across all
languages. When unset, the existing per-language defaults (500MB or
800MB) continue to apply, so no behavior change for existing
deployments.

The setting is exposed under Settings → Jobs and is read at job
execution time, so changes take effect on the next job without a
restart.

Fixes WIN-1963

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* refactor(nsjail): unify default tmpfs size to 800MB

Previously each executor passed its own per-language default (500MB or
800MB) to resolve_nsjail_tmpfs_size. Unify on a single
DEFAULT_NSJAIL_TMPFS_SIZE_BYTES constant (800MB) so the placeholder
behavior is consistent across languages.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(nsjail): resolve tmpfs size outside ruby download closure

The download.ruby config render runs inside a sync closure passed to
par_install_language_dependencies_seq, so `.await` on
resolve_nsjail_tmpfs_size() was a compile error under the `ruby`
feature. Resolve the size once before the closure and capture the
string instead.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* docs(nsjail): rename resolver to *_bytes and clarify fallback

Addresses CI review feedback:
- Rename `resolve_nsjail_tmpfs_size` to `resolve_nsjail_tmpfs_size_bytes`
  so the returned unit is unambiguous at the call site (cubic P2).
- Fix the `NSJAIL_TMPFS_SIZE_MB` doc comment that still said "per-language
  default" — there is no per-language fallback anymore, all unset
  values resolve to the unified 800MB `DEFAULT_NSJAIL_TMPFS_SIZE_BYTES`
  (codex/pi P2).
- Expand the resolver doc to call out that `Some(0)` and negative values
  also fall back, since the match arm is `Some(mb) if mb > 0`.

No behavior change.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-20 13:24:49 +00:00
Ruben Fiszel 9c85565221 fix: bypass OTEL MITM tracing proxy for git sync jobs (#8796)
Git sync runs as a DeploymentCallback job. When the OTEL MITM tracing
proxy is enabled, all HTTP/HTTPS traffic from the script is rerouted
through a local intercepting proxy that chains to the corporate upstream
proxy. Git's HTTPS to GitHub fails in this setup (TLS interception with
chained CONNECT tunneling is fragile, and git's CA env handling diverges
from what the proxy injects), so customers see "GitHub.com URL couldn't
be reached" until they disable OTEL.

Detect DeploymentCallback jobs in get_proxy_envs_for_lang and fall back
to the stock PROXY_ENVS so git talks to the corporate proxy directly,
unmodified. The git sync script is system code; we don't need HTTP spans
for it.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-10 16:32:19 +00:00
Ruben Fiszel 5501b7a729 replace host docker socket with dind sidecar for isolation (#8531)
* feat: replace host docker socket with dind sidecar for isolation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: comment out dind sidecar by default to avoid wasting resources

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: enable dind by default, comment out insecure host socket mount

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 21:33:17 +00:00
hugocasa 65a92d9899 fix: set nsjail time_limit from job timeout so configured defaults are respected (#8389)
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 12:06:31 +00:00
Ruben Fiszel 8b1fe8f9de fix: gracefully handle uninitialized OTEL tracing proxy port (#8274)
* fix: gracefully handle uninitialized OTEL tracing proxy port

When OTEL tracing proxy is enabled but the MITM proxy port hasn't been
assigned yet (race condition at startup, or NUM_WORKERS > 1), fall back
to standard proxy envs instead of failing the job with
"OTEL tracing proxy port not initialized".

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: log to job logs when OTEL tracing proxy is unavailable

When the OTEL tracing proxy is enabled but the port isn't initialized
(race at startup or NUM_WORKERS > 1), append a warning to the job logs
explaining why HTTP request tracing is unavailable for that job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 09:34:21 +00:00
Ruben Fiszel 077779ec52 fix: improve windows compatibility
* ci: add Windows backend integration test workflow

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ci: temporarily add push trigger for testing

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* ci: add --no-fail-fast to run all test binaries

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: Windows path handling for backend integration tests

- WINDMILL_DIR: use std::env::temp_dir() on Windows instead of /tmp/windmill
- HOME_ENV: fall back to USERPROFILE on Windows when HOME is not set
- loader.bun.js: normalize paths to forward slashes for consistent
  comparison with Bun's resolver output on Windows
- bun_executor.rs: convert job_dir to forward slashes in JS template
  strings to avoid backslash escape issues (\t -> tab, etc.)
- go_executor.rs: fix windows_gopath() double backslash bug (r"\\" -> "\\")
- bash_executor.rs: default to "bash" (in PATH) on Windows instead of /bin/bash

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: improve Windows diagnostics and fix onLoad handler

- Include path in create_directory_async/sync panic messages
- Add WINDMILL_DIR initialization debug output
- Fix loader.bun.js onLoad: use properly escaped regex instead of
  returning undefined (Bun requires onLoad to return an object)
- Add env var debug output to CI workflow

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: sanitize Windows-invalid characters in test worker names and fix cargo path

- Replace :: with __ in worker names (colons illegal in Windows dir names)
- Fix HOME_DIR to fall back to USERPROFILE on Windows
- Add PATH fallback for cargo discovery on Windows
- Add debug logging to bun loader for fetch errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: handle single colons in worker names, pass MSVC linker env vars, revert bun debug

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use .exe binary name on Windows and normalize bun import URL paths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use absolute path for rust binary, normalize bun resolve paths

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use .wurl extension instead of .url for bun import resolution on Windows

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use custom namespace for bun plugin to bypass default file resolution

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use virtual namespace for bun import resolution to avoid Windows path issues

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: handle Windows 8.3 paths and namespace-prefixed importers in bun loader

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: strip namespace prefix from args.path and handle absolute imports without leading slash in bun loader

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: simplify bun loader and remove redundant cargo path lookups

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: use platform-specific cargo binary path with .exe on Windows

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: replace HOME_DIR with HOME_ENV in rust_executor to remove duplication

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* refactor: keep original bun loader on linux, use virtual namespace loader only on windows

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-03-04 20:20:18 +00:00
Ruben Fiszel 424ca59dfe feat: make WINDMILL_DIR configurable via environment variable (#8215)
* fix: auto-heal corrupted python runtime cache on remote workers

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* Revert "fix: auto-heal corrupted python runtime cache on remote workers"

This reverts commit 0ea013a554.

* feat: make WINDMILL_DIR configurable via environment variable

Allow users to configure the base directory for Windmill's tmp/cache files
via the WINDMILL_DIR env var (default: /tmp/windmill). This fixes Python
runtime cache corruption on RHEL systems where systemd-tmpfiles-clean
removes files from /tmp.

Converts TMP_DIR (renamed to WINDMILL_DIR) and all derived cache directory
constants from compile-time const &str (concatcp!) to runtime lazy_static
String values.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref ERROR_DIR lazy_static for AsRef<Path> and Display traits

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref to branch name for CI compatibility

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref lazy_static constants in all executor files

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee ref

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: panic if WINDMILL_DIR has trailing slash

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: also reject trailing backslash in WINDMILL_DIR for Windows

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: deref GO_BIN_CACHE_DIR in test utils

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: replace remaining hardcoded /tmp/windmill paths and validate empty WINDMILL_DIR

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: nsjail powershell mount dst, Windows path assumptions, pwsh deref consistency

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore Windows /tmp path translation in go and bun executors

The Windows path translation replaces /tmp with the Windows temp dir
(e.g. C:\tmp) before normalizing slashes. Without this, the default
WINDMILL_DIR=/tmp/windmill produces paths without a drive letter on
Windows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: update ee-repo-ref to 6fd5a2ce908235a17975ad4dbdf0051cd89334f3

This commit updates the EE repository reference after PR #436 was merged in windmill-ee-private.

Previous ee-repo-ref: e8c03e16720833230ebd1878b4c63642ecc6c80f

New ee-repo-ref: 6fd5a2ce908235a17975ad4dbdf0051cd89334f3

Automated by sync-ee-ref workflow.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2026-03-04 08:53:25 +00:00
Ruben Fiszel 2646629194 feat: add force_sandboxing global setting and #sandbox bash annotation (#7816)
* feat: add force_sandboxing global setting and #sandbox bash annotation

Add a new global setting `force_sandboxing` to enable nsjail sandboxing
for all jobs, overriding the DISABLE_NSJAIL environment variable.
Also add a #sandbox annotation for bash scripts to enable sandboxing
on a per-script basis.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* fix: decouple nsjail probing from DISABLE_NSJAIL and apply force_sandboxing to all executors

NSJAIL_AVAILABLE now always probes for the nsjail binary at startup
regardless of the DISABLE_NSJAIL policy flag, fixing #sandbox annotation
and force_sandboxing being unusable in default (DISABLE_NSJAIL=true)
configuration. All language executors now use is_sandboxing_enabled()
instead of checking DISABLE_NSJAIL directly, so force_sandboxing
applies to all job types, not just bash.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: clarify force_sandboxing and DISABLE_NSJAIL relationship

Neither setting overrides the other — sandboxing is enabled when either
force_sandboxing=true OR DISABLE_NSJAIL=false. Updated frontend
description and backend comments to reflect this.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* all

* all

* revert typo

* all

* all

---------

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
2026-02-12 18:20:22 +00:00
Pyra 95df7b9a6a feat: otel REST tracing (#7571) 2026-01-20 13:38:02 +00:00
Alexander Petric c28e77110e fix: do not use unshare for init scripts (#7418) 2025-12-19 19:36:51 +00:00
Alexander Petric 5aa251a2d2 feat(security): unshare pid of worker job process (#7106)
* feat(security): unbind pid for worker jobs to create extra process isolation

* review

* simplify

* cleanup + compose files

* re-add removed comments from nu executor

* simplify

* fail immediately

* updates

* update ping backend

* nsjail / unshare in workers page

* migrations

* frontend + sqlx

* frontend

* frontend

* fix error message

* undo example changes
2025-11-18 23:04:31 +00:00
hugocasa bd3e5e67bb feat(backend): allow specifying powershell module versions (#6781)
* feat(backend): allow specifying powershell module versions

* fix build

* fix windows
2025-10-09 16:23:27 +00:00
hugocasa 898eb6231b feat(backend): array and object params support in pwsh (#6706)
* feat(backend): array and object params support in pwsh

* add bool and improve arg conversion

* fix bash

* update parser

* sqlx
2025-09-30 12:25:12 +00:00
hugocasa 4bbbeb956f feat: powershell private repo support (#6684)
* feat: powershell private repo support

* nits
2025-09-29 13:24:41 +00:00
hugocasa 993baf46bd feat(backend): flow streaming (#6520)
* feat(backend): flow streaming

* all streaming languages + sync api

* sqlx

* fix build

* UI and nits

* nit

* feat: stream last flow step

* sqlx

* nit

* use get for stream endpoints + add snippet in UI

* refactor

* nits

* Update backend/windmill-worker/src/common.rs

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>

* nits

---------

Co-authored-by: graphite-app[bot] <96075541+graphite-app[bot]@users.noreply.github.com>
2025-09-18 15:49:35 +00:00
claude[bot] fcd58191d4 fix: force stdin to Stdio::null for all user code execution (#6575)
Set stdin to Stdio::null for all Commands that execute user code across all supported languages to prevent unwanted input consumption. This affects Python, Deno, Bash, PowerShell, Go, Rust, PHP, Ruby, Java, C#, Ansible, Nu, and Bun executors.

The dedicated worker handler was intentionally left unchanged as it requires stdin for inter-process communication.

Co-authored-by: claude[bot] <209825114+claude[bot]@users.noreply.github.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
2025-09-10 16:14:25 +00:00
Ruben Fiszel 1892895cd9 disable process group for dotnets 2025-08-13 16:46:14 +00:00
Ruben Fiszel ce4177ebae fix: improve docker logs collection in docker mode 2025-07-28 23:48:28 +00:00
hugocasa 79205839b6 fix: add windir env var to pwsh for legacy modules like AD (#6271) 2025-07-24 17:38:26 +00:00
dieriba c4178c05be feat: periodic worker group script (#6159)
* feaat periodic script

* feat periodic script

* fix unused

* check minimun time

* fix unused import

* update repo ref

* update ref

* nits and update repo ref

* add missing checks

* fix

* improve UX

* improve UX

* improve UX

* fix

---------

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2025-07-21 22:57:56 +00:00
Ruben Fiszel b83aca30d2 feat: use process groups to improve zombie job handling (#6157)
* useProcessGroups

* nit
2025-07-10 08:49:56 +00:00
HugoCasa 29f6fab60c fix: pwsh skip already installed modules outside of cache (#6037) 2025-06-24 12:41:54 +02:00
Ruben Fiszel 0f63d03093 dind compile 2025-06-17 08:14:11 +02:00
Ruben Fiszel 38eb71bdf5 fix: always rm containers in docker mode 2025-06-17 02:32:49 +02:00
Ruben Fiszel 5b96bccedd feat: add aws oidc support for instance s3 storage (#5810)
* backend

* iterate

* all

* all

* all

* iterate

* revert

* all

* add tracing to get of authed client

* all

* all

* lal

* all

* update

* fix

* push

* all

* all

* revert

* frontend

* fix checks

* avoid deadlock

* safer

* fix

* fix
2025-05-25 14:03:38 +02:00
Ruben Fiszel af74653b7f improve err message 2025-05-22 19:48:26 +02:00
Ruben Fiszel 7c24fbcef2 fix: improve docker mode unexpected exit handling 2025-05-22 19:37:47 +02:00
Ruben Fiszel dfd8c4cd2a more verbose docker wait errors 2025-05-22 01:07:29 +02:00
Ruben Fiszel d35a7d22f9 fix: add support for log compaction on docker jobs (#5732)
* improve docker compact

* improve docker compact

* update ref

* update

* update agent workers
2025-05-13 21:42:15 +02:00
Ruben Fiszel 63fa499015 feat: agent workers v2 using http (#5588) 2025-04-10 00:56:11 +00:00
pyranota 59740c0478 feat: add java support (#5458)
* feat: add nu (nushell) support

* add worker tests

* deactivate tables and non-any types below top-level

full support will come in V1
for V0 it's better to keep things minimal and simple

* add syntax highlighting

used python's grammar, since nushell isn't supported by monaco nor svelte-highlights

for V1 nu will get it`s own grammar

* add logo

* partially implement plugin support

* change logo + ability to deploy + nsjail draft

* static variables + get_resource + get_variable

* lsp/dev.nu + initial nu lsp (not working yet)

* make it work with nsjail

* nullguard

* Much more flexible signature parsing and better error-messages

* add init script

* rename nulsp to nu

* install nu to dockerfile

* fix merge

* implement Default for MainArgSignature

* stage NU_CACHE_DIR

* improve dockerfiles

* dev.nu for parser-wasm + flake.nix

* update code for windows

* add nushell to flake

* upload Cargo.lock

* make build.sh work on nixos

* build wasm cli parsers

* add docs to README_DEV.md

* add helper script docker/dev.nu

* improve docker/dev.nu

* fix windows

* commit frontend/package(lock).json

* update cargo.lock

* correctly update cargo.lock

* remove lsp

* update flake.nix to include svelte server and nushell

* Revert base.sql to main

* remove PLUGIN_USE_RE

* make CARGO_PATH private

* add nu to cli

* Change flags to build wasm-nu-parser

* remove flake.nix from parser-wasm

* update wasm-build target

* remove unused import

* add cli support for nu

* update github workflows

* wasm-build 0.17 -> 0.19

* update build script

* update cargo.lock

* Fix typographical error

* start working on java

* do java boilerplate

* implement parser for java

* update Cargo.lock

* update ENV_SETTINGS

* use published nu parser

* update package.lock

* java is S3 + Caching enabled

* install nsjail backup

* commit v0

* fix nsjail

* v0.1

* rewrite parser in tree-sitter

* implement parser from scratch

* polishing

* change init script to match new parser

* fix imports

* fix cli build

* fix cli build

* refactor install phase

* implement .valid.windmill atomic verification

* implement java init functionality

* remove quick-xml

* fix windows not recognizing 'mvn'

* create empty settings.xml if there is no config provided

* clean up

* change default settings.xml

* change classpath format for windows

* docs to helper

* java copy bin cache instead of symlink

* remove comments

* merge

* fix package.json

* fix package.json 2

* minor fixing

* migrate to Coursier

* update misc

* Http(s) Proxy + CA certs

* remove unused .wasm

* make requirements insensitive to spaces

* update handle_child refs

* rework save_cache for directories

* fix s3 bug

* compile .wasm for cli

* remove uuid import

* fix compilation

* use reference

* fix zero-dep failure

* removing unsafe stuff

* remove unneeded imports

* revert: we still need winapi

* remove nix store from nsjail

* do not create cache_nomount

* add java to dnt

* remove duplicated dependency in init script

* fix typos

* fix CI

* use published parser
2025-03-27 15:12:55 +01:00
Ruben Fiszel 3edca4bc91 perf: optimize number of queries needed for job run (#5504)
* optPerf

* update sqlx

* update sqlx
2025-03-24 20:12:26 +01:00
Ruben Fiszel a6cf327f74 fix: improve memory metrics graph 2025-03-20 09:54:55 +01:00
Ruben Fiszel 880db319e8 feat: filter by worker + backend perf opt (#5489)
* all

* all

* all

* refactor

* sqlx

* ref

* impr

* fix

* fix

* fix

* ee private
2025-03-19 09:55:53 +01:00
Ruben Fiszel e92a46b088 fix: init_script do not need to use nsjail even in nsjail mode 2025-02-22 01:04:33 +01:00
Ruben Fiszel 52ad48a910 back to pipe in bash for efficiency purposes 2025-02-18 20:08:25 +01:00
Ruben Fiszel 138cedf1da fix(bash): improve bash last line as result reliability using bash process substitution (#5321) 2025-02-18 15:13:28 +01:00
Lucas Abel 8db69ce15e fix(backend): fix zombies left by the bash executor (#4985) 2024-12-27 14:23:05 +01:00
Ruben Fiszel 2c5d07f3d9 feat: docker runtime 2024-12-09 18:02:18 +01:00
Ruben Fiszel 5404ec9b48 fix: pass USERPROFILE on windows 2024-12-05 10:12:44 +01:00
Ruben Fiszel 8bc9a021a8 fix(bash): correctly propagate exit errors 2024-11-08 16:33:01 +01:00
Ruben Fiszel e11cc13c92 fix: apply NO_PROXY and HTTP_PROXY, HTTPS_PROXY more consistently 2024-11-02 13:57:30 +01:00
Ruben Fiszel 2cd46cc55f fix(bash): correctly propagate sigterm for cancelled bash scripts 2024-10-24 14:42:37 +02:00
Alexander Petric c070df12e8 improving pwsh error handling (#4504)
* making pwsh error handling more robust

* cargo fmt
2024-10-09 05:20:46 +02:00
Alexander Petric 6eb7f8ce81 feat(worker): support workers to run natively on windows (#4446)
* minimal code change to get windmill worker on windows for bun and python + rustfmt

* adding support for powershell

* compiling error on unix

* rust linting comments

* comments hugo: PSModulePath

* comments ruben, refactor to simplify

* adding build workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* editing workflow

* skip migration env, ee fixes

* improvements powershell

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* testing windows runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* install postgres on runner

* killing process tree in windows

* sqlx_offline

* install openssl for github windows runner

* used pre-installed openssl

* used pre-installed openssl

* build ee

* build ee

* build ee

* build ee

* adding commented out steps for artifact publishing

* build on tag matchinv v* pattern

* ren instead of mv on Windows

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* fix merging issue

* gate imports for windows

* fixing default cargo home path...

* fixing default cargo home path...

* comments ruben

* make pwsh default modules loading more robust on unix (#4448)

Co-authored-by: Ruben Fiszel <ruben@windmill.dev>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: HugoCasa <hugo@casademont.ch>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
2024-10-03 15:59:12 +02:00
Ruben Fiszel bd6ddc166b feat(cli): add queues, workers and worker-groups commands (#4439)
* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all

* all
2024-09-26 11:27:37 +02:00