* refactor: extract object store code into windmill-object-store crate with filesystem backend
Consolidate all object_store-dependent code from windmill-common into a new
windmill-object-store crate. Add a filesystem-backed object store implementation
using LocalFileSystem for dev/testing without cloud credentials. Includes 30
comprehensive tests covering render_endpoint, lfs_to_object_store_resource,
duckdb_connection_settings, error mapping, and filesystem-backed integration tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* all
* all
* all
* fix: fix raw_app hardcoded path, add missing ObjectStoreResource import, and add tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: move S3ModeFormat to windmill-types, make windmill-parser-sql optional, restore debug logs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Move nsjail config, overlayfs, tar utilities, S3 operations, and sandbox
types from windmill-common and windmill-worker into a dedicated
windmill-sandbox crate. S3 logic lives in windmill-ee-private (symlinked
s3_ee.rs). Both windmill-api and windmill-worker depend on the new crate.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
mount_overlay now tries kernel `mount -t overlay` first (needs root or
CAP_SYS_ADMIN), then falls back to fuse-overlayfs (works unprivileged).
unmount_overlay uses fusermount3 -u for FUSE mounts, umount for kernel.
OverlayMount gains an `is_fuse` field to track which backend was used.
The overlay integration tests now exercise the actual mount_overlay()
and unmount_overlay() Rust functions (via fuse-overlayfs on non-root),
verifying read-through, copy-on-write, and cleanup semantics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace placeholder #[ignore] tests with real integration tests:
- overlay_integration: test overlayfs read/write semantics via unshare
(no root required), and overlayfs rootfs as nsjail sandbox root
- crane_integration: test crane export of alpine image → tar/gz
roundtrip, and running the extracted rootfs in nsjail
Also fix tar_gz to preserve symlinks (follow_symlinks=false) instead
of following them to absolute paths that may not exist on the host.
All tests skip gracefully when tools (nsjail, crane, unshare) are
unavailable, so they won't break CI.
Total: 32 passing (with parquet), 14 parser tests = 46 tests.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add 6 nsjail integration tests that verify actual sandbox execution:
- basic bash execution
- volume read (bind-mount data into sandbox)
- volume write-back (sandbox writes to bind-mounted volume)
- overlay root (snapshot rootfs as sandbox root)
- overlay root + volume mount combined
- result.json output from sandbox
Fix a bug in finalize_nsjail_config where the overlay root mount
(dst: "/") was placed last in the config (via {SHARED_MOUNT}). nsjail
applies mounts in order, so a root bind mount after tmpfs/bind mounts
would overwrite them. The root mount is now moved to be the first mount
block, ensuring subsequent mounts layer correctly on top.
Also add 2 #[ignore] tests for mount_overlay (needs root) and
build_snapshot (needs crane CLI).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add support for custom rootfs snapshots and persistent volumes in nsjail
sandboxes, enabling users to run scripts with pre-built environments and
shared persistent storage.
- DB migration: sandbox_snapshot + sandbox_volume tables
- API: CRUD endpoints under /api/w/{ws}/sandbox/
- Common: annotation parser for # sandbox: and # volume: directives
- Worker: S3-backed snapshot cache, overlayfs mount, volume sync
- Executors: nsjail config post-processing to swap rootfs when overlay active
- Tests: 28 tests covering parsing, config generation, and end-to-end flows
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add read_ee_registry<T> helper that returns None and emits a job log
in CE builds when a private registry is configured. Applied to all
language executors: pip, npm/bun, deno, cargo, go, maven, nuget,
powershell, and ruby. Maven settings.xml file write in monitor.rs
is also gated.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: handle $var: and $res: in arrays for transform_json_value
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve $var/$res arrays inside objects in transform_json_value
Replace top_level: bool with depth: u8 to fix the main bug where arrays
nested inside objects were never resolved. The Object arm recursed with
top_level: false, which caused the Array arm guard to reject the value.
With depth <= 2, arrays inside objects (depth 1-2) are now processed.
Also: transform all array elements unconditionally (handles $encrypted:
in worker, reserved $vars, nested structures), and warn on >1000 items.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add force_sandboxing global setting and #sandbox bash annotation
Add a new global setting `force_sandboxing` to enable nsjail sandboxing
for all jobs, overriding the DISABLE_NSJAIL environment variable.
Also add a #sandbox annotation for bash scripts to enable sandboxing
on a per-script basis.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: decouple nsjail probing from DISABLE_NSJAIL and apply force_sandboxing to all executors
NSJAIL_AVAILABLE now always probes for the nsjail binary at startup
regardless of the DISABLE_NSJAIL policy flag, fixing #sandbox annotation
and force_sandboxing being unusable in default (DISABLE_NSJAIL=true)
configuration. All language executors now use is_sandboxing_enabled()
instead of checking DISABLE_NSJAIL directly, so force_sandboxing
applies to all job types, not just bash.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: clarify force_sandboxing and DISABLE_NSJAIL relationship
Neither setting overrides the other — sandboxing is enabled when either
force_sandboxing=true OR DISABLE_NSJAIL=false. Updated frontend
description and backend comments to reflect this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* all
* revert typo
* all
* all
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* Add AWS Bedrock session token support in API and worker
* Add Bedrock auth mode integration tests for AI agents
* Split Bedrock integration test env vars for IAM and session creds
* cleaning
* Add masked Bedrock bearer-token debug info logs
* Revert "Add masked Bedrock bearer-token debug info logs"
This reverts commit 6b2fc5e7c2d5b1c6db81f416a4439941a084108c.
* cleaning
* feat(ai): support 1M context window for Anthropic resources
Add `enable_1m_context` boolean field to Anthropic resource configuration.
When enabled (and not using Vertex AI), sends the `anthropic-beta: context-1m-2025-08-07`
header in both the API proxy layer and the AI agent worker layer.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix(ai): add serde alias for enable_1M_context DB field name
The resource_type schema uses `enable_1M_context` (uppercase M) but
serde only matched `enable_1m_context` and `enable1mContext`, causing
the field to always deserialize as false.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
When eval_timeout_quickjs hits the timeout (typically due to slow DB
result retrieval), retry up to 2 more times with a 5s interval between
attempts. Non-timeout errors are returned immediately without retry.
Also extract the eval timeout duration as EVAL_TIMEOUT_MS const (set to
20000ms, up from 10000ms) in windmill-jseval.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Alexander Petric <alpetric@users.noreply.github.com>
* feat: add prompt caching support for Anthropic API
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* exclude vertex
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Add new BENCHMARK_KIND variants (sequentialflow, scriptlogs, concurrencylimit,
concurrencykey, mixed, mixed_no_cc) for targeted performance testing. Fix shared
iteration counting across workers using a global atomic counter. Add job_perms
inserts and queue diagnostics for benchmark mode.
Move db connection setup to dedicated module and drop the initial connection pool
before creating the main one, preventing connection starvation when PostgreSQL
max_connections is low.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve Windows build warnings treated as errors
- Gate UV_PATH import behind #[cfg(unix)] in python_versions.rs
- Remove unused tokio::time::sleep import in worker.rs (use fully qualified path)
- Fix unused `file` variable warnings in ansible_executor.rs on Windows
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: add Windows cargo check workflow
Runs cargo check with ee_windows features on push to backend/**
using the blacksmith-16vcpu-windows-2025 runner.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* ci: add cargo check step to Windows build, remove separate check workflow
Add a cargo check step with -D warnings before the full build to fail
fast on any warnings. Remove the separate windows-check.yml workflow.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: parse Python datetime.datetime and datetime.date type annotations correctly
The Python parser only matched ExprKind::Name for type annotations, so
`datetime.datetime` (an Attribute expression) silently fell through to
Typ::Unknown and no datetime picker was shown in the UI.
- Extend parse_expr to resolve `datetime.*` attribute access (alongside
the existing `wmill.*` handling)
- Add Typ::Date variant for `datetime.date` → JSON schema format "date"
- Update python worker to import and convert `date.fromisoformat()`
- Update argSigToJsonSchemaType, AI types, schema validation, and SQL
datatype wasm for the new Date variant
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* all
* all
* all
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
- rquickjs: gated behind `quickjs` feature in windmill-jseval, propagated through windmill-worker/windmill-api, added to oss_core
- windmill-autoscaling: made optional in windmill-api (was unconditional), enabled via enterprise feature
- opentelemetry-proto, prost, hudsucker, rcgen, hyper-http-proxy, hyper-tls, hyper-util: made optional in windmill-worker, enabled via enterprise feature
This significantly reduces compilation time for vanilla `cargo check` without features.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: isolate deno_core into windmill-runtime-nativets subcrate
Remove deno_core from flow eval and isolate nativets V8 runtime into a
dedicated subcrate so deno_core compilation no longer blocks
windmill-worker or windmill-api.
- Create windmill-jseval crate: QuickJS-based JS eval for flow
expressions and batch rerun, extracted from windmill-worker
- Create windmill-runtime-nativets crate: all deno_core/V8 deps and
nativets script execution, with build.rs snapshot generation
- Simplify windmill-worker: remove all deno_* direct deps, empty
build.rs, gate nativets behind optional dep
- Update windmill-api: use windmill-jseval for batch rerun instead of
deno_core, remove deno_core feature entirely
- Add nativets integration tests (nativets_jobs.rs) and parallel
stress test (nativets_stress.rs, 8 workers x 200 jobs)
- Remove dead code: deno flow eval path, USE_QUICKJS env var,
parity tests (replaced with 63 standalone expected-value tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address PR review feedback for deno_core isolation
- Deduplicate unsafe_raw() into windmill-common/src/utils.rs (single source)
- Delete orphaned runtime.js and windmill-client.js from windmill-worker/src/
- Fix operator precedence in windmill-jseval with explicit parentheses
- Remove unnecessary return keyword in heap limit callback
- Remove redundant as usize casts
- Remove ~150 lines of commented-out code from runtime.js
- Remove commented-out #[cfg] in build.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* otel ee
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-dep-map crate for parallel api/worker compilation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve WebhookShared type mismatch and missing enterprise propagation
- Make windmill-api webhook_util re-export from windmill-common instead of
duplicating types, fixing Extension<WebhookShared> mismatch between
windmill-store and windmill-api
- Add windmill-api-jobs/enterprise to windmill-trigger enterprise feature
so check_license_key_valid is available when trigger subcrates enable
enterprise on windmill-trigger
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: stop trigger features from unconditionally enabling enterprise
Move enterprise propagation for all trigger subcrates from individual
trigger feature definitions to the enterprise feature itself, so
enterprise is only enabled when explicitly requested.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: remove unused pub use re-exports and disable CI cargo cache
- Remove unused re-exports from windmill-worker/src/lib.rs:
trigger_dependents_to_recompute_dependencies, handle_job_error,
and unused bun/otel items
- Fix callers to use direct module paths instead
- Add windmill-dep-map as dev-dependency for tests
- Disable cargo cache in backend-check CI (faster from-scratch builds)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore bun re-exports used by tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* chore: re-enable cargo cache for check_ee_full CI job
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: reuse outer tx for schedule push in commit_completed_job
Instead of calling handle_maybe_scheduled_job(db) which opens its own
connections (peak=3), inline the schedule push using a savepoint on the
outer transaction. Auth is fetched via the tx connection using
fetch_authed_from_permissioned_as_conn, and push_scheduled_job runs
on a savepoint so failures roll back only the push, not the completion.
On push failure: savepoint rolls back, schedule is disabled on the outer
tx, and the zombie return path is preserved if disabling also fails.
Peak connections drop from 3 to 1 (or 2 on cold RunnableSettings cache).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* fix: extract shared try_schedule_next_job to unify schedule push paths
Replace the two diverging schedule-push implementations (inlined in
commit_completed_job and standalone handle_maybe_scheduled_job) with a
single try_schedule_next_job that reuses the caller's transaction via
savepoints. This eliminates extra pool connection usage in the
worker_flow.rs path and ensures consistent retry/error semantics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add failpoint markers to try_schedule_next_job
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove plan.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove inner retry loop from try_schedule_next_job, add caller-level retries
The 10-retry x 5s-sleep loop inside try_schedule_next_job held locks on
v2_job_completed/v2_job_queue for up to ~45s when running inside the
outer commit_completed_job transaction.
Now try_schedule_next_job makes a single attempt and returns errors to
the caller. Non-retryable errors (QuotaExceeded, NotFound) disable the
schedule immediately inside the function. Transient errors are returned
for the caller to retry:
- commit_completed_job path: outer backon retry (10x3s) retries the
entire transaction including the schedule push, so no locks are held
during sleep.
- handle_flow path: new backon retry (10x3s) wraps begin/push/commit
with a fresh transaction per attempt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: clear push_err after successful schedule disable to prevent stuck schedules
When try_schedule_next_job disables the schedule for non-retryable errors
(NotFound, QuotaExceeded), clear the error so the caller commits the tx
(persisting the disable). Previously, the error propagated up, causing the
tx to be dropped and rolling back the disable — leaving the schedule
permanently enabled but broken.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add 5s timeout on push_scheduled_job, clean up handle_flow error handling
- Add tokio::time::timeout(5s) around push_scheduled_job inside
try_schedule_next_job to bound worst-case lock holding per attempt
- Remove unreachable QuotaExceeded/NotFound match arms in handle_flow
(these errors are handled internally by try_schedule_next_job)
- Add report_error_to_workspace_handler_or_critical_side_channel in
handle_flow when post-exhaustion schedule disable fails
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: return SchedulePushZombieError when both schedule push and disable fail
When handle_flow cannot push the next scheduled job AND cannot disable the
schedule, return a SchedulePushZombieError so the worker leaves the flow job
in the queue for zombie detection to restart. This prevents stuck schedules
where neither the next tick was pushed nor the schedule was disabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* iam
* fix occupancy + log settings change
* ee ref
* ee ref
* sqlx
* chore: update ee-repo-ref to 7f93a13e96c77292ed4b1e63bc1e8ff1e341d283
This commit updates the EE repository reference after PR #408 was merged in windmill-ee-private.
Previous ee-repo-ref: 5b6a4b2f990b7e5bdf6dea14645c787b42a4d9a6
New ee-repo-ref: 7f93a13e96c77292ed4b1e63bc1e8ff1e341d283
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: mssql integrated auth (kerberos/ntlm)
* install krb5 headers
* also make it work for windows
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Ruben FiszelandClaude Opus 4.5
hugocasa
Zhiyuan Zheng
centdix
Devdatta Talele
