Move K8s operator source code (crd, db_sync, reconciler, resolve) to
windmill-ee-private and gate behind feature = "private". OSS stubs
provide error messages when the feature is disabled.
Add an info Alert banner in the Registries settings section when no
enterprise license is active.
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add type annotations to opaque CRD schema fields
The WindmillInstance CRD failed K8s structural schema validation because
four `Option<serde_json::Value>` fields (`object_store_cache_config`,
`secret_backend`, `slack`, `teams`) generated schemas without a `type`
property. Kubernetes requires all specified object fields to declare a
type.
Add a `schemars(schema_with)` annotation that emits
`{"type": "object", "nullable": true, "x-kubernetes-preserve-unknown-fields": true}`
for these opaque settings fields.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: enable operator in CE builds
The operator feature was only included in `ee_core`, meaning CE images
couldn't run `windmill operator` as a proper k8s controller — the
subcommand silently fell through to standalone mode.
Add `operator` to `ce_core` so CE builds include the k8s controller
that watches and reconciles WindmillInstance CRDs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* Add windmill-operator crate for Kubernetes CRD-based instance config
Introduces a new `windmill-operator` crate that enables declarative
management of Windmill instance configuration via a Kubernetes
`WindmillInstance` CRD. The operator watches CRD resources and performs
full declarative sync of global_settings and worker configs to the
database, supporting GitOps workflows for instance-level configuration.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Add tests for windmill-operator CRD and db_sync
- 9 unit tests for CRD serialization, deserialization, metadata, and
status field behavior
- 15 integration tests for db_sync using #[sqlx::test] with full
declarative sync coverage: upsert, delete, protected keys,
idempotency, worker config prefix handling, and end-to-end sync
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Replace untyped BTreeMap CRD fields with typed structs for schema validation
GlobalSettings, SmtpSettings, IndexerSettings, and WorkerGroupConfig now
have explicit typed fields with serde(flatten) catch-all for forward
compatibility. The generated CRD YAML includes a full OpenAPI v3 schema
that Kubernetes validates on kubectl apply.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Type opaque serde_json::Value CRD fields with real structs
Replace most remaining serde_json::Value fields in WindmillInstance CRD
with properly typed structs derived from the codebase:
- oauths: BTreeMap<String, OAuthClient>
- otel: OtelSettings
- otel_tracing_proxy: OtelTracingProxySettings with ScriptLang enum
- critical_error_channels: Vec<CriticalErrorChannel> (untagged enum)
- critical_alerts_on_db_oversize: DbOversizeAlert
- ducklake_settings: DucklakeSettings with nested catalog/storage types
- custom_instance_pg_databases: CustomInstancePgDatabases
- autoscaling (worker config): AutoscalingConfig with integration struct
- custom_tags, default_tags_workspaces: Vec<String>
- default_tags_per_workspace: bool
Still opaque (serde_json::Value): object_store_cache_config (kube-core
can't generate schemas for internally-tagged enums), secret_backend
(EE-private), slack, teams (no clear struct definitions).
Regenerated CRD YAML with full OpenAPI schema (352→703 lines).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Unify instance config types and add bulk GET/PUT API
Move all typed settings (GlobalSettings, WorkerGroupConfig, etc.) from
windmill-operator/crd.rs into windmill-common/instance_config.rs so both
the API server and operator share a single source of truth. Add diff/apply
logic (Merge mode for UI, Replace mode for operator) and InstanceConfig::from_db().
Add GET/PUT /settings/instance_config endpoints so the frontend loads all
settings in 1 call instead of 42, and saves with a single bulk PUT. The
backend handles the diff internally, running pre-write hooks for changed keys.
Refactor windmill-operator/db_sync.rs to use the shared diff+apply functions
and slim crd.rs down to the CRD wrapper with re-exports.
Includes 32 unit tests and 30 integration tests covering serialization,
diff logic, DB roundtrips, protected settings, and edge cases.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add Form/YAML toggle to instance settings UI
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: show Form/YAML toggle regardless of hideTabs prop
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: replace toggle button group with simple YAML toggle
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: mask sensitive fields in YAML view with show/hide toggle
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: hide internal settings and mask sensitive fields in YAML view
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* feat: hide jwt_secret and min_keep_alive_version from API and config exports
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* all
* feat: add secretKeyRef support for sensitive fields in operator CRD
Allow sensitive fields (license_key, hub_api_secret, scim_token,
smtp_password, OAuthClient.secret, custom PG user_pwd) to reference
Kubernetes Secrets via the standard secretKeyRef pattern instead of
inlining values as plaintext YAML. The reconciler resolves all refs
by reading K8s Secrets before syncing to the database.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* all
* all
* fix: merge main and update dev environment docs
Resolve merge conflicts from origin/main, fix duplicate
UV_INDEX_STRATEGY_SETTING import, and add Playwright MCP
testing instructions to CLAUDE.md.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* fix: init tracing for CLI subcommands and deduplicate setting side-effects
Initialize tracing subscriber before early-return CLI paths (sync-config,
operator) so tracing calls are not silently dropped. Refactor
set_global_setting_internal to call run_setting_pre_write_hook instead of
duplicating the side-effect logic.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* feat: add `wmill instance get-config` CLI command
Dumps the current instance config (global settings + worker configs) as
YAML. Supports --output-file to write to a file instead of stdout.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add 7 #[ignore] e2e tests (one per trigger type) that fire real messages
to external services and verify job creation in v2_job. Also add 9 DB-level
CRUD tests for MQTT, GCP, and Email triggers.
Includes helper shell scripts in tests/fixtures/ to start/stop each
external service (MQTT, WebSocket, Postgres replication, Kafka, NATS,
SQS via LocalStack, GCP Pub/Sub emulator).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The test file used old JobPayload/FlowValue field names that were
refactored into DebouncingSettings/ConcurrencySettings structs.
Remove the test file, fixture, feature flag, and cfg gate in monitor.rs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- rquickjs: gated behind `quickjs` feature in windmill-jseval, propagated through windmill-worker/windmill-api, added to oss_core
- windmill-autoscaling: made optional in windmill-api (was unconditional), enabled via enterprise feature
- opentelemetry-proto, prost, hudsucker, rcgen, hyper-http-proxy, hyper-tls, hyper-util: made optional in windmill-worker, enabled via enterprise feature
This significantly reduces compilation time for vanilla `cargo check` without features.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-api-scripts and windmill-api-users subcrates
Split the monolithic windmill-api crate by extracting scripts.rs, flows.rs,
users.rs, and users_oss.rs into dedicated subcrates. This reduces incremental
rebuild times when editing these modules.
Changes:
- Create windmill-api-scripts crate (scripts.rs + flows.rs, ~4.3K lines)
- Create windmill-api-users crate (users.rs + users_oss.rs, ~2.4K lines)
- Move clear_schedule to windmill-queue (shared by scripts, flows, workspaces)
- Move username utilities (VALID_USERNAME, INVALID_USERNAME_CHARS,
generate_instance_wide_unique_username) to windmill-common/src/usernames.rs
- Move COOKIE_DOMAIN, IS_SECURE, WithStarredInfoQuery, BulkDeleteRequest,
WebhookShared to windmill-common for cross-crate access
- Original files in windmill-api become thin stubs with pub use re-exports
- EE-dependent route handlers remain in windmill-api (create_user, rename_user,
set_password, reset_password, etc.)
- Feature forwarding for enterprise, private, parquet, no_auth
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-api-workspaces subcrate (Step 3)
Move workspaces.rs, workspaces_extra.rs, workspaces_oss.rs, and
workspaces_ee.rs into a new windmill-api-workspaces crate (~7K lines).
Routes that depend on windmill-api internals (AI copilot, teams,
tarball export, critical alerts, stripe) remain in the windmill-api
stub. The subcrate handles all other workspace management routes.
Also moved send_email_if_possible to windmill-common/email_oss.rs
to make it available across subcrates without circular deps.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* refactor: extract windmill-api-groups subcrate (groups.rs + folders.rs)
Extract groups.rs (1,093 lines) and folders.rs (833 lines) into a new
windmill-api-groups subcrate. Both modules had clean dependencies on
already-extracted crates (windmill-api-auth, windmill-common,
windmill-api-workspaces). Also removes unused re-exports of
get_instance_username_or_create_pending and INVALID_USERNAME_CHARS
from windmill-api/src/utils.rs.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: add granular_acls.rs and folder_history.rs to windmill-api-groups
Extract granular_acls.rs (395 lines) and folder_history.rs (68 lines) into
the windmill-api-groups subcrate. Both modules only depend on already-extracted
crates and belong to the same access-control domain as groups and folders.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove unused imports and dead code from subcrate extraction
- Remove unused BASE_URL import from lib.rs
- Remove workspaces_extra.rs and workspaces_oss.rs re-export stubs (no consumers in windmill-api)
- Remove dead send_email_if_possible OSS stub (callers moved to windmill-api-users)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* chore: bust CI cargo cache for subcrate split
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: re-export BASE_URL for EE files that use crate::BASE_URL
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: forward no_auth feature to windmill-api-users, remove dead code
- Add "windmill-api-users/no_auth" to windmill-api's no_auth feature
so the login bypass in users.rs:1600 activates correctly
- Remove dead send_email_if_possible from windmill-api-users/users_oss.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: re-enable cargo cache for backend tests
Cache was disabled to bust stale entries from before subcrate split.
Now that a clean build has run, re-enable for faster CI.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: install mold+clang in CI workflows
The .cargo/config.toml uses mold linker for x86_64-linux.
Build scripts require linking even during cargo check.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: increase cargo test timeout to 30 min
Exit code 143 (SIGTERM) means the 20-min timeout was hit during
compilation without cache. Bump to 30 min as safety net.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: limit cargo build jobs to 4 to prevent OOM in CI
Exit code 143 (SIGTERM) after 8 min = OOM kill during compilation.
8 parallel LLVM codegen jobs exhaust memory on ubicloud-standard-8.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: isolate deno_core into windmill-runtime-nativets subcrate
Remove deno_core from flow eval and isolate nativets V8 runtime into a
dedicated subcrate so deno_core compilation no longer blocks
windmill-worker or windmill-api.
- Create windmill-jseval crate: QuickJS-based JS eval for flow
expressions and batch rerun, extracted from windmill-worker
- Create windmill-runtime-nativets crate: all deno_core/V8 deps and
nativets script execution, with build.rs snapshot generation
- Simplify windmill-worker: remove all deno_* direct deps, empty
build.rs, gate nativets behind optional dep
- Update windmill-api: use windmill-jseval for batch rerun instead of
deno_core, remove deno_core feature entirely
- Add nativets integration tests (nativets_jobs.rs) and parallel
stress test (nativets_stress.rs, 8 workers x 200 jobs)
- Remove dead code: deno flow eval path, USE_QUICKJS env var,
parity tests (replaced with 63 standalone expected-value tests)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: address PR review feedback for deno_core isolation
- Deduplicate unsafe_raw() into windmill-common/src/utils.rs (single source)
- Delete orphaned runtime.js and windmill-client.js from windmill-worker/src/
- Fix operator precedence in windmill-jseval with explicit parentheses
- Remove unnecessary return keyword in heap limit callback
- Remove redundant as usize casts
- Remove ~150 lines of commented-out code from runtime.js
- Remove commented-out #[cfg] in build.rs
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* otel ee
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-dep-map crate for parallel api/worker compilation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve WebhookShared type mismatch and missing enterprise propagation
- Make windmill-api webhook_util re-export from windmill-common instead of
duplicating types, fixing Extension<WebhookShared> mismatch between
windmill-store and windmill-api
- Add windmill-api-jobs/enterprise to windmill-trigger enterprise feature
so check_license_key_valid is available when trigger subcrates enable
enterprise on windmill-trigger
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: stop trigger features from unconditionally enabling enterprise
Move enterprise propagation for all trigger subcrates from individual
trigger feature definitions to the enterprise feature itself, so
enterprise is only enabled when explicitly requested.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: remove unused pub use re-exports and disable CI cargo cache
- Remove unused re-exports from windmill-worker/src/lib.rs:
trigger_dependents_to_recompute_dependencies, handle_job_error,
and unused bun/otel items
- Fix callers to use direct module paths instead
- Add windmill-dep-map as dev-dependency for tests
- Disable cargo cache in backend-check CI (faster from-scratch builds)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: restore bun re-exports used by tests
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* chore: re-enable cargo cache for check_ee_full CI job
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-api into 4 subcrates (api-auth, store, api-sse, api-jobs)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: eliminate refresh_token OnceLock bridge in windmill-store
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: eliminate FromRequestParts OnceLock bridge in windmill-api-auth
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: wire subcrates into workspace and clean up unused re-exports
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: resolve cargo check --all-features errors in subcrate wiring
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* sqlx
* all
* chore: update ee-repo-ref for warning fixes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* refactor: extract windmill-trigger crate and expand windmill-api-jobs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract windmill-trigger-kafka crate from windmill-api
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract windmill-trigger-postgres crate from windmill-api
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract windmill-trigger-websocket and windmill-trigger-mqtt crates
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract windmill-trigger-nats, sqs, gcp, and email crates
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract windmill-trigger-http crate from windmill-api
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: move token creation and permission helpers to windmill-api-auth
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: extract windmill-native-triggers crate from windmill-api
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* sqlx
* all
* refactor: extract windmill-api-embeddings crate and fix CI warnings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: resolve type mismatch in oauth2_oss and remaining warnings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: use correct HTTP_CLIENT config in embeddings crate (30s timeout, cert override)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* all
* fix: gate oauth_refresh_ee on oauth2 feature to fix warnings
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* all
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: reuse outer tx for schedule push in commit_completed_job
Instead of calling handle_maybe_scheduled_job(db) which opens its own
connections (peak=3), inline the schedule push using a savepoint on the
outer transaction. Auth is fetched via the tx connection using
fetch_authed_from_permissioned_as_conn, and push_scheduled_job runs
on a savepoint so failures roll back only the push, not the completion.
On push failure: savepoint rolls back, schedule is disabled on the outer
tx, and the zombie return path is preserved if disabling also fails.
Peak connections drop from 3 to 1 (or 2 on cold RunnableSettings cache).
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* fix: extract shared try_schedule_next_job to unify schedule push paths
Replace the two diverging schedule-push implementations (inlined in
commit_completed_job and standalone handle_maybe_scheduled_job) with a
single try_schedule_next_job that reuses the caller's transaction via
savepoints. This eliminates extra pool connection usage in the
worker_flow.rs path and ensures consistent retry/error semantics.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* test: add failpoint markers to try_schedule_next_job
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* chore: remove plan.md
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: remove inner retry loop from try_schedule_next_job, add caller-level retries
The 10-retry x 5s-sleep loop inside try_schedule_next_job held locks on
v2_job_completed/v2_job_queue for up to ~45s when running inside the
outer commit_completed_job transaction.
Now try_schedule_next_job makes a single attempt and returns errors to
the caller. Non-retryable errors (QuotaExceeded, NotFound) disable the
schedule immediately inside the function. Transient errors are returned
for the caller to retry:
- commit_completed_job path: outer backon retry (10x3s) retries the
entire transaction including the schedule push, so no locks are held
during sleep.
- handle_flow path: new backon retry (10x3s) wraps begin/push/commit
with a fresh transaction per attempt.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: clear push_err after successful schedule disable to prevent stuck schedules
When try_schedule_next_job disables the schedule for non-retryable errors
(NotFound, QuotaExceeded), clear the error so the caller commits the tx
(persisting the disable). Previously, the error propagated up, causing the
tx to be dropped and rolling back the disable — leaving the schedule
permanently enabled but broken.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: add 5s timeout on push_scheduled_job, clean up handle_flow error handling
- Add tokio::time::timeout(5s) around push_scheduled_job inside
try_schedule_next_job to bound worst-case lock holding per attempt
- Remove unreachable QuotaExceeded/NotFound match arms in handle_flow
(these errors are handled internally by try_schedule_next_job)
- Add report_error_to_workspace_handler_or_critical_side_channel in
handle_flow when post-exhaustion schedule disable fails
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: return SchedulePushZombieError when both schedule push and disable fail
When handle_flow cannot push the next scheduled job AND cannot disable the
schedule, return a SchedulePushZombieError so the worker leaves the flow job
in the queue for zombie detection to restart. This prevents stuck schedules
where neither the next tick was pushed nor the schedule was disabled.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* feat: kafka trigger kerberos/gssapi support
* chore: update ee-repo-ref to bb32d562120dc34bbd8d659d92a0d4b8824b8c4c
This commit updates the EE repository reference after PR #405 was merged in windmill-ee-private.
Previous ee-repo-ref: 128c6549d4557895a362fb720f56afa54d6f566b
New ee-repo-ref: bb32d562120dc34bbd8d659d92a0d4b8824b8c4c
Automated by sync-ee-ref workflow.
* adding kafka-gssapi to all_sqlx_features
* ee ref
* ee ref
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
* feat: mssql integrated auth (kerberos/ntlm)
* install krb5 headers
* also make it work for windows
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Ruben Fiszel
centdix
hugocasa
Alexander Petric