* feat: add a toggle to leave forks of non-dev workspaces out of dev isolation
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: track the non-dev forks toggle reactively and bound the forking rule checks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: point ee-repo-ref at the skip non-dev forks telemetry registration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: keep forks of archived dev workspaces isolated and warn about forkable non-dev forks
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: reset tag lineage caches on every archive and unarchive
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* perf: cache the token dev lineage lookup and trim isolation hot-path costs
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: point ee-repo-ref at the merged skip non-dev forks telemetry registration
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* feat: route dev workspace and fork jobs to their own worker groups
* fix: harden dev workspace isolation against review findings
* fix: verify hashed dedicated tags and keep dev lineage across renames
* fix: let tag-scoped tokens reach jobs queued on dev workspace tags
* fix: expand tag scopes only in dev workspaces and forks
* fix: expand tag scopes only while dev workspace isolation is on
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: collapse the dev workspace isolation section by default
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: count dev workspace isolation toggles and dev-serving group flags
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: validate the dev workspace tag suffix on every instance config writer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: update ee-repo-ref to 71e9c0de8317e839f955ebef6a819062f20ac418
This commit updates the EE repository reference after PR #892 was merged in windmill-ee-private.
Previous ee-repo-ref: b5ab84e1705f939c399f6315a03f88b55bce1804
New ee-repo-ref: 71e9c0de8317e839f955ebef6a819062f20ac418
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: export worker cpu, memory and occupancy metrics
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: refresh the exported memory limit with the cgroup readings
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* feat: per-script resource rollup and heaviest scripts table
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: pin the companion ee commit and drop job cpu entries on every exit
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: count the cpu time of suspended workflow rounds
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: count suspended workflow rounds' runtime and guard empty averages
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* perf: flush the runnable rollup hourly in key order and drop its workspace fk
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* fix: order the rollup upsert in sql and sweep rows of deleted workspaces
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* perf: keep the rollup orphan sweep on a primary key range
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
* chore: update ee-repo-ref to a1c00d73a05f227a61d2bf37cc86cd9f5bb1777a
This commit updates the EE repository reference after PR #866 was merged in windmill-ee-private.
Previous ee-repo-ref: c6bcb1bbb5a7e85e870f9de232a0ad1540acf2bd
New ee-repo-ref: a1c00d73a05f227a61d2bf37cc86cd9f5bb1777a
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: let a job see that its worker is draining
* fix: gate worker draining check on job visibility and export sdk helper
* fix: keep draining across re-registration and ignore parked jobs
* fix: reset draining only on worker startup registration
* chore: update ee-repo-ref to 55feb2e84332179faa4e17aeb419f7f480df79a6
This commit updates the EE repository reference after PR #857 was merged in windmill-ee-private.
Previous ee-repo-ref: e5010fa046a3aba93595d1bf44eb6a0cef2b5273
New ee-repo-ref: 55feb2e84332179faa4e17aeb419f7f480df79a6
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* perf: index audit logs by operation and limit each audit table on its own
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: key the audit operation index on id with a trailing timestamp
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: index the legacy audit table by operation too
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: drop the legacy audit index and per-table limit from this PR
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: build the audit operation index in a background migration
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* perf: retire the legacy pre-partitioning audit table
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: bound the legacy audit retirement's lock waits and keep the backfill guard
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: bound the audit index parent's lock wait and retry background steps
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 5370ae3a95a3dc10171f72da74286a654996eee6
This commit updates the EE repository reference after PR #818 was merged in windmill-ee-private.
Previous ee-repo-ref: aa367619a970de231f5c10cad470d9d42169fff8
New ee-repo-ref: 5370ae3a95a3dc10171f72da74286a654996eee6
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(security): a WM_TOKEN job token can never be a global superadmin (GHSA-hfh4-cx4h-3fcr)
Privilege escalation: an app/flow/schedule/trigger execution policy's `on_behalf_of`
(which a `wm_deployers` member can set) could point at a superadmin email. The
resulting job `WM_TOKEN` then passed the email-based superadmin checks, granting
instance superadmin. `forbid_superadmin_job_token` only guarded ~15 of ~75 routes.
Fix at the token layer: a WM_TOKEN must never satisfy a superadmin gate,
regardless of whose email it runs as (sentinel OR a real superadmin).
- `ApiAuthed` gains a `job_id` field, stamped once in `AuthCache::get_opt_job_authed`
from the resolved token's job_id (correct even on cache hits).
- `require_super_admin(db, email)` -> `require_super_admin(db, &ApiAuthed)`, rejects
`authed.job_id.is_some()`. `require_super_admin_email` kept for the few internal
callers without an ApiAuthed.
- `is_super_admin_authed(db, &ApiAuthed)` for the boolean `is_super_admin_email`
authorization branches on request handlers (workspace deletion, fork drops,
dev-workspace attach/archive, object-storage SSRF exemption, custom dbname, EE GHES
+ connected repositories, ...). Migrate ~75 sites (OSS + EE).
- CUSTOM_INSTANCE_DB reads the *authenticated* job_id, not the caller-supplied
`?job_id` query param. Worker-tag check takes a precomputed job-aware `is_super_admin`
on the request path.
Execution-time on-behalf checks (scheduled/flow worker-tag, Cloud enqueue quota,
is_devops_email) are hardened in a follow-up — see
docs/followup-onbehalf-execution-privilege-hardening.md.
Regression tests: a superadmin-email WM_TOKEN is rejected on `require_super_admin`
routes, on `DELETE /workspaces/delete/{w}` (403, workspace preserved), and on the
CUSTOM_INSTANCE_DB lookup with no `?job_id` (401); real superadmin tokens still succeed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: cap devops role at workspace admin and reject reserved on_behalf_of identities
Extends the job-token cap with three pieces:
- `require_devops_role` takes `&ApiAuthed` and rejects job tokens.
`is_devops_email` is true for superadmin emails, so every worker-management,
instance-config and service-log route was reachable by the same superadmin
`WM_TOKEN` that `require_super_admin` already rejects.
- A `job_id` claim that does not parse as a uuid rejects the token rather than
resolving to `None`, which would clear the job provenance and uncap it. Applies
to the internal JWT and the external `jwt_ext_` path.
- Defense in depth at store time: `validate_on_behalf_of` refuses the reserved
internal sentinels as an `on_behalf_of` on apps/flows/scripts/schedules/triggers,
and app execution refuses a policy carrying one — covering already-persisted and
forked-app rows that predate the cap. Deploying on behalf of a real user,
including a real superadmin, stays allowed; the cap handles that at execution.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(mcp): preserve job-token provenance when minting the proxy JWT
The MCP endpoint-tool proxy re-mints a JWT from the caller's ApiAuthed to
forward the proxied request, but passed job_id: None. A job's WM_TOKEN is
capped at workspace admin (GHSA-hfh4-cx4h-3fcr); dropping the job_id here
re-minted an uncapped token that satisfies require_super_admin /
require_devops_role on the proxied route (e.g. listWorkers exposing worker
IPs, job/workspace IDs, and sensitive tags).
Carry api_authed.job_id into create_jwt_token. Adds an in-module regression
that decodes the forwarded JWT and asserts the job_id is preserved for a job
caller and absent for a non-job caller.
Reported by Codex CI review (P1) on #10124.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: cap the admin-or-devops gate at workspace admin for job tokens
require_admin_or_devops (the EE critical-alerts endpoints) grants when the
caller is a workspace admin OR an instance devops. is_devops_email is true
for superadmins, so a WM_TOKEN running on-behalf of a superadmin who is not a
member of the target workspace could clear the devops branch and read/ack that
workspace's critical alerts (GHSA-hfh4-cx4h-3fcr). This gate takes a bare
email, not an ApiAuthed, so the token-layer cap could not see it.
Thread the caller's job-token provenance and reject the devops branch for job
tokens, matching require_devops_role. The workspace-admin branch stays allowed
— that is the cap ceiling. Adds an enterprise-gated regression proving the
bypass is closed and a real superadmin token still clears the gate.
Found while auditing the PR for bare-email gates the choke-point cap misses.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix: cap instance-global is_admin gates at workspace admin for job tokens
Three instance-global routes gate on the caller's own `is_admin` claim, which
`ApiAuthed.is_admin` carries into a WM_TOKEN (it is a workspace-admin claim,
true for superadmins too). A job token is capped at workspace admin
(GHSA-hfh4-cx4h-3fcr), so its is_admin claim must not authorize instance
actions on a route with no workspace binding:
- `unarchive_workspace` — unarchive an arbitrary workspace by id
- `prune_concurrency_group` — delete a global concurrency group
- `list_worker_groups` — return unobfuscated `env_vars_static` (may hold secrets)
Add job-token-aware `is_instance_admin` / `require_instance_admin` helpers (the
same shape as `require_super_admin` / `require_devops_role`) and use them at
these three sites. Workspace-scoped `require_admin(authed.is_admin, ...)` gates
are intentionally left unchanged — a workspace-admin job token is within the
cap there. Regression added covering all three; verified it lets a WM_TOKEN
unarchive/leak without the fix and is blocked with it.
Reported by Codex CI review (P1) on #10124.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* fix(mcp): drop orphaned path_field_renames from EndpointTool test helper
The merge with main adopted main's mcp path-substitution refactor (#10162),
which removed the `path_field_renames` field from `EndpointTool` and its
consumer (`substitute_path_params` no longer takes per-field path renames).
main's `runner.rs` `ep` test helper still constructed the struct with
`path_field_renames: None`, so the workspace test build (cargo test --all,
which compiles windmill-mcp's own #[cfg(test)] module under the `server`
feature) failed with E0560. A plain `cargo check` does not compile that test
module, so it only surfaced in CI's cargo_test.
Remove the orphaned field to match the struct.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
* test: describe the sentinel-rejection policy the forged-identity test asserts
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: complete ApiAuthed initializers in feature-gated tests after merge
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: stop job tokens minting credentials that shed their provenance
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cap the MCP OAuth approval mint at the same elevated-job-token gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cap the self-service password reset at the elevated-job-token gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: cap app embed/SDK mints and scope widening at the elevated-job-token gate
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: keep job tokens from destroying the account they run on behalf of
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: deny job tokens a foreign-workspace admin claim and workspace ejection
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* docs: keep the follow-up inventory in the PR instead of the repo
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: make the session workspace status gate job-token aware
session_workspace_status derived its superadmin branch from a bare email
check, so a job token carrying a superadmin identity resolved the existence
of workspaces it has no relationship with rather than seeing them as
deleted. Switch to is_super_admin_authed, matching every other instance
gate reached from a request ApiAuthed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* revert: leave the global concurrency-group listing on the plain admin gate
The listing exposes concurrency keys across workspaces, which is metadata
rather than a capability, and it 401s rather than degrading. Keep the guard
on the prune route next to it, which is the destructive one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep the instance-admin gate on the global concurrency listing
The listing spans every workspace's concurrency keys, and the gate rejects
only job tokens: the !is_admin branch is the pre-existing check, so
workspaced tokens and interactive admins are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to d30af67d38954f9012f7bad08da23e347344b4c6
This commit updates the EE repository reference after PR #664 was merged in windmill-ee-private.
Previous ee-repo-ref: 7870573dbc3360f99bada143f094c67dce0d9e9c
New ee-repo-ref: d30af67d38954f9012f7bad08da23e347344b4c6
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: hugocasa <hugo@casademont.ch>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat(worker-tags): add `*` fork marker to workspace-scoped custom tags
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test(worker-tags): pin fork-marker tag admission through real lineage
Also render WorkspaceMatcher in its authored form in the operator-facing
"not in the allowed CUSTOM_TAGS" error, and correct the authorization note
on workspace_with_fork_ancestors.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(worker-tags): gate fork-lineage tag lookup on workspace membership
exists_workers_with_tags is a global route whose workspace is a query
param. Resolving its fork lineage for a caller who is not a member
disclosed whether an arbitrary workspace descends from one named by a
tag(parent*) rule. Require membership before the lineage lookup, and
sharpen the workspace_with_fork_ancestors contract accordingly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore(sqlx): add offline cache for fork-marker test INSERT query
The new test_fork_marker_tag_admission_through_lineage integration test
inserts a fork workspace with a bind-parameterized query that had no
cached entry, failing the SQLX_OFFLINE=true CI build.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(queue): audit-log workspace-fairness cap transitions
When the cloud per-workspace fairness mechanism adds a workspace to the
capped set or releases one, write `workspace_fairness.capped` /
`workspace_fairness.uncapped` audit-log entries to the affected workspace.
The cluster admin can review the full timeline from the `admins` workspace
audit view with `all_workspaces=true`; per-workspace owners see their own
events in their normal audit list.
Only the per-cycle refresh winner emits entries (matching where the heavy
aggregation runs), so a fleet of N workers does not produce N duplicates
per transition. The diff is computed against the value already in
`background_task_state` rather than the winner's in-memory cache, so a
freshly-restarted process winning the claim does not spuriously emit
"newly capped" entries for workspaces that were already capped before it
started.
Audit writes are best-effort: failures are logged via tracing and do not
abort the refresh cycle.
Fixes WIN-1984
* feat(queue): scope fairness audit to admins workspace + queue-metrics pane
- Write `workspace_fairness.capped` / `workspace_fairness.uncapped` to the
`admins` workspace (was: per-affected-workspace) with the affected
workspace_id moved to the `resource` field. Cluster admins now get the
full timeline in one place without `all_workspaces=true`.
- Add `GET /workers/workspace_fairness_events` returning the last 100
events. Cloud-gated (returns `[]` on non-cloud) and devops-only.
- Add a `WorkspaceFairnessEvents` Section to the Queue Metrics drawer,
rendered only when `isCloudHosted()` is true. Shows time / event
badge / workspace / parameters with a refresh button.
Fixes WIN-1984
* refactor: extract load helpers from reload_setting family
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert atomic primitive globals to AtomicBool/AtomicI64
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert CRITICAL_*/HUB_API_SECRET/INSTANCE_EVENTS_WEBHOOK/JWT_SECRET to ArcSwap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pin ee-repo-ref to arcswap-refactor EE branch commit
* refactor: convert BASE_URL/HUB_BASE_URL/MIN_VERSION/LICENSE_KEY*/LICENSE_KEY_ID to ArcSwap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert worker hot-path globals to ArcSwap (WORKER_CONFIG et al)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pin ee-repo-ref to combined arcswap-urls+worker EE commit
* chore: update ee-repo-ref to d8be8f88cb8898c8f6b27421989d53528223815d
This commit updates the EE repository reference after PR #532 was merged in windmill-ee-private.
Previous ee-repo-ref: c375aaaac9ec0fc0480993627d0defc8054c31a4
New ee-repo-ref: d8be8f88cb8898c8f6b27421989d53528223815d
Automated by sync-ee-ref workflow.
* fix: cleanup unused imports + fix 2 missed WORKER_CONFIG readers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to ce0f8fbbbde09c4a858312d2d8716d224e99042c
This commit updates the EE repository reference after PR #534 was merged in windmill-ee-private.
Previous ee-repo-ref: 450b601b5aba0ca0b2045f4b5071aa8701b4bfb7
New ee-repo-ref: ce0f8fbbbde09c4a858312d2d8716d224e99042c
Automated by sync-ee-ref workflow.
* fix: secret_backend_integration test — BASE_URL.write().await → .store()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert APP_WORKSPACED_ROUTE to AtomicBool for symmetry with HTTP_ROUTE_WORKSPACED_ROUTE
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to e587df8 (post-#535 merge)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Ruben FiszelandClaude Opus 5.5
wendrul