* fix: trigger git sync for re-encrypted secrets on encryption key change
When changing a workspace encryption key, the secret variables get
re-encrypted with the new key, but the git sync was only dispatched for
the encryption_key.yaml metadata file. Repos with Secrets sync enabled
were left with stale ciphertexts until the next per-variable deployment.
Now, after the transaction commits, we also dispatch a Variable git sync
event for each re-encrypted secret so the new encrypted values are
pushed to the configured repos. Errors are logged but don't roll back
the key rotation.
Fixes WIN-1994
Fixes#9344
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat: batch encryption-key rotation into one git-sync job
Workspace encryption key rotation now re-encrypts every secret variable
and then dispatches a single batched git-sync job carrying the Key event
plus one Variable item per re-encrypted secret. Repos with Secrets sync
enabled receive every new ciphertext in one commit instead of nothing
(previously only `encryption_key.yaml` was pushed) — and instead of N
separate jobs the debouncer might or might not merge.
Wires through the new `handle_deployment_metadata_batch` entry point
added in the companion EE PR; OSS has a no-op shim so the build stays
green.
Adds an integration test (`workspace_encryption_key_git_sync`) asserting
that rotating the key with 3 secret variables in scope produces exactly
one deployment-callback job whose `items` array contains the Key event
+ all 3 variable entries and `skip_secret=false`.
Fixes WIN-1994
Fixes#9344
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref for git-sync helper simplification
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test: cover non-debouncing git-sync fallback on key rotation
Adds a regression test exercising a workspace whose sync script predates
hub version 28103: the rotation must still queue a legacy-format
deployment-callback job per item (encryption_key + each re-encrypted
secret) instead of silently skipping the repo. Bumps ee-repo-ref to the
EE fallback fix.
Addresses the P1 raised in the PR review (Codex/Pi/Claude): batch path
dropped git sync entirely for repos without sync-job debouncing support.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: add sqlx offline cache for encryption-key git-sync test queries
The cargo_test CI job builds with SQLX_OFFLINE=true; the two new
sqlx::query!/query_as! calls in
windmill-api-integration-tests/tests/workspace_encryption_key_git_sync.rs
had no cached entries, failing the build with E0282. Regenerated and
added only the two new query caches (no EE/feature cache loss).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to updated EE companion PR (08e3b9b)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(ai): disable redirect following on AI proxy client to close SSRF
The AI proxy validates the configured base_url against SSRF rules but the
shared HTTP client followed up to 10 redirects without revalidating the
hops, so a public base_url could 3xx the server into a private/internal
address (e.g. the Docker socket or cloud metadata). Disable redirect
following so the validated host is the only one the server connects to.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai): remove heavy redirect SSRF integration test
Drop the integration-test-level regression for redirect following; it
spins up a full API server + DB for a one-line client-config change.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): don't follow symlinks when reading service log files
Defense in depth on top of the existing `..` path-traversal check in
the get_log_file handler: reject the request if the final path
component is a symlink, so a planted symlink in the logs directory
cannot be used to read arbitrary files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(api): authorize and harden the jobs_u get_log_file endpoint
The unauthenticated jobs_u get_log_file endpoint served any job's log
file to anyone who knew the job UUID, with no authorization. Gate it the
same way as get_job_logs: look up the job (the log directory name is the
job id) filtered by workspace and the caller's scope tags, and only allow
non-logged-in callers to read logs of jobs created by the anonymous user.
Also add defense in depth: refuse to read through a symlink so a planted
symlink in the logs directory cannot be used to exfiltrate arbitrary files.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The fork-branch deployment callback runs the hub script
sync-script-to-git-repo-windmill, which imports `windmill-cli` as a pinned
npm dependency and runs it in-process (it does NOT shell out to a PATH wmill).
hub/28236 pinned windmill-cli@1.706.1, whose `git-deploy --only-create-branch`
path returns early without pushing — so the fork branch was checked out
locally but never published to the remote. #9366 fixed the CLI and shipped it
as windmill-cli@1.712.0, but without a hub-script bump the running callback
still used 1.706.1.
Bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28238, which is identical to 28236
except it pins windmill-cli@1.712.0 (content + lockfile). This fixes
test_workspace_fork_creates_branch and production fork-branch creation.
Also add backend/windmill-common/src/workspaces.rs to the git-sync-test
path-gate so future script-path bumps trigger the e2e (the bump alone is not
otherwise covered by the gate).
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
* feat(oauth): support per-provider sandbox URLs in registry + instance settings
* fix(oauth): polish sandbox review nits (cc lookup, header label, ee ref)
* refactor(oauth): drop dead build_oauth_clients duplicate in windmill-oauth
* refactor(oauth): derive sandbox-capable provider list from registry
* chore(docker): copy oauth_connect.json into frontend build stage
* test(oauth): cover sandbox helpers (as_sandbox, canonical_name, resolve)
* chore: update ee-repo-ref to 9297d8f790346e6a6ad540c7bca1a67f91ec11a2
This commit updates the EE repository reference after PR #595 was merged in windmill-ee-private.
Previous ee-repo-ref: 3ab3eca9ac15ebab6db991e7964bc5e48ce21f42
New ee-repo-ref: 9297d8f790346e6a6ad540c7bca1a67f91ec11a2
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* [ee] fix(git-sync): publish fork branch on only_create_branch from the CLI
Fixes WIN-1997. Forking a git-sync-configured workspace must push a
`wm-fork/<branch>/<id>` branch to the repo, but the integration test
`test_workspace_fork_creates_branch` failed: the fork callback job
succeeded yet no branch appeared.
Root cause: the fork-branch callback runs the sync script with
`only_create_branch: true` and no items. The hub sync script delegates
branch checkout to `wmill sync git-deploy --only-create-branch` and runs
its own in-process commit+push ONLY for the `!only_create_branch` path
(`if (!only_create_branch) git_push(...)`). #9284 had moved commit+push
out of the CLI to the caller for the GPG-cache-warmth invariant
(WIN-1974) — but it also dropped the CLI's push for the branch-only
case. A branch-only publish has no commit, so no signing is involved and
the GPG concern does not apply; with neither the CLI nor the hub script
pushing, the empty fork branch was never published.
Restore the CLI push for the `only_create_branch` path (a bare
`git push --porcelain` of the checked-out branch ref). Adds a
deterministic CLI regression test that runs `git-deploy
--only-create-branch` for a fork workspace and asserts the branch
reaches the remote with no caller-side push.
EE companion: format the fork-branch commit message with Display instead
of Debug (no more `Some("...")` leak).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
This commit updates the EE repository reference after PR #597 was merged in windmill-ee-private.
Previous ee-repo-ref: 8b02336fcebdfae4b9d2795cbb74fa7046530bcb
New ee-repo-ref: a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
This commit updates the EE repository reference after PR #597 was merged in windmill-ee-private.
Previous ee-repo-ref: 55c19293232be379a3044eb78f677b545882ffd6
New ee-repo-ref: a30079e75dc5b7d7413aa8ee20e40e80bfea9cbd
Automated by sync-ee-ref workflow.
* [ee] feat(queue): duration-weighted fairness admission atomic
Add the `WORKSPACE_FAIRNESS_ADMISSION_PPM` atomic that the EE
`workspace_fairness_ee::refresh_overloaded` writes on each refresh
(see companion EE PR). The atomic is read on every pull by
`should_admit_capped` to decide whether the dispatch goes down the
standard or fairness path. Defaults to 10_000 (= admit all) so the
pre-fairness behaviour is preserved until the first refresh fires.
OSS stub in `workspace_fairness.rs` continues to return `true`
unconditionally, so non-EE builds are bit-identical.
* docs(queue): consolidate full fairness algorithm into workspace_fairness.rs
Move the algorithm doc — what "overloaded" means in worker-seconds, the
duration-weighted admission derivation, coordinated refresh structure,
audit emission, the SQL perf constraints (no params CTE, drive running
side from v2_job_runtime), and EE gating — into the OSS surface module
where it is readable without EE access. The EE file becomes implementation
only.
Also bump ee-repo-ref to the EE commit that strips the duplicate doc.
* docs(queue): clarify ADMISSION_PPM default is "admit all", not count-based
Addresses CI review (claude[bot]): the `10_000` initial value is the
"admit all" no-op default that applies before the first refresh
classifies an overloaded set — not the count-based value (which would
be `target * 10_000`). The count-based form is the empty-bucket fallback
inside `compute_admission_ppm`, a different thing.
* chore(queue): point ee-repo-ref at EE main (fairness admission merged via #593)
* fix(queue): duration-weighted admission uses unclamped service-time window
Bumps ee-repo-ref to the EE fix (windmill-ee-private#596) that sources
D_c/D_u for the admission probability from a separate 60s service-time
window of true `duration_ms`, instead of the occupancy aggregation whose
per-job contributions are clamped to the 10s occupancy window. The clamp
truncated D_c for capped jobs longer than the window, under-admitting the
duration skew (true 34s jobs → ~86% effective share instead of the target
65%). Occupancy worker-seconds still drive overload classification.
Updates the algorithm doc in workspace_fairness.rs accordingly.
Note: ee-repo-ref points at the EE feature branch; re-point to EE main
once #596 merges.
`announce_server_started` writes a `server_heartbeat:{INSTANCE_NAME}` row
on each startup. INSTANCE_NAME is a fresh random string per process, so
the row is never updated again and a new row is inserted on every
restart, growing background_task_state unboundedly.
Add an hourly monitor task that deletes server_heartbeat:* rows older
than 7 days. Older rows cannot influence check_any_server_started (which
only considers heartbeats refreshed after the restart was initiated), so
they are safe to prune.
Fixes WIN-1990.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(queue): bump EE ref to include worker_ping fairness signal
The current ee-repo-ref.txt pointed to 31cda7c (an unrelated merge
commit on the asset-graph-view-ee branch) instead of ddc9e80, which
contains the workspace-fairness fix that switches the active-share
signal from v2_job_queue.running=true to worker_ping. As a result
cloud was still computing overload off the legacy signal, so a
workspace with many in-flight/suspended flows (lancom01-prod, with
799 suspended flows × 3 v2_job_queue bookkeeping rows each = 2397
running-true rows) was flagged as 95% of cluster activity despite
consuming zero worker slots.
Bumping to ddc9e80 picks up the worker_ping-based signal, which
naturally excludes (a) suspended jobs (no worker pinging them),
(b) zombie running-rows from dead workers, and (c) flow/flownode
orchestration rows that never run on a worker in the first place.
* test(queue): seed v2_job rows + realistic durations for fairness helpers
The new duration-weighted fairness algorithm joins v2_job_queue and
v2_job_completed to v2_job for the `kind` filter (excluding flow
bookkeeping) and reads `duration_ms` for the completed contribution.
Update the test helpers to mirror that schema:
* `insert_completed` now inserts a matching v2_job row (kind=script)
and writes `duration_ms = 1000` with a 1-second [started_at,
completed_at] interval, so each completed row contributes ~1
worker-second when fully inside the refresh window.
* `insert_queued` likewise pre-inserts v2_job, sets `started_at`
to NOW() - 1s when running=true (so running rows contribute ~1
worker-second by the time the refresh runs), and seeds
v2_job_runtime.ping so the running side accrues real-time worker
seconds (the algorithm bounds end-of-interval by ping).
The zombie/suspended insert helpers are intentionally left without
v2_job rows — the new algorithm's INNER JOIN excludes them, so they
still correctly contribute zero worker-seconds.
* chore(queue): bump EE ref to duration-weighted fairness algorithm
Companion to windmill-ee-private#<TBD>: switch the EE workspace
fairness aggregation from a count-based UNION (worker_ping snapshot
+ v2_job_completed count) to a worker-seconds aggregation sourced
directly from v2_job_queue and v2_job_completed, with kind/suspend
filters mirroring handle_zombie_jobs and per-row defenses against
zombie inflation on both halves.
* chore(queue): bump EE ref for fairness perf fix (inline window_start)
* chore(queue): bump EE ref for fairness perf rewrite (driver-side flip)
* update ee ref
`fairness_ignores_zombie_running_rows` and
`fairness_ignores_concurrency_suspended_rows` panic intermittently in CI
(both Linux and Windows runs). Mark them `#[ignore]` until the
underlying flakiness is resolved.
* feat(websocket-trigger): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY (WIN-1988)
`tokio_tungstenite::connect_async` opens a raw TCP socket and ignores
the standard outbound-proxy env vars, so deployments behind a forward
HTTP proxy can't reach the WebSocket endpoint and Test Connection
times out after 30s.
Add a small `proxy` module that resolves the right proxy URL for the
target host (HTTPS_PROXY for wss://, HTTP_PROXY for ws://, NO_PROXY
exclusions, ALL_PROXY fallback, lowercase variants), opens an HTTP
CONNECT tunnel when one applies, and hands the resulting TcpStream to
`client_async_tls_with_config` for the TLS + WS handshake. Direct
connect remains the default when no proxy env is set.
Unit tests cover NO_PROXY matching, proxy URL parsing (including IPv6
literals and basic-auth userinfo), and the CONNECT handshake itself
against an in-process fake proxy (success, basic-auth header, 407
rejection).
Fixes WIN-1988
* refactor(websocket-trigger): reduce blast radius and reuse existing logic
Follow-up to the proxy support change. Three things:
1. Skip the new code path entirely when no proxy is configured.
`connect_async_with_proxy` now checks the env-var snapshots up front
and delegates straight to `tokio_tungstenite::connect_async` if
neither `HTTP_PROXY` nor `HTTPS_PROXY` is set. Same fall-through
applies when proxy env is set but `NO_PROXY` excludes the host or
the proxy URL doesn't parse. Non-proxied deployments now exercise
exactly the previous code path.
2. Move the `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` env-var snapshots
from `windmill-worker::worker` into `windmill-common`. The worker's
`PROXY_ENVS` static now reads from there, and the websocket trigger
reads from the same source — one place reads the env, one source
of truth for both call sites.
3. Replace the hand-rolled proxy-URL parser with `url::Url::parse`
(already a workspace dep, used across the codebase). Half the LoC
and handles edge cases (userinfo percent-encoding, IPv6 literals,
path/query stripping) via the well-tested crate instead of by hand.
All 13 proxy unit tests still pass. `cargo check` is clean.
* fix(websocket-trigger): unbreak EE build + trim proxy tests
- Re-export `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` from
`windmill-worker::worker` (via `pub use windmill_common::...`) so the
EE `otel_tracing_proxy_ee` module's `use crate::{HTTPS_PROXY, ...}`
resolves like it did before. Fixes the `check_ee_full` / `cargo_test`
CI failures from the previous commit.
- Trim the proxy tests to one un-ignored canary
(`http_connect_tunnel_sends_well_formed_request_and_unwraps_stream`)
that exercises the actual on-wire CONNECT handshake plus byte-perfect
tunnel passthrough. The NO_PROXY-matching, URL-parsing, and edge-case
tunnel tests are kept under `#[ignore]` for manual debugging
(`cargo test -- --ignored`) since they're either delegated to
`url::Url::parse` or trivial string matching — low ROI on every CI run.
* fix(jobs): enforce anonymous-only guard on `only_result` job updates
The `jobs_u/getupdate/{id}` and `jobs_u/getupdate_sse/{id}` endpoints
accept `only_result=true`. In that branch, `get_job_update_data` queried
the result solely by (workspace_id, job_id) and skipped the
`created_by == "anonymous"` check that the non-only_result path and
adjacent unauthenticated endpoints apply. An unauthenticated requester
who learned a private job UUID could therefore retrieve that job's
output.
Hoist the guard to the top of `get_job_update_data` so both branches are
covered.
Fixes WIN-1980
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: fold `created_by` check into existing only_result queries
Avoids the extra `SELECT created_by` round-trip per call by joining
`v2_job` once in the two queries that handled the unauth path and
checking inline. Behavior is identical to the prior commit; the SSE
polling loop now does one query per poll instead of two for
unauthenticated callers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: cache anonymous_verified across SSE polls
Replace the LEFT JOIN approach with an upfront `SELECT created_by`
guarded by a new `&mut bool anonymous_verified` parameter that mirrors
`early_return_suppressed`. The SSE polling loop now performs the auth
check exactly once per stream rather than per poll, and the data SQL
reverts to its original form so authenticated callers pay no extra
cost. `created_by` cannot change after job creation, so caching the
verification across polls is safe.
Cost matrix:
- Authed (any path): 0 extra queries
- Unauthed one-shot: 1 extra query (unavoidable)
- Unauthed SSE: 1 extra query at stream start, 0 per poll
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: scope anonymous check to only_result branch
The non-only_result branch already enforces the `created_by` check via
its main query, so a top-level hoisted check duplicated work for
unauthenticated default-path callers. Move the check inside the
`if only_result.unwrap_or(false)` block — exactly where the bypass
lives — and leave the non-only_result path untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ee] feat(service-accounts): allow choosing role at creation time
Previously, service accounts were hardcoded to operator and could not be
used as the CLI sync user since they had no write access. They also only
counted as 0.5 seat each.
This change:
- Extends `NewServiceAccount` to accept optional `is_admin` / `operator`
(defaults to `operator=true` for backward compatibility).
- Exposes a role picker in `AddUser.svelte` when creating a service
account (Operator / Developer / Admin).
- Lets admins update a service account's role from the user list (it
used to be locked to "Operator" with a tooltip).
- Updates the OpenAPI spec + regenerates the frontend client.
A developer/admin service account counts as 1 seat under the existing
seat-cap logic (operators stay at 0.5).
Companion PR on windmill-ee-private updates the `INSERT INTO usr` to
honour the chosen role.
Fixes WIN-1985
* [ee] feat(service-accounts): wm_deployers opt-in for Dev role
When creating a service account with role=Developer, surface a toggle
"Add to wm_deployers" (recommended). Members of wm_deployers can deploy
on behalf of other users — the typical setup when the service account is
used as the CLI sync / CI deploy identity.
- `NewServiceAccount` gains an optional `add_to_deployers` flag.
- Frontend defaults the toggle to on but only shows it under Developer
(admins have it implicitly; operators can't deploy).
- Tooltip links to docs.windmill.dev "Run on behalf of".
Companion EE PR updates the handler to INSERT into usr_to_group for
wm_deployers when the flag is set.
Refs WIN-1985
* chore: update ee-repo-ref to 974ed42067d9f63acb42332b671b8c01ffd4b625
This commit updates the EE repository reference after PR #589 was merged in windmill-ee-private.
Previous ee-repo-ref: f7dbc3cc2ba21c396f4828881e3b9d9ab6f50c69
New ee-repo-ref: 974ed42067d9f63acb42332b671b8c01ffd4b625
Automated by sync-ee-ref workflow.
* [ee] fix(service-accounts): unhardcode role in superadmin user list
Two review issues from the merged #9307 / #589:
1. P1 — The global Users tab in #superadmin-settings still pinned every
service account to "Operator". Now it shows the actual role
(Admin / Operator / Developer), derived from the SA's usr row.
- `list_users_as_super_admin`: replaced `true as operator_only` with
the real `operator` value, and added `is_workspace_admin` from the
row (NULL for password users since their admin status is
per-workspace).
- `global_whoami`: when the email belongs to a service account, look
up its real `operator` / `is_admin` instead of pinning to operator.
- `SuperadminSettingsInner.svelte`: drop the hardcoded "Operator"
badge; render Admin / Operator / Developer using the new fields,
matching the workspace-level view.
2. P2 — Regenerate the bundled `openapi-deref.{yaml,json}` so the
`createServiceAccount` body (now exposing `is_admin`, `operator`,
`add_to_deployers`) and the new `GlobalUserInfo.is_workspace_admin`
field show up at runtime in `/api/openapi.{yaml,json}`.
Bumps `ee-repo-ref.txt` to the EE follow-up that adds the offline
seat-cap check on `create_service_account`.
Refs WIN-1985
* chore: update ee-repo-ref to b7a6068c1f3dc845e012959268b2426f0de4d697
This commit updates the EE repository reference after PR #590 was merged in windmill-ee-private.
Previous ee-repo-ref: 0b1307c21d1bfd6fb43a03c2ba39d2a8bf8e6470
New ee-repo-ref: b7a6068c1f3dc845e012959268b2426f0de4d697
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(settings): skip workspaced-route duplicate checks on cloud
The pre-write validation hooks for `app_workspaced_route` and
`http_route_workspaced_route` query the DB for cross-workspace duplicates
and fail the save when any are found. On cloud both `custom_path_exists`
(apps) and `route_path_key_exists` (HTTP triggers) already scope lookups
by `workspace_id` regardless of these settings, so duplicates across
workspaces are expected and the validation has no runtime meaning. The
result was that any cloud super-admin attempting to save instance
settings with these toggles set to false received
`Duplicate HTTP route paths detected` even though the setting has no
effect on cloud routing.
Fixes WIN-1983
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(error): render JsonErr as readable text and return 400
`Error::JsonErr` previously rendered through `#[error("Error: {0:#?}")]`,
leaking Rust's `Debug` output (`Object { "error": String(...), "details":
Array [...] }`) into the HTTP response body, and was bucketed into the
catch-all 500 branch in `IntoResponse`. The result was a 500 status with
a wall of Rust debug syntax in the toast — confusing and user-hostile.
- Bucket `JsonErr` into 400 (Bad Request): every current call site
(workspaced-route duplicate checks, OAuth client errors, etc.) is a
client/validation issue, not an internal server fault.
- Add `format_json_err_message` which surfaces the `error` field as the
headline, summarises `details` (with a `- key=value` per entry), and
pretty-prints the rest as JSON for unknown shapes. The frontend toast
now reads e.g.
Duplicate HTTP route paths detected
- route_path=a, workspace_id=admins, http_method=post
- route_path=a, workspace_id=starter, http_method=post
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(toast): preserve newlines and escape HTML in multi-line errors
The toast renders via `{@html processMessage(message)}`, so server-side
error bodies that span multiple lines (e.g. the duplicate-route response
from the settings endpoint) collapsed into a single line because HTML
treats consecutive whitespace (including `\n`) as a single space.
When the message contains a newline, escape HTML first (defends against
injected markup in server error bodies) and convert `\n` to `<br />` so
multi-line errors stay readable in the toast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fixup: address CI review feedback
- toast.ts: escape HTML unconditionally. The previous gate on `\n` left
single-line server error bodies unsafe under {@html}, which cubic
flagged as P0. The path regex below only inserts a `<span>` around a
`u/...` or `f/...` capture that can't contain HTML metacharacters, so
escaping the whole input is the simpler and correct fix.
- error.rs: add unit tests pinning the rendered shape of
`format_json_err_message` (error+details, error-only, truncation cap,
non-object fallback to pretty JSON).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(queue): audit-log workspace-fairness cap transitions
When the cloud per-workspace fairness mechanism adds a workspace to the
capped set or releases one, write `workspace_fairness.capped` /
`workspace_fairness.uncapped` audit-log entries to the affected workspace.
The cluster admin can review the full timeline from the `admins` workspace
audit view with `all_workspaces=true`; per-workspace owners see their own
events in their normal audit list.
Only the per-cycle refresh winner emits entries (matching where the heavy
aggregation runs), so a fleet of N workers does not produce N duplicates
per transition. The diff is computed against the value already in
`background_task_state` rather than the winner's in-memory cache, so a
freshly-restarted process winning the claim does not spuriously emit
"newly capped" entries for workspaces that were already capped before it
started.
Audit writes are best-effort: failures are logged via tracing and do not
abort the refresh cycle.
Fixes WIN-1984
* feat(queue): scope fairness audit to admins workspace + queue-metrics pane
- Write `workspace_fairness.capped` / `workspace_fairness.uncapped` to the
`admins` workspace (was: per-affected-workspace) with the affected
workspace_id moved to the `resource` field. Cluster admins now get the
full timeline in one place without `all_workspaces=true`.
- Add `GET /workers/workspace_fairness_events` returning the last 100
events. Cloud-gated (returns `[]` on non-cloud) and devops-only.
- Add a `WorkspaceFairnessEvents` Section to the Queue Metrics drawer,
rendered only when `isCloudHosted()` is true. Shows time / event
badge / workspace / parameters with a refresh button.
Fixes WIN-1984
A token scoped to a single resource (e.g. `resources:read:u/alice/foo`)
could call `GET /api/w/{w}/resources/list_search` and receive `path` and
`value` for unrelated resources in the workspace. Route-level scope
checks only validate `domain:action`; per-resource handlers do a
`check_scopes` against the path, but the listing endpoints did not —
leaking integration credentials, API keys, and other secrets stored as
resource values to narrowly-scoped tokens.
Add `build_scope_path_predicate` to `windmill-api-auth` (mirrors
`check_scopes` semantics but parses the token's scopes once, suitable
for filtering many rows). Apply it to `list_search_resources`,
`list_resources`, `list_names` (resources) and `list_variables`
(non-secret value leak), so a scope-restricted token only ever sees the
paths it is authorized to read. Unscoped tokens and tokens whose only
scopes are `if_jobs:filter_tags:*` are unaffected.
Includes regression tests covering: unscoped, tag-filter-only,
single-resource, wildcard, wrong-domain, and write-implies-read.
Fixes WIN-1981
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(queue): cloud-only per-workspace fairness cap on the shared worker pool
On `app.windmill.dev` the cluster runs a single default worker group, so a
single workspace flooding the queue can degrade quality of service for
everyone else. This adds an opt-in mechanism that caps any single workspace
at a configurable share of the shared worker pool when it has been
dominating cluster activity for more than a configurable window.
Detection signal counts both currently-running jobs and jobs completed in
the rolling window, so it catches workspaces hogging slots with long jobs
**and** workspaces spamming many tiny jobs (where no individual job's
started_at is old, but throughput share dominates).
Refresh is coordinated cluster-wide via a single UPDATE on
`background_task_state`: the `WHERE updated_at < now() - interval` predicate
combined with row-level locking means only one process per refresh cycle
actually runs the aggregation, regardless of fleet size. Every other
process gets the freshly written value in the same round trip via
`UNION ALL ... LIMIT 1`. Heavy aggregation rate stays at ~0.2-0.5 qps for
the whole cluster.
Pull queries are split: the existing query string and its bind shape stay
bit-identical to today, so the planner keeps using the same indexes when
fairness is off or no workspace is currently capped. A separate
`WORKER_PULL_QUERIES_FAIRNESS` adds `AND workspace_id <> ALL($2::text[])`
and is only materialized while the feature is enabled.
Hard-gated to `CLOUD_HOSTED=true` + BASE_URL host == app.windmill.dev at
three layers: frontend `cloudonly: true`, API setter rejection in
`set_global_setting_internal`, runtime check in `fairness_active`. Settings
are exposed under Jobs in the instance-settings UI; defaults are off so
the change is a no-op for self-hosted.
Two-pass pull guarantees no worker idling: if every queued job belongs to
a capped workspace, the second pass uses the unmodified pull queries.
Cap re-asserts on the next refresh.
Fixes WIN-1982
* fix(queue): address CI review findings on workspace fairness
Six fixes from the four-reviewer cross-check on #9303:
1. **Aggregation evaluation (Codex P1).** The previous `INSERT ... ON CONFLICT
DO UPDATE WHERE updated_at < ...` had the heavy `v2_job_queue ∪
v2_job_completed` aggregation inlined into `VALUES`, which Postgres
evaluates for every contender to build the proposed row — losing the
"one heavy aggregation per cycle cluster-wide" property the design
advertises. Split into three small statements: (a) cheap claim with
constant `VALUES`, (b) winner-only `UPDATE ... SET value = jsonb_build_object('overloaded', <agg>)`
(Postgres only evaluates `SET` per row matching `WHERE`, so losers never
compute the aggregation), (c) read for everyone. Heavy query now truly
runs ~0.2-0.5 qps cluster-wide regardless of fleet size.
2. **Numeric setting wraparound (cubic P1).** `u64 as u32` and downstream
`u32 as i32` could silently flip sign and feed `make_interval(secs => -N)`,
making `now() - interval` a future timestamp and disabling the
completed-jobs half of the activity signal. Clamp `duration_secs` to
[1, 86400] and `min_total_jobs` to [0, u32::MAX] before storing.
3. **`/instance_config` bypass (cubic/Claude/Codex P2).** Bulk config endpoint
sidestepped `set_global_setting_internal`'s gate; a self-hosted superadmin
could persist `workspace_fairness_*` rows via the bulk path. Mirror the
per-key check in `set_instance_config` upsert flow.
4. **DB error coerced to false (Claude P2).** `load_workspace_fairness_enabled`
collapsed `Err(_)` to `false` and unconditionally swapped the atomic — a
transient DB blip during notify-event propagation toggled the feature off
cluster-wide (and triggered a `store_pull_query` rebuild precisely when load
is highest). Now propagates the error so the atomic stays at its prior value.
5. **Refresh failure cooldown (Claude P2).** Storing `0` removed the rate
limit entirely; every subsequent pull spawned a new refresh task. Leave
`LAST_REFRESH_MICROS` at `now_us` (already written by the CAS) so the
natural interval acts as the cooldown.
6. **Visibility + duplication (Pi P2).** Mark `make_pull_query_fairness` as
`pub(crate)`. Move the duplicated `BASE_URL host == app.windmill.dev`
parser into `windmill-common::worker::is_cloud_production_host` and share
it between the API setter and the runtime path.
Verified locally:
- `POST /api/settings/global/workspace_fairness_enabled` → 400 (per-key gate)
- `PUT /api/settings/instance_config` with fairness key → 400 (bulk gate)
- `cargo check --workspace --features=private,enterprise,quickjs` — clean
Refs WIN-1982.
* fix(queue): second round of CI review nits on workspace fairness
Three issues raised by the Codex/Claude re-review of commit 0b38ff2:
1. Non-cloud deletes were rejected (Codex P2). The cloud gate ran before
the Null / empty-string deletion branches in both `set_global_setting_internal`
and the bulk `set_instance_config`. A self-hosted instance that inherited
stale `workspace_fairness_*` rows from a cloned cloud DB couldn't clear
them through the API — the rows stayed in `global_settings` and continued
to show up in the YAML export. Now the gate only blocks upserts; Null /
empty-string deletes pass through on any host.
2. Deleted numeric knobs kept stale runtime values (Codex P2). When a
cloud admin cleared `workspace_fairness_max_percent`, `..._duration_secs`,
or `..._min_total_jobs`, the notify-event fired but the numeric loaders
ignored `Ok(None)` and left the previous in-memory value pinned until
process restart. Loaders now distinguish three outcomes:
- `Err(_)`: transient — leave atomic alone (preserves the
previous-round fix).
- `Ok(None)` / `Ok(Some(invalid))`: reset to the documented default.
- `Ok(Some(valid))`: clamp and store.
Defaults are extracted to `WORKSPACE_FAIRNESS_*_DEFAULT` constants kept
in sync with the `AtomicU32::new(...)` initialisers in
`windmill-common/src/worker.rs`.
3. `fairness_active` was `pub` with no cross-crate caller (Claude nit).
Tightened to module-private.
Verified locally on this non-cloud instance:
POST .../workspace_fairness_enabled body=null → 200 (delete passes)
POST .../workspace_fairness_enabled body=true → 400 (set blocked)
PUT .../instance_config {} → 200 (no-op passes)
PUT .../instance_config with fairness key → 400 (bulk set blocked)
Skipped the partial index on `v2_job_queue WHERE running = true` that
Claude flagged as a residual nit — queue stays under 50k rows per the
operator's measurement, so the seq-scan cost (~10 ms × 0.5 qps =
~0.5% of a DB core) is well below the noise floor and the index isn't
worth the maintenance cost on job transitions.
Refs WIN-1982.
* feat(github-app): hide cloud-only UI on self-managed + admin assignment UI
Two related UX fixes for the GitHub App self-managed (GHES) integration:
1. On self-managed instances, the per-installation Export button and the
"Import installation from other instance" section in the workspace UI both
hide. Both round-trip a JWT carrying only {installation_id, account_id} with
no github_base_url, so they would produce broken cloud-style installs on a
self-managed instance. The previous Export attempt also failed with
"No JWT token received from server" because self-managed installs store an
empty JWT by design.
2. New "Workspace assignments" panel in instance settings (GhesAppSettings.svelte)
that auto-discovers installations of the configured GHES App and lets the
super-admin assign them to specific workspaces. Workspace users without
GitHub permissions no longer need to install the App themselves — the admin
provisions the link from instance settings. Admin-provisioned installs show a
"Provisioned by admin" badge in the workspace UI and can only be removed by
the super-admin from instance settings.
Backend support is in the EE companion PR
windmill-labs/windmill-ee-private#588.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to da5189cf69a453de3855057f41be0d84e5910707
This commit updates the EE repository reference after PR #588 was merged in windmill-ee-private.
Previous ee-repo-ref: d959b83ce413ad531e9cc28e0f8199cdecb73a31
New ee-repo-ref: da5189cf69a453de3855057f41be0d84e5910707
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* [ee] fix(secret-backend): pass DB to Vault migrations + surface failure details
Companion to windmill-ee-private fix for WIN-1977. The HashiCorp Vault
migration always failed under JWT/OIDC auth because the migration
constructed VaultBackend without a DB, so every secret hit "Database
connection required for JWT authentication". Creating new secrets worked
because the runtime path passes the DB.
Frontend: when failed_count > 0, the toast and console now show the
per-secret failures (path + error, capped at 5 with "...and N more")
instead of just aggregate counts.
Fixes WIN-1977
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 14315067c083d3361512de621b12e41dbe3b017d
This commit updates the EE repository reference after PR #587 was merged in windmill-ee-private.
Previous ee-repo-ref: 390ed6c851b1915f0b492897c663f8058477680f
New ee-repo-ref: 14315067c083d3361512de621b12e41dbe3b017d
Automated by sync-ee-ref workflow.
* fix(secret-backend): escape failure fields and use <br> in migration toast
Address CI review on PR #9292:
- P1 (cubic/codex): backend-supplied workspace_id/path/error are now
HTML-escaped before being interpolated into the migration toast,
which renders through {@html processMessage(...)} in Toast.svelte.
This prevents stored XSS via secret paths or backend errors that
contain markup. '/' is intentionally left intact so the toast's
path-highlight regex still tags workspace paths.
- P2 (pi): swap '\n' for '<br>' so multi-line failure lists actually
break in the toast instead of collapsing to a single run-on line.
- Extend the same per-secret failure surfacing (toast + console.error)
to the Azure Key Vault and AWS Secrets Manager migration handlers
via a shared reportMigrationFailures() helper so all six migration
paths report identically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(auth): reject unscoped tokens with cross-workspace forged owners (WIN-1978)
An unscoped token (workspace_id IS NULL) whose `owner` field references a
user, group, or unprefixed value that is not present in the target
workspace must not authenticate. The previous fallback in the
`u/<username>` branch granted `(is_admin=false, is_operator=true)` when
no `usr` row matched in the target workspace, letting a token holder
who could mutate the `token` table cross workspace boundaries with
operator privileges.
The `g/<groupname>` branch likewise silently accepted any group name as a
"group user", and the no-prefix branch granted operator state from
arbitrary owner strings. Both are now rejected unless the owner matches
a real user/group membership in the target workspace.
Adds an integration regression covering all three forged-owner shapes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop integration regression for auth fallback
The test added in the previous commit relies on a sqlx::query! that
requires offline-cache regeneration; removing per code-review preference
to keep this PR scoped to the auth-layer fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: use fork-scoped authed for fork visibility in compare_workspaces
* test: add EE end-to-end repro for fork rename visibility
* chore: restore concurrency_locks sqlx cache lost in cleanup
* test: add regression for stale-superadmin-token fork visibility bug
* chore: update sqlx cache for new test queries
Diego Imbert
Ruben Fiszel
hugocasa
centdix
windmill-internal-app[bot]
Diego Imbert
Alexander Petric