slug/project_slug are caller-controlled and were interpolated straight into
the Hub request path; a crafted value (e.g. ../../admin) could reach an
unintended Hub endpoint after URL normalization. Validate against the
frontend charset (lowercase alphanumerics + hyphens, 3-50 chars) in the four
handlers that put the slug in the path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The workspace-switch effect only reset load-derived state, so phase,
draftItems, recordings, hub/bundle metadata, hubVersion, deploymentStatus,
effectiveSlug and hubItemIds survived a switch — a draft built in one
workspace could publish its items/slug under the next workspace's auth.
Reset the full publish session on switch. Also drop explanatory comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hub publish endpoints ignored the {workspace} path and addressed the Hub
project purely by client-supplied project_slug, forwarding with an
instance-wide HUB_DEV_TOKEN. Any workspace admin could mutate or export
another workspace's Hub project by passing its slug.
Stamp the server-trusted workspace from the path onto every forwarded
request as source_id (body for mutations, query param for export) so the
Hub can enforce that the targeted project belongs to the calling
workspace. Requires the matching Hub-side source_id ownership check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an optional path field to the publish bodies and a GET
/projects/{slug}/export route that proxies the hub export (admin-only,
authenticated with HUB_DEV_TOKEN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add POST /w/{workspace}/hub/raw_apps/{id}/embed forwarding to the Hub so a
shared (public) raw app's external_embed_url can be set/cleared. null is
forwarded (not skipped) so unpublish clears the embed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Matches the windmillhub rename: every body now carries `project_slug`
instead of `workspace_slug`, the draft creation forwards to `/projects`,
and the resource_types/resources/triggers proxies hit
`/projects/{slug}/...`. `HubWorkspaceBody` becomes `HubProjectBody`. The
instance-side `Path(workspace)` extractor and the `workspace` URL
parameter stay because that's still the source tenant's identifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the existing /scripts, /flows, /apps thin proxies. Forwards
{ triggers, workspace_slug } to Hub's POST /workspaces/[slug]/triggers
bulk-replace endpoint, with the same require_admin + HUB_DEV_TOKEN
guardrails. Lets the frontend push trigger stubs in a single round-trip
after the items they reference have landed on the Hub.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add ApiAuthed + require_admin to all hub publish handlers; previously any
workspace-authenticated session could trigger Hub-side writes attributed
to the URL workspace via the shared HUB_DEV_TOKEN.
- Track per-kind trigger fetch failures (triggerLoadErrors) so an EE-gated
or transiently failing trigger service no longer silently maps to "0
triggers"; UI surfaces an amber badge listing the missing kinds and a
toast warns the operator before publish.
- Add workspaceLoadSeq cancellation so the parallel loadWorkspace +
loadTriggers stop bleeding stale data when the workspace switches mid
load.
- Drop the silent effectiveSlug fallback to sanitizeSlug(hubName) when
the Hub response can't be parsed; abort the publish instead so items
don't land under a slug the Hub never locked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Forward workspace resource stubs (path + type) to the hub's
/workspaces/{slug}/resources endpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New workspaced router /api/w/:ws/hub/* forwarding to the Hub:
- POST /publish_draft → POST {HUB}/workspaces (slug/name/summary/readme)
- POST /scripts → POST {HUB}/scripts/add (workspace_slug + content)
- POST /flows | /apps | /raw_apps → corresponding hub endpoints
- POST /scripts/:ask_id/recording, /flows/:flow_id/recording → recording uploads
Auth uses HUB_DEV_TOKEN env var (dev shortcut). All bodies are
serde-typed; the helper forward_to_hub centralises the HTTP call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(queue): bump EE ref to include worker_ping fairness signal
The current ee-repo-ref.txt pointed to 31cda7c (an unrelated merge
commit on the asset-graph-view-ee branch) instead of ddc9e80, which
contains the workspace-fairness fix that switches the active-share
signal from v2_job_queue.running=true to worker_ping. As a result
cloud was still computing overload off the legacy signal, so a
workspace with many in-flight/suspended flows (lancom01-prod, with
799 suspended flows × 3 v2_job_queue bookkeeping rows each = 2397
running-true rows) was flagged as 95% of cluster activity despite
consuming zero worker slots.
Bumping to ddc9e80 picks up the worker_ping-based signal, which
naturally excludes (a) suspended jobs (no worker pinging them),
(b) zombie running-rows from dead workers, and (c) flow/flownode
orchestration rows that never run on a worker in the first place.
* test(queue): seed v2_job rows + realistic durations for fairness helpers
The new duration-weighted fairness algorithm joins v2_job_queue and
v2_job_completed to v2_job for the `kind` filter (excluding flow
bookkeeping) and reads `duration_ms` for the completed contribution.
Update the test helpers to mirror that schema:
* `insert_completed` now inserts a matching v2_job row (kind=script)
and writes `duration_ms = 1000` with a 1-second [started_at,
completed_at] interval, so each completed row contributes ~1
worker-second when fully inside the refresh window.
* `insert_queued` likewise pre-inserts v2_job, sets `started_at`
to NOW() - 1s when running=true (so running rows contribute ~1
worker-second by the time the refresh runs), and seeds
v2_job_runtime.ping so the running side accrues real-time worker
seconds (the algorithm bounds end-of-interval by ping).
The zombie/suspended insert helpers are intentionally left without
v2_job rows — the new algorithm's INNER JOIN excludes them, so they
still correctly contribute zero worker-seconds.
* chore(queue): bump EE ref to duration-weighted fairness algorithm
Companion to windmill-ee-private#<TBD>: switch the EE workspace
fairness aggregation from a count-based UNION (worker_ping snapshot
+ v2_job_completed count) to a worker-seconds aggregation sourced
directly from v2_job_queue and v2_job_completed, with kind/suspend
filters mirroring handle_zombie_jobs and per-row defenses against
zombie inflation on both halves.
* chore(queue): bump EE ref for fairness perf fix (inline window_start)
* chore(queue): bump EE ref for fairness perf rewrite (driver-side flip)
* update ee ref
`fairness_ignores_zombie_running_rows` and
`fairness_ignores_concurrency_suspended_rows` panic intermittently in CI
(both Linux and Windows runs). Mark them `#[ignore]` until the
underlying flakiness is resolved.
* feat(websocket-trigger): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY (WIN-1988)
`tokio_tungstenite::connect_async` opens a raw TCP socket and ignores
the standard outbound-proxy env vars, so deployments behind a forward
HTTP proxy can't reach the WebSocket endpoint and Test Connection
times out after 30s.
Add a small `proxy` module that resolves the right proxy URL for the
target host (HTTPS_PROXY for wss://, HTTP_PROXY for ws://, NO_PROXY
exclusions, ALL_PROXY fallback, lowercase variants), opens an HTTP
CONNECT tunnel when one applies, and hands the resulting TcpStream to
`client_async_tls_with_config` for the TLS + WS handshake. Direct
connect remains the default when no proxy env is set.
Unit tests cover NO_PROXY matching, proxy URL parsing (including IPv6
literals and basic-auth userinfo), and the CONNECT handshake itself
against an in-process fake proxy (success, basic-auth header, 407
rejection).
Fixes WIN-1988
* refactor(websocket-trigger): reduce blast radius and reuse existing logic
Follow-up to the proxy support change. Three things:
1. Skip the new code path entirely when no proxy is configured.
`connect_async_with_proxy` now checks the env-var snapshots up front
and delegates straight to `tokio_tungstenite::connect_async` if
neither `HTTP_PROXY` nor `HTTPS_PROXY` is set. Same fall-through
applies when proxy env is set but `NO_PROXY` excludes the host or
the proxy URL doesn't parse. Non-proxied deployments now exercise
exactly the previous code path.
2. Move the `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` env-var snapshots
from `windmill-worker::worker` into `windmill-common`. The worker's
`PROXY_ENVS` static now reads from there, and the websocket trigger
reads from the same source — one place reads the env, one source
of truth for both call sites.
3. Replace the hand-rolled proxy-URL parser with `url::Url::parse`
(already a workspace dep, used across the codebase). Half the LoC
and handles edge cases (userinfo percent-encoding, IPv6 literals,
path/query stripping) via the well-tested crate instead of by hand.
All 13 proxy unit tests still pass. `cargo check` is clean.
* fix(websocket-trigger): unbreak EE build + trim proxy tests
- Re-export `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` from
`windmill-worker::worker` (via `pub use windmill_common::...`) so the
EE `otel_tracing_proxy_ee` module's `use crate::{HTTPS_PROXY, ...}`
resolves like it did before. Fixes the `check_ee_full` / `cargo_test`
CI failures from the previous commit.
- Trim the proxy tests to one un-ignored canary
(`http_connect_tunnel_sends_well_formed_request_and_unwraps_stream`)
that exercises the actual on-wire CONNECT handshake plus byte-perfect
tunnel passthrough. The NO_PROXY-matching, URL-parsing, and edge-case
tunnel tests are kept under `#[ignore]` for manual debugging
(`cargo test -- --ignored`) since they're either delegated to
`url::Url::parse` or trivial string matching — low ROI on every CI run.
* fix(jobs): enforce anonymous-only guard on `only_result` job updates
The `jobs_u/getupdate/{id}` and `jobs_u/getupdate_sse/{id}` endpoints
accept `only_result=true`. In that branch, `get_job_update_data` queried
the result solely by (workspace_id, job_id) and skipped the
`created_by == "anonymous"` check that the non-only_result path and
adjacent unauthenticated endpoints apply. An unauthenticated requester
who learned a private job UUID could therefore retrieve that job's
output.
Hoist the guard to the top of `get_job_update_data` so both branches are
covered.
Fixes WIN-1980
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: fold `created_by` check into existing only_result queries
Avoids the extra `SELECT created_by` round-trip per call by joining
`v2_job` once in the two queries that handled the unauth path and
checking inline. Behavior is identical to the prior commit; the SSE
polling loop now does one query per poll instead of two for
unauthenticated callers.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: cache anonymous_verified across SSE polls
Replace the LEFT JOIN approach with an upfront `SELECT created_by`
guarded by a new `&mut bool anonymous_verified` parameter that mirrors
`early_return_suppressed`. The SSE polling loop now performs the auth
check exactly once per stream rather than per poll, and the data SQL
reverts to its original form so authenticated callers pay no extra
cost. `created_by` cannot change after job creation, so caching the
verification across polls is safe.
Cost matrix:
- Authed (any path): 0 extra queries
- Unauthed one-shot: 1 extra query (unavoidable)
- Unauthed SSE: 1 extra query at stream start, 0 per poll
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* refactor: scope anonymous check to only_result branch
The non-only_result branch already enforces the `created_by` check via
its main query, so a top-level hoisted check duplicated work for
unauthenticated default-path callers. Move the check inside the
`if only_result.unwrap_or(false)` block — exactly where the bypass
lives — and leave the non-only_result path untouched.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ee] feat(service-accounts): allow choosing role at creation time
Previously, service accounts were hardcoded to operator and could not be
used as the CLI sync user since they had no write access. They also only
counted as 0.5 seat each.
This change:
- Extends `NewServiceAccount` to accept optional `is_admin` / `operator`
(defaults to `operator=true` for backward compatibility).
- Exposes a role picker in `AddUser.svelte` when creating a service
account (Operator / Developer / Admin).
- Lets admins update a service account's role from the user list (it
used to be locked to "Operator" with a tooltip).
- Updates the OpenAPI spec + regenerates the frontend client.
A developer/admin service account counts as 1 seat under the existing
seat-cap logic (operators stay at 0.5).
Companion PR on windmill-ee-private updates the `INSERT INTO usr` to
honour the chosen role.
Fixes WIN-1985
* [ee] feat(service-accounts): wm_deployers opt-in for Dev role
When creating a service account with role=Developer, surface a toggle
"Add to wm_deployers" (recommended). Members of wm_deployers can deploy
on behalf of other users — the typical setup when the service account is
used as the CLI sync / CI deploy identity.
- `NewServiceAccount` gains an optional `add_to_deployers` flag.
- Frontend defaults the toggle to on but only shows it under Developer
(admins have it implicitly; operators can't deploy).
- Tooltip links to docs.windmill.dev "Run on behalf of".
Companion EE PR updates the handler to INSERT into usr_to_group for
wm_deployers when the flag is set.
Refs WIN-1985
* chore: update ee-repo-ref to 974ed42067d9f63acb42332b671b8c01ffd4b625
This commit updates the EE repository reference after PR #589 was merged in windmill-ee-private.
Previous ee-repo-ref: f7dbc3cc2ba21c396f4828881e3b9d9ab6f50c69
New ee-repo-ref: 974ed42067d9f63acb42332b671b8c01ffd4b625
Automated by sync-ee-ref workflow.
* [ee] fix(service-accounts): unhardcode role in superadmin user list
Two review issues from the merged #9307 / #589:
1. P1 — The global Users tab in #superadmin-settings still pinned every
service account to "Operator". Now it shows the actual role
(Admin / Operator / Developer), derived from the SA's usr row.
- `list_users_as_super_admin`: replaced `true as operator_only` with
the real `operator` value, and added `is_workspace_admin` from the
row (NULL for password users since their admin status is
per-workspace).
- `global_whoami`: when the email belongs to a service account, look
up its real `operator` / `is_admin` instead of pinning to operator.
- `SuperadminSettingsInner.svelte`: drop the hardcoded "Operator"
badge; render Admin / Operator / Developer using the new fields,
matching the workspace-level view.
2. P2 — Regenerate the bundled `openapi-deref.{yaml,json}` so the
`createServiceAccount` body (now exposing `is_admin`, `operator`,
`add_to_deployers`) and the new `GlobalUserInfo.is_workspace_admin`
field show up at runtime in `/api/openapi.{yaml,json}`.
Bumps `ee-repo-ref.txt` to the EE follow-up that adds the offline
seat-cap check on `create_service_account`.
Refs WIN-1985
* chore: update ee-repo-ref to b7a6068c1f3dc845e012959268b2426f0de4d697
This commit updates the EE repository reference after PR #590 was merged in windmill-ee-private.
Previous ee-repo-ref: 0b1307c21d1bfd6fb43a03c2ba39d2a8bf8e6470
New ee-repo-ref: b7a6068c1f3dc845e012959268b2426f0de4d697
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(settings): skip workspaced-route duplicate checks on cloud
The pre-write validation hooks for `app_workspaced_route` and
`http_route_workspaced_route` query the DB for cross-workspace duplicates
and fail the save when any are found. On cloud both `custom_path_exists`
(apps) and `route_path_key_exists` (HTTP triggers) already scope lookups
by `workspace_id` regardless of these settings, so duplicates across
workspaces are expected and the validation has no runtime meaning. The
result was that any cloud super-admin attempting to save instance
settings with these toggles set to false received
`Duplicate HTTP route paths detected` even though the setting has no
effect on cloud routing.
Fixes WIN-1983
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(error): render JsonErr as readable text and return 400
`Error::JsonErr` previously rendered through `#[error("Error: {0:#?}")]`,
leaking Rust's `Debug` output (`Object { "error": String(...), "details":
Array [...] }`) into the HTTP response body, and was bucketed into the
catch-all 500 branch in `IntoResponse`. The result was a 500 status with
a wall of Rust debug syntax in the toast — confusing and user-hostile.
- Bucket `JsonErr` into 400 (Bad Request): every current call site
(workspaced-route duplicate checks, OAuth client errors, etc.) is a
client/validation issue, not an internal server fault.
- Add `format_json_err_message` which surfaces the `error` field as the
headline, summarises `details` (with a `- key=value` per entry), and
pretty-prints the rest as JSON for unknown shapes. The frontend toast
now reads e.g.
Duplicate HTTP route paths detected
- route_path=a, workspace_id=admins, http_method=post
- route_path=a, workspace_id=starter, http_method=post
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(toast): preserve newlines and escape HTML in multi-line errors
The toast renders via `{@html processMessage(message)}`, so server-side
error bodies that span multiple lines (e.g. the duplicate-route response
from the settings endpoint) collapsed into a single line because HTML
treats consecutive whitespace (including `\n`) as a single space.
When the message contains a newline, escape HTML first (defends against
injected markup in server error bodies) and convert `\n` to `<br />` so
multi-line errors stay readable in the toast.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fixup: address CI review feedback
- toast.ts: escape HTML unconditionally. The previous gate on `\n` left
single-line server error bodies unsafe under {@html}, which cubic
flagged as P0. The path regex below only inserts a `<span>` around a
`u/...` or `f/...` capture that can't contain HTML metacharacters, so
escaping the whole input is the simpler and correct fix.
- error.rs: add unit tests pinning the rendered shape of
`format_json_err_message` (error+details, error-only, truncation cap,
non-object fallback to pretty JSON).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(queue): audit-log workspace-fairness cap transitions
When the cloud per-workspace fairness mechanism adds a workspace to the
capped set or releases one, write `workspace_fairness.capped` /
`workspace_fairness.uncapped` audit-log entries to the affected workspace.
The cluster admin can review the full timeline from the `admins` workspace
audit view with `all_workspaces=true`; per-workspace owners see their own
events in their normal audit list.
Only the per-cycle refresh winner emits entries (matching where the heavy
aggregation runs), so a fleet of N workers does not produce N duplicates
per transition. The diff is computed against the value already in
`background_task_state` rather than the winner's in-memory cache, so a
freshly-restarted process winning the claim does not spuriously emit
"newly capped" entries for workspaces that were already capped before it
started.
Audit writes are best-effort: failures are logged via tracing and do not
abort the refresh cycle.
Fixes WIN-1984
* feat(queue): scope fairness audit to admins workspace + queue-metrics pane
- Write `workspace_fairness.capped` / `workspace_fairness.uncapped` to the
`admins` workspace (was: per-affected-workspace) with the affected
workspace_id moved to the `resource` field. Cluster admins now get the
full timeline in one place without `all_workspaces=true`.
- Add `GET /workers/workspace_fairness_events` returning the last 100
events. Cloud-gated (returns `[]` on non-cloud) and devops-only.
- Add a `WorkspaceFairnessEvents` Section to the Queue Metrics drawer,
rendered only when `isCloudHosted()` is true. Shows time / event
badge / workspace / parameters with a refresh button.
Fixes WIN-1984
A token scoped to a single resource (e.g. `resources:read:u/alice/foo`)
could call `GET /api/w/{w}/resources/list_search` and receive `path` and
`value` for unrelated resources in the workspace. Route-level scope
checks only validate `domain:action`; per-resource handlers do a
`check_scopes` against the path, but the listing endpoints did not —
leaking integration credentials, API keys, and other secrets stored as
resource values to narrowly-scoped tokens.
Add `build_scope_path_predicate` to `windmill-api-auth` (mirrors
`check_scopes` semantics but parses the token's scopes once, suitable
for filtering many rows). Apply it to `list_search_resources`,
`list_resources`, `list_names` (resources) and `list_variables`
(non-secret value leak), so a scope-restricted token only ever sees the
paths it is authorized to read. Unscoped tokens and tokens whose only
scopes are `if_jobs:filter_tags:*` are unaffected.
Includes regression tests covering: unscoped, tag-filter-only,
single-resource, wildcard, wrong-domain, and write-implies-read.
Fixes WIN-1981
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(queue): cloud-only per-workspace fairness cap on the shared worker pool
On `app.windmill.dev` the cluster runs a single default worker group, so a
single workspace flooding the queue can degrade quality of service for
everyone else. This adds an opt-in mechanism that caps any single workspace
at a configurable share of the shared worker pool when it has been
dominating cluster activity for more than a configurable window.
Detection signal counts both currently-running jobs and jobs completed in
the rolling window, so it catches workspaces hogging slots with long jobs
**and** workspaces spamming many tiny jobs (where no individual job's
started_at is old, but throughput share dominates).
Refresh is coordinated cluster-wide via a single UPDATE on
`background_task_state`: the `WHERE updated_at < now() - interval` predicate
combined with row-level locking means only one process per refresh cycle
actually runs the aggregation, regardless of fleet size. Every other
process gets the freshly written value in the same round trip via
`UNION ALL ... LIMIT 1`. Heavy aggregation rate stays at ~0.2-0.5 qps for
the whole cluster.
Pull queries are split: the existing query string and its bind shape stay
bit-identical to today, so the planner keeps using the same indexes when
fairness is off or no workspace is currently capped. A separate
`WORKER_PULL_QUERIES_FAIRNESS` adds `AND workspace_id <> ALL($2::text[])`
and is only materialized while the feature is enabled.
Hard-gated to `CLOUD_HOSTED=true` + BASE_URL host == app.windmill.dev at
three layers: frontend `cloudonly: true`, API setter rejection in
`set_global_setting_internal`, runtime check in `fairness_active`. Settings
are exposed under Jobs in the instance-settings UI; defaults are off so
the change is a no-op for self-hosted.
Two-pass pull guarantees no worker idling: if every queued job belongs to
a capped workspace, the second pass uses the unmodified pull queries.
Cap re-asserts on the next refresh.
Fixes WIN-1982
* fix(queue): address CI review findings on workspace fairness
Six fixes from the four-reviewer cross-check on #9303:
1. **Aggregation evaluation (Codex P1).** The previous `INSERT ... ON CONFLICT
DO UPDATE WHERE updated_at < ...` had the heavy `v2_job_queue ∪
v2_job_completed` aggregation inlined into `VALUES`, which Postgres
evaluates for every contender to build the proposed row — losing the
"one heavy aggregation per cycle cluster-wide" property the design
advertises. Split into three small statements: (a) cheap claim with
constant `VALUES`, (b) winner-only `UPDATE ... SET value = jsonb_build_object('overloaded', <agg>)`
(Postgres only evaluates `SET` per row matching `WHERE`, so losers never
compute the aggregation), (c) read for everyone. Heavy query now truly
runs ~0.2-0.5 qps cluster-wide regardless of fleet size.
2. **Numeric setting wraparound (cubic P1).** `u64 as u32` and downstream
`u32 as i32` could silently flip sign and feed `make_interval(secs => -N)`,
making `now() - interval` a future timestamp and disabling the
completed-jobs half of the activity signal. Clamp `duration_secs` to
[1, 86400] and `min_total_jobs` to [0, u32::MAX] before storing.
3. **`/instance_config` bypass (cubic/Claude/Codex P2).** Bulk config endpoint
sidestepped `set_global_setting_internal`'s gate; a self-hosted superadmin
could persist `workspace_fairness_*` rows via the bulk path. Mirror the
per-key check in `set_instance_config` upsert flow.
4. **DB error coerced to false (Claude P2).** `load_workspace_fairness_enabled`
collapsed `Err(_)` to `false` and unconditionally swapped the atomic — a
transient DB blip during notify-event propagation toggled the feature off
cluster-wide (and triggered a `store_pull_query` rebuild precisely when load
is highest). Now propagates the error so the atomic stays at its prior value.
5. **Refresh failure cooldown (Claude P2).** Storing `0` removed the rate
limit entirely; every subsequent pull spawned a new refresh task. Leave
`LAST_REFRESH_MICROS` at `now_us` (already written by the CAS) so the
natural interval acts as the cooldown.
6. **Visibility + duplication (Pi P2).** Mark `make_pull_query_fairness` as
`pub(crate)`. Move the duplicated `BASE_URL host == app.windmill.dev`
parser into `windmill-common::worker::is_cloud_production_host` and share
it between the API setter and the runtime path.
Verified locally:
- `POST /api/settings/global/workspace_fairness_enabled` → 400 (per-key gate)
- `PUT /api/settings/instance_config` with fairness key → 400 (bulk gate)
- `cargo check --workspace --features=private,enterprise,quickjs` — clean
Refs WIN-1982.
* fix(queue): second round of CI review nits on workspace fairness
Three issues raised by the Codex/Claude re-review of commit 0b38ff2:
1. Non-cloud deletes were rejected (Codex P2). The cloud gate ran before
the Null / empty-string deletion branches in both `set_global_setting_internal`
and the bulk `set_instance_config`. A self-hosted instance that inherited
stale `workspace_fairness_*` rows from a cloned cloud DB couldn't clear
them through the API — the rows stayed in `global_settings` and continued
to show up in the YAML export. Now the gate only blocks upserts; Null /
empty-string deletes pass through on any host.
2. Deleted numeric knobs kept stale runtime values (Codex P2). When a
cloud admin cleared `workspace_fairness_max_percent`, `..._duration_secs`,
or `..._min_total_jobs`, the notify-event fired but the numeric loaders
ignored `Ok(None)` and left the previous in-memory value pinned until
process restart. Loaders now distinguish three outcomes:
- `Err(_)`: transient — leave atomic alone (preserves the
previous-round fix).
- `Ok(None)` / `Ok(Some(invalid))`: reset to the documented default.
- `Ok(Some(valid))`: clamp and store.
Defaults are extracted to `WORKSPACE_FAIRNESS_*_DEFAULT` constants kept
in sync with the `AtomicU32::new(...)` initialisers in
`windmill-common/src/worker.rs`.
3. `fairness_active` was `pub` with no cross-crate caller (Claude nit).
Tightened to module-private.
Verified locally on this non-cloud instance:
POST .../workspace_fairness_enabled body=null → 200 (delete passes)
POST .../workspace_fairness_enabled body=true → 400 (set blocked)
PUT .../instance_config {} → 200 (no-op passes)
PUT .../instance_config with fairness key → 400 (bulk set blocked)
Skipped the partial index on `v2_job_queue WHERE running = true` that
Claude flagged as a residual nit — queue stays under 50k rows per the
operator's measurement, so the seq-scan cost (~10 ms × 0.5 qps =
~0.5% of a DB core) is well below the noise floor and the index isn't
worth the maintenance cost on job transitions.
Refs WIN-1982.
* feat(github-app): hide cloud-only UI on self-managed + admin assignment UI
Two related UX fixes for the GitHub App self-managed (GHES) integration:
1. On self-managed instances, the per-installation Export button and the
"Import installation from other instance" section in the workspace UI both
hide. Both round-trip a JWT carrying only {installation_id, account_id} with
no github_base_url, so they would produce broken cloud-style installs on a
self-managed instance. The previous Export attempt also failed with
"No JWT token received from server" because self-managed installs store an
empty JWT by design.
2. New "Workspace assignments" panel in instance settings (GhesAppSettings.svelte)
that auto-discovers installations of the configured GHES App and lets the
super-admin assign them to specific workspaces. Workspace users without
GitHub permissions no longer need to install the App themselves — the admin
provisions the link from instance settings. Admin-provisioned installs show a
"Provisioned by admin" badge in the workspace UI and can only be removed by
the super-admin from instance settings.
Backend support is in the EE companion PR
windmill-labs/windmill-ee-private#588.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to da5189cf69a453de3855057f41be0d84e5910707
This commit updates the EE repository reference after PR #588 was merged in windmill-ee-private.
Previous ee-repo-ref: d959b83ce413ad531e9cc28e0f8199cdecb73a31
New ee-repo-ref: da5189cf69a453de3855057f41be0d84e5910707
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* [ee] fix(secret-backend): pass DB to Vault migrations + surface failure details
Companion to windmill-ee-private fix for WIN-1977. The HashiCorp Vault
migration always failed under JWT/OIDC auth because the migration
constructed VaultBackend without a DB, so every secret hit "Database
connection required for JWT authentication". Creating new secrets worked
because the runtime path passes the DB.
Frontend: when failed_count > 0, the toast and console now show the
per-secret failures (path + error, capped at 5 with "...and N more")
instead of just aggregate counts.
Fixes WIN-1977
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 14315067c083d3361512de621b12e41dbe3b017d
This commit updates the EE repository reference after PR #587 was merged in windmill-ee-private.
Previous ee-repo-ref: 390ed6c851b1915f0b492897c663f8058477680f
New ee-repo-ref: 14315067c083d3361512de621b12e41dbe3b017d
Automated by sync-ee-ref workflow.
* fix(secret-backend): escape failure fields and use <br> in migration toast
Address CI review on PR #9292:
- P1 (cubic/codex): backend-supplied workspace_id/path/error are now
HTML-escaped before being interpolated into the migration toast,
which renders through {@html processMessage(...)} in Toast.svelte.
This prevents stored XSS via secret paths or backend errors that
contain markup. '/' is intentionally left intact so the toast's
path-highlight regex still tags workspace paths.
- P2 (pi): swap '\n' for '<br>' so multi-line failure lists actually
break in the toast instead of collapsing to a single run-on line.
- Extend the same per-secret failure surfacing (toast + console.error)
to the Azure Key Vault and AWS Secrets Manager migration handlers
via a shared reportMigrationFailures() helper so all six migration
paths report identically.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix(auth): reject unscoped tokens with cross-workspace forged owners (WIN-1978)
An unscoped token (workspace_id IS NULL) whose `owner` field references a
user, group, or unprefixed value that is not present in the target
workspace must not authenticate. The previous fallback in the
`u/<username>` branch granted `(is_admin=false, is_operator=true)` when
no `usr` row matched in the target workspace, letting a token holder
who could mutate the `token` table cross workspace boundaries with
operator privileges.
The `g/<groupname>` branch likewise silently accepted any group name as a
"group user", and the no-prefix branch granted operator state from
arbitrary owner strings. Both are now rejected unless the owner matches
a real user/group membership in the target workspace.
Adds an integration regression covering all three forged-owner shapes.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: drop integration regression for auth fallback
The test added in the previous commit relies on a sqlx::query! that
requires offline-cache regeneration; removing per code-review preference
to keep this PR scoped to the auth-layer fix.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix: use fork-scoped authed for fork visibility in compare_workspaces
* test: add EE end-to-end repro for fork rename visibility
* chore: restore concurrency_locks sqlx cache lost in cleanup
* test: add regression for stale-superadmin-token fork visibility bug
* chore: update sqlx cache for new test queries
* fix(git-sync): revert LATEST_GIT_SYNC_SCRIPT_PATH to hub/28230 to restore GPG-signed deploys (WIN-1974)
hub/28231 (PR #9230) is the "thin" script that hands the actual `git commit`
to the CLI's hidden `sync git-deploy`. The hub script still does the GPG
setup (import key into a fresh GNUPGHOME, dummy `gpg -bsau` to warm the
agent passphrase cache, then `git config user.signingkey` + `commit.gpgsign`
locally), but the commit no longer runs in the same `git_push` flow — it
runs minutes later inside the CLI after workspace API resolution, zip pull,
file extraction, and lockfile autofill. By the time the spawned `git commit`
asks gpg-agent for the cached passphrase, the cache state is no longer
reliable (or the spawned `gpg` ends up talking to a fresh agent), so signing
fails non-interactively with `gpg failed to sign the data`.
hub/28230 is hub/28217's in-script logic rebuilt with windmill-cli@1.703.3:
the GPG setup and the in-script `sh_run("git commit ...")` happen back-to-back
in `git_push`, so the cache is always fresh. It preserves wm_deploy / fork
branch behavior, the EE deployment-callback `main()` signature is unchanged,
and the only min-version check in EE (`is_script_meets_min_version(28103)`)
is comfortably below 28230 — so this revert is safe.
Forward fix (separate PR): publish a new thin script that, alongside the
existing GPG setup, writes a `gpg.program` wrapper using `--pinentry-mode
loopback --passphrase-file` so signing is independent of the agent's cache
state. Re-bump past 28231 then.
Fixes WIN-1974
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(git-sync): check in source-of-truth for the next hub script (gpg.program wrapper)
This is the script that will be published to hub.windmill.dev once verified
on a customer GPG-signed deploy. It replaces hub/28231's agent-cache
pre-warm (`gpg -bsau` with --passphrase) with a stateless gpg.program
wrapper + chmod-600 passphrase file. Every git-invoked gpg call goes
through the wrapper, which always uses --pinentry-mode loopback (and
--passphrase-file when a passphrase exists). Signing no longer depends on
gpg-agent having a cached passphrase by the time the CLI's `git commit`
runs — which closes WIN-1974.
Not wired in yet: LATEST_GIT_SYNC_SCRIPT_PATH stays on hub/28230 until this
script is uploaded and the new hub id is known. This file is checked in so
the diff is reviewable, future bumps have a source of truth, and a CLI
regression test can `cat` it for fixture parity.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(frontend): skip format/pattern validation for $var/$res/$jsonvar references in ArgInput
A resource field with a `pattern` constraint (e.g. the gpg_key.private_key
field, whose pattern enforces a `-----BEGIN PGP PRIVATE KEY BLOCK-----`
prefix) rejects values like `$var:u/me/gpg-private-key` with an "invalid
format" error in the resource editor — even though `$var:`/`$res:`/`$jsonvar:`
are placeholders the backend resolves at runtime, not the actual string
that needs to match the regex.
Bail out of all format/pattern checks (email, ipv4, ipv6, uuid, custom
pattern) when the value is one of these references. Required/numeric
bounds/array checks still apply since they're shape-level, not regex.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(git-sync): bump LATEST_GIT_SYNC_SCRIPT_PATH to hub/28234 (gpg.program-wrapper fix)
hub/28234 is the forward fix for WIN-1974: replaces hub/28231's agent-cache
pre-warm (which became stale by the time the CLI's `git commit` ran) with
a stateless `gpg.program` wrapper that uses `--pinentry-mode loopback`
(and `--passphrase-file` when a passphrase exists) on every gpg invocation.
Bundled CLI is windmill-cli@1.705.0.
Verified via reproducer at /tmp/git-sync-diff/test-gpg-fix.sh: deliberately
killing gpg-agent between GPG setup and `git commit` reproduces the
customer's `gpg failed to sign the data` error verbatim under the old
flow, and the wrapper signs through it. Holds for passphrase-protected
keys, split-subkey [C]+[S] layouts, and unprotected keys.
Drops the local source-of-truth copy (`hub-scripts/`) — hub is canonical
now that 28234 is published.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore(git-sync): drop verbose comment above LATEST_GIT_SYNC_SCRIPT_PATH
The git history (this PR) carries the why; the constant name + value carry
the what.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
The disk_backed_refuses_preexisting_symlink_at_jail_tmp test calls
std::os::unix::fs::symlink directly, which doesn't exist on Windows
targets. Without a cfg gate, `cargo check --tests` fails on Windows
with E0433. Other symlink call sites in this crate (php_executor,
bun_executor, rust_executor, etc.) already follow this pattern.
Fixes WIN-1972
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* feat(nsjail): optional disk-backed /tmp via instance setting
* test(nsjail): unit-test tmp mount resolver and narrow visibility
* refactor(nsjail): switch tmp backing to select + conditional UI
* ui(nsjail): make tmpfs the visible default in /tmp backing select
* fix(nsjail): refuse preexisting jail_tmp to block symlink escape
* fix(nsjail): allow jail_tmp reuse on sequential nsjail calls
Codex flagged that python/ruby/rust executors invoke nsjail twice per
job_dir (install then run). The previous resolver treated any preexisting
jail_tmp as hostile and silently fell back to tmpfs on the second call,
so disk-backed mode never reached the main script run for those langs.
Use symlink_metadata().is_dir() to distinguish a real directory left by
an earlier call in the same job_dir (safe to reuse) from a symlink or
other entity (still refused, as the codebase-tar escape requires).
Also loosen the frontend visibility predicate: only hide nsjail settings
when job_isolation is explicitly 'none' or 'unshare', so deployments
that enable nsjail via DISABLE_NSJAIL=false with no DB setting can
still see the controls.
* fix(ai): enforce RLS and scope check on user-supplied X-Resource-Path
The AI proxy handler accepts an X-Resource-Path header to override the
configured workspace AI provider. When supplied, the handler loaded the
resource value from the resource table using the root DB pool with no
resources:read scope check, so any authenticated workspace user could
point X-Resource-Path at a restricted AI resource (e.g. one in a folder
they cannot read) and the proxy would use that resource's provider
credentials for the outbound AI request.
For user-supplied resource paths, now require resources:read:{path}
scope and fetch the resource through user_db.begin(&authed) so RLS
enforces the same folder/group boundary as the resource API. The RLS-
scoped $var: resolution stays in place as defense in depth. The
admin-configured workspace/instance ai_config path is unchanged.
Fixes WIN-1971
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* test(ai): regression test for X-Resource-Path RLS enforcement
Cover all four cases:
- non-admin pointing X-Resource-Path at a restricted resource is rejected
- non-admin pointing it at a resource they own still works
- admin can point it at any resource
- workspace-configured proxy flow (no X-Resource-Path) is unchanged
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* [ee] fix(indexer): tell admins when ingress routes search to wrong pod
When the IndexReader is absent on the pod handling a search request but
another pod is actively holding the indexer lock, the EE handler now
returns a tailored error pointing at the ingress/load-balancer
configuration instead of the generic "indexer not running" message.
The indexer status endpoint reads the DB lock so it reports "running"
from any pod, but search endpoints need the in-memory IndexReader that
only exists on the lock holder. In multi-replica deployments this looks
like the indexer is healthy but every search 404s.
Companion: windmill-labs/windmill-ee-private#TBD
Fixes WIN-1968.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817
This commit updates the EE repository reference after PR #586 was merged in windmill-ee-private.
Previous ee-repo-ref: 7dd43d1850813071cc18ba49ba090583e7321f4b
New ee-repo-ref: eb18d7b4c0e37fea3f6e1e2cc44e0fddd74ff817
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Operators are read-only and cannot create native triggers, yet the
google/github/nextcloud integration picker routes had no authorization
gate, letting any workspace member drive the admin-configured
integration's upstream API and enumerate its data (Drive files, repo
names, calendars, events).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: authenticate slack callback payload with per-workspace hmac
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: regression tests for unauthenticated slack callback decryption
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: verify slack submission signature before resume + close workspace oracle
Addresses review: verify private_metadata HMAC before handle_resume_action so a
tampered/unsigned submission is rejected up front, and map get_workspace_key
failure to the generic 401 so the status code is not a workspace-existence oracle.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: domain-separate slack payload hmac from resume-secret hmac
Both MAC families key Hmac<Sha256> on the same per-workspace key; resume secrets
are distributed to approvers in resume URLs, so add a fixed domain tag
(slack_payload_v1) to the slack payload MAC to make the two non-interchangeable
by construction rather than by byte-layout coincidence.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A flow inline step whose id is a Python keyword (e.g. `in`) crashed with a
`SyntaxError`: the wrapper emits `from {pkg} import {step_id} as inner_script`,
and `from x import in as y` is invalid Python.
The codegen already prefixes `_` to path segments that start with a digit
(`1234` → `_1234`); this extends that guard to Python hard keywords (`in` →
`_in`) in `compute_python_module_dir` and on the leaf in `compute_py_codegen`
and `prepare_wrapper`. The relative-imports write path inherits it for free.
Fixes#8893
* fix: prevent token label collision bypassing job read access control
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: regression tests for token label collision job read access
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: bind job-read override fast-path to permissioned_as_email
Replaces the reserved-label / label-* exclusion approach: webhook-/http-/email-
labels are created through the public token API by the trigger panels, so they
cannot be reserved, and blocking label-* regressed legitimate re-reads. Instead
the username_override fast-path now requires the job's permissioned_as_email
(non-forgeable, never derived from the label) to equal the caller's email.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(otel): propagate inbound W3C traceparent to job spans
Capture the inbound traceparent header at the run endpoints
(WebhookArgs::to_args_from_format) into a reserved _wm_traceparent arg key
(gated on OTEL_TRACING_ENABLED), riding the args jsonb like
_ENTRYPOINT_OVERRIDE. At pickup, create_span_with_name attaches a span link
from the job's worker span to the originating distributed trace, so a job
triggered by an instrumented service is connected to the caller's trace
while keeping its UUID-derived trace id (trace-by-job-id unaffected).
The link/parse logic lives in the EE otel modules; this OSS side only
captures the header and calls the (no-op outside EE) hook. Companion EE PR
required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to inbound-trace-propagation EE branch
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* docs(agents): don't attribute work to specific customers in repo content
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(otel): relocate job + script spans into the inbound trace
Builds on the captured _wm_traceparent: the worker job span is re-parented on
the inbound caller context, the script subprocess's TRACEPARENT env is the
inbound context (so its spans join the caller's trace), and the context is
propagated to flow steps so the whole flow relocates. Carried to the worker via
a new LogContext.inbound_traceparent field. Non-inbound jobs are unchanged.
Adds a relocation integration test. Companion EE PR required.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: bump ee-repo-ref to inbound-trace-propagation relocate commit
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(otel): harden inbound traceparent capture
Address review feedback:
- strip any caller-supplied _wm_traceparent from args/extra before stashing the
header-captured value, so the reserved key is Windmill-controlled only
- valid_w3c_traceparent: reject version ff and require lowercase hex, so we don't
forward an inbound header that downstream OTel parsers would reject
- clarify that the capture helper does not validate the W3C format (done at use)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 2c7964460327fab5e3a27c0f74b8d6f26ab7f79a
This commit updates the EE repository reference after PR #604 was merged in windmill-ee-private.
Previous ee-repo-ref: 8fc04fb105dc49769205f7174d551a0d134d1bec
New ee-repo-ref: 2c7964460327fab5e3a27c0f74b8d6f26ab7f79a
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: distinguish canceled jobs in runs
* fix: order status=failure|canceled by completed_at to use partial index
The new `status` query param replaced the legacy `success=false` filter on
the Runs page, but the ORDER BY switch in list_completed_jobs_query only
flipped to v2_job_completed.completed_at for success==Some(false). With
status=failure|canceled (and success=None), the query fell back to ordering
by v2_job.created_at, which the partial index
ix_v2_job_completed_failure_workspace (workspace_id, completed_at DESC WHERE
status IN ('failure','canceled')) cannot serve.
EXPLAIN ANALYZE on 500k rows (1% failure/canceled): ordering by completed_at
uses the partial index (~150 buffers, 0.3ms); ordering by created_at scans
the v2_job created_at index and probes/discards 99% of rows via the join
(~49k buffers, 31ms). Switch the ordering to completed_at for
failure/canceled so the partial index serves both filtering and ordering.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* test: trim order-by regression test to the failure/canceled case
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix: only treat canceled as a terminal status icon for completed jobs
Guard the canceled branch in JobStatusIcon and getJobStatusKind with
`'success' in job` so a job that is still running while being canceled keeps
its running icon/favicon until it completes, instead of immediately showing
the gray Canceled state. Also clarify the openapi `status` param is an exact
match (status=success excludes skipped, unlike success=true).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Ruben Fiszel <ruben@windmill.dev>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tristantr
Ruben Fiszel
centdix
Amey Pawar
