* fix: refetch license key from settings when in-memory key is invalid
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: only record accepted license keys so rejected keys stay retryable
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* feat(worker): #ssh directive to run a bash script on a remote SSH host
Add a first-class `#ssh <resource_path>` bash directive that reroutes a
normal bash script to run on a remote host reached over SSH (a
jump/utility node) instead of on the worker, with full parity: typed
positional args in, structured result out, live streamed logs,
cancellation, and remote exit-code propagation.
It mirrors the existing `# sandbox <image>` precedent: the directive is
parsed in handle_bash_job and reroutes to a specialized handler that
reuses handle_child for all execution plumbing.
- windmill-common: BashAnnotations::ssh_target() parser (+ unit test)
and the ssh_execution_enabled instance setting (off by default)
- windmill-worker: reroute hook in bash_executor + ssh_executor_oss
shim. OSS returns a clear "enterprise feature" error; the real
handler lives in ssh_executor_ee.rs (private feature) and is gated by
a valid enterprise license + the instance setting.
- examples/usecase/ssh-execution-wrapper: the ssh_target resource type,
a userland wrapper (no-license fallback), and a README documenting
both paths and the trade-offs vs agent workers.
EE companion: windmill-labs/windmill-ee-private (ee-repo-ref.txt bumped).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(worker): ssh host-key opt-in, 0600 key write, instance setting UI
* chore: update ee-repo-ref
* feat(worker): #ssh $arg form to take the ssh target from a job argument
* fix(worker): #ssh token must look like a target; $arg restricted to path strings
* fix(worker): tighten #ssh parser to exact directive; add -- ssh destination guard
* chore: update ee-repo-ref to d45b9a6cbe40f7fe5d322c850c50f64a6980e4f0
This commit updates the EE repository reference after PR #609 was merged in windmill-ee-private.
Previous ee-repo-ref: 2804f1aa8e74b3a7733aeb6f5044d5085193872a
New ee-repo-ref: d45b9a6cbe40f7fe5d322c850c50f64a6980e4f0
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* feat: clear conflict error + force delete when reusing a fork workspace id
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* fix: guard fork force-delete against double submit
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: investigate pod-deletion-cost for k8s autoscaling scale-in (WIN-2028)
* docs: correct worker_instance aggregation claim per review
* docs: drop investigation doc in favor of implementation
* chore: bump ee-repo-ref for pod-deletion-cost autoscaling scale-in
* chore: update ee-repo-ref to 80c39bf7f3bfeda4cf0974ce32826f53affd9574
This commit updates the EE repository reference after PR #610 was merged in windmill-ee-private.
Previous ee-repo-ref: 51f79d4a49dd6491f4809f3edcb3919571719da7
New ee-repo-ref: 80c39bf7f3bfeda4cf0974ce32826f53affd9574
Automated by sync-ee-ref workflow.
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Missing config is a client/config error, not a server fault. Restores the
BadRequest class lost when hub_token() was extracted.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(flow): support worker tag override on AI agent steps
* refactor: drop ineffective tag passthrough in nested agent tool path
* chore: regenerate openflow-derived system prompt artifacts
When HTTP request tracing is enabled, the NO_PROXY injected into traced jobs
was built solely from the no_proxy_hosts instance setting, ignoring the
worker container's own NO_PROXY. Enabling tracing therefore silently dropped
every exclusion an operator had already configured at the container level,
funneling those hosts into the MITM proxy (and on to any upstream corporate
proxy). The upstream-relay side already honored the container NO_PROXY; this
makes the injected-into-jobs side symmetric by merging both sources.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* feat(api): add endpoint to update token label
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(api): prevent renaming the session token label
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(api): restrict token-label edits to user tokens, not just session
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(frontend): edit token label in the edit modal instead of inline
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(api): reject relabeling tokens to reserved system-token names
Centralize the is_user_token classifier in windmill-common and reuse it
to reject labels colliding with system-token namespaces (ephemeral*,
debugger-token, mcp-oauth-*), not just session.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(api): match ephemeral label case-insensitively and cap label length
Align the canonical is_user_token, the SQL guard and the frontend mirror on
a case-insensitive `ephemeral` match (so a token can't be relabeled to a
casing the backend allows but the UI hides), reject labels over the
VARCHAR(1000) column limit with a 400, and add unit tests for is_user_token.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(forks): defer trigger/schedule state to parent for clean git merge
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(forks): read parent trigger/schedule state on non-RLS pool for complete substitution
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* fix(forks): read schedule fork-ness on non-RLS pool; clarify mutator-rule wording
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
slug/project_slug are caller-controlled and were interpolated straight into
the Hub request path; a crafted value (e.g. ../../admin) could reach an
unintended Hub endpoint after URL normalization. Validate against the
frontend charset (lowercase alphanumerics + hyphens, 3-50 chars) in the four
handlers that put the slug in the path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lazy on-fetch OAuth token refresh persisted the new access token with a raw
`UPDATE variable SET value = <db-encrypted>`, bypassing the secret-backend
abstraction. With an external secret backend (AWS Secrets Manager / Azure Key
Vault / Vault), secret reads resolve through the backend and ignore
`variable.value` entirely, so refresh advanced `account.expires_at` and updated
Postgres but never wrote the new token to the external store. Every read that
did not itself trigger a mint kept serving the frozen connect-time token, which
expired ~1h after connect (RefreshError on Google clients).
`windmill-oauth` can't depend on `windmill-store` (circular), so variable
persistence moves out of `refresh_token{,_for_account}` (which now only exchange
the token + update the `account` row and return the new token) into the
`windmill-store` callers, via a new `store_oauth_token_value` helper that writes
through the configured backend and stores the returned value (encrypted blob for
the DB backend, `$...:` marker for external backends) in `variable.value`.
If persisting the refreshed token fails (more likely now that it can be a
network write to an external backend) after the account was committed fresh,
`store_oauth_token_value` resets `expires_at` to the past and records
`refresh_error` — looking the account up via `variable.account` — so the next
fetch retries instead of serving the stale token for the whole token lifetime.
Also add `windmill-store/tests/oauth_refresh_secret_backend.rs`, an opt-in e2e
regression suite (RUN_SECRET_BACKEND_E2E / RUN_AWS_SM_TESTS) covering database
and external (AWS SM via LocalStack) backends plus the self-healing reset.
Verified against Postgres + LocalStack: 3 passed.
EE companion (oauth_refresh_ee.rs: 3 refresh paths) merged via #607; this OSS
half completes the fix (ee-repo-ref already at EE main 481ea7f).
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The workspace-switch effect only reset load-derived state, so phase,
draftItems, recordings, hub/bundle metadata, hubVersion, deploymentStatus,
effectiveSlug and hubItemIds survived a switch — a draft built in one
workspace could publish its items/slug under the next workspace's auth.
Reset the full publish session on switch. Also drop explanatory comments.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Hub publish endpoints ignored the {workspace} path and addressed the Hub
project purely by client-supplied project_slug, forwarding with an
instance-wide HUB_DEV_TOKEN. Any workspace admin could mutate or export
another workspace's Hub project by passing its slug.
Stamp the server-trusted workspace from the path onto every forwarded
request as source_id (body for mutations, query param for export) so the
Hub can enforce that the targeted project belongs to the calling
workspace. Requires the matching Hub-side source_id ownership check.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add an optional path field to the publish bodies and a GET
/projects/{slug}/export route that proxies the hub export (admin-only,
authenticated with HUB_DEV_TOKEN).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add POST /w/{workspace}/hub/raw_apps/{id}/embed forwarding to the Hub so a
shared (public) raw app's external_embed_url can be set/cleared. null is
forwarded (not skipped) so unpublish clears the embed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Matches the windmillhub rename: every body now carries `project_slug`
instead of `workspace_slug`, the draft creation forwards to `/projects`,
and the resource_types/resources/triggers proxies hit
`/projects/{slug}/...`. `HubWorkspaceBody` becomes `HubProjectBody`. The
instance-side `Path(workspace)` extractor and the `workspace` URL
parameter stay because that's still the source tenant's identifier.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Mirrors the existing /scripts, /flows, /apps thin proxies. Forwards
{ triggers, workspace_slug } to Hub's POST /workspaces/[slug]/triggers
bulk-replace endpoint, with the same require_admin + HUB_DEV_TOKEN
guardrails. Lets the frontend push trigger stubs in a single round-trip
after the items they reference have landed on the Hub.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add ApiAuthed + require_admin to all hub publish handlers; previously any
workspace-authenticated session could trigger Hub-side writes attributed
to the URL workspace via the shared HUB_DEV_TOKEN.
- Track per-kind trigger fetch failures (triggerLoadErrors) so an EE-gated
or transiently failing trigger service no longer silently maps to "0
triggers"; UI surfaces an amber badge listing the missing kinds and a
toast warns the operator before publish.
- Add workspaceLoadSeq cancellation so the parallel loadWorkspace +
loadTriggers stop bleeding stale data when the workspace switches mid
load.
- Drop the silent effectiveSlug fallback to sanitizeSlug(hubName) when
the Hub response can't be parsed; abort the publish instead so items
don't land under a slug the Hub never locked.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Forward workspace resource stubs (path + type) to the hub's
/workspaces/{slug}/resources endpoint.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New workspaced router /api/w/:ws/hub/* forwarding to the Hub:
- POST /publish_draft → POST {HUB}/workspaces (slug/name/summary/readme)
- POST /scripts → POST {HUB}/scripts/add (workspace_slug + content)
- POST /flows | /apps | /raw_apps → corresponding hub endpoints
- POST /scripts/:ask_id/recording, /flows/:flow_id/recording → recording uploads
Auth uses HUB_DEV_TOKEN env var (dev shortcut). All bodies are
serde-typed; the helper forward_to_hub centralises the HTTP call.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(queue): bump EE ref to include worker_ping fairness signal
The current ee-repo-ref.txt pointed to 31cda7c (an unrelated merge
commit on the asset-graph-view-ee branch) instead of ddc9e80, which
contains the workspace-fairness fix that switches the active-share
signal from v2_job_queue.running=true to worker_ping. As a result
cloud was still computing overload off the legacy signal, so a
workspace with many in-flight/suspended flows (lancom01-prod, with
799 suspended flows × 3 v2_job_queue bookkeeping rows each = 2397
running-true rows) was flagged as 95% of cluster activity despite
consuming zero worker slots.
Bumping to ddc9e80 picks up the worker_ping-based signal, which
naturally excludes (a) suspended jobs (no worker pinging them),
(b) zombie running-rows from dead workers, and (c) flow/flownode
orchestration rows that never run on a worker in the first place.
* test(queue): seed v2_job rows + realistic durations for fairness helpers
The new duration-weighted fairness algorithm joins v2_job_queue and
v2_job_completed to v2_job for the `kind` filter (excluding flow
bookkeeping) and reads `duration_ms` for the completed contribution.
Update the test helpers to mirror that schema:
* `insert_completed` now inserts a matching v2_job row (kind=script)
and writes `duration_ms = 1000` with a 1-second [started_at,
completed_at] interval, so each completed row contributes ~1
worker-second when fully inside the refresh window.
* `insert_queued` likewise pre-inserts v2_job, sets `started_at`
to NOW() - 1s when running=true (so running rows contribute ~1
worker-second by the time the refresh runs), and seeds
v2_job_runtime.ping so the running side accrues real-time worker
seconds (the algorithm bounds end-of-interval by ping).
The zombie/suspended insert helpers are intentionally left without
v2_job rows — the new algorithm's INNER JOIN excludes them, so they
still correctly contribute zero worker-seconds.
* chore(queue): bump EE ref to duration-weighted fairness algorithm
Companion to windmill-ee-private#<TBD>: switch the EE workspace
fairness aggregation from a count-based UNION (worker_ping snapshot
+ v2_job_completed count) to a worker-seconds aggregation sourced
directly from v2_job_queue and v2_job_completed, with kind/suspend
filters mirroring handle_zombie_jobs and per-row defenses against
zombie inflation on both halves.
* chore(queue): bump EE ref for fairness perf fix (inline window_start)
* chore(queue): bump EE ref for fairness perf rewrite (driver-side flip)
* update ee ref
`fairness_ignores_zombie_running_rows` and
`fairness_ignores_concurrency_suspended_rows` panic intermittently in CI
(both Linux and Windows runs). Mark them `#[ignore]` until the
underlying flakiness is resolved.
* feat(websocket-trigger): honor HTTPS_PROXY/HTTP_PROXY/NO_PROXY (WIN-1988)
`tokio_tungstenite::connect_async` opens a raw TCP socket and ignores
the standard outbound-proxy env vars, so deployments behind a forward
HTTP proxy can't reach the WebSocket endpoint and Test Connection
times out after 30s.
Add a small `proxy` module that resolves the right proxy URL for the
target host (HTTPS_PROXY for wss://, HTTP_PROXY for ws://, NO_PROXY
exclusions, ALL_PROXY fallback, lowercase variants), opens an HTTP
CONNECT tunnel when one applies, and hands the resulting TcpStream to
`client_async_tls_with_config` for the TLS + WS handshake. Direct
connect remains the default when no proxy env is set.
Unit tests cover NO_PROXY matching, proxy URL parsing (including IPv6
literals and basic-auth userinfo), and the CONNECT handshake itself
against an in-process fake proxy (success, basic-auth header, 407
rejection).
Fixes WIN-1988
* refactor(websocket-trigger): reduce blast radius and reuse existing logic
Follow-up to the proxy support change. Three things:
1. Skip the new code path entirely when no proxy is configured.
`connect_async_with_proxy` now checks the env-var snapshots up front
and delegates straight to `tokio_tungstenite::connect_async` if
neither `HTTP_PROXY` nor `HTTPS_PROXY` is set. Same fall-through
applies when proxy env is set but `NO_PROXY` excludes the host or
the proxy URL doesn't parse. Non-proxied deployments now exercise
exactly the previous code path.
2. Move the `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` env-var snapshots
from `windmill-worker::worker` into `windmill-common`. The worker's
`PROXY_ENVS` static now reads from there, and the websocket trigger
reads from the same source — one place reads the env, one source
of truth for both call sites.
3. Replace the hand-rolled proxy-URL parser with `url::Url::parse`
(already a workspace dep, used across the codebase). Half the LoC
and handles edge cases (userinfo percent-encoding, IPv6 literals,
path/query stripping) via the well-tested crate instead of by hand.
All 13 proxy unit tests still pass. `cargo check` is clean.
* fix(websocket-trigger): unbreak EE build + trim proxy tests
- Re-export `NO_PROXY` / `HTTP_PROXY` / `HTTPS_PROXY` from
`windmill-worker::worker` (via `pub use windmill_common::...`) so the
EE `otel_tracing_proxy_ee` module's `use crate::{HTTPS_PROXY, ...}`
resolves like it did before. Fixes the `check_ee_full` / `cargo_test`
CI failures from the previous commit.
- Trim the proxy tests to one un-ignored canary
(`http_connect_tunnel_sends_well_formed_request_and_unwraps_stream`)
that exercises the actual on-wire CONNECT handshake plus byte-perfect
tunnel passthrough. The NO_PROXY-matching, URL-parsing, and edge-case
tunnel tests are kept under `#[ignore]` for manual debugging
(`cargo test -- --ignored`) since they're either delegated to
`url::Url::parse` or trivial string matching — low ROI on every CI run.
tristantr
Ruben Fiszelandrubenfiszel
centdix
hugocasaandClaude Opus 4.8