* perf: resolve the worker external IP in the background
`run_workers` awaited `external_ip::get_ip()` — an HTTPS GET to
hub.windmill.dev — before spawning any worker, so every worker process paid
that round trip before its first job pull. Measured on a CE debug build it was
120-450 ms of a ~200-500 ms startup, and behind a firewall the call does not
fail fast: it burns its whole 5 s connect timeout, on every process start. That
cost is per-job under EXIT_AFTER_N_JOBS.
The value is informational (it is only written to `worker_ping.ip`, which the
workers list displays so users can whitelist the address), so nothing needs to
wait on it. It now resolves into a process-wide cache off the startup path, and
`WORKER_EXTERNAL_IP` supplies it explicitly for deployments that know their
egress address or have no egress at all.
Until it resolves the ping carries no IP, which `insert_ping_query` now
COALESCEs so a reclaimed row keeps the address the previous process wrote
instead of being blanked. The main loop reports the IP as soon as it lands
rather than on the next periodic tick, so a short-lived process still records
it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep unknown worker IPs out of the whitelist alert
Review follow-ups:
- `WhitelistIp` filtered only the `'unretrievable IP'` sentinel, so the `'NO IP'`
one a pending or failed lookup now leaves in the row would be offered as an
address to whitelist. It filters both.
- Register `WORKER_EXTERNAL_IP` in `ENV_SETTINGS` so operators can confirm from
the instance settings view that it took effect.
- The worker tracked whether it had reported the IP by re-reading the cache
after each ping rather than remembering what the ping carried, so a lookup
landing mid-ping marked it reported without it reaching the row. The value is
read once and threaded through `insert_ping` / `update_worker_ping_full`.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: report a sentinel IP once the lookup has definitively failed
Keeping the previous process's address on a reclaimed `worker_ping` row is right
while the lookup is still in flight, but not once it has failed: the row would
advertise an address nothing has confirmed, and the whitelist alert would offer
it. A failed lookup now reports `UNKNOWN_IP`, leaving NULL to mean "in flight".
`WORKER_EXTERNAL_IP` is rejected when longer than the `varchar(50)` column
rather than panicking the worker on its initial ping, which is a hard failure.
Adds the regression guard for the `ON CONFLICT` semantics: reverting to
`ip = EXCLUDED.ip` would compile and blank every reclaimed row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: keep the agent initial ping acceptable to older servers
An agent worker routinely runs against a server of a different version, and one
predating the background lookup rejects an initial ping carrying no IP — which
`run_worker` turns into a panic, so a newly upgraded agent would crash-loop
against it. The not-resolved-yet case goes over the wire as the sentinel
instead, and the server maps it back so a reclaimed row still keeps its address
while resolution is pending.
Also documents `ip` as the one conditional exception to `insert_ping_query`'s
"only `started_at` and `jobs_executed` survive a restart", and adds
`WORKER_EXTERNAL_IP` to the README env-var table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: deliver the resolved IP to servers that only take it at registration
A server predating the background lookup applies `ip` from the initial ping
only, and ignores it on the periodic ones. An agent registering before its
lookup resolves would therefore keep the sentinel forever on such a server,
where it used to report its real address. It registers a second time once the
address is known, skipping that when the address is still unknown, when the
server is reached over SQL and needs no second registration, or once a job has
run, since registering clears the row's current job.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: re-register the resolved IP even after a job has run
Gating the second registration on "this process has not run a job yet" meant an
agent that pulled queued work before its lookup resolved never delivered the
address to a server that only takes one at registration. No job of the worker is
in flight where that runs, so the gate bought nothing beyond the last job's id,
which the next job refills.
Documents the two cases where WORKER_EXTERNAL_IP stops being an optimisation and
becomes the only way to report an address: an agent against such a server, and a
process shorter-lived than the lookup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* revert: drop the WORKER_EXTERNAL_IP escape hatch
Supplying the address by hand skips the hub lookup, which is not something to
make easy. Resolving it in the background is what keeps it off the startup path;
opting out of it is a separate decision this does not need to take.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: distinguish an IP never established from one that could not be retrieved
`NO IP` was doing double duty: the column default for a row whose lookup has not
resolved, and the marker for one that failed. An operator reading the workers
list could not tell "not resolved yet" from "this instance cannot reach the
hub", and the latter is the actionable one. A failed lookup now reports
`unretrievable IP`, which is also what it reported before the lookup moved off
the startup path.
That leaves `NO IP` meaning only "no address established", which is what an
agent sends while its lookup is in flight and what the server maps back to
"unresolved" — so the wire sentinel no longer collides with the failure marker,
and an agent delivers the failure to a server that only reads an IP at
registration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* feat: add EXIT_AFTER_N_JOBS worker mode for environment cleanup
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address review findings on the EXIT_AFTER_N_JOBS worker mode
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address round-2 review findings on EXIT_AFTER_N_JOBS
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: address round-3 review findings on EXIT_AFTER_N_JOBS
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: bound WORKER_SUFFIX length and document the same-worker drain
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix: validate the assembled worker name length
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
* refactor: extract load helpers from reload_setting family
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert atomic primitive globals to AtomicBool/AtomicI64
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert CRITICAL_*/HUB_API_SECRET/INSTANCE_EVENTS_WEBHOOK/JWT_SECRET to ArcSwap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pin ee-repo-ref to arcswap-refactor EE branch commit
* refactor: convert BASE_URL/HUB_BASE_URL/MIN_VERSION/LICENSE_KEY*/LICENSE_KEY_ID to ArcSwap
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert worker hot-path globals to ArcSwap (WORKER_CONFIG et al)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pin ee-repo-ref to combined arcswap-urls+worker EE commit
* chore: update ee-repo-ref to d8be8f88cb8898c8f6b27421989d53528223815d
This commit updates the EE repository reference after PR #532 was merged in windmill-ee-private.
Previous ee-repo-ref: c375aaaac9ec0fc0480993627d0defc8054c31a4
New ee-repo-ref: d8be8f88cb8898c8f6b27421989d53528223815d
Automated by sync-ee-ref workflow.
* fix: cleanup unused imports + fix 2 missed WORKER_CONFIG readers
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to ce0f8fbbbde09c4a858312d2d8716d224e99042c
This commit updates the EE repository reference after PR #534 was merged in windmill-ee-private.
Previous ee-repo-ref: 450b601b5aba0ca0b2045f4b5071aa8701b4bfb7
New ee-repo-ref: ce0f8fbbbde09c4a858312d2d8716d224e99042c
Automated by sync-ee-ref workflow.
* fix: secret_backend_integration test — BASE_URL.write().await → .store()
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: convert APP_WORKSPACED_ROUTE to AtomicBool for symmetry with HTTP_ROUTE_WORKSPACED_ROUTE
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to e587df8 (post-#535 merge)
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* fix: enrich OTEL log records with per-request LogContext
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: add otlp_smoke example for manual OTEL log bridge verification
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update ee-repo-ref to 5d6b713b74fc46735807f5c32883002e8d976fbc
This commit updates the EE repository reference after PR #529 was merged in windmill-ee-private.
Previous ee-repo-ref: 45959d063bc941c567488d330b5819601cdd2d3d
New ee-repo-ref: 5d6b713b74fc46735807f5c32883002e8d976fbc
Automated by sync-ee-ref workflow.
* refactor: store LogContext in ArcSwap instead of Mutex
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: pin ee-repo-ref to ArcSwap branch commit
* chore: update ee-repo-ref to be2f3d4d11bb7110200524d7157caab3aac53996
This commit updates the EE repository reference after PR #530 was merged in windmill-ee-private.
Previous ee-repo-ref: 45b4d7963a9ebcd583d1a87abe7d07d3d521584a
New ee-repo-ref: be2f3d4d11bb7110200524d7157caab3aac53996
Automated by sync-ee-ref workflow.
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
* Show effective isolation level on workers page, not configured
The workers page displayed the configured isolation level (nsjail/unshare)
even when the binary wasn't actually available, which was misleading.
Now shows "none (nsjail unavailable)" or "none (unshare unavailable)"
when the setting is enabled but the binary failed its startup test,
so admins can immediately see the mismatch from the UI.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* Simplify: use standard 'none' value when isolation binary unavailable
Keep the string as one of the 3 known values (nsjail/unshare/none)
since the frontend checks === 'none' for the warning badge. Now if
nsjail/unshare is configured but the binary is unavailable, it
correctly reports 'none' so the warning badge shows up.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add force_sandboxing global setting and #sandbox bash annotation
Add a new global setting `force_sandboxing` to enable nsjail sandboxing
for all jobs, overriding the DISABLE_NSJAIL environment variable.
Also add a #sandbox annotation for bash scripts to enable sandboxing
on a per-script basis.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* fix: decouple nsjail probing from DISABLE_NSJAIL and apply force_sandboxing to all executors
NSJAIL_AVAILABLE now always probes for the nsjail binary at startup
regardless of the DISABLE_NSJAIL policy flag, fixing #sandbox annotation
and force_sandboxing being unusable in default (DISABLE_NSJAIL=true)
configuration. All language executors now use is_sandboxing_enabled()
instead of checking DISABLE_NSJAIL directly, so force_sandboxing
applies to all job types, not just bash.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* fix: clarify force_sandboxing and DISABLE_NSJAIL relationship
Neither setting overrides the other — sandboxing is enabled when either
force_sandboxing=true OR DISABLE_NSJAIL=false. Updated frontend
description and backend comments to reflect this.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* all
* all
* revert typo
* all
* all
---------
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* feat: workspace dedicated workers
* ref
* chore: update ee-repo-ref to a18ac31062ac092cb9a5fc87629e217d97f4911d
This commit updates the EE repository reference after PR #398 was merged in windmill-ee-private.
Previous ee-repo-ref: 98cfe3fef764d9d815d326d5056c734a03689d33
New ee-repo-ref: a18ac31062ac092cb9a5fc87629e217d97f4911d
Automated by sync-ee-ref workflow.
* fix(frontend): workspace script in flow steps
---------
Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com>
Ruben FiszelandClaude Opus 5
hugocasa
Pyraandwindmill-internal-app[bot]
