mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-21 00:02:30 +00:00
245bbd5d27aa7a2ee26ee9e6ff0591d8042d4bbf
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  hugocasaandClaude Opus 5
|
53afecd458 |
fix: register the job token with the sensitive log masking system (#10943)
* fix(worker): register the job token with the log masking system The masking system covered secrets fetched through `get_value_internal` and `$encrypted:` args, but not the job's own token, so a script that echoed `$WM_TOKEN` wrote it verbatim into logs that are persisted to the database and, when configured, to object storage. `run_worker` now registers the token for the job it just pulled, alongside the existing `register_running_job` call, so it is redacted like any other registered secret. That makes every job carry at least one registered value, where before the per-batch mask snapshot was skipped entirely for the majority of jobs that touched no secret. Cache the compiled Aho-Corasick automaton per job and invalidate it when a new secret is registered, so a chatty job no longer rebuilds it once per log batch. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 * fix(nativets): mask secrets in the in-process log path NativeTS hands `console.log` output to a task that drains a channel into `append_logs`, so it never reaches the masking in `handle_child::write_lines` and a script logging `$WM_TOKEN` persisted the raw JWT. That drain can still be flushing after the job is unregistered, so a plain per-line `snapshot` would leave the tail unmasked. `JobMasker` keeps the last masks it saw for exactly that window, and refreshes while the job is alive so secrets fetched mid-run are covered too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 * fix(nativets): seed the job masker at construction A `JobMasker` that only looked up its masks on the first `mask` call had the same hole at the head of the log that its retention closes at the tail: if the drain task's first productive poll landed after the job was unregistered, the registry was already gone and every line was written raw. `new` now takes the snapshot, and its callers construct it from the job's own execution while the job is still registered. Also cover the nativets sink with an integration test, gated on `deno_core` the way the CI test build is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 * fix(nativets): mask on the producing side of the log channel Masking as the drain task wrote to `append_logs` left two holes, because that task is detached and outlives the job: a secret registered mid-run could still be queued when the job was unregistered and would then be written raw, and the `windmill:job_log` tracing emission that EE forwards job logs on never went through the mask at all. Mask where the line is produced instead. That loop is joined before the job completes, so the job's secrets are always still registered, and one call now covers both the tracing mirror and the channel. The result stream keeps reading the raw text, the way `handle_child` keeps its raw `line` for results. `JobMasker` is no longer load-bearing for the post-unregistration window, so it is documented for what it now does: keep the security notice to once per set of secrets for a sink that masks line by line. Also drop the nativets test's tag override — `DEFAULT_TAGS` does advertise `nativets`, so the comment justifying it was wrong — and pin the automaton cache invalidation, whose failure mode is an unmasked secret. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 * fix(worker): hold the log-masking lifecycle at the job boundary Registering the job around the poller's call left every other way of running a job uncovered: the interactive worker shell and inline AI agent tools both call `handle_queued_job` directly, and a script logging `$WM_TOKEN` from either persisted the live credential. Register from inside `handle_queued_job` instead, under a drop guard, so each path is covered by construction rather than by remembering to add a call. Nothing is lost by unregistering earlier: the writes that follow go through `append_logs`, which never consulted the registry. In nativets, decide the stream/log routing before masking. `MaskSnapshot`'s notice is one-shot, so a secret-bearing `WM_STREAM:` chunk used to spend it on text that is then discarded, leaving later redactions in `job_logs` unexplained. Restore the masker's post-unregistration test: the memory-limit path never joins the producing loop, so that fallback is still load-bearing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 * docs: correct the nativets masking comments The producer loop is not joined on the memory-limit path, so it does not "always" run while the job is registered — say normally, which is what `JobMasker`'s fallback is there for. Name the reason a stream chunk stays raw everywhere it goes, including the tracing mirror: it is result data that no log sink persists, so masking it would be masking a result. State the masker test's invariant without asserting a mechanism behind it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 * fix(masking): keep the security notice on a line of its own `mask` appended the notice as a newline plus the notice text, which assumes the caller hands it a bare log line. nativets hands it a chunk that already ends in a newline, and its sink concatenates chunks verbatim, so the notice arrived after a blank line and the next log line was welded onto the end of it. Emit the notice as its own line for either shape. `handle_child` is unaffected: its input never ends in a newline, so it keeps the original path. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01P316wKe2QCYNcdsx1PwAJ3 --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
 Ruben FiszelandClaude Opus 5
|
785277e0bb |
feat(nativets): bound fetch on a peer that never answers (#11026)
* feat(nativets): bound fetch on a peer that never answers
deno_fetch applies no deadline of any kind. A peer that completes the TCP
handshake, accepts the request and then goes silent leaves `await fetch(...)`
pending indefinitely, holding its worker slot until the *job* timeout -- which
on self-hosted defaults to DEFAULT_SELFHOSTED_TIMEOUT, i.e. 7 days.
Nothing else catches this. Zombie-job detection keys off a stale
v2_job_runtime.ping, and a worker blocked inside a pending fetch keeps pinging
normally throughout: the worker is alive and healthy, only the work is dead.
What this bounds is the wait for a response to begin, and it stops there:
- a peer that never answers -> rejected after N seconds
- a peer slow to answer, but under N -> unaffected
- a body that then streams for an hour,
or is read slowly by the caller -> unaffected, always
That last line rules out the obvious implementation: AbortSignal.timeout(N)
around every fetch would bound the hang and break every streaming response and
long download. This is a hang detector, not a latency budget.
Default 300s via WINDMILL_FETCH_RESPONSE_TIMEOUT_SECS (0 disables), with a
per-script `//fetch_response_timeout <seconds>` annotation alongside the
existing //useragent and //proxy. Both nativets paths inherit it, since
eval_fetch_timeout and the dedicated-worker path in bun_executor both funnel
through create_nativets_runtime.
The ms value is clamped to i32::MAX: deno_web's setTimeout runs its delay
through webidl.converters.long, a 32-bit conversion that *wraps*, so a setting
past ~24.8 days would come out negative and fire immediately -- turning an
over-generous timeout into an instant one on every fetch.
The window covers connect, TLS and request upload as well as server think
time, so a very slow large upload is bounded by it too; the error message says
so rather than claiming the connection went silent.
Not covered: a body that stalls midway. Reaching that needs the response's
InnerBody, which deno_fetch keeps module-private, and every way to wrap it
from outside changes observable Response semantics (locking, bodyUsed,
double-consume errors). Left for a follow-up in deno_fetch itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* test(nativets): cover the instance-wide response-timeout env var
A typo in WINDMILL_FETCH_RESPONSE_TIMEOUT_SECS would compile, pass every
other test, and silently hand every operator the 300s default -- the same
class of silent-default failure the timeout itself exists to prevent. Its
own test binary, since a LazyLock resolves the value once per process.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): inherit the caller's RequestInit, and clear the long-poll ceiling
Two problems with the first cut, both found in review.
`{ ...init, signal }` copied only own enumerable properties, but RequestInit is
a WebIDL dictionary whose members deno_fetch reads with plain property gets
that walk the prototype chain. Anything inherited or non-enumerable was
dropped: `fetch(url, Object.create({method: "POST"}))` silently became a GET.
Worse, a non-object init went from a loud TypeError to a silent GET, because
spreading "POST" yields {0:"P",1:"O",...} -- a valid dictionary with ignored
keys. Now the init is inherited from rather than copied, and a non-dictionary
is handed straight back to deno_fetch for its own TypeError.
The 300s default also sat at half of TIMEOUT_WAIT_RESULT (600s), which
run_wait_result long-polls against with no response headers. A script running
another job synchronously for 300-600s would have timed out client-side while
the server was still legitimately holding the request open -- the long-poll
risk class, instantiated inside the product and reachable without writing a
raw fetch. Default raised to 900s, with the constraint recorded where someone
would break it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): hand the caller's RequestInit to Request untouched
Carrying a WebIDL dictionary across by hand has no safe form, and both
previous attempts were wrong in opposite directions. Spreading a copy drops
inherited and non-enumerable members, and turns a non-object init from a loud
TypeError into a silent GET. Inheriting from it via Object.create fixes those
but makes the child the receiver, so an accessor on the original runs against
an object that lacks its private-field brand:
Cannot read private member #body from an object whose class did not
declare it
So don't carry it at all. fetch()'s own first act is `new Request(input,
init)`; doing that here hands the init to the same constructor, read exactly
as it would be without this wrapper, and our signal travels in an init we own.
`req.signal` is then deno's own resolution of init.signal over an input
Request's signal, which removes the hand-rolled version of that rule too.
The Request is built twice as a result, once here and once inside fetch. That
is cheap: cloneInnerRequest carries method, headers, redirect mode, clientRid
and blob entry, and a body is proxied rather than buffered -- a static body is
a shallow {body, consumed} copy sharing its bytes, a stream gets a one-chunk
pass-through.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): keep an aborted fetch settling in the same tick
deno_fetch keeps its outer fetch non-async on purpose: "WPT has a test that
aborted fetch is settled in the same tick. This means we cannot wrap the
promise if it is already settled" (26_fetch.js). An `async` wrapper adopts
that promise through another one, so a rejection that used to land before any
microtask queued after the call now lands after it.
Made the wrapper non-async, with an early return that hands deno's settled
rejection straight back for an already-aborted signal, and no timer armed
there since there is no response to wait for. Construction still has to reject
rather than throw, so it is caught and returned as a rejection, which is what
the `async` was buying.
The comment claiming this matched deno_fetch's own `async function fetch` was
wrong on two counts -- that function is not async, and the wrapper was not
matching it. Replaced with the constraint that actually holds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): keep fetch's observable shape and reach intrinsics safely
Three ways the wrapper was distinguishable from the fetch it replaces, all
observable from a script sharing the isolate.
`.then` was an ordinary property lookup, so `Promise.prototype.then =
undefined` broke fetch after the request had already gone out. deno's own
modules reach intrinsics through primordials, and this file already captured
setTimeout, clearTimeout and Promise.reject for exactly that reason, so the
lookup was the odd one out. Now captured alongside them.
Declaring `init` without a default made `fetch.length` 2 where the standard
says 1. And the empty-call branch forwarded two explicit `undefined`s, so
deno's required-argument check saw two arguments and raised "Invalid URL:
'undefined'" instead of "1 argument required". Forwarding through
ReflectApply preserves the count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* docs: clarify fetch timeout restart requirements
* fix: capture native fetch abort helpers
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
6d1e12d5e9 |
feat(nativets): expose the standard web-platform globals deno_web provides (#10112)
* feat(nativets): expose standard web-platform globals for bun parity Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(nativets): wire bun-present Event subclasses and add construction smoke test Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(nativets): seed performance.timeOrigin per isolate, drop broken reportError Addresses CI Codex review on #10112: - performance.timeOrigin was undefined (setTimeOrigin never called); seed it per isolate via __wmInitPerIsolate executed from create_nativets_runtime. - reportError needs a global EventTarget this runtime never installs; drop it. - reword the namespace-import comment to not describe drafting history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(nativets): wire DOMException global + broad edge-case smoke sweep DOMException is present in bun and, more importantly, deno_web references it as a global: AbortController.abort() with no reason constructs a DOMException("...", "AbortError"), so the already-wired AbortController/ AbortSignal threw "DOMException is not defined" on abort. Surfaced by a new functional edge-case sweep (smoke_web_globals_edge_cases) that exercises every wired global for real (not just presence) — DOMException/abort, AbortSignal.timeout, EventTarget dispatch, stream tee/reader/writer, all 3 compression formats, structuredClone Map/Set/Date/circular/reject-function, performance mark/measure, MessagePort delivery — plus a check that the merged Web Crypto globals still work. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(nativets): restore arg-default smoke tests dropped in merge, drop history comments Addresses CI Codex/Pi review on the merge commit: - Merge conflict resolution (checkout --ours) dropped smoke_missing_optional_arg_uses_default and smoke_explicit_null_arg_is_preserved (added on main by #10111); restore them. - Reword edge-case-sweep comments to state the constraint, not how the gaps were found. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(nativets): give reportException a global dispatch target; wire stream reader/controller globals Addresses CI Codex review on #10112: - P1: a throwing EventTarget listener (and reportError) is routed through deno_web's reportException, which dispatches on a saved global reference. With none set, dispatchEvent threw a masking error that hid the original. Wire a dedicated EventTarget as that target so the ORIGINAL error is reported (async unhandled, matching bun). Does NOT make globalThis an EventTarget (bun's isn't either). Re-adds reportError, now functional. Regression test asserts the original error is surfaced, not a masking one. - P2: wire the stream reader/controller globals bun also exposes (ReadableStreamDefaultReader/BYOBReader, ReadableStreamDefault/ByteStreamController, ReadableStreamBYOBRequest, WritableStreamDefaultWriter/Controller, TransformStreamDefaultController) for instanceof parity; sweep verifies via real reader/writer/controller instances. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(nativets): make globalThis an EventTarget so globalThis.reportError() works Addresses follow-up CI Codex review on #10112: - P1: the prior fix saved a *separate* EventTarget as the global reference, so globalThis.reportError() still failed its receiver check (this === globalThis_) with 'Illegal invocation'. Make globalThis itself the saved reference by turning it into a functional EventTarget (setPrototypeOf to DedicatedWorkerGlobalScope + setEventTargetData + webidl brand + saveGlobalThisReference), per isolate in __wmInitPerIsolate. Both reportError(e) and globalThis.reportError(e) now surface the original error (async, matching bun) instead of throwing. New test smoke_report_error_both_call_forms covers both call forms. - P2: reword the regression-test comment to state the invariant, not the patch history. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * feat(nativets): wire performance constructor globals for bun parity Addresses the P2 nit in the CI Codex review: bun exposes Performance, PerformanceEntry, PerformanceMark, and PerformanceMeasure as globals (deno_web exports all four), so wire them alongside the performance singleton. The edge-case sweep verifies instanceof against real mark/measure entries. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(nativets): state global-wiring comment as a constraint, not patch history Addresses the P2 in the CI Codex review: reword the block comment to describe the current bun-parity constraint and the deliberate EventSource/ImageData exclusions, without narrating what was or wasn't wired before (per AGENTS.md). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
ba232544e7 |
feat(nativets): add Web Crypto support via deno_crypto (#10109)
The nativets in-process runtime (deno_core) exposed no Web Crypto API:
`crypto` was undefined, so scripts could not use `crypto.getRandomValues`,
`crypto.randomUUID`, or `crypto.subtle`, even though the bun runner provides
them. This closes that parity gap by registering the `deno_crypto` extension
and wiring the crypto globals onto `globalThis`.
- Pin `deno_crypto = "0.223.0"`, the sibling release of the already-pinned
deno_core 0.352 / deno_web 0.240 stack (deps: deno_core ^0.352,
deno_web ^0.240, deno_error =0.6.1), so the rest of the deno stack is
untouched.
- Register `deno_crypto::init(None)` after `deno_web` in both the snapshot
(build.rs) and the runtime (lib.rs) extension lists, keeping the snapshot a
prefix of the runtime list. deno_crypto declares deps = [deno_webidl,
deno_web], which the position satisfies.
- Import `ext:deno_crypto/00_crypto.js` in runtime.js and assign
`crypto` / `Crypto` / `CryptoKey` / `SubtleCrypto` to `globalThis`.
- Add the `smoke_web_crypto` opt-in smoke test asserting the UUIDv4 shape of
`randomUUID`, a non-zero `getRandomValues` fill, and the known
SHA-256("abc") vector via `subtle.digest`.
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
||
|
Ruben FiszelandClaude Opus 4.8
|
ba7f9c065f |
fix(nativets): apply parameter defaults for missing args instead of null (#10111)
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.8
|
41562c7d7c |
fix(nativets): respect custom CA certs in in-process fetch runtime (#9615)
* fix(nativets): respect custom CA certs in in-process fetch runtime Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(nativets): dedupe CA file paths and clarify DENO_TLS_CA_STORE semantics Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix(nativets): resolve CA env vars from worker-group config too Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
Ruben Fiszel
|
3cd0eac8c1 |
deps: bump deno_core / deno_ast / swc to the goldilocks pin set; drop serde ceiling (#9111)
* deps: bump deno_core / deno_ast / swc to the goldilocks pin set; drop serde ceiling
Bumps every deno_* and swc_* workspace dep to a hand-picked "goldilocks"
combination that drops the serde =1.0.220 ceiling without crashing into
the rustls / aws-sdk resolver wall that the obvious deno v2.6.0 target
hits.
## What's the goldilocks set
| crate | old | new | source |
|------------------|----------|-----------|---------------------------------------|
| deno_core | 0.336.0 | 0.352.0 | deno v2.4.0 |
| deno_fetch | 0.214.0 | 0.233.0 | deno v2.4.0 |
| deno_tls | 0.177.0 | 0.196.0 | deno v2.4.0 (last permissive-rustls) |
| deno_console | 0.190.0 | 0.209.0 | deno v2.4.0 |
| deno_url | 0.190.0 | 0.209.0 | deno v2.4.0 |
| deno_webidl | 0.190.0 | 0.209.0 | deno v2.4.0 |
| deno_web | 0.221.0 | 0.240.0 | deno v2.4.0 |
| deno_io | 0.100.0 | 0.119.0 | deno v2.4.0 |
| deno_net | 0.182.0 | 0.201.0 | deno v2.4.0 |
| deno_permissions | 0.49.0 | 0.68.0 | deno v2.4.0 |
| deno_telemetry | 0.12.0 | 0.31.0 | deno v2.4.0 |
| deno_error | =0.5.5 | =0.6.1 | deno v2.4.0 |
| deno_ast | =0.44.0 | =0.51.0 | **override** — see "load-bearing" below |
| deno_fs | (new) | 0.119.0 | new workspace dep — FetchPermissions exposes deno_fs::CheckedPath / GetPath as public API |
| v8 | =130.0.7 | =137.1.0 | deno_core 0.352 transitive |
| swc_common | =0.37.5 | =14.0.4 | **the load-bearing pin** |
| swc_ecma_ast | =0.118.2 | =15.0.0 | matched set with swc_common 14.0.4 |
| swc_ecma_parser | =0.149.1 | =24.0.3 | matched set |
| swc_ecma_visit | =0.104.8 | =15.0.0 | matched set |
| serde | =1.0.220 | ^1 | **freed** (resolves to 1.0.228+) |
## Why this combination and not v2.6.0
The obvious target was deno v2.6.0 (with deno_ast 0.52 → swc_common 17,
well past the `__private` ceiling). That hits three resolver collisions:
1. libsqlite3-sys: deno_cache → rusqlite 0.37 → libsqlite3-sys 0.35
vs sqlx → libsqlite3-sys 0.30. **Already killed by PR #9110** —
we dropped deno_runtime, which is what pulled in deno_cache.
2. fqdn 0.4.6/0.4.7 yanked, required by deno_permissions 0.81.0. Solvable
by injecting the yanked entry into Cargo.lock manually but ugly.
3. rustls: deno_tls 0.198+ hard-pins `=0.23.28`, but aws-sdk-bedrockruntime
1.122.0 → aws-smithy-http-client 1.1.5 wants `^0.23.31`. Within-major
conflict, no resolver path. The unbeatable wall.
Goldilocks-set choice sidesteps (2) and (3) entirely:
- `deno_tls 0.196.0` was the last version before deno tightened
`rustls ^0.23.11` (range, accepts 0.23.31) to exact `=0.23.28`. With
^0.23.11, the resolver picks rustls 0.23.35 (latest 0.23 patch) which
satisfies both deno_tls's `>=0.23.11` and aws-sdk's `>=0.23.31`. Verified
empirically: lockfile has rustls 0.23.35 after this bump.
- `deno_permissions 0.68.0` (v2.4.0's pin) doesn't depend on fqdn at all.
The fqdn dep was added in a later deno_permissions release.
## Why deno_ast =0.51.0 specifically (not 0.48.0 from v2.4.0)
`swc_common 14.0.4` is the first patch that **drops the
`pub use serde::__private as serde;` line** in `src/private/mod.rs`. Older
14.0.x and all 0.37.5–13.x revisions still have it, and that line is
what was capping `serde = "=1.0.220"` (the workspace pin's "stuck because
of swc" comment). Empirically verified by inspecting the tarballs of
14.0.0 / 14.0.1 / 14.0.2 / 14.0.3 / 14.0.4:
14.0.0: has hack
14.0.1: has hack
14.0.2: has hack
14.0.3: has hack
14.0.4: NO HACK ← inflection point
`deno_ast 0.51.0` pins `swc_common =14.0.4` exactly — older deno_ast
versions pin earlier swc_common patches that still have the hack.
Notably, deno v2.4.0 itself pins `deno_ast =0.48.0` (swc_common 9.2.0,
still has hack) — we deliberately deviate from v2.4.0's deno_ast pin
to escape the swc serde wall, while keeping the rest of v2.4.0's pin
set for resolver compatibility with aws-sdk. deno_ast 0.51 was never
shipped in any deno release (v2.4.5 used 0.49, v2.5.0 jumped to 0.50,
v2.6.0 to 0.52), but it's published on crates.io and compatible with
v2.4.0's deno_core 0.352.
## What this unblocks
- PR #9106's `serde = "=1.0.224"` bump variant can rebase onto this
and resolve cleanly (MaterializeInc/rust-postgres' `postgres-types`
needs `serde_core ^1.0.221`, which is satisfied now that we're on
serde 1.0.228).
- Future deno_* / swc_* bumps no longer need to argue about the serde
ceiling — it's gone.
## What changes in source code
This commit is Cargo.toml + Cargo.lock only. Source changes that the
new deno_core / deno_fetch API requires live in the follow-up commits:
- `parsers/windmill-parser-{ts,ts-asset,wac}`: swc 0.37 → 14
(`code.into()` ambiguity fix at 5 sites)
- `windmill-runtime-nativets/build.rs` + `src/lib.rs`: deno_core 0.336
→ 0.352 API moves (`init_ops_and_esm()` → `init()`,
`FetchPermissions` / `NetPermissions` trait signature updates,
`deno_tls::Proxy` enum shape change)
A companion change in windmill-ee-private adjusts
`otel_tracing_proxy_ee.rs:521` for `deno_telemetry::init`'s second arg
becoming by-value (was `&OtelConfig`).
* fix(parsers): adapt to swc_common 14 BytesStr ambiguity
swc 0.37.5 → 14.0.4 changed `SourceMap::new_source_file`'s `src` argument
from `String` to `impl Into<BytesStr>`. With `BytesStr` available, the
existing call sites' `code.into()` on a `&str` becomes ambiguous between
`Into<Bytes>` (from the bytes crate) and `Into<BytesStr>` (from
bytes_str). Switch to `code.to_string()` to produce an owned `String`
that satisfies `From<String> for BytesStr` unambiguously.
Five call sites across three crates:
- windmill-parser-ts/src/lib.rs (3 sites)
- windmill-parser-ts-asset/src/lib.rs (1 site)
- windmill-parser-wac/src/typescript.rs (1 site)
* fix(nativets): adapt to deno_core 0.352 / deno_fetch 0.233 API changes
The goldilocks deno bump (deno_core 0.336 → 0.352, deno_fetch 0.214 →
0.233, etc.) ripples through nativets' build.rs and src/lib.rs.
Source-level changes required:
## 1. `extension!` macro: `init_ops_and_esm()` and `init_ops()` removed
deno_core 0.352's `extension!` macro now generates a single `init()`
function on the extension struct (full: ops + esm), plus `lazy_init()`
(ops only, with `needs_lazy_init = true` and a contract that the
caller invokes `JsRuntime::lazy_init_extensions` after construction).
- `build.rs` (snapshot creation, wants both ops and esm baked in):
`X::init_ops_and_esm(...)` → `X::init(...)`.
- `src/lib.rs:create_nativets_runtime` (runtime, was using `init_ops()`
because the snapshot already provides esm): also → `X::init(...)`.
deno_core's snapshot path skips esm re-execution when the snapshot
provides them, so the esm registration is a no-op at runtime. This
is how deno's own v2.4.0 runtime works.
Avoided `lazy_init` because it requires plumbing
`JsRuntime::lazy_init_extensions(ext_args_vec)` correctly across the
codebase, which is invasive for no behavioural benefit.
## 2. Local `fetch` extension now declared in both build.rs and lib.rs
deno_core 0.352 validates extension order between snapshot and runtime.
Our snapshot's last extension is the local `fetch` ext (which provides
ext:fetch/src/runtime.js). To avoid a runtime panic:
"Extensions from snapshot loaded in wrong order: expected fetch but got windmill"
…the runtime extension list now ends with `fetch::init()` matching the
snapshot order. The macro requires the same `esm` argument to type-check,
even though the ESM is not re-executed at runtime (it's in the snapshot).
## 3. `FetchPermissions` and `NetPermissions` trait shape
`deno_fetch::FetchPermissions` (deno_fetch 0.233.0) added new methods
and changed signatures:
- `check_read` / `check_write`: now take `path: Cow<'a, Path>` plus
a new `get_path: &'a dyn deno_fs::GetPath` parameter, and return
`Result<deno_fs::CheckedPath<'a>, FsError>` instead of `Result<Cow<Path>, FsError>`.
- New `check_write` (didn't exist) and `check_net_vsock` methods.
`deno_net::NetPermissions` (deno_net 0.201.0) gained `check_vsock`
and `check_write_path` now takes `Cow<'_, Path>`.
For `build.rs`'s `PermissionsContainer` (used only during snapshot
creation, where permissions are never actually checked): all methods
`unreachable!("snapshotting")`.
For `src/lib.rs`'s `PermissionsContainer` (used at runtime — the
nativets policy is "allow everything"): `check_read` / `check_write`
return `Ok(CheckedPath::Unresolved(path))`, `check_*_vsock` return
`Ok(())`. Smoke tests confirm fetch/net/url/web/blob/timers/structuredClone
behaviour is intact end-to-end.
## 4. `deno_tls::Proxy` is now an enum
`deno_tls::Proxy` was a struct, is now an enum with `Http`, `Https`,
`Socks5` variants. Our call site uses HTTP proxies — switched the
struct literal `deno_tls::Proxy { url, basic_auth }` to
`deno_tls::Proxy::Http { url, basic_auth }`.
## 5. New `deno_fs` direct workspace dep
`FetchPermissions` exposes `deno_fs::CheckedPath` and `deno_fs::GetPath`
as part of its public API. We can't avoid naming `deno_fs` directly any
more. Pinned to 0.119.0 (v2.4.0's matched version, transitively present
already through deno_fetch). Added to workspace `[dependencies]` plus
nativets's `[dependencies]` and `[build-dependencies]`.
## Validation
`cargo check --features enterprise,deno_core,duckdb,license,python,rust,scoped_cache,parquet,private,private_registry_test,csharp,php,ruby,mysql,quickjs,mcp,run_inline`
→ clean.
`cargo test -p windmill-runtime-nativets smoke -- --ignored --skip smoke_net_`
→ 8 passed; 0 failed (the full local smoke suite covering fetch,
setTimeout/Promise.all, URL/SearchParams, Blob/btoa/atob, large payload
roundtrip, error propagation, concurrent isolates, TS enum/union
transpile).
Network smoke tests (`smoke_net_fetch_example_com`,
`smoke_net_fetch_json_and_headers`) not run as part of the validation
gate but expected to pass — the change preserves deno_fetch behaviour
through the trait surface.
* chore: update ee-repo-ref to pick up deno_telemetry::init by-value fix
Points at windmill-ee-private branch deps/bump-deno-and-swc-goldilocks
which contains the companion otel_tracing_proxy_ee.rs adjustment for
deno_telemetry 0.12 → 0.31 (second arg of `init` is now by-value).
EE-only file, doesn't affect OSS build.
* chore(nix): bump rusty_v8 in flake.nix to 137.1.0 to match Cargo.toml
Cargo.toml's v8 pin moved from =130.0.7 to =137.1.0 as part of the
deno_core 0.336 → 0.352 bump, but I missed the comment directly above
the version pin:
# Exact version NOTE: Do not forget to update version and hash in flake.nix
flake.nix provides the prebuilt librusty_v8 binary that the v8 crate
links against. A version mismatch would either fail to fetch (if the
137.1.0 release didn't exist) or cause link-time symbol mismatches.
Nix is used by rust-client-check.yml and rust_on_release.yml in CI,
plus the dev shell — stale flake pin breaks all of those.
Updates x86_64-linux's sha256 to match the actual hash of
librusty_v8_release_x86_64-unknown-linux-gnu.a.gz at the 137.1.0 tag.
Other targets (aarch64-linux, x86_64-darwin, aarch64-darwin) remain
as lib.fakeHash — they were already placeholders in the previous
pin, so we don't regress on them.
Caught by both cubic and Pi reviewers on PR #9111.
* docs(nativets): clarify snapshot-prefix rule in extension-order comment
Claude reviewer caught that the doc comment claimed the runtime
extension list matches the snapshot's order — implying an exact match.
The truth is more permissive: deno_core 0.352 requires the snapshot's
extension list to be a *prefix* of the runtime's, not an exact match.
Runtime is allowed to append extra extensions (which we do — the
windmill `ext` carrying our ops is the last entry at runtime but absent
from the snapshot).
The code is correct as-is; only the comment wording was misleading.
Also fixes the same wording in PR description.
|
||
|
Ruben Fiszel
|
1e89aff2d6 |
test(nativets): add #[ignore]'d smoke suite for deno_core / deno_ast / swc bumps (#9108)
Adds ten `#[ignore]`'d integration tests in
backend/windmill-runtime-nativets/src/smoke_tests.rs exercising the JS
API surface that the existing nativets tests in tests/worker.rs don't
reach. Run them when bumping the deno_core / deno_ast / deno_runtime /
swc_* pins in backend/Cargo.toml, not on every CI:
cargo test -p windmill-runtime-nativets smoke -- --ignored
# skip network-dependent ones with `--skip smoke_net_`
Why here and not in tests/worker.rs:
windmill-runtime-nativets is the only consumer of the deno_core crate
in the workspace — ScriptLang::Deno spawns the external `deno` binary
via deno_executor.rs, while ScriptLang::Nativets is the only path that
loads the in-process V8 runtime. So a deno_core / deno_ast bump can
only break things downstream of this crate. Co-locating the smoke
tests with the runtime they exercise means they hit the right surface
directly, skip the entire job-queue / worker / API-server stack, and
run in <1s end-to-end (vs. ~30-60s per test for the worker-level
nativets tests).
The tests use the existing `PrewarmedIsolate::spawn` API (already
public for the dedicated-worker path), which gives a clean
"compile-TS → load module → execute main(args) → return JSON" entry
point with no DB or queue plumbing required.
Coverage:
- smoke_basic_value_passing — args binding + return marshaling
- smoke_transpile_enum_and_union — TS-specific syntax (enums,
discriminated unions, casts) through swc_ecma_parser / swc_ecma_ast
- smoke_set_timeout_and_promise_all — deno_web timer ops + V8
microtask drain order
- smoke_url_and_searchparams — deno_url surface
- smoke_web_blob_btoa_atob — deno_web Blob + base64 ops
- smoke_large_payload_roundtrip — 512 KB string in/out through the
op-table boundary
- smoke_error_propagation_with_message — thrown Error must surface
in PrewarmedResult::Err with original message
- smoke_concurrent_isolates — 8 isolates spawned in parallel from
the same tokio runtime; catches V8 isolate-setup races
- smoke_net_fetch_example_com — deno_fetch end-to-end against
example.com
- smoke_net_fetch_json_and_headers — deno_fetch with custom request
headers + Response.json() against httpbin.org/anything
`structuredClone` is not currently wired into the nativets global —
documented in the smoke_web_blob_btoa_atob test in case that ever
changes.
All ten tests pass locally against the current pinned versions
(deno_core 0.336.0 / deno_ast =0.44.0 / swc_common =0.37.5).
|
||
|
Ruben FiszelandClaude Opus 4.7
|
e732004180 |
fix(nativets): forward OTEL-prefixed console logs to tracing events (#8937)
* fix(nativets): forward OTEL-prefixed console logs to tracing events
Nativets jobs run in-process and bypass the handle_child.rs stdout loop
where `OTEL: ` lines are turned into `tracing::event!` calls when
`OTEL_JOB_LOGS=true`. Apply the same prefix handling in the nativets
log receiver so `console.log("OTEL: ...")` reaches the OTEL exporter
like it does for other runtimes.
Moves `OTEL_JOB_LOGS` and `OTEL_PREFIX` into windmill-common so both
crates share the same definition.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* fix(nativets): emit job_log tracing target so logs reach OTEL bridge
For non-native runtimes, `lines_to_stream` → `process_streaming_log_lines`
(EE) emits every stdout line as `tracing::info!(target: "windmill:job_log", ...)`,
which is picked up by the EE `LogContextBridge` and exported to OTEL
(the bridge's filter is `EnvFilter` only, not the targets filter that
drops `windmill:job_log` from stdout/file sinks).
Nativets delivers logs in-process via a channel, so it never goes
through that path and console.log output only reached the Windmill UI.
Emit the same `windmill:job_log` event per line from the nativets log
receiver.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
424ca59dfe |
feat: make WINDMILL_DIR configurable via environment variable (#8215)
* fix: auto-heal corrupted python runtime cache on remote workers
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
* Revert "fix: auto-heal corrupted python runtime cache on remote workers"
This reverts commit
|
||
|
hugocasaandwindmill-internal-app[bot]
|
37c9acb232 |
feat: dedicated nativets (#8021)
* feat: dedicated nativets * review nits * prewarm isolates * ref * chore: update ee-repo-ref to 5f8105b808f3f0186fdf5132d2ee602d8a14aa17 This commit updates the EE repository reference after PR #424 was merged in windmill-ee-private. Previous ee-repo-ref: b7906acabb8ce359230bbd3e30dbb3bba4c42adb New ee-repo-ref: 5f8105b808f3f0186fdf5132d2ee602d8a14aa17 Automated by sync-ee-ref workflow. --------- Co-authored-by: windmill-internal-app[bot] <windmill-internal-app[bot]@users.noreply.github.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
aa9f3da429 |
fix: make V8 runtime init idempotent and auto-initialize before isolate creation
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
90d010347c |
fix: use unprotected V8 platform to prevent SIGSEGV on x86_64 Linux
The default V8 platform uses Memory Protection Keys (pkeys) which require all V8-using threads to be descendants of the thread that called v8::Initialize. Tokio's spawn_blocking pool threads don't satisfy this, causing SIGSEGV in WasmCodePointerTable during isolate creation on x86_64 Linux. Switch to new_unprotected_default_platform which relaxes the pkey requirement. Also remove --single-threaded V8 flag (was degrading performance without fixing the issue) and scope the creation mutex to just JsRuntime::new() instead of the entire lifecycle. See: https://github.com/denoland/deno_core/issues/952 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
05106d7dee |
fix: prevent V8 SIGSEGV by serializing isolate creation and fixing use-after-free
Work around a V8 bug in WasmCodePointerTable::AllocateUninitializedEntry() that causes SIGSEGV when multiple isolates are created concurrently on x86_64 Linux (https://github.com/denoland/deno_core/issues/952). - Add V8_ISOLATE_CREATE_LOCK mutex to serialize JsRuntime::new() calls - Replace oneshot channel with Arc<Mutex<Option<IsolateHandle>>> shared between spawn_blocking and an IsolateDropGuard for proper cancellation - Remove terminate_execution() call on dead isolate handle in error path (was use-after-free: handle dereferenced after JsRuntime already dropped) - Clear handle before drop(js_runtime) to prevent guard from terminating a destroyed isolate Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> |
||
|
Ruben FiszelandClaude Opus 4.6
|
3f68e3a58e |
refactor: isolate deno_core into windmill-runtime-nativets subcrate (#7848)
* refactor: isolate deno_core into windmill-runtime-nativets subcrate Remove deno_core from flow eval and isolate nativets V8 runtime into a dedicated subcrate so deno_core compilation no longer blocks windmill-worker or windmill-api. - Create windmill-jseval crate: QuickJS-based JS eval for flow expressions and batch rerun, extracted from windmill-worker - Create windmill-runtime-nativets crate: all deno_core/V8 deps and nativets script execution, with build.rs snapshot generation - Simplify windmill-worker: remove all deno_* direct deps, empty build.rs, gate nativets behind optional dep - Update windmill-api: use windmill-jseval for batch rerun instead of deno_core, remove deno_core feature entirely - Add nativets integration tests (nativets_jobs.rs) and parallel stress test (nativets_stress.rs, 8 workers x 200 jobs) - Remove dead code: deno flow eval path, USE_QUICKJS env var, parity tests (replaced with 63 standalone expected-value tests) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: address PR review feedback for deno_core isolation - Deduplicate unsafe_raw() into windmill-common/src/utils.rs (single source) - Delete orphaned runtime.js and windmill-client.js from windmill-worker/src/ - Fix operator precedence in windmill-jseval with explicit parentheses - Remove unnecessary return keyword in heap limit callback - Remove redundant as usize casts - Remove ~150 lines of commented-out code from runtime.js - Remove commented-out #[cfg] in build.rs Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * otel ee --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com> |