mirror of
https://github.com/windmill-labs/windmill.git
synced 2026-09-21 08:02:38 +00:00
* feat(nativets): bound fetch on a peer that never answers
deno_fetch applies no deadline of any kind. A peer that completes the TCP
handshake, accepts the request and then goes silent leaves `await fetch(...)`
pending indefinitely, holding its worker slot until the *job* timeout -- which
on self-hosted defaults to DEFAULT_SELFHOSTED_TIMEOUT, i.e. 7 days.
Nothing else catches this. Zombie-job detection keys off a stale
v2_job_runtime.ping, and a worker blocked inside a pending fetch keeps pinging
normally throughout: the worker is alive and healthy, only the work is dead.
What this bounds is the wait for a response to begin, and it stops there:
- a peer that never answers -> rejected after N seconds
- a peer slow to answer, but under N -> unaffected
- a body that then streams for an hour,
or is read slowly by the caller -> unaffected, always
That last line rules out the obvious implementation: AbortSignal.timeout(N)
around every fetch would bound the hang and break every streaming response and
long download. This is a hang detector, not a latency budget.
Default 300s via WINDMILL_FETCH_RESPONSE_TIMEOUT_SECS (0 disables), with a
per-script `//fetch_response_timeout <seconds>` annotation alongside the
existing //useragent and //proxy. Both nativets paths inherit it, since
eval_fetch_timeout and the dedicated-worker path in bun_executor both funnel
through create_nativets_runtime.
The ms value is clamped to i32::MAX: deno_web's setTimeout runs its delay
through webidl.converters.long, a 32-bit conversion that *wraps*, so a setting
past ~24.8 days would come out negative and fire immediately -- turning an
over-generous timeout into an instant one on every fetch.
The window covers connect, TLS and request upload as well as server think
time, so a very slow large upload is bounded by it too; the error message says
so rather than claiming the connection went silent.
Not covered: a body that stalls midway. Reaching that needs the response's
InnerBody, which deno_fetch keeps module-private, and every way to wrap it
from outside changes observable Response semantics (locking, bodyUsed,
double-consume errors). Left for a follow-up in deno_fetch itself.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* test(nativets): cover the instance-wide response-timeout env var
A typo in WINDMILL_FETCH_RESPONSE_TIMEOUT_SECS would compile, pass every
other test, and silently hand every operator the 300s default -- the same
class of silent-default failure the timeout itself exists to prevent. Its
own test binary, since a LazyLock resolves the value once per process.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): inherit the caller's RequestInit, and clear the long-poll ceiling
Two problems with the first cut, both found in review.
`{ ...init, signal }` copied only own enumerable properties, but RequestInit is
a WebIDL dictionary whose members deno_fetch reads with plain property gets
that walk the prototype chain. Anything inherited or non-enumerable was
dropped: `fetch(url, Object.create({method: "POST"}))` silently became a GET.
Worse, a non-object init went from a loud TypeError to a silent GET, because
spreading "POST" yields {0:"P",1:"O",...} -- a valid dictionary with ignored
keys. Now the init is inherited from rather than copied, and a non-dictionary
is handed straight back to deno_fetch for its own TypeError.
The 300s default also sat at half of TIMEOUT_WAIT_RESULT (600s), which
run_wait_result long-polls against with no response headers. A script running
another job synchronously for 300-600s would have timed out client-side while
the server was still legitimately holding the request open -- the long-poll
risk class, instantiated inside the product and reachable without writing a
raw fetch. Default raised to 900s, with the constraint recorded where someone
would break it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): hand the caller's RequestInit to Request untouched
Carrying a WebIDL dictionary across by hand has no safe form, and both
previous attempts were wrong in opposite directions. Spreading a copy drops
inherited and non-enumerable members, and turns a non-object init from a loud
TypeError into a silent GET. Inheriting from it via Object.create fixes those
but makes the child the receiver, so an accessor on the original runs against
an object that lacks its private-field brand:
Cannot read private member #body from an object whose class did not
declare it
So don't carry it at all. fetch()'s own first act is `new Request(input,
init)`; doing that here hands the init to the same constructor, read exactly
as it would be without this wrapper, and our signal travels in an init we own.
`req.signal` is then deno's own resolution of init.signal over an input
Request's signal, which removes the hand-rolled version of that rule too.
The Request is built twice as a result, once here and once inside fetch. That
is cheap: cloneInnerRequest carries method, headers, redirect mode, clientRid
and blob entry, and a body is proxied rather than buffered -- a static body is
a shallow {body, consumed} copy sharing its bytes, a stream gets a one-chunk
pass-through.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): keep an aborted fetch settling in the same tick
deno_fetch keeps its outer fetch non-async on purpose: "WPT has a test that
aborted fetch is settled in the same tick. This means we cannot wrap the
promise if it is already settled" (26_fetch.js). An `async` wrapper adopts
that promise through another one, so a rejection that used to land before any
microtask queued after the call now lands after it.
Made the wrapper non-async, with an early return that hands deno's settled
rejection straight back for an already-aborted signal, and no timer armed
there since there is no response to wait for. Construction still has to reject
rather than throw, so it is caught and returned as a rejection, which is what
the `async` was buying.
The comment claiming this matched deno_fetch's own `async function fetch` was
wrong on two counts -- that function is not async, and the wrapper was not
matching it. Replaced with the constraint that actually holds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* fix(nativets): keep fetch's observable shape and reach intrinsics safely
Three ways the wrapper was distinguishable from the fetch it replaces, all
observable from a script sharing the isolate.
`.then` was an ordinary property lookup, so `Promise.prototype.then =
undefined` broke fetch after the request had already gone out. deno's own
modules reach intrinsics through primordials, and this file already captured
setTimeout, clearTimeout and Promise.reject for exactly that reason, so the
lookup was the odd one out. Now captured alongside them.
Declaring `init` without a default made `fetch.length` 2 where the standard
says 1. And the empty-call branch forwarded two explicit `undefined`s, so
deno's required-argument check saw two arguments and raised "Invalid URL:
'undefined'" instead of "1 argument required". Forwarding through
ReflectApply preserves the count.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01JshNT5XVH78ZsTMvDWfFHb
* docs: clarify fetch timeout restart requirements
* fix: capture native fetch abort helpers
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
269 lines
12 KiB
JavaScript
269 lines
12 KiB
JavaScript
import * as abortSignal from "ext:deno_web/03_abort_signal.js";
|
|
import * as domException from "ext:deno_web/01_dom_exception.js";
|
|
import * as base64 from "ext:deno_web/05_base64.js";
|
|
import * as console from "ext:deno_console/01_console.js";
|
|
import * as encoding from "ext:deno_web/08_text_encoding.js";
|
|
import * as event from "ext:deno_web/02_event.js";
|
|
import * as fetch from "ext:deno_fetch/26_fetch.js";
|
|
import * as file from "ext:deno_web/09_file.js";
|
|
import * as fileReader from "ext:deno_web/10_filereader.js";
|
|
import * as formData from "ext:deno_fetch/21_formdata.js";
|
|
import * as headers from "ext:deno_fetch/20_headers.js";
|
|
import * as streams from "ext:deno_web/06_streams.js";
|
|
import * as timers from "ext:deno_web/02_timers.js";
|
|
import * as url from "ext:deno_url/00_url.js";
|
|
import * as net from "ext:deno_net/01_net.js";
|
|
import * as tls from "ext:deno_net/02_tls.js";
|
|
import * as urlPattern from "ext:deno_url/01_urlpattern.js";
|
|
import * as webidl from "ext:deno_webidl/00_webidl.js";
|
|
import * as crypto from "ext:deno_crypto/00_crypto.js";
|
|
import * as response from "ext:deno_fetch/23_response.js";
|
|
import * as request from "ext:deno_fetch/23_request.js";
|
|
import "ext:deno_web/02_structured_clone.js";
|
|
import * as globalInterfaces from "ext:deno_web/04_global_interfaces.js";
|
|
// Namespace imports (not side-effect-only) so their constructors are reachable
|
|
// for the globalThis wiring below. The module bodies still execute on
|
|
// evaluation, so their side effects apply.
|
|
import * as messagePort from "ext:deno_web/13_message_port.js";
|
|
import * as compression from "ext:deno_web/14_compression.js";
|
|
import * as performance from "ext:deno_web/15_performance.js";
|
|
import "ext:deno_web/16_image_data.js";
|
|
import "ext:deno_fetch/27_eventsource.js";
|
|
|
|
// deno_fetch applies no deadline, so a peer that accepts a request and then
|
|
// never answers leaves `await fetch(...)` pending until the job timeout, which
|
|
// self-hosted defaults to 7 days.
|
|
const ORIGINAL_FETCH = fetch.fetch;
|
|
|
|
// deno_web's timers reject any `this` other than undefined/globalThis, so
|
|
// `timers.setTimeout(...)` passes the module namespace and throws "Illegal
|
|
// invocation".
|
|
const setTimeoutUnbound = timers.setTimeout;
|
|
const clearTimeoutUnbound = timers.clearTimeout;
|
|
// Captured before user code shares the isolate and could redefine them, the
|
|
// way deno's own modules reach intrinsics through primordials.
|
|
const PromiseReject = Promise.reject.bind(Promise);
|
|
const promiseThen = Function.prototype.call.bind(Promise.prototype.then);
|
|
const abortSignalAny = abortSignal.AbortSignal.any.bind(abortSignal.AbortSignal);
|
|
const abortControllerAbort = Function.prototype.call.bind(
|
|
abortSignal.AbortController.prototype.abort,
|
|
);
|
|
const ReflectApply = Reflect.apply;
|
|
|
|
// Installed per isolate by __wmInitPerIsolate; 0 disables. Only a backstop
|
|
// for the impossible case of fetch running before that init.
|
|
let fetchResponseTimeoutMs = 900_000;
|
|
|
|
function fetchResponseTimeoutError(requestUrl, timeoutMs) {
|
|
let target;
|
|
try {
|
|
const parsed = new url.URL(requestUrl);
|
|
// Query and fragment routinely carry tokens, and this reaches a job log.
|
|
target = parsed.origin + parsed.pathname;
|
|
} catch {
|
|
target = "the request target";
|
|
}
|
|
return new domException.DOMException(
|
|
`fetch to ${target} timed out: no response headers arrived within ` +
|
|
`${Math.round(timeoutMs / 1000)}s (this covers connect, request upload ` +
|
|
`and the wait for the server to start replying; once a response begins ` +
|
|
`it is never interrupted). Change it per script with ` +
|
|
`"//fetch_response_timeout <seconds>" (0 disables), or instance-wide ` +
|
|
`with WINDMILL_FETCH_RESPONSE_TIMEOUT_SECS.`,
|
|
"TimeoutError",
|
|
);
|
|
}
|
|
|
|
globalThis.atob = base64.atob;
|
|
globalThis.btoa = base64.btoa;
|
|
// Not `async`, for the same reason deno_fetch's own outer fetch isn't: WPT
|
|
// pins that an aborted fetch settles in the same tick, which adopting its
|
|
// promise through another one would break. Construction still has to reject
|
|
// rather than throw, so it is caught and handed back as a rejection.
|
|
globalThis.fetch = function fetch(input, init = undefined) {
|
|
const timeoutMs = fetchResponseTimeoutMs;
|
|
// Forwarded with the original argument count, so deno still sees an empty
|
|
// call as empty and raises its own "1 argument required". The default on
|
|
// `init` is what keeps `fetch.length` at 1, as the standard has it.
|
|
if (!(timeoutMs > 0) || arguments.length < 1) {
|
|
return ReflectApply(ORIGINAL_FETCH, undefined, arguments);
|
|
}
|
|
|
|
let req;
|
|
let controller;
|
|
let signal;
|
|
try {
|
|
// RequestInit is a WebIDL dictionary: copying it drops inherited and
|
|
// non-enumerable members, and inheriting from it runs accessors against the
|
|
// wrong receiver. Hand it to the same Request constructor fetch() would,
|
|
// and carry our own signal in an init of our own.
|
|
req = new request.Request(input, init);
|
|
|
|
// `req.signal` is deno's own resolution of init.signal over an input
|
|
// Request's signal, so combining with it preserves the caller's abort and
|
|
// reason while ours only adds a ceiling.
|
|
controller = new abortSignal.AbortController();
|
|
signal = abortSignalAny([req.signal, controller.signal]);
|
|
} catch (e) {
|
|
return PromiseReject(e);
|
|
}
|
|
|
|
// Already aborted: hand back deno's own settled rejection untouched, and arm
|
|
// nothing -- there is no response to wait for.
|
|
if (signal.aborted) {
|
|
return ORIGINAL_FETCH(req, { signal });
|
|
}
|
|
|
|
let timer = setTimeoutUnbound(() => {
|
|
timer = undefined;
|
|
abortControllerAbort(controller, fetchResponseTimeoutError(req.url, timeoutMs));
|
|
}, timeoutMs);
|
|
// Disarmed on headers, never on body completion: a response that has begun
|
|
// arriving must be free to stream for as long as it needs.
|
|
const disarm = () => {
|
|
if (timer !== undefined) {
|
|
clearTimeoutUnbound(timer);
|
|
timer = undefined;
|
|
}
|
|
};
|
|
|
|
return promiseThen(
|
|
ORIGINAL_FETCH(req, { signal }),
|
|
(res) => {
|
|
disarm();
|
|
return res;
|
|
},
|
|
(e) => {
|
|
disarm();
|
|
throw e;
|
|
},
|
|
);
|
|
};
|
|
globalThis.Request = request.Request;
|
|
globalThis.Response = response.Response;
|
|
globalThis.Blob = file.Blob;
|
|
globalThis.URL = url.URL;
|
|
globalThis.FormData = formData.FormData;
|
|
globalThis.URLSearchParams = url.URLSearchParams;
|
|
globalThis.Headers = headers.Headers;
|
|
globalThis.FileReader = fileReader.FileReader;
|
|
globalThis.console = new console.Console((msg, level) =>
|
|
globalThis.Deno.core.ops.op_log(msg)
|
|
);
|
|
globalThis.AbortController = abortSignal.AbortController;
|
|
globalThis.AbortSignal = abortSignal.AbortSignal;
|
|
globalThis.crypto = crypto.crypto;
|
|
globalThis.Crypto = crypto.Crypto;
|
|
globalThis.CryptoKey = crypto.CryptoKey;
|
|
globalThis.SubtleCrypto = crypto.SubtleCrypto;
|
|
|
|
Object.assign(globalThis, {
|
|
clearInterval: timers.clearInterval,
|
|
clearTimeout: timers.clearTimeout,
|
|
setInterval: timers.setInterval,
|
|
setTimeout: timers.setTimeout,
|
|
});
|
|
|
|
// Standard web-platform globals from the deno_web / deno_url extensions,
|
|
// exposed to match the bun runner's global surface. Every name below is present
|
|
// in bun; names bun lacks (EventSource, ImageData) are deliberately excluded.
|
|
Object.assign(globalThis, {
|
|
// DOMException. Beyond bun parity, deno_web modules reference it as a global:
|
|
// AbortController.abort() with no reason constructs `new DOMException(...)`, so
|
|
// without this the already-wired AbortController/AbortSignal throw on abort.
|
|
DOMException: domException.DOMException,
|
|
// Text encoding + encoding streams.
|
|
TextEncoder: encoding.TextEncoder,
|
|
TextDecoder: encoding.TextDecoder,
|
|
TextEncoderStream: encoding.TextEncoderStream,
|
|
TextDecoderStream: encoding.TextDecoderStream,
|
|
// File (Blob is already wired above).
|
|
File: file.File,
|
|
// Events (AbortSignal, already wired, extends EventTarget). MessageEvent is
|
|
// the companion to MessagePort/MessageChannel below. Only the event types
|
|
// bun exposes are wired (ProgressEvent / PromiseRejectionEvent are not).
|
|
// reportError works because __wmInitPerIsolate makes globalThis an EventTarget.
|
|
Event: event.Event,
|
|
EventTarget: event.EventTarget,
|
|
CustomEvent: event.CustomEvent,
|
|
MessageEvent: event.MessageEvent,
|
|
CloseEvent: event.CloseEvent,
|
|
ErrorEvent: event.ErrorEvent,
|
|
reportError: event.reportError,
|
|
// Streams + queuing strategies + the reader/controller constructors bun also
|
|
// exposes as globals (used for `x instanceof ReadableStreamDefaultReader` etc.;
|
|
// the controllers throw on direct construction, matching the spec).
|
|
ReadableStream: streams.ReadableStream,
|
|
ReadableStreamDefaultReader: streams.ReadableStreamDefaultReader,
|
|
ReadableStreamBYOBReader: streams.ReadableStreamBYOBReader,
|
|
ReadableStreamDefaultController: streams.ReadableStreamDefaultController,
|
|
ReadableByteStreamController: streams.ReadableByteStreamController,
|
|
ReadableStreamBYOBRequest: streams.ReadableStreamBYOBRequest,
|
|
WritableStream: streams.WritableStream,
|
|
WritableStreamDefaultWriter: streams.WritableStreamDefaultWriter,
|
|
WritableStreamDefaultController: streams.WritableStreamDefaultController,
|
|
TransformStream: streams.TransformStream,
|
|
TransformStreamDefaultController: streams.TransformStreamDefaultController,
|
|
ByteLengthQueuingStrategy: streams.ByteLengthQueuingStrategy,
|
|
CountQueuingStrategy: streams.CountQueuingStrategy,
|
|
// URL pattern matching.
|
|
URLPattern: urlPattern.URLPattern,
|
|
// Compression streams.
|
|
CompressionStream: compression.CompressionStream,
|
|
DecompressionStream: compression.DecompressionStream,
|
|
// Message channel / port.
|
|
MessageChannel: messagePort.MessageChannel,
|
|
MessagePort: messagePort.MessagePort,
|
|
// High-resolution timing: the `performance` singleton and its constructor
|
|
// globals (bun exposes all of these; PerformanceObserver is not in deno_web).
|
|
performance: performance.performance,
|
|
Performance: performance.Performance,
|
|
PerformanceEntry: performance.PerformanceEntry,
|
|
PerformanceMark: performance.PerformanceMark,
|
|
PerformanceMeasure: performance.PerformanceMeasure,
|
|
// Spec structuredClone (validates args + honors the options bag), from the
|
|
// message-port module rather than the single-arg internal helper in
|
|
// 02_structured_clone.js.
|
|
structuredClone: messagePort.structuredClone,
|
|
});
|
|
|
|
// Per-isolate init, invoked from Rust after the snapshot is restored (this
|
|
// module body runs at snapshot-build time, not per isolate).
|
|
globalThis.__wmInitPerIsolate = (config) => {
|
|
if (config != null && typeof config.fetchResponseTimeoutMs === "number") {
|
|
fetchResponseTimeoutMs = config.fetchResponseTimeoutMs;
|
|
}
|
|
|
|
// setTimeOrigin() seeds performance.timeOrigin from the isolate's wall clock;
|
|
// without it timeOrigin is undefined and `timeOrigin + performance.now()` is NaN.
|
|
performance.setTimeOrigin();
|
|
|
|
// Make globalThis a functional EventTarget, as Deno's bootstrap does. deno_web
|
|
// routes uncaught EventTarget-listener errors and reportError through
|
|
// reportException, which dispatches an error event on the saved global
|
|
// reference; reportError also requires its receiver to equal that reference.
|
|
// Both need the reference to be globalThis and globalThis to be an EventTarget,
|
|
// otherwise dispatch throws a masking error and globalThis.reportError() throws
|
|
// "Illegal invocation". Set up per isolate so the reference is the live global.
|
|
// The prototype + brand are what webidl.assertBranded checks in the methods.
|
|
Object.setPrototypeOf(
|
|
globalThis,
|
|
globalInterfaces.DedicatedWorkerGlobalScope.prototype,
|
|
);
|
|
event.setEventTargetData(globalThis);
|
|
globalThis[webidl.brand] = webidl.brand;
|
|
event.saveGlobalThisReference(globalThis);
|
|
};
|
|
|
|
// Expose bootstrapOtel globally so it can be called from Rust after runtime creation.
|
|
// We use dynamic import so deno_telemetry isn't loaded during snapshot creation.
|
|
// Config: [tracingEnabled, metricsEnabled, consoleConfig, deterministic]
|
|
// consoleConfig: 0=ignore, 1=capture, 2=replace
|
|
globalThis.__bootstrapOtel = () => {
|
|
import("ext:deno_telemetry/telemetry.ts").then(({ bootstrap, enterSpan }) => {
|
|
bootstrap([1, 0, 1, 0]);
|
|
// Expose enterSpan for setting parent trace context
|
|
globalThis.__enterSpan = enterSpan;
|
|
});
|
|
};
|