Files
orca/config/build-plugins/plain-node-entry-guard.ts
T
OrcaWinandOrcaWin 4ec6bbf588 Kill hung WSL transcript filesystem operations via child process with route quarantine (#15381)
* fix(native-chat): kill hung WSL operations via child process

Stalled UNC file operations hold libuv permits even after the gate
timeout expires, blocking Chat tab recovery. Two stalled operations
fill both permits and freeze all WSL access until restart.

Fork file I/O for UNC paths into a separate child process. On deadline
expiry, kill the process to force the hung syscall to exit. This frees
the permit for the affected tab's next read. Temporarily quarantine the
stalled route to avoid retry storms.

* chore: drop internal review artifact from the repo root

* fix(native-chat): harden the WSL transcript fs sidecar

Review follow-ups on the sidecar isolation change:

- Only the deadline may abort running gate work. The sole waiter's
  same-duration timeout fired first, killed healthy children on caller
  abandonment, and settled the task before the deadline could quarantine
  a stalled route - leaving the back-off dead for every dedupe:false op.
- Resolve the fork entry from out/main/chunks too: the resolver compiles
  into a shared chunk, and the scanner service child has no
  process.resourcesPath, so packaged WSL vault scans threw entry-not-found
  (masked as an empty tree).
- Allowlist the fork env instead of spreading process.env; ambient
  NODE_OPTIONS would halt or --require code into every child.
- Wrap transport faults (spawn failure, child death) in
  WslTranscriptFsError('unavailable') so discovery reports them as scan
  issues instead of misreading them as missing paths or empty trees.
- Gate the vitest in-process fallback on the vitest worker global so a
  leaked VITEST=true cannot revert production to in-process UNC syscalls.
- Reap idle sidecar processes after 60s instead of holding them for the
  app session.
- Split 'open' into its own protocol union member so the reusable-call
  Exclude actually strips it from the pooled-process API.
- Guard kill('SIGKILL') against the teardown race where an exiting child
  emits an unlistened 'error', and dispatch reads by handle kind before
  path spelling.

* fix(native-chat): probe stalled WSL routes instead of a fixed quarantine

Remaining review follow-ups:

- Escalating route quarantine: first strike lifts after 5s so a distro
  that was cold-booting when its op hit the deadline recovers on the
  next poll (~35s total instead of ~90s); repeat stalls double the
  back-off toward the prior 2x-timeout cap, and any settle the deadline
  did not force clears the strikes. Queued same-route tasks fail fast
  at quarantine instead of stranding one waiter deadline per file in
  sequential scans.
- Single request implementation: the vitest in-process fallback now runs
  the child's own dispatcher (WslTranscriptFsProcessOperations + decode),
  so unit suites exercise exactly what the forked process executes and
  the per-call-site fallback closures are gone. Dirent fixtures gained
  the full kind-flag set the serializer reads.
- Dropped the production-dead per-route close queue; UNC FileHandles
  (test fallback only) mirror the process-handle close contract.
- Error class, messages, and factories move to wsl-transcript-fs-error
  (re-exported from the gate) to keep the gate under the lines budget.

* fix(native-chat): harden WSL transcript fs with route quarantine strike

Extract quarantine logic into a dedicated module with strike decay: stalls older
than 5 minutes restart from base back-off, and concurrent-lane timeouts count as
one incident. Allow joining live in-flight tasks on quarantined routes (they cost
no new I/O). Preserve quarantine across transport faults (child death). Handle
file shrinking during tail reads by detecting short reads and returning empty.
Defer file closes that arrive mid-read instead of refusing, preventing slot
leaks. Separate process slot and boundary-finding concerns into focused modules.

* fix(native-chat): enforce route quarantine windows and isolate lanes per

A late result arriving after the deadline was incorrectly lifting the route
quarantine, allowing subsequent work to start before the back-off period
expired. Now late results are correctly recognized as stale and never cut
the quarantine short.

Process work is now isolated per (route, priority) lane so a scan stall
cannot block exact reads on the same distro. Each lane gets its own client
and process pool; late results and handle faults stay scoped to their lane.

Tests now fake performance.now() alongside timers (the quarantine clock
depends on it) and wait for the full back-off window to expire rather than
advancing by 0. Gate state is reset between test cases since late releases
never lift the quarantine.

---------

Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
2026-08-18 16:57:35 -07:00

282 lines
10 KiB
TypeScript

import { spawn } from 'node:child_process'
import { join } from 'node:path'
import type { Plugin, Rollup } from 'vite'
type NormalizedInputOptions = Rollup.NormalizedInputOptions
type NormalizedOutputOptions = Rollup.NormalizedOutputOptions
type OutputBundle = Rollup.OutputBundle
type OutputChunk = Rollup.OutputChunk
// Why: v1.4.129-rc.1 shipped a dead terminal daemon because a shared main
// chunk gained `require("electron")` (an import edge added in #7642), and the
// daemon is forked as a plain-Node process where electron cannot be required.
// Nothing in CI executes the built daemon-entry under plain Node, so the leak
// stayed invisible until an adopted old daemon died. This guard fails the
// build when any chunk reachable from a plain-Node fork entry requires
// electron, and smoke-loads daemon-entry under plain Node to prove its module
// graph still resolves.
// Entries executed as plain Node (ELECTRON_RUN_AS_NODE / no electron runtime):
// forked daemon, parcel-watcher, WSL filesystem and computer sidecars, and the CLI-run
// agent-hooks entry. require("electron") throws MODULE_NOT_FOUND in all of them.
const PLAIN_NODE_ENTRY_NAMES = [
'daemon-entry',
'parcel-watcher-process-entry',
'computer-sidecar',
'wsl-transcript-fs-process-entry',
'agent-hooks/managed-agent-hook-controls',
'codex/codex-app-server-grant-entry'
] as const
// Entries executed as worker threads of the main process. Electron's module is
// not registered on worker threads, so require("electron") throws
// "Cannot find module 'electron'" there too (verified on Electron 43) and kills
// the worker at startup. These carry hand-written "must stay electron-free"
// comments, which is convention, not enforcement — and the port-scan worker in
// particular sits one import away from a client module that deliberately does
// require electron.
const WORKER_THREAD_ENTRY_NAMES = [
'stt-worker',
'warp-theme-parser-worker',
'session-scanner-opencode-sqlite-worker-entry',
'session-scanner-worker-entry',
'main-thread-hang-watchdog-entry',
'port-scan-command-worker-entry'
] as const
export const GUARDED_ENTRY_NAMES = [
...PLAIN_NODE_ENTRY_NAMES,
...WORKER_THREAD_ENTRY_NAMES
] as const
type EntryRuntime = 'plain-Node process' | 'worker thread'
// Subpaths (electron/main) are as unloadable as the bare module under plain Node.
const ELECTRON_REQUIRE_RE = /require\(\s*["'`]electron(?:\/[^"'`]+)?["'`]\s*\)/
// Why: writeBundle skips any name missing from the bundle, so a renamed or
// removed rollup input would silently drop that entry from the guard and let the
// regression back in. Pin the lists to the input keys at build start instead.
function assertEntryNamesAreRollupInputs(input: NormalizedInputOptions['input']): void {
if (typeof input === 'string' || Array.isArray(input)) {
return
}
const inputNames = new Set(Object.keys(input))
const missing = GUARDED_ENTRY_NAMES.filter((name) => !inputNames.has(name))
if (missing.length > 0) {
throw new Error(
`[plain-node-entry-guard] guarded ${missing.map((name) => `"${name}"`).join(', ')} ` +
`${missing.length === 1 ? 'is not a rollup input' : 'are not rollup inputs'} anymore. ` +
`Update PLAIN_NODE_ENTRY_NAMES/WORKER_THREAD_ENTRY_NAMES in plain-node-entry-guard.ts to ` +
`the current entry names — a stale name silently stops guarding that entry.`
)
}
}
function collectReachableChunks(
entry: OutputChunk,
byFileName: Map<string, OutputChunk>
): OutputChunk[] {
const seen = new Set<string>()
const reachable: OutputChunk[] = []
const stack = [entry.fileName]
while (stack.length > 0) {
const fileName = stack.pop() as string
if (seen.has(fileName)) {
continue
}
seen.add(fileName)
const chunk = byFileName.get(fileName)
if (!chunk) {
continue
}
reachable.push(chunk)
for (const imported of [...chunk.imports, ...chunk.dynamicImports]) {
stack.push(imported)
}
}
return reachable
}
function assertNoElectronRequire(
entryName: string,
entry: OutputChunk,
byFileName: Map<string, OutputChunk>,
runtime: EntryRuntime = 'plain-Node process'
): void {
for (const chunk of collectReachableChunks(entry, byFileName)) {
if (ELECTRON_REQUIRE_RE.test(chunk.code)) {
throw new Error(
`[plain-node-entry-guard] "${entryName}" reaches chunk "${chunk.fileName}" that ` +
`requires electron. "${entryName}" runs as a ${runtime}, where ` +
`require("electron") throws MODULE_NOT_FOUND and kills it at startup (the ` +
`v1.4.129-rc.1 daemon outage). Keep electron imports out of its module graph.`
)
}
}
}
// Owned by the argv parser in src/main/daemon/daemon-entry.ts — keep in sync.
const DAEMON_USAGE_PREFIX = 'Usage: daemon-entry'
export type SmokeTimings = {
timeoutMs: number
// daemon-entry traps SIGTERM and awaits a native shutdown, so the deadline
// needs an uncatchable follow-up to stay a deadline.
killGraceMs: number
}
const DEFAULT_SMOKE_TIMINGS: SmokeTimings = { timeoutMs: 15_000, killGraceMs: 2_000 }
// Bound the wait for stderr to flush after exit; a grandchild inheriting stdio
// can hold the pipes open long after the child is gone.
const SMOKE_STDERR_DRAIN_MS = 250
type SmokeResult = {
status: number | null
signal: NodeJS.Signals | null
stderr: string
error?: Error
timedOut: boolean
}
// Why not spawnSync({ timeout }): its timeout only sends killSignal and then
// keeps blocking until the child exits, so a child that traps SIGTERM hangs the
// build forever. Escalate to SIGKILL instead.
function runDaemonEntry(entryPath: string, timings: SmokeTimings): Promise<SmokeResult> {
return new Promise((resolve) => {
const child = spawn(process.execPath, [entryPath], { stdio: ['ignore', 'ignore', 'pipe'] })
let stderr = ''
let timedOut = false
let settled = false
let forceKillTimer: NodeJS.Timeout | undefined
let drainTimer: NodeJS.Timeout | undefined
child.stderr.setEncoding('utf8')
child.stderr.on('data', (chunk: string) => {
stderr += chunk
})
const deadlineTimer = setTimeout(() => {
timedOut = true
child.kill('SIGTERM')
forceKillTimer = setTimeout(() => child.kill('SIGKILL'), timings.killGraceMs)
}, timings.timeoutMs)
const finish = (status: number | null, signal: NodeJS.Signals | null, error?: Error): void => {
if (settled) {
return
}
settled = true
clearTimeout(deadlineTimer)
clearTimeout(forceKillTimer)
clearTimeout(drainTimer)
resolve({ status, signal, stderr, error, timedOut })
}
child.on('error', (error: Error) => finish(null, null, error))
// 'close' gives the full stderr; 'exit' is the fallback so a held-open pipe
// cannot outlast the process itself.
child.on('close', (status, signal) => finish(status, signal))
child.on('exit', (status, signal) => {
drainTimer = setTimeout(() => finish(status, signal), SMOKE_STDERR_DRAIN_MS)
})
})
}
// Why: proves the whole daemon-entry graph resolves under plain Node (no
// unresolved requires). require("electron") does not throw in a dev tree with
// node_modules present, so the static scan above — not this smoke — is the
// electron regression guard; this only catches gross load failures.
async function smokeLoadDaemonEntry(outputDir: string, timings: SmokeTimings): Promise<void> {
const entryPath = join(outputDir, 'daemon-entry.js')
const result = await runDaemonEntry(entryPath, timings)
if (result.error) {
throw new Error(
`[plain-node-entry-guard] could not smoke-load daemon-entry.js under plain Node: ` +
`${result.error.message}`
)
}
// Almost always means the daemon stopped rejecting an empty argv and started
// listening instead.
if (result.timedOut) {
throw new Error(
`[plain-node-entry-guard] daemon-entry.js did not exit within ${timings.timeoutMs}ms on an ` +
`empty argv under plain Node, so the smoke killed it.`
)
}
if (result.signal) {
throw new Error(
`[plain-node-entry-guard] daemon-entry.js was killed by ${result.signal} under plain Node.`
)
}
const stderr = result.stderr
if (/Cannot find module|MODULE_NOT_FOUND/.test(stderr)) {
throw new Error(
`[plain-node-entry-guard] daemon-entry.js failed to load under plain Node:\n${stderr}`
)
}
if (result.status === 0 || !stderr.includes(DAEMON_USAGE_PREFIX)) {
throw new Error(
`[plain-node-entry-guard] daemon-entry.js did not reject an empty argv under plain Node ` +
`(expected a non-zero exit and the "${DAEMON_USAGE_PREFIX}" error, got exit ` +
`${result.status}). stderr:\n${stderr}`
)
}
}
export function createPlainNodeEntryGuardPlugin(
smokeTimings: SmokeTimings = DEFAULT_SMOKE_TIMINGS
): Plugin {
let daemonOutputDir: string | undefined
return {
name: 'orca-plain-node-entry-guard',
buildStart(options: NormalizedInputOptions) {
assertEntryNamesAreRollupInputs(options.input)
},
writeBundle(options: NormalizedOutputOptions, bundle: OutputBundle) {
// Why: skip in `electron-vite dev` watch mode — the smoke would respawn on
// every rebuild, and the guard only needs to gate produced builds.
if (this.meta.watchMode) {
return
}
const chunks = Object.values(bundle).filter(
(item): item is OutputChunk => item.type === 'chunk'
)
const byFileName = new Map(chunks.map((chunk) => [chunk.fileName, chunk]))
const entryByName = new Map<string, OutputChunk>()
for (const chunk of chunks) {
if (chunk.isEntry && chunk.name) {
entryByName.set(chunk.name, chunk)
}
}
for (const entryName of PLAIN_NODE_ENTRY_NAMES) {
const entry = entryByName.get(entryName)
if (entry) {
assertNoElectronRequire(entryName, entry, byFileName, 'plain-Node process')
}
}
for (const entryName of WORKER_THREAD_ENTRY_NAMES) {
const entry = entryByName.get(entryName)
if (entry) {
assertNoElectronRequire(entryName, entry, byFileName, 'worker thread')
}
}
if (entryByName.has('daemon-entry') && options.dir) {
daemonOutputDir = options.dir
}
},
async closeBundle() {
if (daemonOutputDir) {
const outputDir = daemonOutputDir
daemonOutputDir = undefined
await smokeLoadDaemonEntry(outputDir, smokeTimings)
}
}
}
}