mirror of
https://github.com/stablyai/orca.git
synced 2026-09-22 08:02:28 +00:00
* fix(native-chat): kill hung WSL operations via child process
Stalled UNC file operations hold libuv permits even after the gate
timeout expires, blocking Chat tab recovery. Two stalled operations
fill both permits and freeze all WSL access until restart.
Fork file I/O for UNC paths into a separate child process. On deadline
expiry, kill the process to force the hung syscall to exit. This frees
the permit for the affected tab's next read. Temporarily quarantine the
stalled route to avoid retry storms.
* chore: drop internal review artifact from the repo root
* fix(native-chat): harden the WSL transcript fs sidecar
Review follow-ups on the sidecar isolation change:
- Only the deadline may abort running gate work. The sole waiter's
same-duration timeout fired first, killed healthy children on caller
abandonment, and settled the task before the deadline could quarantine
a stalled route - leaving the back-off dead for every dedupe:false op.
- Resolve the fork entry from out/main/chunks too: the resolver compiles
into a shared chunk, and the scanner service child has no
process.resourcesPath, so packaged WSL vault scans threw entry-not-found
(masked as an empty tree).
- Allowlist the fork env instead of spreading process.env; ambient
NODE_OPTIONS would halt or --require code into every child.
- Wrap transport faults (spawn failure, child death) in
WslTranscriptFsError('unavailable') so discovery reports them as scan
issues instead of misreading them as missing paths or empty trees.
- Gate the vitest in-process fallback on the vitest worker global so a
leaked VITEST=true cannot revert production to in-process UNC syscalls.
- Reap idle sidecar processes after 60s instead of holding them for the
app session.
- Split 'open' into its own protocol union member so the reusable-call
Exclude actually strips it from the pooled-process API.
- Guard kill('SIGKILL') against the teardown race where an exiting child
emits an unlistened 'error', and dispatch reads by handle kind before
path spelling.
* fix(native-chat): probe stalled WSL routes instead of a fixed quarantine
Remaining review follow-ups:
- Escalating route quarantine: first strike lifts after 5s so a distro
that was cold-booting when its op hit the deadline recovers on the
next poll (~35s total instead of ~90s); repeat stalls double the
back-off toward the prior 2x-timeout cap, and any settle the deadline
did not force clears the strikes. Queued same-route tasks fail fast
at quarantine instead of stranding one waiter deadline per file in
sequential scans.
- Single request implementation: the vitest in-process fallback now runs
the child's own dispatcher (WslTranscriptFsProcessOperations + decode),
so unit suites exercise exactly what the forked process executes and
the per-call-site fallback closures are gone. Dirent fixtures gained
the full kind-flag set the serializer reads.
- Dropped the production-dead per-route close queue; UNC FileHandles
(test fallback only) mirror the process-handle close contract.
- Error class, messages, and factories move to wsl-transcript-fs-error
(re-exported from the gate) to keep the gate under the lines budget.
* fix(native-chat): harden WSL transcript fs with route quarantine strike
Extract quarantine logic into a dedicated module with strike decay: stalls older
than 5 minutes restart from base back-off, and concurrent-lane timeouts count as
one incident. Allow joining live in-flight tasks on quarantined routes (they cost
no new I/O). Preserve quarantine across transport faults (child death). Handle
file shrinking during tail reads by detecting short reads and returning empty.
Defer file closes that arrive mid-read instead of refusing, preventing slot
leaks. Separate process slot and boundary-finding concerns into focused modules.
* fix(native-chat): enforce route quarantine windows and isolate lanes per
A late result arriving after the deadline was incorrectly lifting the route
quarantine, allowing subsequent work to start before the back-off period
expired. Now late results are correctly recognized as stale and never cut
the quarantine short.
Process work is now isolated per (route, priority) lane so a scan stall
cannot block exact reads on the same distro. Each lane gets its own client
and process pool; late results and handle faults stay scoped to their lane.
Tests now fake performance.now() alongside timers (the quarantine clock
depends on it) and wait for the full back-off window to expire rather than
advancing by 0. Gate state is reset between test cases since late releases
never lift the quarantine.
---------
Co-authored-by: OrcaWin <293788423+OrcaWin@users.noreply.github.com>
282 lines
10 KiB
TypeScript
282 lines
10 KiB
TypeScript
import { spawn } from 'node:child_process'
|
|
import { join } from 'node:path'
|
|
import type { Plugin, Rollup } from 'vite'
|
|
|
|
type NormalizedInputOptions = Rollup.NormalizedInputOptions
|
|
type NormalizedOutputOptions = Rollup.NormalizedOutputOptions
|
|
type OutputBundle = Rollup.OutputBundle
|
|
type OutputChunk = Rollup.OutputChunk
|
|
|
|
// Why: v1.4.129-rc.1 shipped a dead terminal daemon because a shared main
|
|
// chunk gained `require("electron")` (an import edge added in #7642), and the
|
|
// daemon is forked as a plain-Node process where electron cannot be required.
|
|
// Nothing in CI executes the built daemon-entry under plain Node, so the leak
|
|
// stayed invisible until an adopted old daemon died. This guard fails the
|
|
// build when any chunk reachable from a plain-Node fork entry requires
|
|
// electron, and smoke-loads daemon-entry under plain Node to prove its module
|
|
// graph still resolves.
|
|
|
|
// Entries executed as plain Node (ELECTRON_RUN_AS_NODE / no electron runtime):
|
|
// forked daemon, parcel-watcher, WSL filesystem and computer sidecars, and the CLI-run
|
|
// agent-hooks entry. require("electron") throws MODULE_NOT_FOUND in all of them.
|
|
const PLAIN_NODE_ENTRY_NAMES = [
|
|
'daemon-entry',
|
|
'parcel-watcher-process-entry',
|
|
'computer-sidecar',
|
|
'wsl-transcript-fs-process-entry',
|
|
'agent-hooks/managed-agent-hook-controls',
|
|
'codex/codex-app-server-grant-entry'
|
|
] as const
|
|
|
|
// Entries executed as worker threads of the main process. Electron's module is
|
|
// not registered on worker threads, so require("electron") throws
|
|
// "Cannot find module 'electron'" there too (verified on Electron 43) and kills
|
|
// the worker at startup. These carry hand-written "must stay electron-free"
|
|
// comments, which is convention, not enforcement — and the port-scan worker in
|
|
// particular sits one import away from a client module that deliberately does
|
|
// require electron.
|
|
const WORKER_THREAD_ENTRY_NAMES = [
|
|
'stt-worker',
|
|
'warp-theme-parser-worker',
|
|
'session-scanner-opencode-sqlite-worker-entry',
|
|
'session-scanner-worker-entry',
|
|
'main-thread-hang-watchdog-entry',
|
|
'port-scan-command-worker-entry'
|
|
] as const
|
|
|
|
export const GUARDED_ENTRY_NAMES = [
|
|
...PLAIN_NODE_ENTRY_NAMES,
|
|
...WORKER_THREAD_ENTRY_NAMES
|
|
] as const
|
|
|
|
type EntryRuntime = 'plain-Node process' | 'worker thread'
|
|
|
|
// Subpaths (electron/main) are as unloadable as the bare module under plain Node.
|
|
const ELECTRON_REQUIRE_RE = /require\(\s*["'`]electron(?:\/[^"'`]+)?["'`]\s*\)/
|
|
|
|
// Why: writeBundle skips any name missing from the bundle, so a renamed or
|
|
// removed rollup input would silently drop that entry from the guard and let the
|
|
// regression back in. Pin the lists to the input keys at build start instead.
|
|
function assertEntryNamesAreRollupInputs(input: NormalizedInputOptions['input']): void {
|
|
if (typeof input === 'string' || Array.isArray(input)) {
|
|
return
|
|
}
|
|
const inputNames = new Set(Object.keys(input))
|
|
const missing = GUARDED_ENTRY_NAMES.filter((name) => !inputNames.has(name))
|
|
if (missing.length > 0) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] guarded ${missing.map((name) => `"${name}"`).join(', ')} ` +
|
|
`${missing.length === 1 ? 'is not a rollup input' : 'are not rollup inputs'} anymore. ` +
|
|
`Update PLAIN_NODE_ENTRY_NAMES/WORKER_THREAD_ENTRY_NAMES in plain-node-entry-guard.ts to ` +
|
|
`the current entry names — a stale name silently stops guarding that entry.`
|
|
)
|
|
}
|
|
}
|
|
|
|
function collectReachableChunks(
|
|
entry: OutputChunk,
|
|
byFileName: Map<string, OutputChunk>
|
|
): OutputChunk[] {
|
|
const seen = new Set<string>()
|
|
const reachable: OutputChunk[] = []
|
|
const stack = [entry.fileName]
|
|
while (stack.length > 0) {
|
|
const fileName = stack.pop() as string
|
|
if (seen.has(fileName)) {
|
|
continue
|
|
}
|
|
seen.add(fileName)
|
|
const chunk = byFileName.get(fileName)
|
|
if (!chunk) {
|
|
continue
|
|
}
|
|
reachable.push(chunk)
|
|
for (const imported of [...chunk.imports, ...chunk.dynamicImports]) {
|
|
stack.push(imported)
|
|
}
|
|
}
|
|
return reachable
|
|
}
|
|
|
|
function assertNoElectronRequire(
|
|
entryName: string,
|
|
entry: OutputChunk,
|
|
byFileName: Map<string, OutputChunk>,
|
|
runtime: EntryRuntime = 'plain-Node process'
|
|
): void {
|
|
for (const chunk of collectReachableChunks(entry, byFileName)) {
|
|
if (ELECTRON_REQUIRE_RE.test(chunk.code)) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] "${entryName}" reaches chunk "${chunk.fileName}" that ` +
|
|
`requires electron. "${entryName}" runs as a ${runtime}, where ` +
|
|
`require("electron") throws MODULE_NOT_FOUND and kills it at startup (the ` +
|
|
`v1.4.129-rc.1 daemon outage). Keep electron imports out of its module graph.`
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Owned by the argv parser in src/main/daemon/daemon-entry.ts — keep in sync.
|
|
const DAEMON_USAGE_PREFIX = 'Usage: daemon-entry'
|
|
|
|
export type SmokeTimings = {
|
|
timeoutMs: number
|
|
// daemon-entry traps SIGTERM and awaits a native shutdown, so the deadline
|
|
// needs an uncatchable follow-up to stay a deadline.
|
|
killGraceMs: number
|
|
}
|
|
|
|
const DEFAULT_SMOKE_TIMINGS: SmokeTimings = { timeoutMs: 15_000, killGraceMs: 2_000 }
|
|
|
|
// Bound the wait for stderr to flush after exit; a grandchild inheriting stdio
|
|
// can hold the pipes open long after the child is gone.
|
|
const SMOKE_STDERR_DRAIN_MS = 250
|
|
|
|
type SmokeResult = {
|
|
status: number | null
|
|
signal: NodeJS.Signals | null
|
|
stderr: string
|
|
error?: Error
|
|
timedOut: boolean
|
|
}
|
|
|
|
// Why not spawnSync({ timeout }): its timeout only sends killSignal and then
|
|
// keeps blocking until the child exits, so a child that traps SIGTERM hangs the
|
|
// build forever. Escalate to SIGKILL instead.
|
|
function runDaemonEntry(entryPath: string, timings: SmokeTimings): Promise<SmokeResult> {
|
|
return new Promise((resolve) => {
|
|
const child = spawn(process.execPath, [entryPath], { stdio: ['ignore', 'ignore', 'pipe'] })
|
|
let stderr = ''
|
|
let timedOut = false
|
|
let settled = false
|
|
let forceKillTimer: NodeJS.Timeout | undefined
|
|
let drainTimer: NodeJS.Timeout | undefined
|
|
|
|
child.stderr.setEncoding('utf8')
|
|
child.stderr.on('data', (chunk: string) => {
|
|
stderr += chunk
|
|
})
|
|
|
|
const deadlineTimer = setTimeout(() => {
|
|
timedOut = true
|
|
child.kill('SIGTERM')
|
|
forceKillTimer = setTimeout(() => child.kill('SIGKILL'), timings.killGraceMs)
|
|
}, timings.timeoutMs)
|
|
|
|
const finish = (status: number | null, signal: NodeJS.Signals | null, error?: Error): void => {
|
|
if (settled) {
|
|
return
|
|
}
|
|
settled = true
|
|
clearTimeout(deadlineTimer)
|
|
clearTimeout(forceKillTimer)
|
|
clearTimeout(drainTimer)
|
|
resolve({ status, signal, stderr, error, timedOut })
|
|
}
|
|
|
|
child.on('error', (error: Error) => finish(null, null, error))
|
|
// 'close' gives the full stderr; 'exit' is the fallback so a held-open pipe
|
|
// cannot outlast the process itself.
|
|
child.on('close', (status, signal) => finish(status, signal))
|
|
child.on('exit', (status, signal) => {
|
|
drainTimer = setTimeout(() => finish(status, signal), SMOKE_STDERR_DRAIN_MS)
|
|
})
|
|
})
|
|
}
|
|
|
|
// Why: proves the whole daemon-entry graph resolves under plain Node (no
|
|
// unresolved requires). require("electron") does not throw in a dev tree with
|
|
// node_modules present, so the static scan above — not this smoke — is the
|
|
// electron regression guard; this only catches gross load failures.
|
|
async function smokeLoadDaemonEntry(outputDir: string, timings: SmokeTimings): Promise<void> {
|
|
const entryPath = join(outputDir, 'daemon-entry.js')
|
|
const result = await runDaemonEntry(entryPath, timings)
|
|
if (result.error) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] could not smoke-load daemon-entry.js under plain Node: ` +
|
|
`${result.error.message}`
|
|
)
|
|
}
|
|
// Almost always means the daemon stopped rejecting an empty argv and started
|
|
// listening instead.
|
|
if (result.timedOut) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js did not exit within ${timings.timeoutMs}ms on an ` +
|
|
`empty argv under plain Node, so the smoke killed it.`
|
|
)
|
|
}
|
|
if (result.signal) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js was killed by ${result.signal} under plain Node.`
|
|
)
|
|
}
|
|
const stderr = result.stderr
|
|
if (/Cannot find module|MODULE_NOT_FOUND/.test(stderr)) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js failed to load under plain Node:\n${stderr}`
|
|
)
|
|
}
|
|
if (result.status === 0 || !stderr.includes(DAEMON_USAGE_PREFIX)) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js did not reject an empty argv under plain Node ` +
|
|
`(expected a non-zero exit and the "${DAEMON_USAGE_PREFIX}" error, got exit ` +
|
|
`${result.status}). stderr:\n${stderr}`
|
|
)
|
|
}
|
|
}
|
|
|
|
export function createPlainNodeEntryGuardPlugin(
|
|
smokeTimings: SmokeTimings = DEFAULT_SMOKE_TIMINGS
|
|
): Plugin {
|
|
let daemonOutputDir: string | undefined
|
|
|
|
return {
|
|
name: 'orca-plain-node-entry-guard',
|
|
buildStart(options: NormalizedInputOptions) {
|
|
assertEntryNamesAreRollupInputs(options.input)
|
|
},
|
|
writeBundle(options: NormalizedOutputOptions, bundle: OutputBundle) {
|
|
// Why: skip in `electron-vite dev` watch mode — the smoke would respawn on
|
|
// every rebuild, and the guard only needs to gate produced builds.
|
|
if (this.meta.watchMode) {
|
|
return
|
|
}
|
|
const chunks = Object.values(bundle).filter(
|
|
(item): item is OutputChunk => item.type === 'chunk'
|
|
)
|
|
const byFileName = new Map(chunks.map((chunk) => [chunk.fileName, chunk]))
|
|
const entryByName = new Map<string, OutputChunk>()
|
|
for (const chunk of chunks) {
|
|
if (chunk.isEntry && chunk.name) {
|
|
entryByName.set(chunk.name, chunk)
|
|
}
|
|
}
|
|
|
|
for (const entryName of PLAIN_NODE_ENTRY_NAMES) {
|
|
const entry = entryByName.get(entryName)
|
|
if (entry) {
|
|
assertNoElectronRequire(entryName, entry, byFileName, 'plain-Node process')
|
|
}
|
|
}
|
|
|
|
for (const entryName of WORKER_THREAD_ENTRY_NAMES) {
|
|
const entry = entryByName.get(entryName)
|
|
if (entry) {
|
|
assertNoElectronRequire(entryName, entry, byFileName, 'worker thread')
|
|
}
|
|
}
|
|
|
|
if (entryByName.has('daemon-entry') && options.dir) {
|
|
daemonOutputDir = options.dir
|
|
}
|
|
},
|
|
async closeBundle() {
|
|
if (daemonOutputDir) {
|
|
const outputDir = daemonOutputDir
|
|
daemonOutputDir = undefined
|
|
await smokeLoadDaemonEntry(outputDir, smokeTimings)
|
|
}
|
|
}
|
|
}
|
|
}
|