mirror of
https://github.com/stablyai/orca.git
synced 2026-09-21 16:02:20 +00:00
* perf(codex-usage): resume rollout scans at the last parsed byte Codex rollout files are append-only and grow all day, but any append changed both mtime and size, so `canReuse` discarded the cached entry and the scanner re-read the whole file from byte 0 on the Electron main process. On one real corpus that was 6.59 GB re-read per cycle across 26.63 GB / 21,110 files. Each parsed file now persists a resume point: the offset just past the last newline-terminated line, the parse context at that offset (session id, cwd, model, running totals), a sha256 of the 4 KiB before it, and the file's dev:ino. A grown file resumes there and merges the appended rollup into the cached one; anything unproven falls back to a full reparse — truncation, an in-place rewrite, rotation, a counted tail with no trailing newline, a legacy copied-session suffix offset, or a file that must reclaim deferred fork claims. Resume never depends on mtime equality, so a coarse-mtime filesystem cannot hide an append. Fixture: a 75,737-byte rollout with a 758-byte append re-read 76,495 bytes before and 8,950 after (the append plus two bounded 4 KiB boundary windows). Also bounds the automation-attribution force predicate for both Codex and Claude: it keyed on `lastScanError`, so a persistently failing scan forced a fresh full rescan on every single lookup. It now keys on the most recent scan attempt, which is one forced scan per run regardless of outcome. * perf(usage): run the Claude/Codex/OpenCode usage scans on a worker thread The three first-party usage scans walk whole rollout and transcript corpora and read OpenCode's SQLite synchronously, all on the Electron main process. They rarely produce a long stall — the JSONL reader streams, so it yields to the loop between chunks — but they pin the main-process event loop at ~95% utilization for the scan's whole duration, which is what every IPC message, timer and window event then queues behind. Move that work to one lazily-spawned, unref'd worker thread shared by all three providers, following the OpenCode SQLite scanner precedent (#8864). Measured on a synthetic 4,000-rollout corpus (25.8 MB cache): a cold scan drops from 2,147 ms of main-thread time to 31 ms, and a steady-state incremental scan from 165 ms to 64 ms. The worker is stateless and the cache crosses the boundary both ways. That costs ~64 ms of structured clone at this corpus size, against 2,147 ms saved on the cold path, and it keeps the persisted cache the single source of truth — a worker-owned copy would need an invalidation protocol and a second resident copy of the same multi-MB array. Failure is closed, never a silent empty result: a worker that cannot spawn, times out, or crash-loops rejects, and the store records the scan error and keeps the previous projection. Two clients already carried the same FIFO/timeout/crash-cap machinery, so extract it once as WorkerThreadRequestQueue (with the packaged entry-path resolver as worker-thread-entry-path) and move all three onto it, rather than adding a third copy. Their existing tests pass unchanged. The oracle is event-loop utilization on the calling thread, not a stopwatch: usage-scan-worker-event-loop.test.ts runs the same scan both ways and asserts the worker leg leaves the caller idle while the main-thread leg does not, so CI load moves both legs together (#18788). * test(usage): compare the two scan arms instead of two fixed thresholds The event-loop oracle claimed to be self-calibrating — its header said "the ratio is self-calibrating, so CI load moves both legs together (#18788) instead of tipping a fixed millisecond threshold." It computed no ratio. Two separate `it()` blocks each asserted an absolute threshold against its own arm, run separately, so load moved them independently. The comment described a test nobody wrote, and the flake it promised was impossible is the one that landed: `activeRatio > 0.8` on the calling-thread arm measured 0.764 on an ubuntu runner. Fixing the comment is not enough, because the fraction is the wrong quantity. CPU contention drags the calling-thread arm's active/wall fraction *down* toward the worker's, since the loop parks waiting on a contended libuv pool. A 4-vCPU Linux container measured that arm at 0.175-0.756 across twenty runs, idle and loaded — never once above 0.8. Active *milliseconds* move the other way: contention stretches the caller's JS time far more than it stretches the worker arm's fixed post-and-deserialize cost, so the gap widens under load. Merge the two arms into one case over one corpus and assert the worker arm costs the caller under a fifth of the inline arm's active milliseconds. Same twenty Linux runs: 10.9x-83.6x, passing throughout. Keep the presence preconditions on both arms — an arm that silently scanned nothing satisfies the comparison trivially — and extend them to the calling-thread arm, which previously checked only file and session counts. * fix(ports): name the dropped command when the probe queue is full The shared-queue extraction turned `Port scan command queue is full; dropped ${command}.` into a constant string, because `describeFull` was given no way to see the request. Pile-up is per-probe, so the name is the only thing in that log that identifies which of lsof/ps/netstat was shed. Pass the rejected request to `describeFull` and restore the name. The request is built before the cap check so it exists to be named; the id it burns is a correlation token, so a gap costs nothing. The existing overflow test asserted only the error class, which is why the regression escaped a 29-test suite. It now dispatches the overflow under a different command than the accepted ones and asserts the message text, so a message that names the wrong request fails too. Also add a direct WorkerThreadRequestQueue test. Three subsystems share the queue and each client test only sees the parts its own protocol exercises, with `queueCap` reachable from port-scan alone. Covers one-at-a-time FIFO dispatch, the deadline starting at dispatch rather than enqueue, the consecutive-death cap, and both points where that count clears. And record the child-process hazard at the usage worker entry. `terminate()` reaps nothing the thread spawned, and OpenCode discovery reaches a fork today: `wslGated*` forks the WSL transcript sidecar for a `\\wsl$\...` path, which a Windows `OPENCODE_DB` or `XDG_DATA_HOME` can be. One scan through that entry with a UNC `OPENCODE_DB` forked a sidecar that outlived `terminate()`. * test(ai-vault): assert the OpenCode worker messages exactly, not by fragment Checked every message string in the two clients the shared-queue extraction rewrote against origin/main. Only the port-scan queue-full one regressed (fixed in the previous commit); the OpenCode SQLite client's four messages render identically, the remaining source diffs being renames — `error.message` to `lastError`, `call.timeoutMs` and `CALL_DEADLINE_MS` to `timeoutMs`. `session-scanner-worker-client.ts` was not touched by the extraction. But its suite could not have caught it either. `/timed out/`, `/exited with code/` and a bare `rejects.toThrow()` all still match a message that has lost its interpolated value, which is the same blind spot that let the port-scan regression through. Assert the rendered text instead: the timeout names its deadline, the exit names its code, and the crash-loop drain still carries the text of the fault that killed the run. * fix(usage): correct the worker entry's child-process note The previous note said `worker.terminate()` leaves a forked sidecar orphaned. It does not, and the reproduction that appeared to show it used a stub sidecar missing the `process.on('disconnect', () => process.exit(0))` the real entry has. With a faithful one: the sidecar lives exactly as long as the thread and is gone within 2s of `terminate()`, because tearing the thread down closes the IPC channel it owned. Two worker lifecycles forked two sidecars and leaked neither, and the pre-worker main-thread path reaps its sidecar the same way, on host exit. What is true and worth recording: a fork is reachable from this bundle at all, which is easy to miss; it survives only as long as the channel does; and the sidecar is now re-forked per worker lifecycle instead of pooled for the app's life. State those, and warn that a future child which does not exit on channel close would not get the same free cleanup. * fix(usage): kill a wedged scan worker on no progress, not on wall clock `USAGE_SCAN_TIMEOUT_MS` was a 10-minute deadline on the whole scan. A cold scan of a real history is legitimately minutes — 637 s measured on a 30 GB corpus with 300 worktrees before the per-cwd memo, ~51 s after — so a larger corpus or a slower disk crosses it. Crossing it killed the worker, recorded a scan error and left the cache unadvanced, so the next refresh started cold and died at the same point, forever. The deadline is now a no-progress window. The worker posts a file counter as it walks the corpus (`UsageScanWorkerProgress`, rate-limited to one message a second), and `WorkerThreadRequestQueue` re-arms the active call's timer on each one via the new optional `isProgress`. Clients that do not pass it keep the plain wall-clock deadline. `MAX_CONSECUTIVE_DEATHS` and idle teardown are unchanged. * refactor(usage): report scan progress as a file count, not one call per file Claude's scanner walks batches, so a per-file callback made it loop just to bump a counter.
282 lines
10 KiB
TypeScript
282 lines
10 KiB
TypeScript
import { spawn } from 'node:child_process'
|
|
import { join } from 'node:path'
|
|
import type { Plugin, Rollup } from 'vite'
|
|
|
|
type NormalizedInputOptions = Rollup.NormalizedInputOptions
|
|
type NormalizedOutputOptions = Rollup.NormalizedOutputOptions
|
|
type OutputBundle = Rollup.OutputBundle
|
|
type OutputChunk = Rollup.OutputChunk
|
|
|
|
// Why: v1.4.129-rc.1 shipped a dead terminal daemon because a shared main
|
|
// chunk gained `require("electron")` (an import edge added in #7642), and the
|
|
// daemon is forked as a plain-Node process where electron cannot be required.
|
|
// Nothing in CI executes the built daemon-entry under plain Node, so the leak
|
|
// stayed invisible until an adopted old daemon died. This guard fails the
|
|
// build when any chunk reachable from a plain-Node fork entry requires
|
|
// electron, and smoke-loads daemon-entry under plain Node to prove its module
|
|
// graph still resolves.
|
|
|
|
// Entries executed as plain Node (ELECTRON_RUN_AS_NODE / no electron runtime):
|
|
// forked daemon, parcel-watcher, WSL filesystem and computer sidecars, and the CLI-run
|
|
// agent-hooks entry. require("electron") throws MODULE_NOT_FOUND in all of them.
|
|
const PLAIN_NODE_ENTRY_NAMES = [
|
|
'daemon-entry',
|
|
'parcel-watcher-process-entry',
|
|
'computer-sidecar',
|
|
'wsl-transcript-fs-process-entry',
|
|
'agent-hooks/managed-agent-hook-controls'
|
|
] as const
|
|
|
|
// Entries executed as worker threads of the main process. Electron's module is
|
|
// not registered on worker threads, so require("electron") throws
|
|
// "Cannot find module 'electron'" there too (verified on Electron 43) and kills
|
|
// the worker at startup. These carry hand-written "must stay electron-free"
|
|
// comments, which is convention, not enforcement — and the port-scan worker in
|
|
// particular sits one import away from a client module that deliberately does
|
|
// require electron.
|
|
const WORKER_THREAD_ENTRY_NAMES = [
|
|
'stt-worker',
|
|
'warp-theme-parser-worker',
|
|
'session-scanner-opencode-sqlite-worker-entry',
|
|
'session-scanner-worker-entry',
|
|
'main-thread-hang-watchdog-entry',
|
|
'port-scan-command-worker-entry',
|
|
'usage-scan-worker-entry'
|
|
] as const
|
|
|
|
export const GUARDED_ENTRY_NAMES = [
|
|
...PLAIN_NODE_ENTRY_NAMES,
|
|
...WORKER_THREAD_ENTRY_NAMES
|
|
] as const
|
|
|
|
type EntryRuntime = 'plain-Node process' | 'worker thread'
|
|
|
|
// Subpaths (electron/main) are as unloadable as the bare module under plain Node.
|
|
const ELECTRON_REQUIRE_RE = /require\(\s*["'`]electron(?:\/[^"'`]+)?["'`]\s*\)/
|
|
|
|
// Why: writeBundle skips any name missing from the bundle, so a renamed or
|
|
// removed rollup input would silently drop that entry from the guard and let the
|
|
// regression back in. Pin the lists to the input keys at build start instead.
|
|
function assertEntryNamesAreRollupInputs(input: NormalizedInputOptions['input']): void {
|
|
if (typeof input === 'string' || Array.isArray(input)) {
|
|
return
|
|
}
|
|
const inputNames = new Set(Object.keys(input))
|
|
const missing = GUARDED_ENTRY_NAMES.filter((name) => !inputNames.has(name))
|
|
if (missing.length > 0) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] guarded ${missing.map((name) => `"${name}"`).join(', ')} ` +
|
|
`${missing.length === 1 ? 'is not a rollup input' : 'are not rollup inputs'} anymore. ` +
|
|
`Update PLAIN_NODE_ENTRY_NAMES/WORKER_THREAD_ENTRY_NAMES in plain-node-entry-guard.ts to ` +
|
|
`the current entry names — a stale name silently stops guarding that entry.`
|
|
)
|
|
}
|
|
}
|
|
|
|
function collectReachableChunks(
|
|
entry: OutputChunk,
|
|
byFileName: Map<string, OutputChunk>
|
|
): OutputChunk[] {
|
|
const seen = new Set<string>()
|
|
const reachable: OutputChunk[] = []
|
|
const stack = [entry.fileName]
|
|
while (stack.length > 0) {
|
|
const fileName = stack.pop() as string
|
|
if (seen.has(fileName)) {
|
|
continue
|
|
}
|
|
seen.add(fileName)
|
|
const chunk = byFileName.get(fileName)
|
|
if (!chunk) {
|
|
continue
|
|
}
|
|
reachable.push(chunk)
|
|
for (const imported of [...chunk.imports, ...chunk.dynamicImports]) {
|
|
stack.push(imported)
|
|
}
|
|
}
|
|
return reachable
|
|
}
|
|
|
|
function assertNoElectronRequire(
|
|
entryName: string,
|
|
entry: OutputChunk,
|
|
byFileName: Map<string, OutputChunk>,
|
|
runtime: EntryRuntime = 'plain-Node process'
|
|
): void {
|
|
for (const chunk of collectReachableChunks(entry, byFileName)) {
|
|
if (ELECTRON_REQUIRE_RE.test(chunk.code)) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] "${entryName}" reaches chunk "${chunk.fileName}" that ` +
|
|
`requires electron. "${entryName}" runs as a ${runtime}, where ` +
|
|
`require("electron") throws MODULE_NOT_FOUND and kills it at startup (the ` +
|
|
`v1.4.129-rc.1 daemon outage). Keep electron imports out of its module graph.`
|
|
)
|
|
}
|
|
}
|
|
}
|
|
|
|
// Owned by the argv parser in src/main/daemon/daemon-entry.ts — keep in sync.
|
|
const DAEMON_USAGE_PREFIX = 'Usage: daemon-entry'
|
|
|
|
export type SmokeTimings = {
|
|
timeoutMs: number
|
|
// daemon-entry traps SIGTERM and awaits a native shutdown, so the deadline
|
|
// needs an uncatchable follow-up to stay a deadline.
|
|
killGraceMs: number
|
|
}
|
|
|
|
const DEFAULT_SMOKE_TIMINGS: SmokeTimings = { timeoutMs: 15_000, killGraceMs: 2_000 }
|
|
|
|
// Bound the wait for stderr to flush after exit; a grandchild inheriting stdio
|
|
// can hold the pipes open long after the child is gone.
|
|
const SMOKE_STDERR_DRAIN_MS = 250
|
|
|
|
type SmokeResult = {
|
|
status: number | null
|
|
signal: NodeJS.Signals | null
|
|
stderr: string
|
|
error?: Error
|
|
timedOut: boolean
|
|
}
|
|
|
|
// Why not spawnSync({ timeout }): its timeout only sends killSignal and then
|
|
// keeps blocking until the child exits, so a child that traps SIGTERM hangs the
|
|
// build forever. Escalate to SIGKILL instead.
|
|
function runDaemonEntry(entryPath: string, timings: SmokeTimings): Promise<SmokeResult> {
|
|
return new Promise((resolve) => {
|
|
const child = spawn(process.execPath, [entryPath], { stdio: ['ignore', 'ignore', 'pipe'] })
|
|
let stderr = ''
|
|
let timedOut = false
|
|
let settled = false
|
|
let forceKillTimer: NodeJS.Timeout | undefined
|
|
let drainTimer: NodeJS.Timeout | undefined
|
|
|
|
child.stderr.setEncoding('utf8')
|
|
child.stderr.on('data', (chunk: string) => {
|
|
stderr += chunk
|
|
})
|
|
|
|
const deadlineTimer = setTimeout(() => {
|
|
timedOut = true
|
|
child.kill('SIGTERM')
|
|
forceKillTimer = setTimeout(() => child.kill('SIGKILL'), timings.killGraceMs)
|
|
}, timings.timeoutMs)
|
|
|
|
const finish = (status: number | null, signal: NodeJS.Signals | null, error?: Error): void => {
|
|
if (settled) {
|
|
return
|
|
}
|
|
settled = true
|
|
clearTimeout(deadlineTimer)
|
|
clearTimeout(forceKillTimer)
|
|
clearTimeout(drainTimer)
|
|
resolve({ status, signal, stderr, error, timedOut })
|
|
}
|
|
|
|
child.on('error', (error: Error) => finish(null, null, error))
|
|
// 'close' gives the full stderr; 'exit' is the fallback so a held-open pipe
|
|
// cannot outlast the process itself.
|
|
child.on('close', (status, signal) => finish(status, signal))
|
|
child.on('exit', (status, signal) => {
|
|
drainTimer = setTimeout(() => finish(status, signal), SMOKE_STDERR_DRAIN_MS)
|
|
})
|
|
})
|
|
}
|
|
|
|
// Why: proves the whole daemon-entry graph resolves under plain Node (no
|
|
// unresolved requires). require("electron") does not throw in a dev tree with
|
|
// node_modules present, so the static scan above — not this smoke — is the
|
|
// electron regression guard; this only catches gross load failures.
|
|
async function smokeLoadDaemonEntry(outputDir: string, timings: SmokeTimings): Promise<void> {
|
|
const entryPath = join(outputDir, 'daemon-entry.js')
|
|
const result = await runDaemonEntry(entryPath, timings)
|
|
if (result.error) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] could not smoke-load daemon-entry.js under plain Node: ` +
|
|
`${result.error.message}`
|
|
)
|
|
}
|
|
// Almost always means the daemon stopped rejecting an empty argv and started
|
|
// listening instead.
|
|
if (result.timedOut) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js did not exit within ${timings.timeoutMs}ms on an ` +
|
|
`empty argv under plain Node, so the smoke killed it.`
|
|
)
|
|
}
|
|
if (result.signal) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js was killed by ${result.signal} under plain Node.`
|
|
)
|
|
}
|
|
const stderr = result.stderr
|
|
if (/Cannot find module|MODULE_NOT_FOUND/.test(stderr)) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js failed to load under plain Node:\n${stderr}`
|
|
)
|
|
}
|
|
if (result.status === 0 || !stderr.includes(DAEMON_USAGE_PREFIX)) {
|
|
throw new Error(
|
|
`[plain-node-entry-guard] daemon-entry.js did not reject an empty argv under plain Node ` +
|
|
`(expected a non-zero exit and the "${DAEMON_USAGE_PREFIX}" error, got exit ` +
|
|
`${result.status}). stderr:\n${stderr}`
|
|
)
|
|
}
|
|
}
|
|
|
|
export function createPlainNodeEntryGuardPlugin(
|
|
smokeTimings: SmokeTimings = DEFAULT_SMOKE_TIMINGS
|
|
): Plugin {
|
|
let daemonOutputDir: string | undefined
|
|
|
|
return {
|
|
name: 'orca-plain-node-entry-guard',
|
|
buildStart(options: NormalizedInputOptions) {
|
|
assertEntryNamesAreRollupInputs(options.input)
|
|
},
|
|
writeBundle(options: NormalizedOutputOptions, bundle: OutputBundle) {
|
|
// Why: skip in `electron-vite dev` watch mode — the smoke would respawn on
|
|
// every rebuild, and the guard only needs to gate produced builds.
|
|
if (this.meta.watchMode) {
|
|
return
|
|
}
|
|
const chunks = Object.values(bundle).filter(
|
|
(item): item is OutputChunk => item.type === 'chunk'
|
|
)
|
|
const byFileName = new Map(chunks.map((chunk) => [chunk.fileName, chunk]))
|
|
const entryByName = new Map<string, OutputChunk>()
|
|
for (const chunk of chunks) {
|
|
if (chunk.isEntry && chunk.name) {
|
|
entryByName.set(chunk.name, chunk)
|
|
}
|
|
}
|
|
|
|
for (const entryName of PLAIN_NODE_ENTRY_NAMES) {
|
|
const entry = entryByName.get(entryName)
|
|
if (entry) {
|
|
assertNoElectronRequire(entryName, entry, byFileName, 'plain-Node process')
|
|
}
|
|
}
|
|
|
|
for (const entryName of WORKER_THREAD_ENTRY_NAMES) {
|
|
const entry = entryByName.get(entryName)
|
|
if (entry) {
|
|
assertNoElectronRequire(entryName, entry, byFileName, 'worker thread')
|
|
}
|
|
}
|
|
|
|
if (entryByName.has('daemon-entry') && options.dir) {
|
|
daemonOutputDir = options.dir
|
|
}
|
|
},
|
|
async closeBundle() {
|
|
if (daemonOutputDir) {
|
|
const outputDir = daemonOutputDir
|
|
daemonOutputDir = undefined
|
|
await smokeLoadDaemonEntry(outputDir, smokeTimings)
|
|
}
|
|
}
|
|
}
|
|
}
|