Files
orca/src/shared/process-table-snapshot-reader.ts
T
Neil 756d623955 perf(terminal): bound the in-process macOS process table's blocking calls and ps fallbacks (#31783) (#36282)
* fix(terminal): bound the in-process macOS process table's blocking calls and ps fallbacks (#31783)

Follow-up to #35987 from its post-merge reviews.

- cwd and executable path: node-pty's readProcessCwd/readProcessPath now run
  PROC_PIDVNODEPATHINFO and proc_pidpath on a libuv worker (the synchronous
  exports are gone, so an older pty.node falls back to ps/lsof). JS caps them at
  two outstanding lookups and 250ms; a wedged mount or a full cap answers from
  lsof -b, which never stats, so a dead NFS/SMB/FUSE mount cannot freeze the
  daemon's event loop or take the whole pool.
- Unreadable argv prints "(comm)", as ps does, instead of a proc_pidpath lookup
  under Orca's credentials on the table worker.
- The table exports each row's euid. Argv recovery runs setuid ps only for
  another user's processes, never for our own mid-exec or exiting ones (or
  SIDL/exiting rows), which ps cannot read either: a pane looping short
  commands forked ps on ~15% of full reads, now none.
- Recovery is coalesced per process across concurrent readers and a failed
  recovery waits 5s before costing another ps. The cache keys on comm and start
  time.
- Full-table and shell-foreground captures, which feed strict evidence with
  the capture's age, take only argv ps read after their request; display
  readers keep the 30s cache.
- While the reader is down (stalled-pool breaker open, or a rejecting addon,
  which now backs off 1s..30s), runPsInProcess runs /bin/ps itself through one
  shared budget: identical invocations share a run, a burst of 4 forks then one
  a second, and a run spawned before a request never answers it. Descendant
  teardown and foreground-group signalling keep their own full-timeout ps
  (callerRunsPs). 13 pollers at 750ms: 1,040 ps/min before, 63 after.

No daemon protocol bump: the wire format is unchanged, and each daemon loads
JS and pty.node from the same runtime copy.

* fix(terminal): make the libproc reader survive worker teardown, fail closed and share one fair fork budget (#31783)

Review follow-ups to the previous commit.

- Native: every libproc AsyncWorker derives from OrcaPromiseWorker, which skips
  JS completion once an env cleanup hook marks the env closing and swallows a
  Napi::Error thrown mid-teardown. Terminating a worker_thread with cwd or table
  reads in flight aborted the process; 200 terminates each now survive.
- (comm) in a terminal's foreground group (argv neither reader recovered, e.g.
  a root `su`) judges 'unknown', never 'agent', so the launch line is not typed
  into it.
- Fresh argv only for getFreshSnapshot captures; the 500ms TTL readers keep the
  30s cache. Argv recovery runs through its own fork budget (2, then one per
  2s). Root `su` pane, 13 pollers at 750ms: TTL 79 -> 2 ps/min, fresh 158 -> 31.
- The fork budget serves invocations in FIFO turns, grouping each one's waiters
  onto one run. 13 streams: served 80,42,1,1,2,0x8 -> 10-12 each.
- Forced subprocess captures (a held naming pass) go through the fallback
  budget: 5 readers, 400 direct ps/min -> 63, 395 of 400 captures served.
- A failed recovery is retried 5s after ps gave up, not after it started.
- Tests pin Orca's euid instead of reading the runner's and assuming root is 0.
- process-cwd keeps one lsof per pid until it exits, so a dead mount no longer
  leaks one per poll; shell readiness reads the exe path through runProcess,
  which gives up on an lsof that ignores SIGTERM.
- A #35987 pty.node (table reader, no async path reads) keeps the in-process
  table; cwd/path use lsof, with a one-time warning.
- captureProcessTable moves to process-table-ps-capture.ts.

* fix(terminal): make withheld argv unverifiable and reuse recovered argv for at most 2s (#31783)

Round-2 review follow-ups.

- Worker teardown: drop OrcaPromiseWorker and its env flag (Node drains the
  pool before cleanup hooks, so it never fired). node-pty's mac pty target now
  builds with NODE_API_SWALLOW_UNTHROWABLE_EXCEPTIONS, which covers both the
  call-in and completion paths. The smoke test now calls the reads from a
  timer while terminating: the previous build aborted 2 of 3 runs, this one
  survives 3/3 x 100. pty.cc is back to the first follow-up's exact content.
- A foreground process whose argv ps withheld (`(comm)`: another user's,
  exiting, or a recovery the budget skipped) makes the batch resolver, and so
  host-stamped evidence, `argv_unreadable`/unverifiable rather than live with no
  agent; agent-foreground resolution reports it unavailable.
- judgeTerminalForeground fails closed only for an unreadable shell (`(bash)`
  from a root `su`), not our own exiting `(git)`.
- Recovered argv is reused for at most 2s per identity (pid, start, comm) for
  every reader (the next commit ages evidence by it), and
  fresh reads no longer queue behind the recovery budget. Root `su` pane, 13
  fresh pollers: p50 2,989ms -> 53ms, 31 -> 26 ps/min (TTL reader 1 -> 26).
- The fork budgets serve interactive and fresh readers before display polls
  (cheap and TTL snapshot captures), FIFO within each. 20 display streams plus
  a 3s launch-paste read every 5s: 0/12 -> 12/12 served, polls 4-7 each.
- On a selected terminal, the foreground group's argv is recovered even with no
  process of ours left (`exec su`); the root login is not.
- Budgeted captures keep the 32 MiB output ceiling.

* fix(terminal): age strict evidence by its oldest recovered argv (#31783)

Round-3 review follow-ups.

- Recovered argv carried no read time, so a capture reusing it (cached, or a
  recovery still in flight) stamped its own capture time over argv up to 2s
  older, and the remote write guard could take a same-comm re-exec for the
  agent. Rows now carry when ps read their recovered argv; runPsInProcess
  reports the oldest through onArgvReadAt, and the snapshot reader stamps
  capturedAtMs at that input, so capturedAgeMs is the age of the oldest input
  and consumers' own freshness checks (the write guard's 500ms) decide. A
  recovery begun more than 2s before a reader asked no longer answers it, and
  an older recovery finishing late cannot replace a newer one.
- resolveSpawnFileForegroundFromRows reports a withheld `(node)` foreground
  unavailable.
- One budget queue entry per invocation: an interactive request promotes a
  queued poll's, a poll joins an interactive one.
- parseShellForegroundRows gets its JSDoc back.

* fix(terminal): keep a cached name from authorizing a refused write; integer, under-2s argv ages (#31783)

Round-4 review follow-ups.

- When the relay's evidence declines a write (stale, unverifiable, another
  agent), its foreground name may still prove the shell but no longer proves
  the agent: that name can be the tracker's 1s display cache, e.g. `claude`
  cached just before Claude exited to a root shell under su. Relays that send
  no evidence keep the name as before.
- Snapshot ages round up to whole milliseconds; the wire rejects fractions
  from the monotonic-to-wall conversion.
- Recovered argv is reused for at most 1.5s, so reuse plus the 500ms snapshot
  cache stays under the 2s freshness gate; reuse ages from when that ps
  spawned, not from the wait for its fork token, and only an answer is
  remembered (a timed-out or withheld run proves nothing). Fork budget
  unchanged. Root su pane, 13 pollers: 30 ps/min; fresh p50 51ms; strict
  evidence age p50 752ms, max 1,508ms.

* fix(terminal): keep the relay's agent name behind structurally unverifiable evidence (#31783)

The previous commit refused the name fallback whenever the relay sent any
evidence, so `unverifiable` evidence for structural reasons also blocked
writes: npm Codex (node codex.js spawning native codex) and Gemini (relaunching
itself) put two agent rows in the foreground group (ambiguous_foreground_group),
and remote tmux reads multiplexer_boundary. On SSH POSIX hosts that dropped
launch prompts and failed worker briefs, conversation send/cancel and
automations with agent_not_in_foreground.

The name now cannot authorize only when the evidence saw the group (`live`:
stale, or no agent in it, as in the cached-`claude`-over-root-shell repro) or
withheld a foreground argv (`argv_unreadable`, possibly a root `su` shell).
Other unverifiable reasons keep the name fallback, as before this branch.

* fix(terminal): let a relay name complete only fresh agent-positive evidence (#31783)

The SSH name fallback still let a stale or cached name authorize a write under
any unverifiable reason but argv_unreadable: ps EAGAIN (process_table_unreadable)
after the tracker cached `claude` over a root bash, anchor_missing,
tty_boundary (`sudo -i` with use_pty, a root shell on a new pty), or a 1.5s-old
ambiguous capture.

- The name now completes evidence only when it is fresh (within the guard's
  500ms bound), its reason is one that saw an agent row
  (ambiguous_foreground_group, multiplexer_boundary,
  candidate_start_time_missing), and the name is the launched agent. Every other
  reason, including ones this build does not know, reads unknown. Relays that
  send no evidence keep the name, for mixed versions.
- The relay's evidence names one agent that holds two rows of the group (npm
  Codex: node codex.js spawning native codex; Gemini relaunching itself), so a
  new relay proves them live even when its name is `node`. Two different agents
  stay ambiguous. No wire change.

* fix(terminal): judge a relay name by the host's capture age; keep old relays' npm Codex (#31783)

- Name completion compared the capture's start with the read's start, so a
  slow response made old evidence look fresh: a capture 1,500ms old arriving
  after 1,100ms passed. It now uses the host-reported capturedAgeMs against
  the 500ms bound, which latency cannot shrink. Live evidence keeps its
  capture-start rule.
- Relays before same-agent dedupe report npm Codex (node wrapper plus native
  binary) as ambiguous_foreground_group and name it `node`. For that reason
  only, `node` now completes the evidence when the launched agent is
  npm-installed per the node entrypoint tables; other agents and reasons stay
  fail-closed.

* fix(terminal): bound relay-name completion by twice the snapshot TTL (#31783)

The 500ms bound refused ~9.5% of tmux name completions on a host with 13
pollers (cached snapshot up to one 500ms TTL old plus ~30ms capture, max
528ms), and a refusal drops the launch prompt or throws with no retry. The
bound is now 2 x PROCESS_TABLE_SNAPSHOT_MAX_STALENESS_MS (1s), still on the
host-reported age: 0 of 767 reads exceed it, the delayed 1,500ms repro still
refuses, and the cached-name hazard (the tracker's ~1s display cache) is no
wider.

* fix(terminal): complete relay evidence with a name read now, not the display cache (#31783)

With the 1s bound, a capture at 0-528ms, `codex` cached at 100ms, Codex
exiting at 101ms and the guard at 999ms still let the cached name authorize a
write over the bash that replaced it.

- Name completion (structural reasons, host age within 2x the snapshot TTL)
  now asks confirmForegroundProcess, which reads the foreground now (a fresh
  scan, or node-pty's raw process name) rather than the tracker's 1s display
  cache; a null answer reads unknown. No new wire surface: the local daemon answers
  confirmForegroundProcess, paired runtimes evaluate guards locally, and the
  production controller returns null (read as unknown, failing safe) when a
  provider lacks it. Controllers without the method, and relays sending no
  evidence, keep the cached name as the legacy path.
- Live evidence also accepts a host-reported age within 2x the TTL, so a real
  agent behind a 500-528ms cached snapshot on a fast link is no longer
  refused; a capture begun after the read was asked still counts, and
  5,000ms-old evidence still refuses.

* fix(terminal): confirm live evidence older than one snapshot TTL with a fresh read (#31783)

Accepting live evidence up to 2x the TTL reopened the window the previous
commit closed for name completion: a capture at 0-528ms, the agent exiting at
~529ms, and at 999ms `live claude` still read as agent over a bash.

Live evidence begun no more than one TTL before the read was asked (so any
host age within one TTL, however long ps took) still answers alone. Between
that and 2x the TTL, confirmForegroundProcess reads the foreground now: the
launched agent proves it, a shell reads shell, anything else unknown.
Controllers without that read fall through and refuse, as before.

Reachability: the remote (host.remote) write-guard path has no production
caller since the SSH provider's removal (#27499); the registry dispatches only
local providers and paired runtimes evaluate guards locally. This is defence
in depth for any future remote provider.
2026-10-30 01:05:34 -07:00

326 lines
10 KiB
TypeScript
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
import { readFile } from 'node:fs/promises'
import { monotonicNow } from './deadline-settle-notifier'
import { PS_TIMEOUT_MS, assertWholeCapture, captureProcessTable } from './process-table-ps-capture'
import {
captureDarwinProcessTable,
resetDarwinProcessTableCaptureForTests
} from './darwin-process-table-capture'
import {
PROCESS_TABLE_SNAPSHOT_MAX_STALENESS_MS,
PS_ARGS,
PS_MAX_BUFFER_BYTES,
ProcessTableCaptureError,
SHELL_FOREGROUND_PS_ARGS,
parseProcessTableRows,
parseShellForegroundRows,
parseStrictProcessTableRows,
type ProcessTableRow
} from './process-table-snapshot'
export { PROCESS_TABLE_SNAPSHOT_MAX_STALENESS_MS, PS_ARGS, PS_MAX_BUFFER_BYTES, PS_TIMEOUT_MS }
type Snapshot<T> = { value: T; capturedAtMs: number; completedAtMs: number }
type ProcessTableSnapshotReaderDeps<T> = {
/** `fresh`: an authoritative read waiting on this capture; anything else is a display poll. `inputAt` takes the
* time (in `now()`'s clock) of any input older than the capture itself, such as recovered argv. */
runPs: (fresh: boolean, inputAt: (atMs: number) => void) => Promise<T>
now: () => number
ttlMs?: number
}
/** Build a process-table reader that coalesces concurrent and recent captures. */
export function createProcessTableSnapshotReader<T = string>(
deps: ProcessTableSnapshotReaderDeps<T>
): {
getSnapshot: () => Promise<T>
getSnapshotWithAge: () => Promise<{ value: T; capturedAgeMs: number }>
getFreshSnapshot: () => Promise<T>
reset: () => void
} {
const ttlMs = deps.ttlMs ?? PROCESS_TABLE_SNAPSHOT_MAX_STALENESS_MS
let cached: Snapshot<T> | null = null
let inFlight: Promise<T> | null = null
let sequence = 0
let freshQueued: { promise: Promise<T>; startSequence: number | null } | null = null
async function runSnapshot(fresh = false): Promise<T> {
// Stamp evidence age at capture start (or its oldest input), TTL at completion: slow captures must remain old
// yet still coalesce instead of spawning one whole-machine ps per caller on loaded hosts.
let capturedAtMs = deps.now()
const promise = deps.runPs(fresh, (atMs) => {
capturedAtMs = Math.min(capturedAtMs, atMs)
})
inFlight = promise
try {
const value = await promise
cached = { value, capturedAtMs, completedAtMs: deps.now() }
return value
} finally {
if (inFlight === promise) {
inFlight = null
}
}
}
async function getSnapshot(): Promise<T> {
if (cached && deps.now() - cached.completedAtMs < ttlMs) {
return cached.value
}
if (inFlight) {
return inFlight
}
if (freshQueued) {
return freshQueued.promise
}
return runSnapshot()
}
async function getSnapshotWithAge(): Promise<{ value: T; capturedAgeMs: number }> {
const value = await getSnapshot()
const capturedAtMs = cached?.value === value ? cached.capturedAtMs : deps.now()
// Whole milliseconds, rounded older: the wire requires integers and an input's clock conversion is fractional.
return { value, capturedAgeMs: Math.max(0, Math.ceil(deps.now() - capturedAtMs)) }
}
function getFreshSnapshot(): Promise<T> {
const requestSequence = ++sequence
if (freshQueued?.startSequence === null) {
return freshQueued.promise
}
const priorFresh = freshQueued?.promise ?? null
const priorScan = inFlight
const entry: { promise: Promise<T>; startSequence: number | null } = {
promise: Promise.resolve(undefined as never),
startSequence: null
}
entry.promise = Promise.resolve().then(async () => {
for (const prior of [priorFresh, priorScan]) {
if (!prior) {
continue
}
try {
await prior
} catch {
// The post-boundary scan below owns the confirmation result.
}
}
entry.startSequence = ++sequence
if (entry.startSequence <= requestSequence) {
throw new Error('fresh process snapshot did not start after request')
}
return runSnapshot(true)
})
freshQueued = entry
const clearQueued = (): void => {
if (freshQueued === entry) {
freshQueued = null
}
}
void entry.promise.then(clearQueued, clearQueued)
return entry.promise
}
return {
getSnapshot,
getSnapshotWithAge,
getFreshSnapshot,
reset: () => {
cached = null
inFlight = null
sequence = 0
freshQueued = null
}
}
}
type ProcessTableCapture = {
lenient: () => ProcessTableRow[]
strict: () => ProcessTableRow[]
}
function applyProcessStartTimes(
rows: ProcessTableRow[],
startTimesByPid: ReadonlyMap<number, string> | undefined,
dropUnstableStartTimes = false
): ProcessTableRow[] {
if ((!startTimesByPid || startTimesByPid.size === 0) && !dropUnstableStartTimes) {
return rows
}
return rows.map((row) => {
const startTime = startTimesByPid?.get(row.pid)
if (startTime) {
return { ...row, startTime }
}
if (dropUnstableStartTimes && row.startTime !== undefined) {
const { startTime: _unstable, ...withoutStartTime } = row
return withoutStartTime
}
return row
})
}
function createProcessTableCapture(
stdout: string,
startTimesByPid?: ReadonlyMap<number, string>,
dropUnstableStartTimes = false
): ProcessTableCapture {
let lenientRows: ProcessTableRow[] | null = null
let strictResult: { rows: ProcessTableRow[] } | { error: unknown } | null = null
return {
lenient: () =>
(lenientRows ??= applyProcessStartTimes(
parseProcessTableRows(stdout),
startTimesByPid,
dropUnstableStartTimes
)),
strict: () => {
if (strictResult === null) {
try {
strictResult = {
rows: applyProcessStartTimes(
parseStrictProcessTableRows(stdout),
startTimesByPid,
dropUnstableStartTimes
)
}
} catch (error) {
strictResult = { error }
}
}
if ('error' in strictResult) {
throw strictResult.error
}
return strictResult.rows
}
}
}
/** Field 22 (`starttime`) of `/proc/<pid>/stat`, read past the parenthesised comm. */
export function parseLinuxProcStatStartTime(stat: string): string | null {
const closingParen = stat.lastIndexOf(')')
if (closingParen === -1) {
return null
}
const tail = stat
.slice(closingParen + 1)
.trim()
.split(/\s+/)
return tail[19] || null
}
/** Read Linux's stable PID start-time ticks without spawning another process. */
async function readLinuxProcessStartTimes(
rows: readonly ProcessTableRow[]
): Promise<ReadonlyMap<number, string>> {
const candidates = rows.filter((row) => row.tty !== undefined && row.tty !== '?')
const starts = await Promise.all(
candidates.map(async (row) => {
try {
const startTime = parseLinuxProcStatStartTime(
await readFile(`/proc/${row.pid}/stat`, 'utf8')
)
return startTime ? ([row.pid, startTime] as const) : null
} catch {
return null
}
})
)
return new Map(starts.filter((entry) => entry !== null))
}
/** Recovered argv reports monotonic read times; the snapshot readers stamp ages on the wall clock. */
function onArgvReadAt(inputAt: (atMs: number) => void): (monotonicMs: number) => void {
return (monotonicMs) => inputAt(Date.now() - (monotonicNow() - monotonicMs))
}
const processTableReader = createProcessTableSnapshotReader<ProcessTableCapture>({
runPs: async (fresh, inputAt) => {
const stdout =
process.platform === 'darwin'
? await captureDarwinProcessTable(
(args, timeout, ps) =>
captureProcessTable(args, timeout, ps, !fresh, onArgvReadAt(inputAt)),
assertWholeCapture,
PS_TIMEOUT_MS
)
: await captureProcessTable(PS_ARGS)
if (process.platform !== 'linux') {
return createProcessTableCapture(stdout)
}
const startTimesByPid = await readLinuxProcessStartTimes(parseProcessTableRows(stdout))
return createProcessTableCapture(stdout, startTimesByPid, true)
},
now: () => Date.now()
})
// Separate shell reader avoids macOS terminal-name costs and waiting on full captures it cannot use.
const shellForegroundReader = createProcessTableSnapshotReader<ProcessTableRow[]>({
runPs: async (fresh, inputAt) =>
parseShellForegroundRows(
await captureProcessTable(
SHELL_FOREGROUND_PS_ARGS,
PS_TIMEOUT_MS,
false,
!fresh,
onArgvReadAt(inputAt)
)
),
now: () => Date.now()
})
export async function getFreshShellForegroundSnapshot(): Promise<ProcessTableRow[]> {
return process.platform === 'darwin'
? shellForegroundReader.getFreshSnapshot()
: getFreshProcessTableSnapshot()
}
export async function getProcessTableSnapshot(): Promise<ProcessTableRow[]> {
return (await processTableReader.getSnapshot()).lenient()
}
export async function getFreshProcessTableSnapshot(): Promise<ProcessTableRow[]> {
return (await processTableReader.getFreshSnapshot()).lenient()
}
export async function getStrictProcessTableSnapshot(): Promise<ProcessTableRow[]> {
return (await processTableReader.getSnapshot()).strict()
}
/** Evidence needs current observations, unlike identity probes that wait up to 15s. Idle command= costs ~1.15s/1,948 processes;
* 2s freshness ceiling minus up to 500ms cached age and transit leaves 1.2s. Slower captures fail age gates anyway
* (measured 4–18.6s at load 46, 2.5–9s on a 2,002-process laptop), so return unverifiable promptly. */
export const PROCESS_TABLE_EVIDENCE_BUDGET_MS = 1_200
/** Bound only this wait; leave shared identity captures running to fill the cache instead of spawning another expensive ps. */
export async function withEvidenceBudget<T>(pending: Promise<T>): Promise<T> {
let timer: ReturnType<typeof setTimeout> | undefined
try {
return await Promise.race([
pending,
new Promise<never>((_resolve, reject) => {
timer = setTimeout(
() => reject(new ProcessTableCaptureError('capture_over_budget')),
PROCESS_TABLE_EVIDENCE_BUDGET_MS
)
})
])
} finally {
clearTimeout(timer)
}
}
export async function getStrictProcessTableSnapshotWithAge(): Promise<{
rows: ProcessTableRow[]
capturedAgeMs: number
}> {
const snapshot = await withEvidenceBudget(processTableReader.getSnapshotWithAge())
return { rows: snapshot.value.strict(), capturedAgeMs: snapshot.capturedAgeMs }
}
export function resetProcessTableSnapshotForTests(): void {
processTableReader.reset()
shellForegroundReader.reset()
resetDarwinProcessTableCaptureForTests()
}