Files
orca/src/shared/libproc-process-table.ts
T
Neil 756d623955 perf(terminal): bound the in-process macOS process table's blocking calls and ps fallbacks (#31783) (#36282)
* fix(terminal): bound the in-process macOS process table's blocking calls and ps fallbacks (#31783)

Follow-up to #35987 from its post-merge reviews.

- cwd and executable path: node-pty's readProcessCwd/readProcessPath now run
  PROC_PIDVNODEPATHINFO and proc_pidpath on a libuv worker (the synchronous
  exports are gone, so an older pty.node falls back to ps/lsof). JS caps them at
  two outstanding lookups and 250ms; a wedged mount or a full cap answers from
  lsof -b, which never stats, so a dead NFS/SMB/FUSE mount cannot freeze the
  daemon's event loop or take the whole pool.
- Unreadable argv prints "(comm)", as ps does, instead of a proc_pidpath lookup
  under Orca's credentials on the table worker.
- The table exports each row's euid. Argv recovery runs setuid ps only for
  another user's processes, never for our own mid-exec or exiting ones (or
  SIDL/exiting rows), which ps cannot read either: a pane looping short
  commands forked ps on ~15% of full reads, now none.
- Recovery is coalesced per process across concurrent readers and a failed
  recovery waits 5s before costing another ps. The cache keys on comm and start
  time.
- Full-table and shell-foreground captures, which feed strict evidence with
  the capture's age, take only argv ps read after their request; display
  readers keep the 30s cache.
- While the reader is down (stalled-pool breaker open, or a rejecting addon,
  which now backs off 1s..30s), runPsInProcess runs /bin/ps itself through one
  shared budget: identical invocations share a run, a burst of 4 forks then one
  a second, and a run spawned before a request never answers it. Descendant
  teardown and foreground-group signalling keep their own full-timeout ps
  (callerRunsPs). 13 pollers at 750ms: 1,040 ps/min before, 63 after.

No daemon protocol bump: the wire format is unchanged, and each daemon loads
JS and pty.node from the same runtime copy.

* fix(terminal): make the libproc reader survive worker teardown, fail closed and share one fair fork budget (#31783)

Review follow-ups to the previous commit.

- Native: every libproc AsyncWorker derives from OrcaPromiseWorker, which skips
  JS completion once an env cleanup hook marks the env closing and swallows a
  Napi::Error thrown mid-teardown. Terminating a worker_thread with cwd or table
  reads in flight aborted the process; 200 terminates each now survive.
- (comm) in a terminal's foreground group (argv neither reader recovered, e.g.
  a root `su`) judges 'unknown', never 'agent', so the launch line is not typed
  into it.
- Fresh argv only for getFreshSnapshot captures; the 500ms TTL readers keep the
  30s cache. Argv recovery runs through its own fork budget (2, then one per
  2s). Root `su` pane, 13 pollers at 750ms: TTL 79 -> 2 ps/min, fresh 158 -> 31.
- The fork budget serves invocations in FIFO turns, grouping each one's waiters
  onto one run. 13 streams: served 80,42,1,1,2,0x8 -> 10-12 each.
- Forced subprocess captures (a held naming pass) go through the fallback
  budget: 5 readers, 400 direct ps/min -> 63, 395 of 400 captures served.
- A failed recovery is retried 5s after ps gave up, not after it started.
- Tests pin Orca's euid instead of reading the runner's and assuming root is 0.
- process-cwd keeps one lsof per pid until it exits, so a dead mount no longer
  leaks one per poll; shell readiness reads the exe path through runProcess,
  which gives up on an lsof that ignores SIGTERM.
- A #35987 pty.node (table reader, no async path reads) keeps the in-process
  table; cwd/path use lsof, with a one-time warning.
- captureProcessTable moves to process-table-ps-capture.ts.

* fix(terminal): make withheld argv unverifiable and reuse recovered argv for at most 2s (#31783)

Round-2 review follow-ups.

- Worker teardown: drop OrcaPromiseWorker and its env flag (Node drains the
  pool before cleanup hooks, so it never fired). node-pty's mac pty target now
  builds with NODE_API_SWALLOW_UNTHROWABLE_EXCEPTIONS, which covers both the
  call-in and completion paths. The smoke test now calls the reads from a
  timer while terminating: the previous build aborted 2 of 3 runs, this one
  survives 3/3 x 100. pty.cc is back to the first follow-up's exact content.
- A foreground process whose argv ps withheld (`(comm)`: another user's,
  exiting, or a recovery the budget skipped) makes the batch resolver, and so
  host-stamped evidence, `argv_unreadable`/unverifiable rather than live with no
  agent; agent-foreground resolution reports it unavailable.
- judgeTerminalForeground fails closed only for an unreadable shell (`(bash)`
  from a root `su`), not our own exiting `(git)`.
- Recovered argv is reused for at most 2s per identity (pid, start, comm) for
  every reader (the next commit ages evidence by it), and
  fresh reads no longer queue behind the recovery budget. Root `su` pane, 13
  fresh pollers: p50 2,989ms -> 53ms, 31 -> 26 ps/min (TTL reader 1 -> 26).
- The fork budgets serve interactive and fresh readers before display polls
  (cheap and TTL snapshot captures), FIFO within each. 20 display streams plus
  a 3s launch-paste read every 5s: 0/12 -> 12/12 served, polls 4-7 each.
- On a selected terminal, the foreground group's argv is recovered even with no
  process of ours left (`exec su`); the root login is not.
- Budgeted captures keep the 32 MiB output ceiling.

* fix(terminal): age strict evidence by its oldest recovered argv (#31783)

Round-3 review follow-ups.

- Recovered argv carried no read time, so a capture reusing it (cached, or a
  recovery still in flight) stamped its own capture time over argv up to 2s
  older, and the remote write guard could take a same-comm re-exec for the
  agent. Rows now carry when ps read their recovered argv; runPsInProcess
  reports the oldest through onArgvReadAt, and the snapshot reader stamps
  capturedAtMs at that input, so capturedAgeMs is the age of the oldest input
  and consumers' own freshness checks (the write guard's 500ms) decide. A
  recovery begun more than 2s before a reader asked no longer answers it, and
  an older recovery finishing late cannot replace a newer one.
- resolveSpawnFileForegroundFromRows reports a withheld `(node)` foreground
  unavailable.
- One budget queue entry per invocation: an interactive request promotes a
  queued poll's, a poll joins an interactive one.
- parseShellForegroundRows gets its JSDoc back.

* fix(terminal): keep a cached name from authorizing a refused write; integer, under-2s argv ages (#31783)

Round-4 review follow-ups.

- When the relay's evidence declines a write (stale, unverifiable, another
  agent), its foreground name may still prove the shell but no longer proves
  the agent: that name can be the tracker's 1s display cache, e.g. `claude`
  cached just before Claude exited to a root shell under su. Relays that send
  no evidence keep the name as before.
- Snapshot ages round up to whole milliseconds; the wire rejects fractions
  from the monotonic-to-wall conversion.
- Recovered argv is reused for at most 1.5s, so reuse plus the 500ms snapshot
  cache stays under the 2s freshness gate; reuse ages from when that ps
  spawned, not from the wait for its fork token, and only an answer is
  remembered (a timed-out or withheld run proves nothing). Fork budget
  unchanged. Root su pane, 13 pollers: 30 ps/min; fresh p50 51ms; strict
  evidence age p50 752ms, max 1,508ms.

* fix(terminal): keep the relay's agent name behind structurally unverifiable evidence (#31783)

The previous commit refused the name fallback whenever the relay sent any
evidence, so `unverifiable` evidence for structural reasons also blocked
writes: npm Codex (node codex.js spawning native codex) and Gemini (relaunching
itself) put two agent rows in the foreground group (ambiguous_foreground_group),
and remote tmux reads multiplexer_boundary. On SSH POSIX hosts that dropped
launch prompts and failed worker briefs, conversation send/cancel and
automations with agent_not_in_foreground.

The name now cannot authorize only when the evidence saw the group (`live`:
stale, or no agent in it, as in the cached-`claude`-over-root-shell repro) or
withheld a foreground argv (`argv_unreadable`, possibly a root `su` shell).
Other unverifiable reasons keep the name fallback, as before this branch.

* fix(terminal): let a relay name complete only fresh agent-positive evidence (#31783)

The SSH name fallback still let a stale or cached name authorize a write under
any unverifiable reason but argv_unreadable: ps EAGAIN (process_table_unreadable)
after the tracker cached `claude` over a root bash, anchor_missing,
tty_boundary (`sudo -i` with use_pty, a root shell on a new pty), or a 1.5s-old
ambiguous capture.

- The name now completes evidence only when it is fresh (within the guard's
  500ms bound), its reason is one that saw an agent row
  (ambiguous_foreground_group, multiplexer_boundary,
  candidate_start_time_missing), and the name is the launched agent. Every other
  reason, including ones this build does not know, reads unknown. Relays that
  send no evidence keep the name, for mixed versions.
- The relay's evidence names one agent that holds two rows of the group (npm
  Codex: node codex.js spawning native codex; Gemini relaunching itself), so a
  new relay proves them live even when its name is `node`. Two different agents
  stay ambiguous. No wire change.

* fix(terminal): judge a relay name by the host's capture age; keep old relays' npm Codex (#31783)

- Name completion compared the capture's start with the read's start, so a
  slow response made old evidence look fresh: a capture 1,500ms old arriving
  after 1,100ms passed. It now uses the host-reported capturedAgeMs against
  the 500ms bound, which latency cannot shrink. Live evidence keeps its
  capture-start rule.
- Relays before same-agent dedupe report npm Codex (node wrapper plus native
  binary) as ambiguous_foreground_group and name it `node`. For that reason
  only, `node` now completes the evidence when the launched agent is
  npm-installed per the node entrypoint tables; other agents and reasons stay
  fail-closed.

* fix(terminal): bound relay-name completion by twice the snapshot TTL (#31783)

The 500ms bound refused ~9.5% of tmux name completions on a host with 13
pollers (cached snapshot up to one 500ms TTL old plus ~30ms capture, max
528ms), and a refusal drops the launch prompt or throws with no retry. The
bound is now 2 x PROCESS_TABLE_SNAPSHOT_MAX_STALENESS_MS (1s), still on the
host-reported age: 0 of 767 reads exceed it, the delayed 1,500ms repro still
refuses, and the cached-name hazard (the tracker's ~1s display cache) is no
wider.

* fix(terminal): complete relay evidence with a name read now, not the display cache (#31783)

With the 1s bound, a capture at 0-528ms, `codex` cached at 100ms, Codex
exiting at 101ms and the guard at 999ms still let the cached name authorize a
write over the bash that replaced it.

- Name completion (structural reasons, host age within 2x the snapshot TTL)
  now asks confirmForegroundProcess, which reads the foreground now (a fresh
  scan, or node-pty's raw process name) rather than the tracker's 1s display
  cache; a null answer reads unknown. No new wire surface: the local daemon answers
  confirmForegroundProcess, paired runtimes evaluate guards locally, and the
  production controller returns null (read as unknown, failing safe) when a
  provider lacks it. Controllers without the method, and relays sending no
  evidence, keep the cached name as the legacy path.
- Live evidence also accepts a host-reported age within 2x the TTL, so a real
  agent behind a 500-528ms cached snapshot on a fast link is no longer
  refused; a capture begun after the read was asked still counts, and
  5,000ms-old evidence still refuses.

* fix(terminal): confirm live evidence older than one snapshot TTL with a fresh read (#31783)

Accepting live evidence up to 2x the TTL reopened the window the previous
commit closed for name completion: a capture at 0-528ms, the agent exiting at
~529ms, and at 999ms `live claude` still read as agent over a bash.

Live evidence begun no more than one TTL before the read was asked (so any
host age within one TTL, however long ps took) still answers alone. Between
that and 2x the TTL, confirmForegroundProcess reads the foreground now: the
launched agent proves it, a shell reads shell, anything else unknown.
Controllers without that read fall through and refuse, as before.

Reachability: the remote (host.remote) write-guard path has no production
caller since the SSH provider's removal (#27499); the registry dispatches only
local providers and paired runtimes evaluate guards locally. This is defence
in depth for any future remote provider.
2026-10-30 01:05:34 -07:00

294 lines
10 KiB
TypeScript

import { nameDarwinTerminalDevices } from './darwin-terminal-names'
import {
claimDarwinTerminalNaming,
waitForDarwinTerminalNaming
} from './darwin-terminal-naming-single-flight'
import { loadLibprocAddon, type LibprocProcessRow, type LibprocQuery } from './libproc-addon'
import { DEADLINE_PASSED, beforeDeadline, monotonicNow } from './deadline-settle-notifier'
import {
isNativeReaderHealthy,
readFreshNativeTable,
registerNativeRequest
} from './libproc-native-reads'
import { completeCommands } from './libproc-argv-completion'
import { registerPsFallbackRequest, runPsFallback } from './libproc-ps-fallback'
import {
compareLikePs,
formatDarwinDevice,
formatLibprocStat,
formatLstart,
parsePsQuery
} from './libproc-ps-format'
/** Shaped like a process-host ProcessResult so call sites can substitute it for ps's. */
export type LibprocPsResult = {
code: number | null
signal: NodeJS.Signals | null
stdout: string
stderr: string
timedOut: boolean
outputTruncated?: boolean
}
export type InProcessPsOptions = {
/** The environment ps would run with; its TZ and locale shape lstart and command escaping. */
env?: NodeJS.ProcessEnv
/** The caller's ps timeout: past it the read resolves timed out, as the killed ps would. */
timeoutMs: number
maxOutputBytes?: number
/** A display poll: any ps it needs waits behind interactive and fresh readers in the fork budget. */
background?: boolean
/** Called with when (monotonic ms) the result's oldest recovered argv was read, when that predates this request:
* evidence built on the result is as old as that input, not as the table read. */
onArgvReadAt?: (monotonicMs: number) => void
/** The caller runs its own ps when the in-process reader is down, rather than share the fallback budget, which may
* time it out (signalling paths that must not lose their read). */
callerRunsPs?: boolean
}
const TIMED_OUT: LibprocPsResult = {
code: null,
signal: 'SIGTERM',
stdout: '',
stderr: '',
timedOut: true,
outputTruncated: false
}
/** ps runs setlocale(LC_ALL, ""). */
function localeCategory(env: NodeJS.ProcessEnv, category: 'LC_CTYPE' | 'LC_TIME'): string {
return env.LC_ALL || env[category] || env.LANG || 'C'
}
function lstartZoneIsUtc(env: NodeJS.ProcessEnv): boolean | null {
const zone = env.TZ
if (zone === undefined || zone === process.env.TZ) {
return false
}
return /^(?:UTC|GMT)0?$/.test(zone) ? true : null
}
/** One naming pass at a time: wait out a short one (startup restore) rather than burst /bin/ps; a stalled one
* returns 'stalled' so the caller runs ps, which names terminals out of process. */
async function nameTerminals(
devices: ReadonlySet<string>,
deadline: number
): Promise<ReadonlyMap<string, string> | typeof DEADLINE_PASSED | 'stalled'> {
for (;;) {
const controller = new AbortController()
const naming = claimDarwinTerminalNaming(() =>
nameDarwinTerminalDevices(devices, undefined, controller.signal)
)
if (naming) {
const named = await beforeDeadline(naming, deadline)
if (named === DEADLINE_PASSED) {
controller.abort()
}
return named
}
const waited = await waitForDarwinTerminalNaming(deadline)
if (waited !== 'drained') {
return waited
}
}
}
/** Native reads take milliseconds: a signalling caller tries them for this slice, then gives ps its whole
* original timeout (extending any outer deadline by the slice), so a slow ps succeeds exactly as before. */
export function inProcessSliceMs(timeoutMs: number): number {
return Math.min(150, Math.floor(timeoutMs / 4))
}
/** The outer-deadline allowance for that slice: none where the in-process reader will not run (Linux, no addon, stalled). */
export function inProcessDeadlineAllowanceMs(timeoutMs: number): number {
return loadLibprocAddon() && isNativeReaderHealthy() ? inProcessSliceMs(timeoutMs) : 0
}
/** ps output for `args`, read in process, or by one budgeted /bin/ps while the reader is down; null when the addon or
* the invocation is unsupported (or `callerRunsPs`) so callers run ps. */
export async function runPsInProcess(
args: readonly string[],
options: InProcessPsOptions
): Promise<LibprocPsResult | null> {
const addon = loadLibprocAddon()
const query = addon ? parsePsQuery(args) : null
if (!addon || !query) {
return null
}
const registeredSequence = registerNativeRequest()
const fallbackSequence = registerPsFallbackRequest()
const requestedAtMs = monotonicNow()
const deadline = requestedAtMs + options.timeoutMs
const env = options.env ?? process.env
// ps gets the caller's whole timeout from the moment the reader gives up, as the caller's own ps did.
const fallback = async (): Promise<LibprocPsResult | null> =>
options.callerRunsPs
? null
: runPsFallback(
args,
{ env, maxOutputBytes: options.maxOutputBytes },
fallbackSequence,
monotonicNow() + options.timeoutMs,
options.background
)
const utc = lstartZoneIsUtc(env)
const needsLstart = query.columns.includes('lstart')
// lstart is `%c`: only the C and en_US forms are reproduced.
if (
needsLstart &&
(utc === null || !/^(?:C|POSIX|en_US(?:\..*)?)$/.test(localeCategory(env, 'LC_TIME')))
) {
return null
}
if (!isNativeReaderHealthy()) {
return fallback()
}
const includeCommand = query.columns.includes('command')
const ctype = localeCategory(env, 'LC_CTYPE')
const nativeQuery: LibprocQuery = {
command: includeCommand,
...(query.pids ? { pids: query.pids } : {}),
...(query.tty !== null ? { tty: query.tty } : {}),
...(includeCommand ? { ctype } : {})
}
let rows: LibprocProcessRow[] | null
let ttyNames: ReadonlyMap<string, string> = new Map()
try {
const read = await readFreshNativeTable(addon, nativeQuery, registeredSequence, deadline)
if (read === 'stalled') {
return fallback()
}
if (read === DEADLINE_PASSED) {
return TIMED_OUT
}
// Shared reads hand every caller the same rows; command completion writes per caller.
rows = read?.map((row) => ({ ...row })) ?? null
if (rows && query.columns.includes('tty')) {
const devices = new Set(
rows.map((row) => formatDarwinDevice(row.tdev)).filter((device) => device !== null)
)
if (devices.size > 0) {
const named = await nameTerminals(devices, deadline)
if (named === 'stalled') {
return fallback()
}
if (named === DEADLINE_PASSED) {
return TIMED_OUT
}
ttyNames = named
}
}
if (rows && includeCommand) {
await completeCommands(
rows,
query.pids !== null ? 'pids' : query.tty !== null ? 'tty' : 'all',
ctype,
env,
deadline,
options.background === true
)
}
} catch {
return fallback()
}
if (monotonicNow() > deadline) {
return TIMED_OUT
}
const argvReadAt = Math.min(...(rows ?? []).map((row) => row.commandReadAtMs ?? Infinity))
if (argvReadAt < requestedAtMs) {
options.onArgvReadAt?.(argvReadAt)
}
// ps exits 1 when -t names no terminal or a -p/-t selection matched nothing.
if (rows === null) {
return { ...TIMED_OUT, code: 1, signal: null, timedOut: false }
}
const ttyOf = (row: LibprocProcessRow): string =>
ttyNames.get(formatDarwinDevice(row.tdev) ?? '') ?? '??'
const selected = rows.sort(compareLikePs)
const lines = selected.map((row) =>
query.columns
.map((column) => {
switch (column) {
case 'stat':
return formatLibprocStat(row)
case 'tdev':
return formatDarwinDevice(row.tdev) ?? '??'
case 'tty':
return ttyOf(row)
case 'lstart':
return formatLstart(row.startSec, utc === true)
case 'command':
return row.command ?? `(${row.comm})`
case 'pid':
case 'ppid':
case 'pgid':
case 'tpgid':
return String(row[column])
}
})
.join(' ')
)
let stdout = lines.length > 0 ? `${lines.join('\n')}\n` : ''
const maxBytes = options.maxOutputBytes
const outputTruncated = maxBytes !== undefined && Buffer.byteLength(stdout, 'utf8') > maxBytes
if (outputTruncated) {
stdout = Buffer.from(stdout, 'utf8').subarray(0, maxBytes).toString('utf8')
}
return {
code: query.pids === null && query.tty === null ? 0 : selected.length > 0 ? 0 : 1,
signal: null,
stdout,
stderr: '',
timedOut: false,
outputTruncated
}
}
function stdoutLikeExecFile(args: readonly string[], result: LibprocPsResult): string {
if (result.timedOut || result.code !== 0) {
throw Object.assign(
new Error(
`Command failed: ps ${args.join(' ')}${result.timedOut ? ' (timed out)' : ''}\n${result.stderr}`
),
{ code: result.code, killed: result.timedOut, signal: result.signal }
)
}
if (result.outputTruncated) {
throw Object.assign(new Error(`ps ${args.join(' ')} exceeded maxBuffer`), {
code: 'ERR_CHILD_PROCESS_STDIO_MAXBUFFER'
})
}
return result.stdout
}
/** execFile-shaped: rejects on a nonzero exit or timeout like a failed ps child; null when ps must run. */
export async function readPsStdoutInProcess(
args: readonly string[],
options: InProcessPsOptions
): Promise<string | null> {
const result = await runPsInProcess(args, options)
return result ? stdoutLikeExecFile(args, result) : null
}
/** execFile-shaped /bin/ps through the shared fork budget, for captures that must run ps (a held naming pass);
* only a ps spawned after this call answers it. */
export async function readPsStdoutBudgeted(
args: readonly string[],
options: {
env: NodeJS.ProcessEnv
timeoutMs: number
maxOutputBytes?: number
background?: boolean
}
): Promise<string> {
const registered = registerPsFallbackRequest()
const result = await runPsFallback(
args,
{ env: options.env, maxOutputBytes: options.maxOutputBytes },
registered,
monotonicNow() + options.timeoutMs,
options.background
)
return stdoutLikeExecFile(args, result)
}