Files
orca/src/shared/shell-process-readiness.ts
T
Neil 756d623955 perf(terminal): bound the in-process macOS process table's blocking calls and ps fallbacks (#31783) (#36282)
* fix(terminal): bound the in-process macOS process table's blocking calls and ps fallbacks (#31783)

Follow-up to #35987 from its post-merge reviews.

- cwd and executable path: node-pty's readProcessCwd/readProcessPath now run
  PROC_PIDVNODEPATHINFO and proc_pidpath on a libuv worker (the synchronous
  exports are gone, so an older pty.node falls back to ps/lsof). JS caps them at
  two outstanding lookups and 250ms; a wedged mount or a full cap answers from
  lsof -b, which never stats, so a dead NFS/SMB/FUSE mount cannot freeze the
  daemon's event loop or take the whole pool.
- Unreadable argv prints "(comm)", as ps does, instead of a proc_pidpath lookup
  under Orca's credentials on the table worker.
- The table exports each row's euid. Argv recovery runs setuid ps only for
  another user's processes, never for our own mid-exec or exiting ones (or
  SIDL/exiting rows), which ps cannot read either: a pane looping short
  commands forked ps on ~15% of full reads, now none.
- Recovery is coalesced per process across concurrent readers and a failed
  recovery waits 5s before costing another ps. The cache keys on comm and start
  time.
- Full-table and shell-foreground captures, which feed strict evidence with
  the capture's age, take only argv ps read after their request; display
  readers keep the 30s cache.
- While the reader is down (stalled-pool breaker open, or a rejecting addon,
  which now backs off 1s..30s), runPsInProcess runs /bin/ps itself through one
  shared budget: identical invocations share a run, a burst of 4 forks then one
  a second, and a run spawned before a request never answers it. Descendant
  teardown and foreground-group signalling keep their own full-timeout ps
  (callerRunsPs). 13 pollers at 750ms: 1,040 ps/min before, 63 after.

No daemon protocol bump: the wire format is unchanged, and each daemon loads
JS and pty.node from the same runtime copy.

* fix(terminal): make the libproc reader survive worker teardown, fail closed and share one fair fork budget (#31783)

Review follow-ups to the previous commit.

- Native: every libproc AsyncWorker derives from OrcaPromiseWorker, which skips
  JS completion once an env cleanup hook marks the env closing and swallows a
  Napi::Error thrown mid-teardown. Terminating a worker_thread with cwd or table
  reads in flight aborted the process; 200 terminates each now survive.
- (comm) in a terminal's foreground group (argv neither reader recovered, e.g.
  a root `su`) judges 'unknown', never 'agent', so the launch line is not typed
  into it.
- Fresh argv only for getFreshSnapshot captures; the 500ms TTL readers keep the
  30s cache. Argv recovery runs through its own fork budget (2, then one per
  2s). Root `su` pane, 13 pollers at 750ms: TTL 79 -> 2 ps/min, fresh 158 -> 31.
- The fork budget serves invocations in FIFO turns, grouping each one's waiters
  onto one run. 13 streams: served 80,42,1,1,2,0x8 -> 10-12 each.
- Forced subprocess captures (a held naming pass) go through the fallback
  budget: 5 readers, 400 direct ps/min -> 63, 395 of 400 captures served.
- A failed recovery is retried 5s after ps gave up, not after it started.
- Tests pin Orca's euid instead of reading the runner's and assuming root is 0.
- process-cwd keeps one lsof per pid until it exits, so a dead mount no longer
  leaks one per poll; shell readiness reads the exe path through runProcess,
  which gives up on an lsof that ignores SIGTERM.
- A #35987 pty.node (table reader, no async path reads) keeps the in-process
  table; cwd/path use lsof, with a one-time warning.
- captureProcessTable moves to process-table-ps-capture.ts.

* fix(terminal): make withheld argv unverifiable and reuse recovered argv for at most 2s (#31783)

Round-2 review follow-ups.

- Worker teardown: drop OrcaPromiseWorker and its env flag (Node drains the
  pool before cleanup hooks, so it never fired). node-pty's mac pty target now
  builds with NODE_API_SWALLOW_UNTHROWABLE_EXCEPTIONS, which covers both the
  call-in and completion paths. The smoke test now calls the reads from a
  timer while terminating: the previous build aborted 2 of 3 runs, this one
  survives 3/3 x 100. pty.cc is back to the first follow-up's exact content.
- A foreground process whose argv ps withheld (`(comm)`: another user's,
  exiting, or a recovery the budget skipped) makes the batch resolver, and so
  host-stamped evidence, `argv_unreadable`/unverifiable rather than live with no
  agent; agent-foreground resolution reports it unavailable.
- judgeTerminalForeground fails closed only for an unreadable shell (`(bash)`
  from a root `su`), not our own exiting `(git)`.
- Recovered argv is reused for at most 2s per identity (pid, start, comm) for
  every reader (the next commit ages evidence by it), and
  fresh reads no longer queue behind the recovery budget. Root `su` pane, 13
  fresh pollers: p50 2,989ms -> 53ms, 31 -> 26 ps/min (TTL reader 1 -> 26).
- The fork budgets serve interactive and fresh readers before display polls
  (cheap and TTL snapshot captures), FIFO within each. 20 display streams plus
  a 3s launch-paste read every 5s: 0/12 -> 12/12 served, polls 4-7 each.
- On a selected terminal, the foreground group's argv is recovered even with no
  process of ours left (`exec su`); the root login is not.
- Budgeted captures keep the 32 MiB output ceiling.

* fix(terminal): age strict evidence by its oldest recovered argv (#31783)

Round-3 review follow-ups.

- Recovered argv carried no read time, so a capture reusing it (cached, or a
  recovery still in flight) stamped its own capture time over argv up to 2s
  older, and the remote write guard could take a same-comm re-exec for the
  agent. Rows now carry when ps read their recovered argv; runPsInProcess
  reports the oldest through onArgvReadAt, and the snapshot reader stamps
  capturedAtMs at that input, so capturedAgeMs is the age of the oldest input
  and consumers' own freshness checks (the write guard's 500ms) decide. A
  recovery begun more than 2s before a reader asked no longer answers it, and
  an older recovery finishing late cannot replace a newer one.
- resolveSpawnFileForegroundFromRows reports a withheld `(node)` foreground
  unavailable.
- One budget queue entry per invocation: an interactive request promotes a
  queued poll's, a poll joins an interactive one.
- parseShellForegroundRows gets its JSDoc back.

* fix(terminal): keep a cached name from authorizing a refused write; integer, under-2s argv ages (#31783)

Round-4 review follow-ups.

- When the relay's evidence declines a write (stale, unverifiable, another
  agent), its foreground name may still prove the shell but no longer proves
  the agent: that name can be the tracker's 1s display cache, e.g. `claude`
  cached just before Claude exited to a root shell under su. Relays that send
  no evidence keep the name as before.
- Snapshot ages round up to whole milliseconds; the wire rejects fractions
  from the monotonic-to-wall conversion.
- Recovered argv is reused for at most 1.5s, so reuse plus the 500ms snapshot
  cache stays under the 2s freshness gate; reuse ages from when that ps
  spawned, not from the wait for its fork token, and only an answer is
  remembered (a timed-out or withheld run proves nothing). Fork budget
  unchanged. Root su pane, 13 pollers: 30 ps/min; fresh p50 51ms; strict
  evidence age p50 752ms, max 1,508ms.

* fix(terminal): keep the relay's agent name behind structurally unverifiable evidence (#31783)

The previous commit refused the name fallback whenever the relay sent any
evidence, so `unverifiable` evidence for structural reasons also blocked
writes: npm Codex (node codex.js spawning native codex) and Gemini (relaunching
itself) put two agent rows in the foreground group (ambiguous_foreground_group),
and remote tmux reads multiplexer_boundary. On SSH POSIX hosts that dropped
launch prompts and failed worker briefs, conversation send/cancel and
automations with agent_not_in_foreground.

The name now cannot authorize only when the evidence saw the group (`live`:
stale, or no agent in it, as in the cached-`claude`-over-root-shell repro) or
withheld a foreground argv (`argv_unreadable`, possibly a root `su` shell).
Other unverifiable reasons keep the name fallback, as before this branch.

* fix(terminal): let a relay name complete only fresh agent-positive evidence (#31783)

The SSH name fallback still let a stale or cached name authorize a write under
any unverifiable reason but argv_unreadable: ps EAGAIN (process_table_unreadable)
after the tracker cached `claude` over a root bash, anchor_missing,
tty_boundary (`sudo -i` with use_pty, a root shell on a new pty), or a 1.5s-old
ambiguous capture.

- The name now completes evidence only when it is fresh (within the guard's
  500ms bound), its reason is one that saw an agent row
  (ambiguous_foreground_group, multiplexer_boundary,
  candidate_start_time_missing), and the name is the launched agent. Every other
  reason, including ones this build does not know, reads unknown. Relays that
  send no evidence keep the name, for mixed versions.
- The relay's evidence names one agent that holds two rows of the group (npm
  Codex: node codex.js spawning native codex; Gemini relaunching itself), so a
  new relay proves them live even when its name is `node`. Two different agents
  stay ambiguous. No wire change.

* fix(terminal): judge a relay name by the host's capture age; keep old relays' npm Codex (#31783)

- Name completion compared the capture's start with the read's start, so a
  slow response made old evidence look fresh: a capture 1,500ms old arriving
  after 1,100ms passed. It now uses the host-reported capturedAgeMs against
  the 500ms bound, which latency cannot shrink. Live evidence keeps its
  capture-start rule.
- Relays before same-agent dedupe report npm Codex (node wrapper plus native
  binary) as ambiguous_foreground_group and name it `node`. For that reason
  only, `node` now completes the evidence when the launched agent is
  npm-installed per the node entrypoint tables; other agents and reasons stay
  fail-closed.

* fix(terminal): bound relay-name completion by twice the snapshot TTL (#31783)

The 500ms bound refused ~9.5% of tmux name completions on a host with 13
pollers (cached snapshot up to one 500ms TTL old plus ~30ms capture, max
528ms), and a refusal drops the launch prompt or throws with no retry. The
bound is now 2 x PROCESS_TABLE_SNAPSHOT_MAX_STALENESS_MS (1s), still on the
host-reported age: 0 of 767 reads exceed it, the delayed 1,500ms repro still
refuses, and the cached-name hazard (the tracker's ~1s display cache) is no
wider.

* fix(terminal): complete relay evidence with a name read now, not the display cache (#31783)

With the 1s bound, a capture at 0-528ms, `codex` cached at 100ms, Codex
exiting at 101ms and the guard at 999ms still let the cached name authorize a
write over the bash that replaced it.

- Name completion (structural reasons, host age within 2x the snapshot TTL)
  now asks confirmForegroundProcess, which reads the foreground now (a fresh
  scan, or node-pty's raw process name) rather than the tracker's 1s display
  cache; a null answer reads unknown. No new wire surface: the local daemon answers
  confirmForegroundProcess, paired runtimes evaluate guards locally, and the
  production controller returns null (read as unknown, failing safe) when a
  provider lacks it. Controllers without the method, and relays sending no
  evidence, keep the cached name as the legacy path.
- Live evidence also accepts a host-reported age within 2x the TTL, so a real
  agent behind a 500-528ms cached snapshot on a fast link is no longer
  refused; a capture begun after the read was asked still counts, and
  5,000ms-old evidence still refuses.

* fix(terminal): confirm live evidence older than one snapshot TTL with a fresh read (#31783)

Accepting live evidence up to 2x the TTL reopened the window the previous
commit closed for name completion: a capture at 0-528ms, the agent exiting at
~529ms, and at 999ms `live claude` still read as agent over a bash.

Live evidence begun no more than one TTL before the read was asked (so any
host age within one TTL, however long ps took) still answers alone. Between
that and 2x the TTL, confirmForegroundProcess reads the foreground now: the
launched agent proves it, a shell reads shell, anything else unknown.
Controllers without that read fall through and refuse, as before.

Reachability: the remote (host.remote) write-guard path has no production
caller since the SSH provider's removal (#27499); the registry dispatches only
local providers and paired runtimes evaluate guards locally. This is defence
in depth for any future remote provider.
2026-10-30 01:05:34 -07:00

145 lines
4.9 KiB
TypeScript

import { execFile as execFileCallback } from 'node:child_process'
import { constants } from 'node:fs'
import { access, readlink, realpath, stat } from 'node:fs/promises'
import { delimiter, isAbsolute, resolve } from 'node:path'
import { promisify } from 'node:util'
import { runProcess } from '@orca/process-host'
import { LSOF_NONBLOCKING_ARGS } from './lsof-nonblocking-args'
import { readLibprocProcessFilePath } from './libproc-addon'
import { readPsStdoutInProcess } from './libproc-process-table'
import { PS_MAX_BUFFER_BYTES } from './process-table-snapshot'
const execFile = promisify(execFileCallback)
const PROCESS_READINESS_TIMEOUT_MS = 3000
const DEFAULT_POSIX_EXEC_PATH = '/usr/bin:/bin'
export type ShellProcessReadiness = {
executablePath: string
foreground: boolean
ttyPath: string | null
}
export function parseDarwinExecutablePath(stdout: string): string | null {
const lines = stdout.split(/\r?\n/)
const textIndex = lines.indexOf('ftxt')
const pathLine = textIndex === -1 ? undefined : lines[textIndex + 1]
return pathLine?.startsWith('n') ? pathLine.slice(1) : null
}
async function readExecutablePath(pid: number): Promise<string | null> {
if (process.platform === 'linux') {
return readlink(`/proc/${pid}/exe`)
}
if (process.platform !== 'darwin') {
return null
}
const inProcess = await readLibprocProcessFilePath('executable', pid)
if (inProcess !== undefined) {
return inProcess
}
// execFile settles only once lsof exits, and SIGTERM cannot interrupt it in a dead mount; runProcess gives up.
const result = await runProcess({
program: '/usr/sbin/lsof',
args: [...LSOF_NONBLOCKING_ARGS, '-a', '-p', String(pid), '-d', 'txt', '-Fn'],
timeoutMs: PROCESS_READINESS_TIMEOUT_MS
})
if (result.timedOut || result.code !== 0) {
throw new Error(`lsof -p ${pid} failed${result.timedOut ? ' (timed out)' : ''}`)
}
return parseDarwinExecutablePath(result.stdout)
}
/** Rejects like execFile when ps exits nonzero, so callers keep their error paths. */
async function readPs(args: string[]): Promise<string> {
const inProcess = await readPsStdoutInProcess(args, {
timeoutMs: PROCESS_READINESS_TIMEOUT_MS,
maxOutputBytes: PS_MAX_BUFFER_BYTES
})
if (inProcess !== null) {
return inProcess
}
const { stdout } = await execFile('ps', args, {
encoding: 'utf8',
timeout: PROCESS_READINESS_TIMEOUT_MS,
maxBuffer: PS_MAX_BUFFER_BYTES
})
return stdout
}
export async function readShellProcessReadiness(
pid: number
): Promise<ShellProcessReadiness | null> {
const [executablePath, stdout] = await Promise.all([
readExecutablePath(pid),
readPs(['-p', String(pid), '-o', 'stat=,tty='])
])
const [status, ttyName] = stdout.trim().split(/\s+/)
return status && executablePath
? {
executablePath: await realpath(executablePath),
foreground: status.includes('+'),
ttyPath: ttyName && ttyName !== '?' && ttyName !== '??' ? `/dev/${ttyName}` : null
}
: null
}
export async function listChildProcessIds(pid: number): Promise<number[]> {
const stdout = await readPs(['-axo', 'pid=,ppid='])
return stdout
.split('\n')
.map((line) => line.trim().split(/\s+/))
.filter(([childPid, parentPid]) => childPid && parentPid === String(pid))
.map(([childPid]) => Number(childPid))
}
function shellExecutableCandidates(
shellPath: string,
cwd: string,
pathEnv: string | undefined
): string[] {
return shellPath.includes('/')
? [isAbsolute(shellPath) ? shellPath : resolve(cwd, shellPath)]
: (
pathEnv ??
(process.platform === 'win32' ? (process.env.PATH ?? '') : DEFAULT_POSIX_EXEC_PATH)
)
.split(delimiter)
.map((entry) => resolve(isAbsolute(entry) ? entry : resolve(cwd, entry), shellPath))
}
async function canonicalizeExecutable(candidate: string): Promise<string | null> {
try {
await access(candidate, constants.X_OK)
const canonicalPath = await realpath(candidate)
return (await stat(canonicalPath)).isFile() ? canonicalPath : null
} catch {
return null
}
}
export async function resolveShellExecutablePath(
shellPath: string,
cwd: string,
pathEnv: string | undefined
): Promise<string | null> {
for (const candidate of shellExecutableCandidates(shellPath, cwd, pathEnv)) {
const canonicalPath = await canonicalizeExecutable(candidate)
if (canonicalPath) {
return canonicalPath
}
}
return null
}
/** Resolve all PATH-installed canonical shell executables a startup profile could exec into; reject non-bare shellName inputs. */
export async function resolveInstalledShellExecutablePaths(
shellName: string,
cwd: string,
pathEnv: string | undefined
): Promise<string[]> {
const canonicalPaths = await Promise.all(
shellExecutableCandidates(shellName, cwd, pathEnv).map(canonicalizeExecutable)
)
return [...new Set(canonicalPaths.filter((path): path is string => path !== null))]
}