fix(runtime): preserve exited PTY authority across queued graphs

This commit is contained in:
m4air
2026-09-16 02:24:58 -07:00
parent 291b4ddd6f
commit 33cdcb5ddb
9 changed files with 2662 additions and 9 deletions
@@ -0,0 +1,50 @@
# Queued renderer graph restores an exited pane owner
This is a separate source-level explanation for part of [#19018](https://github.com/stablyai/orca/issues/19018). It reproduces an execution host certifying exit, followed by a queued renderer graph restoring that PTY's runtime `connected` flag and making the actual stable-pane resolver throw `terminal_pane_owner_conflict` against the successor's durable binding.
## Run
From the checkout, with installed dependencies:
```sh
ORCA_BACKGROUND_LAUNCH=1 node docs/audits/queued-terminal-graph-exit/reproduce.mjs /tmp/queued-terminal-graph-exit.json
```
The script uses the real renderer graph publisher, main `Store.persistPtyBinding`, runtime, daemon server, adapter, and local sockets. Only the subprocess and the IPC dispatch boundary are controlled. It creates temporary data/socket paths, runs hidden Node tests, and removes its scratch files. It does not launch an Electron window or install dependencies. The JSON records source hashes and excludes randomly allocated terminal handles/incarnations.
## Ordering
1. Publish the mounted predecessor's graph normally.
2. Capture its next unchanged publication at the IPC dispatch boundary. The real publisher sends the mounted leaf, `mobileSessionTabs: []`, and `unchangedMobileSessionWorktrees`.
3. While that publication is queued, spawn a successor and durably bind it to the same tab and leaf.
4. Deliver the predecessor's physical daemon EXIT. The runtime becomes disconnected with an `exited` verdict.
5. Deliver the already-captured graph, without reordering renderer publications.
6. Resolve the pane, query the owning daemon's fresh inventory, and publish once more. A second variant unmounts the renderer terminal before that inventory and remounts it afterward through the real registration/publisher API.
The replacement-binding commit and the daemon EXIT can run while a renderer invocation is queued. The production spawn commit persists the binding before returning its reply (`ipc/pty/ipc/spawn-commit-persist.ts`); the graph publisher reads the mounted pane's transport independently. The fixture controls that ordering; it does not prove its frequency on the reporting machine.
Retirement correctly refuses to delete the successor's durable binding. Before the fix, the retained surface coordinates then allow the old graph leaf to set the predecessor connected again. A healthy inventory contains only the successor, but the runtime sweep skips records that still have a graph leaf. The list's presentation can label that leaf disconnected while the underlying pane resolver still sees it connected.
## Results
| Variant | First queued graph restores predecessor | After unmount, inventory, and remount | Pane conflict |
| --------------- | --------------------------------------- | ------------------------------------- | ----------------- |
| Before | yes | yes | yes |
| Exit check only | no | yes | no at first check |
| Complete fix | no | no | no |
All three variants also run an ordinary-exit control without a replacement; it remains retired throughout. The middle variant isolates why a weak inventory absence during a renderer mount gap must preserve an already-earned exit certificate. Without that mount gap, the retained disconnected leaf keeps the inventory sweep from forgetting the verdict.
The fix reuses the existing liveness verdict registry to keep an exited graph leaf disconnected and nonwritable, and skips recreating its PTY/URL-watcher ownership. It preserves surface membership: a separate actual `stopExactTerminalsForWorktree({ keepHistory: true })` control passes a changed renderer-built mobile snapshot while physical exit has completed but the stop reply is pending. The history surface remains present before and after the renderer clears its PTY binding. All phases check this control; the fixed phase also checks that mobile projection never combines one PTY's ID with another PTY's handle.
Fresh spawn/registration clears the prior verdict; an owning inventory can establish `live`. A physical exit still records its certificate if the bounded PTY archive was already pruned. Host-only tests cover same-ID replacements, stale predecessor EXIT, fresh renderer-only panes, physical negative exit codes, local unverified stops, SSH disconnects, and retained history. The portable proof runs 15 actual-runtime cases across its three source variants.
## Limits and version evidence
The relevant graph admission, unconditional graph-connected write, durable retirement refusal, and inventory leaf exception are present in the reported **v1.4.197**. That tag already passes `providerExitObserved` from local and daemon physical exit callbacks and computes `processDeathCertified`; this proof's natural-exit path does not depend on #21000's synthetic-notification correction. Executable before/after runs use the current checkout with narrowly asserted source transforms, not the complete historical binary.
This proves stale runtime/pane ownership, not a measured native-process or heap leak. The graph alone does not recreate a headless terminal model. It does not establish that every missing diagnostics row denotes an exited process, nor explain all handle-count growth in the issue.
The existing register retains at most 256 unowned verdicts; PTY/handle/leaf owners keep their verdicts until their own lifecycle ends. The disconnected PTY archive is capped at 128. After both a record and its bounded verdict are evicted, the graph has no remaining per-ID certificate; this change does not add permanent tombstones. Later loss-of-contact writes can still replace an exit verdict with `unverifiable`; a stale positive inventory can separately write a connected record. Those paths are not exercised or fixed by this local queued-publication proof.
A separate audit found that an unreachable unrelated legacy daemon can make aggregate exact-stop verification fail despite absence on the target's own daemon. That is excluded from this fix and from the proof's healthy target-inventory assertion.
@@ -0,0 +1,231 @@
import { vi } from 'vitest'
import assert from 'node:assert/strict'
import { mkdtempSync, rmSync } from 'node:fs'
import { join } from 'node:path'
import { tmpdir } from 'node:os'
import { startLateExitHarness } from '../../../src/main/ipc/pty/daemon-late-exit-test-fixture'
import { OrcaRuntimeService } from '../../../src/main/runtime/orca-runtime'
import { Store } from '../../../src/main/persistence/loading-store/store'
import { resolveStablePaneOwner } from '../../../src/main/ipc/pty/pane/stable-owner'
import {
registerRuntimeTerminalTab,
setRuntimeGraphStoreStateGetter,
setRuntimeGraphSyncEnabled
} from '../../../src/renderer/src/runtime/sync-runtime-graph'
import { graphState } from '../../../src/renderer/src/runtime/sync-runtime-graph/graph-state'
import { syncRuntimeGraph } from '../../../src/renderer/src/runtime/sync-runtime-graph/graph-publication'
import { makeState } from '../../../src/renderer/src/runtime/sync-runtime-graph-test-harness'
import { advertisedUrlWatcher } from '../../../src/main/ports/advertised-url-watcher'
class QueuedGraphExitRuntime extends OrcaRuntimeService {
capture(id: string) {
const pty = this.ptysById.get(id)
return {
connected: pty?.connected,
exitCause: pty?.lastExitCause,
incarnationId: pty?.incarnationId,
liveness: this.getPtyLivenessVerdict(id),
model: this.headlessTerminals.has(id),
urlBound: advertisedUrlWatcher['ptyToWorktree'].has(id),
leaves: this.getLeavesForPty(id).map((leaf) => ({
connected: leaf.connected,
writable: leaf.writable
}))
}
}
mobile(worktreeId: string) {
return this.getMobileSessionTabsForWorktree(worktreeId).tabs.flatMap((tab) =>
tab.type === 'terminal'
? [
{
ptyId: tab.ptyId,
handlePtyId: tab.terminal ? this.handles.get(tab.terminal)?.ptyId : null
}
]
: []
)
}
}
export async function runQueuedGraphExitScenario(
replacement: boolean,
graphGapBeforeInventory = false
) {
const h = await startLateExitHarness()
const predecessor = h.subprocess
const dir = mkdtempSync(join(tmpdir(), 'orca-queued-owner-'))
const store = new Store({ dataFile: join(dir, 'orca-data.json') })
const runtime = new QueuedGraphExitRuntime(store)
h.session.runtime = runtime
const WT = 'repo::/tmp/late-exit-audit'
const TAB = '00000000-0000-4000-8000-000000000001'
const LEAF = '00000000-0000-4000-8000-000000000002'
const successorId = `${WT}@@successor`
let unregister: (() => void) | undefined
try {
store.persistPtyBinding({
worktreeId: WT,
tabId: TAB,
leafId: LEAF,
ptyId: h.id,
incarnationId: h.result.incarnationId
})
runtime.registerPty(h.id, WT, null, {
tabId: TAB,
leafId: LEAF,
incarnationId: h.result.incarnationId
})
runtime.setPtyController({
write: () => true,
kill: () => true,
getForegroundProcess: async () => null,
listProcesses: (_connection, opts) => h.adapter.listProcesses(opts),
hasPty: (id) => h.adapter.hasPty(id)
})
const state = makeState({
tabsByWorktree: {
[WT]: [
{
id: TAB,
worktreeId: WT,
title: 'Terminal',
ptyId: h.id,
customTitle: null,
color: null,
sortOrder: 0,
createdAt: 1
}
]
},
terminalLayoutsByTabId: {
[TAB]: {
root: { type: 'leaf', leafId: LEAF },
activeLeafId: LEAF,
expandedLeafId: null,
ptyIdsByLeafId: { [LEAF]: h.id }
}
}
})
const manager = {
getPanes: () => [{ id: 1, leafId: LEAF }],
getActivePane: () => ({ id: 1, leafId: LEAF }),
getLeafId: () => LEAF,
getNumericIdForLeaf: () => 1
}
setRuntimeGraphSyncEnabled(false)
setRuntimeGraphStoreStateGetter(() => state)
vi.stubGlobal('HTMLElement', class HTMLElement {})
let deliver: (() => void) | undefined
let capture: unknown
let queue = false
vi.stubGlobal('window', {
api: {
runtime: {
syncWindowGraph: (graph: never) => {
if (!queue) {
return Promise.resolve(runtime.syncWindowGraph(1, graph))
}
capture = structuredClone(graph)
return new Promise((resolve) => {
deliver = () => resolve(runtime.syncWindowGraph(1, graph))
})
}
}
}
})
const mount = () =>
registerRuntimeTerminalTab({
tabId: TAB,
worktreeId: WT,
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the publisher reads only the four pane lookup methods supplied by this headless manager.
getManager: () => manager as never,
getContainer: () => null,
getPtyIdForPane: () => h.id,
getTabWideAgentHintLeafId: () => null
})
unregister = mount()
const publish = (): Promise<void> => {
graphState.syncEnabled = true
const pending = syncRuntimeGraph()
graphState.syncEnabled = false
return pending
}
await publish()
queue = true
const inFlight = publish()
assert(deliver, 'Publisher did not dispatch its graph')
if (replacement) {
const next = await h.adapter.spawn({ cols: 80, rows: 24, sessionId: successorId })
assert(
store.persistPtyBinding({
worktreeId: WT,
tabId: TAB,
leafId: LEAF,
ptyId: next.id,
incarnationId: next.incarnationId
})
)
runtime.registerPty(next.id, WT, null, {
tabId: TAB,
leafId: LEAF,
incarnationId: next.incarnationId
})
}
predecessor._simulateExit(0)
await h.waitForExit()
const afterExit = runtime.capture(h.id)
assert.equal(afterExit.connected, false)
deliver()
await inFlight
const afterQueuedGraph = runtime.capture(h.id)
let resolution: unknown
try {
resolution = resolveStablePaneOwner(runtime, store, `${TAB}:${LEAF}`, WT, null)
} catch (e) {
resolution = e instanceof Error ? e.message : String(e)
}
queue = false
if (graphGapBeforeInventory) {
unregister()
unregister = undefined
await publish()
}
const list = await runtime.listTerminals()
const afterFreshList = runtime.capture(h.id)
if (graphGapBeforeInventory) {
unregister = mount()
}
await publish()
return {
scenario: replacement ? 'successor-binding-before-exit' : 'ordinary-exit',
graphGapBeforeInventory,
capture,
afterExit,
afterQueuedGraph,
afterFreshList,
afterRepeatedGraph: runtime.capture(h.id),
mobile: runtime.mobile(WT),
resolution,
inventory: await h.adapter.listProcesses(),
persistedPtyId:
store.getWorkspaceSession().terminalLayoutsByTabId[TAB]?.ptyIdsByLeafId?.[LEAF],
listed: list.terminals.map((t) => ({
id: t.ptyId,
connected: t.connected,
tabId: t.tabId,
leafId: t.leafId
}))
}
} finally {
graphState.syncEnabled = false
unregister?.()
setRuntimeGraphSyncEnabled(false)
setRuntimeGraphStoreStateGetter(null)
vi.unstubAllGlobals()
runtime.onPtyExit(h.id, 0)
runtime.onPtyExit(successorId, 0)
await h.dispose()
store.flushOrThrow()
rmSync(dir, { recursive: true, force: true })
}
}
@@ -0,0 +1,180 @@
import { expect, vi } from 'vitest'
import { OrcaRuntimeService } from '../../../src/main/runtime/orca-runtime'
import {
buildMobileSessionTabSnapshots,
registerRuntimeTerminalTab,
setRuntimeGraphStoreStateGetter,
setRuntimeGraphSyncEnabled
} from '../../../src/renderer/src/runtime/sync-runtime-graph'
import { makeState } from '../../../src/renderer/src/runtime/sync-runtime-graph-test-harness'
import { graphState } from '../../../src/renderer/src/runtime/sync-runtime-graph/graph-state'
if (process.env.ORCA_BACKGROUND_LAUNCH !== '1') {
throw new Error('background required')
}
const TAB = '10000000-0000-4000-8000-000000000001'
const TEST_WORKTREE_PATH = '/tmp/worktree-a'
const TEST_WORKTREE_ID = `repo-1::${TEST_WORKTREE_PATH}`
const LEAF = '10000000-0000-4000-8000-000000000002'
const INC = '10000000-0000-4000-8000-000000000003'
const PTY = `${TEST_WORKTREE_ID}@@sleep-review`
export async function runPreservedHistoryScenario() {
const runtime = new OrcaRuntimeService()
const worktree = {
id: TEST_WORKTREE_ID,
path: TEST_WORKTREE_PATH,
repoId: 'repo-1',
name: 'worktree-a',
branch: 'main',
isMain: false
}
vi.spyOn(runtime, 'resolveWorktreeSelector').mockResolvedValue(worktree)
vi.spyOn(runtime, 'getResolvedWorktreeMap').mockResolvedValue(
new Map([[TEST_WORKTREE_ID, worktree]])
)
let stopped = false
let finishStop!: () => void
const stopGate = new Promise<void>((resolve) => {
finishStop = resolve
})
const stop = vi.fn(async () => {
runtime.onPtyExit(PTY, 0, INC, { providerExitObserved: true })
stopped = true
await stopGate
return true
})
runtime.setPtyController({
write: () => true,
kill: () => true,
stopAndWait: stop,
getForegroundProcess: async () => null,
hasPty: () => !stopped,
listProcesses: async () =>
stopped ? [] : [{ id: PTY, cwd: TEST_WORKTREE_PATH, title: 'terminal', incarnationId: INC }]
})
let currentPty: string | null = PTY
const state = makeState({
tabsByWorktree: {
[TEST_WORKTREE_ID]: [
{
id: TAB,
worktreeId: TEST_WORKTREE_ID,
title: 'Terminal',
ptyId: PTY,
customTitle: null,
color: null,
sortOrder: 0,
createdAt: 1
}
]
},
terminalLayoutsByTabId: {
[TAB]: {
root: { type: 'leaf', leafId: LEAF },
activeLeafId: LEAF,
expandedLeafId: null,
ptyIdsByLeafId: { [LEAF]: PTY }
}
}
})
const manager = {
getPanes: () => [{ id: 1, leafId: LEAF }],
getActivePane: () => ({ id: 1, leafId: LEAF }),
getLeafId: () => LEAF,
getNumericIdForLeaf: () => 1
}
setRuntimeGraphSyncEnabled(false)
setRuntimeGraphStoreStateGetter(() => state)
const unregister = registerRuntimeTerminalTab({
tabId: TAB,
worktreeId: TEST_WORKTREE_ID,
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the snapshot builder reads only these four supplied pane lookup methods.
getManager: () => manager as never,
getContainer: () => null,
getPtyIdForPane: () => currentPty,
getTabWideAgentHintLeafId: () => null
})
const publish = () => {
const snapshots = buildMobileSessionTabSnapshots(state)
runtime.syncWindowGraph(1, {
tabs: [
{
tabId: TAB,
worktreeId: TEST_WORKTREE_ID,
title: 'Terminal',
activeLeafId: LEAF,
layout: null
}
],
leaves: [
{
tabId: TAB,
worktreeId: TEST_WORKTREE_ID,
leafId: LEAF,
paneRuntimeId: 1,
ptyId: currentPty
}
],
mobileSessionTabs: snapshots
})
return snapshots[0]
}
const capture = () =>
runtime['mobileSessionTabsByWorktree'].get(TEST_WORKTREE_ID)?.tabs.map((tab) => ({
type: tab.type,
id: tab.id,
ptyId: tab.type === 'terminal' ? tab.ptyId : null
}))
try {
runtime.registerPty(PTY, TEST_WORKTREE_ID, null, {
tabId: TAB,
leafId: LEAF,
incarnationId: INC
})
publish()
const initial = capture()
expect(initial).toHaveLength(1)
const pending = runtime.stopExactTerminalsForWorktree(`id:${TEST_WORKTREE_ID}`, [PTY], {
keepHistory: true,
targetOnly: true
})
await vi.waitFor(() => expect(stop).toHaveBeenCalledOnce())
const afterExit = capture()
expect(afterExit).toHaveLength(1)
state.runtimePaneTitlesByTabId = { [TAB]: { 1: 'Sleeping terminal' } }
const incoming = publish()
const afterQueued = capture()
const queuedLeaf = runtime['leaves'].get(runtime['getLeafKey'](TAB, LEAF))
const leafState = queuedLeaf
? { connected: queuedLeaf.connected, writable: queuedLeaf.writable, ptyId: queuedLeaf.ptyId }
: null
const model = runtime['headlessTerminals'].has(PTY)
finishStop()
const result = await pending
currentPty = null
state.tabsByWorktree[TEST_WORKTREE_ID] = [
{ ...state.tabsByWorktree[TEST_WORKTREE_ID][0], ptyId: null }
]
state.terminalLayoutsByTabId[TAB] = { ...state.terminalLayoutsByTabId[TAB], ptyIdsByLeafId: {} }
publish()
return {
initial,
afterExit,
incoming,
afterQueued,
leafState,
model,
afterBindingClear: capture(),
result
}
} finally {
finishStop()
graphState.syncEnabled = false
unregister()
runtime.onPtyExit(PTY, 0, INC)
setRuntimeGraphStoreStateGetter(null)
vi.restoreAllMocks()
}
}
@@ -0,0 +1,174 @@
import assert from 'node:assert/strict'
import { createHash } from 'node:crypto'
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
import { tmpdir } from 'node:os'
import { join, resolve } from 'node:path'
import { fileURLToPath, pathToFileURL } from 'node:url'
import { startVitest } from 'vitest/node'
if (process.env.ORCA_BACKGROUND_LAUNCH !== '1') {
throw new Error('Run with ORCA_BACKGROUND_LAUNCH=1.')
}
const root = fileURLToPath(new URL('../../../', import.meta.url))
const paths = [
'src/main/runtime/orca-runtime-sync-window-graph.ts',
'src/main/runtime/orca-runtime-mark-pty-liveness-unverifiable.ts',
'src/main/runtime/orca-runtime-on-pty-exit.ts'
]
const sources = await Promise.all(paths.map((path) => readFile(join(root, path), 'utf8')))
const gate = ` // Retained history stays addressable, but a renderer graph cannot revoke a host-certified exit.
const connected = ptyId !== null && this.getPtyLivenessVerdict(ptyId)?.status !== 'exited'
`
const preserve = ` // An inventory's weak absence cannot revoke an earlier host-certified exit.
if (tracked?.verdict.status === 'exited') {
return
}
`
const certificate = ` if (processDeathCertified) {
// The bounded verdict register also fences late graphs after the PTY record was pruned.
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
}
`
for (const [index, text] of [gate, preserve, certificate].entries()) {
assert(sources[index].includes(text), 'Source changed: review the baseline transform.')
}
const baseline = [
sources[0]
.replace(gate, '')
.replace(
" connected,\n writable: this.graphStatus === 'ready' && connected,",
" connected: ptyId !== null,\n writable: this.graphStatus === 'ready' && ptyId !== null,"
)
.replace(' if (leaf.ptyId && connected) {', ' if (leaf.ptyId) {'),
sources[1].replace(preserve, ''),
sources[2].replace(certificate, '').replace(
' pty.lastExitCause = exitCause\n',
` pty.lastExitCause = exitCause
if (exitCode >= 0 || options.hostExitConfirmed === true) {
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
}
`
)
]
const scratch = await mkdtemp(join(tmpdir(), 'orca-queued-graph-proof-'))
const phases = []
try {
for (const phase of ['before', 'guard-only', 'after']) {
const outputPath = join(scratch, `${phase}.json`)
const testPath = join(scratch, `${phase}.test.ts`)
const configPath = join(scratch, `${phase}.config.mjs`)
await writeFile(
testPath,
`
import { afterAll, it } from ${JSON.stringify(join(root, 'node_modules/vitest/dist/index.js'))}
import { writeFileSync } from 'node:fs'
import { runQueuedGraphExitScenario } from ${JSON.stringify(join(root, 'docs/audits/queued-terminal-graph-exit/fixture.ts'))}
import { runPreservedHistoryScenario } from ${JSON.stringify(join(root, 'docs/audits/queued-terminal-graph-exit/preserved-history-fixture.ts'))}
const rows = []
let history
it("preserved history", async () => { history = await runPreservedHistoryScenario() })
for (const successor of [false, true]) {
for (const graphGapBeforeInventory of [false, true]) {
it(String(successor) + String(graphGapBeforeInventory), async () => rows.push(await runQueuedGraphExitScenario(successor, graphGapBeforeInventory)))
}
}
afterAll(() => writeFileSync(${JSON.stringify(outputPath)}, JSON.stringify({ rows, history })))
`
)
const replacements = Object.fromEntries(
paths.map((path, index) => [
`/${path}`,
phase === 'after' || (phase === 'guard-only' && index === 0)
? sources[index]
: baseline[index]
])
)
await writeFile(
configPath,
`
import base from ${JSON.stringify(pathToFileURL(join(root, 'config/vitest.config.ts')).href)}
const replacements = ${JSON.stringify(replacements)}
export default {
...base,
plugins: [{ name: 'graph-exit-baseline', enforce: 'pre', transform(code, id) {
for (const [path, replacement] of Object.entries(replacements)) {
if (id.replaceAll('\\\\', '/').endsWith(path)) return replacement
}
} }],
test: { ...base.test, include: [${JSON.stringify(testPath)}], maxWorkers: 1, fileParallelism: false }
}
`
)
const runner = await startVitest('test', [], {
root,
config: configPath,
watch: false,
reporters: ['dot']
})
assert(runner, 'Vitest did not start')
await runner.close()
const { rows, history } = JSON.parse(await readFile(outputPath, 'utf8'))
assert.equal(history.afterQueued.length, 1)
assert.equal(history.afterBindingClear.length, 1)
assert.equal(history.model, false)
if (phase !== 'before') {
assert.equal(history.leafState.connected, false)
assert.equal(history.leafState.writable, false)
}
delete history.incoming.publicationEpoch
assert.equal(rows.length, 4)
for (const row of rows) {
const successor = row.scenario === 'successor-binding-before-exit'
assert.equal(row.afterExit.connected, false)
assert.equal(row.afterQueuedGraph.connected, successor && phase === 'before')
assert.equal(
row.afterRepeatedGraph.connected,
successor && (phase === 'before' || (phase === 'guard-only' && row.graphGapBeforeInventory))
)
assert.equal(
row.resolution === 'terminal_pane_owner_conflict',
successor && phase === 'before'
)
assert.equal(row.inventory.length, successor ? 1 : 0)
assert.equal(row.afterRepeatedGraph.model, false)
if (phase === 'after') {
assert.equal(row.afterRepeatedGraph.urlBound, false)
assert(row.afterRepeatedGraph.leaves.every((leaf) => !leaf.connected && !leaf.writable))
assert(row.mobile.every((tab) => !tab.handlePtyId || tab.ptyId === tab.handlePtyId))
}
for (const state of [
row.afterExit,
row.afterQueuedGraph,
row.afterFreshList,
row.afterRepeatedGraph
]) {
delete state.incarnationId
}
delete row.capture.rendererGeneration
if (row.resolution && typeof row.resolution === 'object') {
row.resolution = { ptyId: row.resolution.ptyId }
}
row.inventory = row.inventory.map(({ id }) => ({ id }))
}
phases.push({ phase, rows, history })
}
const output = `${JSON.stringify(
{
sources: Object.fromEntries(
paths.map((path, index) => [
path,
createHash('sha256').update(sources[index]).digest('hex')
])
),
phases
},
null,
2
)}\n`
if (process.argv[2]) {
await writeFile(resolve(process.argv[2]), output)
}
process.stdout.write(output)
} finally {
await rm(scratch, { recursive: true, force: true })
}
File diff suppressed because it is too large Load Diff
@@ -137,6 +137,10 @@ export class OrcaRuntimeWithMarkPtyLivenessUnverifiable extends OrcaRuntimeWithO
protected forgetPtyLivenessVerdict(ptyId: string, observedNoLaterThan?: number): void {
const tracked = this.ptyLivenessVerdictByPtyId.get(ptyId)
// An inventory's weak absence cannot revoke an earlier host-certified exit.
if (tracked?.verdict.status === 'exited') {
return
}
if (observedNoLaterThan !== undefined && tracked && tracked.observedAt > observedNoLaterThan) {
return
}
+4 -6
View File
@@ -198,6 +198,10 @@ export class OrcaRuntimeWithOnPtyExit extends OrcaRuntimeWithOnClientDisconnecte
this.terminalDrivers.clear(ptyId)
this.remoteDesktopFloor.clearPty(ptyId)
this.disposeHeadlessTerminal(ptyId)
if (processDeathCertified) {
// The bounded verdict register also fences late graphs after the PTY record was pruned.
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
}
if (pty) {
pty.connected = false
pty.runtimeSessionOwned = false
@@ -205,12 +209,6 @@ export class OrcaRuntimeWithOnPtyExit extends OrcaRuntimeWithOnClientDisconnecte
pty.disconnectedAt = Date.now()
pty.lastExitCode = exitCode
pty.lastExitCause = exitCause
if (exitCode >= 0 || options.hostExitConfirmed === true) {
// Record the certificate rather than merely dropping the doubt: a reader that has to
// authorize a respawn cannot distinguish "the host reported this process gone" from "this
// runtime has never asked" if both are absence.
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
}
// Why: the exited process's live frames say nothing about a replacement.
// A same-id respawn makes the leaf writable again before any new title,
// so leaving this true would let push delivery type into the new process
@@ -107,14 +107,16 @@ export class OrcaRuntimeWithSyncWindowGraph extends OrcaRuntimeWithAttachWindow
? existing.ptyGeneration + 1
: (existing?.ptyGeneration ?? 0)
const existingPty = ptyId ? this.ptysById.get(ptyId) : undefined
// Retained history stays addressable, but a renderer graph cannot revoke a host-certified exit.
const connected = ptyId !== null && this.getPtyLivenessVerdict(ptyId)?.status !== 'exited'
const tailSource = existing?.ptyId === ptyId ? existing : existingPty
nextLeaves.set(leafKey, {
...leaf,
ptyId,
ptyGeneration,
connected: ptyId !== null,
writable: this.graphStatus === 'ready' && ptyId !== null,
connected,
writable: this.graphStatus === 'ready' && connected,
lastOutputAt: tailSource?.lastOutputAt ?? null,
lastExitCode: tailSource?.lastExitCode ?? null,
lastExitCause: tailSource?.lastExitCause ?? null,
@@ -138,7 +140,7 @@ export class OrcaRuntimeWithSyncWindowGraph extends OrcaRuntimeWithAttachWindow
: graphSyncedAt
})
if (leaf.ptyId) {
if (leaf.ptyId && connected) {
this.recordPtyWorktree(leaf.ptyId, leaf.worktreeId, {
connected: true,
lastOutputAt: existing?.ptyId === leaf.ptyId ? existing.lastOutputAt : null,
@@ -0,0 +1,251 @@
import { describe, expect, it, vi } from 'vitest'
import { OrcaRuntimeService } from './orca-runtime'
const WORKTREE = 'repo::/tmp/graph-exit'
const TAB = '10000000-0000-4000-8000-000000000001'
const LEAF = '10000000-0000-4000-8000-000000000002'
const PTY = `${WORKTREE}@@terminal`
const FIRST = '10000000-0000-4000-8000-000000000003'
const NEXT = '10000000-0000-4000-8000-000000000004'
class ExitAuthorityRuntime extends OrcaRuntimeService {
override resolveWorktreeSelector(selector: string) {
return super.resolveWorktreeSelector(selector)
}
override getResolvedWorktreeMap() {
return super.getResolvedWorktreeMap()
}
capture(id = PTY) {
const pty = this.ptysById.get(id)
return { connected: pty?.connected, incarnationId: pty?.incarnationId }
}
get verdictCount(): number {
return this.ptyLivenessVerdictByPtyId.size
}
dropRecord(id = PTY): void {
this.dropDisconnectedPtyRecord(id)
}
history() {
return {
surfaces: this.mobileSessionTabsByWorktree.get(WORKTREE)?.tabs.length,
leaves: this.getLeavesForPty(PTY).map((leaf) => ({
connected: leaf.connected,
writable: leaf.writable
})),
model: this.headlessTerminals.has(PTY)
}
}
}
function graph(
runtime: OrcaRuntimeService,
ptyId: string | null = PTY,
snapshotVersion?: number
): void {
runtime.syncWindowGraph(1, {
tabs: [
{ tabId: TAB, worktreeId: WORKTREE, title: 'terminal', activeLeafId: LEAF, layout: null }
],
leaves: [{ tabId: TAB, worktreeId: WORKTREE, leafId: LEAF, paneRuntimeId: 1, ptyId }],
...(snapshotVersion === undefined
? {}
: {
mobileSessionTabs: [
{
worktree: WORKTREE,
publicationEpoch: 'renderer:retained-history',
snapshotVersion,
activeGroupId: null,
activeTabId: `${TAB}::${LEAF}`,
activeTabType: 'terminal' as const,
tabs: [
{
type: 'terminal' as const,
id: `${TAB}::${LEAF}`,
parentTabId: TAB,
leafId: LEAF,
...(ptyId ? { ptyId } : {}),
title: 'Terminal',
isActive: true
}
]
}
]
})
})
}
function register(runtime: OrcaRuntimeService, incarnationId = FIRST): void {
runtime.registerPty(PTY, WORKTREE, null, { tabId: TAB, leafId: LEAF, incarnationId })
}
describe('host exit authority over queued renderer graphs', () => {
it('keeps exact-stop history addressable through a changed snapshot before binding clears', async () => {
const runtime = new ExitAuthorityRuntime()
const git = {
path: '/tmp/graph-exit',
head: 'abc',
branch: 'main',
isBare: false,
isMainWorktree: false
}
const worktree = {
...git,
git,
id: WORKTREE,
repoId: 'repo',
displayName: 'graph-exit',
comment: '',
linkedIssue: null,
linkedPR: null,
linkedLinearIssue: null,
isArchived: false,
isUnread: false,
isPinned: false,
sortOrder: 0,
lastActivityAt: 0,
parentWorktreeId: null,
childWorktreeIds: [],
lineage: null
}
const resolve = vi.spyOn(runtime, 'resolveWorktreeSelector').mockResolvedValue(worktree)
const map = vi
.spyOn(runtime, 'getResolvedWorktreeMap')
.mockResolvedValue(new Map([[WORKTREE, worktree]]))
let stopped = false
let finishStop!: () => void
const gate = new Promise<void>((done) => {
finishStop = done
})
const stop = vi.fn(async () => {
runtime.onPtyExit(PTY, 0, FIRST, { providerExitObserved: true })
stopped = true
await gate
return true
})
runtime.setPtyController({
write: () => true,
kill: () => true,
stopAndWait: stop,
getForegroundProcess: async () => null,
hasPty: () => !stopped,
listProcesses: async () =>
stopped ? [] : [{ id: PTY, incarnationId: FIRST, cwd: worktree.path, title: 'terminal' }]
})
let pending: Promise<unknown> | undefined
try {
register(runtime)
graph(runtime, PTY, 1)
pending = runtime.stopExactTerminalsForWorktree(`id:${WORKTREE}`, [PTY], {
keepHistory: true,
targetOnly: true
})
await vi.waitFor(() => expect(stop).toHaveBeenCalledOnce())
graph(runtime, PTY, 2)
expect(runtime.history()).toEqual({
surfaces: 1,
leaves: [{ connected: false, writable: false }],
model: false
})
finishStop()
await expect(pending).resolves.toMatchObject({ postStopVerified: true })
graph(runtime, null, 3)
expect(runtime.history().surfaces).toBe(1)
} finally {
finishStop()
await pending
resolve.mockRestore()
map.mockRestore()
runtime.onPtyExit(PTY, 0, FIRST)
}
})
it.each([0, -1])('retains a physical exit certificate after record pruning, code=%s', (code) => {
const runtime = new ExitAuthorityRuntime()
register(runtime)
runtime.dropRecord()
runtime.onPtyExit(PTY, code, FIRST, { providerExitObserved: true })
graph(runtime)
expect(runtime.capture().connected).toBeUndefined()
expect(runtime.getPtyLivenessVerdict(PTY)).toEqual({ status: 'exited' })
})
it('admits a new renderer pane without inventing a host verdict', () => {
const runtime = new ExitAuthorityRuntime()
graph(runtime)
expect(runtime.capture().connected).toBe(true)
expect(runtime.getPtyLivenessVerdict(PTY)).toBeNull()
})
it('admits a registered successor and ignores the predecessor exit', () => {
const runtime = new ExitAuthorityRuntime()
register(runtime)
runtime.onPtyExit(PTY, 0, FIRST)
register(runtime, NEXT)
runtime.onPtyExit(PTY, 0, FIRST)
graph(runtime)
expect(runtime.capture()).toEqual({ connected: true, incarnationId: NEXT })
expect(runtime.getPtyLivenessVerdict(PTY)).toBeNull()
})
it('admits a same-ID spawn before its registration commits', () => {
const runtime = new ExitAuthorityRuntime()
register(runtime)
runtime.onPtyExit(PTY, 0, FIRST)
runtime.onPtySpawned(PTY, NEXT)
graph(runtime)
expect(runtime.capture()).toEqual({ connected: true, incarnationId: NEXT })
expect(runtime.getPtyLivenessVerdict(PTY)).toBeNull()
})
it('allows owning inventory to prove the same ID live again', async () => {
const runtime = new ExitAuthorityRuntime()
register(runtime)
runtime.onPtyExit(PTY, 0, FIRST)
runtime.setPtyController({
write: () => true,
kill: () => true,
getForegroundProcess: async () => null,
listProcesses: async () => [{ id: PTY, incarnationId: NEXT, cwd: '', title: 'terminal' }]
})
await runtime.listTerminals()
graph(runtime)
expect(runtime.capture()).toEqual({ connected: true, incarnationId: NEXT })
expect(runtime.getPtyLivenessVerdict(PTY)?.status).toBe('live')
})
it('keeps an SSH disconnect unverifiable and admissible', () => {
const runtime = new ExitAuthorityRuntime()
const id = 'ssh:target@@terminal'
runtime.registerPty(id, WORKTREE, 'target', { tabId: TAB, leafId: LEAF, incarnationId: FIRST })
runtime.onPtyExit(id, -1, FIRST)
graph(runtime, id)
expect(runtime.getPtyLivenessVerdict(id)?.status).toBe('unverifiable')
expect(runtime.capture(id).connected).toBe(true)
})
it('does not promote an unverified local stop to an exit certificate', () => {
const runtime = new ExitAuthorityRuntime()
runtime.registerPty(PTY, WORKTREE)
runtime.onPtyExit(PTY, -1, FIRST)
runtime.markPtyLivenessUnverifiable(PTY, 'stop unverified')
graph(runtime)
expect(runtime.getPtyLivenessVerdict(PTY)?.status).toBe('unverifiable')
expect(runtime.capture().connected).toBe(true)
})
it('bounds certificates for exits whose PTY records are already gone', () => {
const runtime = new ExitAuthorityRuntime()
for (let index = 0; index < 1_000; index++) {
runtime.onPtyExit(`${PTY}-${index}`, 0)
}
expect(runtime.verdictCount).toBe(256)
expect(runtime.getPtyLivenessVerdict(`${PTY}-0`)).toBeNull()
expect(runtime.getPtyLivenessVerdict(`${PTY}-999`)).toEqual({ status: 'exited' })
})
})