mirror of
https://github.com/stablyai/orca.git
synced 2026-09-23 16:02:24 +00:00
fix(runtime): preserve exited PTY authority across queued graphs
This commit is contained in:
@@ -0,0 +1,50 @@
|
||||
# Queued renderer graph restores an exited pane owner
|
||||
|
||||
This is a separate source-level explanation for part of [#19018](https://github.com/stablyai/orca/issues/19018). It reproduces an execution host certifying exit, followed by a queued renderer graph restoring that PTY's runtime `connected` flag and making the actual stable-pane resolver throw `terminal_pane_owner_conflict` against the successor's durable binding.
|
||||
|
||||
## Run
|
||||
|
||||
From the checkout, with installed dependencies:
|
||||
|
||||
```sh
|
||||
ORCA_BACKGROUND_LAUNCH=1 node docs/audits/queued-terminal-graph-exit/reproduce.mjs /tmp/queued-terminal-graph-exit.json
|
||||
```
|
||||
|
||||
The script uses the real renderer graph publisher, main `Store.persistPtyBinding`, runtime, daemon server, adapter, and local sockets. Only the subprocess and the IPC dispatch boundary are controlled. It creates temporary data/socket paths, runs hidden Node tests, and removes its scratch files. It does not launch an Electron window or install dependencies. The JSON records source hashes and excludes randomly allocated terminal handles/incarnations.
|
||||
|
||||
## Ordering
|
||||
|
||||
1. Publish the mounted predecessor's graph normally.
|
||||
2. Capture its next unchanged publication at the IPC dispatch boundary. The real publisher sends the mounted leaf, `mobileSessionTabs: []`, and `unchangedMobileSessionWorktrees`.
|
||||
3. While that publication is queued, spawn a successor and durably bind it to the same tab and leaf.
|
||||
4. Deliver the predecessor's physical daemon EXIT. The runtime becomes disconnected with an `exited` verdict.
|
||||
5. Deliver the already-captured graph, without reordering renderer publications.
|
||||
6. Resolve the pane, query the owning daemon's fresh inventory, and publish once more. A second variant unmounts the renderer terminal before that inventory and remounts it afterward through the real registration/publisher API.
|
||||
|
||||
The replacement-binding commit and the daemon EXIT can run while a renderer invocation is queued. The production spawn commit persists the binding before returning its reply (`ipc/pty/ipc/spawn-commit-persist.ts`); the graph publisher reads the mounted pane's transport independently. The fixture controls that ordering; it does not prove its frequency on the reporting machine.
|
||||
|
||||
Retirement correctly refuses to delete the successor's durable binding. Before the fix, the retained surface coordinates then allow the old graph leaf to set the predecessor connected again. A healthy inventory contains only the successor, but the runtime sweep skips records that still have a graph leaf. The list's presentation can label that leaf disconnected while the underlying pane resolver still sees it connected.
|
||||
|
||||
## Results
|
||||
|
||||
| Variant | First queued graph restores predecessor | After unmount, inventory, and remount | Pane conflict |
|
||||
| --------------- | --------------------------------------- | ------------------------------------- | ----------------- |
|
||||
| Before | yes | yes | yes |
|
||||
| Exit check only | no | yes | no at first check |
|
||||
| Complete fix | no | no | no |
|
||||
|
||||
All three variants also run an ordinary-exit control without a replacement; it remains retired throughout. The middle variant isolates why a weak inventory absence during a renderer mount gap must preserve an already-earned exit certificate. Without that mount gap, the retained disconnected leaf keeps the inventory sweep from forgetting the verdict.
|
||||
|
||||
The fix reuses the existing liveness verdict registry to keep an exited graph leaf disconnected and nonwritable, and skips recreating its PTY/URL-watcher ownership. It preserves surface membership: a separate actual `stopExactTerminalsForWorktree({ keepHistory: true })` control passes a changed renderer-built mobile snapshot while physical exit has completed but the stop reply is pending. The history surface remains present before and after the renderer clears its PTY binding. All phases check this control; the fixed phase also checks that mobile projection never combines one PTY's ID with another PTY's handle.
|
||||
|
||||
Fresh spawn/registration clears the prior verdict; an owning inventory can establish `live`. A physical exit still records its certificate if the bounded PTY archive was already pruned. Host-only tests cover same-ID replacements, stale predecessor EXIT, fresh renderer-only panes, physical negative exit codes, local unverified stops, SSH disconnects, and retained history. The portable proof runs 15 actual-runtime cases across its three source variants.
|
||||
|
||||
## Limits and version evidence
|
||||
|
||||
The relevant graph admission, unconditional graph-connected write, durable retirement refusal, and inventory leaf exception are present in the reported **v1.4.197**. That tag already passes `providerExitObserved` from local and daemon physical exit callbacks and computes `processDeathCertified`; this proof's natural-exit path does not depend on #21000's synthetic-notification correction. Executable before/after runs use the current checkout with narrowly asserted source transforms, not the complete historical binary.
|
||||
|
||||
This proves stale runtime/pane ownership, not a measured native-process or heap leak. The graph alone does not recreate a headless terminal model. It does not establish that every missing diagnostics row denotes an exited process, nor explain all handle-count growth in the issue.
|
||||
|
||||
The existing register retains at most 256 unowned verdicts; PTY/handle/leaf owners keep their verdicts until their own lifecycle ends. The disconnected PTY archive is capped at 128. After both a record and its bounded verdict are evicted, the graph has no remaining per-ID certificate; this change does not add permanent tombstones. Later loss-of-contact writes can still replace an exit verdict with `unverifiable`; a stale positive inventory can separately write a connected record. Those paths are not exercised or fixed by this local queued-publication proof.
|
||||
|
||||
A separate audit found that an unreachable unrelated legacy daemon can make aggregate exact-stop verification fail despite absence on the target's own daemon. That is excluded from this fix and from the proof's healthy target-inventory assertion.
|
||||
@@ -0,0 +1,231 @@
|
||||
import { vi } from 'vitest'
|
||||
import assert from 'node:assert/strict'
|
||||
import { mkdtempSync, rmSync } from 'node:fs'
|
||||
import { join } from 'node:path'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { startLateExitHarness } from '../../../src/main/ipc/pty/daemon-late-exit-test-fixture'
|
||||
import { OrcaRuntimeService } from '../../../src/main/runtime/orca-runtime'
|
||||
import { Store } from '../../../src/main/persistence/loading-store/store'
|
||||
import { resolveStablePaneOwner } from '../../../src/main/ipc/pty/pane/stable-owner'
|
||||
import {
|
||||
registerRuntimeTerminalTab,
|
||||
setRuntimeGraphStoreStateGetter,
|
||||
setRuntimeGraphSyncEnabled
|
||||
} from '../../../src/renderer/src/runtime/sync-runtime-graph'
|
||||
import { graphState } from '../../../src/renderer/src/runtime/sync-runtime-graph/graph-state'
|
||||
import { syncRuntimeGraph } from '../../../src/renderer/src/runtime/sync-runtime-graph/graph-publication'
|
||||
import { makeState } from '../../../src/renderer/src/runtime/sync-runtime-graph-test-harness'
|
||||
import { advertisedUrlWatcher } from '../../../src/main/ports/advertised-url-watcher'
|
||||
|
||||
class QueuedGraphExitRuntime extends OrcaRuntimeService {
|
||||
capture(id: string) {
|
||||
const pty = this.ptysById.get(id)
|
||||
return {
|
||||
connected: pty?.connected,
|
||||
exitCause: pty?.lastExitCause,
|
||||
incarnationId: pty?.incarnationId,
|
||||
liveness: this.getPtyLivenessVerdict(id),
|
||||
model: this.headlessTerminals.has(id),
|
||||
urlBound: advertisedUrlWatcher['ptyToWorktree'].has(id),
|
||||
leaves: this.getLeavesForPty(id).map((leaf) => ({
|
||||
connected: leaf.connected,
|
||||
writable: leaf.writable
|
||||
}))
|
||||
}
|
||||
}
|
||||
|
||||
mobile(worktreeId: string) {
|
||||
return this.getMobileSessionTabsForWorktree(worktreeId).tabs.flatMap((tab) =>
|
||||
tab.type === 'terminal'
|
||||
? [
|
||||
{
|
||||
ptyId: tab.ptyId,
|
||||
handlePtyId: tab.terminal ? this.handles.get(tab.terminal)?.ptyId : null
|
||||
}
|
||||
]
|
||||
: []
|
||||
)
|
||||
}
|
||||
}
|
||||
export async function runQueuedGraphExitScenario(
|
||||
replacement: boolean,
|
||||
graphGapBeforeInventory = false
|
||||
) {
|
||||
const h = await startLateExitHarness()
|
||||
const predecessor = h.subprocess
|
||||
const dir = mkdtempSync(join(tmpdir(), 'orca-queued-owner-'))
|
||||
const store = new Store({ dataFile: join(dir, 'orca-data.json') })
|
||||
const runtime = new QueuedGraphExitRuntime(store)
|
||||
h.session.runtime = runtime
|
||||
const WT = 'repo::/tmp/late-exit-audit'
|
||||
const TAB = '00000000-0000-4000-8000-000000000001'
|
||||
const LEAF = '00000000-0000-4000-8000-000000000002'
|
||||
const successorId = `${WT}@@successor`
|
||||
let unregister: (() => void) | undefined
|
||||
try {
|
||||
store.persistPtyBinding({
|
||||
worktreeId: WT,
|
||||
tabId: TAB,
|
||||
leafId: LEAF,
|
||||
ptyId: h.id,
|
||||
incarnationId: h.result.incarnationId
|
||||
})
|
||||
runtime.registerPty(h.id, WT, null, {
|
||||
tabId: TAB,
|
||||
leafId: LEAF,
|
||||
incarnationId: h.result.incarnationId
|
||||
})
|
||||
runtime.setPtyController({
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null,
|
||||
listProcesses: (_connection, opts) => h.adapter.listProcesses(opts),
|
||||
hasPty: (id) => h.adapter.hasPty(id)
|
||||
})
|
||||
const state = makeState({
|
||||
tabsByWorktree: {
|
||||
[WT]: [
|
||||
{
|
||||
id: TAB,
|
||||
worktreeId: WT,
|
||||
title: 'Terminal',
|
||||
ptyId: h.id,
|
||||
customTitle: null,
|
||||
color: null,
|
||||
sortOrder: 0,
|
||||
createdAt: 1
|
||||
}
|
||||
]
|
||||
},
|
||||
terminalLayoutsByTabId: {
|
||||
[TAB]: {
|
||||
root: { type: 'leaf', leafId: LEAF },
|
||||
activeLeafId: LEAF,
|
||||
expandedLeafId: null,
|
||||
ptyIdsByLeafId: { [LEAF]: h.id }
|
||||
}
|
||||
}
|
||||
})
|
||||
const manager = {
|
||||
getPanes: () => [{ id: 1, leafId: LEAF }],
|
||||
getActivePane: () => ({ id: 1, leafId: LEAF }),
|
||||
getLeafId: () => LEAF,
|
||||
getNumericIdForLeaf: () => 1
|
||||
}
|
||||
setRuntimeGraphSyncEnabled(false)
|
||||
setRuntimeGraphStoreStateGetter(() => state)
|
||||
vi.stubGlobal('HTMLElement', class HTMLElement {})
|
||||
let deliver: (() => void) | undefined
|
||||
let capture: unknown
|
||||
let queue = false
|
||||
vi.stubGlobal('window', {
|
||||
api: {
|
||||
runtime: {
|
||||
syncWindowGraph: (graph: never) => {
|
||||
if (!queue) {
|
||||
return Promise.resolve(runtime.syncWindowGraph(1, graph))
|
||||
}
|
||||
capture = structuredClone(graph)
|
||||
return new Promise((resolve) => {
|
||||
deliver = () => resolve(runtime.syncWindowGraph(1, graph))
|
||||
})
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
const mount = () =>
|
||||
registerRuntimeTerminalTab({
|
||||
tabId: TAB,
|
||||
worktreeId: WT,
|
||||
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the publisher reads only the four pane lookup methods supplied by this headless manager.
|
||||
getManager: () => manager as never,
|
||||
getContainer: () => null,
|
||||
getPtyIdForPane: () => h.id,
|
||||
getTabWideAgentHintLeafId: () => null
|
||||
})
|
||||
unregister = mount()
|
||||
const publish = (): Promise<void> => {
|
||||
graphState.syncEnabled = true
|
||||
const pending = syncRuntimeGraph()
|
||||
graphState.syncEnabled = false
|
||||
return pending
|
||||
}
|
||||
await publish()
|
||||
queue = true
|
||||
const inFlight = publish()
|
||||
assert(deliver, 'Publisher did not dispatch its graph')
|
||||
if (replacement) {
|
||||
const next = await h.adapter.spawn({ cols: 80, rows: 24, sessionId: successorId })
|
||||
assert(
|
||||
store.persistPtyBinding({
|
||||
worktreeId: WT,
|
||||
tabId: TAB,
|
||||
leafId: LEAF,
|
||||
ptyId: next.id,
|
||||
incarnationId: next.incarnationId
|
||||
})
|
||||
)
|
||||
runtime.registerPty(next.id, WT, null, {
|
||||
tabId: TAB,
|
||||
leafId: LEAF,
|
||||
incarnationId: next.incarnationId
|
||||
})
|
||||
}
|
||||
predecessor._simulateExit(0)
|
||||
await h.waitForExit()
|
||||
const afterExit = runtime.capture(h.id)
|
||||
assert.equal(afterExit.connected, false)
|
||||
deliver()
|
||||
await inFlight
|
||||
const afterQueuedGraph = runtime.capture(h.id)
|
||||
let resolution: unknown
|
||||
try {
|
||||
resolution = resolveStablePaneOwner(runtime, store, `${TAB}:${LEAF}`, WT, null)
|
||||
} catch (e) {
|
||||
resolution = e instanceof Error ? e.message : String(e)
|
||||
}
|
||||
queue = false
|
||||
if (graphGapBeforeInventory) {
|
||||
unregister()
|
||||
unregister = undefined
|
||||
await publish()
|
||||
}
|
||||
const list = await runtime.listTerminals()
|
||||
const afterFreshList = runtime.capture(h.id)
|
||||
if (graphGapBeforeInventory) {
|
||||
unregister = mount()
|
||||
}
|
||||
await publish()
|
||||
return {
|
||||
scenario: replacement ? 'successor-binding-before-exit' : 'ordinary-exit',
|
||||
graphGapBeforeInventory,
|
||||
capture,
|
||||
afterExit,
|
||||
afterQueuedGraph,
|
||||
afterFreshList,
|
||||
afterRepeatedGraph: runtime.capture(h.id),
|
||||
mobile: runtime.mobile(WT),
|
||||
resolution,
|
||||
inventory: await h.adapter.listProcesses(),
|
||||
persistedPtyId:
|
||||
store.getWorkspaceSession().terminalLayoutsByTabId[TAB]?.ptyIdsByLeafId?.[LEAF],
|
||||
listed: list.terminals.map((t) => ({
|
||||
id: t.ptyId,
|
||||
connected: t.connected,
|
||||
tabId: t.tabId,
|
||||
leafId: t.leafId
|
||||
}))
|
||||
}
|
||||
} finally {
|
||||
graphState.syncEnabled = false
|
||||
unregister?.()
|
||||
setRuntimeGraphSyncEnabled(false)
|
||||
setRuntimeGraphStoreStateGetter(null)
|
||||
vi.unstubAllGlobals()
|
||||
runtime.onPtyExit(h.id, 0)
|
||||
runtime.onPtyExit(successorId, 0)
|
||||
await h.dispose()
|
||||
store.flushOrThrow()
|
||||
rmSync(dir, { recursive: true, force: true })
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
import { expect, vi } from 'vitest'
|
||||
import { OrcaRuntimeService } from '../../../src/main/runtime/orca-runtime'
|
||||
import {
|
||||
buildMobileSessionTabSnapshots,
|
||||
registerRuntimeTerminalTab,
|
||||
setRuntimeGraphStoreStateGetter,
|
||||
setRuntimeGraphSyncEnabled
|
||||
} from '../../../src/renderer/src/runtime/sync-runtime-graph'
|
||||
import { makeState } from '../../../src/renderer/src/runtime/sync-runtime-graph-test-harness'
|
||||
import { graphState } from '../../../src/renderer/src/runtime/sync-runtime-graph/graph-state'
|
||||
|
||||
if (process.env.ORCA_BACKGROUND_LAUNCH !== '1') {
|
||||
throw new Error('background required')
|
||||
}
|
||||
|
||||
const TAB = '10000000-0000-4000-8000-000000000001'
|
||||
const TEST_WORKTREE_PATH = '/tmp/worktree-a'
|
||||
const TEST_WORKTREE_ID = `repo-1::${TEST_WORKTREE_PATH}`
|
||||
const LEAF = '10000000-0000-4000-8000-000000000002'
|
||||
const INC = '10000000-0000-4000-8000-000000000003'
|
||||
const PTY = `${TEST_WORKTREE_ID}@@sleep-review`
|
||||
|
||||
export async function runPreservedHistoryScenario() {
|
||||
const runtime = new OrcaRuntimeService()
|
||||
const worktree = {
|
||||
id: TEST_WORKTREE_ID,
|
||||
path: TEST_WORKTREE_PATH,
|
||||
repoId: 'repo-1',
|
||||
name: 'worktree-a',
|
||||
branch: 'main',
|
||||
isMain: false
|
||||
}
|
||||
vi.spyOn(runtime, 'resolveWorktreeSelector').mockResolvedValue(worktree)
|
||||
vi.spyOn(runtime, 'getResolvedWorktreeMap').mockResolvedValue(
|
||||
new Map([[TEST_WORKTREE_ID, worktree]])
|
||||
)
|
||||
let stopped = false
|
||||
let finishStop!: () => void
|
||||
const stopGate = new Promise<void>((resolve) => {
|
||||
finishStop = resolve
|
||||
})
|
||||
const stop = vi.fn(async () => {
|
||||
runtime.onPtyExit(PTY, 0, INC, { providerExitObserved: true })
|
||||
stopped = true
|
||||
await stopGate
|
||||
return true
|
||||
})
|
||||
runtime.setPtyController({
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
stopAndWait: stop,
|
||||
getForegroundProcess: async () => null,
|
||||
hasPty: () => !stopped,
|
||||
listProcesses: async () =>
|
||||
stopped ? [] : [{ id: PTY, cwd: TEST_WORKTREE_PATH, title: 'terminal', incarnationId: INC }]
|
||||
})
|
||||
let currentPty: string | null = PTY
|
||||
const state = makeState({
|
||||
tabsByWorktree: {
|
||||
[TEST_WORKTREE_ID]: [
|
||||
{
|
||||
id: TAB,
|
||||
worktreeId: TEST_WORKTREE_ID,
|
||||
title: 'Terminal',
|
||||
ptyId: PTY,
|
||||
customTitle: null,
|
||||
color: null,
|
||||
sortOrder: 0,
|
||||
createdAt: 1
|
||||
}
|
||||
]
|
||||
},
|
||||
terminalLayoutsByTabId: {
|
||||
[TAB]: {
|
||||
root: { type: 'leaf', leafId: LEAF },
|
||||
activeLeafId: LEAF,
|
||||
expandedLeafId: null,
|
||||
ptyIdsByLeafId: { [LEAF]: PTY }
|
||||
}
|
||||
}
|
||||
})
|
||||
const manager = {
|
||||
getPanes: () => [{ id: 1, leafId: LEAF }],
|
||||
getActivePane: () => ({ id: 1, leafId: LEAF }),
|
||||
getLeafId: () => LEAF,
|
||||
getNumericIdForLeaf: () => 1
|
||||
}
|
||||
setRuntimeGraphSyncEnabled(false)
|
||||
setRuntimeGraphStoreStateGetter(() => state)
|
||||
const unregister = registerRuntimeTerminalTab({
|
||||
tabId: TAB,
|
||||
worktreeId: TEST_WORKTREE_ID,
|
||||
// oxlint-disable-next-line typescript/consistent-type-assertions -- SAFETY: the snapshot builder reads only these four supplied pane lookup methods.
|
||||
getManager: () => manager as never,
|
||||
getContainer: () => null,
|
||||
getPtyIdForPane: () => currentPty,
|
||||
getTabWideAgentHintLeafId: () => null
|
||||
})
|
||||
const publish = () => {
|
||||
const snapshots = buildMobileSessionTabSnapshots(state)
|
||||
runtime.syncWindowGraph(1, {
|
||||
tabs: [
|
||||
{
|
||||
tabId: TAB,
|
||||
worktreeId: TEST_WORKTREE_ID,
|
||||
title: 'Terminal',
|
||||
activeLeafId: LEAF,
|
||||
layout: null
|
||||
}
|
||||
],
|
||||
leaves: [
|
||||
{
|
||||
tabId: TAB,
|
||||
worktreeId: TEST_WORKTREE_ID,
|
||||
leafId: LEAF,
|
||||
paneRuntimeId: 1,
|
||||
ptyId: currentPty
|
||||
}
|
||||
],
|
||||
mobileSessionTabs: snapshots
|
||||
})
|
||||
return snapshots[0]
|
||||
}
|
||||
const capture = () =>
|
||||
runtime['mobileSessionTabsByWorktree'].get(TEST_WORKTREE_ID)?.tabs.map((tab) => ({
|
||||
type: tab.type,
|
||||
id: tab.id,
|
||||
ptyId: tab.type === 'terminal' ? tab.ptyId : null
|
||||
}))
|
||||
try {
|
||||
runtime.registerPty(PTY, TEST_WORKTREE_ID, null, {
|
||||
tabId: TAB,
|
||||
leafId: LEAF,
|
||||
incarnationId: INC
|
||||
})
|
||||
publish()
|
||||
const initial = capture()
|
||||
expect(initial).toHaveLength(1)
|
||||
const pending = runtime.stopExactTerminalsForWorktree(`id:${TEST_WORKTREE_ID}`, [PTY], {
|
||||
keepHistory: true,
|
||||
targetOnly: true
|
||||
})
|
||||
await vi.waitFor(() => expect(stop).toHaveBeenCalledOnce())
|
||||
const afterExit = capture()
|
||||
expect(afterExit).toHaveLength(1)
|
||||
state.runtimePaneTitlesByTabId = { [TAB]: { 1: 'Sleeping terminal' } }
|
||||
const incoming = publish()
|
||||
const afterQueued = capture()
|
||||
const queuedLeaf = runtime['leaves'].get(runtime['getLeafKey'](TAB, LEAF))
|
||||
const leafState = queuedLeaf
|
||||
? { connected: queuedLeaf.connected, writable: queuedLeaf.writable, ptyId: queuedLeaf.ptyId }
|
||||
: null
|
||||
const model = runtime['headlessTerminals'].has(PTY)
|
||||
finishStop()
|
||||
const result = await pending
|
||||
currentPty = null
|
||||
state.tabsByWorktree[TEST_WORKTREE_ID] = [
|
||||
{ ...state.tabsByWorktree[TEST_WORKTREE_ID][0], ptyId: null }
|
||||
]
|
||||
state.terminalLayoutsByTabId[TAB] = { ...state.terminalLayoutsByTabId[TAB], ptyIdsByLeafId: {} }
|
||||
publish()
|
||||
return {
|
||||
initial,
|
||||
afterExit,
|
||||
incoming,
|
||||
afterQueued,
|
||||
leafState,
|
||||
model,
|
||||
afterBindingClear: capture(),
|
||||
result
|
||||
}
|
||||
} finally {
|
||||
finishStop()
|
||||
graphState.syncEnabled = false
|
||||
unregister()
|
||||
runtime.onPtyExit(PTY, 0, INC)
|
||||
setRuntimeGraphStoreStateGetter(null)
|
||||
vi.restoreAllMocks()
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,174 @@
|
||||
import assert from 'node:assert/strict'
|
||||
import { createHash } from 'node:crypto'
|
||||
import { mkdtemp, readFile, rm, writeFile } from 'node:fs/promises'
|
||||
import { tmpdir } from 'node:os'
|
||||
import { join, resolve } from 'node:path'
|
||||
import { fileURLToPath, pathToFileURL } from 'node:url'
|
||||
import { startVitest } from 'vitest/node'
|
||||
|
||||
if (process.env.ORCA_BACKGROUND_LAUNCH !== '1') {
|
||||
throw new Error('Run with ORCA_BACKGROUND_LAUNCH=1.')
|
||||
}
|
||||
const root = fileURLToPath(new URL('../../../', import.meta.url))
|
||||
const paths = [
|
||||
'src/main/runtime/orca-runtime-sync-window-graph.ts',
|
||||
'src/main/runtime/orca-runtime-mark-pty-liveness-unverifiable.ts',
|
||||
'src/main/runtime/orca-runtime-on-pty-exit.ts'
|
||||
]
|
||||
const sources = await Promise.all(paths.map((path) => readFile(join(root, path), 'utf8')))
|
||||
const gate = ` // Retained history stays addressable, but a renderer graph cannot revoke a host-certified exit.
|
||||
const connected = ptyId !== null && this.getPtyLivenessVerdict(ptyId)?.status !== 'exited'
|
||||
`
|
||||
const preserve = ` // An inventory's weak absence cannot revoke an earlier host-certified exit.
|
||||
if (tracked?.verdict.status === 'exited') {
|
||||
return
|
||||
}
|
||||
`
|
||||
const certificate = ` if (processDeathCertified) {
|
||||
// The bounded verdict register also fences late graphs after the PTY record was pruned.
|
||||
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
|
||||
}
|
||||
`
|
||||
for (const [index, text] of [gate, preserve, certificate].entries()) {
|
||||
assert(sources[index].includes(text), 'Source changed: review the baseline transform.')
|
||||
}
|
||||
const baseline = [
|
||||
sources[0]
|
||||
.replace(gate, '')
|
||||
.replace(
|
||||
" connected,\n writable: this.graphStatus === 'ready' && connected,",
|
||||
" connected: ptyId !== null,\n writable: this.graphStatus === 'ready' && ptyId !== null,"
|
||||
)
|
||||
.replace(' if (leaf.ptyId && connected) {', ' if (leaf.ptyId) {'),
|
||||
sources[1].replace(preserve, ''),
|
||||
sources[2].replace(certificate, '').replace(
|
||||
' pty.lastExitCause = exitCause\n',
|
||||
` pty.lastExitCause = exitCause
|
||||
if (exitCode >= 0 || options.hostExitConfirmed === true) {
|
||||
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
|
||||
}
|
||||
`
|
||||
)
|
||||
]
|
||||
const scratch = await mkdtemp(join(tmpdir(), 'orca-queued-graph-proof-'))
|
||||
const phases = []
|
||||
try {
|
||||
for (const phase of ['before', 'guard-only', 'after']) {
|
||||
const outputPath = join(scratch, `${phase}.json`)
|
||||
const testPath = join(scratch, `${phase}.test.ts`)
|
||||
const configPath = join(scratch, `${phase}.config.mjs`)
|
||||
await writeFile(
|
||||
testPath,
|
||||
`
|
||||
import { afterAll, it } from ${JSON.stringify(join(root, 'node_modules/vitest/dist/index.js'))}
|
||||
import { writeFileSync } from 'node:fs'
|
||||
import { runQueuedGraphExitScenario } from ${JSON.stringify(join(root, 'docs/audits/queued-terminal-graph-exit/fixture.ts'))}
|
||||
import { runPreservedHistoryScenario } from ${JSON.stringify(join(root, 'docs/audits/queued-terminal-graph-exit/preserved-history-fixture.ts'))}
|
||||
const rows = []
|
||||
let history
|
||||
it("preserved history", async () => { history = await runPreservedHistoryScenario() })
|
||||
for (const successor of [false, true]) {
|
||||
for (const graphGapBeforeInventory of [false, true]) {
|
||||
it(String(successor) + String(graphGapBeforeInventory), async () => rows.push(await runQueuedGraphExitScenario(successor, graphGapBeforeInventory)))
|
||||
}
|
||||
}
|
||||
afterAll(() => writeFileSync(${JSON.stringify(outputPath)}, JSON.stringify({ rows, history })))
|
||||
`
|
||||
)
|
||||
const replacements = Object.fromEntries(
|
||||
paths.map((path, index) => [
|
||||
`/${path}`,
|
||||
phase === 'after' || (phase === 'guard-only' && index === 0)
|
||||
? sources[index]
|
||||
: baseline[index]
|
||||
])
|
||||
)
|
||||
await writeFile(
|
||||
configPath,
|
||||
`
|
||||
import base from ${JSON.stringify(pathToFileURL(join(root, 'config/vitest.config.ts')).href)}
|
||||
const replacements = ${JSON.stringify(replacements)}
|
||||
export default {
|
||||
...base,
|
||||
plugins: [{ name: 'graph-exit-baseline', enforce: 'pre', transform(code, id) {
|
||||
for (const [path, replacement] of Object.entries(replacements)) {
|
||||
if (id.replaceAll('\\\\', '/').endsWith(path)) return replacement
|
||||
}
|
||||
} }],
|
||||
test: { ...base.test, include: [${JSON.stringify(testPath)}], maxWorkers: 1, fileParallelism: false }
|
||||
}
|
||||
`
|
||||
)
|
||||
const runner = await startVitest('test', [], {
|
||||
root,
|
||||
config: configPath,
|
||||
watch: false,
|
||||
reporters: ['dot']
|
||||
})
|
||||
assert(runner, 'Vitest did not start')
|
||||
await runner.close()
|
||||
const { rows, history } = JSON.parse(await readFile(outputPath, 'utf8'))
|
||||
assert.equal(history.afterQueued.length, 1)
|
||||
assert.equal(history.afterBindingClear.length, 1)
|
||||
assert.equal(history.model, false)
|
||||
if (phase !== 'before') {
|
||||
assert.equal(history.leafState.connected, false)
|
||||
assert.equal(history.leafState.writable, false)
|
||||
}
|
||||
delete history.incoming.publicationEpoch
|
||||
assert.equal(rows.length, 4)
|
||||
for (const row of rows) {
|
||||
const successor = row.scenario === 'successor-binding-before-exit'
|
||||
assert.equal(row.afterExit.connected, false)
|
||||
assert.equal(row.afterQueuedGraph.connected, successor && phase === 'before')
|
||||
assert.equal(
|
||||
row.afterRepeatedGraph.connected,
|
||||
successor && (phase === 'before' || (phase === 'guard-only' && row.graphGapBeforeInventory))
|
||||
)
|
||||
assert.equal(
|
||||
row.resolution === 'terminal_pane_owner_conflict',
|
||||
successor && phase === 'before'
|
||||
)
|
||||
assert.equal(row.inventory.length, successor ? 1 : 0)
|
||||
assert.equal(row.afterRepeatedGraph.model, false)
|
||||
if (phase === 'after') {
|
||||
assert.equal(row.afterRepeatedGraph.urlBound, false)
|
||||
assert(row.afterRepeatedGraph.leaves.every((leaf) => !leaf.connected && !leaf.writable))
|
||||
assert(row.mobile.every((tab) => !tab.handlePtyId || tab.ptyId === tab.handlePtyId))
|
||||
}
|
||||
for (const state of [
|
||||
row.afterExit,
|
||||
row.afterQueuedGraph,
|
||||
row.afterFreshList,
|
||||
row.afterRepeatedGraph
|
||||
]) {
|
||||
delete state.incarnationId
|
||||
}
|
||||
delete row.capture.rendererGeneration
|
||||
if (row.resolution && typeof row.resolution === 'object') {
|
||||
row.resolution = { ptyId: row.resolution.ptyId }
|
||||
}
|
||||
row.inventory = row.inventory.map(({ id }) => ({ id }))
|
||||
}
|
||||
phases.push({ phase, rows, history })
|
||||
}
|
||||
const output = `${JSON.stringify(
|
||||
{
|
||||
sources: Object.fromEntries(
|
||||
paths.map((path, index) => [
|
||||
path,
|
||||
createHash('sha256').update(sources[index]).digest('hex')
|
||||
])
|
||||
),
|
||||
phases
|
||||
},
|
||||
null,
|
||||
2
|
||||
)}\n`
|
||||
if (process.argv[2]) {
|
||||
await writeFile(resolve(process.argv[2]), output)
|
||||
}
|
||||
process.stdout.write(output)
|
||||
} finally {
|
||||
await rm(scratch, { recursive: true, force: true })
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -137,6 +137,10 @@ export class OrcaRuntimeWithMarkPtyLivenessUnverifiable extends OrcaRuntimeWithO
|
||||
|
||||
protected forgetPtyLivenessVerdict(ptyId: string, observedNoLaterThan?: number): void {
|
||||
const tracked = this.ptyLivenessVerdictByPtyId.get(ptyId)
|
||||
// An inventory's weak absence cannot revoke an earlier host-certified exit.
|
||||
if (tracked?.verdict.status === 'exited') {
|
||||
return
|
||||
}
|
||||
if (observedNoLaterThan !== undefined && tracked && tracked.observedAt > observedNoLaterThan) {
|
||||
return
|
||||
}
|
||||
|
||||
@@ -198,6 +198,10 @@ export class OrcaRuntimeWithOnPtyExit extends OrcaRuntimeWithOnClientDisconnecte
|
||||
this.terminalDrivers.clear(ptyId)
|
||||
this.remoteDesktopFloor.clearPty(ptyId)
|
||||
this.disposeHeadlessTerminal(ptyId)
|
||||
if (processDeathCertified) {
|
||||
// The bounded verdict register also fences late graphs after the PTY record was pruned.
|
||||
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
|
||||
}
|
||||
if (pty) {
|
||||
pty.connected = false
|
||||
pty.runtimeSessionOwned = false
|
||||
@@ -205,12 +209,6 @@ export class OrcaRuntimeWithOnPtyExit extends OrcaRuntimeWithOnClientDisconnecte
|
||||
pty.disconnectedAt = Date.now()
|
||||
pty.lastExitCode = exitCode
|
||||
pty.lastExitCause = exitCause
|
||||
if (exitCode >= 0 || options.hostExitConfirmed === true) {
|
||||
// Record the certificate rather than merely dropping the doubt: a reader that has to
|
||||
// authorize a respawn cannot distinguish "the host reported this process gone" from "this
|
||||
// runtime has never asked" if both are absence.
|
||||
this.rememberPtyLivenessVerdict(ptyId, { status: 'exited' })
|
||||
}
|
||||
// Why: the exited process's live frames say nothing about a replacement.
|
||||
// A same-id respawn makes the leaf writable again before any new title,
|
||||
// so leaving this true would let push delivery type into the new process
|
||||
|
||||
@@ -107,14 +107,16 @@ export class OrcaRuntimeWithSyncWindowGraph extends OrcaRuntimeWithAttachWindow
|
||||
? existing.ptyGeneration + 1
|
||||
: (existing?.ptyGeneration ?? 0)
|
||||
const existingPty = ptyId ? this.ptysById.get(ptyId) : undefined
|
||||
// Retained history stays addressable, but a renderer graph cannot revoke a host-certified exit.
|
||||
const connected = ptyId !== null && this.getPtyLivenessVerdict(ptyId)?.status !== 'exited'
|
||||
const tailSource = existing?.ptyId === ptyId ? existing : existingPty
|
||||
|
||||
nextLeaves.set(leafKey, {
|
||||
...leaf,
|
||||
ptyId,
|
||||
ptyGeneration,
|
||||
connected: ptyId !== null,
|
||||
writable: this.graphStatus === 'ready' && ptyId !== null,
|
||||
connected,
|
||||
writable: this.graphStatus === 'ready' && connected,
|
||||
lastOutputAt: tailSource?.lastOutputAt ?? null,
|
||||
lastExitCode: tailSource?.lastExitCode ?? null,
|
||||
lastExitCause: tailSource?.lastExitCause ?? null,
|
||||
@@ -138,7 +140,7 @@ export class OrcaRuntimeWithSyncWindowGraph extends OrcaRuntimeWithAttachWindow
|
||||
: graphSyncedAt
|
||||
})
|
||||
|
||||
if (leaf.ptyId) {
|
||||
if (leaf.ptyId && connected) {
|
||||
this.recordPtyWorktree(leaf.ptyId, leaf.worktreeId, {
|
||||
connected: true,
|
||||
lastOutputAt: existing?.ptyId === leaf.ptyId ? existing.lastOutputAt : null,
|
||||
|
||||
@@ -0,0 +1,251 @@
|
||||
import { describe, expect, it, vi } from 'vitest'
|
||||
import { OrcaRuntimeService } from './orca-runtime'
|
||||
|
||||
const WORKTREE = 'repo::/tmp/graph-exit'
|
||||
const TAB = '10000000-0000-4000-8000-000000000001'
|
||||
const LEAF = '10000000-0000-4000-8000-000000000002'
|
||||
const PTY = `${WORKTREE}@@terminal`
|
||||
const FIRST = '10000000-0000-4000-8000-000000000003'
|
||||
const NEXT = '10000000-0000-4000-8000-000000000004'
|
||||
|
||||
class ExitAuthorityRuntime extends OrcaRuntimeService {
|
||||
override resolveWorktreeSelector(selector: string) {
|
||||
return super.resolveWorktreeSelector(selector)
|
||||
}
|
||||
|
||||
override getResolvedWorktreeMap() {
|
||||
return super.getResolvedWorktreeMap()
|
||||
}
|
||||
|
||||
capture(id = PTY) {
|
||||
const pty = this.ptysById.get(id)
|
||||
return { connected: pty?.connected, incarnationId: pty?.incarnationId }
|
||||
}
|
||||
|
||||
get verdictCount(): number {
|
||||
return this.ptyLivenessVerdictByPtyId.size
|
||||
}
|
||||
|
||||
dropRecord(id = PTY): void {
|
||||
this.dropDisconnectedPtyRecord(id)
|
||||
}
|
||||
|
||||
history() {
|
||||
return {
|
||||
surfaces: this.mobileSessionTabsByWorktree.get(WORKTREE)?.tabs.length,
|
||||
leaves: this.getLeavesForPty(PTY).map((leaf) => ({
|
||||
connected: leaf.connected,
|
||||
writable: leaf.writable
|
||||
})),
|
||||
model: this.headlessTerminals.has(PTY)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
function graph(
|
||||
runtime: OrcaRuntimeService,
|
||||
ptyId: string | null = PTY,
|
||||
snapshotVersion?: number
|
||||
): void {
|
||||
runtime.syncWindowGraph(1, {
|
||||
tabs: [
|
||||
{ tabId: TAB, worktreeId: WORKTREE, title: 'terminal', activeLeafId: LEAF, layout: null }
|
||||
],
|
||||
leaves: [{ tabId: TAB, worktreeId: WORKTREE, leafId: LEAF, paneRuntimeId: 1, ptyId }],
|
||||
...(snapshotVersion === undefined
|
||||
? {}
|
||||
: {
|
||||
mobileSessionTabs: [
|
||||
{
|
||||
worktree: WORKTREE,
|
||||
publicationEpoch: 'renderer:retained-history',
|
||||
snapshotVersion,
|
||||
activeGroupId: null,
|
||||
activeTabId: `${TAB}::${LEAF}`,
|
||||
activeTabType: 'terminal' as const,
|
||||
tabs: [
|
||||
{
|
||||
type: 'terminal' as const,
|
||||
id: `${TAB}::${LEAF}`,
|
||||
parentTabId: TAB,
|
||||
leafId: LEAF,
|
||||
...(ptyId ? { ptyId } : {}),
|
||||
title: 'Terminal',
|
||||
isActive: true
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
})
|
||||
})
|
||||
}
|
||||
|
||||
function register(runtime: OrcaRuntimeService, incarnationId = FIRST): void {
|
||||
runtime.registerPty(PTY, WORKTREE, null, { tabId: TAB, leafId: LEAF, incarnationId })
|
||||
}
|
||||
|
||||
describe('host exit authority over queued renderer graphs', () => {
|
||||
it('keeps exact-stop history addressable through a changed snapshot before binding clears', async () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
const git = {
|
||||
path: '/tmp/graph-exit',
|
||||
head: 'abc',
|
||||
branch: 'main',
|
||||
isBare: false,
|
||||
isMainWorktree: false
|
||||
}
|
||||
const worktree = {
|
||||
...git,
|
||||
git,
|
||||
id: WORKTREE,
|
||||
repoId: 'repo',
|
||||
displayName: 'graph-exit',
|
||||
comment: '',
|
||||
linkedIssue: null,
|
||||
linkedPR: null,
|
||||
linkedLinearIssue: null,
|
||||
isArchived: false,
|
||||
isUnread: false,
|
||||
isPinned: false,
|
||||
sortOrder: 0,
|
||||
lastActivityAt: 0,
|
||||
parentWorktreeId: null,
|
||||
childWorktreeIds: [],
|
||||
lineage: null
|
||||
}
|
||||
const resolve = vi.spyOn(runtime, 'resolveWorktreeSelector').mockResolvedValue(worktree)
|
||||
const map = vi
|
||||
.spyOn(runtime, 'getResolvedWorktreeMap')
|
||||
.mockResolvedValue(new Map([[WORKTREE, worktree]]))
|
||||
let stopped = false
|
||||
let finishStop!: () => void
|
||||
const gate = new Promise<void>((done) => {
|
||||
finishStop = done
|
||||
})
|
||||
const stop = vi.fn(async () => {
|
||||
runtime.onPtyExit(PTY, 0, FIRST, { providerExitObserved: true })
|
||||
stopped = true
|
||||
await gate
|
||||
return true
|
||||
})
|
||||
runtime.setPtyController({
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
stopAndWait: stop,
|
||||
getForegroundProcess: async () => null,
|
||||
hasPty: () => !stopped,
|
||||
listProcesses: async () =>
|
||||
stopped ? [] : [{ id: PTY, incarnationId: FIRST, cwd: worktree.path, title: 'terminal' }]
|
||||
})
|
||||
let pending: Promise<unknown> | undefined
|
||||
try {
|
||||
register(runtime)
|
||||
graph(runtime, PTY, 1)
|
||||
pending = runtime.stopExactTerminalsForWorktree(`id:${WORKTREE}`, [PTY], {
|
||||
keepHistory: true,
|
||||
targetOnly: true
|
||||
})
|
||||
await vi.waitFor(() => expect(stop).toHaveBeenCalledOnce())
|
||||
graph(runtime, PTY, 2)
|
||||
expect(runtime.history()).toEqual({
|
||||
surfaces: 1,
|
||||
leaves: [{ connected: false, writable: false }],
|
||||
model: false
|
||||
})
|
||||
finishStop()
|
||||
await expect(pending).resolves.toMatchObject({ postStopVerified: true })
|
||||
graph(runtime, null, 3)
|
||||
expect(runtime.history().surfaces).toBe(1)
|
||||
} finally {
|
||||
finishStop()
|
||||
await pending
|
||||
resolve.mockRestore()
|
||||
map.mockRestore()
|
||||
runtime.onPtyExit(PTY, 0, FIRST)
|
||||
}
|
||||
})
|
||||
|
||||
it.each([0, -1])('retains a physical exit certificate after record pruning, code=%s', (code) => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
register(runtime)
|
||||
runtime.dropRecord()
|
||||
runtime.onPtyExit(PTY, code, FIRST, { providerExitObserved: true })
|
||||
graph(runtime)
|
||||
expect(runtime.capture().connected).toBeUndefined()
|
||||
expect(runtime.getPtyLivenessVerdict(PTY)).toEqual({ status: 'exited' })
|
||||
})
|
||||
|
||||
it('admits a new renderer pane without inventing a host verdict', () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
graph(runtime)
|
||||
expect(runtime.capture().connected).toBe(true)
|
||||
expect(runtime.getPtyLivenessVerdict(PTY)).toBeNull()
|
||||
})
|
||||
|
||||
it('admits a registered successor and ignores the predecessor exit', () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
register(runtime)
|
||||
runtime.onPtyExit(PTY, 0, FIRST)
|
||||
register(runtime, NEXT)
|
||||
runtime.onPtyExit(PTY, 0, FIRST)
|
||||
graph(runtime)
|
||||
expect(runtime.capture()).toEqual({ connected: true, incarnationId: NEXT })
|
||||
expect(runtime.getPtyLivenessVerdict(PTY)).toBeNull()
|
||||
})
|
||||
|
||||
it('admits a same-ID spawn before its registration commits', () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
register(runtime)
|
||||
runtime.onPtyExit(PTY, 0, FIRST)
|
||||
runtime.onPtySpawned(PTY, NEXT)
|
||||
graph(runtime)
|
||||
expect(runtime.capture()).toEqual({ connected: true, incarnationId: NEXT })
|
||||
expect(runtime.getPtyLivenessVerdict(PTY)).toBeNull()
|
||||
})
|
||||
|
||||
it('allows owning inventory to prove the same ID live again', async () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
register(runtime)
|
||||
runtime.onPtyExit(PTY, 0, FIRST)
|
||||
runtime.setPtyController({
|
||||
write: () => true,
|
||||
kill: () => true,
|
||||
getForegroundProcess: async () => null,
|
||||
listProcesses: async () => [{ id: PTY, incarnationId: NEXT, cwd: '', title: 'terminal' }]
|
||||
})
|
||||
await runtime.listTerminals()
|
||||
graph(runtime)
|
||||
expect(runtime.capture()).toEqual({ connected: true, incarnationId: NEXT })
|
||||
expect(runtime.getPtyLivenessVerdict(PTY)?.status).toBe('live')
|
||||
})
|
||||
|
||||
it('keeps an SSH disconnect unverifiable and admissible', () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
const id = 'ssh:target@@terminal'
|
||||
runtime.registerPty(id, WORKTREE, 'target', { tabId: TAB, leafId: LEAF, incarnationId: FIRST })
|
||||
runtime.onPtyExit(id, -1, FIRST)
|
||||
graph(runtime, id)
|
||||
expect(runtime.getPtyLivenessVerdict(id)?.status).toBe('unverifiable')
|
||||
expect(runtime.capture(id).connected).toBe(true)
|
||||
})
|
||||
|
||||
it('does not promote an unverified local stop to an exit certificate', () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
runtime.registerPty(PTY, WORKTREE)
|
||||
runtime.onPtyExit(PTY, -1, FIRST)
|
||||
runtime.markPtyLivenessUnverifiable(PTY, 'stop unverified')
|
||||
graph(runtime)
|
||||
expect(runtime.getPtyLivenessVerdict(PTY)?.status).toBe('unverifiable')
|
||||
expect(runtime.capture().connected).toBe(true)
|
||||
})
|
||||
|
||||
it('bounds certificates for exits whose PTY records are already gone', () => {
|
||||
const runtime = new ExitAuthorityRuntime()
|
||||
for (let index = 0; index < 1_000; index++) {
|
||||
runtime.onPtyExit(`${PTY}-${index}`, 0)
|
||||
}
|
||||
expect(runtime.verdictCount).toBe(256)
|
||||
expect(runtime.getPtyLivenessVerdict(`${PTY}-0`)).toBeNull()
|
||||
expect(runtime.getPtyLivenessVerdict(`${PTY}-999`)).toEqual({ status: 'exited' })
|
||||
})
|
||||
})
|
||||
Reference in New Issue
Block a user