fix(sessions): settle canceled handoffs without replacement launches

This commit is contained in:
Neil
2026-09-18 00:12:29 -07:00
parent 5333f0f4e1
commit 2eb0774e62
8 changed files with 218 additions and 52 deletions
@@ -6,9 +6,9 @@ Host teardown stops TUI transcript catchup before draining in-flight handoffs. P
Catchup now registers its state before its first await and owns an abort controller throughout setup. It passes the existing resolver/subscriber cancellation signal, releases late subscriptions, settles the initial-ready wait on stop, and preserves a newer same-session acquisition. `stopAll` permanently closes this host's admission; ordinary per-session `stop` still permits a replacement.
Preparation returns its signal internally so the handoff checks cancellation immediately before launching a TUI. A dedicated internal cancellation error, while no TUI owner or process identity has been committed, releases the unused reservation through the existing fenced `abandonStoredAgentSessionHandoffAttempt` transition. It leaves a recoverable native lease without acquiring a replacement. This distinction matters: ordinary preparation failure invokes native recovery, and a delayed replacement acquisition can finish after the five-second teardown drain. Ordinary read failures retain that recovery behavior. Canceled recovery of a live TUI stops without retrying or relabeling its live lease.
Preparation returns its signal internally so the handoff checks cancellation immediately before and after launching a TUI. A dedicated internal cancellation error, while no TUI owner or process identity has been committed, releases the unused reservation through the existing fenced `abandonStoredAgentSessionHandoffAttempt` transition. If launch returns after cancellation with an owner, the existing proven cleanup path is invoked; if cleanup is unavailable or fails, ownership is retained and manual recovery remains required. It leaves a recoverable native lease without acquiring a replacement. This distinction matters: ordinary preparation failure invokes native recovery, and a delayed replacement acquisition can finish after the five-second teardown drain. Ordinary read failures retain that recovery behavior. Canceled recovery of a live TUI stops without retrying or relabeling its live lease, and settles any original durable operation that was still pending.
These are internal lifecycle changes. They add no wire type and infer no remote process death. An already launched TUI and boundary/import I/O already admitted before cancellation remain outside this change's cancellation guarantee.
These are internal lifecycle changes. They add no wire type and infer no remote process death. A launch that remains in flight beyond the bounded teardown drain, and boundary/import I/O already admitted before cancellation, remain outside this change's cancellation guarantee.
## Reproduce
@@ -18,7 +18,7 @@ ORCA_BACKGROUND_LAUNCH=1 node docs/audits/tui-transcript-acquisition/reproduce.m
The script runs seven tests through the actual host, handoff coordinator, durable record store, journal, and transcript watcher. Real file resolution, watcher installation, and initial read are paused at explicit asynchronous boundaries; provider processes use the existing fake adapter/transport. No real shell or app window launches.
`fix.patch` is reversed inside a temporary Vite transform for the baseline. The new internal error declaration remains available to the same assertions; it does not change baseline control flow. Source hashes and exact failing cases are recorded in `results.json`. Each runner uses a 512 MiB old-space limit, a 90-second deadline, and the repository's cross-platform `runProcess`. Temporary runner/configuration files and acquired watchers are cleaned up.
`fix.patch` is reversed inside a temporary Vite transform for the baseline. The new internal error declaration remains available to the same assertions; it does not change baseline control flow. Source hashes and exact failing cases are recorded in `results.json`. Each runner uses a 512 MiB old-space limit, a 90-second deadline, and the repository's cross-platform `runProcess`. The proof requires the fixed runner to exit successfully in addition to matching its seven-pass/zero-fail report. Temporary runner/configuration files and acquired watchers are cleaned up.
| Version | Passed | Failed |
| ---------- | -----: | -----: |
@@ -27,6 +27,8 @@ The script runs seven tests through the actual host, handoff coordinator, durabl
The five preparation cases cover resolution, subscription return, initial snapshot, admission after teardown, and a completed preparation whose caller has not resumed. The recovery case preserves the live TUI lease. The control delays native acquisition after an ordinary resolver error and verifies the original error and recovery behavior. Six additional ownership tests cover overlapping prepare/recover replacements, per-session restart, repeated shutdown, and the signal returned when no supported record is available. Existing catchup tests preserve live appends and restart gap replay.
Two additional handoff regressions cover a TUI launch returning after cancellation and cancellation during recovery of a pending durable operation. Both fail against the original PR head and pass with the review fix. A late owner is stopped through the existing proven-cleanup contract, then the reservation is abandoned without acquiring a replacement native owner. If cleanup is unavailable or cannot prove the owner stopped, the existing manual-recovery path retains ownership instead. Recovery cancellation marks the original operation failed without relabeling the live TUI lease.
## Version and attribution
Named-path reads confirm the same setup gaps, unguarded forward launch, and stop-before-drain ordering in `v1.4.198`. In that tag the teardown phases are inline in `structured-agent-session-host.ts:254`; current source extracts them into `structured-agent-session-host-teardown.ts`. The executable proof compares current source before/after this fix. It establishes an execution-host watcher retaining path present in the reported version, without proving that #19831 or #19768 exercised this teardown race or explaining either report's memory magnitude.
@@ -1,5 +1,5 @@
diff --git a/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-forward.ts b/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-forward.ts
index a73e8b2111..63d178a67d 100644
index a73e8b2111..b9559868f4 100644
--- a/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-forward.ts
+++ b/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-forward.ts
@@ -12,7 +12,10 @@ import type {
@@ -24,11 +24,19 @@ index a73e8b2111..63d178a67d 100644
owner = await deps.transport!.launchTui({
record,
fence: record.lease.runtimeFence,
@@ -107,6 +111,16 @@ export async function handoffStructuredSessionToTui(
@@ -91,6 +95,7 @@ export async function handoffStructuredSessionToTui(
processIdentityCommitted = true
}
})
} catch (error) {
deps.stopTuiHistoryCatchup?.(sessionId)
+ if (!owner && !processIdentityCommitted && error instanceof StructuredTuiCatchupStoppedError) {
+ prepared?.throwIfAborted()
if (!processIdentityCommitted) {
await deps.store.commitProcessIdentity({
sessionId,
@@ -128,6 +133,16 @@ export async function handoffStructuredSessionToTui(
)
}
}
+ if (error instanceof StructuredTuiCatchupStoppedError && (owner || !processIdentityCommitted)) {
+ await abandonStoredAgentSessionHandoffAttempt(deps.store, {
+ sessionId,
+ expectedFence: record.lease.runtimeFence,
@@ -38,14 +46,35 @@ index a73e8b2111..63d178a67d 100644
+ })
+ throw error
+ }
if (!owner && error instanceof StructuredTuiLaunchCleanupError) {
await markStructuredHandoffManualRecovery(context, sessionId, operationId)
throw error
await recoverNativeAfterTuiFailure(context, sessionId, operationId)
throw error
}
diff --git a/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart-tui.ts b/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart-tui.ts
index c16ac68122..3bf0bc08e8 100644
--- a/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart-tui.ts
+++ b/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart-tui.ts
@@ -96,3 +96,16 @@ export async function persistReprovedTuiOwner(
})
}
}
+
+export async function startRecoveredTuiCatchup(
+ input: StructuredAgentSessionRestartAccess,
+ record: AgentSessionRecord
+): Promise<void> {
+ const prepared = await input.deps.recoverTuiHistoryCatchup?.(
+ record.sessionId,
+ record.lease.runtimeFence
+ )
+ prepared?.throwIfAborted()
+ await input.deps.activateTuiHistoryCatchup?.(record.sessionId)
+ prepared?.throwIfAborted()
+}
diff --git a/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart.ts b/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart.ts
index 13a26f2a7a..8947e4931a 100644
index 13a26f2a7a..a6ad92d91e 100644
--- a/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart.ts
+++ b/src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart.ts
@@ -12,6 +12,7 @@ import {
@@ -12,10 +12,12 @@ import {
structuredTuiRecoveryProofIsAdmissible
} from './structured-agent-session-handoff-status'
import type { StructuredTuiOwner } from './structured-agent-session-handoff-types'
@@ -53,31 +82,42 @@ index 13a26f2a7a..8947e4931a 100644
import {
persistReprovedTuiOwner,
recoverTuiOwnerOrContinue,
@@ -57,6 +58,9 @@ export async function restoreStructuredAgentSessionHandoff(
recoverUnavailableTuiAsNative,
+ startRecoveredTuiCatchup,
type StructuredAgentSessionRestartAccess
} from './structured-agent-session-handoff-restart-tui'
@@ -57,6 +59,15 @@ export async function restoreStructuredAgentSessionHandoff(
}
return
} catch (error) {
+ if (error instanceof StructuredTuiCatchupStoppedError) {
+ if (operationId) {
+ await input.deps.store.recordOperationOutcome({
+ operationId,
+ outcome: { status: 'failed', code: 'agent_session_handoff_failed' }
+ })
+ }
+ throw error
+ }
lastError = error
if (attempt < 2) {
await new Promise((resolve) => setTimeout(resolve, 100 * 2 ** attempt))
@@ -282,8 +286,13 @@ async function startRecoveredTuiCatchup(
input: RestartAccess,
record: AgentSessionRecord
): Promise<void> {
- await input.deps.recoverTuiHistoryCatchup?.(record.sessionId, record.lease.runtimeFence)
+ const prepared = await input.deps.recoverTuiHistoryCatchup?.(
+ record.sessionId,
+ record.lease.runtimeFence
+ )
+ prepared?.throwIfAborted()
await input.deps.activateTuiHistoryCatchup?.(record.sessionId)
+ prepared?.throwIfAborted()
@@ -278,14 +289,6 @@ async function restoreProving(input: RestartAccess, record: AgentSessionRecord):
await continueHandoff(input, stopped)
}
-async function startRecoveredTuiCatchup(
- input: RestartAccess,
- record: AgentSessionRecord
-): Promise<void> {
- await input.deps.recoverTuiHistoryCatchup?.(record.sessionId, record.lease.runtimeFence)
- await input.deps.activateTuiHistoryCatchup?.(record.sessionId)
-}
-
async function continueHandoff(input: RestartAccess, record: AgentSessionRecord): Promise<void> {
const direction = record.lease.runtimeKind === 'native' ? 'to-tui' : 'to-native'
const operationId = record.lease.handoffOperationId!
diff --git a/src/main/native-chat/agent-session-wire/structured-tui-transcript-catchup.ts b/src/main/native-chat/agent-session-wire/structured-tui-transcript-catchup.ts
index cc343c9231..10ce2416a3 100644
--- a/src/main/native-chat/agent-session-wire/structured-tui-transcript-catchup.ts
@@ -123,6 +123,7 @@ export default {...base, test: {...base.test, include: ${JSON.stringify(includes
before.failed === 6 &&
before.passed === 1 &&
before.passed + before.failed === 7 &&
after.exitCode === 0 &&
after.passed === 7 &&
after.failed === 0
console.log(
@@ -3,11 +3,15 @@
"sourceHashes": {
"src/main/native-chat/agent-session-wire/structured-agent-session-handoff-forward.ts": {
"before": "c9bb6fbf8ca3fc3fad815f35a21c73e392dd6be267335984deb0b5c9319210f1",
"after": "89e1b6769dafde03be5de24f8a2e08b48a490527cab7568b9a5ec4a97a0593ea"
"after": "99204872e4432ea841be493012c23b00b67fedabe07a83332c995dec632839cc"
},
"src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart-tui.ts": {
"before": "fcfcbd821816f33d1cf8bb71e6ecb40b03d4139affe8629f5baaa0a45f423921",
"after": "58a1b23f9390234e39bdb9681e43e9b32fd4d741a4242101a8d25e85a7001c6e"
},
"src/main/native-chat/agent-session-wire/structured-agent-session-handoff-restart.ts": {
"before": "8f2dc4f31fd2f96f3e9393afcc0826b712591c5d3bfa80965113e63be65f69ab",
"after": "03b76c1e3ba2389154f3fba4971fc17fbf3ba4ccde471f309e80f67134091d90"
"after": "73461453fba97bd0630471a224fc718ac2ce497c18fc95afb2ee264e7e42f791"
},
"src/main/native-chat/agent-session-wire/structured-tui-transcript-catchup.ts": {
"before": "36085d52e44152c7d8906ac2691242e8e31e54511fc908510c0d3aee10615973",
@@ -95,6 +95,7 @@ export async function handoffStructuredSessionToTui(
processIdentityCommitted = true
}
})
prepared?.throwIfAborted()
if (!processIdentityCommitted) {
await deps.store.commitProcessIdentity({
sessionId,
@@ -111,16 +112,6 @@ export async function handoffStructuredSessionToTui(
})
} catch (error) {
deps.stopTuiHistoryCatchup?.(sessionId)
if (!owner && !processIdentityCommitted && error instanceof StructuredTuiCatchupStoppedError) {
await abandonStoredAgentSessionHandoffAttempt(deps.store, {
sessionId,
expectedFence: record.lease.runtimeFence,
operationId,
recoverableRuntimeKind: 'native',
now: deps.now()
})
throw error
}
if (!owner && error instanceof StructuredTuiLaunchCleanupError) {
await markStructuredHandoffManualRecovery(context, sessionId, operationId)
throw error
@@ -142,6 +133,16 @@ export async function handoffStructuredSessionToTui(
)
}
}
if (error instanceof StructuredTuiCatchupStoppedError && (owner || !processIdentityCommitted)) {
await abandonStoredAgentSessionHandoffAttempt(deps.store, {
sessionId,
expectedFence: record.lease.runtimeFence,
operationId,
recoverableRuntimeKind: 'native',
now: deps.now()
})
throw error
}
await recoverNativeAfterTuiFailure(context, sessionId, operationId)
throw error
}
@@ -96,3 +96,16 @@ export async function persistReprovedTuiOwner(
})
}
}
export async function startRecoveredTuiCatchup(
input: StructuredAgentSessionRestartAccess,
record: AgentSessionRecord
): Promise<void> {
const prepared = await input.deps.recoverTuiHistoryCatchup?.(
record.sessionId,
record.lease.runtimeFence
)
prepared?.throwIfAborted()
await input.deps.activateTuiHistoryCatchup?.(record.sessionId)
prepared?.throwIfAborted()
}
@@ -17,6 +17,7 @@ import {
persistReprovedTuiOwner,
recoverTuiOwnerOrContinue,
recoverUnavailableTuiAsNative,
startRecoveredTuiCatchup,
type StructuredAgentSessionRestartAccess
} from './structured-agent-session-handoff-restart-tui'
@@ -59,6 +60,12 @@ export async function restoreStructuredAgentSessionHandoff(
return
} catch (error) {
if (error instanceof StructuredTuiCatchupStoppedError) {
if (operationId) {
await input.deps.store.recordOperationOutcome({
operationId,
outcome: { status: 'failed', code: 'agent_session_handoff_failed' }
})
}
throw error
}
lastError = error
@@ -282,19 +289,6 @@ async function restoreProving(input: RestartAccess, record: AgentSessionRecord):
await continueHandoff(input, stopped)
}
async function startRecoveredTuiCatchup(
input: RestartAccess,
record: AgentSessionRecord
): Promise<void> {
const prepared = await input.deps.recoverTuiHistoryCatchup?.(
record.sessionId,
record.lease.runtimeFence
)
prepared?.throwIfAborted()
await input.deps.activateTuiHistoryCatchup?.(record.sessionId)
prepared?.throwIfAborted()
}
async function continueHandoff(input: RestartAccess, record: AgentSessionRecord): Promise<void> {
const direction = record.lease.runtimeKind === 'native' ? 'to-tui' : 'to-native'
const operationId = record.lease.handoffOperationId!
@@ -21,6 +21,7 @@ import type {
StructuredAgentSessionHandoffTransport,
StructuredTuiOwner
} from './structured-agent-session-handoff-types'
import { StructuredTuiCatchupStoppedError } from './structured-agent-session-handoff-types'
const journals = createTrackedJournalOpener()
@@ -49,7 +50,7 @@ let acquireNativeStop: ReturnType<typeof vi.fn<(turnId: string) => Promise<boole
let acquireNativeCalls: number
let stopRecoveredOwner: TransportMock<'stopRecoveredOwner'>
let operations: number
type HistoryCatchup = (sessionId: string, fence: number) => Promise<void>
type HistoryCatchup = (sessionId: string, fence: number) => Promise<AbortSignal | void>
let prepareTuiHistoryCatchup: ReturnType<typeof vi.fn<HistoryCatchup>>
let recoverTuiHistoryCatchup: ReturnType<typeof vi.fn<HistoryCatchup>>
let activateTuiHistoryCatchup: ReturnType<typeof vi.fn<(sessionId: string) => Promise<void>>>
@@ -317,6 +318,54 @@ describe('structured session handoff failure handling', () => {
ownerProcess: null
})
})
it('settles cancellation after a TUI launch returns without retaining the new owner', async () => {
const operation = operationId()
const controller = new AbortController()
const launchEntered = Promise.withResolvers<void>()
const launchRelease = Promise.withResolvers<void>()
prepareTuiHistoryCatchup.mockResolvedValueOnce(controller.signal)
launchTui.mockImplementationOnce(async ({ fence, spawnToken }) => {
launchEntered.resolve()
await launchRelease.promise
return makeTuiOwner(fence, spawnToken)
})
await setStoredAgentSessionHandoffStage(store, {
sessionId: SESSION,
fence: 1,
stage: 'preparing',
handoffOperationId: operation,
now: NOW
})
await store.admitOperation({
callerKey: 'test',
operationId: operation,
fingerprint: 'late-launch',
now: NOW
})
const pending = coordinator.restore(SESSION)
const rejection = expect(pending).rejects.toBeInstanceOf(StructuredTuiCatchupStoppedError)
await launchEntered.promise
const acquisitionsBeforeCancellation = acquireNativeCalls
controller.abort(new StructuredTuiCatchupStoppedError())
launchRelease.resolve()
await rejection
expect(stopFailedTuiLaunch).toHaveBeenCalledOnce()
expect(acquireNativeCalls).toBe(acquisitionsBeforeCancellation)
expect(store.getRecord(SESSION)?.lease).toMatchObject({
runtimeKind: 'native',
claimStatus: 'released',
handoffStage: 'old-owner-stopped',
ownerProcess: null
})
expect(store.listOperationRows().find((row) => row.operationId === operation)?.outcome).toEqual(
{
status: 'failed',
code: 'agent_session_handoff_failed'
}
)
})
})
// The direction-agnostic restore path is the crash-during-acquisition recovery every
@@ -392,6 +441,68 @@ describe('structured session ownership recovery on restore', () => {
)
})
it('settles the interrupted recovery operation after catchup cancellation', async () => {
const operation = operationId()
let record = await setStoredAgentSessionHandoffStage(store, {
sessionId: SESSION,
fence: 1,
stage: 'preparing',
handoffOperationId: operation,
now: NOW
})
record = await stopStoredAgentSessionOwnerForHandoff(store, {
sessionId: SESSION,
expectedFence: record.lease.runtimeFence,
operationId: operation,
now: NOW
})
record = await reserveStoredAgentSessionHandoffOwner(store, {
sessionId: SESSION,
expectedFence: record.lease.runtimeFence,
runtimeKind: 'tui',
spawnToken: 'recovery-tui',
operationId: operation,
claimKeyId: 'key-1',
now: NOW
})
await store.commitProcessIdentity({
sessionId: SESSION,
fence: record.lease.runtimeFence,
process: process('recovery-tui', 4401),
now: NOW
})
await store.admitOperation({
callerKey: 'test',
operationId: operation,
fingerprint: 'recovery',
now: NOW
})
const controller = new AbortController()
recoverTuiHistoryCatchup.mockResolvedValueOnce(controller.signal)
activateTuiHistoryCatchup.mockImplementationOnce(async () => {
controller.abort(new StructuredTuiCatchupStoppedError())
})
coordinator = createCoordinator()
await expect(coordinator.restore(SESSION)).rejects.toBeInstanceOf(
StructuredTuiCatchupStoppedError
)
expect(store.getRecord(SESSION)?.lease).toMatchObject({
runtimeKind: 'tui',
claimStatus: 'live',
handoffStage: null,
handoffOperationId: null,
ownerProcess: expect.any(Object)
})
expect(store.listOperationRows().find((row) => row.operationId === operation)?.outcome).toEqual(
{
status: 'failed',
code: 'agent_session_handoff_failed'
}
)
})
it('continues only the persisted TUI handoff after a store restart', async () => {
const plainOperation = operationId()
await store.reserveOwner({