fix(ssh): stop reporting a confirmed kill when the SSH provider is gone (#14977)

* fix(ssh): stop reporting a confirmed kill when the SSH provider is gone

A detached relay PTY is designed to outlive the provider that addressed it
(it ignores SIGHUP and ships with an unlimited grace), so "the SSH provider
is no longer registered" is lost contact, never evidence the remote process
stopped. Both stop primitives in the PTY controller returned `true` from
that branch, and every caller downstream reported the fabricated success:
the CLI printed "PTY killed.", worker-stop settled the dispatch as stopped,
and — because the stop "succeeded" — the unstopped-PTY gate never ran, so
worktree removal walked straight past a live remote agent.

`kill`/`stopAndWait` now still tombstone the local lease but report an
unconfirmed stop and record why, using the three-verdict vocabulary the
worktree teardown gate already spoke (`live` / `unverifiable` / `exited`),
promoted out of that module into `src/shared/pty-liveness-verdict.ts`.
The close receipt, the CLI wording, worker-stop and the removal gate all
read that verdict instead of inferring an exit from silence.

The same rule fixes the mirror-image defect: the aggregate inventory only
enumerates registered providers, so a dropped relay clears `connected` for
every remote PTY at once. The sweep now separates the provider answering
"absent" (an exit) from no provider being able to answer (lost contact), so
worker-stop stops claiming `exited` from a disconnect.

The `connected` wire field is unchanged in meaning and shape.

* fix(orchestration): apply the same honesty to the federation stop path

The federation host runs its own copy of the worker observation and stop
logic, with the same two defects: `inspectRemoteAttachment` read a dropped
relay's `connected: false` as `exited`, and `federationStop` settled the
dispatch as stopped from a close it never confirmed — relaying a fabricated
success all the way home to the coordinator.

Two guards also had to move so the honest verdict does not become a new
refusal. `federationRead` gated on `status !== 'running'`, which would have
rejected a connected terminal the moment a stop lost contact with it; it now
gates on `status === 'exited'`, which is equivalent for every pre-existing
status given the two guards beside it. Local `workerStop` likewise still
attempts the close when the verdict is `unverifiable` — losing contact is a
reason to report the outcome honestly, never a reason to stop trying.

The show observations now carry the reason alongside the status, so a bare
`unverifiable` is actionable. Both are new optional fields.

* fix(ssh): preserve unconfirmed stop verdicts across consumers

* fix(ssh): use canonical live verdict wording

* fix(ssh): refuse wrong-host teardown verification

* test(orchestration): confirm worker release teardown

* fix(orchestration): negotiate honest worker stop receipts

* fix(agent-teams): fence uncertain teammate respawns

* fix(ssh): avoid duplicate missing-provider teardown

* fix(orchestration): preserve archives across release retries

* fix(ssh): preserve verdicts across synthetic kill exits

* fix(ssh): preserve liveness evidence across teardown

* fix(agent-teams): replace panes only after confirmed stop

* fix(ssh): distinguish host exits from relay loss

* fix(ssh): narrow concurrent inventory verdicts

* fix(orchestration): serve archives after uncertain release

* fix(orchestration): expose unverifiable read liveness

* test(ssh): align liveness assertions with verdicts

* fix(ssh): preserve host scope across inventory failures
This commit is contained in:
Brennan Benson
2026-08-17 00:11:19 -07:00
committed by GitHub
parent b279f66c96
commit 7afce2ea41
46 changed files with 1978 additions and 161 deletions
@@ -6,6 +6,7 @@ import type { RemoteDispatchAttachmentRow } from '../../orchestration/types'
import { defineMethod, type RpcMethod } from '../core'
import { OptionalFiniteNumber, requiredString } from '../schemas'
import { readExactWorkerOutput } from './orchestration-worker-output'
import { describeUnconfirmedAgentStop } from '../../../../shared/pty-liveness-verdict'
const FederationDispatchParams = z.object({
dispatchId: requiredString('Missing Dispatch ID')
@@ -36,7 +37,11 @@ export const ORCHESTRATION_FEDERATION_CONTROL_METHODS: RpcMethod[] = [
runtimeEpoch: runtime.getRuntimeId(),
attachment: exposeRemoteAttachment(attachment),
terminal: observation.exact ? observation.terminal : null,
observation: { status: observation.status, exactWorker: observation.exact }
observation: {
status: observation.status,
exactWorker: observation.exact,
...(observation.reason ? { reason: observation.reason } : {})
}
}
}
}),
@@ -46,7 +51,10 @@ export const ORCHESTRATION_FEDERATION_CONTROL_METHODS: RpcMethod[] = [
handler: async (params, { runtime, authenticatedCallerFingerprint }) => {
requireHomeAttachment(runtime, params.dispatchId, authenticatedCallerFingerprint)
const observation = await inspectRemoteAttachment(runtime, params.dispatchId)
if (!observation.exact || !observation.terminal || observation.status !== 'running') {
// Why `=== 'exited'` rather than `!== 'live'`: the other non-live
// statuses are already covered by the two guards, and an unverifiable
// terminal is still readable — losing stop-contact is not an exit.
if (!observation.exact || !observation.terminal || observation.status === 'exited') {
throw new OrchestrationError(
'worker_identity_changed',
`Remote Dispatch ${params.dispatchId} no longer resolves to its exact process.`
@@ -83,7 +91,18 @@ export const ORCHESTRATION_FEDERATION_CONTROL_METHODS: RpcMethod[] = [
dispatchId: params.dispatchId,
terminalHandle: observation.terminal.handle,
workerState: attachment.state,
terminalStatus: observation.status === 'exited' ? 'exited' : 'running',
terminalStatus:
observation.status === 'exited'
? 'exited'
: observation.status === 'unverifiable'
? 'unknown'
: 'running',
terminalLiveness:
observation.status === 'unverifiable'
? 'unverifiable'
: observation.status === 'exited'
? 'exited'
: 'live',
attachedAt: attachment.created_at,
source: params.source,
cursor: params.cursor,
@@ -134,6 +153,22 @@ export const ORCHESTRATION_FEDERATION_CONTROL_METHODS: RpcMethod[] = [
}
try {
const close = await runtime.closeTerminal(observation.terminal.handle)
if (!close.ptyKilled) {
// The tab is retired but the process was never confirmed stopped, so
// the coordinator must not be told this dispatch reached 'stopped'.
const attachment = db.markRemoteAttachmentStopUnknown(
params.dispatchId,
describeUnconfirmedAgentStop(close)
)
return {
dispatchId: params.dispatchId,
state: attachment.state,
alreadySettled: false,
processAction: 'closed_agent_terminal',
lastError: attachment.last_error,
close
}
}
const attachment = db.settleRemoteAttachmentStop(params.dispatchId)
return {
dispatchId: params.dispatchId,
@@ -172,29 +207,44 @@ function requireHomeAttachment(
return attachment
}
async function inspectRemoteAttachment(runtime: OrcaRuntimeService, dispatchId: string) {
async function inspectRemoteAttachment(
runtime: OrcaRuntimeService,
dispatchId: string
): Promise<{
terminal: Awaited<ReturnType<OrcaRuntimeService['showTerminal']>> | null
exact: boolean
status: 'unattached' | 'missing' | 'identity_changed' | 'live' | 'exited' | 'unverifiable'
/** Set with `unverifiable`; names what we lost contact with. */
reason?: string
}> {
const db = runtime.getOrchestrationDb()
const attachment = db.getRemoteDispatchAttachment(dispatchId)
if (!attachment?.terminal_handle) {
return { terminal: null, exact: false, status: 'unattached' as const }
return { terminal: null, exact: false, status: 'unattached' }
}
const terminal = await runtime.showTerminal(attachment.terminal_handle).catch(() => null)
if (!terminal) {
return { terminal: null, exact: false, status: 'missing' as const }
return { terminal: null, exact: false, status: 'missing' }
}
const exact = db.isRemoteAttachmentProcessCurrent({
dispatchId,
paneKey: runtime.getTerminalPaneKey(attachment.terminal_handle),
processIncarnation: runtime.getTerminalProcessIncarnation(attachment.terminal_handle)
})
if (!exact) {
return { terminal, exact, status: 'identity_changed' }
}
// Why: the same rule as the local worker observation — the inventory only
// iterates registered providers, so a dropped relay clears `connected` for
// every remote PTY at once. Lost contact is not a death certificate.
const verdict = runtime.getTerminalLivenessVerdict?.(attachment.terminal_handle) ?? null
if (verdict?.status === 'unverifiable') {
return { terminal, exact, status: 'unverifiable', reason: verdict.reason }
}
return {
terminal,
exact,
status: exact
? terminal.connected === false
? ('exited' as const)
: ('running' as const)
: ('identity_changed' as const)
status: verdict?.status !== 'live' && terminal.connected === false ? 'exited' : 'live'
}
}
@@ -0,0 +1,183 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { ORCHESTRATION_CONTRACT_VERSION } from '../../../../shared/protocol-version'
import { OrcaRuntimeService } from '../../orca-runtime'
import { OrchestrationDb } from '../../orchestration/db'
import { ORCHESTRATION_METHODS } from './orchestration'
// The federation host runs its own copy of the observation and stop logic, so
// it needs the same rule: lost contact with a worker's host is not an exit, and
// a close it could not confirm must not be relayed home as a settled stop.
const HOME_FINGERPRINT = 'home-peer-fingerprint'
const DISPATCH_ID = 'ctx_federation_verdict'
const HANDLE = 'term_remote_worker'
const PANE_KEY = 'tab_remote:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
const INCARNATION = 'runtime:pty:7'
const SSH_PROVIDER_GONE = 'its SSH provider is no longer registered'
describe('federation host liveness verdicts', () => {
let db: OrchestrationDb
let runtime: OrcaRuntimeService
beforeEach(() => {
db = new OrchestrationDb(':memory:')
runtime = new OrcaRuntimeService()
runtime.setOrchestrationDb(db)
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue(PANE_KEY)
vi.spyOn(runtime, 'getTerminalProcessIncarnation').mockReturnValue(INCARNATION)
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
handle: HANDLE,
worktreeId: 'repo::remote-worktree',
connected: false,
status: 'exited'
} as never)
db.createRemoteDispatchAttachment({
dispatchId: DISPATCH_ID,
taskId: 'task_remote',
homePeerFingerprint: HOME_FINGERPRINT,
protocolVersion: ORCHESTRATION_CONTRACT_VERSION,
runtimeEpoch: runtime.getRuntimeId(),
mutationReceipt: {
callerFingerprint: HOME_FINGERPRINT,
requestId: 'rpc_attach',
method: 'orchestration.federationStart',
payloadHash: 'hash'
}
})
db.prepareRemoteAttachmentAuthority({
dispatchId: DISPATCH_ID,
paneKey: PANE_KEY,
processIncarnation: INCARNATION,
worktreeId: 'repo::remote-worktree',
terminalHandle: HANDLE,
setupState: 'not_applicable',
effects: [{ kind: 'terminal', action: 'created', id: HANDLE }]
})
db.markRemoteAttachmentReady(DISPATCH_ID)
})
afterEach(() => db.close())
async function call(name: string, params: Record<string, unknown>) {
const method = ORCHESTRATION_METHODS.find((candidate) => candidate.name === name)
if (!method) {
throw new Error(`Method not found: ${name}`)
}
return method.handler(method.params!.parse(params), {
runtime,
authenticatedCallerFingerprint: HOME_FINGERPRINT
} as never)
}
it('reports lost contact as unverifiable rather than an observed exit', async () => {
vi.spyOn(runtime, 'getTerminalLivenessVerdict').mockReturnValue({
status: 'unverifiable',
reason: SSH_PROVIDER_GONE
})
await expect(
call('orchestration.federationShow', { dispatchId: DISPATCH_ID })
).resolves.toMatchObject({
observation: { status: 'unverifiable', exactWorker: true, reason: SSH_PROVIDER_GONE }
})
})
it('uses the canonical live verdict for an observed process', async () => {
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
handle: HANDLE,
worktreeId: 'repo::remote-worktree',
connected: true,
status: 'running'
} as never)
vi.spyOn(runtime, 'getTerminalLivenessVerdict').mockReturnValue({
status: 'live',
ptyIds: [HANDLE]
})
await expect(
call('orchestration.federationShow', { dispatchId: DISPATCH_ID })
).resolves.toMatchObject({ observation: { status: 'live', exactWorker: true } })
})
it('still reports a locally observed exit as exited', async () => {
await expect(
call('orchestration.federationShow', { dispatchId: DISPATCH_ID })
).resolves.toMatchObject({ observation: { status: 'exited', exactWorker: true } })
})
it('still serves output for a terminal we merely lost stop-contact with', async () => {
// Why this matters: the read gate used to reject every status except live, which
// would refuse a connected terminal the moment a stop lost contact with it.
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
handle: HANDLE,
worktreeId: 'repo::remote-worktree',
connected: true
} as never)
vi.spyOn(runtime, 'getTerminalLivenessVerdict').mockReturnValue({
status: 'unverifiable',
reason: SSH_PROVIDER_GONE
})
const outcome = await call('orchestration.federationRead', {
dispatchId: DISPATCH_ID
}).catch((error: unknown) => error)
expect(outcome).not.toMatchObject({ code: 'worker_identity_changed' })
})
it('does not relay an unconfirmed close home as a settled stop', async () => {
vi.spyOn(runtime, 'getTerminalLivenessVerdict').mockReturnValue({
status: 'unverifiable',
reason: SSH_PROVIDER_GONE
})
const closeTerminal = vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: HANDLE,
tabId: 'tab_remote',
ptyKilled: false,
ptyStopVerdict: 'unverifiable',
ptyStopReason: SSH_PROVIDER_GONE
})
const stopped = (await call('orchestration.federationStop', { dispatchId: DISPATCH_ID })) as {
state: string
lastError?: string
}
// Losing contact is a reason to report honestly, never to stop trying.
expect(closeTerminal).toHaveBeenCalledWith(HANDLE)
expect(stopped.state).not.toBe('stopped')
expect(stopped.lastError).toContain('could not be confirmed stopped')
})
it('does not settle a bare false close as a stop', async () => {
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: HANDLE,
tabId: 'tab_remote',
ptyKilled: false
})
const stopped = (await call('orchestration.federationStop', { dispatchId: DISPATCH_ID })) as {
state: string
lastError?: string
}
expect(stopped.state).not.toBe('stopped')
expect(stopped.lastError).toContain('could not be confirmed stopped')
})
it('still settles a confirmed close as a stop', async () => {
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: HANDLE,
tabId: 'tab_remote',
ptyKilled: true
})
const stopped = (await call('orchestration.federationStop', { dispatchId: DISPATCH_ID })) as {
state: string
processAction: string
}
expect(stopped.state).toBe('stopped')
expect(stopped.processAction).toBe('closed_agent_terminal')
})
})
@@ -192,7 +192,7 @@ describe('orchestration federated worker output', () => {
ok: true,
result: {
server: { environmentId: 'environment_windows', name: 'windows' },
observation: { status: 'running', exactWorker: true },
observation: { status: 'live', exactWorker: true },
terminal: { handle: 'term_windows_worker' }
}
})
@@ -181,6 +181,7 @@ describe('orchestration federated setup evidence', () => {
await expect(
workerShow.handler(workerShow.params!.parse({ dispatch: started.dispatch.id }), { runtime })
).resolves.toMatchObject({
observation: { status: 'live', exactWorker: true },
worker: {
state: 'ready',
stage: 'input_accepted',
@@ -152,7 +152,8 @@ describe('orchestration federation', () => {
} as never)
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: 'term_windows_worker',
closed: true
tabId: 'tab-windows-worker',
ptyKilled: true
} as never)
}
@@ -664,7 +665,7 @@ describe('orchestration federation', () => {
expect(shown).toMatchObject({
ok: true,
result: { observation: { status: 'running', exactWorker: true } }
result: { observation: { status: 'live', exactWorker: true } }
})
expect(homeDb.getFederatedDispatch(dispatch.id)?.remote_runtime_epoch).not.toBe(oldEpoch)
expect(homeDb.getFederatedDispatch(dispatch.id)?.peer_fingerprint).toBe(
@@ -99,7 +99,11 @@ export const ORCHESTRATION_WORKER_CONTROL_METHODS: RpcMethod[] = [
server: { environmentId: server.environmentId, name: server.name },
remoteRuntimeEpoch: remote.runtimeEpoch,
terminal: remote.terminal,
observation: remote.observation
observation: {
...remote.observation,
// Legacy servers published `running`; normalize at the compatibility boundary.
status: remote.observation.status === 'running' ? 'live' : remote.observation.status
}
}
}
if (worker.runtime_epoch && worker.runtime_epoch !== runtime.getRuntimeId()) {
@@ -122,7 +126,12 @@ export const ORCHESTRATION_WORKER_CONTROL_METHODS: RpcMethod[] = [
dispatch,
worker: exposeWorker(worker),
terminal: observation.exact ? observation.terminal : null,
observation: { status: observation.status, exactWorker: observation.exact },
observation: {
status: observation.status,
exactWorker: observation.exact,
// Why: a bare `unverifiable` is not actionable without naming what we lost.
...(observation.reason ? { reason: observation.reason } : {})
},
terminalResource: resource ? exposeWorkerTerminalResource(resource) : null
}
}
@@ -176,7 +185,7 @@ export const ORCHESTRATION_WORKER_CONTROL_METHODS: RpcMethod[] = [
)
}
const resource = db.getWorkerTerminalResourceByOwner(params.dispatch)
if (resource && ['releasing', 'released'].includes(resource.release_state)) {
if (resource && ['releasing', 'unknown', 'released'].includes(resource.release_state)) {
return readArchivedWorkerOutput({
db,
dispatchId: params.dispatch,
@@ -199,7 +208,18 @@ export const ORCHESTRATION_WORKER_CONTROL_METHODS: RpcMethod[] = [
dispatchId: params.dispatch,
terminalHandle: worker.agent_terminal_handle,
workerState: worker.state,
terminalStatus: observation.status === 'exited' ? 'exited' : 'running',
terminalStatus:
observation.status === 'exited'
? 'exited'
: observation.status === 'unverifiable'
? 'unknown'
: 'running',
terminalLiveness:
observation.status === 'unverifiable'
? 'unverifiable'
: observation.status === 'exited'
? 'exited'
: 'live',
attachedAt: worker.created_at,
source: params.source,
cursor: params.cursor,
@@ -10,7 +10,9 @@ export async function inspectWorkerTerminal(
): Promise<{
terminal: Awaited<ReturnType<OrcaRuntimeService['showTerminal']>> | null
exact: boolean
status: 'unattached' | 'missing' | 'identity_changed' | 'running' | 'exited'
status: 'unattached' | 'missing' | 'identity_changed' | 'live' | 'exited' | 'unverifiable'
/** Set with `unverifiable`; names what we lost contact with. */
reason?: string
}> {
const worker = db.getWorkerDispatch(dispatchId)
if (!worker?.agent_terminal_handle) {
@@ -25,10 +27,23 @@ export async function inspectWorkerTerminal(
paneKey: runtime.getTerminalPaneKey(worker.agent_terminal_handle),
processIncarnation: runtime.getTerminalProcessIncarnation(worker.agent_terminal_handle)
})
if (!exact) {
return { terminal, exact, status: 'identity_changed' }
}
// Why: the aggregate inventory only iterates registered providers, so a dropped
// relay clears `connected` for every remote PTY at once. Lost contact is not a
// death certificate, and the verdict is the only field that can tell them apart.
const verdict = runtime.getTerminalLivenessVerdict?.(worker.agent_terminal_handle) ?? null
if (verdict?.status === 'unverifiable') {
return { terminal, exact, status: 'unverifiable', reason: verdict.reason }
}
if (verdict?.status === 'live') {
return { terminal, exact, status: 'live' }
}
return {
terminal,
exact,
status: exact ? (terminal.connected === false ? 'exited' : 'running') : 'identity_changed'
status: terminal.connected === false ? 'exited' : 'live'
}
}
@@ -71,7 +86,7 @@ export async function callFederatedWorkerShow(
residualResources: unknown[]
}
terminal: unknown
observation: { status: string; exactWorker: boolean }
observation: { status: string; exactWorker: boolean; reason?: string }
}> {
return (await runtime.callOrchestrationWorkerServer(
federated.environment_id,
@@ -83,6 +83,19 @@ describe('exact orchestration worker output', () => {
expect(readTerminal).not.toHaveBeenCalled()
})
it('reports unverifiable liveness without claiming the terminal is running', async () => {
const result = await read({
terminalStatus: 'unknown',
terminalLiveness: 'unverifiable'
})
expect(result.status).toEqual({
worker: 'ready',
terminal: 'unknown',
liveness: 'unverifiable'
})
})
it('reads Grok through the shared Native Chat transcript decoder', async () => {
await writeFile(
transcriptA,
@@ -20,6 +20,7 @@ export async function readExactWorkerOutput(args: {
terminalHandle: string
workerState: string
terminalStatus: RuntimeTerminalState
terminalLiveness?: 'live' | 'unverifiable' | 'exited'
attachedAt: string
source?: OrchestrationWorkerReadSource
cursor?: string | number
@@ -100,7 +101,11 @@ export async function readExactWorkerOutput(args: {
returnedMessageCount: transcript.messages.length
},
cursor: nextCursor,
status: { worker: args.workerState, terminal: args.terminalStatus },
status: {
worker: args.workerState,
terminal: args.terminalStatus,
...(args.terminalLiveness ? { liveness: args.terminalLiveness } : {})
},
fallbackReason: null,
warnings: transcript.warnings
}
@@ -145,7 +150,11 @@ async function readTerminalOutput(
sourceIdentity,
terminal: { ...terminal, tail: redactedTerminal.lines },
cursor: nextCursor,
status: { worker: args.workerState, terminal: terminal.status },
status: {
worker: args.workerState,
terminal: args.terminalLiveness === 'unverifiable' ? args.terminalStatus : terminal.status,
...(args.terminalLiveness ? { liveness: args.terminalLiveness } : {})
},
fallbackReason: null,
warnings: redactedTerminal.warnings
}
@@ -10,6 +10,7 @@ import {
type WorkerTerminalTailArchive
} from '../../orchestration/worker-output-archive'
import type { OrcaRuntimeService } from '../../orca-runtime'
import { describeUnconfirmedAgentStop } from '../../../../shared/pty-liveness-verdict'
import { inspectWorkerTerminal } from './orchestration-worker-observation'
import { orchestrationTimestampToMs } from './orchestration-worker-output'
@@ -212,7 +213,19 @@ async function completeWorkerTerminalReleaseOnce(
}
try {
await runtime.closeTerminal(resource.terminal_handle)
const close = await runtime.closeTerminal(resource.terminal_handle)
if (!close.ptyKilled) {
const reason = describeUnconfirmedAgentStop(close)
const unknown = db.markWorkerTerminalReleaseUnknown(resource.id, reason)
return {
dispatchId,
state: 'release_unknown',
processAction: 'closed_agent_terminal',
archive: { source: archiveSource, status: archiveStatus },
lastError: unknown.release_error ?? reason,
recovery: `Inspect with: orca orchestration worker-show --dispatch ${dispatchId} --json — then repeat worker-release with the same --retry-request. Never substitute a broad terminal close.`
}
}
} catch (error) {
const reason = error instanceof Error ? error.message : String(error)
if (/disposed|not connected|unavailable/i.test(reason)) {
@@ -0,0 +1,84 @@
import { describe, expect, it, vi } from 'vitest'
import type { OrcaRuntimeService } from '../../orca-runtime'
import type { OrchestrationDb } from '../../orchestration/db'
import type { WorkerTerminalResourceRow } from '../../orchestration/worker-terminal-ownership'
import { completeWorkerTerminalRelease } from './orchestration-worker-release-completion'
describe('orchestration worker release liveness verdict', () => {
it.each([
{
name: 'an explicit unverifiable verdict',
close: {
handle: 'term_worker',
tabId: 'tab-worker',
ptyKilled: false,
ptyStopVerdict: 'unverifiable' as const,
ptyStopReason: 'its SSH provider is no longer registered'
},
detail: 'its SSH provider is no longer registered'
},
{
name: 'a bare unconfirmed close',
close: { handle: 'term_worker', tabId: 'tab-worker', ptyKilled: false },
detail: 'the stop outcome could not be verified'
}
])('does not release a worker after $name', async ({ close, detail }) => {
const reason = 'its SSH provider is no longer registered'
const resource = {
id: 'resource-1',
terminal_handle: 'term_worker',
host_scope: JSON.stringify({ kind: 'ssh', targetId: 'target-1' }),
archive_source: 'terminal',
archive_status: 'captured',
ownership_state: 'owned',
release_state: 'requested'
} as WorkerTerminalResourceRow
const runtime = {
showTerminal: vi.fn(async () => ({ handle: 'term_worker', connected: false })),
getTerminalPaneKey: vi.fn(() => 'tab-worker:leaf-worker'),
getTerminalProcessIncarnation: vi.fn(() => 'pty-worker:incarnation-1'),
getTerminalLivenessVerdict: vi.fn(() => ({ status: 'unverifiable', reason })),
getOrchestrationDispatchAuthority: vi.fn(() => ({
hostScope: { kind: 'ssh', targetId: 'target-1' }
})),
closeTerminal: vi.fn(async () => close),
notifyMessageArrived: vi.fn()
} as unknown as OrcaRuntimeService
const markWorkerTerminalReleaseUnknown = vi.fn((_resourceId: string, releaseError: string) => ({
...resource,
release_state: 'unknown',
release_error: releaseError
}))
const db = {
getWorkerDispatch: vi.fn(() => ({
agent_terminal_handle: 'term_worker',
created_at: '2026-08-16T00:00:00.000Z'
})),
isDispatchProcessCurrent: vi.fn(() => true),
workerTerminalResourceHasIdentityConflict: vi.fn(() => false),
getWorkerTerminalArchive: vi.fn(() => ({ kind: 'transcript_pin' })),
commitWorkerTerminalArchiveForRelease: vi.fn(() => ({
...resource,
release_state: 'releasing'
})),
markWorkerTerminalReleaseUnknown
} as unknown as OrchestrationDb
await expect(
completeWorkerTerminalRelease({
runtime,
db,
dispatchId: 'ctx-worker',
resource
})
).resolves.toMatchObject({
state: 'release_unknown',
processAction: 'closed_agent_terminal',
lastError: `The agent terminal was closed but its process could not be confirmed stopped: ${detail}.`
})
expect(markWorkerTerminalReleaseUnknown).toHaveBeenCalledWith(
'resource-1',
`The agent terminal was closed but its process could not be confirmed stopped: ${detail}.`
)
})
})
@@ -80,8 +80,9 @@ describe('orchestration worker release recovery', () => {
})
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: 'term_worker',
closed: true
} as never)
tabId: 'tab-worker',
ptyKilled: true
})
vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {})
activeRunId = db.createRun({
objective: 'Release recovery test Run',
@@ -162,6 +163,36 @@ describe('orchestration worker release recovery', () => {
expect(db.getWorkerTerminalResourceByOwner(dispatchId)?.release_state).toBe('releasing')
})
it('preserves archived output when an unconfirmed release retry cannot find the terminal', async () => {
setup()
const { dispatchId } = await startSettledWorker()
vi.mocked(runtime.closeTerminal).mockResolvedValueOnce({
handle: 'term_worker',
tabId: 'tab-worker',
ptyKilled: false,
ptyStopVerdict: 'unverifiable',
ptyStopReason: 'its SSH provider is no longer registered'
})
await expect(
call('orchestration.workerRelease', { dispatch: dispatchId })
).resolves.toMatchObject({ state: 'release_unknown' })
expect(db.getWorkerTerminalArchive(dispatchId)).toBeDefined()
vi.mocked(runtime.showTerminal).mockRejectedValue(new Error('terminal_handle_stale'))
await expect(
call('orchestration.workerRelease', { dispatch: dispatchId })
).resolves.toMatchObject({ state: 'release_unknown' })
const read = (await call('orchestration.workerRead', { dispatch: dispatchId })) as {
archived?: boolean
terminal: { tail: string[] }
}
expect(read).toMatchObject({
archived: true,
terminal: { tail: ['worker output line 1', 'worker output line 2'] }
})
})
it('never touches resources without requested releases', async () => {
setup()
await startSettledWorker()
@@ -93,7 +93,8 @@ describe('orchestration worker release', () => {
})
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: 'term_worker',
closed: true
tabId: 'tab-worker',
ptyKilled: true
} as never)
vi.spyOn(runtime, 'notifyMessageArrived').mockImplementation(() => {})
activeRunId = db.createRun({
@@ -242,7 +243,7 @@ describe('orchestration worker release', () => {
it('reconciles a dead external terminal without closing a process', async () => {
setup()
const { dispatchId } = await startSettledWorker('succeeded', { terminal: 'term_worker' })
inspectProcessLiveness.mockResolvedValue('dead')
inspectProcessLiveness.mockResolvedValue('exited')
await expect(
call('orchestration.workerRelease', { dispatch: dispatchId })
@@ -268,7 +269,7 @@ describe('orchestration worker release', () => {
raw
.prepare('UPDATE worker_terminal_resources SET prior_owner_dispatch_ids = ? WHERE id = ?')
.run('{invalid', resource?.id)
inspectProcessLiveness.mockResolvedValue('dead')
inspectProcessLiveness.mockResolvedValue('exited')
await expect(
call('orchestration.workerRelease', { dispatch: dispatchId })
@@ -297,7 +298,7 @@ describe('orchestration worker release', () => {
setup()
const { dispatchId } = await startSettledWorker()
await call('orchestration.workerTerminalUserInput', { paneKey: workerPaneKey })
inspectProcessLiveness.mockResolvedValue('dead')
inspectProcessLiveness.mockResolvedValue('exited')
await expect(
call('orchestration.workerRelease', { dispatch: dispatchId })
@@ -324,7 +325,7 @@ describe('orchestration worker release', () => {
} else {
db.abandonWorkerDispatch(dispatchId)
}
inspectProcessLiveness.mockResolvedValue('dead')
inspectProcessLiveness.mockResolvedValue('exited')
await expect(
call('orchestration.workerRelease', { dispatch: dispatchId })
@@ -706,7 +707,7 @@ describe('orchestration worker release', () => {
expect(transferred?.terminal_handle).toBe('term_reminted')
expect(db.getWorkerTerminalResourceByOwner(first.dispatchId)).toBeUndefined()
inspectProcessLiveness.mockResolvedValueOnce('dead')
inspectProcessLiveness.mockResolvedValueOnce('exited')
const oldRelease = (await call('orchestration.workerRelease', {
dispatch: first.dispatchId
})) as { state: string; reason?: string }
@@ -727,7 +728,7 @@ describe('orchestration worker release', () => {
const first = await startSettledWorker('succeeded')
const second = await startWorker({ terminal: 'term_reminted' })
settle(second.taskId, second.dispatchId, 'succeeded')
inspectProcessLiveness.mockResolvedValue('dead')
inspectProcessLiveness.mockResolvedValue('exited')
await expect(
call('orchestration.workerRelease', { dispatch: first.dispatchId })
@@ -61,7 +61,7 @@ export const ORCHESTRATION_WORKER_RELEASE_METHODS: RpcMethod[] = [
(await runtime.inspectTerminalProcessIncarnationLiveness(
processIncarnation,
resource.host_scope
)) === 'dead'
)) === 'exited'
) {
const reconciled = db.settleDeadWorkerTerminalRelease({
requestingDispatchId: params.dispatch,
@@ -0,0 +1,57 @@
import { describe, expect, it, vi } from 'vitest'
import { ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY } from '../../../../shared/protocol-version'
import type { OrcaRuntimeService } from '../../orca-runtime'
import type { OrchestrationDb } from '../../orchestration/db'
import { ORCHESTRATION_WORKER_STOP_METHODS } from './orchestration-worker-stop'
describe('federated worker stop capability', () => {
it('does not trust a legacy server stop receipt', async () => {
const markWorkerStopUnknown = vi.fn((_dispatchId: string, reason: string) => ({
state: 'stop_unknown',
last_error: reason
}))
const db = {
getFederatedDispatch: vi.fn(() => ({
dispatch_id: 'ctx_remote',
environment_id: 'environment_linux',
environment_name: 'linux',
peer_fingerprint: 'peer-linux'
})),
beginWorkerStop: vi.fn(() => ({ disposition: 'started' })),
markWorkerStopUnknown
} as unknown as OrchestrationDb
const callOrchestrationWorkerServer = vi.fn(async (_environmentId, method) => {
if (method === 'status.get') {
return { capabilities: [ORCHESTRATION_CONTRACT_RUNTIME_CAPABILITY] }
}
return { state: 'stopped', processAction: 'closed_agent_terminal' }
})
const runtime = {
getOrchestrationDb: () => db,
resolveOrchestrationWorkerServer: () => ({
environmentId: 'environment_linux',
name: 'linux',
peerFingerprint: 'peer-linux'
}),
callOrchestrationWorkerServer
} as unknown as OrcaRuntimeService
const method = ORCHESTRATION_WORKER_STOP_METHODS[0]!
await expect(
method.handler(method.params!.parse({ dispatch: 'ctx_remote' }), {
runtime,
orchestrationMutation: {
callerFingerprint: 'coordinator',
requestId: 'request_stop',
method: 'orchestration.workerStop',
payloadHash: 'hash'
}
})
).resolves.toMatchObject({ state: 'stop_unknown', processAction: 'none' })
expect(markWorkerStopUnknown).toHaveBeenCalledWith(
'ctx_remote',
'Connected server linux cannot prove the worker stop outcome.'
)
expect(callOrchestrationWorkerServer).toHaveBeenCalledTimes(1)
})
})
@@ -0,0 +1,158 @@
import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'
import { OrcaRuntimeService } from '../../orca-runtime'
import { OrchestrationDb } from '../../orchestration/db'
import { ORCHESTRATION_METHODS } from './orchestration'
// The aggregate terminal inventory only iterates registered providers, so a
// dropped relay clears `connected` for every remote PTY at once. That is lost
// contact, not a death certificate.
describe('worker-stop against a terminal we lost contact with', () => {
let db: OrchestrationDb
let runtime: OrcaRuntimeService
beforeEach(() => {
db = new OrchestrationDb(':memory:')
runtime = new OrcaRuntimeService()
runtime.setOrchestrationDb(db)
vi.spyOn(runtime, 'getTerminalPaneKey').mockReturnValue(
'tab_worker:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb'
)
vi.spyOn(runtime, 'getTerminalProcessIncarnation').mockReturnValue('runtime:pty:1')
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
handle: 'term_worker',
worktreeId: 'repo::worktree',
connected: false,
status: 'exited'
} as never)
})
afterEach(() => db.close())
async function call(name: string, params: Record<string, unknown>) {
const method = ORCHESTRATION_METHODS.find((candidate) => candidate.name === name)
if (!method) {
throw new Error(`Method not found: ${name}`)
}
return method.handler(method.params!.parse(params), { runtime })
}
function createWorker() {
const run = db.createRun({
objective: 'Verdict',
coordinatorHandle: 'term_coord',
coordinatorPaneKey: 'tab_coord:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
})
const task = db.createTask({ spec: 'stop worker', runId: run.id })
const started = db.createStartingWorkerDispatch({
taskId: task.id,
startOptions: {},
runtimeEpoch: runtime.getRuntimeId()
})
db.prepareStartingWorkerAuthority({
dispatchId: started.dispatch.id,
handle: 'term_worker',
paneKey: 'tab_worker:bbbbbbbb-bbbb-4bbb-8bbb-bbbbbbbbbbbb',
processIncarnation: 'runtime:pty:1',
worktreeId: 'repo::worktree',
setupState: 'not_applicable',
effects: [{ kind: 'terminal', action: 'created', id: 'term_worker' }]
})
db.markWorkerDispatchReady(started.dispatch.id)
return started.dispatch
}
it('reports the process as unverifiable, not exited, when the link dropped', async () => {
vi.spyOn(runtime, 'getTerminalLivenessVerdict').mockReturnValue({
status: 'unverifiable',
reason: 'its SSH provider is no longer registered'
})
const closeTerminal = vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: 'term_worker',
tabId: 'tab_worker',
ptyKilled: false,
ptyStopVerdict: 'unverifiable',
ptyStopReason: 'its SSH provider is no longer registered'
})
const dispatch = createWorker()
await expect(
call('orchestration.workerShow', { dispatch: dispatch.id })
).resolves.toMatchObject({
observation: {
status: 'unverifiable',
exactWorker: true,
reason: 'its SSH provider is no longer registered'
}
})
const stopped = (await call('orchestration.workerStop', { dispatch: dispatch.id })) as {
state: string
processAction: string
lastError: string
}
// Losing contact is a reason to report honestly, never to stop trying.
expect(closeTerminal).toHaveBeenCalledWith('term_worker')
expect(stopped.processAction).toBe('closed_agent_terminal')
expect(stopped.state).toBe('stop_unknown')
expect(stopped.lastError).toContain('could not be confirmed stopped')
expect(stopped.lastError).not.toContain('exited')
})
it('does not settle a bare false close as stopped', async () => {
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
handle: 'term_worker',
worktreeId: 'repo::worktree',
connected: true,
status: 'running'
} as never)
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: 'term_worker',
tabId: 'tab_worker',
ptyKilled: false
})
const dispatch = createWorker()
const stopped = (await call('orchestration.workerStop', { dispatch: dispatch.id })) as {
state: string
lastError: string
}
expect(stopped.state).toBe('stop_unknown')
expect(stopped.lastError).toContain('could not be confirmed stopped')
})
it('uses the canonical live verdict for an observed process', async () => {
vi.spyOn(runtime, 'showTerminal').mockResolvedValue({
handle: 'term_worker',
worktreeId: 'repo::worktree',
connected: true,
status: 'running'
} as never)
vi.spyOn(runtime, 'getTerminalLivenessVerdict').mockReturnValue({
status: 'live',
ptyIds: ['runtime:pty:1']
})
const dispatch = createWorker()
await expect(
call('orchestration.workerShow', { dispatch: dispatch.id })
).resolves.toMatchObject({
observation: { status: 'live', exactWorker: true }
})
})
it('still reports a locally observed exit as exited', async () => {
const dispatch = createWorker()
await expect(
call('orchestration.workerShow', { dispatch: dispatch.id })
).resolves.toMatchObject({
observation: { status: 'exited', exactWorker: true }
})
const stopped = (await call('orchestration.workerStop', { dispatch: dispatch.id })) as {
lastError: string
}
expect(stopped.lastError).toBe('The recorded worker process is exited; no terminal was closed.')
})
})
@@ -2,6 +2,9 @@ import { z } from 'zod'
import { OrchestrationError } from '../../orchestration/orchestration-error'
import { defineMethod, type RpcMethod } from '../core'
import { requiredString } from '../schemas'
import { describeUnconfirmedAgentStop } from '../../../../shared/pty-liveness-verdict'
import { ORCHESTRATION_WORKER_STOP_VERDICT_RUNTIME_CAPABILITY } from '../../../../shared/protocol-version'
import type { RuntimeStatus } from '../../../../shared/runtime-types'
import {
inspectWorkerTerminal,
resolvePinnedFederatedServer
@@ -29,6 +32,24 @@ export const ORCHESTRATION_WORKER_STOP_METHODS: RpcMethod[] = [
return settledReceipt(params.dispatch, begun.worker.state)
}
try {
const status = (await runtime.callOrchestrationWorkerServer(
server.environmentId,
'status.get',
undefined,
30_000
)) as RuntimeStatus
if (
!status.capabilities?.includes(ORCHESTRATION_WORKER_STOP_VERDICT_RUNTIME_CAPABILITY)
) {
return unknownReceipt(
params.dispatch,
db.markWorkerStopUnknown(
params.dispatch,
`Connected server ${server.name} cannot prove the worker stop outcome.`
),
'none'
)
}
const remote = (await runtime.callOrchestrationWorkerServer(
server.environmentId,
'orchestration.federationStop',
@@ -101,7 +122,12 @@ export const ORCHESTRATION_WORKER_STOP_METHODS: RpcMethod[] = [
)
}
const observation = await inspectWorkerTerminal(runtime, db, params.dispatch)
if (!observation.exact || observation.status !== 'running') {
// Why `unverifiable` still proceeds: losing contact is a reason to report
// the outcome honestly, never a reason to stop trying to stop the worker.
if (
!observation.exact ||
(observation.status !== 'live' && observation.status !== 'unverifiable')
) {
return unknownReceipt(
params.dispatch,
db.markWorkerStopUnknown(
@@ -113,6 +139,15 @@ export const ORCHESTRATION_WORKER_STOP_METHODS: RpcMethod[] = [
}
try {
const close = await runtime.closeTerminal(handle)
if (!close.ptyKilled) {
// The tab is retired, but the agent process was never confirmed stopped —
// settling here is the false success this receipt exists to prevent.
return unknownReceipt(
params.dispatch,
db.markWorkerStopUnknown(params.dispatch, describeUnconfirmedAgentStop(close)),
'closed_agent_terminal'
)
}
const worker = db.settleWorkerStop(params.dispatch)
runtime.notifyMessageArrived(`dispatch:${params.dispatch}`, 'status')
return {
@@ -30,7 +30,8 @@ describe('orchestration worker recovery', () => {
})
vi.spyOn(runtime, 'closeTerminal').mockResolvedValue({
handle: 'term_worker',
closed: true
tabId: 'tab-worker',
ptyKilled: true
} as never)
})
@@ -80,7 +81,7 @@ describe('orchestration worker recovery', () => {
call('orchestration.workerShow', { dispatch: dispatch.id })
).resolves.toMatchObject({
worker: { state: 'ready' },
observation: { status: 'running', exactWorker: true },
observation: { status: 'live', exactWorker: true },
terminal: { handle: 'term_worker' }
})
await expect(