Stop workers from taking over the Run they work for (#29008)

* fix(orchestration): refuse run-use by a worker on the Run it works for

A worker holding an active Dispatch in a Run could bind that Run as its coordinator.
That silently unbound the real coordinator, remembered the worker's address as a
coordinator address (so the worker's own mail in that Run routed to the Run mailbox
from then on), and when the worker later bound its own sub-Run, left the parent Run
with no coordinator at all.

bindRun now refuses a caller whose handle holds an active Dispatch in the Run, with
worker_cannot_coordinate_own_run, before any effect. Creating its own Run for
sub-workers is unchanged.

* fix(orchestration): tell a worker naming its own Run where its mail is

Workers that made a sub-Run ran `check --run <parent>`, were told "This
coordinator terminal is bound to X, not Y", and ran run-use on the parent
to "switch back", unbinding their coordinator. That refusal now says the
Run belongs to their coordinator, that their own mail arrives through
plain `check`, and that they report with `send`; it never mentions run-use.

The run-use refusal now matches the caller the way check and send do
(address, then pane, so a reminted terminal handle is still caught),
also covers Runs above the caller's own through recorded creator_dispatch_id
lineage, skips context-only self-dispatches so a coordinator can still
rebind its own Run, and passes its code through the RPC wire.

* feat(orchestration): refuse run-use over a live coordinator without --takeover

run-use silently took a Run from whoever coordinated it, even a chat or
agent that was still running. It now refuses with run_coordinator_live
when there is positive evidence the current coordinator is running (its
chat is still reachable, or the agent hooks prove its terminal agent
live) unless the caller passes --takeover. Anything unproven (closed chat,
exited or unhooked terminal, another host) rebinds as before, so restart
handoffs need no flag. The probe is async in the RPC handler; bindRun
enforces it only if the Run is still bound to the probed party.

A caller working a Dispatch can never take over, with or without the
flag (worker_cannot_take_over_run). The help text now says run-use is
for resuming a Run and that workers never use it.

* fix(orchestration): stop pointing workers at run-use

Workers learned run-use from error text: "No Run is bound. Use run-create
or run-use first", "Rebind with orchestration run-use", and "Rebind this
terminal with orchestration run-use". Each now says what the caller can
do instead. An unbound worker is told its mail arrives through plain
check and it reports with send; a displaced coordinator is told to stop
and take the Run back only if the user asks. The orchestration skill now
says run-use resumes your own Run, --takeover needs the user's word, and
workers never run it.

* fix(orchestration): stop routing mail to addresses that grabbed a Run's seat

Every address that ever bound a Run is remembered as one of its
coordinator addresses, so mail to it routes to the Run mailbox. When a
worker bound the Run it was working for, its own address joined that
list for good, and its direct mail in that Run kept landing in the
coordinator's inbox. The live database has 45 such rows since
2026-08-20.

All four readers of the remembered addresses now skip one that was first
bound while it worked an unsettled, delegated Dispatch in the same Run.
It is derived from existing rows, so no migration or delete is needed and
rows written by older builds stop mattering. The active-Dispatch check
also matches a worker's orca_session_id spelling, not only its handle.

* fix(cli): keep the orchestration spec file within its line limit

* fix(orchestration): read remembered coordinator Run ids without a type assertion

* fix(orchestration): compare seat-grab times at whole seconds

first_bound_at has no milliseconds, so a worker bound in the same second its
Dispatch settled was counted as grabbing the seat. Compare completed_at at
whole seconds with a strict >.

* fix(orchestration): a worker coordinates only Runs it created

The worker refusals depended on proving the target's coordinator alive, so
a worker could still take any Run whose coordinator was closed, unhooked,
remote or headless. Replace them with one rule derived from existing rows:
a caller working a Dispatch may bind only a Run it legitimately
coordinated before (a remembered, non-seat-grab address of that Run, or a
Run it dispatched work into), never the Run it works for, and never with
--takeover. A same-binding no-op is never refused, and workers are
classified before the legacy gate so they never see its
--takeover-legacy hint. The refusal names the Dispatch and how to end it
with worker-abandon if it is stale. The ancestor walk is gone.

run-use now records which coordinator it probed. If the binding moved
before the transaction, it probes again (three tries) and then refuses
with a retryable run_binding_changed instead of binding over a party
nobody probed. run_coordinator_live names the chat by its title and
worktree, or the terminal by its title and worktree, says it is still
open, and that closing it clears the refusal.

* fix(orchestration): say what happens to mail on takeover and stop naming run-use to readers

The help now says --takeover replaces the coordinator and replays its
unacknowledged mail to you. A replaced mailbox reader is told only that
another session coordinates the Run and to stop reading it.

* refactor(orchestration): throw one non-null refusal from the live gate

* refactor(runtime): host the coordinator presence probe beside the hook presence seam

Main moved the orphan-adoption runtime file; keeping the one-line probe next
to checkHookAgentPresenceFn leaves this PR off that file.

* fix(orchestration): use findLast for the worktree name

* refactor(runtime): move the coordinator presence probe beside the pty presence recheck

Main is moving the runtime construction files; the probe now sits with
recheckHookAgentPresenceForPty, which reads the same seam, so this PR stays
off the files being moved.

* fix(orchestration): keep workers out of their own Run seat

* fix(orchestration): clarify worker Run handoffs in skill

* test: align explicit Run gate refusal wording

* Move coordinator presence beside terminal presence checks

* Walk complete worker creator lineage when binding Runs

* Restore main documentation after merge hook formatting

* perf(orchestration): check run bindings without terminal snapshots

* test(orchestration): narrow SQLite instrumentation before access

* test(orchestration): import the explicit RPC methods module

* test: cover orchestration restart and continued Run handoff

* test: place orchestration CLI integration in CLI project

* test(cli): use explicit module extensions in restart handoff test

* test(orchestration): isolate source CLI integration in subprocesses

* refactor(orchestration): leave the live-coordinator takeover rule to #29292

#29008 is the worker fix: a worker cannot bind the Run it works for or an
ancestor, cannot abandon itself to get around that, and addresses it held while
grabbing a seat stop redirecting its mail. The separate rule that refuses any
second coordinator while the first is still open (the presence probe, the
--takeover flag and the run_coordinator_live / run_binding_changed refusals)
moves to #29292, which models Run ownership and hand-over and already builds on
it. run-use binds synchronously again, as on main.

* test(orchestration): cover delegated worker legacy takeover refusal

* test(orchestration): explain source CLI integration timeout
This commit is contained in:
Brennan Benson
2026-10-26 00:46:27 -07:00
committed by GitHub
parent e4b1e4d22b
commit f71120551f
31 changed files with 1036 additions and 66 deletions
+1
View File
@@ -16,6 +16,7 @@ const contracts = [
'src/main/runtime/rpc/methods/repo.test.ts',
'src/renderer/src/components/tab-bar/use-new-tab-agent-menu-host-preferences.test.tsx',
'src/main/sqlite/sync-database.test.ts',
'src/main/runtime/orchestration/run-binding-performance.test.ts',
'src/main/runtime/orchestration/db/row-column-lists.test.ts',
'src/shared/native-chat/tools/native-chat-tool-attribution-allocation.test.ts',
'src/shared/native-chat/tools/native-chat-tool-pairing.performance.test.ts',
+6
View File
@@ -108,6 +108,12 @@ ORCA orchestration worker-start --spec "<worker B task>" --worktree current --ag
ORCA orchestration check --wait --types "worker_done,escalation,question" --timeout-ms 900000 --json
```
After a restart, resume your own Run with `run-use --id <run_id>`. A worker
cannot bind the Run it works for or an ancestor, and cannot use
`--takeover-legacy`; an already-bound no-op remains allowed. When its
coordinator directs a handoff, a replacement worker may resume a separate Run
with plain `run-use`.
If `worker-start` exits non-zero, do not relaunch. Read the receipt's
`failedStage` and `residualResources`, then load
`references/recovery-and-cleanup.md`.
@@ -51,7 +51,12 @@ run — and once more immediately before `worker_done`, so a redirect or a
cancellation lands before the Task settles. `check` names its caller with
`--terminal`, never `--from`. Stop checking after `worker_done`.
If `check` returns `consumer_fenced`, this process no longer owns its Dispatch:
Do not use `run-use` to check your Dispatch's mail or bind the Run you work for.
You may resume a separate Run when your coordinator directs a handoff, but you
cannot use `--takeover-legacy`. Plain `check` returns your Dispatch's mail
unless you are coordinating a Run; report to your coordinator with `send`.
If a plain `check` (no `--run`) returns `consumer_fenced`, this process no longer owns its Dispatch:
the Attempt was re-attached to another worker or settled without you. Stop, do
not send `worker_done`, and do not retry the check. An empty `check` never means
you were replaced; `consumer_fenced` is the only way you learn that.
File diff suppressed because one or more lines are too long
+1 -1
View File
@@ -109,7 +109,7 @@ export const ROOT_HELP_TEXT_PRIMARY = [
'',
'Orchestration:',
' orchestration run-create Create and bind a lightweight orchestration Run',
' orchestration run-use Bind this coordinator to an existing Run',
' orchestration run-use Resume coordinating a Run (workers never use this)',
" orchestration run-current Show this coordinator's bound Run",
' orchestration run-list List lightweight orchestration Runs',
' orchestration run-show Show one lightweight orchestration Run',
+2 -2
View File
@@ -16,13 +16,13 @@ export const ORCHESTRATION_COMMAND_SPECS: CommandSpec[] = [
},
{
path: ['orchestration', 'run-use'],
summary: 'Bind this coordinator to an existing Run',
summary: 'Resume coordinating a Run',
usage:
'orca orchestration run-use --id <run_id> [--from <handle>] [--takeover-legacy] [--retry-request <id>] [--json]',
allowedFlags: [...GLOBAL_FLAGS, 'id', 'from', 'takeover-legacy', 'retry-request'],
identityFlagRoles: { from: 'caller' },
notes: [
'--takeover-legacy must run in the live coordinator agent terminal it binds; it preserves existing worker assignments.'
'A worker cannot bind its own or ancestor Run or use --takeover-legacy. --takeover-legacy must run in the live coordinator agent terminal it binds; it preserves existing worker assignments.'
]
},
{
@@ -0,0 +1,279 @@
import { spawn } from 'node:child_process'
import { mkdtempSync, rmSync } from 'node:fs'
import { createRequire } from 'node:module'
import { tmpdir } from 'node:os'
import { join } from 'node:path'
import { pathToFileURL } from 'node:url'
import { afterEach, expect, it, vi } from 'vitest'
import { z } from 'zod'
import { OrchestrationDb } from './orchestration/db'
const cliPath = join(process.cwd(), 'src/cli/index.ts')
const tsxLoader = pathToFileURL(createRequire(import.meta.url).resolve('tsx')).href
const entity = z.object({ id: z.string() })
const check = z.object({
result: z.object({
runId: z.string(),
deliveryId: z.string(),
messages: z.array(z.object({ id: z.string() }).passthrough())
})
})
function idOf(response: unknown, key: string): string {
const { result } = z.object({ result: z.record(z.string(), z.unknown()) }).parse(response)
return entity.parse(result[key]).id
}
afterEach(() => {
vi.unstubAllEnvs()
vi.restoreAllMocks()
})
// Sequential source-CLI subprocesses and a disk-backed runtime restart exceed the default test budget.
it("retains CLI work and mail across restart, refuses a worker's own Run, and completes a separate Run handoff", async () => {
const home = mkdtempSync(join(tmpdir(), 'orca-cli-restart-handoff-'))
for (const name of [
'HOME',
'USERPROFILE',
'APPDATA',
'LOCALAPPDATA',
'XDG_CONFIG_HOME',
'XDG_CACHE_HOME',
'XDG_DATA_HOME',
'XDG_STATE_HOME',
'XDG_RUNTIME_DIR',
'CLAUDE_CONFIG_DIR',
'CODEX_HOME',
'ORCA_E2E_HOME_DIR',
'ORCA_E2E_USER_DATA_DIR',
'ORCA_USER_DATA_PATH'
]) {
vi.stubEnv(name, home)
}
vi.stubEnv('ORCA_BACKGROUND_LAUNCH', '1')
vi.stubEnv('ORCA_DISABLE_MACOS_LOGIN_SHELL', '1')
for (const name of [
'ORCA_ENVIRONMENT',
'ORCA_PAIRING_CODE',
'ORCA_REMOTE_PAIRING',
'ORCA_AGENT_SESSION_ID',
'ORCA_RUNTIME_SOURCE_ID',
'ORCA_RUNTIME_SOURCE_INCARNATION',
'ORCA_RUNTIME_SOURCE_PROFILE_PATH'
]) {
vi.stubEnv(name, '')
}
const { OrcaRuntimeRpcServer } = await import('./runtime-rpc')
const fixture = await import('./orchestration-mailbox-notification-test-harness')
const { registerBoundPty } = await import('./bind-first-pane.test-fixture')
const coord = fixture.TERMINAL_HANDLE
const worker = fixture.SECOND_TERMINAL_HANDLE
const predecessor = 'term_predecessor'
const predecessorTab = '55555555-5555-4555-8555-555555555555'
const predecessorLeaf = '66666666-6666-4666-8666-666666666666'
let db = new OrchestrationDb(join(home, 'orchestration.db'))
function runtimeForDatabase() {
const { runtime } = fixture.createRuntime(db)
fixture.registerSecondPane(runtime)
registerBoundPty(runtime, 'pty-predecessor', fixture.WORKTREE_ID, null, {
tabId: predecessorTab,
leafId: predecessorLeaf,
incarnationId: 'predecessor-incarnation'
})
runtime.registerPreAllocatedHandleForPty('pty-predecessor', predecessor)
vi.spyOn(runtime, 'getNestedWorkerMaxDepth').mockReturnValue(2)
return runtime
}
let runtime = runtimeForDatabase()
let server = new OrcaRuntimeRpcServer({ runtime, userDataPath: home })
async function cli(handle: string, args: string[], exitCode = 0): Promise<unknown> {
const child = spawn(
process.execPath,
[
'--import',
tsxLoader,
'--conditions=orca-source',
cliPath,
'orchestration',
...args,
'--json'
],
{
cwd: home,
env: {
...process.env,
ORCA_TERMINAL_HANDLE: handle,
ORCA_PANE_KEY: runtime.getTerminalPaneKey(handle) ?? '',
ORCA_AGENT_LAUNCH_TOKEN:
handle === coord
? fixture.LAUNCH_TOKEN
: handle === worker
? fixture.SECOND_LAUNCH_TOKEN
: ''
},
stdio: ['ignore', 'pipe', 'pipe'],
windowsHide: true,
timeout: 15_000
}
)
let stdout = ''
let stderr = ''
child.stdout.setEncoding('utf8').on('data', (chunk: string) => {
stdout += chunk
})
child.stderr.setEncoding('utf8').on('data', (chunk: string) => {
stderr += chunk
})
const code = await new Promise<number | null>((resolveExit, rejectExit) => {
child.once('close', resolveExit)
child.once('error', rejectExit)
})
expect(code, JSON.stringify({ args, stdout, stderr })).toBe(exitCode)
const response: unknown = JSON.parse(stdout)
expect(response).toMatchObject({ ok: exitCode === 0 })
return response
}
try {
await server.start()
const root = idOf(
await cli(coord, ['run-create', '--objective', 'Retained parent work']),
'run'
)
const task = idOf(await cli(coord, ['task-create', '--spec', 'Finish after restart']), 'task')
const dispatch = idOf(
await cli(coord, ['dispatch', '--task', task, '--to', worker]),
'dispatch'
)
const separate = idOf(
await cli(predecessor, ['run-create', '--objective', 'Predecessor work']),
'run'
)
const continuation = idOf(
await cli(predecessor, ['task-create', '--spec', 'Continue predecessor work']),
'task'
)
const retainedMessage = idOf(
await cli(worker, ['send', '--subject', 'Retain this report']),
'message'
)
const retainedBatch = check.parse(await cli(coord, ['check'])).result
expect(retainedBatch).toMatchObject({ runId: root, messages: [{ id: retainedMessage }] })
await server.stop()
db.close()
db = new OrchestrationDb(join(home, 'orchestration.db'))
runtime = runtimeForDatabase()
server = new OrcaRuntimeRpcServer({ runtime, userDataPath: home })
await server.start()
expect(await cli(coord, ['run-current'])).toMatchObject({ result: { run: { id: root } } })
expect(await cli(coord, ['task-list'])).toMatchObject({
result: { tasks: [{ id: task, status: 'dispatched', dispatch_id: dispatch }] }
})
expect(check.parse(await cli(coord, ['check'])).result).toEqual(retainedBatch)
const bindingBefore = db.getRunRaw(root)
expect(await cli(worker, ['run-use', '--id', root], 1)).toMatchObject({
error: { code: 'worker_cannot_coordinate_own_run', data: { effectsApplied: false } }
})
expect(db.getRunRaw(root)).toEqual(bindingBefore)
expect(check.parse(await cli(coord, ['check'])).result).toEqual(retainedBatch)
expect(await cli(coord, ['check', '--ack', retainedBatch.deliveryId])).toMatchObject({
result: { acknowledged: retainedBatch.deliveryId, count: 0 }
})
expect(await cli(worker, ['run-use', '--id', separate])).toMatchObject({
result: { run: { id: separate } }
})
const question = idOf(
await cli(worker, [
'send',
'--to',
`run:${root}`,
'--type',
'question',
'--subject',
'Proceed with predecessor work?'
]),
'message'
)
const questionBatch = check.parse(await cli(coord, ['check'])).result
expect(questionBatch).toMatchObject({
runId: root,
messages: [{ id: question, type: 'question' }]
})
const reply = idOf(
await cli(coord, ['reply', '--id', question, '--body', 'Proceed with the retained task']),
'message'
)
const replyBatch = check.parse(await cli(worker, ['check'])).result
expect(replyBatch).toMatchObject({
runId: separate,
messages: [
{
id: reply,
thread_id: question,
to_handle: `run:${separate}`,
run_id: separate,
body: 'Proceed with the retained task'
}
]
})
expect(db.getMessageById(reply)?.read).toBe(0)
expect(await cli(worker, ['check', '--ack', replyBatch.deliveryId])).toMatchObject({
result: { runId: separate, acknowledged: replyBatch.deliveryId, count: 0 }
})
expect(db.getMessageById(reply)?.read).toBe(1)
await cli(coord, ['check', '--ack', questionBatch.deliveryId])
const continuedDispatch = idOf(
await cli(worker, ['dispatch', '--task', continuation, '--to', predecessor]),
'dispatch'
)
expect(
await cli(predecessor, [
'send',
'--type',
'worker_done',
'--subject',
'Predecessor work complete',
'--payload',
JSON.stringify({
taskId: continuation,
dispatchId: continuedDispatch,
outcome: 'succeeded'
})
])
).toMatchObject({ result: { lifecycle: { action: 'completed' } } })
expect(await cli(worker, ['task-list'])).toMatchObject({
result: { tasks: [{ id: continuation, status: 'completed' }] }
})
expect(await cli(worker, ['dispatch-show', '--task', continuation])).toMatchObject({
result: { dispatch: { id: continuedDispatch, status: 'completed' } }
})
expect(
await cli(worker, [
'send',
'--type',
'worker_done',
'--subject',
'Parent work complete',
'--payload',
JSON.stringify({ taskId: task, dispatchId: dispatch, outcome: 'succeeded' })
])
).toMatchObject({ result: { lifecycle: { action: 'completed' } } })
expect(await cli(coord, ['task-list'])).toMatchObject({
result: { tasks: [{ id: task, status: 'completed' }] }
})
expect(await cli(coord, ['dispatch-show', '--task', task])).toMatchObject({
result: { dispatch: { id: dispatch, status: 'completed' } }
})
} finally {
await server.stop()
db.close()
rmSync(home, { recursive: true, force: true })
}
}, 120_000)
@@ -281,7 +281,8 @@ describe('orchestration notification mailbox consistency', () => {
.map(([sql]) => sql)
.find((sql) => sql.includes('FROM run_coordinator_handles AS coordinator'))
expect(routingSql).toContain('INDEXED BY idx_messages_undelivered_direct_run')
expect(routingSql).not.toContain('NOT EXISTS')
// An anti-join per coordinator row is fine; one per candidate message would scan the backlog.
expect(routingSql).not.toMatch(/NOT EXISTS \([^)]*candidate\./)
expect(db.getMessageById(owned.id)?.to_handle).toBe(`run:${ownedRun.id}`)
expect(db.getUnreadDirectMessageTypes(detachedHandle)).toEqual(['status'])
db.close()
@@ -54,7 +54,7 @@ export function recordedCreatorIdentity(creator: DispatchCreator): {
}
/** Self-dispatch is bookkeeping; pre-v37 rows lack creator identity and conservatively count as parents. */
function isSelfCreatedDispatch(row: DispatchContextRow): boolean {
export function isSelfCreatedDispatch(row: DispatchContextRow): boolean {
if (row.creator_pane_key && row.assignee_pane_key) {
return isEquivalentPaneKey(row.creator_pane_key, row.assignee_pane_key)
}
@@ -3,6 +3,7 @@ import type { DispatchContextRow, MessageType } from '../../types'
import { DISPATCH_PANE_KEY_MATCH_SUFFIX_SQL, paneKeyMatchSuffix } from '../pane-key-match'
import type { OrchestrationDb } from '../orchestration-db'
import { DISPATCH_CONTEXT_COLUMN_LIST } from '../row-column-lists'
import { workerSeatGrabSql } from '../runs/run-coordinator-mail-routing'
import {
ORCHESTRATION_DELIVERY_BATCH_LIMIT,
type ForeignDirectMailboxRoutingPage
@@ -58,6 +59,7 @@ export function routeForeignDirectMessagesToOwnedMailboxes(
ON candidate.run_id = coordinator.run_id
AND candidate.to_handle = coordinator.terminal_handle
WHERE coordinator.terminal_handle = ?${runExclusion}
AND NOT ${workerSeatGrabSql('coordinator')}
AND candidate.read = 0 AND candidate.delivered_at IS NULL
AND candidate.delivery_contract = 'current_delivery'`,
`SELECT candidate.id, candidate.run_id, candidate.type, candidate.sequence
@@ -4,6 +4,7 @@ import { LEGACY_CONTRACT_VERSION } from '../contract-constants'
import { isEquivalentPaneKey } from '../pane-key-match'
import type { OrchestrationDb } from '../orchestration-db'
import type { OrcaSessionId } from '../../../../../shared/orca-session-address'
import { findCallerWorkerDispatches, workerRunBindingRefusal } from './worker-run-binding'
import {
mailboxAddressOf,
runBoundToCoordinator,
@@ -41,6 +42,17 @@ export function bindRun(
return undefined
}
const sameBinding = runBoundToCoordinator(run, coordinator)
const explicitTakeover = params.takeoverLegacy === true
// Why: before the legacy gate, so a worker never sees its --takeover-legacy recovery command.
const workerRefusal = sameBinding
? null
: workerRunBindingRefusal(this, coordinator, params.runId, {
takeover: explicitTakeover,
bind: true
})
if (workerRefusal) {
throw workerRefusal
}
const adoption = this.getLegacyAdoption()
const adoptedRun = adoption?.adopted_run_id === params.runId
const legacyAuthority = params.legacyCoordinatorAuthority
@@ -111,6 +123,13 @@ export function bindRun(
}
// Only live legacy work requires takeover attestation; settled work has no competing authority.
if (activeLegacyAssignment && !sameBinding && !provenLegacyBinding && !params.takeoverLegacy) {
if (findCallerWorkerDispatches(this, coordinator).length > 0) {
throw new OrchestrationError(
'consumer_fenced',
`Run ${params.runId} still has active legacy work and this caller cannot bind it. Nothing changed.`,
{ effectsApplied: false }
)
}
throw new OrchestrationError(
'consumer_fenced',
'This adopted Run still has live legacy work. Its attested coordinator may rebind it, or a current coordinator may explicitly use run-use --takeover-legacy.',
@@ -1,17 +1,48 @@
import type Database from '../../../../sqlite/sync-database'
import type { OrchestrationDb } from '../orchestration-db'
import { currentRunCoordinatorSessionAddressSql } from './run-coordinator-orca-session'
import { orcaSessionAddressSql } from '../orca-session-address-sql'
/** Mail to an active Dispatch assignee's address in the same Run is that worker's, not coordinator mail. */
export function activeDispatchOwnsAddressSql(runIdSql: string, addressSql: string): string {
return `EXISTS (
SELECT 1 FROM dispatch_contexts
WHERE dispatch_contexts.run_id = ${runIdSql}
AND dispatch_contexts.assignee_handle = ${addressSql}
AND (dispatch_contexts.assignee_handle = ${addressSql}
OR ${orcaSessionAddressSql('dispatch_contexts.assignee_orca_session_id')} = ${addressSql})
AND dispatch_contexts.status IN ('pending', 'dispatched')
)`
}
/**
* A remembered address first bound while it worked an unsettled, delegated Dispatch in the same Run
* was a worker grabbing the seat, not a spelling of the coordinator, so it never routes as one.
* Derived from the rows, so addresses older builds remembered stop mattering without editing them.
* julianday() because completed_at is stored in two text formats; whole seconds and a strict >
* because first_bound_at has no milliseconds, so a bind in the second a Dispatch settled is not a grab.
*/
export function workerSeatGrabSql(aliasRow: string): string {
return `EXISTS (
SELECT 1 FROM dispatch_contexts AS grab
WHERE grab.run_id = ${aliasRow}.run_id
AND (grab.assignee_handle = ${aliasRow}.terminal_handle
OR ${orcaSessionAddressSql('grab.assignee_orca_session_id')} = ${aliasRow}.terminal_handle)
AND NOT (grab.creator_handle IS NOT NULL AND grab.creator_handle = grab.assignee_handle)
AND julianday(grab.created_at) <= julianday(${aliasRow}.first_bound_at)
AND (grab.status IN ('pending', 'dispatched')
OR julianday(datetime(grab.completed_at)) > julianday(${aliasRow}.first_bound_at))
)`
}
/** Whether `addressSql` is a coordinator address of the Run `runIdSql`, seat grabs excluded. */
function coordinatorAliasSql(runIdSql: string, addressSql: string): string {
return `EXISTS (
SELECT 1 FROM run_coordinator_handles AS alias
WHERE alias.run_id = ${runIdSql} AND alias.terminal_handle = ${addressSql}
AND NOT ${workerSeatGrabSql('alias')}
)`
}
export function rememberRunCoordinatorHandle(
this: OrchestrationDb,
runId: string,
@@ -87,10 +118,7 @@ export function createCoordinatorMailRoutingTrigger(this: OrchestrationDb): void
SELECT 1 FROM runs
WHERE runs.id = NEW.run_id AND runs.legacy = 0
)
AND EXISTS (
SELECT 1 FROM run_coordinator_handles
WHERE run_id = NEW.run_id AND terminal_handle = NEW.to_handle
)
AND ${coordinatorAliasSql('NEW.run_id', 'NEW.to_handle')}
AND NOT ${activeDispatchOwnsAddressSql('NEW.run_id', 'NEW.to_handle')}
BEGIN
UPDATE messages SET to_handle = 'run:' || NEW.run_id WHERE sequence = NEW.sequence;
@@ -113,7 +141,12 @@ export function routeAllUnreadDirectMessagesToRunMailbox(
`UPDATE messages SET to_handle = ?
WHERE run_id = ? AND to_handle = ? AND read = 0
AND delivery_contract = 'current_delivery'
AND NOT ${activeDispatchOwnsAddressSql('messages.run_id', 'messages.to_handle')}`
AND NOT ${activeDispatchOwnsAddressSql('messages.run_id', 'messages.to_handle')}
AND NOT EXISTS (
SELECT 1 FROM run_coordinator_handles AS alias
WHERE alias.run_id = messages.run_id AND alias.terminal_handle = messages.to_handle
AND ${workerSeatGrabSql('alias')}
)`
)
.run(`run:${runId}`, runId, directHandle)
}
@@ -12,7 +12,7 @@ export function requireCurrentConsumer(
if (!run || run.legacy === 1 || run.consumer_generation !== consumerGeneration) {
throw new OrchestrationError(
'consumer_fenced',
'This mailbox consumer has been replaced. Rebind with orchestration run-use.'
`Another session now coordinates Run ${runId}; stop reading this mailbox.`
)
}
return run
@@ -16,6 +16,7 @@ import { encodeRunListCursor, decodeRunListCursor } from '../run-list-cursor'
import type { RunListPage } from '../run-list-page'
import type { OrchestrationDb } from '../orchestration-db'
import { RUN_COLUMN_LIST } from '../row-column-lists'
import { workerSeatGrabSql } from './run-coordinator-mail-routing'
export type LegacyAdoptedMailboxOwner = {
runId: string
@@ -57,17 +58,20 @@ export function getRunMailboxOwnerIdsForHandle(
terminalHandle: string,
legacyAdoptedMailboxOwner?: LegacyAdoptedMailboxOwner | null
): string[] {
const runIds = (
this.db
.prepare(
`SELECT coordinator.run_id
FROM run_coordinator_handles AS coordinator
JOIN runs ON runs.id = coordinator.run_id AND runs.legacy = 0
WHERE coordinator.terminal_handle = ?
ORDER BY coordinator.run_id`
)
.all(terminalHandle) as { run_id: string }[]
).map((row) => row.run_id)
const runIds = this.db
.prepare(
`SELECT coordinator.run_id
FROM run_coordinator_handles AS coordinator
JOIN runs ON runs.id = coordinator.run_id AND runs.legacy = 0
WHERE coordinator.terminal_handle = ? AND NOT ${workerSeatGrabSql('coordinator')}
ORDER BY coordinator.run_id`
)
.all(terminalHandle)
.flatMap((row) =>
typeof row === 'object' && row !== null && 'run_id' in row && typeof row.run_id === 'string'
? [row.run_id]
: []
)
const adoptedOwner =
legacyAdoptedMailboxOwner === undefined
? this.getLegacyAdoptedRunMailboxOwner()
@@ -0,0 +1,78 @@
import type { DispatchContextRow } from '../../types'
import { OrchestrationError } from '../../orchestration-error'
import { isSelfCreatedDispatch } from '../dispatch-depth'
import type { OrchestrationDb } from '../orchestration-db'
import {
mailboxAddressOf,
type OrchestrationCoordinatorKey
} from '../../orchestration-caller-identity'
/** The caller's active delegated Dispatches, matched by address then pane as check and send do. */
export function findCallerWorkerDispatches(
db: OrchestrationDb,
caller: OrchestrationCoordinatorKey
): DispatchContextRow[] {
const address = mailboxAddressOf(caller)
if (address === null) {
return []
}
return db
.getActiveDispatchMailboxOwners(address, caller.paneKey ?? undefined)
.filter((dispatch) => !isSelfCreatedDispatch(dispatch))
}
/** A delegated Dispatch in this Run, or started from work in this Run. */
function callerDispatchWorkingForRun(
dispatches: readonly DispatchContextRow[],
db: OrchestrationDb,
runId: string
): DispatchContextRow | undefined {
for (const own of dispatches) {
let row: DispatchContextRow | undefined = own
const visited = new Set<string>()
while (row) {
if (row.run_id === runId || visited.has(row.id)) {
return own
}
visited.add(row.id)
row = row.creator_dispatch_id ? db.getDispatchContextById(row.creator_dispatch_id) : undefined
}
}
return undefined
}
/** A worker cannot bind its own or ancestor Run, or use a takeover flag on another Run. */
export function workerRunBindingRefusal(
db: OrchestrationDb,
caller: OrchestrationCoordinatorKey,
runId: string,
attempted: { takeover: boolean; bind: boolean }
): OrchestrationError | null {
const dispatches = findCallerWorkerDispatches(db, caller)
const own = callerDispatchWorkingForRun(dispatches, db, runId)
if (own) {
const role =
own.run_id === runId
? `You work Dispatch ${own.id} in Run ${runId}, so you cannot coordinate that Run.`
: `You work Dispatch ${own.id} in Run ${own.run_id}, which was started from work in Run ${runId}, so you cannot coordinate Run ${runId}.`
return new OrchestrationError(
'worker_cannot_coordinate_own_run',
`${role} Your mail arrives through plain \`orchestration check\` (no --run); report with \`orchestration send\`. No effects were applied.`,
{ effectsApplied: false, dispatchId: own.id }
)
}
const first = dispatches[0]
if (first && attempted.bind && attempted.takeover) {
return new OrchestrationError(
'worker_cannot_take_over_run',
'A caller working an active Dispatch cannot use a takeover flag. No effects were applied.',
{ effectsApplied: false, dispatchId: first.id }
)
}
return null
}
/** What a worker with no Run binding is told instead of a Run binding command. */
export function unboundWorkerMessage(dispatch: DispatchContextRow): string {
return `You work Dispatch ${dispatch.id} in Run ${dispatch.run_id} and are not bound to a Run. Your mail arrives through plain \`orchestration check\` (no --run); report with \`orchestration send\`. No effects were applied.`
}
@@ -39,6 +39,8 @@ describe('OrchestrationDb v36 to v37 migration', () => {
seed.close()
const raw = new Database(dbPath)
// The current mail-routing trigger reads creator_handle; the next open recreates it.
raw.exec('DROP TRIGGER trg_messages_route_coordinator_mail')
for (const column of CREATOR_COLUMNS) {
raw.exec(`ALTER TABLE dispatch_contexts DROP COLUMN ${column}`)
}
@@ -124,6 +124,62 @@ describe('adopted Run binding without --takeover-legacy', () => {
})
})
it('withholds legacy takeover advice and refuses explicit worker takeovers without effects', () => {
const { db, adoptedRunId, recoveryMessageId } = createAdoptedFixture({ settleWork: false })
const home = db.createRun({
objective: 'worker home',
coordinatorHandle: CURRENT_COORDINATOR_HANDLE,
coordinatorPaneKey: CURRENT_COORDINATOR_PANE
})
const worker = createRootDispatch(
db,
db.createTask({ runId: home.id, spec: 'worker' }).id,
'term_foreign_worker',
'tab_foreign:22222222-2222-4222-8222-222222222222'
)
const before = {
adoptedRun: db.getRun(adoptedRunId),
homeRun: db.getRun(home.id),
dispatch: db.getDispatchContextById(worker.id),
mail: db.getMessageById(recoveryMessageId)
}
try {
db.bindRun({
runId: adoptedRunId,
coordinatorHandle: 'term_foreign_worker',
coordinatorPaneKey: 'tab_foreign:22222222-2222-4222-8222-222222222222'
})
throw new Error('Expected the legacy bind to be fenced')
} catch (error) {
expect(error).toMatchObject({ code: 'consumer_fenced', data: { effectsApplied: false } })
expect(JSON.stringify(error)).not.toContain('run-use')
}
for (const [runId, code] of [
[adoptedRunId, 'worker_cannot_take_over_run'],
[home.id, 'worker_cannot_coordinate_own_run']
] as const) {
expect(() =>
db.bindRun({
runId,
coordinatorHandle: 'term_foreign_worker',
coordinatorPaneKey: 'tab_foreign:22222222-2222-4222-8222-222222222222',
takeoverLegacy: true
})
).toThrowError(
expect.objectContaining({
code,
data: { effectsApplied: false, dispatchId: worker.id }
})
)
expect({
adoptedRun: db.getRun(adoptedRunId),
homeRun: db.getRun(home.id),
dispatch: db.getDispatchContextById(worker.id),
mail: db.getMessageById(recoveryMessageId)
}).toEqual(before)
}
})
it('claims the settled adopted Run and inherits its recovered task graph', () => {
const { db, adoptedRunId } = createAdoptedFixture({ settleWork: true })
@@ -4,6 +4,7 @@ import { join } from 'node:path'
import { afterEach, describe, expect, it } from 'vitest'
import { LEGACY_RUN_ID, OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
import type { DispatchCreator } from './db/dispatch-depth'
describe('OrchestrationDb Run state', () => {
let db: OrchestrationDb | undefined
@@ -17,11 +18,13 @@ describe('OrchestrationDb Run state', () => {
return db
}
const COORD_PANE = 'tab_coord:11111111-1111-4111-8111-111111111111'
function createBoundRun(d: OrchestrationDb) {
return d.createRun({
objective: 'Mailbox test',
coordinatorHandle: 'term_coord',
coordinatorPaneKey: 'tab_coord:11111111-1111-4111-8111-111111111111'
coordinatorPaneKey: COORD_PANE
})
}
@@ -223,6 +226,49 @@ describe('OrchestrationDb Run state', () => {
})
describe('lightweight Run scope', () => {
it('refuses a worker binding an ancestor Run, but not a coordinator with a self-dispatch', () => {
const d = createDb()
const parent = createBoundRun(d)
const coordinator = { kind: 'terminal', handle: 'term_coord', paneKey: COORD_PANE } as const
const dispatchIn = (runId: string, handle: string, creator: DispatchCreator) =>
d.createDispatchContext({
taskId: d.createTask({ spec: 'work', runId }).id,
assigneeHandle: handle,
creator,
maxDepth: Number.MAX_SAFE_INTEGER
})
// A context-only self-dispatch is bookkeeping, so its coordinator keeps its Run.
dispatchIn(parent.id, 'term_coord', coordinator)
expect(
d.bindRun({
runId: parent.id,
coordinatorHandle: 'term_coord',
coordinatorPaneKey: COORD_PANE
})
).toBeDefined()
dispatchIn(parent.id, 'term_worker', coordinator)
const sub = d.createRun({
objective: 'sub',
coordinatorHandle: 'term_worker',
coordinatorPaneKey: null
})
dispatchIn(sub.id, 'term_grand', { kind: 'terminal', handle: 'term_worker' })
expect(() =>
d.bindRun({ runId: parent.id, coordinatorHandle: 'term_grand', coordinatorPaneKey: null })
).toThrow(expect.objectContaining({ code: 'worker_cannot_coordinate_own_run' }))
let creatorHandle = 'term_grand'
for (let depth = 0; depth < 33; depth++) {
const handle = `term_deep_${depth}`
dispatchIn(sub.id, handle, { kind: 'terminal', handle: creatorHandle })
creatorHandle = handle
}
expect(() =>
d.bindRun({ runId: parent.id, coordinatorHandle: creatorHandle, coordinatorPaneKey: null })
).toThrow(expect.objectContaining({ code: 'worker_cannot_coordinate_own_run' }))
expect(d.getRun(parent.id)?.coordinator_handle).toBe('term_coord')
})
it('binds creation to one pane and fences that pane when it creates another Run', () => {
const d = createDb()
const first = d.createRun({
@@ -0,0 +1,265 @@
import { DatabaseSync } from 'node:sqlite'
import { afterEach, describe, expect, it, vi } from 'vitest'
import { OrchestrationDb } from './db'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
import { formatOrcaSessionAddress } from '../../../shared/orca-session-address'
import { testOrcaSessionId } from '../../../shared/orca-session-address-test-fixture'
const databases: OrchestrationDb[] = []
afterEach(() => {
vi.restoreAllMocks()
for (const db of databases.splice(0)) {
db.close()
}
})
function openDatabase(): OrchestrationDb {
const db = new OrchestrationDb(':memory:')
databases.push(db)
return db
}
// Count real SQLite date evaluations, forwarding to SQLite's unchanged implementation on a second connection.
function historyWork(db: OrchestrationDb) {
const adapter: unknown = db.db
if (
typeof adapter !== 'object' ||
adapter === null ||
!('db' in adapter) ||
!(adapter.db instanceof DatabaseSync)
) {
throw new Error('Expected node:sqlite database')
}
const native = adapter.db
const oracle = new DatabaseSync(':memory:')
const date = oracle.prepare('SELECT julianday(?) AS value')
let evaluations = 0
native.function('julianday', { deterministic: true }, (value) => {
evaluations++
return date.get(value)?.value ?? null
})
return {
reset: () => {
evaluations = 0
},
count: () => evaluations,
close: () => oracle.close()
}
}
describe('Run binding scaling contracts', () => {
it('walks to the root with one indexed ancestor read per link at N and 10N', () => {
for (const depth of [4, 40]) {
const db = openDatabase()
const root = db.createRun({
objective: 'root',
coordinatorHandle: 'root',
coordinatorPaneKey: null
})
const child = db.createRun({
objective: 'child',
coordinatorHandle: null,
coordinatorPaneKey: null
})
let parent = createRootDispatch(
db,
db.createTask({ spec: 'root work', runId: root.id }).id,
'worker_0'
)
for (let link = 1; link <= depth; link++) {
parent = db.createDispatchContext({
taskId: db.createTask({ spec: 'child work', runId: child.id }).id,
assigneeHandle: `worker_${link}`,
creator: { kind: 'terminal', handle: `worker_${link - 1}` },
maxDepth: Number.MAX_SAFE_INTEGER
})
}
const separate = db.createRun({
objective: 'handoff',
coordinatorHandle: null,
coordinatorPaneKey: null
})
const before = db.getRunRaw(root.id)
const reads: string[] = []
const observed = new WeakSet<object>()
const originalPrepare = db.db.prepare.bind(db.db)
const prepare = vi.spyOn(db.db, 'prepare').mockImplementation((sql) => {
const statement = originalPrepare(sql)
if (/FROM dispatch_contexts WHERE id = \?/.test(sql) && !observed.has(statement)) {
observed.add(statement)
const get = statement.get.bind(statement)
vi.spyOn(statement, 'get').mockImplementation((...params) => {
reads.push(sql)
return get(...params)
})
}
return statement
})
const bind = (runId: string) =>
db.bindRun({
runId,
coordinatorHandle: parent.assignee_handle,
coordinatorPaneKey: null
})
const assertWalk = () => {
expect(reads).toHaveLength(depth)
expect(prepare.mock.calls.length).toBeLessThanOrEqual(depth + 50)
const sql = reads[0]
expect(sql).toBeDefined()
expect(db.db.prepare(`EXPLAIN QUERY PLAN ${sql}`).all('absent')).toContainEqual(
expect.objectContaining({
detail: expect.stringMatching(/SEARCH dispatch_contexts USING INDEX .*\(id=\?\)/)
})
)
}
expect(() => bind(root.id)).toThrow(
expect.objectContaining({ code: 'worker_cannot_coordinate_own_run' })
)
assertWalk()
expect(db.getRunRaw(root.id)).toEqual(before)
prepare.mockClear()
reads.length = 0
expect(bind(separate.id)?.coordinator_handle).toBe(parent.assignee_handle)
assertWalk()
prepare.mockRestore()
}
})
it.each([
[4, 2],
[40, 2],
[4, 20]
])(
'bounds four real alias consumers with %i historical Dispatches and %i unread messages per alias',
(history, messages) => {
const db = openDatabase()
const run = db.createRun({
objective: 'historical aliases',
coordinatorHandle: 'coord',
coordinatorPaneKey: null
})
const other = db.createRun({
objective: 'unrelated history',
coordinatorHandle: null,
coordinatorPaneKey: null
})
const task = db.createTask({ spec: 'retained work', runId: run.id })
const otherTask = db.createTask({ spec: 'unrelated retained work', runId: other.id })
const sessionId = testOrcaSessionId('11111111-1111-4111-8111-111111111111')
const aliases = ['old_coord', 'old_worker', formatOrcaSessionAddress(sessionId)]
const insert = db.db.prepare(`INSERT INTO dispatch_contexts
(id, task_id, run_id, assignee_handle, assignee_orca_session_id, creator_handle, status, created_at, completed_at)
VALUES (?, ?, ?, ?, ?, 'coord', 'completed', '2000-01-01', ?)`)
for (const [index, alias] of aliases.entries()) {
db.rememberRunCoordinatorHandle(run.id, alias)
for (let row = 0; row < history; row++) {
insert.run(
`history_${index}_${row}`,
task.id,
run.id,
index === 0 ? alias : 'old_worker',
index === 0 ? null : sessionId,
index > 0 && row === history - 1 ? '2000-01-04' : '2000-01-02'
)
}
}
for (let row = 0; row < history * 10; row++) {
insert.run(`unrelated_${row}`, otherTask.id, other.id, 'old_coord', null, '2000-01-04')
}
db.db
.prepare('UPDATE run_coordinator_handles SET first_bound_at = ? WHERE run_id = ?')
.run('2000-01-03', run.id)
const work = historyWork(db)
const prepare = vi.spyOn(db.db, 'prepare')
const ids = new Map<string, string[]>()
const assertWork = (messageMultiplier: number) => {
// Each alias history is visited, but no unrelated Run or quadratic history join is needed.
expect(work.count()).toBeGreaterThanOrEqual(history)
expect(work.count()).toBeLessThanOrEqual(24 * history * messageMultiplier)
expect(prepare.mock.calls.length).toBeLessThanOrEqual(12 * messageMultiplier + 12)
for (const [sql] of prepare.mock.calls.filter(
([query]) =>
query.includes('FROM dispatch_contexts AS grab') && !query.startsWith('EXPLAIN')
)) {
const plan = db.db
.prepare(`EXPLAIN QUERY PLAN ${sql}`)
.all(...Array(sql.match(/\?/g)?.length ?? 0).fill(null))
expect(plan).toContainEqual(
expect.objectContaining({
detail: expect.stringMatching(/SEARCH grab(?: EXISTS)? USING INDEX .*\(run_id=\?\)/)
})
)
expect(plan).not.toContainEqual(
expect.objectContaining({ detail: expect.stringMatching(/SCAN grab/) })
)
}
work.reset()
prepare.mockClear()
}
const restoreDirect = () => {
for (const [alias, messageIds] of ids) {
for (const id of messageIds) {
db.db.prepare('UPDATE messages SET to_handle = ? WHERE id = ?').run(alias, id)
}
}
work.reset()
prepare.mockClear()
}
try {
for (const alias of aliases) {
expect(db.getRunMailboxOwnerIdsForHandle(alias)).toEqual(
alias === 'old_coord' ? [run.id] : []
)
}
assertWork(1)
// Production INSERT fires the actual routing trigger, including both address spellings.
for (const alias of aliases) {
const messageIds: string[] = []
for (let message = 0; message < messages; message++) {
const inserted = db.insertMessage({
from: 'coord',
to: alias,
runId: run.id,
subject: 'direct'
})
expect(inserted.to_handle).toBe(alias === 'old_coord' ? `run:${run.id}` : alias)
messageIds.push(inserted.id)
}
ids.set(alias, messageIds)
}
assertWork(messages)
// Restored pre-upgrade direct mail is swept by binding and foreign-mail reconciliation.
restoreDirect()
for (const alias of aliases) {
db.routeAllUnreadDirectMessagesToRunMailbox(run.id, alias)
}
assertWork(messages)
for (const [alias, messageIds] of ids) {
for (const id of messageIds) {
expect(db.getMessageById(id)?.to_handle).toBe(
alias === 'old_coord' ? `run:${run.id}` : alias
)
}
}
restoreDirect()
for (const alias of aliases) {
expect(db.routeForeignDirectMessagesToOwnedMailboxes(alias).routedCount).toBe(
alias === 'old_coord' ? messages : 0
)
}
assertWork(messages)
for (const [alias, messageIds] of ids) {
for (const id of messageIds) {
expect(db.getMessageById(id)?.to_handle).toBe(
alias === 'old_coord' ? `run:${run.id}` : alias
)
}
}
} finally {
work.close()
}
}
)
})
@@ -5,6 +5,7 @@ import { afterEach, describe, expect, it } from 'vitest'
import Database from '../../sqlite/sync-database'
import { OrchestrationDb } from './db'
import { SCHEMA_VERSION } from './db/contract-constants'
import { createRootDispatch } from './db/root-dispatch-test-fixture'
describe('Run coordinator handle history migration', () => {
let db: OrchestrationDb | undefined
@@ -165,4 +166,36 @@ describe('Run coordinator handle history migration', () => {
expect(db.getMessageById(message.id)?.to_handle).toBe(`run:${run.id}`)
})
it('stops routing an address that grabbed the seat while it worked for the Run', () => {
db = new OrchestrationDb(':memory:')
const run = db.createRun({
objective: 'grabbed',
coordinatorHandle: 'term_coord',
coordinatorPaneKey: 'tab:leaf'
})
const dispatch = createRootDispatch(
db,
db.createTask({ spec: 'w', runId: run.id }).id,
'term_worker'
)
// An older build remembered the worker as a coordinator address when it bound the Run.
db.rememberRunCoordinatorHandle(run.id, 'term_worker')
db.db
.prepare(
`UPDATE dispatch_contexts SET status = 'completed', completed_at = datetime('now', '+1 second') WHERE id = ?`
)
.run(dispatch.id)
const mail = db.insertMessage({
runId: run.id,
from: 'term_coord',
to: 'term_worker',
subject: 'direct'
})
expect(db.getMessageById(mail.id)?.to_handle).toBe('term_worker')
expect(db.getRunMailboxOwnerIdsForHandle('term_worker')).toEqual([])
expect(db.routeForeignDirectMessagesToOwnedMailboxes('term_worker').routedCount).toBe(0)
})
})
+2
View File
@@ -144,6 +144,8 @@ const STRUCTURED_RUNTIME_PASSTHROUGH_CODES: ReadonlySet<string> = new Set([
'worker_prompt_too_large',
'terminal_worktree_mismatch',
'terminal_is_coordinator',
'worker_cannot_coordinate_own_run',
'worker_cannot_take_over_run',
'request_mismatch',
'mutation_ledger_full',
'legacy_read_only',
@@ -195,7 +195,7 @@ describe('orchestration gate Run authorization', () => {
ok: false,
error: {
code: 'consumer_fenced',
message: `This coordinator terminal is bound to ${runA}, not ${strangerRun.id}.`
message: `This caller is bound to Run ${runA}, not Run ${strangerRun.id}. Nothing changed.`
}
})
expect(harness.db.getTask(strangerTask.id)?.status).toBe('ready')
@@ -104,7 +104,7 @@ export const ORCHESTRATION_CHECK_METHODS = [
if (!hasRunBindingKey(caller) && consumingCheck) {
throw new OrchestrationError(
'stable_pane_required',
`Terminal ${handle} has no live pane bound to a Run, so this inbox can never receive Run mail. Rebind this terminal with orchestration run-use, or read the Run mailbox with --run <run_id>.`,
`Terminal ${handle} has no live pane and no active Dispatch, so a consuming check here can never receive mail. Run check from the agent's own live terminal.`,
orchestrationSkillRecoveryData()
)
}
@@ -52,17 +52,22 @@ describe('Run binding during Dispatch checks', () => {
subject
})
}
// A Run this lead created earlier and that lost its binding: the one kind a worker may resume.
function orphanedOwnRun(): string {
const id = state.db.createRun({
objective: 'resume',
coordinatorHandle: 'term_lead',
coordinatorPaneKey: LEAD
}).id
state.db.db
.prepare(
'UPDATE runs SET coordinator_handle = NULL, coordinator_pane_key = NULL WHERE id = ?'
)
.run(id)
return id
}
async function bind(method: 'runCreate' | 'runUse') {
const params =
method === 'runCreate'
? { objective: 'child' }
: {
id: state.db.createRun({
objective: 'adopt',
coordinatorHandle: 'term_other',
coordinatorPaneKey: OTHER
}).id
}
const params = method === 'runCreate' ? { objective: 'child' } : { id: orphanedOwnRun() }
await h.call(`orchestration.${method}`, { from: 'term_lead', ...params }, state.ctx)
const run = state.db.getCurrentRunForPane(LEAD)
if (!run) {
@@ -503,7 +503,9 @@ describe('Run-bound lead mailbox boundaries', () => {
runId: leadRun.id,
messages: [{ subject: 'Run report' }]
})
await expect(check({ run: dispatch.run_id })).rejects.toMatchObject({ code: 'consumer_fenced' })
await expect(check({ run: dispatch.run_id })).rejects.toMatchObject({
code: 'worker_cannot_coordinate_own_run'
})
expect(state.db.getMessageById(pending.id)?.read).toBe(0)
})
@@ -8,10 +8,18 @@ import type {
} from '../../../../orca-runtime'
import {
hasRunBindingKey,
mailboxAddressOf,
runCoordinatorKey,
type OrchestrationCallerIdentity,
type OrchestrationSessionCaller
} from '../../../../orchestration/orchestration-caller-identity'
import { resolveDeclaredCallerParty } from '../../../../orchestration/orchestration-party'
import {
findCallerWorkerDispatches,
unboundWorkerMessage,
workerRunBindingRefusal
} from '../../../../orchestration/db/runs/worker-run-binding'
import type { OrchestrationDb } from '../../../../orchestration/db'
export type RunScopeParams = {
runId?: string
@@ -109,6 +117,33 @@ export function resolveOrchestrationCaller(
return caller
}
const COORDINATOR_RUN_REQUIRED =
'No Run is bound. A coordinator creates one with `orchestration run-create`, or resumes its own with `orchestration run-use --id <run_id>`.'
// Why: a worker is told where its own mail is, never to bind a Run.
function unboundCallerError(
db: OrchestrationDb,
caller: OrchestrationCallerIdentity,
explicit: RunRow | undefined
): OrchestrationError {
if (explicit) {
// Why: consumer_fenced, since a reader that lost this Run mid-check must stop, worker or not.
const holder = mailboxAddressOf(runCoordinatorKey(explicit))
return new OrchestrationError(
'consumer_fenced',
holder
? `This caller is not bound to Run ${explicit.id}, which is bound to ${holder}. Nothing changed.`
: `This caller does not coordinate Run ${explicit.id}, which has no coordinator now. Nothing changed.`
)
}
const worked = findCallerWorkerDispatches(db, caller)[0]
return new OrchestrationError(
'run_required',
worked ? unboundWorkerMessage(worked) : `${COORDINATOR_RUN_REQUIRED} No effects were applied.`,
orchestrationSkillRecoveryData()
)
}
// Why: task and gate mutations must share one Run-binding rule.
export function resolveRunScope(runtime: OrcaRuntimeService, params: RunScopeParams): RunRow {
const db = runtime.getOrchestrationDb()
@@ -123,7 +158,7 @@ export function resolveRunScope(runtime: OrcaRuntimeService, params: RunScopePar
if (!params.callerTerminalHandle) {
throw new OrchestrationError(
'run_required',
'No Run is bound. Use orchestration run-create or run-use first. No effects were applied.',
`${COORDINATOR_RUN_REQUIRED} A worker never binds a Run: its mail arrives through plain \`orchestration check\`, and it reports with \`orchestration send\`. No effects were applied.`,
orchestrationSkillRecoveryData()
)
}
@@ -146,23 +181,21 @@ export function resolveRunScope(runtime: OrcaRuntimeService, params: RunScopePar
)
}
const current = db.getCurrentRunForCoordinator(caller)
// A worker naming its own or ancestor Run needs its Dispatch mailbox, not a binding change.
const workerRefusal =
explicit && current?.id !== explicit.id
? workerRunBindingRefusal(db, caller, explicit.id, { takeover: false, bind: false })
: null
if (workerRefusal) {
throw workerRefusal
}
if (!current) {
if (explicit) {
throw new OrchestrationError(
'consumer_fenced',
`This coordinator terminal is no longer bound to Run ${explicit.id}.`
)
}
throw new OrchestrationError(
'run_required',
'No Run is bound. Use orchestration run-create or run-use first. No effects were applied.',
orchestrationSkillRecoveryData()
)
throw unboundCallerError(db, caller, explicit)
}
if (explicit && current.id !== explicit.id) {
throw new OrchestrationError(
'consumer_fenced',
`This coordinator terminal is bound to ${current.id}, not ${explicit.id}.`
`This caller is bound to Run ${current.id}, not Run ${explicit.id}. Nothing changed.`
)
}
return current
@@ -269,6 +269,50 @@ describe('orchestration.workerAbandon', () => {
)
})
it('refuses self-abandon by a terminal handle, reissued pane, or resolved session before effects', async () => {
const dispatchId = readyWorker()
const pane = 'tab_worker:aaaaaaaa-aaaa-4aaa-8aaa-aaaaaaaaaaaa'
const orcaSessionId = parseOrcaSessionAddress('orca_session_id:worker_1')!
const session: OrchestrationSessionCaller = {
address: 'term_worker',
terminalHandle: 'term_worker',
paneKey: pane,
orcaSessionId,
sessionId: orcaSessionId,
workspaceId: 'wt'
}
const callers: Partial<RpcContext>[] = [
{ orchestrationCompatibilityEvidence: { terminalHandle: 'term_worker', paneKey: pane } },
{ orchestrationCompatibilityEvidence: { terminalHandle: 'term_reminted', paneKey: pane } },
{ orchestrationCaller: session }
]
for (const caller of callers) {
await expect(abandon(dispatchId, caller)).rejects.toMatchObject({
code: 'invalid_argument',
data: { effectsApplied: false, dispatchId }
})
}
expect(db.getDispatchContextById(dispatchId)?.status).toBe('dispatched')
expect(db.getWorkerDispatch(dispatchId)?.state).toBe('ready')
})
it('refuses self-abandon of a context-only Dispatch', async () => {
const task = db.createTask({ runId: 'run_legacy_local', spec: 'context-only worker' })
const dispatch = db.createDispatchContext({
taskId: task.id,
assigneeHandle: 'term_worker',
creator: { kind: 'system' },
maxDepth: 9
})
await expect(
abandon(dispatch.id, {
orchestrationCompatibilityEvidence: { terminalHandle: 'term_worker' }
})
).rejects.toMatchObject({ code: 'invalid_argument', data: { effectsApplied: false } })
expect(db.getDispatchContextById(dispatch.id)?.status).toBe('dispatched')
expect(db.getTask(task.id)?.status).toBe('dispatched')
})
it('reports an already-settled worker as stale and changes nothing', async () => {
const dispatchId = readyWorker()
db.failDispatch(dispatchId, 'tab closed', { workerProcessExited: true })
@@ -201,8 +201,33 @@ export const ORCHESTRATION_WORKER_CONTROL_METHODS = [
name: 'orchestration.workerAbandon',
permission: 'workspace',
params: WorkerDispatchParams,
handler: (params, { runtime, orchestrationCaller }) => {
handler: (params, { runtime, orchestrationCaller, orchestrationCompatibilityEvidence }) => {
const db = runtime.getOrchestrationDb()
// Session identity wins; a terminal-only caller is identified by its launch evidence and pane.
const caller = orchestrationCaller
? { handle: orchestrationCaller.address, paneKey: orchestrationCaller.paneKey }
: orchestrationCompatibilityEvidence?.terminalHandle
? {
handle: orchestrationCompatibilityEvidence.terminalHandle,
paneKey:
runtime.getTerminalPaneKey(orchestrationCompatibilityEvidence.terminalHandle) ??
orchestrationCompatibilityEvidence.paneKey
}
: null
if (
caller &&
db.isDispatchMessageSender({
dispatchId: params.dispatch,
handle: caller.handle,
paneKey: caller.paneKey
})
) {
throw new OrchestrationError(
'invalid_argument',
`You cannot abandon Dispatch ${params.dispatch} while you are working it. Report to its coordinator instead. No effects were applied.`,
{ effectsApplied: false, dispatchId: params.dispatch }
)
}
const abandoned = db.abandonWorkerDispatch(
params.dispatch,
runtime.getRuntimeId(),
@@ -250,7 +250,7 @@ describe('a structured chat coordinates through the same verbs as a terminal', (
ok: false,
error: {
code: 'consumer_fenced',
message: `This coordinator terminal is no longer bound to Run ${runId}.`
message: `This caller is not bound to Run ${runId}, which is bound to ${ADDRESS_Y}. Nothing changed.`
}
})
expect(await as(SESSION_X, 'orchestration.runCurrent', {})).toMatchObject({ run: null })
@@ -600,6 +600,31 @@ describe('a structured worker that names itself by session id', () => {
}
})
it('cannot take or read the Run it works for after making a sub-Run, and keeps its mail', async () => {
// The 10-14 incident: a worker made a sub-Run, `check --run <parent>` said "bound to X, not Y",
// and it ran run-use on the parent, unbinding its coordinator.
const { runId } = dispatchToWorker()
const as = (method: string, params: Row) =>
h.dispatch(orchestrationRequest(method, params, { sessionId: workerSession }))
resultOf(await as('orchestration.runCreate', { objective: 'sub' }))
const read = await as('orchestration.check', { run: runId, peek: true })
expect(read).toMatchObject({ ok: false, error: { code: 'worker_cannot_coordinate_own_run' } })
expect(JSON.stringify(read)).not.toContain('run-use')
expect(resultOf(await as('orchestration.check', { peek: true }))).toMatchObject({
messages: [{ subject: 'instructions' }]
})
expect(await as('orchestration.runUse', { id: runId })).toMatchObject({
ok: false,
error: { code: 'worker_cannot_coordinate_own_run', data: { effectsApplied: false } }
})
expect(h.db.getRunRaw(runId)).toMatchObject({
coordinator_handle: 'term_coord',
consumer_generation: 1
})
})
it.each([
['its handle', handle],
['its session address', ADDRESS_Y]
@@ -31,6 +31,7 @@ function deliveryId(result: Record<string, unknown>): string {
describe('structured lead Run binding through the RPC dispatcher', () => {
let h: SessionCallerHarness
let dispatchId: string
let leadHandle: string
function call(sessionId: string, method: string, params: Record<string, unknown> = {}) {
return h
@@ -41,6 +42,7 @@ describe('structured lead Run binding through the RPC dispatcher', () => {
beforeEach(async () => {
h = createSessionCallerHarness(hostRef)
const handle = mintStructuredWorkerHandle()
leadHandle = handle
const paneKey = mintStructuredWorkerPaneKey(SESSION_Y)
structuredWorkerIdentities.register({
handle,
@@ -61,6 +63,16 @@ describe('structured lead Run binding through the RPC dispatcher', () => {
maxDepth: Number.MAX_SAFE_INTEGER
}).id
})
// A Run this lead created earlier and that lost its binding: the one kind a worker may resume.
function orphanedOwnRun(): string {
const id = h.db.createRun({
objective: 'resume',
coordinatorHandle: leadHandle,
coordinatorPaneKey: null
}).id
h.db.db.prepare('UPDATE runs SET coordinator_handle = NULL WHERE id = ?').run(id)
return id
}
afterEach(() => {
h.runtime.cancelMessageWaiters(`dispatch:${dispatchId}`)
h.close()
@@ -70,16 +82,7 @@ describe('structured lead Run binding through the RPC dispatcher', () => {
it.each(['runCreate', 'runUse'] as const)(
'cancels a parked session Dispatch check on %s',
async (method) => {
const params =
method === 'runCreate'
? { objective: 'child' }
: {
id: h.db.createRun({
objective: 'unbound',
coordinatorHandle: null,
coordinatorPaneKey: null
}).id
}
const params = method === 'runCreate' ? { objective: 'child' } : { id: orphanedOwnRun() }
const wait = vi.spyOn(h.runtime, 'waitForMessage')
const waiting = call(SESSION_Y, 'check', { wait: true, timeoutMs: 500 })
await vi.waitFor(() =>
+1
View File
@@ -51,6 +51,7 @@ export const CLI_BOOLEAN_FLAGS = new Set([
'staged',
'supervise',
'tab',
'takeover-legacy',
'tasks',
'text-stdin',
'unread',